Predicting Human Translation Difficulty Using Automatic Word Alignment
Year: 2023

Abstract:
Translation difficulty arises when translators are required to resolve translation ambiguity from multiple possible translations. Translation difficulty can be measured by recording the diversity of responses provided by human translators and the time taken to provide these responses, but these behavioral measures are costly and do not scale. In this work, we use word alignments computed over large scale bilingual corpora to develop predictors of lexical translation difficulty. We evaluate our approach using behavioural data from translations provided both in and out of context, and report results that improve on a previous embeddingbased approach (Thompson et al., 2020). Our work can therefore contribute to a deeper understanding of cross-lingual differences and of causes of translation difficulty.

heading: 1 Introduction
text: Words can be hard to translate for many reasons including cultural context and differences in semantic subdivisions across languages (Hershcovich et al., 2022; Chaudhary et al., 2021). For instance, the emotional/moral sense of English heart is commonly translated as Malay hati, but in a medical setting heart should be translated as jantung and hati refers to a different bodily organ (the liver). Examples such as this are challenging for language learners and translators because they go beyond simple one-to-one correspondences between source and target words.
Translation difficulty has been studied by researchers from multiple disciplines including psycholinguistics (Degani et al., 2016), computational linguistics (Cotterell et al., 2018), machine translation (Koehn and Knowles, 2017) and translation studies (Carl et al., 2016b). Understanding translation difficulty is an important scientific challenge in its own right, but methods for measuring difficulty
∗Now at Google DeepMind.
can also be applied in a number of ways. First, difficulty measures have previously been used to identify word meanings of cultural significance (Toury, 2021; Thompson et al., 2020). Second, difficulty measures can help develop targeted evaluations for Neural Machine Translation (NMT) systems (Bugliarello et al., 2020; Yin et al., 2021). For example, automatic difficulty ratings allow for the generation of translation samples of varying difficulties and facilitate human evaluation of machine translation. A third potential application is to reweight and calibrate NMT performance across data sets, language pairs and domains based on their varying levels of difficulty – an objective crucial to NMT quality estimation tasks (Fomicheva et al., 2020; Behnke et al., 2022). Finally, in second language learning and human translator training, translation difficulty ratings allow instructors to identify potential challenges to language learners, and to curate translation assignments for translation students of different levels of experience (Sun, 2015; Chaudhary et al., 2021).
In this work, we use surprisal and entropy derived from word alignment to estimate word translation difficulty. Different pairs of aligned words are collected as translation alternatives (e.g., heart-hati and heart-jantung) and used to infer a word’s translation distribution and compute our informationtheoretic difficulty measures. Among previous studies of translation our approach is closest to the work of Chaudhary et al. (2021), as it leverages large-scale human translation data and extracts word-level translations from aligned sentences. Unlike previous studies, however, we are the first to use word alignments to directly address translation difficulty as a psychological aspect of lexical semantics that is measurable by behavioural data. We evaluate our difficulty estimates against translation norms (Tokowicz et al., 2002; Prior et al., 2007; Allen and Conklin, 2014; Bracken et al., 2017; Lee et al., 2022; Tseng et al., 2014) that measure
11590
translation difficulty out of context and translation process features (Carl et al., 2016b) that measure translation difficulty in context. We also compare against a previous approach that uses multilingual embeddings to develop a measure of translation difficulty (Thompson et al., 2020).
Relative to embeddings, we suggest that word alignments better capture lexical and morphological distributions, and hence allow for a better measure of translation difficulty.1 Our measures of translation difficulty are interpretable, and as we show in later sections, help improve the understanding of human language and translation processing.

heading: 2 Related Work
text: Our approach builds on two lines of work from the psycholinguistic literature on translation. One line of work relies on translation norms derived from tasks in which bilingual participants translate single words presented out of context or rate semantic similarity between pairs of words (Tokowicz et al., 2002; Prior et al., 2007; Allen and Conklin, 2014; Bracken et al., 2017; Lee et al., 2022; Tseng et al., 2014). High variation in translation responses to a given word provides evidence of translation ambiguity (Kroll and Tokowicz, 2001; Tokowicz, 2000); whereas perceived degree of cross-lingual semantic overlap informs lexical choice and is predictive of response time (Allen and Conklin, 2013; Van Assche et al., 2009; Dijkstra et al., 2010; Van Hell and De Groot, 1998). A second line of work studies translation in context by measuring reading time and production duration as translators process realistic texts (Carl et al., 2016b). Behavioral approaches like these provide gold-standard measures of translation difficulty but are costly and do not scale.
Within the computational literature, Thompson et al. (2020) and Carl (2021b) derive automatic measures of translation difficulty based on the idea that difficult-to-translate words are hard to align across word embedding spaces. The former use embeddings to compare semantic neighbourhoods of bilingual word pairs and report significant correlations with human semantic similarity judgements. Carl (2021b) learned a cross-lingual embedding projection to estimate word pair similarities, and showed that these estimates predict translation process data. Bugliarello et al. (2020) and Yin
1Code available at https://github.com/ZhengWeiLim/ pred-translation-difficulty.
et al. (2021) probe translation ambiguity from NMT models using cross-mutual information, which is useful for identifying contextual translations in NMT models. Chaudhary et al. (2021) use word alignment distributions to reveal lexical semantic distinctions across languages. Their work shows that word alignments, with properly extracted descriptions, help language learners disambiguate fine-grained lexical distinctions, but does not directly address the general notion of translation difficulty.

heading: 3 Assessing word-level translation difficulty through word alignments
text: Assume that we have a parallel corpus and a word aligner and are interested in the translation distribution of word w from a source language, L1, to a target language, L2. The most natural approach is a count-based distribution, where counts for words aligned with w are normalized by the frequency of w. From here, pal(v|w), the probability of word w being translated to v, can be computed over aligned word pairs. In addition to alignment counts, most word aligners assign a score for each pair of aligned words. Given two parallel sequences, x = [x0, ..., xm] and y = [y0, ..., yn], let xi ↔ yj indicate that the ith token from x is paired with the jth token from y, and let sxi↔yj denote the alignment score.2 This allows a weight-based distribution parallel to the count-based method above. In general, we calculate pal(v|w) by:
pal(v|w) = Sw↔v∑
u∈V Sw↔u . (1)
For the weight-based distribution, Sw↔v represents the sum of alignment scores of all w ↔ v pairings. For the count-based distribution, sxi↔yj = 1, i.e., Sw↔v is the number of times w is aligned with v in the entire corpus.3 The final distribution, pal(v|w), is normalized given the total scores of all possible alignments with w, where V refers to the vocabulary of L2 in the corpus.
The concept of surprisal in psycholinguistics is often associated with cognitive workload, which in translation studies is connected to word translation information (ITra) (Wei, 2022; Carl, 2021a).
2In Dou and Neubig (2021), sxi↔yj is the harmonic mean between p(yj |xi) and p(xi|yj), the probability of xi being aligned to yj over all possible words in y, and vice versa.
3Sw↔v = ∑ x↔y ∑ i,j sxi↔yj1{w,v}(xi, yj)
Translation surprisal is defined as:
Ial(v|w) = − log pal(v|w). (2)
Low surprisal values indicate that v is a stable translation of w, which is expected to require low effort to produce. The translation uncertainty associated with a source word w can be formulated as the entropy (or expected surprisal):
Hal(w) = − ∑
u∈V pal(u|w) log pal(u|w). (3)
Surprisals derived from count-based and weightbased distributions are denoted by Ical and I w al respectively. Likewise, Hcal and H w al will be used as shorthands for their respective entropy values. Word pairs with higher surprisal are expected to be more difficult. Higher translation entropy indicates a greater range of translations for a source word, which is expected to contribute to translation difficulty.

heading: 4 Experiments
text: Dataset and pre-processing We obtain parallel data of English with German (de), Spanish (es), Japanese (ja), Malay (ms), Dutch (nl) and Chinese (zh) from OpenSubtitles (Lison et al., 2018). All sentences are tokenized by the spaCy tokenizer (Honnibal and Montani, 2017), except Malay, for which we use Aksara (Hanifmuti and Alfina, 2020). We choose to preserve word forms in subtitles and evaluation data, because morphological variation, as we see in later sections, partly contributes to translation ambiguity. awesome-align is used to infer word alignments from the tokenized parallel sentences.4 We then calculate surprisal and entropy based on Equations 2 and 3.5
Evaluation. We evaluate our methods against context-free translations compiled in existing norms, which include i) the number of unique translations of a word, and cover Spanish, Japanese, Malay, Dutch and Chinese (to and from English); and ii) semantic similarity ratings of paired words between English and Japanese, Dutch and Chinese (Tokowicz et al., 2002; Prior et al., 2007; Allen and Conklin, 2014; Bracken et al., 2017; Lee et al., 2022; Tseng et al., 2014). Measures of translation in context are derived from CRITT TPR-DB, a behavioural data set extracted from translation
4without --train_co option for consistency optimization. 5Other pre-processing steps are described in Appendix B.
logs collected using key loggers and eye trackers (Carl et al., 2016b). We focus on three such process features:
• Dur specifies the time taken to produce the target token corresponding to a source word.
• Munit describes the number of micro units, which are distinct translation activities marked by pauses of a fixed length. Thus, easier translations correspond to lower values of Munit.
• HTra refers to translation entropy based on manual alignments in TPR-DB.
More details about these three features and about the preprocessing applied are described in Appendix A. We validate against data sets in Japanese (ENJA15, Carl et al., 2016a), German (SG12, Carl et al., 2016b) and Spanish (BML12, Mesa-Lao, 2014), for which information about translation at the token-level is readily available.
Baselines. We compare Ical and I w al with Thompson et al.’s (2020) embedding-based approach, which has been framed explicitly as an account of translation difficulty. Following their work we expand the initial NorthEuraLex translations (Dellert et al., 2020) to include all translation pairs in the evaluation data and recompute word-pair semantic alignments using Common Crawl and Wikipedia fastText embeddings (Grave et al., 2018).
The final values are negated to match the sign of Ial, and denoted here by Semb.6 Thompson et al. (2020) do not provide an embedding-based analog of Hcal and H w al that can be used to estimate
6In Appendix C, we include alternative results computed from OpenSubtitles embedding and translation pairs with additional top 3 aligned translations of the initial vocabulary.
the translation uncertainty associated with a single source word. We therefore compare Hcal and H w al with a simple embedding-based measure Memb defined as the highest value of Semb associated with a source word. We limit all comparisons to the same set of vocabulary and translation pairs.7

heading: 5 Results and Discussion
text: Context-free translations. Table 1 reports the Pearson correlation of all methods given translations to English (→ en) and translations from English (en →). Both Hcal and Hwal achieve moderately high correlations with Spanish and Japanese norms. Hwal is a weight-based entropy, which captures more nuances in its translation distribution than does the count-based approach, and is, in most languages, the most predictive of a source word’s translation difficulty. Table 2 summarizes the correlation of Ical, I w al and Semb of word pairs against semantic similarity ratings.8 The count-based and weight-based entropy measures achieve similar correlations and outperform the embedding-derived measure in 5 out of 6 cases.
Context-dependent translations. Table 3 shows that our corpus-derived entropy measures strongly correlate with entropies based on TPR-DB (HTra), and that Ical and I w al are moderately predictive of Munit. However, Dur correlates weakly but negatively with Ical, I w al and Semb. This finding is surprising — we previously argued that low-surprisal translations and word pairs with high embedding alignment have a larger degree of semantic overlap, which should have contributed to easier transla-
7The vocabulary size, evaluation set and the number of translations in comparison, are reported in Appendix B.
8Unlike alignment distributions, Semb and similarity judgements (except for Dutch norms) are non-directional, resulting in the same values in both directions.
tion and shorter production time. The gap between the embedding and word-alignment approaches for Munit and Dur is also considerably larger than for our previous results. We now offer two partial explanations for these observations.
Lexical and morphological variation. Relative to the embedding-based approach, our wordalignment approach more accurately captures the distribution of lexical choices and morphological variants. Rare and morphologically complex words have long been known to affect NMT modeling difficulty (Belinkov et al., 2017; Cotterell et al., 2018), and have relatively poor representations in both static and contextual embeddings (Bahdanau et al., 2017; Conneau et al., 2017; Schick and Schütze, 2019; Athiwaratkun et al., 2018; Schick and Schütze, 2020; Anastasopoulos and Neubig, 2020). Word embeddings are also typically optimized to minimize the contribution of frequency information (Gong et al., 2018; Mu and Viswanath, 2018; Liang et al., 2021; Spliethöver et al., 2022). Ignoring frequency, however, is problematic for our task because frequency captures information about which translation choices are most typical and natural (Baker, 2018). In our data, varied is more commonly translated to Spanish feminine form, variada, than the masculine form variado. Table 4 suggests that variado took longer to produce because it appears less frequently in parallel text together with varied, as indicated by its surprisal value. Another example that reflects lexical distribution is disliked, where disgustaba is a more popular translation than detestaba. Here Semb fails to distinguish the difference in usage.
Effects of form similarity. Our counterintuitive result for Dur is consistent with previous evidence that cognates are both produced with high probability and associated with relatively long production times.9 Heilmann and Llorca-Bofí (2021) show that the cognate status of a source word increases translation duration (particularly cognateto-cognate translation), due to hesitation and selfmonitoring. Additional evidence that form overlap influences translations is provided by Prior et al. (2011) and Schwartz and Kroll (2006), who found that context helps facilitate non-cognate alternatives to compete in lexical selection. Consistent with these results, we found significant negative correlations between Ical and I w al with cognate rating in Spanish norms (Prior et al., 2007) and Mean Form Sim Rating in Dutch norms (Tokowicz et al., 2002), which shows that cognates are indeed more probable translations.10 For Japanese, we conducted a t-test on surprisals and found a significant difference (p < .001) between borrowings and non-borrowings (Allen and Conklin, 2014).11 Table 4 shows región as a more common but slower translation of region. Unlike variada and diversa, the surprisal differential between variado and diverso is not enough to overcome its cognate effect.

heading: 6 Conclusion
text: We developed predictors of translation difficulty based on word alignment distributions and tested
9Here, we use cognate liberally to include loanwords and words with high form (orthographical and/or phonological) similarity.
10On average, the correlations (p < .001) for Ical are -.213 (es) and -.214 (nl), whereas Iwal are -.214 (es and nl).
11Semb is largely uninfluenced by formal similarity.
them using translation norms and translation processing data. Compared to the embedding-based approach, our measures derived from word alignment do not depend on lexical databases and more accurately capture lexical choice distributions and morphological variation. Our results show improved estimates of translation difficulty, but suggest that a comprehensive account of human translation difficulty must also consider additional factors such as form similarity.

heading: 7 Limitations
text: Although form similarity is demonstrably responsible for slower translation processing, we are unable to ascertain if it is the primary reason. The work also reveals one shortcoming of alignment distributions — the measure tends to be biased towards translations with similar forms and does not always make accurate predictions about cognates. To address this limitation, future work can evaluate more elaborate models of translation that incorporate variables (e.g., form overlap, syntactic complexity, and morphological complexity) identified as relevant by previous empirical work in psycholinguistics.
Ethics Statement
We obtained all data from cited sources. Our experimental procedures and analysis do not involve human participants and are in compliance with ACL Code of Ethics.12

heading: Acknowledgements
text: This work was supported by ARC FT190100200.

heading: A Translation behavioural data
text: We evaluate translation difficulty in context using CRITT TPR-DB, which includes logs for translations of the multiLing corpus (six English source texts) into various languages (Carl et al., 2016b).13 Here we briefly describe all features relevant to translation difficulty.
HTra is similar to Hcal in that these methods quantify the degree of uncertainty in a lexical distribution. Where Hcal measures the entropy of word alignments, HTra does the same for source and target tokens in multiLing translations (Schaeffer et al., 2016). Words with high HTra have less obvious translation choices, which means that the lexical decisions of the translator require more cognitive effort. This measure has been shown to affect total target production duration, First Fixation Duration and Source Token Reading Time (Carl and Schaeffer, 2017; Schaeffer and Carl, 2017; Schaeffer et al., 2016).
Munit refers to the number of micro translation units, which are units of translation activity separated by pauses of a given length, as monitored by a key logger or an eye tracker (Alves and Vale, 2017). This records the number of activities involved in the translation process, where the translator might read, plan, revise, edit or reconsider a previously translated token.
Dur refers to the production duration of a target token given a source token, i.e., the time taken from the first keystroke to last keystroke in producing the relevant token.
Following Heilmann and Llorca-Bofí (2021) and Carl (2021b), we remove all values of Dur smaller than 20ms and log scale all remaining values. Across participants and translation sessions, HTra is averaged by source words, whereas Munit and Dur are averaged by translation pairs.

heading: B Experiment and data specification
text: The pre-processing steps before word alignment include white space cleaning and removal of any sentence pairs containing non-ASCII-decodable characters. After word alignment, we exclude entropy values of words that have been aligned fewer than 20 times, or have frequency lower than 50 in Worldlex (Gimenes and New, 2016).14
13https://sites.google.com/site/ centretranslationinnovation/tpr-db
14http://www.lexique.org/?page_id=250
Table 5 reports the vocabulary size and number of paired words in each measure and evaluation data set. During evaluation, we also limit our comparisons across methods to the same set of vocabulary and translation pairs. The number of comparisons for all result tables in the main text is summarized in Table 6.

heading: C Additional results for Thompson’s embedding-based approach
text: We found the embedding-based method of Thompson et al. (2020) to be highly sensitive to the quality of the input translation pairs — performance degrades with additional word alignment data. Here, we provide results for two alternative measures. M+emb and S + emb are comparable to Memb and Semb in the main text, but incorporate the top 3 word alignments for each word in the initial vocabulary. Another set of measures are M semb and S s emb, which are based on the same translation pairs as Memb/Semb, but are computed with OpenSubtitles embeddings (subs2vec) (Van Paridon and Thompson, 2021).15
Tables 7a and 7b show the results against contextfree translations, which correspond to Tables 1 and 2 in the main text. For context-dependent translations, the correlations with translation process features are reported in Table 8. Note that some values are missing from the tables, because subs2vec embeddings are not available in Japanese and Chinese.

heading: D Terms for use
text: For all relevant data, models and code used in the work, we list licenses permitting research use:
• awesome-align under BSD 3-Clause License
• spaCy tokenizer and subs2vec under MIT License
• Aksara tokenizer under GNU Affero General Public License
• CRITT TPR-DB under CC BY-NC-SA License
• fastText embeddings CC BY-SA 3.0 License
• NorthEuraLex translations under CC BY-SA 4.0 License
We use the code of Thompson et al. (2020) from https://osf.io/tngba/, which can be freely
15https://github.com/jvparidon/subs2vec
used for academic research.16 Lison and Tiedemann (2016) explicitly made OpenSubtitles corpora “freely available to the research community”, whereas translation norms have been created to facilitate multilingual research (Tokowicz et al., 2002; Prior et al., 2007; Lee et al., 2022). The code repository for this project, as referenced from footnote 1, is available under MIT License.
16https://www.nature.com/ nature-portfolio/editorial-policies/ self-archiving-and-license-to-publish# terms-for-use
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix D

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 1,3, Appendix D
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: To address this limitation, future work can evaluate more elaborate models of translation that incorporate variables (e. g. , form overlap, syntactic complexity, and morphological complexity) identified as relevant by previous empirical work in psycholinguistics