Addressing Entity Translation Problem via Translation Difficulty and Context Diversity (2024)

Abstract: Neural machine translation (NMT) systems often produce inadequate translations for named entities. In this study, we conducted preliminary experiments to examine the factors affecting the translation accuracy of named entities, specifically focusing on their translation difficulty and context diversity. Based on our observations, we propose a novel data augmentation strategy to enhance the accuracy of named entity translation. The main concept behind our approach is to increase both the context diversity and translation probability for the targeted named entity pair. To achieve this, we construct additional samples for named entities that exhibit high translation difficulty or low context diversity and use the augmented training data to re-train the final translation model. Furthermore, we propose an entity-aware machine translation metric that prefers the translation output to generate more accurate named entities. Our experimental results demonstrate significant improvements over the baseline in terms of general translation performance and named entity translation accuracy across various test sets, such as WMT news translation and terminology test sets.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 11628–11638 August 11-16, 2024 ©2024 Association for Computational Linguistics
Neural machine translation (NMT) systems often produce inadequate translations for named entities. In this study, we conducted preliminary experiments to examine the factors affecting the translation accuracy of named entities, specifically focusing on their translation difficulty and context diversity. Based on our observations, we propose a novel data augmentation strategy to enhance the accuracy of named entity translation. The main concept behind our approach is to increase both the context diversity and translation probability for the targeted named entity pair. To achieve this, we construct additional samples for named entities that exhibit high translation difficulty or low context diversity and use the augmented training data to re-train the final translation model. Furthermore, we propose an entity-aware machine translation metric that prefers the translation output to generate more accurate named entities. Our experimental results demonstrate significant improvements over the baseline in terms of general translation performance and named entity translation accuracy across various test sets, such as WMT news translation and terminology test sets.

heading: 1 Introduction
text: Neural machine translation (NMT) that leverages the sequence-to-sequence learning paradigm to transform a source sentence into a target sentence has made significant advancements in recent years. However, previous studies have demonstrated that the translation of low-frequency named entities (e.g., person name, location, organization), which plays a central role in improving the customer experience for commercial translation systems, continues to pose a significant challenge for NMT systems (Koehn and Knowles, 2017; Yan et al., 2018).
∗Work done during internship at Tencent AI Lab. †Xing and Yujiu are co-corresponding authors.
Numerous techniques have been proposed to improve the accuracy of named entity translation. These techniques include 1) substituting entity translation in a post-processing step (Luong et al., 2015), 2) utilizing neural translation memory (Wang et al., 2017; Gu et al., 2018; He et al., 2021), 3) employing named entity tag embedding (Ugawa et al., 2018a; Moussallem et al., 2019), 4) applying lexically constrained decoding (Hokamp and Liu, 2017a; Post and Vilar, 2018a; Wang et al., 2022a), and 5) utilizing an interactive translation mechanism (Weng et al., 2019; Xiao et al., 2022). Most of these techniques explicitly decouple the named entity translation from the process of translating the entire sentence.
Along this line, we decompose the sentence into the named entity and its context. We performed preliminary experiments on large-scale in-house data, which allowed us to identify two factors that impact the translation accuracy of named entities:
• translation difficulty: entities comprising highfrequency words are more easily translated than those composed of low-frequency words, since NMT models tend to struggle with translating low-frequency words (Koehn and Knowles, 2017).
• context diversity: diverse and rich context information can prevent overfitting and boost the generalization capability of NMT models for named entities.
Based on the above observations, we propose a novel data augmentation strategy to improve the translation accuracy of named entities. Specifically, given bilingual entity pairs, we first compute the translation difficulty and context diversity scores based on the training corpus. Next, we select the bilingual context from the bilingual sentence pair and combine it with the bilingual entity pair to create synthetic parallel samples. The basic idea is to
11628
generate additional synthetic samples for the entity pair that exhibits high translation difficulty or low context diversity. Consequently, these synthetic parallel samples are used to boost the context diversity and the translation probability associated with the named entity pair. Finally, we use the augmented training data to re-train the final translation model.
We also propose a lexical-based entity-aware machine translation metric to evaluate the quality of translations containing named entities more effectively. The proposed metric assigns a high score for the translation outputs with greater accuracy in named entity translation. We adopt the widely-used BLEU (Post, 2018) and ChrF++ (Popović, 2017) to our entity-aware machine translation metric and find that the proposed entity-aware entity metric correlates better with human judgment than the original metrics.
To evaluate the effectiveness of our data strategy, we conducted extensive machine translation experiments on WMT20 (Barrault et al., 2020) news translation and WMT21 (Alam et al., 2021) terminology translation with general decoding (without lexical constraint). Experimental results demonstrate that the proposed approach outperforms several strong baselines in terms of BLEU and named entity translation accuracy. Furthermore, our strategy can be applied to lexically constrained decoding scenarios. We implement the state-of-the-art template-based approach (Wang et al., 2022a) on top of our strategy and find that our strategy can improve the context translation performance while maintaining the entity translation accuracy.
Our main contributions are:
• We conduct experiments on an in-house test set to identify two key factors influencing the accuracy of named entity translation: translation difficulty and context diversity.
• We propose a novel data augmentation strategy to improve the accuracy of named entity translation. Our approach involves generating synthetic samples for entity pairs with high translation difficulty or low context diversity.
• The strength of our data augmentation method lies in providing theoretically guided augmentation strategies and maintaining a balance between computational cost and performance improvement.
• We propose an entity-aware machine translation metric that prefers the translation output to generate more accurate named entities.

heading: 2 Preliminary Experiment
text: To understand the main characteristics of entity translation in NMT, we performed preliminary experiments on large-scale in-house data 1 to empirically demonstrate that the accuracy of entity translation is strongly correlated to two of its attributes, namely, translation difficulty and contextual diversity.
Notations. Let C = {(xi,yi)}Ni=1 denotes the authentic parallel data, where xi and yi are source and target sentences, and N denotes the total number of sentence pairs. Let E = {(uj ,vj)}Mj=1 denotes the collection of named entity pairs, where M is the number of entity pairs. Furthermore, MX 7→Y refers to a well-trained translation model that trained on C.
Translation Difficulty. Translation difficulty refers to how difficult the level of complexity is associated with translating an entity pair using a translation model without any contextual information. We employ normalized sentence-level probability as the metric for calculating the translation difficulty score to measure this difficulty score.
T (u, v) = 1− pMX 7→Y (v|u) (1)
where the pMX 7→Y (v|u) is the sentence-level probability normalized by the length of target entity phrase v.
A high translation difficulty score indicates that it is hard to generate the entity translation correctly for the translation model MX 7→Y .
Context Diversity. Context diversity refers to the entropy of diverse contexts associated with the entity pair in the training corpus. In accordance with Mikolov et al. (2013), we employ the surrounding words in the source sentences to represent the context and calculate the context diversity score for the entity pair based on the entropy of the context.
C(u, v) = − H∑
h=0
p(ch|u, v) log p(ch|u, v) (2)
1The built in-house data will be publicly available at https://github.com/Skytliang/EntityTranslation.
where p(ch|u, v) is the estimated probability of the context ch of the entity pair (u, v), and is computed as follows,
p(ch|u, v) = count(ch|u, v)∑H h=0 count(ch|u, v)
(3)
where count(ch|u, v) denotes the frequency of the context ch appears around the entity pair (u, v) in training corpus, and H is the number of context.

heading: 2.1 Experimental Setup
text: Data. We conducted experiments on the WMT20 Chinese-to-English translation task. We first utilized TexSmart (Liu et al., 2021)2 to recognize Chinese named entities and used unsupervised neural aligner Mask-Align (Chen et al., 2021a)3 to obtain word alignments. Then, we collect bilingual named entity pairs using the source Chinese entities and word alignment information to build the named entity dictionary D.
To ensure the quality of the named entity dictionary D, we excluded entity pairs that appeared less than five times in the training corpus. We observed that most dictionary entity pairs (83.4%) constituted one-to-one translations, meaning each named entity had only one translation. This preliminary work only considers the entity pair in one-to-one translation mode.
Due to the insufficiency of named entities in the general news translation test set (only 34.8% of the WMT20 Chinese-to-English translation test sentences contained named entities), we constructed a new entity translation test set for the preliminary experiment to evaluate entity translation performance more accurately. Specifically, for each entity pair in the dictionary D, we randomly selected five bilingual sentence pairs containing the entity pair from our in-house data. Entity pairs for which five test samples could not be collected were excluded from the test set.
We show the data statistics of WMT20 and inhouse test set in Table 1. The in-house data offers
2https://ai.tencent.com/ailab/nlp/texsmart/en/ 3https://github.com/THUNLP-MT/Mask-Align
three key advantages: 1) An extensive collection of sentences; 2) A diverse range of entities, ensuring each sentence contains entities; and 3) A comprehensive evaluation system, with each entity being present in five test sentences, facilitates to assess the translation accuracy in various contexts.
Feature Computation. For each entity pair in the dictionary D, we first employed a TransformerBig model (Vaswani et al., 2017) to train the translation model MX 7→Y . Subsequently, this translation model was utilized to calculate the translation difficulty score. Then, we followed Mikolov et al. (2013) to set the context window size to 3 (i.e., concatenating the three preceding tokens and three succeeding tokens to denote the context ch) to calculate the context diversity score.
Entity Translation Accuracy. We use ExactMatch Accuracy (Anastasopoulos et al., 2021) to measure entity translation accuracy. This metric considers only those entity translations that appear in the output translation as correct predictions.

heading: 2.2 Result
text: We computed the translation accuracy for each pair of entities, representing it as a percentage (i.e., 0%,
20%, ..., 100%) and categorized the entities into six groups based on this criterion. For better visualization, we calculated the centroid of each group using Euclidean distance. Figure 1 illustrates the attribute values of the centroid, along with the corresponding percentage of entity pairs for each group.
Our findings indicate that 1) most entity pairs exhibit a high translation accuracy, 85.5% (the right two points in Figure 1, 15.0% + 70.5% = 85.5%) of entity pairs have an accuracy rate higher than 80%. 2) The accuracy of entity translation predominantly depends on both the difficulty of translation and the diversity of the context. Entity pairs with challenging translations and limited contextual variation (represented as sample points in the upper left area of Figure 1) demonstrate lower accuracy.
Our work aims to identify the named entity pairs with low translation accuracy and improve entity translation performance. As shown in Figure 1, our preliminary experiment revealed that entity pairs exhibiting high translation difficulty scores and low context diversity scores generally possess low translation accuracy. In Section 3, we will apply a novel data augmentation technique to improve the translation accuracy of these particular entity pairs.

heading: 3 Methodology
text: In this section, we present a novel data augmentation strategy to improve the accuracy of named entity translation. The translation model training processing of our strategy includes following steps:
(1) Entity-Pair Dictionary Collection We use the named entity toolkit and the unsupervised word alignment toolkit to obtain the bilingual entity pair dictionary. It should be noted that This step can be skipped if an existing entitypair dictionary is available.
(2) Translation Model Training We use the conventional Transformer model to train the translation model, which will be used to compute the normalized sentence-level probability for the named entity pair.
(3) Feature Computation For each entity pair in the entity-pair dictionary, we use the translation model to compute the difficult score as described in Eq.(1) and we follow Mikolov et al. (2013) to set the context window size to 3 to calculate the context diversity score as described in Eq.(2).
Name #Training #Test
News Zh-to-En 20,000,000 2,000 Terminology En-to-Fr 15,523,986 2,100 Terminology En-to-Zh 19,637,866 2,100

heading: 4 Experiment

heading: 4.1 Experimental Setup
text: Data. We conducted experiments on Chinese-toEnglish translation and used WMT20 news translation test sets to evaluate the proposed approach. Additionally, we utilized the WMT21 terminology translation (Alam et al., 2021) for targeted evaluation. We follow the approach outlined by Wang et al. (2022a) to handle entity pairs in the one-tomany translation mode. The statistics of the data are described in Table 2.
Comparison Systems. We use the following approaches as baseline methods in comparison.
• BASELINE We use the TransformerBig (Vaswani et al., 2017) to train the baseline system.
• PLACEHOLDER (Luong et al., 2015) that employs entity tags as placeholders to replace the named entity pairs within the training data. PLACEHOLDER adopts a post-processing step to recover the named entities.
• TERMMIND (Wang et al., 2021) that adopts the code-switching strategy to incorporate named entity translation into the source sentence. TERMMIND promotes the translation model to copy the named entity translation to the target sentence.
• TEMPLATE (Wang et al., 2022a) that rearranges the target sentence of named entities and context through a template in the training data. As for generation, TEMPLATE uses the named entities as the constrained prefix to decode the context part.
System Training. We used the open-source toolkit Fairseq (Ott et al., 2019) to implement the model. Specifically, we chose TRANSFORMERBIG as our model, which consisits of an encoder of 6 layers and a decoder of 6 layers. We followed the settings in the original works to train the models. In brief, we trained the TRANSFORMER model for 25K steps with 131K (4096 × 32) tokens per batch. We warmed up it for 10K steps and decayed the learning rate based on the inverse square root of the update number. We used 8 Nvidia V100 GPUs to conduct the experiments and averaged the last five checkpoints to achieve strong performance.
Table 3 shows the data examples for the listed approaches. We adopted the same training settings but different data construction methods for these approaches. In our augmentation step, we obtained context templates while constructing the dictionary D as introduced in Section 2.1. It is important to note that these two steps are performed simultaneously, and excessive additional computing resources are not required.
To make a fair comparison, we conduct experiments with the same bilingual dictionary D for all systems. For PLACEHOLDER, we add a postprocessing step to the BASELINE system. For TERMMIND and TEMPLATE, we follow Wang et al. (2021) and Wang et al. (2022a) to train the models. For the proposed data augmentation approach, we use the BASELINE to compute the normalized sentence-level probability for the translation difficulty T (u, v). In addition, we set the pre-define hyper-parameter N in Section 3 to 350.
Evaluation. For the automatic evaluation, we used SacreBLEU (Post, 2018), NIST (Doddington, 2002), COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020) for the sentence evaluation. we followed Alam et al. (2021) to adopt Exact-Match Accuracy (Acc), Window Overlap (Overlap), and Terminology-biased TER (1-TERm) to perform named entity evaluation.
We also performed a human evaluation to assess the quality of the translated outputs. We randomly chose 200 sentences from the test set and tasked the annotators with comparing our system against the baseline to determine which system output exhibited superior quality.
Furthermore, we proposed an entity-aware machine translation metric that prefers the translation output to generate more accurate named entities, and used the proposed entity-aware metric to evaluate the sentence quality.

heading: 4.2 Result
text: Table 4 presents the experimental results for Chinese-to-Enlgish translation. In the general decoding scenario, where lexical constraints are not applied, all systems significantly improve entity translation accuracy over the baseline. In the lexically constrained decoding scenario, PLACEHOLDER achieves the highest extract-match accuracy, with a score of 95.46 in the WMT20 test
set. OUR achieves the best overall performance by enhancing entity translation accuracy while preserving the BLEU score.
We conducted a human evaluation of the WMT20 Chinese-to-English translation. Table 5 provides a comprehensive account of the human evaluation. The human evaluation results further support the notion that the proposed strategy yields translation output of superior quality compared to the baseline system.
On the WMT21 terminology translation task, as shown in Table 6, our approach also outperforms the baseline system, TERMMIND and TEMPLATE in both English-to-French and English-to-Chinese
translation tasks, demonstrating the effectiveness and universality of the proposed approach.

heading: 4.3 Ablation Study
text: In the ablation study, we analyze the impact of Translation Difficulty (Factor T) and Context Diversity (Factor C) as described in section 2.
As shown in Table 7, both Factor T and C make significant contributions to the translation accuracy of entities. We find that the impact of factor C on performance is greater. In other words, entities are coupled with certain specific contexts and entity translation errors usually occur when encountering other less common contexts. In our understanding,
we consider T to be the translation difficulty of the entity itself. For example, some entities can be translated literally and are easily learned by the model even if they do not appear frequently.
However, for some non-literal and challenging entities, we aim to enrich their context to help translate them better. We regard T as an internal factor of entities and C as an external factor. Previous work only considers lexical constraints while ignoring the causes of entity translation errors, which is also the motivation of our work.

heading: 5 Entity-aware Machine Translation Evaluation
text: We propose a simple and effective entiy-aware evaluation metric for assessing machine translation quality. The main idea involves considering the translation accuracy of the entities during the calculation of the evaluation score, using both the translation output and the reference translation. In particular, when determining the value, if an entity matches the translation output, its weight will be correspondingly increased.
We apply the entiy-aware evaluation strategy on the SacreBLEU (Post, 2018) and ChfF+ (Popović, 2017). We assign an entity-aware weight of 4, which entails calculating the values of the n-grams (words or characters) by proportionally increasing the weights of the values present in the matching entities. We conducted experiments WMT19 metric tasks4 on four high-resource pairs (from Chinese/zh, German/de, Russian/ru, Finnish/fi to English/en) and three low-resource pairs (from Gujarati/gu, Kazakh/kk, Lithuanian/lt to English/en). To make a fair comparison, we adopt the tok-
4https://www.statmt.org/wmt19/metrics-task.html
enizer_13a5 to tokenize the text and apply the Stanford NER system6 to tokenized text.
Table 8 lists the system-level human correlation results on WMT19 metric tasks. We observe that our entity-aware evaluation metric achieves better human correlation compared to the corresponding baseline system for most high-resource language pairs, with the exception of Finnish.
For the experiments in Table 4 and Table 6, we apply the entity-aware SacreBLEU and ChfF+ to the high-resource pairs (Chinese-to-English, English-to-French and English-to-Chinese) and observe the proposed approach OUR achieves improvements over the baseline system in terms of the entity-aware evaluation metric.

heading: 6 Related Work
text: Decoding Algorithm One line of approaches to lexically constrained NMT focuses on modifying the decoding algorithm to impose lexical constraints. Hokamp and Liu propose grid beam search (GBS) algorithm, which takes target-side pre-specified translation as lexical constraints at each decoding step. However, the decoding speed of GBS scales linearly with the number of constraints. To reduce the computation complexity, Post and Vilar propose the dynamic beam allocation (DBA) method, which limits the decoding complexity by dynamically providing a fixed size of beam to the decoder. VDBA (Hu et al., 2019) gives a fast version of DBA and supports batched decoding. A potential issue with these methods is the lack of consideration for translation fidelity, as there is no indication of a matching source for each
5https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/ tokenizers/tokenizer_13a.py
6https://nlp.stanford.edu/software/CRF-NER.shtml
pre-specified translation. Hasler et al. use decoder attentions to match target-side constraints and their corresponding source words. There are also several other constrained decoding algorithms that utilize word alignments to impose constraints(Song et al., 2020; Chen et al., 2021b). Although the alignmentbased decoding methods are faster than previous works, they remain significantly slower compared to the standard beam search algorithm.
Annotating Strategy Another line of studies focus on annotating named entities to provide type and annotating information. Li et al. introduces more linguistic features by inserting special tokens before and after entities in the source sentence. Other researchers (Ugawa et al., 2018b; Modrzejewski et al., 2020) add entity embeddings when encoding the source sentence, helping models differentiate constrained and unconstrained tokens. Dinu et al. use source factors successfully to enforce terminology. Xie et al. attach entity classifiers to the transformer and design an adaptive loss function for named entities. To make NMT models better learn from and cope with lexical constraints, Wang et al. propose to leverage attention modules to explicitly integrate vectorized lexical constraints. Although these works enhance the representation of sentences, they still face a scalability challenge as they attempt to adapt the architecture of NMT models for this task.
Lexical Constraint Moreover, researchers have shown the benefit of editing the training data to induce constraints. Luong et al. use annotated unk tags to present the out-of-vocabulary (OOV) words in training corpora, where the correspondence between source and target unk symbols are obtained from word alignment. Output unk tags are replaced by the corresponding target constraints in a postprocessing stage. Further research (Crego et al., 2016; Yan et al., 2019) extends unk tags symbols
to specific symbols that can present name entities. Due to the loss of word meaning when representing them with placeholder tags, these methods may hinder next-token prediction in the generation process. Target Lemma Annotation (TLA)(Bergmanis and Pinnis, 2021) annotates source entity words with their target language lemmas instead of meaningless symbols to fix the problem of word meaning loss for placeholders. TermMind (Wang et al., 2021), TADA (Ailem et al., 2021) and Kakao (Bak et al., 2021) use a similar method to TLA besides minor difference. To enhance the proportion of constrained target words’ occurrences, CodeSwitching (Song et al., 2019) directly replaces the source words with their target translations, allowing the model to learn lexicon translations by copying source-side target words. However, it also introduces many target-side tokens and expands the source-side vocabulary. Zeng et al. add target constraints as a prefix of the decoder input to fix the problem of larger vocabulary size for CodeSwitching. Further, (Wang et al., 2022a) proposes a template-based translation framework to handle constraints, achieving high match accuracy while maintaining the inference speed.

heading: 7 Conclusion
text: We demonstrate that the translation accuracy of named entities is influenced by both their level of translation difficulty and the diversity of contextual information. We propose a novel data augmentation strategy that boosts the translation probability and the context diversity of the targeted named entity pair. Experimental results show that the proposed strategy outperforms the baseline in terms of general translation performance and entity translation accuracy. We also propose an entity-aware machine translation metric that prefers the translation output to generate more accurate named entities.
Limitations
A limitation of this work is that our method can not cope with one-to-many (entity with more than one translation candidate) constraints. Moreover, efficiently applying the proposed data augmentation strategy to large-scale training data remains a future challenge.

heading: Future Work
text: Moreover, efficiently applying the proposed data augmentation strategy to large-scale training data remains a future challenge