Venue: NAACL 2022

Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraints

Chun Zeng\({}^{\spadesuit}\), Jiangjie Chen\({}^{\spadesuit}\), Tianyi Zhuang\({}^{\spadesuit}\), Rui Xu\({}^{\spadesuit}\), Hao Yang\({}^{\diamondsuit}\), Ying Qin\({}^{\diamondsuit}\), Shimin Tao\({}^{\diamondsuit}\), Yanghua Xiao\({}^{\spadesuit\spadesuit\dagger}\)

\({}^{\clubsuit}\)Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University Huawei Translation Services Center

\({}^{\clubsuit}\)Fudan-Aishu Cognitive Intelligence Joint Research Center {zengc20,jjchen19,tyzhuang20,shawyh}@fudan.edu.cn, ruixu21@m.fudan.edu.cn, {yanghao30,qinying,taoshimin}@huawei.com

###### Abstract

Lexically constrained neural machine translation (NMT) draws much industrial attention for its practical usage in specific domains. However, current autoregressive approaches suffer from high latency. In this paper, we focus on non-autoregressive translation (NAT) for this problem for its efficiency advantage. We identify that current constrained NAT models, which are based on iterative editing, do not handle low-frequency constraints well. To this end, we propose a plug-in algorithm for this line of work, _i.e._, Aligned Constrained Training (ACT), which alleviates this problem by familiarizing the model with the source-side context of the constraints. Experiments on the general and domain datasets show that our model improves over the backbone constrained NAT model in constraint preservation and translation quality, especially for rare constraints.1

Footnote 1: Our implementation can be found at [https://github.com/sted-byte/ACT4NAT](https://github.com/sted-byte/ACT4NAT).

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5777 - 5790
 

constraint, the model is unable to generate the correct context of the term "gesechrien" as if it does not understand the constraint at all. It is dangerous since terms in specific domains are usually low-frequency words. We argue that the main reasons behind this problem are _1)_ the inconsistency between training and constrained inference and _2)_ the unawareness of the source-side context of the constraints.

To solve this problem, we build our algorithm based on the idea that the context of a rare constraint tends _not_ to be rare as well, _i.e._, "_a stranger's neighbors are not necessarily strangers_", as demonstrated in Table 1. We believe that, when the constraint is aligned to the source text, the context of its source-side counterpart can be utilized to be translated into the context of the target-side constraint, even if the constraint itself is rare. Also, when enforced to learn to preserve designated constraints at training-time, a model should be better at coping with constraints during inference-time.

Driven by these motivations, we propose a plug-in algorithm to improve constrained NAT, namely **A**ligned **C**onstrained **T**raining (ACT). ACT extends the family of editing-based iterative NAT Gu et al. (2019); Susanto et al. (2020); Xu and Carpuat (2021), the current paradigm of constrained NAT. Specifically, ACT is composed of two major components: _Constrained Training_ and _Alignment Prompting_. The former extends regular training of iterative NAT with pseudo training-time constraints into the state transition of imitation learning. The latter incorporates source alignment information of constraints into training and inference, indicating the context of the potentially rare terms.

In summary, this work makes the following contributions:

* We identify and analyse the problems w.r.t. rare lexical constraints in current methods for constrained NAT;
* We propose a plug-in algorithm for current constrained NAT models, _i.e._, aligned constrained training, to improve the translation under rare constraints;
* Experiments show that our approach improves the backbone model w.r.t. constraint preservation and translation quality, especially for rare constraints.

## 2 Related Work

Lexically Constrained TranslationExisting translation methods impose lexical constraints during either inference or training. At training time, constrained MT models include code-switching data augmentation Dinu et al. (2019); Song et al. (2019); Chen et al. (2020) and training with auxiliary tasks such as token or span-level mask-prediction Ailem et al. (2021); Lee et al. (2021). At inference time, autoregressive constrained decoding algorithms include utilizing placeholder tag Luong et al. (2015); Crego et al. (2016), grid beam search Hokamp and Liu (2017); Post and Vilar (2018) and alignment-enhanced decoding Alkhouli et al. (2018); Song et al. (2020); Chen et al. (2021). For the purpose of efficiency, recent studies also focus on non-autoregressive constrained translation. Susanto et al. (2020) proposes to modify the inference procedure of Levenshtein Transformer Gu et al. (2019) where they disallow the deletion of constraint words during iterative editing. Xu and Carpuat (2021) further develops this idea and introduces a reposition operation that can reorder the constraint tokens. Our work absorbs the idea of both lines of work. Based on NAT methods, we brings alignment information by terminologies to help learn the contextual information for lexical constraints, especially the rare ones.

Non-Autoregressive TranslationAlthough enjoy the speed advantage, NAT models suffer from performance degradation due to the multi-modality problem, _i.e._, generating text when multiple translations are plausible. Gu et al. (2018) applies sequence-level knowledge distillation KD Kim and Rush (2016) that uses an AT's output as an NAT's new target, which reduces word diversity and reordering complexity in reference, resulting in fewer modes Zhou et al. (2020); Xu et al. (2021). Various algorithms have also been proposed to alleviate this problem, including incorporating latent variables Kaiser et al. (2018); Shu et al. (2020), iterative refinement Ghazvininejad et al. (2019); Stern et al. (2019); Gu et al. (2019); Guo et al. (2020), advanced training objective Wang et al. (2019); Du et al. (2021) and gradually learning target-side word inter-dependency by curriculum learning Qian et al. (2021). Our work extends the family of editing-based iterative NAT models for its flexibility to impose lexical constraints Susanto et al. (2020); Xu and Carpuat (2021) 



## 3 Background

### Non-Autoregressive Translation

Given a source sentence as \({\mathbf{x}}\) and a target sentence as \({\mathbf{y}}=\{y_{1},\cdots,y_{n}\}\), an AT model generates in a left-to-right order, _i.e._, generating \(y_{t}\) by conditioning on \({\mathbf{x}}\) and \(y_{<t}\). An NAT model Gu et al. (2018), however, discards the word inter-dependency in output tokens, with the conditional independent probability distribution modeled as:

\[P({\mathbf{y}}|{\mathbf{x}})=\prod_{t=1}^{n}P(y_{t}|{\mathbf{x}}). \tag{1}\]

Such factorization is featured with _high efficiency_ at the cost of _performance drop_ in translation tasks due to the _multi-modality_ problem, _i.e._, translating in mixed modes and resulting in token repetition, missing, or incoherence.

### Editing-based Iterative NAT

Iterative refinement by editing is an NAT paradigm that suits constrained translations due to its flexibility. It alleviates the multi-modality problem by being autoregressive in editing previously generated sequences while maintaining nonautoregressiveness within each iteration. Thus, it achieves better performance than fully NATs while is faster than ATs.

Levenshtein TransformerTo better illustrate our idea, we use Levenshtein Transformer LeVT et al. (2019) as the backbone model in this work, which is a representative model for constrained NAT based on iterative editing.

LevT is based on the Transformer architecture Vaswani et al. (2017), but more flexible and fast than autoregressive ones. It models the generation of sentences as Markov Decision Process (MDP) defined by a tuple \((\mathcal{Y},\mathcal{A},\mathcal{E},\mathcal{R},\mathbf{y}^{0})\). At each decoding iteration, the agent \(\mathcal{E}\) receives an input \({\mathbf{y}}\in\mathcal{Y}\), chooses an action \(a\in\mathcal{A}\) and gets reward \(r\). \(\mathcal{Y}\) is a set of discrete sentences and \(\mathcal{R}\) is the reward function. \({\mathbf{y}}^{0}\in\mathcal{A}\) is the initial sentence to be edited.

Each iteration consists of two basic operations, _i.e._, _deletion_ and _insertion_, which is described in Table 2. For the \(k\)-th iteration of the sentence \({\mathbf{y}}^{k}=({<}{S>},y_{1},...,y_{n},{<}/{S>})\), the insertion consists of placeholder and token classifiers, and the deletion is achieved by a deletion classifier. LevT trains the model with imitation learning to insert and delete, which lets the agent imitate the behaviors drawn from the expert policy:

* **Learning to insert**: edit to reference by inserting tokens from a fragmented sentence (_e.g._, random deletion of reference).
* **Learning to delete**: delete from the insertion result of the current training status to the reference.

The key idea is to learn how to edit from a ground truth after adding noise or the output of an adversary policy to the reference. The ground truth of the editing process is derived from the Levenshtein distance Levenshtein (1965).

Lexically Constrained InferenceLexical constraints can be imposed upon a translation model in: _1) soft constraints_: allowing the constraints not to appear in the translation; and _2) hard constraints_: forcing the constraints to appear in the translation. In NAT, the constraints are generally incorporated at inference time. Susanto et al. (2020) injects constraints as the initial sequence for iterative editing in Levenshtein Transformer LeVT et al. (2019), achieving soft constrained translation. And hard constrained translation can be easily done by dissolving the deletion of the constraints. Xu and Carpuat (2021) alters the deletion action in LevT with the reposition operation, allowing the reordering of multiple constraints.

### Motivating Study: Self-Constrained Translation

According to Table 1, constrained NAT models seem to suffer from the low-frequency of lexical constraints, which is dangerous as most terms in practice are rare. To further explore the impact of constraint frequency upon NATs, we conduct a preliminary analysis on constrained LevT Susanto et al. (2020). We sort words in each reference text based on frequency, dividing them into _six_ buckets by frequency order (as in Figure 1), and sample a

\begin{table}
\begin{tabular}{p{42.7pt}|p{284.5pt}} \hline
**Action** & **Implementation** \\ \hline
**Insertion** & **Placeholder Classifier**: predicts the number of tokens (\(0\sim K_{max}\)) to be inserted at every consecutive position pairs and then inserts the corresponding number of [PLH]. \\  & **Token Classifier**: predicts the actual target token of the [PLH]. \\ \hline
**Deletion** & **Deletion Classifier**: predicts whether each token (except for the boundaries) should be “kept” or “deleted”. \\ \hline \end{tabular}
\end{table}
Table 2: The implementation details of insertion and deletion operations in LevT.



word from each bucket as lexical constraints for translation.2 We denote these constraints as _self-constraints_. In this way, we have six times the data, and the six samples derived from one raw sample only differ in the lexical constraints.

Footnote 2: Sentences that cannot be divided into six buckets are removed.

As shown in Figure 1, translation performance generally keeps improving as the self-constraint gets rarer. This is because setting low-frequency words in a sentence as constraints, which are often hard to translate, actually lightens the load of an NAT model. However, there are two noticeable performance drops around relative frequency ranges of 10%-30% (bucket 2) and 90%-100% (bucket 6), denoted as _Drop#1_ (-0.3 BLEU) and _Drop#2_ (-0.6 BLEU). Note that Drop#1 is mainly due to the the fact that there are mostly unknown tokens (_i.e._, \(<\)UNK\(>\)) in the bucket 2. We leave detailed discussions about buckets and Drop#1 to Appendix C.

In this experiment, we are more interested in the reasons for Drop#2 when constraints are low-frequency words. We assume a _trade-off_ in _self-constrained_ NAT: the model does not have to translate rare words as they are set as an initial sequence (constraints), but it will have a hard time understanding the context of the rare constraint due to 1) the rareness itself and 2) the lack of the alignment information between target-side constraint tokens and source tokens. Thus, the model does not know how many tokens should be inserted to the left and right of the constraint, which is consistent with the findings in Table 1.

## 4 Proposed Approach

The findings and assumptions discussed above motivate us to propose a plug-in algorithm for lexically constrained NAT models, _i.e._, Aligned Constrained Training (ACT). ACT is designed based on two major ideas: _1) Constrained Training_: bridging the discrepancy between training and constrained inference; _2) Alignment Prompting_: helping the model understand the context of the constraints.

### Constrained Training

As introduced in SS3.2, constraints are typically imposed during inference time in NAT Susanto et al. (2020); Xu and Carpuat (2021). Specifically, lexical constraints are imposed by setting the initial sequence \({\mathbf{y}^{0}}\) as \((<\)S\(>,C_{1},C_{2},...,C_{k},</S>)\), where \(C_{i}=(c_{1},c_{2},...,c_{l})\) is the \(i\)-th lexical constrained word, \(l\) is the number of tokens in the \(i\)-th constraint, and \(k\) is the number of constraints.

However, such mandatory preservation of the constraints is not carried out during training. During imitation learning, _random deletion_ is applied for ground-truth \({\mathbf{y}^{*}}\) to get the incomplete sentences \({\mathbf{y}^{\prime}}\), producing the data samples for expert policies of how to insert from \({\mathbf{y}^{\prime}}\) to \({\mathbf{y}^{*}}\). This leads to a situation where the model does not learn to preserve fixed tokens and organize the translation around the tokens. Such discrepancy could harm the applications of soft constrained translation.

To solve this problem, we propose a simple but effective Constrained Training (CT) algorithm. We first build _pseudo terms_ from the target by sampling 0-3 words (more tokens after tokenization) from reference as the pre-defined constraints for training.3 Afterward, we disallow the deletion of pseudo term tokens during building data samples for imitation learning. This encourages the model to edit incomplete sentences containing lexical constraints into complete ones, bridging the gap between training and inference.

Footnote 3: In the experiments, these pseudo constraints are sampled based on TF-IDF score to mimic the rare but important terminology constraints in practice.

### Alignment Prompting

As stated in SS3.3, we assume the rareness of constraints hinders the model to insert proper tokens of its contexts (_i.e._, _a stranger's neighbors are also strangers_). To make the matter worse, previous research Ding et al. (2021) has also shown that lexical choice errors on low-frequency words tend

Figure 1: Ablation study of self-constrained translation on WMT14 En\(\rightarrow\)De test set with Wiktionary terminology constraints Dinu et al. (2019). The absolute average frequencies of self-constraints in a bucket decrease from left to right in the x-axis.

 

to be propagated from the teacher (an AT model) to the student (an NAT model) in knowledge distillation.

However, terminologies, by nature, provide hard alignment information for source and target which the model can conveniently utilize. Thus, on top of constrained training, we propose an enhanced approach named **A**ligned **C**onstrained **T**raining (ACT). As illustrated in Figure 2, we propose to directly align the target-side constraints with the source words and prompt the alignment information to the model during both training and inference.

Building Alignment for ConstraintsWe first align the source words to the target-side constraints, which are either pseudo constraints during training or actual constraints during inference. For each translated sentence constraints \({\cal C}_{\text{tgt}}=(C_{1},C_{2},...,C_{k})\), we use an external alignment tool external aligner, such as GIZ++ Brown et al. (1993); Och and Ney (2003), to find the corresponding source words, denoted as \({\cal C}_{\text{src}}=(C^{\prime}_{1},C^{\prime}_{2},...,C^{\prime}_{k})\).

Prompting Alignment into LevTThe encoder in LeVT, besides token embedding and position embedding, is further added with a learnable alignment embedding that comes from \({\cal C}_{\text{src}}\) and \({\cal C}_{\text{tgt}}\). We set the alignment value for each token in \(C^{\prime}_{i}\) to \(i\) and the others to 0, which are further encoded into embeddings. The prompting of alignment is not limited to training, as we also add such alignment embeddings to source tokens aligned to target-side constraints during inference.

## 5 Experiments

### Data and Evaluation

Parallel Data and Knowledge DistillationWe consider the English\(\rightarrow\)German (En\(\rightarrow\)De) translation task and train all of the MT models on WMT14 En-De (3,961K sentence pairs), a benchmark translation dataset. All sentences are pre-processed via byte-pair encoding (BPE) Sennrich et al. (2016) into sub-word units. Following the common practice of training an NAT model, we use the sentence-level knowledge distillation data generated by a Transformer, Vaswani et al. (2017) provided by Kasai et al. (2020).

Datasets with Lexical ConstraintsGiven models trained on the above-mentioned training sets, we evaluate them on the _test sets_ of several lexically constrained translation datasets. These test sets are categorized into two types of standard lexically constrained translation datasets: _1)_ Type#1: tasks from WMT14 Vaswani et al. (2017) and WMT17 Bojar et al. (2017), which are of the same general domain (news) as training sets; _2)_ Type#2: tasks from OPUS Tiedemann (2012) that are of specific domains (medical and law). Particularly, the real application scenarios of lexically constrained MT models are usually domain-specific, and the constrained words in these domain datasets are relatively less frequent and more important.

Figure 2: An example of alignment prompting. The constraint tokens \(y_{*}\) are given by users during inference, and can also be sampled from target sentence during training. Given \(y_{*}\), we align them with tokens \(x_{*}\) in the source and build alignment embeddings to be fed into the encoder.

 

Following previous work Dinu et al. (2019); Susanto et al. (2020); Xu and Carpuat (2021), the lexical constraints in Type#1 tasks are extracted from existing terminology databases such as Interactive Terminology for Europe (IATE)4 and Wiktionary (WIKT)5 accordingly. The OPUS-EMEA (medical domain) and OPUS-JRC (legal domain) in Type#2 tasks are datasets from OPUS. The constraints are extracted by randomly sampling 1 to 3 words from the reference Post and Vilar (2018). These constraints are then tokenized with BPE, yielding a larger number of tokens as constraints. The statistical report is shown in Table 3, indicating the frequencies of Type#2 datasets are generally much lower than Type#1 ones.

Footnote 4: [https://iate.europa.eu](https://iate.europa.eu)

Footnote 5: [https://www.wiktionary.org](https://www.wiktionary.org)

Evaluation MetricsWe use BLEU Papineni et al. (2002) for estimating the general quality of translation. We also use _Term Usage Rate_Term%, Dinu et al. (2019); Susanto et al. (2020); Lee et al. (2021) to evaluate lexically constrained translation, which is the ratio of term constraints appearing in the translated text.

### Models

We use Levenshtein Transformer LevT, Gu et al. (2019) as the backbone model to ACT algorithm for constrained NAT. We compare our approach with a series of previous MT models on applying lexical constraints:

* _Transformer_ Vaswani et al. (2017), set as the AT baseline;
* _Dynamic Beam Allocation_ (DBA) Post and Vilar (2018) for constrained decoding with dynamic beam allocation over Transformer;
* _Train-by-sep_Dinu et al. (2019), trained on augmented code-switched data by replacing the source terms with target constraints or append on source terms during training;
* Constrained LevT Susanto et al. (2020), which develops LevT Gu et al. (2019) by setting constraints as initial editing sequence;
* EDITOR Xu and Carpuat (2021), a variant of LevT, replacing the delete action with a reposition action.

Implementation DetailsWe use and extend the FairSeq framework Ott et al. (2019) for training our models. We keep mostly the default parameters of FairSeq, such as setting \(d_{model}\) = 512, \(d_{hidden}\) = 2,048, \(n_{heads}\) = 8, \(n_{layers}\) = 6 and \(p_{dropout}\) = 0.3. The learning rate is set as 0.0005, the warmup step is set as 4,000 steps. All models are trained with a batch size of 16,000 tokens for maximum of 300,000 steps with Adam optimizer Kingma and Ba (2014) on 2 NVIDIA GeForce RTX 3090 GPUs with gradient accumulation of 4 batches. Checkpoints for testing are selected from the average weights of the last 5 checkpoints. For Transformer Vaswani et al. (2017), we use the

\begin{table}
\begin{tabular}{l||c c|c c|c c|c} \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c|}{**WMT17-IATE**} & \multicolumn{2}{c|}{**WMT17-WIKT**} & \multicolumn{2}{c|}{**WMT14-WHKT**} & \multirow{2}{*}{**Latency**} \\  & **Term\%** & **BLEU** & **Term\%** & **BLEU** & **Term\%** & **BLEU** & (ms) \\ \hline _Reported results in previous work_ & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \hline Transformer Vaswani et al. (2017)† & 79.65 & **29.58** & 79.75 & **30.80** & **76.77** & **31.75** & **244.5** \\ DBA Post and Vilar (2018) & 82.00 & 25.30 & **99.50** & 25.80 & - & - & 434.4 \\ Train-by-rep Dinuu et al. (2019) & **94.50** & 26.00 & 93.40 & 26.30 & - & - & - \\ \cline{2-7} LevT Gü Gu et al. (2019)† & 80.31 & 28.97 & 81.11 & 30.24 & 80.23 & 29.86 & **92.0** \\ \multicolumn{7}{l}{_w/ soft constraint_Susanto et al. (2020)} & 93.81 & 29.73 & 93.44 & 30.82 & 94.43 & 29.93 & - \\ \multicolumn{7}{l}{_w/ hard constraint_Susanto et al. (2020)} & **100.00** & **30.13** & **100.00** & **31.20** & **100.00** & **30.49** & - \\ EDITOR Xu and Carpuat (2021)� & 83.00 & 27.90 & 83.50 & 28.80 & - & - & 121.7 \\ \multicolumn{7}{l}{_w/ soft constraint_} & 97.10 & 28.80 & 96.80 & 29.30 & - & - & - \\ \multicolumn{7}{l}{_w/ hard constraint_} & **100.00** & 28.90 & 99.80 & 29.30 & - & - & 134.1 \\ \hline _Our implementation_ & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \hline LevT�� & 78.32 & **29.80** & 80.20 & 30.75 & **79.53** & 29.95 & **71.9** \\ \multicolumn{7}{l}{+ constrained training (CT)† } & 78.76 & 29.46 & **80.77** & **30.82** & 79.13 & 30.24 & 78.6 \\ \multicolumn{7}{l}{+ aligned constrained training (ACT)† } & **79.43** & 29.57 & 80.20 & 30.63 & 77.17 & **30.35** & 77.0 \\ \multicolumn{7}{l}{LeV† _soft constraint_} & 94.25 & 30.11 & 93.78 & 30.92 & 94.88 & 30.38 & 79.5 \\ \multicolumn{7}{l}{+ constrained training (CT) † } & 96.24 & 30.19 & 96.61 & 30.96 & 97.44 & 31.01 & **75.4** \\ \multicolumn{7}{l}{+ aligned constrained training (ACT)† } & **96.90** & **30.56** & **97.62** & **31.06** & **98.82** & **31.08** & 76.3 \\ \multicolumn{7}{l}{LeV† _hard constraint_} & **100.00** & 30.31 & **100.00** & 30.65 & **100.00** & 30.49 & 82.7 \\ \multicolumn{7}{l}{+ constrained training (CT)† } & **100.00** & 30.31 & **100.00** & 30.99 & **100.00** & 31.01 & 78.1 \\ \multicolumn{7}{l}{+ aligned constrained training (ACT)† } & **100.00** & **30.68** & **100.00** & **31.18** & **100.00** & **31.11** & **77.0** \\ \hline \end{tabular}
\end{table}
Table 4: Translation results with lexical constraints. **Term%** is the constraint term usage rate. Method† translates _without_ lexical constraints in input.

 

checkpoint released by Ott et al. (2018).

### Main Results

Table 4 reports the performance of LevT with ACT (as well as the CT ablation) on the type 1 tasks (WIKT and IATE as terminologies), compared with baselines. In general, the results indicate the proposed CT/ACT algorithms achieve a consistent gain in performance, term coverage, and speed over the backbone model mainly in the setting of constrained translation.

When translating with _soft_ constraints, _i.e._, the constraints need not appear in the output, adding ACT to LevT helps preserve the terminology constraints (+\(\sim\)5 Term%) and improves translation performance (+0.31-0.88 on BLEU). If we enforce _hard_ constraints, the term usage rate doubtlessly reaches 100%, with reasonable improvements on BLEU. When translating _without_ constraints, however, adding ACT does not bring consistent improvements as hard and soft constraints do.

As for the ablation for CT and ACT, we have two observations: 1) term usage rate increases mainly because of CT, and can be further improved by ACT; 2) translation quality (BLEU) increases due to the additional hard alignment of ACT over CT. The former could be attributed to the behavior of _not deleting the constraints_ in CT. The latter is because of the introduction of source-side information of constraints that familiarize the model with the constraint context.

Table 4 also shows the efficiency advantage of non-autoregressive methods compared with autoregressive ones, which is widely reported in the NAT research literature. The proposed methods do not cause drops in translation speed against the backbone LevT. When translating with lexical constraints, LevT with CT or ACT is even faster than LevT. In contrast, constrained decoding methods for autoregressive models (_i.e._, DBA) nearly double the translation latency. Since the main purpose of non-autoregressive research is developing efficient algorithms, such findings could facilitate the industrial usage for constrained translation.

Translation Results on Domain DatasetsFor a generalized evaluation of our methods, we apply the models trained on the general domain dataset (WMT14 En-De) to medical (OPUS-EMEA) and legal domains (OPUS-JRC). As seen in Table 5, even greater performance boosts are witnessed. When trained with ACT, both term usage (+\(\sim\)8-10 Term%) and translation performance (up to 4 BLEU points) largely increase, which is more significant than the general domain.

The reason behind this observation is that the backbone LevT would have a hard time recognizing them as constraints since the lexical constraints in these datasets are much rarer. Therefore, forcing LevT to translate with these rare constraints would generate worse text, _e.g._, BLEU drops for 2.45 points on OPUS-JRC than with soft constraints. And when translating with soft constraints, LevT over-deletes these rare constraints. In contrast, the context information around constraints is effectively pin-pointed by ACT, so ACT would know the context ("neighbors") of the rare constraint ("strangers") and insert the translated context around the lexical constraints. In this way, more terms are preserved by ACT, and the translation achieves better results.

## 6 Analysis

### Self-Constrained Translation Revisited

As a direct response to our motivation in this paper, we revisit the ablation study of self-constrained NAT in SS3.3 with the proposed ACT algorithm. Same as before, we build self-constraints from each target sentence and sort them by frequency. As shown in Figure 3(a), different from constrained LevT that suffers from Drop#2 (SS3.3), ACT managed to handle this scenario pretty well. Following the motivations given in SS3.3, when constraints become rarer, ACT successfully breaks the _trade-off_ with better understanding of the provided contextual information.

What if the self-constraints are sorted based on TF-IDF?We also study the importance of different words in a sentence via TF-IDF by forcing

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**OPUS-EMEA**} & \multicolumn{2}{c}{**OPUS-JRC**} \\ \cline{2-5}  & Term\% & BLEU & Term\% & BLEU \\ \hline LevT\({}^{\dagger}\) & 52.40 & 27.90 & **55.39** & 30.24 \\   + ACT\({}^{\dagger}\) & **53.41** & **28.30** & 55.35 & **31.01** \\    LeVt w/ _soft_ & 83.37 & 30.35 & 84.32 & 32.53 \\   + ACT & **92.09** & **32.02** & **91.94** & **33.70** \\    LeVt w/ _hard_ & 100.00 & 30.77 & 100.00 & 30.08 \\   + ACT & 100.00 & **32.30** & 100.00 & **34.09** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on test sets from OPUS, which is out of the training domain (WMT14 En\(\rightarrow\)De). Results show that ACT brings larger performance for lower-frequency lexical constraints within these datasets.

 

them as constraints. As results in Figure 3(b) show, we have very similar observations from frequency-based self-constraints at Figure 3(a), and the gap between LevT and LevT + ACT is even higher as TF-IDF score reaches the highest.

### How does ACT perform under different kinds of lexical constraints?

The experiments in SS6.1 create pseudo lexical constraints by traversing the target-side reference for understanding the proposed ACT. In the following analyses, we study different properties of lexical constraints, _e.g._, frequency and numbers, and how they affect constrained translation.

Are improvements by ACT robust against constraints of different frequencies?Given terminology constraints in the samples, we sort them by (averaged) frequency and evenly average the corresponding data samples into _high_, _medium_ and _low_ categories.6 The results on translation quality of each category for the En\(\rightarrow\)De translation tasks are presented in Table 6. We find that LevT benef its mostly from ACT in the scenarios of lower-frequency terms for three datasets. Although, in some settings such as High in WMT14-WIKT and Med in WMT17-WIKT, the introduction of ACT for constrained LevT seems to bring performance drops for those higher-frequency terms. Since terms from IATE are rarer than WIKT as in Table 3, the improvements brought by ACT are steady.

Footnote 6: For multi-word terms, we take the average frequency of the words.

Are improvements by ACT robust against constraints of different numbers?In more practical settings, the number of constraints is usually more than one. To simulate this, we randomly sample 1-5 words from each reference as lexical constraints, and results are presented in Figure 4. We find that, as the number of constraints grows, the translation quality ostensibly becomes better for LevT with or without ACT. And ACT consistently brings extra improvements, indicating the help by ACT for constrained decoding in constrained NAT.

### Limitations

Although the proposed ACT algorithm is effective to improve NAT models on constrained translation, we also find it does not bring much performance gain on translation quality (_i.e._, BLEU) over the backbone LevT for unconstrained translation. The results on the full set of WMT14 En\(\rightarrow\)De test set further corroborate this finding, which is shown in Appendix A.

Another limitation of our work is that we do not propose a new paradigm for constrained NAT. The

Figure 4: Ablation results of constrained translation with one-to-multiple constraints.

Figure 3: Extended self-constrained translation results on WMT14-WIKT. Each and every word of a reference is used as a lexical constraint (_i.e._, self-constraint) for translation, sorted by frequency or TF-IDF.

 

purpose of this work is to enhance existing methods for constrained NAT, _i.e._, editing-based iterative NAT methods, under rare lexical constraints. It would be interesting for future research to explore new ways to impose lexical constraints on NAT models, perhaps on non-iterative NAT.

Note that, machine translation in real scenario still falls behind human performance. Moreover, since we primary focus on improving constrained NAT, real applications calls for refinement in various aspects that we do not consider in this work.

## 7 Conclusion

In this work, we propose a plug-in algorithm (ACT) to improve lexically constrained non-autoregressive translation, especially under low-frequency constraints. ACT bridges the gap between training and constrained inference and prompts the context information of the constraints to the constrained NAT model. Experiments show that ACT improves translation quality and term preservation over the backbone NAT model Levenshtein Transformer. Further analyses show that the findings are consistent over constraints varied from frequency, TF-IDF, and lengths. In the future, we will explore the application of this approach to more languages. We also encourage future research to explore a new paradigm of constrained NAT methods beyond editing-based iterative NAT.

## Acknowledgement

We would like to thank Xinyao Shen and Shineng Fang at Fudan University as well as Yimeng Chen at Huawei for the support in implementation. We also thank Rong Ye at ByteDance and the anonymous reviewers for their valuable comments and suggestions for this work. This work was supported by National Key Research and Development Project (No. 2020AAA0109302), Shanghai Science and Technology Innovation Action Plan (No.19511120400) and Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103).

## References

* Ailem et al. [2021] Melissa Ailem, Jingshu Liu, and Raheel Qader. 2021. Encouraging neural machine translation to satisfy terminology constraints. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 1450-1455, Online. Association for Computational Linguistics.
* Alkhouli et al. [2018] Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney. 2018. On the alignment problem in multi-head attention-based neural machine translation. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 177-185, Brussels, Belgium. Association for Computational Linguistics.
* Bahdanau et al. [2015] Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations, ICLR 2015 ; Conference date: 07-05-2015 Through 09-05-2015.
* Barrault et al. [2020] Loic Barrault, Magdalena Biesialska, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubesic, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 conference on machine translation (WMT20). In _Proceedings of the Fifth Conference on Machine Translation_, pages 1-55, Online. Association for Computational Linguistics.
* Bird et al. [2009] Steven Bird, Ewan Klein, and Edward Loper. 2009. _Natural language processing with Python: analyzing text with the natural language toolkit_. " O'Reilly Media, Inc..".
* Bojar et al. [2017] Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{6}{c}{**WMT14-WIKT**} & \multicolumn{6}{c}{**WMT17-IATE**} & \multicolumn{6}{c}{**WMT17-WIKT**} \\ \cline{2-13}  & All & High & Med. & Low & All & High & Med. & Low & All & High & Med. & Low \\ \hline LevT\({}^{\dagger}\) & 29.95 & 30.46 & **28.03** & 31.49 & **29.80** & **30.08** & **29.72** & **29.45** & **30.75** & **30.96** & 29.09 & 32.16 \\ + ACT\({}^{\dagger}\) & **30.35** & **30.68** & 28.00 & **32.54** & 29.57 & 29.63 & 29.57 & 29.20 & 30.63 & 30.35 & **29.11** & **32.46** \\ LevT w/ _soft_ & 30.38 & 30.37 & 28.50 & 32.19 & 30.11 & 29.25 & 30.67 & 30.04 & 30.92 & 30.70 & **29.58** & 32.23 \\ + ACT & **31.08** & **30.48** & **29.18** & **33.85** & **30.56** & **29.93** & **31.05** & **30.36** & **31.06** & **30.72** & 29.53 & **32.73** \\ LevT w/ _hard_ & 30.49 & **30.50** & 28.67 & 31.99 & 30.31 & 29.46 & 30.66 & 30.37 & 30.65 & 30.28 & 29.44 & 32.00 \\ + ACT & **31.11** & 30.23 & **29.32** & **33.85** & **30.68** & **29.97** & **31.18** & **30.67** & **31.18** & **30.58** & **29.71** & **32.90** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Ablation results of terminology-constrained En\(\rightarrow\)De translation tasks w.r.t. word frequency of terms.

 

translation (WMT17). In Proceedings of the Second Conference on Machine Translation, pp. 169-214. Cited by: SS3.
* P. F. Brown, S. A. Della Pietra, V. J. Della Pietra, and R. L. Mercer (1993)The mathematics of statistical machine translation: parameter estimation. Computational Linguistics19 (2), pp. 263-311. External Links: ISSN 0854-499X, Document, Link Cited by: SS3.
* G. Chen, Y. Chen, and V. O.K. Li (2021)Lexically constrained neural machine translation with explicit alignment guidance. Proceedings of the AAAI Conference on Artificial Intelligence35 (14), pp. 12630-12638. External Links: ISSN 0037-1273, Link, Document Cited by: SS3.
* G. Chen, Y. Chen, Y. Wang, and V. O.K. Li (2020)Lexical-constraint-aware neural machine translation via data augmentation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3587-3593. External Links: ISBN 978-1-4673-319-1, Link, Document Cited by: SS3.
* J. Crego, J. Kim, G. Klein, A. Repbollo, K. Yang, J. Senellart, E. Akhanov, P. Brunelle, A. Coquard, Y. Deng, S. Enoue, C. Geiss, J. Johanson, A. Khalsa, R. Khiari, B. Ko, C. Kobus, J. Lorieux, L. Martins, D. Nguyen, A. Priori, T. Riccardi, N. Segal, C. Servan, C. Tiquet, B. Wang, J. Yang, D. Zhang, J. Zhou, and P. Zoldan (2016)Systran's pure neural machine translation systems. External Links: 1605.02146 Cited by: SS3.
* L. Ding, L. Wang, X. Liu, D. F. Wong, D. Tao, and Z. Tu (2021)Understanding and improving lexical choice in non-autoregressive translation. In International Conference on Learning Representations, Cited by: SS3.
* G. Dinu, P. Mathur, M. Federico, and Y. Al-Onaizan (2019)Training neural machine translation to apply terminology constraints. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3063-3068. External Links: ISBN 978-1-4673-319-1, Link, Document Cited by: SS3.
* C. Du, Z. Tu, and J. Jiang (2021)Order-agnostic cross entropy for non-autoregressive machine translation. In Proc. of ICML, Cited by: SS3.
* M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer (2019)Mask-predict: parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 6112-6121. External Links: ISBN 978-1-4673-319-1, Link, Document Cited by: SS3.
* J. Gu, J. Bradbury, C. Xiong, V. O.K. Li, and R. Socher (2018)Non-autoregressive neural machine translation. In International Conference on Learning Representations, Cited by: SS3.
* J. Guo, Z. Zhang, L. Xu, H. Wei, B. Chen, and E. Chen (2020)Incorporating bert into parallel sequence decoding with adapters. In Advances in Neural Information Processing Systems, Vol. 33, pp. 10843-10854. External Links: ISSN 0037-1273, Link, Document Cited by: SS3.
* C. Hokamp and Q. Liu (2017)Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, pp. 1535-1546. External Links: ISBN 978-1-4673-319-1, Link, Document Cited by: SS3.
* L. Kaiser, S. Bengio, A. Roy, A. Vaswani, N. Parmar, J. Uszkoreit, and N. Shazeer (2018)Fast decoding in sequence models using discrete latent variables. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80, pp. 2390-2399. External Links: ISBN 978-1-4673-319-1, Link, Document Cited by: SS3.
* J. Kasai, J. Cross, M. Ghazvininejad, and J. Gu (2020)Non-autoregressive machine translation with disentangled context transformer. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119, pp. 5144-5155. External Links: ISBN 978-1-4673-319-1, Link, Document Cited by: SS3.
* Y. Kim and A. M. Rush (2016)Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, pp. 1317-1327. External Links: ISBN 978-1-4673-319-1, Link, Document Cited by: SS3.
* D. P. Kingma and J. Ba (2014)Adam: a method for stochastic optimization. Note: cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego External Links: ISBN 978-1-4673-319-1, Link, Document Cited by: SS3.
* Gyubok Lee, Seongjun Yang, and Edward Choi (2021)Improving lexically constrained neural machine translation with source-conditioned masked span prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Online, pp. 743-753. External Links: ISBN 978-1-4673-319-1, Link, Document Cited by: SS3.
* Vladimir I. Levenshtein (1965)Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics. Doklady10, pp. 707-710.

 

Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 11-19, Beijing, China. Association for Computational Linguistics.
* Och and Ney [2003] Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. _Computational Linguistics_, 29(1):19-51.
* Ott et al. [2019] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ott et al. [2018] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
* Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Post and Vilar [2018] Matt Post and David Vilar. 2018. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1314-1324, New Orleans, Louisiana. Association for Computational Linguistics.
* Qian et al. [2021] Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2021. Glancing transformer for non-autoregressive neural machine translation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1993-2003, Online. Association for Computational Linguistics.
* Sennrich et al. [2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Shu et al. [2020] Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. 2020. Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(05):8846-8853.
* Song et al. [2020] Kai Song, Kun Wang, Heng Yu, Yue Zhang, Zhongqiang Huang, Weihua Luo, Xiangyu Duan, and Min Zhang. 2020. Alignment-enhanced transformer for constraining nmt with pre-specified translations. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(05):8886-8893.
* Song et al. [2019] Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang. 2019. Code-switching for enhancing NMT with pre-specified translation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 449-459, Minneapolis, Minnesota. Association for Computational Linguistics.
* Stern et al. [2019] Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible sequence generation via insertion operations. In _Proceedings of the 36th International Conference on Machine Learning_, volume 97 of _Proceedings of Machine Learning Research_, pages 5976-5985. PMLR.
* Susanto et al. [2020] Raymond Hendy Susanto, Shamil Chollampatt, and Liling Tan. 2020. Lexically constrained neural machine translation with Levenshtein transformer. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3536-3543, Online. Association for Computational Linguistics.
* Tiedemann [2012] Jorg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)_, pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
* Ture et al. [2012] Ferhan Ture, Douglas W. Oard, and Philip Resnik. 2012. Encouraging consistent translation choices. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 417-426, Montreal, Canada. Association for Computational Linguistics.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Wang et al. [2019] Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019. Non-autoregressive machine translation with auxiliary regularization. _Proceedings of the AAAI Conference on Artificial Intelligence_, 33(01):5377-5384 

Weijia Xu and Marine Carpuat. 2021. EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints. _Transactions of the Association for Computational Linguistics_, 9:311-328.
* Xu et al. (2021) Weijia Xu, Shuming Ma, Dongdong Zhang, and Marine Carpuat. 2021. How does distilled data complexity impact the quality and confidence of non-autoregressive machine translation? In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 4392-4400, Online. Association for Computational Linguistics.
* Zhou et al. (2020) Chunting Zhou, Jiatao Gu, and Graham Neubig. 2020. Understanding knowledge distillation in non-autoregressive machine translation. In _International Conference on Learning Representations_.

 

Results on Full Test Set of WMT14 (En\(\rightarrow\)De)

We extend the experiment on WMT14 En\(\rightarrow\)De task to the full test set (3,003 samples) in Table 7. Following Susanto et al., we report results on both the filtered test set for sentence pairs that contain at least one target constraint ("Con.", 454 sentences) and the full test set ("Full", 3,003 sentences), which contains samples that do not have lexical constraints. When trained on the full test set, term usage rate raises from 94.88% to 98.82% when trained with ACT under soft constrained decoding, but the BLEU score has marginal improvements. The conclusion is consistent with the experiments in the main body of the paper that LevT with ACT is not significantly better than LevT on unconstrained translation, though our main claim rests on the scenario of constrained NAT.

## Appendix B Case Study

The case study of LevT and LevT with ACT is presented in Table 8. In the case of unconstrained or soft constrained translation, LevT incorrectly translates low frequency constraint words (_e.g._, _Hihnterfeierm_ in case 1). In the case of hard constrained translation, LevT tends to have more interfering words around the constraint words (_e.g._, _sind_ in case 1). After incorporating ACT, we witness consistent improvements in the translation of the constraints for LevT, especially for soft constrained translation where it successfully translates given constraints. However, when the translation is not constrained on lexical terms (_i.e._, unconstrained translation), LevT with ACT still struggles at translating the term correctly (both case 1 and 2).

\begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Term\%**} & \multicolumn{2}{c}{**BLEU**} \\ \cline{3-4}  & & Full (3,003) & Con. (454) \\ \hline LevT\({}^{\dagger}\) & **79.53** & **26.95** & 29.95 \\  + ACT\({}^{\dagger}\) & 77.17 & 26.93 & **30.35** \\   LeV\(\nabla\) w/ soft & 94.88 & 27.04 & 30.38 \\  + ACT & **98.82** & **27.06** & **31.08** \\   LeV\(\nabla\) w/ hard & 100.00 & 27.06 & 30.49 \\  + ACT & 100.00 & **27.07** & **31.11** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Experiments on the test set of WMT14 En\(\rightarrow\)De task, which shares the same domain of training set. Following Susanto et al. (2020), “Con.” is the subset of WMT14-Full as shown in Table 3, where every sample has at least one lexical term as constraint.

\begin{table}
\begin{tabular}{|p{142.3pt}||} \hline \hline \multicolumn{2}{|c|}{**Case 1**} \\ \hline \hline \multicolumn{2}{|c|}{**Source**} \\ \hline However, carriages are also popular for hen parties, he commented. \\ \hline
**Target** Kutschen sind aber auch für Jungesellinnenabschiedde beleibt, meint er. \\ \hline
**Terminology Constraints** ben parties \(\rightarrow\) Jungesellinnenabschiede \\ \hline \hline \multicolumn{2}{|c|}{LeVT} \\ \hline
**Unconstrained translation** Kutschen sind aber auch für _Hihnreferäern_ beliebt, kommentière er. \\ \hline
**Soft constrained translation** Allerdings sind auch für _Hünnenabschebeliebelt_, kommentierté er. \\ \hline
**Hard constrained translation** Aber Auch für _Junggesillinnenabschiede_ sidnt beliebt, sagte er. \\ \hline \hline \multicolumn{2}{|c|}{LeVT + ACT} \\ \hline
**Unconstrained translation** Wagen sind aber auch für _Hühnerpartys_ beliebt, kommentiere er. \\ \hline
**Soft constrained translation** Kutschen sind aber auch fur _Junggesellinnenabschiede_beit, sagte er. \\ \hline
**Hard constrained translation** Kutschen sind aber auch für _Junggesellinnenabschiede_beleibt, sagte er. \\ \hline
**Hard constrained translation** Kutschen sind aber auch für _Junggesellinnenabschiede_beleibt, sagte er. \\ \hline \hline \multicolumn{2}{|c|}{**Case 2**} \\ \hline
**Source** The media also reported that several people injured. \\ \hline
**Target** Median berichtetten außedem von mehreren Verletzten. \\ \hline
**Terminology Constraints** injured \(\rightarrow\) Verletzen \\ \hline \hline \multicolumn{2}{|c|}{LeVT} \\ \hline
**Unconstrained translation** Die Medien berichtetten auch, dass mehrere Menschen verletzyt warden. \\ \hline
**Soft constrained translation** Die Medicien berichteten auch, dass mehrere Verletztre wanden. \\ \hline
**Word constrained translation** Die Medicien berichteten auch, dass mehrere Verletzte wunden. \\ \hline
**Unconstrained translation** Die Medien berichteten auch, dass mehrere Versletzyt wurden. \\ \hline
**Soft constrained translation** Die Medicien berichteten auch, dass mehrere Verletzten. \\ \hline
**Hard constrained translation** Die Medicien berichteten auch, dass mehrere Verletzten. \\ \hline \hline \multicolumn{2}{|c|}{**LvT + ACT**} \\ \hline
**Unconstrained translation** Die Medien berichteten auch, dass mehrere Menschen verletzyt wurden. \\ \hline
**Soft constrained translation** Die Medicien berichteten auch, dass mehrere Verletzten. \\ \hline
**Hard constrained translation** Die Medienic berichteten auch, dass mehrere Verletzten. \\ \hline \hline \end{tabular}
\end{table}
Table 8: Case study of LevT and LevT with ACT. Text in brown denotes the constraint word, text in red denotes the translation error of constraints, and \(\Rightarrow\) denotes analysis of the translation errors.

 



## Appendix C Unraveling the Buckets in Self-Constrained Translation

In this section, we dig further into the buckets in self-constrained translation (SS3.3, SS6.1), especially for understanding why Drop#1 happens.

As seen in Table 9, we categorize and count the constraints into five classes based on their Part-Of-Speech tagging with NLTK Bird et al. (2009). We find that, 1) punctuation (PUNK) dominates bucket 1; 2) as the constraint frequency decreases (from bucket 1 to bucket 6), the number of constraints identified as nouns (NN*) grows; 3) bucket 2 has the most UNK constraints. The third finding is because, as the BPE training was only done on the training set of the datasets, there will be \(<\)UNK\(>\) on the target side of the test set. Thus, cases in bucket 2 have a relatively large number of UNK tokens as constraints, resulting in the Drop#1.

To give a clearer view about how is UNK causing Drop#1, we exclude samples with UNK as constraints, and obtain a revised self-constrained translation results, as in Figure 5. Clearly, Drop#1 disappears in the given setting. Of course, Drop#2 still verifies our claim in the paper.

Figure 5: Extended self-constrained translation results on WMT14-WIKT by removing UNK as constraints. The settings are the same as in §6.1.