SlowGuess/NAACL_2022_OCR huggingface dataset:
Chun Zeng\*, Jiangjie Chen\*, Tianyi Zhuang\*, Rui Xu\*, Hao Yang\*, Ying Qin\*, Shimin Tao\*, Yanghua Xiao\*
$\spadesuit$ Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University  $\diamondsuit$ Huawei Translation Services Center
\*Fudan-Aishu Cognitive Intelligence Joint Research Center {zengc20, jjchen19,tyzhuang20, shawyh}@fudan.edu.cn, ruixu21@m.fudan.edu.cn, {yanghao30,qinying,taoshimin}@huawei.com
Lexically constrained neural machine translation (NMT) draws much industrial attention for its practical usage in specific domains. However, current autoregressive approaches suffer from high latency. In this paper, we focus on non-autoregressive translation (NAT) for this problem for its efficiency advantage. We identify that current constrained NAT models, which are based on iterative editing, do not handle low-frequency constraints well. To this end, we propose a plug-in algorithm for this line of work, i.e., Aligned Constrained Training (ACT), which alleviates this problem by familiarizing the model with the source-side context of the constraints. Experiments on the general and domain datasets show that our model improves over the backbone constrained NAT model in constraint preservation and translation quality, especially for rare constraints. $^{1}$
Despite the success of neural machine translation (NMT) (Bahdanau et al., 2015; Vaswani et al., 2017; Barrault et al., 2020), real applications usually require the precise (if not exact) translation of specific terms. One popular solution is to incorporate dictionaries of pre-defined terminologies as lexical constraints to ensure the correct translation of terms, which has been demonstrated to be effective in many areas such as domain adaptation, interactive translation, etc.
Previous methods on lexically constrained translation are mainly built upon Autoregressive Translation (AT) models, imposing constraints at inference-time (Ture et al., 2012; Hokamp and Liu,
Source
Travellers screamed and children cried.
1.8K 24 2.8M 30.0K 122
Target
Reisende hatten geschrien und Kinder geweint.
944 9.9K 13 2.6M 20.1K 13
Terminology Constraints
scream → geschrien
Unconstrained translation
Reisende schieren und Kinder rieen. ⇒ wrong term
Soft constrained translation
Reisende rien. ⇒ incomplete sentence & wrong term
Hard constrained translation
Reisende geschieren. ⇒ incomplete sentence
Table 1: Translation examples of a lexically constrained non-autoregressive translation (NAT) model (Gu et al., 2019) under a low-frequency word as constraint. The underbraced word frequencies (uncased) are calculated from the vast WMT14 English-German translation (En-De) datasets (Vaswani et al., 2017).
2017; Post and Vilar, 2018) or training-time (Luong et al., 2015; Ailem et al., 2021). However, such methods either are time-consuming in real-time applications or do not ensure the appearance of constraints in the output. To develop faster MT models for industrial applications, Non-Autoregressive Translation (NAT) has been put forth (Gu et al., 2018; Ghazvininejad et al., 2019; Gu et al., 2019; Qian et al., 2021), which aims to generate tokens in parallel, boosting inference efficiency compared with left-to-right autoregressive decoding.
Researches on lexically constrained NAT are relatively under-explored. Recent studies (Susanto et al., 2020; Xu and Carpuat, 2021) impose lexical constraints at inference time upon editing-based iterative NAT models, where constraint tokens are set as the initial sequence for further editing. However, such methods are vulnerable when encountered with low-frequency words as constraints. As illustrated in Table 1, when translated with a rare
Our implementation can be found at https://github.com/sted-byte/ACT4NAT.
Human Language Technologies, pages 5777 - 5790
constraint, the model is unable to generate the correct context of the term "geschrien" as if it does not understand the constraint at all. It is dangerous since terms in specific domains are usually low-frequency words. We argue that the main reasons behind this problem are 1) the inconsistency between training and constrained inference and 2) the unawareness of the source-side context of the constraints.
To solve this problem, we build our algorithm based on the idea that the context of a rare constraint tends not to be rare as well, i.e., "a stranger's neighbors are not necessarily strangers", as demonstrated in Table 1. We believe that, when the constraint is aligned to the source text, the context of its source-side counterpart can be utilized to be translated into the context of the target-side constraint, even if the constraint itself is rare. Also, when enforced to learn to preserve designated constraints at training-time, a model should be better at coping with constraints during inference-time.
Driven by these motivations, we propose a plugin algorithm to improve constrained NAT, namely Aligned Constrained Training (ACT). ACT extends the family of editing-based iterative NAT (Gu et al., 2019; Susanto et al., 2020; Xu and Carpuat, 2021), the current paradigm of constrained NAT. Specifically, ACT is composed of two major components: Constrained Training and Alignment Prompting. The former extends regular training of iterative NAT with pseudo training-time constraints into the state transition of imitation learning. The latter incorporates source alignment information of constraints into training and inference, indicating the context of the potentially rare terms.
In summary, this work makes the following contributions:
- We identify and analyse the problems w.r.t. rare lexical constraints in current methods for constrained NAT;
- We propose a plug-in algorithm for current constrained NAT models, i.e., aligned constrained training, to improve the translation under rare constraints;
- Experiments show that our approach improves the backbone model w.r.t. constraint preservation and translation quality, especially for rare constraints.
Lexically Constrained Translation Existing translation methods impose lexical constraints during either inference or training. At training time, constrained MT models include code-switching data augmentation (Dinu et al., 2019; Song et al., 2019; Chen et al., 2020) and training with auxiliary tasks such as token or span-level mask-prediction (Ailem et al., 2021; Lee et al., 2021). At inference time, autoregressive constrained decoding algorithms include utilizing placeholder tag (Luong et al., 2015; Crego et al., 2016), grid beam search (Hokamp and Liu, 2017; Post and Vilar, 2018) and alignment-enhanced decoding (Alkhouli et al., 2018; Song et al., 2020; Chen et al., 2021). For the purpose of efficiency, recent studies also focus on non-autoregressive constrained translation. Susanto et al. (2020) proposes to modify the inference procedure of Levenshtein Transformer (Gu et al., 2019) where they disallow the deletion of constraint words during iterative editing. Xu and Carpuat (2021) further develops this idea and introduces a reposition operation that can reorder the constraint tokens. Our work absorbs the idea of both lines of work. Based on NAT methods, we brings alignment information by terminologies to help learn the contextual information for lexical constraints, especially the rare ones.
Non-Autoregressive Translation Although enjoy the speed advantage, NAT models suffer from performance degradation due to the multi-modality problem, i.e., generating text when multiple translations are plausible. Gu et al. (2018) applies sequence-level knowledge distillation (KD) (Kim and Rush, 2016) that uses an AT's output as an NAT's new target, which reduces word diversity and reordering complexity in reference, resulting in fewer modes (Zhou et al., 2020; Xu et al., 2021). Various algorithms have also been proposed to alleviate this problem, including incorporating latent variables (Kaiser et al., 2018; Shu et al., 2020), iterative refinement (Ghazvininejad et al., 2019; Stern et al., 2019; Gu et al., 2019; Guo et al., 2020), advanced training objective (Wang et al., 2019; Du et al., 2021) and gradually learning target-side word inter-dependency by curriculum learning (Qian et al., 2021). Our work extends the family of editing-based iterative NAT models for its flexibility to impose lexical constraints (Susanto et al., 2020; Xu and Carpuat, 2021).
Action	Implementation
Insertion	Placeholder Classifier: predicts the number of tokens (0 ~ Kmax) to be inserted at every consecutive position pairs and then inserts the corresponding number of [PLH]. Token Classifier: predicts the actual target token of the [PLH].
Deletion	Deletion Classifier: predicts whether each token (except for the boundaries) should be “kept” or “deleted”.
Table 2: The implementation details of insertion and deletion operations in LevT.
3.1 Non-Autoregressive Translation
Given a source sentence as  $\pmb{x}$  and a target sentence as  $\pmb{y} = \{y_1, \dots, y_n\}$ , an AT model generates in a left-to-right order, i.e., generating  $y_t$  by conditioning on  $\pmb{x}$  and  $y_{<t}$ . An NAT model (Gu et al., 2018), however, discards the word inter-dependency in output tokens, with the conditional independent probability distribution modeled as:
$$
P (\boldsymbol {y} | \boldsymbol {x}) = \prod_ {t = 1} ^ {n} P \left(y _ {t} | \boldsymbol {x}\right). \tag {1}
$$
Such factorization is featured with high efficiency at the cost of performance drop in translation tasks due to the multi-modality problem, i.e., translating in mixed modes and resulting in token repetition, missing, or incoherence.
3.2 Editing-based Iterative NAT
Iterative refinement by editing is an NAT paradigm that suits constrained translations due to its flexibility. It alleviates the multi-modality problem by being autoregressive in editing previously generated sequences while maintaining non-autoregressiveness within each iteration. Thus, it achieves better performance than fully NATs while is faster than ATs.
Levenshtein Transformer To better illustrate our idea, we use Levenshtein Transformer (LevT, Gu et al., 2019) as the backbone model in this work, which is a representative model for constrained NAT based on iterative editing.
LevT is based on the Transformer architecture (Vaswani et al., 2017), but more flexible and fast than autoregressive ones. It models the generation of sentences as Markov Decision Process (MDP) defined by a tuple  $(\mathcal{V},\mathcal{A},\mathcal{E},\mathcal{R},\boldsymbol{y}^0)$ . At each decoding iteration, the agent  $\mathcal{E}$  receives an input  $\boldsymbol{y}\in \mathcal{V}$
chooses an action  $a \in \mathcal{A}$  and gets reward  $r$ .  $\mathcal{V}$  is a set of discrete sentences and  $\mathcal{R}$  is the reward function.  $\pmb{y}^{0} \in \mathcal{A}$  is the initial sentence to be edited.
Each iteration consists of two basic operations, i.e., deletion and insertion, which is described in Table 2. For the  $k$ -th iteration of the sentence  $\pmb{y}^{k} = (
, y_{1}, \dots, y_{n}, 
)$ , the insertion consists of placeholder and token classifiers, and the deletion is achieved by a deletion classifier. LevT trains the model with imitation learning to insert and delete, which lets the agent imitate the behaviors drawn from the expert policy:
- Learning to insert: edit to reference by inserting tokens from a fragmented sentence (e.g., random deletion of reference).
- Learning to delete: delete from the insertion result of the current training status to the reference.
The key idea is to learn how to edit from a ground truth after adding noise or the output of an adversary policy to the reference. The ground truth of the editing process is derived from the Levenshtein distance (Levenshtein, 1965).
Lexically Constrained Inference Lexical constraints can be imposed upon a translation model in: 1) soft constraints: allowing the constraints not to appear in the translation; and 2) hard constraints: forcing the constraints to appear in the translation. In NAT, the constraints are generally incorporated at inference time. Susanto et al. (2020) injects constraints as the initial sequence for iterative editing in Levenshtein Transformer (LevT, Gu et al., 2019), achieving soft constrained translation. And hard constrained translation can be easily done by disallowing the deletion of the constraints. Xu and Carpuat (2021) alters the deletion action in LevT with the reposition operation, allowing the reordering of multiple constraints.
3.3 Motivating Study: Self-Constrained Translation
According to Table 1, constrained NAT models seem to suffer from the low-frequency of lexical constraints, which is dangerous as most terms in practice are rare. To further explore the impact of constraint frequency upon NATs, we conduct a preliminary analysis on constrained LevT (Susanto et al., 2020). We sort words in each reference text based on frequency, dividing them into six buckets by frequency order (as in Figure 1), and sample a
Figure 1: Ablation study of self-constrained translation on WMT14 En→De test set with Wiktionary terminology constraints (Dinu et al., 2019). The absolute average frequencies of self-constraints in a bucket decrease from left to right in the x-axis.
word from each bucket as lexical constraints for translation.2 We denote these constraints as self-constraints. In this way, we have six times the data, and the six samples derived from one raw sample only differ in the lexical constraints.
As shown in Figure 1, translation performance generally keeps improving as the self-constraint gets rarer. This is because setting low-frequency words in a sentence as constraints, which are often hard to translate, actually lightens the load of an NAT model. However, there are two noticeable performance drops around relative frequency ranges of  $10\% -30\%$  (bucket 2) and  $90\% -100\%$  (bucket 6), denoted as  $Drop\# 1$  (-0.3 BLEU) and  $Drop\# 2$  (-0.6 BLEU). Note that  $Drop\# 1$  is mainly due to the fact that there are mostly unknown tokens (i.e.,  $\langle \mathrm{UNK}\rangle$ ) in the bucket 2. We leave detailed discussions about buckets and  $Drop\# 1$  to Appendix C.
In this experiment, we are more interested in the reasons for Drop#2 when constraints are low-frequency words. We assume a trade-off in self-constrained NAT: the model does not have to translate rare words as they are set as an initial sequence (constraints), but it will have a hard time understanding the context of the rare constraint due to 1) the rareness itself and 2) the lack of the alignment information between target-side constraint tokens and source tokens. Thus, the model does not know how many tokens should be inserted to the left and right of the constraint, which is consistent with the findings in Table 1.
The findings and assumptions discussed above motivate us to propose a plug-in algorithm for lexically constrained NAT models, i.e., Aligned Constrained Training (ACT). ACT is designed based on two major ideas: 1) Constrained Training: bridging the discrepancy between training and constrained inference; 2) Alignment Prompting: helping the model understand the context of the constraints.
4.1 Constrained Training
As introduced in §3.2, constraints are typically imposed during inference time in NAT (Susanto et al., 2020; Xu and Carpuat, 2021). Specifically, lexical constraints are imposed by setting the initial sequence  $\mathbf{y}^0$  as  $(<\mathrm{S}>, C_1, C_2, \ldots, C_k, </\mathrm{S}>)$ , where  $C_i = (c_1, c_2, \ldots, c_l)$  is the  $i$ -th lexical constrained word,  $l$  is the number of tokens in the  $i$ -th constraint, and  $k$  is the number of constraints.
However, such mandatory preservation of the constraints is not carried out during training. During imitation learning, random deletion is applied for ground-truth  $\pmb{y}^{*}$  to get the incomplete sentences  $\pmb{y}'$ , producing the data samples for expert policies of how to insert from  $\pmb{y}'$  to  $\pmb{y}^{*}$ . This leads to a situation where the model does not learn to preserve fixed tokens and organize the translation around the tokens. Such discrepancy could harm the applications of soft constrained translation.
To solve this problem, we propose a simple but effective Constrained Training (CT) algorithm. We first build pseudo terms from the target by sampling 0-3 words (more tokens after tokenization) from reference as the pre-defined constraints for training.<sup>3</sup> Afterward, we disallow the deletion of pseudo term tokens during building data samples for imitation learning. This encourages the model to edit incomplete sentences containing lexical constraints into complete ones, bridging the gap between training and inference.
4.2 Alignment Prompting
As stated in §3.3, we assume the rareness of constraints hinders the model to insert proper tokens of its contexts (i.e., a stranger's neighbors are also strangers). To make the matter worse, previous research (Ding et al., 2021) has also shown that lexical choice errors on low-frequency words tend
<sup>3</sup>In the experiments, these pseudo constraints are sampled based on TF-IDF score to mimic the rare but important terminology constraints in practice.
Sentences that cannot be divided into six buckets are removed.
Figure 2: An example of alignment prompting. The constraint tokens  $y_*$  are given by users during inference, and can also be sampled from target sentence during training. Given  $y_*$ , we align them with tokens  $x_*$  in the source and build alignment embeddings to be fed into the encoder.
to be propagated from the teacher (an AT model) to the student (an NAT model) in knowledge distillation.
However, terminologies, by nature, provide hard alignment information for source and target which the model can conveniently utilize. Thus, on top of constrained training, we propose an enhanced approach named Aligned Constrained Training (ACT). As illustrated in Figure 2, we propose to directly align the target-side constraints with the source words and prompt the alignment information to the model during both training and inference.
Building Alignment for Constraints We first align the source words to the target-side constraints, which are either pseudo constraints during training or actual constraints during inference. For each translated sentence constraints  $\mathcal{C}_{\mathrm{tgt}} = (C_1,C_2,\dots,C_k)$ , we use an external alignment tool external aligner, such as GIZA++ (Brown et al., 1993; Och and Ney, 2003), to find the corresponding source words, denoted as  $\mathcal{C}_{\mathrm{src}} = (C_1',C_2',\dots,C_k')$ .
Prompting Alignment into LevT The encoder in LevT, besides token embedding and position embedding, is further added with a learnable align-
Dataset (test set)	# Sent.	Avg. Len. of Con.	Avg. Con. Freq.
WMT14-WIKT	454	1.15	25,724.73
WMT17-IATE	414	1.09	3,685.42
WMT17-WIKT	728	1.22	26,252.70
OPUS-EMEA	2,996	1.95	2,187.63
OPUS-JRC	2,984	1.99	3,725.71
Table 3: Statistics of the test sets with target-side lexical constraints. "Avg. Len. of Con." denotes the average number of words in a constraint. "Avg. Con. Freq." is the average frequency of lexical constraints calculated with the training vocabularies of corresponding language.
ment embedding that comes from  $\mathcal{C}_{\mathrm{src}}$  and  $\mathcal{C}_{\mathrm{tgt}}$ . We set the alignment value for each token in  $C_i^\prime$  to  $i$  and the others to 0, which are further encoded into embeddings. The prompting of alignment is not limited to training, as we also add such alignment embeddings to source tokens aligned to target-side constraints during inference.
5.1 Data and Evaluation
Parallel Data and Knowledge Distillation We consider the English  $\rightarrow$  German (En  $\rightarrow$  De) translation task and train all of the MT models on WMT14 En-De (3,961K sentence pairs), a benchmark translation dataset. All sentences are pre-processed via byte-pair encoding (BPE) (Sennrich et al., 2016) into sub-word units. Following the common practice of training an NAT model, we use the sentence-level knowledge distillation data generated by a Transformer, (Vaswani et al., 2017) provided by Kasai et al. (2020).
Datasets with Lexical Constraints Given models trained on the above-mentioned training sets, we evaluate them on the test sets of several lexically constrained translation datasets. These test sets are categorized into two types of standard lexically constrained translation datasets: 1) Type#1: tasks from WMT14 (Vaswani et al., 2017) and WMT17 (Bojar et al., 2017), which are of the same general domain (news) as training sets; 2) Type#2: tasks from OPUS (Tiedemann, 2012) that are of specific domains (medical and law). Particularly, the real application scenarios of lexically constrained MT models are usually domain-specific, and the constrained words in these domain datasets are relatively less frequent and more important.
Models	WMT17-IATE	WMT17-WIKT	WMT14-WIKT	Latency (ms)
Term%	BLEU	Term%	BLEU	Term%	BLEU
Reported results in previous work
Transformer (Vaswani et al., 2017)†	79.65	29.58	79.75	30.80	76.77	31.75	244.5
DBA (Post and Vilar, 2018)	82.00	25.30	99.50	25.80	-	-	434.4
Train-by-rep (Dinu et al., 2019)	94.50	26.00	93.40	26.30	-	-	-
LevT (Gu et al., 2019)†	80.31	28.97	81.11	30.24	80.23	29.86	92.0
w/ soft constraint (Susanto et al., 2020)	93.81	29.73	93.44	30.82	94.43	29.93	-
w/ hard constraint (Susanto et al., 2020)	100.00	30.13	100.00	31.20	100.00	30.49	-
EDITOR (Xu and Carpuat, 2021)†	83.00	27.90	83.50	28.80	-	-	121.7
w/ soft constraint	97.10	28.80	96.80	29.30	-	-	-
w/ hard constraint	100.00	28.90	99.80	29.30	-	-	134.1
Our implementation
LevT†	78.32	29.80	80.20	30.75	79.53	29.95	71.9
+ constrained training (CT)†	78.76	29.46	80.77	30.82	79.13	30.24	78.6
+ aligned constrained training (ACT)†	79.43	29.57	80.20	30.63	77.17	30.35	77.0
LevT w/ soft constraint	94.25	30.11	93.78	30.92	94.88	30.38	79.5
+ constrained training (CT)	96.24	30.19	96.61	30.96	97.44	31.01	75.4
+ aligned constrained training (ACT)	96.90	30.56	97.62	31.06	98.82	31.08	76.3
LevT w/ hard constraint	100.00	30.31	100.00	30.65	100.00	30.49	82.7
+ constrained training (CT)	100.00	30.31	100.00	30.99	100.00	31.01	78.1
+ aligned constrained training (ACT)	100.00	30.68	100.00	31.18	100.00	31.11	77.0
Table 4: Translation results with lexical constraints. Term% is the constraint term usage rate. Method† translates without lexical constraints in input.
Following previous work (Dinu et al., 2019; Susanto et al., 2020; Xu and Carpuat, 2021), the lexical constraints in Type#1 tasks are extracted from existing terminology databases such as Interactive Terminology for Europe (IATE)<sup>4</sup> and Wiktionary (WIKT)<sup>5</sup> accordingly. The OPUS-EMEA (medical domain) and OPUS-JRC (legal domain) in Type#2 tasks are datasets from OPUS. The constraints are extracted by randomly sampling 1 to 3 words from the reference (Post and Vilar, 2018). These constraints are then tokenized with BPE, yielding a larger number of tokens as constraints. The statistical report is shown in Table 3, indicating the frequencies of Type#2 datasets are generally much lower than Type#1 ones.
Evaluation Metrics We use BLEU (Papineni et al., 2002) for estimating the general quality of translation. We also use Term Usage Rate (Term%, Dinu et al., 2019; Susanto et al., 2020; Lee et al., 2021) to evaluate lexically constrained translation, which is the ratio of term constraints appearing in the translated text.
We use Levenshtein Transformer (LevT, Gu et al., 2019) as the backbone model to ACT algorithm for constrained NAT. We compare our approach with a series of previous MT models on applying lexical
constraints:
- Transformer (Vaswani et al., 2017), set as the AT baseline;
- Dynamic Beam Allocation (DBA) (Post and Vilar, 2018) for constrained decoding with dynamic beam allocation over Transformer;
- Train-by-sep (Dinu et al., 2019), trained on augmented code-switched data by replacing the source terms with target constraints or append on source terms during training;
- Constrained LevT (Susanto et al., 2020), which develops LevT (Gu et al., 2019) by setting constraints as initial editing sequence;
- EDITOR (Xu and Carpuat, 2021), a variant of LevT, replacing the delete action with a reposition action.