SlowGuess/NAACL_2022_OCR huggingface dataset:
Model	Term%	BLEU
Full (3,003)	Con. (454)
LevT†	79.53	26.95	29.95
+ ACT†	77.17	26.93	30.35
LevT w/ soft	94.88	27.04	30.38
+ ACT	98.82	27.06	31.08
LevT w/ hard	100.00	27.06	30.49
+ ACT	100.00	27.07	31.11
The case study of LevT and LevT with ACT is presented in Table 8. In the case of unconstrained or soft constrained translation, LevT incorrectly translates low frequency constraint words (e.g., Hühnerfeiern in case 1). In the case of hard constrained translation, LevT tends to have more interfering words around the constraint words (e.g., sind in case 1). After incorporating ACT, we witness consistent improvements in the translation of the constraints for LevT, especially for soft constrained translation where it successfully translates given constraints. However, when the translation is not constrained on lexical terms (i.e., unconstrained translation), LevT with ACT still struggles at translating the term correctly (both case 1 and 2).
Table 7: Experiments on the test set of WMT14 En→De task, which shares the same domain of training set. Following Susanto et al. (2020), "Con." is the subset of WMT14-Full as shown in Table 3, where every sample has at least one lexical term as constraint.
Case 1
Source
However, carriages are also popular for hen parties, he commented.
Target
Kutschen sind aber auch für Jungesellinnenabschiede beliebt, meint er.
Terminology Constraints
hen parties → Jungesellinnenabschiede
LevT
Unconstrained translation
Kutschen sind aber auch für Hühnerfeiern beliebt, kom-mentierte er. ⇒ wrong term
Soft constrained translation
Allerdings sind auch für Hinnenabschiedebeliebt, kommen-tierte er. ⇒ wrong term
Hard constrained translation
Aber Auch für Jungesellinnenabschiede sind beliebt, sagte er. ⇒ incomplete sentence
LevT + ACT
Unconstrained translation
Wagen sind aber auch für Hühnerpartys beliebt,kommen-tierte er. ⇒ wrong term
Soft constrained translation
Kutschen sind aber auch für Jungesellinnenabschiede beliebt, sagte er.
Hard constrained translation
Kutschen sind aber auch für Jungesellinnenabschiede beliebt, sagte er.
Case 2
Source
The media also reported that several people injured.
Target
Medien berichteten außer dem von mehreren Verletzten.
Terminology Constraints
injured → Verletzten
LevT
Unconstrained translation
Die Medien berichteten auch, dass mehrere Menschen verletzt wurden. ⇒ wrong term
Soft constrained translation
Die Medien berichteten auch, dass mehrere Verletzte[wurden. ⇒ wrong term
Hard constrained translation
Die Medien berichteten auch, dass mehrere Verletzte[wurden. ⇒ wrong term
LevT + ACT
Unconstrained translation
Die Medien berichteten auch, dass mehrere Menschen verletzt wurden. ⇒ wrong term
Soft constrained translation
Die Medien berichteten auch, dass mehrere Verletzten.
Hard constrained translation
Die Medien berichteten auch, dass mehrere Verletzten.
Table 8: Case study of LevT and LevT with ACT. Text in brown denotes the constraint word, text in red denotes the translation error of constraints, and  $\Rightarrow$  denotes analysis of the translation errors.
Bucket	# PUNC	# NN*	# (JJ*,RB*,VB*)	# UNK	# OTHER	# ALL
1	1,300	971	433	0	63	2,767
2	148	1,520	567	186	346	2,767
3	12	1,926	531	97	201	2,767
4	2	2,298	308	4	155	2,767
5	0	2,377	208	3	179	2,767
6	0	2,336	134	5	292	2,767
Table 9: The counted statistics of constraint tokens within each bucket in self-constrained translation study, where tokens are categorized according to their Part-Of-Speech tags. Among them, PUNC denotes punctuation; NN* denotes all sets of nouns (whose POS tags start with NN, including NN, NNP, NNS, NNPS, etc.); JJ*, RB* and VB* denotes all kinds of adjectives, adverbs and verbs; UNK is the constraint with UNK token and some special symbols; and others as denoted as OTHER.
(b) Sorting self-constraints by TF-IDF.
Figure 5: Extended self-constrained translation results on WMT14-WIKT by removing UNK as constraints. The settings are the same as in §6.1.
C Unraveling the Buckets in Self-Constrained Translation
In this section, we dig further into the buckets in self-constrained translation (§3.3, §6.1), especially for understanding why Drop#1 happens.
As seen in Table 9, we categorize and count the constraints into five classes based on their Part-Of-Speech tagging with NLTK (Bird et al., 2009). We find that, 1) punctuation (PUNK) dominates bucket 1; 2) as the constraint frequency decreases (from bucket 1 to bucket 6), the number of constraints identified as nouns  $(\mathrm{NN}^{*})$  grows; 3) bucket 2 has the most UNK constraints. The third finding is because, as the BPE training was only done on the training set of the datasets, there will be  $<\mathrm{UNK}>$  on the target side of the test set. Thus, cases in bucket 2 have a relatively large number of UNK tokens as constraints, resulting in the Drop#1.
To give a clearer view about how is UNK caus
ing Drop#1, we exclude samples with UNK as constraints, and obtain a revised self-constrained translation results, as in Figure 5. Clearly, Drop#1 disappears in the given setting. Of course, Drop#2 still verifies our claim in the paper.