SlowGuess/NAACL_2022_OCR huggingface dataset:
Jessy Lin Geza Kovacs Aditya Shastry Joern Wuebker John DeNero University of California, Berkeley Google Lilt jessy_lin@berkeley.edu, joern@lilt.com, john@lilt.com
We introduce translation error correction (TEC), the task of automatically correcting human-generated translations. Imperfections in machine translations (MT) have long motivated systems for improving translations post-hoc with automatic post-editing. In contrast, little attention has been devoted to the problem of automatically correcting human translations, despite the intuition that humans make distinct errors that machines would be well-suited to assist with, from typos to inconsistencies in translation conventions. To investigate this, we build and release the ACED corpus with three TEC datasets<sup>1</sup>. We show that human errors in TEC exhibit a more diverse range of errors and far fewer translation fluency errors than the MT errors in automatic post-editing datasets, suggesting the need for dedicated TEC models that are specialized to correct human errors. We show that pre-training instead on synthetic errors based on human errors improves TEC F-score by as much as 5.1 points. We conducted a human-in-the-loop user study with nine professional translation editors and found that the assistance of our TEC system led them to produce significantly higher quality revised translations.
Despite recent progress in machine translation (MT), a tremendous amount of translated content in the world is still written by humans (DePalma, 2021). Humans are often assumed to produce trusted, high-quality translations. In reality, they do make errors, including spelling, grammar, and translation errors (Hansen, 2009). This paper introduces the task of translation error correction (TEC). Given a source sentence  $s$  and a human-generated translation  $t$ , the goal of TEC is to produce an improved translation  $t'$  by correcting all errors in  $t$ .
"Translation correction" has long been studied in the MT community through the task of automatic post-editing (APE), which aims to correct errors in machine-generated translations (Simard et al., 2007). TEC is structurally identical to APE. However, it requires modeling a different data distribution: errors made by humans, which differ from those made by MT systems (Freitag et al., 2021). We characterize the error distribution in TEC by building, analyzing, and releasing the ACED corpus, a collection of three TEC datasets from varying domains, with a total of 35,261 English-German translations produced and corrected by professional translators in the natural course of their work. While APE is dominated by the fluency errors that are characteristic of MT systems (74% of sentences), our TEC corpus exhibits a broader distribution of errors that human translators are prone to make.
Using this error analysis, we propose an approach for TEC that pre-trains on synthetic corruptions more similar to errors made by humans, outperforming models that were developed for the related tasks of MT, grammatical error correction, and APE on all ACED datasets.
The task of TEC is often currently performed by humans, e.g. translators hired to review and edit translations ("reviewers"). Can a TEC system help reviewers edit faster, or produce higher quality final translations than they would have without assistance? We ran a human-in-the-loop user study with nine professional translators using our best-performing TEC model. We found that the reviews produced when assisted with a TEC system were rated as higher quality than those produced without, and produced with less manual effort. Qualitatively, users commented that trust and consistency of the suggestions were critical. They speculated future automated assistance could be helpful for onboarding to new content, spotting technical errors, and improving their own awareness of errors to catch.
Dataset available at: github.com/lilt/tec
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 494 - 507 July 10-15, 2022 ©2022 Association for Computational Linguistics
Error Type	Example Text
Monolingual: typos	s: Do your feet roll inwards when running?
t: Klappen keine Fälle beim Laufen nachinnen?
t': Klappen keine Fälle beim Laufen nachinnen?
Monolingual: grammar	s: Own tough winter runs in the ...
t: Bei harten Winterläufe sorgt der ...
t': Bei harten Winterläufe sorgt der ...
Monolingual: fluency	s: The traffic emerges from the VPN server and ...
t: Der Verkehr wird vom VPN-Server ausgegeben und ...
t': Der Datenverkehr wird vom VPN-Server ausgegeben und ...
Bilingual	s: Quad Core XEON E3-1501M, 2.9GHz
t: Quad Core XEON 2,9 GHz
t': Quad Core XEON E3-1501M, 2,9 GHz
Preferential	s: VersaMax I / O auxiliary spring clamp style
t: VersaMax Zusammen-E / A Federklemmenart
t': VersaMax Zusammen-E / A Federklemmenbauform
Table 1: Error taxonomy for the ACED corpus, with examples from the dataset.
Looking forward, a natural question arises of whether the research community should focus on learning to revise model outputs (APE) or human outputs (TEC). With recent improvements in MT, it has been increasingly difficult for APE models to improve model output that is already high quality (Chollampatt et al., 2020). On the other hand, we should expect that humans will continue to make errors. TEC models will continue to provide benefit as a way of assisting humans, whether for professional translation work or everyday language learners. TEC is synergistic with continuing advancements in MT: improved MT will lead to improved error correction for human-generated translations. While APE pits models against models, TEC is an opportunity to combine the best of humans and models because humans and models make different errors.
In sum, this paper revisits the notion of "translation correction" conceived narrowly as the MT-centered task of APE, with an empirical investigation of translation error correction (TEC), the task of learning to correct human translations. Our contributions are:
1. We release ACED, the first corpus for TEC containing three datasets of human translations and revisions generated naturally from a commercial translation workflow.
2. We analyze the kinds of errors humans make in ACED, finding that while APE is dominated by correcting translation fluency, TEC focuses on correcting a broader range of errors that
appear in translation.
3. We propose a pre-training approach for TEC that outperforms approaches developed for similar tasks such as APE. Together, our results suggest the need for distinct approaches to correct human translation errors.
4. We perform a human-in-the-loop user study, finding that professional translators produce higher quality translations when assisted by a TEC model.
2 The  $\spadesuit$  ACED Corpus for TEC
Given a source language sentence  $s$  and a human-generated translation  $t$ , the goal of TEC is to produce a corrected target language sentence  $t'$ .
We introduce the ACED corpus, a set of three TEC datasets: ASICS, EMERSON, and DIGITALO-CEAN (DO), each consisting of English-German sentence triples  $(s,t,t^{\prime})$  from varying domains.
ACED is a real-world benchmark, containing naturalistic data from a task humans perform, rather than manually annotated data. All translations were created by professional translators working with Lilt, a localization services provider. All translators have at least 5 years of professional translation experience and experience working with the customer and domain. Each document was translated from scratch (i.e. not post-edited) by a human translator using an interactive neural MT system. Each translated document was then reviewed by a reviewer, who Lilt selects as one of the more senior translators. As a result, the examples in our corpus exhibit
Dataset	Domain	# sentences	% edited	# edits (mean)	Edit distance (mean)
train	dev	test
ASICS	Marketing	1395	525	616	29 %	1.6	7.5
EMERSON	Technical	4287	1255	1662	20 %	1.5	5.8
DIGITALOCEAN (DO)	Technical	11773	6104	7644	8 %	1.5	7.1
Table 2: Corpus statistics for each dataset in the ACED corpus, including edit statistics on % of sentences that edited, and for edited sentences, the average number of edits and average edit distance.
real errors that translators make, and the corrected translations are publication quality.
Secondly, ACED is diverse, with the three datasets from varying domains exhibiting different error distributions and difficulty for initial work on the TEC task. Information for each dataset is shown in Table 2. The ASICS dataset consists of marketing content with product names and descriptions for an activewear company. EMERSON consists of industrial product names for a manufacturing company. DO consists of software engineering tutorials. The various content types pose different challenges for translators and thus for TEC systems, which we discuss in Section 2.1 and Section 2.2.
Duplicate sentences with the same source  $s$  were removed. A portion of sentences were rewritten by the reviewer rather than being edited (a relative edit distance of more than  $25\%$  and a minimum of two edited words). We replace  $t$  in these sentences with  $t'$  in the corpus, so that training and evaluation focuses on local edits rather than re-translations. The train, dev, and test splits were constructed by splitting along document boundaries.
2.1 How do TEC and APE errors differ?
To understand how the human errors in the TEC task differ from model errors in APE, we compare the types of errors in ACED with 100 randomly sampled errors in the WMT 2021 APE shared task dev set $^{2}$ , which we then annotate with error types. We define an error taxonomy that classifies each edit as one of three types: (1) Monolingual edits are identifiable from only the target-side text. We divide these further into subcategories that highlight different capabilities needed to correct edits: typos (including spelling, punctuation, spacing, orthographic issues), grammar, and fluency (awkward phrasing, word choice, or non-native-sounding disfluencies); (2) Bilingual edits concern mismatches between the source and target text, e.g. over- or under-translation, mis-translations; (3) Preferen
tial edits correct text that is inconsistent with the preferences of the customer, as described in extralinguistic project requirements (e.g. terminology or stylistic preferences). Examples of each error type are shown in Table 1. Our error taxonomy closely mirrors those of previous analyses of human translation errors (Specia and Shah, 2014; Yuan and Sharoff, 2020; Gupta et al., 2021), and we confirm their findings that human translation errors differ from MT errors. However, while previous work focuses on error detection and quality estimation, TEC is concerned with error correction. Our error types are intended to isolate the capabilities that models need to learn to correct edits (e.g., target-side language models can learn to correct monolingual errors, but cannot do well on bilingual edits).
We annotate and release error labels for all test sentences in ASICS to enable its use as a diagnostic set for per-type evaluation of models. On the larger EMERSON and DO datasets, we randomly sample 50 errors to annotate for this analysis. Error types were annotated by a professional German translator. Each segment can have multiple error types. In Table 3, we report the percentage of sentences with at least one error of each type.  $74\%$  of sentences in APE exhibit a fluency error, in contrast to up to  $22\%$  of sentences in ACED, while other types like monolingual grammar, bilingual, and preferential errors are notably underrepresented in APE. We also note that all sentences in the APE shared task are edited, while a key feature of TEC is identifying when a sentence does not need to be edited. The error distributions suggest that different modeling techniques may shine in each: while APE challenges models to correct disfluent translations characteristic of MT systems, our task is designed to focus on identifying and correcting the typos, mismatches, and grammatical errors more commonly exhibited by humans. Guided by this observation, we describe an approach in Section 3 that pre-trains on synthetic edits that are more representative of this
$^{2}$ https://www.statmt.org/wmt21/ape-task.html
TEC	APE
ASICS	EMERSON	DO	WMT '21
Monolingual
typos	13	16	22	16
grammar	41	4	2	6
fluency	22	0	20	74
Bilingual	22	70	32	5
Preferential	7	24	40	6
Table 3: Percentages of erroneous sentences that contain at least one error of each type for ASICS, EMERSON, DO, and the dev set of the WMT 2021 APE shared task. As a task, APE exhibits many more fluency errors than TEC.
error distribution.
2.2 How difficult is it to learn to edit?
To quantify how difficult it may be to learn the correct edits in ACED, we report statistics on edit overlap: what proportion of edits that we expect models to perform (e.g. adding a comma) appear exactly in the training set? We use the errant toolkit to identify discrete edits (Bryant et al., 2017). Each edit is represented as a tuple (original span, replacement span), e.g. ("auf", "an") to replace "auf" with "an." Edit statistics are reported in Table 4: in ASICS and DO,  $\sim 20\%$  of the total number of edits in dev and test appear in the training set, while EMERSON has  $\sim 60\%$  of dev and test edits appear in the training set.
While the edit overlap rate provides a relative sense of scale for precision and recall numbers, it does not provide an upper bound on recall. It is possible to learn edits that do not exactly appear in the training set. For example, capitalizing product names ("winterized"  $\rightarrow$  "WINTER-IZED") is a learnable pattern that would appear as many distinct edits. Additionally, some errors can be corrected without fine-tuning because they are generic typo, grammatical, fluency, or bilingual errors. Conversely, it is also possible that it is wrong to make an edit that appears in the training set, depending on the surrounding sentential context.
3 Approaches to TEC
We propose a TEC model and compare it to several models designed for related tasks to determine whether they are also effective for TEC. An overview of the differences between the models is shown in Figure 1. All models use the Transformer neural architecture (Vaswani et al., 2017) that generates the target sequence  $t'$  from left to right. All are pre-trained on 36M sentences from the WMT18
ASICS	EMERSON	DO
Train	Total Edits	606	1436	1212
Unique Edits	418	486	940
Dev	Total Edits	246	381	1004
% in train	14	63	21
Test	Total Edits	287	364	766
% in train	23	60	21
Table 4: Edit and edit overlap statistics for each ACED dataset: total number of edits, unique edits in each train split, and percentage of total edits in each dev and test split that appear in the train split.
translation task $^3$  and fine-tuned on ACED, unless indicated otherwise. All pre-training and fine-tuning data is pre-processed by normalizing punctuation with the Moses toolkit (Koehn et al., 2007).
All models have 6 encoder and decoder layers, model dimension of 256, feed-forward dimension of 512, and 8 attention heads. We use a joint English-German vocabulary with 33k byte pair encoding subwords (Sennrich et al., 2016). During pre-training, we set the dropout to 0.1 and use the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 0.0002. During fine-tuning, we decrease the learning rate to 0.0001 and reset the Adam momentum parameters. We select the best fine-tuning checkpoint with edit-level  $\mathrm{F}_{0.5}$  score on our dev set. We use greedy inference.
3.1 Dual-Source Encoder-Decoder Model
We first describe the dual-source encoder-decoder we use for the APE and TEC models. Formally, the original Transformer architecture (Vaswani et al., 2017) takes a sequence of  $J$  source tokens  $\mathbf{s}_{1\dots J}$  and predicts a sequence of  $I'$  target tokens  $\mathbf{t}_{1\dots I'}^{\prime}$ . We adapt the architecture to additionally encode the original translation  $t$ , a sequence of  $I$  tokens,  $\mathbf{t}_{1\dots I}$ . We independently project  $\mathbf{t}$  into the embedding space, add an offset vector  $\mathbf{o}$ , and then concatenate the embedding with the embedding of the source s to form the encoder input. To allow the dual-source model to copy tokens from the original translation  $t$ , we implement the copy-mechanism proposed by Zhao et al. (2019), which augments the model with an additional encoder-decoder attention layer. An expanded description of the model can be found in Appendix Section A.
3http://www.statmt.org/wmt18
Figure 1: Overview of model architectures, pre-training data, and fine-tuning data for each approach to TEC. Transformer encoders and decoders are depicted as yellow and blue rectangles, respectively. The GEC and TEC models are pre-trained with synthetic corruptions of  $t'$  ( $t_{\mathrm{corrupted}}'$ ), as detailed in the description of the TEC and GEC models. APE uses MT-generated translations of  $s$  ( $t_{\mathrm{MT}}$ ) as synthetic data. BERT-APE is a state-of-the-art pre-trained APE model made available by Correia and Martins (2019).
3.2 Synthetic Data Generation
For the TEC and GEC model, we generate synthetic triples  $(s,t,t^{\prime})$  for pre-training. We generate a synthetic  $t$  by corrupting the German side of the translation data into  $t_{\mathrm{corrupted}}^{\prime}$ . For each sentence, we sample the probability of corruption  $p_c\sim \mathcal{N}(\mu = 0.01,\sigma = 0.04)$  clipped at 0. On each character and word in that sentence, with probability  $p_c$ , we randomly select one of the following perturbations to apply at that position: insertion, deletion, transposition, repetition.
3.3 TEC Models
The five approaches we compare are:
TEC (this work) We implement the dual-source encoder-decoder model that encodes two inputs  $(s, t)$  and outputs  $t'$ , as described in Section 3.1, and then pre-train on synthetic data generated with the procedure in Section 3.2. We then fine-tune on ACED.
MT We train an English-German neural machine translation model (with the standard architecture described previously) and fine-tune it on  $(s, t')$  ACED pairs, ignoring the original translation  $t$ .
GEC We evaluate a encoder-decoder (monolingual) GEC model that takes an incorrect German sentence  $t$  as input and outputs a corrected  $t'$ . We use the same copy mechanism to attend to  $t$  as our TEC model. To pre-train, we perturb  $t'$  using the procedure described in Section 3.2, throwing away the source side to obtain  $(t, t') = (t_{\text{corrupted}}, t')$  pairs. We then fine-tune on the ACED corpus, ignoring  $s$ .
APE We implement a dual-source encoder-decoder model that is identical to our TEC model. Following common practice in APE (Junczys-Dowmunt and Grundkiewicz, 2016; Negri et al., 2018), we pre-train on synthetic "post-editing" triples  $(s, t, t')$  where  $t = t_{\mathrm{MT}}$  is generated by translating  $s$  with an MT system. We split the training dataset into two parts, train an MT model on each half, and use each model to translate the other half of the dataset not seen during training. We then fine-tune on ACED.
BERT-APE We also evaluate whether a state-of-the-art APE model can be directly applied to our task. We evaluate the BERT-based encoder-decoder of Correia and Martins (2019), on which the WMT 2019 shared task winner was based (Lopes et al., 2019). Following Correia and Martins (2019), we fine-tuned on 23K English-German SMT triplets from the WMT18 shared task $^4$ . We reproduce their results on the APE shared task test sets, and continue fine-tuning this model on ACED. Following their paper, the inputs are pre-processed by tokenizing and joining the two inputs with a separator to form  $(s[\mathrm{SEP}]t,t^{\prime})$  pairs.
The primary metric for TEC is MaxMatch scores  $(\mathbf{M}^2)$  (Dahlmeier and Ng, 2012) computed with the errant toolkit (Bryant et al., 2017).  $\mathbf{M}^2$  is a standard metric for GEC that aligns  $t$  and  $t'$  to extract discrete " edits." We choose to follow the GEC evaluation practice of up-weighting precision by comparing  $\mathrm{F}_{0.5}$ , since the original translation is
<sup>4</sup>http://www.statmt.org/wmt18/ape-task.html, with 12K train triplets from 2016 and 11k from 2017
Model	ASICS	EMERSON	DO
Prec.	Rec.	F0.5	Prec.	Rec.	F0.5	Prec.	Rec.	F0.5
MT	3.3	31.4	4.0	16.2	78.6	19.2	1.2	41.9	1.5
GEC	52.8	6.6	22.0	78.0	53.3	71.4	20.5	2.0	7.1
APE	51.2	7.3	23.3	78.1	54.4	71.8	14.4	1.7	5.8
BERT-APE	6.8	10.8	7.3	32.0	57.8	35.1	2.3	3.8	2.5
TEC	57.4	9.4	28.4	82.1	57.2	75.5	21.7	2.0	7.2
Table 5: Main results on ACED. Our fine-tuned TEC model outperforms on  $\mathrm{F}_{0.5}$ . The fine-tuned MT model scores highest on recall because it makes many edits, but at the cost of unacceptably low precision.
mostly correct: it is better to suggest few correct edits than potentially introduce new errors.
Table 5 shows that TEC achieves the best overall  $\mathrm{F}_{0.5}$  score on all datasets, from +0.1 (on DO) up to +5.1 (on ASICS) above the next-best model. Fine-tuning on actual human corrections provides substantial gains; results without fine-tuning can be found in Appendix Section B.
Both the MT model (which ignores  $t$ ) and the GEC model (which ignores  $s$ ) underperform TEC. The MT model's high edit recall can be attributed to the fact that it proposes many edits, greatly trading off precision. Without conditioning on  $t$ , direct MT translations of the source diverge from the reference. The GEC model obtains high precision but underperforms on recall. Conditioning on  $s$  not only makes it possible to propose bilingual edits, but also provides additional information to correct monolingual edits, as we show in Section 4.1.
Can APE models be directly adapted for TEC?
Since our task is structurally identical to APE, a natural question is whether models that are trained on the APE objective can be directly adapted for TEC. The APE and TEC models differ only in pretraining, but the performance difference between them is substantial, indicating that the more GEC-like data synthesis procedure is a better fit for TEC than APE-style data synthesis via MT. Even more, the BERT-APE model, which is first fine-tuned to achieve state-of-the-art on APE before fine-tuning on ACED, achieves a particularly low  $\mathrm{F}_{0.5}$  score because it makes too many edits (low precision). Although future work may find insights in APE, our results emphasize that models that excel at correcting machine errors cannot be assumed to work well on human translations.
4.1 Fluency & Per-category Error Analysis
We perform a more in-depth comparison using alternative metrics on ASICs, which includes an-
notated error labels as a diagnostic tool. First, to understand how much models are editing, we look at  $n$ -gram overlap with the GLEU metric (Napoles et al., 2015), a variant of BLEU used in GEC evaluation to measure the fluency of holistic rewrites (Sakaguchi et al., 2016). Next, we compare sentence-level accuracy, which measures exact match with  $t'$ . We compute overall sentence-level accuracy, which includes unedited sentences (which some models may incorrectly edit). We also report accuracy per error type over (edited) sentences annotated with that error type. These metrics need to be interpreted carefully: a no-edit baseline achieves a GLEU score of 87.85 (since original translations are mostly close in edit distance to the final) and sentence-level accuracy of  $70.62\%$  (the % of unedited sentences), outperforming models like MT and BERT-APE that make too many incorrect edits.
Our TEC model achieves the best score overall on both alternative metrics over all sentences, but various models outperform on specific error types. The full results are shown in Table 6. Examples of system outputs for different error types can be found in Appendix Section C.