SlowGuess/NAACL_2022_OCR huggingface dataset:
Notably, APE models lose the most accuracy relative to our model on monolingual typo edits. This may be because neural MT decoders much less frequently introduce target-side errors that would be similar to typos (compared to the frequency of fluency errors). Still, the observation that different models do well at different errors suggests that future work can improve on TEC by leveraging the strengths of different models, e.g. using MT models to propose alternative translations.
5 User Study: Assisting Professional Translators with TEC
Our automatic evaluation shows how our TEC model can outperform other baseline systems, but
Model	GLEU	Sentence-level Accuracy (%)
Overall /616	Unedited /435	Mono. Typos /16	Mono. Grammar /78	Mono. Fluency /41	Bilingual /41	Preferential /14
No-edit	87.85	70.62	(100)	-	-	-	-	-
MT	44.79	31.66	40.00	0.00	11.54	2.44	24.39	14.29
GEC	88.46	71.10	97.01	31.25	14.10	0.00	0.00	0.00
APE	88.39	71.43	97.47	12.50	16.67	2.44	0.00	0.00
BERT-APE	82.35	39.77	52.18	6.25	19.23	0.00	4.88	0.00
TEC (this work)	88.81	71.92	97.01	25.00	17.95	0.00	7.32	0.00
Table 6: Additional analysis of n-gram overlap (via GLEU) and exact-match sentence accuracy over all test sentences (Overall) and per error category for the ASICS dataset. Number of sentences with an error of each type are indicated in gray, with some sentences containing errors from multiple categories. For each error category, we report the percentage of those sentences with at least one error of that type that a system predicted exactly.
we are ultimately interested in whether any TEC system is indeed useful in practice. Presently, TEC is done manually by humans. To investigate whether TEC systems can already be useful to humans—improving the quality, speed, or ease of human review—we performed a human-in-the-loop user study with our TEC model.
We recruited 9 professional translators to serve as reviewers. None of them had prior experience with ASICS content. They were allowed to read and reference the sentences in the ASICS training set to familiarize themselves with the content and preferred terminology. Then, they were each assigned to review 74 sentences from the test set of ASICS. Of the 74 sentences, our TEC system predicted a suggested edit for 57 sentences, and for the remaining 17 sentences our TEC system did not predict any edits.
We opt for a within-subjects design to control for speed and experience differences between reviewers. For each reviewer, the 74 sentences were randomized such that half were in the "assisted condition" showing the TEC suggestion if available for the sentence, and the other half were in the "unassisted condition" where no TEC suggestion was shown. The reviewing interface is shown in Figure 2. If the sentence has suggestions available, the reviewer is asked to first accept or reject each of the suggestions. Then, they are asked to make edits to the text until they are satisfied with the translation. They then click a button to confirm their translation and move to the next sentence.
During the review process, we track:
1. Whether the TEC suggestion, if shown, was accepted or declined
Segment 20/74
Suggestions shown
Featuring a jacquard mesh upper and FLYTEFOAM™ Propel technology for a lightweight quality that allows your feet to breathe, this ASICS running shoe offers excellent support and stability for those big mile days.
Dieser  $\oplus$  ASICS Laufschuh mit einem Jacquard-Mesh-Obermaterial und der FLYTEFOAM™ Propel-Technologie ist leicht,{last'se FuBe atmen und bietet dir bei denen Langstreckenläufen hervorgrende Unterstützung und Stabilität.
Figure 2: The interface used to show suggestions to reviewers in our user study.
CS Laufschuh ASICS-Laufschuh
Please accept or reject the suggestions above before making any other revisions
2. Total time spent reviewing each sentence
3. Number of edit operations (insertions and deletions) the user made
4. Levenshtein edit distance from the original text to the final text
Finally, to evaluate whether TEC has an effect on quality, we asked a 10th translator to compare the quality of the reviewed sentences by ranking the 9 reviewed translations, with ties allowed. This translator was the translator who had reviewed the reference translations in the corpus, as the explicit goal is to ensure consistency with conventions in the training documents.
In the user study,  $79\%$  of TEC suggestions were accepted. For the purpose of analyzing effects of the TEC suggestions on time spent and translation quality, we will focus on only the 255 sentences (across 9 reviewers) where a TEC suggestion exists. Results are shown in Table 7. For all statistical significance tests, we use the Mann-Whitney (MW) U-test for testing statistical significance as all quantities are neither normally distributed nor log-normal.
Suggestions	Review Time (ms/char)	Inserts + Deletes	Levenshtein Distance
Hidden	361	0.0625	0.0347
Shown (Overall)	367	0	0.0185
Accepted	328	0	0.0176
Declined	841	0.0625	0.0426
Table 7: Length-normalized medians from our user study for review times, number of characters the user inserted and deleted, and final Levenshtein edit distances from the original to the final translations. Data is split based on whether the suggestion was hidden or shown, and "Suggestion Shown" is further broken down according to whether the user accepted or declined the suggestion.
5.2.1 Effects of Suggestions on Time Spent During the Review Process
We first analyze how TEC suggestions influence the time spent reviewing a sentence. We compare the time durations normalized by the length of the sentence that needed to be reviewed (the length-normalized review time), as longer sentences require more time to be read and reviewed.
There is no significant difference in length-normalized review time when the suggestion is hidden vs. shown (MW  $U = 31654$ ,  $p = 0.460$ ). When suggestions are shown, the length-normalized review time is significantly less on sentences where reviewers accepted the suggestion, compared to sentences where they declined (MW  $U = 3555$ ,  $p < 0.0005$ ).
A potential explanation for these results is that when reviewers are shown incorrect TEC suggestions, they are distracted and slowed down, providing some evidence that precision should indeed be emphasized in automatic evaluations of TEC.
5.2.2 Effects of Suggestions on Edits Made During the Review Process
We also analyze the effects of suggestions on the editing effort, as measured by the number of characters the reviewer had to insert and delete, as well as how different the final reviewed sentences were from the original.
When suggestions are shown vs. hidden, there is a significant reduction in the number of insertions+deletions  $(\mathrm{MW}U = 41348.5,p < 0.0001)$ . There is also a significant reduction when a shown suggestion is accepted vs. declined  $(\mathrm{MW}U = 4007.5,p < 0.005)$ . There is no significant difference in the Levenshtein distance from the original translation to the final translation, between when a suggestion is shown vs. hidden (MW
Figure 3: Box plot showing quality rankings of segments reviewed with suggestions hidden vs shown. Lower is better. Notches indicate median rankings. Bars indicate the upper fence  $(3^{rd}$  quartile + IQR*1.5).
$U = 33750.0, p = 0.611$ , or between when a shown suggestion is accepted vs. declined (MW  $U = 5485.0, p = 0.783$ ).
Thus, the TEC system suggestions help to significantly reduce the amount of manual typing that the user must perform.
5.2.3 Effects of suggestions on reviewed sentence quality
To assess the effects of TEC assistance on quality, we used the quality rankings produced by the independent reviewer. Quality rankings were not normally distributed, so we use the Mann-Whitney U-test for testing statistical significance. A box plot of the quality rankings is shown in Figure 3. The median quality ranking when the suggestion is shown is 1, vs. 2 when the suggestion is hidden. The quality ranking is significantly lower (meaning quality is higher) when the suggestion is shown, vs. hidden  $(\mathrm{MW}U = 28738.0,p < 0.01)$ .
This suggests that showing TEC suggestions may be helping reviewers correct errors they may not have otherwise noticed, or help nudge them towards desired corrections.
5.3 Qualitative Findings
We also conducted a post-study survey for reviewers to report qualitative feedback. To understand common themes in the responses, we present all themes that at least two reviewers mention in their commentary.
5.3.1 The Role of Reliability and Trust
Five reviewers commented that reliability is critical: it was difficult to trust the system when they noticed some suggestions were incorrect, or the system did not reliably make an edit when applicable (e.g. always hyphenating when appropriate):
"Because I wasn't sure I can trust the suggestions (because I saw several incorrect ones) so it took me longer to think/check whether the suggestion is right. And I have to read the entire sentence again anyway to check for other errors the suggestion didn't catch...would only work if I knew  $100\%$  that the suggestions are always right"
These comments are in concordance with our quantitative findings. Perhaps unlike other assistive applications, it is not enough to only have high precision: if reviewers cannot trust that the system has caught most or all errors, they will not save time as they still have to read the entire sentence carefully. Conversely, a high-recall, low-precision system is not only distracting, but also leads reviewers to be suspicious of whether suggestions are correct in general. In general, future TEC systems must manage this balance of precision and recall for user trust.
5.3.2 Use Cases for TEC
Many reviewers highlighted scenarios where trustworthy TEC systems could be particularly useful.
Two reviewers said TEC is helpful for corrections and typos, similar to the use cases for GEC in the wild (Omelianchuk et al., 2021):
"If the tool would manage to reliably show missing punctuation marks, or numbers, or that the translation contains different numbers than the source, that would be helpful and save time"; "recurring mistakes"
On the other hand, three reviewers mentioned that they hoped such a system would make more substantial corrections in order to save a non-negligible amount of time, although of course these edits may come at the expense of precision:
"There were not many suggestions, and they only offered small improvements...Not clear whether I would save time or not."
Three reviewers commented that a TEC system could be a memory aid or substitute for researching client-specific requirements, which is often an intensive part of the production translation process. One reviewer pointed out it could be particularly useful as an instructive tool for translators who are new to a client:
"If I am new to an account and don't yet know whether this client wants hyphens or not (always an issue with German). So usually I have to research... (or guess), but if the QA suggestions knew this client's preference and would tell me, that would save me time."
Finally, three reviewers commented that it could be useful as an attention-directing tool by making them aware of what errors they might look out for, especially in repetitive content where it may be easy to miss details:
"makes you more sensitive for spotting similar errors"; "makes you aware of what kind of errors to look for in upcoming segments"; "maybe it helps with [repetitive sentences] that you would otherwise just quickly glance at."
We introduced the task of translation error correction (TEC) and released the ACED corpus to study automatic correction of human translations, consisting of three TEC datasets across varying domains. In our analysis of TEC data, we showed how the errors that humans make differ from those made by MT systems, suggesting that this task warrants different approaches from those previously studied in the task of automatic post-editing. We confirm this empirically by proposing a synthetic data generation procedure that more closely matches the distribution of human translation errors and showing that our TEC model, pre-trained on this data, consistently outperforms models developed for APE, as well as those for MT and GEC. Finally, we showed how our TEC system is helpful to real humans, assisting professional reviewers and leading them to produce higher quality reviewed translations.
Future work may improve on our TEC system by investigating how to leverage the strengths of recent MT systems (e.g. for initializing systems or proposing edits) or developing more sophisticated synthetic data generation techniques (e.g. using the source sentence or linguistic knowledge). Beyond our benchmark, it would be interesting to apply TEC systems to other settings in which human translation errors appear, e.g., to correct translations written by language learners, denoise MT training sets, or clean up MT evaluation sets.
From the perspective of human-AI interaction, TEC presents a real-world use case and testbed to study how to assist experts with modern NLP systems, hinting at the opportunity to combine the best of humans and machines.
We thank Sai Gouravajhala, Yunsu Kim, Eric Wallace, and the other members of the Lilt research team and Berkeley NLP group for helpful discussion and feedback. We thank Morgan Raymond and Spence Green for their support in releasing the dataset. Finally, we are grateful to the professional translators who annotated the dataset and participated in the user study.
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793-805, Vancouver, Canada. Association for Computational Linguistics.
Shamil Chollampatt, Raymond Hendy Susanto, Liling Tan, and Ewa Szymanska. 2020. Can automatic post-editing improve NMT? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2736-2746, Online. Association for Computational Linguistics.
Gonçalo M. Correia and André F. T. Martins. 2019. A simple and effective approach to automatic post-editing with transfer learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568-572, Montreal, Canada. Association for Computational Linguistics.
Donald A. DePalma. 2021. The language sector in eight charts. Accessed: 2022-05-01.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation.
Prabhakar Gupta, Ridha Juneja, Anil Kumar Nelakanti, and Tamojit Chatterjee. 2021. Detecting over/under-translation errors for determining adequacy in human translations. ArXiv, abs/2104.00267.
Gyde Hansen. 2009. A classification of errors in translation and revision. In CIUTI-Forum: Enhancing Translation Quality: Ways, Means, Methods.
Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2016. Log-linear combinations of monolingual and bilingual neural machine translation models for automatic post-editing. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 751-758, Berlin, Germany. Association for Computational Linguistics.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 595-606, New Orleans, Louisiana. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
António V. Lopes, M. Amin Farajian, Gonçalo M. Correia, Jonay Trénous, and André F. T. Martins. 2019. Unbabel's submission to the WMT2019 APE shared task: BERT-based encoder-decoder for automatic post-editing. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 118-123, Florence, Italy. Association for Computational Linguistics.
Courtney Napolles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 588-593, Beijing, China. Association for Computational Linguistics.
Matteo Negri, Marco Turchi, Rajen Chatterjee, and Nicola Bertoldi. 2018. ESCAPE: a large-scale synthetic corpus for automatic post-editing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Kostiantyn Omelianchuk, Vipul Raheja, and Oleksandr Skurzhanskyi. 2021. Text Simplification by Tagging. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 11-25, Online. Association for Computational Linguistics.
Keisuke Sakaguchi, Courtney Naples, Matt Post, and Joel Tetreault. 2016. Reassessing the goals of grammatical error correction: Fluency instead of grammaticality. Transactions of the Association for Computational Linguistics, 4:169-182.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Michel Simard, Nicola Ueffing, Pierre Isabelle, and Roland Kuhn. 2007. Rule-based translation with statistical phrase-based post-editing. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 203-206, Prague, Czech Republic. Association for Computational Linguistics.
Lucia Specia and Kashif Shah. 2014. Predicting human translation quality. In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas: MT Researchers Track, pages 288-300, Vancouver, Canada. Association for Machine Translation in the Americas.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.
Yu Yuan and Serge Sharoff. 2020. Sentence level human translation quality estimation with attention-based neural networks. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1858-1865, Marseille, France. European Language Resources Association.
Thomas Zenkel, Joern Wuebker, and John DeNero. 2019. Adding interpretable attention to neural translation models improves word alignment. arXiv preprint arXiv:1901.11359.
Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 156-165, Minneapolis, Minnesota. Association for Computational Linguistics.
A Transformer Architecture Background and Model Description
A.1 Transformer Architecture
The neural models implemented in this work are based on the self-attentional Transformer architecture (Vaswani et al., 2017). Formally, given a sequence of source tokens (encoded as one-hot vectors)  $\mathbf{s}_{1\dots J} = (s_1,\dots ,s_J)$ ,  $s_j\in \mathbf{V}$ , the goal is to predict a sequence of target tokens  $\mathbf{t^{\prime}}_{1\dots I^{\prime}} = (t_1^{\prime},\ldots ,t_{I^{\prime}}^{\prime})$ ,  $t_i^\prime \in \mathbf{V}$ , that is a translation of the source sequence, where  $\mathbf{V}$  is the vocabulary. The model has two main components, the encoder and the decoder. The encoder transforms the source sequence  $\mathbf{s}_{1\dots J}$  into a sequence of hidden states by first mapping each individual token into a continuous embedding space, adding a positional embedding and then processing it through a sequence of self-attention and feed-forward layers:
$$
\mathbf {x} _ {1 \dots J} = \mathbf {E s} _ {1 \dots J} + \mathbf {p} _ {1 \dots J} \tag {1}
$$
$$
\mathbf {h} _ {1 \dots J} ^ {\text {e n c}} = \operatorname {e n c o d e r} \left(\mathbf {x} _ {1 \dots J}\right), \tag {2}
$$
where  $x_{j}\in \mathbf{V},j\in (1,\ldots ,J),\mathbf{E}$  is the embedding matrix for vocabulary  $\mathbf{E}$  and  $\mathbf{p}_{1\dots J}$  is the sequence of positional embeddings described in Sec. 3.5 of (Vaswani et al., 2017). At a given time step  $i$  ,the decoder defines a probability distribution  $P_{i}$  over all vocabulary items in  $\mathbf{V}$  ..
$$
\mathbf {y} ^ {\prime} _ {1 \dots i - 1} = \mathbf {E t} ^ {\prime} _ {1 \dots i - 1} + \mathbf {p} _ {1 \dots i - 1} \tag {3}
$$
$$
\mathbf {h} _ {i} ^ {\mathrm {d e c}} = \operatorname {d e c o d e r} \left(\mathbf {y} _ {1 \dots i - 1} ^ {\prime}, \mathbf {h} _ {1 \dots J} ^ {\mathrm {e n c}}\right) \tag {4}
$$
$$
P _ {i} \left(t _ {i} ^ {\prime}\right) = \operatorname {s o f t m a x} \left(\mathbf {h} _ {i} ^ {\mathrm {d e c}} \mathbf {E} ^ {\top}\right) \tag {5}
$$
where we assume a single shared vocabulary  $\mathbf{V}$  and embedding matrix  $\mathbf{E}$ . At training time we optimize the cross-entropy loss
$$
\mathcal {L} _ {\mathrm {C E}} (P) = - \sum_ {i} \log \left(P _ {i} \left(t _ {i} ^ {\prime}\right)\right). \tag {6}
$$
A.2 Dual-Source Encoder-Decoder Model
Given an additional input sequence  $\mathbf{t}_{1\dots I} = (t_1,\dots ,t_I)$ . The dual-source model used for the APE and TEC models is implemented by independently projecting  $\mathbf{t}_{1\dots I}$  into the embedding space, adding an offset vector  $\mathbf{o}$  and concatenating the embedding sequences. Equations 2 and 4 are rewritten as
$$
\mathbf {y} _ {1 \dots I} = \mathbf {E t} _ {1 \dots I} + \mathbf {p} _ {1 \dots I} + \mathbf {o} \tag {7}
$$
$$
\mathbf {h} _ {1 \dots (J + I)} ^ {\text {e n c}} = \operatorname {e n c o d e r} ([ \mathbf {x} _ {1 \dots J}; \mathbf {y} _ {1 \dots I} ]) \tag {8}
$$
$$
\mathbf {h} _ {i} ^ {\mathrm {d e c}} = \operatorname {d e c o d e r} \left(\mathbf {y} _ {1 \dots i - 1} ^ {\prime}, \mathbf {h} _ {1 \dots (J + I)} ^ {\mathrm {e n c}}\right), \tag {9}
$$
where  $\mathbf{o}$  is a single learned vector that is broadcast to all positions  $i\in (1,\dots ,I)$  and  $[\cdot ;\cdot ]$  denotes the concatenate operation.
A.3 Copy-Attention Mechanism
The new output probability distribution for the next target token  $P_{i}(t_{i}^{\prime})$  is a weighted sum of the probability of generating and the probability of copying token  $t_{i}^{\prime}$ :
$$
\hat {P} _ {i} \left(t _ {i} ^ {\prime}\right) = \left(1 - \alpha_ {i} ^ {\text {c o p y}}\right) P _ {i} \left(t _ {i} ^ {\prime}\right) + \alpha_ {i} ^ {\text {c o p y}} P _ {i} ^ {\text {c o p y}} \left(t _ {i} ^ {\prime}\right), \tag {10}
$$