SlowGuess/NAACL_2021_OCR huggingface dataset:
Data Normalization. Because of data imbalance in favor of high confidence, we normalized the self-reported user confidence using the following formula:  $x' = \frac{x - \min}{\max - \min} \times 4 + 1$ . The min and max values were taken individually for every participant. This only affected those who never used 1 or 2 in their self-reported confidence. We did not apply this normalization to the quality annotations, because the annotators used the whole scale in almost all cases. The overall average of all confidence judgments decreased from 3.72 to 3.59 by this normalization.
This only helped with the imbalance a little.
9This rule discredits inputs meant to be viable, where the very last token was later edited, though. Manual examination of the data verified the efficacy of the heuristic.
Figure 3: Distribution (percentage and absolute count) of quality annotation (rounded overall) and self-reported user confidence.
To avoid strong assumptions about the underlying process, we did not normalize the data to have zero mean and standard deviation of 1 for every feature dimension. This would also have made any interpretation less intuitive.
Results on final inputs. Table 5 shows the average evaluation scores of final confirmed inputs, accompanied by average self-confidence scores across various configurations. For clarity, we illustrate the same results in Figure 2. Comparing the Czech MT systems, their ranking with respect to the Overall score corresponds to the results of the automatic evaluation in the news domain shown in Table 1.
Interestingly enough, Czech 2 received an average confidence score comparable to its teacher model Czech 3 (see in Figure 2). The results of comparison across different combinations of cues suggest that configurations with backtranslation feedback enabled achieved better performance in terms of the overall quality. In such cases, the users also felt more confident. Unlike for overall quality, the effect of an available backward translation cue on user confidence was statistically significant by Mann-Whitney U test for 0.6 point difference  $(U = 24243.5, p < 0.0001)$ .
Conversely, quality estimation cues appear not to be useful, which the users also noted. Unfortunately, the presence of paraphrases increased user confidence, but decreased the objective translation quality. These results are in contrast with the work of Zouhar and Novák (2020). We attribute this difference to an insufficient number of samples and also a more homogeneous composition of partic
TGT SRC	TGT STI	Fluency	Overall	Conf.
SRC-STI	0.07	0.64	0.23	0.50	0.08
TGT-SRC	0.69	0.68	0.71	0.14
TGT-STI	0.67	0.88	0.15
Fluency	0.84	0.13
Overall	0.14
Table 6: Correlation between all quality annotations variables and self-reported user confidence.
ipants (all foreign PhD students studying in the Czech Republic) in their work.
Note that users who had knowledge of some other Slavic language (Polish or Russian) on average expressed higher confidence (3.95) and also produced translations of higher quality (4.44). The effects of different modules on their work were closer to the effects described in Zouhar and Novák (2020).
As seen in Figure 3, a significant proportion of the scenes ( $\sim 41\%$ ) received 4 or 5 on both self-reported confidence and overall translation quality. Although these high scores are positive in terms of industry progress, it makes the quality-confidence dependency harder to analyze.
Table 6 shows expected rating behavior in terms of correlations. We can see that Fluency is mostly correlated with TGT-SRC and TGT-STI adequacies and less with SRC-STI adequacy, which should affect the translation fluency only slightly.[10] We also see that TGT-STI adequacy and Fluency affects the Overall rating the most, which accords with its definition. Self-reported user confidence correlates the least with all the rest, but slightly more with TGT-STI, TGT-SRC and Overall scores, which we consider positive.
MT comparison in detail. Figure 4 shows the average spent time per stimulus as well as the number of forward translation requests and input length in characters with respect to the confidence and overall translation quality for submitted translations. The figure is split into three graphs, each corresponding to one of the Czech MT systems.
Input text length does not appear to affect the overall translation quality significantly, while it seems to affect users' self-reported confidence.
The curves for time spent, although different in
In a scenario where the SRC-STI adequacy is lowered by typos in Source, which then also negatively affects the translation process and also the Fluency.
Text length, conf.  
Text length, qual.
Requests, conf. Requests, qual.
Time, conf. Time, qual.
Self-reported user confidence, Overall translation quality
Figure 4: Relationship between the scores (self-reported user confidence in darker colors and overall translation quality in lighter colors) and average scene time, forward translation request count and input length in characters (y-axes). Best viewed in color.
absolute values, peak in the middle (rating 3) and have the lowest values for scores of 1 and 5. This may happen because the stimulus was either easy to complete, or the users did not work on this stimulus diligently. It is supported by the fact that they did not report low confidence in these instances. A similar trend, although less pronounced, can be seen with the number of requests.
We can also notice that the Czech 2 system has the lowest times despite also having a vastly higher number of executed requests. The request delay was the same for all MT systems, so in this case, the users recognized that they did not have to wait so long for getting a translation back and hence sent more requests. This is one of the possible explanations for why in Figure 2 the average self-reported confidence for this system is on par with its teacher model, Czech 3, despite being less performant objectively.
The degree of interactivity appears to be the main factor affecting these MT systems profiles. The figures of Czech 1 and Czech 3 look very similar even though they vary greatly in performance and only have their speeds in common (slower than Czech 2).
Intermediate vs. final. Having also intermediate viable inputs at our disposal, we explored how quality changes in the transition from intermediate to final inputs. We excluded those scenes that contain no viable intermediate input, which accounts for almost  $69\%$ .
Although our heuristics can filter out most of the intermediate inputs which are not viable, some
Config	SRC STI	TGT SRC	TGT STI	Fluency	Overall
BT QE PP	-0.19†	+0.10	+0.04	+0.05	+0.08
BT QE	-0.14†	+0.16◇	+0.03	+0.04	+0.03
BT PP	-0.12	+0.14	+0.16	+0.12	+0.17
QE PP	-0.20†	+0.03	-0.13*	+0.02	-0.07*
BT	-0.24†	+0.33†	+0.10	+0.10	+0.11
QE	-0.11*	-0.01	-0.10	-0.05	-0.04
PP	-0.11◇	+0.09	-0.05	-0.00	-0.04
-	-0.02	+0.16	+0.16	+0.04	+0.06
Total	-0.15†	+0.11†	+0.01	+0.04	+0.03
Table 7: Average difference of quality between intermediate viable and final inputs and their translations for all combinations of available cue modules. Statistical significance was calculated by Wilcoxon signed-rank test  $(p < 0.05^{*}, 0.01^{\diamond}, 0.001^{\dagger})$
of those remaining can be still considered defective. They may contain a typo, artifacts of unfinished rephrasing or may miss important information. These non-viable inputs must be excluded from the comparison, as the user would unlikely submit them or they could be easily fixed by a spell-checker. We manually examined all intermediate viables and excluded the defective ones from the following statistics.
Table 7 shows the average difference in the quality of intermediate and corresponding final inputs and their translations. The greatest improvement in the Overall score is again achieved by configurations utilizing backtranslation feedback, although the difference is not statistically significant. What is significant, though, are some
Inter	I teach my son English with the 'Learning Time with Timmy' series on Youtube.
Final	Ućím svého syna Angličana /Englishman/ se seriálem „Learning Time with Timmy“ na Youtube. I teach my son English language with the series 'Learning Time with Timmy' series on Youtube. Ućím svého syna anglický jazyk se seriálem „Learning Time with Timmy“ na Youtube.
Inter	Why was I not able to make a payment by mobile?
Final	Proć jsem nemohl zaplatit za /for/ mobil? Why was I not able to make a payment from my mobile?
Proć jsem nemohl zaplatit z mobilu?
Inter	What documents do I need to have if my ID has expired?
Final	Jaké doklady potřebuji, když mi vypršel průkaz totožnosti? What documents do I need to have if my ID is out of date? Jaké dokumenty potřebuji, když je m subj průkaz zastaral /got obsoleste/?
Table 8: Examples of user interaction with the Ptakopět system. In the top two, the rephrasing of the intermediate input resulted in an improved final translation, in the bottom one the final translation worsened.
TGT-SRC scores including the BT configuration. It shows that the translation of the final input is on average more adequate to the source than the translation of the intermediate inputs. Nevertheless, the effect on the TGT-STI adequacy is marginal due to negative differences in the SRC-STI adequacy score. These can be justified by the fact that any modification of the original query in the stimulus might have been considered as a shift in meaning by the annotators, although in reality the original intention could be still understandable.
In Table 8, we show three examples of the intermediate and the final inputs with their translations to Czech. In the top two, the rephrasing helped to improve the translation quality: (1) by adding a word "language" to prevent translating "English" as a Czech word for "Englishmen", or (2) by substituting a preposition. Conversely, the replacement of the verb "has expired" by a phrase "out of date" led to a drop in translation quality. This is due to a grammatical error and use of the Czech expression meaning "got obsolete", which indeed sounds old-fashioned in this context.
In this paper, we demonstrated through an experiment the effect of three translation cues on user confidence and translation quality.
The backward translation cue proves to be a powerful means to enhance user confidence in MT. At the same time, it neither increase nor decrease significantly the translation quality. The fact that backtranslation feedback has a marginal effect to objective quality but greatly increases user confidence is surprising because it is the most intuitive low-effort approach to outbound translation scenarios which can be done even with publicly available MT systems.
The paraphraseer seems to increase user confi
dence less (compared to not being present), with no or slightly negative impact on the translation quality. Without a better method to generate diverse and still adequate paraphrases, employing this cue is questionable. The effect of word-level quality estimation appears to be even more questionable. We attribute it mainly to the underlying word-level models, which may not be mature enough for user-facing applications.
Despite the loss in objective translation quality, the CPU-optimized student MT model either managed to maintain its teacher's high trustworthiness or compensated for it by its speed.
Future work. Scores in both user confidence and overall translation quality annotation cluster together. Having the distribution less concentrated by changing the underlying task with stimuli or by working with more low resource languages could reveal stronger dependencies between individual variables.
We limited ourselves to only three baseline solutions to help in outbound translation. In the future work, inspiration could be drawn from the approaches of interactive machine translation systems and these could be adapted for the purposes of outbound translation.
Sincere thanks to Chris Burns and the three anonymous reviewers for their thorough review and helpful comments.
This project has received funding from the grants H2020-ICT-2018-2-825303 (Bergamot) of the European Union and 19-26934X (NEUREM3) of the Czech Science Foundation. The work has also been supported by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2018101 LINDAT/CLARIAH-CZ.
All participants were recruited online and had to complete an informed consent form using a secure Qualtrics survey before they could progress to taking part in the experiment. Data was anonymized, recorded and stored in accordance with ACM protocol. Ethical clearance was confirmed by the School of Informatics Ethics Committee at the University of Edinburgh (Reference RT 4058). Participants were offered a £20 Amazon voucher as compensation for their time upon completion of the experiment.
Vicent Alabau, Michael Carl, Francisco Casacuberta, Mercedes García Martínez, Jesús González-Rubio, Bartolomé Mesa-Lao, Daniel Ortiz-Martínez, Moritz Schaeffer, and Germán Sanchis-Trilles. 2016. Learning Advanced Post-editing, New Frontiers in Translation Studies, pages 95-110. Springer, Cham, Switzerland.
John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence Estimation for Machine Translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 315-321, Geneva, Switzerland. COLING.
Ondrej Bojar, Ondrej Dušek, Tom Kocmi, Jindrich Libovický, Michal Novák, Martin Popel, Roman Sudarikov, and Dušan Variš. 2016. CzEng 1.6: Enlarged Czech-English Parallel Corpus with Processing Tools Dockered. In Text, Speech, and Dialogue: 19th International Conference, TSD 2016, number 9924 in Lecture Notes in Computer Science, pages 231–238, Cham, Switzerland. Springer International Publishing.
Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 Conference on Machine Translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272-303, Belgium, Brussels. Association for Computational Linguistics.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Christian Federmann, Oussama Elachqar, and Chris Quirk. 2019. Multilingual Whispers: Generating
Paraphrases with Translation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 17-26, Hong Kong, China. Association for Computational Linguistics.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised Quality Estimation for Neural Machine Translation. Transactions of the Association for Computational Linguistics, 8:539-555.
Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian Federmann. 2019. Findings of the WMT 2019 Shared Tasks on Quality Estimation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 1-10, Florence, Italy. Association for Computational Linguistics.
Ulrich Germann, Roman Grundkiewicz, Martin Popel, Radina Dobreva, Nikolay Bogoychev, and Kenneth Heafield. 2020. Speed-optimized, Compact Student Models that Distill Knowledge from a Larger Teacher Model: the UEDIN-CUNI Submission to the WMT 2020 News Translation Task. In Proceedings of the Fifth Conference on Machine Translation, pages 191-196, Online. Association for Computational Linguistics.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving Human Parity on Automatic Chinese to English News Translation. arXiv preprint arXiv:1803.05567.
Yoon Kim and Alexander M. Rush. 2016. Sequence-Level Knowledge Distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
Tom Kocmi, Martin Popel, and Ondrej Bojar. 2020. Announcing CzEng 2.0 Parallel Corpus with over 2 Gigawords. arXiv preprint arXiv:2007.03006.
Dominik Macháček, Jonas Kratochvil, Sangeet Sagar, Matúš Žilinec, Ondrej Bojar, Thai-Son Nguyen, Felix Schneider, Philip Williams, and Yuekun Yao. 2020. ELITR non-native speech translation at IWSLT 2020. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 200-208, Online. Association for Computational Linguistics.
Pascual Martínez-Gómez, Germán Sanchis-Trilles, and Francisco Casacuberta. 2012. Online adaptation strategies for statistical machine translation in post-editing scenarios. Pattern Recognition,
45(9):3193 - 3203. Best Papers of Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA'2011).
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Martin Popel, Dominik Machacek, Michal Auersperger, Ondrej Bojar, and Pavel Pecina. 2019. English-Czech Systems in WMT19: Document-Level Transformer. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 342-348, Florence, Italy. Association for Computational Linguistics.
Martin Popel, Marketa Tomkova, Jakub Tomek, Lukasz Kaiser, Jakob Uszkoreit, Ondrej Bojar, and Zdenek Zabokrtsky. 2020. Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals. Nature communications, 11(1):1-15.
Germán Sanchis-Trilles, Vicent Alabau, Christian Buck, Michael Carl, Francisco Casacuberta, Mercedes García-Martínez, Ulrich Germann, Jesús González-Rubio, Robin L Hill, Philipp Koehn, LuisA Leiva, Bartolomé Mesa-Lao, Daniel Ortiz-Martínez, Herve Saint-Amand, Chara Tsoukala, and Enrique Vidal. 2014. Interactive translation prediction versus conventional post-editing in practice: a study with the CasMaCat workbench. Machine Translation, 28(3-4):217-235.
Sebastin Santy, Sandipan Dandapat, Monojit Choudhury, and Kalika Bali. 2019. INMT: Interactive Neural Machine Translation Prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 103-108, Hong Kong, China. Association for Computational Linguistics.
Lucia Specia, Kashif Shah, Jose G.C. de Souza, and Trevor Cohn. 2013. QuEst - A translation quality estimation framework. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 79-84, Sofia, Bulgaria. Association for Computational Linguistics.
Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
Nancy Underwood, Bartolomé Mesa-Lao, Mercedes García Martínez, Michael Carl, Vicent Alabau, Jesús González-Rubio, Luis A. Leiva, German Sanchis-Trilles, Daniel Ortiz-Martínez, and Francisco Casacuberta. 2014. Evaluating the Effects of Interactivity in a Post-Editing Workbench. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 553-559, Reykjavik, Iceland. European Language Resources Association (ELRA).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pages 5998-6008, Red Hook, NY, USA. Curran Associates Inc.
Vilém Zouhar and Michal Novák. 2020. Extending Ptakopět for Machine Translation User Interaction Experiments. The Prague Bulletin of Mathematical Linguistics, 115:129-142.
Vilém Zouhar. 2020. Enabling Outbound Machine Translation. Bachelor thesis, Charles University, Faculty of Mathematics and Physics.
Vilém Zouhar and Ondrej Bojar. 2020. Outbound Translation User Interface Ptakopét: A Pilot Study. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6969-6977, Marseille, France. European Language Resources Association.