SlowGuess/ACL_2020_OCR huggingface dataset:
As expected, deletions were faster than insertions. Here, MK, T, and P were the fastest, followed by S; MM was slowest by far. Regarding significance, all modalities were significantly faster than MM, and MK was significantly faster than S (all  $p < 0.01$ , all  $r > 0.88$ ).
For reordering, P and T were the fastest, followed by MK and S. The statistical analysis revealed that T is significantly faster than all modalities except P, both P and MK are significantly faster than S, and S is significantly faster than MM (all  $p < 0.05$ , all  $r > 0.83$ ).
Replacements with MK were the fastest, followed by P, T, S, and MM. MK was significantly faster than all other modalities, and P significantly faster than S and MM (all  $p < 0.05$ , all  $r > 0.83$ ), while no significant differences exist between the other three.
Apart from the ratings and timings, we present the main qualitative feedback from the interviews.
5.5.1 Pen & Touch
Especially for short insertions and replacements, handwriting was seen as a suitable input mode; for more extended changes, one should instead fall back on typing or dictation. Both touch/pen deletion mechanisms (strike-through and double-tap) and touch/pen reordering were highlighted as very useful or even "perfect" as they "nicely resemble a standard correction task". Most participants seemed to prefer the pen to finger handwriting for insertions and replacements due to its precision, although it was considered less direct.
(a) Insertions.
(c) Replacements.
Figure 2: Subjective ratings.
(b)Deletions.
(d) Reorderings.
A major concern was thinking about and creating sufficient space to handwrite into. A suggested improvement was to make the available space configurable to one's own handwriting. Furthermore, placing the palm of the hand on the screen should not be interpreted as input. Six participants also noted that the text jumps around when reordering a word from the end of a line, as the picked-up word is removed from the text, resulting in all remaining words being moved to the front, which could be prevented by adapting the text only on drop.
5.5.2 Speech & Multi-Modal Combinations
Perceptions regarding speech recognition were somewhat mixed, with some thinking it worked "super" while two participants found it exhausting to formulate commands while mentally working with text. Furthermore, speech was considered impractical for translators working in shared offices. Both insertions and replacements using speech received lots of positive feedback (from 8 and 7 participants, respectively), interesting findings being that "the longer the insertion, the more interesting speech becomes". Speech deletion was considered to "work fine" and to be simpler than insertion as there is usually no need to specify the position. However, it would be unsatisfactory to have to read 10 words to delete them.
The main advantage of the multi-modal ap
proach was that "one has to speak/think less". However, it was also argued that "when you talk, you can also just say everything", meaning that the simplified MM command was not seen as an advantage for this participant. An interesting statement was that "if there are no ambiguities, speech is better, but if there are, multi-modal is cool".
Ideas on how to improve speech ranged from better highlighting the changes in the target view, to adding the possibility to restate the whole segment. While the ASR tool used (IBM Watson) is one of the state-of-the-art APIs, it might still have negatively impacted the results for S and MM, as a few times a word was wrongly recognized (e.g., when replacing an ending, the ASR did not always correctly recognize the word form). To improve this aspect, participants discussed the idea of passing the text to the speech recognition (Dymetman et al., 1994) or training the ASR towards the user.
5.5.3 Mouse & Keyboard
Due to daily usage, participants stated they were strongly biased regarding mouse and keyboard, where "the muscle memory" helps. However, many actually considered MK as very unintuitive if they imagined never having used it before, especially compared to pen and touch, or as one participant stated for reordering: "why do I have to do all of this, why is it not as simple as the pen".
Figure 3: Editing durations (in ms) per operation and modality.
5.5.4 General Feedback
In general, we received lots of positive feedback in the final discussion about the prototype, where participants made statements such as "I am going to buy this once you are ready" or expressed "respect for the prototype". Multiple participants reported that it would be nice to have multiple options to vary between the modalities. It was frequently suggested to combine the two editing views, e.g. by having a switch to enable/disable the drawing mode. Participants also commented positively on the large typeface for the current segment ("you really see what you are working on"). Suggestions for further improvements included adaptation possibilities for the size of the editing fields and a switch between vertical and horizontal source-target layout.
5.6 Discussion
This section discusses the main takeaways regarding each modality.
5.6.1 Pen
According to ordering scores, subjective ratings, and comments, we see that the pen is among the best modalities for deletions and reordering. However, other modalities are superior for insertions and replacements, where it was seen as suitable for short modifications, but to be avoided for more extended changes. In terms of timings, P was also among the fastest for deletions and reorderings, and among the slowest for insertions. What is interest
ing, however, is that P was significantly faster than S and MM for replacements, even though it was rated lower. The main concern for handwriting was the need to think about space and to create space before actually writing.
5.6.2 Touch
Results for touch were similar, but it was considered worse for insertions and replacements. Furthermore, and as we expected due to its precision, pen was preferred to finger touch by most participants. However, in terms of timings, the two did not differ significantly apart from replace operations, and even for replacements, where it was clearly rated as the worst modality, it actually turned out to be (non-significantly) faster than S and MM.
5.6.3 Speech & Multi-modal Combinations
Speech and multi-modal PE were considered the worst and were also the slowest modalities for reordering and deletions. For insertions and replacements, however, these two modalities were rated and ordered  $2^{\text{nd}}$  (after MK) and in particular much better than P and T. Timing analysis agrees for insertions, being  $2^{\text{nd}}$  after MK. For replacements, however, S and MM were the slowest even though the ratings put them before P and T. An explanation of why MM was slower than S for deletion is that our implementation did not support MM deletions of multiple words in a single command. Still, we would have expected a comparable speed of MM and S for reordering. Insertions are the only oper-
ation where the multi-modal approach was (nonsignificantly) faster than S since the position did not have to be verbally specified.
Furthermore, the participants' comments highlighted their concern regarding formulating commands while already mentally processing text. Still, S and MM received a lot of positive feedback for insertions and replacements, where they would be more interesting the more text was to be added. The main advantage of the MM approach, as argued by the participants, was that one has to speak less, albeit at the cost of doing two things at once.
5.6.4 Mouse & Keyboard
Mouse & keyboard received the best scores for insertions and replacements, where it was the fastest modality. Furthermore, it got good ratings for deletions and reordering, where it was also fast (but not the fastest) for reordering. However, some participants commented negatively, stating that it only works well because of "years of expertise".
5.6.5 General
Interestingly, our findings are not entirely in line with translators' intuitions reported in our previous elicitation study (Herbig et al., 2019a): while touch worked much better than expected, handwriting of whole subphrases did not work as well as they thought. Additionally, it is interesting to note that some newly introduced modalities could compete with mouse & keyboard even though participants are biased by years of training with the latter.
Overall, many participants provided very positive feedback on this first prototype combining pen, touch, speech, and multi-modal combinations for PE MT, encouraging us to continue. Furthermore, several promising ideas for improving and extending the prototype have been proposed.
The focus of our study was to explore the implemented interactions in detail, i.e., each modality for each operation irrespective of frequency. The chosen methodology guaranteed that we receive comparable feedback on all interactions from professional translators by having them correct the same mistakes using different modalities. Nevertheless, a more realistic "natural" workflow follow-up study should be conducted in the future, which will also show if participants swap modalities within sentences depending on the error type, or if they stick to single modalities to avoid frequent modality switches.
While more and more professional translators are switching to the use of PE to increase productivity and reduce errors, current CAT interfaces still heavily focus on traditional mouse and keyboard input, even though the literature suggests that other modalities could support PE operations well. This paper therefore presents MMPE, a CAT prototype combining pen, touch, speech, and multi-modal interaction together with common mouse and keyboard input possibilities, and explores the use of these modalities by professional translators. The study shows a high level of interest and enthusiasm for using these new modalities. For deletions and reordering, pen and touch both received high subjective ratings, with pen being even better than mouse & keyboard. In terms of timings, they were also among the fastest for these two operations. For insertions and replacements, speech and multimodal interaction were seen as suitable interaction modes; however, mouse & keyboard were still favored and faster here.
As a next step, we will integrate the participants' valuable feedback to improve the prototype. While the presented study provided interesting first insights regarding participants' use of and preferences for the implemented modalities, it did not allow us to see how they would use the modalities over a longer time period in day-to-day work, which we also want to investigate in the future.
Furthermore, participants in Herbig et al. (2019a) were positive regarding the idea of a user interface that adapts to measured cognitive load, especially if it automatically provides additional resources like TM matches or MT proposals. An exploration of multi-modal measuring approaches (Herbig et al., 2019b) shows the feasibility of this, so we will try to combine explicit multi-modal input, as done in this work, with implicit multi-modal sensor input to better model and support the user during PE.
This research was funded in part by the German Research Foundation (DFG) under grant number GE 2819/2-1 (project MMPE). We thank AMPLEXOR (https://www.amplexor.com) for their excellent support in providing access to professional human translators for our experiments.
Vicent Alabau, Ragnar Bonk, Christian Buck, Michael Carl, Francisco Casacuberta, Mercedes García-Martínez, Jesús González, Philipp Koehn, Luis Leiva, Bartolomé Mesa-Lao, et al. 2013. CAS-MACAT: An open source workbench for advanced computer aided translation. The Prague Bulletin of Mathematical Linguistics, 100:101-112.
Vicent Alabau and Francisco Casacuberta. 2012. Study of electronic pen commands for interactive-predictive machine translation. In Proceedings of the International Workshop on Expertise in Translation and Post-Editing - Research and Application, pages 17-18.
Nora Aranberri, Gorka Labaka, A Diaz de Ilarraza, and Kepa Sarasola. 2014. Comparison of post-editing productivity between professional translators and lay users. In Proceeding of AMTA Third Workshop on Post-editing Technology and Practice, pages 20-33.
Jan van den Bergh, Eva Geurts, Donald Degraen, Mieke Haesen, Iulianna van der Lek-Ciudin, Karin Coninx, et al. 2015. Recommendations for translation environments to improve translators' workflows. In Proceedings of the 37th Conference Translating and the Computer, pages 106-119. Tradulex.
Julie Brousseau, Caroline Drouin, George Foster, Pierre Isabelle, Roland Kuhn, Yves Normandin, and Pierre Plamondon. 1995. French speech recognition in an automatic dictation system for translators: The TransTalk project. In Proceedings of Eurospeech Fourth European Conference on Speech Communication and Technology, pages 193-196.
Michael Carl, Martin Jensen, and Kay Kristian. 2010. Long distance revisions in drafting and post-editing. CICling Special Issue on Natural Language Processing and its Applications, pages 193-204.
Sven Coppers, Jan van den Bergh, Kris Luyten, Karin Coninx, Iulianna van der Lek-Ciudin, Tom Vanalle-meersch, and Vincent Vandeghiste. 2018. Intellingo: An intelligible translation environment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1-13. ACM.
Barbara Dragsted, Inger Margrethe Mees, and Inge Gorm Hansen. 2011. Speaking your translation: Students' first encounter with speech recognition technology. Translation & Interpreting, 3(1):10-43.
Marc Dymetman, Julie Brousseau, George Foster, Pierre Isabelle, Yves Normandin, and Pierre Plamondon. 1994. Towards an automatic dictation system for translators: The TransTalk project. In Proceedings of the ICSLP International Conference on Spoken Language Processing.
Marcello Federico, Nicola Bertoldi, Mauro Cettolo, Matteo Negri, Marco Turchi, Marco Trombetti,
Alessandro Cattelan, Antonio Farina, Domenico Lupinetti, Andrea Martines, et al. 2014. The MateCat tool. In Proceedings of the 25th International Conference on Computational Linguistics: System Demonstrations, pages 129-132.
Federico Gaspari, Antonio Toral, Sudip Kumar Naskar, Declan Groves, and Andy Way. 2014. Perception vs reality: Measuring machine translation post-editing productivity. In Third Workshop on Post-Editing Technology and Practice, page 60.
Spence Green, Jason Chuang, Jeffrey Heer, and Christopher D Manning. 2014a. Predictive translation memory: A mixed-initiative system for human language translation. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, pages 177-187. ACM.
Spence Green, Jeffrey Heer, and Christopher D Manning. 2013. The efficacy of human post-editing for language translation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 439-448. ACM.
Spence Green, Sida I Wang, Jason Chuang, Jeffrey Heer, Sebastian Schuster, and Christopher D Manning. 2014b. Human effort and machine learnability in computer aided translation. In Proceedings of the EMNLP Conference on Empirical Methods in Natural Language Processing, pages 1225-1236.
Nico Herbig, Santanu Pal, Tim Düwel, Kalliopi Meladaki, Mahsa Monshizadeh, Vladislav Hnatovskiy, Antonio Krüger, and Josef van Genabith. 2020. MMPE: A multi-modal interface using handwriting, touch reordering, and speech commands for post-editing machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics.
Nico Herbig, Santanu Pal, Josef van Genabith, and Antonio Krüger. 2019a. Multi-modal approaches for post-editing machine translation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, page 231. ACM.
Nico Herbig, Santanu Pal, Mihaela Vela, Antonio Krüger, and Josef Genabith. 2019b. Multi-modal indicators for estimating perceived cognitive load in post-editing of machine translation. Machine Translation, 33(1-2):91-115.
Maarit Koponen. 2012. Comparing human perceptions of post-editing effort with post-editing operations. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 181-190. Association for Computational Linguistics.
Elina Lagoudaki. 2009. Translation editing environments. In MT Summit XII: Workshop on Beyond Translation Memories.
Samuel Läubli, Mark Fishel, Gary Massey, Maureen Ehrensberger-Dow, and Martin Volk. 2013. Assessing post-editing efficiency in a realistic translation environment. In Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice, pages 83-91.
Mercedes Garcia Martinez, Karan Singla, Aniruddha Tammewar, Bartolomé Mesa-Lao, Ankita Thakur, MA Anusuya, Banglore Srinivas, and Michael Carl. 2014. SEECAT: ASR & eye-tracking enabled computer assisted translation. In The 17th Annual Conference of the European Association for Machine Translation, pages 81-88. European Association for Machine Translation.
Bartolomé Mesa-Lao. 2014. Speech-enabled computer-aided translation: A satisfaction survey with post-editor trainees. In Proceedings of the EACL 2014 Workshop on Humans and Computer-assisted Translation, pages 99–103.
Joss Moorkens. 2018. What to expect from neural machine translation: A practical in-class translation evaluation exercise. The Interpreter and Translator Trainer, 12(4):375-387.
Joss Moorkens and Sharon O'Brien. 2015. Post-editing evaluations: Trade-offs between novice and professional participants. In Proceedings of the 18th Annual Conference of the European Association for Machine Translation, pages 75-81.
Joss Moorkens and Sharon O'Brien. 2017. Assessing user interface needs of post-editors of machine translation. In Human Issues in Translation Technology, pages 127-148. Routledge.
Sharon O'Brien, Joss Moorkens, and Joris Vreeke. 2014. Kanjingo - a mobile app for post-editing. In Proceedings of the 17th Annual Conference of the European Association for Machine Translation.
Lane Schwartz, Isabel Lacruz, and Tatyana Bystrova. 2015. Effects of word alignment visualization on post-editing quality & speed. Proceedings of MT Summit XV, 1:186-199.
Carlos S.C. Teixeira, Joss Moorkens, Daniel Turner, Joris Vreeke, and Andy Way. 2019. Creating a multimodal translation tool and testing machine translation integration using touch and voice. Informatics, 6.
Dimitri Theologitis. 1998. Language tools at the EC translation service: The theory and the practice. In Proceedings of the 20th Conference Translating and the Computer, pages 12-13.
Antonio Toral, Martijn Wieling, and Andy Way. 2018. Post-editing effort of a novel with statistical and neural machine translation. Frontiers in Digital Humanities, 5:9.
Olga Torres-Hostench, Joss Moorkens, Sharon O'Brien, Joris Vreeke, et al. 2017. Testing interaction with a mobile MT post-editing app. *Translation & Interpreting*, 9(2):138.
Vincent Vandeghiste, Tom Vanallemeersch, Liesbeth Augustinus, Bram Bulté, Frank Van Eynde, Joris Pelemans, Lyan Verwimp, Patrick Wambacq, Geert Heyman, Marie-Francine Moens, et al. 2019. Improving the translation environment for professional translators. Informatics, 6(2):24.
Vincent Vandeghiste, Tom Vanallemeersch, Liesbeth Augustinus, Joris Pelemans, Geert Heyman, Iulianna van der Lek-Ciudin, Arda Tezcan, Donald Degraen, Jan van den Bergh, Lieve Macken, et al. 2016. Scale - Smart Computer-Aided Translation Environment. Baltic Journal of Modern Computing, 4(2):382-382.
Mihaela Vela, Santanu Pal, Marcos Zampieri, Sudip Kumar Naskar, and Josef van Genabith. 2019. Improving CAT tools in the translation workflow: New approaches and evaluation. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, pages 8-15.
Julian Wallis. 2006. Interactive Translation vs Pretranslation in the Context of Translation Memory Systems: Investigating the Effects of Translation Method on Productivity, Quality and Translator Satisfaction. Ph.D. thesis, University of Ottawa.
Masaru Yamada. 2015. Can college students be posteditors? An investigation into employing language learners in machine translation plus post-editing settings. Machine Translation, 29(1):49-67.
Marcos Zampieri and Mihaela Vela. 2014. Quantifying the influence of MT output in the translators' performance: A case study in technical translation. In Proceedings of the EACL 2014 Workshop on Humans and Computer-assisted Translation, pages 93-98.
Julián Zapata. 2016. Translating on the go? Investigating the potential of multimodal mobile devices for interactive translation dictation. Tradumática: Traduccioni Tecnologies de la Informacion i la Comunicacion, 1(14):66-74.
Julián Zapata, Sheila Castilho, and Joss Moorkens. 2017. Translation dictation vs. post-editing with cloud-based voice recognition: A pilot experiment. Proceedings of MT Summit XVI, 2.
Anna Zaretskaya and Miriam Seghiri. 2018. User Perspective on Translation Tools: Findings of a User Survey. Ph.D. thesis, University of Malaga.
Anna Zaretskaya, Mihaela Vela, Gloria Corpas Pastor, and Miriam Seghiri. 2016. Comparing post-editing difficulty of different machine translation errors in Spanish and German translations from English. International Journal of Language and Linguistics, 3(3):91-100.
Andre Zenner and Antonio Krüger. 2017. Shifty: A weight-shifting dynamic passive haptic proxy to enhance object perception in virtual reality. IEEE Transactions on Visualization and Computer Graphics, 23(4):1285-1294.