SlowGuess/ACL_2020_OCR huggingface dataset:
Nico Herbig<sup>1</sup>, Tim Düwel<sup>1</sup>, Santanu Pal<sup>1,2</sup>, Kalliopi Meladaki<sup>1</sup>, Mahsa Monshizadeh<sup>2</sup>, Antonio Krüger<sup>1</sup>, Josef van Genabith<sup>1,2</sup>
German Research Center for Artificial Intelligence (DFK)  
Saarland Informatics Campus, Germany  
<sup>2</sup>Department of Language Science and Technology, Saarland University, Germany  
{firstname_lastname}@dfki.de  
{firstname_lastname}@uni-saarland.de
Current advances in machine translation (MT) increase the need for translators to switch from traditional translation to post-editing (PE) of machine-translated text, a process that saves time and reduces errors. This affects the design of translation interfaces, as the task changes from mainly generating text to correcting errors within otherwise helpful translation proposals. Since this paradigm shift offers potential for modalities other than mouse and keyboard, we present MMPE, the first prototype to combine traditional input modes with pen, touch, and speech modalities for PE of MT. The results of an evaluation with professional translators suggest that pen and touch interaction are suitable for deletion and reordering tasks, while they are of limited use for longer insertions. On the other hand, speech and multi-modal combinations of select & speech are considered suitable for replacements and insertions but offer less potential for deletion and reordering. Overall, participants were enthusiastic about the new modalities and saw them as good extensions to mouse & keyboard, but not as a complete substitute.
As machine translation (MT) has been making substantial improvements in recent years $^{1}$ , more and more professional translators are integrating this technology into their translation workflows (Zaretskaya et al., 2016; Zaretskaya and Seghiri, 2018). The process of using a pre-translated text as a basis and improving it to create the final translation is called post-editing (PE). Older research showed a strong dislike of translators towards PE (Lagoudaki, 2009; Wallis, 2006), and more recent studies agree that translators are still cautious about PE and question its benefits (Gaspari et al., 2014; Koponen,
2012), partly because they see it as a threat to their profession (Moorkens, 2018). Experienced translators in particular exhibit rather negative attitudes (Moorkens and O'Brien, 2015). Conversely, novice translators have been shown to have more positive views on PE (Yamada, 2015). Green et al. (2013) demonstrated that some translators actually strongly prefer PE and argue that "users might have dated perceptions of MT quality".
Apart from translators' preference, productivity gains of  $36\%$  when using modern neural MT for PE (Toral et al., 2018) already result in substantial changes in translation workflows (Zaretskaya and Seghiri, 2018) and will probably continue to do so the better MT becomes. Thus, PE requires thorough investigation in terms of interface design, since the task changes from mostly text production to comparing and adapting MT and translation memory (TM) proposals, or put differently, from control to supervision. Previous elicitation-based research (Herbig et al., 2019a) investigated how translation environments could better support the PE process and found that translators envision PE interfaces relying on touch, pen, and speech input combined with mouse and keyboard as particularly useful. A small number of prototypes exploring some of these modalities also showed promising results (Teixeira et al., 2019).
This paper presents MMPE, the first translation environment combining standard mouse & keyboard input with touch, pen, and speech interactions for PE of MT. The results of a study with 11 professional translators show that participants are enthusiastic about having these alternatives, even though time measurements and subjective ratings do not always agree. Overall, pen and touch modalities are well suited for deletion and reordering operations, while speech and multi-modal interaction are suitable for insertions and replacements.
$^{1}$ WMT 2019 translation task: http://matrix.statmt.org/, accessed 16/04/2020
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1691-1702 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
In this section, we present related research on translation environments and particularly focus on existing multi-modal approaches to PE.
2.1 CAT and Post-Editing
Most professional translators nowadays use so-called CAT (computer-aided translation) tools (van den Bergh et al., 2015). These provide features like MT and TM together with quality estimation and concordance functionality (Federico et al., 2014), alignments between source and MT (Schwartz et al., 2015), interactive MT offering assistance like auto-completion (Green et al., 2014b,a), or intelligibility assessments (Coppers et al., 2018; Vandeghinste et al., 2016, 2019).
While TM is still often valued higher than MT (Moorkens and O'Brien, 2017), a recent study by Vela et al. (2019) shows that professional translators who were given a choice between translation from scratch, TM, and MT, chose MT in  $80\%$  of the cases, highlighting the importance of PE of MT. Regarding the time savings achieved through PE, Zampieri and Vela (2014) find that PE was on average  $28\%$  faster for technical translations, Aranberri et al. (2014) show that PE increases translation throughput for both professionals and lay users, and Läubli et al. (2013) find that PE also increases productivity in realistic environments. Furthermore, it has been shown that PE not only leads to reduced time but also reduces errors (Green et al., 2013).
Furthermore, PE changes the interaction pattern (Carl et al., 2010), leading to a significantly reduced amount of mouse and keyboard events (Green et al., 2013). Therefore, we believe that other modalities or combinations thereof might be more useful for PE.
2.2 Multi-Modal Approaches
Dictating translations dates back to the time when secretaries transcribed dictaphone content on a typewriter (Theologitis, 1998); however, the use of automatic speech recognition also has a long history for translation (Dymetman et al., 1994; Brousseau et al., 1995). A more recent approach, called SEECAT (Martinez et al., 2014), investigates the use of automatic speech recognition (ASR) in PE and argues that its combination with typing could boost productivity. A survey regarding speech usage with PE trainees (Mesa-Lao, 2014) finds that they have a positive attitude towards
speech input and would consider adopting it, but only as a complement to other modalities. In a small-scale study, Zapata et al. (2017) found that ASR for PE was faster than ASR for translation from scratch. Due to these benefits, commercial CAT tools like memoQ and MateCat are also beginning to integrate ASR.
The CASMACAT tool (Alabau et al., 2013) allows the user to input text by writing with e-pens in a special area. A vision paper (Alabau and Casacuberta, 2012) proposes to instead use e-pens for PE sentences with few errors in place and showcases symbols that could be used for this. Studies on mobile PE via touch and speech (O'Brien et al., 2014; Torres-Hostench et al., 2017) show that participants especially liked reordering words through touch drag and drop, and preferred voice when translating from scratch, but used the iPhone keyboard for small changes. Zapata (2016) also explores the use of voice- and touch-enabled devices; however, the study did not focus on PE, and used Microsoft Word instead of a proper CAT environment.
Teixeira et al. (2019) explore a combination of touch and speech for translation from scratch, translation using TM, and translation using MT. In their studies, touch input received poor feedback since (a) their tile view (where each word is a tile that can be dragged around) made reading more complicated, and (b) touch insertions were rather complex to achieve within their implementation. In contrast, integrating dictation functionality using speech was shown to be quite useful and even preferred to mouse and keyboard by half of the participants.
The results of an elicitation study by Herbig et al. (2019a) indicate that pen, touch, and speech interaction should be combined with mouse and keyboard to improve PE of MT. In contrast, other modalities like eye tracking or gestures were seen as less promising.
In summary, previous research suggests that professional translators should switch to PE to increase productivity and reduce errors; however, translators themselves are not always eager to do so. It has been argued that the PE process might be better supported by using different modalities in addition to the common mouse and keyboard approaches, and an elicitation study suggests concrete modalities that should be well suited for various editing tasks. A few of these modalities have already been explored in practice, showing promising results. However, the elicited combination of pen, touch,
and speech, together with mouse and keyboard, has not yet been implemented and evaluated.
3 The MMPE Prototype
We present the MMPE prototype (see Figure 1) which combines these modalities for PE of MT. A more detailed description of the prototype can be found in Herbig et al. (2020), and a video demonstration is available at https://youtu.be/H2YM2R8Wfd8.
3.1 Apparatus & Overall Layout
On the software side, we decided to use Angular for the frontend, and node.js for the backend. As requested in Herbig et al. (2019a), we use a large tiltable touch & pen screen for the study (see Figure 1b): the Wacom Cintiq Pro 32 inch display with the Flex Arm that allows the screen to be tilted and moved flat on the table, or to be moved up to work in a standing position. We further use the Sennheiser PC 8 Headset for speech input. The goal of this hardware setup was to limit induced bias as much as possible, in order to get results on the modalities and not on a flawed apparatus.
We implemented a horizontal source-target layout (see Figure 1a), where each segment's status (unedited, edited, confirmed) is visualized between source and target. On the far right, support tools are offered as requested in Herbig et al. (2019a): (1) the unedited MT output, to which the users can revert their editing using a button, and (2) a corpus combined with a dictionary.
The current segment is enlarged, thereby offering space for handwritten input and allowing the user to view a lot of context while still seeing the current segment in a comfortable manner (Herbig et al. (2019a); see Figure 1a). The view for the current segment is further divided into the source segment (left) and two editing planes for the target, one for handwriting and drawing gestures (middle), and one for touch deletion & reordering, as well as standard mouse and keyboard input (right). Both initially show the MT proposal and synchronize on changes to either one. The reason for having two editing fields instead of only one is that some interactions are overloaded, e.g., a touch drag can be interpreted as both handwriting (middle) and reordering (right). Undo and redo functionality, as well as confirming segments, are also implemented through buttons between the source and target texts, and can further be triggered through hotkeys. The
target text is spell-checked, as a lack of this feature was criticized in Teixeira et al. (2019).
3.2 Left Target View: Handwriting
For handwriting recognition (see Figure 1c), we use the MyScript Interactive Ink SDK. Apart from merely recognizing the written input, it offers gestures like strike-through or scribble for deletions. For inserting words, one can directly write into an empty space, or create such a space first by breaking the line (draw a long line from top to bottom), and then handwriting the word. All changes are immediately interpreted, i.e., striking through a word deletes it immediately, instead of showing it in a struck-through visualization. The editor further shows the recognized text immediately at the very top of the drawing view in a small gray font, where alternatives for the current recognition are offered. Apart from using the pen, the user can also use his/her finger or the mouse on the left-hand editing view for handwriting.
3.3 Right Target View: Touch Reordering, Mouse & Keyboard
On the right-hand editing view, the user can delete words by simply double-tapping them with pen/finger touch, or reorder them through a simple drag and drop procedure (see Figure 1d), which visualizes the picked-up word as well as the current drop position, and automatically fixes spaces between words and punctuation marks. This reordering functionality is strongly related to Teixeira et al. (2019); however, only the currently dragged word is temporarily visualized as a tile to offer better readability. Naturally, the user can also edit using mouse and keyboard, where all common navigation inputs work as expected from other software.
3.4 Speech Input
For speech recognition, we stream the audio recorded by the headset to IBM Watson servers to receive a transcription, which is then analyzed in a command-based fashion. Thus, our speech module not only handles dictations as in Teixeira et al. (2019), but can correct mistakes in place.
As commands, the user has the option to "insert", "delete", "replace", and "reorder" words or subphrases. To specify the position, if it is ambiguous, one can define anchors as in "after"/"before"/"between", or define the occurrence
2see https://developer.myscript.com/docs/concepts/editing-gestures/,accessed 16/04/2020
(a) Screenshot of the interface.
(b) Apparatus.
(c) Handwriting on left target view.
(d) Touch reordering on right target view.
Figure 1: Overview of the MMPE prototype.
of the entity ("first"/"second"/"last"). A full example is "insert A after second B", where A and B can be words or subphrases. Character-level commands are not supported, so instead of e.g. deleting a suffix, one should replace the word.
3.5 Multi-Modal Combinations
Last, the user can use a multi-modal combination, i.e., pen/touch/mouse combined with speech. For this, the cursor first needs to be positioned on or next to a word, or the word needs to be long-pressed with pen/touch, resulting in a pickup visualization. Afterwards, the user can then use a simplified voice command like "delete", "insert  $A$ ", "move after/before  $A$ / between  $A$  and  $B$ ", or "replace by  $A$ " without needing to specify the position/word.
3.6 Logging
In a log file, we store all concrete key presses, touched pixel coordinates, etc. Much more importantly, we directly log all UI interactions (like segmentChange), as well as all text manipulations (like replaceWord) together with the concrete changes (e.g. with the oldWord, newWord, and complete segmentText).
4 Evaluation Method
The prototype was evaluated by professional translators<sup>3</sup>. We used EN-DE text, as our participants were German natives and we wanted to avoid ASR recognition errors as reported in Dragsted et al. (2011). In the following, "modalities" refers to Touch (T), Pen (P), Speech (S), Mouse & Keyboard (MK), and Multi-Modal combinations (MM, see Section 3.5), while "operations" refers to Insertions, Deletions, Replacements, and Reorderings. The experiment consisted of the following phases and took approximately 2 hours per participant:
4.1 Introduction & Independent PE
First, participants filled in a questionnaire capturing demographics as well as information on CAT usage. Then the experimenter introduced all of the prototype's features in a prepared order to ensure a similar presentation for all participants.
After that, participants were given 10-15 minutes to explore the prototype on their own. We
3The study has been approved by the university's ethical review board. Freelance participants were paid their usual fee, while in-house translators participated during working hours. The data and analysis scripts can be found at https://mmpe.dfki.de/data/ACL2020/
specifically told them that we are more interested in them exploring the presented features than in receiving high-quality translations. This phase had two main purposes: (1) to let the participants become familiar with the interface (e.g., how best to hold the pen) and to resolve questions early on; (2) to see how participants intuitively work with the prototype. Two experimenters carefully observed the participants and took notes on interesting behavior and questions asked.
4.2 Feature-Wise & General Feedback
The central part of the study was a structured test of each modality for each of our four operations. For this, we used text from the WMT news test set 2018. Instead of actually running an MT system, we manually introduced errors into the reference set to ensure that there was only a single error per segment. Overall, four sentences had to be corrected per operation using each modality, which results in  $4 \times 4 \times 5 = 80$  segments per participant. Within the four sentences per operation, we tried to capture slightly different cases, like deleting single words or a group of words. For this, we adapted the prototype, such that a pop-up occurs when changing the segment, which shows (1) the operation to perform and which modality to use, (2) the source and the "MT", which is the reference with the introduced error, as well as (3) the correction to apply, which uses color, bold font, and strike-through to easily show the required change to perform. The reason why we provided the correction to apply was to ensure a consistent editing behavior across all participants, thereby making subjective ratings and feedback as well as time measurements comparable. The logging functionality was extended, such that times between clicking "Start" and confirming the segment were also logged.
To avoid ordering effects, the participants went through the operations in counter-balanced order, and through the modalities in random order. After every operation (i.e., after  $4 \times 5 = 20$  segments) and similar to Herbig et al. (2019a), participants rated each modality for that operation on three 7-point Likert scales ranging from "strongly disagree" to "strongly agree", namely as to whether the interaction "is a good match for its intended purpose", whether it "is easy to perform", and whether it "is a good alternative to the current mouse and keyboard approach". Furthermore, we asked the translators to give us their thoughts on advantages and disad
vantages of the modalities, and how they could be improved. Afterward, participants further had to order the 5 modalities from best to worst.
In the end, after completing all 80 segments, we performed a final unstructured interview to capture high-level feedback on the interface as well as things we missed in our implementation.
4.3 Remarks Regarding Methodology
While a direct comparison to state-of-the-art CAT tools would be interesting, the results would be highly questionable as the participants would be expert users of their day-to-day tool and novice users of our tool. Furthermore, the focus of our prototype was on the implemented modalities, while widely used features like a TM or consistency checker are currently missing. Since our main question was whether the newly implemented features have potential for PE of MT or not, we focus on qualitative feedback, ratings, and timing information, which is more relevant to this research question.
5 Evaluation Results and Discussion
In this section, we present and discuss the study's main findings.
5.1 Participants
Overall, 11 (f=10, m=1, 2 left-handed) professional EN-DE translators participated in the experiment, 3 freelance and 8 in-house translators. Their ages ranged from 30 to 64 (avg=41.6,  $\sigma = 9.3$ )<sup>4</sup>, with 3 to 30 years of professional experience (avg=13.3,  $\sigma = 7.4$ ) and a total of 27 language pairs (avg=2.6). All translators translate from EN to DE, and all describe their German Language skills as native and their English skills as C1 to native level. For most participants, the self-rated CAT knowledge was good (6 times) or very good (4 times, 1 neutral). However, participants were less confident about their PE skills (4 neutral, 4 good, 3 very good), thereby matching well with the CAT usage surveys. Years of experience with CAT tools ranged from 3 to 20 (avg=11.5,  $\sigma = 5.1$ ), where participants had used between 1 and 10 distinct CAT tools (avg=4.9,  $\sigma = 2.7$ ).
5.2 Subjective Ratings
Figure 2 shows the subjective ratings provided for each modality and operation on the three scales
4The small number of participants and their age distribution (with 10 participants of age 30 to 48, and only one aged 64) did not us allow to analyze the effect of age on the results.
"Goodness", "Ease of use", and "Good alternative to mouse & keyboard" after having tested each feature (see Section 4.2). As can be seen, participants tended to give similar ratings on all three scales.
For insertions and replacements, which required the most text input, the classical mouse & keyboard approach was rated highest; however, the multi-modal combination and speech were also perceived as good, while pen and especially touch received lower scores.
For deletions and reorderingings, pen, touch, and mouse & keyboard were all perceived as very good, where P and T were ranked even slightly higher than MK for reorderingings. Speech and multi-modal were considered worse here.
5.3 Orderings
After each operation, participants ordered the modalities from best to worst, with ties being allowed. As an example, for "MM & S best, then P, then MK, and last T" we assigned 0.5 times the  $1^{\text{st}}$  and 0.5 times the  $2^{\text{nd}}$  position to both MM and S, while P got  $3^{\text{rd}}$ , MK  $4^{\text{th}}$ , and T the  $5^{\text{th}}$  position. To get an overall ordering across participants, we then multiplied the total amount of times a modality was rated  $1^{\text{st}}/2^{\text{nd}}/3^{\text{rd}}/4^{\text{th}}/5^{\text{th}}$  by  $1/2/3/4/5$  (similar to Zenner and Krüger (2017)). Consequently, a lower score indicates that this modality is better suited for the operation. The scores for each modality and operation are:
- Insertions:  $1^{\mathrm{st}}$ : MK(20.5),  $2^{\mathrm{nd}}$ : MM(26.5),  $3^{\mathrm{rd}}$ : S(31.5),  $4^{\mathrm{th}}$ : P(38.5),  $5^{\mathrm{th}}$ : T(48)
- Deletions:  $1^{\mathrm{st}}$ :  $\mathbf{P}(21.5)$ ,  $2^{\mathrm{nd}}$ :  $\mathbf{MK}(29)$ ,  $3^{\mathrm{rd}}$ :  $\mathbf{T}(31.5)$ ,  $4^{\mathrm{th}}$ :  $\mathbf{MM}(41)$ ,  $5^{\mathrm{th}}$ :  $\mathbf{S}(42)$
- Replacements:  $1^{\mathrm{st}}$ : MK(21),  $2^{\mathrm{nd}}$ : MM(29),  $3^{\mathrm{rd}}$ : S(30),  $4^{\mathrm{th}}$ : P(35),  $5^{\mathrm{th}}$ : T(50)
Reorderings:  $1^{\mathrm{st}}$  : P(21.5),  $2^{\mathrm{nd}}$  : T(31),  $3^{\mathrm{rd}}$  : S(35.5),  $4^{\mathrm{th}}$  : MK(36),  $5^{\mathrm{th}}$  : MM(41)
5.4 Timings
We analyzed the logged duration of each modality-operation pair. Note that this is the time from clicking "Start" until confirming the segment; thus, it includes recognition times (for speech and handwriting) and really measures how long it takes until a participant is satisfied with the edit. Even though participants were instructed to provide feedback or ask questions only while the popup is shown, i.e., while the time is not measured, participants
infrequently did so during editing. We filtered out such outliers and averaged the 4 sentences of each modality-operation pair per participant to get a single value, thereby making the samples independent for the remaining analyses.
Figure 3 shows boxplots of the dataset for the 20 modality-operation pairs. For statistical analysis, we first conducted Friedman tests per operation, showing us that significant differences exist for each operation (all  $p < 0.001$ ). Afterward, posthoc analyses using Wilcoxon tests with Bonferroni-Holm correction showed which pairs of modalities are significant and how large the effect  $r$  is.
For insertions, MK was by far the fastest modality, followed by MM and S. All differences except for MM vs. S and T vs. P are statistically significant with large effect sizes (all  $p < 0.01$ , all  $r > 0.83$ ).