Venue: NAACL 2021

# Backtranslation Feedback Improves User Confidence in MT, Not Quality

Vilem Zouhar1, Michal Novak1, Matus Zilinec1, Ondrej Bojar1, Mateo Obregon2, Robin L. Hill2, Frederic Blain3,4, Marina Fomicheva3, Lucia Specia3, Lisa Yankovskaya5

###### Abstract

Translating text into a language unknown to the text's author, dubbed outbound translation, is a modern need for which the user experience has significant room for improvement, beyond the basic machine translation facility. We demonstrate this by showing three ways in which user confidence in the outbound translation, as well as its overall final quality, can be affected: backward translation, quality estimation (with alignment) and source paraphrasing. In this paper, we describe an experiment on outbound translation from English to Czech and Estonian. We examine the effects of each proposed feedback module and further focus on how the quality of machine translation systems influence these findings and the user perception of success. We show that backward translation feedback has a mixed effect on the whole process: it increases user confidence in the produced translation, but not the objective quality.

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 151–161

June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

When dealing with machine translation (MT) on the web, most of the attention of the research community is paid to _inbound translation_. In this scenario, the recipients are aware of the MT process, and thus it is their responsibility to interpret and understand the translated content correctly. For an MT system, it is sufficient to achieve such quality that allows a recipient to get the gist of the meaning of texts on webpages.

For _outbound translation_, it is the other way round: the responsibility to create the content in the way that it is correctly interpreted by a recipient lies on the authors of the message. The main issue is that the target language might be entirely unknown to them. Prototypically it is communication by email, filling in foreign language forms, or involving some other kind of interactive medium. The focus in this scenario is placed not only on producing high-quality translations but also on measuring the author that the MT output is correct.

One of the approaches to improving both quality and authors' confidence, first employed in this scenario by Zouhar and Bojar (2020), is to provide cues that indicate the quality of MT output as well as suggest possible rephrasing of the source. They may include backward translation to the source language, highlighting of the potentially problematic parts of the input, or suggesting paraphrases. Except for preliminary work by Zouhar and Novak (2020), the impact of individual cues has not yet been properly explored.

In this paper, we present the results of a new experiment on outbound translation. Building on the previous works, the focus was expanded to investigate the influence of different levels of performance of the underlying MT systems, as well as utilizing a much greater range and diversity of participants and evaluation methods.

Native English speakers were tasked to produce text either in Czech or in Estonian with an outbound translation system in an e-commerce context. Every user also reported a confidence score upon finishing each stimulus trial. A native Czech or Estonian speaker later evaluated each final translation for fluency and adequacy. The set of available cues varied for each participant from stimuli to stimuli, following a controlled experimental design, in order to determine the impact of specific combinations of cues on the self-reported confidence and the final translation quality.

For our study, we made use of the Ptakopat system Zouhar (2020). This bespoke software was specifically developed to examine user behavior when testing machine translation user interfaces, especially in the context of outbound translation.1

Footnote 1: The code for this project and also the experiment data are available as open-source.github.com/zouharvi/ptakopet

The structure of the paper is as follows. After an overview of the related work in Section 2, we

present the environment for the outbound translation we used for the experiment, including the MT systems and modules that provided cues to the users, in Section 3. Section 4 describes the data that we collected during the experiment, and in Section 5 we further analyze them to reveal and discuss various aspects of our approach to outbound translation. We conclude with the main findings in Section 6.

## 2 Related Work

Despite recent advances in neural machine translation (NMT) quality, resulting in output comparable to human professionals in specific settings Hassan et al. (2018); Popel et al. (2020), it is far from reasonable to blindly believe that the output of MT systems is perfectly accurate. It should thus not be simply included in an email or another message without some means of verification. Feedback in this scenario is needed, which would tell users if the translation is correct and ideally even give instructions on how to improve it.

A related area of interactive machine translation (IMT) focuses mainly on either post-editor scenarios Martinez-Gomez et al. (2012); Sanchis-Trilles et al. (2014); Underwood et al. (2014); Alabau et al. (2016) or generally scenarios in which users are able to produce the translation themselves and the system only aims to speed it up or improve it Santy et al. (2019).

Outbound translation differs from common IMT scenarios by the fact that the user does not speak the target language, and hence operates on the MT result only in a limited way.

The first work to deal with this task by Zouhar and Bojar (2020) focused on working with Czech German MT in context of asking and reformulating questions. A preliminary experiment on the effect of translation cues has been carried out by Zouhar and Novak (2020), but it was conducted on a much smaller scale both in terms of participants and annotators and with non-native speakers of English. This may have affected the results that differ in some aspects, especially in the usefulness of the word-level quality estimation.

## 3 Environment for Testing Outbound Translation

In order to test the effect of different cues, we utilized Prakopet, a web-based tool for outbound translation. The tool provides machine translation together with cues in the form of backward translation, quality estimation and paraphrasing. These cues are intended to help the user arrive at a better translation and increase their confidence in the produced output. The tool is modular, allowing the modules for MT and cues to be either replaced with others or turned on and off.

By linking a collection of sample stimuli to the tool it can also be used to conduct experiments. Participants are asked to react to stimuli by formulating texts in a language known to them and producing and editing translations in a language they do not know. The set of cues they are pre

Figure 1: Screenshot of the Ptkopét experiment interface with all modules enabled. Only the two white text areas on the right are editable. The first is the source (from which forward translation is made) and the second is the final output (from which backtranslation is shown). Editing the second text area was purely optional as it was a language unknown to the participant.

 

sented with may vary. The users are also asked to report their confidence in the produced output.

In this experiment, each participant was presented with a sequence of scenes, interacting with the outbound translation system in each of them. Figure 1 shows an example of a scene and user interaction. In the following sections, we describe the main components of the experiment.

### Stimuli

We used screenshots of web forms (real-world examples from the e-commerce domain) as stimuli. Every screenshot displayed an excerpt of a web form containing a text field for open queries with a specific query already pre-filled and highlighted in a green rectangle. For example, Figure 1 shows a form at hotel webpages with a pre-filled special request.

This query, or rather its message, is what should be translated. Apart from the query, the screenshot captured elements of the webpage that should make it easier and faster for the user to understand the intended message and its context. The stimuli are also accompanied by a short description of the website's domain (e.g. _acommodation_) above the screenshot for the same purpose.

The dataset consists of 70 screenshots and corresponding pre-filled queries in English.2 It was selected from a collection of 462 such screenshots, collated by six annotators.3 The annotators were instructed to look for web forms with text boxes that could be filled with text which would require translation. We were not interested in fields such as names, addresses, numbers or pre-defined lists of values (e.g. countries). We emphasized that the collection should consist of a broad variety of domains, but the particular choice of domains and websites was up to the annotators.

Footnote 2: As the pre-filled queries were conceived by non-native speakers of English, they may contain grammatical errors. The intention behind them is always understandable, though.

Footnote 3: Available at hdl.handle.net/11234/1-3622.

### Modules

The set of available modules (backward translation BT, quality estimation QE, paraphrasing PP), as well as the choice of the MT system, was randomized for every user for every stimulus. We denote a specific cue configuration by the modules present, e.g. BTPP. Figure 1 shows an example of modules' outputs, given a user's rephrasing of the query from the stimulus.

Machine Translation.We used three MT systems for Czech (differing in speed and training data size) and one for Estonian. All of the systems were trained in both directions: the forward systems translate from English, whereas the opposite direction is used as a backward translation cue. All the MT systems follow the Transformer model architecture Vaswani et al. (2017) design, though student systems make use of the simplified simple recurrent unit and other modifications described in Germann et al. (2020). Table 1 shows how the MT systems performed in terms of BLEU score Papineni et al. (2002) on the test set of WMT18 News task Bojar et al. (2018).

The _Czech 3_ system is the winning MT model of Czech-English News Translation in WMT 2019 Popel et al. (2019), having been trained on 58M authentic sentence pairs and 65M backtranslated monolingual sentences.4

Footnote 4: In the opposite direction, 48M monolingual sentences have been used to create synthetic data.

The training proposed by Germann et al. (2020) was used for a CPU-optimized student model _Czech 2_. It was created by the knowledge distillation Kim and Rush (2016) method on translations generated by Czech 3. Although it has been trained solely on synthetic data, its performance in the news domain falls behind the teacher only by 0.5 to 3.0 BLEU points, depending on the translation direction. We included it mainly due to its speed as shown in Section 4.

The design of the _Czech 1_ system is identical to Czech 3. The only difference is that the former was trained only on a subsample of 5M sentence pairs from CzEng 1.7 Bojar et al. (2016). This system was chosen to simulate performance on less resourceful language pairs.

The _Estonian_ system uses the same construction procedure as Czech 2. The teacher system utilized in knowledge distillation was internally trained for us by the authors of Germann et al. (2020).

\begin{table}
\begin{tabular}{l c r} \hline \hline  & \multicolumn{1}{c}{\(X\leftarrow\)EN} & \multicolumn{1}{c}{\(X\rightarrow\)EN} \\ \hline Czech 1 & 19.57 & 25.04 \\ Czech 2 & 23.85 & 32.71 \\ Czech 3 & 26.00 & 33.11 \\ Estonian & 25.85 & 31.61 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance of utilized MT systems in BLEU score evaluated on WMT18 test set; higher is better.

 

Quality Estimation.QE is the task of predicting the quality of an MT output without relying on reference translation, as opposed to traditional evaluation based on automatic metrics (BLEU, TER, etc.). We have used QE to predict potential translation errors at the word-level which in turn, combined with a source-target token-level alignment algorithm,5 enables us to identify the source words that have led to those translation errors. QE suggestions are presented by red word highlighting (see Figure 1).

Footnote 5: It was provided by FastAlign [43] models trained on bitext from CzEng 2.0 [67] and OPUS collection [98] for English-Czech and English-Estonian, respectively. Measured on 10 queries sampled from the dataset of stimuli and their translations produced by the Czech 3 and Estonian systems, the F1 score of English tokens alignment exceeds 80% in both cases.

We note that word-level error annotation is a hard and costly task. Thus, available data for building systems to predict word-level errors is scarce. To circumvent this issue we relied on a feature-based approach which exploited information from the neural MT system (i.e. a glass-box approach to QE) and did not require large amounts of data for training. Glass-box features have been successfully used for QE of statistical MT [10, 11] and have been recently shown to be effective for sentence-level QE of neural MT systems [12]. To accommodate for the different types of MT models used in this work, including a student model _Czech 2_, we did not use the full set of features from [12] but instead relied on simple subset of log-probability based features:

* Log-probability of the word
* Log-prob. of the previous word
* Log-prob. of the next word
* Average log-prob. of the translated sentence
* Number of characters in the word

We build a binary gradient boosting classifier to predict word-level quality. To train the classifier we collected a small curated dataset with translation error annotation. Although the annotation is binary6 (OK/BAD class), the dataset is heavily imbalanced. To alleviate this issue, we over-sampled the minority class (BAD).

Footnote 6: In addition, each translated word labeled as BAD was manually annotated with a subcategory: minor, major or critical. However, due to the heavy imbalance of the data, we did not use this fine-grained annotation to train the QE system.

We randomly split the data for each MT system into train (80%) and test (20%). In addition to accuracy, we report F1 for each class and Matthews correlation coefficient (MCC) as proposed by [12] for imbalanced data. Table 2 shows these results for Estonian and Czech.

We observed that F1 for the BAD class is much lower than F1 for OK. This indicates the difficulty of our QE models in correctly predicting the minority class. The reasons for that are as follows. First, log-probabilities might not contain enough information to predict major or critical issues. In particular, critical issues concern the mistranslation of specific elements in the text (e.g. numbers or named entities), which is beyond the scope of the glass-box features used in our experiments. We plan to investigate other light-weight features that could better capture this information. Secondly, on average, MT quality is quite high (even for weaker models) and therefore, the vast majority of the words belong to the positive class.

Paraphraser.This module was expected to provide users with a potential rephrasing of their inputs from which they may draw inspiration for alternative translations. The paraphraser is based on pivoting, i.e. a round-trip translation via a pivot language. [10] showed that pivoting is an effective way of generating diverse paraphrases, especially if done via linguistically unrelated languages. A larger set of pivot languages should further increase the diversity of paraphrases.

Our paraphrasing system performed two-step English-to-English translation through 41 pivot languages. It is based on T2T-multi-big model from [10], a multi-lingual Transformer-big [11] model with a shared encoder and decoder. It has been trained on 231M sentence pairs sampled from the OPUS collection [98]. Given a sentence, the model yielded 41 variants. In order not to overwhelm users, the paraphrases are then grouped so that two paraphrases with the same bag of words

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & EN\(\rightarrow\)ET & \multicolumn{2}{c}{EN\(\rightarrow\)CS} \\  & & Czech 1 & Czech 3 \\ \hline Accuracy & 0.74 & 0.70 & 0.77 \\ F1\({}_{\text{BAD}}\) & 0.37 & 0.32 & 0.14 \\ F1\({}_{\text{OK}}\) & 0.83 & 0.81 & 0.87 \\ MCC & 0.28 & 0.23 & 0.12 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of the word-level QE system on the outputs of our MT systems.

 

excluding stop words end up in the same group. In the end, users are presented with a list of one random representative from each group, sorted by the group size in descending order. The paraphrases suggested by multiple languages should thus appear at the top. To achieve reasonable response time (ca. 3s), the service has been run on a GPU.

Table 3 shows the performance of the paraphraser in terms of BLEU score, evaluated on a subset of the Quora Question Pairs dataset.7 The subset consists of 4000 question pairs, with 2000 pairs containing real paraphrases, and 2000 containing similar sentences with a different meaning. The two cases are respectively denoted by \(+\) and \(-\). The produced outputs seem to be more similar to real paraphrases than to fake ones, which corresponds to what we observed for source sentences with twice as high BLEU scores.

Footnote 7: quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs

### Self-reported confidence

Users were asked to submit their rephrased English query and its translation by reporting their confidence in the produced translation. They specified how much they trusted the translation on a standard Likert scale from 1 (least) to 5 (most).

## 4 Data Collected in the Experiment

During a single scene, the participant saw a stimulus, worked on it and then finished it either by rating their confidence or by describing the reason for skipping. The participant was continuously presented with the translation output and the cues. We logged all incoming data as well as requests to the modules and their responses together with timestamps.

In total, \(52\) English speaking participants joined our experiment, out of whom \(49\) were native speakers of English. There were \(70\) scenes, each with a unique stimulus, prepared for every participant. After filtering out the scenes which we found invalid as they contained either no input from the users or were not finished, the total number of scenes to be analyzed was \(2486\). The participants thus succeeded in completing \(48\) scenes on average. As shown in Table 4, the distribution of completed scenes over different configurations appears to be balanced.

Since one of the goals of Ptakopet is to facilitate work with MT, we also focused on the time participants had to spent in the interface together with the number of their actions8 needed to finish stimuli. They are summarized in Table 4. It is clear that the short response times of student models Czech MT 2 and Estonian) encourage the users to perform more actions, while still spending less time on one scene on average.

Footnote 8: We measure actions by the number of forward translation requests because they are present in every configuration.

## 5 Evaluation and Results

Having recorded the essential interactions of participants with Ptakopet, we further analyzed the collected data, especially user inputs and their translations.

Viable inputs.Unless a participant skipped a scene, it was concluded by confirming the _final input_ and its translation. We were also interested in examining intermediate complete sentences which

\begin{table}
\begin{tabular}{l c c c c} \hline Config. & \# Scenes & Time [s] & Actions & Pace [s] \\ \hline Czech 1 & 610 & 59 & 1.85 & 44 \\ Czech 2 & 643 & 42 & 3.66 & 17 \\ Czech 3 & 601 & 52 & 1.66 & 41 \\ Estonian & 632 & 46 & 3.06 & 22 \\ \hline BT QE PP & 307 & 49 & 2.77 & 27 \\ BT QE & 331 & 47 & 2.70 & 25 \\ BT PP & 304 & 51 & 2.42 & 31 \\ QE PP & 298 & 55 & 3.11 & 27 \\ BT & 311 & 50 & 2.31 & 31 \\ QE & 304 & 46 & 2.84 & 25 \\ PP & 302 & 49 & 2.68 & 28 \\ - & 285 & 46 & 2.08 & 34 \\ \hline Total & 2486 & 49 & 2.62 & 28 \\ \hline \end{tabular}
\end{table}
Table 4: Summary of collected scenes, median time, mean number of actions, and median pace (time per one action) aggregated over all scenes across different configurations. Time and pace are in seconds; actions are computed from translation requests made. For the two variables involving time, median was used instead of mean in order to avoid the effect of outlier scenes where the user was inactive for a longer time period.

\begin{table}
\begin{tabular}{l c c c} \hline  & Source & Ref\(+\) & Ref\(-\) \\ \hline Paraphrase & 35.46 & 13.59 & 7.42 \\ Source & – & 29.04 & 15.71 \\ \hline \end{tabular}
\end{table}
Table 3: Performance of the paraphraser in BLEU averaged across all languages. The produced paraphrase is compared either with the source sentence, or with the reference, which can be a real paraphrase (\(+\)), or a similar sentence with a different meaning (\(-\)). For illustration, a comparison of the source sentence with the two types of reference is added.

 

users considered and later abandoned. We call these _viable intermediate inputs_. The collection of such inputs was possible because the Ptakopet tool continually records user's interaction. We set the minimum time without any edit for an input to be sent to the forward translation module to 1000 ms. Despite this relatively long period, still many incomplete or erroneous inputs were recorded, perhaps while the user was deliberating. We thus used a simple heuristic to extract the viable ones.

For an input to be considered viable, it had to end with a full stop, an exclamation mark, or the same token as the final input ended. Furthermore, its length had to be within a 25% margin around the length of the final input without whitespaces.9

Footnote 9: This rule discredits inputs meant to be viable, where the very last token was later edited, though. Manual examination of the data verified the efficacy of the heuristic.

Whereas each confirmed scene by design resulted in \(1\) final input and its translation, the number of intermediate viable inputs (non-final) was \(0.62\). Their average length was \(98.43\%\) of the final input.

Evaluation of translation quality.The extracted viable inputs and their translations were rated for quality and adequacy by 12 Czech and 3 Estonian native speakers. For each viable input, the annotators were shown the source, its translation and the corresponding stimulus. They were asked to rate on the scale from 1 (least) to 5 (most) the following statements:

* _SRC-STI:_ The meaning of user input corresponds to what is entered in the form shown in the image.
* _TGT-SRC:_ The meaning of the translation corresponds to the user input.
* _TGT-STI:_ The meaning of the translation corresponds to what is entered in the form shown in the image.
* _Fluency:_ The translation is fluent (including typography, punctuation, etc.)
* _Overall:_ The overall translation quality including both adequacy with respect to the stimulus and fluency is high.

On average, we collected \(7.15\) assessments per viable input. The inter-rater agreement measured by Kripendorff's alpha was 0.47 and 0.48 for Czech and Estonian, respectively.

Data Normalization.Because of data imbalance in favor of high confidence, we normalized the self-reported user confidences using the following formula: \(x^{\prime}=\frac{x-min}{max-min}\times 4+1\). The \(min\) and \(max\) values were taken individually for every participant. This only affected those who never used \(1\) or \(2\) in their self-reported confidences. We did not apply this normalization to the quality annotations, because the annotators used the whole scale in almost all cases. The overall average of all confidence judgments decreased from \(3.72\) to \(3.59\) by this normalization.

This only helped with the imbalance a little.

Figure 2: Effect of different MT systems and the presence and absence of every module on self-reported user confidence and translation quality.

 

To avoid strong assumptions about the underlying process, we did not normalize the data to have zero mean and standard deviation of 1 for every feature dimension. This would also have made any interpretation less intuitive.

Results on final inputs.Table 5 shows the average evaluation scores of final confirmed inputs, accompanied by average self-confidence scores across various configurations. For clarity, we illustrate the same results in Figure 2. Comparing the Czech MT systems, their ranking with respect to the Overall score corresponds to the results of the automatic evaluation in the news domain shown in Table 1.

Interestingly enough, Czech 2 received an average confidence score comparable to its teacher model Czech 3 (see in Figure 2). The results of comparison across different combinations of cues suggest that configurations with backtranslation feedback enabled achieved better performance in terms of the overall quality. In such cases, the users also felt more confident. Unlike for overall quality, the effect of an available backward translation cue on user confidence was statistically significant by Mann-Whitney U test for \(0.6\) point difference (\(U=24243.5,p<0.0001\)).

Conversely, quality estimation cues appear not to be useful, which the users also noted. Unfortunately, the presence of paraphrases increased user confidence, but decreased the objective translation quality. These results are in contrast with the work of Zouhar and Novak (2020). We attribute this difference to an insufficient number of samples and also a more homogeneous composition of participants (all foreign PhD students studying in the Czech Republic) in their work.

Note that users who had knowledge of some other Slavic language (Polish or Russian) on average expressed higher confidence (\(3.95\)) and also produced translations of higher quality (\(4.44\)). The effects of different modules on their work were closer to the effects described in Zouhar and Novak (2020).

As seen in Figure 3, a significant proportion of the scenes (~41%) received 4 or 5 on both self-reported confidence and overall translation quality. Although these high scores are positive in terms of industry progress, it makes the quality-confidence dependency harder to analyze.

Table 6 shows expected rating behavior in terms of correlations. We can see that Fluency is mostly correlated with TGT-SRC and TGT-STI adequacies and less with SRC-STI adequacy, which should affect the translation fluency only slightly.10 We also see that TGT-STI adequacy and Fluency affects the Overall rating the most, which accords with its definition. Self-reported user confidence correlates the least with all the rest, but slightly more with TGT-STI, TGT-SRC and Overall scores, which we consider positive.

Footnote 10: In a scenario where the SRC-STI adequacy is lowered by typos in Source, which then also negatively affects the translation process and also the Fluency.

MT comparison in detail.Figure 4 shows the average spent time per stimulus as well as the number of forward translation requests and input length in characters with respect to the confidence and overall translation quality for submitted translations. The figure is split into three graphs, each corresponding to one of the Czech MT systems.

Input text length does not appear to affect the overall translation quality significantly, while it seems to affect users' self-reported confidence.

The curves for time spent, although different in

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & TGT & TGT & Fluency & Overall & Conf. \\  & SRC & STI & Fluency & Overall & Conf. \\ \hline SRC-STI & 0.07 & 0.64 & 0.23 & 0.50 & 0.08 \\ TGT-SRC & & 0.69 & 0.68 & 0.71 & 0.14 \\ TGT-STI & & & 0.67 & 0.88 & 0.15 \\ Fluency & & & & 0.84 & 0.13 \\ Overall & & & & 0.14 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Correlation between all quality annotations variables and self-reported user confidence.

Figure 3: Distribution (percentage and absolute count) of quality annotation (rounded overall) and self-reported user confidence.

 

absolute values, peak in the middle (rating 3) and have the lowest values for scores of 1 and 5. This may happen because the stimulus was either easy to complete, or the users did not work on this stimulus diligently. It is supported by the fact that they did not report low confidences in these instances. A similar trend, although less pronounced, can be seen with the number of requests.

We can also notice that the Czech 2 system has the lowest times despite also having a vastly higher number of executed requests. The request delay was the same for all MT systems, so in this case, the users recognized that they did not have to wait so long for getting a translation back and hence sent more requests. This is one of the possible explanations for why in Figure 2 the average self-reported confidence for this system is on par with its teacher model, Czech 3, despite being less performant objectively.

The degree of interactivity appears to be the main factor affecting these MT systems profiles. The figures of Czech 1 and Czech 3 look very similar even though they vary greatly in performance and only have their speeds in common (slower than Czech 2).

Intermediate vs. final.Having also intermediate viable inputs at our disposal, we explored how quality changes in the transition from intermediate to final inputs. We excluded those scenes that contain no viable intermediate input, which accounts for almost 69%.

Although our heuristics can filter out most of the intermediate inputs which are not viable, some of those remaining can be still considered defective. They may contain a typo, artifacts of unfinished rephrasing or may miss important information. These non-viable inputs must be excluded from the comparison, as the user would unlikely submit them or they could be easily fixed by a spell-checker. We manually examined all intermediate viables and excluded the defective ones from the following statistics.

Table 7 shows the average difference in the quality of intermediate and corresponding final inputs and their translations. The greatest improvement in the Overall score is again achieved by configurations utilizing backtranslation feedback, although the difference is not statistically significant. What is significant, though, are some

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Config & SRC & TGT & TGT & Fluency & Overall \\  & STI & SRC & STI & Fluency & Overall \\ \hline BT QE PP & -0.19\({}^{\ddagger}\) & +0.10 & +0.04 & +0.05 & +0.08 \\ BT QE & -0.14\({}^{\dagger}\) & +0.16\({}^{\diamond}\) & +0.03 & +0.04 & +0.03 \\ BT PP & -0.12 & +0.14 & +0.16 & +0.12 & +0.17 \\ QE PP & -0.20\({}^{\dagger}\) & +0.03 & -0.13* & +0.02 & -0.07*** \\ BT & -0.24\({}^{\ddagger}\) & +0.33\({}^{\lx@paragraphsign}\) & +0.10 & +0.10 & +0.11 \\ QE & -0.11* & -0.01 & -0.10 & -0.05 & -0.04 \\ PP & -0.11\({}^{\circ}\) & +0.09 & -0.05 & -0.00 & -0.04 \\ – & -0.02 & +0.16 & +0.16 & +0.04 & +0.06 \\ \hline Total & -0.15\({}^{\ddagger}\) & +0.11\({}^{\dagger}\) & +0.01 & +0.04 & +0.03 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Average difference of quality between intermediate viable and final inputs and their translations for all combinations of available cue modules. Statistical significance was calculated by Wilcoxon signed-rank test (\(p<0.05\text{$\ast$}\), \(0.01\text{$\char 6$}\), \(0.001\text{$\char 6$}\))

Figure 4: Relationship between the scores (self-reported user confidence in darker colors and overall translation quality in lighter colors) and average scene time, forward translation request count and input length in characters (y-axes). Best viewed in color.

 

TGT-SRC scores including the BT configuration. It shows that the translation of the final input is on average more adequate to the source than the translation of the intermediate inputs. Nevertheless, the effect on the TGT-STI adequacy is marginal due to negative differences in the SRC- STI adequacy score. These can be justified by the fact that any modification of the original query in the stimulus might have been considered as a shift in meaning by the annotators, although in reality the original intention could be still understandable.

In Table 8, we show three examples of the intermediate and the final inputs with their translations to Czech. In the top two, the rephrasing helped to improve the translation quality: (1) by adding a word "language" to prevent translating "English" as a Czech word for "Englishmen", or (2) by substituting a preposition. Conversely, the replacement of the verb "has expired" by a phrase "out of date" led to a drop in translation quality. This is due to a grammatical error and use of the Czech expression meaning "got obsolete", which indeed sounds old-fashioned in this context.

## 6 Conclusion

In this paper, we demonstrated through an experiment the effect of three translation cues on user confidence and translation quality.

The backward translation cue proves to be a powerful means to enhance user confidence in MT. At the same time, it neither increase nor decrease significantly the translation quality. The fact that backtranslation feedback has a marginal effect to objective quality but greatly increases user confidence is surprising because it is the most intuitive low-effort approach to outbound translation scenarios which can be done even with publicly available MT systems.

The paraphraser seems to increase user confidence less (compared to not being present), with no or slightly negative impact on the translation quality. Without a better method to generate diverse and still adequate paraphrases, employing this cue is questionable. The effect of word-level quality estimation appears to be even more questionable. We attribute it mainly to the underlying word-level models, which may not be mature enough for user-facing applications.

Despite the loss in objective translation quality, the CPU-optimized student MT model either managed to maintain its teacher's high trustworthiness or compensated for it by its speed.

Future workScores in both user confidence and overall translation quality annotation cluster together. Having the distribution less concentrated by changing the underlying task with stimuli or by working with more low resource languages could reveal stronger dependencies between individual variables.

We limited ourselves to only three baseline solutions to help in outbound translation. In the future work, inspiration could be drawn from the approaches of interactive machine translation systems and these could be adapted for the purposes of outbound translation.

## Acknowledgments

Sincere thanks to Chris Burns and the three anonymous reviewers for their thorough review and helpful comments.

This project has received funding from the grants H2020-ICT-2018-2-825303 (Bergamot) of the European Union and 19-26934X (NEUREM3) of the Czech Science Foundation. The work has also been supported by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2018101 LINDAT/CLARIAH-CZ.

\begin{table}
\begin{tabular}{l l} \hline \hline Inter & I teach my son **English** with the ‘Learning Time with Timmy’ series on Youtube. \\  & Uéim sveho syna **Angilkanç**/_Englishman_/se seriéalém_,_Learning Time with Timmy” na Voutube. \\ Final & I teach my son **English language** with the series ‘Learning Time with Timmy’ series on Youtube. \\  & Uéim sveho syna **maglicky jäx** sex serialen ,_Learning Time with Timmy” na Voutube. \\ \hline Inter & Why was I not able to make a payment by mobile? \\  & Proc jsenn remohl zaptalità raz /oqr’ mobil? \\ Final & Why was I not able to make a payment from my mobile? \\  & Proc jsem nemohl zaptalität z mohitu? \\ \hline Inter & What documents do I need to have if my ID has expired? \\  & Jake dokulatory potkebuj, kdyz im **vypk** prókaz totoznosti? \\ Final & What documents do I need to have if my ID is out of date? \\  & Jake dokumenty potkebuj, kdyz je mij póhraz **zastral**/_goû obstetric_? \\ \hline \hline \end{tabular}
\end{table}
Table 8: Examples of user interaction with the Ptakopét system. In the top two, the rephrasing of the intermediate input resulted in an improved final translation, in the bottom one the final translation worsened.

 



## Ethics

All participants were recruited online and had to complete an informed consent form using a secure Qualtrics survey before they could progress to taking part in the experiment. Data was anonymized, recorded and stored in accordance with ACM protocol. Ethical clearance was confirmed by the School of Informatics Ethics Committee at the University of Edinburgh (Reference RT 4058). Participants were offered a f20 Amazon voucher as compensation for their time upon completion of the experiment.

## References

* Alabau et al. [2016] Vicent Alabau, Michael Carl, Francisco Casacuberta, Mercedes Garcia Martinez, Jesus Gonzalez-Rubio, Bartolome Mesa-Lao, Daniel Ortiz-Martinez, Moritz Schaeffer, and German Sanchis-Trilles. 2016. _Learning Advanced Post-editing_, New Frontiers in Translation Studies, pages 95-110. Springer, Cham, Switzerland.
* Blatz et al. [2004] John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence Estimation for Machine Translation. In _COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics_, pages 315-321, Geneva, Switzerland. COLING.
* Bojar et al. [2016] Ondrej Bojar, Ondrej Dusek, Tom Kocmi, Jindrich Libovicky, Michal Novak, Martin Popel, Roman Sudarikov, and Duan Varisi. 2016. CzEng 1.6: Enlarged Czech-English Parallel Corpus with Processing Tools Dockered. In _Text, Speech, and Dialogue: 19th International Conference, TSD 2016_, number 9924 in Lecture Notes in Computer Science, pages 231-238, Cham, Switzerland. Springer International Publishing.
* Bojar et al. [2018] Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 Conference on Machine Translation (WMT'18). In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 272-303, Belgium, Brussels. Association for Computational Linguistics.
* Dyer et al. [2013] Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
* Federmann et al. [2019] Christian Federmann, Oussama Elachqar, and Chris Quirk. 2019. Multilingual Whispers: Generating Paraphrases with Translation. In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_, pages 17-26, Hong Kong, China. Association for Computational Linguistics.
* Fornicheva et al. [2020] Marina Fornicheva, Shuo Sun, Lisa Yankovskaya, Frederic Blain, Francisco Guzman, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised Quality Estimation for Neural Machine Translation. _Transactions of the Association for Computational Linguistics_, 8:539-555.
* Fonseca et al. [2019] Erick Fonseca, Lisa Yankovskaya, Andre F. T. Martins, Mark Fishel, and Christian Federmann. 2019. Findings of the WMT 2019 Shared Tasks on Quality Estimation. In _Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)_, pages 1-10, Florence, Italy. Association for Computational Linguistics.
* Germann et al. [2020] Ulrich Germann, Roman Grundkiewicz, Martin Popel, Radima Dobreva, Nikolay Bogoychev, and Kenneth Heafield. 2020. Speed-optimized, Compact Student Models that Distill Knowledge from a Larger Teacher Model: the UEDIN-CUNI Submission to the WMT 2020 News Translation Task. In _Proceedings of the Fifth Conference on Machine Translation_, pages 191-196, Online. Association for Computational Linguistics.
* Hassan et al. [2018] Hansyan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunut, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, ShuangZhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving Human Parity on Automatic Chinese to English News Translation. _arXiv preprint arXiv:1803.05567_.
* Kim and Rush [2016] Yoon Kim and Alexander M. Rush. 2016. Sequence-Level Knowledge Distillation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
* Kocmi et al. [2020] Tom Kocmi, Martin Popel, and Ondrej Bojar. 2020. Announcing CxEng 2.0 Parallel Corpus with over 2 Gigawords. _arXiv preprint arXiv:2007.03006_.
* Machacek et al. [2020] Dominik Machacek, Jonas Kratochvil, Sangeet Sagar, Matus Zlinec, Ondrej Bojar, Thai-Son Nguyen, Felix Schneider, Philip Williams, and Yuehu Yao. 2020. ELITR non-native speech translation at IWSLT 2020. In _Proceedings of the 17th International Conference on Spoken Language Translation_, pages 200-208, Online. Association for Computational Linguistics.
* Martinez-Gomez et al. [2012] Pascual Martinez-Gomez, German Sanchis-Trilles, and Francisco Cascauberta. 2012. Online adaptation strategies for statistical machine translation in post-editing scenarios. _Pattern Recognition_,

45(9):3193 - 3203. Best Papers of Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA'2011).
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In _Proceedings of 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Popel et al. (2019) Martin Popel, Dominik Machacek, Michal Auersperger, Ondrej Bojar, and Pavel Pecina. 2019. English-Czech Systems in WMT19: Document-Level Transformer. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 342-348, Florence, Italy. Association for Computational Linguistics.
* Popel et al. (2020) Martin Popel, Marketa Tomkova, Jakub Tomek, Lukasz Kaiser, Jakob Uszkoreit, Ondrej Bojar, and Zdenek Zabokrtsky. 2020. Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals. _Nature communications_, 11(1):1-15.
* Sanchis-Trilles et al. (2014) German Sanchis-Trilles, Vicent Alabau, Christian Buck, Michael Carl, Francisco Casacuberta, Mercedes Garcia-Martinez, Ulrich Germann, Jesus Gonzalez-Rubio, Robin L Hill, Philipp Koehn, LuisA Leiva, Bartolome Mesa-Lao, Daniel Ortiz-Martinez, Herve Saint-Amand, Chara Tsoukala, and Enrique Vidal. 2014. Interactive translation prediction versus conventional post-editing in practice: a study with the CasMaCat workbench. _Machine Translation_, 28(3-4):217-235.
* Santy et al. (2019) Sebastian Santy, Sandipan Dandapat, Monojit Choudhury, and Kalika Bali. 2019. INMT: Interactive Neural Machine Translation Prediction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations_, pages 103-108, Hong Kong, China. Association for Computational Linguistics.
* A translation quality estimation framework. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 79-84, Sofia, Bulgaria. Association for Computational Linguistics.
* Tiedemann (2012) Jorg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)_, pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
* Underwood et al. (2014) Nancy Underwood, Bartolome Mesa-Lao, Meredecs Garcia Martinez, Michael Carl, Vicent Alabau, Jesus Gonzalez-Rubio, Luis A. Leiva, German Sanchis-Trilles, Daniel Ortiz-Martinez, and Francisco Cascoucheta. 2014. Evaluating the Effects of Interactivity in a Post-Editing Workbench. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)_, pages 553-559, Reykjavic, Iceland. European Language Resources Association (ELRA).
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_, NIPS'17, pages 5998-6008, Red Hook, NY, USA. Curran Associates Inc.
* Zouhar and Novak (2020) Vilem Zouhar and Michal Novak. 2020. Extending Ptakopet for Machine Translation User Interaction Experiments. _The Prague Bulletin of Mathematical Linguistics_, 115:129-142.
* Zouhar (2020) Vilem Zouhar. 2020. Enabling Outbound Machine Translation. Bachelor thesis, Charles University, Faculty of Mathematics and Physics.
* Zouhar and Bojar (2020) Vilem Zouhar and Ondrej Bojar. 2020. Outbound Translation User Interface Ptakopet: A Pilot Study. In _Proceedings of The 12th Language Resources and Evaluation Conference_, pages 6969-6977, Marseille, France. European Language Resources Association.