IMTLab: An Open-Source Platform for Building, Evaluating, and Diagnosing Interactive Machine Translation Systems
Authors: Xu Huang
Zhirui Zhang
Ruize Gao
Yichao Du
Lemao Liu
Guoping Huang
Shuming Shi
Jiajun Chen
Shujian Huang
Venue: EMNLP 2023 Main (2023)
Keywords: Interactive Machine Translation; Neural Machine Translation; Lexical-constrained Translation
TL;DR: We present IMTLab, an open-source end-to-end interactive machine translation system platform and evaluate popular IMT systems.

Abstract: We present IMTLab, an open-source end-to-end interactive machine translation (IMT) system platform that enables researchers to quickly build IMT systems with state-of-the-art models, perform an end-to-end evaluation, and diagnose the weakness of systems.
IMTLab treats the whole interactive translation process as a task-oriented dialogue with a human-in-the-loop setting, in which human interventions can be explicitly incorporated to produce high-quality, error-free translations.
To this end, a general communication interface is designed to support the flexible IMT architectures and user policies.
Based on the proposed design, we construct a simulated and real interactive environment to achieve end-to-end evaluation and leverage the framework to systematically evaluate previous IMT systems.
Our simulated and manual experiments show that the prefix-constrained decoding approach still gains the lowest editing cost in the end-to-end evaluation, while BiTIIMT achieves comparable editing cost with a better interactive experience.

IMTLAB: An Open-Source Platform for Building, Evaluating, and Diagnosing Interactive Machine Translation Systems
Xu Huang Zhirui Zhang Ruize Gao
Yichao Du
Lemao Liu
Gouping Huang² Shuming Shi Jiajun Chen' Shujian Huang
National Key Laboratory for Novel Software Technology, Nanjing University
Tennent Al Lab $Shanghai Jiao Tong University
*University of Science and Technology of China
'xuhuang@smail. edu. cn, {chenjj, huangsj edu. cn 2 zrustc1 @eggail. com, (redmondliu, donkeyhuang, shumingshi)etemeeen duyichao@mail.. wstc. edu. cn
com
ruizgaonlp@@mail com
Abstract
machines, with multiple interactive steps to obtain a satisfactory translation.
We present IMTLAB, an open-souree source end-to- end interactive machine translation (IMT) sys- tem platform that enables researchers to quickly build IMT systems with statc-of-the-as models, perform an end-to-end evaluation, and diagnose the weakness of systems. IMTLAB treats the whole interactive translation process task- oriented dialogue with a human in- in-the-loop set- ting, which human interventions can ex- plicitly incorporated to produce high-qualing error-free translations. To this end, general communication interface designed support the flexible IMT architectures and user policies. Based on the proposed design, we construct a simulated and real interactive environment to achieve end-to-end evaluation and leverage the framework to systematically evaluate previ- IMT systems. Our simulated and manual experiments show that the prefix-constrained decoding approach still gains the lowest editing cost in the end-to-end evaluation, while BiTI- IMT (Xiao et al., 2022) achieves comparable editing cost with better interactive experience.
Traditional IMT systems use a left-to-right com-
pletion paradigm (Barrachina et al., 2009; Knowles and Koehn, 2016; Wuebker et al., 2016) where human translators revise words in the translation prefix. This paradigm can be easily implemented using prefix-constrained decoding strategy. How- ever, this strict left-to-right manner limits its flex- ibility, as some translators may prefer revise words in different order. Recently, an alternative IMT paradigm has been proposed, allowing human translators to revise words at arbitrary positions in the translation. The essential technique for this paradigm is lexical-constrained translation (Cheng et al., 2016; Hokamp and Liu, 2017; Post and Vilar, 2018; Chen et al., 2020; Xiao et al., 2022), which leverages modified words constraints to generate satisfactory translation.
Although various IMT techniques have been pro- posed, there is currently a lack of unified platform to fairly and systematically evaluate these methods, especially in end-to-end manner. One of the difficulties is that previous IMT methods differ in their interactive policies, evaluation environments and metrics, making it challenging to fairly com- pare their performance and efficiency. Moreover, the evaluation methods previously employed, such as randomly deleting words or phrases and then comparing BLEU scores after one or several inter- actions (Weng al., 2019; Chen et al., 2020; Xiao et al., 2022), are far from the real-world interac- tive experience. The final hypothesis may be not the golden translation and post-editing still indis- pensable. Instead, the end-to-end paradigm, which evaluates the performance of IMT systems when the human translator finishes editing the translation, is more accurate measure of the total cost of the whole iterative collaboration process.
1 Introduction
recent years, there has been significant develop- ment in neural machine translation (NMT) (Bah- danau et al., 2015; Vaswani et al.. 2017; Hassan et al., 2018). However, the quality of machine- translated texts still cannot meet the rigorous de- mands of industrial applications, necessitating costly and inefficient human intervention. Inter- active machine translation (IMT) (Foster et al., 1997; Langlais et al., 2000; Barrachina al., 2009; Cheng et al., 2016; Peris et al., 2017; Chen et al., 2021) is a promising solution that can guarantee high-quality, error-free translations. It involves an iterative collaboration process between humans and
WWork was done during internship at Tencent Lab. *Corresponding author
In this paper, we introduce an open-source end-

---

to-end IMT system platform, namely IMTLAB, to
and construct simulated real interactive envi- ronment achieve an end-to-end evaluation.
fill this gap. This platform enables both academia and industry to quickly build IMT systems us- ing state-of-the-art models, perform end-to-end evaluations, and diagnose weaknesses in IMT sys-
We conduct simulated and manual experiments to systematically compare several popular IMT sys- tems. The user interaction data collected during these experiments will be released to the commu- nity for future research.
tems. IMTLAB treats the entire interactive trans- lation process as task-oriented dialogue, which includes human-in-the-lowe setting and allows IMT systems to leverage explicit human interven- tions to produce high-quality, error-free transla- tions. Specifically, the user's goal to obtain cor- rect translation, while the IMT system constantly generates translation based on the user's behaviors. During this process, the user iteratively performs editing operations to request the response of IMT systems until the pre-defined goal is achieved or the user loses patience with this interactive pro- cess. To support the flexible architectures of vari- ous IMT systems, we propose a general communi- cation interface between IMT systems and users, where users are limited to five common types of editing operations: keep, insert, replace, delete and blank-filling. Then IMTLAB leverages the revised translation and the corresponding character-leve editing operations from users to form lexical con- straints for IMT systems. Moreover, we build simulated or real interactive environment for IMT systems and introduce new evaluation metries to better verify the effectiveness of IMT systems in an end-to-end manner.
2 Related Work
IMT has been developed to assist professional translators since the era of statistical machine trans- lation (SMT) (Langlais et al., 2000; Foster et al., 1997; Barrachina et al., 2009; Cheng et al., 2016; Simianer et al., 2016). Recently, with the devel- opment of NMT (Bahdanau et al., 2015; Vaswani et al., 2017; Hassan et al., 2018), the field of IMT has undergone a major shift towards powerful deep neural models (Hokamp and Liu, 2017; Post and Vilar, 2018).
Early IMT systems typically adopt left-to-right sentence completing framework (Barrachina et al.. 2009; Knowles and Koehn, 2016; Wuebker et al., 2016; Peris et al., 2017), where users process the translation from the beginning and revise the left- most error. However, this left-to-right approach is inflexible and inefficient for modifying critical translation errors near the end of the sentence. As a result, many researchers have explored alternative IMT paradigms that allow human translators to revise critical translation errors at any position.
We conduct simulated and manual experiments to systematically evaluate several popular IMT sys- tems with this platform. Experimental results indi- decoding approach
One such paradigm is the pick-revise framework proposed by Cheng al. (2016) to improve effi- ciency. With respect to neural translation models, Exxical-consrriigg decoding (LCD) methods are employed to satisfy arbitrary constraints, such as Grid Beam Search (GBS) (Hokamp and Liu, 2017) and Dynamic Beam Allocation (DBA) (Post and Vilar, 2018). Then Weng et al. (2019) design bidi- rectional IMT framework on top of LCD method. While LCD methods suffer from slow inference speed, some methods treat the lexical constraints as part of the input and let vanilla NMT directly learn the decoding strategy. For example, LeCA (Chen et al., 2020) and BiTHIMT (Xiao et al., 2022) are two such methods that employ soft and hard lexi- cal constraints, respectively, to improve translation quality and inference speed. Despite the numer- ous IMT techniques, there is currently no unified platform to evaluate these methods in fair and systematic end-to-end manner. To fill this gap, we develop the first open-source platform for building,
cate that the still obtains the lowest editing cost in the end-to- end evaluation, while BiTIIMT (Xiao et al., 2022) achieves comparable editing cost with a better in- teractive experience, i.c., better success rate, lower average turns and response time. IMTLAB is also compatible with large language models, such as ChatGPT. Additional experiments show that Chat- GPT yields promising performance of editing cost in end-to-end evaluation, but is not very ro- bust flexible lexical constraints. summary, our contributions are as follows:
To the best of our knowledge, we develop the first open-source end-to-end IMT system platform.'
We design a general communication interface to support the flexible architectures of IMT systems,
'Codes are available at ht hptps://e //gi httgg/tttttteeee// htt..coooooo/a.aa..a/ooooooo xuuHuang/ IMTLab.

---

My goal: The biggest advantage for VR works to enable the audience to have a unique movie watching "immersion"
et al., 2018: Zhang al., 2019; Wang et al., 2021), since IMT involves an iterative collaboration pro- cess between humans and machines, with multiple interactive steps to obtain satisfactory translation. As illustrated in Figure 1, the user's objective this dialogue is to obtain a reference translation for the source sentence input. The IMT system is designed to produce translations based on the user's feedback, and the user provides editing op- erations (such as keep, insert, replace, delete, and blank-filling) request the system's 's response. This interactive process continues until the user's objec- tive is met or the user becomes frustrated with the system. In this way, IMTLAB allows the IMT system to leverage explicit human interventions to refine translations and enables researchers to evalu- ate different IMT systems in an end-to-ent manner.
VRY Sarrs
ESALES
The greatest advantage of VR works is that the audience hus a unique sense of "immersion" in the film.
The biggest advaniage for VR works is the pudlience haave a unique "immersion" -in film
The biggest advantage for VR works that the audiencc will have a unique sense of "immersion".
The biggest advantage for VR works is to the audieence to unique movie "immersion".
The biggest advantage for VR works to enable the audience to have a unique movie watching "immersion'
Keep
Insert
Replace
Betete
Blank:
Figure I: Overview architecture of IMTLAB, involving the iterative collaboration process between user and IMT system. During this process, the user provides editing operations (i.c., keep, insert, replace, delete and blank-filling) interactively to query IMT systems until the user's goal is achieved or the user loses patience with this interactive process.
3.2 Communication Interface
In order to support the flexible architectures of var- ious IMT systems, we design a general communi- cation interface between IMT systems and users. Users are limited to five common types of editing operations: (i) keep words unchanged; (ii) insert a
evaluating and diagnosing IMT systems. Another research line related to this topic is online learning from translation memory or hu- man feedback, which is particularly effective when translating documents in an unseen domain. This approach typically enhances the performance of NMT systems by fine-tuning through human- corrected translations (Turchi et al., 2017; Kothur et al., 2018; Peris and Casacuberta, 2019), or by utilizing TM-NMT frameworks (Bapna and Firat, 2019; Xia et al., 2019; Hao al., 2023) or KNN- MT methods (Khandelwal et al., 2021; Zheng et al., 2021a,b; Meng et al., 2022; Wang et al., 2022; Dai et al., 2023; Du et al., 2023) to incorporate human- corrected translation pairs. In this paper, we focus on the interactive process within single sentence, rather than the knowledge transfer across sentences. We leave the incorporation of online learning in platform for achieving stronger IMT system future direction.
continuous word span any position; (iii) replace
continuous word span with one; (iv) delete
continuous word span at any position; (v) replace continuous word span with a special placeholder (or just insert) to prompt the IMT system for blank- filling. These five common types could be com- bined to cover interactive policies in most IMT paradigms, including left-to-right completion, pick- revise framework, and arbitrary lexical constraints. To preserve user editing information, maintain the record of editing operations at the character level, which is language-frient feedback can be decomposed into two parts, the revised translation and the corresponding opera- tion tags, providing the IMT system with as much information as possible about the user's edits.
Therefore, user
Figure 2 illustrates an example of how IMT- LAB records editing operations from different users. Specifically, when the IMT system produces an unsatisfactory translation that requires human in- tervention, users can perform the aforementioned editing operations according to their goals. It's worth noting that the editing processes of different people may vary, such as Human and 2. The re- vised translation and corresponding operation tags from user feedback are then used to query the IMT system for the desired translation. These different
3 IMTLAB
This section details the design of IMTLAB and its flexibility to support wide range of experiments.
3.1 Overall Design
this work, we consider the entire interactive translation process as a task-oriented dialogue with a numan-in-ther loop setting (Li et al., 2017; Liu

---

Source: V SAAAIEATTEETETAEEEAAAAAAAAAEEEEE
User Goal:
The biggest advantage for VR works is to enable the audience to have unique movie watching "immersion".
IMT System: The greatest advantage of VR works is that the audience has a unique sense of "immersion" in the film.
Human (prefix constraint)
Text: The biggest advantage for VR works is to that the audience has a Operations: kkkk --r-- k---- |rrr| -------------- ---------------
a unique sense of "immersion" in the film.
Human 2 (complex constraints)
Text: The biggest advantage for VR works is that the audience hasve bbbkkkdij
Operations: kkkk
|rr|| 1-----1a0 |bbbb]
unique sense of Limmersion"- in the film. k
-1------------------------------
Keep(k) Insert(i) Replace(r) Delete(d) Blank(b)
Figure 2: An example of the communication interface, where perform editing operations (i.e., keep, insert, replace, delete and blank-filling) on the output of the IMT system according their own goals. We show two different editing processes from users: one of which contains a prefix constraint, while the other one contains complex constraints. The editing operations at the character level and the kept, inserted and replaced characters actually are lexical constraints. is a special placeholder for the blank-filling operation.
editing operations bring different constraints, e.g.. constraints prefixes or arbitrary positions.
kept, inserted and replaced characters are lexical constraints for IMT systems.
Note that our current version only supports the kcyboard-related operations mentioned above, but is capable of evaluating most common IMT sys- tems. We will leave the adaptation to other special operations like mouse clicks (Sanchis-Trilles al., 2008; Navarro and Casacuberta, 2022) or screen touches for future work.
For the implementation details, when users per- form keep, insert and replace operations, it is nat- ural to directly label the revised translation with corresponding operations tags, i.e., "k", "i" and "Y" tags. For the deletion operation, we preserve deleted text in revised translations and label them with "d" tags to explicitly display the deletion pro- cess, which may provide more editorial informa- tion to the IMT system. The blank-filling and dele- tion operations are very similar operations, with the difference that the former means the words are redundant, while the latter means words needed replace original ones. Therefore, we use a similar way record the editing process of the blank-filling operation, which labels deleted text in revised translations with "b" tags. When the user only wants to insert new words without dele- tion, we append the special placeholder **p in the revised translation and label it with "b" tags. Addi- tionally, when users perform the combined opera- tions of delete+insert insert+delete on a contin- uous character span, treat these combined oper- ations as replace operations in practice. Finally, all editing operations from users could be converted into a lexical-constraained template t, where the
3.3 IMT System
During the interactive process, the IMT system must constantly generate translations based on the user's editing operations, which requires the sys- tem to support both normal and seiccl--onsttriied translations. Specifically, given source sentence the user first obtains initial translation ())) by querying the IMT system without any constraints:
(0) IMT(x) arg max P(y[x). (I)
If this translation does not meet the user's require- ments, the following interaction process begins. At the i-th turn, users modify the translation (1) re- turned by the IMT system according to the user's goal yo. and corresponding editing operations are converted into a coniicai-onstrineed template (i)

---

obtain the next translation g(++1);
translation, randomly selects direction (left right), and corrects the first incorrect word in that direction using a replace operation. Then blank- filling operations are used to remove the prefix and the suffix of this continuous word span.
t( = Policy($) (y(), ,y)).
10811) = IMT(X, t (1))
(2)
arg max P(y) x, )()),
L.21: Recently, some IMT systems (Peris et al., 2017; Xiao al.. 2022) employ text-infilling policy that retains all correct words in a sentence and utilizes blank-filling operations to complete the remaining parts, so we develop L.2rl policy to simulate this scenario. In this policy, the simu- lated user replaces all incorrect word spans with blank-filling operations and appends one correct word using insert operation at the position of the leftmost incorrect word each turn.
where constraints in (() should appear in the trans- lation (((11), This interactive process continues until we obtain yo- If the user loses patience with this interactive process, they could directly apply a post-editing strategy at any time to completely correct the translation.
3.4 Simulated Environment
We construct simulated interactive environment where the IMT system interacts with user simula- tors instead of real users. Although there may be differences between simulated and real users, this setting enables researchers to quickly conduct a detailed analysis of IMT systems without any real- world cost. During the interactive translation pro- cess, the reference serves as an oracle to determine whether the words phrases in the translation are correct, and the simulator provides simulated user response each turn according to their pre-defined interactive policy. The translation is only accepted when matches the reference. In IMTLAB, we im- plement five common interactive policies for sim- ulated users, including machine translation post- editing (MTPE), left-to-right sentence completion (L2r), random sentence completion (Rand), left-to- right infilling (L2rI) and random infilling (RandI):
This strategy behaves almost the same as L2rl, but it appends one correct word using an insert operation at position near the preserved words. The position is randomly selected.
RandI:
To model situations where users become impatient with the interactive process, we assume that the simulated user will lose patience if the constraints in L2r not met. As Rand, L2rI, and RandI offer greater flexibility in selecting editing operations. we assume that the simulated user is more likely to tolerate constraint violations up to three times. If a constraint violation does occur, we will restart the interactive policies based on the current transla- tion. In addition, the maximum interaction rounds threshold is determined by the number of editing sequences provided by MTPE to simulate situations where the interaction process is too inefficient for users to continue. If the number of interaction rounds exceeds this threshold, we assume that the simulated user will lose patience. MTPE is leveraged to correct the current translation once the simulated user loses patience with the interactive process.
The simulated user corrects the initial translation directly in one turn to match the ref- erence using the optimal editing sequence based on the Levenshtein distance (Levenshtein, 1965).
MTPE:
L2r: The simulated user first identifies the left- most incorrect word and corrects it using re- place operation. Then blank-filling operation is employed to remove the suffix. This inter- active policy is widely used in traditional IMT systems (Barrachina et al., 2009).
3.5 Evaluation Metrics
The evaluation methods used in previous IMT meth- ods, such as randomly deleting words or phrases and comparing BLEU scores after interactions, do not accurately reflect real-world interactive experiences. To address this issue, IMT- LAB introduces several end-to-end evaluation met- rics for IMT systems that provide more accurate measures of the total cost of the entire iterative collaboration process.
several
Rand: As advanced IMT methods (Hokamp and Liu, 2017; Post and Vilar, 2018) now allow users to modify errors at any position, we extend I.2r policy to simulate this scenario. The simulated user randomly selects one correct word from the initial translation as the start point and uses blank- filling operations to remove the prefix and the suf- fix it. In subsequent iterations, the user identi- fies the position of the last constraint in the new
Editing Cost (EC). For the interactive experi- ence, the editing cost is the most crucial metric,

---

rather than BLEU scores, as the user is expected to achieve their desired translation through the iterative collaboration process. To this end, we define the cost of each editing operation based on the actual number of required keystrokes:
preprocess the data and learn joint vocabulary of 60k. To evaluate the IMT systems in both simulated and manual experiments, we randomly sample 500 or 100 sentence pairs from each original test set, respectively, for the end-to-end evaluation.
keep: 0
Models. We implement four popular IMT sys- tems based on fairseq toolkit (Ott et al., 2019).
insert: #chars inserted
delete: 1
Prefix: vanilla Transformer model. This model only supports L2r, in which prefix-constraieed decoding is used to produce translations.
replace: #chars inserted +
blank-filling:
During the interactive translation process, we calculate the total editing cost by summing the
DBA (Post and Vilar, 2018): a vanilla Trans- former model same as the one above. During decoding, adopts the DBA method to satisfy the lexical constraints.
cumulative cost of the user's editing operations at each turn. While we acknowledge that the cost of using mouse or touchpad could be considered, we have chosen to ignore it in the current version for the sake of simplicity.
BiTHIMT (Xiao et al., 2022): a Transformer model that learns with the bilingual text-infilling task and can fill missing segments in revised translation.
Success Rate (SR). We track the rate of user sat- isfaction with the interactive process to implicitly reflect the interaction experience.
LeCA (Chen et al., 2020): lexical constraint- aware Transformer model that simply packs con- straints and source sentence together with sep- arating symbol to generate the final translation. We use the same augmented data as BiTIIMT. The pointer network is also used in this model.
Consistency (Con.). To measure the user's cog- nitive cost of new words phrases, we calculate the average edit distance between two adjacent outputs from IMT systems.
Average Turns (AT). We count the average turns of the entire interactive translation process for each source sentence.
All models use the Transformer-big architecture and share all embeddings. The learning rate is 7 and the warmup step is 4000 for all models. The maximum update is set to 300k for vanilla Trans- formers and 400k for models using data augmen- tation. The batch size is set to 32k for the En tasks and 64k for the Zh En tasks. For inference, we average 10 best checkpoints of each model and the beam size is 5. We run each model on single NVIDIA Tesla V100 GPU with a batch size of 1.
Response Time (RT). The average response time of IMT systems in the entire interactive process.
Among these metries, the editing cost reflects the workload of users, while the other metries reflect the user's experience with the interactive process.
4
Experiments
4.1 Setup
4.2 Simulation Evaluation
Data. We conduct simulated and manual exper- iments on Anglish-Germin (En-De) and Chinese- English (Zh-En) language pairs in both directions. For En we use the WMT14 En-De dataset consisting of 4.5 million parallel sentences as the training set, newstest2013 as the development set, and newstest2014 4 as the test set. We preprocess the data similar to the script in fairseq except that we sentencepiece (Kudo and Richardson, 2018) tokenize the sentences and learn joint vocabulary of 40k. For Zh< EEn, the training data is from the WMT17 Zh- Eh dataset containing 20M parallel sentences. We also use sentencepiece to
We evaluate the end-to-end performance of IMT models using different interactive policies, includ- ing MTPE, L2r, Rand, L2rI and RandI In this sim- ulation evaluation, we first record the editing cost of each model, where we average the editing cost of 500 source sentences. As shown Table 1, MTPE serves as the baseline without interaction and all methods obtain similar editing cost using MTPE, meaning that the translation performance of them
We run three experiments with different seeds for Rand and RandI interactive policies. The variance of editing cost in the random experiments can he found in Appendix A.

---

En-De
De-En
Zh-En
En-Zh
Policy
Prefix
DBA
BITMMT
LeCA
Prefix
DBA
BiTUMT LeCA
Prefix
DBA
BITIMT
LeCA
Prefix
DBA
BiTIMT LeCA
NTPE
86.41
86.41
85.62
85.39
73.41
73.41
73.51
70.71
104.82
104.82
105.20
105.64
36.71
36.71
36.56
37.34
L2r
62.78
65.76
66.67
65.50
57.66
59.21
60.45
58.62
80.42
84.28
81.96
86.41
29.60
36.80
31.49
32.51
Rand
104.49
74.27
86.41
96.54
68.52
77.00
123.65
93.42
126.01
53.30
40.09
64.17
L2rl
70,54
67.80
68.77
65.17
62.45
63.27
/
87.92
84,79
90.92
43.29
42.42
47.87
RandI
80.18
68.08
70.52
72.15
63.00
64.23
98.80
82.87
88.84
50.28
43.05
47.65
Avg.
62.78
80.24
69.21
72.80
57.66
73.27
63.61
65.78
80.42
98.66
85.76
98.05
29.60 45.92
39.26
48.05
Table 1: The editing cost (L) of each model with different interactive policies in simulation evaluation. The lowest cost in each column is in bold and "Avg." shows the average of the above four editing costs.
SR (1)
Con, (L)
AI (J)
RT (L)
Prefix
DBA
B.THMT
LeCA
Prefix
DBA
BiTHIMT LcCA
Prefix
DBA
BITIIMT
LeCA
Prefix
DBA
BiTIIMT LeCA
En-De
98.2%
88. 88.1%
98.6%
90.2%
3,94
4.72
4.00
4.00
7.8
8.1
7.5
7.3
307
673
167
282
De-En
96.8%
87.7%
97.2%
89.6%
3,64
3.54
8.2
7.9
7.6
158
Zh-En
97.6%
87.1%
98.2%
80.2%
5.37
6.05
4.97
5.18
11.3
11.3
10.8
10.7
398
791
200
354
En-Zh
97.0%
85.5%
96.3%
55.2%
4.41
6.22
4.32
5.41
11.2
11.2
11.0
10.7
325
759
170
310
Avg.
97.44 87.1%
97.6%
78.8%
4.31
5.36
4.23
4.53
9.62
9.74
9.28
9.05
331
719
174
Table The success rate, consistency, average turns and respouse time (ms) of each method simulation evaluation. where we average the results of each model different interactive policies. "Avg." shows the average of the above four translation directions. The detailed results are shown in Appendix A.
Model
Metric
En-De
De-En
Zh-En
En-Zh
Spearman's
BLEU
54.55
54.31
46.37
49.55
p=0.00
BITIMT
CSR
100%
100%
100%
100%
Human editing cost
BLEU
55.32
54.96
45 88
48.95
LeCA
CSR
99.55%
99.30%
98.51%
98.21%
40-
Table BLEU and Copy Success Rate (CSR) of BiTI-
IMT and LeCA on the test sets with sampled constraints.
30-
is very close due the same training dataset. Cur- rent IMT methods achieve significant improvement over MTPE in most interactive policies, indicating the benefits of introducing an interactive process. When considering the same model with different interactive policies, L2r outperforms in most cases, showing that this conventional approach is still competitive. Surprisingly, Prefix achieves the best editing cost compared other models in all trans- lation directions. BiTHIMT achieves comparable performance with Prefix when using L2r and is more robust to all interactive policies.
Simulation editing cost
Figure The correlation between the editing costs of simulated and real users.
We also note that lexical constraint-awarc meth- ods, such as LeCA, have the lowest success rate in end-to-end evaluation. As listed in Table 3, we randomly select some words or phrases as con- straints to conduct experiments, following previous work (Chen et al., 2020), and find that the perfor- mance gap between BiTIIMT and LeCA is very small. These findings highlight the importance of end-to-end evaluation, which amplifies the error in each turn and closely resembles real-world interac- tive experience.
In addition to the editing cost, we also analyze the success rate, consistency, average turns and re- sponse time of these models, where we average the results of each model on L2r, Rand, L2rI and RandI. From Table 2, we observe that BiTHIMT obtains lower average turns than Prefix and has the best success rate, consistency and response time. These results demonstrate that BiTTIMT provides a better interactive experience than other methods while achieving comparable editing cost to Prefix.
4.3 Human Evaluation
For human evaluation, real users interact with dif- ferent IMT models to finish the translation task and we build the website for this purpose. To obtain

---

EC (U)
SR
AT) (
RI DBA
Prefix
DBA
BITIMTT
LeCA
Prefix
DBA
BITHIMT
LeCA
Prefix
DBA
BITIIMT
LeCA
Prefix
BITIIMT LeCA
En-De
46.36 49.41
45.19
45.84
95.7%
96.3%
99.8%
96.3%
4.6
3.9
3.7
4.1
264
617
154
247
De-En
38.91
40.07
40.63
37.20
98.7%
98.7%
99.7%
98.3%
3.6
3.2
3.5
3.4
249
559
143
236
Zh-En
54.79
57.53
54,50
59.00
99.0%
95.3%
99.0%
90.7%
4.1
3.4
3,9
3.9
368
717
206
344
En-Zh
21.19
22.26
22.27
25.16
100%
99.0%
99.7%
84.3%
3.7
2.8
3.1
3.0
296
632
176
284
Avg.
40.31 42.48
40.65
41.80
98.3%
97.3%
99.3%
92.4%
4.0
3.3
3.6
3.6
294
631
170
278
Table 4: The editing cost, success rate, average turns and response time (ms) of each method in human evaluation, where we average the results of each model with three annotators. More results are shown in Appendix B.
user feedback on real interactive experience, provide a "MTPE" checkbox that the user could click to express their dissatisfaction with the in- teractive process. Therefore, in this experiment, the success rate is recorded as the unclicking rate of this checkbox, and we ignore the consistency metric, which is also reflected in user satisfaction. Specifically, we randomly sample 100 sentences from each test set and then ask three human trans- lators to interact with IMT systems. Translators use flexible operations (such as keep, insert, re- place, delete, and blank-filling) to interact with IMT models without any requirements. The hu- man evaluation results are listed in Table 4. Similar to the conclusion of the simulation evaluation, we observe that Prefix still obtains the lowest editing cost in the end-to-end human evaluation, while BiTIIMT achieves comparable editing cost to Pre- fix with a better interactive experience, i.e., better success rate, lower average turns and response time. We also calculate the Spearman's 's correlation be- tween the average editing cost over four simulated policies and the average editing cost over three hu- man translators, as shown in Figure 3. This result demonstrates good correlation between simulated and manual experiments IMTLAB. In addition, we observe that the editing cost of real users is sig- nificantly lower than that of simulated users. The real users feedback that they learn to select the best operations by observing the output of IMT systems to improve their editing efficiency, which currently could not be simulated by IMTLAB. We hope the release of real user interaction data will aid in the construction of such simulated user.
EC (1)
SR (1)
En-De
De-Ea Zh-Fn En-Zh En-De De-Fn Zh-En En En-Zh
MTPE
81.58
66.56 100.33 35.39
L2r
74.00
57.38
89.68
33,95
66.00%
71.0% 60.4% 53.0%
Rand
126.55
114.20
158.56
59.34
28.4%
25.2% 25.8% 17.4%
L2r1 Rand]
74,61
65.90
99.28
48.23
78.6 %
80.8%
72.2% 67.8%
93.57
77.89 114.84
53.03
52.8%
62.4%
46.2% 43.6%
Avg.
92.18 78.84 115.59
48.64
55.7%
59.9% 51.2% 45.5%
Table S: The editing cost and success rate of ChatGPT in simulation evaluation.
CharGPT
BITIMTT
EC (L) AT (U) RT (1)
EC (R) (1) (L) RT (
En-De
48.03 90.0%
3.2
3144
45.19 99.8%
3.7
154
De-En
33.36 97.3%
2.9
2627
40.63
99.7%
3.5
143
Zh-En
54.85 91.3%%
3.3
3006
54.50
99.0%
206
En-Zh
21.03 98.0%
2.7
3455
22.27
99.7%
3.1
176
Avg. 39.32 94.2% 3.0
3058
40.65 99.3%
3.6
170
Table 6: The editing cost, success rate, average turns and response time (ms) of ChatGPT and BiTIIMT in human evaluation.
constraints into natural language. We adopt the approach of BiTIIMT to use ChatGPT as the IMT system that fills missing segments in a revised trans- lation, and evaluate the performance using simu- lated and manual settings. More details for Chat- GPT are presented in Appendix C.
Table presents the editing cost and success rate of ChatGPT in simulation evaluation. is evident that L2r is still in a leading position compared to other interactive policies. However, the editing cost of ChatGPT is worse than BiTIIMT on av- erage, since the success rate of ChatGPT is very low, more than 81%. ChatGPT struggles sat- isfy lexical constraints, even though we explicitly require it to strictly follow the template.
4.4 Evaluation for ChatGPT
Large language models (LLMs) such as ChatGPT and GPT-4 have demonstrated remarkable machine translation ability during a chat. IMTLAB is com- patible with these models by converting lexical
We conduct the human evaluation for ChatGPT following the setting of the previous subsection and list all results in Table 6. Surprisingly, the editing cost of ChatGPT is better than BiTIIMT but the
We inform translators that Prefix only supports the left-to- right completion manner.
"We call the gpt-3.5-turbo--00 API.

---

success rate is still unsatisfactory. Additionally, we compare the editing cost of ChatGPT and BiTIIMT using MTPE in human evaluation. ChatGPT and BiTIIMT achieve 66.9 and 72 points, respectively, but this 5-point gap is reduced to 1.3 during the in- teractive process, indicating the unsatisfied success rate of ChatGPT hinders the editing cost.
the incorporation of online learning or translation memory in our platform for achieving a stronger IMT system as future direction.
In the current platform, words that are not modi- fied edited in any way are automatically con- sidered as right words during the interactive pro- cess, However, there an alternative interaction strategy where unmodified words are automati- cally considered incorrect. Actually, we could introduce a default system mode to unify the two interaction strategies, in which this system mode would trigger different automatic post-processing steps after user editing. In this way, the total cost of both the keep and delete operations is set of IMT systems and a simulated or real interactive environment is further constructed for the end-to- end evaluation. Experiments demonstrate that the prefix-constrained decoding approach still achieves the lowest editing cost in the end-to-end evaluation, while BiTIMTT achieves comparable editing cost with a better interactive experience. IMTLAB is also compatible with LLMs, such as ChatGPT.
5 Conclusion
In this paper, we introduce IMTLAB, an open- source platform for building, evaluating and diag- nosing IMT systems. IMTLAB treats the whole interactive translation process as a task-oriented di- alogue. To this end, we design general communi- cation interface to support the flexible architectures
I and then one of these costs would not be calculated when using different system modes, We leave this refined framework as future work, providing the flexibility needed to accommodate more interaction strategies.
A Acknowledgements
We would like to thank the anonymous review- ers for their insightful comments. Shujian Huang is the corresponding author. This work is sup- ported by National Science Foundation of China (No. 62176120, 623761 16), the Liaoning Provin- cial Research Foundation for Basic Research (No. 1022-KF-26-02
6 Limitations
In this section, we discuss the limitations and future research directions of our work:
Although the simulated and manual experiments show a strong correlation, there is still gap be- tween simulated and real users. Real users could learn to select the best operations by observing the output of IMT systems, which can improve their editing efficiency. We hope that the release of real user interaction data will aid in the con- struction of such simulated user in the future.
References
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
To remove the effect of multiple translation refer- ences on the interactive process, human transla- tors are required to strictly follow the same ref- erence, rather than engaging in a more realistic, completely unconstrained manner. In the future, we would like to extend our human evaluation to a more complex setting.
Ankur Bapna and Orhan Firat. 2019. Non-parametric adaptation for neural machine translation. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACI- HLT 2019, Minneapolis, MN, USA. June 2-7, 2019, Volume I (Long and Short Papers), pages 1921-1931. Association for Computational Linguistics.
Our current evaluation is limited to four typical IMT systems and ChatGPT, excluding other IMT systems. In the future, we hope more researchers could construct other IMT systems in IMTLAB.
Sergio Barrachina, Oliver Bender, Francisco Casacu- berta, Jorge Civera, Elsa Cubel, Shahram Khadivi, Antonio I. Lagarda, Hermann Ney, Jesús Tomás, En- rique Vidal, and Juan Miguel Vilar. 2009. Statistical approaches to Linguistics, 3-(1))3-28.
translation. Comput.
In this work, we mainly focus on the interac- tive process within a single sentence, rather than knowledge transfer across sentences. We leave
Guanhua Chen, Yun Chen, and Victor O. K. Li. 2021. Lexically constrained neural machine translation with

---

explicit alignment guidance. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12630-12638.
Rebecca Knowles and Philipp Koehn. 2016. Neural
interactive translation prediction. In 12th Confer- ences of the Association for Machine Translation in the Americas: MT Researchers' Track, AMTA 2016, Austin, TX. USA, October 28 November 1, 2016, pages 107-120. The Association for Machine Trans- lation in the Americas.
Guanhua Chen, Yun Chen, Yong Wang, and Victor O. K. Li. 2020. exxical-conttrait-awaar neural machine translation via data augmentation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intellience, IJCAI 2020.
Sachith Sri Ram Kothur, Rebecca Knowles, and Philipp Koehn. 2018. Document-level adaptation for neu- ral machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Gener- ation, NMT@ ACL 2018, Melbourne, Australia, July 20, 2018, pages 64-73. Association for Computa- tional Linguistics.
Shanbo Cheng, Shujian Huang, Huadong Chen, Xinyu Dai, and Jiajun Chen. 2016. PRIMT: A pick-revise framework for interactive machine translation. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 1240-1249, The Association for Computational Lin- guistics.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 November 4, 2018, pages 66-71. Asso- ciation for Computational Linguistics.
Yuhan Dai, Zhirui Zhang, Qiuzhi Liu, Qu Cui, Weihua Li, Yichao Du, and Tong Xu. 2023. Simple and scalable nearest neighbor machine translation. In The Eleventh International Conference on Learning Representations.
Yichao Du, Zhirui Zhang, Bingzhe Wu, Lemao Liu, Tong Xu, and Enhong Chen. 2023. Federated near- est neighbor machine translation. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Openkeview.net.
Philippe Langlais, George F. Foster, and Guy Lapalme. 2000. Transtype: a computer-aided translation typing system.
Vladimir I. Levenshtein. 1965. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics. Doklady, 10:707-710.
George F. Foster, Pierre Isabelle, and Pierre Plamon- don. 1997. Target-text mediated interactive machine translation. Machine Translation, 12(1-2):175-195))
Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end task- completion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natu- ral Language Processing, IJCNLP 2017, Taipei, Tai- wan, November 27 December 1, 2017- Volume 1: Long Papers, pages 733-743, Asian Federation of Natural Language Processing.
Hongkun Hao, Guoping Huang, Lemao Liu, Zhirui Zhang, Shuming Shi, and Rui Wang. 2023. Rethink- ing translation memory augmented neural machine translation. In Findings of the Association for Com- putational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 2589-2605. Association for Computational Linguistics.
Bing Liu, Gökhan Tür, Dilek Hakkani-Tie Pararth Shah, and Larry P. Heck. 2018. Dialogue learn- ing with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, NAACL- HLT 2018. New Orleans, Louisiana. USA, June 1-6, 2018, Volume I (Long Papers), pages 2060-2069. Association for Computational Linguistics.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Downnl Lewis, Mu Li, et al. 2018. Achieving human par- ity on automatic chinese to english news translation. arXiv preprint arXiv: 0503.05567.
William
Chris Hokamp and Qun Liu. 2017. Lexically con- strained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL 2017, Vancouver, Canada, July August 4, Volume 1: Long Papers, pages 1535-1546, Association for Computational Linguistics.
Yuxian Meng, Xiaoya Li, Xiayu Zheng, Fei Wu, Xi- aofei Sun, Tianwei Zhang, and Jiwei Li. 2022. Fast nearest neighbor machine translation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 555 555-565, Dublin, Ireland, Association for Computational Linguistics.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemayer, and Mike Lewis. 2021. Nearest neigh- bor machine translation. In 9th International Confer- ence on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net.
Ángel Navarro and Francisco Casacuberta. 2022. On the use mouse actions at the character level. Infor- mation, 13(6).

---

Myle Ott, Sergey Edunov, Alexii Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli, 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Weizhi Wang, Zhirui Zhang, Junliang Guo, Yinpei Dai, Boxing Chen, and Weihua Luo. 2021. Task-oriented dialogue system as natural language generation. Pro- ceedings of the 45th International ACM SIGIR Con- Jerence on Research and Development in Information Retrieval.
Álvaro Peris and Francisco Casacuberta. 2019, Online learning for effort reduction in interactive neural ma- chine translation. Comput. Speech Lang., 58:98-126.
Rongxiang Weng, Hao Zhou, Shujian Huang, Lei Li, Yifan Xia, and Jiajun Chen. 2019. Correct-and- memorize: Learning to translate from interactive revisions. In Proceedings of the Twenty- Eightt Inter- national Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019.
Álvaro Peris, Miguel Domingo, and Francisco Casacu- berta. 2017. Interactive neural machine translation. Comput. Speech Lang., 45:201-220.
Joem Wuebker, Spence Green, John DeNero, Sasa Hasan, and Minh-Thang Luong. 2016. Models and inference for prefix-cosstraieed machine translation. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume I: Long Papers. The Association for Computer Linguistics.
Matt Post and David Vilar. 2018. Fast lexically con- strained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Vol- ume I (Long Papers), pages 1314-1324, Association for Computational Linguistics.
Mengzhou Xia, Guoping Huang, Lemao Liu, and Shum- ing Shi. 2019. Graph based translation memory for neural machine translation. In The Thirty-Thise AAAI Conference Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial In- telligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - Feeruary 1, 2019, pages 7297-7304. AAAI Press.
Germán Trilless Daniel Ortiz-Marting Jorge Civera, Francisco Casacuberta, Enrique Vidal, and Hieu Hoang. 2008. Improving interactive machine translation via mouse actions. In Proceedings of the 2008 Conference on Empirical Methods in Natu- ral Language Processing, pages 485-494, Honolulu, Hawaii. Association for Computational Linguistics.
Patrick Simianer, Sariya Karimova, and Stefan Riezler. 2016. A post-editing interface for immediate adapta- tion in statistical machine translation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstru- tions, pages 16-20, Osaka, Japan. The COLING 2016 Organizing Committee.
Yanling Xiao, Lemao Liu, Guoping Huang, Qu Cui, Shujian Huang. Shuming Shi, and Jiajun Chen, 2022. Bitiimt: A bilingual text-infilingg method for inter- active machine translation. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers). ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1958- 1969. Association for Computational Linguisties.
Marco Turchi, Mattco Negri, M. Amin Farajian, and Marcello Federico. 2017. Continuous learning from human post-edits for neural machine translation. Prague Bull. Math. Linguistics, 108:233-244.
Zhirui Zhang, Xiujun Li, Jianfeng Gao, and Enhong Chen. 2019. Budgeted policy learning for task- oriented dialogue systems. In Proceedings of the 57th Conference of the Association for Computa- tional Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3742-3751. Association for Computational Linguis- tics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Miaa Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-59980 -608..
Xin Zheng, Zhirui Zhang, Junliang Guo. Shujian Huang. Boxing Chen, Weihua Luo, and Jiajun Chen. 2021a. Adaptive nearest neighbor machine translation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 368-374, Online. Association for Computational Linguistics.
Dongqi Wang, Haoran Wei, Zhirui Zhang, Shujian Huang, Jun Xie, and Jiajun Chen. 2022. Non- parametric online learning from human feedback for neural machine translation. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Hhirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial In- telligence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pages 11431-11493 1439. AAAI Press.
Xin Zheng, Zhirui Zhang, Shujian Huang, Boxing Chen. Jun Xie, Weihua Luo, and Jiajun Chen. 2021b. Non- parametric unsupervised domain adaptation for neu- machine translation. Findings of the Associa- tion for Computational Linguistics: EMNLP 2021,

---

pages 4234-4241, Punta Cana, Dominican Republic. Association for Computational Linguistics.
'he user can also use defined hot keys to replace span with blank placcholder just insert a blank. The revised translation shown in Figure 4(b), where the remaining texts the area are lexical constraints. After clicking the "Translate" button, a new translation is generated by the backend IMT system, shown in Figure 4(c). To distinguish be- tween newly generated texts and lexical constraints, they have different colors. The user continues this eycle until the translation is satisfactory and clicks the "Submit" button to move on.
Simulation Evaluation Results
We provide more detailed statistics of the simula- tion results in Table 7, including the success rate, consistency, average turns and response time of each method. The variance of editing cost in the random experiments is shown in Table 9.
Human Evaluation Results
We record the evaluation results of three human translators, as shown in Table 8.
Examples
Figure 5 shows two examples of human interactive translation processes. The font color in this figure is the same as the font color of the human interface. In the first example, the user revises the translation a left-to-right manner, while the user in the second example adopts an infilling-style policy.
C ChatGPT
We design some prompts for machine translation with templates so that ChatGPT can complete the translation task with lexical constraints, thus can do interactive machine translation. We compare five different candidate prompts designed by human or advised by ChatGPT and choose the best one for simulated and manual experiments. We test five different prompts on the En-De test using L2r interactive policy. The prompts and results are shown Table 12. The temperature set to 0 and max tokens is 200. For the initial translation, we just use the following prompt:
Translate the following [SRC] text [TG
where [SRC] and [TGT] are the source language and the target language, and [X] is the source sen- tence. For the translation task with lexical con- straints, adopt the following prompt:
Translate the [SRC] sentence by filling in the [TGT] template. Strictly follow the given [TGT] template and generate a whole translation.
[SRC] sentence: [X]
[TGT] template: [T]
[TGT] translation:
where [T] the lexieal-constrraneed template string and deeotee blank. More results of ChatGPT in simulation and human evaluation are shown in Table 10 and Table 11.
D Human interface
Figure 4 demonstrates the GUI of our platform. Figure 4(a) shows an initial translation of the given source sentence. Then the user can edit the transla- tion in the target text area by inserting, deleting, etc. The newly inserted characters by the user are black.

---

En-De
De-En
Zh-En
Ea-Zh
Policy
Prefix
DBA
BiTTIMT
LeCA
Prefix
DBA
BITIMT
LeCA
Prefix
DBA
BITIMMT
LeCA
Prefix
DBA
BITIIMT
LeCA
L2r
98.2%
88.4%
98.2%
83.6%
96.8%
90.0%
95.6%
86.4%
97.6%
82.4%
97.6%
69.0%
97.0%
86.0%
94.8%
44.6%
Rand
63.8%
96.3%
86.9%
60.9%
93.3%
83.1%
/ 65.8%
95.3%
75.7%
58.6%
90.2%
46.3%
SR())
L2r1
100%
100%
93.2%
100%
100%
93.0%
/ 100%
100%
83. 83.2%
99.8%
100%
52.8%
RandI
100%
100%
97.1%
100%
100%
96.0%
/ 100%
100%
92.7%
97.7%
100%
77.0%
L2r
3.94
3.93
4.16
3.84
3.50
3.54
3.61
3.36
5.37
5,36
5,33
5.30
4,41
5.49
4.36
4.46
Rand
5.85
4.14
4.25
5.86
3.71
3.91
/
6.37
5.08
5.87
8.28
4.79
5.85
(on.
L2r1
4.76
3.89
3.92
4.29
3.75
3.33
/
6,78
4.84
4.79
5.78
4.34
5.68
Rand1
4.34
3.79
3.97
4,10
3.50
3.57
7
5.69
4.62
4.75
5.34
4.18
5.65
L2r
7.8
7.2
8.2
7.0
8,2
7.7
8.6
7.6
11.3
9.4
11.8
9.7
11.2
10.0
12.1
8.9
Rand
/
10.6
8.5
9.1
10.9
9.1
9.3
15.0
12.4
14.1
14,5
12.7
15.3
AT(L)
L
6.9
6.7
6.5
7.1
6.9
6.7
9.8
9.6
9.4
9.4
9.6
8.9
RandI
7.8
6.5
7.8
6.9
6.7
11.0
9.3
9.4
10.7
9.6
9.7
L2r
307
811
176
281
293
803
171
271
398
938
212
355
325
1072
177
294
Rand
704
193
281
678
186
275
/
822
233
353
862
211
306
KT())
L2rI
615
150
283
591
138
269
745
177
353
522
142
313
RandI
563
137
276
581
326
Table 7: The success rate, consistency, average turns and response time (ms) of each method in simulation evaluation.
En-De
De-En DBA
Zh-En
En-Zh
Metric
Prefix
DBA
BITHIMT LeCA
Prefix
RiTHIMT LeCA
Prefix
DBA
BiTIMMT
LeCA
Prefix
DBA
BITIMMT LeCA
EC
44.03
45.32
41.99
44.09
36.08
37.9
37.8
35.34
50.39
53.33
50.18
55.52
20.56
22.22
22.86
26.17
Human
SR
96%
95%
98%
95%
99%
97%
99%
97%
99%
95%
99%
88%
100%
99%
99%
80%
AT
4.1
4.3
3,9
4.3
3.6
3.4
3.7
3.9
4.6
3.4
4.1
4.2
3.1
2.9
3.4
3.2
269
592
151
250
252
541
139
239
384
737
192
363
304
638
169
300
EC
49.48
50.21
45.87
46.81
39.92
42.20
41.24
36.77
57.40
58.50
53.30
58. 14
21 14
22.14 14
22.95
26.88
Human
SR
95%
95%
99%
97%
97%
100%
100%
99%
100%
95%
100%
93%
100%
99%
100%
84%
AT
4.2
2.8
3.5
4.7
2.8
2.4
3.4
3.4
3.3
3.0
3.7
3.7
3.0
2.2
2.7
3.3
RT
262
513
139
245
245
462
131
241
366
581
205
342
296
447
169
286
FCC
45.58
52,70
47.70
46.63
40.74
42.00
42.86
39.48
56.58
60.77
60.03
63.34
21.56
22.43
21.00
22.42
Human 3
SR
96%
99%
100%
97%
100%
99%
100%
99%
98%
96%
98%
9100%
100%
99%
100%
89%
AT
5.4
4.5
3.8
3.3
4.0
3.8
3.5
2.9
4.4
3.7
3.9
3.9
5.1
3.4
3.2
2.4
RT
264
617
154
247
249
5559
143
236
368
717
206
344
296
632
176
284
Table 8: The editing cost, success rate, average turns and response time (ms) of each method in human evaluation.
En-De
De-En
Zh-En
En-ZH
Rand Rand]
Rand Rand]
Rand
Rand]
Rand RandI
DBA
0.161 0.592
2,784
0.048
3.702 0.197
10.120 0.024
Metric
En-De
De-En
Zh-En
En-Zh
BITIMMT
0.678 0.403
0.001
0.151
0.217
0.318 0.020 0.015
EC
42.42
31.97
$3.89
21.27
LeCA 0.069 0.041 0.563
0.572
0.105
0.484 0.725 0.252
SR
89%
95%
86%
99%
Human
AT
3.1
3.0
3.7
2.9
Table 9: The variance editing cost in the random experiments.
RT
3548
3766
3426
4071
EC
51.26
35.70
$6.55
22.62
SR
90%
98%
93%
96%
Human 2
AT
3.7
3.0
3.2
2.9
RT
2895
2132
2938
3175
EC
50.40
32.41
$4,10
19.19
Con.
AT
SR
91%
99%
95%
99%
Human 3
En-De De-En Zh-En En-
En-De De-En Zh-En En-Zh
AT 2.9
2.7
2.9
2.2
L2r
4.47
3.57
5.84
4.24
5.6
6.2
8.4
8.2
RT
2989
1984
2654
3120
Rand
5.44
6.99
10.23
5.68
7.9
8.3
10.4
10.4
Table II: The editing cost, success rate, average turns and response time (ms) of ChatGPT in human evalua-
L2rl
4.30
3.69
6.22
4.25
6.1
8.6
8.5
Rand]
3.61
3.21
5.79
3.77
7.6
7.9
tion.
Table 10: The consistency and average turns of Chat- GPT in simulation evaluation.

---

Prompt
Editing Cost
Success Rate
Translate the [SRC] sentence by filling in the [TGT] template. Strictly follow the given
74.00
0.63
[TGT] template and generate whole translation
[SRC] sentence: [X]
[TGT] template: [T]
[TGT] translation:
Striclyy follow the provided [TGT] template and information to generate grammatically correct [TGT] sentence that accurately conveys the sume meaning us the given [SRC] sentence. You must generate complete sentence and deviation from the template should be avoided.
99.88
0.22
[SRC] sentence: [X]
[TGT] template: [T]
[TGT] sentence:
Use the provided [TGT] template and information to generate sentence in [TGT] that conveys the same meaning as the given [SRC] sentence. Ensure that the sentence follows the given lemplate exactly.
95.24
0.42
(SRC] sentence: [X]
(TGT) template: [T]
Complete [TGT] sentence:
[SRC] sentence: [X]
74.32
0.48
[TGT] template: [T]
Create [TGT] sentence using the given template and information that accurately translates the provided [SRC] sentence. You must conform to the template and generate the whole translation.
[TGT] sentence:
[SRC] sentence: [X]
74.33
0.45
(TGT) template: [T]
Your task is to provide a German translation of the given English sentence. You must use the given [TGT] template and information exactly as provided without making any changes, and generate complete translation.
[TGT] translation:
Table 12: Results of different prompts of ChatGPT.
IMTLab
IMTLab
modeer commule
2 AARREEOEE QINER
Rww.uRReeeaaaaenn SAALLEERR RINOR
d AUSEIRR DERERRERSSSSSSSSS SIREIEI R442R
It. 167352206, BEG
1t. 19755-224 001000
1T. 169322381 1E5-89990
by Dieken anch bect Jon Unbemon in 5975 and riceivee aboun 0 miloor
boogg bi by Deniin and vise cam Juice aedill in 1572.
designt Damir arteoot dke USan bulf in 1997, rocewes
Nath BUT Rats
befrc cfee
The initial translation
(b) The revised translation
(c) The translation
Figure 4: Demonstrations of the human interface.

---

Er erklärte, es sei "gut möglich", dass Franziskus die Kardinäle um Rat bitte.
He declared that it was "good possible" for Francis ask the cardinals for advice.
He said it was "quite possible" that Francis would
He said it was "quite possible" that Francis would ask the cardinals for advice.
(a) Example
dit. 5553 AHERE & 1 18
As result, the problems of high pollution, large energy resources, and land occupation have not been fundamentally addressed by the transformation of production and lifestyle.
Therefore, the problem of high pollution, energy and land occupation has not been fundamentally addressed in the transformation of production and life styles.
S
Therefore, the problem of high pollution, large energy source and land occupation has not been fundamentally addressed in the transformation of production and life styles.
Therefore, the problem of high pollution, heavy energy and land accupation has not been fundamentally addressed in the transformation of production and life styles.
Therefore, the problem of high pollution, heavy energy and land occupation has not been fundamentally addressed in the transformation of production and life styles.
Therefore, the problem of high pollution, heavy energy consumption and land occupation has not been fundamentally addressed in the transformation of production and life styles.
(b) Example
Figure 5: Examples of interactive translation processes


Reviews:
[{"review_id": "FYPFsIuDey", "reviewer": "Reviewer_nTMj", "summary": "", "questions": "Question A: Given the current staten of user simulation and given the results from your paper, which advise would you give users of your platform on which simulation policy to use how, in order to predict human performance with an IMT system evaluated by this platform?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide a comprehensive review of our work. We appreciate your thoughtful feedback and constructive comments. Below, we address the concerns and questions you raised.\n\n> Question A: Given the current staten of user simulation and given the results from your paper, which advise would you give users of your platform on which simulation policy to use how, in order to predict human performance with an IMT system evaluated by this platform?\n\nAs mentioned in section 4.3, we have observed a positive correlation between the average editing cost across four simulated policies and the average editing cost across three human translators (see figure 3). Based on this finding, we recommend users to evaluate an IMT system using all supported simulated policies and take the average performance as the prediction of human performance. Furthermore, we encourage users to incorporate more simulated policies into our platform to capture the editing behavior of real translators as comprehensively as possible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper offers a comprehensive platform for fast, automatic evaluation if IMT. The paper presents a comprehensive set of evaluations on different metrics with different systems and compares their results against results obtained with human translators.", "reasons_to_reject": "The simulation of users still has room for improvement (as also commented by the authors) and will probably improve over the future work.", "questions_to_authors": "Question A: Given the current staten of user simulation and given the results from your paper, which advise would you give users of your platform on which simulation policy to use how, in order to predict human performance with an IMT system evaluated by this platform?", "paper_topic_and_main_contributions": "The paper describes an Open-Source Platform for evaluating interactive machine translation systems. For this, the platform offers an API that covers current IMT approaches and allows to simulate user behaviour. With the help of the platform, the papers evaluates several popular user strategies in combination with several different machine translation strategies in IMT, for several language directions. It also evaluates ChatGPT as (constrained) MT system. The paper uses various metrics for evaluation. Several of these are taken from the area of dialog modeling, as the paper treats IMT as a human-machine-dialog problem. Finally the paper validates the simulation on the metric of editing costs by comparing an evaluation with simulated users against an evaluation conducted with human users.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "For Table 1: Maybe readability can be improved, if you would underline in every row the result for the best MT system."}, "extra_text": {}}, {"review_id": "iaHP8aUE8D", "reviewer": "Reviewer_HeBz", "summary": "", "questions": "I am curios as why consider the \"keep\" cost as 0 and the \"delete\" as 1. On the one hand, I don't think that a fixed cost of 1 is fair (even if it can be performed by keeping the \"delete\" key pressed, it is still not comparable with the cost of single-pressing any other key).\n\nOn the other hand, wouldn't it be better to leave the delete cost as 0 (i.e., not performing any operation to a word sequence equal to a deletion) and considering the cost of the word sequences to keep? On average, I believe there are going to be more word sequences to delete than to keep (considering that once you select a word sequence as \"keep\", this could be stored and automatically performed for the subsequent interactions). Of course, which cost to assign to that operation would be critical, but it could be leveraged by adding the use of the mouse (e.g, Peris et al. consider one mouse action for keeping single words and two for any other word sequence).\n\nIn any case, with this ramble I wanted to point out the importance of designing a more-complex, good evaluation scenario that could cover as many IMT protocols as possible.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide a comprehensive review of our work. We appreciate your thoughtful feedback and constructive comments. Below, we address the concerns and questions you raised.\n\n> On the one hand, I don't think the proposed editing cost is equally fair for all the IMT policies.\n\n> I am curios as why consider the \"keep\" cost as 0 and the \"delete\" as 1. On the one hand, I don't think that a fixed cost of 1 is fair (even if it can be performed by keeping the \"delete\" key pressed, it is still not comparable with the cost of single-pressing any other key).\n\n> On the other hand, wouldn't it be better to leave the delete cost as 0 (i.e., not performing any operation to a word sequence equal to a deletion) and considering the cost of the word sequences to keep?\n\nThank you for providing your valuable feedback on the new interactive policy and editing cost. We greatly appreciate your thoughtful suggestions, as they will help us enhance our platform to support a wider range of interactive policies and IMT systems.  \n(a) In response to your concerns regarding the choice of considering the \"keep\" cost as 0 and the \"delete\" cost as 1, we understand the need for clarification and would like to provide a detailed explanation to address any potential confusion. In our platform, words that are not modified or edited in any way are automatically considered as **right** words during the interactive process. This approach aligns with the post editing behavior of users. Therefore, the cost associated with the \"keep\" operation is set to 0, as there is no additional effort or modification required for these words. Regarding the \"delete\" operation, our platform allows users to select a word span using a mouse or touchpad and then simply press the \"delete\" key. This method closely resembles the observed user behavior in human evaluation. While we acknowledge that the cost of using a mouse or touchpad could be considered, we have chosen to ignore it in the current version for the sake of simplicity. As a result, the total cost for the \"delete\" operation is set to 1. We hope this explanation clarifies the rationale behind our chosen costs for the \"keep\" and \"delete\" operations in our platform.\n\n(b) Your suggested policy has motivated us to develop a refined framework that can effectively accommodate both our proposed interaction strategy and your suggested approach. We appreciate the different perspective you have brought to the table. In your suggested setting, words that are not modified or edited in any way are automatically considered as **wrong** words during the interactive process. This implies that we would need to introduce a \"new\" key to facilitate the \"keep\" operation, similar to the \"delete\" operation mentioned earlier. Consequently, the cost associated with the \"keep\" operation would be set to 1.  To address these differing approaches, we could introduce a default system mode that unifies the two methods. This system mode would trigger different automatic post-processing steps after user editing. Under this unified approach, we would set the total cost of both the \"keep\" and \"delete\" operations to 1. Then, when using different system modes, one of these costs would not be calculated. It's important to note that our current communication interface is also compatible with this setting. We believe that this refined framework will provide the flexibility needed to accommodate both our proposed interaction strategy and your suggested approach.\n\n> On the other hand, author are using different IMT systems which have not been trained to perform with a particular policy. This, as far as I seen, what they are doing is not assessing which IMT policy performs better (overall) but how each IMT systems performs for each policy.\n\nYes, we also want to emphesize that assesssing how each IMT system performs for each policy can bring out a deeper understanding of the relationship between the IMT system and each policy, which is helpful to guide the human translators on how to better interact with the IMT system.\n\n> Finally, the proposed protocol limits the user interaction to several keyboard-related operations, missing the opportunity to use other peripherals to reduce the editing cost even more\n\nWe will append more discussions and cite these papers in our final version. We admit that our protocol is limited to some given operations at this time, yet it is capable of evaluating most common IMT systems. More efforts are needed to update our this protocal to support more operations, and we also hope that the community will participate in this project.\n\n> In any case, with this ramble I wanted to point out the importance of designing a more-complex, good evaluation scenario that could cover as many IMT protocols as possible.\n\nWe agree with the importance of designing a more complex, good evaluation scenario that could cover as many IMT protocols as possible, which is also our goal. Our current version is able to cover the most of common evaluation scenarios. We will also be dedicated to this task and strive to make necessary progress. We hope that the community will participate in this project and collectively advance the field of IMT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It deploys an open-source platform which could lead towards a unification of designing and evaluating IMT protocols.", "reasons_to_reject": "* The edit operations need to be carefully revised and extended to offer more freedom to IMT protocols.\n* While interesting, the evaluation does not meet its intended purpose.", "questions_to_authors": "I am curios as why consider the \"keep\" cost as 0 and the \"delete\" as 1. On the one hand, I don't think that a fixed cost of 1 is fair (even if it can be performed by keeping the \"delete\" key pressed, it is still not comparable with the cost of single-pressing any other key).\n\nOn the other hand, wouldn't it be better to leave the delete cost as 0 (i.e., not performing any operation to a word sequence equal to a deletion) and considering the cost of the word sequences to keep? On average, I believe there are going to be more word sequences to delete than to keep (considering that once you select a word sequence as \"keep\", this could be stored and automatically performed for the subsequent interactions). Of course, which cost to assign to that operation would be critical, but it could be leveraged by adding the use of the mouse (e.g, Peris et al. consider one mouse action for keeping single words and two for any other word sequence).\n\nIn any case, with this ramble I wanted to point out the importance of designing a more-complex, good evaluation scenario that could cover as many IMT protocols as possible.", "paper_topic_and_main_contributions": "This work contributes an open-source platform to build and evaluate IMT systems. It also proposes some evaluation criteria and performs the evaluation of several IMT policies over four different models.\n\nOn the one hand, I don't think the proposed editing cost is equally fair for all the IMT policies. In fact, I believe this is the reason why the prefix-based policy always yields the lowest cost: it is reduced to the number of character in the word correction + 1, while other more complex policies need to deal with word deletions.\n\nOn the other hand, author are using different IMT systems which have not been trained to perform with a particular policy. This, as far as I seen, what they are doing is not assessing which IMT policy performs better (overall) but how each IMT systems performs for each policy.\n\nFinally, the proposed protocol limits the user interaction to several keyboard-related operations, missing the opportunity to use other peripherals to reduce the editing cost even more (e.g., the use of the mouse has showed significant improvements in the literature (e.g., Sanchis et al., 2008; Navarro and Casacuberta, 2022).\n\nReferences\nSanchis-Trilles, G.; Ortíz-Martínez, D.; Civera, J.; Casacuberta, F.; Vidal, E.; Hoang, H. Improving Interactive Machine Translation via Mouse Actions. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Honolulu, HI, USA, 25–27 October 2008; pp. 485–494\nNavarro, Ángel, and Francisco Casacuberta. 2022. \"On the Use of Mouse Actions at the Character Level\" Information 13, no. 6: 294. https://doi.org/10.3390/info13060294", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WDwMonJ5Es", "reviewer": "Reviewer_UJug", "summary": "", "questions": "- Line 323: I'm not entirely sure what MTPE is: Is the MT output corrected in 1 step? If not, which edit is applied first? In general it would be helpful to have examples for each policy.\n- How is prefix decoding implemented in the policies and in the user interface? First removal of the remaining suffix and then blanking?\n- It would be intersting to compare the overall editing time in the human evaluation.\n- Line 450: What is the variance in the Rand experiments?\n- Line 405: What is the success rate in the simulated experiments?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide a comprehensive review of our work. We appreciate your thoughtful feedback and constructive comments. Below, we address the concerns and questions you raised.\n\n> Line 323: I'm not entirely sure what MTPE is: Is the MT output corrected in 1 step? If not, which edit is applied first? In general it would be helpful to have examples for each policy.\n\nYes, the output is corrected in 1 step. There are some examples for each policy in the supplementary materials. We will also include more examples for each policy in the appendix of the final version. \n\n> How is prefix decoding implemented in the policies and in the user interface? First removal of the remaining suffix and then blanking?\n\nIn the **L2r** policy, the wrong suffix is directly replaced with a blank using the *blank-infilling* operation. In the human interface, the user can also use the hot key to directly replace texts to the right of the cursor with a blank.\n\n> It would be interesting to compare the overall editing time in the human evaluation.\n\nWe apologize that the editing time is not recorded in the human evaluation. Thank you for pointing it out and we would like to incorporate this indicator in the future version. \n\n> Line 450: What is the variance in the Rand experiments?\n\nThe variance of editing cost is shown in the following table. Most of the variance is less than 1. We will append these results in the future version.\n\n|         | ende  |       | deen  |       | zhen  |       | enzh  |       |\n|---------|-------|-------|-------|-------|-------|-------|-------|-------|\n|         | Rand  | RandI | Rand  | RandI | Rand  | RandI | Rand  | RandI |\n| DBA     | 0.161 | 0.592 | 2.784 | 0.048 | 3.702 | 0.197 | 0.120 | 0.024 |\n| BiTIIMT | 0.678 | 0.403 | 0.001 | 0.151 | 0.217 | 0.318 | 0.020 | 0.015 |\n| LeCA    | 0.069 | 0.041 | 0.563 | 0.572 | 0.105 | 0.484 | 0.725 | 0.252 |\n\n> Line 405: What is the success rate in the simulated experiments?\n\nTo model situations where users become impatient with the interactive process, we assume that the simulated user will lose patience once the prefix constraints in **L2r** are not met. As **Rand**, **L2rI**, and **RandI** offer greater flexibility in selecting editing operations, we assume that the simulated user is more likely to tolerate constraint violations up to three times. In addition, if the number of turns exceeds the threshold defined as the length of MTPE editing sequence, the system also fails (it means that the interactive process could not bring improvement in editing efficiency). More details can be found from line 361 to line 378 on paper.\n\n> At times the paper is lacking important details which may hinder replicability.\n\nAs mentioned in the paper, we will release our code, models and data after the acceptance of this paper.\n\n> Missing references and presentation improvements\n\nWe will add more related work and revise the presentation in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The unified toolkit for evaluating interactive machine translation is likely to be very useful to compare interactive machine translation approaches.\n- The proposed evaluation metrics and abstractions should further help in this regard.", "reasons_to_reject": "- At times the paper is lacking important details which may hinder replicability.", "questions_to_authors": "- Line 323: I'm not entirely sure what MTPE is: Is the MT output corrected in 1 step? If not, which edit is applied first? In general it would be helpful to have examples for each policy.\n- How is prefix decoding implemented in the policies and in the user interface? First removal of the remaining suffix and then blanking?\n- It would be intersting to compare the overall editing time in the human evaluation.\n- Line 450: What is the variance in the Rand experiments?\n- Line 405: What is the success rate in the simulated experiments?", "paper_topic_and_main_contributions": "The authors present a unified toolkit for the evaluation if interactive machine translation approaches. Formally, they define a set of operations (keep, insert, replace, delete, blank filling) and policies (prefix decoding and several variations of simulating in-filling). Given a policy and a set of edits, the backing machine translation systems are queried to provide a step towards an improved translation. In an evaluation, four different models are compared employing the four different policies. Metrics include the model's response time and the average number of turns needed to arrive at the final translation. In simulated experiments the authors show that a traditional transformer model doing prefix-decoding performs best in most metrics, an in-filling approach comes however quite close.\nLastly, the authors present a short human evaluation, where first good correlation is shown to the simulated experiments.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The tables are somewhat too dense."}, "extra_text": {"Missing_References": "- \"A Post-editing Interface for Immediate Adaptation in Statistical Machine Translation\" presents a variant of interactive machine translation where phrases of a phrase-based MT system can be moved inidivudally."}}]

Author rebuttal:

Meta-review: The reviewers were unanimous in their thoughts about this paper as being exciting and has potential to be a strong addition to the program.
Decision: Accept-Main