Rethinking Word-Level Auto-Completion in Computer-Aided Translation
Authors: Xingyu Chen
Lemao Liu
Guoping Huang
Zhirui Zhang
Mingming Yang
Shuming Shi
Rui Wang
Venue: EMNLP 2023 Main (2023)
Keywords: machine translation
interactive machine translation
computer assisted translation
word level auto completion
TL;DR: This paper propose a joint-training method to improve the performance of the word-level auto-completion task.

Abstract: Word-level auto-completion (WLAC) plays a crucial role in Computer-Assisted Translation. While previous studies have primarily focused on designing complex model architectures, this paper takes a different perspective by rethinking the fundamental question: what kind of words are good auto-completions? We introduce a measurable criterion to address this question and discover that existing WLAC models often fail to meet this criterion. Building upon this observation, we propose an effective approach to enhance WLAC performance by promoting adherence to the criterion. Notably, the proposed approach is general and can be applied to various encoder-based architectures. Through extensive experiments, we demonstrate that our approach outperforms the top-performing system submitted to the WLAC shared tasks in WMT2022, while utilizing significantly smaller model sizes.

Rethinking Word-Level Auto-Completion in Computer-Aided Translation
Xingyu Chen Lemao Liu Guoping Huang, # Zhirui Zhan Mingming Yang: Shuming Shiş Rui Wang
Shanghai Jiao Tong University, China
8 Tencent Al Lab, China
(galaxychen, wangrui 12] @ @sjtu.edu.cn (redmondliu, donkeyhuang, jackzrzhang, shanemmyang, shumingshi] | @ @tencent.com
Abstract
Tronslate Chinese sentence to English
Word-Level Auto-Completion (WLAC) plays a erucial role in Computer-Assisted Transla- tion. It aims at providing word-level auto- completion suggestions for human translators. While previous studies have primarily focused on designing complex model architectures, this paper takes different perspective by rethink- ing the fundamental question: what kind of words are good auto-completions? We intro- duce measurable criterion answer this ques- tion and discover that existing WLAC mod- els often fail to meet this criterion. Building upon this observation, we propose an effec- tive approach to enhance WLAC performance by promoting adherence to the criterion. No- tably, the proposed approach is general and can be applied to various encoder- based archi- tectures. Through extensive experiments, we demonstrate that our approach outperforms the lop-performing system submitted to the WLAC shared tasks in WMT2022, while utilizing sig- nificantly smaller model sizes?.
Chinase:
A Ask At.
one step mon, one giont loop for markind.
Translation
Winhout hysothesis
That's small
so
5eo small
With MT hypothess:
That's one small stop for ma, but a big step for humority.
stop
angle
egnifient
An example of word-level auto completion. Assume the human translator is going to imput the Golden Translation. The complcomplltion suggests the possible word candidates given the typed characters. It can be more accurate with the help of translation hy- pothesis from MT models.
Figure 1:
user survey indicates that 90.2% of participants find the word-level auto-suggestion feature help- ful (Moslem et al., 2022). Therefore, WLAC plays an important role in CAT.
1 Introduction
There are many existing methods for modeling WLAC, and they mainly differ in model architec- tures (Li et al., 2021; Yang al., 2022b; Moslem al., 2022: Yang al., 2022a; Ailem et al., 2022). For example, Li al. (2021); Yang al. (2022a) design a BERT-like architecture to directly predict the target word while Yang et al. (2022b) employ a model similar to the auto-regressive NMT to pre- dict the BPE tokens of the target word.
recent years, more and more researchers have studied computer-aided translation (CAT) that aims to assist human translators to translate the input text (Alabau et al., 2014; Knowles and Koehn, 2016; Hokamp and Liu, 2017: Santy et al., 2019; Huang et al., 2021; Weng al., 2019). The word- level auto-completion (WLAC) task (Casacuberta et al., 2022) is the core function of CAT, which in- volves predicting the word being typed by the trans- lator given the translation context, as illustrated in Figure! 1. Effective auto-completion has the poten- tial to reduce keystrokes by at least 60% during the translation process (Langlais et al., 2000). A
Take Figure as an example, the task of WLAC can be described as: given the source sentence (the Chinese sentence), the partial translated sentence ("That's one small") and the typed character quence ("s"), predict the word w that the human translator going to input ("step"). This paper goes beyond model architectures and reconsiders the essence of WLAC, posing fundamental ques- tion ($3): what defines a correct word w? Theo-
"This work completed during the Tencent Al Lab internship.
'The conresponding authors
1Our code is publicly available at https://giith: com/ galaxyChen/Mi CWAeeenn//AC---aae Joint-Training.
Training.

---

retically, good should appear in the reference translation, as illustrated in Figure 1. However, in practice, this criterion not directly applicable for improving the performance WLAC because the reference translation unavailable during infer- ence. Therefore, we attempt to relax this criterion by replacing the reference with the output from a trained machine translation system. The exper- imental results show that the relaxed criterion is reasonable: the predicted w that are present in the hypothesis exhibit significantly higher accuracy compared to those absent in the hy- pothesis. Furthermore, we find that existing WLAC models usually fail to meet the relaxed criterion (see Table 2 later). This suggests an opportunity for improving their performance by meeting the criterion.
Rethinking the WLAC task by investigating the criterion for golden WLAC prediction. and we find that the representative baseline models often fail to meet this criterion.
To address this criterion violation, we pro- pose an effective approach that leverages joint training to implicitly encourage agreement be- tween WLAC and Machine Translation.
Experimental results demonstrate that our pro- posed approach achieves remarkable improve- ments, surpassing tfate-of-the-art (SOTA) models by significant margin. Notably, our approach also exhibits advantages in terms of model size and generality.
Backbone Models for WLAC
Based on the above finding, this paper presents a novel approach to enhance WLAC systems, which can be applied to different model architectures ($4). The key idea is to encourage WLAC's output to sat- isfy the relaxed criterion as much as possible, i.e., enhancing the agreement between WLAC and MT system such that the output exists in the machine- translated hypothesis. Guided by this principle. two distinct methods are proposed. The first method achieves this agreement during the inference phase. This method requires rumning MT decoding during inference, which is inefficient. Moreover, its per- formance is limited by the quality of MT. To alle- viate both issues, we introduce the second method, which achieves agreement through joint training of WLAC and MT. The intuition behind this method is that the training stage, we generate the train- ing data for the WLAC task from parallel corpus, so the WLAC's ground truth naturally exists in the reference translation. By jointly training the two models, we can implicitly learn the agreement between WLAC and MT. Notably, this method by- passes the need for MT decoding during inference, resulting in improved efficiency and removing the constraints imposed by MT's quality.
The WLAC task comes from a real translation sce- nario: human translator is translating source sentence, who has already translated part of the sentence, and is typing a new word. The input con- tains three parts: the source sentence s, the partial translation e, and the typed character sequence As shown in Figure 1, The WLAC task is to predict the word w that the translator is going to input (Li et al., 2021; Casacuberta et al., formulaee as:
P(w) F(s.
(1)
where P(w) is the probability distribution of w, and F is the word prediction model. According toLi et al. (2021), the context four types depending on the input position, here we only consider the left context ej and the right context c, to the input, and both e and e, could be empty.
be categorized into
In this section, we introduce two types of back- bone models for the WLAC task. These backbone models serve the foundations for our proposed techniques and experiments in the subsequent sec- tions.
Word-level Model The first backbone is called All-In-One Encoder (AIOE), which adopts a BERT- like(Devlin et al., 2019) Transformer Encoder ar- chitecture for word prediction similar to Li et al. (2021). The AIOE takes the concatenation of the source sentence, context, and typed sequence its input. The input format is: s <sep> et <tip> <mask> c,, where c; the left context the input and e, is the right context. Specifically, we append a <mask> token at the end of the typed character
The effectiveness of our proposed method is val- idated through experiments on the four language di- rections of the WLAC shared task in WMT22 ($5). Our approach achieves substantial improvements across two distinct backbone models. Notably, it surpasses the performance of the WMT22's best system with only 20% of the parameters and much simpler architecture.
This paper makes the following contributions:

---

sequence and leverage the final hidden state of the <mask> token for word prediction.
criterion next.
relaxed criterion Since the golden translation is not available during inference, we propose to utilize the output of a machine translation model, named T. as the approximation of T. Thus, we establish the relaxed criterion based on the T: is good prediction if is contained in the machine- translated hypothesis T. In this case, we call agrees with T or the WLAC model agrees with the machine translation model. Given a data point (st, C22 t,, w.). we can compute the agreement by:
Despite its simplicity and efficiency, the AIOE model suffers from the out-of-vocabulary (OOV) problem, which can significantly hinder its perfor- mance. To this end, we introduce a variance of AIOE model that predicts word in sub-word level.
Sub-word-level Model Extending the word-level AIOE model to sub-word-level straightforward: we consider the task of predicting sequence of sub-words as a generation problem, and introduce a Transformer Decoder to the AIOE model to per- form the generation. We use Byte-Pair Encoding (BPE) (Sennrich et al., 2016) for the sub-word to- kenization, and call this model AIOE-BPE. Yang al. (2022b) also propose sub-word prediction at the BPE level, but their model treats it as se- quence of mask- preni-predition tasks, requiring interac- tively insert generated sub-words into the input. In comparison, the AIOE-BPE model offers a simpler architecture and has the ability to utilize advanced generation techniques such as beam search.
T. = T(s;, Ci2 tú);
(2)
Agreement, = I(w; E T.),
(3)
where T is a translation model and I[()) is the indi- cator function. For the machine translation output T may introduce noise and deviate from T, it is nat- ural to ask, how effective is this relaxed criterion? Therefore, we conduct preliminary experiments to justify the usefulness of this relaxed criterion.
3.2 Empirical Justification
Due to the difficulty of labeling the WLAC data, we generate training data from parallel corpus for training the WLAC models, following the standard practice (Li et al., 2021; Casacuberta et al., 2022).
To evaluate whether the relaxed criterion can be used as indicator of good predictions, we exam- ine the accuracy of words that agree (Agr. Acc.) and disagree (Disagr. Acc.) with the machine- translated outputs. We train an NMT model to gen- erate top-5 translations for each source sentence in the test set, which serve as the noisy translation T. The analysis is conducted on the WMT22-WLAC testset for zh-en direction.
3 What Is a Good Prediction in WLAC?
In this paper, we rethink the WLAC task by posing a fundamental question: What is "good" predic- tion given the source sentence s, the context e and a typed character sequence t? In the subsequent sec- tions, we explore the criterion to assess candidate predictions and use it to improve the performance during the inference.
WLAC AIOE
Agr. Acc. Disagr. Acc. A
63.75%
44.97%
18.78%
AIOE-BPE
64.82%
49.82%
15.00%
3.1 A Criterion to Being a Good Prediction
Table I: Accuracy of agreement and disagreement un- der the relaxed criterion. Agr/Disagr: Acc. repre- sents the percentage of correct predictions for the agree- ments/disagreemmetts between WLAC and MT model.
A principle criterion Rooted in the translation natural, the word w predicted by a WLAC model is good prediction if w and the context с has the potential to lead to a high-quality translation T (e.g. the translation the human translator is go- ing to input). In other words, w is likely to be a good prediction if it is contained in high-quality (golden) translation T. This can be considered as example:
Is the relaxed criterion reasonable? Table shows the Arg. Acc. and Disagr. Acc for two backbone models, which reveals insightful find- ings. Although the agreement accuracy (about 64%) is limited by the discrepancy between noisy and golden translation, the accuracy drops dramati- cally if the prediction disagrees with the translation T. Therefore, despite the presence of noisy transla- tions, these results support the relaxed criterion to some extent: predictions that agree with translation
principle criterion. Take Figure the prediction step is good because it exists in the golden translation. However, this principled crite- rion is hard to apply in practice because we can not access the ground truth translation T during infer- ence. Therefore, we explore a practical, relaxed

---

are more likely to be correct, whereas violations of the relaxed criterion (disagreements) would lead to poor accuracy.
examine each word in the predictions and check if included in the translation. The first word in the top-k list that exists the translation selected as the final prediction. This strategy manually align the prediction with translation in a flexible way: the choice of WLAC model and translation model is arbitrary. The final performance is closely related to the choices of models.
WLAC
Agreement
Disagreement
AIOE
47.36%
52.64%
AIOE-BPE
49.03%
50.97%
Table 2:
The percentage of agreement between NMT and each backbone WLAC model.
However, this approach heavily relies on the quality of translation. A preliminary analysis show that for a naive MT model, only 44.6% of the WLAC labels exist in the translation. In such sce- narios, enhancing agreement alone does not guar- antee performance improvements. One possible solution is to enhance the input of MT model. We propose Context MT model, which takes addi- tional translation context and typed sequence input, and generates full target sentence. The input of Context MT is the same as WL.AC, so it's a better approximation of the golden translation model.
Backbone models usually fail to meet the relaxed criterion. Additionally, we analyze the degree of agreement between each backbone model with MT and the result is illustrated in Table 2. Surprisingly, we observe that less than a half of the predictions agree with the translations, i.e., WLAC model vio- lates the relaxed criterion with large probability, Considering the substantial accuracy gap between agreement and disagreement, there is significant room for performance enhancement.
4.2 Enhancing agreement via Joint Training
3.3 The relation between accuracy and agreement
One drawback of joint inference method is that the WLAC model isn't aware of the translation task during training, which means that the top-k pre- dictions may deviate from the ground truth. To overcome this limitation, we propose a joint train- ing approach, wherein the WLAC model and the MT model are trained together using shared back- bone encoder. Specifically, we extend the backhone model by introducing MT decoder, transform- ing the backbone model into MT model. Here the MT model is the same as Context MT model described in $4.1. We define the training loss of the joint training model as the combination of the WLAC loss and the translation loss, represented as follows:
One might wonder why we want to introduce a new criterion to assess the prediction, in addition to the original accuracy metric. The primary dis- tinction between accuracy and agreement lies in their roles within the WLAC task. Accuracy, being the official metric, directly evaluates the correct- ness of label predictions. In contrast, agreement operates independently of labels, quantifying the presence of prediction candidates in translation hy- potheses. While accuracy assesses "correetness". it can't 't provide insights for model optimization. In the subsequent section, we describe how to lever- age agreement enhance WLAAC models.
4 Enhancing WLAC with MT by Agreement
C L= a-LAL + (1-a) a) LMrr,
(4)
Motivated by the findings from Section 3, we pro- pose two different approaches which improve the agreement between WLAC and machine translation for overall improvements in WLAC performance.
where a is a hyper-parameter controlling the bal- ance between the two losses. To enhance the in- teraction between two tasks, we also share the fi- nal word prediction layer between the backbone model and the decoder. As described in section 5.1, the training data of WLAC is generated from parallel corpus, so there will be full agreement between WLAC label and ground truth translation at the training stage. This agreement enables the WLAC model to learn how to accurately predict words within the translations. Besides, the MT
4.1 Enhancing agreement via Joint Inference
One approach to enhance agreement is jointly con- sider the WLAC predictions and machine transla- tion results during inference. We begin by gener- ating the top-k predictions from the WLAC model. Then, we use a MT model to generate translation hypothesis based on the source sentence. Next, we

---

model can learn to generate translations based on the context provided by the WLAC predictions. By jointly training the two models, we enable them to mutually benefit from each other's 's knowledge and improve their respective tasks.
AIOE-BPE This is the sub-word-leve model de- scribed in section 2. The BPE decoder is layer Transformer Decoder.
AIOE-Joint The AIOE model with machine trans-
The key advantage of joint training is that once the training is completed, we can only keep the backbone model and discard the MT decoder. Note that the backbone encoder can receive optimization signals from both the WLAC task and the trans- lation task, so the backbone model has acquired the skill to agrec with translation during training process. This enables us to maintain the agreement capabilities while preserving a small and efficient inference model.
lation joint training. The MT decoder contains 6 Transformer Decoder layers.
AIOE-BPE-Joint The AIOE-BPE model with
machine translation joint training. We share the final word prediction layer of MT decoder and AIOE-BPE backbone decoder.
5.3 Comparison among Agreement Methods
We firstly compare the performance of joint infer- ence method and joint training method. For joint inference method, we use the word-level backbone AIOE model for the WLAC model, and consider two kinds of machine translation model: translation model trained on parallel corpus (MT) and trans- lation model trained on WLAC input and transla- tion output (Context MT). We also consider the Google Translator(GTTF to see the impact of trans- lation quality. For the joint training method, we use AIOF-Joint model. All the experiments are conduct in zh-en direction. The result is reported in Table 3.
5 Experiment
5.1 Datasets
We conduct experiments on two language pairs: English- Chiisse-hhheeeeeeee and English-German. The zh-en dataset we used is the UN Parallel Corpus V1.0 from WMT17. For en-de, we use the training data from WMT14. We adopt the following strategy on parallel sentences to generate WLAC training data": firstly, we sample a target word w from the target language sentence, then we sample spans respectively from the left and right context of the target word, denoted as and c,. Additionally, sample typed sequence from the target word. To sample typed sequence from Chinese words we use the pypinyin' tool. All models are trained on the generated training data, with WMT21 data serving as the validation set. For evaluation, we utilize the test set from the WMT22 WLAC shared task.
Method AIOE
Acc.
Agr.
Agr. Acc.
53.87
47.36%
63.75%
AIOE+MT
54.20
58.61%
57.70%
AIOE+CMT
56.01
74.51%
61.98%
AIOE+GT
56.16
44.51%
62.50%
AIOE+JT 59.75
65.50%
67.51%
Table 3: Comparison of joint-methods. Acc. is the accuracy of WLAC task. Agr. is the percentage of agreements. Agr. Acc. is the percentage of the accurate prediction among agreements.
5.2 Models for Comparison
GWLAN Proposed by Li et al. (2021), GWLAN model is word-level model with two encoders.
HW-TSC The models proposed by Yang et al. (2022b) the winners of the WMT22 WLAC shared task across three language di- rections. It's BPE-level model with encoder- decoder architecture
It is observed that joint inference with MT and CMT models significantly improve the agreement, although the percentage of accurate agreement de- creases when combined with the translation mod- els. This discrepancy can be attributed to the fact that the machine translation models are unaware of the WLAC task, and their translations may deviate from the ground truth. Consequently, enhancing agreement does not guarantee an improvement in the quality of WLAC predictions under this situa- tion. On the other hand, using better MT model
AIOE
This model the word-level model de- scribed in section 2. The AIOE model con- tains 6 Transformer Encoder layers. See ap- pendix B for more training details.
hhttps::/gitthin http://tthuu..commmmmmml.lc wtwm//emmlli//MAA//bloooooo htttt//ggeeertt_aam_llll semplee.pyy. py
Thttps: / httpt/gttttttttte/e cww/mozim 11 lazg/
Chttps://clooe http://lloo.ooogllleee/oeoo wttc hom/translate/

---

(Google Translator here) can bring better perfor- mance than using NMT system or Context MT system, but the agreement drops greatly. We at- tribute this phenomenon to two primary factors: 1. we only use top translation from Google Transla- tor while we use top 5 translation from MT/CMT model. 2. The high agreement accuracy suggests that Google Translator can closely approximate human translators. As consequence, it is plausi- ble that the performance of the foundational AIOE model may serve as the limiting factor in this con- text. From the perspective of overall accuracy, al- though the joint inference method outperforms the backbone AIOE model, there is still a large gap compared to the joint training method. Based on these findings, we only focus on the joint training method for the subsequent experiments.
We employ the Context MT model, as described in Section 4.1, to generate the top-5 translations using beam search. Then we apply the aforemen- tioned inference strategies to these translations. We also evaluate the performance of GPT-text-davinci- 003 (we set the temperature as 0), including di- rectly solving the task and matching from the trans- lation. The prompts used in our experiments are provided in Appendix A. The experimental results for the zh-en language pair are presented in Table 5. Notably, while the translation upper bound is comparable to our joint training method, there still exists a large gup between the matching method and the upper bound. Considering that the inputs of Context MT and AIOE model are the same, the huge difference in their performance indicates that our joint training method is more effective in uti- lizing translation knowledge to solve WLAC task. The GPT performs badly in the WLAC task, in- dicating that we need to specifically optimize the model for the WLAC task.
5.4
Main Results
The evaluation result tth WMT22 WLAC shared task reported on Table 4. Compared to HW-TSC, our word-level methods have obtained better performance on zh-en and de-en. One ex- ception is en-de, the word-level model performed badly because it suffers from OOV problem, where about 17% labels are OOV. After replacing the backbone with BPE-level model, our method show superior performance compared to the SOTA in all directions. No matter which backbone is used, our joint training method can greatly improve the backbone performance, indicating that our method is a general framework and has the potential to be applied to more encoder based models. Another obvious advantage of our model is its superior pa- rameter efficiency. Despite being only 15% of the size of the HW-TSC model during inference, our best AIOE-BPE-Joint model outperforms the HW- TSC model significantly.
The impact of MT task The influence of the hyper-parameter a on the model performance, as outlined in equation 4, directly reflects the impact of translation task. By setting a to O, the model is essentially a translation model with additional context input. If 1, the model corresponds the AIOE model without joint training. In Figure 2. we present the accuracy achieved at varying values of a. Notably, as a increases from 0 to 0.75, the accuracy increases rapidly. This observation high- lights the difference between the translation task and the WLAC task, emphasizing the necessity of optimizing the model specifically for the WLAC task to achieve better performance. Interestingly, with set 0.99, the performance remains comparable to the best achieved performance. This finding remarkable, as it suggests that even a small signal from the translation task can greatly enhance the WLAC task's performance when com- pared to the model with set to 1. Consequently, our proposed joint training method effectively in- tegrates the translation task into the WLAC task, resulting in substantial improvements.
5.5 Ablation Study
Translation performance Our method has demonstrated the importance of leveraging ma- chine translation knowledge to solve the WLAC task, but is it possible to solve the task directly using a translation model? We use the same par- allel training data to train translation model and compare the performance of different inference strategy: 1) Upper Bound: if the top-k generated translations contain the target word, we treat it as "correct". 2) Prefix Match: We use the typed se- quence to match the top-k translations and take the most common matched word as the prediction.
6 Analysis & Discussion
The experiment results have demonstrated the ef- fectiveness of our method, but how does the MT task help WLAC? In this section, conduct de-
#https: openai. wwwpeai..woooo.ooooooeoooe/oeooo ggwdes/

---

Model
#Parameters
zh-en
en-zh
en-de
de-en
GWLAN
105M
51.11
48,90
40.69
53.87
HW-TSC
526M
59.40
63.82
62.06
AIOE
80M
54.13
52.88
39.43
56.40
AIOE-BPE
74M
57.17
52.97
56.83
61.62
AIOE-Joint
80M(105M)
59.75
56.59
44.67
62.77
AIOE-BPE-Join
74M(100M)
61.08
58.09
64.59
66.91
Table Experiment results on WMT22 WLAAC test set. Results are reported as accuracy. The number of parameters in brackets means parameters in training stage.
Method
Accuracy
UpperBound
58.05
No-Agree 22. 22.55%
PrefixMatch
45.69
Terse Eroor 8.39%
GPT-UpperBound
42.33
AIOE- Agree 19 89%
GPT-Direct
17.89
AIOE
53.87
Frequency sleadiig 9.18%
AIOE+Joint Training
59.75
Only-Joint 57 57.56%
Cehers
Table 5: Zh-En results for translation-only methods.
2.35M
0.58
Figure Improvement analysis on cases where AIOE prediction is wrong while AIOE-Joint is right. AIOE- Agree: the AIOE model agrees with the translation. Only-Joint: only the AIOE-Joint model agrees with the translation. No-Agree: none of the models agrees with the translation.
0.57
Accuracy
0.56
0.55
0.54
where we Wm and Wm. Based on whether We and Wm agree with T, we define three groups: AIOE-Agree, Only-Joint and No-Agree. The percentage of each group is illus- trated in Figure 3. Next, we will conduct detailed analysis on each group.
analysis
1.0
WLAC weight
Figure 2: The impact of different the AIOE accu- racy. Red dashed line is the best performance and the green represents the accuracy without joint training.
Does the translation help the WLAC prediction? The Only-Joini group contains cases that only AIOE-Joint model agrees with translation, which is the most notable part of improvement. The transla- tion process can be viewed as a planning procedure, guiding word prediction by leveraging knowledge of the complete translation. In this scenario, the model simply needs to select the most likely word from the translation. Though we do not explicitly generate translation during inference, the backbone model of AIOE-Joint has learnt how to encode the input for the MT task, so the translation implicitly helps the model to make correct prediction.
tailed analysis on the influence of the translation task and study the relation between generated trans- lations and predicted words. All the experiments are based on word-level AIOE-Joint model in zh-en direction.
6.1 Improvement Analysis
To gain insights into the contributions of the trans- lation task to WLAC prediction, we conduct a de- tailed analysis of the improvements observed after introducing joint training. Let We represent the predictions of the AIOE model, Wm denote the pre- dictions of the A1OE-Joint model, w denote the ground truth, and represent the translation gen- erated by the AIOE-Joint model. We focus our
Why ATOE model fails when agrees with trans- lation? From Figure 3, there are about 22.55% of the cases where the AIOE model agrees with

---

the translation. However, despite the agreement, the model still fails to make the correct predic- tion. Further analysis of this group reveals two types of errors: tense errors and frequency mislead- ing. Tense errors occur when the AIOE prediction and the label correspond to different tenses of the same word. We utilize the nltk.stem tool 9 to iden- tify tense errors. Interestingly, the introduction of MT joint training resolves the tense errors in this group, indicating that the model can generate mor- phologically accurate predictions under translation context. Frequency mislcading refers to the phc- nomenon where the AIOE model tends to predict high-frequency words. We compute the percentage of cases where AIOE predicted words are more frequent than AIOE-Joint predictions. Remarkably, approximately half of the cases in the AIOE-Agred group fall under this error category. This suggests that, with the assistance of translation, the WLAC model can prioritize words that are contextually appropriate rather than relying on repetitive high- frequency words.
ground truth w, and the translation T generated by the AIOE-Joint model. We classify the error cases based on whether the translation T includes the word Wm and w. Additionally, we report the cosine similarity between Wm and w using the fastText tool'. The results are illustrated in Figure 4.
MT failure The majority of errors can be at- tributed to the MT-Error group, where the labels are absent in the translation. The low similarity between the labels and predictions in this group in- dicates that the AIOE-Joint model fails to capture essential information from the input under such cireumstances. The significant divergence between the model's translation and the golden translation is a key factor contributing to this failure. We at- tribute the translation fails in these cases to the lack of sufficient context. In Table 6, we present the average length of context and the percentage of zero-context cases for different groups. Notably, the MT-Error group exhibits the lowest average context length and the highest percentage of zero- context cases. These findings strongly suggest that the model struggles to generate reasonable trans- lations when there is limited context. This issue could potentially be addressed by employing more advanced sampling techniques or using more pow- erful MT decoder. We will leave it future work.
Does the AIOE-Joint model "copy" words from translation? The No-Agree group offers insight about how the joint model works: substan- tial portion of correct predictions is not present in the translation, suggesting that the word prediction module does not merely copy words from the trans- lations. Instead, it utilizes translations as references to achieve the most accurate predictions.
Group
Avg. Context
Zero-Context%
Err.
5.63
26.67%
AIOE-Er
6.00
22.82%
MT-Er.
5.50
28.02%
Similarity we tanse enro
Table 6: Statistics about context length. Group Era contains all the AIOE-Joint failures. Group AIOE-Err and Group MT-Err. represents A1OE-Error and MT- Error respectively. Avg. Context the averaged length of context, and Zero-Contex % is the percentage of zero- context cases.
Simi lanntty with label
65
63
60
AIOE-Error
MT-Erroor
N/DE-Emor
Backbone failure The AIOE-Error group indi- cates that the model may deviate during word pre- diction and struggle to identify the correct target from the translation. However, the similarity this group is unexpectedly high. To figure out why AIOE model fails in these cases, we look into spe- cific error types of this group. Further analysis reveals that there are approximately 20% tense rors. After eliminating the tense errors from this group. the similarity decreases dramatically, sug-
Figure 4: Error Analysis. AIOF-Error: WLAC label exists in translation, MT-Error: urasllttoon does not the contain WLAC label.
6.2 Error Analysis
Lastly, we dive into the cases where the AIOE- Joint model encounters failures. In this analysis, we consider the AIOE-Joint prediction Wm. the
hhtps:/wwww.nkk.oaaaphhlt.tt..tt
Inttps://fasttext.ce

---

gesting that the model fails capture crucial infor- mation from the input under this situation. These errors could be mitigated by enhancing the encoder. Through our comprehensive analysis, a significant advantage of our proposed method emerges: the model's behavior is more interpretable, enabling easier identification of bottlenecks.
of pure translation methods without training on WLAC task. We emphasize the importance of em- ploying translation knowledge during the WLAC training, so we proposed to jointly train the two tasks, incorporating translation into word predic- tion by soft yet effective way.
Conclusion
7
Related Work
This paper firstly rethinks the essence of the "cor- rect" prediction for WLAC and we propose agree- ment, a measurable criterion to judge whether a prediction word is good not. Surprisingly, we find that existing WLAC models usually violate this criterion with high probability. Based on this findings, we present two effective strategies; joint inference and joint training with machine transla- tion (MT), aimed at enhancing WLAC performance by encouraging the model to adhere to the agree- ment criterion. Extensive experiments show that the proposed approach surpasses the best system submitted to the WLAC shared tasks in WMT2022, with much smaller model size. Additionally, our result analysis highlights the effective integration of MT knowledge into the WLAC task, resulting improved performance. The error analysis also re- veals the directions for further optimization, which we will leave as future work.
7.1 Interactive Machine Translation
Interactive machine translation is a sub-field of computer aided translation, where human translator would write the translation interactively. An IMT system would generate translation hypothesis based on the current context, and the human trans- lator would be asked to post edit the sentence. Each time the translator edit the sentence, a new hypoth- esis would be generated, and this process would continue until the translator satisfied with the translation. Different IMT systems have emerged (Green et al., 2014; Hokamp and Liu, 2017; Weng et al., 2019; Wang et al., 2020; Huang et al., 2021), and it's reported that IMT systems can improve translation quality (Green et al., 2014) 4) and is ben- efit the potential users (Casacuberta al., 2009; Barrachina al., 2009).
7.2 Auto Completion
Acknowledgements
Auto completion is the core function of IMT, in- cluding sentence-level auto completion and word- level auto completion. Previous work mainly focus on sentence-level auto completion (Alabau al., 2014; Zhao al., 2020), where the system would complete the whole translation based on user typed prefix. However, sentence-level auto completion requires to generate a complete sentence every time the user inputs, and the overhead of the system is very high.
Xingyu and Rui are with the MT-Lab, Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineer- ing, and also with the MoE Key Lab of Artifi- cial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200204, China. Rui is sup- ported by the General Program of the National Natural Science Foundation of China (62176153), the Shanghai Pujiang Program (21PJ1406800), the Shanghai Municipal Science and Technology Ma- jor Project (2021SHZDZX0102,, the Alibaba-AIR Program (22088682), and the Tencent AI Lab RBFR2023012.
Word-level auto completion only predicts the word that the user going to input. Previous stud- ies have explored predicting the next word based on the prefix and typed chars (Langlais et al., 2000; Santy et al., 2019), while our work focuses on real- world translation scenarios like post-editing (Vas- concellos and León, 1985; Green et al., 2014). Yang et al. (2022a) proposed to encoder-like architecture predict the target word and Yang al. (2023) designed an energy based model for reranking, Yang et al. (2022b) additionally intro- duce machine translation as pre-training task, and Moslem et al. (2022) examined the performance
Limitations
Based on our error analysis, a significant portion of errors can still be attributed to MT errors, indicating that even with joint training, there are discrepancies between the MT model and the golden translation.
Furthermore, our analytical experiments were conducted specifically on the zh-en language pair,

---

and the generalizability of our findings other lan- guages may vary. While our main experiments have demonstrated the effectiveness of the proposed joint training method, it is essential to examine the improvement analysis and error analysis for spe- cific linguistic phenomena, such as the prediction of German grammatical gender. To gain compre- hensive understanding of the effectiveness of the joint training approach, our future work includes extensive experiments on multiple languages.
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Spence Green, Sida I Wang, Jason Chuang, Jeffrey Heer, Sebastian Schuster, and Christopher D Man- ning. 2014. Human effort and machine learnability in computer aided translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1225-1236.
Chris Hokamp and Qun Liu. 2017. Lexically con- strained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1535-1546, Vancouver, Canada. Association for Computational Linguistics.
Lastly, the current WLAC task primarily focuses on high-resource languages. Exploring the perfor- mance of our method in low low-resource scenarios is also an important area for future investigation.
Guoping Huang, Lemao Liu, Xing Wang, Longyue Wang, Huayang Li. Zhaopeng Tu, Chengyan Huang. and Shuming Shi. 2021. Transmart: A practical in- teractive machine translation system. arXiv preprint orXiv:2105.13072
References
Melissa Ailem, Jingshu Liu, Jean- Gabriel Barthélemy, and Raheel Qader. 2022. Lingua custodia's partici- pation at the wmn 2022 word-level auto-completion shared task. In Proceedings of the Seventh Confer- ence on Machine Translation (WMT), pages 170- 1175.
Rebecca Knowles and Philipp Kochn. 2016. Neural interactive translation prediction. In 12th Confer- ences of the Association for Machine Translation in the Americas: MI Researchers' Track, AMTA 2016, Austin, TX, USA, October 28- November 1, 2016. pages 107-120. The Association for Machine Trans- lation in the Americas.
Vicent Alabau, Christian Buck, Michael Carl, Francisco Casacuberta, Mercedes Garcfa-Martinez. Ulrich Ger- mann, Jesús Konzález-Rubio, Robin Hill, Philipp Koehn, Luis A Leiva, et al. 2014. Casmacat: A computer- assisted translation workbench. In Pro- ceedings of the Demonstrations the 14th Confer- ence of the European Chapter of the Association for Compistational Linguistics.
Philippe Langlais, George Foster, and Guy Lapalme. 2000. Transtype: a computer-aided translation typ- ing system. In ANI.-NAACI 2000 Workshop: Em- bedded Machine Translation Systems.
Sergio Barrachina, Oliver Bender, Francisco Casacu- berta, Jorge Civera. Elsa Cubel, Shahram Khadivi, Antonio Lagarda, Hermann Ney, Jesús Tomás, En- rique Vidal, et al. 2009. Statistical approaches to computer-assisted translation. Computational Lin- guistics, 35(1):3 28.
Huayang Li, Lemao Liu, Guoping Huang, and Shuming Shi. 2021. GWLAN: General word level Autocom- pletioN for computer-aided translation. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online.
Francisco Casacuberta, Jorge Civera, Elsa Cubel. An- tonio L Lagarda, Guy Lapalme, Elliott Macklovitch, and Enrique Vidal. 2009. Human interaction for high- quality machine translation. Communications of the ACM. 52(10))135-13)
Yasmin Moslem, Rejwanul Haque, and Andy Way. 2022. Translation word-level auto-completion: What can we achieve out of the box? arXiv preprint arXiv:2210.12802.
Francisco Casacuberta. George Foster, Guoping Huang. Philipp Koehn, Geza Kovacs, Lemao Liu, Shum- ing Shi, Taro Watanabe, and Chengging Zong. 2022. Findings of the word-level autocompletion shared task in wmt 2022. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 812-820.
Sebastin Santy, Sandipan Dandapat, Monojit Choud- hury, and Kalika Bali. 2019. INMT: interactive neu- ral machine translation prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing, WANNP-IICNLP 2019, Hong Kong, China, November 3-7, 2019 System Demonstrations, pages Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1715-1725,

---

Berlin, Germany, Association for Computational Lin- guistics.
is "source sentence". The translator has al- ready translated part of the sentence, the left context is "left context". The translator now typing a new word after the left context, the typed sequence is "typed sequence", what's s the word the translator is going to imput? Your answer should only contain the word.
Muriel Vasconcellos and Marjorie León. 1985. Spanam and engspan: machine translation at the pan amen- can health organization. Computational Linguistics, 11(--3:122--36.
Qian Wang. Jiajun Zhang, Lemao Liu, Guoping Huang. and Chengqing Zong. 2020. Touch editing: flex- ible one-time interaction approach for translation. In Proceedings of the 1st Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Confer- ence Natural Language Processing, pages 1-11, Suzhou, China. Association for Computational Lin- guistics.
Suffix human translator translating a Chinese sentence into English. The Chinese sentence is "source sentence". The translator has al- ready translated part of the sentence, the right context is "right context". The translator is now typing a new word before the right con- text, the typed sequence is "typed sequence", what's the word the translator is going in- put? Your answer should only contain the word.
Rongxiang Weng, Hao Zhou, Shujian Huang, Lei Li, Yifan Xia, and Jiajun Chen. 2019. Correct-and- memorize: Learning to translate from interactive revisions. In Proceedings of the Twenty-Eighe Inier- national Joint Conference Artificial Intelligence ICCAI-19, pages ferences on Artificial Intelligence Organization.
International Joint Con-
Bi-context A human translator is translating Chi- nese sentence into English. The Chinese sen- tence is "source sentence". The translator has already translated part of the sentence, the left context is "left context". and the right context is "right context". The translator now typ- ing a new word between the left context and right context, the typed sequence is "typed sequence". what's the word the translator is going to input? Your answer should only con- tain the word.
Cheng Yang, Guoping Huang. Mo Yu, Zhirui Zhang, Si- heng Li, Mingming Yang, Shuming Shi, Yujiu Yang. and Lemao Liu. 2023. An energy-based model for word-level autocompletion in computer-aided trans- lation. Transactions of ACL.
Cheng Yang. Siheng Li, Chufan Shi, and Yujiu Yang. 2022a. ligroup submissions for wmf22 word-level autocompletion task. In Proceedings the Seventh Conference on Machine Translation (WMT).
Hao Yang, Hengchao Shang, Zongyao Li, Daimeng Wei, Xianghui He, Xiaoyu Chen, Zhengzhe Yu, Ji- axin Guo, Jinlong Yang. Shaojun Li, al. 2022b. Hw-tsc's submissions to the wmt22 word-level auto completion task. In Proceedings of the Seventh Con- ference on Machine Translation Shared Tasks Papers. Association for Computational Linguistics.
Zero-context human translator is translating a Chinese sentence into English. The Chinese sentence is "source sentence". The translator is now typing word, the typed sequence is "ryped sequence", what's the word the trans- lator is going input? Your answer should only contain the word.
Tianxiang Zhao, Lemao Liu, Guoping Huang, Huayang Li, Yingling Liu, Liu GuiQuan, and Shuming Shi. 2020. Balancing quality and human involvement: An effective approach to interactive neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9660-
B Training Details
For all AIOE model, we use Transformer En- coder for layers. The embedding size is 512, the dimension for feed-forward layer 2048. Each layer has 8 attention heads. For AIOE-BPE model, we additionally add a Transformer Decoder with 6 layers.
9667.
A GPT prompts
A.1 Prompts for directly solving problems
For AIOE model, we use joint-vocabulary with the size of 120000. For AIOE-BPE model, the vocabulary size is 66630 for English-Chiness pair and 59918 for English-German pair.
We design different prompts for different types of context as described in (Li et al., 2021). Words in italic will be replaced by the actual input for each case.
The learning rate for training 5e-4. We opti- mize the model for 200000 steps with a batch size of 32000 tokens.
Prefix A human translator is translating a Chinese sentence into English. The Chinese sentence


Reviews:
[{"review_id": "FA980cddPq", "reviewer": "Reviewer_1XZn", "summary": "", "questions": "- Is it possible to combine joint training and joint inference? If so, does it result in performance improvements?\n- Can you provide more details or codes for generating the synthetic dataset? While it is well-written, some minor details are required for reproduction (e.g., span sampling probability, ratio of MT and WLAC data, and dynamic/static sampling of synthetic data)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments. We address your comments below.\n\n**Q1. Is it possible to combine joint training and joint inference? If so, does it result in performance improvements?**\n\nYes, it's possible to combine these two methods. We show the result of zh-en below:\n\n| Model                               | Acc.  |\n| ----------------------------------- | ----- |\n| AIOE                                | 53.87 |\n| AIOE+Joint Inference                | 56.01 |\n| AIOE+Joint Training                 | 59.75 |\n| AIOE+Joint Training+Joint Inference | 59.28 |\n\nAs we can see, the performance of combining two methods is comparable with using joint training only. The procedure of joint inference is: 1. Use MT model to generate translation hypotheses. 2. Check whether the top-5 predictions of WLAC model exist in the translation hypotheses, if so, use it as the final prediction. The performance of joint inference is greatly hindered by the quality of MT model. Even the joint training method can produce more accurate prediction candidates, the poor MT hypotheses would mislead the prediction selection.\n\n**Q2. Can you provide more details or codes for generating the synthetic dataset? While it is well-written, some minor details are required for reproduction (e.g., span sampling probability, ratio of MT and WLAC data, and dynamic/static sampling of synthetic data)**\n\nYes, we will release our reproducible code along with the next version of paper. We follow the official WLAC data generation script (https://github.com/lemaoliu/WLAC/blob/main/scripts/generate_samples.py) to generate training data for the WLAC task. The ratio of translation data and the WLAC data is 1:1, and we dynamically generate training examples at the training time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The method is well-motivated, simple, and the results are promising.\n- The experiments and analysis provided are sufficient to support the effectiveness of the method.", "reasons_to_reject": "- I don't see any clear reason to reject this paper.", "questions_to_authors": "- Is it possible to combine joint training and joint inference? If so, does it result in performance improvements?\n- Can you provide more details or codes for generating the synthetic dataset? While it is well-written, some minor details are required for reproduction (e.g., span sampling probability, ratio of MT and WLAC data, and dynamic/static sampling of synthetic data)", "paper_topic_and_main_contributions": "This paper revisits the problem of word-level auto-completion by studying its relation to machine translation. It analyzes the limitations of previous methods and proposes joint inference and joint training of WLAC and MT. The proposed method outperforms previous approaches, and further analysis also support the paper.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XFn8Ei4Wmz", "reviewer": "Reviewer_4EMT", "summary": "", "questions": "Out of curiosity: Do you have an idea how much MT quality affects the performance of the AIOE+MT model? For example if you use Google Translate or GPT-4, can you do better than AIOE-Joint?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments. We admit that some concepts in our paper should be clarified and we will improve it in the next version. We address your comments below.\n\n**Q1 Do you have an idea how much MT quality affects the performance of the AIOE+MT model? For example if you use Google Translate or GPT-4, can you do better than AIOE-Joint?**\n\nThe MT quality does affects the performance of AIOE+MT model. We use the Google Translation API to generate translation hypothese and apply joint inference on it. The result is shown below:\n\n| Model                  | Acc.  |\n| ---------------------- | ----- |\n| AIOE+MT                | 54.20 |\n| AIOE+CMT（Context MT） | 56.01 |\n| AIOE+Google Translator | 56.27 |\n| AIOE+Joint Training    | 59.75 |\n\nAs we can see, the performance of the joint inference method is influenced by the quality of MT model. Using a better MT model  (Google Translator here) can bring better performance than using our NMT systems or Context MT system, but the performance still lacks behind the joint training method. This results highlight the importance of incorporating translation signal into the WLAC model.\n\n**Q2 Are you proposing a new metric?**\n\nNo, the proposed criterion \"agreement\" is *not* a metric for the WLAC task, because \"agreement\" doesn't access the labels of WLAC task, it just computes how many predictions exist in the translation hypotheses.  The criterion \"agreement\" reveals the relations between the WLAC task and the MT task, which serves as a preliminary analysis that motivates the subsequent model improvements.\n\n**Q3 The relationship between \"accuracy\" and \"agreement\"**\n\nYes, your understanding of \"accuracy\" and \"agreement\" is correct. We admit that these two concepts are not well described in the current version of paper, so we will further clarify them in the next version. Thank you for pointing out this problem.\n\n**Typos Grammar Style And Presentation Improvements**\n\nThanks for your detailed comments, we will fix the typos and improve the presentation according to your suggestions. We will re-organize the Table 2 to include more information. We will also update the Figure 4 to make it less confusing in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposal is very well-motivated by the preliminary analysis.\n2. Experiment results are strong.\n3. The error analysis and subsequent discussion give good insights into where the model improves and where it still falls short.", "reasons_to_reject": "My main complaint with this paper is its presentation quality. A lot of concepts could be described more clearly to avoid confusing the readers, especially those who are not familiar with the WLAC task. I will elaborate on my feedback in the presentation improvement box.\n\nIn my opinion, this is one of these cases where the \"required edit\" option in journals would have been useful but is not an option for EMNLP. I think the content is good enough so I would rather not block this paper for this reason but I hope the authors do address my comments before camera-ready.", "questions_to_authors": "Out of curiosity: Do you have an idea how much MT quality affects the performance of the AIOE+MT model? For example if you use Google Translate or GPT-4, can you do better than AIOE-Joint?", "paper_topic_and_main_contributions": "This paper addresses the problem of word-level auto-completion (WLAC), where the model tries to complete a partially typed word sequence $\\mathbf{t}$ into a full word $w$, given the source sentence $\\mathbf{s}$ and target context $\\mathbf{c}$.\n\nThe first main contribution of the paper is a preliminary analysis that shows that MT output is also a reasonable proxy to estimate WLAC models' performance, and existing models (AIOE) don't work well when evaluated with this proxy. This paves the way for the other contribution, which are two different improvements: (1) at inference time, use MT output to help select a good completion among the top-k list predicted by the model; (2) jointly train the model with MT capabilities, so the model can predict something that will likely be in the MT output.\n\nExperiments show that the proposed method is very effective -- on the WMT22 WLAC benchmark, it beats the SToA model with only 15% of its parameter size.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I think I had two big confusions when I read the paper that took me a while to recover from them:\n\n- You are not proposing a new metric (which I misunderstood from the word choice \"criterion\"), but Section 3 mainly serves as a preliminary analysis that motivates the subsequent model improvements. If you are indeed proposing a metric, you will need to do a very different set of evaluations here.\n- The notion of \"accuracy\" is what you compute when comparing with the ground truth, as in the test set, while \"agreement\" is checking if the word exists in the translated sentence. This is used extensively throughout the paper, so I think it's worth to be explained more explicitly.\n\nOther than that, a few more small things:\n\n- L56: I would say during \"inference\" instead of during \"testing\" here\n- L120: typed *character* sequence $\\mathbf{t}$\n- L225-234: I would add how much ground truth agrees & disagrees with MT output for comparison in Table 2, in order to strengthen your argument there\n- L257: This is talking about WLAC *gold* labels, right? If yes this is what you want to include in Table 2.\n- L483: AIOR -> AIOE\n- Figure 4: I find the lines confusing. Probably remove them and use some larger markers in the figure (e.g. cross or triangle)."}}, {"review_id": "weZj1wBeGj", "reviewer": "Reviewer_Nd7e", "summary": "", "questions": "Line 537-539: \"We attribute the translation fails in these cases to the lack of sufficient context.\"\nTranslation error is common since the MT model can not be perfect, as also mentioned in Limitations. Hence it is suggested to provide evaluation metrics, such as BLEU, of the Transformer-based MT model on the four translation directions in Table 3. This will help readers better understand the basic translation abilities of the jointly trained MT model.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments. We will release our reproducible code along with the next version of paper. We address your comments below.\n\n**Q1 Provide evaluation metrics, such as BLEU, of the Transformer-based MT model on the four translation directions in Table 3**\n\nThank you for point out this problem. We understand that a translation metric such as BLEU can provide better insights about how the MT model fails. However, the test set provided by WMT WLAC shared task does not provide a golden translation, so we can't compute the BLEU score on it. We will try to report the BLEU results on our synthetic dataset in the next version of paper.\n\n**Typos Grammar Style And Presentation Improvements**\n\nThank you for the detailed suggestions. We will adjust the location of tables and improve figure quality for better presentation. The temperature of GPT-text-davinci-003 is 0, we will also provide it in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes an effective method to handle WLAC tasks with the help of MT models. The proposed method performs effectively on the WMT22 WLAC test set, surpassing previous SOTA baseline models.\n2. The intuition and motivation, method introduction, experimental setups, and detailed result analysis are well elaborated, making the work sound and reproducible.", "reasons_to_reject": "Code releasing is encouraged for better reproducibility.", "questions_to_authors": "Line 537-539: \"We attribute the translation fails in these cases to the lack of sufficient context.\"\nTranslation error is common since the MT model can not be perfect, as also mentioned in Limitations. Hence it is suggested to provide evaluation metrics, such as BLEU, of the Transformer-based MT model on the four translation directions in Table 3. This will help readers better understand the basic translation abilities of the jointly trained MT model.", "paper_topic_and_main_contributions": "By rethinking the definition of a good WLAC prediction, this paper introduces a criterion that a better prediction is more likely to be contained in a high-quality translation. Then this paper proposes to jointly train WLAC and MT models to encourage agreement between the predicted words of WLAC and the translation from MT models. AIOE and its variants outperform previous SOTA baselines on the WMT22 WLAC test set, demonstrating the effectiveness of the proposed method. The intuition and motivation, method introduction, experimental setups, and detailed result analysis are well elaborated, making the work sound and reproducible.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Presentation Improvements:\n1. Table 4 appears in Section 5.3, which precedes Table 3 in Section 5.4. It would be better to adjust the table format and reorder them.\n2. Line 411: A citation or URL link is needed when mentioning \"GPT-text-davinci-003\". Besides, the generation temperature of the GPT model should be provided.\n3. Figure 3: The font style and size should be improved for better presentation."}}]

Author rebuttal:

Meta-review: All the reviewers agree that the paper presents a well motivated approach, which is is also evaluated in a satisfactory manner and provides strong results. Not only did all reviewers agree on the soundness of the work, but also provided high excitement scores. This paper is an easy recommendation for the main conference.

However I would like to highly encourage the authors to pay attention to the recommendations of the reviewers to improve the presentation of the paper, as well as to the questions for clarification. Specifically, reviewer 4EMT pointed out that the paper may be hard to understand for people not completely familiar with WLAC. Please take the advice of all the reviewers into account when preparing the final submission.
Decision: Accept-Main