INarIG: Iterative Non-autoregressive Instruct Generation Model For Word-Level Auto Completion
Authors: Hengchao Shang
Zongyao Li
Daimeng Wei
Jiaxin GUO
Minghan Wang
Xiaoyu Chen
lizhi Lei
Hao Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: Word-Level Auto Completion
Computer-Aided Translation

Abstract: Computer-aided translation (CAT) aims to enhance human translation efficiency and is still important in scenarios where machine translation cannot meet quality requirements. One fundamental task within this field is Word-Level Auto Completion (WLAC). WLAC predicts a target word given a source sentence, translation context, and a human typed character sequence. Previous works either employ word classification models to exploit contextual information from both sides of the target word or directly disregarded the dependencies from the right-side context. Furthermore, the key information, i.e. human typed sequences, is only used as prefix constraints in the decoding module. In this paper, we propose the INarIG (Iterative Non-autoregressive Instruct Generation) model, which constructs the human typed sequence into Instruction Unit and employs iterative decoding with subwords to fully utilize input information given in the task. Our model is more competent in dealing with low-frequency words (core scenario of this task), and achieves state-of-the-art results on the WMT22 and benchmark datasets, with a maximum increase of over 10\% prediction accuracy.

INarIG: Iterative Non-autoregressive Instruct Generation Model For Word-Level Auto Completion
Hengchao Shang, Zongyao Li, Daimeng Wei, Jiaxin Guo, Minghan Wang, Xiaoyu Chen, Lizhi Lei, Hao Yang Huawei Translation Service Center, Beijing, China (shanghengchao, httpechho,,,,ooogg lizongyao, wwwdaimeng, ien wangminghan, chenxiaoyu35, lei 1 Leilizhi, yanghao30 www.://iiewwwww..eecmmoo com
Abstract
Computer-aided translation (CAT) aims to en- hance human translation efficiency and is still important in scenarios where machine transla- tion cannot meet quality requirements. One fun- damental task within this field is Word-Level Auto Completion (WLAC). WLAC predicts target word given a source sentence, translation context, and human typed character sequence. Previous works either employ word classifica- tion models to exploit contextual information from both sides of the target word or directly disregarded the dependencies from the right- side context. Furthermore, the key information, i.e. human typed sequences, is only used as prefix constraints in the decoding module. In this paper, propose the INarIG (Iterative Non-autoregressive Instruct Generation) model, which constructs human typed sequence into Instruction Unit and employs iterative de- coding with subwords to fully utilize input in- formation given in the task. Our model more competent in dealing with low-frequency words (core scenario of this task), and achieves state- of-the-art results the WMT22 and bench- mark datasets, with a maximum increase of over 10% prediction accuracy.
Sp
We asked to ap
sp their opinions
We ap opinions
spccilless sperifi apiil
specillass specific
spccilias spoiffe
secilles seetffc qnii
qxki
Translation
We asked specialists for their opinions.
Source
Wir haben Meinung von Fachirzten cingcholt.
Figure I: Illustration WLAC task. e: = (4t Gy)) is the translation context for source sentence human typed sequence. Words in the rounded rectangles are predictions generated by the model given the tuple (zz, s). As ct, Gr can be empty, the task has four types of context data: (a) zero-context, (b) prefix, (c) suffix, and (d) bi-context.
is the
search on computer-aided translation (CAT) (Bar- rachina al., 2009; Green et al., 2014a; Knowles and Koehn, 2016; Santy et al., 2019) aims at en- hancing manual translation. Among these works, sentence/word-level autocompletion (Knowles and Koehn, 2016; Zhao et al., 2020; Li et al., 2021) is fundamental task and has been widely applied to Post Editing (PE) and Interactive Translation Prediction (ITP) (Knowles and Koehn, 2016).
Li et al. (2021) offer a more general definition of Word-Level AutoCompletion (WLAC) (Figure 1): predicting target word w given the source sentence translation context (left and right to the target word), and human typed sequence (one or several character(s) of the target word). And they open-source the first benchmark system with training datasets and baseline results.
1 Introduction
Transformer and its variants (Vaswani al., 2017; Shaw al., 2018; So al., 2019; Dehghani al., 2018), coupled with large-scale parallel and syn- thetic corpora (Sennrich et al., 2016; Edunov et al., 2018; Wu et al., 2019; Pham et al., 2020), have boosted machine translation quality to large ex- tent. Now, Machine Translation (MT) can basi- cally meet users' requirements in scenarios where translation quality is undemanding, for instance, understanding social media posts. However, in other scenarios where audience expect highly ac- curate translations (e.g. reading product man- ual or a government document), manual proof- reading and post-editing are still required. Re-
According to the definition, the task faces two challenges: First, the target word may have con- textual dependencies both left and right sides. Consequently, during decoding, it imperative consider information from both sides. Second, the human typed sequence merely a prefix sequence of the target word with an uncertain length. Thus, how to fully utilize this information is key to the

---

task.
Constructing the human typed sequence into Instruction Unit and performing character- level embedding to ensure deep information fusion at the encoding phase.
In prior works (Navarro et al., 2022; Moslem al., 2022), dependencies from the right side disregarded during decoding, and only informa- tion from the left side utilized. Additionally, the human typed sequence is merely used as a prefix constraint in the decoding module. As result, it is difficcult to achieve further performance en- hancement based on these methods. Some other works (Li et al., 2021; Yang et al., 2022) use word- level models to perform word classification. These methods are capable of integrating information de- pendencies from both sides, but word-level models are incapable of predicting low-frequency and out- voccauablyyyy (OOV) words, thus cannot satisfy translators' major needs as they type low-frequency words the most (Casacuberta et al., 2022).
Iterative decoding subword level ensures compatibility with context dependencies on both sides, making the model more competent handling wwwfrrquencyy words.
Utilizing pre-training and multi-task learning strategies efficiently address incomplete translation context.
Background
2.1 Task Definition
According to Li et al. (2021), the WLAC task il- lustration is shown Figure 1, and a detailed defi- nition is as follows: Suppose r = (21, ..... Tm) is source sequence and = ($1, $2,..., sk) is sequence of human typed characters. The trans- lation context is denoted
In this paper, we propose a new Iterative Non- autoregressive Instruct Generation (INarIG) model for WLAC. First, we construct the human typed se- quence s into an Instruction Unit to instruct model generation and encode it together with the transla- tion context to fully leverage available informa- tion. As human typed sequence is merely one or several characters, we perform character-level embedding. Second, we use conditional masked decoding, which similar to Non-autoregressive Translation (NAT) model, to ensure compatibility with dependencies from both side of the decoding anchor (target word). This decoding strategy en- ables model process four types of context. Moreover, we employ iterative decoding of sub- words form the target word, which makes the model more friendly to low-frequency words and thus more appealing to translators.
(c), where C (c,1, 912 G,i), Gr (G,,1, Cr.21 Cr.)). The translation pieces q and c, are on the left and right hand side of s, respectively. WLAC aims predict a target word w, which starts with and is to be placed in the middle of a and G, to constitute a complete translation. Note that w is not necessary consecutive to C₁ Gr,1- More generally, Cr, can be empty, which lead four types of context: zero-context (no context), suffix (only right con- text), prefix (only left context), and bi-context (left and right context).
2.2 Main Challenges
As a new task, WLAC presents unique challenges for modeling both its input and constraints. These challenges include:
Another major challenge in this task, which was ignored previous research, the incom- plete translation text. As e, and a only provide fragmented information, modeling the target lan- guage is challenging, which in turn affects the model performance. We propose two strategies enhance the model's language modelling perfor- mance: (1) fine-tuning on pre-trained MT model or Conditional Masked Language Model (CMLM) (Ghazvininejad et al., 2019); and (2) multi-task learning with the CMLM task.
The human typed sequence contains prefix information of the target word, but its length is uncertain, which poses a challenge for its utilization.
The target word may have contextual depen- dencies to both the left and right sides. As re- sult, the decoding module must be adapted to simultancously utilize information from both sides.
Our models have achieved state-of-the-art per- formance, with prediction accuracies that exceed the benchmarks by an average of 7.76%, and a maximum increase of over 10%.
Translation context (c, and q) only infor- mation segments and may not be contiguous with the target word. Incomplete information on the translation side poses a significant chal- lenge for modeling the target language.
Our main contributions include:

---

opinoos
CTE6E stterionn
Insrrctioon Constrccing
[505] [TPPss MASKI opiiioas (308)
salf stterriurs
X/V
EIONS
INarG decoding: Der
[SO5] we [TPP) [SE2] [MASK] opaniara [EOS]
[00]]
IMASK
PUSS
DNrrlG decodings for
[508] we (TT)) sp (SEP) gpt ialats (MASK]. opptioon [D08]
Hos
Bulh
INatki decoding: Sa
F(Sc8]
FLve
FrEOSt
[808] _ee [TTP] sp (SFP) sper iilss (KOW) MASS]] oppiiions (BO6)
Enk
Tare
[S0S]
opiriona
[BOS]
dereciion and Deviken
specialios
Usi.
Figure 2: The figure on the left illustrates the decoder structure of our model. The human typed sequence "sp" is constructed Instruction Unit with character-level embedding. On the right, we provide an example of our iterative decoding process. (TIP) and (SEP) special tokens used in Instruction Unit. The [SEP] token is only present when decoded subwords exist. The [EOW] token is used to mark the end of a word.
In languages such Chinese where the writ- ing and phonetic symbols inconsistent, the translation-side language model should be able simultaneously process Chinese char- acters, Pinyin, and their relationship.
adopt the most similar approach ours: subword-level model for autoregressive decoding. The human typed sequence is used soft con- straint and the target word is generated by the decoder. However, they do not particularly ad- dress the incomplete translation, which hinders the model performance.
2.3 Related Work
Prior to general definition of this task provided by Li al. (2021), related researches include in- teractive MT (IMT) (Green al. (2014b). Santy et al. (2019)) or translation prediction (TP) (Koehn (2009), Alabau et al. (2014), Kochn et al. (2014), Green et al. (2014b), Huang et al. (2015), Knowles and Koehn (2016) and Coppers et al. (2018)), Langlais et al. (2000), Santy et al. (2019), Lee et al. (2021). Theses works focus on predicting the next word/phrase given the left-side translation context. The settings of the task share great similarity with WLAC, so these works have referential importance to WLAC.
Methodology
In this section, first present the overall structure of our model. Then, introduce the Instruction Unit used, well as our decoding procedure. Figure 2 shows the details.
3.1 Conditional Masked Decoding
Our model's overall structure is Conditional Masked Language Model (Ghazvininejad al., 2019), which is similar to that the benchmark system (Li et al., 2021). The source-side informa- tion is encoded by the encoder, and its represen- tation is passed to the decoder by cross attention. A MASK] token serves as decoding anchor and translation context (cr. cς) is integrated as inputs to be passed to the decoder. When decoding with [MASK] token, the model can utilize contextual information from both sides and also allows us process four types of context using a single model. We denote Dulac Dzero U D pre U D U D as the training data for WLAC, and try to minimize the following objective:
Navarro et al. (2022) and Moslem et al. (2022) adapt the strategy mentioned above to this task. Un- fortunately, the models do not fully fit the WLAC task setting. According to their model designs, some inputs (s, c,) or constraints (w is not neces- sary consecutive to è or (4) are not leveraged.
Li et al. (2021) and Yang et al. (2022) treat WLAC as a word classification task using word- level model. The human typed sequence is used vocabulary constraint during decoding. Such models are well-suited for the WLAC task setting, but does not make full use of human typed se- quence and not very friendly to low-frequency and out-of vocabulary words. Ailem et al. (2022)
L(Dwlaod 0)
log P(w]z, 8;
(1)

---

3.2 Instruction Unit
Human typed sequence, as the prefix of the tar- get word, is key to the WLAC task. In order to simplify the model and ensure maximum utiliza- tion of information, we choose to directly encode it into the model. For this purpose, we construct the sequence into an Instruction Unit using a spe- cial token [TIP] and integrate directly into the decoder's 's input together with translation context, Additionally, the human typed sequence is merely prefix sequence of uncertain length character- level embedding is used. Figure 2 shows the details. section 6.2, we demonstrate that the Instruction Unit can guide the model to generate the correct target word.
2 Ine
xih
3 #o
Original: & 1F
+ 2
Code-Switch: &
AF xih i an
+
80
Random Mask: # # x monn an +p
45
28
Figure 3: The upper part shows an example of Chi- nese translation. The Chinese words need predicted with the human typed sequence "xih", which is prefix of the corresponding Pinyin "xihuan". The lower part demonstrates Code-Switch and Random Mask for Chi- nese sentence.
the same data structure. The overall optimization objective is:
3.3 Iterative Decoding with Subword
L(0) E
log (,
To better adapt to the task setting, we choose to subword the encoding unit. Consequently, in order predict a complete target word, need generate of subwords. Inspired by the non-autoregresssive model, we introduce iterative decoding, which decodes one subword per iteration. Then the decoded subword also integrated into the Instruction Unit to guide the generation of the next subword. This process repeats until the final word is formed. The decoding process is carried out from left right, the human typed sequence serves the prefix of the target word.
(2)
Σ
log P(wmaskl a :)
(I,,e' Deantin
Note that the Dentm is the CMLM data generated from bilingual data. é is the complete translation context in CMLM data.
4.2 Code-Switching for Chinese
Another typical issue regarding Chinese is the version between Chinese characters and Pinyin (phonetic symbols). Most people need input the Pinyin first and then use an input method editor to convert into Chinese words, see the upper part of Figure Since Pinyin is not used pre-training or multi-task learning, the model has no knowledge of the structure of Pinyin and the mapping between Chinese words and Pinyin, which poses another challenge to this task.
An example is shown in the right part of Figure 2. The [[OW] token is used to mark the end of a word, and the (SEP) token serves as separator between the human type sequence and decoded subword tokens.
4 Training Strategy
To address this issue, we refer to the code- switching strategy and convert portion of Chinese words into Pinyin sequence when constructing the CMLM training data, thus the model is able to learn the mapping between Pinyin and Chinese charac- ters. We also mask character(s) in Pinyin sequence with certain probability, allowing the model to learn the inner structure of Pinyin. A case is shown in the lower part of Figure
4.1 Pre-training & Multi-Task learning
As mentioned in section 2.2, and e, are randomly simulated incomplete translation pieces, posing great challenges train the target-side language model (LM)). To address this, propose the fol- lowing two strategies.
First, inheriting the target-side LM ability from a pre-trained model. Related pre-training tasks include MT and CMLM.
Another solution multi-task joint training. By incorporating an additional task, can ensure that the model can maintain language learning capabil- ity throughout training. CMLM can be an addi- tional task trained together with WLAC as it uses
5
Experiment
5.1 Datasets
To
validate our model's effectiveness, we per- form experiments on two datasets: one from

---

WMT22 and the other from the Benchmark sys- tem. Each dataset includes data of two language pairs (Chinese-English and English-Gernane both directions. Regarding Chinese++ English data, the benchmark dataset contains 1 1.23M bilin- gual sentence pairs from LDC corpora while the WMT22 dataset contains 15M pairs from UN Par- allel Corpus V1.0². pypinyin' is used to con- vert Chinese word phonetic symbols. For the English++German tasks, the two datasets use the same data: 4.5M from WMT14 and is pre- processed by Stanford'. We first use the data gen- eration script" provided by the WMT22 task to construct training data Ditac from bilingual pairs. Both datasets contain their corresponding test sets. We adopt sentencepiece (Kudo and Richardson, 2018) for subword modeling and set the vocabu- lary size 32K.
data 1:1, and the probability of mask token in CMLM data is 20%.
on
Code-switching for the EN=ZH model is per- formed during the multi-task training phase. We randomly select 50% of the Damim data for code switching. Each Chinese token has a 30% chance of being selected and converted to Pinyin charac- ters.
5,4 Inference & Evaluation
During inference, we average the last 10 check- points to obtain the final model and report results from a single model without ensemble. We adopt beam search during decoding and the beam size is set to 4 for models of all language pairs.
We use accuracy the evaluation metric (Li et al., 2021):
Nmatch
5.2 Model Configuration
ACC
Naul
We adopt the Transformer-base configuration for the purpose of fair comparison. The batch size is set to 64K tokens with a learning rate of 5e-4 and a 4000 step warmup. Other parameters are set to the default values of fairseq (Ott et al., 2019) All of our experiments are performed on NVIDIA 8*V 100 and we save model every 2K updates.
where Nmatch is the number of correctly predicted words and Nat is the number of all test examples.
Results
6.1 Main Results
We first compare method with other related works the WMT22 test sets in Table The results demonstrate that our model achieves higher accuracy (an average increase of 4.8% and a max- imum increase of 7.58%) compared with the best results from other works, resulting in a new state- of-thc-art performance. Additionally, we also com- pare our model with the benchmark system (Li et al., 2021) the benchmark test sets. Results are shown in Table 2. And the results remain consistent with that of WMT22 dataset, with an average accu- racy improvement of over 7% and a maximum im- provement of over 10%. The results fully validate the effectiveness and robustness of our method.
5.3 Training Process
We train a single model process all types of translation context. The model leverages the joint training strategy mentioned in $3.1. During train- ing, we divide the training process into three phases according to the two strategies presented in 4.1:
1) Generate the final training data Dirain. from
D Dwlae according to 3.3.
2) Pre-train an MT or CMLM model with the original bilingual pairs.
3) Finetune it with Demtm and Dirain using
multi-task joint training.
Models are measured by loss on dev sets. Partic- ularly, for CMLM task pre-training, we set the prob- ability of mask token to 15%-50% to ensure the model learn sadget-side LM. In the multi-task learn- D Dutac
6.2 Effectiveness of Instruction Unit
One of our primary contributions is constructing the human typed sequence into character-level In- struction Unit, which efficiently encodes prefix con- straint information into the model to guide word generation. To demonstrate its effectiveness, we conduct comparative experiment by replacing the Instruction Unit with vocabulary filter module similar to the one used in the benchmark system. Results are presented in Table
ing phase,
the ratio of Demtm and
hhtts://wwwsstammm http://wwwwtt..r///e.wtaooo www.s/tttattrrgg//ee///oooooo autocompletion.he
"https//cooffereence unite ong////uncosus
"htps://gittub.. cttttttuubooo////////eee wom/mozzilaaz///aaoooo wht.:///ythhooiiiiiiii
https://nlp.stanfrr...m stanford.edu/prooooo...oo /w//////cccts///mm/////////////
com/temaolin/
/tttps://gttubb..o

---

Models
ZH=
EN=ZH
DE=EN
DN=DE
Li et al. (2021) (our reproduction)
52.06
50.86
60.13
56.18
Moslem et al. (2022)
50.41
31.94
61.44
58.94
Ailem et al. (2022)
57.36
48,97
Yang et al. (2022)
54.05
53.98
57.27
41.83
Navarro et al. (2022)
39.02
33.97
INarlG (Ours)
59.10
56.23
69.02
63.52
Table 1: Experiment results on WMT22 dataset and "_" means not provided. The results of benchmark system (Li et al., 2021) are reproduced by us.
ZH-EEN
EN-
DE~EN
EN+DE
Models
NISTOS
NIST06
NISTO5
NIST06
NT13
NT14
NT13
NT14
Li et al. (2021)
55.54
55.85
53.64
54.25
57.84
56.75
56.91
52.68
INrrG (Ours)
62.50
64.16
59.31
60.76
65.21
65.76
61.22
62.76
Table 2: Experiment results on benchmark dataset. NISTOS and NIST06 are test sets used in the benchmark system for Chinese-English (both directions), and test sets for Fermann-Englise tasks are newstest13 and newstest 14.
ZH
DE~EN
Models
NISTO5
NIST06
NT13
NT14
INarIG
59.31
60.76
65.21
65.76
<ip
w/o IU
53.60
53.85
58.64
57.71
w/o Iter-D
55.61
56.82
60.45
61.22
Table 3: Results of the comparative experiments based on benchmark dataset. "w/o IU" means replace Instruc- tion Unit with vocabulary filter module. "w/o Iter-D" means replace subword based iterative decoding with word-level model.
<lip
posh
the
According to the results, Instruction Unit achieves consistent enhancement across various language pairs. Particularly, in terms of accuracy, we observe 6.6% to 8% increase on DE=EN and 5.7% 6.9% increase on EN=ZH. The results indicate that model-based deep information fusion is much more effective than shallow fusion in the decoding module.
<tipo
Kaep>
people
the
Figure 4: A case study of cross attention-based word alignment for [MASK] token with source sentence.
6.3 Effectiveness of Iterative Decoding
Anlysis
To improve performance with low-frequency words, we utilize a subword-level model and an iterative decoding strategy. To verify the effective- ness of this strategy, we train a word-level model to perform word classification and use it for com- parison. We compare the accuracies benchmark set, as shown in Table 3. The accuracies of our models outperform the word-level model, with an average accuracy improvement of around 4.2%, verifying the effectiveness of our decoding strategy.
In this chapter, analyzes the effectiveness of our innovative points and optimization strategies used in training to better validate our method. Unless otherwise specified, the experiments are conducted on the benchmark dataset.
7.1 Instruct Generation
We analyze the Instruct Generation method used in our model from the perspective of word align-

---

EN=ZH
DE=EN
Mudil
Models
NISTO5
NIST06
NT13
NT14
Raw-Model
50.69
52.34
53.21
51.82
70
+NMT-FT
56.24
58.33
63.66
64.12
65-
+CMLM-FT
56.44
57.25
63.58
63.80
+Multi- Task
57.28
59.41
64.00
64.75
INarG
59.31
60.76
65.21
65.76
50
Table 4: The results of our experiments different training strategies. Raw-Model refers to the baseline model trained directly on WLAC data. NMT-FT and CMLM-FT uses NMT and CMLMM pre-trained models, respectively, Multi-Tass means multi-task training with both CMLM and WLAC data. The INarIG bination of an NMT pre-trained model and multi-task training.
45
2 8mn
8
100
Figure 5: The accuracies of words at different frequen- cies for DE task. The test sets 1 and newstest14 are merged and divided to bins with equal size according to the frequency of target word. Bin I and Bin 10 denote the most frequent and infrequent bin.
com-
is superior at all frequency intervals, especially for low-frequency words. For 30% less-frequent words, the accuracy improves by more than 8%. The results of the EN=ZH task shown in appendix $A, also exhibit almost the same trend, confirming that our model has an advantage in predicting low- frequency words.
ment. According to the task setting, the translation context is incomplete, making it difficult for the [MASK] token to align with the corresponding in- formation in the source text. The prefix information of the target word in Instruction Unit enhances the representation of the [MASK]. facilitating align- ment and guiding decoding. We use an example to visualize the last cross-attention layer's attention weights, as shown in Figure 4.
7.3 Pre-training & Multi-Task learning
As stated above, we realize that incomplete trans- lation context challenges the model performance. To prove the effectiveness of our pre-training and multi-task strategy, conduct experiments the benchmark dataset, and the results are shown in Table 4.
In this case, the candidate words are "princi- ple" and "people-oriented" with the corresponding source tokens " and "B"." When com- paring the top and middle figures, the addition of prefix "p" increases the attention weights for the correct original token, especially for "[ ad- dition, if subword "people" has already be decoded, the target word should only be "people-centered" As shown in the bottom figure, the model shift the attention to focus on "4" in the source side.
According to the results, the accuracy of our models drops 7% to 13% when no related strategy is applied. To be more specific, we observe pre- training on either NMT or CMLM task achieves equivalent results and multi-task training leads greater improvements than pre-training. The final model which combines the two strategies, obtains further 1.2% improvement on average.
The case confirms that our Instruction Unit can effectively guide the model generate correct words, improving the accuracy of the model.
Furthermore, our analysis of the training loss curve is consistent with the findings. In addition, evaluate the performance of our models on four types of context and observe consistent improve- ments, demonstrating the robustness of our model. Details of this analysis is presented in the appendix $B.
7.2 Accuracy & Word Frequency
As mentioned before, in real-world scenarios, trans- lators have a stronger demand for autocompletions low-frequency words (Casacuberta et al., 2022). In Section 6.3, we have already demonstrated the effectiveness of our subword-level model in terms of overall accuracy. Here, we conduct further anal- ysis on the accuracy of words different word frequencies for additional validation.
7.4 Code-Switching for Chinese
In order to verify the effectiveness of code- switching strategy for Chinese, we conduct exper- iments on EN-ZH using both datasets, and the
The results for DE~EN task are shown in Figure 5. which demonstrate that the subword-level model

---

EN=ZH
7.6 Inference Performance
Models
NIST05
NIST06
WMT22
Theoretically, our model has higher computational complexity compared to word-based classification models. Regarding the iterative decoding process of a non-autoregressivee model, the representations of all tokens on the decoder side need to be com- puted repeatedly. However the impact inference speed not serious in practical applications for two reasons: (1) The task predicts only one word so the sequence length limited. (2) Computing can be processed in parallel, and shallow-decold (Kasai et al., 2020; Wang et al., 2019) Transformer models are widely used.
INarIG
59.31
60.76
56.23
w/o CS
58.02
59.77
55.07
Table Results the code-switching strategy compar- ative experiments for Chinese. "w/o CS" means without code-switch strategy during training.
EN
EN DE
Models
NIST05
NIST06
NT13
NT14
INarIG 59.31
60.76
65.21
65.76
+ BT
60.66
61.83
67.89
68.12 12
+Big
61.56
63.75
69.97
70.80
Discussion & Future Work
Table 6: Results of BT style synthetic data on bench- mark dataset. "+ BT means back translation style synthetic data is added for training, and "+Big" means changing model configuration to transformer Big.
Two issues requires more research and analysis: 1. Our model allows us to apply more main- stream enhancement strategies in the field of nat- ural language processing, such as multilingual en- hancement and word alignment. How to efficiently use these strategies should be further explored.
accuracy results are summarized in Table 5.
2. Our decoding strategy is inspired by non- autoregressive models. More general decoding strategies that integrate autoregressive and non- autoregressive approaches have been studied by Wang et al. (2022); Li al. (2022), and future researches may find better strategies.
The code-switching strategy increase the ac- curacy of the model by an additional 1-2%, which also confirms the necessity of modeling the inter- nal structure of Pinyin and the relationship between pinyin and Chinese words. It is worth noting that Chinese a representative example of this type of language, and the code-switching strategy can be easily extended to similar languages.
8.1 LLM for WLAC
Upon the release of ChatGPT", observe that large language models exhibit exceptional perfor- mance across a range of NLP tasks. To evaluate ChatGPT's capability on WLAC, we extract a sub- set of 1000 instances from the EN=DE newstest test set and construct a corresponding task prompt (provided in the appendix $C).
7.5 Enhancement with Monolingual data
Due to the high degree of similarity between our model and the standard MT models, we are able to transfer existing optimization strategies from MT tasks. Back translation (Sennrich et al., 2016; Edunov et al., 2018; Wu et al., 2019) is the sim- plest and most effective one. To confirm that back translation can be applied to our model, conduct additional experiment by adding 10M Chinese and 20M English monolingual data on top of the Benchmark dataset. The beam-search back trans- lation (Sennrich et al., 2016) is used to construct synthetic data. The results are shown in Table
The accuracy on our test set is 44.2%. Chat- GPT's performance the WLAC task is inferior to method (63.5% in terms of accuracy). Im- proving prompts may improve its efficacy. Fur- thermore, large language models suffer from slow inference speeds, and the word completion task re- quires minimal latency to ensure seamless input. Overall, further exploration and adaptation may be needed for the application of large models to the WLAC task.
Overall, the BT-style synthetic data improve the accuracy of the model, and increasing the model capacity can further enhance the model perfor- mance. This demonstrates that our model can effec- tively adopt optimization strategies from other NLP tasks, such as MT, and it is worth further research.
Conclusion
In this paper, propose novel iterative non- autoregressive instruct generation model for the
"https//chat.ppemai.co/

---

WLAC task, and validate the effectiveness of our model on two mainstream datasets. Subsequently, through carefully-designed experiments, we ver- ify that our model's ' better capability of utilizing avaialbe information compared to previous works, as our approach uses model-based deep informa- tion fusion. We also demonstrate that our iterative decoding based on subwords can improve the accu- racy of low-frequency words and better meet trans- lators' requirements. Afterwards, through com- parative experiments, we confirm that a series of training strategies used in our work help improve accuracy.
task in wmt 2022. In Proceedings of the Seventh Conference on Machine Translation, pages 812-820, Abu Dhabi. Association for Computational Linguis- tics.
Sven Coppers, Jan Van den Bergh, Kris Luyten, Karin Coninx, Tulianna Van der Lek-Ciudin, Tom Vanalle- meersch, and Vincent Vandeghinste. 2018. Intellingo: An intelligible translation environment. In Proceed- ings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1-13.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2018. Universal transformers. In International Conference on Learn- ing Representations.
Sergey Edunov. Myle Ott, Michael Auli, and David Grangier. 2018. Understanding tack-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
10 Limitations
The iterative NAT decoding strategy we used can decode one or several tokens in a single iteration, depending on the number of masks being input. We choose to decode one token per iteration. Further research should be done to explore the effective- ness of decoding more than one tokens per iteration. be performed from
Marjan Ghazvininejad. Omer Levy. Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel de- coding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 61 12-6121.
In addition, NAT decoding left right, right to left, or randomly. We employ the most straightforward approach: left to right. Further research is required to analyze the perfor- mance of other decoding directions.
Spence Green, Sida T Wang, Jason Chuang, Jeffrey Heer, Sebastian Schuster, and Christopher D Man- ning. 2014a. Human effort and machine learnability in computer aided translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 0225-1234
References
Melissa Ailem. Jingshu Liu, Jean-Gabrie Barthelemy, and Raheel Qader. 2022. Lingua custodia's partici- pation at the wmt 2022 word-level completinn shared task. In Proceedings of the Seventh Confer- ence on Machine Translation, pages 1170-1175, Abu Dhabi, Association for Computational Linguistics.
Spence Green, Sida I. Wang, Jason Chuang. Jeffrey Heer, Sebastian Schuster, and Christopher D. Man- ning. 2014b, Human effort and machine learnability in computer aided translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1225-1236. Doha, Qatar. Association for Computational Linguis- tics.
Vicente Alabau, Christian Buck, Michael Carl, Fran- cisco Casacuberta, Mercedes Garefa-Martinez, UI- rich Germann, Jesús González-Rubio, Robin I.. Hill, Philipp Koehn, Luis A. Leiva, Bartolomé Mesa-Lao, Daniel Ortiz-Martine Sanchis- Triiles, and Chara Tsoukala. 2014. Cas- macat: A computer- assisted translation workbench. In Conference of the European Chapter of the Asso- ciation for Computational Linguistics.
Guoping Huang. Jiajun Zhang, Yu Zhou, and Chengging Zong. 2015. A new input method for human transla- tors: integrating machine translation effectively and imperceptibly. In Twenty- Fourth International Joint Conference on Artificial Intelligence.
Herve Saint-Amand, Germán
Jungo Kasai, Nikolaos Pappas. Hao Peng, James Cross. and Noah Smith. 2020. Deep encoder, shallow decoder: Reevaluating noo-euvtreeresssvve machine translation. In International Conference on Learning Representations.
Sergio Barrachina, Oliver Bender, Francisco Casacu- berta, Jorge Civera, Elsa Cubel, Shahram Khadivi, Antonio Lagarda, Hermann Ney, Jesús Tomás, En- rique Vidal, et al. 2009. Statistical approaches to computer-assisted translation. Computational Lin- guistics, 3-122
Rebecca Knowles and Philipp Koehn. 2016. Neural interactive translation prediction. In AMTA (!), pages 107-120.
Francisco Casacuberta, George Foster, Guoping Huang, Philipp Koehn, Geza Kovaes, Lemao Liu, Shum- ing Shi, Taro Watanabe, and Chengging Zong. 2022. Findings of the word-level autocompletion shared
Philipp Koehn. 2009. A process study of computer- aided translation. Machine Translation, 23(4):241- 263.

---

Philipp Kochn, Chara Tsoukala, and Herve Saint- Amand. 2014. Refinements to interactive translation prediction based on search graphs. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 574-578.
Sebastin Santy, Sandipan Dandapat, Monojit Choud- hury, and Kalika Bali. 2019. Inmt: Interactive neural machine translation prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNLP-IJCNIPP): System Demonstrations, pages 103-108.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 86-96, Berlin, Germany, Association for Computational Lin- guistics.
Philippe Langlais, George Foster, and Guy Lapalme. 2000. TyansType: a computer-aided translation typ- ing system. In ANLP-NAACL 2000 Workshop: Em- bedded Machine Translation Systems.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Short Papers), pages 464-468, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Dongjun Lee, Junhyeong Ahn, Heesoo Park, and Jaemin Jo. 2021. IntelliCAT: Intelligent machine translation post-editing with quality estimation and translation suggestion. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Nat- ural Language Processing: System Demonstrations, pages 11-19, Online. Association for Computational Linguistics.
David So, Quoc Le, and Chen Liang. 2019. The evolved transformer. In International Conference on Machine Learning, pages 5877-5886 PMLR.
Huayang Li, Lemao Liu, Guoping Huang, and Shuminng Shi. 2021. Gwlan: General word-level autocomple- tion for computer-aided translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4792-4802.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, Advances in neural information processing systems, 30.
Minghan Wang. Jiaxin Guo, Yuxia Wang. Daimeng Wei, Hengchao Shang, Yinglu Li. Chang Su, Yimeng Chen, Min Zhang, Shimin Tao, et al. 2022. Diformer: Directional transformer for neural machine transla- tion. In Proceedings of the 23rd Annual Conference of the Europea A Association for Machine Translation, pages 81 1--
Pengfei Li, Liangyou Li. Meng Zhang, Minghao Wu. and Qun Liu. 2022. Universal conditional masked language pre-training for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6379-6391.
Yasmin Moslem, Rejwanul Haque, and Andy Way. 2022. Translation word-level auto-completion: What can we achieve out of the box? In Proceedings of the Seventh Conference on Machine Translation, pages 1176-1181, Abu Dhabi. Association for Computa- tional Linguistics.
Qiang Wang, Bei Li. Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics.
Lijun Wu, Yiren Wang. Yingce Xia, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2019. Exploiting monolin- gual data at scale for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP- IJCNLP,, pages 4207- 4216, Hong Kong, China. Association for Computa- tional Linguistics.
Angel Navarro, Miguel Domingo, and Francisco Casacuberta. 2022. Prhit's 's submission to wlac 2022. In Proceedings of the Seventh Conference on Ma- chine Translation, pages 1182-1186, Abu Dhabi. As- sociation for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Cheng Yang, Siheng Li, Chufan Shi, and Yujiu Yang. 2022. ligroup submissions for wmt22 word-level au- tocompletion task. In Proceedings of the Seventh Conference Machine Translation, pages 1187 1191, Abu Dhabi. Association for Computational Linguistics.
Hieu Pham, Xinyi Wang, Yiming Yang, and Graham Neubig. 2020. Meta back-translatton Interna- tional Conference on Learning Representations.

---

Tianxiang Zhao, Lemao Liu, Guoping Huang, Huayang Li, Yingling Liu, Liu GuiQuan, and Shuming Shi. 2020. Balancing quality and human ievolvement: An effective approach to interactive neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9660)
accuracy is basically consistent with the loss perfor- mance, which again verifies our analysis. (2) The accuracy curves under differeet types of context are consistent too, which proves generalizability our optimization strategy. We also perform the same ablation experiment on the EN model, and the results are consistent with those of the DE- model. The detailed results are in Figure 8.
9667.
A Accuracy & Word Frequency for EN=ZH task
C Prompt for WLAC
Accuracy results of words with different frequen- cies for EN task is presented in Figure 6.
The specific prompt structure can found in Fig- ure 9.
Wend-Leve Model Heeraiive Occoding
80
70
50
30
20
10
10
8m
Figure 6: The accuracies of words at different fre- quencies for EN=ZH task. The test sets NISTO5 and NIST06 are merged and divided to bins with equal size according to the frequency of target word. Bin I and Bin 10 denote the most frequent and infrequent bin.
Pre-training vs Multi-Task learning
Taking DE+EN an example, we visualize loss curves of our models on the dev set, as shown in Figure The model without any strategy gets stuck in a local optima in few updates, causing loss increase in subsequent training. After we add CMLM data (complete translations) for multi-task training, the lowest loss drops from 4.2 to 3.5. The loss reduces well for the model using pre- training strategy. However, we observe loss in- crease after 250K update steps, probably due to catastrophic forgetting during fine-tuning. The LM ability learned in the pre- pre-traiming stage deteriorates in the fine-tuning stage, so the lowest loss of the model is still higher than modal using the multi- task training strategy. The combination of NMT-FT and multi-task strategies ensures stable training and leads to the lowest loss.
Finally, we visualize the accuracy of each model under different types of context the right side of Figure 7. Two conclusions can be drawn: (1) The

---

DE2EN's
Accuracies
Rap-Mocel
*MutoTook
MMut--Tas--NTTTTT
55
1626
bl-context
3.6
0K
190K
200K
200K
400K
5006
Baw Rodd
-NKI-21
-Mulb-ask
Mutb Maue-laskeeMM KIMEEPT
Updatte Steps
Thiining strategys
Figure 7: The loss curves and accuracies for DE=EN models with different training strategies, All results are measured on the NT13 dev test.
EN2ZH's
Accuracies data
5.2
5.0
4.8
60.0
4.5
37.5
4.4
4.2
52.5
bloeanttet
4.0
53.0
Rsw-Mode
-NNT-FT
-Muth-Tokk
-Mith BORENN
Update
Training stratogys
Figure 8: The loss curves and accuracies for EN over training strategies. All results are hased on the NIST02 dev test.
QQ: Word-Level Compoomeenio (WLAC), which aims predict target word given source sentence, translation icontext and human typed character sequence, make task more general in real-world scenarios, the translation "context is made that the left context and right context, which be empty. The original text is English, the stranslation is German.
Source sentenee is you can do that, , khan said, the public gets more comfortuble.". lleft context is "damit, so khan, wãre auch".
rright context is ".".
thuman typed character sequence is "be". So what's the word that starts with "be"? (It's best to give the answer without explanation.)
A: The word that starts with "be" is "bereit".
Figure 9: The example of ChatGPT prompt about the WLAC task.


Reviews:
[{"review_id": "T5wT72T4oE", "reviewer": "Reviewer_DbUp", "summary": "", "questions": "1. Regarding the training data analysis without using the code-switching strategy in Section 7.4: Specifically, how are these data composed? To be more precise, what would the user's input sequence look like if this strategy were not employed? Concerning the experimental results in Section 7.4, which indicate that the impact of the code-switching strategy is only 1-2%: Why is this influence so minimal?\n\n2. The right panel in Figure 2 and the [MASK] symbol in Figure 3 are not the same, and there are distinct differences in their functionalities. In Figure 2, [MASK] is used as a separator, and it is not intended for predicting words or characters. However, in Figure 3, [MASK] serves as a placeholder that the model needs to predict, whether it is a word or a character. Is this difference expected?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review.\n\n### For question 1：\n\nRegarding the code-switching strategy used in the EN-ZH model, let us explain it using the example shown in Figure 3 in our paper.  \n\nIn this example, the Chinese words used are \"喜欢\" (like) and \"瓷\" (porcelain). Their corresponding pinyin is \"xihuan\" and \"ci\".  \n\nHere is a table summarizing the different types of data used during model training:\n\n| Data type       | Source                              | Target                                         | mask token |\n| :-------------- | :---------------------------------- | ---------------------------------------------- | ---------- |\n| $D_{bilingual}$ | I like Chinese porcelain very much. | 我 非常  喜欢 中国  的 瓷  器 ！               | -          |\n| $D_{wlac}$      | I like Chinese porcelain very much. | 我   [TIP] x i h [MASK]  瓷  器 ！             | 喜欢       |\n| $D_{cmlm}$      | I like Chinese porcelain very much. | 我 非常  [MASK] 中国  的 [MASK] 器 ！          | 喜欢  瓷   |\n| $D_{cmlm+ cs}$  | I like Chinese porcelain very much. | 我 非常 x i [MASK] u a n 中国  的 [MASK] 器 ！ | h  瓷      |\n\n**Q:**  Regarding the training data analysis without using the code-switching strategy in Section 7.4: Specifically, how are these data composed? \n\n**A:** The code-switching strategy is performed during the multi-task learning stage, when the model needs to learn $D_{wlac}$ and $D_{cmlm}$ (both generated from the original $D_{bilingual}$) simultaneously. \n\nAt this time, the correspondence between pinyin and Chinese word only appears in the $D_{wlac}$ , and the pinyin characters \"xih\" entered by the user are only a prefix of the complete pinyin string\"xihuan\". \n\nThe code-switching strategy converts Chinese words in the $D_{cmlm}$ into corresponding pinyin with a certain probability, and the pinyin characters also participate in random masking. Then we get a new dataset denoted as $D_{cmlm+ cs}$ for training.  $D_{cmlm+ cs}$ allows the model to learn the pinyin structure and correspondence between pinyin and Chinese words too.\n\n**Q:** To be more precise, what would the user's input sequence look like if this strategy were not employed?\n\n**A:**  From the previous examples, we can see that whether the code-switching strategy is used or not, the user's input is a prefix (e.g. \"xi\", \"xih\") of the pinyin string (\"xihuan\") corresponding to the target Chinese word (\"喜欢\").\n\nDetails for this example are shown in the table below:\n\n| Information type    | Information content                             |\n| ------------------- | ----------------------------------------------- |\n| Source              | I like Chinese porcelain very much.             |\n| Translation context | left context: 我         right context: 瓷 器 ! |\n| Human-input         | xih                                             |\n| Decoder input       | 我 [TIP] x i h [MASK] 瓷 器 !                   |\n| Target word         | 喜欢                                            |\n\n**Q:** Concerning the experimental results in Section 7.4, which indicate that the impact of the code-switching strategy is only 1-2%: Why is this influence so minimal?\n\n**A:**  Regarding the effect of code-switching strategy, we speculate the reason is: When there is sufficient simulated $D_{wlac}$ generated, the model has already established a strong baseline, so the supplementary pinyin information in $D_{cmlm+ cs}$  only lead to minor improvement. This is worth further study.\n\n### For question 2：\n\n**Q:** The right panel in Figure 2 and the [MASK] symbol in Figure 3 are not the same, and there are distinct differences in their functionalities. In Figure 2, [MASK] is used as a separator, and it is not intended for predicting words or characters. However, in Figure 3, [MASK] serves as a placeholder that the model needs to predict, whether it is a word or a character. Is this difference expected?\n\n**A:** Regarding the role of [MASK], our paper might not explain it clearly enough. The [MASK] in both Figure 2 and Figure 3 serves as a placeholder for decoding.\n\nFirst, the right side of Figure 2 describes the iterative decoding process of the model, where each round of iterative decoding is based on the [MASK]. Figure 3 mainly shows the process of constructing training data for the CMLM task during multi-task training. Here [MASK] also serves as a placeholder used to predict the target word.\n\n### About decoding speed:\n\nOur model has higher computational cost during inference compared to word-based classification models, because the representations in the decoder need to be recomputed at each iteration. However, since the current task only predicts one word, the number of iterations is limited. Moreover, computations at the decoder side during inference can be fully parallelized, so the overall inference latency does not have a significant increase. In our experiment, the latency to predict one word is within 50ms with a Tesla T4 GPU powered by pytorch. Further optimizations (adapting acceleration libraries: FasterTransformer [1] or LightSeq [2]) can reduce the inference latency to within 15ms. Even for online systems, this level of performance is sufficient.\n\n[1]  https://github.com/NVIDIA/FasterTransformer\n\n[2] Wang et al., 2021 LightSeq: A high performance inference library for transformers. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Propose INarIG to fully utilize the human-typed sequences in the WLAC task.\n\n2. The model achieves new state-of-the-art results on two standard datasets, demonstrating its effectiveness. \n", "reasons_to_reject": "1. Limited novelty. Several model components like the NAT decoder and code-switching have been applied in other contexts before. The innovations seem more incremental to existing ideas.\n\n2. The results regarding the decoding speed are missing.", "questions_to_authors": "1. Regarding the training data analysis without using the code-switching strategy in Section 7.4: Specifically, how are these data composed? To be more precise, what would the user's input sequence look like if this strategy were not employed? Concerning the experimental results in Section 7.4, which indicate that the impact of the code-switching strategy is only 1-2%: Why is this influence so minimal?\n\n2. The right panel in Figure 2 and the [MASK] symbol in Figure 3 are not the same, and there are distinct differences in their functionalities. In Figure 2, [MASK] is used as a separator, and it is not intended for predicting words or characters. However, in Figure 3, [MASK] serves as a placeholder that the model needs to predict, whether it is a word or a character. Is this difference expected?", "paper_topic_and_main_contributions": "This paper presents a new neural architecture called INarIG for word-level autocompletion in computer-aided translation. Word autocompletion aims to predict a target word given a source sentence, translation context, and a human-typed prefix. Specifically, it designs an Instruction Unit to fully encode the human prefix using characters, which provides more direct guidance to the model's generation process. The model uses an iterative, non-autoregressive decoder with subwords for compatibility with contextual dependencies on both sides of the target word. This also handles low-frequency words better. The model employs pre-training and multi-task learning to address the challenge of incomplete context. For Chinese, it uses a code-switching strategy to learn mappings between characters and pinyin. The experiments demonstrate state-of-the-art results on two standard datasets, with especially strong improvements on low-frequency words. The paper provides extensive analysis and ablation studies to validate the advantages of the proposed techniques. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "y6pBRrDfE3", "reviewer": "Reviewer_4ENL", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review.\n\nOur model shares some similarities with Ailem et al. (2022), but achieves 11-14% higher accuracy on the same data and model size. We use no additional training data for pretraining and multitask learning, and use no other pre-training models as well.\n\nThe key to our significant improvement is that we identify the adverse impact of incomplete translations on model training, and propose a novel solution accordingly. We design a model that is highly compatible with the standard MT task and address such adverse impact using pre-training (MT models) and multitask learning (WLAC and CMLM). We believe this is one of the major contributions of our work, with detailed descriptions in Sections 1, 4.1 and 7.3. Ailem et al. (2022) do not analyze and propose solutions in this respect.\n\nIn addition, the model architecture we employed is different from that in Ailem et al. (2022). In order to accommodate context dependencies of the right-side target words, we choose to use an NAT model, so as to put all translation information at the decoder side. This design ensures seamless compatibility with MT tasks and allows us to easily transfer optimization strategies in the MT field (e.g. pre-training, multitask learning, back translation augmentation). On the contrary, Ailem et al. (2022) encode both source and target context at the encoder side, and the decoder only generates target words. Their model design is significantly different from a standard MT task setting, so it requires customized optimization strategies, making optimization hard to be performed.\n\nIn all, our contributions to the WLAC task include:\n1. Analyze the characteristics and challenges of the WLAC task;\n2. Design a feasible and efficient model tailored for the WLAC task;\n3. Propose a training strategy to address major challenges of the task and achieve SOTA results with significant improvement;\n\nThank you again for your questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an innovative Iterative Non-autoregressive Instruction Generation (INarIG) model, which effectively handles intricacies in Word-Level AutoCompletion (WLAC) through subword-level decoding. The model is fine-tuned on a pre-trained MT Conditional Masked Language Model (CMLM). \n\nThe INarIG model demonstrates a significant improvement in accuracy compared to current methods and benchmark systems. This advancement raises the bar in WLAC.\n", "reasons_to_reject": "In contrast to Li et al. (2021), which predicts target words at the word level, the proposed model iteratively generates a subword to compose the prediction word. The proposed method also fine-tunes on a pre-trained MT Conditional Masked Language Model(CMLM) to enhance the model since the translation text is incomplete. Similar to this proposed model, Ailem et al. (2022) also use a subword-level model for autoregressive decoding to generate a sequence of subwords to constitute a target word step by step. So, the major difference between Ailem et al. (2022) and the proposed model is finetuning from a pre-trained CMLM. Convincing validation would require a comparative analysis of both models (the proposed model and Ailem et al. 2022), either both with or without fine-tuning.\n\nThe paper predominantly focuses on a specific task (Word-Level AutoCompletion), potentially lacking broader applicability to other NLP domains, which could limit its significance to a diverse conference audience.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The concept of Word-Level AutoCompletion (WLAC) involves predicting a desired word when provided with a source sentence, translation context, and at least one character input from a human. This research introduces the Iterative Non-autoregressive Instruct Generation (INarIG) model, which systematically produces subwords to form the final predicted word. Additionally, the proposed approach refines its performance by fine-tuning a pre-trained MT Conditional Masked Language Model (CMLM) to address incomplete translation context. Notably, the integration of character embedding is employed to incorporate human-typed sequence information into the model effectively.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DbgegnvZNZ", "reviewer": "Reviewer_Tsoo", "summary": "", "questions": "- What is the theoretical complexity of the method in comparison with previous approaches?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q:** What is the theoretical complexity of the method in comparison with previous approaches?\n\n**A:**   Thanks for your thoughtful review.\n\nTheoretically, our model has higher computational complexity compared to that of word-based classification models. Regarding the iterative decoding process of a NAT model, the representations of all tokens on the decoder side need to be computed repeatedly. However, the inference of the NAT model decoder can be fully parallelized (including generating multiple candidate sets), and the number of iterations N is limited (average 2-3), so the inference latency of the model is acceptable in practical application.\n\nIn our experiment, the latency to predict one word is within 50ms with a Tesla T4 GPU powered by pytorch. Further optimizations (adapting acceleration libraries: FasterTransformer [1] or LightSeq [2]) can reduce the inference latency to within 15ms. Even for on-line systems, this level of performance is sufficient.\n\n\n\n[1]  https://github.com/NVIDIA/FasterTransformer\n\n[2] Wang et al., 2021 LightSeq: A high performance inference library for transformers. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The method is well described and the main design decisions are justified. The improvements over previous methods are large, which shows the effectiveness of the proposed approach.\n\t\t\t\n- The experiments are sound, they show ablations over different components and a variety of analysis that covers the most relevant questions. This includes: how the model works with low frequency words, different pre-training strategies, additional monolingual data, and comparison with ChatGPT.\n", "reasons_to_reject": "- The main application of this task is probably on-line, thus latency of the model’s inference should be a factor to consider. Predicting one subword at the time requires running the model N times (N being the number of subword). Also, having a list of suggested candidates increases the latency by C times (C being the number of candidates). The paper does not include latency analysis. ", "questions_to_authors": "- What is the theoretical complexity of the method in comparison with previous approaches?", "paper_topic_and_main_contributions": "This work propose a new model for the task of word auto completion in assisted translation. They use an iterative non-autoregressive decoder that is trained to predict characters as conditional masked LM (one subword at the time) until completing the word. In addition, they use pre-training and multi-task learning strategies to boost the task performance. The evaluation results on datasets performed on 2 languages pairs (in both directions) show large improvement over previous methods.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the word-level auto-completion task (WLAC), useful for computer-assisted translation. The authors propose a new method that, unlike previous approaches, can use target right-context in additional to left context, and handles the fact that either left and/or right context may be empty or incomplete. They experiment with pretraining and co-training on the MLM task along with WLAC to account for the fact that language modelling is done with incomplete target context. They also integrate an experiment with code-switched Pinyin-Chinese character data. They achieve better performance than the benchmark model, perform ablation studies on the model components and perform better on low-frequency words, which are the most important in the task.

The method surpasses the previous state-of-the-art results and although it builds on previous work, introduces a couple of novel elements (character-level encoding for the target prefix, pretraining and multi-task fine-tuning, iterative decoding process with an non-autoregressive MLM model). Ablations are included and the experiments appear sound. It is also important that the method improves results for low-frequency words. The main missing detail is the decoding speed and complexity, which would be important for real-world CAT applications. The authors provide some details about this in their rebuttal. This discussion should be added to the paper and would benefit from a comparison with other methods.
Decision: Accept-Findings