MT2: Towards a Multi-Task Machine Translation Model with Translation-Specific In-Context Learning
Authors: Chunyou Li
Mingtong Liu
Hongxiao Zhang
Yufeng Chen
Jinan Xu
Ming Zhou
Venue: EMNLP 2023 Main (2023)
Keywords: Machine Translation
In-Context Learning

Abstract: Sentence-level translation, document-level translation, translation memory, and terminology constrained translation play an important role in machine translation. Most of the previous work uses separate models or methods to solve these tasks, which is not conducive to knowledge transfer of different tasks and increases the complexity of system construction. In this work, we explore the potential of pre-trained language model in machine translation tasks and propose a Multi-Task Machine Translation (MT2) model to integrate these translation tasks. We design a novel translation-specific In-Context Learning (ICL) paradigm for model training, in which all of the translation tasks can be modeled as context-learning tasks that integrate contextual information for performance improvement. Specifically, we propose a retrieval and alignment method to obtain a large scale context-enhancement training data, then we train the model in an in-context learning manner. Furthermore, we adopt two context-dependent training strategies to encourage the model to better understand and utilize contextual information for translation.
Extensive experiments on translation memory, terminology constrained translation, document-level translation, and few-shot domain-adaptation tasks demonstrate the superior performance of our model, verifying the effectiveness of our proposed approach.

MT2: Towards a Multi-Task Machine Translation Model with TranslationSSpeeffie In-Context Learning
Chunyou Li"; Mingtong Liu', Hongxiao Zhang', Yufeng Chen', Jinan Xu Ming Zhou?
Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China 2 Beijing Lanzhou Technology Co., Ltd., Beijing, China. ( hwwgxiozzaang chenyf, jaxu]ebjtu edu. cn iiiiinggong zhoumi Booong@@llannbooo com
[chunyoulis i,
Abstract
Query: (he foandaiaer "HIMSS" io "Bee7 Al Chémmotta momiomaiitiioo "HIVES is ghoal. loed oow eegeication
procade
Sentence-level translation, document-level translation, translation memory, and terminol- ogy constrained translation play an important role in machine translation. Most of the pre- vious work uses separate models or methods solve these tasks, which not conducive to knowledge transfer of different tasks and increases the complexity system construc- tion. In this work, we explore the potential of pre-trained language model in machine transla- tion tasks and propose a Multi-Task Machine Translation (MT2) model integrate these translation tasks. We design a novel translation- specific In-Context Learning (ICI.) paradigm for model training, in which all of the transla- tion tasks can be modeled as context-learning tasks that integrate contextual information for performance improvement. Specifically, we propose a retrieval and alignment method to ob- tain large scale contextenhaaneement training data, then we train the model in an in- in-context learning manner. Furthermore, we adopt two context-dependent training strategies to encour- age the model to better understand and utilize contextual information for translation. Exten- sive experiments translation memory, termi- nology constrained translation, document-Ied level translation, and few-shot domain-adaen adaptatiin tasks demonstrate the superior performance of our model, verifying the effectiveness of our proposed approach.
Anower imssssit fi Mi FV it Hee
ChaiGPT
Querys
cutting snongy
(UE. plase prosike
doockap offictiee
linging Mestegin
prpieeiis to to
Quers: the imsdaain of TInuSs a 1000 t ye dee plaws peovide the Our
Chinostaniitin "HIMES in ghond hond
Anowen Boff it fotee 61 it ts al Ho 8 it Hite
The willnn
looe well secived 1-
Axoerr is du Wes ie te
The output of our model and ChatGPT. It shows that model can effectively use the term and translation memory information for translation while ChatGPT does not perform well.
Figure 1:
context information (Tan et al., 2019) in document- level translation task, or changing the model struc- ture (Gu et al., 2018; He et al., 2021, 2019; Khan- delwal al., 2020a) to integrate translation mem- ory sentences. This hampers the transfer of knowl- edge across diverse tasks and increases to the intri- cacy of system development.
Recently, large language models (LLMs) have gained significant attention in the field of natural language processing. There are several work that are interested in the application of LLMs in ma- chine translation and have conducted many eval- uation work with LLMs, such as GPT3.5 and BLOOM-176B (Scao et al., 2022; Moslem et al., 2023; Peng et al., 2023). These work find that even the currently best-performed LLM, ChatGPT, still lags behind the powerful supervised models in many cases (Zhu et al., 2023; Jiao et al., 2023). Meanwhile, some work has shown that the LLMSs can perform better when given appropriate prompts, which shows the in-context learning ability of large models (Ghazvininejad et al., 2023: Peng et al., 2023; He et al., 2023; Brown et al., 2020). These work mainly focus on evaluating and how to stim- ulate the performance of large language models,
1 Introduction
Translation memory, terminology constrained trans- lation, and document-level machine translation play an important role in machine translation. Most previous studies employ separate models or meth- ods to handle these tasks, such as introducing ad- ditional context encoders (Zheng et al., 2020; Xu et al., 2021) and hierarchical attention to modeling
Contribution during internship at Beijing Lanzhou Technology Co., Ltd., Beijing. China. Corresponding author.

---

however, focus on training transsation-specific model enhance in-context translation capability. In this work, we explore training a unified model to address multiple machine translation tasks. We propose Multi-Task Machine Translation (MT2) model, which follows the in-context learn- ing paradigm to tackle the context-learning trans- lation tasks that integrate contextual information for performance improvement, such as transla- tion memory, terminology constrained translation, and document-level machine translation et al. We propose a two-stage training method, where the first stage focuses on training the general trans- lation ability, and the second stage aims to im- prove its mondext- ability with context- enhancement data and training strategies. Espe- cially, we propose a method based on retrieval and alignment to obtain a large-scale context- enhancement training data, where the retrieved sim- ilar sentences and aligned translation pairs are used as contexts for improving translation. Moreover, we adopt two context-dependen training strate- gies to enhance the context-modeling ability of the model, which encourage the model to better understand and utilize the knowledge transmitted through the context. We give some examples to compare our model with the ChatGPT, and it shown in Figure! 1.
We construct extensive experiments on multi- ple machine translation tasks, and the results demonstrate that our model-7B is comparable to the ChatGPT in in-context learning ability.
2 Related Work
2.1 Machine Translation Tasks
There are many tasks in the field of machine trans- lation, and most studies have proposed various so- lutions for different tasks (Liang et al., 2021a,b, 2023). In the translation memory task, the previ- ous work tried to change model structure for in- tegrating translation memory sentence (Gu et al., 201 8; He al., 2021), introduce additional mod- ules (He al., 2019; Khandelwal al., 2020a). Reheman et al. (2023) proposed to treat similar sentences as prompts to the neural machine trans- lation model during the decoding process, which can leverage the knowledge provided by prompts to improve the translation. Recent research showed that LLMs can improve translation performance when prompted by translation memory (Moslem et al., 2023). Typical solutions for terminology constrained translation are based on slot replace- ment and term injection (Ailem et al., 2021; Anas- tasopoulos et al., 2021; Zhang et al., 2021). Recent work explored providing phrase-level prompts to the model to improve translation results, which achieves significant gains and demonstrates transla- tion controllability (Sun et al., 2022a; Ghazvinine- jad et al., 2023). With document-level translation task, some methods focus on changing the archi- tecture of the model, such as introducing addi- tional context encoders (Zheng et al., 2020; Xu et al., 2021), cache-like memory network (Tu et al., 2018), and hierarchical attention (Tan et al., 2019). Sun et al. (2022b) proposes a multi-resolutional training method, which involves multiple levels of sentences and can translate document in end- to-end manner. There are two common solutions in domain adaptation tasks, which are data cen- tric (Britz et al., 2017) and model centric (Lachaux et al., 2020). More recent methods aim to adapt to a new domain during inference without addi- tional training (Khandelwal et al., 2020b; Zheng et al., 2021; Agrawal et al., 2022; Sun et al., 2022a), where their practice is often to provide some bene- ficial prompting information.
We conduct extensive experiments on multiple translation tasks, including translation memory, ter- minology constrained translation, document-level translation, and domain adaptation tasks. With the help of in-context learning paradigm, the transla- tion quality and term translation accuracy achieve significant improvement. In addition, the model can perform Nomain-specift. translation directly by concatenating few in-domain examples without updating parameters.
Our contributions can be summarized as follows:
We propose a multi-task machine transla- tion model with translation-speccfic in-context learning, which can address multiple trans- lation tasks with the in-context learning paradigm.
We propose a context-enhanceneen data con- struction method based on retrieval and align- ment to improve the context learning ability of the model and apply two context-dependent training strategies to promote to better un- derstand and utilize contextual information.
However, performing each task with separate models may not be conducive to knowledge trans- fer between different tasks. Our method integrates

---

these tasks into unified model with in-context learning paradigm, benefiting knowledge transfer and reducing the intricacy of system development.
Outpur
Language Model
2.2 In-Context Learning
lepal
Recently, in-context learning becomes new paradigm for natural language processing (NLP), which allows the language models make pre- dictions only based a few examples in the form demonstration (Brown et al., 2020; Dong et al., 2022; Moslem et al., 2023). From a principle per- spective, the LLMs will learn the pattern hidden in the demonstrations and make correct predictions accordingly. On the other hand, ICL is similar to the decision-making process of human beings, it involves learning from analogy (Dong et al., 2022). Specifically, given a query input text q and series of candidate answers Y {yn...., ..... 3r J. a well- trained model M will give the answer conditional on n demonstrations C' [(1,,11),-... (In, ) where (g:, 34) is query-answer pair and generally written in task-related natural language templates.
the imnuiation of hove invertment RRi--M4 if Bl- the irselation This the tuslitia of "busance" au 20-9 the innulation of 'Deciding on the businees policy and investmene pian 'ie
pollcy" F8
Contest
New Query
The In-context learning paradigm for machine translation, where the context can be the translation memory sentences, terminologies and preceding trans- lated sentence pairs in the document.
Figure 2:
tion, which affects the current prediction with at- tention mechanism. Like traditional translation task (Vaswani et al., 2017), we adopt the cross- entropy loss function as follow:
b
[(0) - Σ
Togp(y: Vog contert; 0),
ICL has multiple attractive advantages, the main one that it can directly perform predictions based on the pre-trained models without updating param- eters. This could greatly reduce the computation costs for adapting new tasks (Chen et al., 2021; Min et al., 2022). In addition, it can easily inte- grate human knowledge into language models by modifying templates and demonstrations (Lu et al., 2022; Wei et al., 2022; He et al., 2023). There are recent works that leverage in-context learning paradigm to solve specific tasks such as informa- tion extraction, sentiment analysis (Yoo et al., 2021; Liu et al., 2022; Xu et al., 2023), and machine trans- lation (Moslem et al., 2023; Zhu et al., 2023).
where 0 is the model parameters, Yi and |y| are the i-th token in the input sentence and length of y, respectively. contert represents the context information.
In the translation memory task and terminology constrained translation task, conterl represents the similar parallel sentences pairs retrieved and ter- minology constrained pairs, respectively, For the document-level translation task, contezt the pre- ceding source sentences and their corresponding translations.
Specifically, take the translation memory task as example, given parallel translation pair (Ir, y) and set of retrieved similar sentence pairs C = {(=]) yf") ..., (- . v4 ). we wrap each pair (at", , v/")) with an instruction template and then con- catenate them front of æ as context-enhancement examples. The examples of training data is illus- trated in the right of Figure 3.
Our work is based on machine translation with in-context learning paradigm, we explore training unified model to solve multiple translation tasks. Most previous work mainly focus on how to apply on LLMs, while we concentrate on how to train and enhance in-context learning ability of the model for translation task.
As illustrated in Figure 2, with the help of the in-context learning paradigm, the model can obtain and utilize knowledge transmitted through context to guide the translation process, so as make the translation more accurate and improve the quality of translation.
3 Methodology
3.1 Translation-Specifi--h Context Learning In this work, we extend the traditional machine translation training task. When predicting the cur- rent token, our model not only considers the source sentence and a part of translated tokens, but also rich context information such as term informa-
3.2 In-Context Learning Data Construction To train the model with the in-context learning paradigm, we propose a method based on retrieval

---

Resieve
Irsobtien csangg:
liforreett prllien Ru1: RETTTTRRRA
De inytih zusckree of to haara eoe chauune eo peance aane ext osty bob: seviedde
TM dislww
SEC Kcuifurtlatad Flansas poldion
Jubthh Wel Re Bii
TGT: 12.20::
the contayy 1 in 2.28 F
Et si A T
Decemuer mample t:
sec: neijing lotta orchier k.
Alignnure
cipeal
Majee
1:3
Rejiig
connEtt dalanci
unized moond
tusabionn of 1e Mill 86 Je undend
txear das dhy barrs syclnder
Stalis
cumpple
Clow
Chiv
tratrbatttnn or exstrpppcceee mnnucc. The personal compater sssrade And inkonet and aemeriic also cureee ci M.
the isan kaiee "wccattiioon aschimtioon agrerment i titt insoistion
1o 1t. BR16- NEarZ
"laal poosecing "ibeiigg peosade the Chaise bundlate aaaia puccciip hane
They gr mound burallatiin the
8-1 St St. 02:21 Nov
Treniree
All is thipp End group lien.
TGT.
Terns coample
-- WIB1.
Tlay Inaalaaion like
"prnoope pordle
Do trandaaic modeers lond prociiee titee inseilation
All the Biigo
insrration practes inamatiinal rectpnecty volana Weill: s1Blli -t 86.00
........
........
they wil thane
Figure 3: Overview of data construction. The left is methods for obtaining context data, and the right is training data samples. 'Reverse" means to reverse the 'SRC' and "TGT'. '</end>' is the defined instruction end symbol.
and alignment to construct context-enhanccmment data. The overview of data construction flow is shown on the left of Figure 3.
for the lack of data in term-constrained translation task.
Document-level Context Construction. We ex- pect the system can model the context of document- level data. However, the parallel document data is scarce, so try to translate the monolingual document data into the target language with pow- erful translation system obtain document-level training corpus. Specifically, given an English doc- ument corpus, we employ a translation model translation it into Chinese. When used for train- ing, the Chinese is used as the source side and the English is used as target side.
Sentence-level Context Retrieval For each source sentence, we assign a similarity score be- tween and each sample in the translation memory database, where the similarity score is calculated according to the BM25 algorithm (Robertson et al., 2009), and the calculation formula is as follows:
[q:) : (k1 + 1)
score = Σ w(()) [(x)) + kn (1 - b avgal where qi is the i-th word in the retrieval sentence, D is an example in the translation memory database. wo() denotes inverse document frequency (IDF), and (9:) represents the frequency of word qi in D, in and b are two adjustable parameters. [D] and wegdl denotes the length of D and average length, respectively. We then rank the similarity scores from largest smallest and select the top 20 sam- ples as candidate translation memory sentences. Phrase-level Context Extraction. Limited by the scarcity of terminology translation data, we generalize it to phrase-level constrains to construct context-enhancemeen data. As shown in Figure 3, for each parallel translation pair, we first get the phrase alignment dictionary and then filter the phrase alignment dictionary according to part of speech, in which we mainly keep noun phrases and adjectives. In this case, we can construct large- scale context-enhanced training data, which is con- ducive to the training of ICL strategy and makes up
Next, we construct the document-leved context- enhancement data two ways.
The document data is first split into sentence level. Then like the document example in Figure 3, the source and corresponding trans- lation are concatenated in sentence order.
To enhance document-level context modeling capability, inspired by (Sun al., 2022b), we collect n sentences as a segment on the basis of following the sentence order, like document example in Figure 3, where n is a random number.
3.3 Context-Learning Enhancement with Two-Stage Training
In order to train the model better utilize context, we propose to train the model two stages, namely translation pre-training and in-context learning

---

training. The first stage mainly cultivates the gen- eral translation ability of the model. We use large number of parallel sentence pairs for training on the basis of the pre-trained language model to enhance its translation ability.
Domain
Education
Subtitle
Thesis Spoken
Zh-En En-Zh
w/o TM
36.66
22.61
39.78
34.52
w/ TM
38.83
25.22
41.85
36.38
w/o TM
25.02
20.64
17.79
24.56
The second stage of training focuses on improv- ing the in-context modeling ability of the model. To this end, the model is trained with the context- enhancement data described above. In addition, maintain the general translation ability in the model, we also mix some general parallel data. Furthermore, we adopt two context-dependent training strategies to enhance the contextual Iearn- ing ability of the model. The first is mask strat- egy, given a translation memory training sample (21,21 V1, 11) O (I22, y2, T2) O.. (Iar Vn: In) >, there a 50% probability to perform the following processing: Iterate over each token in sentence Zn. and if it has appeared in a context sentence Zen, will be masked with a 25% probability and this to- ken in the context will also be masked with a small probability. Where e denotes concatenation oper- ation, 4 is the instruction template, and z; and yi denote the source and target sentence, respectively. In this manner, the model will pay more attention in context when predicting the next token.
w/ TM
28.14
22.19
21.80
26.64
Table I: Experiments results on the translation memory and domain- comain-adaptatton tasks.
dataset, whereas the remaining data utilized the translation memory database. In the document- level translation task, we combine Ist2010-2013 from IWSLT2015 as the test dataset in the Zh-En direction. We also select a representative document- level novel translation in the En-Zh direction. Pre- vious work demonstrated the importance of con- textual information in ensuring consistency and coherence in novel translation (Wang al., 2023; Karpinska and Iyyer, 2023). We manually con- structed high-quality test dataset of 1000 sen- tences. In the terminology constrained translation task, we construct test dataset for Aviation, Law, and Entertainment domains based on a terminology dictionary, and each test dataset contains 2000 sen- tences. Specifically, we use 30w collated termi- nology dictionary. The evaluation metric is Sacre- BLEU We also the report COMET (Rei et al., 2022) scores Appendix A.
In order to enhance the robustness of the model, we propose the second training strategy. Specif- ically, we preprocess the alignment dictionary to get new dictionary where each phrase has multi- ple meanings, that is, polysemy. In practice, each phrase maintains the correct translation with 50% ratio, and the rest case we will add noise. If the i-th phrase is chosen to be corrupted, it may be replaced with another meaning of the word 30% of the time or an unrelated translation 20% of the time. In this case, it can improve contextual adaptation of terms.
4.2 Implementation Details
For the model setting, we choose the BLOOM- 7b1 as the base model, which is an decoder-only transformer language trained by Bigscience (Scao et al., 2022). All our methods are implemented with Huggingface and Deepspeed -hatt" We apply zero-stage-3 (Rajbhandari et al., 2020) and gradi- ent checkpointing strategy to train the model with 32 A 100 GPUs We set the aatch_size and gradi- ent accumulation_ steps hyperparameters to 5, 32, respectively. The learning rate for the first stage is le-4 and the learning rate for second stage is 5e-5. And the parameters are optimized by using Adam optimizer with 31 = 0.9, /32 = 0.95 and e = 10-$, We obtain the phrase alignment dictionary with fast align (Dyer et al., 2013).
4 Experiments
4.1 Datasets
For the training data, we collect about 100 million parallel data and about 100 million sackstranslttion data, and we take about 40 million parallel data as the translation memory database. In the second training stage, we supplement about 1.7 million document-level translation corpus.
In the translation memory and domain- adaptation tasks, we evaluate our model in Edu- cation, Subtitle, Thesis, and Spoken domains from UM-Corpus (Tian et al., 2014). And we randomly select 1000 sentences for each domain as the test
httppp///gtttuo..o////o////a/rra cnttps://gt thw.b wtt/cocmooccaeecccea com/miccosoott/////oocoooc/oo DeepSpeedExamples

---

Domain
Aviation
Law
Entertainment
Method
BLEU
Accuracy
BLEU
Accuracy
BLEU
Accuracy
w/o Term
41.61
81.44%
53.62
84.37%
30.77
78.57%
Term
44.93
92.27%
56.13
92.28%
31.81
87.07%
Table 2: The performance of model the terminology constrained translation task in En-Zh direction. Accuracy represents the rate of the target phrase appears in the translation output.
Task
Translation memory task
Terminology constrained translation task w/o ICL
Model
w/o ICL
w/ ICL
A
W/CL
ChatGPT
32.26
36.41
4.15
48.61/85.03
51.33/94.94
2.72/9.91
Ours
36.03
40.27
4.24
51.69/83.97
56.18/93.67
4.49/9.70
Table 3:
The comparison between the ChatGPT and our model in translation memory and terminology constrained tasks. We also test the accuracy of term translation terminology constrained translation task. Our model achieves greater growth than ChatGPT with the in-context learning paradigm.
4.3 Main Results
Direction
System
ICL
BLEU
Translation Memory and Domain Adaptation. this experiment, we evaluate how well the model performs domain-adaptation task directly without parameter updates. To be more convincing, we dis- card the translation memory sentence which is the same the sentence to be translated and the results are given in Table 1. When the similar in-domain sentences retrieved are provided, the performance increases significantly, with average BLEU score improvements of 2.18 and 2.69 points in En-Zh and Zh-En directions, respectively. The results show that the model has the ability to capture the knowledge contained in the context and apply it to improve translations, and the higher the sentence similarity, the more useful information is provided. Terminology Constrained Translation. To eval- uate the performance the model in terminology task, we conduct experiments in three datascts and the results are reported in Table 2. With the help of terminology constrained information, our model significantly improve the accuracy of terminol- ogy constrained translation in all datasets without additional training. There was also an improve- ment in the overall translation quality evaluated by BLEU scores, with approximately points. These experiments show that our model can effectively utilize the knowledge transmitted from the contexts and apply it to translation processing.
Base
No
30.72
Zh-En
Base
Yes
29.38
MT2
No
30.50
MT2
Yes
31.33
Base
No
28.54
En-Zh
Base
Yes
29.08
MT2
No
28.78
MT2
Yes
29.50
Table 4: Experimental results of document-level transla- tion Zh-En and En-Zh directions. 'Base' and 'MT2' represent the models trained from the translation pre- training and in-context learning stage, respectively.
without the in-context learning paradigm and the results shown in Table 4. The value of ICL is No indicates that no contextual information is pro- vided, that is, traditional sentence-level translation. Instead, Yes means that when translating the sen- tence, we will concatenate some preceding source sentences and corresponding translations. In line with the process of human document translation, method takes into account the historical trans- lation information, ensuring better consistency be- tween the translated segments. From the table, the MT2 performs better than the base model, and we can find that the model can effectively improve the overall translation quality with the in-context learn
Document Translation. To show the context- learning performance of the model in document- level translation, we conduct experiments with and

---

I: give the English translation of 1960
T.My parents were married in 1960.
C: the translation of $5 "6* is 'get married'
T: My parents got married in 1960.
I: give the Chinese translation of is to develop effective culling strategies for the bison population."
T: & fh å # A मं Alt 2 3i + A # # f n m
C: the Chinese translation of 'The culling strategy has been well received. 'is , 7 T
T: è & 3 şj 4 ft # t1 E कं * b5 i #
I: give the English translation of # I # 22 b $ f A â + Hf 1X, $ # 26 E
T: Li Si is younger, work experience also insufficient, education is not high, but no matter do what thing, he is conscientious and responsible, so, the leader thinks highly of him very much. D: If we translate this paragraph with in-context learning paradigm, the translation is as follows:
T: Li Si is younger and has less work experience and less education. But no matter what he does, he is serious and responsible. So, the leader thinks highly of him.
Table 5: Some cases showing different outputs of the model with and without context. Where I, T, C, D represent Instruction, Translation, Context and Description, respectively. The third example shows subject consistency with in-context learning paradigm.
ing paradigm. addition, as shown in Table 5, our model can improve the translation quality in consistency and coherence.
our proposed method for translation task.
4.5 Case Study
4.4 Compared With ChatGPT
We give some cases of the output of model with and without providing context in Table 5. We can see that the prompt context can affect the translation re- sults, which indicates that the model has the ability to obtain the information conveyed by the context prompts and apply them to translation process. An interesting example is example 1, where the tense of the information we provide is present simple, and the translation results given by the model apply the correct past tense.
In order to explore the performance our model and ChatGPT in solving translation tasks with in- context learning paradigm, we select 50 examples each for terminology constrained translation and translation memory tasks. We obtain the translation results of ChatGPT through web version We use two prompt templates and record the higher scores for ChatGPT, <trc>=<igt> and the translation of <sre> is <igi>, where the <sre> and <igl> denote the placeholder for source and target sentence re- spectively. The results are given in Table 3. Due to different training data and model sizes, it is dif- ficult to directly compare with ChatGPT. Results show that our model achieves better translation per- formance with supervised training data, which is consistent with previous studies (Zhu al., 2023). In this work, we mainly focus on the performance improvement of the model with and without the in-context learning paradigm, which is the value of A in the table. The results show that our model generally perform better than ChatGPT in terms of performance increase, except for term accuracy. Although our model is small in scale, it can achieve results comparable to ChaIGPT after training with
(0) BooZhhEEn)
DoaEinJhh
51.4
29.5
23:2
21.8
30.8
29.5
caraed
contant
Thesks
(9) Subiile
25.0
41.6
24.2
41.4
24-A
41.2
24.2
Figure 4: The effect of the number of context.

---

Domain Education Subtitle
Thesis
Spoken
you like area whare you Livingt
you
testeggt
farthest 37.97
24.02
41.21
34.88
@i (u tt siling?
[f--Tom will stteal nummen counci Bl Parto stendd fiti0 KT L.
middle
38.06
24.04
41.29
34.93
tymowr A2 phr, Mhet will unord eurreerr courre +B 41 sinn ii-l "M diIEN,
nearest 38.64
24.22
41.42
35.71
ID-Do like area where you listegt-tale to Sest Ie-8 "suea 16 &f 4 Hu
A2-ibo you like where lisingh -leddd
Table 6:
Results the impact of similar sentences order.
Pi-iiver will vorrd cammmor coumo H8 19 dood 1688 8: 1- H4
"uteed Bordre/ Ad #3-Ton will useed sammrrr counse -8 22 slend wod--ii bt
Domain Education Subtitle Thesis Spoken
The comparison results, where the top the output without the second comlext-enhancement strategy and the bottom is the output with the strategy. The bottom results more robust than the one above.
Figure 5:
w/o mask
39.02
24.87
41.60
35.92
w/ mask
38.83
25.22
41.85
36.38
Table 7: Experiments the impact of mask strategy.
Method
Aviation
Law
Entertainment
No
41.38
52.45
29.73
5 Analysis
Yes
43.66
54.98
31.50
5.1
Effect of the Number of Context
Table 8: The comparison results with and without the second strategy
In this subsection, we investigate the effect of the number of concatenated contexts on the translation results. The results is in Figure 4. Overall, the performance improves initially as the number of contexts increases, but as we continue to add more contexts, the scores start to drop. We speculate that as the number of contexts increases, noise will be introduced, resulting in decrease in transla- tion quality. In the translation memory task, con- catenating more contexts means that the retrieved sentences get lower similarity scores and the cor- relation between texts with longer spans will also decrease in decument-level translation, leading to the introduction of noise which degrades the trans- lation quality.
5.3 Impact of Context-Enhanceemnnl Strategy In this subsection, we conduct ablation study to ex- plore the benefits these two strategies. From the Table 7, we can observe that the model with mask strategy performs better in all datasets except Edu- cation, with maximum BLEU score improvements of 0.35, 0.25, 0.46 in Subtitle, Thesis and Spoken suggest that
datasets, respectively. Therefore, with the help of this strategy, the model can ef- fectively pay attention to the useful information contained in the context and utilize improve the translation quality.
The reason we introduce the second strategy is to allow the model to adapt to the noise in the con- text without breaking the semantics of the sentence itself. In order verify the effectiveness of the method, we introduced some noise into the test set, and the experimental results are shown in Table 8. We also provide a few examples to evaluate the performance of the model in two scenarios in the Figure 5. We can observe that when we give a correct translation pairs in the context, the model can use this information to improve the transla- tion results. However, when give incorrect term translation, the model will discard this prompt information
5.2 Efffec of the Order of Context
It is obvious that the translation results will be affected by the similarity score of the retrieved sen- tences. Besides, we are interested in evaluating the effect of the concatenation order of similar sen- tences on the translation results. Hence, we take top three similar sentences retrieved as a baseline to test the influence of the most similar sentence being farthest, middle, and nearest to the sentence to be translated, respectively. And the experiments results are shown in Table 6. In line with the intu- itive idea, when the most similar sentence is closer the sentence to be translated, the performance improvement is more obvious, indicating that the model is more sensitive to closer context informa- tion and pays more attention to it.
destroys the semantics of the sen- tence itself. That shows the performance of the model after adopting the second strategy is more robust and it can reduce the impact of noise in the

---

context when translating.
context examples selection for machine translation. arXiv preprint arXi::2212.02337.
6 Conclusion
Melissa Ailem, Jinghsu Liu, and Raheel Qader. 2021. Encouraging neural machine translation to satisty terminology constraints. arXiv preprint arXiv:2106.03330.
In this work, we propose a multi-task machine translation model with translition-specific in- context learning, which can address multiple context-leaming type translation tasks with the in- context learning paradigm. We propose a method based retrieval and alignment to construct the comtext-enhancement data and then train the model with in-context learning paradigm in two stages. In addition, we also adopt two context-dependent training strategies to make model better understand and utilize contextual information for performance improvement. Finally, we conduct experiments on terminology constrained translation, document- level translation, translation memory and few-shot domain- -adaaaataoa tasks to evaluate the effective- ness of our approach.
Antonios Anastasopoulos, Laurent Besacier, James Cross, Matthias Gallé, Philipp Koehn, Vassilina Nikoulina, et al. 2021. On the evaluation of ma- chine translation for terminology consistency. arXiv preprint arXiv:2106. 1891.
Denny Britz. Quoc Le, and Reid Pryzant. 2017. Effec- tive domain mixing for neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 118-126.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 877-1901.
We will extend the model to multilingual trans- lation system and explore instruction transfer learn- ing in the future.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He Hc. 2021. Meta-lcarning via language model in-context tuning. arXiv preprint arXiv:2 2/ 10.07814.
Limitations
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.0022
Here we discuss the limitations of our work. First, for the convenience of data processing and training, we set the maximum length of input text at training time, which may cause the model to perform poorly when dealing with too long input text. Secondly, the noise contained in the context has a negative impact on translation quality, and we adopted some strategies to enhance the robustness the model. However, it is still worth exploring how to avoid or alleviate the impact of noise on translation qual- ity. We will conduct further research on the above issues in future work.
Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644 648.
Marjan Ghazvininejad, Hila Gonen, and Luke Zettle- moyer. 2023. Dictionary-hasee phrase-level prompt- ing of large language models for machine translation. arXiv preprint arXiv:2302.0755
Jiatao Gu. Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018. Search engine guided neural machine trans- lation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Acknowledgement
The research work descried in this paper has been supported by the National Key R&D Program of China (220AAA01080001) and the National Na- ture Science Foundation of China (No. 61976015, 61976016, 61876198 and 61370130) The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to im- prove this paper.
Qiuxiang He, Guoping Huang. Qu Cui, Li Li, and Lemao Liu, 2021. Fast and accurate neural machine translation with tramslation memory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 31 3170-3180.
Qiuxiang He, Guoping Huang, Lemao Liu, and Li Li. 2019. Word position aware translation memory for neural machine translation. Natural Language Processing and Chinese Computing: 8th CCF In- ternational Conference, NLPCC 2019, Dunhuang, China, October 9-14. 2019, Proceedings, Part 8. pages 367-379. Springer.
References
Sweta Agrawal, Chunting Zhou, Mike Lewis. Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. In-

---

Zhiwei He, Tian Liang. Wenxiang Jiao, Zhuosheng Zhang. Yuju Yang, Rui Wang, Zhaopeng Tu, Shum- ing Shi, and Xing Wang. 2023. Exploring human- like translation strategy with large language models. arXiv preprint 1rXiv:2305.04/14
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2022. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2791-2809.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt good translator? a preliminary study. arXiv preprint arXiv:2301.08745.
Yasmin Moslem, Rejwanul Haque, and Andy Way. 2023, Adaptive machine translation with large language models. arXiv preprint 230..9949..1
Marzena Karpinska and Mohit lyyer. 2023, Large lan- guage models effectively leverage document- level context for literary translation, but critical emrors per- sist. arviv preprint arXiv:2304.0 032445.
Keqin Peng. Liang Ding, Qihuang Zhong, Li Shen. Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation. arXiv preprint 2rXiv2303.133880
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer. and Mike Lewis. 2020a. Nearest neighbor machine translation. arXiv preprint arXiv:2010.00110.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Com- puling. Networking, Storage and Analysis, pages 1- 16. IEEE
Urvashi Khandelwal, Angela Fan. Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020b. Near- est neighbor machine translation. arXiv preprint
Abudurexiti Reheman, Tao Zhou, Yingfeng Luo. Di Yang, Tong Xiao, and Jingbo Zhu, 2023. Prompt- ing neural machine translation with translation mem- ories, arXiv preprint arXiv:2301. 053380.
arXiv:2010.00710.
Marie-Anne Lachaux, Armand Joulin, and Guillaume Lample, 2020. Target conditioning for one-to-many generation. arXiv preprint ar xixv:20090097550
Ricardo Rei, Ana C Farinha, José GC de Souza, Pedro G Ramos, André FT Martins, Luisa Coheur, and Alon Lavie. 2022. Searching for cometinho: The little metric that could. In Proceedings of the 23nd Annual Conference of the European Association for Machine Translation, pages 61-70.
Yunlong Liang, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. 2021a. Modeling bilingual con- versational characteristics for neural chat translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Pupers), pages 5711-5724, Online. Association for Computational Linguistics.
Stephen Robertson, Hugo 7aragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 344:3333-38,,
Yunlong Liang, Fandong Meng, Jinan Xu, Jiaan Wang, Yufeng Chen, and Jie Zhou. 2023. Unified model learning for various neural machine translation. arXiv preprint 205.0:2305.02777
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221 20.000000.
Yunlong Liang, Chulun Zhou, Fandong Meng, Jinan Xu, Yufeng Chen, Jinsong Su, and Jie Zhou. 2021b. Towards making the most of dialogue characteris- ties for neural chat translation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 67-79, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Zewei Sun, Qingnan Jiang, Shujian Huang, Jun Cao. Shanbo Cheng, and Mingxuan Wang. 2022a. Zero- shot domain adaptation for neural machine trans- lation with retrieved perase-level prompts. arXiv preprint arXi 1409.
Zewei Sun, Mingxuan Wang, Hao Zhou, Chenggi Zhao, Shujian Huang. Jiajun Chen, and Lei Li. 2022b. Re- thinking document-level neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3537-3548.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? DeeLIO 2022, page 100.
Yao Lu, Max Bartolo, Alastair Moore, Scbastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers), pages 8086-8098.
Xin Tan, Longyin Zhang, Deyi Xiong, and Guodong Zhou. 2019. Hierarchical modeling of global context for document- neural machine translation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP) pages 1576-1585.

---

Liang Tian, Derek F Wong, Lidia S Chao, Paulo Quaresma, Francisco Oliveira, and Lu Yi. 2014. Um- corpus: A large english-chinese parallel corpus for statistical machine translation. LREC, pages 1837- 1842.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong. Jiajun Chen, Lei Li, and Shujian Huang. 2023. Multilingual machine translation with large language models: Empirical results and analy- sis. arXiv preprinl arXiv:2304.04677
Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2018. Learning to remember translation history with a continuous cache. Transactions of the Association for Compulational Linguistics, 6:407-420
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processsing systems, 30.
Longyue Wang, Siyou Liu, Mingzhou Xu, Linfeng Song. Shuming Shi, and Zhaopeng Tu. 2023. A survey on zero pronoun translation. arXiv preprint 1001::2305.10166.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Lc, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.
Hongfei Xu, Deyi Xiong, Josef Van Genabith, and Qi- uhui Liu. 2021. Efficient context-awes aware neural ma- chine translation with layer-wise weighting and input- aware gating. In Proceedings of the Twenty-Ninth International Conference on International Joint Con- ferences on Artificial Intelligence, pages 3933-3940.
Xin Xu, Yuqi Zhu, Xiaohan Wang, and Ningyu Zhang, 2023. How to unleash the power of large language models for few-shot relation extraction? arXiv preprint 2rXiv22005.05555
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang- Woo Lee, and Woomyeong Park. 2021. Gpt3mix: Leveraging large-scale language models for text aug- mentation. arXiv preprint 2 08826..
Tong Zhang, Long Zhang, Wei Ye. Bo Li, Jinan Sun, Xiaoyu Zhu, Wen Zhao, and Shikun Zhang. 2021. Point, disambiguate and copy: Incorporating bilin- gual dictionaries for neural machine translation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3970- 3979.
Xin Zheng, Zhirui Zhang, Shujian Huang, Boxing Chen, Jun Xie, Weihua Luo, and Jiajun Chen. 2021. Mon-parametric unsupervised domain adapta- tion for neural machine translation. arXiv preprint arXiv:2109.06604
Zaixiang Zheng Xiang Yue, Shujian Huang, Jiajun Chen, and Alexandra Birch. 2020. Towards mak- ing the most of context neural machine translation. arXiv preprint urXiv:2002.07982

---

A
Appendix
Domain
Education Subtitle
Thesis
Spoken
En-Zh
w/o TM
55.3
17.2
64.4
52.4
w/ TM
57.3
21.6
65.4
53.4
Zh-Bn
w/o TM
37.4
6.40
15.6
36.2
w/ TM
41.7
7.31
19.1
36.5
Table 9: The COMET scores on the translation memory and domain-ane addpttaion tasks.
Direction
System
ICL
COMETT
MT2
No
47.2
Zh-En
MT2
Yes
48.1
MT2
No
35.7
En-Zh
MT2
Yes
38.0
Table COMET scores document-level trans- lation in Zh-En and En Zh directions.
Domain
Aviation
Law
Entertainment
w/o Term
66.1
89.4
56.7
Term
70.5
93.0
57.9
Table 11:
The COMET scores of terminology con- strained translation En-Zh direction.


Reviews:
[{"review_id": "w5lMk6Z6vG", "reviewer": "Reviewer_afzn", "summary": "", "questions": "Please see the above section", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer afzn:\n\n  We sincerely thank you for the time and effort you have dedicated to reviewing our paper. Your valuable comments and constructive suggestions have been instrumental in improving the quality of our work. Below, we reply to the comments raised by the reviewer point-by-point.\n\n**Q1: Knowledge transfer of different tasks lacks clarity and substantiation.**\n\nA1: Previous work demonstrates that multi-task learning benefits the learning of machine translation models, such as [R1] and [R2]. In the paper [R3], single-sentence and multi-sentence joint training has also improved the performance of the model. We followed the successful experience and ideas of previous work, as this is not our main contribution point, we did not analyze it in detail. Thank you very much for your suggestion, we will add corresponding statements in the Related Work section to better enable readers to understand the relevant content.\n\n**Q2: The clarification of the mentioned alignment method is essential to enhance the understanding of readers regarding the practicality of this alignment method.**\n\nA1: One of its contributions is that it can make up for the lack of data in term translation tasks. This alignment-based method can construct large-scale context-enhanced training data, which is conducive to the training of ICL strategies and improves the in-context learning ability of the model. In addition, the data can be generalized not only at the term level but into phrase-level constraints, which helps to improve the generalization of the model. Thanks for your suggestion, we’ll add more detailed descriptions in the later version.\n\n**Q3: Missing reference for BM25 algorithm.**\n\nA3: Thanks for pointing this out, this is our negligence, we will add the reference of BM25 algorithm (R[4]).\\\n\\\nThanking you again for your generosity with your valuable time invested in improving our paper.\n\nYours sincerely,\n\nAll author.\\\n\\\n\\\n**References**\n\n[R1] Zhou S, Zeng X, Zhou Y, et al. Improving robustness of neural machine translation with multi-task learning[C]//Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). 2019: 565-571.\n\n[R2] Luong M T, Le Q V, Sutskever I, et al. Multi-task sequence to sequence learning[J]. arXiv preprint arXiv:1511.06114, 2015.\n\n[R3] Sun Z, Wang M, Zhou H, et al. Rethinking document-level neural machine translation[J]. arXiv preprint arXiv:2010.08961, 2020.\n\n[R4] Robertson S, Zaragoza H. The probabilistic relevance framework: BM25 and beyond[J]. Foundations and Trends® in Information Retrieval, 2009, 3(4): 333-389.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is thoughtfully organized and skillfully composed, demonstrating a clear and logical structure that aids in understanding the content. The author's analysis is robust, contributing to the paper's credibility by offering thorough insights and well-founded conclusions. Moreover, the paper effectively illustrates the applicability of pretrained language models (LLMs) across multiple tasks in machine translation (MT) and highlights the concept of translation via in-context learning.\n", "reasons_to_reject": "The abstract commences by discussing MT2's exploration of a singular model engaged in various tasks, particularly emphasizing its advantage in \"knowledge transfer of different tasks\" compared to using distinct models for each task. However, the evaluation illustrating MT2's benefits from other tasks, such as \"knowledge transfer of different tasks,\" lacks clarity and substantiation. Despite this, the research effectively addresses other pertinent inquiries, displaying solid analysis.\n\nThe method of alignment mentioned in the abstract remains unclear within the paper, lacking explicit details on its application in the research. For example, its implementation in context retrieval should be explicitly detailed, especially in relation to its contribution to the in-context learning strategy employed in MT2. This clarification is essential to enhance the understanding of readers regarding the practicality of this alignment method.", "questions_to_authors": "Please see the above section", "paper_topic_and_main_contributions": "In this paper, the authors delve into the utilization of pretrained language models for machine translation tasks. They introduce the MT2 model, which is a unified model capable of handling various machine translation tasks: translation memory and domain adaptation, document-level machine translation, and terminology-constrained translation. The paper exhibits strong motivation, a clear structure, and well-crafted writing. Furthermore, it is accompanied by a robust analysis and compelling experimental evidence.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "BM25 algorithm"}, "extra_text": {}}, {"review_id": "uQkkjWTJEw", "reviewer": "Reviewer_1ZMP", "summary": "", "questions": "What is the difference in performance between the non-finetuned bloom-7b1 model and the model after finetuning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 1ZMP:\n\nWe sincerely thank you for the time and effort you have dedicated to reviewing our paper. Your valuable comments and constructive suggestions have been instrumental in improving the quality of our work. Below, we reply to the comments raised by the reviewer point-by-point.\n\n**Q1: Consider adding more evaluation metrics and methods.**\n\nA1: Thanks for your suggestion, we use the BLEU metric in the paper, showing the effectiveness of the method. In order to overcome the shortcomings of a single automatic evaluation metrics (BLEU score), we will use more diverse evaluation metrics to evaluate the translation quality of the model, such as chrf, COMET or GPT-4 etc. \n\n**Q2: More analysis about translation quality consistency and coherence.**\n\nA2: Automatically evaluating the consistency and coherence of document translations is difficult. On the one hand, we use BLEU score to evaluate it and show improvement in Table 4. On the other hand, we also use the method of manual evaluation. Specifically, we manually evaluated 100 pieces of test data, and about 30% of the samples showed improvement. We will follow your suggestions, which will be added to the appendix in the final version.\n\n**Q3: Quantitative analysis results for the second training strategy.**\n\nA3: Thank you for your suggestion. In our experiment, in order to enhance the robustness of the model, we adopted the second strategy. The manual evaluation demonstrated improvements, and we conducted case studies, but we apologize for not including the corresponding statistical analysis process in the article. We adopt your suggestion and give a quantitative analysis experiment, where we introduce some noise in the terminology constrained translation task. The results is (Aviation: 41.38 vs. 43.66;  Law: 52.45 vs. 54.98;  Entertainment: 29.73 vs. 31.50;). The experimental results show that the model with the second training strategy is more robust. We will include corresponding analysis in future version for better understanding by readers. Thanks again for your suggestion.\n\n\n**Q4: More details about test data.**\n\nA4: In the translation memory task and domain adaptation task, we use the public dataset UM-Corpus, and in the document translation task, we use IWSLT2015 as the test dataset. In the terminology constrained translation task, we were unable to find a suitable public test dataset. Therefore, based on a high-quality term dictionary, we extract the term information contained in each sentence to construct a test set and evaluate the performance of the model with it. Specifically, we use a 30w collated terminology dictionary. Thanks for your suggestion, we include more details in the future.\n\n**Q5: The difference in performance between the bloom-7b1 model and the model after finetuning.**\n\nWe evaluated the origin bloom-7b1 model, and the results are not good. The BLEU score is basically below 5, and it is prone to translation off-target and repeated generation problems. The [R1] paper also reached the same conclusion. Typically, larger models have better translation results, however even a large model of 176B is still weaker than supervised training translation models (R[2]). Thank you for your suggestion, we will add the corresponding experimental results in the appendix later.\\\n\\\n\\\nThanking you again for your generosity with your valuable time invested in improving our paper.\n\nYours sincerely,\n\nAll author.\\\n\\\n\\\n**References**\n\n[R1] Bawden R, Yvon F. Investigating the translation performance of a large multilingual language model: the case of bloom[J]. arXiv preprint arXiv:2303.01911, 2023.\n\n[R2] Jiao W, Wang W, Huang J, et al. Is ChatGPT a good translator? A preliminary study[J]. arXiv preprint arXiv:2301.08745, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors proposed an innovative In-Context Learning (ICL) paradigm, which effectively utilizes information from the historical translation database to enhance translation performance.\n\n2. The authors conducted extensive experiments to verify the effectiveness of the Translation-Specific In Context Learning method for translation tasks, especially conditional machine translation tasks.\n\n3. The authors provided a reproducible data construction method, which allows for the quick and easy collection of a large amount of data, facilitating replication and further exploration by future researchers.", "reasons_to_reject": "1. In the era of Large Language Models, traditional evaluation metrics such as BLEU and ROUGE cannot accurately judge the translation quality. Therefore, it is best to combine manual evaluation, training-based automated metrics, or incorporate GPT-4 for scoring, especially when the test dataset is limited and small. This would provide a more credible assessment.\n\n2. The paper relies heavily on case analysis to support its conclusions, which is not sufficiently persuasive. For example, the statement in the paper, \"In addition, as shown in Table 5, our model can improve the translation quality in consistency and coherence.\"\n\n3. The paper should include quantitative analysis results for the second training strategy, rather than just relying on a case analysis. This is necessary to validate the effectiveness and necessity of the strategy.\n\n4. It would be preferable to disclose more details about the data evaluation, such as the test dataset. This would facilitate replication and further exploration by other researchers.", "questions_to_authors": "What is the difference in performance between the non-finetuned bloom-7b1 model and the model after finetuning?", "paper_topic_and_main_contributions": "The paper presents a novel Multi-Task Machine Translation (MT2) model that effectively incorporates various translation tasks, including sentence-level translation, document-level translation, translation memory, and terminology constrained translation. The model introduces an innovative In-Context Learning (ICL) paradigm, treating all translation tasks as context-learning tasks. Additionally, the authors devise a retrieval and alignment technique for acquiring extensive context-enhancement training data and implement two context-dependent training strategies to enhance the model's comprehension and utilization of contextual information. Experimental results demonstrate the model's excellent performance across multiple machine translation tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A6H3ANic8N", "reviewer": "Reviewer_7odN", "summary": "", "questions": "A. In section 3.3, how do you ensure that each phrase has multiple meanings?\n\nB. What is the performance of BLOOM-7b1 if you don't finetune the model, but use the same prompts for inference?\n\nC. Do you have quantitative results for the second context-enhancement strategy?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7odN：\n\nWe sincerely thank you for the time and effort you have dedicated to reviewing our paper. Your valuable comments and constructive suggestions have been instrumental in improving the quality of our work. Below, we reply to the comments raised by the reviewer point-by-point.\n\n\n**Q1: The importance of multi-task training is not clearly demonstrated.**\n\nA1: Our main contribution is to unify multiple tasks into a unified model through the in-context learning paradigm, and propose several solutions to enhance in-context learning. The traditional approach is to model these tasks separately, and some tasks are limited by the limited training data, such as terminology constrained translation task. Our proposed two methods for constructing training data and multi-task training can alleviate the problem of task data sparsity. The importance of multi-task learning is not our main concern,we did not analyze it in detail. Previous work demonstrates that multi-task learning benefits the learning of machine translation models, such as [R1] and [R2]. We followed the successful experience and ideas of previous work. But a major difference is that we propose a method based on the in-context-learning paradigm, which integrates traditional machine translation methods into a unified model, thus different tasks can be jointly trained to improve the effect.\n\n**Q2: Missing References**\n\nA2: Thank you for your reminder, we will add corresponding references [R3], [R4]  and [R7] in later editions. \n\n**Q3: Each phrase has multiple meaning.**\n\nA3: Our motivation is to enhance the robustness of the model. We obtain alignment dictionaries with an alignment tool from a large amount of parallel data in advance, each segment can get multiple meanings with high probability. And it doesn't matter even if it doesn't have multiple meanings, we will keep the original correct interpretation and add random noise when constructing the training data. Thank you for raising this question, we will add corresponding details later.\n\n**Q4: The difference in performance between the bloom-7b1 model and the model after finetuning.**\n\nA4: We evaluated the origin bloom-7b1 model, and the results are not good. The BLEU score is basically below 5, and it is prone to translation off-target and repeated generation problems. The [R5] paper also reached the same conclusion.. Typically, larger models have better translation results, however evan a large model of 176B is still weaker than supervised training translation models (R[6]). Thank you for your suggestion, we will add the corresponding experimental results in the appendix later.\n\n\n**Q5: A quantitative results for the second context-enhancement strategy.**\n\nA5: Thank you for your suggestion. In our experiment, in order to enhance the robustness of the model, we adopted the second strategy. The manual evaluation demonstrated improvements, and we conducted case studies, but we apologize for not including the corresponding statistical analysis process in the article.  We adopt your suggestion and give a quantitative analysis experiment, where we introduce some noise in the terminology translation task. The results is (Aviation: 41.38 vs. 43.66;  Law: 52.45 vs. 54.98;  Entertainment: 29.73 vs. 31.50;). The experimental results show that the model with the second training strategy is more robust. We will include corresponding analysis in future version for better understanding by readers. Thanks again for your suggestion.\\\n\\\n\\\nThanking you again for your generosity with your valuable time invested in improving our paper.\n\nYours sincerely,\n\nAll author.\\\n\\\n\\\n**References**\n\n[R1] Zhou S, Zeng X, Zhou Y, et al. Improving robustness of neural machine translation with multi-task learning[C]//Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). 2019: 565-571.\n\n[R2] Luong M T, Le Q V, Sutskever I, et al. Multi-task sequence to sequence learning[J]. arXiv preprint arXiv:1511.06114, 2015.\n\n[R3] Min S, Lewis M, Zettlemoyer L, et al. Metaicl: Learning to learn in context[J]. arXiv preprint arXiv:2110.15943, 2021.\n\n[R4] Chen Y, Zhong R, Zha S, et al. Meta-learning via language model in-context tuning[J]. arXiv preprint arXiv:2110.07814, 2021.\n\n[R5] Bawden R, Yvon F. Investigating the translation performance of a large multilingual language model: the case of bloom[J]. arXiv preprint arXiv:2303.01911, 2023.\n\n[R6] Jiao W, Wang W, Huang J, et al. Is ChatGPT a good translator? A preliminary study[J]. arXiv preprint arXiv:2301.08745, 2023.\n\n[R7] Robertson S, Zaragoza H. The probabilistic relevance framework: BM25 and beyond[J]. Foundations and Trends® in Information Retrieval, 2009, 3(4): 333-389.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A unified model can handle multiple translation tasks, and the additional context clearly improves results.\n\nThe paper includes many comparative examples that help understand the behavior and benefits of the approach.", "reasons_to_reject": "The importance of multi-task training is not clearly demonstrated. The MT2 model should be compared with models for which the 2nd stage only contains examples for 1 task (plus general parallel data).\n\nImportant related work (see \"Missing References\") is missed, especially in terms of training a language/translation model on in-context learning examples (i.e. learning to learn in-context).", "questions_to_authors": "A. In section 3.3, how do you ensure that each phrase has multiple meanings?\n\nB. What is the performance of BLOOM-7b1 if you don't finetune the model, but use the same prompts for inference?\n\nC. Do you have quantitative results for the second context-enhancement strategy?", "paper_topic_and_main_contributions": "This paper presents an approach (MT2) to train a model that handles multiple in-context translation tasks (e.g. terminology constrained translation, document-level translation, translation memory). A pre-trained language model is first fine-tuned on parallel sentences to improve general translation ability. It is then further finetuned on a dataset, created based on retrieval and alignment, containing in-context prompts for multiple specific translation tasks. During this second stage, some tokens are masked to encourage the model to rely on the context more. Some phrases are also replaced to improve model robustness.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L90: we -> We\n\nBM25: add citation\n\nGive a name to the second context-enhancement strategy."}, "extra_text": {"Missing_References": "Min et al. MetaICL: Learning to Learn In Context. NAACL 2022. https://aclanthology.org/2022.naacl-main.201/\n\nChen et al. Meta-learning via Language Model In-context Tuning. ACL 2022. https://aclanthology.org/2022.acl-long.53/"}}, {"review_id": "OC2UjMiloJ", "reviewer": "Reviewer_qFRK", "summary": "", "questions": "When constructing sentence-level context, where does the source sentence mentioned in L250 comes from?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer qFRK:\n\nWe sincerely thank you for the time and effort you have dedicated to reviewing our paper. Your valuable comments and constructive suggestions have been instrumental in improving the quality of our work. Below, we reply to the comments raised by the reviewer point-by-point.\n\n**Q1: This model is bilingual, not multilingual.**\n\nA1: As with previous work, in order to verify the effectiveness of the method, we conducted experiments on English-Chinese bidirectional data. However, we can easily extend it to multiple language translation, because BLOOM is a multilingual pre-trained language model. In the future, as you mentioned, we will extend it to a multilingual translation model.\n\n\n**Q2: The performance in the small-scale datasets.**\n\nA2: With the in-context learning paradigm, there is also improvement when there is less data. Under the same data, the gain brought with the in-context learning paradigm is more. When we did the preliminary experiment, we conducted it on a relatively small-scale data, and the final effect tends to be roughly the same on each task. Specifically, we use about 10 million data, and with the in-context learning paradigm, the BLEU score could be increased by 1-2 points. Therefore, our method can also improve the performance of model on small-scale datasets. Thanks for your suggestion, we will add these conclusions in the future to better illustrate our work.\n\n\n**Q3: Where does the source sentence mentioned in L250 comes from?**\n\nA3: There may be ambiguity in my expression, the “source sentence” here refers to the source sentence in the parallel sentence pair. We will correct this description, thanks for the suggestion.\\\n\\\n\\\nThanking you again for your generosity with your valuable time invested in improving our paper.\n\nYours sincerely,\n\nAll author.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* An integrated translation model is useful for real applications.\n* The improvement of empirical results is significant compared with the base model and ChatGPT.\n* According to the shown examples, the proposed training strategy effectively enhances the translation quality.\n", "reasons_to_reject": "* This model is bilingual, not multilingual. It is unclear its performance when generalizing to other languages.\n* While the results are impressive, training such a model is expensive (100 million data for one language). Though this may not be a problem for an off-the-shelf model, I am wondering if such amount of data is necessary. If not, MT2 can conveniently expand to other foundation models.\n", "questions_to_authors": "When constructing sentence-level context, where does the source sentence mentioned in L250 comes from?", "paper_topic_and_main_contributions": "This work proposes a multi-task translation model MT2 with in-context learning, which integrates translation memory, terminology constrained translation, document-level translation, and few-shot domain adaptation tasks. The model is first pre-trained on a huge parallel corpus and then trained on an in-context learning dataset constructed by retrieval and alignment. Extensive experiments show MT2 achieves significant improvement over the base model and ChatGPT.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a multi-task machine translation model called MT2 and uses in-context learning (ICL) paradigm for model training. It has a clear structure, extensive experiments and analyses.
Authors demonstrate the effectiveness of large language models (LLMs) for various machine translation tasks via in-context learning with convincing results. Some minor aspects could be improved for the camera-ready: references (see reviewers comments), use more diverse evaluation metrics of translation quality (not only BLEU), inclusion of detailed results for the second training strategy, minor improvements in grammar and style.
Decision: Accept-Main