DecoMT: Decomposed Prompting for Machine Translation Between Related Languages using Large Language Models
Authors: Ratish Puduppully
Anoop Kunchukuttan
Raj Dabre
AiTi Aw
Nancy F. Chen
Venue: EMNLP 2023 Main (2023)
Keywords: NLP
LLM
few-shot prompting
Machine Translation
TL;DR: This paper proposes an approach for few-shot decomposed prompting for machine translation between related languages

Abstract: This study investigates machine translation between related languages i.e., languages within the same family that share linguistic characteristics such as word order and lexical similarity. Machine translation through few-shot prompting leverages a small set of translation pair examples to generate translations for test sentences. This procedure requires the model to learn how to generate translations while simultaneously ensuring that token ordering is maintained to produce a fluent and accurate translation. We propose that for related languages, the task of machine translation can be simplified by leveraging the monotonic alignment characteristic of such languages. We introduce DecoMT, a novel approach of few-shot prompting that decomposes the translation process into a sequence of word chunk translations. Through automatic and human evaluation conducted on multiple related language pairs across various language families, we demonstrate that our proposed approach of decomposed prompting surpasses multiple established few-shot baseline approaches. 
For example, DecoMT outperforms the strong few-shot prompting BLOOM model with an average improvement of 8 chrF++ scores across the examined languages.

DecoMT:
Decomposed Prompting for Machine Translation Between Related Languages using Large Language Models
Ratish Puduppully' Anoop Kunchukuttan 3.5,6 Ai Ti Aw Nancy F. Chen
3.5,6
Raj Dabre
Institute for Infocomm Research (I'R), A STAR, Singapore
CCNS @ CREATE, Singapore MMicrosoft, India
National Institute of Information and Communications Technology
iIT Madras 6 A14Bharat
puduppullyei2r.a-sstee .ed...edu.s.. sg ankunchuêmi crosoft. com nfychenei2r. a edu.
raj. wtw.ddbbeeict.go....
Abstract
significantly streamline trade and enhance social connections.
This study investigates machine translation be- tween related languages i.e., languages within the same family that share linguistic charac- teristies such as word order and lexical simi- larity. Machine translation through few-shot prompting leverages a small set of translation pair examples generate translations sentences. This procedure requires the model to learn how to generate translations while si- multaneously ensuring that token ordering is maintained to produce fluent and accurate translation. We propose that for related lan- guages, the task of machine translation simplified by leveraging the monotonic align- ment characteristic of such languages. We introduce DecoMT, novel approach of few- shot prompting that decomposes the transla- tion process into sequence of word chunk translations. Through automatic and human evaluation conducted on multiple related lan- guage pairs across various language families, we demonstrate that our proposed approach of decomposed prompting surpasses multiple es- tablished few-shot baseline approaches. For example, DecoMT outperforms the strong few- shot prompting BLOOM model with an aver- age improvement of chrF++ scores across the examined languages.
Related languages, often from the same fam- ily, share word order and lexical characteristics, leading to predominantly monotonic translations where word order is largely preserved. This is seen in languages like Hindi, Marathi, Malayalam, Tamil, Bengali, etc. from the Indian subcontinent, which follow Subject-Object Verb (SOV) struc- ture. Similar monotonic translation relationships are also observed among other language pairs, such as Indonesian and Malay or Ukrainian and Russian. Recent work has shown the power of few-shot prompting with large language models (LLMs) for tasks like machine translation, summarization, and question answering (Lin al., 2022; Workshop et al., 2023). machine translation, this approach prompts an LLM with a handful of example pairs and test example. This requires the model to generate translations while ensuring a fluent word ordering, a process that fails to account for any unique characteristics intrinsic to the languages involved. For instance, it neglects the monotonic alignment-ane an integral trait evident in translations between related languages.
test
be
LLMs are often biased towards English in their training data. For example, in mT5 (Xue et al., 2021), Hindi and Malayalam tokens represent just 0.8% and 0.07% respectively. This imbalance hin- ders LLM performance in tasks involving non- English languages and English non-English translations (Lin et al., 2022). In particular, for few- shot translation tasks between related languages, these models may not have encountered sufficient data in these languages. Overcoming these limita- tions can be achieved by incorporating inductive biases about related languages.
1 Introduction
In this work, we focus on the translation between re- lated languages, a vital aspect from both economic and social perspectives. A considerable amount of commercial activity and social interaction occur between neighboring regions speaking two related languages. In these situations, pivot translation via third language, such as English, can prove inef- ficient due two inference steps which can also cause cascading errors (Dabre et al., 20.1). Instcad, direct translation between related languages could
Recently. Khot et al. (2023) introduced an ap- proach known as decomposed prompting. This

---

(DecoMT), a novel approach that simplifies the translation task by dividing it into the in- cremental translation of word chunks.
Hi
M.
Independentt translation
Hb
M:
000
Hp
We perform extensive evaluations on closely related languages from diverse language fami- lies, including pairs such Hindi = Marathi, Hindi = Malayalam, Hindi = Telugu, Hindi = Gujarati, Indonesian = Malay, Russian = Ukrainian, and Spanish Portuguese.
Hi+Hi: <mask++M
Ri
HitHi+H:
R
+++
Contextusl transfation
000
Hiii
We compare DecoMTT against several robust baselines, including few-shot prompting of LLMs (Lin et al., 2022; Workshop al., 2023), as well as sequential autoregressive prompting of bidirectional LLMs (Patel et al., 2023). We demonstrate that DecoMT delivers robust results when compared to these base- lines, particularly outperforming them in sce- narios involving low-resource languages.
Ri
R+<mask>+M_.
000
Hp *H|
Raussmask
Figure The diagram provides an overview of De- composed Prompting for Machine Translation (De- coMT). The source text (H) is divided into sev- eral chunks (H₁, H....HH.HHH,HHH........ chunk is translated independently using few-shot prompting, yielding corresponding target chunks (M₁, M,... M142...MaM) The DecoMT process leverages the source chunks, their respective transla- tions, and the previously predicted contextual transla- tion incrementally predict the contextually appropri- ate translation of the subsequent chunk.
Each
We release code and model outputs on github !
Related Work
Few-shot Prompting for MT Few-shot prompt- ing for MT leverages an autoregressive LLM, which is prompted with small number of sentence pairs alongside their translations. The LLM then predicts the translation when provided with test sentence. Examples of such LLMs include XGLM (Lin et al., 2022) and BLOOM (Workshop et al., 2023). We interchangeably refer to this approach as Standard Prompting.
technique dissects a complex task into simpler, more manageable subtasks, each of which ad- dressed through few-shot prompting of LLMs.
We aim to enhance translations by harnessing the inductive bias of monotonicity related languages. We posit that by relieving LLMs from implicit re- ordering and focusing on sub-sentence structures, more accurate translations, particularly in longer sentences, can be achieved. This leads us to pro- pose a decomposed prompting approach, termed Decomposed Prompting for Machine Translation (DecoMT) (Figure 1), which splits input sen- tence into chunks, translates each independently, and incrementally generates context-aware transla- tions.
Garcia al. (2023) have shown the effective- ness of few-shot prompting in machine translation. Yet, their method necessitates training a decoder- only LLM from scratch. In comparison, we use an off-the-shelf LLM, mT5, for DecoMT. A series of recent research delves into example selection for prompt construction (Vilar et al., 2023; Zhang et al., 2023; Kumar et al., 2023; Agrawal et al.. 2023). In our method, rely amples for prompting. Jiao et al. (2023) analyzed machine translation using ChatGPT and found that ChatGPT's performance aligns closely with com- mercial translation systems when utilizing GPT-4. In the interest of reproducibility, our emphasis lies on publicly accessible LLMs like BLOOM and mT55
While much of the existing research on prompt- ing focuses on decoder-only LLMs, recent studies (Patel et al., 2023) show the potential of encoder- decoder models like mT5 (Xue et al., 2021) for such tasks. Our DecoMT approach builds upon this premise, utilizing the mT5 encoder-devoder LLM.
fixed set of ex-
The following are our contributions:
We introduce Decomposed Prompting for MT

---

Sequential Autoregressive Prompting Patel al. (2023) introduced an approach for prompt- ing bidirectional LLMs, such as m'T3 (Xue al., 2021). Their Sequential Autoregressive Prompting (SAP) method generates token autoregressively, appends it back to the input, and predicts the subse- quent token. They demonstrated that SAP outper- forms traditional few-shot prompting for LLMs. Our method also leverages bidirectional LLMs. However, while they primarily exploit the autore- gressive nature of these models, we further harness the bidirectional capability of LLMs to generate context-aware translations.
English Spanish English it beautful day
<mask> hermoto dia
Spanish: <mask
Translane from English Spanish English: is beautiful day Spanish: Es <mask> dia
<mask> hermoso
Encoder
Decoder
Figure 2:
Depiction of bidirectional encoder- decoder LLM prompting strategies for translation tasks.
The upper part (a) uses an autoregressive translation, while part (b) employs the LLM for masked token infill- ing using surrounding context.
research aligns with this direction, concentrating translations between related languages, many of which are characterized as low-resource.
Decomposed Prompting Khot et al. (2023) pro- posed decomposed prompting, an approach that breaks down complex tasks into simpler ones, each tackled using few-shot prompting of LLMs. We ap- ply this prompting strategy to the task of machine translation between related languages.
DecoMT
In this section, we present the DecoMT Approach, our technique for decomposed prompting in Ma- chine Translation. Our method involves two-stage translation process for word chunks: firstly, an in- dependent translation stage where each chunk is translated in isolation; and secondly, a contextual translation stage where translation occurs while considering the surrounding context.
Incremental Generation In the field data- to-text generation, Puduppully et al. (2022) pre- sented a strategy for document generation that de- composes the process into generating sequence of paragraphs, interleaved with predicting plan for each paragraph. Our DecoMT method can be viewed extension of this approach for the task of translating monotonically aligned sentences, where the plan is implicitly specified through the monotonic chunk alignment.
3.1 Employed Pretrained Model
In implementing DecoMT, we use the mT5 model (Xue et al., 2021), specifically the XL variant with 3.7 billion parameters. mT5 is encoder-decoder model that is trained with a span-cormiption objec- tive. During the training process of mT5, random spans within the input text are replaced with place- holders such as (mask_Op 0), (mask_ 1), and so forth. In the output text, these correspond to mask to- kens followed by the respective spans that were substituted in the input. Just like in the case of T5 (Raffel et al., 2020), the spans being replaced during training of lengths varying from 2 to 5 tokens.
Press and Smith (2018) proposed an eager trans- lation approach, in which the model begins translat- ing without having to wait until the entire sentence has been processed. Our DecoMT method shares this characteristic, it similarly doesn't require the whole sentence be available before initiat- ing translation. However, unlike their method, De- coMT's translation units extend beyond single token. Moreover, DecoMT incorporates a contex- tual translation phase where the translation of an independent chunk is further refined through infill- ing.
One approach machine translation with mT5 follows the Standard Prompting method, as de- picted in Figure (a) (Workshop et al., 2023; Lin et al., 2022). In this setup, the mT5 encoder re- ceives an input sequence: source language label, source sentence, target language label, followed by (mask) token. The decoder then generates the translation. In our independent translation frame- work, we employ this technique to produce M, from H,, as depicted in Figure 1.
Machine Translation for Low Resource Lan- guages There have been studies on machine translation models for low-resource languages (Haddow et al., 2022; Team et al., 2022; Ramesh al., 2022; A14Bharat et al., 2023; Dabre al., 2022). While most of these focus on translations between English and other languages, Fan al. (2021) is notable for its emphasis on improving translations among -onglihlish languages. Our
Another technique to utilize mT5 for translation is by leveraging its bidirectional infilling capability,

---

exhibited in Figure 2 (b). The prompt includes the source language label, source sentence, target language label and a partially masked translation. The mT5 decoder then generates the masked tokens. This specific approach is used in generating our contextual translations R₃ as shown in Figure I.
as a product of conditional probabilities:
ply|2, = aa x222)
3
p(ui 24 1 184 10
Depending where the (mask) placeholder is inserted, the model will perform either text comple- tion or infilling. It's important to note that a single mask can yield more than one token.
*y p
3.4 Prompt Construction
Our methodology employs fow-shot prompting, a technique that allows an LLM to make predic- tions based on a limited number of examples. This section will elucidate the process of constructing prompts for independent and contextual translation. We utilize five examples for few-shot prompting.
3.2 Creating Aligned Monotonic Translations through Human Annotation
We select the first five examples from the dev set of the FLORES dataset (Goyal et al., 2022). Each example consists of a pair of corresponding sen- tences in two different languages. Annotators tasked align these sentences a monotonic man- ner, maintaining the same sequence of information. Importantly, annotators have the liberty to modify the sentences as required to achieve this.
Word count in Each Chunk Let us consider the token count within each word chunk in both prompt templates and test examples. For the prompt tem- plates, k and j denote the number of tokens in a word chunk for independent and contextual transla- tion, respectively. Conversely, in test example, signifies the token count within word chunk for independent translation.
3.3 Translation Model
Let æ represent the input sentence and B denote the number of chunks in æ. We define ý as the prelim- inary translation of æ, obtained by concatenating independently translated chunks. Furthermore, y represents the final translation, which is assembled from contextually translated chunks. For the pur- pose of simplification in our formulation, we omit the prompt template and focus on the translation of test examples.
and 5 and 10, respec- tively. Nevertheless, the morphological richness of languages varies as a single token in one language might equate to several tokens in another. Hence, during the construction of prompt templates, we programmatically align each chunk fully with its translated equivalent, causing potential deviations from the standard values of and 10 for and j- Lastly, treat m hyperparameter, which is tuned using the FLORES development set.
We typically
In the case of independent translation, make the assumption that each ii is only dependent on its corresponding z4, where indicates the index of the chunk within a sentence. This captured by the equation:
Independent Translation Each translation ex- ample for independent translation (Figure 3) com- mences with "Translate from [Source language] to [Targee language]:", followed by a line break. then "[Source language]." and the first chunk of the source language sentence. Subsequently, we present "[Target language]:" and the corresponding translated chunk on a new line. This sequence is replicated for all the chunks in sentence.
3 p = II p())))))))
(1)
In the case of contextual translation, we parame- terise y as dependent on x and ÿ. represented as:
Upon completing a sentence, we use a newline separator and proceed to the next example. This procedure is repeated for all five examples in the prompt template.
plyl ú) = p(u122 ... y2l=122......................... ... vs. Ú102 ûs)
(2)
We make a conditional independence assumption that, any position i, yi dependent on x;-1-
In the case of the test example, the prompt begins with "Translate from [Source language] [Target language]:", followed by line break and "[Source language]:" with a chunk from the source language. The subsequent line is "[Target language]: (mask)".
Tú, Fy the previous contextual translation yi 1- and the next independent translation ÿi+1- This assumption allows us to rewrite the joint probability

---

Translate from Hindi to Malayalam:
Translate from Hindi to Malavalam:
Hindi: सोयता аř, अेाााड of सशक
Hindi: लोमार की, अोाinड ofrafich सşा sfi Riefa is पeा Malayalanz: diggggg agoode aendsocavenógi ordigcc anad nanuseneeeeeeeee eonggeerrong
Malayalaur disgggugg ggodbe (ggoratooooorreentttatttoogg cennteearvtiiig ondig (On Monday, Stanford Univessity School)
Handi afts सहडीरय às dेाprrrी 7
(On Monday, seientists at the Stanford University School of Medicine)
Maloyalam mocedd manuScnkkreeee veogr werocodd (of medicine scientists)
Hindi: altiasss ae 3i yea
ate sad बात
Hindi: witirersff vs seth अहा
Malavalamz: eaocesreeg soarges 1200, mocooroddly aooootr movustil) 4b00b dt-Papeero
Malayalam: seouoeneg mecaoos gotss
(cells into their types)
(eapable of sorting cells arcording to their types)
Hindi smare ofe सहाी
Hindi: 1a zy अाातोारि œweरपा tे suiveure di योपपग tft.
Malayalau ge @ood eshajrro (sort based on)
Malayalam: am apmllw авoсneйierrr gaJamBom. earrgalt mown go
Hiudi: qa सय gाााोाe अपकररय
(ammonnced invention new diagnostic tool)
Malayalam: syml seavorilere @wieaaa (a now diagnoetic tool)
3 more examples luvv
Hindi- िोया
Malayalam: charga aymoolll (2Mujjnagg
Translate from Hindi Malayalam:
(announeed the invention
Hindi: यहनोरययर 1ê ahe and शमस que पपरयोट अnfनररााा पार yia my ीिाा Malayalam: crovsecu crunasecrm anw. onccr orsavemenn aith agase- swoddg" aorobb caoaooree alsttaat adleommouu
mncee exampies here
[sn sirport fire enginer rolled over its way to the scene) Hindi: सााीयय सfre सााप t.
Translate from Tindi to Malayalam:
Hindi: Vesrरer di sit साय
Malayalam: lejpersoluu, ancayaamuh dhacooigg natgera. (local media tokl)
Malayalam: crsoners cranevaroma ad auieerr cooasmmen (on the way to the scene)
Hindi qaa queeйé affrhnær eieh qeक nh रसा
Translate from Hindi to Malayalam:
Malayalam: em aprneen cown aioum, allshaa acleom- moul
Hindi: कालाात áी यबयाीी (पााfeet)) @ अा à aाद è टिहारर à असर (Since moving to the Catalan capital (Bureelona), Vidal has for the club) Malayalam: daoggeperoops mejomnnccaao smosks qymes, cinoct pogled"
(an airport fire engine rolled over)
Hindi: सानीय #ffiem a
Malayalam: (ppocc0lle accoyaarec (local muslia)
(Catalan's capital <mask> since Vidal for club)
Hindi: कााा
Figure Prompt Template for Contextual Translation with a Test Example: Similar to Figure 3, but with longer word chunks (approx. 10 tokens). The test prompt pairs source language label with three con- catenated word chunks. Following the target language label is the previous contextual translation, a (mask) placeholder, and the third chunk's independent transla- tion. The model's 's goal is to complete the masked chunk. English bracketed text is explanatory and not part of the prompt. The aligned chunks are colored identically.
Malayalau: diewsodg ourgere. (bas told)
Translate from Hindi to Malayalam: Hindi काााकत की सपाायीी (Catalan's is capital) Malayalam: smasio
Figure 3: Prompt Template for Independent Translation with a Test Example: The template includes five sen- tences in the source (Hindi) and target (Malayalam) lan- guages divided into word chunks. The model receives a test example source chunk and a target language prompt with a (mask) placeholder, aiming to predict the corre- sponding target chunk. English text in brackets is for clarification, not in the actual prompt.
translation]". Here, the model's task is to infill the translation for the second source language chunk. Appendix A contains an example of independent and contextual translation prompt templates for translation between Indonesian and Malay.
The model's objective at this point is to predict the translation for the source language chunk.
3.5 Inference
Contextual Translation The prompt template for contextual translation (Figure 4) mirrors that of independent translation, with one key difference: the examples in prompt template are around twice long as that of the lengths of examples in inde- pendent translation template prompt. the test ex- ample for contextual translation, the prompt starts with "Translate from [Source language] to [Target language]:", followed by "[Source language]:" and a concatenation of three chunks from the source language.
Figure 1 provides an overview of our DecoMT ap- proach. We omit the prompt template from the block diagram for simplicity. We segment the in- put sentence into multiple chunks, denoted as H1, H2, H,, H: 1. H H3, each comprising m tokens. We then independently translate cach chunk into corresponding translations, labelled as M1. Mg....,M, M, M,+1, M .... M2.
The key innovation in our approach lies in the contextual translation, which is performed incre- mentally for each chunk. Initially, we concatenate the first two chunks, H, and Hy, with the place-
The next line reads "[Target language:: Iprevious contextual translation] (mask) [next independent

---

holder (mask) and the translation of the second chunk Mg. This forms the input to predict the first contextual translation, R
2022), which utilizees a subword-based tokenizer. Conversely, chrf++ evaluates character n n-gram matches for n values ranging from to 4, in addi- tion to word n-gram matches that include unigram and bigram. Given its demonstrated higher correla- tion with human annotator scores for low-resource languages (Popović, 2017), chrF++ serves valu- able metric for study. We use the SacreBLEU library (Post, 2018) to compute these metrics, We provide signatures for both BLEU and chrF+++ 3. For hyperparameter tuning, we utilize the FLO- RES development set. We evaluate chunk sizes for m from the set (3,4.5).
Subsequently, we concatenate the first three chunks, H1, H2, and H3, with the contextual transla- tion obtained from the previous step, R1, alongside the placeholder (mask) and the translation of the third chunk, M3. This is used to predict the next contextual translation, Rg.
This process is continued iteratively. At an inter- mediate step, the chunks H; H4+1. and H3+2, along with the previously computed contextual transla- tion R2, the placeholder (mask), and the translation of the chunk M,+2, are used to predict the next contextual translation, R++1.
4.2 Evaluation
Finally, for the last chunk, the input is the con- catenation of the penultimate and final chunks, H3-1 and Hg, the last computed contextual transla- tion R 1, and the placeholder (mask). The model then predicts the final contextual translation, Rg. Appendix contains a worked out example for translation from Hindi to Malayalam.
We conducted evaluations on multiple languages using the Flores devtest set, focusing specifically on translations between closely related languages: Hindi (hin) Marathi (mar), hin ++ Malayalam (mal), hin <+ Gujarati (guj), hin Telugu (tel), Indonesian (ind) ++ Malay (zsm), Ukrainian (ukr) + Russian (rus), and Portuguese (por) ++ Spanish (spa). The latter pair represents high-resource language setup for comparison.
4 Experimental Setup
We conduct comparative study of our DecoMT approach, which is based on mT5 (Xue al., 2021) with 3.7B parameters, against various established approaches. These include the Standard Prompting technique applied to 7.1B parameters variant of BLOOM (Workshop ct al., 2023), and 7.5B param- eters variant of XGLM (Lin et al., 2022). We also compare our method with the Standard Prompt- ing technique applied to the mT5 model. In this case, as mT5 generates only few tokens at time, we append the generated text back to the input to prompt further text generation. Furthermore, we compare our approach with SAP (Patel et al., 2023), a technique that also utilizes mT5 with 3.7B param- eters.
Results
5.1 Automatic Evaluation
The results of our evaluations are summarized in Table I. We conducted statistical significance test- ing via paired bootstrap sampling (Koehn, 2004) (p < 0.05). Regarding performance, XGLM (Lin et al., 2022) when used with Standard Prompt- ing, demonstrated low spBLEU and chrF++ scores for low-resource language pairs such hinl->mal, hin<>mar, hin++guj, and ind++zsm. It performed somewhat better with the ukr-tru pair, likely due to the greater availability of resources for Russian compared to Ukrainian.
BLOOM (Workshop et al., 2023), outperformed XGLM across all directions and language pairs except tel >hin. However, BLOOM does not cur- rently support languages such as zsm, rus, and ukr.
4.1 Evaluation Metrics
Our approach's performance is assessed using spBLEU (Goyal et al., 2022), a variant of BLEU(Papineni et al., 2002), and chrF++ (Popović, 2017) metrics. The BLEU metric measures word n-gram matches, encompassing unigram, bigram, trigram, and four-grams. However, due the mor- phological richness of the languages we are work- ing with, BLEU scores can often be underestimated. To counteract this, we employ spBLEU as sug- gested by NLLB (Goyal et al., 2022; Team et al.,
When implemented with Standard Prompting. mT5 outperformed XGLM for most low-resource language pairs and even outperformed BLOOM on hin-+mal, hin-tguj. and hin-+tel pairs, underscor- ing its effectiveness as robust baseline.
PBLEU Signature: mrffx:llassemmiieeleffnood htt.leeseDsssoooteeppeessrrraoo 'chrF++ Signature: nrefs: Iessemmicdeleefffsseeeee nww.2lspacemooverrion2...11

---

spBLEU
chrl++
SP
SAP
DecoMT
SP
SAP
DecoMT
mi1
m11
hin-+mal
3,0
0.0
10.7
17.6
18.7
15.7
0.1
23.2
34.3
37.0
mal -hii
10.6
0.0
8.9
14.9
16.3
29.3
0.0
24.8
34.2
36.8
hin -mar
11.7
0.0
7.2
12.5
13.9
30.8
2.8
22.4
32.1
35.6
mar-hhin
19.7
0.0
13.5
19.5
21.0
39.9
4.9
31.3
39.6
41.9
hin gujj
6.8
0.0
15.3
21.4
22.0
26.2
0.1
30.9
39.2
41.1
guj-+hin
20.8
0.0
16.2
22.5
23.2
40.6
3.1
34.0
42.2
43.7
hin-ttll
3.5
0.3
9.2
19.31
19.5
19.9
1.6
24.0
37.2
38.5
lel->hin
9.2
12.9
9.6
16.6
17.8
28.7
30.6
26.2
35.9
38.6
zsm-=ind
0.0
18.1
28.7
29.6
7.4
40.8
53.9
55.9
>zmm
0.0
14.9
26.9
28.2
3.4
37.2
53.1
54.5
nus-tukr
30.1
31.0
36.4
49.9
ukr->rus
23.0
32.3
40.7
34.6
49.1
51.5
spa spa->por
29.1
28.3
13.6
27.6
26.5
51.5
49.4
32.0
48.4
50.0
por >spa
28.2
26.0
13.4
24.8
26.3
50.1
48.2
33.1
46.4
48.9
Table The table presents spBLEU and chrF+++ scores for standard prompting (SP) with BLOOM XGLM, SAP with mT5, and proposed DecoMT approach with m'5 across several language pairs, all tested on the FLORES devtest set. The highest performing results are highlighted in bold, and the second best scores are underlined for clarity. All comparisons with DecoMT demonstrate statistical significance (p 0.05) (except results marked with 1) per paired bootstrap sampling (Koehn, 2004).
SAP proved to strong approach, echoing the findings of Patel et al. (2023). It outperformed Stan- dard Prompting with BLOOM, XGLM and mT5 on the hin+-mal, min+ hin+-guj, hin+ttel, indo >zsm, and sukr language pairs. Never- theless, BLOOM outperformed SAP for the high- resource spr+>por pair.
hension skills in the source language and demon- strated fluency in the target language. These an- notators were remunerated alignment with lo- cal hourly wage standards. The language pairs hinę >ar,, hink hin<>>guj >guj, zsm zsm->ind, and por->spa >spa were selected for evaluation, contingent upon the avail- ability of annotators well-suited for each pair. It should be noted that only single annotator assigned each language pair. We sampled 50 sentences for each approach for a total of 200.
Lastly, DecoMT surpassed all other approaches on the low-resource language pairs min++ma, hine>man hinj>gujj. hin< hte, ind >zsm, and ruse>ukr. While it also achieved impressive results with the high-resource spat>> por pair, it fell short of BLOOM's performance in this particular scenario. It's worth noting that DecoMT demonstrated an average improvement of 13.8 points in the chrl+++ score over Standard Prompting with mT5, which presents a more direct comparison for DecoMT due to the same base model and their similar prompting and inference strategies.
Our human evaluation strategy employs the Cross-Lingual Semantic Textual Similarity (XSTS) methodology (Licht et al., 2022) adopted by NLLB (Team et al., 2022) and IndicTrans2 (A14Bharat et al., 2023). Within this approach, annotators presented with the source sentence alongside trans- lations produced by various approaches, omitting any auman-annotated references. XSTS empha- sizes translation adequacy over fluency, well- suited to our focus on translation between related, typically low-resource languages, where adequacy takes precedence.
5.2 Human Evaluation
To further analyze the quality of the outputs and val- idate the enhancements indicated by the automatic evaluation scores, we carry out a human evaluation study. This involves a comparative examination of our DecoMT approach, SAP, and Standard Prompt- ing with mT5 and BLOOM.
The XSTS metric is composed of scale ranging from to 5, where score of signifies completely dissimilar sentence pairs and a score of represents semantically identical sentences. Appendix con-
tains details of the score values.
We engaged annotators who possessed compre-
As shown in Table 2, DecoMT significantly out-

---

SAP
DecoMT
tion in the second stage of DecoMT significantly improves performance. We report the improvement in chrF+++ scores Table 3. The improvement
BLOOM
m11
wrs
hin mar
2.4
1.9
2
3.0
-hin
3.4
2.34
3.4
3.6
spBLEU is presented in Appendix E.
hin hin-
2,0"
2.1
3.4
3.2
guj +hin
3.0
2.1
3.3
3.6
ind
1.0"
3.4
4.8
4.9
Lang Pair
chrF+++ (Single Stage)
chrP++
por >spa
4.7
2.5
4.5
hin->mal
33.7
+3.3
mal-shin
34.7
+2.1
Table 2: Human evaluation scores for standard prompt- ing (SP) with BLOOM and XGLM, SAP with mT5, and our proposed DecoMT approach with m'15, Results marked with * indicate statistically significant differ- ence (p < 0.05) from DecoMT using ANOVA with post-hoc Tukey HSD test.
hin-2mar
33.1
+2.5
mar-shin
39.6
+2.3
hin->guj guj->hin
39.6
+1.5
41.8
+1.9
hin->tel
36.3
+2,2
tel->hin
35.7
+2,9
zsm->ind
53.8
+2.1
performs Standard Prompting with mT5 across all language pairs. DecoMT is significantly better than BLOOM for hin->mar, hin++guj and ind-tzsm but comparable with BLOOM mar-thin and por-+spa. DecoMT is significantly better than SAP for hin mar, while demonstrating comparable per- formance for the remaining language pairs.
ind->2sm
54.3
+0,2
rus->ukr
48.5
+1.4
ukr->rus
+1.6
spa->por
48.7
+1.3
por->spa
47.1
+1.8
Table 3: Improvement in chrl++ scores gained by the DecoMT approach compared to the Single Stage.
6 Diseussion
Off-target Translations To quantify the off- target translation rate among various approach's outputs, we employed the Language Identification tool developed by the NLLBB (Team al., 2022). The off-target translation rate is represented as a percentage, with a lower percentage denoting supe- rior performance, as shown in Table 4. We see that the DecoMT approach consistently outperforms other approaches with lower off-target translation rate across various translation tasks. We conduct further analysis in Appendix
Scores of Translation across different Sentence Lengths The DecoMT strategy involves trans- lating source sentences in consecutive chunks, a method we hypothesize will lead to enhanced trans- lation adequacy. To explore this, we group source sentences into length-basc based buckets, each with width equivalent to the standard deviation of the source sentence lengths. If bucket contains fewer than 20 instances, we merge with its neighbour.
Figure depicts the relationship between source sentence length and chrF+++ scores for the hin hmnl and zsm-tind language pairs. As hypothesized, as the length of the source sentence increases, the per- formance of DecoMT, measured by chrF++, im- proves. For the zsm ism+iind language pair, the chrl+++ scores of DecoMT and SAP are nearly identical for the first two buckets. However, as we move to the next three buckets with longer sentences, we ob- serve a steady increase in DecoMT's chrF++ scores. This is in contrast with the declining scores of SAP, highlighting DecoMT's superiority in translating longer sentences.
Extension to Autoregressive and other Encoder- Decoder LLMs At present, utilize mTT5 for both independent and contextual translations. How- ever, it's worth noting that any autoregressive LLM could potentially be used for independent trans- lation. As for contextual translation, an autore- gressive LLM could be prompted with a fill-in- the-blanks type of prompt avenue we intend to explore in future work. Additionally, the explo- ration of other encoder-decoder LLMs such UI.2 (Tay al., 2023) or AlexaTM (Soltan al., 2022) for contextual translations presents a promising re- search direction.
Improvement by Adding the Contextual Trans- lation Compared to the Independent Transla- tion We compared the single-stage independent translation to the two-stage DecoMT. The experi- ments show that the inclusion of contextual transla-
Experiments with Zero-shot and One-shot Prompting We undertook zero-shot translation experiments for select language pairs, specifically

---

hin, Deva mai Mlym
LaTh ind Lath
60
-0-
(each is chat lqual is standaard desixiion
(each buccel accul to the standaar doration
Figure 5: The plots show the relationship between source sentence length and chrF++ scores for hin- and zsm-xind pairs. Lengths are bucketed, each equal to the sentence lengths' standard deviation, with any bucket with less than 20 sentences merged with neighbour. The data implies DecoMT's chrF++ scores outperform SAP's with increasing sentence length, indicating DecoMT's proficiency with longer sentences.
SAP
DecoMT
Both SAP and SP experience significant per- formance drops transitioning from five-shot one-shot. For instance, the spBLEU score for hin->tel in SAP drops from 19.3 (five-shot) to just 1.3 (one-shot).
BL.0M
XGLM
mT5
mTS
hin-*mal
23.6
100.0
14.4
0.4
0.0
mal-+hin
8.4
0.0
4.4
1.4
0.2
hin->mar
21.2
96.3
35.2
10.0
0.8
mar->hin
1.3
20.0
2.6
1.1
0.2
Inference Times As highlighted in Patel et al. (2023), to generate sentence comprising 7' words, SAP necessitates T' forward passes through the model. This approach stands in contrast to Stan- dard Prompting, which only requires single pass. In the case of DecoMT, the independent translation stage can be parallelized with relative ease. For the contextual translation stage, T/m forward passes through the model are needed, where m denotes the chunk size. As a result, the inference time for DecoMT is less than that of SAP. Appendix H contains more details of runtime analysis.
hin Lijj
10.2
99.7
3.8
0.2
0.0
guj->hin
3.3
0.0
1.9
0.4
0.2
z5m ssm->ind ind +2sm
48.8
23.3
17.7
13.1
94.2
59.7
47.3
30.1
rus rus->ukr ukr-Armn
84.3
1.7
0.0
0.5
0.0
spa-apor 0.2
0.4
3.4
0.9
0.2
por por-
0.0
0.5
0.6
0.3
0.1
Table 4: The percentage of sentences off-target for a translation direction. Lower is better.
hin<->guj, hin<->tel, and hin<->mal. We com- pared different approaches applied to mT55 includ- ing DecoMT, SAP and Standard Prompting. We found that all approaches yielded near-zero BLEU scores. In most instances, the models merely copied the input as the output. We hypothesize that this because in zero-shot setting the model may not understand that has to perform transla- tion to the target language.
7 Conclusion
In this study, we introduced DecoMT, novel ap- proach using decomposed prompting for Machine Translation of related languages. DecoMT demon- strated superior performance over established few- shot prompting baselines in translating between low-resource related languages, as evidenced by our experiments with the FLORES dataset. Addi- tionally, DecoMT showed robust performance even in high-resource scenarios.
We compared one-shot and five-shot settings for three language pairs (hin<->guj, hin<->tel and minl using Standard Prompting (SP), SAP, and DecoMT with mTS. Our results in Appendix G indicate that:
Limitations
Despite its advantages, DecoMT does possess cer- tain limitations. Notably, the approach requires hu- man annotation for constructing the five example- aligned prompts in the template. However, our
DecoMT maintains strong performance even in the one-shot setting.

---

observations suggest that the annotators primar- ily need to modify existing translations, which is less laborious than generating translations from scratch, an activity that can be done in under 30 minutes. Conversely, other baseline approaches don't require such annotation and are able to di- rectly utilize translation examples.
A14Bharat, Jay Gala, Pranjal A. Chitale, Raghavan AK. Sumanth Doddapaneni, Varun Gumma, Aswanth Ku- mar, Janki Nawale, Anupama Sujatha, Ratish Pudup- pully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. Indictrans2: Towards high-quality and accessible ma- chine translation models for all 22 scheduled indian languages.
When considering the translation time, DecoMT, given its two-stage process encompassing inde- pendent and contextual translations, inherently re- quires a longer duration to generate outputs com- pared to traditional few-shot prompting methodolo- gies.
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2021. A survey of multilingual neural machine trans- lation. ACM Comput. Surv., 13155:99:1199:388
Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh Khapra, and Pratyush Ku- mar. 2022. IndicBART: A pre-trained model for indic natural language generation. In Findings of the As- sociation for Computational Linguisties: ACL 2022, pages 1849-1863. Dublin, Ireland. Association for Computational Linguistics.
Another limitation of DecoMT is its dependency on an LM with infixing capabilities during the con- textual translation stage. In the absence of infixing capabilities. this can be simulated on other LLM with appropriate prompting, and we plan to explore that in future work.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed Kl-Kishky. Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary. Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Ar- mand Joulin. 2021. Beyond english-centric multilin- gual machine translation. Journal of Machine Learn- ing Research, 22(107):
Ethics Statement
This study does not involve any new data collection. We solely utilize publicly accessible datasets for conducting the experiments reported herein. Fur- thermore, for the purpose of annotation of trans- lation examples and the human evaluation of ma- chine translation outputs, we employ annotators who are duly compensated for their time and exper- tise, ensuring fair practices in line with established standards.
Xavier Garcia, Yamini Bansal, Colin Cherry, George F. Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat, 2023. The unreasonable effectiveness of few- shot learning for machine translation. In Interna- tional Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10867-10878. PMLR.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-538.
Acknowledgements
We would like to thank the reviewers for their feed- back. This research was supported by funding from the Institute for Infocomm Research (I2R) under A A'STAR ARES, Singapore. We extend our grat- itude to Litton Kurisinkel, Aswanth Kumar, Siti Umairah, Ivan Kukanov, Swapnali Waghunde, and Fabian Gutterrettierezzztte for their work in annotating the few-shot prompts. Additionally, we'd like to Kunal
Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindrich Helel, and Alexandra Birch. 2022. Survey of low-resource machine translation. Comput. Linguistics, 4873)667.
thank Siti Umairah, Fabian Gandhi, and Faiz Masi for their contributions to the human evaluation experiments.
Wenxiang Jiao, Wenxuan Wang. Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks.
References
Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. In- context examples selection for machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8857-8873, Toronto, Canada, Association for Computational Linguistics.
Philipp Koehn. 2004, Statistical significance tests for machine tramslation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona. Spain, Association for Computational Linguistics.

---

Aswanth Kumar, Anoop Kunchukuttan, Ratish Pudup- pully, and Raj Dabre. 2023. In-context example selec- tion for machine translation using multiple features.
Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj. Mayank Jobanputra, Raghavan AK. Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Ma- halakshmi J. Divyanshu Kakwani, Navnect Kumar, Aswin Pradeep. Srihari Nagaraj. Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Ku- mar, and Mitesh Shantadevi Khapra. 2022. Samanan- tar: The largest publicly available parallel corpora collection for 11 indic languages. Transactions of the Association for Computational Linguistics, 10:145. 162.
Daniel Licht, Cynthia Gao, Janice Lam, Francisco Guz- man, Mona Diab, and Philipp Kochn. 2022. Consis- tent human evaluation of machine translation across language pairs. In Proceedings of the 15th biennial conference of the Association for Machine Transla- tion in the Americas (Volume 1: Research Track). pages 309-321, Orlando. USA. Association for Ma- chine Translation in the Americas.
Salch Soltan, Shankar Ananthakrishnan, Jack FitzGer- ald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris. Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Srid- har, Fabian Triefenbach, Apury Verma, Gokhan Tur, and Prem Natarajan. 2022. Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang. Shuohui Chen. Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary. Brian O' Horo, Jeff Wang, Luke Zettle- moyer, Zornitsa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019-9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yi Tay, Mostafa Dehghami, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. U12: Unifying language learning paradigms.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
NLLB Team, Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad. Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran. Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko. Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang- 2022. No language left behind: Scaling human- centered machine translation.
Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison- Burch. 2023. Bidirectional language models are also sew-shot learners, In The Eleventh International Con- ference on Learning Representations.
Maja Popović. 2017. chrF++: words helping charac- ter n-grams. In Proceedings of the Second Confer- ence on Machine Translation, pages 612-618, Copen- hagen, Denmark. Association for Computational Lin- guistics.
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023. Prompt- ing PaLM for translation: Assessing strategies and performance. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 15406- 15427, Toronto, Canada. Association for Computa- tional Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguisties.
Ofir Press and Noah A. Smith. 2018 You may not need attention. CoRR, 1t/18100.13400
BigSeience Workshop. Teven Le Scao, Angela Fan, et al. 2023. Bloom: A 176b-parameter open-access multi- lingual language model.
Ratish Puduppully, Yao Fu, and Mirella Lapata. 2022. Data-to-text generation with variational sequential planning. Transactions of the Association for Com- putational Lingustics, 10:697-715
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach, Learn Res,, 21:140: 1-14677

---

Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine transla- tion: A case study. In Proceedings of the 40th Inter- national Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 41092-41 10. PMLR.
Translate from Malay to Indonesian:
Malay: Saintis Stamford Universiti Sekolah Indonesian: Imuwaa dari Stanford University School of Malay: Perubatan pada hari Isnin
Indonesian: Medicine pada hari Senin
Malay: mengumumkan penemuan alat diagnostik baharu Indonesian: mengumumkan penemuan alat diagnostik baru
Malay: yang bolch menyusun sel herdasarkan
Indonesian: yang bisa mengurutkan sel berdasarkan
Malay: jenis: cip kecil dapat dicetak
Indonesian: tipe: cip kecil dapat dicetak
Malay: yang bolch dihasilkan menggunakan printer
Indonesian: yang bisa diproduksi menggunakan printer
Malay: inkjet standard dengan kos sekitar
Indonesian: inkjet standar dengan biaya sekitar
Malay: satu sen AS se cip.
Indonesian: satu sen AS per cip.
Translate from Malay to Indonesian:
Malay: Ketua penyelidik mengatakan bahawa diagnosis Indonesian: Ketua peneliti mengatakan bahwa diagnosis
Malay: ini mungkin dapat menghasilkan pengesanan
Indonesian: ini mungkin dapat menghasilkan deteksi
Malay: awal kanser, tuherkulosis, HIV. dan Indonesian: dini kanker, tuberkulosis, HIV, dan Malay: malaria kepada pesakit- pesakit-pesakit di negara Indonesian: malaria kepada pasien pastenasinnnn di negara Malay: berpendapatan rendah, di mana kadar Indonesian: berpenghasilan rendah, di mana tingkat kesembuhan dari ponyakit-penyakee seperti kanser Indonesian: kesembuhan dari penyakit- penyakit-penyaki penyakit seperti kanker
Malay:
Malay: payudara bolch mencapai setengah dari
Indonesian: payudara bisa mencapai setengah dari negara- negara kaya.
Malay:
Indonesian: negara negara-negara negara kaya.
Translate from Malay to Indonesian:
Malay: 39C Gripen terhempas ke Indonesian: JAS 39C Gripen jatuh ke
Malay: landasan sekitar jam 9:30
A Examples of Prompts
Indonesian: landasan pacu sekitar pukul 9:30
Malay: waktu tempatan (0230 UTC) dan
Indonesian: waktu setempat (0230 UTC) dan
Malay: meletup, mengakibatkan ditutup lapangan terbang Indonesian: meledak, menyebabkan ditutupnya bandara Malay: untuk penerbangan komensial.
Indonesian: untuk penerbangan komersial.
Translate from Malay to Indonesian:
Malay: Juruterbang tersebut dikenalpasti sebugai Ketua Indonesian: Pilot tersebut diidentifikasi schagai Pemimpin Malay: Pasukan Dilokrit Pattavee.
Indonesian:
Skuadron Dilokrit Pattavee.
Translate from Malay to Indonesian:
Malay: Media tempatan melaporkan sebuah kenderaan
Indonesian: Media lokal melaporkan sebuah kendaraan Malay: pemadam api di lapangan terbang tergolek
Indonesian: pemadam api di bandara terguling Malay: ketika dikendalikan.
The prompts used for independent and contex- tual translations by DecoMT for the language pair Malay-+Indonesian are presented in Table and Table 6, respectively. Meanwhile, Table 7 illus- trates the prompts utilized for Standard Prompting and SAP.
Indonesian: sedang dioperasikan.
Translate from Malay to Indonesian:
Table 5: Prompt for Independent translation in De from Malay to Indonesian

---

Translate from Malay to Indonesian:
Translate from Malay to Indonesian:
Malay: Saintis dari Stamford Universiti Sekolah Peru- batan pada hari Isnin
Malay: Pada hari Ismin, Saintis daripada Sekolah Pe- rubatan Universiti Stamford mengumumkan penemuan alat diagnostik baru yang bolch mengasingkan sel-sel mengikut jenis: cip kecil yang bolch dicetak yang holch dihasilakn menggunakan pencetak standard inkjet untuk kira-kira satu sen A.S setiap satu.
Indonesian: Ilmuwan dari Stanford University School of Medicine pada hari Senin
Malay; mengumumkan penemuan alat diagnostik baharu yang bolch menyusun sel berdasarkan
Indonesian; mengumumkan penemuan alat diagnostik baru yang bisa mengurutkan sel herdasarkan
Indonesian: Ilmuwan dari Stanford University School of Medicine pada hari Senin mengumumkan penemuan alat diagnostik baru yang bisa mengurutkan sel berdasarkan tipe: cip kecil dapat dicetak yang bisa diproduksi meng- gunakan printer inkjet standar dengan biaya sekitar satu sen AS per cip.
Malay: jenis: cip kecil dapat dicetak yang boleh dihasilkan menggunakan printer
Indonesian: tipe: cip kecil dapat dicetak yang bisa dipro- duksi menggunakan printer
Malay: inkjet standard dengan kos sekitar satu sen AS se cip.
Translate from Malay to Indonesian:
Indonesian: inkjet standar dengan biaya sekitar satu sen AS per cip.
Malay: Penyclidik utama mengatakan bahawa ia mungkin menghasilkan pengesanan awal kanser, tuberkulosis, HIV dan malaria kepada pesakit di negara-negara berpendap- atan rendah, di mana kadar kemandirian untuk penyakit seperti kanser payu dara ialah separuh daripada di negara- negara yang lebih kaya.
Translate from Malay to Indonesian:
Malay: Ketua penyelidik mengatakan bahawa diagnosis ini mungkin dapat menghasilkan pengesanan Indonesian: Ketua peneliti mengatakan bahwa diagnosis ini mungkin dapat menghasilkan deteksi
Ketua peneliti mengatakan bahwa diagno- sis ini mungkin dapat menghasilkan deteksi dini kanker, tuberkulosis, HIV, dan malaria kepada pasien-pasinn di ne- gara herpenghasilan rendah, di mana tingkat kesembuhan dari penyakit-penyaki seperti kanker payudara bisa men- capai setengah dari vegara-negane kaya.
Indonesian:
Malay: awal kanser, tuberkulosis, HIV, dan malaria pesakit di negara
kepada pesakit-pesakit
Indonesian: dini kanker, tuberkulosis, HIV, dan malaria kepada pasien-pant pasienn--aiinn di negara
Malay: berpendapatan rendah, di mana kadar kesembuhan dari penyakit-penyee seperti kanser
Translate from Malay to Indonesian:
Indonesian: berpenghasilan rendah, di mana tingkat ke- sembuhan dari aaaat-keny-akiii seperti kanker
Malay: JAS 39C Gripen telah terhempas ke atas landasan sekitar jam 9:30 pagi waktu tempatan (0230 UTC) dan meletup, mengakibatkan lapangan terbang ditutup bagi penerhangan komersial.
Malay: payudara boleh mencupai setengah dari negaru- negara kaya.
Indonesian: payudara bisa mencapai setengah dari negura- negara kaya.
Indonesian: JAS 39C Gripen jatuh ke landasan pacu seki- tar pukul 9.30 waktu setempat (0230 UTC) dan meledak, menyebabkan ditutupnya bandara untuk penerbangan komersial.
Translate from Malay Indonesian:
Malay: JAS 39C Gripen terhempas ke landasan sekitar jam 9:30
Translate from Malay to Indonesian:
Indonesian: JAS 39C Gripen jatuh ke landasan pacu seki- tar pukul 9:30
Malay: Juruterbang telah dikenal pasti sebagai Ketua Pa- sukan Dilokrit Pattavee.
Malay: waktu tempatan (0230 UTC) dan meletup, men- gakibatkan ditutup lapangan terbang
Indonesian: Pilot tersebut diidentifikasi schagai Pemimpin Skuadron Dilokrit Pattavee.
Indonesian: waktu setempat (0230 UTC) dan meledak, menyebabkan ditutupnya bandara
Translate from Malay to Indonesian:
Malay: untuk penerbangan komersial, Indonesian: untuk penerbangan komersial.
Malay: Media tempatan melaporkan kenderaan api lapan- gan terbang terguling ketika memberi maklum balas. Indonesian: Media lokal melaporkan sebuah kendaruan pe- madam api di bandara terguling saat sedang dioperasikan.
Translate from Malay to Indonesian:
Malay: Juruterbang tersebut dikenalpusti sebagai Ketua
Pasukan Dilokrit Pattavec. Indonesian: Pilot tersebut diidentifikasi scbagai Pemimpin Skuadron Dilokrit Pattavee.
Translate from Malay to Indonesian:
Table 7: Prompt for Standard Prompting and SAP from Malay to Indonesian
Translate from Malay to Indonesian:
Malay: Media tempatan melaporkan sebuah kenderaan
pemadam api di lapangan terbang tergolek Indonesian: Media lokal melaporkan sebuah kendaraan pemadam api di bandara terguling
B Example for DecoMT
Malay: ketika dikendalikan. Indonesian: saat sedang dioperasikan.
Figure 6 presents a block diagram which explains DecoMT with the help of an example. The task at hand is translation from Hindi to Malayalam. The Hindi sentence is divided into four consecutive chunks: H1, H2, H3. and H4, each consisting of m = tokens. Using few-shot prompting, these chunks are independently translated into Malay- alam, resulting in M,, M2, M3. and M4. However,
Translate from Malay Indonesian:
Table 6: Prompt for Contextual translation in DecoMT from Malay to Indonesian

---

laenn & fee Mhe kastalaan kee rasjathaance (baarsiona) mein jaane ke baad se, vidaal ne klab ke 49 gem khele the. Since moving to the Cotalan capital (Barcelona), Vidal played 49 games for the club.
Mhstesel (Kff@tebet)
kastalaan kee rasjadhaanee
H.
Catalanns capiaal
(4118 H amri # <id (baarsilona) mein jaane ke baad after moving (barcelona)
(mnddm]leeeccm)) milcod a rommmirr aue-ats (Dsarsilona) poyathinu sheshem going (barcelons)
Hi+Hl <masko+M2
साननतात dt tratucct (en(itttt)) at अार aie. bnaaskossuormmlleee allaé dudomamlt auóvédo
kastialooneyude Thaleshhanemaaya capiaal
esopeesermaggg meimunomago
R1
After moving to the Catalan capital (Barcelona)
T:
स,, (िssei se, vidaal ne klab
agcascó, Callentt spaggii
Ho
Anotnal, videal clubinu
Mi
since. Vidal has for club
HitHiHHi Ri+smasko+
माानcाग iettri ( and tn mrz fourer were aageeneeaage meirunomapop <mask> agmmoo, ailaooi genglin Since moving the Catalan capital (Saroalona), Vidal has the club
(seuoer) wlseued e-peggmaf
(baarsilora) yilekku pokunnathu
(Barcelona)
R:
टितर 49 ior बrt it. gem the, for pleyed 49 games.
workknaral 44r Al- avdll aid ithaayy kalkal kalichu. played 49 games.
स,, fasta in five सेι R
HitHitHi Ransmasko- M.
(ouo@@mkkeereen) allaued duidemmee smaeko goslmool 49 æuglav dواla)) agaev, culearn eyeail
aggrvn,, oileooé qerrl muthal, videal clab Since, Vidal Club
R.
Since moving (Barcelons), Vidal had played 49 the club.
8, Fiesrrt verr fिre 49 सhe Q;; agavar, rálesoot qenn smask>
mouil 49 uvtrussemmu avell a/gaamii neayi malsarangal kalichitundu.. has played 49 matches
Ri-mmakk>
Since, Vidal has played 49 for the club.
R
Figure 6: This diagram provides step-by-step illustration of the DecoMT process. For the sake of simplifying our explanation, we have excluded the prompt template from block diagram. The chunks of Hindi input, represented as H,. H2. H3. and H4. are initially translated into Malayalam independently using few-shot prompting, resulting in Mj, M2, M3. and M4. Subsequently, infilling is used to derive contextual translations, denoted as R1, Rg. R3. and R₁. Each block of H;, M, and R; presents three lines: the original text, its English transliteration, and its translation into English. The blocks marked T; illustrate the contextual translation tasks. The input block for T; includes a concatenation of input chunks, the previous contextual translation, a mask placeholder, and an independent translation, along with their Finglish translation. The final translation into Malayalam, is produced by piecing together the contextual translations R,, R,, R3, and R,, It should be noted that the English translations and transliterations are included for sake clarity and are not an integral part DecoMT process.
we observe that these translated chunks can occa- sionally lack coherence.
The model then infills the mask, producing a con- textually appropriate translation of Mj, which we denote R
For instance, consider the translation of the H4 chunk. The chunk commences with रिर which can translate to 'reason' or 'for' (indicating possession) in English. The M4 translation into Malayalam, T adopts the former meaning, whereas the sentence context implies that the latter interpre- tation would be more suitable.
Next, we feed a concatenation of H,, H2, H3. along with concatenation of R1, mask place- holder, and M3 into the mT5 model. The result is contextually appropriate translation, R2, of M2.
This procedure is repeated for all the intermedi- chunks. For the final chunk, we input a concate- nation of H3, H4, R3, and mask placeholder. The mT5 model then predicts the contextually appro- priate translation, R4, of the M, translation. Given the context of Hg. H,, and Rg, the contextual trans-
To rectify this, we introduce process gener- ate contextually appropriate translations. We input a concatenation H₁, H2, and a mask placcholder, along with Mz, into the bidirectional m55 model.

---

Language pair m
there may be differences in important informa- tion related to the primary subject/verllobject.
hin -aal
5
mall -hil
3
A score of 3 indicates that the sentences mostly similar, but there may be some minor omissions of unimportant information. There should not be any significant conflict in the information.
hin >mar 5
mar mar->hin >hin
4
hin hin-tguj
5
guj-thin
4
hin->tel
tel->hn
score of 4 indicates that the sentences are paraphrases of each other. There are no major differences or missing information, although there may be variations in expression such as tone, style, emphasis, or formality.
7sm-+ind
ind-+zsm
nis-tukr 4
ukr->rus 4
por-tspa 4
spa->por 4
A score of 5 indicates that the sentences are completely equivalent in meaning and usage, including expression aspects such as formality, tones, style, and emphasis.
Table 8: Optimum value of m found through hyperpa- rameter search in (3,4.5).
lation correctly interprets the intended meaning.
For more details and examples, see Licht et al. (2022).
Hyperparameter m
E Improvement by Adding the Contextual Translation Compared to the Independent Translation
The optimum value of m for different language pairs is presented in Table 8. We posit that the optimal value of is contingent the relative morphological complexity of the source language. Take the example of hine>mal. Since Hindi (hin) is less morphologically complex than Malayalam (mal), a larger number tokens are required in a chunk for hin-tmal than for mal-+hin to produce satisfactory outputs in the independent translation stage.
Table showcases the improvements in spBLEU scores achieved by the DecoMT approach in com- parison to the Single Stage method.
Lang Pair
spBLEU (Single Stage)
spBLEU
hin->mal mal-shin
15.9
13.3
+3.0
In the of zsm++>ind both languages exhibit similar morphological complexity, resulting in identical optimum value of m, which is 4. The same applies to the rust-xukr and spa<+tpor pairs. For these three pairs, a value of m smaller than 4 results in subpar independent translation qual- ity. Conversely, value exceeding 4 might lead to truncated translations.
hin->mar
12.1
+1.8
mar->hin
17.0
+4.0
hin->guj
20.2
+1.8
guj->hin
21.0
+2.2
hin->tel
16.7
+2.8
tel->hin
11.3
+6.5
zsm->ind
26.4
+3.2
ind->zsm
27.7
+0.5
rus->ukr
28.3
D Details of Human Annotation Guidelines
ukr->rus
spa->por
24.4
+2.1
por->spa 23.7
+2.6
The XSTS metric provides ratings between and representing different levels similarity between sentences.
Table 9: Improvement in spBLEU scores gained by the DecoMT approach compared to the Single Stage.
A score of indicates that the sentences share little content may be about different topics. If they share content, it is less than 50
F Off-target Translations
In Table 4, focusing on the relatively high off- target translation rate for ind++zsm, particularly for ind ind- we analyzed 50 mislabeled DecoMT
A score indicates that the sentences are about similar topics but are not equivalent, and

---

spBLEU
chrl++
Language Pair
SP
SAP
DecoMT
SP
SAP
DecoMT
m11
m13
hin->guj one-shot
10.0
16,1
22.2
20.5
29.6
41.1
hin->guj five-shots
15.3
21.4
22.0
30.9
39.2
41.1
guj->hin one-shot
17.1
22.9
23.0
34.7
42.7
43.4
guj->hin five-shots
16.2
22.5
23.2
34.0
42.2
43.7
hin->tel one-shot
0,4
18.9
2.8
38.2
hin->tel five-shots
9.2
19.3
19.5
24.0
37.2
38.5
tel-shin one-shot
5.3
8.7
17.4
12.6
18.6
38.4
tel-shin five-shots
9.6
16.6
17.8
26.2
35.9
38.6
hin->mal one-shot
1.3
2.9
18.2
3.2
5.7
36.7
hin->mal five-shots
10.7
17.6
18.7
23.2
34.3
37.0
mal-shin one-shot
9.1
13.4
16.5
22.7
29.8
36.9
mal->hin five-shots
8.9
14.9
16.3
24.8
34.2
36.8
Table 10: Comparison of one-shot and five-shot translation results across three language pairs using SP, SAP, and DecoMT with mT5. Notably, DecoMT exhibits robust performance in one-shot settings, whereas SP and SAP show marked performance reductions, exemplified by the spBLEU drop for hin->tel in SAP from 19.3 (five-shot) to 1.3 (one-shot).
output sentences from ind >7sm. annotator from our human evaluation study (Section 5.2) found that 64% these sentences were in fact Malay, not Indonesian. This suggests potential shortcomings in automatic language identification for closely related languages such as ind and zsm.
SAP Analysis: For the SAP system, due to the unpredictability the expected target length, we do decoding at 1.5 times the maximum source length. This is based on our studies of lengths of examples from validation dataset. For example, for our given source batch, the reference Hindi translation encompasses 55
G Comparison between One-shot and Five-shot Prompting
tokens for the Marathi sentence which is 41
tokens long. As the longest example 41 tokens, we run inference for 1.5 61 steps. Table contains partial trace of per- formance profiling using eprofile. We see that for SAP, there are 61 calls to predict_outpin method. The method predict_outiee output is respon- sible for rumning inference the LLM. Each method takes 2.384 seconds. The inference of the batch takes 145,455 seconds.
As detailed in Table 10, our evaluations span three language pairs and compare the efficacy of Stan- dard Prompting (SP), SAP, and DecoMT method- ologies when evaluated on mT5. In comparison between one-shot and five-shot scenarios, we find that DecoMT consistently demonstrates strong per- formance in one-shot settings, in contrast to the pronounced performance dips observed for both SP and SAP.
ncalls
cumtime
percall 145.455
filenanc-lienooffunttion (built-in method builtins. Nuiltins.exec)
3/1
145.455
H Analysis of Runtime
145.428
2.384
163 (predict_output)
To ensure fair comparison, we profile the codes using eprofile during the inference phase, exe- cuted on an A40 48GB GPU, eprofile examines the time taken by various API calls. In this case, our chosen task is translating from Marathi to Hindi using the initial batch of 5 examples from the FLO- RES test set, with the longest Marathi sample in the batch being 41 tokens long.
Table 11: Performance Profiling Data for SAP
DecoMT Analysis: For Marathi Hindi trans- chunk size of 4. We first sider the independent translation stage. Break- ing down the sentence lengths of the batch tokens: 16, 30, 24, 41, and 28, we get respec- tive chunk counts of 4, 8, 6, 11, and 7 7-ag-
lations,
httts::/ddos..python.oo pytppn.toooooo www.pho..or/////irrrrr/oooooooo
html

---

gregating 36 chunks. Split into batches of 8, this leads to 5 API calls predict_orr _otpput. With the longest sentence in the batch hav- ing 41 tokens, the contextual translation stage demands 11 API calls to predict_outpint cu- mulating to 16 calls. These 16 api calls in total amount to 96.868 seconds (Table 12). While predict_ in DecoMT tends to take longer than in SAP (owing to DecoMTT pre- dicting multiple tokens as opposed to SAP's single-token approach), the overall fewer API calls render DecoMT more efficient.
ncalls
cumtime
percall 96.883
fiename:/ineno(uuccion)) (built-in method builtins.exec)
3/1
96.883
16
96.868
6.054
decomt. decomt.py:199 19: (predict_output)
Table 12: Performance Profiling Data for DecoMT


Reviews:
[{"review_id": "jp9GwsT7vm", "reviewer": "Reviewer_F6Qp", "summary": "", "questions": "- How is the input chunked to ensure that the chunks can be translated monotonically?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, particularly for appreciating our evaluation on multiple language pairs from different language families. We address your concerns below.\n\n**Q1. The innovation is not enough for an EMNLP full-length paper**\n\nWe've introduced Decomposed Prompting for Machine Translation (DecoMT).\n* Incremental Translation: Unlike conventional methods using autoregressive prompting of LLMs like BLOOM and XGLM that process entire sentences, DecoMT translates word chunks incrementally. This approach yields an 8 chrF++ improvement over BLOOM across language pairs.\n* Monotonic Alignment: Current few-shot prompting methods neglect intrinsic traits of related languages. DecoMT addresses this by focusing on inherent monotonic alignment, improving translation accuracy.\n* Focus on Less-explored Language Pairs: We've experimented with less-explored language pairs such as hindi<->malayalam, indonesian<->malay, and ukrainian<->russian.\n* Bidirectional Context: Distinct from Sequential Autoregressive Prompting (SAP) from Patel et al. (2023), DecoMT uses bidirectional context in translation, as referenced in lines 139-143.\n\nEmpirically, our evaluations cover a wide range of language pairs. Through both automatic and human evaluation, we consistently outperform established baselines. Additionally, our analysis reveals that DecoMT achieves higher chrF++ scores than SAP as source sentence length increases.\n\n**Q2: The method has an inefficiency,  since it required multiple inference calls…**\n\nWe acknowledge DecoMT's inefficiency compared to standard autoregressive prompting based approaches, as noted in lines 557-562 in the Limitations section. However, it's pertinent to note that DecoMT still stands as more efficient than SAP, as detailed in lines 524-534. \nMost LLMs have limited representation of the majority of non-English languages, hence it is imperative to introduce a specific inductive bias. In our case, the inductive bias is that of monotonicity between related languages. The inductive bias we chose significantly improves translation quality, which we believe justifies the increased cost.\n\n**Q3: How is the input chunked to ensure that the chunks can be translated monotonically?**\n\nDuring the annotation of the 5 examples, annotators were instructed to maintain a monotonic alignment between the source and target. Take, for instance, the translation from Marathi to Hindi. In DecoMT's first stage, there were 33 chunks spanning the 5 examples in the prompt template. We expect that, given these 33 in-context examples, the model learns the task of monotonic translation. In DecoMT's second stage, the chunks are longer, resulting in fewer of them. Continuing with the Marathi to Hindi translation example, there were 19 chunks in the prompt template. We observed that given these in-context examples, the model adeptly learns infilling. In this stage, the model can leverage a broader local context from translations of neighboring chunks as well as the associated source chunks to ensure accurate translation and refine the first stage's translation. Consequently, our straightforward chunking approach effectively ensures that translations proceed in a monotonic manner.\n\n**Q4: Reproducibility**\n\nUpon acceptance, we will release the code and model outputs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Evaluation on multiple language pairs from different language families is included\n\n- The paper is well written and the method is well-explained\n\n", "reasons_to_reject": "- The innovation is not enough for an EMNLP full-length paper\n\n- The method has an inefficiency, since it required multiple inference calls (as many as the number of chunks) in the contextual translation stage, which cannot be parallelized since the outputs are used successively", "questions_to_authors": "- How is the input chunked to ensure that the chunks can be translated monotonically?", "paper_topic_and_main_contributions": "The paper proposes a prompt scheme for MT of closely related languages. The method is in two stages: \n  - In a first stage, the input is chunked and each chunk is translated separately, using a few shots of monotonic-translated examples\n  - In a second stage, which the authors refer to as \"contextual translation\", the translated chunks are retranslated iteratively, with the previous output supplied in the target as context.\n\nDecomposed prompting is inspired by Khot et at 2023. The innovation is claimed to be in the second stage, the \"contextual translation\"\nThe motivation for this scheme is that for closely related languages, the first stage encourages monotonic translation (through the few shots presented in the prompt). This is more efficient since these languages tend to be low-resource. \n\nThe experiments use mT5 \n \nThe paper is thin on innovation, and the context is not enough for a full-length EMNLP paper", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bwgEhXQRjQ", "reviewer": "Reviewer_TKfW", "summary": "", "questions": "1. I guess even for related languages, there still exists word order differences, how would this affect the translation quality? \n2. Is it possible to use syntactic parser to split the input sentence into a number of chunks so that each chunk keeps a relatively complete semantic meaning?\n3. How much is it improved by adding the contextual translation (2nd stage) compared to the independent translation (1st stage)?\n4. If we remove the first stage, and only use the second stage, i.e. source chunks + previously translations + <mask>, what would the performance be?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. We address the concerns below:\n\n**Q: From the experimental results we can see although the DecoMT performs better than SAP, it is not a huge margin, but the two-stage process makes the translation more complicated and not straightforward.**\n\nDespite the two-stage design, DecoMT's methodology is intuitive and straightforward to implement. More importantly, our results reveals that DecoMT holds a statistically significant advantage over SAP across various criteria.\nWe evaluated DecoMT against SAP using 7 language pairs, analyzed in both directions. This results in a comprehensive comparison across 14 directions, as tabulated in Table 1.\nIn terms of the chrF++ metric, DecoMT considerably outperforms SAP across all 14 directions.\nWhen assessed using spBLEU, DecoMT exhibits superior performance in 12 directions, is comparable with SAP in 1 direction, and falls short in another.\nWe've established the statistical significance of these findings using paired bootstrap sampling (Koehn, 2004) with a confidence level set at p<0.05.\nOur analyses, particularly depicted in Figure 5, further illustrate that DecoMT's superiority grows more pronounced as sentence lengths increase.\n\n**Q: Even authors claim that using parallel computing for independent chunk translation, it is not fair to compare the efficiency or speed in this way. It's fairer to compare under same computing resources.**\n\nTo ensure a fair comparison, we've now profiled the codes using cprofile (https://docs.python.org/3/library/profile.html) during the inference phase, executed on an A40 48GB GPU. cprofile examines the time taken by various API calls. Our chosen task was translating from Marathi to Hindi using the initial batch of 5 examples from the FLORES test set, with the longest Marathi sample in the batch being 41 tokens long.\n1. SAP Analysis: For the SAP system, due to the unpredictability of the expected target length, we do decoding at 1.5 times the maximum source length. This is based on our studies of lengths of examples from validation dataset. For example, for our given source batch, the reference Hindi translation encompasses 55 tokens for the Marathi sentence which is 41 tokens long.\nAs the longest example is 41 tokens, we run inference for 41 * 1.5 = 61 steps.\nBelow is a partial trace using cprofile.\nWe see that for SAP, there are 61 calls to predict_output method. The method predict_output is responsible for running inference on the LLM. Each method takes 2.384 seconds. The inference of the batch takes 145.455 seconds.\n\n| ncalls | tottime | percall | cumtime | percall | filename:lineno(function)             |\n|--------|---------|---------|---------|---------|--------------------------------------|\n| 3/1    | 0.000   | 0.000   | 145.455 | 145.455 | {built-in method builtins.exec}      |\n| ...    |         |         |         |         |                                      |\n| 61     | 0.002   | 0.000   | 145.428 | 2.384   | few-shot-sap.py:163(predict_output)  |\n| ...    |         |         |         |         |                                      |\n\n2. DecoMT Analysis: For Marathi-Hindi translations, we use a chunk size of 4. \nWe first consider the independent translation stage. Breaking down the sentence lengths of the batch in tokens: 16, 30, 24, 41, and 28, we get respective chunk counts of 4, 8, 6, 11, and 7—aggregating to 36 chunks. Split into batches of 8, this leads to 5 API calls to predict_output.\nWith the longest sentence in the batch having 41 tokens, the contextual translation stage demands 11 API calls to predict_output, cumulating to 16 calls.\nThese 16 api calls in total amount to 96.883 seconds.\nWhile predict_output in DecoMT tends to take longer than in SAP (owing to DecoMT predicting multiple tokens as opposed to SAP's single-token approach), the overall fewer API calls render DecoMT more efficient.\n\nBelow is a partial trace using cprofile.\n| ncalls | tottime | percall | cumtime | percall | filename:lineno(function)                    |\n|--------|---------|---------|---------|---------|----------------------------------------------|\n| 3/1    | 0.000   | 0.000   | 96.883  | 96.883  | {built-in method builtins.exec}              |\n| ...    |         |         |         |         |                                              |\n| 16     | 0.001   | 0.000   | 96.868  | 6.054   | few-shot-decomp-mt.py:199(predict_output)    |\n| ...    |         |         |         |         |                                              |\n\nWe will incorporate this detailed analysis into the main paper for comprehensive understanding.\n\n**Q1: I guess even for related languages, there still exists word order differences, how would this affect the translation quality?**\n\nDuring the annotation of the 5 examples, annotators were instructed to maintain a monotonic alignment between the source and target. The annotators could reorder as well as modify the sentences to ensure this monotonicity. For the languages we have studied, the annotators were able to ensure monotonicity in the 5 examples. \nWe believe that when word order differences arise during testing, they are typically localized. As a result, they can be effectively managed through intra-chunk reordering. Nonetheless, a deeper exploration of this question is part of our plan for future work.\n\n**Q2: Is it possible to use syntactic parser to split the input sentence into a number of chunks so that each chunk keeps a relatively complete semantic meaning?**\n\nAt present, our method prioritizes a simpler strategy, employing a fixed chunk size for each source-target language combination. This approach has yielded satisfactory outcomes.  Leveraging syntactic parsers to achieve more semantically cohesive chunks is an interesting idea. However, most of the languages we consider in our work lack a good syntactic parser.\n\n**Q3: How much is it improved by adding the contextual translation (2nd stage) compared to the independent translation (1st stage)?**\n\nIn a new set of experiments, we compared the single-stage independent translation to the two-stage DecoMT. The experiments show that the inclusion of contextual translation in the second stage of DecoMT significantly improves performance. Here are the results:\n\n| lang pair     \t| spBLEU (Single Stage)  | spBLEU (DecoMT)  \t| chrF++ (Single Stage)  | chrF++ (DecoMT)  \t|\n|----------|------------------------|----------------------|------------------------|----------------------|\n| mar->hin | 17.0               \t| 21.0             \t| 39.6               \t| 41.9             \t|\n| hin->mar | 12.1               \t| 13.9             \t| 33.1               \t| 35.6             \t|\n| guj->hin | 21.0               \t| 23.2             \t| 41.8               \t| 43.7             \t|\n| hin->guj | 20.2               \t| 22.0             \t| 39.6               \t| 41.1             \t|\n| mal->hin | 13.3               \t| 16.3             \t| 34.7               \t| 36.8             \t|\n| hin->mal | 15.9               \t| 18.7             \t| 33.7               \t| 37.0             \t|\n| hin->tel | 16.7               \t| 19.5             \t| 36.3               \t| 38.5             \t|\n| tel->hin | 11.3               \t| 17.8             \t| 35.7               \t| 38.6             \t|\n| spa->por | 24.4               \t| 26.5             \t| 48.7               \t| 50.0             \t|\n| por->spa | 23.7               \t| 26.3             \t| 47.1               \t| 48.9             \t|\n| zsm->ind | 26.4               \t| 29.6             \t| 53.8               \t| 55.9             \t|\n| ind->zsm | 27.7               \t| 28.2             \t| 54.3               \t| 54.5             \t|\n| ukr->rus | 32.1               \t| 34.4             \t| 49.9               \t| 51.5             \t|\n| rus->ukr | 28.3               \t| 31.0             \t| 48.5               \t| 49.9             \t|\n\nThese findings emphasize the importance of contextual translation in the DecoMT approach. We will include this ablation study in our paper.\n\n**Q4: If we remove the first stage, and only use the second stage, i.e. source chunks + previously translations + <mask>, what would the performance be?**\n\nThe method you described is referred to as “Standard Prompting applied to mT5” in our paper (see lines 350-354). A direct comparison can be found in Table 1 under the column SP(mT5). Across all language pairs, SP(mT5) consistently scores lower in both spBLEU and chrF++ metrics when compared to DecoMT. This underscores the efficacy of the two-stage DecoMT approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main contribution of this paper is the method to decompose the prompts, realising a two-stage translation for related languages.", "reasons_to_reject": "From the experimental results we can see although the DecoMT performs better than SAP, it is not a huge margin, but the two-stage process makes the translation more complicated and not straightforward. Even authors claim that using parallel computing for independent chunk translation, it is not fair to compare the efficiency or speed in this way. It's fairer to compare under same computing resources.", "questions_to_authors": "1. I guess even for related languages, there still exists word order differences, how would this affect the translation quality? \n2. Is it possible to use syntactic parser to split the input sentence into a number of chunks so that each chunk keeps a relatively complete semantic meaning?\n3. How much is it improved by adding the contextual translation (2nd stage) compared to the independent translation (1st stage)?\n4. If we remove the first stage, and only use the second stage, i.e. source chunks + previously translations + <mask>, what would the performance be?", "paper_topic_and_main_contributions": "This paper proposes a framework for related languages translation which decomposes the prompts into two stages, i.e. independent chunk translation and then a contextual translation. The related languages have the similar word order and linguistic characteristics so that it can be translated monotonically. To better use these features, the paper first separate the input sentence into a number of chunks and translate them independently, then by combining the contexts of the target translation, the framework performs an incremental/contextual translation. Experiments on different related languages show significant improvements on different evaluation metrics compared to baselines.\n\nThe main contribution of this paper is the method to decompose the prompts, realising a two-stage translation for related languages. However, this method is only suitable for similar family of languages.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cOI032KVKi", "reviewer": "Reviewer_C2ZS", "summary": "", "questions": "- L449, the authors mentioned that each language pair only have one single annotator. Does this cause randomness in the human evaluation results?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback including appreciation of our approach and evaluation. \n\nWe first want to make the clarification regarding our approach:\n\n**Q: During the first decoding round, the authors split a source sentence into several chunks and performs independent translation of each chunk. (H_i -> M_i) In the second decoding round, each source chunk is concatenated with the neighbor chunks (left and right) as context (H_i-1 H_i H_i+1), and the corresponding translated context segment from round 1 is reused as target context (M_i-1 <mask> M_i+1). The model then performs a text in-filling task to output the translation correponds to the <mask> in the given context (H_i-1 H_i H_i+1: M_i-1 <mask> M_i+1 -> R_i). Finally the generated R_i are concatenated as the final translation.**\n\nThe reviewer has aptly summarized the first stage of our two-stage approach. However, we'd like to clarity the approach in the second stage: In this stage, the input is H_i-1 H_i H_i+1 along with R_i-1 <mask> M_i+1 to predict R_i. So, the transformation can be represented as:\n(H_i-1 H_i H_i+1: R_i-1 <mask> M_i+1 -> R_i). This process is explained in Figure 1, Section 3.5, and in equations (2) and the subsequent equation.\n\n**Q: This paper lacks experiments to show the necessity of using two-round decoding. If the independent chunk translation is already satifactory, then second decoding round using context is not necessary. Translation performance of the first and second decoding round should be compared to have a clear view of the usefulness of the second decoding round.**\n\nAs addressed in our response to Reviewer 2 (Q3), we have presented a comparison between the single-stage independent translation and DecoMT. The results consistently demonstrate DecoMT's superiority across all language pairs and directions.\n\n**Q: Also the effect of using few-shot prompting with 5 examples is not revealed in this paper. Since the bidirectional mT5 already performs both the direct translation task by filling the mask at the end of a source sequence, and the text in-filling task by predicting the mask in the middle of a source sequence, it seems that zero-shot translation using the two round decoding strategy also works.**\n\nAs suggested, we undertook zero-shot translation experiments for select language pairs, specifically hin<->gujr, hin<->tel, and hin<->mal, given our computational constraints. We compared different approaches applied to mT5 including DecoMT, SAP and Standard Prompting. We found that all approaches yielded near-zero BLEU scores.  In most instances, the models merely copied the input as the output. We hypothesize that this is because in a zero-shot setting the model may not understand that it has to perform translation to the target language.\n\n**Q: L449, the authors mentioned that each language pair only have one single annotator. Does this cause randomness in the human evaluation results?**\n\nThe human evaluations are useful to take a closer look and confirm what the automated metrics have shown. We concur that relying on a single annotator may introduce some variance in human evaluation. Nevertheless, to mitigate biases, we ensured that evaluation tasks were randomized, ensuring no system gets an undue advantage.\n\nWe're also grateful for the additional references provided and will incorporate them to better contextualize our work. We will also fix the typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed two-pass decoding method is interesting. It reuses the generated translation through independent chunk translation as pseudo target context to perform the second decoding round. It takes the advantage of bidirectional LM to perform both direct translation and text in-filling, which may not be possible for unidirectional LM if not specifically fine-tuned.\n- The proposed method is effective. It delivers better translation performance than previous prompting methods based on both unidirectional LM and bidirectional LM over several language pairs.\n- This proposed method may be an inspiration for future work to perform translation between non-English or low-resource languages using LLMs.", "reasons_to_reject": "- This paper lacks experiments to show the necessity of using two-round decoding. If the independent chunk translation is already satifactory, then second decoding round using context is not necessary. Translation performance of the first and second decoding round should be compared to have a clear view of the usefulness of the second decoding round.\n- Also the effect of using few-shot prompting with 5 examples is not revealed in this paper. Since the bidirectional mT5 already performs both the direct translation task by filling the mask at the end of a source sequence, and the text in-filling task by predicting the mask in the middle of a source sequence, it seems that zero-shot translation using the two round decoding strategy also works.", "questions_to_authors": "- L449, the authors mentioned that each language pair only have one single annotator. Does this cause randomness in the human evaluation results?", "paper_topic_and_main_contributions": "This paper propose a two-pass decoding method to use a mT5 model for direct translation between related languages. \n\nDuring the first decoding round, the authors split a source sentence into several chunks and performs independent translation of each chunk. (H_i -> M_i) In the second decoding round, each source chunk is concatenated with the neighbor chunks (left and right) as context (H_i-1 H_i H_i+1), and the corresponding translated context segment from round 1 is reused as target context (M_i-1 <mask> M_i+1). The model then performs a text in-filling task to output the translation correponds to the <mask> in the given context (H_i-1 H_i H_i+1: R_i-1 <mask> M_i+1 -> R_i). Finally the generated R_i are concatenated as the final translation. The translation process is performed under a few-shot prompting manner, where the source and target language is indicated in the prompting template.\n\nExperiments show that the proposed method deliver better translation results than previous prompting methods using mT5 on several related language pairs, especially when translation longer sentences. LLMs are known to have limited performance on low resource languages, especially translation between non-English languages. This paper gives a solution to perform direct translation between related non-English languages without passing through English using LLMs.\n\nHowever, this paper could still benefit from some more experiments and analyses to better demonstrate the effectiveness of the proposed method. For example, this paper only compares few-shot prompting performance. As all compared methods, including the proposed method, seem applicable in zero-shot scenarios, it is better to see the effectiveness of using few-shot examples. Also, since the final translation only comes from the second decoding round, what is the performance of the first decoding round? Does the second round really improves the translation quality? Is it necessary?", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Since there are quite a lot papers working prompting LLMs for machine translation that appeared quickly this year, it would be better to mention some of them appeared three months before the submission, even though they may not be officially reviewed:\n\n- The unreasonable effectiveness of few-shot learning for machine translation. Garcia et al., 2023\n- Prompting PaLM for Translation: Assessing Strategies and Performance. Vilar et al., 2022\n- Prompting Large Language Model for Machine Translation: A Case Study. Zhang et al., 2023\n- Is ChatGPT A Good Translator? A Preliminary Study. Jiao et al., 2023", "Typos_Grammar_Style_and_Presentation_Improvements": "- L36: any references to support this statement?\n- L64: mT5 is proposed in Xue et al instead of Lin et al\n- Figure 3: It would be better if it shares the same form as Figure 4, i.e. texts instead of picture"}}]

Author rebuttal:

Meta-review: The paper proposes an interesting two-pass method for promting LLMs for translation between related languages. 

The proposed method yields better performance in terms of automatic evaluation scores than previous prompting methods over several language pairs from different families. 

The method may be an inspiration for future work on translation between non-English or low-resourced languages using LLMs.

The revisions arised during authors' response should be applied in the final version.
Decision: Accept-Main