The Reasonableness Behind Unreasonable Translation Capability of Large Language Model
Authors: Tingchen Fu
Lemao Liu
Deng Cai
Guoping Huang
Shuming Shi
Rui Yan
Venue: ICLR 2024 poster (2024)
Keywords: large language model
translation
unintentional bilingualism
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Multilingual large language models trained on non-parallel data yield impressive translation capabilities. Existing studies demonstrate that incidental sentence-level bilingualism within pre-training data contributes to the LLM's translation abilities. However, it has also been observed that LLM's translation capabilities persist even when incidental sentence-level bilingualism are excluded from the training corpus.
In this study, we comprehensively investigate the unreasonable effectiveness and the underlying mechanism for LLM's translation abilities, specifically addressing the question why large language models learn to translate without parallel data, using the BLOOM model series as a representative example. Through extensive experiments, our findings suggest the existence of unintentional bilingualism in the pre-training corpus, especially word alignment data significantly contributes to the large language model's acquisition of translation ability. Moreover, the translation signal derived from word alignment data is comparable to that from sentence-level bilingualism. Additionally, we study the effects of monolingual data and parameter-sharing in assisting large language model to learn to translate. Together, these findings present another piece of the broader puzzle of trying to understand how large language models acquire translation capability.

Published as a conference paper at ICLR 2024
THE REASONABLENESS BEHIND UNREASONABLE TRANSLATION CAPABILITY OF LARGE LANGUAGE MODEL
Tingehen Fu 00+ Lemao Liu Deng Cai Guoping Huang Shuming Shi Rui Yan 9
Gaoling School of Artificial Intelligence, Renmin University of China
Tencent AI Lab
lucas. fut Fut.incchen@gail.cco redmondl.ieteeneet..cmm com
ruiyaneruc.edu.co
ABSTRACT
Multilingual large language models (LLM) trained on non-parallel data yield impressive translation capabilities. Existing studies demonstrate that inciden- tal sentence-level bilingualism within pre-training data contributes to the trans- lation abilities of large language models. However, it has been observed that the translation capabilities persist even when incidental sentence-level bilingual- this study, we compre-
ism is excluded from the training corpus. Therefore, hensively investigate the question why LLM can acquire translation capability without sentence-level bilingualism data. To this end, we examine the impacts of word-level bilingualism data (i.e., word alignment data and code-switching data) even purified monolingual data in addition sentence-level bilingualism data. Through extensive experiments, we have made significant findings. turns out that word alignment data plays crucial role in enabling LLMs to acquire translation ability. Surprisingly, the translation signal derived from word align- ment data is even comparable to that obtained from sentence-level bilingualism. Moreover, pre-training on purified monolingual data may enable slight transla- tion signal for LLM, thanks to the shared parameters in Transformer and some shared tokens across both source and target languages. Our code is available in httpp://gitthbb com/ ct http///gcchen//CCR244444 Tranoo----raancttttatntatiioo
INTRODUCTION
The performance modern neural machine translation (NMT) systems highly subject to the qual- ity and volume of available parallel corpus. NMT systems trained in low-resource unsupervised settings (Ravi & Knight 2011 Artetxe et al. (018)) are usually less comparable to the ones trained in fully supervised setting (Lample et al. 2018 Lin et al. 2022, Garcia al., 2021). However, recent multilingual large language models (LLMs), exemplified by ChatGPT (OpenAl, 2022) and
GPT4 (OpenAll 2023). seemingly defy conventional wisdom and exhibit unreasonable effectiveness in machine transl lation (García et al. [2023] Vilar et al. 2022 Zhang et al. 2023a). Intriguingly, these models trained non-parallel corpus, as opposed to the explicit parallel corpora commonly em- ployed for NMT training. Therefore, it is imperative to comprehend why LLMs enable translation by learning from non-parale non-parallel corpus.
Concerning the reason why LLMs could learn to translate, a natural assumption is the existence of incidental sentence-level bilingualism in the pre-training corpus, which provides supervision signals and has been demonstrated sufficient to train an NMT system (Briakou et al., (023). Nevertheless, Briakou et al.] (2023) discover that PaLM (Chowdhery et al. 2022) still demonstrates non-trivial translation capability even after removing such incidental sentence-level bilingualism, indicating that only partially accounts for LLM's translation capability. As a result, there must be other sources contributing the LLM's 's translation capability beyond the presence of incidental sentence- level bilingualism.
This work was done during an internship at Tencent AI Lab.
1 "Corrsponniig author: Lemao Liu (redmondl iu@tencent. com) and Rui Yan (ruiyan@ruc.

---

Published as a conference paper at ICLR 2024
Therefore in this study, we aim to comprehensively answer the in-depth question of why LLMs capable of translation even without incidental sentence-level bilingualism. We hypothesize that the finer granularity of unintentional bilingualism, specifically word alignment data code-switching data (exemplified in Table D. may also play a substantial role. To this end, we propose to measure and compare the impact of unintentional bilingualism across various levels of granularity. Specifi- cally, we first identify and collect three types of unintentional bilingualism data from existing multi- lingual corpora, namely sentence alignment, word alignment, and code-switching ($3). To quantity the effect of three types of bilingualism, it is too costly to train LLM from scratch and thereby we develop two computationally feasible methods as surrogates measure the impact of data ($4)
Subsequently, we apply the two surrogate methods to BLOOM-family model (Scao et al. 2022) with our collected three types of unintentional bilingual data and compare their effects on transla- tion capacity ($5). Moreover, extensive experiments are conducted to glean insights into the impact of other factors (c.g., monolingual data, parameter-sharing data volume) on the acquisition of trans- lation capacity for LLM $
To summarize, our analysis yields two main findings:
Aside from sentence alignment data, other forms of unintentional bilingualism also play important role in assisting large language model to learn translation capability. In par- ticular, we discover that word alignment data exhibit comparable or sometimes superior effectiveness in providing translation signals when compared with sentence alignment data.
It is possible observe a slight translation signal by pre-training solely on purified mono- lingual corpora in addition to unintentional bilingualism, thanks to some shared tokens (such as digits and symbols) and the shared parameters in model architecture across both source and target languages.
RELATED WORK
Large Language Models for Translation. Multilingual large language models, with Chat- GPT (OpenAl 2022) and GPT4 (OpenAl) 2023) as representatives, have garnered heated attention for their impressive capacity in neural machine translation (Hendy et al. 2023 Zhu et al.] [023] 2023) that is par with not only commercial translation systems like Google Translate and Microsoft Translator (Hendy et al.] 2023) but also the winner of WMT 2023a). Numerous previous works have devoted considerable efforts to evaluating and analyzing the translation abilities of these LLMs in both high and low resource lan- guages (Hendy et al., 22033 Jiao et al.) 2023b Robinson et al. (023). multilingual translation (Zhu et al. 2023). document-leve) translation <Wang et al.] 2023 Karpinska & lyyer] 2023). However, most existing works put emphasis on enhancing the performance of I.L.Ms with fine-tuning (Xu et al. 2023a; Jiao et al., 2023a| Yang et al. 2023 Zeng et al.
Vilar et al. 2022, García et al.
competition (Zhang et al.
2023) /Zhang et al.) (023b) seeking a better prompting strategy (Bawden & Yvon 2023 He et al. [202]] Ghazvininejad et al., 2003 Lu
cet al. 2023), In contrast, little attention is paid on the question why are these LLMs able to translate when they are only trained on multilingual corpus but witness no parallel data?
Data Contamination Data contamination (Magar & Schwartz, 2022) refers to the possible over- lap between the pre-training corpus and the benchmark for downstream evaluation (Brown et al.) 2020). Data contamination can lead to inflated evaluation results and overestimation of the model's true performance since the model might inadvertently memorize the "leaked" information (Dodge ct al., 2021] Lewis et al. 2021 Carlini et al.] 2018) that was already present in the pre-training corpus. The phenomenon has been observed and reported on close-book QA (Lewis et al. 2021 Kandpal et al., 2022). numerical reasoning (Razeghi et al., 2022) and so on. On the other hand, Blevins & Zettlemoyer (2022) observe the mixing of other languages in "officially" English corpora (e.g., BookCoroou Chooppus/uuuaa Zhu et al. 2015), and C4.en Raffel et al., 2020). christened "language contamina- tion", may explain the acquisition of the language model's cross-lingual ability. Inspired by [levins] & Zeutlemoeer (2022), we posit that language contamination, or in other words the mixing and co- existing of multiple languages, may also be present in non non-paalllel multilingual corpora and could potentially account for the translation capacity of multilingual LLMs. Most relevant to our work, Briakou et al. (2023) observe and extract unintentional bilingualism from the pre- pre-training corpus of Pal.M (Chowdhery et al. 2022) and examine the impact of various types of bilingualism. How-
2

---

Published as a conference paper at ICLR 2024
ever, their experiments and analysis primarily involve sentence- level bilinguallssm closed-source LLM (Chowdhery et al. 2022). but overlook translation signals at a finer gramularity.
3 SENTENCE LEVEL AND WORD LEVEL BILINGUALISM IN CORPUS
Usually derived from the Common Craw with a sophisticated data filtering pipeline, existing multi- lingual corpora such mC4 (Xue al.] 202T), CC-100 (Conneau et al., (2020a), ROOTS (Laurençon itt al. 2022b) and OSCAR (Ortiz Su' arez et al. 2019) are composed of multiple monolingual splits with their languages detected by language classification tools (Joulin et al. (0016). Nevertheless, due to the imperfections of the language classifier (Blevins & Zettlemoyer, 2022), a monolingual split may inadvertently contain some unintentional bilingual data. In this study, we focus on unintentional bilingualism between English and Chinese and identify three types of unintentional bilingualism present in three widely used multilingual corpora:
The first case is sentence alignment (SA), wherein sentence and translation co- exist within close proximity in a docu- ment. Sentence alignment data consti- tutes a form of sentence-level bilingualism Analogously, word aligmment (WA) per- tains to the co-occurrence of one or words (though not an entire sentence) and their translations within close proximity in a single document. Both sentence align- ment and word alignment involve bilin- gual translation and differ solely in the granularity of translation. Additionally, we identify code-switching (CS), which specifically refers to the co-occurrence of two languages within close proximity in a document, where the content in the two languages is semantically related rather than bearing a direct translation relation- ship. Both word alignment and code- switching are regarded as word-level bilin- gualism. Table I showcases three kinds of unintentional bilingualism.
Type
Example
Sentence Alignment
This news, like o light as an indescribable speed, In the blink of spread throughout the entire Martial Dragon Continent.
bullet, landed on the tranquil middle, instantly exploded!
news
Word Alignment
Beijingg will procure RMB million social organization
services.
Seijing News (88 (8088 January
Code- Switching
16 Provious Article):: New Polio immunization Drive to
Start Nigeria's
T- 22/ Next Articlel: Hong Kong's Top Health Official Resigns
Over
Table 1: Examples of three types of unintentional bilin- gualism found in mC4. For sentence alignment and word alignment, the text in italics is a parallel bilin- gualism in English and Chinese. For code-switching, the text highlighted in gray is our self-added transla- tion for illustration.
To excavate three types of unintentional
bilingualism, we develop data mining pipeline. Briefly, to identify and gather unintentional bilin- gualism from monolingual Chinese split: (1) We first search for alphahetic-composse fragments in each document with regular expression. (2) The fragments recognized to be English by an off-the-shelf language detection tool (Joulin et al., 2016) are then translated into Chinese with an external commercial translation system (Huang et al. 2021) D. (3) Finally, we measure the similarity between the obtained translation and the nearby context of the found fragments and categorize the fragment together with its nearby context into three types of unintentional bilingualism accordingly. Aside from collecting unintentional bilingualism, for the purpose of experimental comparison, we also gather "pure" monolingual Chinese (English) corpus by meticulously eliminating uninten- tional bilingualism as much as possible. More specifically, we again use regular expressions to remove any English letters (Chinese Characteristics) from the Chinese corpus (English corpus) in mC4, More details can be found in Appendix A
The distribution of unintentional bilingualism in three corpora is depicted in Figure As evidenced by the figure, unintentional bilingualism only accounts for a very small proportion (less than 5%) in the entire corpus. Nonetheless, given the vast scale of the corpus, the absolute quantity of con- taminated documents is indeed non-trivial. Furthermore, as a common attribute spanning the three corpora, word alignment and code-switching predominantly comprise the primary components of unintentional bilingualism.
httoss//commnncaalll org/
3

---

Published as a conference paper at ICLR 2024
Dataset
Language
Test
Example pool
WMT21
Baglish-Chiness
newstest2021 (1948/1002)
newstest((011,,211,,,
English
eng devtest (1012)
eng Latn. dev (997)
Chinese
zho
(1012)
zho Hans. dev (997)
Catalan
cat. devtest (1012)
cat.Latn, dev (997)
FLORES-200
Eastem Panjabi
pan. ban_Guru. devtest (1012)
pan_Guru, dev (997)
Igbo
Latn. devtest (1012)
ibo Latn dev (997)
Tswana
tsn Latn.devtest (1012)
tsn Latn. dev (997)
Table 2: stances.
Statistics of our evaluation benchmarks. Numbers in brackets denote the number
sendence wiggmmn: word sinmmnt code-swilching
mC4.en
mC4.zh
sentence
# Doc
210,931
2,462
alignment
Seg
355,320
432
Proportion (%)
word
Doc 658,643
1,972,764
alignment
Seq
500,550
659,456
code-
Doc
2,021,502
5,086,373
switching
Seq
903,810
997,376
0.5
The statistics of our mined un- intentional bilingual text from mC4.en and mC4.zh. We concatenate all the unin- tentional bilingual data together with their nearby context in documents to form input sequences fixed length. More details could be found in Appendix A
Table 3:
0.0
m/44
ROOTS
OSCAR
Corpora
Figure The proportion of documents con- taining three types of unintentional bilingual text three corpora. The proportion is esti- mated by subset sampling.
4 METHODOLOGY TO QUANTIFY TRANSLATION CAPABILITY
To quantify and analyze the contribution of unintentional bilingual data, Briakou et al. (023 pri- marily extract sentence alignment instances from the PaL. pre-training corpus, convert them into translation pairs (ar, y). thereby train external parison with another NMT model trained with official WMT parallel data to verify the effectiveness of sentence alignment data. Unfortunately, the strategy is not applicable when investigating the im- pact of other types of unintentional bilingualism since we can hardly convert word alignment data or code-switching data into translation pairs (z, y) train an external NMT system.
NMT model and draw com-
Key Idea Alternatively, we do not train an external NMT model but directly train LLM on different data sources and evaluate their performances as follows:
Train LLM on our collected X data. X is one of three types of unintentional bilingualism, i.e., sentence aligmment (SA), word alignment (WA), and code-switching (CS).
Train another L.M on a X-rand dataset as comparison X. X-rand data randomly sam- pled from the original C4.en and C4.zh with the same number of examples as X. Note that it is contaminated with unintentional bilingualism and contains SA, WA and CS as shown in Figure [ 3. Compare different LLMs (X vs. X-rand) with various metrics to measure their translation quality.
Accordingly, it can be inferred that: 1) the LLM trained on data performs better than that on X- rand data, then X contributes to LLM's translation capability; otherwise, X may have little influence on translation ability. 2) if the LLM trained on X, performs better than the models trained on X2, then conclude that X, contributes more to LLM's translation capability than X, (X1 X₂ represents one of SA, WA or CS) and vice versa.
Implementation However, a naive implementation of the above strategy is rather resource- intensive and infeasible in practice, it involves the pre-training of multiple LLMS corresponding
4

---

Published as a conference paper at ICLR 2024
to various data types. To put it into practice, employ two surrogate methods that compu- tationally efficient. The first approach, dubbed post-training, involves the continued training of the LLM under various data configurations based on their released checkpoints. The second approach, referred to as pre-training, is to pre-train a smaller LMM with random initialization which serves as simulation of training large language models from seratch. More implementation details can be found in Appendix C
Metries Since n-gram metrics like BLEU (Papineni et al. 2002) are known to underestimate the performance of large language models GGarefa al., 2023). ) we mainly use COMET (Rei et al., 2020) and BLEURT (Sellam et al. (2020) to measure translation performance following Zhang et al. (2023a) Besides, we obtain fine aene-graieed and explainable evaluation of the translation performance employing InstructScore (Xu et al. b0.3clbb. a compact yet competitive metric closely matching COMET translation evaluation. Unfortunately, for the pre-trained smaller language models, their translation capability is not strong enough to yield readable translations through decoding, as evi- denced in $ $5.3 and would be meaningless to measure the quality of these translations in terms of COMET and BLEURT. Instead, for evaluating these pre-trained models, we use metric that is independent of decoding, i.c. perplexity for the conditional distribution of reference translation on the source measure their translation ability.
WHY LLMS CAN TRANSLATE? AN EMPIRICAL ANALYSIS
5.1 EXPERIMENT SETUP
Dataset If not otherwise specified, we use the unintentional bilingual data and purified monolin- gual data (i.e., excluding SA, WA and CS) from mC4.en and mC4.zh to perform the experiments, and their statisties are shown in Table 3 We use WMT21 news translation task (Akhbardeh et al.) 2021] and the FLORES--000 (team et al. 2022) our evaluation benchmarks, with their statistics presented in Table
Language Model In this study, we mainly focus on decoder-only multilingual LLM (Lin et al,) 2022 Scao et al. 20222 Wei et al. (223) that is not explicitly trained parallel corpus and choose BLOOM (Scao et al. (2222) to perform our experiments for its diverse language category (45 natural languages and programme codes) and public availability.
Prompt Given test case (Itest ytest), randomly sample in-context translation examples (I1, }1 (222,
122... ./.)...................... (In, Vn from the example pool (Table and then insert into the following
template borrowed from Gareia et al. (023)
(source__anguage) {w1)
[target.language) {
[source__anguage] {øn}
(target.language) {yn}
(source_language)
{Rtest
(target.language)
where source language and target language are the names of source and target languages.
5.2 POST-TRAINING EXPERIMENT ON BILINGUALISM
We first consider post-training BLOOM-7.16 lb and the experiment results WMT21 are displayed in Table As shown in the table, both sentence alignment data and word alignment data contribute to the translation ability, they significantly outperform their X-rand counterparts in general. No- tably, the overall performance of post-trained models is akin to the original BLOOM-7.1H model, which may be attributed to the marginal influence of post-training data in light of the pre-existing
5

---

Published as a conference paper at ICLR 2024
ZH-EN
EN-ZH
3-shot
5-shot
3-shot
5-shot
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
BL.00M-..Ib
59.58
37.21
60.38
38.01
79.84
57.87
80.34
58.58
SA
62.05*
41.24
61.79*
40.47*
79.77
58.32'
80.18
58.64
SA-rand
59.13
37.60
59.28
37.73
79.47
57.48
79.99
58.33
WA
58.36'
36.34"
58.15"
35.75*
79.59
57.61
80.11
58.46
WA-rand
56.21
32.91
56.51
33.32
79:48
57.42
79.86
58.14
60.00"
38.59'
59.54"
37.82*
78.59
56.63
79.48
57.53
CS-rand
56.64
33.39
57.50
34.44
79.20
57.34
80.24
58.30
Table 4: Translation performance for post-training BLOOM-7.1b. CS = code-switching, WA = word alignment, SA = sentence alignment. Numbers marked with asterisk are significant improvements (t-test, < 0.05) compared with the second-best model in the same block.
ZH EN
EN ZH
3-shot
S-shot
3-shol
S-shot
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
BL BLLOMM-56m
53.62
34.00
54.55
35.14
66.84
43.23
67.85 47.88
44.40
61.55*
61.57*
43.04
46.32'
69.98*
47.14"
SA-rand
54.87
36.41
55.26
36.60
61.80
38.58
63.71
40,33
WA
60.99*
42.09*
60.77*
41.72"
71.82*
49.03*
72.47*
50.24
WA-rand
58.47
37.83
57.47
36.39
67.55
43.44
68.23
44,27
CS CS-rand
59.02*
39.66*
59.22*
39.90'
68.24
45.41
69.35
46.60"
58.43
38.56
57.95
37.49
68.53
44.70
69.26
45,27
Table Translation performance for post-training smaller-scale LLM (BLO0M-560m) CS= code- switching. WA = word alignment, SA = sentence alignment. Numbers marked with asterisk are significant improvements (t-test, < 0.05) compared with the second best model the same block.
translation capabilities possessed by the original BLOOM-7.1b Therefore, for BLOOM-7.15. the comparison should be limited to X and X-rand counterparts as the performance of post-training models dominated by the original BLOOM-7.1b. In addition, 5-shot performance is usually better than 3-shot ones in general, which echoes prior findings (Zhang et al. 2023a) that more in-context examples usually help.
Therefore, we conduct similar post-training experiments on top of a smaller LLM (i.e., BLOOM- 560m). Table [5] summarizes the experiment results. From this table, we can observe that both SA and WA significantly outperform their X-rand counterparts by a large margin in all cases, demon- strating that both SA and WA substantially contribute to LLM's translation ability. Notably, to our surprise, we find the effect of word alignment data is comparable or even superior to that of sentence alignment. We gauge one possible reason is that the number of WA examples contained in the pre- training corpus greatly exceeds the number of SA examples, as shown in Table 3 This unexpected discovery can further be used to elucidate an important phenomenon wherein the LLM's translation capability persists when sentence-level bilingualism is excluded from the training corpus an ob- servation previously noted in (Briakou et al., 2023) but lacking a clear explanation. In addition, we also find that CS outperforms CS-rand by modest margin. This fact indicates that code-switching data imparts weak translation knowledge to LLMs.
5.3 PRE-TRAINING EXPERIMENT ON BILINGUALISM
Aside from the post-training experiment, we pre-train a self-implemented BL.0MM5600m from scratch with our collected data the simulation of training LLM. Owing the constraints of our computational resources, we are regrettably unable to train a fully-converged BLOOM-560n| but alternatively train for a fixed number of updates, resulting in self-implemented pre-trained mod-
We present the results of the original BLOOM-7.1h 1b only for reference but for basic haseline compare since post-training may incur domain shift or hyper-paramete mismatching and thus lead to inferior performance than the original BL.0MM7.1Ib.
'Training a BL.0MM-600 necessitates 92.61 days on 32 A A100 GPUs.

---

Published as a conference paper at ICLR 2024
ZII- EN
EN ZI1
target
I-shot
3-shot
S-shot
target
1-shot
3-shot
S-shor
SA
149.09
117.69
121.74
123.09
1303.59
523.11
525.32"
527.93*
SA-rand
85,50
110.85
109.13
109.00
WA
115.72
80,58*
81,21
81,30"
346.65
216.33*
212.53*
212.10'
WA-rand
130.63
154.89
151.66
150.60
489.26
375.24
363.67
363.34
CS
138.36
129.82
131.35
132.37
343.39
270.67
273.18
CS rand
91.34
112.20
109.21
108.21
351.53
281.94
269.61
266.18
Table 7: Translation performance for pre-training smaller-scale LLM (BL00M-560m) in terms of perplexity. CS code-switching, WA word alignment, SA sentence alignment. Numbers marked with an asterisk are significant improvements (t-test, < 0.05) compared with the second- best model in the same block; The Numbers underlined are smaller translation perplexity than target language modeling perplexity. "_" denotes the number is above 2 x 10
els being incapable of producing meaningful translations upon decoding. As is shown in Table 6 the majority of pre-trained models under various data settings exhibit only near-random" COMET scores that are much lower than the numbers in Table [ indicating the potential risk of drawing a conclusion based solely on the comparison of the rather subpar translations upon decoding.
Alternatively, we use bilingual perplexity with var- to
ious numbers of examples 1-sshot/3-shot/--sse measure their translation ability, which is obtained by concatenating the Ytest into the prompt outlined in $5.1 and computing the perplexity on Viest Apart from that, we also present the monolingual perplex- ity of target language modeling (the "target" column in Table). In implementation, we solely input the Utest into language models to compute its perplex- ity. From the information theory perspective (Xu al. 2020), when bilingual perplexity is lower than monolingual perplexity, the LLM encapsulates posi- tive mutual information between two languages and is able to employ information in source languages, which can be interpreted as translation capacity to a certain degree. Otherwise, it indicates that the lan- guage model treats the source language as noise and its translation capacity is too weak to be observed.
ZH-EN
EN-ZH
COMET
BLEURT
COMET
BLEURT
SA
38.07
18.47
32.54
5.16
SA-man
23.96
8.48
WA
35.96
16.36
41.22
3.73
WA-rand
30.35
6.25
33.29
2.45
CS
29.40
18.71
37.10
CS-rand
37.45
17.63
37.91
6.49
random
36.15
3.54
31.18
0.73
Table 6: Very weak translation performance of pre-training BLOOM-560m evaluated on WMT21 with 5 in-context examples. The near-random performance suggests the num- bers might be noisy and thus unreliable.
We can observe that the effect word alignment data is clearly evidenced by the reduced bilingual perplexity of WA in comparison to WA-rand. In the case of SA SA-rand, SA-rand attains better multilingual perplexity in ZH-EN yet fails at EN-ZH direction, possibly because of the imbalanced proportion of Chinese data in sentence alignment (refer to Table 3). Notably, since the bilingual per- plexity exceeds the monolingual ones, from the perspective information theory we can conclude that SA-rand does not exhibit translation capacity. In contrast, SA, WA, and CS all achieve positive mutual information, although CS does not outperform CS-rand in terms of multilingual perplexity. In summary, both WA and SA contribute significantly to LLM's translation capability, mirroring the findings of the post-training experiments.
5.4 TRANSLATION CAPABILITY FROM PURE MONOLINGUAL DATA
So far we have verified the role of unintentional bilingualism in enhancing the LLM's translation capabilities. Inspired from Pires et al. (2019); Artetxe et al. (2020), we pose a further question: Does the presence of unintentional bilingualism within the pre-training corpus constitute prerequisite for translation ability? In other words, after climinating unintentional bilingualism. can the LLM
*The "random" in Table 6 refers directly outputring random shuffle of source sentences in the test set as system hypothesis.
Mattheattiaalyy. if the perplexity of target language modeling p(y) if higher than the conditional perplexity ply æ)) then the mutual information I, Hy(y) Hy [x) is positive (Xu et al. 2020
7

---

Published as a conference paper at ICLR 2024
ZH-EN
EN ZH
step
target
1-shot
3-shot
5-shot
target
1-shot
3-shot
5-shot
4.5k
145.54
155.
148.94
147.00
562.75
804.49
727.77
711.32
7.5k
114.55
141.42
132.69
129.77
450.33
517.05
477.31
511.61
10k
66.13
90.49
83.75
81.87
242.68
200.21
182.51
179.54
Table 8: The translation capacity (in terms of perplexity) WMT21 for pre-training smaller-scale LLM (BLO0M-560m) on purified monolingual data. The underlined numbers represent bilingual perplexity values that are lower than the monolingual perplexity in the target language.
largel
1-shot
3-shot
S-shol
Default
242.68
200.21
182.51
179.54
Sep Layer
241.83
304.03
290.31
288.03
Digit Substitution
410.69
734.54
512.70
432.36
Table 9: The translation capacity (in terms of perplexity) WMT21 EN-ZH for pre-training smaller-scale LLM (BLOOM-560m) on purified monolingual data.
acquire translation ubilities simply by training on pure monolingual data in English and Chinese? To answer the question, we pre-train BLOOM-560m from scratch exclusively using pure monolingual data in English and Chinese. The experiment results are shown in Table g
We observe that the LLM exhibits positive mutual information EN-ZH direction after training for 10k steps, proving that LLM could acquire a slight translation signal through pure monolingual data, though possibly conditioned on sufficient data and specific translation direction, which may be interesting to most researchers. To account for the counter-intuitive translation ability from pure monolingual corpora, we postulate two crucial factors: (1) some common tokens (e.g., digits and symbols) shared across both languages, which mapped to shared space and act anchor- ing points to initially align their adjacent context and progressively extend to the entire semantic space (Pires et al.) 2019 Conneau et al. 2020b). Meanwhile, (2) the shared parameters in Trans- former across two languages may enable the model to detect the language-univent thereby learning align the representation of multiple languages with the help of anchor points, or unintentional bilingualism in our scenario (Artetxe et al. 2020 Dufter & Schlitze 2020).
structures,
To investigate the effect of shared tokens such digits and symbols, we experiment with a variant Digit Substitution in which we substitute all dig- its in pure monolingual corpora with correspond- ing words respective languages [ The results are
EN
ZH
LM Head
shown in Table | and it becomes apparent that the positive mutual information on EN-ZH direction dis-
EN layer m
ZH layer m ..
EN layer 2
ZH layer 2
appears, which primarily agrees with our hypothesis.
EN layer
ZH layer
To verify the second factor regarding the impact of parameter-sharing architecture, we conduct an experiment comparing the performance of Default BLOOM-560m architecture with a variant in which the shared transformer layers are replaced with language-spede ones, dubbed Sep Layer. To be more specific, English data and Chinese data have respective transformer blocks and shared word em- bedding, as shown in Figure 2 The experiment re- sults are shown in Table 9 substantiating the crucial role of parameter-sharin default variants exhibit significantly lower monolin- gual perplexity.
Embedding
EN
ZH
Figure 2: The model architecture of the Sep Layer variant. The modules in blue and yellow are exclusive for English and Chi- nese, respectively. The modules in green shared modules of two languages.
between languages, as the
For example, substitute "one" for "1" in English and "_" for "1" in Chinese.
8

---

Published as a conference paper at ICLR 2024
Logarithmn of parplextty dtty
Logarithm of parple plyy
bo 00/50
p3n on
on pan
en-100
Lsn
en-tan
lmgo
Lebigo
200gge
650rp0
Ionigo
250apo
325rgp
steps
Training stepe
(a) Perplexity dynamics for BLOOM-560m
(b) Perplexity dynamics for BLOOM-7.IB
Figure 3: The perplexity dynamics of BLOOM model family on data size (training step) for both X-EN and EN-X.
5.5 EFFECTS OF TRANSLATION CAPABILITY ON DATA SIZE
To better understand how the translation ability of LLMs emerged, we examine the influence of pre- training data size utilizing the officially released intermediate checkpoints of the BLOOMyffmmll model. 'he trend for BL00M-560m and BLOOM--.16 on multiple language pairs (English <>> tttitaaann/anjaa/lggoo///
are shown in Figure Please refer to Appendix [ for our rationale for choosing language pairs. From the figures, we can observe a similar trend:
1. The translation perplexity on a specific language is correlated its makeup proportion in the pre- training corpus. The LLM achieves lower perplexity for high-resource languages (e.g., Chinese, Catalan) compared to low-resource ones (e.g.. Igbo, Tswana).
2. The translation ability experiences a surge at the early stage (approximately 1/6 of the whole training process) training but plateaus or gradually increases thereafter. Note that this cannot be solely attributed to changes in learning rate since the warmup phase ends and the learning rate begins to decay at around 1/1000 of the whole training process;
3. Overall, X-EN and EN-X exhibit similar trend for any language X. Except for one or two outliers, X-EN and EN-X usually surge, plateau, fluctuate, or decline in tandem, implying the two translation directions X-EN and EN-X may constitute duality relationship (He et al.. 2016). 4. The dynamics of BLOOM-560m and BLOOM-7.16 (and other variants shown in Appendix DD exhibit similar patterns and trends, despite the differing parameter scales. This implies that small- sized PLMs might share the same underlying mechanisms as LLMs when learning to translate.
Translation into low-resource languages noticeably worse than from low-resource languages into English, showing how important target language modeling is in the estimation of translation quality. With more English data, models are able produce better English output whereas low- resource language output is less polished.
6 CONCLUSION
Our study concentrates on the acquisition of translation capability in multilingual LLMs. To com- prehend the origins of this capacity, we verify the presence of word alignment and code-switching data in three mainstream multilingual corpora. Our experiments emphasize that the finer granular- ity of unintentional bilingualism (i.c., word alignment data) yields significant impact, which is comparable to or surpasses the effect of sentence alignment data. The translation ability of LLM still has room for improvement, particularly in low-resource languages (Robinson et al.] 2023 Zhu et al. 2023) where LLM usually lags behind supervised translation model (team et al.] 2022). As possible remedy, our discoveries may inspire the research community in devising data augmentation or supervised fine-tuning techniques to address this challenge, which would also be our future work.
https://hhugginggace.o wwwpaaac...bbbggiianeee ca/bigscience
9

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
All Our code, data and model checkpoints involved in this study will be released to facilitate re- lated research. We present the details our data collection and process pipeline in Appendix [A] The implementation details are described in Appendix C with the selection of hyper-parameters in Table 16
ETHICS STATEMENT
This paper will pose any ethical problems. First, machine translation is an old task in natural language processing and representation learning with numerous papers about this task published at ICLR conferences. Second, all the corpora used in this paper have been used in previous papers. Our method should only be used to understand the translation capacity of multilingual large language models or other relevant research uses but not for any malicious purpose.
ACKNOWLEDGEMENT
This work was supported by National Natural Science Foundation of China (NSFC Grant No. 62122089), Beijing Outstanding Young Scientist Program NO. 200WZYH0H020110000000 Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University China, the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China.
and
REFERENCES
Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina Espaa-Bonet, Angela Fan, Christian Feder- mann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyinn Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 conference on machine translation (wmt21). In Proceedings of the Sixth Conference Machine Translation, pp. 1-88, Online, November 2021. Association for Compu- tational Linguistics. URL htt.httts///cccanhoolooo org/2021. 1ttp//2221.mmt11111 wmt-1.1
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In International Conference Learning Representations, 2018. URL https://
openreview. www.e//wwnnet/oommmmd//y/y/e/eee
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. the cross- linsul transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4623-4637, Online, July 2020. Association for Computational URL.
Linguistics. doi: 10. 1116655/11122222lcllmma.. acl ocl-main.421
org/ 2020.
Rachel Bawden and Franccois Yvon. Investigating the translation performance of large multilin- gual language model: the case of bloom. ArXiv, abs/2303.01911, 2023.
Terra Blevins and Luke Zettlemoyer. Language contamination helps explains the cross-lingual ca- pabilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3563-3574, Abu Dhabi, United Arab Emirates, De- cember 2022. Association for Computational Linguisties. URL https:///aclanthoooos org/
2022.emmlp-mm wwweeeennlmmmie main. 233
Eleftheria Briakou, Colin Cherry, and George F. Foster. Searching for needles in a haystack: On the role of incidental bilingualism in palm's translation capability. ArXiv, abs/2305. 10266, 2023.
10

---

Published as a conference paper at ICLR 2024
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ily Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL
Nicholas Carlini, Chang Liu, Jemej Kos, Úlfar Erlingsson, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. ArXiv e-prints, 1802.08232, 2018. URL/httss://arat wwtpi/..ogg///////eeeee 08232
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, and et al. Palm: Scaling language modeling with pathways. 2022.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudbary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1440-8451, Online, July 2020a. Association for Computational Linguistics. doi: 10.18663///22222aal-mm. 747. https: //aclanthology.oo .rgg 2020. acl-main. 747
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. Emerging cross- lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting
of the Association for Computational Linguistics, pp. 6022-6034, Online, July 2020b. Asso-
ciation for Computational Linguisties. doi: 1/6653//122220.a--mmaa.... 7/aclanthology, wrg//020...22 acl-main... 536
URL https:
Jesse Dodge, Maarten Sap. Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference in Naturul Language Processing, pp. 1286-1305 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/w1/2021.emmm main. 98. URL
Empirical Methods
0221 emnlp-main....m 98
Philipp Dufter and Hinrich Schütze. Identifying elements essential for BERT's multilinguality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 4423-4437, Online, November 2020. Association for Computational Lin-
guistics. doi: 10. (00.18653/1122000.mmml main.
URL hntpss://aclanthologgy htt.tthhooooooooggo/ooooooo orw./220..00..
emnlp- mailp-mmin..mm 358
Xavier Garcia, Aditya Siddhant, Orhan Firat, and Ankur Parikh. Harnessing multilinguality in unsupervised machine translation for rare languages. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1126-1137, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 .naacc- mmacl-main..a ma...89.
www.:lloooooooogggggge/g 2021 naacl -aneel-main..mm 89
Xavier Gareía, Yamini Bansal, Colin Cherry, George F. Foster, Maxim Krikun, Fan Feng, Melvin Johnson, and Orhan Firat. The unreasonable effectiveness of few-shot learning for machine trans- lation. ArXiv, abs/2302.01398. 2023.
Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. Dietionary-basee phrase-level prompt- ing of large language models for machine translation. ArXiv, abs/2302.07856, 2023.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei- Ying Ma. Dual learning for machine translation. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016. URL h/tp:::::///////ot/tttet httphtttpss////roeeeninnnnn www.:gg...eursss..omm cctpscccrpprrrrrrp paper /2Les//pper/201/ffile// Bb5S2c0333644433577577777000---3

---

Published as a conference paper at ICLR 2024
Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. Exploring human-like translation strategy with large language models. ArXiv, Mww/2335504110 2023.
Amr Hendy, Mohamed Gomaa Abdelrchim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Atify, and Hany Hassan Awadalla. How good are gpt models at machine translation? comprehensive evaluation. ArXiv, abs/2302.09210, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL
net/forummid
Yf
Guoping Huang, Lemao Liu, Xing Wang, Longyue Wang, Huayang Li, Zhaopeng Tu, Chengyang Huang, and Shuming Shi. Transmart: A practical interactive machine translation system. ArXiv, abs/2105. 13072, 2021.
Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Xing Wang. Shuming Shi, and Zhaopeng Tu. Parrot: Translating during chat using large language models. arXiv preprint 2rXiv2330..02466 2023a.
Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. Is chatgpt a good translator? a preliminary study. ArXiv, abs/2301.08875,, 2023b.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttextzip: Compressing text classification models. arXiv preprint 111.v:161..03651, 2016.
Nikhil Kandpal, H. Deng. Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, 2022. wtt.cccchhoaaaarrrororrossS.S... URL htttss:/ppppp semanticscholar. org//orpuusD:225522999
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0888 2020.
Marzena Karpinska and Mohit lyyer. Large language models effectively leverage document-level context for literary translation, but critical errors persist. ArXiv, abs/2304.03225, 2023.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations, 2018.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:3 15533880931866 2022a.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, and et al. The bigscience ROOTS corpus: A 1.6TB composite multilingual dataset. Thirty-sixth Conference on Neural Information Procesy- ing Systems Datasets and Benchmarks Track, 2022b. URL http:://pppeneviieeeeeeeeeeeeeee
Id=UoEw6KigkUn
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 1000-1008, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 Leac--mai.886 URL httpthsss//aaaaatlllleteoo hwwphhooooooooooooooooooooo 2021. wrt..//.rr.......mmma..... 86
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary. Brian O' OoHoro, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual generative lan- guage models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. Computational Linguistics. URL https:://aclanthol.l. www.lloyyoogooooooooooooooo org/ 2022.
Abu Dhabi, United Arab Emirates, December 2022. Association for
emn ewnle-nnnmmmm 1p-main. 616
12

---

Published as a conference paper at ICLR 2024
Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Haoran Yang, Wai Lam, and Furu Wei. Chain-of- dictionary prompting elicits translation in large language models. ArXiv, abs/2305.06577 2023.
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers), pp. 157-165, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 12.18663/1//0222.cc-shhoo acl-short. 18
URL http:://aad.antttoooyyoo///////
OpenAL. Chatgpt: Optimizing language models for dialogue, 2022. URL https::/openai.ccoom #log/chatgpt/
OpenAI. Gpt-4, 2023. URL https:tts/////eeaaa htt.://ttt///pppaa..moooooo//o//
Pedro Javier Ortiz Su' arez, Benoit Sagot, and Laurent Romary. Asynchronous pipelines for process- ing huge corpora medium to low resource infrastructures. Proceedings of the Workshop Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pp. 9 16, Mannheim, 2019. Lestniz-Institut f'ur Deutsche Sprache. doi: 10.1461 10.14618/ids-put pu2-90.. URL http:://nbn-reeollviinn de/ur:: nbn: de bsz 001339---2155
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing 7hu. Bleu: a method for auto-
matic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the
Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.31 10.31155/107333.0033135 URL
ottps://aclanntooogyy.orrgg///
Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Alexan- dra Sasha Luccioni, Yacine Jernite, and Anna Rogers. The roots search tool: Data transparency for Ilms. arXiv preprint arXiv:2302.1403: 2023.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1996-5001, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10.18653/v1/P199 14933. URL
http://acaaaattoooooy..//////1/9 org/
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. Associ- ation for Computational Linguistics. URL https: / /wwww acwweb. www.weeb.orrg/aatoooo///////i 0tt////11866388888 6319
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text- to-text transformer. Journal of Machine Learning Research, 21(140): 2020. URL http: hrjjmr..org/ppprr///ee
07.. html
Sujith Ravi and Kevin Knight. Deciphering foreign language. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pp. 12-21, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL
https://acl.nthhoogyy.rrr. wwtp://hoooooooooor11/1111000
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Compu- tational Linguistics: EMNLP 2022, pp. 840-854, Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguistics. URthttps://acl wwntholog..org//000 2022.
fFindings-emnlp.55
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2702, Online, November 2020. Association for Computational mww.emmmmnnnnnn11..3..
Linguistics. doi: 10.18653/v http://1122000mmm... main.213. URL https::/aaclanthologg org/ 2020. eemllpmmiin2 213
Nathaniel R Robinson, Perez Ogayo, David R Mortensen, and Graham Neubig. Chatgpt mt: Com- petitive for high-(but not low-) resource languages. arXiv preprint arXiv:2309.07442,, 2023.
13

---

Published as a conference paper at ICLR 2024
Teven Le Scao, Angela Fan, Christopher Akiki, and et al. Elizabeth-Jane Pavlick. Bloom: A 176b- parameter open-access multilingual language model. ArXiv, abs/2211.05100, 2022.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, Online, July 2020. Association for Computational Linguistics. doi: 18.13//3/v
N...8653/11/222.ccm acl-200..clmainn.
URL httpps:///alaantoollgg http://cllanooooooooo////////// 22002020..o wtl.22....lmaiinnn acl-maln.....m 704
Nllb team, Marta Ruiz Costa-jussä, James Cross, Onur cCelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Alison Youngblood, Bapi Akula, Lofe Barrault, Gabriel Mejia Gonza- lez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, C. Tran, Pierre Yves Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm' an, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-center centered machine translation. ArXiv, abs/2207.0461 2022. https:t:s://// httpps::/ap..mmmatisssciiiiieeie
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. Prompting palm for translation: Assessing strategies and performance. ArXiv, abs/2211.09100 2022.
Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuminn Shi, and Zhaopeng Tu. Document-level machine translation with large language models. ArXiv, abp.a3a4..0221....... 2023.
Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. Polylm: An open source polyglot large language model. arXiv preprint (xXxx23077.0611 2023.
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine translation: Boosting translation performance of large language models. 2023a. URL https:
s/api.sematticscool/ .rg. org/CCrpusIDD 262084016
Wenda Xu, Xian Qian, Mingxuan Wang, Lei Li, and William Yang Wang. SESCORE2: Learning text generation evaluation via synthesizing realistic mistakes. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Voluue 1: Long Papers), pp. 3166-5183. Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023. http:///.2.....oonn.n.oooo acl-long. URL
httpss:/aaclantholoog..oo
acl long. 283
Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. Instructscore: Towards explainable text generation evaluation with automatic feed- back. arXiv preprint 23X.:23355.12222.... 2023c.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In International Conference on Learning Represen- tations, 2020. httt:hhtts////penreeet
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v/2201. maacl-main.4e URL https: //aclanthology.org www.0221aaalmmmii...mmmmm
Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages. 2023. URL https: htttpp::/appp //api.
semanticscholar.co .org httr://oops1D:225666.
Jiali Zeng, Fandong Meng, Yongjing Yin. and Jie Zhou. Tim: Teaching large language models to translate with comparison. arXiv preprint arXiv:2307. 20744,230..00448, 2023.
14

---

Published as a conference paper at ICLR 2024
Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: A case study. ArXiv, abs/2301.07069, 2023a.
Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, al. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. arXiv preprint arXiv:2306.10900 2023b.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. Multilingual machine translation with large language models: Empirical results and analysis. ArXiv, abs/2304.04675. 2023.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
15

---

Published as a conference paper at ICLR 2024
A MORE DATA COLLECTION DETAILS
Our experiments are conducted on the excavated data from the mC4 corpus (Xue et al. 2021). mC4
is commonly used multilingual corpus, from which we randomly sample 192, 715, 980 English documents and use all of 54, 542, 336 Chinese documents to collect three types of unintentional bilingualism and "pure" monolingual English/Chiness, corpora. Specifically,
1) To search for English fragments from Chinese documents, we first use the regular expression [A-Za-zØ-9 N,AAA "N " -===&*[[a=-a==[]]
JAA-Za-z-9 _\, A \\MM\\WA fragment. We use fasttext as our language detection tool. The
to detect rophabtic-compoon fragment recognized to be English by fasttext with prediction over 90 is sent to TranSmart system for translation.
2) Next, English fragment and the obtained translation. The fragment and its nearby contex 10 are categorized as sentence alignment (SA) if the fragment contains at least 5 words and the sacreBLEU score is above 10%. Otherwise, it would be categorized as word alignment (WA) if the similarity is non-zero or code-switching (CS) if it is zero.
sacrebleu Post 2018 measure the similarity between the nearby context of the
3) The procedure is almost the same when it comes to detecting unintentional Chinese text from English documents. The only difference is how we seek possible Chinese fragments. Specifically, we search for bharacter-composee fragments whose Unicode lies between \u4e00 and \u9fff, the interval corresponding CJK unified ideographs.
4) Analogously, to obtain "pure" monolingual English/Chinese corpus for experimental use, we again use regular expressions eliminate possible bilingual fragments from a "monolin- gual" corpus to obtain its purified version. Specifically, get rid of all the strings matched by [A-Za-z0-9 \ ( [A-Za-z0-9 A
from Chinese corpus and all strings whose Unicode lies within \u2e80 and \u9fff from an English corpus.
In our experiment, first pre-process documents and group them into fixed-length sequences. In other words, sequences are assembled by concatenating and/or splitting documents to the appropriate length. Multiple short documents may compose only single sequence, a long document will be cut into a few sequences. Different from Briakou et al. (2023), we do not add a special document- boundary token.
A.1 DATA QUALITY CONTROL
To check the quality of our collected unintentional bilingual data, we recruited three human annota- tors from our institute, each possessing bachelor's degree and excelling in both languages. We ran- domly sample 100 instances from the SA/WA/CS data derived from the EN/ZH corpus and then in- quire whether the presented texts align with the def- initions of SA/WA/CS. The final decision is reached based on the majority vote of the three annotators. The annotators remain unaware of the origin of sen- tences so as to eliminate potential bias. The human evaluation results shown in the Table T0
EN
ZH
sentence alignment
87%
90%
word alignment
91%
93%
code-switching
90%
92%
Table 10: The human evaluation on the
quality of our curated unintentional bilingual data.
httos://transnartt .o..o.oo///oo////////ooooooooott con/
Inttos://githu. con/njposst/accebleel
We define the nearby context or close proximity as the adjacent lines of the fragment.
We expand the scope here including CJK strokes, CJK compatibility, CJK punctuation and so on to mitigate false negatives.
16

---

Published as a conference paper at ICLR 2024
ZH-EN
EN-ZH
3-shot
5-shot
3-shot
5-shot
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
SA(threshol==15.0)
60.76
42.33
60.64
42.21
69.08
46.51
69.69
47.15
1otthreshold=1550--aammm
52.97
34.59
53.12
34.62
60.47
36.71
62.46
38.92
SA(threshol==1255
60.58
41.97
60.61
42.00
68,93
46.19
69.68
47.11
14thheshhodd:12.5)-ranm
50.55
37.73
50.59
34.01
59.89
37.19
62.27
39.40
Table 11: Translation performance for post-training smaller-scale LLM (BLO0M-560m) with dif- ferent sentence alignment threshold. SA = sentence alignment.
zh-en
en-zh
1-shot
3-shot
5-shot
1-shot
3-shot
5-shot
BLOOM-560m
17.50
17.39
17.44
25.09
22.36
21.69
SA
15.02
14.87
14.87
23.37
22.31
22.26
SA-rand
17.38
17.36
17.36
37.22
31.73
30.83
15.33
15.33
15.36
23.10
21.11
20.97
CS-rand
16.27
16.52
16.52
25.63
23.08
22.94
WA
11.37
11.50
11.62
18.51
17.63
WA-rand
16.52
16.66
16.65
25.64
22.81
22.64
Table 12:
The perplexities WMT21 after post-training BLO0M-5600 with different data set- tings, CS = code switch, WA = word alignment, SA = sentence alignment.
A.2 MORE ANALYSIS ON CLASSIFICATION THRESHOLD
To verify whether varying syper-parameters for classifying sentence alignment yield different con- clusions, we modify the similarity threshold for sentence alignment classification from 10.0 12.5 and 15.0 (measured in BLEU). Utilizing the updated sentence alignment thresholds, the experimen- tal outcomes for post-training BLOOM-560m are displayed in Table
As shown in the table, alterations to the threshold may impact specific translation performance on WMT21, but not our overall conclusion. essence, our findings remain robust under varying similarity thresholds and are relatively unaffected by minor adjustments of the threshold.
A.3 TRAIN-TEST OVERLAP
ROOTS corpus (Laurençon et al. 2022a) only partially publicly available so it would be infeasible accurately compute the overlap between the BLOOM pre-training data and the WMT21 test set. Therefore, we randomly sample 50 cases from WMT21 EN-ZH and ZH-EN respectively and approach the overlap proportion by manually searching similar passages in the pre-training corpus with the ROOTS fuzzy match tool (Piktus et al.) 2023). Following previous work (Vilar et al. (022 Chowdhery al. 2022), we regard test case as leaked a sub-string of its target sentence with minimum length of 15 tokens is contained in top-10 search results returned by the ROOTS fuzzy search tool. Through our analysis, the overlap proportion of both WMT21 EN-ZH and Z.-EEN are less than 2.0%.
B MORE EXPERIMENT RESULTS ON POST-TRAINING
We present the perplexity results for two post-training experiments (BLO0M-560m and BLOOM- 7b) in Table 12 and Table 13 respectively. From the tables, we could see the perplexity results are consistent with those of BLEURT and COMET 4 Table verifying that perplexity is an acceptable proxy for measuring translation ability.
Besides, employ InstructScore (Xu al. 2023c) to have fine- fine-grained and explainable eval- uation of translation performance. The experiment results on post-training BLOOM-560m and BLOOM--.11 lb shown in Table 14] Table 15 respectively.
17

---

Published as a conference paper at ICLR 2024
ZH-EN
EN-ZH
1-shot
3-shot
5-shot
1-shot
3-shot
5-shot
BLOOM-7.Ib
6.65
6.84
6.86
9.64
9.38
9.34
SA
6.31
6.62
6.76
9.75
9.64
9.62
SA-rand
10,44
WA
6.69
9.39
9.25
9.25
WA-rand
6.81
7.02
9.80
9.58
9.56
CS
6.60
6.79
9.85
9.60
9.58
CS-rand
6.69
6.92
Table 13: The perplexities on WMT21 after post-training BLOOO--.11 under different data set- tings, CS code switch, WA word alignment, SA sentence alignment.
ZH-EN
EN-ZH
3-shot
5-shot
3-shot
5-shot
BL00M-560m
-9.16
-9.08
-11.51
-11.70
-9.01
-9.09
-11.95
-11.91
SA-rand
-9.32
-9.32
-11.98
-12.12
WA
-9.15
-9.24
-11.72
-11.66
WA-rand
-9.87
-10.33
-11.78
-11.59
.9.25
.9.05
-1..74
-11.63
CS-rand
-9.43
-9.74
-11.75
-11.75
Table 14: The InstructScore on WMT21 after post-training BLOOM-560m under different data settings. CS code switch, WA word alignment, SA sentence alignment.
ZH-EN
EN-ZH
3-shot
5-shot
3-shot
5-shot
BLOOM-7.I6
-9.88
-9.89
-10.51
-10.55
SA
-9.64
-9.76
-9.55
.9.57
-and
-9.34
-9.27
-10.48
-10.55
WA
-9.76
-9.76
-9.71
-9.61
WA-rand
10.30
-10.40
-10.73
-10.64
CS
-9.87
-9.46
-9.21
-9.24
CS-rand
-9.83
-9.82
-10.88
10.60
Table 15: The InstructScore on WMT21 after post-training BLOOM-7.16 under different data set- tings. CS = code switch, WA word alignment, SA sentence alignment.
18

---

Published as a conference paper at ICLR 2024
post-training (BLOOM-560m)
pre-traiing (BL00M-560m)
post-training (BLO0M-7.16)
Precision
float16
float16
float16
Batch Size
256
512
128
Optimizer
AdamW
AdamW
Adam W
Adam (8 [2)
(0.9.0.95)
(0,9.0995)
0.99.
Learning Rate
le-5
3e-4
le-4
Sequence Length
1024
1024
Warmup Step
500
Decay style
cosine
cosine
cosine
Min. Learning Rate
0
0
Weight Decay
le-1
le-1
le-1
Gradient clip
1.0
1.0
1.0
LoRA rank
NA
NA
8
LoRA
NA
NA
16
Table 16: The paraeeeas for post-training and pre-training.
2.0
jausable Information
3.5
amch
0.5
p20
360
11.00
1760
Todo
Model Size
Figure 4: The perplexity dynamics of BLOOM model family model size for both X-EN and EN-X.
C MORE IMPLEMENTATION DETAILS
Our experiments conducted cloud Linux server with Ubuntu 16.04 operating system. The codes are written in Python 3.10 using the code from huggingface library 12 The GPU type is Nvidia Tesla V100 with 32GB GPU memory.
The detailed hyper-parameter settings for post-training and from-scratch training are shown in Ta- T6 Note that for post-training of BLO0M-7.1b without loss of generality,
the LoRA (Hu
et al. 2022 as parameter-efficiet training technique rather than full- aull-parameter training. We apply the low-rank adaptation for the query, key, value and output projection matrices in the self-attention module within every transformer layer. We train the model for one epoch for both post-training and pre-training- To ensure fair comparison, we maintain the size and language composition of X-addd to be consistent with X. Specially, when preparing X-rand data, the sampling ratio between C4.en and C4.zh is alignment with the composition proportion shown in Table
For prompting, we randomly sample in-context examples from the candidate pool. For decoding, we use greedy search with a minimal generation length of It is possible that with a more sophis- ticated prompting and decoding algorithm we may get better results but the decoding algorithm or prompting strategy is not the focus of this study.
htttss:/hhuggingaaccccc
19

---

Published as a conference paper at ICLR 2024
Logarithm of parplextyy
Logarithm of perplexth;
pan on
Ibo
en-io
Gn
envonn
Lebigo
200rge
850r00
1epo
Sergp
10000
Ziarpo
325r00
steps
stepe
Perplexity dynamies for BLOOM-1.ID
(b) Perplexity dynamics for BLOOM-1.7b
Logarithm of parplexity
Iiero
Segrp
Iholp
Jaugso 12/ar
Training steps
(c) Perplexity dynamics for BLOOM-3b
Figure 5: The perplexity dynamics of BLOOM model family on data size (training step) for both X-EN and EN-X.
D MORE EXPERIMENTS RESULTS ON LEARNING DYNAMICS
As BLOOM models are trained on English (33%) and other 45 natural languages, we rank these lan- guages according to their make-up up proportion in ROOTS (Laurençon et al. 2022b) and thereby se-
lect languages, namely Simplified Chinese (1st, 18%), Catalan (10th, 1.2%) Eastern Punjabi (19th, 0.1%), Igbo (28th, 0.001%) 1)) and Setswana (37th, 0.0001 ()) to perform to (X-EN) and from (EN-X) English translation experiments on the FLORES-200 (team al. 2022) benchmark. The effect of data size (training steps) for BLO0M-560m and BL.OOM-7.1b are shown in Figure 33a and Fig- ure 3b respectvelly. Besides, we also track the learning dynamics of BLOOM-I.Ib, BLOOM- 1.7b and BLOOM-3b. The results shown in Figure 5 In addition, plot the trend of translation capacity (measured multilingual perplexity) with the model size on Figure
We observe that the shape of the curve initially exhibits obvious decline from 560m to 1.7b, followed by a period of gradual tapering off. Notably, for low-resource languages (e.g.. Igbo and Setswana) which constitute small portion of the training corpus, solely expanding the model size enhance their translation quality a suboptimal approach. This strategy necessitates considerably larger model size but might only yield marginal returns, which echoes previous findings (Kaplan et al. 2020 Kandpal et al. 2022 to certain degree.
20


Reviews:
[{"review_id": "vkLY9MCt1u", "reviewer": "Reviewer_ZA8F", "summary": "The paper studies the translation ability in multilingual LLMs, showing that the presence of sentence alignment, word alignment and code-switching data may contribute maybe an important role in explaining this amazing ability of multilingual LLMs. It is interesting that word alignment data yields a significant impact that surpasses the effect of sentence alignment data in certain cases (e.g. for post-training smaller-scale LLM). Despite several cons I see from the work, I think this work is a nice and very timing contribution to the community and thus recommend for an acceptance.", "questions": "- I get lost at random baseline in Table 6. Please elaborate more.\n- what is the precision/recall of \"WA\", \"SA\", \"CS\" classifier?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The findings from the paper are interesting and may shed light on the amazing translation ability from LLM. I enjoy reading the work a lot.", "weaknesses": "My biggest concern about the work is that while it is true that maybe the presence of sentence alignment, word alignment and code-switching data contribute an important role in explaining this amazing ability of multilingual LLMs. We just don't know how important they are and is there any other reason (e.g. the presence of other stuff) that is actually even way more important than the three. I don't think the paper provide any data points on this.\n\nAnother smaller concern is that I see some paragraphs are just half-baked from curiosity perspective. For instance, section 5.4 shows that after eliminating unintentional bilingual data, the translation ability is still there, apparently. But there is no data points on why that happens, just some postulating about the reasons and that is it.\n\nThe final weakness point of the paper to me is the presentation in Section 5.5. I could not follow exactly what \"the shared transformer layers in the BLOOM-560m model\". To make the work self-contained I think the paper should at least present some high level details about the shared transfomer layers before presenting how this may influence the effect of unintentional bilingualism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8VSwKLyzGm", "reviewer": "Reviewer_5xLr", "summary": "Multilingual large language model has shown impressive translation capabilities, while the model is trained on non-parallel data. This paper is motivated by a question of \"Why can multilingual large language model can learn to translate without parallel data?\" The authors report that there exist unintentional bilingualism in the training data that can be categorized into 3 types of bilingualism: 1) sentence alignment, 2) word alignment, and 3) code-switching and experimentally showed that, with these bilingualism, a large language model can learn to translate. Specifically, word alignment data provides the model with translation signals when compared with sentence alignment data. These findings will be helpful in data collection, data augmentation to improve translation capability of a large language model, since the translation ability lags behind a general translation model that is trained on parallel data.", "questions": "- In Table 3, why is #seq smaller than #Doc? The paper describes \"We concatenate all the sentences in documents to form input sequences of fixed length\" and I assume that #seq would be larger in #Doc.\n- The authors focus on English and Chinese data to identify unintentional bilingualism types, and I was wondering if any other bilingualism types exist when you check the other languages.\n\n- In Tables 9 and 10, please consider rephrasing \"the translation performance\" since both tables report PPL, not general translation performance metrics such as BLEU or COMET. \n- n-gram -> $n$-gram", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Extensive experiments are conducted, to address the main question of \"Why can multilingual large language model can learn to translate without parallel data?\"\n- The authors provide detailed empirical analyses to identify the underlying mechanism of translation capability in multilingual large language model. They also discuss the cases where single monolingual data are available with different sizes. This part might be helpful on how to improve low-resource language translation quality in the language model\n- The paper is well organized and clearly described in most parts.", "weaknesses": "- The authors focus on English and Chinese data to identify unintentional bilingualism types, and I was wondering if any other bilingualism types exist when you check the other languages.\n- This paper looks interesting in light of the empirical analyses of bilingualism's role in translation capabilities; however, the question still remains on how to effectively enhance translation capability further against the supervised translation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "soiLX4dfXh", "reviewer": "Reviewer_stf2", "summary": "This submission addresses an important and relevant question in recent machine translation research: how do large language models manage to perform so well on multiple language pairs without being trained specifically on bilingual (parallel) data? It outlines three different types of unintentional bilingual data (UBD): sentence and word alignment, as well as code switching, and analyse their influence on machine translation performance, suggesting that word-level alignment may play a larger role than was previously thought, possibly because it is much more prevalent than sentence alignment.", "questions": "p.8 -- Shouldn't perplexity de lower in Table 9 (using UBD) than in Table 10 (purified) -- Can you comment on why it would be the opposite?\n\nSection 5.2: Although WA is better than WA-rand in Table 4, it is also inferior to both SA and CS, often by a large margin. Could you better support the claim that \"the effect of word alignment data is comparable or even superior to that of sentence alignment\"?\n\nAn additional final point in Section 5.6. could be that translation *into* low resource languages is noticeably worse than *from* low resource language into English, showing how important fluency is in the estimation of translation quality. With more EN data, models are able to produce better English output whereas low resource language output is less polished.\n\nMisc/typos:\n\"... translation quality is less comparable to NMT systems trained with parallel data.\" is a bit ambiguous as low-resource systems are typically trained on parallel data as well, but much less of it.\n\n\"other forms of unintentional bilingualism also plays\" -> play\n\n\"Sine\" (p.5) -> Since\n\n\"CS outperforms its X-rand counterparts\" could more clearly be \"CS outperforms CS-rand\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The underlying question has been addressed before, suggesting that the presence of incidental sentence alignment in \"supposedly\" monolingual data plays a large role in LLM performance on MT. This submission pushes the investigation by looking at word-level alignment and code-switched data which as far as I know is novel.\n\nThe positive impact of word alignment is demonstrated in a number of experiments showing improvements in either translation quality or perplexity. The hypothesis that the larger amount of word alignment data (as compared to sentence-aligned) allows it to be as useful as sentence-aligned data is well supported by the experiments.\n\nMT quality is evaluated using neural metrics rather than BLEU, which aligns with recent results from the WMT22 metrics task (https://aclanthology.org/2022.wmt-1.2/).\n\nSome results are presented with significance tests of the differences, yay!", "weaknesses": "The UBD (sentence & word aligned + code switched) is extracted using an automated methods with arbitrary parameters that seem fairly ad hoc (eg 10 BLEU poins threshold, appdx A). This raises the question of how good that data is... Are sentence-aligned segments even aligned sentences? At 10 BLEU point, this is not very clear. This data is at the core of the argument of the paper, better quality control would make a more convincing case.\n\nExperiments use mostly smaller models, as well as surrogate methods, and some models are not even converged. Clearly the amount of computation is significant. However, would one make conclusion on a chemical process from a reaction that has not completed? These imperfect or incomplete experimental conditions make the conclusions less convincing. The switch to perplexity does not help -- the claims in Section 5.3 are only mildly convincing if it is not possible to extract minimally useful translations from the models.\n\nSection 5.4 would benefit from a comparison with a model using UBD data somehow -- this would allow to gauge whether the performance in Table 8 is getting remotely close to acceptable translation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "efI5Unca3A", "reviewer": "Reviewer_6ach", "summary": "This paper explored the reasons for the emergent translation capabilities of LLM from the perspective of data composition and provided empirical evidence to support its findings.", "questions": "1. How can we be certain that it's related to numbers and punctuation? If we replace numbers with their unique expressions in different languages, such as 'one' for '1', 'two' for '2'; '壹' for '1', '贰' for '2' (Traditional Chinese); '一' for '1', '二' for '2' (Simplified Chinese), ensuring that each language's corpus is entirely unrelated, would the translation capability still exist?\n\n2. Is the spontaneous alignment capability of large models based on the power of the attention framework? Regardless, does joint training of an LLM with two or more languages always generalize to translation capabilities, given a fixed number of languages in the data? This point is crucial for data augmentation methods in LLM.\n\n3. Compared to supervised NMT models, the translation capability of large models is more evident in their flexible interactions, producing translations that better meet requirements. This flexibility also leads to the unpredictability of the translation outputs from large models. Even a minor change in a prompt can result in different translations. In light of this paper, how do the authors view this issue?\n\n4. If possible, please consider using the 'Instruct Score' from EMNLP 2023 (Instructscore: Towards Explainable Text Generation Evaluation with Automatic Feedback) as a metric.\n\n5. The title is quite bold, but I don't believe the content of the paper fully justifies it. For instance, if the focus is on the data composition of LLM, deeper issues like data leakage (where training data includes test data or text similar to test data) should be considered. The 'bloomz' dataset, which is open-source and includes translation tasks, doesn't seem to have been addressed in the paper. LLMs are trained on vast amounts of data, so there's a need to check for (partial) overlap with the test set.\n\n6. Has there been any consideration that an excessive amount of bilingual data might dilute the pre-trained knowledge in LLM and impair its translation capability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper innovatively delved into the emergent translation abilities of LLM by analyzing its training data composition.", "weaknesses": "The evaluation methods have its limitations.\n\nThere are flaws in the methodology, suggesting that the prompts might influence the results.\n\nAt its core, it's an ablation study at the data level.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)