On Bilingual Lexicon Induction with Large Language Models
Authors: Yaoyiran Li
Anna Korhonen
Ivan Vulić
Venue: EMNLP 2023 Main (2023)
Keywords: Bilingual Lexicon Induction
Large Language Models
TL;DR: On Bilingual Lexicon Induction with Large Language Models

Abstract: Bilingual Lexicon Induction (BLI) is a core task in multilingual NLP that still, to a large extent, relies on calculating cross-lingual word representations. Inspired by the global paradigm shift in NLP towards Large Language Models (LLMs), we examine the potential of the latest generation of LLMs for the development of bilingual lexicons. We ask the following research question: Is it possible to prompt and fine-tune multilingual LLMs (mLLMs) for BLI, and how does this approach compare against and complement current BLI approaches? To this end, we systematically study 1) zero-shot prompting for unsupervised BLI and 2) few-shot in-context prompting with a set of seed translation pairs, both without any LLM fine-tuning, as well as 3) standard BLI-oriented fine-tuning of smaller LLMs. We experiment with 18 open-source text-to-text mLLMs of different sizes (from 0.3B to 13B parameters) on two standard BLI benchmarks covering a range of typologically diverse languages. Our work is the first to demonstrate strong BLI capabilities of text-to-text mLLMs. The results reveal that few-shot prompting with in-context examples from nearest neighbours achieves the best performance, establishing new state-of-the-art BLI scores for many language pairs. We also conduct a series of in-depth analyses and ablation studies, providing more insights on BLI with (m)LLMs, also along with their limitations.

On Bilingual Lexicon Induction with Large Language Models
Yaoyiran Li Anna Korhonen Ivan Vulié Language Technology Lab, TAL, University of Cambridge (y1711, alk23, vv250 J@cam. ac.
Abstract
(SotA) BLI approaches have been predominantly supported by learning a cross-lingual word embed- ding (CLWE) space, with which BLI is tackled via nearest neighbour retrieval (Artetxe et al., 201 8a; Heyman et al., 2019; Peng et al.. 2021; Li et al.. 2022a; Marchisio et al., 2022, inter alia).
Bilingual Lexicon Induction (BLI) is task in multilingual NI.P that still, to large extent, relies on calculating cross- gross-lingual word repre- sentations. Inspired by the global paradigm shift NLP towards Large Language Models (LLMs)), we examine the potential of the lat- est generation of LLMs for the development bilingual lexicons. We ask the following re- search question: Is it possible to prompt and fine-tune multilingual LLMs (mLLMs) for BLI, and how does this approach compare against and complement current BLI approaches? To this end, we systematically study 1) zero-shot prompting for unsupervised BLI and 2) few- shot in io-tentext prompting with a set of seed translation pairs, both without any LLM fine- tuning, well standard BLI-oriented fine- tuning of smaller LLMs. We experiment with open-source text-to-text mLLMs of differ- ent sizes (from 0.3B to 13B parameters) two standard BLJI benchmarks covering a range of typologically diverse languages. Our work is the first to demonstrate strong BLI capabilities text-to-text mLLMs. The results reveal that few-shot prompting with in-context examples from nearest neighbours achieves the best per- formance, establishing new state-of-the-art BLI scores for many language pairs. We also con- duct a series of in-depth analyses and ablation studies, providing more insights on with (m)LLMs, also along with their limitations.
Meanwhile, autoregressivve text-to-text large lan- guage models (LLMs) have emerged as the comer- stone of cutting-edge NLP research (Raffel et al.. 2020; Brown et al., 2020; Ouyang et al., 2022; Chowdhery et al., 2022). For example, multilin- gual LLMS (mLLMs) have shown (sentence-level) machine translation capabilities (Vilar et al., 2022; Briakou et al., 2023), although they have not been pretrained for machine translation in supervised manner. Motivated by the recent remarkable suc- cess of (m)LLMM, in this work we investigate 1) the potential of prompting and fine-tuning of mLLMs for BLI and 2) how their capabilities compare against and complement current BLI approaches. We focus on how to expose word-level bilingual knowledge and elicit word translations from multi- lingual LLMs. To our best knowledge, we are the first to leverage autoregressive mLLMs for BLI."
We systematically study zero-shot and few- shot prompting for BLI with off-the-shelf encoder-decoder and decoder-only autoregressive mLLMs (Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020), respectively, In the few-shot scenario, propose to incorporate in-context amples from nearest neighbours into the prompts to boost the BLI performance. In order to guide the mLLMs* generation, we hand-craft 'mask-filling- style' and 'GPT-style' templates catering to the characteristics of different LLMs and conduct ex- tensive template search for BLI. In addition to pro-
1 Introduction and Motivation
Bilingual Lexicon Induction (BI.), also known as word translation, is a fundamental research topic in multilingual NLP that aims to bridge the lexi- cal gap between languages (Ruder et al., 2019). It has wide range of applications such machine translation (Artetxe et al., 2018b; Marchisio al., 2020; Chronopoulou et al., 2021) and cross-lingual transfer learning, especially for low-resource lan- guages (Sun et al., 2021; Zhou et al., 2021; Wang et al., 2022). Over the past decade, ftate-of-the-a
We point out that the work of Li al. (2022a) leverages only the encoder part of mT5 (the decoder is dropped) in one of their experiments to extract CLWEs for BLL In contrast, our experiments with m'5 utilise its full encoder-decader structure and generate the target words autorogressively, completely different from CLWE- E-Eaaad approaches.

---

viding complete and effective pipeline for BLI via prompting off-the-shelf mLLMs, we also in- vestigate BLI-oriented fine-tuning with the LLMs' own pretraining objectives, aiming at specialising mLLMs into 'few-shot word translators'.
space, BLI has been conducted via nearest neigh- bour retrieval. A detailed overview of different BLI principles can be found, e.g., in the work of Ruder et al. (2019).
More recently, researchers have attempted BLI by leveraging encoder-only multilingual masked language models (mMLMs) such as mBERT (De- vlin et al., 2019) and XLM-R (Conneau et al., 2020) whose neural architecture consists of only Trans- former encoders (Vaswani et al., 2017). Gonen et al. (2020) prompt mBERT with templates where the target word is replaced with a '<mask>' to- ken, and the language modelling head of mBERT outputs subword token fill the mask. This method is theoretically flawed because it cannot address the cases where the target word comprises two or more subword tokens. Therefore, Gonen et al. (2020) only evaluate BLI on a small set of 'toy" examples rather than standard BLI datasets. In terms of performance, this method lags far behind traditional BLI approaches. A more successful way of leveraging mMLMs is to extract decontex- tualised word representations from them (Zhang et al., 2021). The strongest CLWEs for BLI so far are learned via a two-stage contrastive approach combining both static (e.g., fastText) and mMLM- extracted features (Li et al., 2022a).
We conduct extensive experiments on two stan- dard BLI benchmarks, XLING (Glavaš al., 2019) and PanLex-BLI (Vulié et al., 2019), investigat- ing the word translation capabilittess of off-the- shelf mL.MMs (we adopt 18 models from L.M families) in various BLI setups. Our compre- hensive comparisons between mLLMs confirm, expected, that 1) different LLM families dis- play varying word translation capabilities and 2) stronger BLI performance tends to be associated with larger model sizes. To demonstrate the effec- tiveness of our prompt-based approach, we bench- mark our method against two SotA CLWE-bass based baselines. Notably, our approach with LLaMA130 outperforms the CLWE-based SotA on the XLING dataset by a considerable margin, establishing new SotA results on many language pairs in all BLI se- tups. Meanwhile, we also identify two limitations of BLI with mLLMs: 1) they are less competitive on the PanLex-BIe benchmark for rewourre languages; 2) CLWE-based approaches usually sup- port more languages than mLLMs. Finally, we run a series of insightful ablations and discuss the use-
Text-to-Text LLMs. Autoregressive LLMs have established new state-of-the-ar results on many NLP tasks. The prominent model groups include 1) encoder-decoder LLMs such as BART (Lewis et al., 2020) and T5 (Raffel et al., 2020); 2) Ope- nAl's decoder-only GPT series such as GPT-2 (Rad- ford et al., 2019), GPT-3 (Brown et al., 2020), and InstructGPT (Ouyang et al., 2022): 3) other GPT- like LLMs with specific improvements such as Chinchilla (Hoffmann et al., 2022), PaLM (Chowd- hery et al., 2022), and LLaMA LLM series (Tou- vron et al., 2023).
fulness of BLI-oriented fine-tuning, In short, our work validates the BLI capabilities of mLLMs and proposes new methodology for BLL. We hope that the combination of our comprehensive analyses and discussions, including on limitations, will pave the way for the development of stronger BLI sys- tems in the future. Our code is publicly available at github.ccom.
2 Related Work
Biliigual Lexicon Induction. Over the past decade, predominant BLI approaches have relied on the calculation of cross-lingual word embed- dings (CLWEs) where, in the most popular BLI variant, two transformation functions are learned to respectively map source and target monolingual static word embedding spaces into a shared cross- lingual space (Xing et al., 2015; Lample et al., 2018; Joulin et al., 2018; Artetxe al., 2018a; Alvarez-Melis and Jaakkola, 2018; Patra et al.. 2019; Mohiuddin et al., 2020; Glavaš and Vulić, 2020; Peng et al., 2021; Li et al., 2022a; Marchisio et al., 2022). Then, relying on the learned CLWE
Our work adopts five families of open-source text-to-text multilingual LLMs for BLI, including mT5 (Xue et al., 2021), mTO (Muennighoff et al., 2022), XGLM (Lin et al., 2022), mGPT (Shliazhko et al., 2022), and LLaMA (Touvron et al., 2023). We introduce each of these in more detail in $3.1. Unlike the encoder-only MLMs, text-to-text LLMs are theoretically capable of generating words con- sisting of arbitrary numbers of subword tokens.
Using mMLMs alone still underperforms purely fastText- hased methods since Ms are contextualised encoders pretrained for sentence-level tasks (Li et al., 2022a).

---

3 Methodology
categories; in what follows, we briefty introduce each of them and showcase some simple templates used for BLI-prompting' the LLMs.
Preliminaries and Terminology. As- suming a bilingual scenario with a souree language L" and target language L with their respec- tive vocabularies denoted as X' and y, the BLI task is typically formulated as a standard informa- tion retrieval task (Gaussier et al., 2004; Glavaš et al., 2019). The goal is to rank the words from ) with respect to their similarity to the input source word w The vocabulary size for each language is typically set 200k (Li et al., 2022a), covering the most frequent 200k word types in each lan- guage. A bilingual lexicon then comprises a set of one-to-one source and target word translation pairs (Mikolov et al., 2013), and we denote a word pair as -==w"", w²) where w E X, w E y.
BLI Task:
The first category includes mT5 and mTO,, two encoder-decoder LLM families that leverage the full Transformer architecture (Vaswani al., 2017). Each model family comes in five different sizes, and we evaluate all these ten models.
mT5 (Xue et al., 2021) is pretrained on the mC4 dataset covering 101 languages. The LLM lever- ages a span-corruption objective that tries to re- construct consecutive spans of dropped-out tokens replaced with special mask tokens.
m (Muennighoff et al., 2022) is a multitask- finetuned mLLM based instruction fine-tuning from the original mT5 model. The fine-tuning is conducted with English prompts on mTO's xP3 dataset spanning 46 languages.*
We assume set D3 of N available seed trans- lation pairs, constituting the so-called seed dictio- nary, which are used as the training set. Depending on the number of training pairs, the task is usually referred to as supervised BLI (typically, N 2 5K), semi-supervised BLI (e.g.. 0 < N < 1K), and un- supervised BLI (N = 0) in the literature (Artetxe et al., 2018a; Zhao et al., 2020; Li et al., 2022a). For convenience, we also refer to the unsupervised setup as zero-shot BLJ (N 0) and denote the setup with a handful of seed translation pairs as few-shot BLI (N > 0), corresponding to how we prompt mLLMs for BLI (we describe zero-shot and few-shot prompts for BLI later in $3). A test set Dr, where Ds 0 Dr = 0, is used for evaluation. In some cases, source word may have more than one ground-truth translation (i.e., there exist two or more word pairs in a BLI dictionary that share the same source word). Following previous work (Lample et al., 20;8; Glavaš et al., 2019; Li et al., 2022a), we consider a prediction correct as long as it is any of the ground-truth translations. The BLI scores are reported based on the standard Precision @ K (P@ K) BLI measure, where K de- notes the length of the ranked list.
For these two encoder-decoder style mLLMs, we aim derive prompts such that the first word of the output sequence serves as its guess for wV, Cater- ing to its span-corruption objective, for mT5 we propose to design mouk--filing-styyee English tem- plates where '<mask>' tokens are used as place- holders for the target words. Here is an example template: 'The L word w" in L is <mask>.' where 1.", T,M, and w are placeholders for the source language, target language, and the input source word, respectively." When a prompt based on this template is fed into mT5, its decoder will then output a sequence to fill the mask. Since mTO is based on mTS, we found that mask-fillng--stlle prompts are also applicable to mTO. However, un- like for mT5, the instruction-tuned mTO fits tem- plates without the '<mask>' token." For simplic- ity, we will denote all such templates without any '<mask>' tokens as *GPT-style templates'.
The second model category comprises XGLM, mGPT, and LLaMA as three decoder-only LLMs pretrained with causal LM losses. Our experiments involve five XGLM and two LLaMA models whose
3.1 Prompting Multilingual LLMs for BLI
"Among all the LLMs covered in our work, mTO is the only one trained for sentence-level translation tasks (machinc translation is one of the tasks during its multitusk fine-tuning). However, experimental results reported later indicate that this does not benefit BLI in prompting setups.
This study employs five families of mainstream multilingual text-to-text LLMs (mLLMs): mT5, mTo, XGLM, mGPT, and LLaMA.3 Based on their model structures, we group these models into two
'The '<mask>" token mT5 and mTO is actually 'cex- tra _i__ddd'', Therefore, an example of an actual prompt would be Thhe German word gebouw in French <extra id O>.'
We also experimented with mBART (Liu et al., 2020). However, we do not report the results with mBART since its BLI performance proved inferior in our preliminary investiga- tions.
'Tor instance. 'The L word w" in L is' may prompt m'TO to output u" to complete the input sentence, and 'How do you say w" in I.² would prompt mTO to generate w² to answer the question.

---

model sizes are no larger than 13B parameters. while mGPT only releases one model of size 1.4B. Unlike encoder-decoder LLMs for conditional gen- eration, the decoder-only causal LLMs first repeat the input sequence in their output, and we construct prompts that induce LLMs to produce wn immedi- ately after the repeated input sequence.
similarity measure for the retrieval. We again design mask-filling-style and GPT-style few-shot templates for the mLLMs, as discussed in $3.1. Similar to zero-shot prompts, for few-shot prompts we also extract the first word after removing spe- cial tokens (c.g., start-of-sentence,, padding, and '<mask>' tokens) and repeated input sequence (for decoder-only models) as the prediction of w".
XGLM (Lin et al., 2022) offers multilingual LLMs similar to GPT-3 (Brown et al., 2020) and is reported to outperform GPT-3 of comparable size in a series of tasks. The work builds a CC100-XL dataset based on Conneau et al. (2020) and Wenzek et al. (2020), and XGLM is pretrained with a subset of it covering 30 languages.
3.3 Template Design and BLI Inference
Template Design. We hand-craft in total 102 En- glish zero-shot and few-shot templates, respectively listed in Tables 10 and 11 of Appendix C. A small set of basic templates is fully manually designed, and additional variants then created by modi- fying or replacing the punctuation (see the tables). For each L.M,, we search for its best zero-shot template and best few-shot template on a randomly chosen language pair (German, French) and fix the template choices for experiments on all other lan- guage pairs. The best template choices for cach LLM are provided in Table 12 (Appendix C).
mGPT (Shliazhko et al., 2022) reproduces the GPT-3 structure and trained on 60 languages using Wikipedia and C4 data (Raffel et al., 2020).
LLaMA (Touvron et al., 2023) is a recently re- leased SotA LLM family trained on trillions of tokens exclusively from publicly available datasets; it supports 20 languages. LLaMA also features its efficient implementation, and adopts a series of recent improvements on normalisation, activation functions, and positional embeddings.
BLI Inference. At inference, we adopt beam search for both encoder-decoder and decoder-only LLMs and make the generator return the final beam ranked by their sequence scores. For each input prompt corresponding to w", we iterate through the returned set of sequences, and for each sequence extract the word after removing any redundant prefix content, as described in $3.2. The first word extracted that appears the target vocabulary is returned as our prediction of w
Our decoder-only LLMs solely leverage GPT- style prompts introduced above for mTOO since their tokenisers usually do not support '<mask>' tokens.
3.2 Retrieval-Augmented In-Context Learning
In $3.1, we presented some simple zero-shot prompts (i.e., prompts without in-context exam- ples) for BLI. However, recent work highlights the sho- capabilities of modern LLMs (Brown al., 2020). Therefore, we also investigate few- shot templates for improved BLI performance."
3.4 BLI-Oriented Fine-Tuning
This work predominantly focuses on 'learningless' experiments based on zero-shot and few-shot in- context setups with off-the-shelf mLLMSs for BLI without any fine-tuning. As a side experiment, we also aim to fine-tune smaller-scale mLLMs, making them specialise into few-shot word translators with our few-shot prompts as input. Our training set is still Ds, but now exclude retrieving input w" itself as in-context example. We combine the Ds of each language pair with which we fine- tune encoder-decoder mLLMS with mT5's span- corruption loss and fine-tune decoder-only LLMs with the standard causal LM objective.
We propose to retrieve the nearest neighbours of a source word which we use to construct in-context samples to boost BLI performance. More specifi- cally, given Ds and an input source word w", we extract n word pairs (wff, w.²) c Ds, < < such that w < < n are n nearest neigh- bours of wh in the auxiliary static monolingual word embedding space of X. This auxiliary space is based on pretrained fastText word embeddings (Bojanowski et al., 2017)$ and we use the cosine
"Note again that few-shot in-context learning does not require any actual fine-tuning LLMs. The word 'learning' here only refers inserting in-context examples into the input prompt sequence.
"For the rare cases where an in-context source word re- trieved may have more than one translation in Ds, we only keep the target word with the highest word frequency in fast- Text's training data.
"See Appendix B for more details about the fastText WEs used in our work.

---

4 Experimental Setup
detailed BLI scores are reported in Appendix D. Side Experiments. We conduct series of addi- tional experiments to further understand the BLI capabilities of mLLMs. 1) We investigate how the BLI performance is related the number of in- context examples (5K and 1K setups). 2) As an ab- lation study, we validate the usefulness of (our pro- posed) in-context samples extracted from nearest neighbours by comparing with randomly sampled in-context examples. 3) Finally, we fine-tune some of our relatively smaller-scale LLMs, including m tage XGLM56aM: and XGLM on our 5-shot templated BLI data (XLING) and fur- ther study the effectiveness of our BLI-oriented fine-tuning (5K and 1K setups). The training set includes all XLING language pairs, where the 5K and 1 K setups have 271, 754 and 55, 228 training instances respectively.
Training and Evaluation Data. Our experiments adopt two standard and publicly available BLI datasets, also used in a body of very recent BLI research (Vulić et al., 2020; Sachidananda et al., 2021; Aboagye et al., 2022: Li et al., 2022a,b; Vulié al., 2023). 1) XLING (Glavaš al., 2019) pro- vides BLI dictionaries covering 8 languages and 56 BLI directions. Among these 8 languages, are supported by all of our mLLMs: English (EN), French (FR), German (DL), Italian (IT), and Rus- sian (RU). Therefore, $5 mainly focuses on and reports results on all the 20 = 5 4 BLI direc- tions for the languages." 10 For each language pair, XLING provides test set DT of 2K transla- tion pairs. It also provides training sets Ds of 5K and 1K translation pairs, where the former is the superset of the latter. For brevity, we denote the cases |Dsl= = 5K, [Dsl = 1K, and [Dsl = 0 as the 5K setup, 1K setup, and unsupervised setup, respectively." 11 2) PanLex-BLI (Vulié et al., 2019) offers BLI lexicons spanning 15 lower-resource languages and all 210 BLI directions. We select three languages that are supported by most of our mLLMs Bulgarian (BG), Catalan (CA), and Hun- garian (HU). The test set size of PanLex-BIe BLI is also 2K; under the loworrresource assumption, we only focus on unsupervised and 1K BLI setups.
Hyperparameters. We first introduce our hyper- parameters for BLI inference. In our main experi- ments, we adopt v = 5 14 while in side experiments we further investigate and compare using different numbers of in-context examples n. Concerning the generation of output sequences, we adopt a beam size of 5 for all LLMs, and the maximum sequence length is for encoder-decoder models and 5 plus the input sequence length for decoder-only mod- els which first repeat the input sequence before generating new content. As for encoder-decoder LLMs, we use an evaluation batch size of 100 for smaller models and 8 for larger models as listed in Table 8 (Appendix B). Since the pretraining of decoder-only LLMs usually does not see padding tokens, adopt batch size of 1. 15 Following prior work (Li al., 2022a,b), all our hyperparam- eters are tuned on (German, French), randomly selected language pair.
Main Experiments. In our main experiments, we prompt 18 off-the-shelf models from 5 mLLM fam- ilies mentioned in $3.1 for BLI without any fine- un and systematically evaluate their BLI per- formance in three different BLI setups XLING and PanLex-BIa datasets introduced above. In 5K and 1K setups, 5-shot in-context learning is adopted for our mLLMs, while the unsupervised setup, zero-shot prompts are used. We compare the BLI scores between different mI.Mss from the perspectives of LLM family and model size, and we also benchmark their performance against two SotA CLWE-based baselines, introduced later. Se- lected results are summarised in $5.1 while full and
For 'BLI-oriented' fine-tuning, use the XLING data combining all language pairs, and the batch size is 16 for XGLM1.1B and 32 for bassaac.arge and XGLM564M We use AdamW (Loshchilov and Hutter, 2019) with be- tas = (0.9, 0.98) and a weight decay of 0.1. The learning rate is 2e-6 for mTS5 1e-6 for maTStarge. and 5 for XGLM5aMM concerning XGLM 5e-9 is adopted for the 5K setup and 2c-8 for the 1K setup. All the hyperparameters are tuned on
10 "oowever, in the appendix, we also provide the results with the remaining 3 languages: Croatian (IRR), Finnish (FI). and Turkish (TR).
11 Relattd work often also refers the 5K and IK cases supervised and semi- semi-supervised BLI setups, respectively (Li et al., 2022a).
12 A summary concerning the detailed information each
Unnees otherwise stated, we report 5-shot BLI results throughout our experiments. We found that a larger batch size may cause a drop in the BII performance.
mLLM available Appendix B.
13 "Experiments on BI 1ll--orice -orieeeeed fine-tuning for selection of ml.LMs are discussed later.

---

the same randomly chosen language pair (German, French). Each LLM is fine-tuned for at most 20 epochs capped at 12 hours on 1 80GB A100 GPU.
DE-+1
27.85
54.12
49.1
35.73
4345
57.64
56.44
17.54
53.64
46.37
32,66
66.56
32.47
54.76
13:54
0/130
19.15
64.2
61.3
4500
01.11
57.50
01.97
57:34
41.45
5618
45.15
$2.35
Re
36
59.51
5345
56.7
54.1
4112
M6.11
56.2
00353
57.29
410
57.12
4357
66.19
Baselines. We adopt the following two SotA CLWE-based approaches as our baselines; both are open-source. We follow their original sug- gested hyperparameter choices respectively for 5K (supervised), 1K semi-ssuerv and unsuper- vised BLI setups, and we re-verify that the hyper- parameters recommended are (ucar-)optimall The Cross-domain Similarity Local Scaling (CSLS) re- trieval (Lample et al., 2018) is adopted as recom- mended in the baselines.
the
96.5
13.34
16.71
45.5
98.66
59380
40.5
59.0
4710
014
82.15
40.38
34.9
5016
76.36
11aMA
64.64
51.79
46007
11:57
66.3
1516
53,12
-106
44.0
46.2
43.3
32:55
64.0
11.5
55.36
ES a
47:3
55.11
56.19
40,01
59.3
44.55
68.27
89/6
$2.46
59.6
51.51
43.8
51:06
55.31
19.45
11:13
51.6
4157
98-11
56.11
19.35
83:34
33.78
56.86
38:15
62-57
53,81
57.85
Zeo-Skal
YecHtr
CONTANTIVELLL
a770gl
XGLMM
Llaht/
DE >
44.64
35.75
29:31
58.57
22.65
43.3
VECMAP (Artetxe et al., 2018a) is one of the most representative BLI approaches based on static CLWES. It induces fastText-based CLWEs in vari- ous BLI supervision setups, and is notable for its effective self-learning mechanism, especially in weakly supervised and unsupervised BLI setups.
5/5
90/44
42.4
31:75
35-45
40.76
11
11:36
$4.41
47.51
43.15
35.77
52:34
35.00
51/11
32.16
53.87
53.09
32.19
46.83
36.45
36.76
-6.11
4199
32.92
42.91
26.16
46.72
Table I: Main results on 20 XLING BLI directions in
5K, 1K, and zero-shot (unsupervised) setups. Off-the- shelf mLLMs are used without any fine-tuning. Average P@ x 100% scores of each language going to and from other languages are reported. '-:: CONTRASTIVEBLI does not support unsupervised BLI
CONTRASTIVEBL.J (Li et al., 2022a) refines CLWES with a two-stage contrastive learning pro- cedure and reports the currently highest CLWE- based BLI scores on XLING and PanLex-BLI in 5K and 1K BLI setups. We adopt its strongest CLWES derived with both fastText and mBERT (Devlin et al., 2019). CONTRASTIVEBLI does not support unsupervised BLI.
XLING: Main Results. In Table 1, we report the BLI performance of the strongest single model from each LLM family (full results for each of the 18 LLMs are available in Appendix D). Our results on 20 XLING BLI directions reaffirm the leading position of LLaMA where the LLaMA L3B variant achieves the highest overall average BLI scores in all BLI setups, also outperforming CON- TRASTIVEBLI, the previous CLWe-based SotA on the same dataset. We speculate that LLMs are more adept at few-shot learning: LLaMA outper- forms VECMAP by circa 10 P@1 points in few-shot setups, but only by about 2 points in the zero-shot setup. It is also worth mentioning that in 5K and 1K setups, mT5 and XGLM4.5B beat VECMAP although they underperform CONTRASTIVEBLI; how- ever, in the zero-shot setup they still cannot match VECMAP.
BLI Evaluation. Following previous work, report the standard Precision @1 I (P@1) scores both for our methods and for baseline methods.
5 Results and Discussion
5.1 Main Results
Comparison between mLLMs. We compare the average BLI scores on 20 XLING BLI directions derived from all our 18 models from 5 LLM fam- ilies in Figure 1. In all (5K, 1K, and zero-shot) BLI setups, the same general trends are observed. 1) As expected, within the same mLLM family, larger models usually present stronger BLI ca- pabilities, although exceptional cases exist (e.g.. XGLM7,5B underperforms XGLM1,5B). 2) For encoder-decoder models, we find that mT5 outper- forms TT, showing that the instruction fine-tuning of mTO does not benefit BLI in our experimental setups; 3) LLaMA models achieve the strongest BLI performance among our 5 model families.
PanLex-BLI: Main Results. We present our re- sults on lower-resource languages from PanLex- BLI in Table 2. We also provide here only strongest model of each LLM family, while the full results from all 18 LLMs are available in Ap- pendix D). This time, LLaMA still outperforms other LLMs, but we report here XGLM7.0B instead which beats XGLM Unlike for XLING, we
"P@@ is the most authoritative metric for BLJ Other mea- sures such as P and Mean Reciprocal Rank (MRR) show similar trends (Lample et al., 2018; Li et al., 2022a).

---

XOLM
WG77
12
Noddd
Medd
Figure 1: Averaged BLI score model size (0.3B 13B): (left) 15K; (middle) =IK; (right) -
find that traditional CLWE-based approaches still outperform LLM-clicited BLI in general. This may reveal that current SotA mLLMs (sizeş 13B) still
[IK Betupl
e115.
4/70
XGLM
DCET
L.JMMuu
RG-CA
32.65
63.95
15.57
11:72
40.10
41:31
CA-1B0
31.54
10.06
36.2
22:73
41.53
80-00
14:77
25.45
36-57
36.33
CA-Hi
lack strong word translation capabilities for large number of languages and language pairs, even for those they currently cover.
Aug.
37.31
43.37
40.72
36,04
28,45
26.5
12.17
33.6
21.47
16,33
20.17
37.03
Put simply, while current mLLMs do exhibit strong performance for arguably high-resource lan- guages (from XLING), they still have deficien- cies with lower-resource languages as well as with their portability much larger number of languages, currently covered by more traditional BLI approaches (Li et al., 2022a). We leave to future work the investigation of larger mLLMs (e.g., LLaMAs0B) for BLI with lower-resource lan- guages and languages unseen by the mLLMs.
80-100
76.46
29:34
32.47
17.35
24.07
16.91
7.23
9.81
33.61
36.5
34.
36.99
22.41
26.55
RE-eCA
37.79
15.47
26.48
36.37
Ang
36.8
36.85
22.35
25.77
Table 2: Main results on PanLex-BLL BLI directions
in 1K and zero-shot (unsupervised) setups. Off- the-ff mLLMs are used without any fine-tuning. P@ x 100% scores are reported. by the LLM; 2) CONTRAsTIVEBLI does not support unsupervised BLI.
1) a language not supported
Statistical Significance. We conduct x" test com- paring LI LLaMA13B against the strongest single base- line in each BLI setup (i.e., CONTRASTIVEBLI in few- shot setups and VECMAP in the zero-shot setup) on the average BLI performance over 20 XLING and 6 PanLex-BLI BLI directions respectively, and estimate the p-values follows. 1) On XLING, p is 2. in the 5K setup, 8.5c-32 the 1K setup, and 4.3e-9 the zero-shot setup. 2) For PanLex- BLI, p is 1e-4 in the 1K setup and 1.9e-35 in the zero-shot setup. The p-values show that our main findings are clearly statistically significant."
80
of Saa SeeslDi-- lod-K
(DS--K
(sho0)
Figure 2: BLI scores averaged 20 BLI directions from XLING with respect to the number in-context examples to 10), with magge and LLaMA13e in both 5K and 1K BLI setups.
ant outperforms its zero-shot results by 5 P@) points. However, with higher values for (i.e., 2 5), the gains become saturated.
5.2 Further Analyses
n-Shot Prompting. To better understand the in- fluence of the number of in-context examples, we pick mT5e (an encoder-decoder LLM) and LLaMA13B 13B (a decoder-only LLM) and run exper- iments ranging from 0-shot to 10-shot. Figure 2 depicts their average BLI scores 20 XLING BLI directions in 5K and 1K setups, respectively. The results clearly demonstrate the usefulness of in-context learning. Even when having only one in- context example (one-shot), the same model vari-
Ablation Study. One key contribution of our work is that we propose to extract in-context examples from nearest neighbours. To validate the effec- tiveness this approach, we conduct an essential ablation study where we use randomly sampled in-context examples instead. As with our main ex- periments, present average scores 20 BLI directions from only one best LLM from each LLM family, and full results on all LLMs are available in Appendix D. Our results in Table demonstrate the following. 1) The nearest neighbour based 'NN (*K) scores for every LLM outperform the 'Ran-
17 By convention, < 0.05: statistically significant; 1e-3: statistically highly significant.

---

dom (*K)" scores by salient margin: this shows the effectiveness of in-context examples from near- est neighbours. 2) 'Random (*K)' outperforms 'Zero-Shot', showing that even randomly picked in-context examples can benefit BLI. 3) 'NN (5K) outperforms 'NN (IK)', which means that better in-context examples can be retrieved from larger (and more varied) 5K seed dictionary. 18 We further show that these findings are statistically significant via x² test and report the v-vuuess in Table 4.
Maeaa
KGLLM
Fine Tuned (5K)
36.11
46.68.00055
28.94
40,92:0000
Off The (helF(5K)
28.33
43:25
24.51
29.49
Fine-Tunal (1K)
25.61
38 18.48-0195
25.5
39.59.n017
Oh-The-Shelr (IK)
24.91
38.84
24.18
38,
Table 5: Comparisons between MLLMs before and after BLJ-oriented fine-tuning. Averaged BLI scores @P@ 1 x100%) on 20 XLING BLI directions.
BLI-oriented fine-tuning (i.e., training the mLLMs on BLI data with their own pretraining objectives) and may indicate the following. 1) With the current training approach and fixed amount of computa- tional budget, one may prioritise adopting off-the- shelf larger L.Ms (with in-context learning) rather than training smaller-scale oncs. 2) In future work, other training objectives and strategies should be investigated for improved BLI performance with mLLMs As another future research avenue, it is also worth extending the training to larger mLLMs and adopting novel fine-tuning techniques such prompt tuning (Lester et al., 2021), adapters (Li et al., 2020, 2023) and LoRA (Hu al., 2022).
mTTS.
XG an
@CPT
LLaMA pm
(S(
$2.83
38.00
$3.08
41.2
60.23
(IK)
49.37
35.42
51,18
40.13
$7.85
(5K)
34.97
Random (18)
45.93
33.90
49.92
39.38
56.11
Zero Zero-Shoe (Unsupervised)
41.99
32.90
42.91
26.18 18
48.72
Table 3: Ablation results. Averaged BLI scores (P@ 100%) on 20 XLING BLI directions. Rows 1-2: 5-shot prompting with in-context examples extracted from NN in Ds of size 5K and 1K. Rows 3-4; 5-shot prompting with random in-context examples in Ds of size 5K and 1K. Row 5: zero-shot prompting without any in-context examples.
Templates. Now, we additionally provide some preliminary findings from our template search." 19 1) Models from the same mLLM family may tend to prefer the same template. For example, Table 12 (Appendix C) shows that all five XGLM models prefer the same best zero-shot template and four of them share one best few-shot template. This phenomenon is to some extent seen also on mT5 (zero-shot and few-shot), m'TO (zero-shot and few- shot), and LLaMA (few-shot). This should be due to the same training data, training strategy, and model architecture adopted for all models in the same LLM family. 2) As already mentioned in $3.1, mTO is compatible with both malk-filling-- style and GPT-style templates: Table 12 shows that some mTO models prefer templates with '<mask>' and others do not. 3) Under the 'GPT-style' tem- plates, decoder-only models all prefer templates for sentence completion while some of the instruction- tuned mTO models prefer questions with "?"
poralse
4T55a
mT00a
XGLMM
IGPT
LLaMAup
NN(3K) Random (580
133:00
199-19
2.46:19
476-11
1.41e221
(IK) v. Random (IK)
9.3e-22
9.1e66
4.6e4
1.1e6
Random (SK) v. Zem-Shaa
1.7-43
18c00
14:-8
le411
84c-114
Randon (IK) Shot
2:46-28
3963
322-45
5.5--29
28:944
NN NN
22
1.3e
1.277
3
1.
Table 4: Statistical significance associated with Table 3. We conduct x2 tests and report p-values.
BLI-Oriented Fine- Fine-Tuning. The fine-tuning ex- periments, due to computational constraints, are conducted on four relatively smaller-scale LLMs. Table 5 reports each model's 's average performance on 20 XLING BLI directions before and after fine- tuning. We run fine-tuning experiments three times with different random seeds and report both mean scores and standard deviations. We only observe salient gains on mT5 models and XGLM in the 5K setups. For XGLM1,7B in both setups and all models in the 1K setup, the gains are smaller or even non-existent. Even in the 5K setup, the tuned mT5 still cannot match the off-the-shelf mTS and the tuned mT underperforms mT55 (cf. Table 18 in Appendix D). This may indicate some of the limitations of our proposed
5.3 Further Discussion
Few-Shot Learning for BLI. Our main results demonstrate that few-shot learning derives consis- tent gains over zero-shot prompting, For instance, HR- and IN saw 345 and 272 cases in
We also observe a slight edge of 'Random (5K)' over 'Random (1K)', and we speculate this might have to do with how XLING's G test set and seed dictionaries were created. In fact, the IK seed dictionary contains the most frequent IK words, and the test set words include less frequent 2K words.
19 Since we conduct template search only on a random lan- guage pair, these findings are yet to he verified hy future work for other language pairs.

---

their test sets respectively where few-shot learning makes the correct prediction but zero-shot learning fails (positive cases). There are only 85 and 87 cases where zero-shot prompting beats few-shot prompting (negative cases). We present 8 posi- tive examples and 4 negative examples for each of HR->EN and I comparing five-shot (5K setup) and zero-shot results with LLaMA in Ta- ble 19 (Appendix E). For instance, 'gušter (HR) lizard (EN)' and 'sezam (HR) + sesame' are two positive cases, their in-context examples being five different animal names and five plant names, which may help 1aB to narrow down the scope of the target word to animal and plant names respectively. Similarly, 'valcer (HR) >> waltz (EN) (a positive case) is associated with five in-context examples related to either music or dance. How- ever, few-shot learning does not always help. For example, in "eventuale (IT) + eventual (EN)' and 'scopre (IT) + discovers (EN)' translation tasks, the LLM seems to make a mistake due to directly copying one of the words provided in the in-context examples, whereas zero-shot prompting predicts the correct answers.
ing and opens up new possibilities for BLI research. In fact, LLM prompting provides a generic and straightforward way of leveraging external knowl- edge for BLI While we have demonstrated the effectiveness of in-context word translation exam- plcs, external information such as word defimition, parts-of-speech, spelling, and sentence translation pairs can also be integrated into text prompts. 2) On NMT and other related fields. Recent work has incorporated word translation pairs into text tem- plates to prompt L.LMs for sentence-level neural machine translation (NMT) and demonstrates that the bilingual lexical 'hints' lead to significant gains in NMT (Ghazvininejad et al., 2023; Jones et al., 2023). While ground-truth bilingual dictionary can be leveraged, BLI is able to provide word trans- lations for language pairs and words not covered in existing bilingual lexica. 20 Our work can provide strong word translation pairs for lexicon-enhanced MT, and the improved MT may further benefit, e.g., the field of cross- cronsalingual transfer learning via ALNSLAAE-TRANN
approaches (Conneau et al., 2018; Li et al., 2023).
Conclusion
BLI for EN and non-EN Languages. It is notewor- thy that the volume of data in English for mLLM pretraining often exceeds that in any other language (T5, mTO and LLaMA), and thus mLLMs may be biased, favouring BLI directions involving EN. However, we did not identify very clear clues indicating that their BLI performance (heavily) biased. In fact, in the 5K setup (see Table 1), al- though 'EN-** surpasses 'non-EN-*4 in absolute BLI scores (for each of our LLMs and also CLWE- based baselines), we meanwhile observe that 1) mTO,. achieves lower average score in '*->EN' than '*>''', and 2) for LLaMA13B. *E- lags behind both **-FRR and '*->IT'. Moreover, as an example, the I.LMM model supports 6 lan- guages resulting in 30 BLI directions, and 20 of them are between non-EN languages. LLaMA131 outperforms CLWe-based SotA in 16/20 cases and in 18/20 cases respectively in the 5K and IK setups for the non-EN pairs (cf. Tables 13 and 14). How- ever, again note that this might hold only for high-resource languages such as the ones covered in XLING.
This paper presents the first study on bilingual lexi- con induction (BLI) with multilingual text-to-text large language models (mLLMs). We develop the methodology to prompt mLLMs for BLI, conduct extensive template search, and systematically ex- periment with representative mLLM families (18 models) on a variety of zero-shot and few-shot BLI tasks. Relying on off theelff mLLMs, our ex- periments on the standard XLING dataset offer strong performance in all BLI setups, where our proposed few-shot prompting with in-context ex- amples from nearest neighbours outperforms the strongest CLWE-based SotA by considerable margin. However, our study also points out that prompting- based methods still need be success- fully extended to lower-resource languages. Fi- nally, we conduct a series of in-depth analyses covering variants of our few-shot prompting and preliminary investigations on BLI-oriented fine- tuning. Our key findings and comprehensive anal- yses may pave the way for the development of stronger mLLM-based BLI systems in the future.
Impact Statement. Here, we discuss the potential impact of our study on the following two aspects. 1) On future BLI research. Our work minimises the technical gap between BLI and based learn-
Moreover, given an input word, BLI can offer multiple plausible translations for the downstream NMT to consider, and we speculate this may, to some extent, increase the diver- sity of MT output.

---

Limitations
ported by Grace & Thomas C. H. Chan Cambridge International Scholarship. Ivan Vulié is supported by a personal Royal Society University Research Fellowship 'Inclusive and Sustainable Language Technology for a Truly Multilingual World' (no 221137; 2022-).
First, most recently released state-of-the-art mLLMs are still unable to support as many lan- guages as static word embeddings, which currently limits their wider portability. For instance, LLaMA supports 20 languages and XGLM supports 30 lan- guages, while fastText provides pretrained static WEs for 294 languages that can be used for the induction of static CLWEs, Intuitively, this is be- cause training LLMs that support more languages would require higher computational costs (with more training data and typically larger model sizes). We hope that researchers in the future can pretrain and release mLLMs that support larger set of linguistically diverse languages, which can thus probably extend the success of our approach to more languages and language families.
References
Prince Osei Aboagye, Jeff Phillips. Yan Zheng, Junpeng Wang, Chin-Chia Michael Yeh, Wei Zhang. Liang Wang, and Hao Yang. 2022. Normalization of lan- guage embeddings for rross-lingual alignment. International Conference on Learning Representa- tions.
David Alvarez-Melis and Tommi Jaakkola. 2018. Gromov-Wasserstrr Wasserstinn alignment word embedding spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 1881-1890, Brussels, Belgium. Association for Computational Linguistics.
Second, our work did not investigate open- source LLMs with more than 13B parameters" due to a large number of experiments conducted com- bined with our limited computing resources, and we did not evaluate any closed-source LLMs. Quite few tech companies and AT research labs have been training LLMs with 100+B and even 500+B parameters. We encourage interested readers who have access to adequate computing resources or specific closed-source LLMs to take a step further and investigate if larger LLMs can provide an even stronger BLI performance than reported in this par- ticular work, following the recipe presented in this work.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. A robust self-learning method for fully unsupervised cross-lingual mappings word embeddings. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics, pages 789-798. Melbourne, Australia, Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018b. Unsupervised neural ma- chine translation. In International Conference on Learning Representations.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- tion for Computational Linguistics,
Third, as also discussed in other BLI work (Li al., 2022b), existing BLI datasets did not control the synonyms and polysemy well and a sufficient detail. In fact, when constructing BLI datasets, is very difficult to collect all correct translations for each source word. Therefore, one limitation of BLI evaluation is that it cannot give credit to correct answers that are not included in the ground- truth translation set, and evaluation is typically con- ducted out-of-context. Constructing finer-gres graine BLI datasets with the help of qualified annotators (e.g.. linguists, typologists and bilingual speakers) is beyond the scope of this work.
Eleftheria Briakou, Colin Cherry, and George F. Foster. 2023. Searching for needles in a haystack: On the role of incidental bilingualism in palm's translation capability. ArXiv, abs/2305. 10266.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners In Ad- vances in Neural Information Processing Systems. volume 33, pages 1877-1901. Curran Associates, Inc.
Acknowledgements
We thank the anonymous reviewers and area chairs for their valuable feedback. Yaoyiran Li is sup-
html 22 For instance, I.LaMA also offers larger 30B and 65B model variants.
21 fasttext Faasstext.cc//ocs//eeeee hwtp///occeeeeepprrrreeeeeeeeeee
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul

---

Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- kumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gareía, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Re- won Child, Oleksandr Polozov, Katherine Lee, Zong- wei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Hellserellsteem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. ArXiv, abs/2204.02311.
Marjan Ghazvininejad. Hila Gonen, and Luke Zettle- moyer. 2023. Dictionary baasond phrase-level prompt- ing of large language models for machine translation. ArXiv, abs/2332.007856.
Goran Glavaš, Robert L Litschko, Sebastian Ruder, and Ivan Vulić, 2019. How to (properly) evaluate cross- lingual word embeddings: On strong baselines, com- parative analyses, and some misconceptions. In Pro- ceedings of the 57th Annual Meeting of the Associa- lion for Computational Linguistics. pages 710-721, Florence, Italy, Association for Computational Lin- guistics.
Goran Glavaš and Ivan Vulié. 2020. Non-linear instance-
tross-lingual mapping for non- isomorphic embedding spaces. In Proceedings of the 58th Anmual Meeting of the Association for Com- putational Linguistics, pages 7548-7555, Online. As- sociation for Computational Linguistics.
Hila Gonen, Shauli Ravfogel, Yanai Elazar, and Yoav Goldberg. 2020. It's not Greek to mBERT: Inducing word-level translations from multilingual BERT. Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 45-56, Online, Association for Computational Linguistics.
Alexandra Chronopoulou, Dario Stojanovski. and Alexander Fraser. 2021. Improving the lexical abil- ity of pretrained language models for unsupervised neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 173-180, Online. Association for Computational Linguistics.
Geert Heyman, Bregt Verreet, Ivan Vulié, and Marie- Francine Moens. 2019. Learning unsupervised multi- lingual word embeddings with incremental multilin- gual hubs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 1890-1902, Minneapolis, Minnesota. Association for Computational Linguistics.
Alexis Conncau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Oit, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 8440- 8451, Online. Association for Computational Lin- guistics.
Jordan Hoffmann. Sebastian Borgeaud. Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl. Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan. Erich Elsen, Jack W. Rae, Oriol Vinyals. and L. Sifre. 2022. Training compute-optimal large language models. ArXiv, abs/2203. 15556.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams. Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Jacob Devlin, Ming- Weig-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4171-4186, Minncapolis, Minnesota. Association for Computational Linguistics.
Alex Jones, Isaac Caswell, Ishan Saxena, and Orhan Firat. 2023. Bilex rx: Lexical data augmentation for massively multilingual machine translation. ArXiv, abs/2303.
Eric Gaussier, J.M. Renders. I. Matveeva, C. Goutte, and H. Dejean. 2004. A geometric view on bilingual lexicon extraction from comparable corpora. In Pro- ceedings of the 42nd Annual Meeting of the Associa- tion for Computational Linguistics, pages 526-533, Barcelona, Spain.
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave 2018. Loss in translation: Learning bilingual word mapping with retrieval criterion. Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language

---

Processing, pages 2979-2984, Brussels, Belgium. Association for Computational Linguistics.
multilingual generative language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 901 9-99052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Guillaume Lample. Alexis Conneau, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Proceed- ings of the International Conference on Learning Representations.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Chazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- rics, 8:726-742.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages -045- Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Ilya Loshchilov and Frank Hutter, 2019, Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Reprexenta- tions.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-seqqence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880. Online. Association for Computa- tional Linguisties.
Sasha Luccioni, Victor Schmidt, Alexandre Lacoste, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. In NeurlPS 2019 Workshop on Tackling Climate Change with Machine Learning.
Kelly Marchisio, Kevin Duh, and Philipp Kochn. 2020. When does unsupervised machine translation work? In Proceedings of the Fifth Conference on Machine Translation, pages 571-583, Online. Association for Computational Linguistics.
Yaoyiran Li, Ching- Yun Chang. Stephen Rawls, Ivan Vulié, and Anna Korhonen. 2023. Translation- enhanced multilingual text-to-image generation. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 91 9174-9193, Toronto, Canada. Association for Computational Linguisties.
Kelly Marchisio. Neha Verma, Kevin Duh, and Philipp Koehn. 2022. Iso Controlling the relative iso- morphism of word embedding spaces. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6019-6033, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yaoyiran Li, Fangyu Liu, Nigel Collier, Anna Korhonen, and Ivan Vulić. 2022a. Improving word translation via two-stage contrastive learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3353-43744 Dublin, Ireland. Association for Computational Linguistics.
Tomás Mikolov, Quoe V. Le, and Ilya Sutskever. 2013, Exploiting similarities among languages for machine translation. CoRR, abs/1309.4416
Tasnim Mohiuddin, M Saiful Bari, and Shafiq Joty. 2020. LNMap: Departures from isomorphic assump- tion in bilingual lexicon induction through non-linear mapping in latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2712-2723, Online. Associ- ation for Computational Linguistics.
Yaoyiran Li, Fangyu Liu, Ivan Vulić, and Anna Korho- nen. 2022b. Improving bilingual lexicon induction with cross-encoder reranking. In Findings of the As- sociation for Computational Linguistics: EMNLP 2022, pages 4 4100-4116, 11, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yaoyiran Li, Edoardo Maria Ponti, Ivan Vulié, and Anna Korhonen. 2020. Emergent communication pretrain- ing for few-shot machine translation. In Proceed- ings of the 28th International Conference on Com- putational Linguisties, pages 4716 1471, Barcelona. Spain (Online). International Committee on Compu- tational Linguistics.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Rose Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong. Hailey Schoelkopf, Xiangru Tang, Dragomir R. Radev, Alham Fikri Aji. Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual gen- eralization through multitask finetuning. ArXiv, abs/2211.01785
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang. Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal. Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' Horo, Jeff Wang, Luke Zettle- moyer, Zornitsa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2022. Few-shot learning with
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder,

---

Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744. Curran Associates, Inc.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg. Matthew R. Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with -umi-supervisoon in non-isometric embedding spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 184-193, Florence, Italy, Association for Computational Linguistics.
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George F. Foster. 2022. Prompting palm for translation: Assessing strategies and performance. ArXiv, abs/2211.09102.
Ivan Vulić, Goran Glavaš, Fangyu Liu, Nigel Collier, Edoardo Maria Ponti, and Anna Korhonen. 2023, Probing cross-lingual lexical knowledge from mul- tilingual sentence encoders. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 2089- 2105, Dubrovnik, Croatia, Association for Computa- tional Linguistics.
Xutan Peng, Chenghua Lin, and Mark Stevenson. 2021. Cross-lingual word embedding refinement by f norm optimisation. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 2690-2701, Online. As- sociation for Computational Linguistics.
Ivan Vulié, Goran Glavaš, Roi Reichart, and Anna Ko- rhonen. 2019. Do we really need fully unsupervised cross-lingual embeddings? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4407-4418, Hong Kong, China. Association for Com- putational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl Blog.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 11(14)):1667
Ivan Vulié. Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Prob- ing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference Empiri- cal Methods Natural Language Processing, pages 7222-7240, Online. Association for Computational Linguisties.
Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Inteligence Research, 65:569- 631.
Xinyi Wang. Sebastian Ruder, and Graham Neubig. 2022. Expanding pretrained models to thousands more languages via lexicon-bases adaptation. In Pro- ceedings of the 60th Annual Meeting of the Associa- rion for Computational Linguistics, pages 863-877, Dublin, Ireland. Association for Computational Lin- guistics.
Vin Sachidananda, Ziyi Yang. and Chenguang Zhu 2021. Filtered inner product projection for crosslin- gual embedding alignment. In International Confer- ence on Learning Representations.
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. mgpt: Few-shot learners go multilin- gual. ArXiv, abs/2204.07580.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Lan- guage Resources and Evaluation Conference, pages 4003 2003-401,, Marseille, France. European Language Resources Association.
Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, and David R. Mortensen. 2021. Cross- cultural similarity features for cross- lingual transfer learning of pragmatically motivated tasks. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 2403-2414, Online. Association for Computational Linguisties.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal trans- form for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1006-1011 I, Denver, Colorado. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' elien Rodriguez, Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 13971..
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and

---

Colin Raffel. 2021. mT3: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguisties.
Jinpeng Zhang, Baijun Ji, Nini Xiao, Xiangyu Duan, Min Zhang, Yangbin Shi, and Weihua Luo. 2021. Combining static word embeddings and contextual representations for bilingual lexicon induction. In Findings of the Association for Computational Lin- guistics: ACI-IJCNLP 2021, pages 2943-2955, On- line. Association for Computational Linguistics.
Xu Zhao, Zihao Wang, Yong Zhang, and Hao Wu. 2020. relaxed matching procedure for unsupervised BLI. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 3036-
3041, Online. Association for Computational Lin- guistics.
Yucheng Zhou, Xiubo Geng, Tao Shen, Wenqiang Zhang, and Daxin Jiang. 2021. Improving zero-shot cross-lingual transfer multilingual question an- swering over knowledge graph. In Proceedings of the 2021 Conference the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5822-5834, Online. Association for Computational Linguistics.

---

A
Languages
search Computing Services at the University of Cambridge, where each run leverages single Nvidia 80GB A100 GPU and 32x CPU cores.
Family
Language
Code
LLMs
Germanic
English
Software: Slurm 20.11.9 19.11 19, Python 3.9.7, PyTorch L.10.1 Transformers 4.28.1.
Germon
DC
Catalan
CA
mTS M.LLLMM
Romance
French
mTSS.mTIXXGLMM M.M.IIXGGG...MMLLLLM...
Italian
mDS5mTUXXG..MMMGPLLLLM
Runtime (Wall Time): We present the average single BLI direction (i.e., circa 2, 000 word pairs in XLING test set; the time required for loading the LLM and the dataset is not included) for each LLM Table 8. The per- epoch training time for BLJ-oriented fine-tuning is provided in Table 9.
Bulgarion
inference time
Slavie
Creatian HR
LLaMA
Russian
RU
mT5..TU.XGL.M M115:ATTUXGG.MMMGTPLLL.....
Turkic
Turkish
TR
mII3.nTO.GGLMMm.PP
Finnish
Uralic
Hungarian
MI1.mIOOmGGPLLLMM
Table 6: Languages covered in our experiments with their ISO 639-1 codes and the mL.M families that sup- port that language, categorized by language family. IE = Indo-European
Hyperparameter Search: As introduced in $3.3 and $4, our template selection and all our hyperpa- rameter search are conducted on a single randomly chosen language pair (German, French), following previous work (Li et al., 2022a,b). The learning rate for LLM fine-tuning is selected from [le 9, 5e 9, lc 2, 8, le 7, le 6, 2c 6, 2c 5, le 3].
B Reproducibility Checklist
BLI Data: We adopt two publicly available BLI
datasets." 23 24
Significance: We have discussed the significance of our main results and ablation results in the last paragraph of $5.1 and in Table 4 respectively, which demonstrates that our findings statisti- cally significant.
Static Word Embeddings: Following the datasets' own recommendations and other previ- ous work, we use the XLING-preprocessed fast- Text WEs trained on Wikipedia for XLING data and fastText WEs trained on Common Crawl Wikipedia for PanLex-Bii -LII, and the WEs trimmed to the most frequent 200K words for each language. For fair comparisons, we use the same set of fastText WEs both for the retrieval of near- est neighbours (in-context examples) and for the CLWE-based baselines.
Randomness: Our main experiments com- pletely deterministic since we rely on off-the-shelff LLMs without any fine-tuning, nearest neighbour retrieval for in-context examples (a deterministic retrieval algorithm), and the deterministic beam search. The randomness only exists in two parts of our side analysis. First, we use random in-context examples in our ablation study, and we verify our findings with statistical tests in Table 4. Second, the fine-tuning experiments do have randomness, and run fine-tuning three times for each model, reporting both average BLI performance and the standard deviation.
Pretrained LLMs and Parameter Counts: All the LLMs used in our experiments are publicly available from the huggingface, co model hub. We summarise their model identifiers and model sizes in Table 7. Please refer to each LLM's own copyright and licence before downloading, using, fine fineitunin, redistributing any LLM.
Carbon Footprint: All the experiments involved in this project including hyperparameter tuning, template search, BLI inference, and BLI-oriented fine-tuning of our LLMs consume circa 1, 650 A 100 GPU hours. Based on a publicly available 'machine learning emissions calculator' (Luccioni et al., 2019) and our computational infrastructure, we estimate that our work causes the emission of circa 200kg CO2 equivalents.
Source Code: Our code is publicly available at https://gii thub. com/ cambridge /p li.
Computing Infrastructure: We have run our code on Wilkes3, GPU cluster hosted by Re-
httt:s///ggihbb...o///ooo///////
rs. rs.html ottptppss/fstttte..ccceaeeeleler
tml
/nttps://mlco2. http::///llooo..hhbbbbb. www.gtiitttiiieeeeeeeeeeeeeeeeee
it/iii////iie io/impact/#compute

---

LLM
Number Parameters
C
Templates
"googlo/mtt-anne -mall"""""""
300, 176, 708
m T5ase
"google/mt5-bas
582, 280
We summarise all our zero-shot templates in Ta- ble 10 and few-shot templates in Table 11: these 102 templates constitute our 'template pool'. Each of Tables 10 and 11 is split into two parts for mask- filling-styli and GPT-style templates respectively as introduced in $3.1 and $3.2. In addition, we list the best zero-shot template and the best few- shot template for each of our 18 LLMs in Table 12. Again, as already mentioned $3.3, our template selection conducted on a randomly chosen lan- guage pair (German, French) where for few-show templates, the in-context examples are derived from seed dictionary of size 5K. While we do not have enough computational resources to calculate and do not have enough space present the performance of each template for each LLM on each XLING BLI direction (102 18 x 56 102, 816 scores), in the last paragraph of $5.2 we have discussed some preliminary findings only from our template scarch.
mT5ue
google/mtt
1, 129. 681,312
mT
"oogle/mm5-x1"
742, 619,
google/muS-xxl"
12, 921, 057,
300, 176, 708
m Thase
"bigscience/mtt--bar
582, 401,280
"bigseicncc//mtO-eee
229, 581,312
mTOa
bbigscience/ant--tt
742, 619,648
bigsccienc//mtO-xxl"
12,921, 057, 280
MreaM
Taacehook/sglm 564M"
564, 616
XGLM,
"facebookksllm122
732, 907, 008
XGLM2.8
'facebook/gll-22.998
2,941, 536
XGLM
"fccebookkglin-4555
4,532, 481,488
XGLM2,58
Tacebookksglm-7.5BB
7,492, 740,
mGPT
"serbhank ainhG
1,417,500 526
LLaMATB
"buggyllama/llana-00
738, 415, 616
LLaMA13
"huggyllama/lama-
13,015, 864,320
Table LLMs used in our experiments with their hugg ingface. model IDs and model sizes.
LLM
Batch Size (Inference)
0-Shot
5-Shot
mT'. m5
100
6s
7s
m Tusse
100
7s
8s
uuge
100
8s
12s
D Full BLI Results
mT55i
8
45s
50s
ml
52s
Here we present our full results on both XLING and PanLex-BLL. Table 13, 14, and 16 are our re- sults on all 56 XLING BLI directions in 5K, 1K, and zero-shot (unsupervised) BLI setups respec- tively. Table 15 and 17 are results for PanLex- BLII lower-resource languages (6 BLI directions) in 1K and shot (unsupervised) BLI setups. Note that an (m)LLM usually cannot support every language. and we use '-' denote this scenario. Throughout this paper. our expression 'a language is not sup- ported by an LLM' means that the language is not used for pretraining the L.M even the LLM's to- keniser may still be able to tokenise possibly many input sentences in the language. Table 18 shows the full ablation results for each of our 18 mLLMs.
m am
100
5s
6s
m0
6s
mT0 muguee
100
8s
14s
mTO,1
46s
50s
mT00al
57s
78s
XGLMs84M
213s
225s
XGLM1,7B
228s
237s
XGI.M2.8
343s
366s
XGLM,.B
336s
XGLM2,68
382s
461s
mGPT
192s
210s
LLaMARR
328s
463s
LLaMA11B
131s
636s
Table 8: Inference time (in seconds) of each LLM with
0-Shot and S-hhot prompts respectively.
E Translation Examples
LLM
Batch Size (Training)
Setup
1K Setup
To illustrate how few-shot learning improves
32
17 min
4 min
BLI we present some of our BLI results with
32
38 min
8 min
LLaMA13B in Table 19 comparing five-shot and
XGLMssuM
zero-shot prompting on HR EEN and IT->EN BLI test sets.
min Table 9: Per-epoch training time (in minutes) each LLM with 5-Shot prompts in 5K and 1K setups respec- tively.
XGLM1.7B
15 min

---

MaskkFilling-Styyl Templates (Zero-Shot Prompting)
The word wn in 1 is: <masks.
2
The word of in T is: <mask>.
3
The word wf in L is: <mask>
4
The word wf in D is <mask>
5 The I." word w in T is: <masks.
6
The I." word w/ in E <masko.
The L word "f in _ is: <masko. The word w in L is: <masks The L word "f in 1 <masks "w"" in L <mask>. 'n"" in 1 <mask>
8
The L word 'w in [ is <mask>,
10
The word w in L is <mask>
12
The word 'w in Z <mask>
w
<mask>.
16
ut is: <mask>
17 What the translation of the word wt into TM? emasks,
18
What the translation of the word into I.? <masks.
19 What the translation of the E word w"" into [P? <masks.
20 What the translation of the E word w" into I <mask>,
21 The translation of the word "w" into I is <masko.
22 The translation of the word u² into 12 is <masks.
23 The translation of the L word 'w2" into I is <masks.
24
How do you say "w"" in EP? <masks.
25 How do you say w" in Ld? <mask>.
26 How do you say the L word 'w"" in Lx? <mask>.
27 How do you say the L" word w" in Ld? <masks,
28 Translate the word 'w"' into LP; <masks.
Translate the word w" into L <masks.
30 Translate the word w" into D.; <mask>
Translatc w" into <masks.
Translatc the word w into L <masks,
Translate the word w" into I2; <mask>
Translate from 13; w <masks. Translate from 18; emasks.
Translate from I3;; w* <mask>
Translate from L 1 w"=> <mask>
GPT-Style Templates (Zero-Shot Prompting)
The word 'tp 1.2 word
The word
word w in TM is:
word wff in L The f." word "n in 1.2 is
word in I is: 'n" T
46
w" in E
47
Translate the word 'w"" into L*; Translate from L to L.; w"
48
Translate the word wn into L.: Translate from L to IP: wF=>
49
50
51
Translate w" into L?:
52
Translate the L word w into 1.;
53
Translate the L word 'w." into L*;
54
What the translation of the word 'w"' into LP?
55
What the translation of the word w into 1 The translation of the word uz into I is
56
The translation of the word "w=" into E is
57
The translation of the E word 'w=" into L is
59
The translation of the L word w² into E is
How you say "n in I??
How you say w" in LP?
How you say in
you say w
How you say the word 'w=" in
How you say the word wf in
Q:
L translation Ac
Table 10: Our 66 templates for zero-shot prompting. These include 37 mask- mask-filling-stee silling-style templates (template IDs: ~ 37) and 29 GPT-style templates (template IDs: 38 66). In our experiments, the '<mask>> token is "sxxtraidd id_ 0>' for mT5 and mTO.

---

Mask-Filling-Style Templates (Few-Shof Prompting)
Translate from 1 to L uf->uf wf->ww w <mask>,
68
Translate from 1 to L": u uf- ut->u = <mask>.
69
Translate from L to L": w -uf wg->ww w* <mask>
70
Translate from 1 to L": of uf-suf us->ug. =² <mask>
Translate from L to L"; uf= ud=sud w"=> <mask>.
72
Translate from 1. to IP: wif=bud oif=bug ===> <mask>.
73
Translate from I. to L": of==e ug=suly <masko
74
Translate from L to L": wf=suuf mf=suu, ="=> <mask>>>
75
The word wf in I. is w/. The word ug L is uy.. The word w" in 1." <masks.
76
The L word wf in L is w.. The L word ws in L is w/. The L word w"
<mask>.
77
The L word 'usf" in L is "wf The L word w in 1 is w The L' word "w" in L is 'mmask''.
78
The L word wf in L is w/. The L word wy in 1 is wy. The L word w² in L is <mask>.
GPT-Style Templates Shot Prompting)
79
Translate from 1, to LP: of soalf wi of- wn Translate from 1 to L.: of off- wug-ww. uf-> Translate from 1 to 1 ufe ug=swy w">>
80
82
Translate from 1 to L"; uf= u bif= =²=>
83
The word wf in L is w/. The word wg in L is wy. The word wf in 1 is
84
The word wf in L is wf. The word wg L wy. The word w² in L is:
85
The 1, word uf L is up. The L word wg in L8 is wy. The L word w
86
The L word uf in L is wp. The L word wp in L is wz. The L word w" in L is:
87
The word wff in 1.2 is w/. The word wz in 12" is way The word w" in 1 is
88
The word wf in L is w.. The word wg in L is w.. The word w" in L is:
The 1/" word u/ L uy. The L word wy in L is wy. The L word wt in L is The L word wi in L is wf. The L word wif in L is wp. The L word w" in L The word "wf" in L is wf The word woy in L is wy. The word 'wff in L is
90
91
92
The word "wf" in L is wf. The word "uz" in L is wo. The word 'w in L is:
93
The L word 'uf" in L w.. The L word 'wg" in L is wy. The L word wn in L is The I," word 'uf" 1 wf- The I word wws" in T is w... The I/" word "n in T." The word 'ws" in L is wi. The word 'ug"" in
95
wy. The word "=" in i
96
The word "wf" in L is uy, The word 'ug" in E is wy. The word 'w"" in I is:
97
The L word 'wf" in L is wf. The L word 'wo'" in L is wy. The L word "w"" in 1 is
98
The L word 'uf" L wp. The L word 'ug'' in L is wy. The L word "w"" in L is:
99
The word wf in L is wp. The word ui in L is wf. How do you say w² in Ld?
100
The L word wf in L is wf. The L word wi in L is uyy How do you say the L word w² in L
101
The word' "ws" in L is wf. The word 'ws' in L is wo. How do you say "w" in L
102
The 1/2 word 'usf in L is w.. The 122 word 'ng" in T is wy. How do you say the I word "w"" 1.2
Table 11: Our 36 templates for few-shot prompting. For simplicity, we present only two in-context examples in each template. These include 12 mask-filling- style templates (template IDs: 67 ~ 78) and 24 GPT-style templates (template IDs: 79 102). In our experiments, the '<mask>> token '<extra_id id O> for mT5 and mT0.

---

LLM
Best Template (Zero-Shot)
The word 'w/" 1.9 is: <masks.
buse
Translate the word wi into 1%: <mask>,
mT Sarge
The L word 'w"" in L is; <mask>. The I." word 'f in 12 is: <mask>. The L word " in L is: <masks
mT5a1
m p5
mT Tamalll
Translate from L to Lt; w <mask> Translate from I. to DM; =/=> <mask>.
m Touse
maaage
Q: What L translation of A:
mT0,1
How du you say the L word 'w"" in LP?
mT0xa
How do you say the L word "w" in L
XGLM5
The I." word uz in is:
XGLM
The I." word u/" in 1." is:
XGLM2.0B
The L word w in L is:
XGLM2.5B
The L" word uz in L is:
XGLMM7.BB
The I." word az" in is:
mGPT
Translate the L word w" into L":
LLaMAyB
The L word w in L is:
LLaMA
Translate from 1 to L": =/=>>
LLM
Best Template (Few-Shot)
&mall
The word u/ L wif. The word wy in L wyz The word w" in L <mask>. The word ui L is wf. The word wi in is udy. The word w" in
b1se
<mask>. The L." word w/f in L is w.. The I." word wy in I is wy. The L word u² in L is <mask>.
maage
mT5a
The I," word u/ in 1 is wp, The E" word wg in 1 is my. The 1/" word wn in 1." is <mask>.
m mT5
The L" word ui in L w.. The L word wn L is uy.. The L word w in L <masks.
m m000mm
The word 'uf" in _ is wV. The word 'ws" in L is w.. How du you say "w" in L
m Touse
The L word of in L wff. The I word w5 in L is wy.. The L word w2 in L8
mToange
The L word 'wf"' in L is 'u.". The L word 'uj" L is 'wy"" The L word 'w." in L 'emaske"'
mT0a
The L word of in L is w². The L word wz in L is wy. The L word w= in L is:
mTUsat
The L word wi in L is wy. The L word wi in L is wy The L word wf in L is:
XGLM550M
The word uf in f." is wx. The word wi in E uzy. The word w² 1."
XGLM
The I." word w/f in /." is wo.. The I." word wi in 1 is w The E. word wf in 1.2 is
XGLM2.0B
The L" word uif in L is w.. The 1 word
in L is w.. The L word w² in L is
XGLM2.5
The L. word u/f in 1 w.. The 1." word wg in 1 is w.. The L word u² 1
M7,5B
The E." word w/f in
w.. The 1." word wz in 1 is w.. The I." word ar" in T.2
mGPT
The L word un in L w/. The L word wi in L is w.. The L word w" L
LLaMAyB
The L. word 'wf"" in L is w.. The L word 'un"" in L is wy. The L word "w"" in L
15aMA
The E" word 'wff" in 1 is n.. The I. word 'ug" in L is wy. The 1/" word "w" in 1." is
Table 12: The best template for each LLM respectively for zero-shot and few-shot prompting. For simplicity, we present only two in-context examples in each template in the few-shot setup.

---

Bil.u
DTLL
RITSL
XCLM
MILM
XCLMMs
XCLM
11.50
6145
7.45
11.39
25.66
31.13
35.3
0:5
4.99
3,0
12:01
27.18
21.55
31.03
25.33
12.77
7006
196T
56.55
2355
2012
43.09
16-06
1375
63.33
13.1
2011
3135
41.51
4448
317
371
12.05
34.6
27-2
4032
15:90
12.00
$1.5
23335
01-188
01-41
2135
56.77
1230
2137
300
4345
40.13
141
24.55
30A7
1541
12.00
43.1
48.11
5735
2235
3516
3455
41.0
44.35
6400
45.45
61-00
1332
40.1
10311
215
313
43.77
44:1
1.5
451
635
2111
2055
$4.0
2003
40.4
43.0
1335
56.39
2675
538
26.00
51.0
EX-00
318
162
10.55
633
05.35
K35
9555
21.15
365
13.5
3435
131
5000
63
1635
$6.35
25.
313
4666
4331
10.3
1355
21.77
300
4331
4881
016
435
635
2123
3600
2586
4615
2034
4814
5506
17.0
2339
5U33
1738
54.2
872
556
2145
5633
635
261
6435
625
340
1217
0633
23.35
33.35
200
3432
031
19.2
$4.00
26.11
13.33
5775
2011
2000
4335
54.99
25.99
4316
1130
35.12
43.5
53.55
5535
5356
5837
15-1
1335
1545
285
16%
205
206
1836
155
1541
0.35
130
100
13.55
152
6.
7153
535
1150
200
1991
15.5
1998
1133
41.
2216
21.06
4421
4699
1916
636
1630
3053
1805
1100
605
355
068
1155
2700
19.55
301
125
3490
115
1881
430
1151
3672
3431
037
239
36.29
21/15
3135
3010
1141
1E1
16.1
316
3186
10-
(3.0
1637
60-1
4540
6472
672
11.1
3180
5131
23.08
66433
1.5
533
2.55
12.04
10.55
33.77
11.54
51.37
03.37
9415
(8.3
48.35
51.13
80-I1
4576
5156
1139
28.4
41.5
40.3
0.9
4.3
715
2435
3626
23.47
12.38
46.5
2515
238
41.3
535
2333
2332
363
022
23.30
363
43.30
3398
4-13
251
747
1339
2336
31.0
36.2
1.00
5.5
384
1426
2013
3401
23.4
3121
33.22
3577
361
203
336
22.0
3130
3421
Te-th
3599
3633
138
459
349
3607
36.95
2001
11-15
303
4002
811
1431
2633
3354
3125
212
337
335
1632
230
1231
21.7
3180
1299
3400
208
434
3833
Table 13: Full 5-shot BLI results (P@ 100%% 56 XLING BLI directions with 5K seed translation pairs.
Off-the-shelf LLMs are used without any fine-tuning. 1-: language in the pair is not supported by the LLM.

---

8.400
44.1
5.31
13.57
36.3
13.35
0.35
124
25.74
33:47
23:34
112
54.01
10:12
3533
43:37
5577
16.76
01-18
43.55
08.85
43.90
5015
13.55
15.00
16-3
58.07
4323
6.8
32.6
192
40.5
43.3
16-11K
13.9
14.99
110
11.3
1.44
12.
33.00
17.45
159
16
36.7
3111
330
2941
0.37
3.78
2317
1123
2103
2855
2330
2721
1450
166
3833
16.6
1595
1663
0.1
641
9.10
1000
14.60
9671
13.5
1201
151
51.
60-01
2738
450
231
23
233
31/6
462
2135
432
188
655
6135
11.5
11m
86
15
16.5
110
03.55
87.3
16.0
-51
614
9485
2234
31.7
4331
23.09
0.5
1457
5000
45.22
61.5
5122
2534
2185
5409
4542
60.5
3346
4335
1561
916
380
3000
21.33
751
1585
113
273
1236
14.55
11.5
085
015
615
136
13.6
$3.5
111
53.5
56.
53.5
11.
K3-04
2537
233
3535
100
35.3
23.9
42.4
40.03
21.44
454
2003
2105
2243
16-13
1117
4334
530
1822
2506
34.5
4642
0.06
201
234
1225
2611
13.0
2655
75.55
8-18
2429
655
13.55
2708
38.55
-127
30
2123
2335
2223
4403
1226
6723
2508
368
14.8
00-0
2614
6.7
16:55
3631
234
15.35
5112
37.1
-22
L210
3-0
26-1
4145
45
2126
2339
3231
0.7
43
360
1334
21,5
2010
2457
15.5
27113
1335
230
042
021
3600
1721
3629
15.5
423
13-4h
2014
4531
44.5
47.65
46.19
2566
41.5
15/0
43:5
14-40
44.54
11-4k
1386
1555
25.77
43.33
36.44
1.20
11.9
2015
22.99
40.27
24.05
25.35
3507
1431
5128
7.51
22.55
36.34
23.11
36.14
41.35
45.4
55.47
1537
199
11
31.00
16.00
00.7
16.3
17.6
1916
F331
43.55
0.51
1814
33.
022
23.05
3664
48.55
45.55
44.56
1236
15.69
196
521
1851
0.7
185
1734
17.75
75.03
78:13
13115
19.00
14.1
1t-i
41.09
3651
0.99
2.5
3431
26.14
3331
35:42
$7.00
3519
18.11
1.81
18511
3638
0.50
233
13.33
11.1
3133
26:44
3645
11/66
36/
3384
5.4
11:1
367
300
17.3%
3000
15.99
15.00
15:05
1155
331
7505
56
10
17.01
2.00
012
1300
3019
75.6
3006
314
1855
2538
1.57
31
30.6
18.0
354
35.5
32.66
32.31
Table 14: Full 5-shot BLI results (P@ 1 x 100%% on 56 XLING BLI directions with 1K seed translation pairs. Off-the-shelf LLMs are used without any fine-tuning. 1-: language in the pair is not supported by the LLM..
4333
233
018
6.12
3000
35.02
23.00
10
1577
61.1
0/6
2234
51.65
36.1
8.6
1.56
436
1437
2548
2061
11.32
111
1:33
2264
27
6.1
3017
43.2
14.00
11.11
6.8
13.5
3016
1448
1748
841
5.90
638
1655
3656
34.55
44:01
Avg.
4337
411
1.35
2233
0.34
231
338
Table 15: 5-shot BLI results (P@1 100%%) on PanLex-BLa BLI with 1K seed translation pairs. Off-the-shelf LLMs are used without any fine-tuning. '-": a language in the pair is not supported by the LLM.

---

25.21
0.31
1:25
12:00
25.77
21.33
4.80
4.22
240
14.24
22:00
9.25
15/20
26.0
25:57
3611
11.39
34.4
1:45
1:43
11-2)
24:14
2133
3332
12:44
5.99
51.23
2145
25.35
13:34
11:11
1123
56.55
51.4
41.33
7.0
11:45
37.66
13.57
me-iit
57 18
28.66
78.56
IR 10c
7212
31.1
35.72
48.41
2647
11.23
25.5
25.67
01:33
20.36
35.31
20.59
1.32
55.35
1.01
19.00
20.5
33.55
10.2
1351
3315
14.07
1.76
21.95
13.07
41.33
18.3
73.6
6.71
15.57
S2.11
37.07
15.33
1558
03.8
43.11
02 ima
23.5
5.0
6.99
15.25
1852
36.4
4.5
3.51
11.33
13.1
422
10.22
16.9
19.51
1983
11.48
18
26.46
7:89
6.26
1536
18.59
4.5
14:20
17:3
24-17
21.33
14.35
54.3
15.5
13.35
23.47
36.35
96.33
16.14
1136
33.38
31.55
15.96
26.11
56.11
41.93
51.
21.65
42.1
26:55
5.35
535
17:35
2611
37.15
62
435
522
15.05
27.45
21.5
56.25
31.1
37.7
12.1
m-
4694
8.84
5.7
1696
357
33.0
594
55
13,45
19.00
22.23
533
1770
3076
26.88
31.3
16.14
13
1155
10.5
367
53.15
359
310
24.1
M63
31.33
52.5
574
1176
11.55
41.5
12.
154
67.85
55.55
90.55
12.85
40.43
2.25
1259
23.55
36.55
40.97
3.91
27:25
32.33
535
1725
40090
42.43
1933
16.2
4622
22.5
4.53
33
210
25.5
35.15
201
49
4.00
1310
2431
522
21.
22.33
211
10.3
M-9
13.36
22.17
1.6
133
1320
12.25
71.8
550
15:12
26.1
76.66
75.99
7.6
1901
11.41
18.8
21.12
24.8
6:33
6.55
13.1
3133
26.5
4.7
15.00
2194
21.55
280
1641
14.97
106
121
21.44
25.25
25.42
22.44
12.94
21.36
31.74
0.4
1.5
120
30.59
458
9.64
16.34
24.86
2955
1.9
in i
2683
3151
9634
22.55
31.1
31.2
24.14
33.58
1n-1nt
34.24
35.1
31.77
10-11
24.11
3531
189
10-114
85:22
1.43
13.60
2531
41.31
47.99
13.25
0.23
2646
52.97
41.93
42.21
21.41
3741
18-11
1531
15.14
56.05
17.31
5.00
10.38
50.35
665
16.10
$1.85
38.55
5000
1964
19-480
33.50
0.36
0.11
26.66
11 11
36.3
033
1.65
E356
1573
533
23.95
55.01
56.3
33:33
17.00
10-417
4184
11:11
25.55
935
540
15.55
8.6
14.09
56.35
49:51
38.87
55.55
47:3
23.94
0.41
1958
20.45
2644
1.55
13:33
13
2033
2032
31.21
28.12
15.33
3132
35
28.7
4.25
3,32
7.26
13.87
19.75
4.3
8.37
10.99
4.5
12.5
1897
12.33
2033
0.30
10-4m
32.66
5.11
6:11
143
23.35
33..
1.47
3.01
11.02
13.34
17.30
378
1540
18.31
2231
35.55
3138
21.14
3.33
5.99
1557
3135
25.47
121
433
1.66
16.5
3.0
436
15.
3138
3038
11:38
18.
1K
31.44
3.43
6.31
1228
20.3
36.81
5.13
5.99
1.71
14.54
20.35
40
14.3
20.3
36.2
25.35
21.55
25.86
1.5
579
1366
17.3
36.11
343
433
196
19:3
18.04
4.8
10.99
16.99
15.28
18.
13.08
12.89
0.13
0.45
114
1933
3038
1.38
0.7
7.0
14.34
0.3
14.
11:85
23.85
5433
11.00
1537
3335
2.99
835
Table 16: Full zero-shot BLI results (P@ 100%) on 56 XLING BLJ directions without any seed translation pair (our zero-shot setup is also known the unsupervised BLI setup for CLWE- based approaches). tff-the-shelf LLMs are used without any fine-tuning. "-": language in the pair is not supported by the L.M.
Ir75.ad
6I75:
MCLMuun
REPT
E1aMbby
12-MA
R0-HCA
0.83
4.79
17:35
15.35
3.6
14
15.6
31.44
515
1.1
22.13
34.34
14.1
1.25
80 1ad
36.46
0.35
0.29
0.62
118
22.4
0.33
0.58
0.2
333
16.44
935
38.34
25.5
CA. 100
31.5
0.55
8334
15.31
21.55
38.99
65
4.34
1.59
1254
22.99
23.7
34.33
A0HHCA
77.19
8.6
6.27
16:72
312
25.07
431
14:01
11.3
31:88
36.17
Table 17: Zero-shot BLI results (P@ I x (00%) on ex-LLI without any seed translation pair (our zero-shot setup is also known the unsupervised BLI setup for CLWE-based approaches). Off-the-shelf LLMs used without any fine-tuning. '- a language in the pair is not supported by the L.M..
MULMue
X0LMi
MMLMoo
NOLM
aAPI
LLaMhhu
MN
14,56
25.33
43.58
52.53
1,43
635
24.11
33.40
41.54
2008
11.28
41.2
$1.53
50
11/77
Kindon (SK)
11.36
2588
33.33
4333
4635
1121
533
416
25.3
3430
21.1
23.44
4635
2038
4810
3634
53.32
55335
1645
19.9
0.55
4545
014
44
128
730
359
21.77
3401
3531
99.91
$611
7.6
11:33
33.65
01.0
790
333
2517
33.0
100
3580
36.0
25.85
1830
Table 18: Full ablation results with all our 18 mLLMs. Avg. BLI scores (P@) 1000% 20 XLING BLI directions. Row 1 ~ 2: five-shot prompting with in-context examples extracted from nearest neighbours (NN) in Ds of size 5K and IK. Row ~ 4: five-shot prompting with random in-context examples in Ds of size 5K and 1K. Row 5: zero-shot prompting without any in-context examples.

---

Ground- Truth Trandation Pair
Method
Context
Prediction
brisati
D Shot
beeak
S-Shot
isbrisati -delete, kopintti sprviti samend, depuniti deguniti- haappeemen, plsati-wwite
crase
B-Shot
Jryful
Jesty (HK) - edible (EN)
S Shot
jodt-neat, Grbnjk--gariic portt--wegetbble grack--piss. bijka-ppam
edible
slap (MR) watcrfall (CN)
0 Shot
slaposi kunjon oeasyon, jezero lloke, porok screck, poteci-istreats
waterfall
step
D-Shot
(HK) (EN)
odkestur -iouceessa, koscett rcencort bolet shallei, simfonija saymphouy, Marir--piaao
Socan (UX) -3 carp (CN)
B Shot
5 Shht
puons-immei, sihu-fishh obolowe-ficher, geran-vaww. hismnn-hirarrz
shurk
turp
(HB) owls (FK)
D. Shot
-aninals poea-beed malke 2rotiije-atiimal
slept
gubter (uk) Ulard (UN)
S-Shot
Sakur-rrt, reija-nonkk,, zijer- kerzjast-stttt -ttrril mxjmn-
saram (IR) senomm (PN)
B Shot
Shot
graakk-peas, raficc-nommma,, povste-negetblle caflcc-romaannn rommates, smoken--tig
sesame
cumua
odgodena postponed (uN)
D. Shot
S-Shot
nógoditi -ddaay, suupooedd rajac-
provedess scondicted
pudqpued delayod
lijenet (In) werrecy (FN)
Shot
kestted-rsttreee, okyno-- odlinoo-rrrhh bjju--secrel, -acerryy
discretion
secrecy
B Shot
progrosis
progtoza (HK) prognosts (EN)
S.Sbot
peognoe -oorecasss dijanno :dagness, tempija -hemgy, lijeleij treatment, asini -astlmm
fxeccsts
0-Shet
disoulation
otspanje disenuttonn
ddrinnj--dnnogg. kiveline-aacih firinnje--fiheingg duteearge- conrtje-elok
melling
Mrppurtare (IT) endure (EN)
Shot
alievine -esse, portar -bring superare suparar--iexced rexceed, odine -hate, sininizzan -aininize
endure
suppoct
8-Shot
quick
snello (IT) slender (EN)
Shhtt
flosible--lss ilesi rodi-aail sutile-mblll, legoomenc uqueoo-uppaanne
slender
bindi Mindi
Shot
tulladdss -tai, ingun -oongus anbo--anabic
D-Shot
Neycle
will (PK)
Shot
cepello-sbaa, mastelin -cape vie--faes, concin-spilloo vodin-drrss
vall
6 Shot
prikly
(1)) plum (UN)
Shot
ciliegia -iheery smigli--wwille sazzro- iginger mannelats -imm, aglio-aparii
plom
propuroo propane
Progeno-hyyrroge acsto-racid, cctonno--cuttoon comtustibbl fuels, polimero-polymer
propane
poneelo (19) kmob (sN)
Shot
masiglis -undle regolablle vadjustable, manigle -aandles copertioo -lid, cinghia
apple
kunh
alopiane (IT) plateau (12)
0 Shot
Shot
mutagne-mmounans, plonee-pliin, valbni-ddale piames-plain. valle-vtaleey
high
B-Shat
eventiale (EN)
5.httt
accessssio ruccessary, valutare -gauge possibie- sposible. recessatio snceded, riciedee veguuie
possible
seopec (mT) discovers (LN)
0 Shot
Shit
soprine-chhowrr mnde--hhliees, chinle-woking sying--rophinn, rageon-ggrl
discovers dhaweer
B-Shet
Dregs
S-ottt
logi-mmics, animali aatimalssssssss.. -ltop, morcle-diiss creatus -cceuuess
sosprese surprises (UN)
S-bbi supposs supras-upriee pnaprttatoo--moppeeee seprntitt-uurriingg aepptttttttiiiiieeeriiip..p sapptaaittiiieee-isssiinnnn
surprises
vurprisu!
Table 19: Translation examples on HR EN and IT-EEN We include here ground truth translation pairs and show the predictions derived from zero-shot and five-shot prompting with LLLMA


Reviews:
[{"review_id": "DU7jZAEwJI", "reviewer": "Reviewer_5V65", "summary": "", "questions": "A: The paper is lacking insights into the stability of the results (e.g. there is no mention of repeating experiments for different random seeds). The authors should add this information, especially in the context of prompting LLMs and the choice of few-shot examples. Furthermore, when reporting averages, standard deviations should be added, in addition to mean scores, e.g. when averaging multiple language pairs in Tables 1 and 3 , also potentially in Figures 1 and 2. This concern regarding the stability to different random initializations should also be addressed in the fine-tuning experiments (Tables 4 and 17).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, and hope that our clarifications below will remove the major concerns and motivate the reviewer to reassess their initial scores.\n\n>**Reasons To Reject - Concerns on the Stability of results / Randomness.** This is mentioned as the only “Reasons to Reject” and also asked in the “Questions for the Authors”. \n\n**Response: Our method (in the main experiments and without fine-tuning) expects to yield non-varied output (without any significant randomness)**, as also confirmed in our preliminary experiments, given the same/fixed prompt. In more detail: \n\n1. For our **main experiments (the whole Tables 1-2, 12-16, Figures 1-2)** the models are fixed (downloaded from Huggingface), and are not fine-tuned at all. The choice of in-context examples is nearest neighbour (NN) retrieval in the fastText embedding space, as stated in L298-L308, which is a deterministic choice. For generation, we use deterministic beam search without any randomness (rather than beam sampling). **The whole process is completely deterministic, does not imply any stochasticity**, and should be fully reproducible as such.\n\n2. We stated in footnote 7 that our few-shot “learning” also does not involve any fine-tuning. The term “few-shot learning” in our setup is inherited from the GPT-3 paper, where several in-context examples are appended to the input sentence but there is also no fine-tuning on downstream tasks. \n\n3. For our main experiments, we provide extremely significant p-values (L530-L542), which support our claims. \n\n4. The **randomness exists in two minor parts of our side analysis** as follows. \n     - The “Random 5K” and “Random 1K” rows in our ablation studies in Table 3 and also Table 17 (which is a more detailed version of Table 3). The ablation study also did not involve any fine-tuning. We did not run it multiple times because the results are significant enough to support our findings. After all, the results are averaged scores over 20 language pairs each including about 2k test samples (in total 38776 test samples). Here, we additionally estimate the significance level (p-values) related to “Random 5K” and “Random 1K” via $\\chi^2$ test in the following Table to supplement our Table 3, and we’ll add this information into the appendix. Statistically significant: p<0.05 (also see our footnote 17). The results sufficiently prove the usefulness of in-context examples as stated in L567-L576. \n|   |  mT5-xxl |  mT0-xxl | mGPT  |  XGLM-4.5 | LLAMA-13b |\n|---|---|---|---|---|---|\n|P-value (NN-5k vs Random-5k)|  1.27e-60 |  1.95e-19 |  2.35e-19 | 4.67e-11  |  1.41e-21 |\n|P-value (NN-1k vs Random-1k)| 9.34e-22  | 9.13e-6  |  4.6e-4 |  6.4e-7 |  1.1e-6 |\n|P-value (Random5k vs Zero-shot)| 1.7e-43  |  1.8e-9 |  1.4e-83 | 1.0e-311  | 8.4e-114  |\n|P-value (Random1k vs zero-shot)| 2.4e-28  | 3.9e-3  |  3.2e-85 |  5.8e-289 | 2.8e-94  |\n\n    - Our fine-tuning experiments (Table 4) do imply some possible randomness during training. However, our analysis in L580-L588 already indicates that fine-tuning **smaller** models still obviously underperforms off-the-shelf **larger** models (this is our main claim/findings for the fine-tuning experiments as stated in L592-L598). \n\n5. Because of our limited access to computational resources, we prioritized other more important experiments over multiple runs, but we will include a subset of results averaged across 3 or 5 runs in the camera-ready following your suggestion.\n\n> **Paper Topic And Main Contributions - Concerns on Novelty.**\n\n**Response:** \n\n- **We stress that we are the first to tackle BLI via LLMs** in a systematic and didactic fashion, aiming to understand the current capabilities of different models. We conduct extensive experiments and thorough analyses on BLI with LLMs and our proposed **retrieval-augmented few-shot learning** achieves strong results on many language pairs. Insightful results are presented through comparisons between LLMs, investigations on the influence of model size and the number of in-context examples, templates, BLI-oriented fine-tuning, etc. We also consider the finding that the LLMs provide sub-optimal performance for lower-resource languages as an important contribution and a reference building block for future research. \n\n- NMT and BLI are two very related but still distinct areas of research: previous research also showed how to combine BLI output to improve NMT, so gains in BLI might also translate to benefits in NMT research.\n\n> **Typos.** \n\n**Response:** Thanks for pointing this out. L1135-L1136 refers to the results on PanLex-BLI lower-resource languages. We will make it more clear in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The primary strengths of this paper are tied to its thorough examination of the topic. It encompasses 5 distinct LLM families, including both encoder-decoder and decoder-only models. It also covers 20 high-resource and 6 low-resource language pairs, from 2 standard BLI benchmark datasets, while comparing to strong baselines from previous work on this topic. Given the already acknowledged of power of LLMs, this work contributes a comprehensive evaluation for BLI tasks and contributes an overview of the capabilities and limitations to the research field.", "reasons_to_reject": "The main concern is the lack of insights into the stability of the presented results. The paper doesn't state whether the experiments were repeated multiple times, and the degree to which randomness influences the outcomes when language models are prompted.", "questions_to_authors": "A: The paper is lacking insights into the stability of the results (e.g. there is no mention of repeating experiments for different random seeds). The authors should add this information, especially in the context of prompting LLMs and the choice of few-shot examples. Furthermore, when reporting averages, standard deviations should be added, in addition to mean scores, e.g. when averaging multiple language pairs in Tables 1 and 3 , also potentially in Figures 1 and 2. This concern regarding the stability to different random initializations should also be addressed in the fine-tuning experiments (Tables 4 and 17).", "paper_topic_and_main_contributions": "This paper explores the use of large language models (LLMs) for the task of bilingual lexicon induction (BLI). It attempts to answer whether and to what extent it is possible to prompt and fine-tune LLMs for BLI and how do they compare on standard BLI benchmarks. In addition to zero-shot prompting of existing LLMs, the authors examine in-context few-shot prompting. This work demonstrates the strong performance of this approach for high-resource languages and in few shot approaches, when compared to previous approaches using cross-lingual word representations. However, when looking at the results for low-resource languages, the proposed approach with prompting out-of-the-box language models is still lagging behind previous methods. In addition, this paper shows that fine-tuning a smaller model is less successful than prompting larger models out-of-the-box, which could be a finding of interest for other tasks. Regarding the novelty of this paper, LLMs have already shown success in machine translation which makes it expected that they will perform well for the related task of BLI. Nevertheless, this paper presents an explicit evaluation on BLI benchmarks, which is a significant contribution.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "A: clarification of the text in Appendix D is needed (lines 1135-1136)"}, "extra_text": {}}, {"review_id": "EHELOML7CV", "reviewer": "Reviewer_om7d", "summary": "", "questions": "* A: Although P@1 shows similar trends as P@5 of MRR, I think it would be\ninteresting to explore top 5/10 LLM outputs. Static cross-lingual embeddings,\nsuch as aligned fastText vectors often retrieve morphological variants of the\nsame word in the top-n output. Would LLMs behave similarly or maybe output a more\nvaried set of words, e.g., synonyms?\n* B: BLI of named entities is often problematic with static word embeddings, e.g.,\nthe output contains a set of politicians of the given language/country instead of\nthe correct translation. Would LLMs be different?\n* C: I am wondering whether the authors have an intuition of why LLaMA has the\nbest performance and whether this finding could correlate with the performance\non other cross-lingual tasks?\n* D: I am also wondering about the long-term implications/use of the findings.\nThe authors mention a few applications of BLI, such as using it to generate\ntranslation options or embeddings to initialize unsupervised MT systems, however\nend-to-end MT is possible with LLMs and no embeddings were involved in this\nwork. Would BLI performance correlate with other cross-lingual task performance\nfor example?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your feedback and encouragement! Here we respond to the questions:\n\n**Question A.** P@1 is still the most standard and most used metric in BLI, hence it is adopted in our work. Concerning P@K and MRR, by manually inspecting the generated output texts, we found that many output words and phrases that rank 2nd, 3rd, 4th or 5th according to the beam score also make sense. Therefore, the trend for P@5/10, MRR should also hold (previous work, mentioned in footnote 16, states that the same trend is observed for P@1, P@5/10 or MRR).\n\n**Question B.** This is a very interesting observation. We are not familiar with the standard BLI-for-named-entities tasks and benchmarks, and this would definitely be a very interesting analysis. Following your suggestion, we will run a small-scale side analysis in the current work, but will leave the bulk of additional exploration for future research.  Concerning the issue that the output may contain “politicians of a given language instead of the correct translation”, our intuition is that maybe providing some in-context examples (few-shot learning) like what is done in our work could help, which can ‘teach’ LLMs to produce better output that is more likely to comply with some specific requirements (domain, content, format, etc.) as demonstrated in the in-context examples - this is another interesting experiment to try out.\n\n**Question C.** LLaMA is the 2023 new SotA LLM pretrained on a larger corpus compared to earlier LLMs. LLaMA also adopts a series of recent improvements such as (1) rotary embeddings, (2) SwiGLU activation, and (3) pre-normalization which are highlighted in the LLaMA paper. Please see our response to Question D below to see how BLI can improve NMT.\n\n**Question D.** Here we show how our text-to-text BLI (without calculating word embeddings) can potentially improve **end-to-end NMT**. Once BLI can improve NMT, the “Translate Train” and “Translate Test” **cross-lingual transfer learning** approaches can be improved accordingly which rely on the quality of NMT. Following your question and suggestion, we will expand the discussion on the relationship between BLI and NMT in the context of LLMs. \n\n- First, we found some initial recent works on using bilingual dictionaries to improve end-to-end NMT [1,2], where pure texts are used and no word embeddings are calculated at all. E.g., in order to translate a sentence $S_{x}$ in language X into language Y, and given some useful word translation pairs ($w_{x}^1$, $w_{y}^1$), ($w_{x}^2$, $w_{y}^2$) all in pure texts, [1] designs templates for prompting: “Translate the following sentence to Y: $S_{x}$. In this context, the word $w_{x}^1$ means $w_{y}^1$, the word $w_{x}^2$ means $w_{y}^2$. The full translation to Y is:”. [2] presents two related but different tricks.  \n\n    - [1] Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation. Ghazvininejad et al. arXiv. 2023. (please see its Figure 1 as an example)\n    - [2] BILEX Rx: Lexical Data Augmentation for Massively Multilingual Machine Translation. Jones et al. arXiv.2023.\n\n- Second, previous work [1-2] actually uses ground-truth bilingual dictionaries to aid NMT. The BLI output can supplement gold-standard dictionaries in two scenarios: (1) BLI can provide word translations for words and language pairs not covered in existing gold-standard dictionaries; (2) BLI can provide multiple (top k) possible translations for downstream NMT models to take into account.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* LLMs were not explored for BLI before.\n* The paper presents detailed experiments and analyses, giving an extensive\noverview of LLM's lexical capabilities.\n", "reasons_to_reject": "* No significant reasons to reject.\n", "questions_to_authors": "* A: Although P@1 shows similar trends as P@5 of MRR, I think it would be\ninteresting to explore top 5/10 LLM outputs. Static cross-lingual embeddings,\nsuch as aligned fastText vectors often retrieve morphological variants of the\nsame word in the top-n output. Would LLMs behave similarly or maybe output a more\nvaried set of words, e.g., synonyms?\n* B: BLI of named entities is often problematic with static word embeddings, e.g.,\nthe output contains a set of politicians of the given language/country instead of\nthe correct translation. Would LLMs be different?\n* C: I am wondering whether the authors have an intuition of why LLaMA has the\nbest performance and whether this finding could correlate with the performance\non other cross-lingual tasks?\n* D: I am also wondering about the long-term implications/use of the findings.\nThe authors mention a few applications of BLI, such as using it to generate\ntranslation options or embeddings to initialize unsupervised MT systems, however\nend-to-end MT is possible with LLMs and no embeddings were involved in this\nwork. Would BLI performance correlate with other cross-lingual task performance\nfor example?\n", "paper_topic_and_main_contributions": "This work investigates large text-to-text language models for bilingual lexicon\ninduction, a task that was not explored with LLMs before. The authors compare\nvarious models, including different model types (encoder-decoder and\ndecoder-only) and sizes, using various prompt templates, in zero- and few-shot\n(in-context learning and model fine-tuning) setups. Although the main goal of\nthe paper is the analysis of LLMs on BLI, it contributes novel techniques\nas well, such as in-context learning with nearest neighbors examples or prompt\ntemplates for the BLI task. The experiments are performed on various language\npairs, including high-resource and moderate-resource pairs. On top of presenting\nLLM's BLI performance in general, the authors perform various analyses and\nablation studies. The paper is detailed and easy to follow.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uLTDZCsckr", "reviewer": "Reviewer_cKpL", "summary": "", "questions": "Could BLI be affected by the fact that LLMs tend to be biased toward dominant languages like English?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback and helpful suggestions! Here we respond to the three minor weaknesses and the question for the authors respectively.\n\n**Minor Weakness 1.** Thanks for pointing this out! We admit that the existing BLI datasets actually do not cover a really large amount of diverse languages and we will be more explicit about it (we also already mentioned in L644-L658, that the range of languages for LLMs is one limitation of our method and LLMs for BLI in general). We agree that ‘language genus’ would be more proper and will adopt more accurate wording following your suggestion.\n\n**Minor Weakness 2.** We agree that previous approaches for BLI also show lower results in low-resource scenarios, and we understand that similar findings for LLMs may not make readers feel more excited. However, please consider the BLI performance in the following four setups: (A) traditional methods on high-resource language pairs (B) traditional methods on low-resource language pairs (C) LLMs on high-resource language pairs and (D) LLMs on low-resource language pairs. Previous work tells us that A>B, and analogously we can guess (before reading our paper) C>D. But some **more findings validated by our experiments** are that e.g., for LLaMA-13b we show C>A but D<B. How to further boost language coverage and capabilities of LLMs for lower-resource languages is an open question for the entire field of multilingual NLP, and some improvements on that front should also be reflected in BLI results as well in future research.\n\n**Minor Weakness 3.** We fully agree and acknowledge that LLMs are biased towards English. Our work shows that even such biased LLMs (e.g., LLaMA) can already outperform traditional BLI methods on many language pairs including non-English pairs. If stronger, more multilingual and less biased LLMs get developed in the future, our method may achieve even better results. Thank you for **implicitly** indicating that our method will potentially become more influential in the future, we will expand the discussion of (currently limited) multilinguality of LLMs (with the focus on the BLI task) following your feedback.\n\n**Question For The Authors.** Our preliminary conclusion is that LLMs also achieve strong BLI performance for non-English pairs. In our appendices, Tables 12,13,15 show detailed results on each language pair for the XLING dataset. For example, LLaMA supports 6 out of 8 languages. The 6 languages result in 30 language pairs and 20 of them are between non-English languages. Here are some statistics comparing LLaMA-13b with CLWE-based SotA: \n\n| |10 Language pairs involving English|20 non-English Language pairs |All 30 pairs|\n|---|---|---|---|\n|5K Setup|LLaMA wins in 8/10 cases|LLaMA wins in 16/20 cases|LLaMA wins in 24/30 cases|\n|1K Setup|LLaMA wins in 5/10 cases|LLaMA wins in 18/20 cases|LLaMA wins in 23/30 cases|\n|Zero-shot Setup|LLaMA wins in 5/10 cases|LLaMA wins in 7/20 cases|LLaMA wins in 12/30 cases|\n\nFrom this table we did **not** find very clear clues showing that LLaMA-13b may favour language pairs that involve English. We only know that LLMs (with <=13b parameters) may struggle with low-resource languages (corresponding to Minor Weakness 2 above).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I find the article well structured and easy to read. The experimental setting seems sound.\n\n- The authors claim that this work is the first one to explore BLI with multilingual LLMs.\n", "reasons_to_reject": "Some minor weaknesses \n\n- The authors say they include a typologically diverse sample. However, the criterion for choosing the languages seems more opportunistic (whichever languages are available on the current datasets). This is not necessarily bad; I just recommend being more explicit about this. If we look at Appendix A, we can see that most languages are actually Indo-European. In fact, in this table (Appendix A), the authors write ”linguistic family” but they are referring to linguistic genus, so the table needs to be corrected.\n\n- The results show bad performance in low-resource scenarios. In a way, this is something we already knew with previous approaches for BLI: bad quality embeddings lead to deficient induction of translation pairs. \n\n- Large language models, e.g., LLaMA might be biased towards English. \n", "questions_to_authors": "Could BLI be affected by the fact that LLMs tend to be biased toward dominant languages like English?", "paper_topic_and_main_contributions": "The paper focuses on Bilingual lexicon induction (BLI). The novelty relies on using multilingual LLMs and prompting to induce the translation pairs. The article includes a comprehensive set of main and side experiments. They obtain competitive performances, particularly using models like LLaMA\n\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines the use of LLMs for the task of Bilingual Lexicon Induction (BLI), and in particular examine and evaluate different prompting strategies. They find that LLMs have strong capability to generate bilingual lexica. 

Soundness: All reviewers agree that the soundness of the paper is very high (giving scores of 4/4/5), and that comprehensive experiments are performed to support the conclusions in the paper.  

Excitement: Reviewers generally score high excitement for the paper (4/4/4) and list the readability and thoroughness as main reasons. Some challenges to excitement are raised in that (1) it is not surprising that LLMs can generate BLIs well, and (2) it is somewhat unclear why one would need an LLM-generated BLI instead of simply using the LLM (which generated the BLI) to perform a downstream task itself. The authors reply with an example in machine translation and point to 2 arXiv papers that investigate this issue.
Decision: Accept-Main