mAggretriever: A Simple yet Effective Approach to Zero-Shot Multilingual Dense Retrieval
Authors: Sheng-Chieh Lin
Amin Ahmad
Jimmy Lin
Venue: EMNLP 2023 Main (2023)
Keywords: Multilingual Dense Retrieval
Zero-Shot Language Transferability
Lexical and Semantic Matching
TL;DR: We introduce mAggretriever, which effectively leverages semantic and lexical features from pre-trained multilingual transformer, and demonstrate its strong zero-shot multilingual retrieval capability.

Abstract: Multilingual information retrieval (MLIR) is a crucial yet challenging task due to the need for human annotations in multiple languages, making training data creation labor-intensive. 
In this paper, we introduce mAggretriever, which effectively leverages semantic and lexical features from pre-trained multilingual transformers (e.g., mBERT and XLM-R) for dense retrieval. 
To enhance training and inference efficiency, we employ approximate masked-language modeling prediction for computing lexical features, reducing 70--85\% GPU memory requirement for mAggretriever fine-tuning. 
Empirical results demonstrate that mAggretriever, fine-tuned solely on English training data, surpasses existing state-of-the-art multilingual dense retrieval models that undergo further training on large-scale MLIR training data.
Our code is available at url.

mAggretriever: A Simple Yet Effective Approach to Zero-Shot Multilingual Dense Retrieval
Gheng-Chiess Lin',, Amin Ahmad Jimmy Lin'
University of Waterloo
2 Vectara
(s2691in, jimmylin| @uwaterloo.c ca, amin@vectara.co
Abstract
Linliihh datasets alone (Zhang et al., 2023b). How- ever, Izacard (2022) emphasize the impor- tance of contrastive pre-training on multilingual corpora to achieve improved language transferabil- ity in IR. Other approaches utilize multilingual par- allel corpora or translation pairs for contrastive pre- training (Feng et al.. 2022) or fine-tuning (Reimers and Gurevych, 2020; Bonifacio et al., 2021). How- ever, these solutions are viable only when signifi- cant computational resources or multilingual paral- lel data are available.
Multilingual information retrieval (MLIR) is crucial yet challenging task due to the need for human annotations in multiple languages, mak- ing training data creation labor-intes inteesiitensvve. In this paper, we introduce m mAggretriever, which effectively leverages semantic and lexical fea- tures from pre-trained multilingual transform- ers (e.g., mBERT and XLM-R) for dense re- trieval. To enhance training and inference efficiency, we employ approximate masked- language modeling prediction for computing lexical features, reducing 70-85% GPU mem- ory requirement for mAggretriever fine-tuning. Empirical results demonstrate that mAggre- triever. fine-tuned solely on English training data, surpasses existing state-of-the-art multi- lingual dense retrieval models that undergo fur- ther training on large-scale MIIR training data. Our code is available at https: :/ //github. com/castors cas.orini.inooo/ooooooo /innnd/hrrr. /dhr.
Recently, Lin et al. (2023b) demonstrate that the existing DR models solely using [CLS] (or aver- aged pooling) do not fully exploit the capability from pre-trained transformers for retrieval. Lin et al. (2023b) propose Aggretriever by combining the semantic and lexical features from the respec- tive two components, [CLS] and masked language modeling (MLM) prediction, which shows supe- rior effectiveness on diverse English retrieval tasks. This inspires us to ask the question: Can Aggre- triever benefit multilingual retrieval?"
1 Introduction
Fine-tuning a pre-trained transformer has proven to be highly effective in many tasks of natural lan- guage processing, including information retrieval (IR). Despite its success, the recent tfate-oftthe- art dense retrieval (DR) models (Ni al., 2022: Lin al.. 2023a) predominantly focus English. This bias arises from the fact that constructing an effective DR model requires a substantial amount annotated training data, which is predominantly available in English datasets (Bajaj et al., 2016; Kwiatkowski et al., 2019). This makes it challeng- ing for users of low-resource languages to benefit from the recent IR progress.
Extending Aggretriever support multilingual retrieval poses challenges due to increased com- putation costs for extracting lexical features from the MLM component as the vocabulary size of the pre-trained model grows, making fine-tuning with limited resources challenging. In this work, we pro- pose two simple approaches to approximate MLM prediction to extract lexical features from multi- lingual pre-trained transformers, making the train- ing of mAggretriever possible in one GPU within 40 GBs of memory. Remarkably, mAggretriever exhibits strong retrieval capability across multiple languages despite being fine-tuned on English data.
To address this issue, researchers have explored leveraging pre-trained multilingual transformers, such as mBERT (Devlin et al., 2019) and XL.M- R (Conneau et al., 2020), which exhibit promising language transfcrability even when fine-tuned on
The paper is structured as follows: we begin by providing background information on standard DR and Aggretriever. We then present proposed approaches tackle the computational challenges
In this paper, we refer multilingual retrieval to monolingual retricval across multiple languages.
Work done during Sheng-Chich's internship at Vectara.

---

involved in extending Aggretriever to support mul- tilingual retrieval. Subsequently, we compare mAg- gretriever with other state-of-the-art multilingual DR models and explore the potential for extending mAggretriever to enable cross-lingual retrieval.
prediction
P
(4)
(3) English
(2) source language
63
(1) whole-vocabollery
Sofimax
2
Background
Embedding table
W mll
Dense Retrieval. Given a query with sequential tokens q = (ICLS), 41 - qn), our task is to re- trieve a list of passages to maximize some ranking metric such nDCG or MRR. Standard dense re- trieval (DR) models (Reimers and Gurevych, 2019;
Enghish rekarn
Spantis token
Contextunlized embedding
%
% temis
%
juega
com
frecuencia
Karpukhin et al., 2020) based on pre-trained lan- guage models encode queries and passages as low dimensional [CLS] J vectors with a bi-encoder ar- chitecture and use the dot product between the en- coded vectors as the similarity score:
Figure I: Illustration of MLM prediction for the input token 91- (1) Whole-vocabulary prediction considers all languages while target-language prediction only consid- ers the tokens in one language; e.g., (2) source language (Spanish) or (3) English. (4) Self prediction assigns probability of one to itself. 0. denotes less than 0.1.
simcis(q,P) p) Eqicist
eprciar>
(1)
where @qiossi and respective query and passage
next-sentence prediction during pre-training; thus, does not carry much lexical information.
epicie) are the [CLS] vectors the last layer of pre-trained language model (e.g., BERT).
The wordpiece lexical representation Vg is then compressed, without supervision, into low- dimensional vector agg; and concatenated with the [CLS] vector. The similarity score between a given q-p pair computed with their dot product:
Aggretriever. In addition to using [CLS] vec- tors to capture semantic textual features, Lin et al. (2023b) further propose capture lexical textual features from the pre-trained MLM prediction head by projecting each contextualized token embedding 6g. into a high- digh-diiensional vector in the wordpiece lexical space:
sim (om((g1) Q.P. p) A (Cqiers) Vg) (Epiers vp)
(4)
(Cqiens agg;)) (Cpias) agg"):
where Equess and v, capture semantic and lexical textual features from BERT, respectively. Follow- ing Lin et al. (2023b), we linearly project errecss into 128 dimension and compress Vg into 640 di- mension; i.e., Qqicas E R128 and agg" C R400 We refer readers to Lin et al. (2023b) for the details of compressing V, into agg".
Pr, softmax
W mim Dmim);
(2)
where eq Rdd, Wilm E ddx] and bmim E R/lll are the weights of the pre-trained MLM lin- ear projector, and Pg. E Vapl is the i-th contex- tualized token represented by probability distri- bution over the BERT wordpiece vocabulary, V Wwp Weighted max pooling is then performed over the sequential representations (Pm P....... ... .p.) to
3 Our Approach
In this work, we extend Aggretriever multilin- gual retrieval, maggretriever. However, directly applying Eq. (2) to multilingual pre-trained MLM heads over the whole vocabulary increases com- putation cost for both training and inference due to large vocabulary size |V in MLM projector, W (Nair et al., 2022; Lassance, 2023). For ex- ample, the mBERT and XLM-R have respective cabulary size of 120K and 250K (vs BERT's 35K) in the MLM projector. To address the issue, we propose two computationally efficient approaches to approximate MLM prediction in Eq. (2).
obtain a single-vector lexical representation:
vglu] (E(1.2.....)
Wi [v.
where W; = |eg W b/ E R is positive scalar and 21 ... /V J; W E dxxl and b I are trainable weights. Note that the scalar w; for each token 4i is essential capture term importance, which Py: alone cannot capture since it is normalized by softmax. Note that the [CLS] token embedding is excluded since it is used for

---

Table 1: Training and inference cost comparisons on MS MARCO with XLM-R base backbone.
vector components
training cost
query encoding latency
Models
[CLS]
agg"
MLM prediction
GPU memory
total time
CPU
GPU
(1) XLM-
768 dim.
dim.
21 GBs
5.5 hrs
105 ms/q
9.1 ms/q
(2) XLM-B R RAGG
whole vocabulary
OOM (140 GBs)
400 ms/q
15.1 ms/q
(3) XLM-R AGG
128 dim.
640 dim.
target language self
39 GBs
9.5 hrs
248 ms/4
11.4 ms/q
(4) XLM-R AGG
21 GBs
5.5 hrs
123 ms/y
9.1 ms/q
Target-languaat prediction. One intuitive ap- proach is to compute the probability distribution over the target-language token set of interest, de- noted as V by replacing Wmlm and mmim in Eq. (2) with W and btg respectively.
4 Experimental Setups
4.1 Datasets and Metries
We use the MS MARCO passage ranking dataset introduced by Bajaj et al. (2016), comprising a cor- pus with 8.8M passages and 500K training queries. Model supervised language (English) retrieval ef- fectiveness is evaluated on the 6980 (MARCO dev) queries from the development set with one rele- vant passage per query on average. Following the established procedure, we report MRR @ 10 and R @ 1000 as the metrics.
mim
Wig
mim [] = W mlm [: t];
big [1] bmim(1)) (r),
if U E Vop
W/g
mlm [. v] = 0' b mlm [c] = 0, otherwise.
From the above equation, we only have to compute the matrix multiplication and softmax among V instead of wpl tokens. This approach assumes that only the tokens corresponding to the target language have responses when conducting MLM prediction; i.e., P., [t] = 0 if & vlg wwp
We evaluate model zero-shot retrieval effective- ness in other languages using MIRACL dataset introduced by Zhang et al., comprising around 77k queries over Wikipedia in 18 languages with over 700k high-quality relevance judgments by native speakers. We use the publicly available develop- ment queries and their relevance judgements in 15 languages (two surprise languages and English are excluded). Following Zhang et al., port macro averaged nDCG10 and R 100 over the 15 languages and list the full numbers in Ap- pendix A.1.
Self prediction. Since BERT MLM pre-training task includes recovering the unmasked token itself, to further save computation cost, may approxi- mate MLM prediction by assigning a value of one to the token qi itself, and zero the others:
Pq: = xン C 10. for j E (tok_id(a))). (5)
The operation removes the computationally expen- sive matrix multiplication and softmax operation in Eq. (2). Note that, combining Eq. (3) and (5), the lexical representations, Vg. from self predic- tion can be considered bag-of-word vectors with learned term weights.
Finally, we study how to conduct cross- lingual retrieval using mAggretriever on XQuAD- R dataset introduced by Roy et al. (2020), consist- ing of parallel queries and corpora with 11 lan- guages.2 2 We conduct retrieval using the queries with XQ language against the corpus with X lan- guage and report the macro-averaged MAP@ 100 over all the cross- lous-llnguul combinations of the 110 language pairs (Xq * Xc), and the other 11 mono- lingual combinations (Xq = Xc).
Figure 1 illustrates whole-vocabulary, our pro- posed turget-language and self MLM predictions. Table compares the training and inference cost of the standard DR (row 1) and mAggretriever with different MLM prediction strategies on MS MARCO dataset using the backbone of XLM-R base The training and query encoding settings are detailed in Section 4.2. We observe training with target-language (English in our case) MLM prediction reduces the GPU memory requirement compared to whole vocabulary (row 3 vs 2) while self prediction yields training and inference effi- ciency on par with standard DR (row 4 vs 1).
Table 2 reports the data statisties of MIRACL and XQuAD-R. Note that the candidates of MIR- ACL are passages while the candidates of XQuAD- R are chunked sentences from XQuAD corpora. Note that although XQuAD corpora is the manu- ally rewritten multilingual parallel corpora (Artetxe et al., 2020), the numbers of chunked sentences are
2 Datasets: MIRACL, XQuAD-R

---

Table 2: MIRACL and XQuAD-R data statistics.
7e-6. Each batch includes 24 queries, and for each query, we randomly sample one positive and seven negative passages. All the negatives are sampled from the MS MARCO "small" triples training set, which is created using BM25. During training, we minimize the negative log likelihood as the stan- dard contrastive loss and following Karpukhin et al. (2020), for each query, consider all the (positive and negative) passages from the other triplets in the batch as in-batch negative samples. We set the max- imum input length for the query and the passage to 32 and 128, respectively, at both training and in- ference stages for MS MARCO. For MIRACL and XQUAD-R, we use the maximum input length of 128 and 256 for the query and passage, respectively.
MIRACL Dev
XQuAD-R
language
ISO
queries
# candidates
queries
candidates
Arabic ar
2,869
2,061,414
1,190
1,222
Bengali
bn
411
297.265
German de
1.190
1,276
Greek
1,190
1,234
English cn
648
32,893.221
1.190
1,180
Spanish es
799
10.373,953
1,190
1,215
Persian
632
2,207,172
Finnish
1,271
1,883,509
French
343
14,636,953
Hindi
350
506.264
1.190
1,244
Indonesian
960
1,446,315
Japanese
860
6,953,614
Korean
ko
213
1,466,752
Russian
1,252
9,543,918
1,190
1,219
Swahili
482
131.924
Telugu
828
518.079
Thai
733
162,166
1,190
852
Note that we lowercase all the queries and passages for mAggretriever. We measure query encoding latency on the 6980 MS MARCO development queries with the batch size of 1 and single thread on a Linux machine with 12 Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz and 88G of RAM.
Turkish
1.190
1,167
Victnamese vi
1,190
1,209
Chinese
2h
393
4,934,368
1,190
1,196
different between languages. Also note that for each query, MIRACL has multiple relevant candi- dates while XQuAD only has one.
Target Token Set Construction. For each lan- guage of corpus in MIRACL and XQuAD-R, we tokenize and lowercase all the passages and col- lect the unique tokens in the corpus as the target token set. For example, when fine-tuning on MS MARCO dataset, we use the token set built from MS MARCO corpus. While conducting target- language MLM prediction on MIRACL Arabic queries and corpus, we use the token set collected from Arabic corpus our target token set. Note that self MLM prediction does not require collect- ing the token set for the target language.
4.2 Implementation Details
Models. We apply mAggretriever to two 12-layer pre-trained multilingual models: (1) mBERT; (2) XLM-R.3 3 We compare models fine-tuned solely using [CLS] vector and based on mAggretriever using whole-vocabulary MLM prediction with the subscripts "CLS" and "AGG", respectively, e.g.. mBERTcLs and mBERTAGG: AGG. We report the two variants of mAggretriever with larget-lagguage and self prediction: e.g., MBERT AGG For target-language prediction, we train mAggre- triever using English token prediction and run infer- ence using the corresponding language of each cor- pus, and for whole-vocabulary prediction, we fine- tune with half of the batch size. In addition, we re- port the numbers of BM25 and mDPR from Zhang et al. as reference points, and the two state-of-the- art multilingual retrievers: (1) mContriever (Izac- ard al., 2022), pre-trained on multilingual cor- pora with 29 languages and further fine-tuned on MS MARCO d (2) Cohere (API), whose numbers are copied from Kamalloo et al. (2023).
AGG and mBERTsclf AGG
Results
5.1 Results on MIRACL
Table reports models' retrieval effectiveness on MS MARCO and MIRACL development queries. We first observe that mAggretriever, incorporat- ing lexical features, not only outperforms its CLS counterpart in supervised English retrieval (MARCO), but also exhibits superior transferabil- ity to other languages (MIRACL) regardless of backbone. In addition, mBERT AGG. without in- troducing contrastive pre- trainingn on multilingual corpora, outperforms mContriever in MIRACL in terms of nDCG@ 10. Note that mBERT AGG and mContriever are both initialized from mBERT. We
Training and Inference. We train our models on a single A100 GPU with 80 GB memory for 6 epochs (around 100k steps) with learning rate
'Our preliminary experiments on MS MARCO show that lowercase improves mAggretrievar while degrades its CLSS counterpart.
Model checkpoints: mBERT, XLM-R, mContriever 'Cohcre multilingual retrieval model

---

using respective query and corpus source language the language Table 3: Supervised (MARCO) English and zero-shot (MIRACL) multilingual retrieval effectiveness compar- isons. Full numbers are listed in Appendix A.I.
Table 4: Zero-shot retrieval effectiveness on XQuAD-R.
XQ (Xc) denotes the language of queries (corpus).
MLM target lang.
XQUAD-R
MARCO Dev
MIRACL Dev
Models
query
corpus
Xg
X
Xg of X
Models
Fnnglish
15 lang. macrn avg.
MAP@ 100
MRR@ 1O R@ 1K
nDCG@ 10 R@ 100
XLM-RcLs
73.1
57.5
(a) BM25
85.8
39.6
78.5
XLM-RaGo
77.4
(b) mDPR
29.6
94.6
41.7
78.9
Xq
X
77.4
36.0
(c) mContriever
27.4
97.1
43.8
85.9
(4) XLM-R AGG
X
Xc
77.4
44.9
English
English
73.5
51.7
(d) Cohere (API)
50.1
XLM-RAG
77.3
36.2
(I) mBFRTcLs
29.1
93.6
36.1
71.6
(2) mBERTAGO AGO
34.3
95.8
44.4
79.5
(3) mBERT 10
AGG
34.5
96.1
44.4
80.1
(4) mBERT AGG
34,2
95.6
46.9
82.3
(5) XLM LLM-RRLi
31.1
93.8
39.3
73.9
as the target language (row 3),
(6) XLM-R
34.7
96.1
52.9
86.4
corresponding to each corpus (row 4) or English (row 5) as target language for both queries and corpus. Note that rows and are the same when both queries and corpus are in the same language (XQ = Xc).
XLM-R AGG MaGi
35.0
96.2
53.3
86.0
(8) XL.M Reell
35.0
96.0
53.3
86.3
hypothesize that mContriever's high recall (i.c., R@ IK) comes from its pre-training on multilin- gual corpora with 29 languages.
We observe that mggretrriever shows relatively poor cross-lingual retrieval effectiveness (Xq * Xc) compared to its CLS counterpart (rows 2,3,6 vs 1). When aligning the MLM prediction target language for queries and corpus, the cross-lingual retrieval effectiveness sees improvement (rows 3 vs 4,5). These results show that MLLM prediction head potentially can be used as a translation layer to project query and corpus into the same language, which is also reported by Nair et al. (2022). It is possible to leverage the transformers pre-trained with translation language modeling (Chi et al., 2021; Feng et al., 2022) and the established parallel training data (Bonifacio al., 2021) improve mAggretriever's cross -ooss-ln retrieval capability, which we leave for future work.
Switching to XLM-R backbone, mAggretriever even outperforms Cohere (API). Furthermore, notice that mAggretriever with XLM-R backbone improves over mBERT more than its CLS coun- terpart does. For example, in the case of MIR- ACL, XLM-R AGG exhibits a significant improve- ment over MBERT AGG from an nDCG@ 10 score of 44.4 to 53.3, whereas XLM-ReLS only sees modest improvement over mBERTcLs from 36.1 to 39.3. This notable enhancement highlights mAg- gretriever's ability to effectively utilize superior pre-trained language model.
Finally, compared to whole-vocabulary and pro- posed approximate MLM predictions, we observe that self prediction shows comparable and even strong language transferability. We hypothesize that MLM prediction learned from English data cannot transfer well to other languages. It is worth mentioning that compared whole-vocabulary MLM prediction, the proposed approximate MLM prediction, target-language and self prediction, are advantagcous for real-world deployment since they show almost no effectiveness drop (sometimes even better) but require far less training and inference cost as shown in Table 1.
Conclusion
In this paper, we introduce mAggretriever, an ex- tension of Aggretriever for multilingual retrieval, by combining lexical and semantic features in pre- trained language models for dense retrieval. We propose target-language and self MLM predictions to enhance the efficiency of mAggretriever. Our study highlights the efficiency advantage of self MLM prediction in multilingual retrieval, while target-language MLM prediction offers fiexibility for cross-lingual retrieval. Importantly, mAggre- triever, solely fine-tuned on English data, demon- strates competitive multilingual retrieval capability compared to other state-of of-thh-art dense retrievers.
5.2 Results on XQuAD-R
In Table directly apply XLM-R based models fine-tuned on MS MARCO to XQuAD-R dataset. the experiment, we try different MLM prediction settings for XLM-R AGG For example, instead of

---

Limitations
Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proc. ACL, pages 8440-8451.
Our research primarily focuses on enhancing mul- tilingual retrieval, specifically targeting monolin- gual retrieval zero-shot transfer to non-English languages. We plan to extend our study im- prove cross-lingual retrieval by leveraging trans- formers pre-trained with translation language mod- cling (Chi et al., 2021; Feng et al., 2022). In ad- dition, we only discuss how to improve zero-shot language transferability of dense retrieval. It is possible to further improve model effectiveness by leveraging existing multilingual training data (Boni- facio et al., 2021; Zhang et al., 2021) and better negative mining strategies (Shen et al., 2022). Fi- nally, due to space limitation, we compare mAggre- triever with previous state-of-the-art multilingual retrievers on Mr. TyDi (Zhang et al., 2021) in Ap- pendix A.2.
Jacob Devlin, Ming-Was King-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proc. NAACL, pages 4171-4186.
Fangxiaoyu Feng, Yinfei Yang. Daniel Cer, Naveen Ari- vazhagan, and Wei Wang. 2022. Language-agnosti BERT sentence embedding. In Proc. ACL, pages
Gautier Izacard, Mathilde Caron, Lucas Hosseini. Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research.
Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo. Nandan Thakur, David Alfonso-Hermelo, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. Evalu- ating embedding APIs for information retrieval. 2305:2300006300
Acknowledgements
This research was supported in part by the Canada First Research Excellence Fund and the Natural Sci- ences and Engineering Research Council (NSERC) of Canada. We thank the anonymous referees who provided useful feedback to improve this work.
Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question amswering. In Proc. EMNLP, pages 6769-6784
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova. Llion Jones, Ming-Wer Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral Questions: a benchmark for question answering research. Transactions of the Association of Compu- tational Linguistics. 7:452-466.
References
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proc. ACI. pages 4623- 4637.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, An- drew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO: A human generated machine reading comprehension dataset.
Carlos Lassance. 2023. Extending English IR methods to multi-lingus IR. arXiv:2302.14723:
Sheng-Chie Chich Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen tau Yih, and Xilun Chen. 2023a. How to train your DRAGON: Diverse augmentation towards generalizable dense retrieval. arXiv:2302.07452.
Luiz
Henrique Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti. Marzich Fadace, Roberto Lotufo, and Rodrigo Nogueira. 2021.
Sheng-Chieh Lin, Minghan Li, and Jimmy Lin. 2023b. Aggretriever: A Simple Approach to Aggregate Tex- tual Representations for Robust Dense Passage Re- trieval. Transactions of the Association for Computa- tional Linguistics, pages 436-452.
mMARCO: multilingual version of MS MARCO passage ranking dataset. arXiv:2108. 13877
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang. Xia Song. Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An cetemattoo-thhoorefe framework for cross-lingual language model pre-training. In Proc. NAACL, pages 3576-3588.
Suraj Nair, Eugene Yang, Dawn Lawrie, James May- field, and Douglas W. Oard. 2022. Learning a sparse representation model for neural CLIR. In Proc. DE- SIRES.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wet Chang, and Yinfei Yang. 2022.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco

---

Large dual encoders are generalizable retrievers. In Proc. EMNLP, pages 9844-9855
Nils Reimers and Iryna Gurevych. 2019. Sentence-
BERT: Sentence embeddings using Siamese BERT- networks. Proc. EMNLP, pages 3982-3992.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. In Proc. EMNLP, pages 4512 4525.
Uma Roy, Noah Constant, Rami AI-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020. LAReQA: Languago-agnostic answer retrieval from multilin- gual pool. In Proc. EMNLP, pages 5919-5930.
Tianhao Shen, Mingtong Liu, Ming Zhou, and Deyi Xiong. 2022. Recovering gold from black sand: Mul- tilingual dense passage retrieval with hard and false negative samples. In Proc. EMNLP, pages 10659- 10670.
Ning Wu, Yaobo Liang, Houxing Ren, Linjun Shou, Nan Duan, Ming Gong, and Daxin Jiang. 2022. Unsupervised context aware sentence representa- tion pretraining for multi-lingual dense retrieval. arXiv:2206.03281.
Shunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang, and Nan Duan. 2023a. Modeling sequential sentence relation to improve cross poss-llnguua dense retrieval. In Proc. ICLR.
Xinyu Zhang, Xueguang Ma. Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: multi-lingual benchmark for dense retrieval. In Proc. Workshop Multilingual Representation Learning, pages 127-137.
Xinyu Zhang, Kelechi Ogueji. Xueguang Ma. and Jimmy Lin. 2023b. Toward best practices for training multilingual dense retrieval models. ACM Trans. Inf. Syst.
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermell, Xi- aoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. Transac- tions of the Association for Computational Linguis- ties, pages 1114 1131.

---

A Appendix
A.1 Full Results on MIRACL
Table 5 reports the detailed nDCG@ 10 and @@ 100 numbers on MIRACL 15 languages for all the com- pared models. Note that we do not English dataset when evaluating MIRACL.
A.2 Comparisons Mr. TyDi
Some previous state-of-the-art multilingual re- trievers are evaluated on the test queries of Mr. TyDi (Zhang ct al., 2021), the multilingual retrieval dataset similar to MIRACL but with sparse rele- vance judgements and less languages. In order to compare with the previous models, following the model inference settings in MIRACL, evaluate our mAggretriever with the backbone of XLM-R. The full results are tabulated in Table 6. We still observe that all the variants of mAggretriever out- perform previous state of the art in out 10 languages. Note that most of the previous retriev- ers undergo multilingual contrastive learning. For example, mContriever are pre- pre-trained on the corpus with 29 languages while mColBERT are fine-tuned multilingual MS MARCO dataset introduced by Bonifacio et al. (2021).

---

Table 5: MIRACL multilingual retrieval comparisons. Bold denotes the best effectiveness.
MIRACL Dev
Models
ar
hn
BS
fr
ja
ko
111
SW
te
th
avg.
nDCG@ 10
BM25
48.1
50.8
31.9
33.3
55.1
18.3
45.3
44.9
36,9
41,9
33,4
38.3
49.4
48.4
18.0
39.6
mDPR
44.3
47.8
43.5
38.3
43.9
41.9
40.7
35.6
35.8
51.2
mContriever
52.5
50.0
41.8
21.5
60.2
31.4
28.6
42.4
48.3
39.1
56.0
52.8
51.7
41.0
(d) Cohere (API)
61.7
59.4
23.3
47.1
63.4
46.2
49.3
44.6
46.0
49.6
46.9
61.1
61.3
54.6
36.5
50.1
(1) mBFRTcls
50.9
45.2
31.7
30.5
48.5
30.5
37.7
19.8
43.4
40.0
27.0
21.2
36.4
37.7
43.9
36.1
mBERT AGe
56.9
48.5
44.0
43.1
63.4
40.8
34.1
40.3
49.3
43.9
44.9
44.6
44.7
27.2
40.2
44.4
mBERT AGG
45.5
45.4
42.9
34.3
47.7
43.4
45.2
43.9
32.5
41.5
(4) mBERT
59.4
51.0
44.6
44.5
65.3
43.6
37.4
42.1
50.2
47.8
46.3
48.5
48.0
31.1
44.3
46.9
(5) XcM--ReLs
46.6
46.6
29.9
43.5
44.4
28.5
41.7
31.7
40.7
45.6
27.5
22.5
50.5
53.9
35.9
39.3
XLM-
60.6
60.4
42.5
46.2
66.2
43.0
44.8
47.8
53.3
58.2
44.6
46.6
72.1
66.0
41.4
52.9
XLM-R Ago
60.2
60.9
44.3
46.7
43.8
48.0
47.2
53.3
59.1
45.8
45.2
70.2
67.1
42.9
(8) XLMM-R
61.4
61.4
42.9
46.5
66.2
41.3
46.2
48.4
53.9
57.9
46.5
47.5
71.2
66.8
41.0
53.3
@
(a) BM25
88.9
90.9
70.2
73.1
89.1
65.3
86.8
90.4
80.5
78.3
66.1
70.1
83.1
88.7
56.0
78.5
(b) mDPR
84.1
81.9
86.4
89.8
78.8
91.5
77.6
57.3
82,5
73.7
79.7
61.6
76.2
67.8
94.4
78.9
mContriever
92.5
92.1
84.1
65.4
95.3
82.4
64.6
80.2
87.8
87.5
85.0
91.1
96.1
93.6
90.3
85.9
(d) Cohere (APT)
mBERT
81.9
67.1
84.1
72.9
76.3
47.7
80.7
71.9
61.1
69.7
(2) mBERT AGG
86.9
83.8
81.0
81.0
90.2
83.2
70.9
76.6
84.1
71.7
80.8
79.4
82.4
60.0
81.3
79.5
(3) MBERT AGG
86.2
82.2
82.1
79.9
89.7
82.9
72.6
75.7
84.3
76.3
80.5
80.9
81.7
63.8
82.6
80.1
(4) mBERT AGG
88.7
85.1
80.1
82.4
92.1
86.5
77.6
80.2
85.5
78.4
82.0
81.5
83.4
65.9
84.5
82.3
XLM-RcLS
79.2
82.6
63.6
79.9
75.2
66.8
76.9
77.2
77.6
62.5
48.8
87.0
89.2
(6) XLM-RAGEE MAGG
89.8
92.5
78.8
85,4
92.6
81.8
85.1
83.5
89.0
88.5
78.9
79.1
95.4
95.1
80.6
86.4
(7) XLM-R R
89.3
93.1
80.4
84.6
91.8
81.4
84.5
82.6
88.9
86.6
79.5
77.7
94.8
95.1
79.9
86.0
(8) XLM-Ri0L ACG
90.2
92.8
78.2
84.5
93.1
81.2
82.5
84.6
89.8
88.3
80.8
79.4
95.5
94.5
79.0
86.3
Table 6: Mr. TyDi multilingual retrieval comparisons with late-of-the-art multilingual retrievers. Bold denotes the best effectiveness. multi CL denotes multilingual contrastive learning.
Mr. TyDi Test
Models
multi CL
bn
fi
id
ko
th
avg.
MRR 100
(a) BM25 (Zhang et al., 2021)
X
36.7
41.3
28.8
38.2
21.7
28.
32.9
39.6
42.4
41.7
35.1
(b) MColBERT (Bonifacio et al., 2021)
/
55.3
48.8
41.3
55.5
36.6
36.7
48.2
44.8
61.6
mContriever (Izacard al., 2022)
/
43.4
42.3
35.1
42.6
32.4
34.2
36.1
51.2
37.4
40.2
39.5
CCP (Wu 2022)
42.6
37.2
46.2
37.7
34.6
36.0
39.2
47.0
48.9
41.5
(c) MSM (Zhang al., 2023a)
51.6
39.4
50.5
32.0
36.8
37.2
43.4
62.6
53.5
44.7
(1) 1LM-RRLL
X
41.9
40.8
27.8
39.9
32.5
33.0
27.7
23.7
$4.2
46.1
36.8
(2) XLM-R, MAGA
X
52.3
55.8
43.2
55.0
40.4
40.5
41.5
45.1
77.5
57.3
50.8
XLM- AGG
52.3
43.0
54.8
41.1
40.4
44,9
46.0
76.2
58.7
51.2
(4) XLM
X
52.0
58.5
42.6
54.8
39.2
41.6
44.3
47.3
74.7
58.3
51.3


Reviews:
[{"review_id": "yciT5wgHxO", "reviewer": "Reviewer_xtfQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s insightful questions. See below response.\n\n1. Why do the approximate methods exhibit superior retrieval performance: \nWe can divide the full MLM prediction task into two components: self-prediction and lexical expansion, i.e. predicting tokens other than those explicitly indicated. From the experimental results, we hypothesize that the key element for good transferability into different languages is self-prediction, not lexical expansion. In fact, when conducting full MLM prediction, the lexical expansion may cause errors in the agg* vector. Stated differently, the lexical expansion for English does not transfer to other languages as well as self-prediction of the token itself. We will discuss this further in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents mAggretriever, an extended version of Aggretriever, for multilingual retrieval. To improve the efficiency of mAggretriever, they introduced target-language and self MLM predictions. Although the proposed approaches are straightforward and simple, they demonstrate the effectiveness of the propose methods on two different datasets, using two pre-trained models, namely mBERTR and XLM-R.", "reasons_to_reject": "As depicted in Table 2, the proposed methods outperform mAggretriever using the whole-vocabulary MLM prediction, namely mBERTAGG and XLM-RAGG. \nLines 147-149 state, \"we propose two computationally efficient approaches to approximate MLM prediction.\" \nSince the proposed methods are approximations of the original approach, the question arises: Why do these approximate methods exhibit superior retrieval performance (R@1K, R@100) compared to the original MLM prediction?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents mAggretriever, a novel method that efficiently harnesses semantic and lexical attributes from pre-trained multilingual transformers like mBERT and XLM-R for dense retrieval. The authors propose two computationally efficient strategies to approximate MLM prediction. Despite being fine-tuned on English data, mAggretriever demonstrates impressive retrieval performance across various languages.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uNZsiKqX2M", "reviewer": "Reviewer_rgyC", "summary": "", "questions": "A. How do you determine if a token belongs to the target language? As some tokens may be shared across multiple languages.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s insightful discussion. See below responses to the concerns and questions.\n\n1. Results in Table2 do not look very high in MS MARCO: \nWhile we agree that knowledge distillation and hard negative mining might further improve our models’ retrieval effectiveness on MS MARCO, our main intention in presenting Table 2 was to provide a fair comparison with all baselines, including, for example, mDPR and mContriever, which only use MS MARCO training data constructed by BM25 mined negatives. \n\n2. Many works using [CLS] or average pooling have shown good results:\nWe admit that the short paper format limits our ability to showcase the full advantages of mAggretriever. Due to these limitations, we cannot compare with every state-of-the-art model, since many of those were evaluated on Mr. TyDi rather than MIRACL. But we report the comparisons in Table 6 (see Appendix A.4). Our mAggretriever consistently beats recent state-of-the-art models, which were trained on either multilingual training data or parallel translation data. We will point this out explicitly in our revised version.\n\n3. The assumption of only English training data is available is not valid: \nWe agree, but, as you mentioned, data from other languages is less plentiful than English, or, in extreme cases, non-existent. In addition, this data, when it is available, is often lower quality than English datasets (e.g. MS MARCO and NQ). Thus, using these non-English datasets would require exploration of how to improve the datasets’ quality as well as sampling strategies for the languages during training. On the other hand, mAggretriever provides an elegant solution to bypass these issues: namely, to train on English data and directly transfer the retrieval performance to other languages. We will clarify this further in our revised version.\n\n4. How do you determine if a token belongs to the target language?\nDue to space limitations, we describe the implementation details in Appendix A.1 (see target-token set construction). We will add the pointer to the appending in the revised version.\n\n5. The self prediction part in Line 162-169 is not very clear to me: \nThe self prediction is a bag-of-words representation with learned term weights. The learned term weight for each token is $w_i$ in Eq.(3). In our approach section, we mainly discuss how to output the MLM prediction probability; i.e., $p_{q_i}$ in Eq.(3). Since $p_{q_i}$ is normalized by softmax as shown in Eq.(2), directly conducting max pooling over all $p_{q_i}$ in Eq.(3) would ignore the importance of each input token $q_i$. Thus, we also use learned term weights to reweight the probability from each token $q_i$. This design is the same as Aggretriever. Therefore, self prediction is a bag-of-words representation where each appearing token $q_i$ has term weight $w_i$. Thank you for pointing out the unclarity and we will further clarify it in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper extends the study scope of Aggretriever from monolingual to multilingual scenarios and proposes several variants for representation aggregation.\n2. The experiments are very thorough and some of the results are interesting.\n", "reasons_to_reject": "1. Results in Table 2 do not look very high, especially on the MS-MARCO dataset. But this may be due to the fact that the authors do not use mined hard negatives.\n2. This paper builds on two assumptions: only English training data is available; aggregation from MLM distribution is necessary. However, these assumptions may not hold in real-world scenarios. Multilingual data are still available in many cases, though they may be much less than English data. Also, many papers utilizing only the [CLS] vector or mean pooling have shown good results.\n", "questions_to_authors": "A. How do you determine if a token belongs to the target language? As some tokens may be shared across multiple languages.\n", "paper_topic_and_main_contributions": "This paper studies the problem of aggregating [CLS] and MLM softmax predictions for multilingual information retrieval.\nIn the authors' setting, a multilingual pre-trained model is fine-tuned on English data only\nbut is evaluated on multilingual data.\nTo solve this problem,\nthe authors propose several variants for aggregation,\nincluding whole vocabulary, target language, and self prediction.\nExperiments on the MS-MARCO and MIRACL datasets find that the target language aggregation\nachieves competitive results while reducing the training and inference costs.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The self prediction part in Line 162-169 is not very clear to me. Is it a bag-of-words representation without any learned term weighting? Clarification would be appreciated.\n"}, "extra_text": {}}, {"review_id": "cOx5iPDnUn", "reviewer": "Reviewer_MRVN", "summary": "", "questions": "- The proposed method demonstrates substantial improvements in MRR and nDCG over mContriever, while showing only a slight improvement in recall. It would be valuable for the authors to provide insights into the reasons for this gap in performance.\n- It would be beneficial to include a comprehensive comparison of computation cost and inference latency with other baselines such as mContriever and mDPR to better evaluate the overall efficiency-effectiveness trade-off. Specifically, I would be interested in GPU usage and retrieval latency of mDPR/mContriever comapred to mAggretriever.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the insightful discussion and help comments. See below responses.\n\n1. The comparison is not entirely fair. A comprehensive comparison with mDPR and mContriever should be included:\nFirst of all, we would like to point out that mAggretriever’s final text representation is the same as other dense retrieval models; i.e., a  768-dimensional single-vector. The only difference is that mAggretriever’s vector is the concatenation of  128-dimensional [CLS] and 640-dimensional agg* vectors, all of which are dense vectors. Thus, there is no latency difference in end-to-end vector search using mAggretriever and other dense models. The only additional efficiency bottleneck of mAggretriever is MLM prediction and compressing the high-dimensional lexical vector into a 640-dimensional vector, which may sacrifice the training cost and query encoding latency as shown in Table1. Furthermore, our most efficient design (i.e., self prediction), due to no MLM prediction required, only shows a minor additional cost on query encoding latency on CPU (this additional cost comes from vector compression). Thus, we believe the comparison of mAggretriever with all the other dense models is fair and comprehensive enough. We will point out this explicitly and thanks for the comments.\n\t\n2. The proposed approach is limited; i.e., lexical representation is required:\nThe proposed approach is based on pre-trained transformers and as far as we know, all the pre-trained transformer language models can output lexical representations. Thus, we believe our approach can widely apply to different pre-trained transformer backbones.\n\n\n3. Recall comparison to mContriever:\nThere are two main differences between mContriever (with the backbone of mBERT) and our trained models, $\\text{mBERT}_{\\text{CLS}}$ and $\\text{mBERT}_{\\text{AGG}}$. First, mContriever experiences contrastive pre-training on 29 languages while our trained models do not; Second, mContriever is further fine-tuned on MS MARCO with a batch size of 1024 while our trained models use a batch size of 96. Although we cannot conduct such experiments to verify the effectiveness of the two factors, we hypothesize the high recall of mContriever comes from the multilingual contrastive pre-training. We thank you for the suggestions and agree that it is worth discussing. We will discuss this point in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is simple, improving both efficient and effective for multilingual retrieval.\n- The experimental methodology and analysis conducted in this study are extensive and thorough, with plenty insights and discussion provided", "reasons_to_reject": "- The motivation behind the Aggretriever-like design is clear, but it seems limited to this specific approach (or say approaches that require lexical representation).\n- The comparison made in the paper is not entirely fair, as it only considers Aggretriever-based methods. The efficiency workload stems from the design of Aggretriever which requires the high-dimensional lexical representation, and mDPR/mContriever would not face the such challenge. While the author highlight the effiency of the proposed method, it would be more appropriate to compare the efficiency-effectiveness balance with other methods like mDPR/mContriever.\n- [minor] Some notations are undefined and lead to confusion", "questions_to_authors": "- The proposed method demonstrates substantial improvements in MRR and nDCG over mContriever, while showing only a slight improvement in recall. It would be valuable for the authors to provide insights into the reasons for this gap in performance.\n- It would be beneficial to include a comprehensive comparison of computation cost and inference latency with other baselines such as mContriever and mDPR to better evaluate the overall efficiency-effectiveness trade-off. Specifically, I would be interested in GPU usage and retrieval latency of mDPR/mContriever comapred to mAggretriever.", "paper_topic_and_main_contributions": "This paper extend Aggretriever under multilingual retrieval senarios with two approaches proposed to alliviate the computation cost when incorperate with a large multilingual vocabulary space. One is target-language prediction, which mask the source langugae token during MLM projection. Another is self prediction, which mask all the other tokens except for the input ones. Experiments show that mAggretriever surpasses existing state-of-the-art multilingual dense retrieval models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- A task formulation of multilingual retrieval is expected.\n- Some abbreviations need clarification, such as \"tg\" (target) and \"wp\" (word piece), to avoid confusion for readers."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers found merits in the submission. Especially, it extends the study scope of Aggretriever from monolingual to multilingual scenarios. Although the performance improvement is not so great, I think the technical contribution may be inspiring to the EMNLP community.
Decision: Accept-Main