M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024)

Abstract: In this paper, we introduce a new embedding model called M3-Embedding, which is distinguished for its versatility in Multi-Linguality, Multi-Functionality, and Multi-Granularity. It provides a uniform support for the semantic retrieval of more than 100 working languages. It can simultaneously accomplish the three common retrieval functionalities: dense retrieval, multi-vector retrieval, and sparse retrieval. Besides, it is also capable of processing inputs of different granularities, spanning from short sentences to long documents of up to 8,192 tokens. The effective training of M3-Embedding presents a series of technical contributions. Notably, we propose a novel self-knowledge distillation approach, where the relevance scores from different retrieval functionalities can be integrated as the teacher signal to enhance the training quality. We also optimize the batching strategy, which enables a large batch size and high training throughput to improve the discriminativeness of embeddings. M3Embedding exhibits a superior performance in our experiment, leading to new state-of-the-art results on multilingual, cross-lingual, and longdocument retrieval benchmarks.1

heading: 1 Introduction
text: Embedding models are a critical form of DNN application in natural language processing. They encode the textual data in the latent space, where the underlying semantics of the data can be expressed by the output embeddings (Reimers and Gurevych, 2019; Ni et al., 2022). With the advent of pre-trained language models, the quality of text embeddings have been substantially improved, making them imperative components for the information retrieval (IR) system. One common form of embedding-based IR application is
†. Co-first author ∗. Corresponding authors 1. The model, code, and data is publicly available at
https://github.com/FlagOpen/FlagEmbedding.
BGE M3-Embedding
Multi-Lingual
Cross-Lingual
100+ Languages
Sparse Retrieval
Multi-Vec Retrieval
Dense Retrieval
Passage-Level
Doc-Level (≤8192)
Sentence-Level
Multi-Linguality Multi-Functionality Multi-Granularity

heading: Sparse Retrieval

heading: Dense Retrieval

heading: Multi-Linguality Multi-Functionality Multi-Granularity
text: dense retrieval, where relevant answers to the query can be retrieved based on the embedding similarity (Karpukhin et al., 2020; Xiong et al., 2020; Neelakantan et al., 2022; Wang et al., 2022; Xiao et al., 2023). Besides, the embedding model can also be applied to other IR tasks, such as multi-vector retrieval where the fine-grained relevance between query and document is computed based on the interaction score of multiple embeddings (Khattab and Zaharia, 2020), and sparse or lexical retrieval where the importance of each term is estimated by its output embedding (Gao et al., 2021a; Lin and Ma, 2021; Dai and Callan, 2020).
Despite the widespread popularity of text embeddings, the existing methods are still limited in versatility. First of all, most of the embedding models are tailored only for English, leaving few viable options for the other languages. Secondly, the existing embedding models are usually trained for one single retrieval functionality. However, typical IR systems call for the compound workflow of multiple retrieval methods. Thirdly, it is challenging to train a competitive long-document retriever due to the overwhelming training cost, where most of the embedding models can only support short inputs.
To address the above challenges, we introduce M3-Embedding, which is pronounced for its breakthrough of versatility in working languages, retrieval functionalities, and input granularities. Particularly, M3-Embedding is proficient in multi-
2318
linguality, which is able to support more than 100 world languages. By learning a common semantic space for different languages, enables both multilingual retrieval within each language and crosslingual retrieval between different languages. Besides, it is able to generate versatile embeddings to support different retrieval functionalities, not just dense retrieval, but also sparse retrieval and multivector retrieval. Finally, M3-Embedding is learned to process different input granularities, spanning from short inputs like sentences and passages, to long documents of up to 8,192 input tokens.
The training of M3-Embedding poses a significant challenge. In our work, the following technical contributions are made to optimize the embedding quality. Firstly, we propose a novel self knowledge distillation framework, where the multiple retrieval functionalities can be jointly learned and mutually reinforced. In M3-Embedding, the [CLS] embedding is used for dense retrieval, while embeddings from other tokens are used for sparse retrieval and multi-vector retrieval. Based on the principle of ensemble learning (Bühlmann, 2012), such heterogenous predictors can be combined as a stronger predictor. Thus, we integrate the relevance scores from different retrieval functions as the teacher signal, which is used to enhance the learning process via knowledge distillation. Secondly, we optimize the batching strategy to achieve a large batch size and high training throughput, which substantially contributes to the discriminativeness of embeddings. Last but not least, we perform extensive and high-quality data curation. Our dataset includes three sources: 1) the extraction of unsupervised data from massive multi-lingual corpora, 2) the integration of closely related supervised data, 3) the synthesization of scarce training data. The three data sources are complement to each other and applied to different training stages, which lays a solid foundation for the versatile text embeddings.
M3-Embedding exhibits a remarkable versatility in our experiments. It achieves superior retrieval quality for a variety of languages, leading to state-of-the-art performances on popular multilingual and cross-lingual benchmarks like MIRACL (Zhang et al., 2023c) and MKQA (Longpre et al., 2021). It effectively learns the three retrieval functionalities, which can not only work individually but also work together for an even stronger retrieval quality. It also well maintains its superior capability across different input granularities within 8192 tokens, which outperforms the existing
methods by a notable advantage. Our contributions are summarized as follows. 1) We present M3-Embedding, which achieves unprecedented versatility in multi-linguality, multifunctionality, and multi-granularity. 2) We propose a novel training framework of self-knowledge distillation and optimize the batching strategy for efficient training. We also create high-quality training resource based on comprehensive data curation. 3) Our model, code, and data is publicly available, offering critical resources for both direct usage and future development of text embeddings.

heading: 2 Related Work
text: The related works are reviewed from three aspects: general text embeddings, embedding models for neural retrieval, embeddings of multi-linguality.
In the past few years, substantial progress has been achieved in the field of text embedding. One major driving force is the popularity of pre-trained language models, where the underlying semantic of the data can be effectively encoded by such powerful text encoders (Reimers and Gurevych, 2019; Karpukhin et al., 2020; Ni et al., 2022). In addition, the progress of contrastive learning is another critical factor, especially the improvement of negative sampling (Xiong et al., 2020; Qu et al., 2021) and the exploitation of knowledge distillation (Hofstätter et al., 2021; Ren et al., 2021; Zhang et al., 2021a). On top of these well-established techniques, it becomes increasingly popular to learn versatile embedding models, which are able to uniformly support a variety of application scenarios. So far, there have been many impactful methods in the direction, like Contriever (Izacard et al., 2022), LLM-Embedder (Zhang et al., 2023a), E5 (Wang et al., 2022), BGE (Xiao et al., 2023), SGPT (Muennighoff, 2022), and Open Text Embedding (Neelakantan et al., 2022), which significantly advance the usage of text embeddings for general tasks.
One major application of embedding models is neural retrieval (Lin et al., 2022). By measuring the semantic relationship with the text embeddings, the relevant answers to the input query can be retrieved based on the embedding similarity. The most common form of embedding-based retrieval method is dense retrieval (Karpukhin et al., 2020), where the text encoder’s outputs are aggregated (e.g., via [CLS] or mean-pooling) to compute the embedding similarity. Another common alternative is known as multi-vecor retrieval (Khattab and Za-
haria, 2020; Humeau et al., 2020), which applies fine-grained interactions for the text encoder’s outputs to compute the embedding similarity. Finally, the text embeddings can also be transformed into term weights, which facilitates sparse or lexical retrieval (Luan et al., 2021; Dai and Callan, 2020; Lin and Ma, 2021). Typically, the above retrieval methods are realized by different embedding models. To the best of our knowledge, no existing method is able to unify all these functionalities.
Despite the substantial technical advancement, most of the existing text embeddings are developed only for English, where other languages are lagging behind. To mitigate this problem, continual efforts are presented from multiple directions. One is the development of pre-trained multi-lingual text encoders, such as mBERT (Pires et al., 2019), mT5 (Xue et al., 2021), XLM-R (Conneau et al., 2020). Another one is the curation of training and evaluation data for multi-lingual text embeddings, e.g., MIRACL (Zhang et al., 2023c), mMARCO (Bonifacio et al., 2021), Mr. TyDi (Zhang et al., 2021b), MKQA (Longpre et al., 2021). At the same time, the multi-lingual text embeddings are continually developed from the community, e.g., mDPR (Zhang et al., 2023b), mContriever (Izacard et al., 2022), mE5 (Wang et al., 2022), etc. However, the current progress is still far from enough given the notable gap with English models and the huge imbalance between different languages.

heading: 3 M3-Embedding
text: M3-Embedding realizes three-fold versatility. It supports a wide variety of languages and handles input data of different granularities. Besides, it unifies the common retrieval functionalities of text embeddings. Formally, given a query q in an arbitrary language x, it is able to retrieve document d in language y from the corpus Dy: dy ← fn∗(qx, Dy). In this place, fn∗(·) belongs to any of the functions: dense, lexical, or multi-vector retrieval; y can be another language or the same language as x.

heading: 3.1 Data Curation
text: M3-Embedding calls for a large-scale and diverse multi-lingual dataset. In this work, we perform comprehensive data collection from three sources: the unsupervised data from unlabeled corpora, the fine-tuning data from labeled corpora, and the finetuning data via synthesization (shown as Table 8). The three data sources complement to each other,
which are applied to different stages of the training process. Particularly, the unsupervised data is curated by extracting the rich-semantic structures, e.g., title-body, title-abstract, instruction-output, etc., within a wide variety of multi-lingual corpora, including Wikipedia, S2ORC (Lo et al., 2020), xP3 (Muennighoff et al., 2023), mC4 (Raffel et al., 2020), CC-News (Hamborg et al., 2017) and the well-curated data from MTP (Xiao et al., 2023). To learn the unified embedding space for cross-lingual semantic matching, the parallel sentences are introduced from two translation datasets, NLLB (NLLB Team et al., 2022) and CCMatrix (Schwenk et al., 2021). The raw data is filtered to remove potential bad contents and low-relevance samples. In total, it brings in 1.2 billion text pairs of 194 languages and 2655 cross-lingual correspondences.
Besides, we collect relatively small but diverse and high-quality fine-tuning data from labeled corpora. For English, we incorporate 8 datasets, including HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), NQ (Kwiatkowski et al., 2019), MS MARCO (Nguyen et al., 2016), COLIEE (Kim et al., 2023), PubMedQA (Jin et al., 2019), SQuAD (Rajpurkar et al., 2016), and NLI data from SimCSE (Gao et al., 2021b). For Chinese, we integrate 7 datasets, including DuReader (He et al., 2018), mMARCO-ZH (Bonifacio et al., 2021), T2-Ranking (Xie et al., 2023), LawGPT(Liu et al., 2023), CMedQAv2 (Zhang et al., 2018), NLIzh2, and LeCaRDv2 (Li et al., 2023). For other languages, we leverage the training data from Mr. Tydi (Zhang et al., 2021b) and MIRACL (Zhang et al., 2023c).
Finally, we generate synthetic data to mitigate the shortage of long document retrieval tasks and introduce extra multi-lingual fine-tuning data (denoted as MultiLongDoc). Specifically, we sample lengthy articles from Wikipedia, Wudao (Yuan et al., 2021) and mC4 datasets and randomly choose paragraphs from them. Then we use GPT3.5 to generate questions based on these paragraphs. The generated question and the sampled article constitute a new text pair to the fine-tuning data. Detailed specifications are presented in Appendix A.2.

heading: 3.2 Hybrid Retrieval
text: M3-Embedding unifies the common retrieval functionalities of the embedding model, i.e. dense retrieval, lexical (sparse) retrieval, and multi-vector
2. https://huggingface.co/datasets/shibing624/nli-zh-all
retrieval. The formulation is presented as follows. • Dense retrieval. The input query q is transformed into the hidden states Hq based on a text encoder. We use the normalized hidden state of the special token “[CLS]” for the representation of the query: eq = norm(Hq[0]). Similarly, we can get the embedding of passage p as ep = norm(Hp[0]). Thus, the relevance score between query and passage is measured by the inner product between the two embeddings eq and ep: sdense ← ⟨ep, eq⟩. • Lexical Retrieval. The output embeddings are also used to estimate the importance of each term to facilitate lexical retrieval. For each term t within the query (a term is corresponding to a token in our work), the term weight is computed as wqt ← Relu(WTlexHq[i])), where Wlex ∈ Rd×1 is the matrix mapping the hidden state to a float number. If a term t appears multiple times in the query, we only retain its max weight. We use the same way to compute the weight of each term in the passage. Based on the estimation term weights, the relevance score between query and passage is computed by the joint importance of the co-existed terms (denoted as q ∩ p) within the query and passage: slex ← ∑ t∈q∩p(wqt ∗ wpt).
•Multi-Vector Retrieval. As an extension of dense retrieval, the multi-vector method utilizes the entire output embeddings for the representation of query and passage: Eq = norm(WTmulHq), Ep = norm(W T mulHp), where Wmul ∈ Rd×d is the learnable projection matrix. Following ColBert (Khattab and Zaharia, 2020), we use lateinteraction to compute the fine-grained relevance score: smul ← 1N ∑N i=1max M j=1Eq[i] · ETp [j]; N and M are the lengths of query and passage. Thanks to the multi-functionality of the embedding model, the retrieval process can be conducted
in a hybrid process. First of all, the candidate results can be individually retrieved by each of the methods (the multi-vector method can be exempted from this step due to its heavy cost). Then, the final retrieval result is re-ranked based on the integrated relevance score:
srank ← w1 · sdense + w2 · slex + w3 · smul (1)
where the values of w1, w2 and w3 depend on the downstream scenario.

heading: 3.3 Self-Knowledge Distillation
text: The embedding model is trained to discriminate the positive samples from the negative ones. For each of the retrieval methods, it is expected to assign a higher score for the query’s positive samples compared with the negative ones. Therefore, the training process is conducted to minimize the InfoNCE loss, whose general form is presented by the following loss function:
Ls(·) = − log exp(s(q, p∗)/τ)∑
p∈{p∗,P ′} exp(s(q, p)/τ) . (2)
Here, p∗ and P ′ stand for the positive and negative samples to the query q; s(·) is any of the functions within {sdense(·), slex(·), smul(·)}.
The training objectives of different retrieval methods can be mutually conflicting with each their. Therefore, the native multi-objective training can be unfavorable to the embedding’s quality. To facilitate the optimization of multiple retrieval functions, we propose to unify the training process on top of self-knowledge distillation. Particularly, based on the principle of ensemble learning (Bühlmann, 2012), the predictions from different retrieval methods can be integrated as a more accurate relevance score given their heterogeneous
nature. In the simplest form, the integration can just be the weighted sum of different prediction scores:
sinter ← w1 · sdense +w2 · slex +w3 · smul. (3)
Then we compute the weighted sum of Ldense, Llex, Lmul and Linter as the loss without selfknowledge distillation: L ← ( λ1·Ldense+λ2·Llex+λ3·Lmul+Linter
) /4. (4)
In previous studies, the training quality of embedding model can benefit from knowledge distillation, which takes advantage of fine-grained soft labels from another ranking model (Hofstätter et al., 2021). In this place, we simply employ the integration score sinter as the teacher, where the loss function of each retrieval method is modified as:
L′∗ ← −p(sinter) ∗ log p(s∗). (5)
Here, p(·) is the softmax activation; s∗ is any of the members within sdense, slex, and smul. We further integrate and normalize the modified loss function:
L′ ← ( λ1 ·L′dense+λ2 ·L′lex+λ3 ·L′mul ) /3. (6)
Finally, we derive the final loss function for selfknowledge distillation with the linear combination of L and L′: Lfinal ← ( L+ L′ ) /2.
The training process constitutes a multi-stage workflow (Figure 2). In the first place, the text encoder (an XLM-RoBERTa (Conneau et al., 2020) model adapted by RetroMAE (Xiao et al., 2022) method) is pre-trained with the massive unsupervised data, where only the dense retrieval is trained in the basic form of contrastive learning. The self-knowledge distillation is applied to the second stage, where the embedding model is fine-tuned to establish the three retrieval functionalities. The random initialization of Wlex led to poor slex accuracy and high Llex at the beginning of the training. In order to reduce the impact of this, we set w1 = 1, w2 = 0.3, w3 = 1, λ1 = 1, λ2 = 0.1 and λ3 = 1 during the training process. Both labeled and synthetic data are used in this stage, where hard negative samples are introduced for each query following the ANCE method (Xiong et al., 2020). (See Appendix B.1 for more details.)

heading: 3.4 Efficient Batching
text: The embedding model needs to learn from diverse and massive multi-lingual data to fully capture the
general semantic of different languages. It also needs to keep the batch size as large as possible (introducing a huge amount of in-batch negatives) to ensure the discriminativeness of text embeddings. Given the limitations on GPU’s memory and computation power, people usually truncate the input data into short sequences for high throughput of training and a large batch size. However, the common practice is not a feasible option for M3-Embedding because it needs to learn from both short and long-sequence data to effectively handle the input of different granularities. In our work, we improve the training efficiency by optimizing the batching strategy, which enables high training throughput and large batch sizes.
Particularly, the training data is pre-processed by being grouped by sequence length. When producing a mini-batch, the training instances are sampled from the same group. Due to the similar sequence lengths, it significantly reduces sequence padding (Figure 3, marked in red) and facilitates a more effective utilization of GPUs. Besides, when sampling the training data for different GPUs, the random seed is always fixed, which ensures the load balance and minimizes the waiting time in each training step. Besides, when handling longsequence training data, the mini-batch is further divided into sub-batches, which takes less memory footprint. We iteratively encode each sub-batch using gradient checkpointing (Chen et al., 2016) and gather all generated embeddings. This method can significantly increase the batch size. For example, when processing text with a length of 8192, the batch size can be increased by more than 20 times. (see Appendx B.3 for more details.) Finally, the embeddings from different GPUs are broadcasted, allowing each device to obtain all embeddings in the distributed environment, which notably expands the scale of in-bath negative samples.
Model Avg ar bn en es fa fi fr hi id ja ko ru sw te th zh de yo Baselines (Prior Work) BM25 31.9 39.5 48.2 26.7 7.7 28.7 45.8 11.5 35.0 29.7 31.2 37.1 25.6 35.1 38.3 49.1 17.5 12.0 56.1 mDPR 41.8 49.9 44.3 39.4 47.8 48.0 47.2 43.5 38.3 27.2 43.9 41.9 40.7 29.9 35.6 35.8 51.2 49.0 39.6 mContriever 43.1 52.5 50.1 36.4 41.8 21.5 60.2 31.4 28.6 39.2 42.4 48.3 39.1 56.0 52.8 51.7 41.0 40.8 41.5 mE5large 66.6 76.0 75.9 52.9 52.9 59.0 77.8 54.5 62.0 52.9 70.6 66.5 67.4 74.9 84.6 80.2 56.0 56.4 78.3 E5mistral-7b 63.4 73.3 70.3 57.3 52.2 52.1 74.7 55.2 52.1 52.7 66.8 61.8 67.7 68.4 73.9 74.0 54.0 54.1 79.7 OpenAI-3 54.9 - - - - - - - - - - - - - - - - - - M3-Embedding (Our Work) Dense 69.2 78.4 80.0 56.9 56.1 60.9 78.6 58.3 59.5 56.1 72.8 69.9 70.1 78.7 86.2 82.6 62.7 56.7 81.8 Sparse 53.9 67.1 68.9 43.8 38.6 45.1 65.4 35.3 48.2 48.9 56.1 61.5 44.5 57.9 79.1 70.9 36.1 32.5 70.0 Multi-vec 70.5 79.6 81.0 59.3 57.8 62.0 80.1 59.4 61.5 58.3 74.5 71.2 71.2 79.1 87.9 83.0 63.7 58.0 82.4 Dense+Sparse 70.4 79.6 80.7 58.8 58.1 62.3 79.7 58.0 62.9 58.3 73.9 71.2 69.8 78.5 87.2 83.1 63.5 57.7 83.3 All 71.5 80.2 81.5 59.6 59.7 63.4 80.4 61.2 63.3 59.0 75.2 72.1 71.7 79.6 88.1 83.7 64.9 59.8 83.5
Table 1: Multi-lingual retrieval performance on the MIRACL dev set (measured by nDCG@10).
For users who are severely limited in computation or data resource, we present an even simpler method called MCLS (Multi-CLS), which simply inserts multiple CLS tokens to the long document during inference, and takes the average of all CLS embeddings as the ultimate embedding of the document. Despite simplicity, it is surprisingly effective in practice. (See Appendix B.2 for more details.)

heading: 4 Experiment
text: In this section, we investigate M3-Embedding’s performance in terms of multi-lingual retrieval, crosslingual retrieval, and long-doc retrieval. We also explore the impact of its technical factors.

heading: 4.1 Multi-Lingual Retrieval
text: We evaluate the multi-lingual retrieval performance with MIRACL (Zhang et al., 2023c), which consists of ad-hoc retrieval tasks in 18 languages. Each task is made up of query and passage presented in the same language. Following the official benchmark, we evaluate our method using Pyserini (Lin et al., 2021), and use nDCG@10 as the primary evaluation metric (Recall@100 is also measured and reported in Appendix C.1). Specifically, for the dense method (denoted as Dense), we first use it to generate the embeddings of the corpus and then build the dense index for searching top-1000 candidates with Faiss. For the sparse method (denoted as Sparse), we first use it to generate the weights of the corpus and then build the sparse index for searching top-1000 candidates with Lucene. For the multi-vector method (denoted as Multi-vec), considering its heavy cost, we use it as reranker to re-rank the top-200 candidates from dense method. For the hybrid retrieval of dense method and sparse method (denoted as Dense+Sparse), we set w1 = 1, w2 = 0.3 and
w3 = 0 in equation(1) to re-rank the union set of top-1000 candidates from Dense and top-1000 candidate from Sparse. For the hybrid retrieval of all three methods (denoted as All), we set w1 = 1, w2 = 0.3 and w3 = 1 in equation(1) to re-rank the top-200 candidates from Dense.
We incorporate the following baselines in our experiment: the lexical retrieval method: BM25 (Robertson and Zaragoza, 2009); the dense retrieval methods: mDPR3 (Zhang et al., 2023b), mContriever4 (Izacard et al., 2022), mE5large (Wang et al., 2022) and E5mistral-7b (Wang et al., 2023). To make the BM25 and M3 more comparable, in the experiment, we use the same tokenizer as M3 (i.e., the tokenizer of XLM-Roberta) for BM25. Using the same vocabulary from XLM-Roberta can also ensure that both approaches have the same retrieval latency. The results of BM25 with different tokenizers are shown in Appendix C.2. We also make a comparison with Text-Embedding-3-Large(abbreviated as OpenAI3), which was recently released by OpenAI5.
We can make the following observations according to the experiment result in Table 1. Firstly, M3Embedding already achieves a superior retrieval performance with only its dense retrieval functionality (Dense). It not only outperforms other baseline methods in the average performance, but also maintains a consistent empirical advantage in most of individual languages. Even compared with E5mistral-7b, which leverages a much larger Mistral-7B model as the text encoder and specifically trained with English data, our method is able to produce a similar result in English and notably higher results in the other languages. Besides, the sparse retrieval functionality (Sparse) is also ef3. https://huggingface.co/castorini/mdpr-tied-pft-msmarco 4. https://huggingface.co/facebook/mcontriever-msmarco 5. https://platform.openai.com/docs/guides/embeddings
fectively trained by M3-Embedding, as it outperforms the typical BM25 methods in all languages. We can also observe the additional improvement from multi-vector retrieval, which relies on finegrained interactions between query and passage’s embeddings to compute the relevance score. Finally, the collaboration of dense and sparse method (Dense+Sparse) leads to a further improvement over each individual method, and the collaboration of all three methods (All) brings forth the best performance.

heading: 4.2 Cross-Lingual Retrieval
text: We make evaluation for the cross-lingual retrieval performance with the MKQA benchmark (Longpre et al., 2021), which includes queries in 25 nonEnglish languages. For each query, it needs to retrieve the passages containing answers from the English Wikipedia corpus. In our experiment, we make use of the well-processed corpus offered by the BEIR6 (Thakur et al., 2021). Following the previous study (Izacard et al., 2022), we report Recall@100 as the primary metric (Recall@20 is reported as an auxiliary metric in the Appendix C.1). For Dense+Sparse method and All method, we set the same weights as in MIRACL dataset.
The experiment result is shown in Table 2. Sim-
6. https://huggingface.co/datasets/BeIR/nq
ilar to our observation in multi-lingual retrieval, M3-Embedding continues to produce a superior performance, where it notably outperforms other baseline methods purely with its dense retrieval functionality (Dense). The collaboration of different retrieval methods brings in further improvements, leading to the best empirical performance of cross-lingual retrieval. Besides, we can also observe the following interesting results which are unique to this benchmark. Firstly, the performance gaps are not as significant as MIRACL, where competitive baselines like E5mistral-7b is able to produce similar or even better results on some of the testing languages. However, the baselines are prone to bad performances in many other languages, especially the low-resource languages, such as ar, km, he, etc. In contrast, M3-Embedding maintains relatively stable performances in all languages, which can largely be attributed to its pre-training over comprehensive unsupervised data. Secondly, although M3Embedding (Sparse) is still better than BM25, it performs badly compared with other methods. This is because there are only very limited co-existed terms for cross-lingual retrieval as the query and passage are presented in different languages.

heading: 4.3 Multilingual Long-Doc Retrieval
text: We evaluate the retrieval performance with longer sequences with two benchmarks: MLDR (Multi-
Max Length Avg ar de en es fr hi it ja ko pt ru th zh Baselines (Prior Work) BM25 8192 53.6 45.1 52.6 57.0 78.0 75.7 43.7 70.9 36.2 25.7 82.6 61.3 33.6 34.6 mDPR 512 23.5 15.6 17.1 23.9 34.1 39.6 14.6 35.4 23.7 16.5 43.3 28.8 3.4 9.5 mContriever 512 31.0 25.4 24.2 28.7 44.6 50.3 17.2 43.2 27.3 23.6 56.6 37.7 9.0 15.3 mE5large 512 34.2 33.0 26.9 33.0 51.1 49.5 21.0 43.1 29.9 27.1 58.7 42.4 15.9 13.2 E5mistral-7b 8192 42.6 29.6 40.6 43.3 70.2 60.5 23.2 55.3 41.6 32.7 69.5 52.4 18.2 16.8 text-embedding-ada-002 8191 32.5 16.3 34.4 38.7 59.8 53.9 8.0 46.5 28.6 20.7 60.6 34.8 9.0 11.2 jina-embeddings-v2-base-en 8192 - - - 37.0 - - - - - - - - - - M3-Embedding (Our Work) Dense 8192 52.5 47.6 46.1 48.9 74.8 73.8 40.7 62.7 50.9 42.9 74.4 59.5 33.6 26.0 Sparse 8192 62.2 58.7 53.0 62.1 87.4 82.7 49.6 74.7 53.9 47.9 85.2 72.9 40.3 40.5 Multi-vec 8192 57.6 56.6 50.4 55.8 79.5 77.2 46.6 66.8 52.8 48.8 77.5 64.2 39.4 32.7 Dense+Sparse 8192 64.8 63.0 56.4 64.2 88.7 84.2 52.3 75.8 58.5 53.1 86.0 75.6 42.9 42.0 All 8192 65.0 64.7 57.9 63.8 86.8 83.9 52.2 75.5 60.1 55.7 85.4 73.8 44.7 40.0 M3-w.o.long Dense-w.o.long 8192 41.2 35.4 35.2 37.5 64.0 59.3 28.8 53.1 41.7 29.8 63.5 51.1 19.5 16.5 Dense-w.o.long (MCLS) 8192 45.0 37.9 43.3 41.2 67.7 64.6 32.0 55.8 43.4 33.1 67.8 52.8 27.2 18.2
Table 3: Evaluation of multilingual long-doc retrieval on the MLDR test set (measured by nDCG@10).
lingual Long-Doc Retrieval), which is curated by the multilingual articles from Wikipedia, Wudao and mC4 (see Table 7), and NarrativeQA (Kočiský et al., 2018; Günther et al., 2023), which is only for English. In addition to the previous baselines, we further introduce JinaEmbeddingv2 (Günther et al., 2023), text-embeddingada-002 and text-embedding-3-large from OpenAI given their outstanding long-doc retrieval capability. For Dense+Sparse method, we set w1 = 0.2, w2 = 0.8 and w3 = 0 in equation(1). For All method, we set w1 = 0.15, w2 = 0.5 and w3 = 0.35 in equation(1).
The evaluation result on MLDR is presented in Table 3. Interestingly, M3 (Sparse) turns out to be a more effective method for long document retrieval, which achieves another about 10 points improvement over the dense method. Besides, the multivector retrieval is also impressive, which brings 5.1+ points improvement over M3 (Dense). Finally, the combination of different retrieval methods leads to a remarkable average performance of 65.0.
To explore the reason for M3-Embedding’s competitiveness in long-document retrieval, we perform the ablation study by removing the long document data from the fine-tuning stage (denoted as w.o. long). After this modification, the dense method, i.e. Dense-w.o.long, can still outperform the majority of baselines, which indicates that its empirical advantage has been well established during the pre-training stage. We also propose a simple strategy, MCLS, to address this situation (no data or no GPU resource for document-retrieval fine-tuning). Experimental results indicate that MCLS can significantly improve the performance of document retrieval without training (41.2→ 45.0).
We make further analysis with NarrativeQA (Table 4), where we can make a similar observation as MLDR. Besides, with the growth of sequence length, our method gradually expands its advantage over baseline methods (Figure 5), which reflects its proficiency in handling long inputs.

heading: 4.4 Ablation study
text: Self-knowledge distillation. The ablation study is performed to analyze the impact of self-knowledge distillation (skd). Particularly, we disable the distillation processing and have each retrieval method trained independently (denoted as M3-w.o.skd). According to our evaluation on MIRACL (Table 5), the original method, i.e. M3-w.skd, is able to achieve better performances than the ablation method in all settings, i.e., Dense, Sparse, Multivec. Notably, the impact is more pronounced for sparse retrieval. Such a result also reflects the incompatibility between dense and sparse retrieval methods. With skd, the incompatibility can be largely overcome. (More detailed results are avail-
able in Appendix C.1.) Impact of multi-stage training. We also make explorations for the impacts from different training stages. Fine-tuning indicates the direct fine-tuning from XLM-RoBERTA (Conneau et al., 2020); RetroMAE+Fine-tuning refers to the fine-tuning on the pre-trained model from RetroMAE (Xiao et al., 2022). Meanwhile, RetroMAE+Unsup+Finetuning involves fine-tuning on a model that is trained with RetroMAE and then pre-trained on unsupervised data. The results are presented in Table 6. We can observe that RetroMAE can significantly improve the retrieval performance, and pre-training on unsupervised data can further enhance the retrieval quality of the embedding model. (More detailed results are available in Appendix C.1.)

heading: 5 Conclusion
text: In this paper, we introduce M3-Embedding, which substantially advances the versatility of text embeddings in terms of supporting multi-lingual retrieval, handling input of diverse granularities, and unifying different retrieval functionalities. M3Embedding presents three technical contributions: self-knowledge distillation, efficient batching, and high-quality curation of data. The effectiveness of M3-Embedding is empirically verified, where it leads to superior performances on multi-lingual retrieval, cross-lingual retrieval, and multi-lingual long-document retrieval tasks.

heading: Limitations
text: First of all, while our proposed M3-Embedding model achieves state-of-the-art performance on popular multi-lingual and cross-lingual bench-
marks such as MIRACL and MKQA, it is important to acknowledge that the generalizability of our approach to diverse datasets and real-world scenarios needs to be further investigated. Different datasets may have varying characteristics and challenges that could affect the performance of our model. Secondly, while M3-Embedding is designed to process inputs of different granularities, including long documents of up to 8192 tokens, we acknowledge that processing extremely long documents could pose challenges in terms of computational resources and model efficiency. The performance of our model on very long documents or documents exceeding the specified token limit needs to be further investigated. Furthermore, we claim support for more than 100 working languages in M3-Embedding. However, the potential variations in performance across different languages are not thoroughly discussed. Further analysis and evaluation on a broader range of languages are necessary to understand the robustness and effectiveness of our model across different language families and linguistic characteristics.

heading: Ethics Consideration
text: Our work proposes a new embedding model called M3-Embedding, which is distingulished for its versality in multi-linguality, multi-functionality and multi-granularity. Because our model will be publicly avaliable, it is influenced by the inherent impacts of open-source model. Moreover, we use the multilingual data including all kinds of languages in the training of M3-Embedding. However, due to the uneven distribution of training data for different languages, the model’s performance may vary across languages, which could potentially be seen as discriminatory or unfair. We ensure that our work is conformant to the ACL Ethics Policy7.

heading: Acknowledgements
text: We would like to thank anonymous reviewers for their helpful feedback, and ACL 2024 and ACL Rolling Review organizers for their efforts. This research is supported by National Science and Technology Major Project (2023ZD0121504).

heading: A Details of Datasets

heading: A.1 Collected Data
text: The language and length distribution (the number of tokens) of the unsupervised data are illustrated in Figure 4.
We observed that for long texts (e.g., the news in cc-news), the initial sentences tend to be summarizing statements, and the model can rely solely on the information presented in these initial sentences to establish relevant relationships. To prevent the model from focusing solely on these starting sentences, we implemented a strategy of randomly shuffling the order of segments within entire texts. Specifically, we divided the text into three segments, shuffled their order randomly, and recombined them. This approach allows relevant text segments to appear randomly at any position within the long sequence. During training, we applied this operation to passages with a probability of 0.2%.

heading: A.2 Synthetic Data
text: The prompt for GPT3.5 is “You are a curious AI assistant, please generate one specific and valuable question based on the following text. The generated question should revolve around the core content of this text, and avoid using pronouns (e.g., ”this”). Note that you should generate only one question, without including additional content:”. The details of generated dataset are shown in Table 7.
B Implementation Details

heading: B.1 Experimental Hyperparameters
text: We adopt a further pre-trained XLM-RoBERTa8 as the foundational model. We extend the max position to 8192 and update the model via the RetroMAE (Xiao et al., 2022) method. The data comprises Pile (Gao et al., 2020), Wudao (Yuan et al., 2021), and mC4 (Raffel et al., 2020) datasets. We sampled a total of 184 million text samples from these sources, covering 105 languages. The maximum sequence length is 8192 and the learning rate is 7 × 10−5. The batch size is set to 32 and we accumulate the gradient over 16 steps. Pre-training is conducted on 32 A100(40GB) GPUs for 20,000 steps.
For the pre-training with the massive unsupervised data, the max length of query and passage is set to 512 and 8192, respectively. The learning rate is 5× 10−5, the warmup ratio is 0.1 and the weight 8. https://huggingface.co/FacebookAI/xlm-roberta-large
decay is 0.01. This training process takes 25,000 steps. For training data with different sequence length ranges (e.g., 0-500, 500-1000, etc.), we use different batch sizes. The details are represented in Table 9. The second stage is conducted on 96 A800(80GB) GPUs.
In the fine-tuning stage, we sample 7 negatives for each query. Refer to Table 9 for the batch size. In the initial phase, we employed approximately 6000 steps to perform warm-up on dense embedding, sparse embedding and multi-vectors. Subsequently, we conducted unified training with selfknowledge distillation. These experiments were carried out on 24 A800(80GB) GPUs.

heading: B.2 MCLS Method
text: The fine-tuning using long text can be constrained due to the absence of long text data or computation resources. In this situation, we propose a simple but effective method: MCLS(Multiple CLS) to enhance the model’s ability without fine-tuning on long text. The MCLS method aims to utilize multiple CLS tokens to jointly capture the semantics of long texts. Specifically, we insert a CLS token for every fixed number of tokens (in our experiments, we insert a “[CLS]” for each 256 tokens), and each CLS token can capture semantic information from its neighboring tokens. Ultimately, the final text embedding is obtained by averaging the last hidden states of all CLS tokens.

heading: B.3 Split-batch Method
text: Algorithm 1 Pseudocode of split-batch.
# enable gradient-checkpointing M3.gradient_checkpointing_enable()
embs = [] for batch_data in loader:
# split the large batch into multiple sub-batch for sub_batch_data in batch_data:
sub_emb = M3(sub_batch_data) # only collect the embs embs.append(sub_emb)
# concatenate the outputs to get final embeddings embs = cat(embs)
Algorthm 1 provides the pseudo-code of the splitbatch strategy. For the current batch, we partition it into multiple smaller sub-batches. For each subbatch we utilize the model to generate embeddings, discarding all intermediate activations via gradient checkpointing during the forward pass. Finally, we gather the encoded results from all sub-batch, and obtain the embeddings for the current batch. It is crucial to enable the gradient-checkpointing
strategy; otherwise, the intermediate activations for each sub-batch will continuously accumulate, ultimately occupying the same amount of GPU memory as traditional methods.
In Table 10, we investigate the impact of splitbatch on batch size. It can be observed that, with the split-batch enabled, there is a significant increase in batch size. Simultaneously, the increase becomes more pronounced with longer text lengths, and in the case of a length of 8192, enabling splitbatch results in a growth of batch size by over 20 times.

heading: C More Results

heading: C.1 Additional Resutls
text: In this section, we present additional evaluation results on the MIRACL and MKQA benchmarks. As shown in Table 12 and 13, M3-Embedding outperforms all baselines on average.
The detailed results of ablation studies of selfknowledge distillation and multi-stage training on the MIRACL dev set are shown in Table 14 and Table 15.

heading: C.2 Different Tokenizer for BM25
text: We investigate the impact of different tokenizers on the BM25 method, and the results are shown in Table 11. We can observe that:
Use Split-batch Max Length
1024 4096 8192
× 262 25 6 √
855 258 130
unique tokens after encoding documents (for example, on the MLDR dataset, the tokenizer of XLM-RoBERTa produces 1056 unique terms per article, while Lucene’s analyzer generates 1451 unique terms, which is over 37% more and will increase retrieval latency).
• M3 outperforms BM25 models using the same tokenizer on all datasets, indicating that the learned weights are significantly better than the weights calculated by BM25.

heading: Future Work
text: 3) Our model, code, and data is publicly available, offering critical resources for both direct usage and future development of text embeddings