Update README.md
Browse files
README.md
CHANGED
@@ -16,17 +16,9 @@ In this project, we introduce BGE-M3, which is distinguished for its versatility
|
|
16 |
- Multi-Linguality: It can support more than 100 working languages.
|
17 |
- Multi-Granularity: It is able to process inputs of different granularities, spanning from short sentences to long documents of up to 8192 tokens.
|
18 |
|
19 |
-
**Some suggestions for retrieval pipeline in RAG:**
|
20 |
-
We recommend to use following pipeline: hybrid retrieval + re-ranking.
|
21 |
-
- Hybrid retrieval leverages the strengths of various methods, offering higher accuracy and stronger generalization capabilities.
|
22 |
-
A classic example: using both embedding retrieval and the BM25 algorithm.
|
23 |
-
Now, you can try to use BGE-M3, which supports both embedding and sparse retrieval.
|
24 |
-
This allows you to obtain token weights (similar to the BM25) without any additional cost when generate dense embeddings.
|
25 |
-
- As cross-encoder models, re-ranker demonstrates higher accuracy than bi-encoder embedding model.
|
26 |
-
Utilizing the re-ranking model (e.g., [bge-reranker](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/reranker), [cohere-reranker](https://txt.cohere.com/rerank/)) after retrieval can further filter the selected text.
|
27 |
-
|
28 |
|
29 |
## News:
|
|
|
30 |
- 2024/3/2: Release unified fine-tuning [example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/unified_finetune) and [data](https://huggingface.co/datasets/Shitao/bge-m3-data)
|
31 |
- 2024/2/6: We release the [MLDR](https://huggingface.co/datasets/Shitao/MLDR) (a long document retrieval dataset covering 13 languages) and [evaluation pipeline](https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB/MLDR).
|
32 |
- 2024/2/1: **Thanks for the excellent tool from Vespa.** You can easily use multiple modes of BGE-M3 following this [notebook](https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/mother-of-all-embedding-models-cloud.ipynb)
|
@@ -95,6 +87,15 @@ If you want to fine-tune all embedding function of m3, you can refer to the [uni
|
|
95 |
|
96 |
|
97 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
98 |
|
99 |
|
100 |
## Usage
|
@@ -217,9 +218,13 @@ print(model.compute_score(sentence_pairs,
|
|
217 |
|
218 |
## Evaluation
|
219 |
|
|
|
|
|
|
|
|
|
220 |
|
221 |
-
We compare BGE-M3 with some popular methods, including BM25, openAI embedding, etc.
|
222 |
|
|
|
223 |
- Multilingual (Miracl dataset)
|
224 |
|
225 |
![avatar](./imgs/miracl.jpg)
|
@@ -252,6 +257,7 @@ especially in long document retrieval.
|
|
252 |
![avatar](./imgs/bm25.jpg)
|
253 |
|
254 |
|
|
|
255 |
## Training
|
256 |
- Self-knowledge Distillation: combining multiple outputs from different
|
257 |
retrieval modes as reward signal to enhance the performance of single mode(especially for sparse retrieval and multi-vec(colbert) retrival)
|
@@ -266,8 +272,8 @@ Refer to our [report](https://arxiv.org/pdf/2402.03216.pdf) for more details.
|
|
266 |
|
267 |
## Acknowledgement
|
268 |
|
269 |
-
Thanks the authors of open-sourced datasets, including Miracl, MKQA, NarritiveQA, etc.
|
270 |
-
Thanks the open-sourced libraries like [Tevatron](https://github.com/texttron/tevatron), [Pyserini](https://github.com/castorini/pyserini).
|
271 |
|
272 |
|
273 |
|
@@ -284,4 +290,4 @@ If you find this repository useful, please consider giving a star :star: and cit
|
|
284 |
archivePrefix={arXiv},
|
285 |
primaryClass={cs.CL}
|
286 |
}
|
287 |
-
```
|
|
|
16 |
- Multi-Linguality: It can support more than 100 working languages.
|
17 |
- Multi-Granularity: It is able to process inputs of different granularities, spanning from short sentences to long documents of up to 8192 tokens.
|
18 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
19 |
|
20 |
## News:
|
21 |
+
- 2024/3/8: **Thanks for the [experimental results](https://towardsdatascience.com/openai-vs-open-source-multilingual-embedding-models-e5ccb7c90f05) from @[Yannael](https://huggingface.co/Yannael). In this benchmark, BGE-M3 achieves top performance in both English and other languages, surpassing models such as OpenAI.**
|
22 |
- 2024/3/2: Release unified fine-tuning [example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/unified_finetune) and [data](https://huggingface.co/datasets/Shitao/bge-m3-data)
|
23 |
- 2024/2/6: We release the [MLDR](https://huggingface.co/datasets/Shitao/MLDR) (a long document retrieval dataset covering 13 languages) and [evaluation pipeline](https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB/MLDR).
|
24 |
- 2024/2/1: **Thanks for the excellent tool from Vespa.** You can easily use multiple modes of BGE-M3 following this [notebook](https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/mother-of-all-embedding-models-cloud.ipynb)
|
|
|
87 |
|
88 |
|
89 |
|
90 |
+
**5. Some suggestions for retrieval pipeline in RAG**
|
91 |
+
We recommend to use following pipeline: hybrid retrieval + re-ranking.
|
92 |
+
- Hybrid retrieval leverages the strengths of various methods, offering higher accuracy and stronger generalization capabilities.
|
93 |
+
A classic example: using both embedding retrieval and the BM25 algorithm.
|
94 |
+
Now, you can try to use BGE-M3, which supports both embedding and sparse retrieval.
|
95 |
+
This allows you to obtain token weights (similar to the BM25) without any additional cost when generate dense embeddings.
|
96 |
+
- As cross-encoder models, re-ranker demonstrates higher accuracy than bi-encoder embedding model.
|
97 |
+
Utilizing the re-ranking model (e.g., [bge-reranker](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/reranker), [cohere-reranker](https://txt.cohere.com/rerank/)) after retrieval can further filter the selected text.
|
98 |
+
|
99 |
|
100 |
|
101 |
## Usage
|
|
|
218 |
|
219 |
## Evaluation
|
220 |
|
221 |
+
### Benchmarks from the open-source community
|
222 |
+
![avatar](./imgs/others.webp)
|
223 |
+
The BGE-M3 model emerged as the top performer on this benchmark (OAI is short for OpenAI).
|
224 |
+
For more details, please refer to the [article](https://towardsdatascience.com/openai-vs-open-source-multilingual-embedding-models-e5ccb7c90f05) and [Github Repo](https://github.com/Yannael/multilingual-embeddings)
|
225 |
|
|
|
226 |
|
227 |
+
### Our results
|
228 |
- Multilingual (Miracl dataset)
|
229 |
|
230 |
![avatar](./imgs/miracl.jpg)
|
|
|
257 |
![avatar](./imgs/bm25.jpg)
|
258 |
|
259 |
|
260 |
+
|
261 |
## Training
|
262 |
- Self-knowledge Distillation: combining multiple outputs from different
|
263 |
retrieval modes as reward signal to enhance the performance of single mode(especially for sparse retrieval and multi-vec(colbert) retrival)
|
|
|
272 |
|
273 |
## Acknowledgement
|
274 |
|
275 |
+
Thanks to the authors of open-sourced datasets, including Miracl, MKQA, NarritiveQA, etc.
|
276 |
+
Thanks to the open-sourced libraries like [Tevatron](https://github.com/texttron/tevatron), [Pyserini](https://github.com/castorini/pyserini).
|
277 |
|
278 |
|
279 |
|
|
|
290 |
archivePrefix={arXiv},
|
291 |
primaryClass={cs.CL}
|
292 |
}
|
293 |
+
```
|