update
Browse files
README.md
CHANGED
@@ -35,25 +35,36 @@ FlagEmbedding can map any text to a low-dimensional dense vector which can be us
|
|
35 |
And it also can be used in vector databases for LLMs.
|
36 |
|
37 |
************* 🌟**Updates**🌟 *************
|
38 |
-
-
|
39 |
-
- 09/
|
|
|
|
|
40 |
- **New reranker model**: release cross-encoder models `BAAI/bge-reranker-base` and `BAAI/bge-reranker-large`, which are more powerful than embedding model. We recommend to use/fine-tune them to re-rank top-k documents returned by embedding models.
|
41 |
- **update embedding model**: release `bge-*-v1.5` embedding model to alleviate the issue of the similarity distribution, and enhance its retrieval ability without instruction.
|
42 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
43 |
- 08/09/2023: BGE Models are integrated into **Langchain**, you can use it like [this](#using-langchain); C-MTEB **leaderboard** is [available](https://huggingface.co/spaces/mteb/leaderboard).
|
44 |
-
- 08/05/2023: Release base-scale and small-scale models, **best performance among the models of the same size 🤗**
|
45 |
-
- 08/02/2023: Release `bge-large-*`(short for BAAI General Embedding) Models, **rank 1st on MTEB and C-MTEB benchmark!** :tada: :tada:
|
46 |
-
- 08/01/2023: We release the [Chinese Massive Text Embedding Benchmark](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB) (**C-MTEB**), consisting of 31 test dataset.
|
|
|
|
|
47 |
|
48 |
|
49 |
## Model List
|
50 |
|
51 |
`bge` is short for `BAAI general embedding`.
|
52 |
|
53 |
-
| Model | Language | | Description | query instruction for retrieval
|
54 |
|:-------------------------------|:--------:| :--------:| :--------:|:--------:|
|
55 |
-
| [BAAI/
|
56 |
-
| [BAAI/bge-reranker-
|
|
|
57 |
| [BAAI/bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) | English | [Inference](#usage-for-embedding-model) [Fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) | version 1.5 with more reasonable similarity distribution | `Represent this sentence for searching relevant passages: ` |
|
58 |
| [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) | English | [Inference](#usage-for-embedding-model) [Fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) | version 1.5 with more reasonable similarity distribution | `Represent this sentence for searching relevant passages: ` |
|
59 |
| [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) | English | [Inference](#usage-for-embedding-model) [Fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) | version 1.5 with more reasonable similarity distribution | `Represent this sentence for searching relevant passages: ` |
|
@@ -68,11 +79,15 @@ And it also can be used in vector databases for LLMs.
|
|
68 |
| [BAAI/bge-small-zh](https://huggingface.co/BAAI/bge-small-zh) | Chinese | [Inference](#usage-for-embedding-model) [Fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) | a small-scale model but with competitive performance | `为这个句子生成表示以用于检索相关文章:` |
|
69 |
|
70 |
|
71 |
-
|
72 |
|
73 |
-
|
74 |
For examples, use bge embedding model to retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 document to get the final top-3 results.
|
75 |
|
|
|
|
|
|
|
|
|
76 |
## Frequently asked questions
|
77 |
|
78 |
<details>
|
@@ -109,7 +124,11 @@ please select an appropriate similarity threshold based on the similarity distri
|
|
109 |
<summary>3. When does the query instruction need to be used</summary>
|
110 |
|
111 |
<!-- ### When does the query instruction need to be used -->
|
112 |
-
|
|
|
|
|
|
|
|
|
113 |
For a retrieval task that uses short queries to find long related documents,
|
114 |
it is recommended to add instructions for these short queries.
|
115 |
**The best method to decide whether to add instructions for queries is choosing the setting that achieves better performance on your task.**
|
@@ -369,7 +388,7 @@ which is more accurate than embedding model (i.e., bi-encoder) but more time-con
|
|
369 |
Therefore, it can be used to re-rank the top-k documents returned by embedding model.
|
370 |
We train the cross-encoder on a multilingual pair data,
|
371 |
The data format is the same as embedding model, so you can fine-tune it easily following our [example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/reranker).
|
372 |
-
More details
|
373 |
|
374 |
|
375 |
## Contact
|
@@ -379,7 +398,8 @@ You also can email Shitao Xiao(stxiao@baai.ac.cn) and Zheng Liu(liuzheng@baai.ac
|
|
379 |
|
380 |
## Citation
|
381 |
|
382 |
-
If you find
|
|
|
383 |
```
|
384 |
@misc{bge_embedding,
|
385 |
title={C-Pack: Packaged Resources To Advance General Chinese Embedding},
|
@@ -394,5 +414,3 @@ If you find our work helpful, please cite us:
|
|
394 |
## License
|
395 |
FlagEmbedding is licensed under the [MIT License](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE). The released models can be used for commercial purposes free of charge.
|
396 |
|
397 |
-
|
398 |
-
|
|
|
35 |
And it also can be used in vector databases for LLMs.
|
36 |
|
37 |
************* 🌟**Updates**🌟 *************
|
38 |
+
- 10/12/2023: Release [LLM-Embedder](./FlagEmbedding/llm_embedder/README.md), a unified embedding model to support diverse retrieval augmentation needs for LLMs. [Paper](https://arxiv.org/pdf/2310.07554.pdf) :fire:
|
39 |
+
- 09/15/2023: The [technical report](https://arxiv.org/pdf/2309.07597.pdf) of BGE has been released
|
40 |
+
- 09/15/2023: The [masive training data](https://data.baai.ac.cn/details/BAAI-MTP) of BGE has been released
|
41 |
+
- 09/12/2023: New models:
|
42 |
- **New reranker model**: release cross-encoder models `BAAI/bge-reranker-base` and `BAAI/bge-reranker-large`, which are more powerful than embedding model. We recommend to use/fine-tune them to re-rank top-k documents returned by embedding models.
|
43 |
- **update embedding model**: release `bge-*-v1.5` embedding model to alleviate the issue of the similarity distribution, and enhance its retrieval ability without instruction.
|
44 |
+
|
45 |
+
|
46 |
+
<details>
|
47 |
+
<summary>More</summary>
|
48 |
+
<!-- ### More -->
|
49 |
+
|
50 |
+
- 09/07/2023: Update [fine-tune code](https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/baai_general_embedding/README.md): Add script to mine hard negatives and support adding instruction during fine-tuning.
|
51 |
- 08/09/2023: BGE Models are integrated into **Langchain**, you can use it like [this](#using-langchain); C-MTEB **leaderboard** is [available](https://huggingface.co/spaces/mteb/leaderboard).
|
52 |
+
- 08/05/2023: Release base-scale and small-scale models, **best performance among the models of the same size 🤗**
|
53 |
+
- 08/02/2023: Release `bge-large-*`(short for BAAI General Embedding) Models, **rank 1st on MTEB and C-MTEB benchmark!** :tada: :tada:
|
54 |
+
- 08/01/2023: We release the [Chinese Massive Text Embedding Benchmark](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB) (**C-MTEB**), consisting of 31 test dataset.
|
55 |
+
|
56 |
+
</details>
|
57 |
|
58 |
|
59 |
## Model List
|
60 |
|
61 |
`bge` is short for `BAAI general embedding`.
|
62 |
|
63 |
+
| Model | Language | | Description | query instruction for retrieval [1] |
|
64 |
|:-------------------------------|:--------:| :--------:| :--------:|:--------:|
|
65 |
+
| [BAAI/llm-embedder](https://huggingface.co/BAAI/llm-embedder) | English | [Inference](./FlagEmbedding/llm_embedder/README.md) [Fine-tune](./FlagEmbedding/llm_embedder/README.md) | a unified embedding model to support diverse retrieval augmentation needs for LLMs | See [README](./FlagEmbedding/llm_embedder/README.md) |
|
66 |
+
| [BAAI/bge-reranker-large](https://huggingface.co/BAAI/bge-reranker-large) | Chinese and English | [Inference](#usage-for-reranker) [Fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/reranker) | a cross-encoder model which is more accurate but less efficient [2] | |
|
67 |
+
| [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) | Chinese and English | [Inference](#usage-for-reranker) [Fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/reranker) | a cross-encoder model which is more accurate but less efficient [2] | |
|
68 |
| [BAAI/bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) | English | [Inference](#usage-for-embedding-model) [Fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) | version 1.5 with more reasonable similarity distribution | `Represent this sentence for searching relevant passages: ` |
|
69 |
| [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) | English | [Inference](#usage-for-embedding-model) [Fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) | version 1.5 with more reasonable similarity distribution | `Represent this sentence for searching relevant passages: ` |
|
70 |
| [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) | English | [Inference](#usage-for-embedding-model) [Fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) | version 1.5 with more reasonable similarity distribution | `Represent this sentence for searching relevant passages: ` |
|
|
|
79 |
| [BAAI/bge-small-zh](https://huggingface.co/BAAI/bge-small-zh) | Chinese | [Inference](#usage-for-embedding-model) [Fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) | a small-scale model but with competitive performance | `为这个句子生成表示以用于检索相关文章:` |
|
80 |
|
81 |
|
82 |
+
[1\]: If you need to search the relevant passages to a query, we suggest to add the instruction to the query; in other cases, no instruction is needed, just use the original query directly. In all cases, **no instruction** needs to be added to passages.
|
83 |
|
84 |
+
[2\]: Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. To balance the accuracy and time cost, cross-encoder is widely used to re-rank top-k documents retrieved by other simple models.
|
85 |
For examples, use bge embedding model to retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 document to get the final top-3 results.
|
86 |
|
87 |
+
All models have been uploaded to Huggingface Hub, and you can see them at https://huggingface.co/BAAI.
|
88 |
+
If you cannot open the Huggingface Hub, you also can download the models at https://model.baai.ac.cn/models .
|
89 |
+
|
90 |
+
|
91 |
## Frequently asked questions
|
92 |
|
93 |
<details>
|
|
|
124 |
<summary>3. When does the query instruction need to be used</summary>
|
125 |
|
126 |
<!-- ### When does the query instruction need to be used -->
|
127 |
+
|
128 |
+
For the `bge-*-v1.5`, we improve its retrieval ability when not using instruction.
|
129 |
+
No instruction only has a slight degradation in retrieval performance compared with using instruction.
|
130 |
+
So you can generate embedding without instruction in all cases for convenience.
|
131 |
+
|
132 |
For a retrieval task that uses short queries to find long related documents,
|
133 |
it is recommended to add instructions for these short queries.
|
134 |
**The best method to decide whether to add instructions for queries is choosing the setting that achieves better performance on your task.**
|
|
|
388 |
Therefore, it can be used to re-rank the top-k documents returned by embedding model.
|
389 |
We train the cross-encoder on a multilingual pair data,
|
390 |
The data format is the same as embedding model, so you can fine-tune it easily following our [example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/reranker).
|
391 |
+
More details please refer to [./FlagEmbedding/reranker/README.md](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/reranker)
|
392 |
|
393 |
|
394 |
## Contact
|
|
|
398 |
|
399 |
## Citation
|
400 |
|
401 |
+
If you find this repository useful, please consider giving a star :star: and citation
|
402 |
+
|
403 |
```
|
404 |
@misc{bge_embedding,
|
405 |
title={C-Pack: Packaged Resources To Advance General Chinese Embedding},
|
|
|
414 |
## License
|
415 |
FlagEmbedding is licensed under the [MIT License](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE). The released models can be used for commercial purposes free of charge.
|
416 |
|
|
|
|