Update README.md
Browse files
README.md
CHANGED
@@ -2611,22 +2611,21 @@ pipeline_tag: sentence-similarity
|
|
2611 |
<a href=#usage>Usage</a> |
|
2612 |
<a href="#evaluation">Evaluation</a> |
|
2613 |
<a href="#train">Train</a> |
|
2614 |
-
<a href="#contact">Contact</a> |
|
2615 |
<a href="#license">License</a>
|
2616 |
<p>
|
2617 |
</h4>
|
2618 |
|
2619 |
-
|
2620 |
|
2621 |
-
[English](README.md) | [中文](README_zh.md)
|
2622 |
|
2623 |
FlagEmbedding can map any text to a low-dimensional dense vector which can be used for tasks like retrieval, classification, clustering, or semantic search.
|
2624 |
-
And it also can be used in vector
|
2625 |
|
2626 |
************* 🌟**Updates**🌟 *************
|
2627 |
- 08/05/2023: Release base-scale and small-scale models, **best performance among the models of the same size 🤗**
|
2628 |
-
- 08/02/2023: Release `bge-large-*`(short for BAAI General Embedding) Models, **rank 1st on MTEB and C-MTEB benchmark!**
|
2629 |
-
- 08/01/2023: We release the Chinese Massive Text Embedding Benchmark (**C-MTEB**), consisting of 31 test dataset.
|
2630 |
|
2631 |
|
2632 |
## Model List
|
@@ -2635,12 +2634,12 @@ And it also can be used in vector database for LLMs.
|
|
2635 |
|
2636 |
| Model | Language | Description | query instruction for retrieval |
|
2637 |
|:-------------------------------|:--------:| :--------:| :--------:|
|
2638 |
-
| [BAAI/bge-large-en](https://huggingface.co/BAAI/bge-large-en) | English |
|
2639 |
-
| [BAAI/bge-base-en](https://huggingface.co/BAAI/bge-base-en) | English |
|
2640 |
| [BAAI/bge-small-en](https://huggingface.co/BAAI/bge-small-en) | English | a small-scale model but with competitive performance | `Represent this sentence for searching relevant passages: ` |
|
2641 |
-
| [BAAI/bge-large-zh](https://huggingface.co/BAAI/bge-large-zh) | Chinese |
|
2642 |
-
| [BAAI/bge-large-zh-noinstruct](https://huggingface.co/BAAI/bge-large-zh-noinstruct) | Chinese | This model is trained without instruction, and
|
2643 |
-
| [BAAI/bge-base-zh](https://huggingface.co/BAAI/bge-base-zh) | Chinese | a base-scale model but with
|
2644 |
| [BAAI/bge-small-zh](https://huggingface.co/BAAI/bge-small-zh) | Chinese | a small-scale model but with competitive performance | `为这个句子生成表示以用于检索相关文章:` |
|
2645 |
|
2646 |
|
@@ -2649,10 +2648,12 @@ And it also can be used in vector database for LLMs.
|
|
2649 |
|
2650 |
* **Using FlagEmbedding**
|
2651 |
```
|
2652 |
-
pip install
|
2653 |
```
|
|
|
|
|
2654 |
```python
|
2655 |
-
from
|
2656 |
sentences = ["样例数据-1", "样例数据-2"]
|
2657 |
model = FlagModel('BAAI/bge-large-zh', query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章:")
|
2658 |
embeddings = model.encode(sentences)
|
@@ -2686,7 +2687,7 @@ embeddings = model.encode(sentences, normalize_embeddings=True)
|
|
2686 |
print(embeddings)
|
2687 |
```
|
2688 |
For retrieval task,
|
2689 |
-
each query should start with
|
2690 |
```python
|
2691 |
from sentence_transformers import SentenceTransformer
|
2692 |
queries = ["手机开不了机怎么办?"]
|
@@ -2715,7 +2716,7 @@ model = AutoModel.from_pretrained('BAAI/bge-large-zh')
|
|
2715 |
|
2716 |
# Tokenize sentences
|
2717 |
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
|
2718 |
-
# for retrieval task, add
|
2719 |
# encoded_input = tokenizer([instruction + q for q in queries], padding=True, truncation=True, return_tensors='pt')
|
2720 |
|
2721 |
# Compute token embeddings
|
@@ -2731,7 +2732,7 @@ print("Sentence embeddings:", sentence_embeddings)
|
|
2731 |
|
2732 |
## Evaluation
|
2733 |
`baai-general-embedding` models achieve **state-of-the-art performance on both MTEB and C-MTEB leaderboard!**
|
2734 |
-
More details and evaluation
|
2735 |
|
2736 |
- **MTEB**:
|
2737 |
|
@@ -2760,7 +2761,7 @@ More details and evaluation scripts see [benchemark](benchmark/README.md).
|
|
2760 |
|
2761 |
- **C-MTEB**:
|
2762 |
We create a benchmark C-MTEB for Chinese text embedding which consists of 31 datasets from 6 tasks.
|
2763 |
-
Please refer to [
|
2764 |
|
2765 |
| Model | Embedding dimension | Avg | Retrieval | STS | PairClassification | Classification | Reranking | Clustering |
|
2766 |
|:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
|
@@ -2777,18 +2778,17 @@ Please refer to [benchemark](benchmark/README.md) for a detailed introduction.
|
|
2777 |
|
2778 |
|
2779 |
|
2780 |
-
|
2781 |
## Train
|
2782 |
This section will introduce the way we used to train the general embedding.
|
2783 |
-
The training scripts are in [
|
2784 |
-
and we provide some examples to do [pre-train](https://github.com/FlagOpen/FlagEmbedding/
|
2785 |
|
2786 |
|
2787 |
**1. RetroMAE Pre-train**
|
2788 |
We pre-train the model following the method [retromae](https://github.com/staoxiao/RetroMAE),
|
2789 |
which shows promising improvement in retrieval task ([paper](https://aclanthology.org/2022.emnlp-main.35.pdf)).
|
2790 |
The pre-training was conducted on 24 A100(40G) GPUs with a batch size of 720.
|
2791 |
-
In retromae, the mask ratio of
|
2792 |
We used the AdamW optimizer and the learning rate is 2e-5.
|
2793 |
|
2794 |
**Pre-training data**:
|
@@ -2812,26 +2812,25 @@ We trained our model on 48 A100(40G) GPUs with a large batch size of 32,768 (so
|
|
2812 |
We used the AdamW optimizer and the learning rate is 1e-5.
|
2813 |
The temperature for contrastive loss is 0.01.
|
2814 |
|
2815 |
-
For the version with `*-instrcution`, we add instruction to the query for
|
2816 |
-
For
|
2817 |
-
For
|
2818 |
In the evaluation, the instruction should be added for sentence to passages retrieval task, not be added for other tasks.
|
2819 |
|
2820 |
|
2821 |
-
The finetune script is accessible in this repository: [
|
2822 |
You can easily finetune your model with it.
|
2823 |
|
2824 |
**Training data**:
|
2825 |
|
2826 |
- For English, we collect 230M text pairs from [wikipedia](https://huggingface.co/datasets/wikipedia), [cc-net](https://github.com/facebookresearch/cc_net), and so on.
|
2827 |
|
2828 |
-
- For
|
2829 |
|
2830 |
**The data collection is to be released in the future.**
|
2831 |
|
2832 |
We will continually update the embedding models and training codes,
|
2833 |
hoping to promote the development of the embedding model community.
|
2834 |
|
2835 |
-
|
2836 |
## License
|
2837 |
-
FlagEmbedding is licensed under [MIT License](). The released models can be used for commercial purposes free of charge.
|
|
|
2611 |
<a href=#usage>Usage</a> |
|
2612 |
<a href="#evaluation">Evaluation</a> |
|
2613 |
<a href="#train">Train</a> |
|
|
|
2614 |
<a href="#license">License</a>
|
2615 |
<p>
|
2616 |
</h4>
|
2617 |
|
2618 |
+
For more details please refer to our GitHub repo: [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding).
|
2619 |
|
2620 |
+
[English](README.md) | [中文](https://github.com/FlagOpen/FlagEmbedding/blob/master/README_zh.md)
|
2621 |
|
2622 |
FlagEmbedding can map any text to a low-dimensional dense vector which can be used for tasks like retrieval, classification, clustering, or semantic search.
|
2623 |
+
And it also can be used in vector databases for LLMs.
|
2624 |
|
2625 |
************* 🌟**Updates**🌟 *************
|
2626 |
- 08/05/2023: Release base-scale and small-scale models, **best performance among the models of the same size 🤗**
|
2627 |
+
- 08/02/2023: Release `bge-large-*`(short for BAAI General Embedding) Models, **rank 1st on MTEB and C-MTEB benchmark!** :tada: :tada:
|
2628 |
+
- 08/01/2023: We release the [Chinese Massive Text Embedding Benchmark](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB) (**C-MTEB**), consisting of 31 test dataset.
|
2629 |
|
2630 |
|
2631 |
## Model List
|
|
|
2634 |
|
2635 |
| Model | Language | Description | query instruction for retrieval |
|
2636 |
|:-------------------------------|:--------:| :--------:| :--------:|
|
2637 |
+
| [BAAI/bge-large-en](https://huggingface.co/BAAI/bge-large-en) | English | :trophy: rank **1st** in [MTEB](https://huggingface.co/spaces/mteb/leaderboard) leaderboard | `Represent this sentence for searching relevant passages: ` |
|
2638 |
+
| [BAAI/bge-base-en](https://huggingface.co/BAAI/bge-base-en) | English | rank **2nd** in [MTEB](https://huggingface.co/spaces/mteb/leaderboard) leaderboard | `Represent this sentence for searching relevant passages: ` |
|
2639 |
| [BAAI/bge-small-en](https://huggingface.co/BAAI/bge-small-en) | English | a small-scale model but with competitive performance | `Represent this sentence for searching relevant passages: ` |
|
2640 |
+
| [BAAI/bge-large-zh](https://huggingface.co/BAAI/bge-large-zh) | Chinese | :trophy: rank **1st** in [C-MTEB](https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB) benchmark | `为这个句子生成表示以用于检索相关文章:` |
|
2641 |
+
| [BAAI/bge-large-zh-noinstruct](https://huggingface.co/BAAI/bge-large-zh-noinstruct) | Chinese | This model is trained without instruction, and rank **2nd** in [C-MTEB](https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB) benchmark | |
|
2642 |
+
| [BAAI/bge-base-zh](https://huggingface.co/BAAI/bge-base-zh) | Chinese | a base-scale model but has similar ability with `bge-large-zh` | `为这个句子生成表示以用于检索相关文章:` |
|
2643 |
| [BAAI/bge-small-zh](https://huggingface.co/BAAI/bge-small-zh) | Chinese | a small-scale model but with competitive performance | `为这个句子生成表示以用于检索相关文章:` |
|
2644 |
|
2645 |
|
|
|
2648 |
|
2649 |
* **Using FlagEmbedding**
|
2650 |
```
|
2651 |
+
pip install FlagEmbedding
|
2652 |
```
|
2653 |
+
See [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/baai_general_embedding/README.md) for more methods to install FlagEmbedding.
|
2654 |
+
|
2655 |
```python
|
2656 |
+
from FlagEmbedding import FlagModel
|
2657 |
sentences = ["样例数据-1", "样例数据-2"]
|
2658 |
model = FlagModel('BAAI/bge-large-zh', query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章:")
|
2659 |
embeddings = model.encode(sentences)
|
|
|
2687 |
print(embeddings)
|
2688 |
```
|
2689 |
For retrieval task,
|
2690 |
+
each query should start with an instruction (instructions see [Model List](https://github.com/FlagOpen/FlagEmbedding/tree/master#model-list)).
|
2691 |
```python
|
2692 |
from sentence_transformers import SentenceTransformer
|
2693 |
queries = ["手机开不了机怎么办?"]
|
|
|
2716 |
|
2717 |
# Tokenize sentences
|
2718 |
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
|
2719 |
+
# for retrieval task, add an instruction to query
|
2720 |
# encoded_input = tokenizer([instruction + q for q in queries], padding=True, truncation=True, return_tensors='pt')
|
2721 |
|
2722 |
# Compute token embeddings
|
|
|
2732 |
|
2733 |
## Evaluation
|
2734 |
`baai-general-embedding` models achieve **state-of-the-art performance on both MTEB and C-MTEB leaderboard!**
|
2735 |
+
More details and evaluation tools see our [scripts](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/README.md).
|
2736 |
|
2737 |
- **MTEB**:
|
2738 |
|
|
|
2761 |
|
2762 |
- **C-MTEB**:
|
2763 |
We create a benchmark C-MTEB for Chinese text embedding which consists of 31 datasets from 6 tasks.
|
2764 |
+
Please refer to [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/README.md) for a detailed introduction.
|
2765 |
|
2766 |
| Model | Embedding dimension | Avg | Retrieval | STS | PairClassification | Classification | Reranking | Clustering |
|
2767 |
|:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
|
|
|
2778 |
|
2779 |
|
2780 |
|
|
|
2781 |
## Train
|
2782 |
This section will introduce the way we used to train the general embedding.
|
2783 |
+
The training scripts are in [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/baai_general_embedding/README.md),
|
2784 |
+
and we provide some examples to do [pre-train](https://github.com/FlagOpen/FlagEmbedding/blob/master/examples/pretrain/README.md) and [fine-tune](https://github.com/FlagOpen/FlagEmbedding/blob/master/examples/finetune/README.md).
|
2785 |
|
2786 |
|
2787 |
**1. RetroMAE Pre-train**
|
2788 |
We pre-train the model following the method [retromae](https://github.com/staoxiao/RetroMAE),
|
2789 |
which shows promising improvement in retrieval task ([paper](https://aclanthology.org/2022.emnlp-main.35.pdf)).
|
2790 |
The pre-training was conducted on 24 A100(40G) GPUs with a batch size of 720.
|
2791 |
+
In retromae, the mask ratio of encoder and decoder are 0.3, and 0.5 respectively.
|
2792 |
We used the AdamW optimizer and the learning rate is 2e-5.
|
2793 |
|
2794 |
**Pre-training data**:
|
|
|
2812 |
We used the AdamW optimizer and the learning rate is 1e-5.
|
2813 |
The temperature for contrastive loss is 0.01.
|
2814 |
|
2815 |
+
For the version with `*-instrcution`, we add instruction to the query for retrieval task in the training.
|
2816 |
+
For english, the instruction is `Represent this sentence for searching relevant passages: `;
|
2817 |
+
For chinese, the instruction is `为这个句子生成表示以用于检索相关文章:`.
|
2818 |
In the evaluation, the instruction should be added for sentence to passages retrieval task, not be added for other tasks.
|
2819 |
|
2820 |
|
2821 |
+
The finetune script is accessible in this repository: [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/baai_general_embedding/README.md).
|
2822 |
You can easily finetune your model with it.
|
2823 |
|
2824 |
**Training data**:
|
2825 |
|
2826 |
- For English, we collect 230M text pairs from [wikipedia](https://huggingface.co/datasets/wikipedia), [cc-net](https://github.com/facebookresearch/cc_net), and so on.
|
2827 |
|
2828 |
+
- For chinese, we collect 120M text pairs from [wudao](https://github.com/BAAI-WuDao/Data), [simclue](https://github.com/CLUEbenchmark/SimCLUE) and so on.
|
2829 |
|
2830 |
**The data collection is to be released in the future.**
|
2831 |
|
2832 |
We will continually update the embedding models and training codes,
|
2833 |
hoping to promote the development of the embedding model community.
|
2834 |
|
|
|
2835 |
## License
|
2836 |
+
FlagEmbedding is licensed under [MIT License](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE). The released models can be used for commercial purposes free of charge.
|