ldwang commited on
Commit
2f68ed7
1 Parent(s): 1241160

update readme

Browse files
Files changed (1) hide show
  1. README.md +87 -31
README.md CHANGED
@@ -2602,8 +2602,22 @@ language:
2602
  pipeline_tag: sentence-similarity
2603
  ---
2604
 
2605
- <h1 align="center">FlagEmbedding</h1>
2606
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2607
 
2608
  <h4 align="center">
2609
  <p>
@@ -2611,18 +2625,19 @@ pipeline_tag: sentence-similarity
2611
  <a href=#usage>Usage</a> |
2612
  <a href="#evaluation">Evaluation</a> |
2613
  <a href="#train">Train</a> |
 
2614
  <a href="#license">License</a>
2615
  <p>
2616
  </h4>
2617
 
2618
- For more details please refer to our GitHub repo: [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding).
2619
 
2620
  [English](README.md) | [中文](https://github.com/FlagOpen/FlagEmbedding/blob/master/README_zh.md)
2621
 
2622
  FlagEmbedding can map any text to a low-dimensional dense vector which can be used for tasks like retrieval, classification, clustering, or semantic search.
2623
- And it also can be used in vector databases for LLMs.
2624
 
2625
  ************* 🌟**Updates**🌟 *************
 
2626
  - 08/05/2023: Release base-scale and small-scale models, **best performance among the models of the same size 🤗**
2627
  - 08/02/2023: Release `bge-large-*`(short for BAAI General Embedding) Models, **rank 1st on MTEB and C-MTEB benchmark!** :tada: :tada:
2628
  - 08/01/2023: We release the [Chinese Massive Text Embedding Benchmark](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB) (**C-MTEB**), consisting of 31 test dataset.
@@ -2632,7 +2647,7 @@ And it also can be used in vector databases for LLMs.
2632
 
2633
  `bge` is short for `BAAI general embedding`.
2634
 
2635
- | Model | Language | Description | query instruction for retrieval |
2636
  |:-------------------------------|:--------:| :--------:| :--------:|
2637
  | [BAAI/bge-large-en](https://huggingface.co/BAAI/bge-large-en) | English | :trophy: rank **1st** in [MTEB](https://huggingface.co/spaces/mteb/leaderboard) leaderboard | `Represent this sentence for searching relevant passages: ` |
2638
  | [BAAI/bge-base-en](https://huggingface.co/BAAI/bge-base-en) | English | rank **2nd** in [MTEB](https://huggingface.co/spaces/mteb/leaderboard) leaderboard | `Represent this sentence for searching relevant passages: ` |
@@ -2642,27 +2657,32 @@ And it also can be used in vector databases for LLMs.
2642
  | [BAAI/bge-base-zh](https://huggingface.co/BAAI/bge-base-zh) | Chinese | a base-scale model but has similar ability with `bge-large-zh` | `为这个句子生成表示以用于检索相关文章:` |
2643
  | [BAAI/bge-small-zh](https://huggingface.co/BAAI/bge-small-zh) | Chinese | a small-scale model but with competitive performance | `为这个句子生成表示以用于检索相关文章:` |
2644
 
2645
-
2646
 
2647
  ## Usage
2648
 
2649
- * **Using FlagEmbedding**
 
 
 
2650
  ```
2651
  pip install -U FlagEmbedding
2652
  ```
2653
- See [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/baai_general_embedding/README.md) for more methods to install FlagEmbedding.
2654
 
2655
  ```python
2656
  from FlagEmbedding import FlagModel
2657
  sentences = ["样例数据-1", "样例数据-2"]
2658
  model = FlagModel('BAAI/bge-large-zh', query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章:")
2659
- embeddings = model.encode(sentences)
2660
- print(embeddings)
 
 
2661
 
2662
- # for retrieval task, please use encode_queries() which will automatically add the instruction to each query
2663
- # corpus in retrieval task can still use encode() or encode_corpus()
2664
  queries = ['query_1', 'query_2']
2665
- passages = ["样例段落-1", "样例段落-2"]
2666
  q_embeddings = model.encode_queries(queries)
2667
  p_embeddings = model.encode(passages)
2668
  scores = q_embeddings @ p_embeddings.T
@@ -2672,7 +2692,7 @@ The value of argument `query_instruction_for_retrieval` see [Model List](https:/
2672
  FlagModel will use all available GPUs when encoding, please set `os.environ["CUDA_VISIBLE_DEVICES"]` to choose GPU.
2673
 
2674
 
2675
- * **Using Sentence-Transformers**
2676
 
2677
  Using this model also is easy when you have [sentence-transformers](https://www.SBERT.net) installed:
2678
 
@@ -2683,15 +2703,18 @@ pip install -U sentence-transformers
2683
  from sentence_transformers import SentenceTransformer
2684
  sentences = ["样例数据-1", "样例数据-2"]
2685
  model = SentenceTransformer('BAAI/bge-large-zh')
2686
- embeddings = model.encode(sentences, normalize_embeddings=True)
2687
- print(embeddings)
 
 
2688
  ```
2689
- For retrieval task,
2690
- each query should start with an instruction (instructions see [Model List](https://github.com/FlagOpen/FlagEmbedding/tree/master#model-list)).
 
2691
  ```python
2692
  from sentence_transformers import SentenceTransformer
2693
- queries = ["手机开不了机怎么办?"]
2694
- passages = ["样例段落-1", "样例段落-2"]
2695
  instruction = "为这个句子生成表示以用于检索相关文章:"
2696
 
2697
  model = SentenceTransformer('BAAI/bge-large-zh')
@@ -2700,7 +2723,23 @@ p_embeddings = model.encode(passages, normalize_embeddings=True)
2700
  scores = q_embeddings @ p_embeddings.T
2701
  ```
2702
 
2703
- * **Using HuggingFace Transformers**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2704
 
2705
  With transformers package, you can use the model like this: First, you pass your input through the transformer model, then you select the last hidden state of first token (i.e., [CLS]) as the sentence embedding.
2706
 
@@ -2716,7 +2755,7 @@ model = AutoModel.from_pretrained('BAAI/bge-large-zh')
2716
 
2717
  # Tokenize sentences
2718
  encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
2719
- # for retrieval task, add an instruction to query
2720
  # encoded_input = tokenizer([instruction + q for q in queries], padding=True, truncation=True, return_tensors='pt')
2721
 
2722
  # Compute token embeddings
@@ -2760,7 +2799,7 @@ More details and evaluation tools see our [scripts](https://github.com/FlagOpen/
2760
 
2761
 
2762
  - **C-MTEB**:
2763
- We create a benchmark C-MTEB for Chinese text embedding which consists of 31 datasets from 6 tasks.
2764
  Please refer to [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/README.md) for a detailed introduction.
2765
 
2766
  | Model | Embedding dimension | Avg | Retrieval | STS | PairClassification | Classification | Reranking | Clustering |
@@ -2788,7 +2827,7 @@ and we provide some examples to do [pre-train](https://github.com/FlagOpen/FlagE
2788
  We pre-train the model following the method [retromae](https://github.com/staoxiao/RetroMAE),
2789
  which shows promising improvement in retrieval task ([paper](https://aclanthology.org/2022.emnlp-main.35.pdf)).
2790
  The pre-training was conducted on 24 A100(40G) GPUs with a batch size of 720.
2791
- In retromae, the mask ratio of encoder and decoder are 0.3, and 0.5 respectively.
2792
  We used the AdamW optimizer and the learning rate is 2e-5.
2793
 
2794
  **Pre-training data**:
@@ -2797,8 +2836,7 @@ We used the AdamW optimizer and the learning rate is 2e-5.
2797
  - [wikipedia](https://huggingface.co/datasets/wikipedia)
2798
  - [msmarco](https://huggingface.co/datasets/Tevatron/msmarco-passage-corpus)
2799
  - Chinese:
2800
- - Subset of [wudao](https://github.com/BAAI-WuDao/Data)
2801
- - [baidu-baike](https://baike.baidu.com/)
2802
 
2803
 
2804
  **2. Finetune**
@@ -2812,11 +2850,11 @@ We trained our model on 48 A100(40G) GPUs with a large batch size of 32,768 (so
2812
  We used the AdamW optimizer and the learning rate is 1e-5.
2813
  The temperature for contrastive loss is 0.01.
2814
 
2815
- For the version with `*-instrcution`, we add instruction to the query for retrieval task in the training.
2816
- For english, the instruction is `Represent this sentence for searching relevant passages: `;
2817
- For chinese, the instruction is `为这个句子生成表示以用于检索相关文章:`.
2818
- In the evaluation, the instruction should be added for sentence to passages retrieval task, not be added for other tasks.
2819
-
2820
 
2821
  The finetune script is accessible in this repository: [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/baai_general_embedding/README.md).
2822
  You can easily finetune your model with it.
@@ -2829,8 +2867,26 @@ You can easily finetune your model with it.
2829
 
2830
  **The data collection is to be released in the future.**
2831
 
 
 
 
 
 
 
 
 
 
2832
  We will continually update the embedding models and training codes,
2833
  hoping to promote the development of the embedding model community.
2834
 
 
 
 
 
 
 
2835
  ## License
2836
- FlagEmbedding is licensed under [MIT License](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE). The released models can be used for commercial purposes free of charge.
 
 
 
 
2602
  pipeline_tag: sentence-similarity
2603
  ---
2604
 
 
2605
 
2606
+ <h1 align="center">FlagEmbedding</h1>
2607
+ <p align="center">
2608
+ <a href="https://www.python.org/">
2609
+ <img alt="Build" src="https://img.shields.io/badge/Contribution-Welcome-blue">
2610
+ </a>
2611
+ <a href="https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE">
2612
+ <img alt="License" src="https://img.shields.io/badge/LICENSE-MIT-green">
2613
+ </a>
2614
+ <a href="https://huggingface.co/C-MTEB">
2615
+ <img alt="Build" src="https://img.shields.io/badge/C_MTEB-🤗-yellow">
2616
+ </a>
2617
+ <a href="https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding">
2618
+ <img alt="Build" src="https://img.shields.io/badge/FlagEmbedding-1.0-red">
2619
+ </a>
2620
+ </p>
2621
 
2622
  <h4 align="center">
2623
  <p>
 
2625
  <a href=#usage>Usage</a> |
2626
  <a href="#evaluation">Evaluation</a> |
2627
  <a href="#train">Train</a> |
2628
+ <a href="#contact">Contact</a> |
2629
  <a href="#license">License</a>
2630
  <p>
2631
  </h4>
2632
 
 
2633
 
2634
  [English](README.md) | [中文](https://github.com/FlagOpen/FlagEmbedding/blob/master/README_zh.md)
2635
 
2636
  FlagEmbedding can map any text to a low-dimensional dense vector which can be used for tasks like retrieval, classification, clustering, or semantic search.
2637
+ And it also can be used in vector database for LLMs.
2638
 
2639
  ************* 🌟**Updates**🌟 *************
2640
+ - 08/09/2023: BGE Models are integrated into Langchain, you can use it like [this](#using-langchain); C-MTEB leaderboard is [avaliable](https://huggingface.co/spaces/mteb/leaderboard).
2641
  - 08/05/2023: Release base-scale and small-scale models, **best performance among the models of the same size 🤗**
2642
  - 08/02/2023: Release `bge-large-*`(short for BAAI General Embedding) Models, **rank 1st on MTEB and C-MTEB benchmark!** :tada: :tada:
2643
  - 08/01/2023: We release the [Chinese Massive Text Embedding Benchmark](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB) (**C-MTEB**), consisting of 31 test dataset.
 
2647
 
2648
  `bge` is short for `BAAI general embedding`.
2649
 
2650
+ | Model | Language | Description | query instruction for retrieval\* |
2651
  |:-------------------------------|:--------:| :--------:| :--------:|
2652
  | [BAAI/bge-large-en](https://huggingface.co/BAAI/bge-large-en) | English | :trophy: rank **1st** in [MTEB](https://huggingface.co/spaces/mteb/leaderboard) leaderboard | `Represent this sentence for searching relevant passages: ` |
2653
  | [BAAI/bge-base-en](https://huggingface.co/BAAI/bge-base-en) | English | rank **2nd** in [MTEB](https://huggingface.co/spaces/mteb/leaderboard) leaderboard | `Represent this sentence for searching relevant passages: ` |
 
2657
  | [BAAI/bge-base-zh](https://huggingface.co/BAAI/bge-base-zh) | Chinese | a base-scale model but has similar ability with `bge-large-zh` | `为这个句子生成表示以用于检索相关文章:` |
2658
  | [BAAI/bge-small-zh](https://huggingface.co/BAAI/bge-small-zh) | Chinese | a small-scale model but with competitive performance | `为这个句子生成表示以用于检索相关文章:` |
2659
 
2660
+ \*: If you need to search the **long** relevant passages to a **short** query (s2p retrieval task), you need to add the instruction to the query; in other cases, no instruction is needed, just use the original query directly. In all cases, **no instruction** need to be added to passages.
2661
 
2662
  ## Usage
2663
 
2664
+ Here are some examples to use `bge` models with
2665
+ [FlagEmbedding](#using-flagembedding), [Sentence-Transformers](#using-sentence-transformers), [Langchain](#using-langchain), or [Huggingface Transformers](#using-huggingface-transformers).
2666
+
2667
+ #### Using FlagEmbedding
2668
  ```
2669
  pip install -U FlagEmbedding
2670
  ```
2671
+ If it doesn't work for you, you can see [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/baai_general_embedding/README.md) for more methods to install FlagEmbedding.
2672
 
2673
  ```python
2674
  from FlagEmbedding import FlagModel
2675
  sentences = ["样例数据-1", "样例数据-2"]
2676
  model = FlagModel('BAAI/bge-large-zh', query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章:")
2677
+ embeddings_1 = model.encode(sentences)
2678
+ embeddings_2 = model.encode(sentences)
2679
+ similarity = embeddings_1 @ embeddings_2.T
2680
+ print(similarity)
2681
 
2682
+ # for s2p(short query to long passage) retrieval task, please use encode_queries() which will automatically add the instruction to each query
2683
+ # corpus in retrieval task can still use encode() or encode_corpus(), since they don't need instruction
2684
  queries = ['query_1', 'query_2']
2685
+ passages = ["样例文档-1", "样例文档-2"]
2686
  q_embeddings = model.encode_queries(queries)
2687
  p_embeddings = model.encode(passages)
2688
  scores = q_embeddings @ p_embeddings.T
 
2692
  FlagModel will use all available GPUs when encoding, please set `os.environ["CUDA_VISIBLE_DEVICES"]` to choose GPU.
2693
 
2694
 
2695
+ #### Using Sentence-Transformers
2696
 
2697
  Using this model also is easy when you have [sentence-transformers](https://www.SBERT.net) installed:
2698
 
 
2703
  from sentence_transformers import SentenceTransformer
2704
  sentences = ["样例数据-1", "样例数据-2"]
2705
  model = SentenceTransformer('BAAI/bge-large-zh')
2706
+ embeddings_1 = model.encode(sentences, normalize_embeddings=True)
2707
+ embeddings_2 = model.encode(sentences, normalize_embeddings=True)
2708
+ similarity = embeddings_1 @ embeddings_2.T
2709
+ print(similarity)
2710
  ```
2711
+ For s2p(short query to long passage) retrieval task,
2712
+ each short query should start with an instruction (instructions see [Model List](https://github.com/FlagOpen/FlagEmbedding/tree/master#model-list)).
2713
+ But the instruction is not needed for passages.
2714
  ```python
2715
  from sentence_transformers import SentenceTransformer
2716
+ queries = ['query_1', 'query_2']
2717
+ passages = ["样例文档-1", "样例文档-2"]
2718
  instruction = "为这个句子生成表示以用于检索相关文章:"
2719
 
2720
  model = SentenceTransformer('BAAI/bge-large-zh')
 
2723
  scores = q_embeddings @ p_embeddings.T
2724
  ```
2725
 
2726
+ #### Using Langchain
2727
+
2728
+ You can use `bge` in langchain like this:
2729
+ ```python
2730
+ from langchain.embeddings import HuggingFaceBgeEmbeddings
2731
+ model_name = "BAAI/bge-small-en"
2732
+ model_kwargs = {'device': 'cuda'}
2733
+ encode_kwargs = {'normalize_embeddings': True} # set True to compute cosine similarity
2734
+ model_norm = HuggingFaceBgeEmbeddings(
2735
+ model_name=model_name,
2736
+ model_kwargs=model_kwargs,
2737
+ encode_kwargs=encode_kwargs
2738
+ )
2739
+ ```
2740
+
2741
+
2742
+ #### Using HuggingFace Transformers
2743
 
2744
  With transformers package, you can use the model like this: First, you pass your input through the transformer model, then you select the last hidden state of first token (i.e., [CLS]) as the sentence embedding.
2745
 
 
2755
 
2756
  # Tokenize sentences
2757
  encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
2758
+ # for s2p(short query to long passage) retrieval task, add an instruction to query (not add instruction for passages)
2759
  # encoded_input = tokenizer([instruction + q for q in queries], padding=True, truncation=True, return_tensors='pt')
2760
 
2761
  # Compute token embeddings
 
2799
 
2800
 
2801
  - **C-MTEB**:
2802
+ We create a benchmark C-MTEB for chinese text embedding which consists of 31 datasets from 6 tasks.
2803
  Please refer to [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/README.md) for a detailed introduction.
2804
 
2805
  | Model | Embedding dimension | Avg | Retrieval | STS | PairClassification | Classification | Reranking | Clustering |
 
2827
  We pre-train the model following the method [retromae](https://github.com/staoxiao/RetroMAE),
2828
  which shows promising improvement in retrieval task ([paper](https://aclanthology.org/2022.emnlp-main.35.pdf)).
2829
  The pre-training was conducted on 24 A100(40G) GPUs with a batch size of 720.
2830
+ In retromae, the mask ratio of encoder and decoder are 0.3, 0.5 respectively.
2831
  We used the AdamW optimizer and the learning rate is 2e-5.
2832
 
2833
  **Pre-training data**:
 
2836
  - [wikipedia](https://huggingface.co/datasets/wikipedia)
2837
  - [msmarco](https://huggingface.co/datasets/Tevatron/msmarco-passage-corpus)
2838
  - Chinese:
2839
+ - [wudao](https://github.com/BAAI-WuDao/Data)
 
2840
 
2841
 
2842
  **2. Finetune**
 
2850
  We used the AdamW optimizer and the learning rate is 1e-5.
2851
  The temperature for contrastive loss is 0.01.
2852
 
2853
+ Besides, we add instruction to the query for s2p(short query to long passage) retrieval task in the training (add nothing to passages).
2854
+ For English, the instruction is `Represent this sentence for searching relevant passages: `;
2855
+ For Chinese, the instruction is `为这个句子生成表示以用于检索相关文章:`.
2856
+ In the evaluation, the instruction should be added for queries in retrieval task, not be added for other tasks.
2857
+ Noted that the instruction is not needed for passages.
2858
 
2859
  The finetune script is accessible in this repository: [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/baai_general_embedding/README.md).
2860
  You can easily finetune your model with it.
 
2867
 
2868
  **The data collection is to be released in the future.**
2869
 
2870
+
2871
+ ## Schedule
2872
+ - [x] Chinese Massive Text Embedding Benchmark
2873
+ - [x] release baai-general-embedding models
2874
+ - [x] release codes for training
2875
+ - [ ] Multilingual model
2876
+ - [ ] Training Datasets
2877
+ - [ ] ...
2878
+
2879
  We will continually update the embedding models and training codes,
2880
  hoping to promote the development of the embedding model community.
2881
 
2882
+
2883
+ ## Contact
2884
+ If you have any question or suggestion related to this project, feel free to open an issue or pull a request.
2885
+ You also can email Shitao Xiao(stxiao@baai.ac.cn) and Zheng Liu(liuzheng@baai.ac.cn).
2886
+
2887
+
2888
  ## License
2889
+ FlagEmbedding is licensed under [MIT License](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE). The released models can be used for commercial purposes free of charge.
2890
+
2891
+
2892
+