ldwang commited on
Commit
6747e44
1 Parent(s): 03bc1d2
Files changed (1) hide show
  1. README.md +34 -16
README.md CHANGED
@@ -2615,6 +2615,7 @@ language:
2615
  <a href="#evaluation">Evaluation</a> |
2616
  <a href="#train">Train</a> |
2617
  <a href="#contact">Contact</a> |
 
2618
  <a href="#license">License</a>
2619
  <p>
2620
  </h4>
@@ -2628,6 +2629,7 @@ FlagEmbedding can map any text to a low-dimensional dense vector which can be us
2628
  And it also can be used in vector databases for LLMs.
2629
 
2630
  ************* 🌟**Updates**🌟 *************
 
2631
  - 09/12/2023: New Release:
2632
  - **New reranker model**: release cross-encoder models `BAAI/bge-reranker-base` and `BAAI/bge-reranker-large`, which are more powerful than embedding model. We recommend to use/fine-tune them to re-rank top-k documents returned by embedding models.
2633
  - **update embedding model**: release `bge-*-v1.5` embedding model to alleviate the issue of the similarity distribution, and enhance its retrieval ability without instruction.
@@ -2662,10 +2664,9 @@ And it also can be used in vector databases for LLMs.
2662
 
2663
  \*: If you need to search the relevant passages to a query, we suggest to add the instruction to the query; in other cases, no instruction is needed, just use the original query directly. In all cases, **no instruction** needs to be added to passages.
2664
 
2665
- \**: Different embedding model, reranker is a cross-encoder, which cannot be used to generate embedding. To balance the accuracy and time cost, cross-encoder is widely used to re-rank top-k documents retrieved by other simple models.
2666
  For examples, use bge embedding model to retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 document to get the final top-3 results.
2667
 
2668
-
2669
  ## Frequently asked questions
2670
 
2671
  <details>
@@ -2728,7 +2729,9 @@ If it doesn't work for you, you can see [FlagEmbedding](https://github.com/FlagO
2728
  from FlagEmbedding import FlagModel
2729
  sentences_1 = ["样例数据-1", "样例数据-2"]
2730
  sentences_2 = ["样例数据-3", "样例数据-4"]
2731
- model = FlagModel('BAAI/bge-large-zh', query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章:")
 
 
2732
  embeddings_1 = model.encode(sentences_1)
2733
  embeddings_2 = model.encode(sentences_2)
2734
  similarity = embeddings_1 @ embeddings_2.T
@@ -2759,7 +2762,7 @@ pip install -U sentence-transformers
2759
  from sentence_transformers import SentenceTransformer
2760
  sentences_1 = ["样例数据-1", "样例数据-2"]
2761
  sentences_2 = ["样例数据-3", "样例数据-4"]
2762
- model = SentenceTransformer('BAAI/bge-large-zh')
2763
  embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
2764
  embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
2765
  similarity = embeddings_1 @ embeddings_2.T
@@ -2774,7 +2777,7 @@ queries = ['query_1', 'query_2']
2774
  passages = ["样例文档-1", "样例文档-2"]
2775
  instruction = "为这个句子生成表示以用于检索相关文章:"
2776
 
2777
- model = SentenceTransformer('BAAI/bge-large-zh')
2778
  q_embeddings = model.encode([instruction+q for q in queries], normalize_embeddings=True)
2779
  p_embeddings = model.encode(passages, normalize_embeddings=True)
2780
  scores = q_embeddings @ p_embeddings.T
@@ -2785,7 +2788,7 @@ scores = q_embeddings @ p_embeddings.T
2785
  You can use `bge` in langchain like this:
2786
  ```python
2787
  from langchain.embeddings import HuggingFaceBgeEmbeddings
2788
- model_name = "BAAI/bge-small-en"
2789
  model_kwargs = {'device': 'cuda'}
2790
  encode_kwargs = {'normalize_embeddings': True} # set True to compute cosine similarity
2791
  model = HuggingFaceBgeEmbeddings(
@@ -2809,8 +2812,8 @@ import torch
2809
  sentences = ["样例数据-1", "样例数据-2"]
2810
 
2811
  # Load model from HuggingFace Hub
2812
- tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-large-zh')
2813
- model = AutoModel.from_pretrained('BAAI/bge-large-zh')
2814
  model.eval()
2815
 
2816
  # Tokenize sentences
@@ -2830,6 +2833,7 @@ print("Sentence embeddings:", sentence_embeddings)
2830
 
2831
  ### Usage for Reranker
2832
 
 
2833
  You can get a relevance score by inputting query and passage to the reranker.
2834
  The reranker is optimized based cross-entropy loss, so the relevance score is not bounded to a specific range.
2835
 
@@ -2839,10 +2843,10 @@ The reranker is optimized based cross-entropy loss, so the relevance score is no
2839
  pip install -U FlagEmbedding
2840
  ```
2841
 
2842
- Get relevance score:
2843
  ```python
2844
  from FlagEmbedding import FlagReranker
2845
- reranker = FlagReranker('BAAI/bge-reranker-base', use_fp16=True) #use fp16 can speed up computing
2846
 
2847
  score = reranker.compute_score(['query', 'passage'])
2848
  print(score)
@@ -2856,10 +2860,10 @@ print(scores)
2856
 
2857
  ```python
2858
  import torch
2859
- from transformers import AutoModelForSequenceClassification, AutoTokenizer, BatchEncoding, PreTrainedTokenizerFast
2860
 
2861
- tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-base')
2862
- model = AutoModelForSequenceClassification.from_pretrained('BAAI/bge-reranker-base')
2863
  model.eval()
2864
 
2865
  pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
@@ -2925,7 +2929,7 @@ Please refer to [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C
2925
  - **Reranking**:
2926
  See [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/) for evaluation script.
2927
 
2928
- | Model | T2Reranking | T2RerankingZh2En\* | T2RerankingEn2Zh\* | MmarcoReranking | CMedQAv1 | CMedQAv2 | Avg |
2929
  |:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
2930
  | text2vec-base-multilingual | 64.66 | 62.94 | 62.51 | 14.37 | 48.46 | 48.6 | 50.26 |
2931
  | multilingual-e5-small | 65.62 | 60.94 | 56.41 | 29.91 | 67.26 | 66.54 | 57.78 |
@@ -2938,13 +2942,13 @@ See [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/) for
2938
  | [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) | 67.28 | 63.95 | 60.45 | 35.46 | 81.26 | 84.1 | 65.42 |
2939
  | [BAAI/bge-reranker-large](https://huggingface.co/BAAI/bge-reranker-large) | 67.6 | 64.03 | 61.44 | 37.16 | 82.15 | 84.18 | 66.09 |
2940
 
2941
- \* : T2RerankingZh2En and T2RerankingEn2Zh are cross-language retrieval task
2942
 
2943
  ## Train
2944
 
2945
  ### BAAI Embedding
2946
 
2947
- We pre-train the models using retromae and train them on large-scale pairs data using contrastive learning.
2948
  **You can fine-tune the embedding model on your data following our [examples](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune).**
2949
  We also provide a [pre-train example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/pretrain).
2950
  Note that the goal of pre-training is to reconstruct the text, and the pre-trained model cannot be used for similarity calculation directly, it needs to be fine-tuned.
@@ -2967,6 +2971,20 @@ If you have any question or suggestion related to this project, feel free to ope
2967
  You also can email Shitao Xiao(stxiao@baai.ac.cn) and Zheng Liu(liuzheng@baai.ac.cn).
2968
 
2969
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2970
  ## License
2971
  FlagEmbedding is licensed under the [MIT License](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE). The released models can be used for commercial purposes free of charge.
2972
 
 
2615
  <a href="#evaluation">Evaluation</a> |
2616
  <a href="#train">Train</a> |
2617
  <a href="#contact">Contact</a> |
2618
+ <a href="#citation">Citation</a> |
2619
  <a href="#license">License</a>
2620
  <p>
2621
  </h4>
 
2629
  And it also can be used in vector databases for LLMs.
2630
 
2631
  ************* 🌟**Updates**🌟 *************
2632
+ - 09/15/2023: Release [paper](https://arxiv.org/pdf/2309.07597.pdf) and [dataset](https://data.baai.ac.cn/details/BAAI-MTP).
2633
  - 09/12/2023: New Release:
2634
  - **New reranker model**: release cross-encoder models `BAAI/bge-reranker-base` and `BAAI/bge-reranker-large`, which are more powerful than embedding model. We recommend to use/fine-tune them to re-rank top-k documents returned by embedding models.
2635
  - **update embedding model**: release `bge-*-v1.5` embedding model to alleviate the issue of the similarity distribution, and enhance its retrieval ability without instruction.
 
2664
 
2665
  \*: If you need to search the relevant passages to a query, we suggest to add the instruction to the query; in other cases, no instruction is needed, just use the original query directly. In all cases, **no instruction** needs to be added to passages.
2666
 
2667
+ \**: Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. To balance the accuracy and time cost, cross-encoder is widely used to re-rank top-k documents retrieved by other simple models.
2668
  For examples, use bge embedding model to retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 document to get the final top-3 results.
2669
 
 
2670
  ## Frequently asked questions
2671
 
2672
  <details>
 
2729
  from FlagEmbedding import FlagModel
2730
  sentences_1 = ["样例数据-1", "样例数据-2"]
2731
  sentences_2 = ["样例数据-3", "样例数据-4"]
2732
+ model = FlagModel('BAAI/bge-large-zh-v1.5',
2733
+ query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章:",
2734
+ use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
2735
  embeddings_1 = model.encode(sentences_1)
2736
  embeddings_2 = model.encode(sentences_2)
2737
  similarity = embeddings_1 @ embeddings_2.T
 
2762
  from sentence_transformers import SentenceTransformer
2763
  sentences_1 = ["样例数据-1", "样例数据-2"]
2764
  sentences_2 = ["样例数据-3", "样例数据-4"]
2765
+ model = SentenceTransformer('BAAI/bge-large-zh-v1.5')
2766
  embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
2767
  embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
2768
  similarity = embeddings_1 @ embeddings_2.T
 
2777
  passages = ["样例文档-1", "样例文档-2"]
2778
  instruction = "为这个句子生成表示以用于检索相关文章:"
2779
 
2780
+ model = SentenceTransformer('BAAI/bge-large-zh-v1.5')
2781
  q_embeddings = model.encode([instruction+q for q in queries], normalize_embeddings=True)
2782
  p_embeddings = model.encode(passages, normalize_embeddings=True)
2783
  scores = q_embeddings @ p_embeddings.T
 
2788
  You can use `bge` in langchain like this:
2789
  ```python
2790
  from langchain.embeddings import HuggingFaceBgeEmbeddings
2791
+ model_name = "BAAI/bge-large-en-v1.5"
2792
  model_kwargs = {'device': 'cuda'}
2793
  encode_kwargs = {'normalize_embeddings': True} # set True to compute cosine similarity
2794
  model = HuggingFaceBgeEmbeddings(
 
2812
  sentences = ["样例数据-1", "样例数据-2"]
2813
 
2814
  # Load model from HuggingFace Hub
2815
+ tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-large-zh-v1.5')
2816
+ model = AutoModel.from_pretrained('BAAI/bge-large-zh-v1.5')
2817
  model.eval()
2818
 
2819
  # Tokenize sentences
 
2833
 
2834
  ### Usage for Reranker
2835
 
2836
+ Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding.
2837
  You can get a relevance score by inputting query and passage to the reranker.
2838
  The reranker is optimized based cross-entropy loss, so the relevance score is not bounded to a specific range.
2839
 
 
2843
  pip install -U FlagEmbedding
2844
  ```
2845
 
2846
+ Get relevance scores (higher scores indicate more relevance):
2847
  ```python
2848
  from FlagEmbedding import FlagReranker
2849
+ reranker = FlagReranker('BAAI/bge-reranker-large', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
2850
 
2851
  score = reranker.compute_score(['query', 'passage'])
2852
  print(score)
 
2860
 
2861
  ```python
2862
  import torch
2863
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
2864
 
2865
+ tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-large')
2866
+ model = AutoModelForSequenceClassification.from_pretrained('BAAI/bge-reranker-large')
2867
  model.eval()
2868
 
2869
  pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
 
2929
  - **Reranking**:
2930
  See [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/) for evaluation script.
2931
 
2932
+ | Model | T2Reranking | T2RerankingZh2En\* | T2RerankingEn2Zh\* | MMarcoReranking | CMedQAv1 | CMedQAv2 | Avg |
2933
  |:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
2934
  | text2vec-base-multilingual | 64.66 | 62.94 | 62.51 | 14.37 | 48.46 | 48.6 | 50.26 |
2935
  | multilingual-e5-small | 65.62 | 60.94 | 56.41 | 29.91 | 67.26 | 66.54 | 57.78 |
 
2942
  | [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) | 67.28 | 63.95 | 60.45 | 35.46 | 81.26 | 84.1 | 65.42 |
2943
  | [BAAI/bge-reranker-large](https://huggingface.co/BAAI/bge-reranker-large) | 67.6 | 64.03 | 61.44 | 37.16 | 82.15 | 84.18 | 66.09 |
2944
 
2945
+ \* : T2RerankingZh2En and T2RerankingEn2Zh are cross-language retrieval tasks
2946
 
2947
  ## Train
2948
 
2949
  ### BAAI Embedding
2950
 
2951
+ We pre-train the models using [retromae](https://github.com/staoxiao/RetroMAE) and train them on large-scale pairs data using contrastive learning.
2952
  **You can fine-tune the embedding model on your data following our [examples](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune).**
2953
  We also provide a [pre-train example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/pretrain).
2954
  Note that the goal of pre-training is to reconstruct the text, and the pre-trained model cannot be used for similarity calculation directly, it needs to be fine-tuned.
 
2971
  You also can email Shitao Xiao(stxiao@baai.ac.cn) and Zheng Liu(liuzheng@baai.ac.cn).
2972
 
2973
 
2974
+ ## Citation
2975
+
2976
+ If you find our work helpful, please cite us:
2977
+ ```
2978
+ @misc{bge_embedding,
2979
+ title={C-Pack: Packaged Resources To Advance General Chinese Embedding},
2980
+ author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff},
2981
+ year={2023},
2982
+ eprint={2309.07597},
2983
+ archivePrefix={arXiv},
2984
+ primaryClass={cs.CL}
2985
+ }
2986
+ ```
2987
+
2988
  ## License
2989
  FlagEmbedding is licensed under the [MIT License](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE). The released models can be used for commercial purposes free of charge.
2990