ldwang commited on
Commit
13942ee
1 Parent(s): 0cfcd48
Files changed (1) hide show
  1. README.md +34 -16
README.md CHANGED
@@ -16,6 +16,7 @@ language:
16
  <a href="#evaluation">Evaluation</a> |
17
  <a href="#train">Train</a> |
18
  <a href="#contact">Contact</a> |
 
19
  <a href="#license">License</a>
20
  <p>
21
  </h4>
@@ -29,6 +30,7 @@ FlagEmbedding can map any text to a low-dimensional dense vector which can be us
29
  And it also can be used in vector databases for LLMs.
30
 
31
  ************* 🌟**Updates**🌟 *************
 
32
  - 09/12/2023: New Release:
33
  - **New reranker model**: release cross-encoder models `BAAI/bge-reranker-base` and `BAAI/bge-reranker-large`, which are more powerful than embedding model. We recommend to use/fine-tune them to re-rank top-k documents returned by embedding models.
34
  - **update embedding model**: release `bge-*-v1.5` embedding model to alleviate the issue of the similarity distribution, and enhance its retrieval ability without instruction.
@@ -63,10 +65,9 @@ And it also can be used in vector databases for LLMs.
63
 
64
  \*: If you need to search the relevant passages to a query, we suggest to add the instruction to the query; in other cases, no instruction is needed, just use the original query directly. In all cases, **no instruction** needs to be added to passages.
65
 
66
- \**: Different embedding model, reranker is a cross-encoder, which cannot be used to generate embedding. To balance the accuracy and time cost, cross-encoder is widely used to re-rank top-k documents retrieved by other simple models.
67
  For examples, use bge embedding model to retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 document to get the final top-3 results.
68
 
69
-
70
  ## Frequently asked questions
71
 
72
  <details>
@@ -129,7 +130,9 @@ If it doesn't work for you, you can see [FlagEmbedding](https://github.com/FlagO
129
  from FlagEmbedding import FlagModel
130
  sentences_1 = ["样例数据-1", "样例数据-2"]
131
  sentences_2 = ["样例数据-3", "样例数据-4"]
132
- model = FlagModel('BAAI/bge-large-zh', query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章:")
 
 
133
  embeddings_1 = model.encode(sentences_1)
134
  embeddings_2 = model.encode(sentences_2)
135
  similarity = embeddings_1 @ embeddings_2.T
@@ -160,7 +163,7 @@ pip install -U sentence-transformers
160
  from sentence_transformers import SentenceTransformer
161
  sentences_1 = ["样例数据-1", "样例数据-2"]
162
  sentences_2 = ["样例数据-3", "样例数据-4"]
163
- model = SentenceTransformer('BAAI/bge-large-zh')
164
  embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
165
  embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
166
  similarity = embeddings_1 @ embeddings_2.T
@@ -175,7 +178,7 @@ queries = ['query_1', 'query_2']
175
  passages = ["样例文档-1", "样例文档-2"]
176
  instruction = "为这个句子生成表示以用于检索相关文章:"
177
 
178
- model = SentenceTransformer('BAAI/bge-large-zh')
179
  q_embeddings = model.encode([instruction+q for q in queries], normalize_embeddings=True)
180
  p_embeddings = model.encode(passages, normalize_embeddings=True)
181
  scores = q_embeddings @ p_embeddings.T
@@ -186,7 +189,7 @@ scores = q_embeddings @ p_embeddings.T
186
  You can use `bge` in langchain like this:
187
  ```python
188
  from langchain.embeddings import HuggingFaceBgeEmbeddings
189
- model_name = "BAAI/bge-small-en"
190
  model_kwargs = {'device': 'cuda'}
191
  encode_kwargs = {'normalize_embeddings': True} # set True to compute cosine similarity
192
  model = HuggingFaceBgeEmbeddings(
@@ -210,8 +213,8 @@ import torch
210
  sentences = ["样例数据-1", "样例数据-2"]
211
 
212
  # Load model from HuggingFace Hub
213
- tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-large-zh')
214
- model = AutoModel.from_pretrained('BAAI/bge-large-zh')
215
  model.eval()
216
 
217
  # Tokenize sentences
@@ -231,6 +234,7 @@ print("Sentence embeddings:", sentence_embeddings)
231
 
232
  ### Usage for Reranker
233
 
 
234
  You can get a relevance score by inputting query and passage to the reranker.
235
  The reranker is optimized based cross-entropy loss, so the relevance score is not bounded to a specific range.
236
 
@@ -240,10 +244,10 @@ The reranker is optimized based cross-entropy loss, so the relevance score is no
240
  pip install -U FlagEmbedding
241
  ```
242
 
243
- Get relevance score:
244
  ```python
245
  from FlagEmbedding import FlagReranker
246
- reranker = FlagReranker('BAAI/bge-reranker-base', use_fp16=True) #use fp16 can speed up computing
247
 
248
  score = reranker.compute_score(['query', 'passage'])
249
  print(score)
@@ -257,10 +261,10 @@ print(scores)
257
 
258
  ```python
259
  import torch
260
- from transformers import AutoModelForSequenceClassification, AutoTokenizer, BatchEncoding, PreTrainedTokenizerFast
261
 
262
- tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-base')
263
- model = AutoModelForSequenceClassification.from_pretrained('BAAI/bge-reranker-base')
264
  model.eval()
265
 
266
  pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
@@ -326,7 +330,7 @@ Please refer to [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C
326
  - **Reranking**:
327
  See [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/) for evaluation script.
328
 
329
- | Model | T2Reranking | T2RerankingZh2En\* | T2RerankingEn2Zh\* | MmarcoReranking | CMedQAv1 | CMedQAv2 | Avg |
330
  |:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
331
  | text2vec-base-multilingual | 64.66 | 62.94 | 62.51 | 14.37 | 48.46 | 48.6 | 50.26 |
332
  | multilingual-e5-small | 65.62 | 60.94 | 56.41 | 29.91 | 67.26 | 66.54 | 57.78 |
@@ -339,13 +343,13 @@ See [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/) for
339
  | [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) | 67.28 | 63.95 | 60.45 | 35.46 | 81.26 | 84.1 | 65.42 |
340
  | [BAAI/bge-reranker-large](https://huggingface.co/BAAI/bge-reranker-large) | 67.6 | 64.03 | 61.44 | 37.16 | 82.15 | 84.18 | 66.09 |
341
 
342
- \* : T2RerankingZh2En and T2RerankingEn2Zh are cross-language retrieval task
343
 
344
  ## Train
345
 
346
  ### BAAI Embedding
347
 
348
- We pre-train the models using retromae and train them on large-scale pairs data using contrastive learning.
349
  **You can fine-tune the embedding model on your data following our [examples](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune).**
350
  We also provide a [pre-train example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/pretrain).
351
  Note that the goal of pre-training is to reconstruct the text, and the pre-trained model cannot be used for similarity calculation directly, it needs to be fine-tuned.
@@ -368,6 +372,20 @@ If you have any question or suggestion related to this project, feel free to ope
368
  You also can email Shitao Xiao(stxiao@baai.ac.cn) and Zheng Liu(liuzheng@baai.ac.cn).
369
 
370
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
371
  ## License
372
  FlagEmbedding is licensed under the [MIT License](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE). The released models can be used for commercial purposes free of charge.
373
 
 
16
  <a href="#evaluation">Evaluation</a> |
17
  <a href="#train">Train</a> |
18
  <a href="#contact">Contact</a> |
19
+ <a href="#citation">Citation</a> |
20
  <a href="#license">License</a>
21
  <p>
22
  </h4>
 
30
  And it also can be used in vector databases for LLMs.
31
 
32
  ************* 🌟**Updates**🌟 *************
33
+ - 09/15/2023: Release [paper](https://arxiv.org/pdf/2309.07597.pdf) and [dataset](https://data.baai.ac.cn/details/BAAI-MTP).
34
  - 09/12/2023: New Release:
35
  - **New reranker model**: release cross-encoder models `BAAI/bge-reranker-base` and `BAAI/bge-reranker-large`, which are more powerful than embedding model. We recommend to use/fine-tune them to re-rank top-k documents returned by embedding models.
36
  - **update embedding model**: release `bge-*-v1.5` embedding model to alleviate the issue of the similarity distribution, and enhance its retrieval ability without instruction.
 
65
 
66
  \*: If you need to search the relevant passages to a query, we suggest to add the instruction to the query; in other cases, no instruction is needed, just use the original query directly. In all cases, **no instruction** needs to be added to passages.
67
 
68
+ \**: Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. To balance the accuracy and time cost, cross-encoder is widely used to re-rank top-k documents retrieved by other simple models.
69
  For examples, use bge embedding model to retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 document to get the final top-3 results.
70
 
 
71
  ## Frequently asked questions
72
 
73
  <details>
 
130
  from FlagEmbedding import FlagModel
131
  sentences_1 = ["样例数据-1", "样例数据-2"]
132
  sentences_2 = ["样例数据-3", "样例数据-4"]
133
+ model = FlagModel('BAAI/bge-large-zh-v1.5',
134
+ query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章:",
135
+ use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
136
  embeddings_1 = model.encode(sentences_1)
137
  embeddings_2 = model.encode(sentences_2)
138
  similarity = embeddings_1 @ embeddings_2.T
 
163
  from sentence_transformers import SentenceTransformer
164
  sentences_1 = ["样例数据-1", "样例数据-2"]
165
  sentences_2 = ["样例数据-3", "样例数据-4"]
166
+ model = SentenceTransformer('BAAI/bge-large-zh-v1.5')
167
  embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
168
  embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
169
  similarity = embeddings_1 @ embeddings_2.T
 
178
  passages = ["样例文档-1", "样例文档-2"]
179
  instruction = "为这个句子生成表示以用于检索相关文章:"
180
 
181
+ model = SentenceTransformer('BAAI/bge-large-zh-v1.5')
182
  q_embeddings = model.encode([instruction+q for q in queries], normalize_embeddings=True)
183
  p_embeddings = model.encode(passages, normalize_embeddings=True)
184
  scores = q_embeddings @ p_embeddings.T
 
189
  You can use `bge` in langchain like this:
190
  ```python
191
  from langchain.embeddings import HuggingFaceBgeEmbeddings
192
+ model_name = "BAAI/bge-large-en-v1.5"
193
  model_kwargs = {'device': 'cuda'}
194
  encode_kwargs = {'normalize_embeddings': True} # set True to compute cosine similarity
195
  model = HuggingFaceBgeEmbeddings(
 
213
  sentences = ["样例数据-1", "样例数据-2"]
214
 
215
  # Load model from HuggingFace Hub
216
+ tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-large-zh-v1.5')
217
+ model = AutoModel.from_pretrained('BAAI/bge-large-zh-v1.5')
218
  model.eval()
219
 
220
  # Tokenize sentences
 
234
 
235
  ### Usage for Reranker
236
 
237
+ Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding.
238
  You can get a relevance score by inputting query and passage to the reranker.
239
  The reranker is optimized based cross-entropy loss, so the relevance score is not bounded to a specific range.
240
 
 
244
  pip install -U FlagEmbedding
245
  ```
246
 
247
+ Get relevance scores (higher scores indicate more relevance):
248
  ```python
249
  from FlagEmbedding import FlagReranker
250
+ reranker = FlagReranker('BAAI/bge-reranker-large', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
251
 
252
  score = reranker.compute_score(['query', 'passage'])
253
  print(score)
 
261
 
262
  ```python
263
  import torch
264
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
265
 
266
+ tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-large')
267
+ model = AutoModelForSequenceClassification.from_pretrained('BAAI/bge-reranker-large')
268
  model.eval()
269
 
270
  pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
 
330
  - **Reranking**:
331
  See [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/) for evaluation script.
332
 
333
+ | Model | T2Reranking | T2RerankingZh2En\* | T2RerankingEn2Zh\* | MMarcoReranking | CMedQAv1 | CMedQAv2 | Avg |
334
  |:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
335
  | text2vec-base-multilingual | 64.66 | 62.94 | 62.51 | 14.37 | 48.46 | 48.6 | 50.26 |
336
  | multilingual-e5-small | 65.62 | 60.94 | 56.41 | 29.91 | 67.26 | 66.54 | 57.78 |
 
343
  | [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) | 67.28 | 63.95 | 60.45 | 35.46 | 81.26 | 84.1 | 65.42 |
344
  | [BAAI/bge-reranker-large](https://huggingface.co/BAAI/bge-reranker-large) | 67.6 | 64.03 | 61.44 | 37.16 | 82.15 | 84.18 | 66.09 |
345
 
346
+ \* : T2RerankingZh2En and T2RerankingEn2Zh are cross-language retrieval tasks
347
 
348
  ## Train
349
 
350
  ### BAAI Embedding
351
 
352
+ We pre-train the models using [retromae](https://github.com/staoxiao/RetroMAE) and train them on large-scale pairs data using contrastive learning.
353
  **You can fine-tune the embedding model on your data following our [examples](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune).**
354
  We also provide a [pre-train example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/pretrain).
355
  Note that the goal of pre-training is to reconstruct the text, and the pre-trained model cannot be used for similarity calculation directly, it needs to be fine-tuned.
 
372
  You also can email Shitao Xiao(stxiao@baai.ac.cn) and Zheng Liu(liuzheng@baai.ac.cn).
373
 
374
 
375
+ ## Citation
376
+
377
+ If you find our work helpful, please cite us:
378
+ ```
379
+ @misc{bge_embedding,
380
+ title={C-Pack: Packaged Resources To Advance General Chinese Embedding},
381
+ author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff},
382
+ year={2023},
383
+ eprint={2309.07597},
384
+ archivePrefix={arXiv},
385
+ primaryClass={cs.CL}
386
+ }
387
+ ```
388
+
389
  ## License
390
  FlagEmbedding is licensed under the [MIT License](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE). The released models can be used for commercial purposes free of charge.
391