|
--- |
|
language: |
|
- zh |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- PEG |
|
- feature-extraction |
|
- sentence-similarity |
|
- transformers |
|
- mteb |
|
model-index: |
|
- name: PEG |
|
results: |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/CMedQAv1-reranking |
|
name: MTEB CMedQAv1 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 84.09137463267582 |
|
- type: mrr |
|
value: 86.6288888888889 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/CMedQAv2-reranking |
|
name: MTEB CMedQAv2 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 86.55765031914974 |
|
- type: mrr |
|
value: 89.4325396825397 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C_MTEB/CmedqaRetrieval |
|
name: MTEB CmedqaRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 26.101000000000003 |
|
- type: map_at_10 |
|
value: 38.239000000000004 |
|
- type: map_at_100 |
|
value: 40.083 |
|
- type: map_at_1000 |
|
value: 40.205 |
|
- type: map_at_3 |
|
value: 34.386 |
|
- type: map_at_5 |
|
value: 36.425999999999995 |
|
- type: mrr_at_1 |
|
value: 39.434999999999995 |
|
- type: mrr_at_10 |
|
value: 46.967999999999996 |
|
- type: mrr_at_100 |
|
value: 47.946 |
|
- type: mrr_at_1000 |
|
value: 47.997 |
|
- type: mrr_at_3 |
|
value: 44.803 |
|
- type: mrr_at_5 |
|
value: 45.911 |
|
- type: ndcg_at_1 |
|
value: 39.434999999999995 |
|
- type: ndcg_at_10 |
|
value: 44.416 |
|
- type: ndcg_at_100 |
|
value: 51.773 |
|
- type: ndcg_at_1000 |
|
value: 53.888000000000005 |
|
- type: ndcg_at_3 |
|
value: 39.816 |
|
- type: ndcg_at_5 |
|
value: 41.467999999999996 |
|
- type: precision_at_1 |
|
value: 39.434999999999995 |
|
- type: precision_at_10 |
|
value: 9.786999999999999 |
|
- type: precision_at_100 |
|
value: 1.5810000000000002 |
|
- type: precision_at_1000 |
|
value: 0.184 |
|
- type: precision_at_3 |
|
value: 22.414 |
|
- type: precision_at_5 |
|
value: 15.943999999999999 |
|
- type: recall_at_1 |
|
value: 26.101000000000003 |
|
- type: recall_at_10 |
|
value: 53.82900000000001 |
|
- type: recall_at_100 |
|
value: 84.63199999999999 |
|
- type: recall_at_1000 |
|
value: 98.782 |
|
- type: recall_at_3 |
|
value: 39.585 |
|
- type: recall_at_5 |
|
value: 45.141 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C_MTEB/CovidRetrieval |
|
name: MTEB CovidRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 70.39 |
|
- type: map_at_10 |
|
value: 78.93599999999999 |
|
- type: map_at_100 |
|
value: 79.202 |
|
- type: map_at_1000 |
|
value: 79.205 |
|
- type: map_at_3 |
|
value: 77.538 |
|
- type: map_at_5 |
|
value: 78.312 |
|
- type: mrr_at_1 |
|
value: 70.706 |
|
- type: mrr_at_10 |
|
value: 79.018 |
|
- type: mrr_at_100 |
|
value: 79.28399999999999 |
|
- type: mrr_at_1000 |
|
value: 79.288 |
|
- type: mrr_at_3 |
|
value: 77.713 |
|
- type: mrr_at_5 |
|
value: 78.462 |
|
- type: ndcg_at_1 |
|
value: 70.601 |
|
- type: ndcg_at_10 |
|
value: 82.555 |
|
- type: ndcg_at_100 |
|
value: 83.718 |
|
- type: ndcg_at_1000 |
|
value: 83.855 |
|
- type: ndcg_at_3 |
|
value: 79.779 |
|
- type: ndcg_at_5 |
|
value: 81.149 |
|
- type: precision_at_1 |
|
value: 70.601 |
|
- type: precision_at_10 |
|
value: 9.463000000000001 |
|
- type: precision_at_100 |
|
value: 0.9979999999999999 |
|
- type: precision_at_1000 |
|
value: 0.101 |
|
- type: precision_at_3 |
|
value: 28.871999999999996 |
|
- type: precision_at_5 |
|
value: 18.019 |
|
- type: recall_at_1 |
|
value: 70.39 |
|
- type: recall_at_10 |
|
value: 93.572 |
|
- type: recall_at_100 |
|
value: 98.736 |
|
- type: recall_at_1000 |
|
value: 99.895 |
|
- type: recall_at_3 |
|
value: 86.091 |
|
- type: recall_at_5 |
|
value: 89.384 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C_MTEB/DuRetrieval |
|
name: MTEB DuRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 26.147 |
|
- type: map_at_10 |
|
value: 80.205 |
|
- type: map_at_100 |
|
value: 82.96 |
|
- type: map_at_1000 |
|
value: 82.999 |
|
- type: map_at_3 |
|
value: 55.16799999999999 |
|
- type: map_at_5 |
|
value: 69.798 |
|
- type: mrr_at_1 |
|
value: 89.8 |
|
- type: mrr_at_10 |
|
value: 93.16799999999999 |
|
- type: mrr_at_100 |
|
value: 93.22500000000001 |
|
- type: mrr_at_1000 |
|
value: 93.228 |
|
- type: mrr_at_3 |
|
value: 92.85 |
|
- type: mrr_at_5 |
|
value: 93.067 |
|
- type: ndcg_at_1 |
|
value: 89.8 |
|
- type: ndcg_at_10 |
|
value: 87.668 |
|
- type: ndcg_at_100 |
|
value: 90.16 |
|
- type: ndcg_at_1000 |
|
value: 90.505 |
|
- type: ndcg_at_3 |
|
value: 85.842 |
|
- type: ndcg_at_5 |
|
value: 85.101 |
|
- type: precision_at_1 |
|
value: 89.8 |
|
- type: precision_at_10 |
|
value: 42.225 |
|
- type: precision_at_100 |
|
value: 4.8149999999999995 |
|
- type: precision_at_1000 |
|
value: 0.48900000000000005 |
|
- type: precision_at_3 |
|
value: 76.967 |
|
- type: precision_at_5 |
|
value: 65.32 |
|
- type: recall_at_1 |
|
value: 26.147 |
|
- type: recall_at_10 |
|
value: 89.30399999999999 |
|
- type: recall_at_100 |
|
value: 97.609 |
|
- type: recall_at_1000 |
|
value: 99.409 |
|
- type: recall_at_3 |
|
value: 57.56 |
|
- type: recall_at_5 |
|
value: 74.78200000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C_MTEB/EcomRetrieval |
|
name: MTEB EcomRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 53.300000000000004 |
|
- type: map_at_10 |
|
value: 62.507000000000005 |
|
- type: map_at_100 |
|
value: 63.068000000000005 |
|
- type: map_at_1000 |
|
value: 63.08200000000001 |
|
- type: map_at_3 |
|
value: 60.050000000000004 |
|
- type: map_at_5 |
|
value: 61.41 |
|
- type: mrr_at_1 |
|
value: 53.300000000000004 |
|
- type: mrr_at_10 |
|
value: 62.507000000000005 |
|
- type: mrr_at_100 |
|
value: 63.068000000000005 |
|
- type: mrr_at_1000 |
|
value: 63.08200000000001 |
|
- type: mrr_at_3 |
|
value: 60.050000000000004 |
|
- type: mrr_at_5 |
|
value: 61.41 |
|
- type: ndcg_at_1 |
|
value: 53.300000000000004 |
|
- type: ndcg_at_10 |
|
value: 67.31700000000001 |
|
- type: ndcg_at_100 |
|
value: 69.862 |
|
- type: ndcg_at_1000 |
|
value: 70.231 |
|
- type: ndcg_at_3 |
|
value: 62.222 |
|
- type: ndcg_at_5 |
|
value: 64.66300000000001 |
|
- type: precision_at_1 |
|
value: 53.300000000000004 |
|
- type: precision_at_10 |
|
value: 8.260000000000002 |
|
- type: precision_at_100 |
|
value: 0.941 |
|
- type: precision_at_1000 |
|
value: 0.097 |
|
- type: precision_at_3 |
|
value: 22.833000000000002 |
|
- type: precision_at_5 |
|
value: 14.879999999999999 |
|
- type: recall_at_1 |
|
value: 53.300000000000004 |
|
- type: recall_at_10 |
|
value: 82.6 |
|
- type: recall_at_100 |
|
value: 94.1 |
|
- type: recall_at_1000 |
|
value: 97.0 |
|
- type: recall_at_3 |
|
value: 68.5 |
|
- type: recall_at_5 |
|
value: 74.4 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C_MTEB/MMarcoRetrieval |
|
name: MTEB MMarcoRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 70.68799999999999 |
|
- type: map_at_10 |
|
value: 79.28399999999999 |
|
- type: map_at_100 |
|
value: 79.537 |
|
- type: map_at_1000 |
|
value: 79.545 |
|
- type: map_at_3 |
|
value: 77.643 |
|
- type: map_at_5 |
|
value: 78.694 |
|
- type: mrr_at_1 |
|
value: 73.05199999999999 |
|
- type: mrr_at_10 |
|
value: 79.794 |
|
- type: mrr_at_100 |
|
value: 80.024 |
|
- type: mrr_at_1000 |
|
value: 80.03099999999999 |
|
- type: mrr_at_3 |
|
value: 78.441 |
|
- type: mrr_at_5 |
|
value: 79.29 |
|
- type: ndcg_at_1 |
|
value: 73.05199999999999 |
|
- type: ndcg_at_10 |
|
value: 82.627 |
|
- type: ndcg_at_100 |
|
value: 83.737 |
|
- type: ndcg_at_1000 |
|
value: 83.946 |
|
- type: ndcg_at_3 |
|
value: 79.585 |
|
- type: ndcg_at_5 |
|
value: 81.306 |
|
- type: precision_at_1 |
|
value: 73.05199999999999 |
|
- type: precision_at_10 |
|
value: 9.835 |
|
- type: precision_at_100 |
|
value: 1.038 |
|
- type: precision_at_1000 |
|
value: 0.106 |
|
- type: precision_at_3 |
|
value: 29.756 |
|
- type: precision_at_5 |
|
value: 18.788 |
|
- type: recall_at_1 |
|
value: 70.68799999999999 |
|
- type: recall_at_10 |
|
value: 92.38300000000001 |
|
- type: recall_at_100 |
|
value: 97.347 |
|
- type: recall_at_1000 |
|
value: 98.992 |
|
- type: recall_at_3 |
|
value: 84.37 |
|
- type: recall_at_5 |
|
value: 88.434 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C_MTEB/MedicalRetrieval |
|
name: MTEB MedicalRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 53.1 |
|
- type: map_at_10 |
|
value: 58.36599999999999 |
|
- type: map_at_100 |
|
value: 58.939 |
|
- type: map_at_1000 |
|
value: 58.99100000000001 |
|
- type: map_at_3 |
|
value: 57.15 |
|
- type: map_at_5 |
|
value: 57.794999999999995 |
|
- type: mrr_at_1 |
|
value: 53.2 |
|
- type: mrr_at_10 |
|
value: 58.416000000000004 |
|
- type: mrr_at_100 |
|
value: 58.989999999999995 |
|
- type: mrr_at_1000 |
|
value: 59.041 |
|
- type: mrr_at_3 |
|
value: 57.199999999999996 |
|
- type: mrr_at_5 |
|
value: 57.845 |
|
- type: ndcg_at_1 |
|
value: 53.1 |
|
- type: ndcg_at_10 |
|
value: 60.989000000000004 |
|
- type: ndcg_at_100 |
|
value: 63.967 |
|
- type: ndcg_at_1000 |
|
value: 65.436 |
|
- type: ndcg_at_3 |
|
value: 58.425000000000004 |
|
- type: ndcg_at_5 |
|
value: 59.583 |
|
- type: precision_at_1 |
|
value: 53.1 |
|
- type: precision_at_10 |
|
value: 6.93 |
|
- type: precision_at_100 |
|
value: 0.8370000000000001 |
|
- type: precision_at_1000 |
|
value: 0.096 |
|
- type: precision_at_3 |
|
value: 20.7 |
|
- type: precision_at_5 |
|
value: 12.98 |
|
- type: recall_at_1 |
|
value: 53.1 |
|
- type: recall_at_10 |
|
value: 69.3 |
|
- type: recall_at_100 |
|
value: 83.7 |
|
- type: recall_at_1000 |
|
value: 95.5 |
|
- type: recall_at_3 |
|
value: 62.1 |
|
- type: recall_at_5 |
|
value: 64.9 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/Mmarco-reranking |
|
name: MTEB MMarcoReranking |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 33.548800108363665 |
|
- type: mrr |
|
value: 32.529761904761905 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/T2Reranking |
|
name: MTEB T2Reranking |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 69.43381583724414 |
|
- type: mrr |
|
value: 80.47879657392181 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C_MTEB/T2Retrieval |
|
name: MTEB T2Retrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 28.116000000000003 |
|
- type: map_at_10 |
|
value: 80.026 |
|
- type: map_at_100 |
|
value: 83.541 |
|
- type: map_at_1000 |
|
value: 83.592 |
|
- type: map_at_3 |
|
value: 56.092 |
|
- type: map_at_5 |
|
value: 69.114 |
|
- type: mrr_at_1 |
|
value: 91.557 |
|
- type: mrr_at_10 |
|
value: 93.73700000000001 |
|
- type: mrr_at_100 |
|
value: 93.808 |
|
- type: mrr_at_1000 |
|
value: 93.811 |
|
- type: mrr_at_3 |
|
value: 93.384 |
|
- type: mrr_at_5 |
|
value: 93.614 |
|
- type: ndcg_at_1 |
|
value: 91.553 |
|
- type: ndcg_at_10 |
|
value: 87.003 |
|
- type: ndcg_at_100 |
|
value: 90.128 |
|
- type: ndcg_at_1000 |
|
value: 90.615 |
|
- type: ndcg_at_3 |
|
value: 88.205 |
|
- type: ndcg_at_5 |
|
value: 86.978 |
|
- type: precision_at_1 |
|
value: 91.553 |
|
- type: precision_at_10 |
|
value: 43.25 |
|
- type: precision_at_100 |
|
value: 5.067 |
|
- type: precision_at_1000 |
|
value: 0.518 |
|
- type: precision_at_3 |
|
value: 77.25 |
|
- type: precision_at_5 |
|
value: 64.902 |
|
- type: recall_at_1 |
|
value: 28.116000000000003 |
|
- type: recall_at_10 |
|
value: 85.994 |
|
- type: recall_at_100 |
|
value: 96.345 |
|
- type: recall_at_1000 |
|
value: 98.867 |
|
- type: recall_at_3 |
|
value: 57.67099999999999 |
|
- type: recall_at_5 |
|
value: 72.26 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C_MTEB/VideoRetrieval |
|
name: MTEB VideoRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 64.9 |
|
- type: map_at_10 |
|
value: 73.763 |
|
- type: map_at_100 |
|
value: 74.116 |
|
- type: map_at_1000 |
|
value: 74.12100000000001 |
|
- type: map_at_3 |
|
value: 72.15 |
|
- type: map_at_5 |
|
value: 73.25 |
|
- type: mrr_at_1 |
|
value: 64.9 |
|
- type: mrr_at_10 |
|
value: 73.763 |
|
- type: mrr_at_100 |
|
value: 74.116 |
|
- type: mrr_at_1000 |
|
value: 74.12100000000001 |
|
- type: mrr_at_3 |
|
value: 72.15 |
|
- type: mrr_at_5 |
|
value: 73.25 |
|
- type: ndcg_at_1 |
|
value: 64.9 |
|
- type: ndcg_at_10 |
|
value: 77.639 |
|
- type: ndcg_at_100 |
|
value: 79.396 |
|
- type: ndcg_at_1000 |
|
value: 79.554 |
|
- type: ndcg_at_3 |
|
value: 74.406 |
|
- type: ndcg_at_5 |
|
value: 76.385 |
|
- type: precision_at_1 |
|
value: 64.9 |
|
- type: precision_at_10 |
|
value: 8.959999999999999 |
|
- type: precision_at_100 |
|
value: 0.979 |
|
- type: precision_at_1000 |
|
value: 0.099 |
|
- type: precision_at_3 |
|
value: 26.967000000000002 |
|
- type: precision_at_5 |
|
value: 17.14 |
|
- type: recall_at_1 |
|
value: 64.9 |
|
- type: recall_at_10 |
|
value: 89.60000000000001 |
|
- type: recall_at_100 |
|
value: 97.89999999999999 |
|
- type: recall_at_1000 |
|
value: 99.2 |
|
- type: recall_at_3 |
|
value: 80.9 |
|
- type: recall_at_5 |
|
value: 85.7 |
|
--- |
|
license: apache-2.0 |
|
library_name: transformers |
|
--- |
|
|
|
<h1 align="center">PEG: Towards Robust Text Retrieval with Progressive Learning</h1> |
|
|
|
## Model Details |
|
We propose the PEG model (a Progressively Learned Textual Embedding), which progressively adjusts the weights of samples contributing to the loss within an extremely large batch, based on the difficulty levels of negative samples. |
|
we have amassed an extensive collection of over 110 million data, spanning a wide range of fields such as general knowledge, finance, tourism, medicine, and more. |
|
|
|
Our technical report is available at [Paper](https://arxiv.org/pdf/2311.11691.pdf) |
|
|
|
## Usage (HuggingFace Transformers) |
|
|
|
Install transformers: |
|
``` |
|
pip install transformers |
|
``` |
|
|
|
Then load model and predict: |
|
```python |
|
from transformers import AutoModel, AutoTokenizer |
|
import torch |
|
|
|
|
|
# Load model from HuggingFace Hub |
|
tokenizer = AutoTokenizer.from_pretrained('TownsWu/PEG') |
|
model = AutoModel.from_pretrained('TownsWu/PEG') |
|
sentences = ['如何更换花呗绑定银行卡', '花呗更改绑定银行卡'] |
|
# Tokenize sentences |
|
inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') |
|
|
|
# Compute token embeddings |
|
with torch.no_grad(): |
|
last_hidden_state = model(**inputs, return_dict=True).last_hidden_state |
|
embeddings = last_hidden_state[:, 0] |
|
print("embeddings:") |
|
print(embeddings) |
|
``` |
|
|
|
## Contact |
|
If you have any question or suggestion related to this project, feel free to open an issue or pull request. |
|
You also can email Tong Wu(townswu@tencent.com). |
|
|
|
|
|
## Citation |
|
|
|
If you find our work helpful for your research, please consider citing the following BibTeX entry: |
|
|
|
``` |
|
|
|
@article{wu2023towards, |
|
title={Towards Robust Text Retrieval with Progressive Learning}, |
|
author={Wu, Tong and Qin, Yulei and Zhang, Enwei and Xu, Zihan and Gao, Yuting and Li, Ke and Sun, Xing}, |
|
journal={arXiv preprint arXiv:2311.11691}, |
|
year={2023} |
|
} |
|
|
|
``` |
