File size: 11,386 Bytes
27f5d2f 7ac7d85 27f5d2f 7ac7d85 27f5d2f 7ac7d85 27f5d2f 7ac7d85 27f5d2f 7ac7d85 27f5d2f 2aee0f7 27f5d2f 7ac7d85 27f5d2f 7ac7d85 27f5d2f 7ac7d85 27f5d2f 7ac7d85 27f5d2f 7ac7d85 27f5d2f 7ac7d85 27f5d2f 7ac7d85 27f5d2f 7ac7d85 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 |
---
license: mit
language:
- en
tags:
- ColBERT
- Passage Retrieval
---
# Model Card for ColBERT
<!-- Provide a quick summary of what the model is/does. -->
`ColBERT-v2` is a passage retrieval model that leverages late interaction to allow token-level interaction between query and document embeddings. It outperforms traditional retrievers (such as Dense Passage Retrieval) that encode queries and documents into one-dimensional embeddings.
With the approval of the original author, this model is re-implemented with the huggingface-compatible implementation of [PreFLMR](https://preflmr.github.io/). All the weights are imported from the [official ColBERTv2 checkpoint](https://github.com/stanford-futuredata/ColBERT?tab=readme-ov-file).
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, Matei Zaharia
- **Re-implemented by:** Weizhe Lin, Howard Mei, Jinghong Chen, Bill Byrne
- **Model type:** FLMRModelForRetrieval
- **Language(s) (NLP):** English
- **License:** MIT License
### Model Sources
<!-- Provide the basic links for the model. -->
- **Repository:** https://github.com/stanford-futuredata/ColBERT
- **Repository of implementation:** https://github.com/LinWeizheDragon/FLMR
- **Paper:** https://arxiv.org/abs/2112.01488
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
This model can be used directly to retrieve documents from a large corpus using text queries. The retrieval usage can be found in the [PreFLMR implementation](https://github.com/LinWeizheDragon/FLMR).
### Downstream Use
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
This model can be used combined with language models to create a retrieval-augmented language model.
## How to Get Started with the Model
For details of training, indexing, and performing retrieval, please refer to [here](https://github.com/LinWeizheDragon/FLMR).
1. Install the [FLMR package](https://github.com/LinWeizheDragon/FLMR).
2. A simple example use of this model:
```python
from flmr import FLMRConfig, FLMRModelForRetrieval, FLMRQueryEncoderTokenizer, FLMRContextEncoderTokenizer
checkpoint_path = "LinWeizheDragon/ColBERT-v2"
query_tokenizer = FLMRQueryEncoderTokenizer.from_pretrained(checkpoint_path, subfolder="query_tokenizer")
context_tokenizer = FLMRContextEncoderTokenizer.from_pretrained(checkpoint_path, subfolder="context_tokenizer")
model = FLMRModelForRetrieval.from_pretrained(checkpoint_path,
query_tokenizer=query_tokenizer,
context_tokenizer=context_tokenizer,
)
Q_encoding = query_tokenizer(["What is the capital of France?", "What is the capital of China?"])
D_encoding = context_tokenizer(["Paris is the capital of France.", "Beijing is the capital of China.",
"Paris is the capital of France.", "Beijing is the capital of China."])
inputs = dict(
query_input_ids=Q_encoding['input_ids'],
query_attention_mask=Q_encoding['attention_mask'],
context_input_ids=D_encoding['input_ids'],
context_attention_mask=D_encoding['attention_mask'],
use_in_batch_negatives=True,
)
res = model.forward(**inputs)
```
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
The model was trained on [MS MARCO Passage Ranking](https://github.com/microsoft/MSMARCO-Passage-Ranking)
More details can be found in the [official ColBERT repository](https://github.com/stanford-futuredata/ColBERT?tab=readme-ov-file)
### Training Procedure
The training details can be found in the [official ColBERT repository](https://github.com/stanford-futuredata/ColBERT?tab=readme-ov-file) and the [ColBERTv2 paper](https://arxiv.org/abs/2112.01488)
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
The model is evaluated on a range of retrieval tasks, such as:
- [MS MARCO Passage Ranking](https://github.com/microsoft/MSMARCO-Passage-Ranking)
- [BEIR](https://arxiv.org/abs/2104.08663)
- [LoTTE](https://github.com/stanford-futuredata/ColBERT/blob/main/LoTTE.md)
- [OOD Wikipedia Open QA](https://arxiv.org/abs/2112.01488)
### Results
Find the evaluation results in the [ColBERTv2 paper](https://arxiv.org/abs/2112.01488)
## Citation
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
Please cite the original papers:
```
@inproceedings{10.1145/3397271.3401075,
author = {Khattab, Omar and Zaharia, Matei},
title = {ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT},
year = {2020},
isbn = {9781450380164},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3397271.3401075},
doi = {10.1145/3397271.3401075},
abstract = {Recent progress in Natural Language Understanding (NLU) is driving fast-paced advances in Information Retrieval (IR), largely owed to fine-tuning deep language models (LMs) for document ranking. While remarkably effective, the ranking models based on these LMs increase computational cost by orders of magnitude over prior approaches, particularly as they must feed each query-document pair through a massive neural network to compute a single relevance score. To tackle this, we present ColBERT, a novel ranking model that adapts deep LMs (in particular, BERT) for efficient retrieval. ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step that models their fine-grained similarity. By delaying and yet retaining this fine-granular interaction, ColBERT can leverage the expressiveness of deep LMs while simultaneously gaining the ability to pre-compute document representations offline, considerably speeding up query processing. Crucially, ColBERT's pruning-friendly interaction mechanism enables leveraging vector-similarity indexes for end-to-end retrieval directly from millions of documents. We extensively evaluate ColBERT using two recent passage search datasets. Results show that ColBERT's effectiveness is competitive with existing BERT-based models (and outperforms every non-BERT baseline), while executing two orders-of-magnitude faster and requiring up to four orders-of-magnitude fewer FLOPs per query.},
booktitle = {Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval},
pages = {39–48},
numpages = {10},
keywords = {bert, deep language models, efficiency, neural ir},
location = {Virtual Event, China},
series = {SIGIR '20}
}
@inproceedings{santhanam-etal-2022-colbertv2,
title = "{C}ol{BERT}v2: Effective and Efficient Retrieval via Lightweight Late Interaction",
author = "Santhanam, Keshav and
Khattab, Omar and
Saad-Falcon, Jon and
Potts, Christopher and
Zaharia, Matei",
editor = "Carpuat, Marine and
de Marneffe, Marie-Catherine and
Meza Ruiz, Ivan Vladimir",
booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = jul,
year = "2022",
address = "Seattle, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.naacl-main.272",
doi = "10.18653/v1/2022.naacl-main.272",
pages = "3715--3734",
abstract = "Neural information retrieval (IR) has greatly advanced search and other knowledge-intensive language tasks. While many neural IR methods encode queries and documents into single-vector representations, late interaction models produce multi-vector representations at the granularity of each token and decompose relevance modeling into scalable token-level computations. This decomposition has been shown to make late interaction more effective, but it inflates the space footprint of these models by an order of magnitude. In this work, we introduce ColBERTv2, a retriever that couples an aggressive residual compression mechanism with a denoised supervision strategy to simultaneously improve the quality and space footprint of late interaction. We evaluate ColBERTv2 across a wide range of benchmarks, establishing state-of-the-art quality within and outside the training domain while reducing the space footprint of late interaction models by 6{--}10x.",
}
@inproceedings{10.1145/3511808.3557325,
author = {Santhanam, Keshav and Khattab, Omar and Potts, Christopher and Zaharia, Matei},
title = {PLAID: An Efficient Engine for Late Interaction Retrieval},
year = {2022},
isbn = {9781450392365},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3511808.3557325},
doi = {10.1145/3511808.3557325},
abstract = {Pre-trained language models are increasingly important components across multiple information retrieval (IR) paradigms. Late interaction, introduced with the ColBERT model and recently refined in ColBERTv2, is a popular paradigm that holds state-of-the-art status across many benchmarks. To dramatically speed up the search latency of late interaction, we introduce the Performance-optimized Late Interaction Driver (PLAID) engine. Without impacting quality, PLAID swiftly eliminates low-scoring passages using a novel centroid interaction mechanism that treats every passage as a lightweight bag of centroids. PLAID uses centroid interaction as well as centroid pruning, a mechanism for sparsifying the bag of centroids, within a highly-optimized engine to reduce late interaction search latency by up to 7x on a GPU and 45x on a CPU against vanilla ColBERTv2, while continuing to deliver state-of-the-art retrieval quality. This allows the PLAID engine with ColBERTv2 to achieve latency of tens of milliseconds on a GPU and tens or just few hundreds of milliseconds on a CPU at large scale, even at the largest scales we evaluate with 140M passages.},
booktitle = {Proceedings of the 31st ACM International Conference on Information \& Knowledge Management},
pages = {1747–1756},
numpages = {10},
keywords = {colbert, dynamic pruning, efficient search, late interaction, neural information retrieval},
location = {Atlanta, GA, USA},
series = {CIKM '22}
}
```
If the implementation helps your research, please cite:
```
@article{Lin_Mei_Chen_Byrne_2024,
title={PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers},
url={http://arxiv.org/abs/2402.08327},
number={arXiv:2402.08327},
publisher={arXiv},
author={Lin, Weizhe and Mei, Jingbiao and Chen, Jinghong and Byrne, Bill},
year={2024}}
``` |