SlowGuess/NAACL_2021_OCR huggingface dataset:
COIL-full achieves a very narrow performance gap to ColBERT. Recall that ColBERT computes all-to-all soft matches between all token pairs. For retrieval, it needs to consider for each query token all mentions of all tokens in the collection (MS-MARCO passage collection has around 500M token mentions). COIL-full is able to capture matching patterns as effectively with exact match signals from only query tokens' mentions and a single CLS matching to bridge the vocabulary gap.
We observe a similar pattern in the rerank task. COIL-tok is already able to outperform dense retriever and COIL-full further adds up to performance with CLS matching, being on-par with Colbert. Meanwhile, previous BERT rerankers have little performance advantage over COIL  $^{5}$ . In practice, we found BERT rerankers to be much more
${}^{5}$  Close performance between COIL and BERT rerankers is partially due to the bottleneck of BM25 candidates.
Table 1: MSMARCO passage collection results. Results not applicable are denoted  $-$  and no available n.a'.
MS MARCO Passage Ranking
Model	Dev Rerank MRR@10	Dev Retrieval MRR@10	Recall@1K	DL2019 Retrieval NDCG@10	MRR@1K
Lexical Retriever
BM25	-	0.184	0.853	0.506	0.825
DeepCT	-	0.243	0.909	0.572	0.883
DocT5Query	-	0.278	0.945	0.642	0.888
BM25+BERT reranker	0.347	-	-	-	-
Dense Retriever
Dense (BM25 neg)	n.a.	0.299	0.928	0.600	n.a.
Dense (rand + BM25 neg)	n.a.	0.311	0.952	0.576	n.a.
Dense (our train)	0.312	0.304	0.932	0.635	0.898
ColBERT	0.349	0.360	0.968	n.a.	n.a.
COIL-tok	0.336	0.341	0.949	0.660	0.915
COIL-full	0.348	0.355	0.963	0.704	0.924
Table 2: MSMARCO document collection results. Results not applicable are denoted  ${}^{ - }$  ’ and no available ’n.a’.
MS MARCO Document Ranking
Model	Dev Rerank MRR@10	Dev Retrieval MRR@10	Recall@1K	DL2019 Retrieval NDCG@10	MRR@1K
Lexical Retriever
BM25	-	0.230	0.886	0.519	0.805
DeepCT	-	0.320	0.942	0.544	0.891
DocT5Query	-	0.288	0.926	0.597	0.837
BM25+BERT reranker	0.383	-	-	-	-
Dense Retriever
Dense (BM25 neg)	n.a.	0.299	0.928	0.600	n.a.
Dense (rand + BM25 neg)	n.a.	0.311	0.952	0.576	n.a.
Dense (our train)	0.358	0.340	0.883	0.546	0.785
COIL-tok	0.381	0.385	0.952	0.626	0.921
COIL-full	0.388	0.397	0.962	0.636	0.913
expensive, requiring over  $2700\mathrm{ms}$  for reranking compared to around 10ms in the case of COIL.
Table 2 reports the results on MSMARCO document collection. In general, we observe a similar pattern as with the passage case. COIL systems significantly outperform both lexical and dense systems in MRR and NDCG and retain a small advantage measured in recall. The results suggest that COIL can be applicable to longer documents with a consistent advantage in effectiveness.
The results indicate exact lexical match mechanism can be greatly improved with the introduction of contextualized representation in COIL. COIL's token-level match also yields better fine-grained signals than dense retriever's global match signal. COIL-full further combines the lexical signals with dense CLS match, forming a system that can deal with both vocabulary and semantic mismatch, being as effective as all-to-all system.
5.2 Analysis of Dimensionality
The second experiment tests how varying COIL's token dimension  $n_t$  and CLS dimension  $n_c$  affect model effectiveness and efficiency. We record retrieval performance and latency on MARCO passage collection in Table 3.
In COIL-full systems, reducing CLS dimension from 768 to 128 leads to a small drop in performance on the Dev set, indicating that a full 768 dimension may not be necessary for COIL. Keeping CLS dimension at 128, systems with token dimension 32 and 8 have very small performance difference, suggesting that token-specific semantic consumes much fewer dimensions. Similar pattern in  $n_t$  is also observed in COIL-tok ( $n_c = 0$ ).
On the DL2019 queries, we observe that reducing dimension actually achieves better MRR. We believe this is due to a regulatory effect, as the
Table 3: Performance and latency of COIL systems with different representation dimensions. Results not applicable are denoted  $-$  and no available 'n.a.' Here  $n_c$  denotes COIL CLS dimension and  $n_t$  token vector dimension. *: ColBERT use approximate search and quantization. We exclude I/O time from measurements.
Model	Dev Retrieval	DL2019 Retrieval	Latency/ms
MRR@10	Recall@1K	NDCG@10	MRR	CPU	GPU
BM25	0.184	0.853	0.506	0.825	36	n.a.
Dense	0.304	0.932	0.635	0.898	293	32
ColBERT	0.360	0.968	n.a.	n.a.	458*	-
COIL
nc	nt
768	32	0.355	0.963	0.704	0.924	380	41
128	32	0.350	0.953	0.692	0.956	125	23
128	8	0.347	0.956	0.694	0.977	113	21
0	32	0.341	0.949	0.660	0.915	67	18
0	8	0.336	0.940	0.678	0.953	55	16
Table 4: Sample query document pairs with similarity scores produced by COIL. Tokens in examination are colored blue. Numbers in brackets are query-document vector similarities computed with vectors generated by COIL.
Query Token	COIL Contextualized Exact Match Score	Relevance
what is a cabinet in govt	Cabinet [16.28] (government) A cabinet [16.75] is a body of high-ranking state officials, typically consisting of the top leaders of the ...	+
Cabinet [7.23] is 20x60 and top is 28x72. .... I had a 2cm granite counter-top installed with a 10 inch overhang on one side and a 14 inch....	-
what is priority pass	Priority Pass [11.61] is an independent airport lounge access program. A membership provides you with access to their network of over 700 ....	+
Snoqualmie Pass [7.98] is a mountain pass [6.83] that carries Interstate 90 through the Cascade Range in the U.S. State of Washington....	-
what is njstart	NJSTART is [1.25] a self-service online platform that allows vendors to manage forms, certifications, submit proposals, access training ....	+
Contract awardees will receive their Blanket P.O. once it is [-0.10] converted, and details regarding that process will also be sent...	-
test queries were labeled differently from the MS-MARCO train/dev queries (Craswell et al., 2020).
We also record CPU and GPU search latency in Table 3. Lowering COIL-full's CLS dimension from 768 to 128 gives a big speedup, making COIL faster than DPR system. Further dropping token dimensions provide some extra speedup. The COIL-tok systems run faster than COIL-full, with a latency of the same order of magnitude as the traditional BM25 system. Importantly, lower dimension COIL systems still retain a performance advantage over dense systems while being much faster. We include ColBERT's latency reported in the original paper, which was optimized by approximate search and quantization. All COIL systems have lower latency than ColBERT even though our current implementation does not use those optimization techniques. We however note that approximate search and quantization are applicable to COIL, and leave the study of speeding up COIL to future work.
COIL differs from all previous embedding-based models in that it does not use a single unified embedding space. Instead, for a specific token, COIL learns an embedding space to encode and measure the semantic similarity of the token in different contexts. In this section, we show examples where COIL differentiates different senses of a word under different contexts. In Table 4, we show how the token similarity scores differ across contexts in relevant and irrelevant query document pairs.
The first query looks for "cabinet" in the context of "govt" (abbreviation for "government"). The two documents both include query token "cabinet" but of a different concept. The first one refers to the government cabinet and the second to a case or cupboard. COIL manages to match "cabinet" in the query to "cabinet" in the first document with a much higher score. In the second query, "pass" in both documents refer to the concept of permis
sion. However, through contextualization, COIL captures the variation of the same concept and assigns a higher score to "pass" in the first document.
Stop words like "it", "a", and "the" are commonly removed in classic exact match IR systems as they are not informative on their own. In the third query, on the other hand, we observe that COIL is able to differentiate "is" in an explanatory sentence and "is" in a passive form, assigning the first higher score to match query context.
All examples here show that COIL can go beyond matching token surface form and introduce rich context information to estimate matching. Differences in similarity scores across mentions under different contexts demonstrate how COIL systems gain strength over lexical systems.
Exact lexical match systems have been widely used for decades in classical IR systems and prove to be effective and efficient. In this paper, we point out a critical problem, semantic mismatch, that generally limits all IR systems based on surface token for matching. To fix semantic mismatch, we introduce contextualized exact match to differentiate the same token in different contexts, providing effective semantic-aware token match signals. We further propose contextualized inverted list (COIL) search index which swaps token statistics in inverted lists with contextualized vector representations to perform effective search.
On two large-scale ad hoc retrieval benchmarks, we find COIL substantially improves lexical retrieval and outperforms state-of-the-art dense retrieval systems. These results indicate large headroom of the simple-but-efficient exact lexical match scheme. When the introduction of contextualization handles the issue of semantic mismatch, exact match system gains the capability of modeling complicated matching patterns that were not captured by classical systems.
Vocabulary mismatch in COIL can also be largely mitigated with a high-level CLS vector matching. The full system performs on par with more expensive and complex all-to-all match retrievers. The success of the full system also shows that dense retrieval and COIL's exact token matching give complementary effects, with COIL making up dense system's lost token level matching signals and dense solving the vocabulary mismatch probably for COIL.
With our COIL systems showing viable search latency, we believe this paper makes a solid step towards building next-generation index that stores semantics. At the intersection of lexical and neural systems, efficient algorithms proposed for both can push COIL towards real-world systems.
S. Blackford, J. Demmel, J. Dongarra, I. Duff, S. Hammarling, Greg Henry, M. Héroux, L. Kaufman, Andrew Lumsdaine, A. Petitet, R. Pozo, K. Remington, and C. Whaley. 2002. An updated set of basic linear algebra subprograms (bias). ACM Transactions on Mathematical Software, 28.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820.
Zhuyun Dai and J. Callan. 2019a. Context-aware sentence/passage term importance estimation for first stage retrieval. ArXiv, abs/1910.10687.
Zhuyun Dai and J. Callan. 2020. Context-aware document term weighting for ad-hoc search. Proceedings of The Web Conference 2020.
Zhuyun Dai and Jamie Callan. 2019b. Deeper text understanding for IR with contextual neural language modeling. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 985-988. ACM.
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
Fernando Diaz, Bhaskar Mitra, and Nick Craswell. 2016. Query expansion with locally-trained word embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
Debasis Ganguly, Dwaipayan Roy, Mandar Mitra, and Gareth J. F. Jones. 2015. Word embedding based generalized language model for information retrieval. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval.
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2020. Modularized transformer-based ranking framework. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. Association for Computational Linguistics.
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021a. Rethink training of BERT rerankers in multi-stage retrieval pipeline. In Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II.
Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2021b. Complement lexical retrieval model with semantic residual embeddings. In Advances in Information Retrieval - 43rd European Conference on IR Research,
ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part I.
J. Guo, Y. Fan, Qingyao Ai, and W. Croft. 2016. A deep relevance matching model for ad-hoc retrieval. Proceedings of the 25th ACM International on Conference on Information and Knowledge Management.
R. Guo, Philip Y. Sun, E. Lindgren, Quan Geng, David Simcha, Felix Chern, and S. Kumar. 2019. Accelerating large-scale inference with anisotropic vector quantization. arXiv: Learning.
Charles R. Harris, K. Jarrod Millman, Stefan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernandez del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. 2020. Array programming with NumPy. Nature.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and J. Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In ICLR.
J. Johnson, M. Douze, and H. Jégou. 2017. Billion-scale similarity search with gpus. ArXiv, abs/1702.08734.
V. Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and W. Yih. 2020. Dense passage retrieval for open-domain question answering. ArXiv, abs/2004.04906.
O. Khattab and M. Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP.
John Lafferty and Chengxiang Zhai. 2001. Document language models, query models, and risk minimization for information retrieval. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
Victor Lavrenko and W. Bruce Croft. 2001. Relevance-based language models. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and M. Collins. 2020. Sparse, dense, and attentional representations for text retrieval. ArXiv, abs/2005.00181.
Sean MacAvaney, F. Nardini, R. Perego, N. Tonellotto, Nazli Goharian, and O. Frieder. 2020. Efficient document re-ranking for transformers by precomputing term representations. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
Donald Metzler and W. Bruce Croft. 2005. A markov random field model for term dependencies. In SIGIR 2005: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
Tomas Mikolov, Ilya Sutskever, Kai Chen, G. S. Corrado, and J. Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS.
Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with bert. ArXiv, abs/1901.04085.
Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to doctttttquery.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32. Curran Associates, Inc.
Jeffrey Pennington, R. Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. ArXiv, abs/1802.05365.
Stephen E Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson
model for probabilistic weighted retrieval. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Chenyan Xiong, Zhuyun Dai, J. Callan, Zhiyuan Liu, and R. Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, J. Liu, P. Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval. ArXiv, abs/2007.00808.
Z. Yang, Zihang Dai, Yiming Yang, J. Carbonell, R. Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
A.1 Index Building Illustration
The following figure demonstrates how the document "apple pie baked ..." is indexed by COIL. The document is first processed by a fine-tuned deep LM to produce for each token a contextualized vector. The vectors of each term "apple" and "juice" are collected to the corresponding inverted list index along with the document id for lookup.
Document #10 - apple pie baked ...
Figure 4: COIL Index Building of document "apple pie baked..."
A.2 Search Illustration
The following figure demonstrates how the query "apple juice" is processed by COIL. Contextualized vectors of each term "apple" and "juice" go to the corresponding inverted list index consisting of a lookup id array and a matrix stacked from document term vectors. For each index, a matrix vector product is run to produce an array of scores. Afterwards a max-scatter of scores followed by a sort produces the final ranking. Note for each index, we show only operations for a subset of vectors (3 vectors) in the index matrix.
Query: apple juice
Figure 5: COIL Search of query "apple juice".