SlowGuess/NAACL_2021_OCR huggingface dataset:
Marco Wrzalik Dirk Krechel
RheinMain University of Applied Sciences, Germany
{firstname_lastname}@hs-rm.de
Many recent approaches towards neural information retrieval mitigate their computational costs by using a multi-stage ranking pipeline. In the first stage, a number of potentially relevant candidates are retrieved using an efficient retrieval model such as BM25. Although BM25 has proven decent performance as a first-stage ranker, it tends to miss relevant passages. In this context we propose CoRT, a simple neural first-stage ranking model that leverages contextual representations from pretrained language models such as BERT to complement term-based ranking functions while causing no significant delay at query time. Using the MS MARCO dataset, we show that CoRT significantly increases the candidate recall by complementing BM25 with missing candidates. Consequently, we find subsequent re-rankers achieve superior results with less candidates. We further demonstrate that passage retrieval using CoRT can be realized with surprisingly low latencies.
The successful development of neural ranking models over the past few years has rapidly advanced state-of-the-art performance in information retrieval (Guo et al., 2020; Craswell et al., 2020). One key aspect of the success is the exploitation of query-document interactions based on token representations from self-supervised language models (LMs) (Devlin et al., 2019; Radford et al., 2019; Pennington et al., 2014). Due to high computational effort, however, these interaction-focused approaches are limited to re-ranking scenarios and thus they depend on the effectiveness of first-stage ranking models for candidate retrieval. Term-based retrieval models such as BM25 have proven decent performance in this task, but tend to miss relevant passages. In this context, we propose Complementary Rankings from Transformers (CoRT), a simple neural first-stage ranking model that lever
ages contextual representations from transformer-based language models (Vaswani et al., 2017; Devlin et al., 2019) to complement term-based first-stage rankings. CoRT optimizes an underlying text encoder towards representations that reflect relevance through vector similarity. The model is trained to act complementary to term-based retrieval by using passages from BM25 rankings as negative examples. We study the characteristics of CoRT with four types of experiments based on the MS MARCO dataset. First, we measure various ranking metrics and compare the results with first-stage ranking baselines and competitors. In course of this, we demonstrate the portion of relevant candidates that are added by CoRT. Second, we combine the candidates from CoRT and BM25 with a state-of-the-art re-ranker based on BERT (Nogueira and Cho, 2019) and investigate how many candidates are needed to saturate the ranking quality. Third, we train CoRT with various representation sizes and measure its impact on the first-stage ranking quality. Fourth, we measure the retrieval latencies of CoRT with two retrieval modalities: a distributed exhaustive search on four GPUs and an approximate search based on a graph-based nearest-neighbor index with pruning heuristics (Iwasaki and Miyazaki, 2018). Finally, we build an exemplary end-to-end ranking pipeline using our first-stage ranking to demonstrate its efficiency. Our contribution is a first-stage ranking framework with the potential to improve end-to-end ranking pipelines by adding candidates that term-based retrieval models typically miss. With this it is possible to reduce the total number of re-ranking candidates without hurting end-to-end ranking quality. As a secondary contribution, we provide an open-source implementation<sup>1</sup> that enables other researchers to reproduce our results and test CoRT on other datasets.
<https://github.com/lavis-nlp/CoRT>
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4194-4204 June 6-11, 2021. ©2021 Association for Computational Linguistics
In this section we describe key concepts of neural ranking and refer to related work. We then present neural first-stage ranking approaches that allow direct comparison with our results.
2.1 Key Concepts of Neural Ranking
According to Guo et al. (2016), Neural ranking approaches can be categorized into two types of models depending on the architecture. Representation-focused approaches (Huang et al., 2013; Shen et al., 2014; Zamani et al., 2018) produce representations for queries and documents to predict relevancy scores using a simple distance or similarity measure. In this context, exploiting local interactions between neighboring terms is a commonly used technique (Shen et al., 2014; Zamani et al., 2018). Models of the interaction-focused type exploit interactions between query and document terms (Guo et al., 2016; Xiong et al., 2017; Dai et al., 2018; Nogueira and Cho, 2019). Although this leads to superior ranking quality (Guo et al., 2016, 2020; Qiao et al., 2019), it is computationally much more intensive, since a given query has to be processed together with each potentially relevant document. Hence, this type of neural ranking model is only applicable in a ranking pipeline, where limited numbers of documents are given as potentially relevant candidates. These candidates are selected by an efficient retrieval model that is able to retrieve documents directly from the corpus in a reasonable amount of time. The multi-stage ranking technique is also known as cascade ranking and optimizing the configuration of such a pipeline towards maximized efficiency and effectiveness has been extensively studied in the past (Wang et al., 2011; Chen et al., 2017). Many interaction-focused neural ranking models employ a dedicated layer to explicitly perform a matching between query and document terms (Guo et al., 2016; Xiong et al., 2017; Dai et al., 2018). Another approach is using the attention mechanism (Vaswani et al., 2017), or more specifically, a pretrained transformer encoder such as BERT (Devlin et al., 2019) to exploit both local and query-document interactions (Nogueira and Cho, 2019; Qiao et al., 2019). Recently, some hybrid approaches have been proposed that combine typical representation-focused techniques with interaction-focused approaches to reduce computational cost: Gao et al. (2020) propose a model architecture comprising three modules for document
understanding, query understanding and relevance judging respectively. The understanding modules produce token-level representations, which can be cached as usual in representation-focused approaches. The relevance judging module uses those cached representations to apply query-document interactions more quickly. Each module is a stack of transformer layers (Vaswani et al., 2017), initialized with weights from BERT. In a related approach, MacAvaney et al. (2020) investigate the relationship between different numbers of dedicated layers of BERT for query-document interactions and measure the resulting speedup that is due to token representation caching, as well as its impact on the end-to-end ranking quality. Khattab and Zaharia (2020) propose a related approach, namely ColBERT. The model architecture incorporates an inexpensive max-similarity mechanism to perform token-level query-document interactions. The authors propose to store token representations in an Approximate Nearest Neighbor (ANN) index to quickly retrieve only those documents that have token representations in the proximity to those of the query. Thus, ColBERT can be described as an end-to-end ranking approach that brings its own first-stage retrieval mechanism allowing to perform end-to-end ranking in a reasonable amount of time.
2.2 Neural First-stage Ranking
Now we discuss neural ranking approaches that can be used to retrieve passages or documents directly from an entire corpus in a reasonable amount of time and thus qualify for first-stage ranking. Many proposed methods make use of existing infrastructure for sparse bag-of-words retrieval or at least inverted indexing (Manning et al., 2008). Zamani et al. (2018) propose SNRM, a representation-focused approach with sparse representations that can be used with an inverted index as if each feature dimensions corresponds to a term in a bag-of-words representation. SNRM uses pretrained GloVe Word Embeddings (Pennington et al., 2014) to model soft-matched n-grams which are encoded in sparse representations. Nogueira et al. (2019) predict queries for given documents to expand those documents by corresponding query terms. In their first work, known as doc2query, they used a sequence-to-sequence (seq2seq) transformer model (Vaswani et al., 2017). In a subsequent work, Nogueira and Lin (2019) reported large effectiveness gains for their follow-up model docTTTTTquery by replac
ing the seq2seq model with  $T5$  (Raffel et al., 2020). Another approach aims at predicting optimal document term weights as a function of the term's context. DeepCT, proposed by Dai and Callan (2020), utilizes BERT to predict these context-aware weights based on associated queries in the training data.
Inverted indexing is only applicable to sparse representations. Representation-focused models using dense representations can instead employ an ANN index, which heuristically prunes documents that are unlikely to be in the top- $k$  proximity of the query representation to realize low response latencies (Boytsov et al., 2016; Gysel et al., 2018). Karpukhin et al. (2020) recently used this technique in combination with a fine-tuned BERT encoder for open question answering.
We describe a first-stage ranking model that acts as a complementary ranker to existing term-based retrieval models such as BM25. To achieve this, we make use of a transformer-based pretrained language model and its inherent ability to make use of token-level local interactions. Its complementary behavior is further supported by negative sampling from BM25 rankings.
The model architecture of CoRT, illustrated in Figure 1, follows the idea of a Siamese Neural Network (Bromley et al., 1993). Passages and queries are encoded using an identical model with shared weights except for one detail: The passage encoder  $\psi_{\alpha}$  and the query encoder  $\psi_{\beta}$  use different segment embeddings (Devlin et al., 2019). CoRT computes relevance scores as angular similarity between query and passage representations while training a pairwise ranking objective.
CoRT can incorporate any BERT-like encoder as underlying text encoder. Here, we use a pretrained ALBERT (Lan et al., 2020) encoder for its smaller model size, the tougher sentence coherence pretraining and increased first-stage ranking quality throughout our early-stage experiments compared to BERT. The tokenizer of ALBERT is a Word-Piece tokenizer (Wu et al., 2016) including the special tokens [CLS] and [SEP] known from BERT. From the text encoder we seek a single representa
Figure 1: CoRT's model architecture and pair-wise learning objective (simplified).
tion vector for the whole passage or query, which we call context representation. From ALBERT we take the [CLS] embedding of the last layer for this purpose. We denote the context representation obtained from the underlying encoder for an arbitrary string  $s$  with  $\tau(s) \in \mathbb{R}^h$  where  $h$  is the output representation size.
ALBERT's language modeling approach involves sentence coherence prediction for which segment embeddings are used to signal different input segments. Although we only feed single segments to the encoder, i.e. a query or a passage, we use segment embeddings which allow the model to encode queries differently from passages. The segment embeddings  $E_{A}$  and  $E_{B}$  (illustrated in Figure 1) are part of the context encoder functions  $\tau_{\alpha}$  and  $\tau_{\beta}$  for passages and queries respectively. The context representation is further projected to the desired representation size  $e$  using a linear layer followed by a tanh activation function. Thus, the complete passage encoder function is  $\psi_{\alpha}(s) := \tanh(W\tau_{\alpha}(s) + b)$  where  $W \in \mathbb{R}^{h \times e}$  and  $b \in \mathbb{R}^e$  are parameters of the linear layer. The query encoder  $\psi_{\beta}$  is defined analogous.
Training CoRT corresponds to updating the parameters of the encoder  $\psi$  towards representations that reflect relevance between queries and passages through vector similarity. Each training sample is a triple comprising a query  $q$ , a positive passage  $d^{+}$  and a negative passage  $d^{-}$ . While positive passages are taken from relevance assessments, negative passages are sampled from term-based rankings (i.e. BM25) to support the complementary property of CoRT. The relevance score for a query-passage pair  $(q,d)$  is calculated using the angular cosine similarity function:
Similar to Cer et al. (2018), we found angular similarity performs better than cosine similarity.
$$
s i m (q, d) := 1 - \arccos \left(\frac {\psi_ {\beta} (q) \cdot \psi_ {\alpha} (d)}{| | \psi_ {\beta} (q) | | | | \psi_ {\alpha} (d) | |}\right) / \pi
$$
As illustrated in Figure 1, the training objective is to score the positive example  $d^{+}$  by at least the margin  $\mu$  higher than the negative one  $d^{-}$ . As part of our loss function, we use the triplet margin objective:
$$
l (q, d ^ {+}, d ^ {-}) := m a x (0, s i m (q, d ^ {-}) - s i m (q, d ^ {+}) + \mu)
$$
Inspired by Song et al. (2016), we aim to take full advantage of the whole training batch. For each query, each passage in the batch is used as a negative example, except for the given positive passage. Thus, we define our batch-wise loss function as follows:
$$
\mathcal {L} := \sum_ {1 \leq i \leq n} \left(\sum_ {1 \leq j \leq n} l \left(q _ {i}, d _ {i} ^ {+}, d _ {j} ^ {-}\right) + \sum_ {1 \leq k \leq n, k \neq i} l \left(q _ {i}, d _ {i} ^ {+}, d _ {k} ^ {+}\right)\right)
$$
$q_{i},d_{i}^{+}$  and  $d_{i}^{-}$  denote the triple of the  $i_{th}$  sample in the batch and  $n$  the number of samples per batch. We found this technique to make the training process more robust against exploding gradients. Otherwise we need to employ gradient clipping (Zhang et al., 2020) to stabilize the training process. Also, it positively affects first-stage ranking results<sup>3</sup>.
3.4 Indexing and Retrieval
For retrieval with CoRT, each passage must be encoded by the passage encoder  $\psi_{\alpha}$ . Subsequent normalization of each vector allows us to use the dot product as a proxy score function for sim, which is sufficient to form rankings accurately. Given a query  $q$ , we calculate its representation  $\psi_{\beta}(q)$  and the dot product with each normalized passage vector. From those, the  $k$  highest scores are selected and sorted to form the CoRT ranking. This procedure can be implemented heavily parallelized using GPU matrix operations. Alternatively, the passage representations can be indexed in an ANN index to avoid exhaustive similarity search. In contrast to the first-stage ranking of Khattab and Zaharia (2020) and MacAvaney et al. (2020), we
only index one representation per passage rather than one per token. Finally, we combine the resulting ranking of CoRT with the respective BM25 ranking by interleaving the positions beginning with CoRT to create a single merged ranking of equal length. During this process, each passage that was already added by the other ranking is omitted. For example, merging two ranking lists beginning with  $[a,b,c,d,\ldots]$  and  $[e,c,f,a,\ldots]$  would result in  $[a,e,b,c,\not e,f,d,\not e,\ldots]$ . The interleaving procedure stops as soon as the desired ranking size has been reached. The result is a compound ranking of CoRT and BM25, which we denote with  $\mathrm{CoRT}_{\mathrm{BM25}}$ .
We present four experiments studying the ranking quality and recall of CoRT, the connection between the number of candidates and re-ranking effectiveness, the impact of the representation size  $e$ , and CoRT's retrieval latencies. Finally, we outline a competitive end-to-end ranking setup with CoRT and a BERT-based re-ranker.
4.1.1 MS MARCO Passage Retrieval
The Microsoft Machine Reading Comprehension (Nguyen et al., 2016) dataset for passage ranking was introduced in 2018. It provides a benchmark for passage retrieval with real-world queries and passages gathered from Microsoft's Bing search. The MS MARCO passage ranking task comprises 8.8M passages sampled from web pages and about 1M queries that are formulated as questions. The objective is to rank those passages high that were labeled as relevant to answer the respective question. The annotations, however, are sparse. There are 530k positive relevance labels distributed over 808k queries in the training set, whereby most queries are associated to one passage. The validation and evaluation sets, dev and eval, comprise 101k queries each. An official subset of dev, called dev. small comprises 6980 queries and 7437 relevance labels and is often used for publicly reported evaluations. We follow this convention and use dev. small for testing. The creators suggest to use the mean reciprocal rank cut at the tenth position (MRR@10) as primary evaluation measure. Additionally, we measure NDCG@20 (Manning et al., 2008) as less punishing ranking quality measure and the recall at various positions
<sup>3</sup>We achieve 2.0 p.p. higher MRR@10 compared to the plain triplet margin loss on the MS MARCO passage task.
Table 1: First-stage ranking results on the MS MARCO dev. small set. The asterisk (*) denotes merged rankings using an instance of CoRT that was not specifically trained to complement the corresponding term-based ranker.
MS MARCO
Passage (dev. small)	MRR
@10	NDCG
@20	@50	@100	RECALL
@200	@500	@1000
BM25	18.7	25.8	59.2	67.0	73.8	81.2	85.7
doc2query	21.5	-	-	-	-	-	89.3
DeepCT	24.3	32.1	68.5	75.2	81.0	87.3	90.9
docTTTTTquery	27.7	36.5	75.6	81.9	86.9	91.6	94.7
CoRT	27.1	34.0	66.4	73.1	78.6	84.4	88.0
CoRTBM25	27.4	35.9	74.3	81.6	87.3	92.5	94.9
CoRTDeepCT*	28.3	36.8	75.6	82.3	87.6	92.5	94.9
CoRTDocTTTTTquery*	28.8	38.0	78.5	85.3	90.0	94.4	96.5
to indicate how many relevant passages a re-ranker would miss if the number of candidates is reduced.
4.1.2 TREC 2019 DL Passage Retrieval
The passage retrieval section of the TREC 2019 Deep Learning Track (Craswell et al., 2020) provides on average 215 manual relevance assessments per query for a set of 43 MS MARCO queries. Each assessment corresponds to a rating on a scale from 0 (not relevant) to 3 (perfectly relevant). We adopt the evaluation metrics MRR (uncut), NDCG@10 and MAP from the official TREC overview. In contrast to the original MS MARCO benchmark, this evaluation set provides dense annotations, but only for few queries.
4.2 First-Stage Ranking
We train CoRT as described in Section 3.3 while using a representation size of  $e = 768$ . In this section we discuss the first-stage ranking results of our model using the datasets and their associated metrics described in Section 4.1.
4.2.1 MS MARCO Passage Retrieval
The results of CoRT and its baselines on the MS MARCO passage retrieval task (dev. small) are reported in Table 1. Next to BM25 as a baseline, we include DeepCT (Dai and Callan, 2020), doc2query (Nogueira et al., 2019) and its successor docTTTTTquery (Nogueira and Lin, 2019). All three are recent first-stage rankers with average retrieval latencies below 100ms per query on the MS MARCO passage corpus. The metrics MRR@10 and NDCG@20 reveal a quite decent ranking quality for the standalone CoRT ranker. Since CoRT's primary use is candidate retrieval rather than standalone ranking, we pay particular attention to the recall at various cuts. From the perspective of BM25, the absolute increase of recall due to merging with CoRT ranges between
Table 2: First-stage ranking results on the TREC 2019 DL passage task.
TREC DL 2019	MRR @1000	NDCG @10	MAP @1000	RECALL @500
BM25	68.5	49.7	29.0	69.4
CoRT	84.3	60.0	29.7	58.3
CoRTBM25	86.2	59.7	35.1	76.9
15.1 (RECALL@50) and 9.2 (RECALL@1000), which we consider the complementary portion of CoRT. Greater increases of recall can be noticed for lower cuts, which is particular useful when reranking is performed with low numbers of candidates. The top-200 candidates from  $\mathrm{CoRT}_{\mathrm{BM25}}$  comprise higher recall than the top-1000 candidates from BM25. The metrics for DeepCT and docTTTTquery have been calculated using published top-1000 rankings from the respective authors. Thus, we were able to merge those rankings with CoRT. However, the used instance of CoRT was trained on BM25 and not on the external ranker.
4.2.2 TREC 2019 DL Passage Retrieval
Although we consider the relevance assessments from TREC 2019 DL to be dense, we found 112 unlabeled passages among the 43 top-10 CoRT rankings while the assessments for the BM25 rankings are complete. This means there are, on average, 2.6 unlabeled passages in the top-10 CoRT rankings, which might make this evaluation somewhat unfavourable for CoRT and explain the drop in recall. Still, Table 2 shows superior results for CoRT compared to BM25 in terms of ranking quality (MRR, NDCG and MAP). Merging CoRT with BM25 slightly increases MRR and NDCG, while a decent gain in terms of recall can be noticed. We can not report any results for DeepCT or docTTT
Tquery, since only rankings for the MS MARCO dev. small set are available online from the respective authors.
4.3 Candidate Re-ranking
We re-rank candidates from both BM25 and  $\mathrm{CoRT}_{\mathrm{BM25}}$  to study the impact of the candidates on a subsequent interaction-focused re-ranking. By varying the numbers of candidates, we investigate at which point adding more candidates becomes ineffective.
4.3.1 Re-ranking Model
Similar to (Nogueira and Cho, 2019), we use a simple binary classifier based on BERT. The query-passage pair  $(q,p)$  is concatenated to one token sequence of two segments. This sequence is processed by the BERT encoder while the [CLS] embedding of the last layer, which we denote with  $\phi (q,p)$ , is projected to a single classification logit. We then apply the sigmoid activation function  $\sigma$  to obtain the relevance confidence for query  $q$  and passage  $p$ . This procedure can be formalized as  $\zeta (q,p) = \sigma (W^{\prime}\phi (q,p) + b^{\prime})$  where  $W^{\prime}\in \mathbb{R}^{h\times 1}$  and  $b^{\prime}\in \mathbb{R}$  are the parameters of a linear layer with a single output activation. To form a ranking at inference time, we sort the candidates by the model's confidence. Following (Nogueira and Cho, 2019), this model is trained using a point-wise objective. We sample query-passage pairs, each associated with a binary relevance label  $y\in \{0,1\}$  and minimize the binary cross-entropy loss:
$$
l ^ {\prime} (q, p, y) = y \cdot l o g \zeta (q, p) + (1 - y) \cdot l o g (1 - \zeta (q, p))
$$
4.3.2 Re-ranking Results
As illustrated in figure 2, using  $\mathrm{CoRT}_{\mathrm{BM25}}$  as first-stage ranking appears to result in superior end-to-end ranking quality in terms of MRR@10. This is especially true, if low numbers of candidates are used. We also notice earlier saturation<sup>4</sup> of MRR@10 for  $\mathrm{CoRT}_{\mathrm{BM25}}$ , which is illustrated in Figure 2. Only 64 candidates from  $\mathrm{CoRT}_{\mathrm{BM25}}$  are sufficient to achieve top results with this re-ranker. In contrast, 256 candidates from BM25 are needed to reach the point of saturation, which translates in quadrupled re-ranking time.
Figure 2: Re-ranking quality by number of candidates for BM25 (black) and  $\mathrm{CoRT}_{\mathrm{BM25}}$  (red) on the MS MARCO passage task. Dashed lines indicate effectiveness saturation (<0.5% increase).
4.4 Impact of Representation Size