In defense of dual-encoders for neural ranking

Transformer-based models such as BERT have proven successful in information retrieval problem , which seek to identify relevant documents for a given query . There are two broad flavours of such models : cross-attention ( CA ) models , which learn a joint embedding for the query and document , and dual-encoder ( DE ) models , which learn separate embeddings for the query and document . Empirically , CA models are often more accurate , which has motivated a series of works seeking to bridge this gap . However , a fundamental question remains less explored : does this performance gap reflect a limitation in the capacity of DE models , or in the training of such models ? In this paper , we study this question , with three contributions . First , we establish theoretically that with a sufficiently large encoder size , DE models can capture a broad class of scores without cross-attention . Second , we show empirically that on real-world problems , the gap between CA and DE models may be due to the latter overfitting to the training set . To mitigate this behaviour , we propose a distillation strategy that focuses on preserving the ordering amongst documents , and confirm its efficacy on benchmark neural re-ranking datasets . 1 Introduction : transformer-based neural ranking . Information retrieval ( Mitra & Craswell , 2018 ) is the classic problem of identifying relevant documents for a given query . Typically , such retrieval is performed in a two-step manner ( Matveeva et al. , 2006 ) : one uses an initial model to efficiently retrieve a candidate set of documents for a query , and then uses a second model to re-rank these candidates . The retrieval stage is generally implemented by a model with low inference cost , for which candidate document generation is tractable ; if this candidate set is reasonably small , it is feasible to use more complex models in the re-ranking stage . Transformer-based models such as BERT ( Devlin et al. , 2019 ) have proven successful in both the retrieval and re-ranking stages . Such neural ranking models have two flavours : dual-encoder ( DE ) models ( Lee et al. , 2019 ; Chang et al. , 2020 ; Karpukhin et al. , 2020 ; Xiong et al. , 2021 ; Luan et al. , 2021 ) , which learn separate ( factorised ) embeddings for the query and document ; and cross-attention ( CA ) models ( Nogueira & Cho , 2019 ; Dai & Callan , 2019 ; Yilmaz et al. , 2019 ; MacAvaney et al. , 2019 ; Gao et al. , 2020 ) , which learn a joint embedding for the query and document . Only DE models are applicable for retrieval , as they admit efficient nearest neighbour search ( Guo et al. , 2020 ; Johnson et al. , 2021 ) to identify candidate documents . Both CA and DE models are applicable for re-ranking ; however , empirically , CA models have proven more accurate ( Hofstätter et al. , 2020a ) . Does the re-ranking performance gap between CA and DE models reflect a limitation in the inherent capacity of the latter ’ s factorised representation , or in the training of DE models ? These questions are of conceptual interest , but are also practically relevant : DE models allow for more efficient inference owing to the ability to pre-compute document embeddings , which enables fast similarity search when presented with a candidate query ( Khattab & Zaharia , 2020 ; Hofstätter et al. , 2020b ) . Several works have explored means of improving DE models , including by changing1 the scoring layer ( Khattab & Zaharia , 2020 ; MacAvaney et al. , 2020 ; Hofstätter et al. , 2020b ) , and by distilling predictions from a CA model ( Lu et al. , 2020 ; Izacard & Grave , 2020 ; Yang & Seo , 2020 ; Hofstätter et al. , 2020a ; Miech et al. , 2021 ) . However , the root cause of the gap between CA and DE models remains elusive . In this paper , we study these questions , with the following contributions : 1Unless otherwise stated , we use “ DE ” to mean a dual-encoder with dot-product scoring per Equation 2 . ( i ) we establish theoretically that with a sufficiently large embedding dimension ( and under mild assumptions ) , any continuous ground-truth scores can be modelled by generic DE models ( Proposition 1 ) , and in particular by sufficiently deep transformer-based DEmodels ( Proposition 2 ) . Thus , in principle , there is no fundamental restriction in using DE versus CAmodels for re-ranking . ( ii ) we show empirically ( Figure 1 ) that on real-world problems , CA and DE models can achieve similar training performance , but DE models may do worse on test data due to overfitting ( §3.2 ) . Thus , the DE models may suffer due to poorer generalisation ability rather than capacity . ( iii ) to mitigate the above , we propose a distillation strategy focussing on mimicking the teacher ’ s ordering amongst documents ( §4 ) . This generalises the recent margin MSE loss ( Hofstätter et al. , 2020a ) ( §4.1 ) , and justifies the utility of softmax cross-entropy based distillation for re-ranking ( Proposition 3 , §4.2 ) . We confirm the efficacy of this strategy on the re-reranking benchmarks MSMARCO-Passage ( Nguyen et al. , 2016 ) and Natural Questions ( Kwiatkowski et al. , 2019 ) . Overall , our results aim to shed light on the fundamental differences between CA and DE models , and give a simple yet effective distillation strategy to boost the performance of the latter . 2 Background and notation . Retrieval problems . Suppose we have query space Q and document space D. For a given query q ∈ Q , the goal of information retrieval is to identify a set of relevant documents Drel ( q ) ⊆ D ( Mitra & Craswell , 2018 ) . This is typically achieved by learning a scorer s : Q × D → R that predicts the relevance of a query and document , and simply selecting the top-k highest scoring documents for a given query . Typically , this scorer is itself implemented in two phases ( Matveeva et al. , 2006 ; Nogueira & Cho , 2019 ; Chang et al. , 2020 ; Miech et al. , 2021 ; Hofstätter et al. , 2021 ) . In the retrieval phase , one identifies an initial candidate set of documents via an initial scorer sret . In the re-ranking phase , one re-scores only these candidate documents via a distinct scorer srrk to obtain Drel ( q ) . We may thus regard s : Q×D→ R as a suitable composition of both sret and srrk . To learn a scorer , suppose we have supervision { ( qn , dn , yn ) } Nn=1 , where for each query qn we have a list of K associated documents dn . = ( dn1 , . . . , dnK ) ∈ DK with ground-truth relevance labels yn . = ( yn1 , . . . , ynK ) ∈ { 0 , 1 } K . Typically , dn comprises a small subset of D ( i.e. , K |D| ) , since it is rarely feasible to collect relevance labels for all documents . Consider a parameterised class of scorers S .= { s ( · , · ; θ ) : Q × D → R | θ ∈ Θ } . For any given θ ∈ Θ , let sn ( θ ) . = ( s ( qn , dn1 ; θ ) , . . . , s ( qn , dnK ; θ ) ) ∈ RK denote the vector of model scores on the provided documents . One may then learn a scorer s via minimising R ( θ ) .= 1N ∑ n∈ [ N ] ` ( yn , sn ( θ ) ) for loss ` : RK × RK → R+ , such as the mean square error ` mse ( yn , sn ( θ ) ) . = ∑ k∈ [ K ] ( ynk − snk ( θ ) ) 2 ; and the softmax cross-entropy ( assuming for simplicity 1 > y = 1 ) : for temperature τ > 0 , ` ce ( yn , sn ( θ ) ) . = − ∑ k∈ [ K ] ynk · log pnk ( θ ) where pnk ( θ ) . = exp ( τ−1 · snk ( θ ) ) ∑ k′∈ [ K ] exp ( τ −1 · snk′ ( θ ) ) . ( 1 ) To instantiate either objective , one missing ingredient is the precise parameterisation of S. We now describe one possible parameterisation , given by a transformer model . Cross-attention and dual-encoders . Transformers ( Vaswani et al. , 2017 ) are sequence-to-sequence models with good empirical performance on language tasks . Suppose we have a sequence ( x1 , . . . , xL ) of L tokens ( e.g. , words in a language ) , with X .= ( x1 , x2 , . . . , xL ) ∈ RD×L being a corresponding sequence of token embeddings ( e.g. , word embeddings ) . A transformer encoder maps this sequence to Z .= ( z1 , z2 , . . . , zL ) ∈ RD×L , via a composition of attention and feedforward layers . The former is a mapping attn : RD×L → RD×L , which in its simplest ( “ single head ” ) form is attn ( X ) = X + WVXA , where A . = σ ( X > WQWKX/ √ D ) for softmax σ ( · ) and weights WQ , WK , WV . Intuitively , this embeds a token via a weighted sum of all tokens ’ embeddings . The weighting is given by the attention matrix A ∈ RL×L , which is a learned similarity measure between pairs of tokens . BERT ( Devlin et al. , 2019 ) is a canonical instantiation of the transform encoder , coupled with effective pre-training tasks shown to improve downstream performance in small-sample regimes . Transformers can readily score ( query , document ) relevance : given a pair ( q , d ) ∈ Q×D of tokenised queries and documents , one may pass their concatenation through a transformer encoder T . The resulting embedding may be pooled ( e.g. , using the [ CLS ] token ( Devlin et al. , 2019 ) ) to yield s ( q , d ) = w > pool ( T ( concat ( q , d ) ) ) for weights w ∈ RD . Such cross-attention ( CA ) models apply attention layers on the queries and documents jointly , which intuitively allow for rich interactions . A distinct strategy is to separately embed queries and documents ( e.g. , via BERT encoders ) , and score them with their dot-product ( Lee et al. , 2019 ; Chang et al. , 2020 ; Karpukhin et al. , 2020 ; Zhan et al. , 2020 ; Ma et al. , 2021 ; Xiong et al. , 2021 ; Luan et al. , 2021 ; Qu et al. , 2021 ; Zhan et al. , 2021a ) : s ( q , d ) = pool ( T ( q ) ) > pool ( T ( d ) ) . ( 2 ) Such dual-encoder ( DE ) models have their roots in Siamese networks for image processing ( Bromley et al. , 1993 ) , and have received recent interest in a range of language problems ( Reimers & Gurevych , 2019 ; Gillick et al. , 2019 ; Karpukhin et al. , 2020 ) . A salient feature of Equation 2 is that one can index the query and document embeddings . This implies that DE models can have significantly lower the inference cost compared to CA models , which makes them appealing to use for both the retrieval and re-ranking stages . Indeed , DE models are widely used for retrieval , owing to their amenability to fast approximate nearest neighbour search ( Guo et al. , 2020 ; Johnson et al. , 2021 ) . Unfortunately , for re-ranking , one typically observes a large gap between DE and CA performance ( see §3.2 ) . Cross-attention to dual-encoder distillation . One strategy to bridge the performance gap between CA and DE models is distillation ( Ba & Caruana , 2014 ; Hinton et al. , 2015 ) . Suppose we have supervision { ( qn , dn , tn , yn ) } Nn=1 , where tn ∈ RK are the scores from a “ teacher ” ( e.g. , CA ) model . Then , we may minimise Rdist ( θ ) . = 1N ∑ n∈ [ N ] ` ( tn , sn ( θ ) ) , noting that we use the teacher supervision tn rather than “ one-hot ” labels yn ; one may also combine the teacher and one-hot labels . Two canonical instantiations are logit matching ( Ba & Caruana , 2014 ) , which uses the the mean square error ` mse ( tn , sn ( θ ) ) between the teacher and student scores , and probability matching ( Hinton et al. , 2015 ) , which uses the softmax cross-entropy between the teacher and student softmax probabilities , i.e. , ` ( tn , sn ) = ` ce ( p teach n , sn ) where pteachnk . = exp ( τ −1·tnk ) ∑ k′∈ [ K ] exp ( τ −1·tnk′ ) for temperature τ > 0 . Empirically , distillation has been shown to at least partially bridge the chasm between CA and DE performance ( Lu et al. , 2020 ; Izacard & Grave , 2020 ; Yang & Seo , 2020 ; Hofstätter et al. , 2020a ; Miech et al. , 2021 ) .

Summary: This paper investigate the gap problem between cross-attention (CA) and dual-encoder (DE) models for document reranking. In general, cross-attention models perform much better than dual-encoder models while consuming more computational cost. The authors first leverage Mercer's theorem to prove that dual-encoders models are sufficiently expressive to model a broad class of scores using countably infinite dimension, which mainly the gap between CA and DE models should not exist from theoretical analysis. They further identified the gap is caused by the generalization ability of the DE models. To reduce the overfitting on the training data, they proposed to use distillation algorithms. Specially, they design the loss function which matches the margins between the teacher (CA) and student (DE) models. They also extend the margin to probability matching by using softmax cross-entropy loss. Through the experiments, they demonstrated that their proposed distillation approaches can further reduce the gap between DE and CA models.  The main contributions of this paper is that they show there should not be a gap between DE and CA models from theoretical analysis. They found that the gap is caused by the generalization ability of CA model and proposed distillation approach to reduce the gap.