Drowning in Documents: Consequences of Scaling Reranker Inference
Authors: name: Mathew Jacob, name: Erik Lindgren, name: Matei Zaharia, name: Michael Carbin, name: Omar Khattab, name: Andrew Drozdov

Rerankers, typically cross-encoders, are often used to re-score the documents
retrieved by cheaper initial IR systems. This is because, though expensive,
rerankers are assumed to be more effective. We challenge this assumption by
measuring reranker performance for full retrieval, not just re-scoring
first-stage retrieval. Our experiments reveal a surprising trend: the best
existing rerankers provide diminishing returns when scoring progressively more
documents and actually degrade quality beyond a certain limit. In fact, in this
setting, rerankers can frequently assign high scores to documents with no
lexical or semantic overlap with the query. We hope that our findings will spur
future research to improve reranking.