Understanding Performance of Long-Document Ranking Models through Comprehensive Evaluation and Leaderboarding
Authors: Leonid Boytsov, David Akinpelu, Tianyi Lin, Fangwei Gao, Yutian Zhao, Jeffrey Huang, Nipun Katyal, Eric Nyberg


Reviewer paper summaries: This paper conducts a comprehensive evaluation of over 20 Transformer models for long-document ranking, revealing that most models show minimal improvement over a simple FirstP baseline due to a positional bias in document relevance distribution.  It introduces a new dataset, MS MARCO FarRelevant, to demonstrate the robustness of these models against such biases and highlights the limited yet significant benefits of long-context processing in certain conditions.
Summary of strengths: 1. The paper offers a broad assessment of various Transformer models on long-document ranking.
2. It identifies the impact of positional bias on retrieval performance, a critical finding for improving document ranking algorithms.
3. The creation of MS MARCO FarRelevant provides a tool for evaluating model robustness to document structure biases.
Summary of weaknesses: 1. As mentioned in the limitations section, this paper mainly focuses on cross-encoding rankers, and mainly conducts experiments on English datasets. I think these are some limitations, but these are acceptable to some extent.
2. Although this study raised the problem of position bias in long-document ranking and verified it based on a new dataset, MS MARCO FarRelevant. But given the high standards of EMNLP, I expect conclusions beyond "positional bias" regarding the long document ranking task according to comprehensive evaluation and leaderboarding.
Comments, suggestions and typos: 1. See the summary of weaknesses.
2. There is a mistake of citation on 488 line. (JINA (?))
Overall assessment scores: 3