Length Normalization in the Era of Neural Rankers

Document length was rarely a factor in traditional retrieval evaluation metrics. As a result, traditional rankers could take advantage of this property by favoring longer documents, which were more likely to be relevant. Returning a long, non-relevant document was in no way penalized by these traditional metrics, even if in practice the searcher would spend time fruitlessly scanning the document for relevant material. As we show by way of an illustrative experiment, a policy of ignoring document length may have had unfortunate impacts even in traditional contexts. But this policy becomes increasingly inappropriate in an era of neural rankers, where query-document similarity may be computed directly from text. Freed from ranking based on documentand corpus-level statistics, it should be possible to return precisely the required information and no more. Future evaluation efforts and metrics should reflect this goal. 1. TRADITIONAL EVALUATION Traditional retrieval evaluation metrics, including AP, MRR, NDCG, RBP, ERR, and others, treat all documents equally, regardless of length. In traditional test collections, a relevance judgment applies to a document as a whole, and if a document includes any relevant material at all, the entire document is judged relevant for the purpose of these metrics. Even when graded relevance judgments are available, these relevance grades do not explicitly consider document length. Although it might the case that a long document containing a small amount of relevant material will receive a lower grade than a similarly sized document dedicated to the topic of the query, these grades reflect only the value of relevant material to the searcher, and fail to reflect the time the searcher might waste viewing non-relevant material. It has long been known that in many traditional test collections longer documents are more likely to be relevant [25]. This observation, together with the length blindness of traditional evaluation metrics, encourages traditional rankers to effectively assign a larger prior to longer documents, even though retrieving a long but non-relevant document may waste a searcher’s time. For example, the BM25 [22] ranking formula still serves as a frequent baseline in current research and as a standard feature in learned rankers. This formula explicitly incorporates a length normalization parameter (b) which works in EVIA 2019, June 10, 2019, Tokyo, Japan. concert with a term saturation parameter (k1) to adjust document scores to account for document length. When tuned on traditional test collections, the value of these parameters will reflect a bias for longer documents. By assigning equal value to all documents, regardless of length, traditional evaluation metrics both fail to appropriately penalize rankers for returning long non-relevant documents, and fail to appropriately reward them for returning long relevant documents. While a longer relevant document may indeed may be more useful to a searcher than a shorter relevant document, since it may treat the associated topic at greater depth, neither this potential value, nor the associated risks, are recognized by traditional evaluation metrics. 2. FROM THERE TO HERE All of these observations and concerns mattered little during the 18-year hiatus between 1998 and 2016, when relatively modest progress was made on pure content-based ranking [3, 29], i.e., rankers that use only the human language in the document itself as the source of ranking features. During this era some progress did come through exploiting internal document features, such as fields [20] and proximity[16], as well as through new theoretical insights [14, 32], and improvements to pseudo-relevance feedback [1]. But most progress came through the user of externally computed document properties, such as link analysis [5] and user behavior [2], as well as through the development of learned rankers designed to extract benefits from large numbers of carefully engineered features [7]. During this era, research attention was largely focused on Web search, where searcher intent is often navigational or transactional in nature, and document length is less of a consideration [6]. Over this period, some consideration was given to how document length relates to relevance from a theoretical perspective. In particular, Fang et al. [11] define several lengthrelated constraints for retrieval models. One constraint, called LNC2, suggests that “the score of a document should decrease if we add an extra occurrence of a ‘non-relevant word’ (i.e., a word not in the query)”.