CIKM 2020 Talk: The Impact of Negative Relevance Judgments on NDCG
Channel: webis
Date: 2020-12-07

Title: The Impact of Negative Relevance Judgments on NDCG
Authors: Lukas Gienapp, Maik Fröbe, Matthias Hagen, and Martin Potthast
Conference: CIKM 2020.
Paper: https://webis.de/publications.html#gienapp_2020c

Abstract:
NDCG is one of the most commonly used measures to quantify system performance in retrieval experiments. Though originally not considered, graded relevance judgments nowadays frequently include negative labels. Negative relevance labels cause NDCG to be unbounded. This is probably why widely used implementations of NDCG map negative relevance labels to zero, thus ensuring the resulting scores to originate from the [0, 1] range. But zeroing nega- tive labels discards valuable relevance information, e.g., by treating spam documents the same as unjudged ones, which are assigned the relevance label of zero by default. We show that, instead of zeroing negative labels, a min-max-normalization of NDCG retains its statistical power while improving its reliability and stability.