Continuous Evaluation in Information Retrieval Across Methods and Time
Authors: Jüri Keller
Venue: SIGIR
Year: 2025

Evaluating Information Retrieval (IR) systems is essential yet challenging. The dynamic nature of information and relevance further complicates IR. For example, Adar et al. observed that as early as 2009, websites frequently changed multiple times per hour. In response, recent IR systems have become more personalized, semantic, and context-aware. They evolved from lexical ranking functions to complex neural models embedded in feature-rich systems. While this often improves retrieval quality, it also challenges their reliability and robustness. The proposed research is motivated by the overarching goal of maintaining the trustworthiness of IR systems. To do so a rigorous evaluation is needed. Only by assessing the quality of a system can it be maintained and improved. Such an evaluation can not take place in isolation but must consider the dynamics of the search setting at all stages of a system-from development to maintenance and improvement. Based on the CRISP-DM methodology, these stages are sketched out as an ''IR Life Cycle'', which we will further define and oppose with a continuous evaluation framework in future work. In this regard, the changing search setting and the alignment between evaluation methods are key problems to be investigated. The dynamic nature of the search setting makes it difficult to compare evaluation outcomes from different points in time. As systems and testbeds evolve, a common ground for comparison becomes elusive. Furthermore, when evaluations are conducted under dynamic conditions, their temporal validity becomes limited once the testbeds no longer match the current search setting. This calls into question how temporally robust evaluations are. Therefore, we aim to systematically reintroduce changes into evaluations and to identify synergies among evaluation methods. This will be addressed along the following research areas: Classifying changes: Initially, a formal definition of the search setting and its dynamics is created. Accordingly, we will examine how search settings and systems evolve. To track the effectiveness over time, especially the relation between observable queries and latent information needs, need to be investigated. Comparing results: The changing setting makes a direct comparison of retrieval results difficult. To retain expressible and interpretable results and attribute the measured effects, we explored comparison strategies as a reproducibility problem. Repeating evaluations: When the search setting deviates from the testbed, its validity becomes uncertain, and systems need to be re-evaluated. By estimating this uncertainty, it should be assessed if it is worth to update the testbed and re-evaluate the system. Aligning evaluation methods: To improve the feasibility of repeated evaluations, we aim to align evaluation methods to identify alternatives for demanding components and to uncover synergies between complementary components. In summary, it is intended to scrutinize the abstractions made by conventional IR evaluations and assess systems from multiple perspectives to gain a more holistic understanding of IR systems.