Building new test collections for Information Retrieval evaluation
Channel: IIIA Hub
Date: 2022-06-07

David Otero Freijeiro, University of A Coruña, Spain is visiting our research group and will give a talk entitled:

Building new test collections for Information Retrieval evaluation

## Abstract: ##
Reproducible offline evaluation of retrieval systems depends on test collections. These benchmarks comprise topics, documents, and relevance judgements indicating which documents should be retrieved for each topic. Gathering the latter is costly since expert assessors must review the documents. For this reason, judges only see a portion of the whole corpus. The most common approach for selecting that subset is through pooling. Pooling has allowed the construction of large benchmarks at a low cost. However, the proliferation of new tasks and the need to scale offline evaluation require new and larger test collections. The construction of those datasets poses some challenges to traditional methods. In this talk, we will cover some of these problems and how we can tackle them. In particular, we will see what we can do when we need to release annotated training data before building the collection or in scenarios where only few participants are available. Additionally, we will also see how we can obtain robust and reusable judgements while reducing the assessor effort using pooling methods based on multi-armed bandits.