New Perspectives on Query Performance Prediction
Creators: name: Zendel, Oleg
Publisher: name: RMIT University
Year: 2024
Subjects: classificationCode: 460508
subject: Information retrieval and web search

description: Effectively navigating the vast online data landscape is crucial in our daily
lives in the digital era. The internet serves as our primary hub for work and
personal interactions, involving the consumption and production of information.
However, the overwhelming volume of online information complicates the task of
finding accurate and relevant information. Search engines are the primary means
of accessing online information and face the critical challenge of delivering
pertinent results swiftly, especially with short and occasionally ambiguous
queries. This underscores the need to ensure the retrieval of the right
information. At the core of effective search engines lies information retrieval,
providing users with relevant information in response to queries. Traditional
methods of user feedback for evaluation prove impractical due to scalability,
time constraints, and cost. This thesis addresses these challenges by delving
into information retrieval, specifically focusing on Query Performance
Prediction (QPP). QPP estimates search result quality without relying on human
judgments, offering insights to enhance search effectiveness.

Acknowledging the flaws in even well-performing search systems, there is a
growing imperative to predict search effectiveness. QPP, as explored in this
thesis, serves as a means to anticipate potential shortcomings in meeting user
information needs. The system can adapt and pivot to alternative strategies or
alert the user, enhancing its adaptability to users' preferences and information
needs. This thesis comprehensively inspects QPP from user and system
perspectives, examining diverse evaluation methods. It deepens the understanding
of QPP, identifying critical aspects and propelling the field forward. The
introduction of two distribution-based evaluation frameworks for QPP methods
with statistical analysis enhances the evaluation process.

Initiating our exploration, we investigate the interaction between query
variations and QPP from the user's perspective, aiming to understand user
assessments and expectations. Our focus includes scrutinizing users' ability to
assess the usefulness of diverse queries for a predefined information need,
employing a crowd-sourcing user study for that purpose. Notably, our study
reveals consistent user assessments of different query variants across varying
information needs, contributing to advancing our knowledge of the user
perspective towards a more user-centric information retrieval system evaluation.

Continuing our research, we categorize different information needs into
different cognitive complexity categories, testing the efficacy of Large
Language Models (LLMs) against human experts. Our findings indicate the
comparability of LLMs to human experts in classifying cognitive complexity,
contributing to the development of more automated and scalable methods for data
annotation and classification in information retrieval.

The study then scrutinizes the dynamics of QPP, investigating factors
influencing variance in prediction quality across different information needs
and query variations. We propose a new evaluation method based on pairwise
comparisons, revealing that the variance in prediction quality is primarily due
to inherent task differences rather than the introduction of query variations.
This work enhances understanding of QPP dynamics and contributes to the
development of more accurate and robust evaluation methods across different QPP
tasks.

Introducing a distribution-based framework for QPP method evaluation, we propose
a new evaluation method to generate a distribution for the predicted and actual
search results. Using this framework, we study the variance in prediction
quality across different factors in the retrieval pipeline, advancing the
development of more accurate and robust evaluation methods for QPP models.

Finally, we present an entropy-based QPP method for neural information retrieval
models, evaluating its performance thoroughly. Our findings indicate
comparability to current unsupervised state-of-the-art QPP methods, identifying
strengths and weaknesses. This research contributes to the development of more
accurate and robust QPP methods, emphasizing the potential of simple score-based
QPP methods and the importance of thorough evaluation.

In conclusion, this thesis advances research in information retrieval,
specifically QPP, by offering insights into user assessments, cognitive
complexity evaluation, key factors affecting QPP, and improved evaluation
frameworks. These contributions enhance the development of more accurate and
adaptable information retrieval systems, opening avenues for future
investigations in addressing the evolving challenges of the digital information
landscape.