Query Performance Prediction for Conversational Search and Beyond
Authors: Chuan Meng
Venue: SIGIR
Year: 2024

Query performance prediction (QPP) is a key task in information retrieval (IR) [1]. The QPP task is to estimate the retrieval quality of a search system for a query without human relevance judgments. In summary, I aim to solve 4 limitations identified in previous QPP studies: I have published 3 papers that address 3 of these limitations, while the remaining one is the focus of my future work. While extensively explored for traditional ad-hoc search, QPP for conversational search (CS) [4] has been little studied. I have identified limitation 1 in previous QPP studies: There is a lack of a comprehensive investigation into how well existing QPP methods designed for ad-hoc search perform in the context of CS. To fill this research gap, I have conducted a comprehensive reproducibility study [5], where I examined various QPP methods that were designed for ad-hoc search in the CS setting. I have made the code and data publicly available on https://github.com/ChuanMeng/QPP4CS. Moreover, I have identified limitation 2 in previous studies on QPP for CS: There is a lack of research in investigating and leveraging the CS-specific features that do not exist in ad-hoc search to improve QPP quality for CS. I have authored a paper to fill this research gap [3]. Specifically, my empirical analysis indicates a correlation between query rewriting quality in CS and the actual retrieval quality. Based on this finding, I have proposed a perplexity-based pre-retrieval QPP framework (PPL-QPP) for CS, which integrates query rewriting quality into existing QPP methods. Experimental results show that PPL-QPP improves QPP quality. Beyond the scope of QPP for CS, I have identified drawbacks in general QPP methods. Existing QPP methods typically return a single scalar value that indicates the retrieval quality, which results in two issues: (i) relying on a single value to represent different IR metrics leads to a "one size fits all" issue, and (ii) a single value constraints the interpretability of QPP. Thus, I have identified limitation 3: there is a shortage of QPP methods that are capable of effectively predicting various IR evaluation metrics while maintaining interpretability. To address the limitation, I have proposed a QPP framework using automatically generated reevance judgments (QPP-GenRE); it decomposes QPP into independent subtasks of judging the relevance of each item in a ranked list to a given query [6]. QPP-GenRE enables the prediction of any IR metric using generated relevance judgments as pseudo-labels, and enables the interpretation of predicted IR metrics based on generated judgments. I have fine-tuned an open-source large language model (LLM) for judging relevance. Experimental results show that QPP-GenRE achieves state-of-the-art QPP quality; my fine-tuned LLM demonstrates a high relevance judgment agreement with human assessors. I have made the code and data publicly available on https://github.com/ChuanMeng/QPP-GenRE. As part of my future work, I plan to solve limitation 4: No study has explored the application of QPP in retrieval-augmented generation (RAG) to predict when not to rely on low-quality retrieved items that have the potential to hurt RAG's text generation.