# A Study of a Gain Based Approach for Query Aspects in Recall Oriented Tasks
## 4 Experiments and Analysis
### Using Traditional Query Performance Predictors Applied to Recall Prediction for Systematic Reviews  
To have a better grasp on the peculiarities of the problem, we first try to apply traditional techniques of Query Performance Prediction (QPP) to our specific setting. We aim at showing that, traditional QPP techniques fail to correctly order formulations when (i) the recall is the key performance indicator; (ii) we sort formulations of the same topic and not queries representing different topics. Showing this, is a further evidence of the importance of using appropriate tools, such as the _gain_ as described in Section 2 to correctly tackle the problem. More in detail, we select a set of very well-know QPP models, in order to determine whether they can be satisfactory applied to the prediction of the recall and can be used with the documents and queries that we have at hand. Traditionally, Query Performance Predictors are divided into two macro-categories, according to the information they exploit to formulate the prediction: Pre-retrieval predictors and Post-retrieval Predictors. _Pre-retrieval predictors_ analyze query and corpus statistics prior to retrieval [14; 23; 24; 25; 26; 27; 28] and _post-retrieval predictors_ that also analyze the retrieval results [15; 29; 30; 31; 32; 33; 34; 35; 36]. Even though Pre-retrieval predictors have the advantage of being faster, since they do not need to retrieve the documents for a certain run, post-retrieval predictors typically perform better. Table 1 reports the predictors that we include in our analyses and a brief description of how they work. It is important to notice that, as for many QPP models, the models that we selected do not actually predict the performance measure. They associate a score to each of the queries, which is expected to correlate with the performance measure, but is on a different scale and cannot be used directly as estimate of the performance. The traditional strategy to evaluate how good a query performance predictor is, consists in computing a traditional retrieval performance measure, such as Average Precision (AP), for each of the query, and determine how much such measure correlates with the prediction scores computed by the QPP model [23; 24; 25; 26; 28; 29; 30; 31; 32; 33; 34; 35; 38; 41; 42; 43]. Notice that, there are two main aspects that might impair traditional QPP models in our specific setting:  
* Remember that we are in the setting of the systematic reviews. Therefore, it is by far more important to retrieve as many as possible relevant documents, rather than putting them in the first positions. Therefore, we are not interested in estimating the AP, which is a precision based measure, but our aim is to predict which query will have the best recall;
* We do not compare queries meant for different information needs, which is the typical evaluation scenario for QPP models. On the other hand, we aim at understanding which one, among a set of queries representing the same information need, achieve the best result. To determine whether we are impaired by the first problem, we first apply the traditional QPP considering only the default formulation of each topic, and we compare whether the predictors are capable of correctly determining the inter-topic performance. More in detail, with this first experiment, we are interested in understanding whether the baseline predictors are capable of predicting which _topic_ will have the best recall, using a single formulation for each of them. Table 2 reports the result of such analysis. We can observe that, by looking at Table 2, the results are in line with previous similar experiments in the literature, such as [5; 44]. Almost all the predictors are able to achieve a significant correlation with the recall (with level \(\alpha=0.01\)). Two noticeable exceptions are represented by nqc and smv: traditionally, they are considered among the best predictors, but in this specific scenario they fail, with correlations not statistically different from 0.