# A Study of a Gain Based Approach for Query Aspects in Recall Oriented Tasks
## 4 Experiments and Analysis
### Using Traditional Query Performance Predictors Applied to Recall Prediction for Systematic Reviews
Our hypothesis is that, while pre-retrieval predictors tend to be estimators of the recall base of a query, and therefore tend to correlate with the recall itself, post-retrieval predictors tend to compute their predictors based on the scores that the retrieval model assigns to the top-ranked documents. In this sense, post-retrieval predictors are \"top-heavy\": they focus on the upper part of the ranked list of documents. This behaviour favours predicting the performance for top-heavy measures, such as Average Precision or nDCG. Instead, our task consists in predicting the recall, given a _long_ list of documents. It is not unlikely that the \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Type** & **Predictor** & **Description** \\ \hline pre-retrieval & max-idf [27] & It considers the maximum value of the idf (inverse document frequency) over the query \\  & mean-idf [37] & It computes the mean value of the idf over the query terms \\  & std-idf [37] & It uses the standard deviation of the idf over the query terms \\  & sum-scq [28] & Measures similarity based on cf.idf to the corpus, summed over the query terms. \\  & mean-scq [28] & It relies on the same value of sum-scq, but it \\  & max-scq [28] & It relies on the same value of sum-scq, \\  & & but considers only the maximum value \\ \hline post-retrieval & wig [38] & Standard deviation of the top documents scores in the retrieval list. \\  & & Difference between the mean retrieval score of \\  & nqc [39] & the top documents, scaled by the score of the \\  & & entire corpus \\  & smv [40] & It computes the prediction considering the \\  & & standard deviation of the retrieval scores \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Pre- and Post-retrieval predictive baseline models considered. upper part of the list of retrieved documents is saturated with relevant ones; nevertheless, we are more interested in being sure that _every_ relevant document has been considered, rather than saying whether the top part of the ranked list contains relevant documents. We now switch the focus from predicting the performance _across_ topics, to predict the performance _within_ topics. Instead of comparing the performance that the standard formulation is expected to achieve for each topic, we try to sort different formulations for the same topic, according to the predicted performance. Table 3 reports the results of our analysis. Compared to the results observed in Table 2, the performance achieved by traditional predictors for the \"within\"-topics prediction, is extremely lower, with very few cases of significantly positive correlation between the predicted and observed recall. Note that, even though we agree with [13] on the fact that predicting the best query among a series of formulations of a topic is a hard task, we end up with diametrically opposite conclusions. [PERSON] et al. [13] observed severe flaws in traditional QPP techniques when _predicting the performance across topics_. On the other hand, they found the task of predicting the performance within topics (which they refer to as Query Variation Performance Prediction (QVPP)) to be easier, achieving higher (although still very low) results. What we observe here, is diametrically opposite: we found the worst results when predicting results within topics, and performance in line with previous literature for the predictions across topics. A possible explanation for this phenomenon is that we use the traditional QPP models for a different task compared to [PERSON] et al. [13]. In fact, our aim is to predict the recall, while [PERSON] et al. [13] aim at predicting the Average Precision. As a final remark, we want to point out that, [PERSON] et al. [45] recently showed how the \"QVPP\" is a harder task, compared to traditional QPP, confirming in this sense our findings.