Normalised Precision at Fixed Recall for Evaluating TAR

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-26 13:17:56

Abstract: A popular approach to High-Recall Information Retrieval (HRIR) is Technology-Assisted Review (TAR), which uses information retrieval and machine learning techniques to aid the review of large document collections. TAR systems are commonly used in legal eDiscovery and medical systematic literature reviews. Successful TAR systems are able to find the majority of relevant documents using the least number of manual assessments. Previous work typically evaluated TAR models retrospectively, assuming that the system achieves a specific, fixed Recall level first and then measuring model quality (for instance, work saved at r\% Recall).

This paper presents an analysis of one of such measures: \emph{Precision at r\% Recall (P@r\%)}. We show that minimum Precision at r\% scores depends on the dataset, and therefore, this measure should not be used for evaluation across topics or datasets. We propose its min-max normalised version ($nP@r\%$), and show that it is equal to a product of TNR and Precision scores. Our analysis shows that $nP@r\%$ is least correlated with the percentage of relevant documents in the dataset and can be used to focus on additional aspects of the TAR tasks that are not captured with current measures. Finally, we introduce a variation of $nP@r\%$, that is a geometric mean of TNR and Precision, preserving the properties of $nP@r\%$ and having a lower coefficient of variation.

Main paper: 
 introduction: 
 High-Recall Information Retrieval (HRIR) focuses on identifying nearly all relevant documents within a given collection. Technology-Assisted Review (TAR) is a prevalent method in HRIR that combines information retrieval and machine learning techniques to enhance the review of large document sets. The primary objective of TAR is to augment human effort by automating routine tasks and prioritising documents for review, thereby saving time and resources for organisations.Citation screening for systematic literature reviews is a key application of TAR [20, 22, 23, 32] . In this task, researchers screen a large number of publications initially identified through a literature search to determine those relevant to the review. This process, traditionally manual, is time-consuming and demands extensive effort, involving numerous eligibility decisions. Other TAR applications include legal electronic discovery [11, 40] or constructing evaluation collections [28] . Initiatives such as TREC Legal [2, 11, 31, 36] , TREC Total Recall [16] , and CLEF eHealth TAR [20] [21] [22] , have facilitated HRIR research by providing datasets and standardised evaluation methods.A critical metric for HRIR systems is Recall, indicating the fraction of relevant documents retrieved. TAR aims to maximise relevant document identification (True Positives, 𝑇 𝑃) while minimising the inclusion of irrelevant ones (False Positives, 𝐹 𝑃). By decreasing 𝐹 𝑃 counts, TAR systems enhance efficiency for reviewers. Nonetheless, implementing TAR requires care, as subpar performance can lead to legal repercussions, personal liability, and financial losses, especially in legal discovery contexts [14] .Various evaluation measures have been proposed to assess the effectiveness of TAR systems [37] . One prevalent approach is to evaluate the system at a fixed Recall level. This approach has been popularised by methods measuring work saved compared to the random ordering of documents (e.g., Work Saved over Sampling, 𝑊 𝑆𝑆@𝑟 % [9] ) and by counting True Negatives at an r% Recall (𝑇 𝑁 𝑅@𝑟 %) [25] . Evaluating TAR systems at a fixed Recall level aids in determining the trade-off between Precision and Recall. Traditionally, this has been particularly useful under the assumption that a minimum acceptable level of Recall exists for a task.As the number of potential applications for TAR grows, so too does the need for enhanced evaluation techniques. In this paper, we examine one of the measures used for evaluating TAR systems: Precision at r% Recall (𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑟 %, 𝑃@𝑟 %) [23, 26] . We find that it does not fulfil the zero Axiom #3 introduced by Busin and Mizzaro [5] . To address this limitation, following the approach of the nDCG measure [19] , we propose to min-max normalise it.Our contributions are as follows:• We analyse the Precision at r% Recall measure and propose a min-max normalised Precision at r% (𝑛𝑃@𝑟 %), equating to the product of 𝑃@𝑟 % and 𝑇 𝑁 𝑅@𝑟 %. • We conduct experiments to investigate the differences in evaluations and rankings using 𝑛𝑃@𝑟 % compared to other TAR metrics. We show that 𝑛𝑃@𝑟 % is the least correlated with the percentage of relevant documents in datasets among considered metrics. • We introduce 𝑠𝑛𝑃@𝑟 %, a geometric mean of 𝑇 𝑁 𝑅 and Precision, preserving the properties of 𝑛𝑃@𝑟 % and having lower coefficient of variation.We first briefly describe Technology-Assisted Reviews. Then, we propose an analytical formulation of normalised Precision at a Recall rate. Finally, we conduct experiments to compare 𝑛𝑃@𝑟 % and 𝑠𝑛𝑃@𝑟 % with other popular TAR measures. The source code for our experiments is publicly available. 1  
background: 
 All TAR automation models can be coarsely categorised into prioritisation (ranking) or classification approaches [32] . An effective TAR algorithm aims to maximise the number of relevant documents found and save the reviewers' time by removing irrelevant documents.When treating the TAR as a ranking task (e.g., for the sub-task of screening prioritisation or stopping prediction), then rank-based measures and measures at a fixed cut-off are commonly used, e.g., 𝑛𝐷𝐶𝐺@𝑛, 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑛, 𝑅𝑒𝑐𝑎𝑙𝑙@𝑛, R-Precision [16] , and last relevant found.When TAR is treated as a classification task, measures based on the confusion matrix and the notion of Precision and Recall are commonly used [32, 37] . Aside from Precision and Recall, measures include variations of the harmonised mean between the two, i.e., F 𝛽 -score, Yield, Burden [39] , Utility 𝛽 [38] , sensitivity-maximising thresholds [12] , and AUC [8] . Another measure, Work Saved over Sampling (𝑊 𝑆𝑆), measures the amount of work saved when using machine learning models to screen irrelevant publications [9] . The True Negative Rate (𝑇 𝑁 𝑅) was proposed as an alternative as it addresses some of the limitations of 𝑊 𝑆𝑆 regarding averaging scores from multiple datasets [25] . Retrospectively evaluating models at different levels of Recall takes into account the number of relevant documents found and the trade-off between reviewing more documents and potentially finding more relevant ones, versus stopping the review and potentially missing some relevant documents.Recall versus effort plots using the knee method [10] have been proposed as a more generalised extension, plotting the scores over the full range of values of Recall. However, these methods, similarly to the ROC curve do not provide users with a single number score, which might be crucial for some users.Yang et al. [41] proposed a mathematical model that predicts how varying document and reviewer costs affect total TAR workflows. However this framework focuses on cost modelling for reviewing one specific query.Previous work used Precision at r% Recall as an evaluation measure for automated citation screening algorithms [23, 24] :𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑟 % = 𝑃@𝑟 % = 𝑇 𝑃 𝑇 𝑃 + 𝐹 𝑃 , when 𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑟 % (1)Researchers used 𝑃@𝑟 % to evaluate also other tasks like classification [7, 30, 33] or object detection [17] . Another application was mining user query logs to refine component description [29] . In the medical and healthcare domains, 𝑃@𝑟 % is referred to as "PPV at sensitivity level" and has been used for evaluation in several other works, such as in [3, 4, 6, 13, 18] .Consider an example scenario in which a search for a review returns a collection of 𝑁 = 2, 000 documents. Of these, 200 are relevant to the study and should be included in the final review (we 1 https://anonymous.4open.science/r/normalised-precision-at-recall-D246 call these ground truth relevant items includes, I = 𝑇 𝑃 + 𝐹 𝑁 ), while the remaining 1, 800 are irrelevant and should be excluded (we call them excludes, E = 𝑇 𝑁 + 𝐹 𝑃 ). In manual screening, annotators must review all 2, 000 documents to identify only the 200 relevant ones. In the case of TAR systems, we consider that some of these irrelevant documents will be correctly identified by the model.The domain and characteristics of the review influence the choice of Recall level. Past studies on the automation of citation screening in medicine typically used 95% Recall as the threshold to preserve a satisfactory quality of the systematic literature review in medicine [9] . In other technology-assisted review domains, Recall levels might be lower, for instance, in eDiscovery, a commonly used Recall is 80% [40, 42] . Sometimes the choice of Recall is influenced by the time or money limitations of the task. 
normalised precision at 𝑟 % recall: 
 Defining a Recall level for assessing TAR systems assumes that the number of true positive and false negative documents remains constant. Achieving a specific 𝑟 % Recall assumes that exactly (1 -𝑟 )% of documents that should be included will be misclassified. Therefore, for a specific 𝑟 % Recall, the number of True Positives (𝑇 𝑃) and False Negatives (𝐹 𝑁 ) will be equal to:EQUATIONEQUATIONThis means that these terms will also be a constant for every model for the same dataset. For instance, from the example in the previous section, a Recall of 95% is achieved when the model accurately identifies 190 relevant documents (𝑇 𝑃) and misclassifies the remaining 10, i.e., these are False Negatives (𝐹 𝑁 ). The Precision of the model depends on the number of False Positives (𝐹 𝑃), which can range from zero (best score) to the number of all excludes (|𝐸|, worst score). Using the above equations, we can define maximum and minimum Precision@r% values as follows:EQUATIONEQUATION)Maximum Precision@r% value will always be equal to 1. However, the minimum Precision value, similarly to WSS measure [25] , depends on the I/E ratio of the dataset:EQUATIONlimEQUATION)For datasets highly imbalanced towards the negative class, the minimum value of P@r% will be close to 0. On the other hand, with a growing presence of the positive class, the minimum value of P@r% will be growing towards 1.Busin and Mizzaro [5] introduced an axiomatic approach to IR evaluation measures proposing eight axioms that every effectiveness metric should satisfy. Axiom #3 (Zero and maximum) states: "An effectiveness metric should have a true zero in 0 and a maximum value 𝑀. The theoretically worst (best) performances ⊥ should give 0 (𝑀) as the metric value. As a normalisation convention let 𝑀 = 1 such that ∀ metric, 𝑟𝑎𝑛𝑔𝑒 (metric) = [0, 1], metric(𝛼, 𝛼) = 1, and metric(𝛼, ⊥) = 0."The minimum Precision value, depending on the class imbalance, violates the aforementioned Axiom #3. This becomes crucial, especially in retrieval tasks, where the scores are almost always averaged across several topics or datasets. 𝑃@𝑟 % is favouring those models underperforming on easier topics, which consequently narrows the gap between good and poor models. Therefore, we argue that this measure should not be employed for such evaluations. To address this problem and facilitate averaging across datasets, we propose defining a min-max normalised version of 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑟 % (𝑛𝑃@𝑟 %):EQUATIONwhere the following equation can be resubstituted as:𝑛𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑟 % = 𝑇 𝑃 𝑇 𝑃 + 𝐹 𝑃 • 𝑇 𝑁 𝑇 𝑁 + 𝐹 𝑃 = 𝑃@𝑟 % • 𝑇 𝑁 𝑅@𝑟 % (9)𝑛𝑃@𝑟 % = 𝑃@𝑟 % • 𝑇 𝑁 𝑅@𝑟 %.(10) Equation (10) shows that 𝑛𝑃@𝑟 % is interconnected with Precision and True Negative Rate. Interestingly, both measures relate to type I error (𝐹 𝑃). Achieving high normalised Precision requires a balance between identifying relevant documents (Precision) and disregarding irrelevant ones (Specificity). This relationship can be important for evaluating and improving information retrieval models, especially in contexts of high-recall search tasks.As Precision scores tend to have high variance in comparison with other measures, we propose to further introduce a variation of the 𝑛𝑃@𝑟 % which is a geometric mean of its components: 𝑠𝑛𝑃@𝑟 % = √︁ 𝑛𝑃@𝑟 % = √︁ 𝑃@𝑟 % • 𝑇 𝑁 𝑅@𝑟 % (11) By introducing the square root, we intend to decrease the influence of Precision. The formulation in Equation ( 11) is analogous to the Fowlkes-Mallows index [15] , a clustering similarity measure, where the 𝑇 𝑃𝑅 term would be replaced with 𝑇 𝑁 𝑅. 𝑠𝑛𝑃@𝑟 % also preserves the zero Axiom. 
experiment setup: 
 To assess the importance of our findings, we conduct experiments comparing 𝑛𝑃@𝑟 % scores (also abbreviated as 𝑛𝑃 in subsequent sections) with other measures. We select the task of ranking documents for a systematic review search. We conduct the experiments using 100 systematic reviews (topics) from the CSMeD-Cochrane-dev benchmark [27] . CSMeD-Cochrane is a meta-dataset combining five different test collections [1, [20] [21] [22] 35] . CSMeD-Cochrane is the most extensive collection of systematic reviews used to evaluate document screening algorithms.We select this dataset due to its extensive coverage of topics and public availability of baseline runs. 2 We reuse runs described in the original CSMeD paper, which includes five different models: two statistical models (BM25 and TF-IDF), and three Transformer-based models (MiniLM-L6-v23 , MPNet-base-v2 4 and BioBERT-snli 5 ) from the SentenceTransformers library [34] . Each of the five models uses four different systematic review meta-data as input query representations: 'title', 'abstract', 'eligibility criteria' and 'search strategy'. This configuration results in a total of 20 different combinations of runs.We re-evaluate the runs at the Recall level of 95%, using 𝑛𝑃, 𝑠𝑛𝑃 and the two measures that are part of the equation: 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 and 𝑇 𝑁 𝑅. We also calculate other standard TAR evaluation measures: Mean Average Precision (𝑀𝐴𝑃) and average position at which the last relevant item is found calculated as a percentage of the dataset size (𝐿𝑎𝑠𝑡𝑅𝑒𝑙) [20] . We intentionally refrain from using 𝑊 𝑆𝑆 measure as previous work highlighted its limitations and demonstrated that 𝑊 𝑆𝑆 is a special version of 𝑇 𝑁 𝑅 [25] . 
results and discussion: 
 We first look at correlation between 𝑛𝑃 and other measures. Then we investigate the change in run rankings for each measure and finally we evaluate the impact of different levels of Recall.Table 1 presents correlations between measures using Spearman's method. There is a moderate correlation between 𝑛𝑃@95% (and 𝑠𝑛𝑃@95%) and all other measures (between .655 and .533). Especially between 𝑃@95% and 𝑇 𝑁 𝑅@95% correlations are comparable meaning a comparable influence of both components of the equation. Interestingly, 𝑛𝑃@95% (and 𝑠𝑛𝑃@95%) exhibits the weakest, almost negligible, correlation between percentage of relevant examples, in contrast to all other considered measures. We also measure a correlation with a dataset size defined as a total number of documents found by a search query (|E | + |I|). We find that 𝑛𝑃@95% shows a weaker correlation to dataset size when compared to MAP. This difference highlights that 𝑛𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 focuses on distinct aspects of the screening task. Detailed plots presenting correlations between 𝑛𝑃@95% and 𝑃@95% and 𝑇 𝑁 𝑅@95% are in Appendix A.Figure 1 presents presents the coefficient of variation (CV) in evaluation measure scores between topics as depicted through violin plots for normalised measures. 𝑛𝑃@95% shows a high variance Table 1 : Correlation matrix of selected metrics calculated using Spearman's method. 𝑛𝑃@95% and 𝑠𝑛𝑃@95% have identical correlation coefficients.nP@95% snP@95% P@95% TNR@95% LastRel MAP P@95% 0.602 1.-0.027 0.140 0.910 TNR@95% 0.655 -0.027 1.- Variance Figure 1 : Coefficient of variation in evaluation measure scores between topics presented as violin plots for normalised measures.as their mean CV from 20 runs is equal to 1.5. This behaviour is influenced by Precision, which disproportionately favours betterperforming systems. 𝑇 𝑁 𝑅 and 𝑀𝐴𝑃 exhibit comparably lower variances, might be considered better metrics for discriminating between good and bad systems, as they show less sensitivity to variations across different queries. However, we observe that the mean CV for 𝑠𝑛𝑃@95% falls within the range of the mean CV for MAP, which is considered a reliable evaluation measure. This validates our assumption to use the geometric mean for reducing the impact created by the high variance in Precision.We can observe that the ordering of run changes when different metrics are applied (see Table 2 ). Especially, 𝑛𝑃@𝑟 % offers a different perspective for ordering when contrasted with all other measures and, especially with the incorrect usage of 𝑃@𝑟 %. However, these differences are not statistically significant for the top 10 runs. We hypothesise it is due to the large collection size, which contains various topics of very different types and characteristics. Analysis on a larger number of datasets and models could enhance these findings. The 𝑛𝑃 measure is sensitive to changes in both Precision and TNR, and the trend in 𝑛𝑃 indicates that it is likely being more heavily influenced by Precision than TNR, given the shape of its curve in relation to the other two measures. This observation underscores the utility of the 𝑛𝑃 in scenarios where both False Positives and False Negatives carry significant costs. 
conclusion: 
 This paper analyses Precision at 𝑟 % Recall behaviour as an evaluation measures in a high-recall setting. We show the problems with using Precision@𝑟 % and propose its min-max normalised version. nPrecision at 𝑟 % is equal to the product of Precision and True Negative Rate, offering a comprehensive measure for benchmarking IR systems, emphasising the need for models to optimise both True Positives and True Negatives. We also introduced 𝑠𝑛𝑃, a variation of 𝑛𝑃 that is the geometric mean of Precision and TNR.We presented empirical analysis of 𝑛𝑃@𝑟 % and compared it to other TAR measures. We showed how these evaluation measures can be used to focus on models' performance on different aspects of the screening process. Notably, 𝑛𝑃@𝑟 % and 𝑠𝑛𝑃@𝑟 %, among all tested measures, has the lowest correlation with the percentage of relevant documents in dataset, making it more robust to evaluating screening models. For Recall-oriented tasks, high TNR is desirable but not sufficient on its own, as it does not account for the ranking of retrieved items. 𝑛𝑃 and 𝑠𝑛𝑃 can be important measures since they also assesses the quality of the ranking. In future work, we will focus on evaluating and estimating 𝑠𝑛𝑃 scores within legal eDiscovery workflows.Springer International Publishing, Cham, 502-517. 
 
 Appendix: Figure 3 presents scatter plots contrasting 𝑛𝑃@95% with 𝑃@95% and 𝑇 𝑁 𝑅@95% scores across runs and datasets. The plot reveals a range of values for both metrics across the tested models, indicating variability in performance. The size of each marker represents the relative percentage of relevant documents in the dataset with larger markers meaning datasets with higher ratio of relevant documents. Correlations mentioned in Section 5.1 can be observed for both component measures of 𝑛𝑃@95%. For example, datasets consisting of a larger number of relevant documents (represented by larger circles) exhibit higher 𝑃@95% scores. However, this cannot be observed for 𝑛𝑃@95%. The size of each marker represents the relative percentage of relevant documents in the dataset. Average 𝑛𝑃@95%, 𝑃@95% and 𝑇 𝑁 𝑅@95% are indicated by dashed grey lines.Available from https://github.com/WojciechKusa/CSMeD-baselineshttps://huggingface.co/sentence-transformers/all-MiniLM-L6-v2https://huggingface.co/sentence-transformers/all-mpnet-base-v2https://huggingface.co/pritamdeka/S-BioBert-snli-multinli-stsb 


Decision: The authors proposed normalized versions of the precision@recall measure for recall-oriented tasks, such as TAR.
The reviewers' feedback is mostly positive. They noted that the paper is easy to follow and that the proposed metrics are intuitive and clearly explained. The reviewers agreed that the paper makes a small yet interesting theoretical contribution.
However, the reviews also requested some improvements to the presentation of the paper and additional clarifications and discussion, including:

- Discussing edge cases when there are no non-relevant documents.
- Providing additional explanations when discussing Equation 8.
- Correcting some typos, grammatical errors, and notation issues.

The authors should address these issues when preparing the final version.

Reviews: review: main_review: main_review: To sum-up the paper, the proposal is to normalize the precision@r measure to 0 to 1. Some descriptive experimental results are given for this new measure, which are not very insightful. At the end, I do not know whether the proposed normalization really makes a difference in comparing different TAR methods.

Also, the related work lists many performance measured without clear definitions. It would help the readability of the paper to also define most of these mentioned metrics, at least those that are used in the experimental results section.

Finally, I must confess that I am not an expert at all in HRIR. Maybe, I am underestimating the importance of the work for this field.
main_review: main_review: The paper was easy to read and understand and the contribution was made clear early on. It makes a nice theoretical contribution and is therefore very much in the scope of ICTIR. I only wished for some more discussion at times (see my official comments).
main_review: main_review: I am not very familiar with the specific topic "TAR systems" of the paper, but all presented evidence and literature looks reasonable to me. The presented metrics, correlations etc. are convincing to me. The problem is described clearly and the proposed solution seems to alleviate it, so I think this is a good contribution - perhaps for a subsection of the conference attendees, but still.
main_review: main_review: The paper puts a good case for using the new metric.  The derivation is appropriate.  The experiments appear good. And it is well written.