# Computer-Assisted Relevance Assessment: A Case Study of Updating Systematic Medical Reviews
## 4 Results
### Performance  
Relevance is subjective even at the expert level, different judgments could be inferred for the same article. Accordingly, it is essential to define, within the context of the experiment, what documents count as relevant, and why. In the case study of systematic reviews update, relevance judgments are determined by the authors of the review through the reference sections, as described in Section 3.3.1. In our experiment, we considered the authors' judgments as the gold-standard annotations since they initially set the inclusion and exclusion criteria for each review. For each article, we counted documents as relevant if both the participant and the author of the systematic review agree on their relevancy, i.e., the participant clicked on the \"Relevant\" button, and the article is in the _included_ or _excluded_ reference sections, and similarly for the irrelevant documents. The correctly judged relevant and irrelevant sets are denoted \(S_{R}\) and \(S_{IRR}\), respectively.  
To compare the difference in relevancy judgment between sentence and abstract views, a logical measure to use is the number of correct relevant articles reported by the user, \(S_{R}\), for both tasks. We interpret accuracy based on correctly assessed documents, namely the true positives (TP) and true negatives (TN), which indicate if the participants agree or disagree with the ground truth judgments. \(S_{R}\) represents the TP as the participant judgment aligns with the author judgment positively while \(S_{IRR}\) represents TN.  
\[Accuracy=\frac{S_{R}+S_{IRR}}{N} \tag{1}\]  
where \(N\) is the total number of articles per systematic review. Similar to the legal e-discovery task, conducting a systematic review aims at finding all available relevant documents. Missing documents could lead to legal issues for e-discovery or could affect the conclusions reported by the systematic review. Therefore, there is a need for a better suited measure of performance for this case study, such as recall, to measure the fraction of all relevant documents found by participants. In our evaluation, we calculated recall by dividing the number of correctly judged relevant documents (\(S_{R}\)) by the total number of relevant documents (\(R\)) available for each systematic review.  
\[Recall=\frac{S_{R}}{R} \tag{2}\]  
In statistical analysis of binary classification, the F-score, or F-measure, is a measure of a test's accuracy that combines both recall and precision to compute the score. The higher is the score, the better, reaching its best value at 1. The general formula to calculate the F-measure requires a positive real beta parameter (\(\beta\)) as follows:  
\[F_{\beta}=(1+\beta^{2})\frac{Precision.Recall}{(\beta^{2}.Precision)+Recall} \tag{3}\]  
Increasing the beta parameter consequently means emphasizing recall over precision. Given the recall-oriented nature of the task, we report the F2 score, which weights the recall twice as much as precision as opposed to the F1 score that portrays the harmonic mean of the precision and recall. Additionally, we tracked the time needed to judge each article in both the _Abstract-View_ and _Sentence-View_ tasks. As shown in Table 1, each topic has a different number of relevant documents. To minimize the skewness of reported relevant articles, we normalized each systematic review according to the number of relevant documents in its set. Table 3 shows the values for the computed measures; the corresponding values are reported on the average basis among all participants, across all systematic reviews from the data collection.