# Active Learning for Biomedical Article Classification with Bag of Words and _FastText_ Embeddings
## 3 Results
### Predictive Performance Evaluation
#### 3.3.2 SLR Workload Reduction  
To evaluate the workload reduction possible to obtain in an SLR process, the _work saved over sampling_ (WSS) metric is used, which represents the share of irrelevant articles that would not have to be manually screened (because they were correctly identified by the classification model) over a random sampling baseline. However, the original definition of WSS [24] was not intended for active learning and does only take into account the human effort of screening articles classified as relevant, ignoring the effort needed to provide training data labels. To account for the latter, the WSS is defined as  
\[\text{WSS}=1-\text{burden}-(1-\text{yield}) \tag{5}\]  
where, after [61]  
\[\text{burden} =\frac{\text{TP}^{T}+\text{TN}^{T}+\text{FP}^{T}+\text{FN}^{T}+ \text{TP}^{P}+\text{FP}^{P}}{N} \tag{6}\] \[\text{yield} =\frac{\text{TP}^{T}+\text{TP}^{P}}{\text{TP}^{T}+\text{TP}^{P}+ \text{FN}^{T}+\text{FN}^{P}} \tag{7}\]  
Here, \(N\) is the total number of articles and \(\text{TP},\text{TN},\text{FP},\text{ and FN}\) denote as usual the number of true positives, true negatives, false positives, and false negatives, with the \(T\) and \(P\) superscripts indicating whether the corresponding number is for the training set (i.e., manually labeled by the human evaluator), for which actual labels are used, or for the pool (i.e., unlabeled), for which model predictions are used. In the case of experiments reported in this article, where class labels for training articles correspond to their true relevant or irrelevant status, \(\text{FP}^{T}\) and \(\text{FN}^{T}\) are both 0. The burden represents the total manual screening effort, which includes screening training articles (both from the initially selected training set and subsequent active learning queries) to provide their class labels and screening unlabeled articles predicted to be relevant by the model. The yield is actually the same as the recall, which is the share of actually relevant articles that are correctly identified. Any given value of yield would be achieved by screening the \((1-\text{yield})\) share of randomly selected articles; hence, the latter term is subtracted from \(1-\text{burden}\) to obtain WSS.  
Unlike standard machine learning model evaluation, WSS calculation is performed on the entire dataset used for active learning, which at any iteration is the sum of the current labeled training set and the remaining unlabeled pool. This may be therefore referred to as the _train+pool_ evaluation procedure as opposed to the standard train-test split or \(k\)-fold cross-validation procedures. To reduce the variance, the experiments using train+pool evaluation are repeated \(n=10\) times, with true and predicted class labels combined from these repeats and used to calculate WSS.  
Following [24], many SLR research studies report WSS@95% which is the WSS value obtained when the recall (or yield) reaches 0.95. There are two possible interpretations of this condition. One, natural for passive learning, would be to adjust the decision threshold used to assign predicted class labels based on predicted class probabilities or decision function values provided by the model so that the required recall is achieved. Thisthreshold would be usually different for different algorithm configurations and datasets, and such a threshold tuning could be hardly performed in an actual application. The other, more realistic and making more sense for active learning, is to keep the decision threshold fixed and identify the active learning iteration at which the combined true class labels for the training set and predictive class labels for the pool achieve the required recall. This is the approach adopted here.