# Active Learning for Biomedical Article Classification with Bag of Words and _FastText_ Embeddings
## 3 Results
### Early Stopping Criteria: Aggregated  
Table 7 presents the percentage of the pool used for training when different stop criteria were satisfied during active learning with random initialization and uncertainty sampling, for all classification algorithm and text representation combinations, averaged over all datasets. Row labels provide the type of the early stopping criterion and the threshold value. Similarly, Table 8 presents the area under the precision-recall curve of the  
Figure 1: Learning curves for SVM-FT and RF-BOW with random initialization and uncertainty sampling.  
model obtained when different stop criteria were satisfied, for all classification algorithm and text representation combinations, averaged over all datasets.  
The following observations can be made based on these results.  
* The performance of early stopping criteria heavily depends on the classification algorithm: the same stopping criteria tend to stop active learning with SVM earlier than with RF with the biggest differences for stability-based criteria.
* Differences between the early stopping criteria performance for the same classification algorithm with different text representations are less pronounced, although stability-based criteria stop SVM with the _FastText_ representation considerably earlier than with bag of words.
* The most universal early stopping criterion, providing roughly comparable performance for all algorithm-representation configurations, is the confidence criterion with a 0.6-0.8 threshold.
* Stability-based stopping may work well, but it definitely requires quite different thresholds for the SVM and random forest algorithms (above 0.98 for the former, between 0.93 and 0.98 for the latter).  
\begin{table}
\begin{tabular}{c c c c c} \hline  & **SVM-BOW** & **SVM-FT** & **RF-BOW** & **RF-FT** \\ \hline confidence 0.5 & 10\% & 10\% & 10\% & 10\% \\ confidence 0.6 & 19\% & 23\% & 34\% & 29\% \\ confidence 0.7 & 26\% & 30\% & 48\% & 40\% \\ confidence 0.8 & 36\% & 38\% & 61\% & 54\% \\ confidence 0.9 & 54\% & 55\% & 77\% & 72\% \\ confidence 0.95 & 73\% & 71\% & 89\% & 84\% \\ confidence 0.98 & 85\% & 83\% & 98\% & 95\% \\ stability 0.9 & 14\% & 12\% & 25\% & 28\% \\ stability 0.93 & 17\% & 13\% & 38\% & 44\% \\ stability 0.95 & 21\% & 15\% & 58\% & 54\% \\ stability 0.98 & 43\% & 25\% & 93\% & 87\% \\ stability 0.99 & 64\% & 37\% & 97\% & 95\% \\ stability 0.995 & 76\% & 47\% & 100\% & 97\% \\ stability 0.998 & 84\% & 56\% & 100\% & 100\% \\ \hline \end{tabular}
\end{table}
Table 7: Percentage of the pool used for training when different early stopping criteria were satisfied, averaged over all datasets.  
\begin{table}
\begin{tabular}{c c c c c} \hline  & **SVM-BOW** & **SVM-FT** & **RF-BOW** & **RF-FT** \\ \hline confidence 0.5 & 0.27 & 0.40 & 0.42 & 0.40 \\ confidence 0.6 & 0.34 & 0.48 & 0.52 & 0.49 \\ confidence 0.7 & 0.37 & 0.50 & 0.54 & 0.51 \\ confidence 0.8 & 0.40 & 0.51 & 0.54 & 0.53 \\ confidence 0.9 & 0.45 & 0.53 & 0.55 & 0.53 \\ confidence 0.95 & 0.47 & 0.54 & 0.55 & 0.53  
The following observations can be made:  
* No single stopping criterion and threshold parameter value works best for all datasets.
* The confidence-based criterion with a fixed threshold value of 0.8 for SVM-FT or 0.7 for RF-BOW provide quite reliable stopping signals for most datasets with some notable exceptions being _Oipods_ and _SkeletalMuscleRelaxants_.
* The stability-based criterion produces mixed results, and while for SVM-FT, it delivers useful stopping signals for most datasets with the 0.99 threshold, it is hard to recommend a similarly reasonable threshold value for RF-BOW.