# Active Learning for Medical Article Classification with Bag of Words and Bag of Concepts Embeddings
## 1 Introduction
### Motivation and Contributions  
As summarized above, most published research on the use of text classification for systematic literature reviews has used the bag of words representation, which, despite its simplicity, handles the task of mapping texts to a vector space useful for classification quite well. Although alternative embedding-based representations provide some clear advantages, such as usually reduced dimensionality and the potential to capture contextual information, this does not necessarily make them better for the text classification task. The utility of several such representations, including word2 vec [19], doc2 vec [20], Glove [21], fastText [22], and BioBERT [23], was evaluated in previous work [17; 24], and they did not outperform the bag of words representation. While some of them were found to reach a similar level of predictive power, with the advantage of lower dimensionality, the former can be still considered the method of choice for text classification with conventional (non-neural) machine learning algorithms. While more complex neural network-based embeddings are promising and deserve further interest [25], their widespread adoption in SLR systems encounters limitations related to computational cost, challenges in interpretability, and complexities in integration with active learning pipelines [26; 27]. Moreover, recent benchmark analyses continue to show that the bag of words representation remains competitive in comparison with modern Transformer architectures in various text classification tasks [27].  
Another approach to improve text representation based on the bag of concepts (BoC) approach has not yet received as much attention as it deserves, and its utility for active learning-based SLR systems has not been investigated with sufficient depth. We find this direction promising because it potentially augments the simplicity and utility of bag of words, with domain knowledge and semantic information represented by ontologies. Bag of concepts can be actually considered a family of representations sharing the same basic principle of using features corresponding to concepts but differing in several detailed design decisions, such as the choice of ontologies and annotation tools or the method of concept vocabulary formation. This research partially fills the gap left by prior work by experimentally verifying the utility of several instantiations of the bag of concepts representation for semi-automated SLRs, in combinations with different classification algorithms and active learning setups. Unlike the form of bag of concepts previously described in the literature [28], the applied feature extraction method does not rely on the UMLS storage repository and management tools, which makes it more general and potentially applicable with different ontologies for different domains.  
More specifically, the contributions of this work can be summarized as follows.
* A general bag of concepts text representation procedure with several variants corresponding to different vocabulary formation methods;
* Systematic experiments comparing the utility of the bag of words, bag of concepts, and combined bag of concepts and bag of words representations, using different classification algorithms and active learning setups, with evaluation methods based on workload savings and performance profiles;
* Practical recommendations for semi-automated SLR system development.  
A preliminary and limited investigation of the utility of the bag of concepts representation for active learning applied to systematic literature reviews was reported before [17], but the study presented in this article is considerably more comprehensive with respect to the scope of bag of concepts variants, classification algorithms, and active learning configurations.