# Active Learning for Medical Article Classification with Bag of Words and Bag of Concepts Embeddings
## 3 Experimental Study
### Experimental Procedure  
In the experimental study, we present the performance of active learning using two types of text representations--bag of words and bag of concepts--as well as the combined bag of concepts and -words (BoC+W) representation with the random forest (RF), linear support vector machine (SVL ), and multinomial naive Bayes (NB) algorithms. Text cleaning and normalization were performed using the Python spaCy library version 3.0.5. [52] with its en_core_web_sm English language model. The process includes removing punctuation and other special characters, conversion to lower case, lemmatization, and stopword removal. This is only applied when using the bag of words representation, since for the bag of concepts representation, text preprocessing is handled by the annotator. The bag of words representation is implemented using the TfidfVectorizer class in the Python scikit-learn library [53]. To determine the vocabulary, frequency-based filtering \begin{table}
\begin{tabular}{l l l} \hline \hline  & **Total** & **Relevant (\%)** \\ \hline ACE Inhibitors & 2544 & 183 (7.19\%) \\ ADHD & 851 & 84 (9.87\%) \\ Antihistamines & 310 & 92 (29.68\%) \\ Atypical Antipsychotics & 1120 & 363 (32.41\%) \\ Beta Blockers & 2072 & 302 (14.58\%) \\ Calcium Channel Blockers & 1218 & 279 (22.91\%) \\ Estrogens & 368 & 80 (21.74\%) \\ NSAIDs & 393 & 88 (22.39\%) \\ Opioids & 1915 & 48 (2.51\%) \\ Oral Hypoglycemics & 503 & 139 (27.63\%) \\ Proton Pump Inhibitors & 1333 & 238 (17.85\%) \\ Skeletal Muscle Relaxants & 1643 & 34 (2.07\%) \\ Statins & 3465 & 173 (4.99\%) \\ Triptans & 671 & 218 (32.49\%) \\ Urinary Incontinence & 327 & 78 (23.85\%) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Case study datasets. is applied, skipping words with document frequency below 0.01 or above 0.95, which corresponds to the setting of the min_df = 0.01 and max_df = 0.95 parameters. For the bag of concepts representation, a custom implementation of the procedure described in Section 2.1 was developed, using the NOBLE annotator with the SNOMED CT and MeSH ontologies and the depth threshold (\(TL\)) for extended representation set to 3. These choices follow [28]. All datasets are from the medical domain, so using the most common ontologies in this domain--SNOMED CT and MeSH--in our paper, as in [28], is well justified. The following representation variants corresponding to different vocabulary formation methods are used:  
**Normal (n):** All concepts identified by the annotator are included in the representation;  
**Short (s):** As above, but concepts with document frequency less than 1% or greater than 95% of the maximum document frequency are omitted;  
**Extended (x):** The concept vocabulary is extended to include concepts lying on the path from the concept identified by the annotator to the ontology root;  
**Extended short (xs):** As above, but concepts with document frequency less than 1% or greater than 95% of the maximum document frequency are omitted. It should be noted that the short variants are characterized by significantly shorter representation vectors; for example, in the case of the ACE Inhibitors study, the representation vector for the normal variant contains 10,462 elements, whereas for the short variant, it contains 1380 elements. On the other hand, the size of the vector representations in the extended variant does not differ that much from that of the normal one: for the same case study, these representations have sizes of 14,624 and 10,462, respectively. Our experiments showed that the most common medical ontologies--SNOMED CT and MeSH--provide sufficiently rich vocabularies for creating numerical representations of medical texts. In our tests, we used SNOMED CT with 354,319 concepts and MeSH with 305,350 concepts. Table 2 presents the dimensions of features vectors for BoW, the short version of BoC, and the extended short version of BoC representations for each considered case study.