# Active Bagging Ensemble Selection

Vangjel Kazllarof, Sotiris Kotsiantis

► To cite this version:

Vangjel Kazllarof, Sotiris Kotsiantis. Active Bagging Ensemble Selection. 17th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2021, Hersonissos, Crete, Greece. pp.455-465, 10.1007/978-3-030-79157-5_37. hal-03789005

HAL Id: hal-03789005

https://inria.hal.science/hal-03789005

Submitted on 27 Sep 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Active Bagging Ensemble Selection

Vangjel Kazllarof¹[0000-0003-4545-3866] and Sotiris Kotsiantis²[0000-0002-2247-3082]

¹ University of Patras, 26504 Rio Achaia, Greece

² University of Patras, 26504 Rio Achaia, Greece

**Abstract.** As technology progresses with more and more data collected, the need of finding the appropriate label for them increases. However, many times the labeling process is a very difficult or/and expensive task and in most cases a help of an expert or expensive equipment is needed. For this reason the need of labeling only the most appropriate instances rises. Active Learning techniques can accomplish this by querying only those instances that a trained model finds the greatest amount of information and providing them to a human expert in order to label them. Combining these techniques with a fast ensemble classifier, a very performant in terms of classification accuracy schema can emerge where a trained model in a small amount of labeled instances can grow by adding only the most informative instances from a much greater pool of unlabeled instances. In this paper, we will propose such a schema using Bagging Ensemble Selection that uses REPTree as base classifier under Active Learning techniques and we will compare it to four well-known ensemble classifiers under the same techniques on 61 real world datasets.

**Keywords:** Active Learning, Ensemble Classifiers, Algorithms.

## 1 Introduction

Nowadays large amount of data are collected from various sources. However, not all of this data comes with a classified label because it might be quite difficult, time consuming [1] or/and expensive [2] to find it. Many times the labeling process is bounded by a budget plan allowing only a small amount of the unlabeled data for annotating. For these reasons creating accurate models with the use of as least as possible data becomes essential. This rises the need of finding a way to discard the data that provide no useful information and find the label for only the ones that will improve the initial model. A field in Machine Learning called Weakly Supervised Learning has emerged that tackles the problems described above, with Active Learning (AL) [3] playing a leading role in the field. The AL process is utilized by performing queries to the unlabeled data in order to find those instances that provide the most amount of information. Essential fact to create accurate models under AL schema is the use of robust and accurate ensemble classifiers with fast and accurate base classifiers.

In this work we will propose an AL schema with the use of Bagging Ensemble Selection (BES) that uses Reduced Error Pruning Tree (REPTree) [4] as base classifier and we will compare it with other well-known ensemble classifiers under the same AL
schema proving that the proposed one can outperform the rest in terms of classification accuracy, especially when the portion of initial labeled data is very small compared to the unlabeled one.

In the following section a brief introduction of the AL theory and the BES with the use of REPTree as base classifier will be presented along with the related work. In section 3, we will demonstrate the proposed algorithm. The experimental procedure and the results of our experiments will be described in section 4 following with the conclusions and the future work in section 5.

## 2 Related Work

To begin with, a number of techniques in querying unlabeled instances have been developed and they are called Scenarios. Membership Query Synthesis [5] is one of them in which the model generates instances de novo and then queries them. Another is the Stream Based Selective Sampling [6] in which unlabeled instances come one-by-one from an input stream and the model decides whether will be queried or discarded. The last one is Pool Based [7] scenario, probably the most popular one, in which assumes that there is a small pool of labeled instances (L) and a much larger pool of unlabeled ones (U). The model rates all the instances of U and selects those with the greatest value to query.

For measuring the informativeness of the unlabeled instances, several Query Strategies (QS) have been developed [3]. Uncertainty Sampling [7] is one of the most known strategies in which the model rates the instances based on the uncertainty level. The selected instances for querying are the one that the model is most uncertain.

AL methods with the use of ensemble classifiers have been widely used in both experimental and real life problems. In Gesture Recognition area, algorithms that use AL with the use of a number of ensemble methods and Random Forest have been proposed in [8] and [9] respectively. In both works a good classification accuracy has been achieved by using only a small subset of the whole dataset. Continuing in the Image Recognition field, an ensemble of Convolutional Neural Networks has been proposed in [10] under an AL schema for the uncertainty estimation of unlabeled data. It was compared against Monte Carlo Dropout and geometric approaches showing that the proposed algorithm outperformed its rivals in classification image datasets used in the literature and in real-world images for diabetic retinopathy detection. A study on the Data Stream Mining have been demonstrated in [11] where Active Online Ensembles is proposed in which accurate models are built with the use small-sized ensembles. In the study online Bagging and online Boosting that use Hoeffding tree and k-Nearest Neighbors as base classifiers have been extended with an AL component and showed great results in classification accuracy with much smaller ensemble size against traditional online Bagging and online Boosting contributing in the training performance as well.

Moreover, algorithms that combine AL and ensemble trees have been proposed in [12] and [13] in which Rotation Forest and Logitboost with the use of M5P as base
classifier have been exploited respectively, showing good results in classification ac-
curacy against other simple and ensemble classifiers under the same AL schemas on
real-world datasets. Similar approach is implemented in the current work, adding BES
with the use of REPTree base classifier in the pool of accurate ensembles under AL
schema.

3 Active Learning BESTrees

3.1 Bagging Ensemble Selection

For the ensemble strategy, an extension of Ensemble Selection (ES) [14] that uses a
library of base learners to construct the ensemble one is chosen in this work. It uses a
construction strategy in order to extract a well performing subset of models that are
trained with the base learners. Forward step-wise selection strategy was proposed in
[14] that started with an empty ensemble and it was gradually populated by the models
that maximize the ensemble's performance on a hillclimb set, until all models are ex-
amined.

Although ES has shown great advantages compared to other ensemble strategies,
many times it has suffered from overfitting on the hillclimb set, hitting the performance
of the model on the test set. To overcome this, BES [15] was proposed that in its sim-
plest implementation, ES is treated as a base classifier for bagging.

Moreover, an extension of BES, named BES-out-of-bag (BES-OOB), was proposed in [15] that uses all of the bootstrap sample in order to generate the models and the respective out-of-bag (OOB) instances as the hillclimb set. Then, base classifiers were trained in the bootstrap sample and ensemble selection was applied to them according to their performance in the OOB sample.

The implementation of BES-OOB method with the use of REPTree (BESTrees) as
base classifier was compared with other ensemble strategies in [16] showing better re-
sults against Stochastic Gradient Boosting and Bagging and being comparable against
an ensemble that combines both Bagging and Stochastic Gradient Boosting.

Although BES has shown great performance, it has not been widely used in the lit-
erature. BESTrees was found to be used for Sentiment Classification in [17] resulting,
however, worse results than Random Forest, on a dataset produced by tweets. More
promising results was shown on a lightweight extension of BESTrees in [18] where
BES classification was used for Activity Recognition on a smart home system using
mobile phones and iBeacons. It was compared against other existing lightweight algo-
rithms outperforming them in both classification accuracy and efficiency in terms of
hardware resources.

## 3.2 Proposed Algorithm

First of all, the Labeled Ratio (R) is defined as the percentage of the size of L compared to the sum of the sizes of L and U and it is described by the following formula:
$$R = \frac{Size(L)}{Size(L) + Size(U)} * 100 \quad (1)$$

Starting now with L sized with a small R, the AL procedure creates an initial model and then uses it to rate the instances of U, given the selected QS. After finding the most informative instances, the next step is to provide them to a Human Expert (HE) one-by-one or in batches of fixed or dynamic size, depending of the size of the initial L. The HE is an ideal labeler in this work that always annotates correctly the provided instances. Then, the new labeled instances are removed from U and added to L, ending an AL circle [3].

In the next step, the new formatted L is trained again and this procedure goes on until the stopping criterion is met. For the stopping criterion we used the max iterations method by stopping the whole process after 15 iterations. The output of the algorithm is the trained model on L formatted in the last iteration of the algorithm. This model then is used to predict the test set.

The chosen ensemble classifier is the BESTrees as described above and under AL schema the Active Learning BESTrees (ALBESTrees) emerges and it is described in the following algorithm:

Algorithm ALBESTrees:
HE: Human Expert (Oracle);
CL: Classifier (BESTrees);
MI: Max Iterations (15);
QS: Query Strategy (Uncertainty Sampling);
I: Current Iteration (0);
L: Initial Labeled Set;
U: Initial Unlabeled Set;
R: Labeled Ratio;
T: Test Set;
B: Batch Size (Ceil((Size(L)+Size(U)) * R / MI));
TR: Top Rated Instances;
Begin:
    Train CL in L;
    While I < MI:
        Assign uncertainty values to U using QS(CL);
        Add B top rated instances in TR;
        Remove TR from U;
        Ask HE for labeling instances in TR;
        Merge TR in L;
        Empty TR;
        Retrain CL in new L;
        Assign I+1 in I;
    End
    Predict labels of T using last trained CL;
End
End
For each iteration we used the batch mode sampling (B) where the top rated instances are selected in batches from the U set. The size of the batches is different for each dataset and it is calculated with the following equation:

$$B = \left\lfloor \frac{(Size(L) + Size(U)) * R}{Max Iterations} \right\rfloor \quad (2)$$

This way, we eliminate the size dependency of the datasets compared to the iterations. This way, the size of L in the last iteration will be always doubled from starting the size of the procedure.

# 4 Experiments and results

## 4.1 Datasets

For our experiments we selected 61 real-world datasets from UCI repository [19]. The datasets consist of 27 binary and 34 multiclass classification problems. They have a great variety of sizes with the smallest one to have only 208 (sonar) instances while the largest 67557 instances (connect-4). The number of classes vary from 2 classes for binary datasets to a maximum of 28 classes for multiclass (abalone). As for the number attributes, all datasets have a range of 2 (banana) to 90 (movement_libras) different attributes where many of them are constructed from only numerical or categorical types while other from both.

For the imbalance factor, the binary datasets are almost all balanced with only one exception (coil200) being quite imbalance with almost 6% of instances classified in the minority class. On the other hand, many multiclass datasets have a great imbalance factor with the instances classified to the minority class less than 5%.

## 4.2 Experimental procedure

The AL schema starts with Pool Based scenario with the use of a small L and big U as starting sets. For R we selected four different values to experiment with and these are 5%, 10%, 15% and 20% in order to examine the behavior of the algorithm in both small and big initial L. The model in the last iteration is trained to an L doubled in size being 10%, 20%, 30% and 40% of the whole dataset for each R value respectively.

In order to rate the instances in the U set, we used Uncertainty Sampling (UncSamp) with three different methods in order to measure the uncertainty. The first is Entropy in which the top rated instances are the one that increase the entropy of the model, the second is Least Confidence that rates the instances in favor of the one that the model has the least confidence and the last one is Smallest Margin that aims to select the instances that have the smallest margin between the two most probable classes that the model decides. To compare with, we also run the experiments with Random Sampling that selects the instances from U in a random manner.

For rivals we selected 3 well-known ensemble techniques with the use of 3 different tree classifiers as base classifier. The first one is Bagging technique with base classifiers the REPTree, that is used in the proposed algorithm as well, and J48 that is a Java
implementation of the C4.5 decision tree. The next two ensemble techniques belong to the Boosting family and these are LogitBoost [20] and AdaBoostM1 [21] with the use of Decision Stump as base classifier.

In the training process we used 3-fold validation that separates the train set in three folds and uses 2/3 for training and the rest 1/3 for testing, repeating the process three times until all folds are tested. The classification accuracy of the model is the average of results of each testing phase.

It is worth mentioning that every experiment ran 3 times for every dataset, QS, R and classifier and the final result is the average of the results of each experiment. For software we used WEKA [4] with the use of JCLAL [22] framework for the AL setup and for each classifier we used the default parameters provided by the software.

## 4.3 Results

In Table 1 and Table 2 we demonstrate the results of the classification accuracy of each algorithm for both binary and multiclass datasets respectively annotating with bold text the best classification accuracy for each dataset. Due to lack of space, we only included the results of classification accuracy for R=5% using the Uncertainty Sampling with Entropy method against the rest of the selected ensemble methods. The rest of the results can be found in the link:

http://ml.math.upatras.gr/wp-content/uploads/2021/03/Active-BES-Results.zip.

Moreover, we calculated the total winnings of each algorithm in terms of classification accuracy for every initial R and every query strategy method for both binary and multiclass datasets. The proposed algorithm outperformed its rivals in all of the uncertainty sampling metrics and initial labeled ratios with a total of 574 winnings followed by Bagging(J48) with 127 winnings. It is worth mentioning that most winnings of the proposed algorithm are accomplished for R = 5% proving that it is a better choice when there are very few labeled instances compared to the unlabeled one.

For the statistical analysis of the performance of the algorithms, we used non-parametrical Friedman tests that examines if the null hypothesis of the similarity of the algorithms that it compares holds [23]. In Fig. 1, we demonstrate these results comparing all algorithms with the use of Uncertainty Sampling and Random Sampling query strategies using violin plot.

From the results it is shown that the proposed algorithm has the best ranking in all cases compared to its rivals. Moreover, Uncertainty Sampling shows better rankings compared to Random Sampling proving that AL techniques can be very beneficial in order to choose the most appropriate instances.

# 5 Conclusions and Future Work

In this work we proposed BES with the use of REPTree as base classifier under AL schema. We compared it with other well-known ensemble methods on both binary and multiclass classification datasets under the same AL schema. From the results it is shown that the proposed algorithm outperforms its rivals with statistical significance
according to non-parametrical Friedman tests, making it a great candidate for problems where acquiring labels of unlabeled data is constrained by a budget plan.

For future work, we will compare the proposed algorithm with other state-of-the-art ensemble algorithms like Random Forest, Rotation Forest and Neural Networks Ensembles in both classification accuracy and training efficiency. Moreover, we will experiment with Noisy Oracles where HE has an error ratio on its annotations, in order to simulate more real life scenarios [24].

<table><caption>**Table 1.** Classification accuracy of the ALBESTrees against the selected ensemble methods for binary datasets under UncSamp(Entropy) strategy for R = 5%</caption><thead><tr><th>Dataset</th><th>BES<br>(REPTree)</th><th>Bagging<br>(REPTree)</th><th>Bagging<br>(J48)</th><th>Logitboost<br>(Decision St.)</th><th>Adaboost<br>(Decision St.)</th></tr></thead><tbody><tr><td>banana</td><td>70.81</td><td>82.10</td><td>83.35</td><td><strong>84.48</strong></td><td>59.25</td></tr><tr><td>bands</td><td><strong>56.90</strong></td><td>44.74</td><td>47.21</td><td>49.32</td><td>55.14</td></tr><tr><td>chess</td><td>91.94</td><td>93.19</td><td><strong>95.78</strong></td><td>90.00</td><td>84.38</td></tr><tr><td>coil2000</td><td><strong>94.03</strong></td><td>93.96</td><td>93.49</td><td>92.30</td><td><strong>94.03</strong></td></tr><tr><td>credit-a</td><td><strong>85.94</strong></td><td>71.40</td><td>81.06</td><td>72.75</td><td>84.88</td></tr><tr><td>credit-g</td><td><strong>71.57</strong></td><td>70.13</td><td>69.53</td><td>68.53</td><td>70.63</td></tr><tr><td>german</td><td><strong>71.57</strong></td><td>70.10</td><td>68.73</td><td>68.37</td><td>70.47</td></tr><tr><td>heart-statlog</td><td><strong>72.59</strong></td><td>55.56</td><td>60.25</td><td>69.14</td><td>70.49</td></tr><tr><td>housevotes</td><td><strong>96.98</strong></td><td>95.25</td><td>94.67</td><td>91.82</td><td>94.82</td></tr><tr><td>ionosphere</td><td><strong>87.84</strong></td><td>56.89</td><td>62.20</td><td>69.92</td><td>74.45</td></tr><tr><td>kr-vs-kp</td><td>92.09</td><td>94.50</td><td><strong>95.58</strong></td><td>90.39</td><td>86.57</td></tr><tr><td>magic</td><td><strong>85.59</strong></td><td>83.26</td><td>82.88</td><td>81.73</td><td>77.14</td></tr><tr><td>mammo-<br>graphic</td><td><strong>83.41</strong></td><td>70.68</td><td>79.52</td><td>74.66</td><td>79.92</td></tr><tr><td>monk-2</td><td><strong>98.07</strong></td><td>97.22</td><td>97.22</td><td>90.48</td><td>95.76</td></tr><tr><td>mushroom</td><td><strong>99.94</strong></td><td>99.74</td><td>99.36</td><td>99.41</td><td>97.58</td></tr><tr><td>phoneme</td><td><strong>83.78</strong></td><td>79.16</td><td>80.47</td><td>78.26</td><td>72.25</td></tr><tr><td>pima</td><td><strong>73.39</strong></td><td>67.97</td><td>69.01</td><td>69.84</td><td>70.66</td></tr><tr><td>ring</td><td><strong>95.58</strong></td><td>85.67</td><td>87.62</td><td>82.30</td><td>49.51</td></tr><tr><td>sonar</td><td><strong>62.85</strong></td><td>45.53</td><td>52.57</td><td>59.48</td><td>60.73</td></tr><tr><td>spambase</td><td><strong>92.94</strong></td><td>88.80</td><td>89.66</td><td>85.08</td><td>83.76</td></tr><tr><td>spectfheart</td><td><strong>76.78</strong></td><td>43.07</td><td>49.44</td><td>59.70</td><td>69.91</td></tr><tr><td>tic-tac-toe</td><td>71.40</td><td>67.88</td><td>67.40</td><td><strong>75.01</strong></td><td>69.07</td></tr><tr><td>titanic</td><td>71.20</td><td>77.65</td><td><strong>78.27</strong></td><td>77.15</td><td>77.66</td></tr><tr><td>twonorm</td><td><strong>95.82</strong></td><td>84.77</td><td>86.36</td><td>85.82</td><td>84.81</td></tr><tr><td>vote</td><td>90.50</td><td>78.08</td><td>88.35</td><td>91.56</td><td><strong>92.72</strong></td></tr><tr><td>wdbc</td><td><strong>94.43</strong></td><td>85.18</td><td>85.76</td><td>89.87</td><td>94.38</td></tr><tr><td>wisconsin</td><td><strong>96.88</strong></td><td>89.95</td><td>92.97</td><td>92.02</td><td>92.97</td></tr></tbody></table>
<table><caption>**Table 2.** Classification accuracy of the ALBESTrees against the selected ensemble methods for multiclass datasets under UncSamp(Entropy) strategy for R = 5%</caption><thead><tr><th>Dataset</th><th>BES<br>(REPTree)</th><th>Bagging<br>(REPTree)</th><th>Bagging<br>(J48)</th><th>Logitboost<br>(Decision St.)</th><th>Adaboost<br>(Decision St.)</th></tr></thead><tbody><tr><td>abalone</td><td><strong>24.91</strong></td><td>21.88</td><td>19.94</td><td>18.73</td><td>16.73</td></tr><tr><td>anneal</td><td><strong>90.05</strong></td><td>84.11</td><td>85.15</td><td>89.03</td><td>77.02</td></tr><tr><td>audiology</td><td>29.80</td><td>33.04</td><td><strong>49.10</strong></td><td>38.91</td><td>33.90</td></tr><tr><td>balance-scale</td><td>73.92</td><td>62.61</td><td>71.84</td><td><strong>74.96</strong></td><td>49.81</td></tr><tr><td>balance</td><td>74.93</td><td>69.17</td><td>66.98</td><td><strong>76.28</strong></td><td>55.90</td></tr><tr><td>car</td><td><strong>83.82</strong></td><td>74.19</td><td>78.34</td><td>80.56</td><td>70.79</td></tr><tr><td>cleveland</td><td><strong>56.57</strong></td><td>53.87</td><td>53.87</td><td>51.70</td><td>53.98</td></tr><tr><td>connect-4</td><td>69.41</td><td>71.71</td><td><strong>72.67</strong></td><td>70.24</td><td>65.83</td></tr><tr><td>dermatology</td><td><strong>93.48</strong></td><td>75.69</td><td>86.31</td><td>76.17</td><td>48.70</td></tr><tr><td>ecoli</td><td><strong>74.80</strong></td><td>68.35</td><td>68.15</td><td>63.99</td><td>62.00</td></tr><tr><td>flare</td><td>69.23</td><td><strong>70.04</strong></td><td>68.73</td><td>68.49</td><td>53.47</td></tr><tr><td>kr-vs-kp</td><td><strong>36.34</strong></td><td>27.27</td><td>33.56</td><td>35.84</td><td>10.04</td></tr><tr><td>led7digit</td><td><strong>54.73</strong></td><td>41.59</td><td>47.00</td><td>48.78</td><td>14.94</td></tr><tr><td>letter</td><td><strong>80.84</strong></td><td>62.47</td><td>68.15</td><td>71.02</td><td>6.91</td></tr><tr><td>marketing</td><td><strong>31.53</strong></td><td>28.56</td><td>26.59</td><td>25.89</td><td>18.64</td></tr><tr><td>movement_libras</td><td><strong>35.74</strong></td><td>20.74</td><td>24.35</td><td>27.24</td><td>10.93</td></tr><tr><td>newthyroid</td><td><strong>81.41</strong></td><td>81.11</td><td>78.79</td><td>80.70</td><td>81.26</td></tr><tr><td>nursery</td><td>88.79</td><td>89.17</td><td><strong>90.74</strong></td><td>90.41</td><td>64.54</td></tr><tr><td>optdigits</td><td><strong>91.48</strong></td><td>79.94</td><td>81.60</td><td>78.35</td><td>18.74</td></tr><tr><td>page-blocks</td><td><strong>96.91</strong></td><td>95.74</td><td>96.06</td><td>95.04</td><td>92.60</td></tr><tr><td>penbased</td><td><strong>96.31</strong></td><td>89.92</td><td>91.95</td><td>89.72</td><td>20.52</td></tr><tr><td>primary-tumor</td><td>25.86</td><td>25.17</td><td>24.39</td><td><strong>28.81</strong></td><td>25.86</td></tr><tr><td>satimage</td><td><strong>86.23</strong></td><td>80.95</td><td>81.66</td><td>73.37</td><td>33.63</td></tr><tr><td>segment</td><td><strong>93.67</strong></td><td>88.66</td><td>89.25</td><td>85.34</td><td>28.51</td></tr><tr><td>shuttle</td><td><strong>99.96</strong></td><td>99.90</td><td><strong>99.96</strong></td><td>99.83</td><td>84.23</td></tr><tr><td>soybean</td><td><strong>64.13</strong></td><td>20.45</td><td>43.88</td><td>60.24</td><td>13.47</td></tr><tr><td>texture</td><td><strong>92.28</strong></td><td>81.48</td><td>84.93</td><td>83.45</td><td>16.08</td></tr><tr><td>thyroid</td><td>99.11</td><td>99.30</td><td><strong>99.36</strong></td><td>96.62</td><td>96.87</td></tr><tr><td>vehicle</td><td><strong>62.37</strong></td><td>48.35</td><td>52.44</td><td>56.34</td><td>26.04</td></tr><tr><td>vowel</td><td><strong>43.30</strong></td><td>10.91</td><td>39.66</td><td>35.93</td><td>14.14</td></tr><tr><td>waveform-5000</td><td><strong>82.03</strong></td><td>75.11</td><td>75.00</td><td>70.28</td><td>55.37</td></tr><tr><td>winequality-red</td><td><strong>49.86</strong></td><td>45.03</td><td>46.32</td><td>40.70</td><td>42.21</td></tr><tr><td>winequality-white</td><td><strong>50.52</strong></td><td>46.86</td><td>47.39</td><td>40.94</td><td>31.19</td></tr><tr><td>yeast</td><td><strong>49.10</strong></td><td>45.15</td><td>47.15</td><td>41.40</td><td>21.29</td></tr></tbody></table>
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Violin plot presenting the distribution of Friedman ranking of the ensemble classifiers comparing the selected query strategy against random sampling</figcaption></figure>

References

1. Zhu, X., Lafferty, J., Ghahramani, Z.: Combining Active Learning and Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions. In: ICML 2003 Workshow on the Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining. pp. 58–65 (2003)

2. Settles, B., Craven, M., Friedl, L.: Active Learning with Real Annotation Costs. In: Proceedings of the NIPS Workshop on Cost-Sensitive Learning. pp. 1–10 (2008)

3. Settles, B.: Active learning. Synth. Lect. Artif. Intell. Mach. Learn. 18, 1–111 (2012). https://doi.org/10.2200/S00429ED1V01Y201207AIM018

4. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA Data Mining Software: An Update. SIGKDD Explor. 11, (2009)

5. Angluin, D.: Queries and Concept Learning. Mach. Learn. 2, 319–342 (1987)

6. Cohn, D., Ladner, R., Waibel, A.: Improving generalization with active learning. Mach. Learn. 15, 201–221 (1994)

7. Lewis, D.D., Gale, W.A.: A Sequential Algorithm for Training Text Classifiers. In: Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 3–12. ACM/Springer (1994)

8. Schumacher, J., Sakič, D., Grumpe, A., Fink, G.A., Wöhler, C.: Active learning of ensemble classifiers for gesture recognition. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). pp. 498–507. Springer, Berlin, Heidelberg (2012)

9. Kazllarof, V., Kotsiantis, S., Karlos, S., Xenos, M.: Automated hand gesture recognition exploiting active learning methods. In: PCI 2017: Proceedings of the 21st Pan-Hellenic Conference on Informatics. pp. 1–6. Association for Computing Machinery (2017)
10. Beluch, W.H., Genewein, T., Nürnberger, A., Köhler, J.M.: The Power of Ensembles for Active Learning in Image Classification. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 9368–9377. IEEE Computer Society (2018)

11. Alabdulrahman, R., Viktor, H., Paquet, E.: An active learning approach for ensemble-based data stream mining. In: IC3K 2016 - Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management. pp. 275–282. SciTePress (2016)

12. Kazllarof, V., Karlos, S., Kotsiantis, S.: Active learning Rotation Forest for multiclass classification. Comput. Intell. 35, 891–918 (2019). https://doi.org/10.1111/coin.12217

13. Kazllarof, V., Karlos, S., Kotsiantis, S.: Investigation of Combining Logitboost(M5P) under Active Learning Classification Tasks. Informatics. 7, 50 (2020). https://doi.org/10.3390/informatics7040050

14. Caruana, R., Niculescu-Mizil, A., Crew, G., Ksikes, A.: Ensemble Selection from Libraries of Models. In: Proceedings of the 21st International Conference on Machine Learning. pp. 137–144 (2004)

15. Sun, Q., Pfahringer, B.: Bagging Ensemble Selection. In: Proceedings of the 24th Australasian Joint Conference on Artificial Intelligence (AI’11). pp. 251–260. Springer, Perth (2011)

16. Sun, Q., Pfahringer, B.: Bagging Ensemble Selection for Regression. In: Proceedings of the 25th Australasian Joint Conference on Artificial Intelligence (AI’12). pp. 695–706. Springer, Sydney (2012)

17. Moreira, S., Filgueiras, J., Martins, B., Couto, F., Silva, M.J.: REACTION: A naive machine learning approach for sentiment classification. In: Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). pp. 490–494. Association for Computational Linguistics, Atlanta (2013)

18. Alam, M.A.U., Pathak, N., Roy, N.: Mobeacon: An iBeacon-Assisted Smartphone-Based Real Time Activity Recognition Framework. In: Proceedings of the 12th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services. pp. 130–139. Institute for Computer Sciences, Social Informatics and Telecommunications Engineering (ICST) (2015)

19. Dua, D., Graff, C.: UCI Machine Learning Repository, http://archive.ics.uci.edu/ml

20. Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: A statistical view of boosting. Ann. Stat. 28, 337–407 (2000). https://doi.org/10.1214/aos/1016218223

21. Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. Proc. Thirteen. Int. Conf. Int. Conf. Mach. Learn. 148–156 (1996)

22. Reyes, O., Pérez, E., Fardoun, H.M., Ventura, S.: JCLAL: A Java Framework for Active Learning. J. Mach. Learn. Res. 17, 1–5 (2016). https://doi.org/10.5555/2946645.3007048

23. Eisinga, R., Heskes, T., Pelzer, B., Te Grotenhuis, M.: Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers. BMC Bioinformatics. 18, 68 (2017). https://doi.org/10.1186/s12859-017-1486-2

24. Gupta, G., Sahu, A.K., Lin, W.Y.: Noisy Batch Active Learning with Deterministic Annealing. (2019)