<figure><img src="image_1.png" /></figure>

HAL
open science

# Active Learning Query Strategy Selection Using Dataset Meta-features Extraction

Vangjel Kazllarof, Sotiris Kotsiantis

► To cite this version:

Vangjel Kazllarof, Sotiris Kotsiantis. Active Learning Query Strategy Selection Using Dataset Meta-features Extraction. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.185-194, 10.1007/978-3-031-34107-6_15. hal-05498101

HAL Id: hal-05498101

https://inria.hal.science/hal-05498101v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_4.png" /></figure>

# Active Learning Query Strategy Selection Using Dataset Meta-features Extraction

Vangjel Kazllarof and Sotiris Kotsiantis

Department of Mathematics, University of Patras, 26504 Rio Achaia, Greece
vkazlarof@upatras.gr

**Abstract.** In the age of information, data abundance has enabled scientists to create models that have great positive impact in our life and society. However, many times the rate of data production is much bigger than the rate of classifying them in the appropriate label due to complexity, personnel or cost of equipment for the labeling task. For this reason, Active Learning techniques have been developed with the Uncertainty Sampling being one of the most popular techniques for querying the unlabeled data. However, selecting the correct Query Strategy for ranking the uncertainty in order to create the best possible model is a time and cost consuming task and most of the times the Active Learning process needs to be repeated multiple times during training. In this work, we exploit the Meta-Features extracted by 123 datasets and select the winning Query Strategy among Least Confidence, Smallest Margin and Entropy for each dataset. In the sequence, we create a dataset with a subset of the extracted Meta-Features and the winning Query Strategy for each dataset and train it in order to create a Decision Tree that can be used in order to select the most suitable Query Strategy.

**Keywords:** Active Learning · Uncertainty Sampling · Meta-Features

## 1 Introduction

Active Learning have played a major role in creating performant models when the knowledge of the data labels is limited. Expensive equipment, specialized personnel and limited time have always been a burden in labeling acquired data [1]. For this reason, Active Learning techniques have been developed that help scientists to query and acquire the label only of the most important and informative data in order to achieve a performant model with limited knowledge [2].

On the one hand, the development of Active Learning techniques have helped a lot in acquiring efficiently the most informative instances. On the other hand, selecting the best of these techniques is not something trivial because every technique has its advantages and disadvantages and most of the times their results are dependent to the dataset itself [3]. In this work we take advantage of the Meta-Features extracted by a number of datasets in order to select the most performant Active Learning technique. Meta-Features are measures that are used mainly in the Meta-Learning field in order to characterize datasets along with their relation to algorithm bias [4].
The outline of the paper continues with Sect. 2 where we explain the Active Learning techniques used in the work while in Sect. 3 we demonstrate the Meta-Features in more detail. In Sect. 4 we analyze the related works on the subject. Section 5 is dedicated to the experiments and the results and we conclude in Sect. 6 with brief conclusions and the work that we plan to do in the future.

## 2 Active Learning

Active Learning process starts with a small amount of labeled data (L) and a much bigger amount of unlabeled data (U). In the beginning, a model is trained using the labeled data. Then, in the Pool-based scenario that is used in this work, the trained model is used for ranking the unlabeled instances on the informativeness they provide to it. A number of instances with the highest rank are then provided to an expert in order to label them. After acquiring the label, the instances are added to the labeled pool and the model is trained again, concluding an Active Learning cycle. The Active Learning cycles go on until a stopping criterion is met and the last trained model is the outcome of the whole process [5]. In Fig. 1 a visualization of Active Learning cycle in the Pool-based scenario is demonstrated.

<figure><img src="image_5.png" /><figcaption>Fig. 1. Pool-based Active Learning cycle</figcaption></figure>

For the sampling process, one of the most popular methods is the Uncertainty Sampling. In this method, the top ranked instances are the one that the model has the lowest confidence for its class. In the work, we used three popular query strategies for measuring the uncertainty, the Least Confidence (LC), the Smallest Margin (SM) and the Entropy (Ent) [5].
In the Least Confidence Query Strategy the active learner selects the instances that has the highest belief that they are misclassified and it is described by Eq. 1.

$$x_{LC} = \underset{x}{\operatorname{argmax}} 1 - P_{\text{model}}(\hat{y}|x) \qquad (1)$$

where $\hat{y}$ is the most probable label and equals and it is described by Eq. 2

$$\hat{y} = \underset{y}{\operatorname{argmax}} P_{\text{model}}(y|x) \qquad (2)$$

In the Smallest Margin Query Strategy the active learner also takes under consideration the distribution of the remaining labels. For this reason tries to find the instances that their top two ranked instances has the smallest margin between them and it is described by Eq. 3

$$x_{SM} = \underset{x}{\operatorname{argmin}} P_{\text{model}}(\hat{y}_1|x) - P_{\text{model}}(\hat{y}_2|x) \qquad (3)$$

where $\hat{y}_1$ and $\hat{y}_2$ are the first and second most probable labels respectively.

Entropy Query Strategy seeks to find the instances that increases the overall entropy of the model and it is described by the Eq. 4

$$x_{Ent} = \underset{x}{\operatorname{argmax}} 1 - \sum_{i} P_{\text{model}}(y_i|x) \log P_{\text{model}}(y_i|x) \qquad (4)$$

where $y_i$ ranges over all labels [5].

## 3 Meta-features Extraction

Meta-Features extraction is a process of obtaining knowledge of a dataset by measuring value characteristics and use them for data mining tasks. The meta-knowledge, as referred in the literature, is mainly used in Meta-Learning [6] and AutoML [7] fields in order to understand and characterize a dataset along with its learning biases and then provide this information to a learner in order to optimize it.

Generally a Meta-Feature is a function $f: D \to \mathbb{R}^s$ where $D$ is the dataset space using initially a measure function $m: D \to \mathbb{R}^m$ and then a summarization function $s: \mathbb{R}^m \to \mathbb{R}^s$ along with their hyper-parameters $h_m$ and $h_s$ respectively. A Meta-Feature function can be shortly described by Eq. 5 [8].

$$f(D) = s(m(D, h_m), h_s) \qquad (5)$$

Meta-Features can be distinguished on their nature of the measurement. This way they can be grouped in the following groups. General is the simplest group of measurements like number of instances, number of attributes or number of classes. Statistical is the group of measurements that describe the numerical attributes of data distribution while Information-theoretic group is the one that describe the categorical attributes and their relationship with the classes. Model-based Meta-Features are measurements that
describe characteristics that are extracted by simple models. Landmarking, Relative-Landmarking and Subsampling-Landmarking are a groups of performances, relative-performances and performances of dataset subsamples respectively of simple and efficient learners. Clustering is a group of measurements that describe the information extracted from external validation indexes. Concept measurements estimate the density of the instances along with the variability of the classes. Itemset group is the one that computes the correlation between binary attributes and finally the Complexity group consists of measurements that estimate how difficult the separation of instances into their classes is [9].

# 4 Related Works

In the literature, Meta-Learning is combined with Active Learning techniques mainly for reducing the meta-examples required to train by selecting the most informative one when trying to construct a reliable meta-learner. Such process has been developed in [10] in which Uncertainty Sampling methods have been combined with Active Meta-Learning in order to increase the Meta-Learning performance by generating more relevant meta-examples. Similar approach has been used in [11] in which they managed to achieve similar Meta-Learning performance by selecting only up to 20% of labeled meta-examples using Uncertainty Sampling methods. Moreover, in [12] Outlier Detection techniques were applied along with Active Learning and Meta-learning in order to remove meta-examples that were considered as outliers to the meta-learner resulting to an improved performance. Another similar approach was conducted in [13] where a combination of Meta-Learning and Active Learning has been used in improving the performance of peptide datasets using only few training examples.

Although Meta-Learning has been combined with Active Learning in order to improve the meta-learner, few have been published for improving the active-learner. In [14] Meta-Learning has been used for automatic tuning of the Uncertainty Sampling threshold in stream classification, improving the performance and the cost efficiency of the active-learner. Similarly, in this work we try to improve the cost efficiency of the Active Learning process by taking advantage of the Meta-Features and create a Decision Tree that can be used in order to predict the best Query Strategy for the active-learner.

# 5 Experiments and Results

## 5.1 Dataset Construction

For the dataset construction, we use 123 datasets sourced by OpenML [15]. Initially we extract all the Meta-Features listed in the groups mentioned above. For this task we used the Pymfe [9] package written in Python resulting to a dataset of 123 instances and 182 attributes that each one describe an extracted Meta-Feature.

In the sequence, we determine the winning Active Learning Query Strategy between Least Confidence, Smallest margin and Entropy methods for each dataset in order to label each instance. Initially every dataset is splitted into train and test sets. For the train set we use the 80% of instances and for the test set we use 20% of instances of each
dataset. Train set will be used for the Active Learning process while test set will be used for evaluating the performance of the trained model obtained after process finishes.

For the Active Learning process, every dataset train set starts with a small pool of initially labeled instances of 5% of the total train set in order simulate more realistically real-world problems. The remaining 95% will be the pool of the unlabeled instances in which the query strategies will search for the most informative one. The stopping criterion for the Active Learning task is the 15 iterations of Active Learning cycles and in every cycle a dynamic batch of top rated instances is selected in order to double the pool of the labeled instances and reach to 10% of the total train set. For the labeling process a simulated oracle is used that always predicts the correct label for the queried instances. For base learner we chose is simple Decision Tree classifier in order to have a fast training process.

The Active Learning task described above classified every instance on the winning Query Strategy resulting to a dataset of 42 instances classified to Least Confidence, 41 instances classified to Smallest Margin and the remaining 40 instances classified to Entropy Query Strategy. For the implementing the Active Learning task we used Libact [16] package along with Scikit-Learn [17] package, both written in Python.

## 5.2 Decision Tree Creation

For the Decision Tree construction we used a Java implementation of C4.5 [18] algorithm called J48 from the WEKA [19] software. After pruning, the resulted Decision Tree contained only 17 from the 182 Meta-Features meaning that only with a small subset of Meta-Features we can obtain a winning Query Strategy efficiently. Below we list the Meta-Features used by the Decision Tree along with their category group and their description:

* General

- **NrInst:** Number of instances [20]

* Statistical

- **Cor(mean):** Mean of absolute values of the correlation of distinct dataset column pairs [21]

- **MAD(mean):** Mean of the Median Absolute Deviation adjusted by a factor [22]

- **CanCor sd):** Standard distribution of canonical correlations of the data [23]

- **NrNorm:** Number of attributes normally distributed based in a given method [24]

* Model-based

- **LeavesPerClass sd):** Standard distribution of proportion of the leaves per class in Decision Tree model [25]

- **NodesPerAttr:** Ratio of nodes per number of attributes in Decision Tree model [26]

- **LeavesCorrob (sd):** Standard distribution of leaves corroboration of the Decision Tree model [26]
* Complexity

- **T1 sd):** Standard distribution of the fraction of hyperspheres covering data [27]

- **Density:** Average density of the network [27]

* Clustering

- **PB:** Pearson correlation between class matching and instance distances [28]

* Landmarking

- **NaiveBayesRel sd):** Standard distribution of relative performances of the Naive Bayes classifier [29]

- **EliteNN(mean):** Mean of the performances of Elite Nearest Neighbor [29]

- **EliteNN sd):** Standard distribution of performances of Elite Nearest Neighbor classifier [29]

- **OneNNRel sd):** Standard distribution of the relative performances of the 1-Nearest Neighbor classifier [29]

- **RndNodeRel sd):** Standard distribution of the relative performances of the single Decision Tree node model induced by a random attribute [29]

- **LinearDiscrRel sd):** Standard distribution of relative performances of the Linear Discriminant classifier [29]

In the following Rule Set, the rules of the produced Decision Tree are demonstrated. The tree consists of 45 nodes and has a height of 13 branches. From the nodes, 23 are leaves of the tree that describe the classification class along with the classified instances and the wrongly classified instances. The rest of the nodes describe the Meta-Features used and the vertices the value of each Meta-feature for the decision making process.
NrInst<=140
Cor(mean) <= 0.130823: LC(8.0)
Cor(mean) > 0.130823
EliteNN sd) <= 0.184284
PB <= -0.082785: LC(7.0/1.0)
PB > -0.082785: SM(5.0/1.0)
EliteNN (sd) > 0.184284: SM(6.0)
NrInst>140
NaiveBayesRel (sd) <= 2.5
LeavesPerClass (sd) <= 0.117851: SM(3.0)
LeavesPerClass (sd) > 0.117851
EliteNN (sd) <= 0.073421: SM(3.0/1.0)
EliteNN (sd) > 0.073421: LC(8.0/1.0)
NaiveBayesRel (sd) > 2.5
T1 (sd) <= 0.047321
OneNNRel (sd) <= 3
RndNodeRel (sd) <= 6: SM(6.0/1.0)
RndNodeRel (sd) > 6: Ent (3.0)
OneNNRel (sd) > 3
NodesPerAttr <= 176.333333
RndNodeRel (sd) <= 3.5
LeavesCorrob (sd) <= 0.003182: Ent (6.0)
LeavesCorrob (sd) > 0.003182
NaiveBayesRel (sd) <= 3: SM(2.0)
NaiveBayesRel (sd) > 3
LinearDiscrRel (sd) <= 3: SM(3.0/1.0)
LinearDiscrRel (sd) > 3
CanCor (sd) <= 0.309755
NrNorm <= 0
LinearDiscrRel (sd) <= 5
MAD (mean) <= 245.13684: LC(9.0/1.0)
MAD (mean) > 245.13684: Ent (2.0)
LinearDiscrRel (sd) > 5: LC(7.0/1.0)
NrNorm > 0: Ent (3.0/1.0)
CanCor (sd) > 0.309755: Ent (11.0/3.0)
RndNodeRel (sd) > 3.5: LC(3.0)
NodesPerAttr > 176.333333: SM(3.0)
T1 (sd) > 0.047321
MAD (mean) <= 0.320958
Density <= 0.998825: SM(8.0)
Density > 0.998825: Ent (2.0)
MAD (mean) > 0.320958
EliteNN (mean) <= 0.75: Ent (13.0)
EliteNN (mean) > 0.75: SM(2.0)
**Decision Tree (Rule Set).** Produced Decision Tree as a set of rules for selecting a winning Query Strategy using Meta-Features.

In order to evaluate the produced Decision Tree, we used the constructed dataset as the test set. In the Table 1 we demonstrate the confusion matrix of the tree.

<table><caption>**Table 1.** Confusion matrix of the produced Decision Tree</caption><thead><tr><td>Query Strategy</td><td>Least Confidence</td><td>Smallest Margin</td><td>Entropy</td></tr></thead><tbody><tr><td>Least Confidence</td><td>38</td><td>2</td><td>2</td></tr><tr><td>Smallest Margin</td><td>2</td><td>37</td><td>2</td></tr><tr><td>Entropy</td><td>2</td><td>2</td><td>36</td></tr></tbody></table>

From the results, it is shown that 111 instances are classified correctly and 12 incorrectly with each class having 4 incorrectly classified instances resulting to a classification accuracy of 90.24%.

In the Table 2, Precision, Recall and F-Measure of the produced Decision Tree are demonstrated.

<table><caption>**Table 2.** Precision, Recall and F-Measure of the produced Decision Tree</caption><thead><tr><td>Query Strategy</td><td>Precision</td><td>Recall</td><td>F-Measure</td></tr></thead><tbody><tr><td>Least Confidence</td><td>90.5%</td><td>90.5%</td><td>90.5%</td></tr><tr><td>Smallest Margin</td><td>90.2%</td><td>90.2%</td><td>90.2%</td></tr><tr><td>Entropy</td><td>90%</td><td>90%</td><td>90%</td></tr><tr><td><strong>Weighted Average</strong></td><td>90.2%</td><td>90.2%</td><td>90.2%</td></tr></tbody></table>

From the performance metrics, it is shown that the Decision Tree produces reliable classification results since both Precision and Recall are over 90% along with the F-Measure.

# 6 Conclusion and Future Work

In this work we constructed a dataset of Meta-Features and then produced a Decision Tree that can be used for selecting the appropriate Uncertainty Sampling Query Strategy for Active Learning tasks. The classification accuracy of the Decision Tree indicates that calculating a small set of Meta-Features in new datasets can be enough for selecting the appropriate Query Strategy without having to run all of them, saving a lot of time and resources.

In the future we would like to try using a more sophisticated base learner like Random Forest [30] for the Active Learning process and evaluate how the classes change and
how biased the produced Decision Tree can be. Moreover, we would like to extend the research to imbalance datasets [31] and to regression problems [32]. Finally, we would like to use a noisy oracle in order to give in the Active Learning process an error rate when evaluating the label of the datasets [33].

## References

1. Settles, B., Craven, M., Friedl, L.: Active learning with real annotation costs. In: Proceedings of the NIPS Workshop on Cost-Sensitive Learning, pp. 1–10 (2008)

2. Tharwat, A., Schenck, W.: A survey on active learning: state-of-the-art practical challenges and research directions. Mathematics **11**(4), 820 (2023). https://doi.org/10.3390/MATH11040820

3. Kazllarof, V., Karlos, S., Kotsiantis, S.: Investigation of combining logitboost (M5P) under active learning classification tasks. Informatics **7**(4), 50 (2020). https://doi.org/10.3390/informatics7040050

4. Brazdil, P., Giraud-Carrier, C., Soares, C., Vilalta, R.: Metalearning (2009). https://doi.org/10.1007/978-3-540-73263-1

5. Settles, B.: Active learning. Synth. Lect. Artif. Intell. Mach. Learn. **18**, 1–111 (2012). https://doi.org/10.2200/S00429ED1V01Y201207AIM018

6. Rivolli, A., Garcia, L.P.F., Soares, C., Vanschoren, J., de Carvalho, A.C.P.L.F.: Meta-features for meta-learning. Knowl.-Based Syst. **240**, 108101 (2022). https://doi.org/10.1016/J.KNO SYS.2021.108101

7. Rakotoarison, H., Milijaona, L., Rasoanaivo, A., Sebag, M., Schoenauer, M.: Learning Meta-features for AutoML (2022). https://hal.inria.fr/hal-03583789v2. Accessed 07 Mar 2023

8. Rivolli, A., Garcia, L.P.F., Soares, C., Vanschoren, J., de Carvalho, A.C.P.L.F.: Characterizing classification datasets: a study of meta-features for meta-learning (2018). https://doi.org/10.48550/arxiv.1808.10406

9. Alcobaça, E., Siqueira, F., Rivolli, A., Garcia, L.P.F., Oliva, J.T., de Carvalho, A.C.P.L.F.: MFE: towards reproducible meta-feature extraction. J. Mach. Learn. Res. **21**(111), 1–5 (2020). http://jmlr.org/papers/v21/19-348.html. Accessed 07 Mar 2023

10. Prudêncio, R.B.C., Ludermir, T.B.: Combining uncertainty sampling methods for active meta-learning. In: ISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications, pp. 220–225 (2009). https://doi.org/10.1109/ISDA.2009.160

11. Prudêncio, R.B.C., Soares, C., Ludermir, T.B.: Uncertainty sampling-based active selection of datasetoids for meta-learning. In: Honkela, T., Duch, W., Girolami, M., Kaski, S. (eds.) ICANN 2011. LNCS, vol. 6792, pp. 454–461. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21738-8_58

12. Prudêncio, R.B.C., Ludermir, T.B.: Active meta-learning with uncertainty sampling and outlier detection. In: Proceedings of International Joint Conference on Neural Networks, pp. 346–351 (2008). https://doi.org/10.1109/IJCNN.2008.4633815

13. Barrett, R., White, A.D.: Investigating active learning and meta-learning for iterative peptide design. J. Chem. Inf. Model. **61**(1), 95–105 (2021). https://doi.org/10.1021/ACS.JCIM.0C00946/ASSET/IMAGES/LARGE/C10C00946_0006.JPEG

14. Martins, V.E., Cano, A., Junior, S.B.: Meta-learning for dynamic tuning of active learning on stream classification. Pattern Recognit. **138**, 109359 (2023). https://doi.org/10.1016/J.PAT COG.2023.109359

15. Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: OpenML: networked science in machine learning. ACM SIGKDD Explor. Newsl. **15**(2), 49–60 (2014). https://doi.org/10.1145/2641190.2641198
16. Yang, Y.-Y., Lee, S.-C., Chung, Y.-A., Wu, T.-E., Chen, S.-A., Lin, H.-T.: libact: Pool-based Active Learning in Python (2017). https://doi.org/10.48550/arxiv.1710.00379

17. Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. **12**, 2825–2830 (2012). https://doi.org/10.48550/arxiv.1201.0490

18. Salzberg, S.L.: C4.5: programs for machine learning by J. Ross Quinlan. Morgan Kaufmann Publishers, Inc., 1993. Mach. Learn. **16**(3), 235–240 (1994). https://doi.org/10.1007/BF00993309

19. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: an update. SIGKDD Explor. **11**(1) (2009). https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.148.3671. Accessed 07 Mar 2021

20. Fulkerson, B., Michie, D., Spiegelhalter, D.J., Taylor, C.C.: Machine learning, neural and statistical classification. Technometrics **37**(4), 459 (1995). https://doi.org/10.2307/1269742

21. Castiello, C., Castellano, G., Fanelli, A.M.: Meta-data: characterization of input features for meta-learning. In: Torra, V., Narukawa, Y., Miyamoto, S. (eds.) MDAI 2005. LNCS (LNAI), vol. 3558, pp. 457–468. Springer, Heidelberg (2005). https://doi.org/10.1007/11526018_45

22. Ali, S., Smith, K.A.: On learning algorithm selection for classification. Appl. Soft Comput. **6**(2), 119–138 (2006). https://doi.org/10.1016/J.ASOC.2004.12.002

23. Kalousis, A.: Algorithm selection via meta-learning (2002). https://doi.org/10.13097/ARC_HIVE-OUVERTE/UNIGE:104435

24. Kopf, C., Taylor, C.: Meta-Analysis: From Data Characterisation for Meta-Learning to Meta-Regression (2000)

25. Filchenkov, A., Pendryak, A.: Datasets meta-feature description for recommending feature selection algorithm. In: Proceedings of Artificial Intelligence and Natural Language and Information Extraction, Social Media and Web Search FRUCT Conference AINL-ISMW FRUCT 2015, pp. 11–18 (2016). https://doi.org/10.1109/AINL-ISMW-FRUCT.2015.7382962

26. Bensusan, H., Giraud-Carrier, C., Kennedy, C.: A higher-order approach to meta-learning, pp. 109–117 (2000). https://research-information.bris.ac.uk/en/publications/a-higher-order-approach-to-meta-learning. Accessed 07 Mar 2023

27. Lorena, A.C., Garcia, L.P.F., Lehmann, J., Souto, M.C.P., Ho, T.K.A.M.: How complex is your classification problem? ACM Comput. Surv. **52**(5), 1–34 (2019). https://doi.org/10.1145/3347711

28. Gupta, S.D.: Point biserial correlation coefficient and its generalization. Psychometrika **25**(4), 393–408 (1960). https://doi.org/10.1007/BF02289756/METRICS

29. Bensusan, H., Giraud-Carrier, C.: Discovering task neighbourhoods through landmark learning performances. In: Zighed, D.A., Komorowski, J., Zytkow, J. (eds.) PKDD 2000. LNCS (LNAI), vol. 1910, pp. 325–330. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45372-5_32

30. Breiman, L.: Random forests. Mach. Learn. **45**(1), 5–32 (2001). https://doi.org/10.1023/A:1010933404324/METRICS

31. Aggarwal, U., Popescu, A., Hudelot, C.: Active Learning for Imbalanced Datasets, pp. 1428–1437 (2020)

32. Wu, D., Lin, C.T., Huang, J.: Active learning for regression using greedy sampling. Inf. Sci. **474**, 90–105 (2018). https://doi.org/10.1016/j.ins.2018.09.060

33. Gupta, G., Sahu, A.K., Lin, W.Y.: Noisy Batch Active Learning with Deterministic Annealing (2019). http://arxiv.org/abs/1909.12473. Accessed 13 Mar 2021