# Improving Active Learning Performance through the Use of Data Augmentation

Authors: Joao Fonseca; Fernando Bacao
Journal: International Journal of Intelligent Systems
 (2023)

---

## Abstract

Active learning (AL) is a well-known technique to optimize data usage in training, through the interactive selection of unlabeled observations, out of a large pool of unlabeled data, to be labeled by a supervisor. Its focus is to find the unlabeled observations that, once labeled, will maximize the informativeness of the training dataset, therefore reducing data-related costs. The literature describes several methods to improve the effectiveness of this process. Nonetheless, there is a paucity of research developed around the application of artificial data sources in AL, especially outside image classification or NLP. This paper proposes a new AL framework, which relies on the effective use of artificial data. It may be used with any classifier, generation mechanism, and data type and can be integrated with multiple other state-of-the-art AL contributions. This combination is expected to increase the ML classifier’s performance and reduce both the supervisor’s involvement and the amount of required labeled data at the expense of a marginal increase in computational time. The proposed method introduces a hyperparameter optimization component to improve the generation of artificial instances during the AL process as well as an uncertainty-based data generation mechanism. We compare the proposed method to the standard framework and an oversampling-based active learning method for more informed data generation in an AL context. The models’ performance was tested using four different classifiers, two AL-specific performance metrics, and three classification performance metrics over 15 different datasets. We demonstrated that the proposed framework, using data augmentation, significantly improved the performance of AL, both in terms of classification performance and data selection efficiency (all the codes and preprocessed data developed for this study are available athttps://github.com/joaopfonseca/publications/).


---

## Body

## 1. Introduction

The importance of training robust ML models with minimal data requirements is substantially increasing [1–3]. Although the growing amount of valuable data sources and formats being developed and explored is affecting various domains [4], these data are often unlabeled. Only a tiny amount of the data being produced and stored can be helpful in supervised learning tasks. In addition, it is often difficult and expensive to label data for specific machine learning (ML) projects, especially when data-intensive ML techniques are involved (e.g., deep learning classifiers) [1]. In this scenario, labeling the full dataset becomes impractical, time-consuming, and expensive. Two different ML techniques attempt to address this problem: semisupervised learning (SSL) and active learning (AL). Even though they address the same problem, the two follow different approaches. SSL focuses on observations with the most certain predictions, whereas AL focuses on observations with the least certain predictions [5].SSL attempts to use a small, predefined set of labeled and unlabeled data to produce a classifier with superior performance. This method uses the unlabeled observations to help define the classifier’s decision boundaries [6]. Simultaneously, the amount of labeled data required to reach a given performance threshold is also reduced. It is a particular case of ML because it falls between the supervised and unsupervised learning perspectives. AL, instead of optimizing the informativeness of an existing training set, expands the dataset to include the most informative and/or representative observations [7]. It is an iterative process where a supervised model is trained and simultaneously identifies the most informative unlabeled observations to increase the performance of that classifier. The combination of SSL with AL has been explored in the past, achieving state-of-the-art results [8].Several studies have pointed out the limitations of AL within an imbalanced learning context [9, 10]. With imbalanced data, AL approaches frequently have low performance, high computational time, or data annotation costs. Studies addressing this issue tend to adopt classifier-level modifications, such as the weighted extreme learning machine [9, 11, 12]. However, classifier or query function-level modifications (see Section 2.1) have limited applicability since a universally good AL strategy has not yet been found [7]. Other methods address imbalanced learning by weighing the observations as the function of the observation’s class imbalance ratio [13]. Alternatively, other techniques reduce the imbalanced learning bias by combining informative and representative-based query approaches (see Section 2.1) [14]. Another approach to deal with imbalanced data and data scarcity, in general, is generating synthetic data [15]. This approach has the advantage of being classifier-agnostic; it potentially reduces the imbalanced learning bias and also works as a regularization method in data-scarce environments, such as AL implementations [16]. However, most recent studies improve the AL performance by modifying the design/choice of the classifier and query functions used.Recently, synthetic data generation techniques gathered attention among ML researchers for its effectiveness over a wide range of applications: regularization, oversampling, semisupervised learning, self-supervised learning, etc. Data augmentation generates synthetic observations to complement naturally occurring observations. It aims to reinforce the definition of a ML classifier’s decision boundary during the learning phase and improve the generalization of the algorithm. These techniques have the advantage of being a data level technique (despite the existence of augmentation methods applied internally in the ML classifier). Therefore, they can be implemented in a way that will not affect the choice of classifier and does not exclude the usage of other regularization approaches. In an AL context, the generation of synthetic data becomes particularly appealing, especially with randomized and statistical-based approaches; it ensures better model performance with reduced involvement of a human agent, at the expense of a marginal increase in computational power. In addition, synthetic data is expected to reduce the amount of labeled data required for a good AL implementation.Figure1 illustrates the difference across AL iterations between the standard AL approach and the proposed method. Synthetic data allowed for a quicker expansion of the labeled input area at an early stage of the process, with better defined decision boundaries and near-convergence of the ML classifier’s performance at the third iteration. Data augmentation influences the choice of unlabeled observations for labeling into regions where synthetic data is not being able to represent the unlabeled data pool.Figure 1
Illustration of the different acquisition processes in AL using a K-nearest neighbors classifier and Shannon’s entropy as the uncertainty estimation function, with five observations being collected and labeled per iteration. The top row shows the behavior of a standard AL implementation, while the bottom row shows the behavior of the proposed method. ((a)–(c)) The decision boundaries at iterations 1 (after the collection of five random initial training observations), 2 (with 10 labeled observations), and 3 (with 15 labeled observations), respectively. The initial labeled dataset for both approaches is the same. The two classes were distinguished with△ and × and are colored as red and blue, respectively, if they are labeled. The transparent green observations are synthetic observations (bottom row).
(a)(b)(c)


### 1.1. Motivation and Contributions

The usage of data augmentation in AL is not new. The literature found on the topic (see Section2.3) focuses on either image classification or natural language processing and uses deep learning-based data augmentation to improve the performance of neural network architectures in AL. These methods, although showing promising results, represent a limited perspective of the potential of data augmentation in a real-world setting:(1)
Using deep learning in an iterative setting requires access to significant computational power.(2)
These models tend to use sophisticated data augmentation methods, whose implementation may not be accessible to nonsophisticated users.(3)
They require a significant amount of processing time per iteration and are inappropriate for settings with limited time budgets.(4)
The studies found on the topic are specific to the domain, classifier, and data augmentation method. In addition, all of the related methods found (except one) focus on either image or natural language processing classification problems.Consequently, the direct effect of data augmentation is unclear: these studies implement different neural network-based techniques for different classification problems, whose performance may be attributed to various elements within the AL framework.In this study, we explore the effect of data augmentation in AL in a context-agnostic setting, along with two different data augmentation policies: oversampling (where the amount of data generated for each class equals the amount of data belonging to the majority class) and nonconstant data augmentation policies (where the amount of data generated exceed the amount of data belonging to the majority class in varying quantities) between iterations. We start by conceptualizing the AL framework and each of its elements, as well as the modifications involved to implement data augmentation in the AL iterative process. We argue that simple nondomain-specific data augmentation heuristics are sufficient to improve the performance of AL implementations, without the need to resort to deep learning-based data augmentation algorithms. These contributions can be summarized as follows:(1)
We propose a flexible AL framework with pipelined data augmentation for tabular data that may be adapted for any domain or data type. This implementation is directed towards the use cases with limited computational power and/or processing time.(2)
We use a geometric-based data augmentation method for non-network-based classifiers and adapt it to leverage information from the AL process. To the best of our knowledge, most existing methods use domain/classifier-specific augmentations or the mixup approach, which is incompatible with most classifiers that are not neural network-based.(3)
We provide empirical evidence that the integration of varying data augmentation policies between iterations in the AL framework not only further reduces the amount of labeled data required, but is also a viable training strategy for fully supervised learning settings.When compared to the standard AL framework, the proposed framework contains two additional components: the generator and the hyperparameter optimizer. We implemented a modified version of the Geometric Synthetic Minority Oversampling Technique (G-SMOTE) [17] as a data augmentation method with an optimized generation policy (explained in Section 2.2). We also proposed a hyperparameter optimization module, which is used to find the best data augmentation policy at each iteration. We tested the effectiveness of the proposed method in 15 datasets of different domains. We implement three AL frameworks (standard, oversampling, and varying data augmentation) using four different classifiers and three different performance metrics and calculated two AL-specific performance metrics.The remainder of this article is structured as follows: Section2 introduces relevant topics discussed in the paper and describes the related work. Section 3 elucidates the proposed method. Section 4 details the methodology of the study’s experiment. Section 5 presents the results obtained from the experiment, as well as a discussion of these results. Section 6 presents the conclusion and future directions drawn from this study.


## 1.1. Motivation and Contributions

The usage of data augmentation in AL is not new. The literature found on the topic (see Section2.3) focuses on either image classification or natural language processing and uses deep learning-based data augmentation to improve the performance of neural network architectures in AL. These methods, although showing promising results, represent a limited perspective of the potential of data augmentation in a real-world setting:(1)
Using deep learning in an iterative setting requires access to significant computational power.(2)
These models tend to use sophisticated data augmentation methods, whose implementation may not be accessible to nonsophisticated users.(3)
They require a significant amount of processing time per iteration and are inappropriate for settings with limited time budgets.(4)
The studies found on the topic are specific to the domain, classifier, and data augmentation method. In addition, all of the related methods found (except one) focus on either image or natural language processing classification problems.Consequently, the direct effect of data augmentation is unclear: these studies implement different neural network-based techniques for different classification problems, whose performance may be attributed to various elements within the AL framework.In this study, we explore the effect of data augmentation in AL in a context-agnostic setting, along with two different data augmentation policies: oversampling (where the amount of data generated for each class equals the amount of data belonging to the majority class) and nonconstant data augmentation policies (where the amount of data generated exceed the amount of data belonging to the majority class in varying quantities) between iterations. We start by conceptualizing the AL framework and each of its elements, as well as the modifications involved to implement data augmentation in the AL iterative process. We argue that simple nondomain-specific data augmentation heuristics are sufficient to improve the performance of AL implementations, without the need to resort to deep learning-based data augmentation algorithms. These contributions can be summarized as follows:(1)
We propose a flexible AL framework with pipelined data augmentation for tabular data that may be adapted for any domain or data type. This implementation is directed towards the use cases with limited computational power and/or processing time.(2)
We use a geometric-based data augmentation method for non-network-based classifiers and adapt it to leverage information from the AL process. To the best of our knowledge, most existing methods use domain/classifier-specific augmentations or the mixup approach, which is incompatible with most classifiers that are not neural network-based.(3)
We provide empirical evidence that the integration of varying data augmentation policies between iterations in the AL framework not only further reduces the amount of labeled data required, but is also a viable training strategy for fully supervised learning settings.When compared to the standard AL framework, the proposed framework contains two additional components: the generator and the hyperparameter optimizer. We implemented a modified version of the Geometric Synthetic Minority Oversampling Technique (G-SMOTE) [17] as a data augmentation method with an optimized generation policy (explained in Section 2.2). We also proposed a hyperparameter optimization module, which is used to find the best data augmentation policy at each iteration. We tested the effectiveness of the proposed method in 15 datasets of different domains. We implement three AL frameworks (standard, oversampling, and varying data augmentation) using four different classifiers and three different performance metrics and calculated two AL-specific performance metrics.The remainder of this article is structured as follows: Section2 introduces relevant topics discussed in the paper and describes the related work. Section 3 elucidates the proposed method. Section 4 details the methodology of the study’s experiment. Section 5 presents the results obtained from the experiment, as well as a discussion of these results. Section 6 presents the conclusion and future directions drawn from this study.


## 2. Background

In this section, we describe the AL problem and data augmentation techniques and review the literature that combines AL with data augmentation. Table1 describes the notations used throughout the rest of this study.Table 1
Description of all notations and symbols used throughout the article.
SymbolMeaningfcML classifierxiObservation at indexifacqxi;fcAcquisition functionfaugxi;τAugmentation functionτAugmentation policyDData pool contains both labeled and unlabeled dataDlabtLabeled data set at iterationtDpooltUnlabeled data pool at iterationtDnewtSet of observations fromDpoolt to be labeled and added to Dlabt+1TIteration budgetnAnnotation budget per iteration


### 2.1. Active Learning

This paper focuses on pool-based AL methods as defined in [18]. The goal of AL models is to maximize the performance of a classifier, fc, while annotating as least observations, xi, as possible. They use a data pool, D, where D=Dlab∪Dpool and Dpool≫Dlab. Dpool and Dlab refer to the sets of unlabeled and labeled data, respectively. Having a budget of T iterations (where t∈1,2,…,T) and n annotations per iteration, at iteration t, fc is trained using Dlabt to produce, for each xi∈Dpoolt, an uncertainty score using an acquisition function facqxi;fc. These uncertainty scores are used to annotate the n observations with highest uncertainty from Dpoolt to form Dnewt. The iteration ends with the update of Dlabt+1=Dlabt∪Dnewt and Dpoolt+1=Dpoolt/Dnewt [2, 19]. This process is shown in Figure 2. Before the start of the iterative process, assuming Dlabt=0=∅, the data used to populate Dlabt=1 are typically collected randomly from D=Dpoolt=0 and are labeled by a supervisor [20–22].Figure 2
Diagram depicting a typical AL iteration. In the first iteration, the training set collected during the initialization process becomes the “current training dataset.”Research focused on AL has typically been focused on the specification offacq [23] and domain-specific applications, such as malware detection [24] or land use/land cover classification [25]. Acquisition functions can be divided into two different categories [26, 27]:(1)
Informative-Based. These strategies use the classifier’s output to assess the importance of each observation towards the performance of the classifier [28].(2)
Representative-Based. These strategies estimate the optimal set of observations that will optimize the classifier’s performance [27].Although there are significant contributions toward the development of more robust query functions and classifiers in AL, modifications to AL’s basic structure are rarely explored. In [21], Yoo and K introduce a loss prediction module in the AL framework to replace the uncertainty criterion. This model implements a second classifier to predict the expected loss of the unlabeled observations (using the actual losses collected during the training of the original classifier) and return the unlabeled observations with the highest expected loss. However, this contribution is specific to deep neural networks and was only tested for image classification.AL techniques may also be used to complement other well-known learning challenges. For example, security bug report prediction tasks are typically developed in imbalanced learning environment, where it is necessary to manually label large amounts of data, which may result in mislabeled data [29]. Another related example is machinery fault diagnostics, where the quality and quantity of the data collected is often recognized as a bottleneck [30]. In this case, ML-based techniques frequently leverage unlabeled data to improve the classification performance [31] and rely on manual data acquisition [32]. In these examples, the application of an AL technique could reduce the amount of labeled data required, reduce the strain in the supervisor’s labeling process, and reduce the amount of label noise.An under explored challenge in the AL literature is the effective handling of different data structures. One method to address this problem is autoencoder architectures [33] or, in the case of text data, semantic representation networks [34]. However, understanding how to integrate these two types of methods is a subject of future research. Within other research streams, such as deep reinforcement learning, some research also focus on optimizing observation efficiency during the learning process [35].


### 2.2. Data Augmentation

The standard AL model can be complemented with a data augmentation function,faugxi;τ, where τ defines the augmentation policy. In this context, τ refers to the transformation applied and its hyperparameters and faug produces a modified observation, x˜∈Daug where Daug is the set of modified observations. This involves the usage of a new set of data, Dtraint=Dlabt∪Daugt, to train the classifier.Data augmentation methods expand the training dataset by introducing new and informative observations [36]. The production of artificial data may be conducted via the introduction of perturbations on the input [37], feature [38], or output space [36]. Data augmentation methods may be divided into two categories [39]:(1)
Heuristic approaches attempt to generate new and relevant observations by applying a predefined procedure, usually incorporating some degree of randomness [40]. Since these methods typically occur in the input space, they require fewer data and computational power when compared to neural network methods.(2)
Neural network approaches, on the other hand, map the original input space into a lower-dimensional representation, known as the feature space [38]. The generation of artificial data occurs in the feature space and is reconstructed into the input space. Although these methods allow the generation of less noisy data in high-dimensional contexts and more plausible artificial data, they are significantly more computationally intensive.While some techniques may depend on the domain, others are domain-agnostic. Random erasing [41], translation, cropping, and flipping are the examples of image data-specific augmentation methods. Other methods, such as autoencoders, may be considered domain agnostic.


### 2.3. Data Augmentation in Active Learning

The only AL model found that uses data augmentation outside of the computer vision or NLP domains implements a pipelined approach, described in [20]. In this study, the AL model proposed was applied for tabular data using an oversampling data augmentation policy (i.e., the artificial data was only generated to balance the target class frequencies). However, this AL model was applied in a land use/land cover classification context with specific characteristics that are not necessarily found in other supervised learning problems. Specifically, these types of datasets are high dimensional and have limited data variability within each class (i.e., cohesive spectral signatures within classes) due to their geographical proximity. Furthermore, this method does not allow augmentation policy optimization (i.e., every hyperparameter has to be hard-coded a priori).The Bayesian generative active deep learning (BGDAL) [42] is another example of a pipelined combination of facq and faug, applied to image classification. BGDAL uses a variational autoencoder (VAE) architecture to generate artificial observations. However, the proposed model is computationally expensive, requires a large data pool to train the VAE, and is not only dependent on the quality of the augmentations performed but also on the performance of the discriminator and classifiers used.The method proposed in [16], look-ahead data acquisition for deep active learning, implements data augmentation to train a deep learning classifier. However, adapting existing AL applications to use this approach is often impractical and implies the usage of image data since the augmentations used are image data specific and occur on the unlabeled observations, before the unlabeled data selection.The variational adversarial active learning (VAAL) model [43] is a deep AL approach to image classification that uses inputs as the embeddings produced by a VAE into a secondary classifier, working as facq, to predict if xi∈D belongs to Dpool. The n true positives with the highest uncertainty are labeled by the supervisor and Dpool and Dlab are updated as described in Section 2.1. The task-aware VAAL model [44] extends the VAAL model by introducing a ranker, which consists of the learning loss module introduced in [21]. These models use data augmentation techniques to train the different neural network-based components of the proposed models. However, the AL components used are specific image classification, computationally expensive, and the analysis of the effect of data augmentation in these AL models is not discussed.In [45], the proposed AL method was explicitly designed for image data classification, where a deep learning model was implemented as a classifier, but its architecture is not described, the augmentation policies used are unknown and the results reported correspond to single runs of the discussed model. The remaining AL models found implement data augmentation for NLP applications in [46, 47]. However, these methods were designed for specific applications within that domain and are not necessarily transferable to other domains or tasks.


## 2.1. Active Learning

This paper focuses on pool-based AL methods as defined in [18]. The goal of AL models is to maximize the performance of a classifier, fc, while annotating as least observations, xi, as possible. They use a data pool, D, where D=Dlab∪Dpool and Dpool≫Dlab. Dpool and Dlab refer to the sets of unlabeled and labeled data, respectively. Having a budget of T iterations (where t∈1,2,…,T) and n annotations per iteration, at iteration t, fc is trained using Dlabt to produce, for each xi∈Dpoolt, an uncertainty score using an acquisition function facqxi;fc. These uncertainty scores are used to annotate the n observations with highest uncertainty from Dpoolt to form Dnewt. The iteration ends with the update of Dlabt+1=Dlabt∪Dnewt and Dpoolt+1=Dpoolt/Dnewt [2, 19]. This process is shown in Figure 2. Before the start of the iterative process, assuming Dlabt=0=∅, the data used to populate Dlabt=1 are typically collected randomly from D=Dpoolt=0 and are labeled by a supervisor [20–22].Figure 2
Diagram depicting a typical AL iteration. In the first iteration, the training set collected during the initialization process becomes the “current training dataset.”Research focused on AL has typically been focused on the specification offacq [23] and domain-specific applications, such as malware detection [24] or land use/land cover classification [25]. Acquisition functions can be divided into two different categories [26, 27]:(1)
Informative-Based. These strategies use the classifier’s output to assess the importance of each observation towards the performance of the classifier [28].(2)
Representative-Based. These strategies estimate the optimal set of observations that will optimize the classifier’s performance [27].Although there are significant contributions toward the development of more robust query functions and classifiers in AL, modifications to AL’s basic structure are rarely explored. In [21], Yoo and K introduce a loss prediction module in the AL framework to replace the uncertainty criterion. This model implements a second classifier to predict the expected loss of the unlabeled observations (using the actual losses collected during the training of the original classifier) and return the unlabeled observations with the highest expected loss. However, this contribution is specific to deep neural networks and was only tested for image classification.AL techniques may also be used to complement other well-known learning challenges. For example, security bug report prediction tasks are typically developed in imbalanced learning environment, where it is necessary to manually label large amounts of data, which may result in mislabeled data [29]. Another related example is machinery fault diagnostics, where the quality and quantity of the data collected is often recognized as a bottleneck [30]. In this case, ML-based techniques frequently leverage unlabeled data to improve the classification performance [31] and rely on manual data acquisition [32]. In these examples, the application of an AL technique could reduce the amount of labeled data required, reduce the strain in the supervisor’s labeling process, and reduce the amount of label noise.An under explored challenge in the AL literature is the effective handling of different data structures. One method to address this problem is autoencoder architectures [33] or, in the case of text data, semantic representation networks [34]. However, understanding how to integrate these two types of methods is a subject of future research. Within other research streams, such as deep reinforcement learning, some research also focus on optimizing observation efficiency during the learning process [35].


## 2.2. Data Augmentation

The standard AL model can be complemented with a data augmentation function,faugxi;τ, where τ defines the augmentation policy. In this context, τ refers to the transformation applied and its hyperparameters and faug produces a modified observation, x˜∈Daug where Daug is the set of modified observations. This involves the usage of a new set of data, Dtraint=Dlabt∪Daugt, to train the classifier.Data augmentation methods expand the training dataset by introducing new and informative observations [36]. The production of artificial data may be conducted via the introduction of perturbations on the input [37], feature [38], or output space [36]. Data augmentation methods may be divided into two categories [39]:(1)
Heuristic approaches attempt to generate new and relevant observations by applying a predefined procedure, usually incorporating some degree of randomness [40]. Since these methods typically occur in the input space, they require fewer data and computational power when compared to neural network methods.(2)
Neural network approaches, on the other hand, map the original input space into a lower-dimensional representation, known as the feature space [38]. The generation of artificial data occurs in the feature space and is reconstructed into the input space. Although these methods allow the generation of less noisy data in high-dimensional contexts and more plausible artificial data, they are significantly more computationally intensive.While some techniques may depend on the domain, others are domain-agnostic. Random erasing [41], translation, cropping, and flipping are the examples of image data-specific augmentation methods. Other methods, such as autoencoders, may be considered domain agnostic.


## 2.3. Data Augmentation in Active Learning

The only AL model found that uses data augmentation outside of the computer vision or NLP domains implements a pipelined approach, described in [20]. In this study, the AL model proposed was applied for tabular data using an oversampling data augmentation policy (i.e., the artificial data was only generated to balance the target class frequencies). However, this AL model was applied in a land use/land cover classification context with specific characteristics that are not necessarily found in other supervised learning problems. Specifically, these types of datasets are high dimensional and have limited data variability within each class (i.e., cohesive spectral signatures within classes) due to their geographical proximity. Furthermore, this method does not allow augmentation policy optimization (i.e., every hyperparameter has to be hard-coded a priori).The Bayesian generative active deep learning (BGDAL) [42] is another example of a pipelined combination of facq and faug, applied to image classification. BGDAL uses a variational autoencoder (VAE) architecture to generate artificial observations. However, the proposed model is computationally expensive, requires a large data pool to train the VAE, and is not only dependent on the quality of the augmentations performed but also on the performance of the discriminator and classifiers used.The method proposed in [16], look-ahead data acquisition for deep active learning, implements data augmentation to train a deep learning classifier. However, adapting existing AL applications to use this approach is often impractical and implies the usage of image data since the augmentations used are image data specific and occur on the unlabeled observations, before the unlabeled data selection.The variational adversarial active learning (VAAL) model [43] is a deep AL approach to image classification that uses inputs as the embeddings produced by a VAE into a secondary classifier, working as facq, to predict if xi∈D belongs to Dpool. The n true positives with the highest uncertainty are labeled by the supervisor and Dpool and Dlab are updated as described in Section 2.1. The task-aware VAAL model [44] extends the VAAL model by introducing a ranker, which consists of the learning loss module introduced in [21]. These models use data augmentation techniques to train the different neural network-based components of the proposed models. However, the AL components used are specific image classification, computationally expensive, and the analysis of the effect of data augmentation in these AL models is not discussed.In [45], the proposed AL method was explicitly designed for image data classification, where a deep learning model was implemented as a classifier, but its architecture is not described, the augmentation policies used are unknown and the results reported correspond to single runs of the discussed model. The remaining AL models found implement data augmentation for NLP applications in [46, 47]. However, these methods were designed for specific applications within that domain and are not necessarily transferable to other domains or tasks.


## 3. Proposed Method

Based on the literature found on AL, most of the contributions and novel implementations of AL algorithms have focused on the improvement of the choice/architecture of the classifier or the improvement of the uncertainty criterion. In addition, the resulting classification performance of AL-trained classifiers is frequently inconsistent and marginally improves the classification performance when compared to classifiers trained over the entire training set. In addition, there is also significant variability in the data selection efficiency during different runs of the AL iterative process [20].This paper provides a context-agnostic AL framework for the integration of data augmentation within AL, with the following contributions:(1)
Improvement of the AL framework by introducing a parameter tuning stage only using the labeled dataset available at the current iteration (i.e., no labeled hold-out set is needed).(2)
Generalization of the generator module proposed in [20] from oversampling techniques to any other data augmentation mechanism and/or policy.(3)
Implementation of data augmentation outside the deep AL realm, which was not previously found in the literature.(4)
Analysis of the impact of data augmentation and oversampling in AL over 15 different datasets of different domains, while comparing them with the standard AL framework.The proposed AL framework is depicted in Figure3. The generator element becomes an additional source of data and is expected to introduce additional data variability into the training dataset. This aspect should allow the classifier to generalize better and perform more consistently over unseen observations. However, in this scenario, the amount of data to generate per class at each iteration is unknown. Consequently, the hyperparameter tuning step was introduced to estimate the optimal data augmentation policy at each iteration. In our implementation, this step uses the current training dataset to perform an exhaustive search over specified generator parameters, tested over a 5-fold cross-validation method. The best augmentation policy found is used to train the iteration’s classifier in the following step. This procedure is described in Algorithm 1.Figure 3
Diagram depicting the proposed AL iteration. The proposed modifications are comprised within the red polygon and marked with a boldface “C.”We implemented a simple modification in the selection mechanism of the G-SMOTE algorithm to show the effectiveness of data augmentation in an AL implementation. We use the uncertainties produced byfacq to compute the probabilities of observations to be selected for augmentation as an additional parameter. This modification is described in Algorithm 2.Algorithm 1: Proposed AL framework (single iteration).
Given:t≥1, performance metric fpmInput:Dpool, Dlab, fc, faug, facq, τgrid, k, nOutput:Dpool, DlabFunctionParameterTuning (fc, faug,τgrid,Dlab, k):p←0τ←∅Dlab1,…Dlabk←Dlab//Dlabn∩Dlabm=∅,∀n,m∈1,…,kforallτ′∈τgriddop′←∅forallDlabi∈Dlab1,…DlabkdoDtest′←DlabiDtrain′←Dlab/DlabiDtrain′←faugDtrain′;τ′trainfc using Dtrain′p′←p′∪fpmfcDtestp′←∑xi∈p′xi/kifp′>pthenp←p′τ←τ′returnτbeginτ←ParameterTuningfc,faug,τgrid,Dlab,kDtrain←faugDlab;τtrainfc using DtrainDnew=argmaxDpool′⊂Dpool,Dpool′=n∑x∈Dpool′facqx;fcannotateDnewDpool←Dpool/DnewDlab←Dlab∪DnewThis modification facilitates the usage of G-SMOTE beyond its original oversampling purposes. However, in this paper, the data augmentation strategies are also used to ensure that class frequencies are balanced. Furthermore, the amount of artificial data produced for each class is defined by theaugmentation factor, αaf, which represents a percentage of the majority class Cmaj (e.g., an augmentation factor of 1.2 will ensure there are countCmaj×1.2 observations in every class). In this paper’s experiment, the data generation mechanism is similar to the one in [20]. This factor allows the direct comparison of the two frameworks and establishes a causality of the performance variations to the data generation mechanism (i.e., augmentation vs. normal oversampling) and hyperparameter tuning steps. However, in this case, the hyperparameter tuning is solely going to be used for augmentation policy optimization.Algorithm 2: G-SMOTE modified for data augmentation in AL.
Given:t≥1, Dlabt≠∅, Dlab=Dlabmin∪Dlabmaj, GSMOTEInput:Dpoolt, Dlabt, fct−1, facq, τOutput:DtraintFunctionDataSelection (Dlabt, facq, fct−1):U←∅P←∅ps∼U0,1forallxi∈Dlabtdouxi←facqxi;fct−1U←U∪uxiforalluxi∈Udopxi←uxi/∑U+∑PP←P∪pxii←argmaxP<psreturn i-th element in DlabtbeginDaugmin←∅Daugmaj←∅αaf,αtrunc,αdef←τN←countCmaj×αafforallDaug′∈Daugmin,Daugmaj,Dlab′∈Dlabmin,DlabmajdowhileDaug′<Ndoxcenter←DataSelectionDlab′,facq,fct−1xgen←GSMOTExcenter,Dlabt,αtrunc,αdefDaug′←Daug′∪xgenDaug←Daugmin∪DaugmajDtraint←Dlabt∪DaugIn the proposed framework, we (1) generalize the generator module to accept any data augmentation method or policy and (2) introduce a hyperparameter tuning module to estimate the optimal data augmentation policy. This framework was designed to be task-agnostic. Specifically, any data augmentation method (domain-specific or not) may be applied, as well as any other parameter search method. It is also expected to be compatible with other AL modifications, including those that do not affect solely the classifier or uncertainty criterion, such as the one proposed in [21].


## 4. Methodology

This section describes the different elements included in the experimental procedure. The datasets used were acquired in open data repositories. Their sources and preprocessing steps are defined in Subsection4.1. The classifiers used in the experiment are defined in Subsection 4.2. The metrics chosen to measure AL performance and overall classification performance are defined in Subsection 4.3. The experimental procedure is described in Subsection 4.4.The methodology developed serves a two-fold purpose: (1) comparing classification performance once all the AL procedures are completed (i.e., optimal performance of a classifier trained via iterative data selection) and (2) comparing the amount of data required to reach specific performance thresholds (i.e., the number of AL iterations required to reach similar classification performances).


### 4.1. Datasets

The datasets used to test the proposed method are publicly available in open data repositories. Specifically, they were retrieved from the OpenML and the UCI machine learning repository websites. They were chosen considering diverse application domains, imbalance ratios, dimensionality, and number of target classes, all of them focused on classification tasks. The goal is to demonstrate the performance of the different AL frameworks in various scenarios and domains. The data preprocessing approach was similar across all datasets. Table2 describes the key properties of the 15 preprocessed datasets where the experimental procedure was applied.Table 2
Description of the datasets collected after data preprocessing. The sampling strategy is similar across datasets. IR, imbalance ratio.
DatasetFeatInstMin. instMaj. instIRClassImage segmentation1411551651651.07Mfeat zernike4719941982001.0110Texture4018241651661.0111Waveform4016665515641.023Pendigits1618321761911.0910Vehicle188461992181.14Mice protein6910731051501.438Gas drift12819872344301.846Japanese vowels1219921563232.079Usps25618591423102.1810Gesture segmentation3219742005902.955Volkert1471943454279.4910Steel plates2419415567312.247Baseball15132057119620.983Wine quality1115991068168.16The data preprocessing pipeline is depicted as a flowchart in Figure4. The missing values are removed from each dataset by removing the corresponding observations. This step ensures that the input data in the experiment is kept as close to its original form as possible. The nonmetric features (i.e., binary, categorical, and ordinal variables) were removed since the application of G-SMOTE is limited to continuous and discrete features. The datasets containing over 2000 observations were downsampled in order to maintain the datasets to a manageable size. The data sampling procedure preserves the relative class frequency of the dataset, in order to maintain the imbalance ratio (IR) originally found in each dataset (where IR=countCmaj/countCmin). The remaining features of each dataset are scaled to the range of −1,1 to ensure a common range across features.Figure 4
Data preprocessing pipeline.The preprocessed datasets were stored into an SQLite database file and are available along with the experiment’s source code in the project’s GitHub repository (see final remarks regarding data and software availability).


### 4.2. Machine Learning Algorithms

We used a total of four classification algorithms and a heuristic data augmentation mechanism. The choice of classifiers was based on the popularity and family of the classifiers (tree-based, nearest neighbors-based, ensemble-based, and linear models). Our proposed method was tested using a decision tree (DT) [48], a K-nearest neighbors classifier (KNN) [49], a random forest classifier (RF) [50], and a logistic regression (LR) [51]. Since the target variables are multiclass, the LR classifier was implemented using the one-versus-all approach. The predicted class is assigned to the label with the highest likelihood.The oversampler G-SMOTE was used as a data augmentation method. The typical data generation policy of oversampling methods is to generate artificial observations on nonmajority classes such that the number of majority class observations matches those of each nonmajority class. We modified this data generation policy to generate observations for all classes, as a percentage of the number of observations in the majority class. In addition, the original G-SMOTE algorithm was modified to accept data selection probabilities based on classification uncertainty. These modifications are discussed in Section3.Every AL procedure was tested with different selection criteria: random selection, entropy, and breaking ties. The baseline used is the standard AL procedure. As a benchmark, we add the AL procedure using G-SMOTE as a standard oversampling method, as proposed in [20]. Our proposed method was implemented using G-SMOTE as a data augmentation method to generate artificial observations for all classes, while still balancing the class distribution, as described in Section 3.


### 4.3. Evaluation Metrics

Considering the imbalanced nature of the datasets used in the experiment, commonly used performance metrics such as overall accuracy (OA), although being intuitive to interpret, are insufficient to quantify a model’s classification performance [52]. The Cohen’s Kappa performance metric, similar to OA, is also biased towards high-frequency classes since its definition is closely related to the OA metric, making its behavior consistent with OA [53]. However, these metrics remain popular choices for the evaluation of classification performance. Other performance metrics such as precision=TP/TP+FP, recall=TP/TP+FN (also known as sensitivity), or specificity=TN/TN+FP are calculated as a function of true/false positives (TP and FP) and true/false negatives (TN and FN) and can be used on a per-class basis instead. In a multiple dataset scenario with varying amounts of target classes and meanings, comparing the performance of different models using these metrics becomes impractical.Based on the recommendations found in [52, 54], we used two metrics found to be less sensitive to the class imbalance bias, along with OA as a reference for easier interpretability:(i)
The geometric-mean scorer (G-mean) consists of the geometric mean of specificity and recall [54]. Both metrics are calculated in a multi-class context considering a one-versus-all approach. For multiclass problems, the G-mean scorer is calculated as its average per class values as follows:(1)G−mean=recall¯×specificity¯.(ii)
The F-score metric consists of the harmonic mean of precision and recall. The two metrics are also calculated considering a one-versus-all approach. The F-score for the multiclass case can be calculated using its average per class values as follows [52]:(2)F−score=2×precision¯×recall¯precision¯+recall¯.(iii)
The OA consists of the number of TP divided by the total amount of observations. Consideringc as the label for the different classes present in a target class, OA is given by the following formula:(3)OA=∑cTPc∑cTPc+FPc.The comparison of the performance of AL frameworks is based on its data selection and augmentation efficacy. Specifically, an efficient data selection/generation policy allows the production of classifiers with high performance on unseen data while using as least nonartificial training data as possible. We follow the recommendations found in [55]. To measure the performance of the different AL setups, the performance of an AL setup will be compared using two AL-specific performance metrics.(iv)
Area under the Learning Curve (AULC). It is the sum of the classification performance over a validation/test set of the classifiers trained of all AL iterations. The resulting AULC scores are fixed within the range 0,1 by dividing the AULC scores by the total amount of iterations (i.e., the maximum performance area) to facilitate the interpretability of this metric.(v)
Data Utilization Rate (DUR) [56]. It measures the percentage of training data required to reach a given performance threshold, as a ratio of the percentage of training data required by the baseline framework. This metric is also presented as a percentage of the total amount of training data, without making it relative to the baseline framework. The DUR metric is measured at 45 different performance thresholds, ranging between 0.10,1.00 at a 0.02 step.


### 4.4. Experimental Procedure

The evaluation of different active learners in a live setting is generally expensive, time-consuming, and prone to human error. Instead, a common practice is to compare them in an offline environment using labeled datasets [57]. Since the dataset is already labeled, the annotation process is performed at zero cost in this scenario. Figure 5 depicts the experiment designed for one dataset over a single run.Figure 5
Experimental procedure flowchart. The preprocessed datasets are split into five folds. One of the folds is used to test the best-found classifiers using AL and the classifiers trained using the entire training dataset (containing the remaining folds). The training set is used to run both the AL simulations as well as train the normal classifiers. The validation set is used to measure AL-specific performance metrics over each iteration. We use different subsets for overall classification performance and AL-specific performance to avoid data leakage.A single run starts with the splitting of a preprocessed dataset into five different partitions, stratified according to the class frequencies of the target variable using the K-fold cross-validation method. During this run, an active learner or classifier is trained five times using a different partition as the test set each time. For each training process, a validation set containing 25% of the subset is created and is used to measure the data selection efficiency (i.e., AULC and DUR using the classification performance metrics, specific to AL). Therefore, for a single training procedure, 20% of the original dataset is used as the validation set, 20% is used as the test set, and 60% is used as the training set. The AL simulations and the classifiers’ training occur within the training set. However, the classifiers used to find the maximum performance classification scores are trained over the full training set. The AL simulations are run over a maximum of 50 iterations (including the initialization step), adding 1.6% of the training set each time (i.e., all AL simulations use less than 80% of the training set). Once the training phase is completed, the test set classification scores are calculated using the trained classifiers. For the case of AL, the classifier with the optimal validation set score is used to estimate the AL’s optimal classification performance over unseen data.The process shown in Figure5 is repeated over three runs using different random seeds over the 15 different datasets collected. The final scores of each AL configuration and classifier correspond to the average of the three runs and 5-fold cross-validation estimations (i.e., the mean score of 15 fits, across 15 datasets).The hyperparameters defined for the AL frameworks, classifiers, and generators are shown in Table3. In the generators table, we distinguish the G-SMOTE algorithm working as a normal oversampling method from G-SMOTE-AUGM, which generates additional artificial data on top of the usual oversampling mechanism. Since the G-SMOTE-AUGM method is intended to be used with varying parameter values (via within-iteration parameter tuning), the parameters were defined as a list of various possible values. The remaining parameters were selected based on knowledge gathered in previous literature and typical default values for each of the algorithms. This choice was motivated by the impossibility of parameter tuning in a real-world setting when applying the benchmark AL methods. Although the proposed method addresses this limitation, we show that exclusively tuning the parameters on the augmentation policy is already sufficient to achieve superior, statistically significant performance.Table 3
Hyperparameter definition for the active learners, classifiers, and generators used in the experiment.
Active learnersHyperparametersInputsStandard# Initial obs1.6%# Additional obs. per iteration1.6%Max. Iterations + initialization50Evaluation metricsG-mean, F-score, OASelection strategyRandom, entropy, breaking tiesWithin-iteration param. tuningNoneGeneratorNoneClassifierDT, LR, KNN, RFOversamplingGeneratorG-SMOTEProposedGeneratorG-SMOTE-AUGMWithin-iteration param. tuningGrid search K-fold CVClassifierDTMin. samples split2CriterionGiniLRMaximum iterations100MulticlassOne-vs-allSolverLiblinearPenaltyL2 (ridge)KNN# Neighbors5WeightsUniformMetricEuclideanRFMin. samples split2# Estimators100CriterionGiniGeneratorG-SMOTE# Neighbors4Deformation factor0.5Truncation factor0.5G-SMOTE-AUGM# Neighbors3, 4, 5Deformation factor0.5Truncation factor0.5Augmentation factor1.1,2.0 at 0.1 step


## 4.1. Datasets

The datasets used to test the proposed method are publicly available in open data repositories. Specifically, they were retrieved from the OpenML and the UCI machine learning repository websites. They were chosen considering diverse application domains, imbalance ratios, dimensionality, and number of target classes, all of them focused on classification tasks. The goal is to demonstrate the performance of the different AL frameworks in various scenarios and domains. The data preprocessing approach was similar across all datasets. Table2 describes the key properties of the 15 preprocessed datasets where the experimental procedure was applied.Table 2
Description of the datasets collected after data preprocessing. The sampling strategy is similar across datasets. IR, imbalance ratio.
DatasetFeatInstMin. instMaj. instIRClassImage segmentation1411551651651.07Mfeat zernike4719941982001.0110Texture4018241651661.0111Waveform4016665515641.023Pendigits1618321761911.0910Vehicle188461992181.14Mice protein6910731051501.438Gas drift12819872344301.846Japanese vowels1219921563232.079Usps25618591423102.1810Gesture segmentation3219742005902.955Volkert1471943454279.4910Steel plates2419415567312.247Baseball15132057119620.983Wine quality1115991068168.16The data preprocessing pipeline is depicted as a flowchart in Figure4. The missing values are removed from each dataset by removing the corresponding observations. This step ensures that the input data in the experiment is kept as close to its original form as possible. The nonmetric features (i.e., binary, categorical, and ordinal variables) were removed since the application of G-SMOTE is limited to continuous and discrete features. The datasets containing over 2000 observations were downsampled in order to maintain the datasets to a manageable size. The data sampling procedure preserves the relative class frequency of the dataset, in order to maintain the imbalance ratio (IR) originally found in each dataset (where IR=countCmaj/countCmin). The remaining features of each dataset are scaled to the range of −1,1 to ensure a common range across features.Figure 4
Data preprocessing pipeline.The preprocessed datasets were stored into an SQLite database file and are available along with the experiment’s source code in the project’s GitHub repository (see final remarks regarding data and software availability).


## 4.2. Machine Learning Algorithms

We used a total of four classification algorithms and a heuristic data augmentation mechanism. The choice of classifiers was based on the popularity and family of the classifiers (tree-based, nearest neighbors-based, ensemble-based, and linear models). Our proposed method was tested using a decision tree (DT) [48], a K-nearest neighbors classifier (KNN) [49], a random forest classifier (RF) [50], and a logistic regression (LR) [51]. Since the target variables are multiclass, the LR classifier was implemented using the one-versus-all approach. The predicted class is assigned to the label with the highest likelihood.The oversampler G-SMOTE was used as a data augmentation method. The typical data generation policy of oversampling methods is to generate artificial observations on nonmajority classes such that the number of majority class observations matches those of each nonmajority class. We modified this data generation policy to generate observations for all classes, as a percentage of the number of observations in the majority class. In addition, the original G-SMOTE algorithm was modified to accept data selection probabilities based on classification uncertainty. These modifications are discussed in Section3.Every AL procedure was tested with different selection criteria: random selection, entropy, and breaking ties. The baseline used is the standard AL procedure. As a benchmark, we add the AL procedure using G-SMOTE as a standard oversampling method, as proposed in [20]. Our proposed method was implemented using G-SMOTE as a data augmentation method to generate artificial observations for all classes, while still balancing the class distribution, as described in Section 3.


## 4.3. Evaluation Metrics

Considering the imbalanced nature of the datasets used in the experiment, commonly used performance metrics such as overall accuracy (OA), although being intuitive to interpret, are insufficient to quantify a model’s classification performance [52]. The Cohen’s Kappa performance metric, similar to OA, is also biased towards high-frequency classes since its definition is closely related to the OA metric, making its behavior consistent with OA [53]. However, these metrics remain popular choices for the evaluation of classification performance. Other performance metrics such as precision=TP/TP+FP, recall=TP/TP+FN (also known as sensitivity), or specificity=TN/TN+FP are calculated as a function of true/false positives (TP and FP) and true/false negatives (TN and FN) and can be used on a per-class basis instead. In a multiple dataset scenario with varying amounts of target classes and meanings, comparing the performance of different models using these metrics becomes impractical.Based on the recommendations found in [52, 54], we used two metrics found to be less sensitive to the class imbalance bias, along with OA as a reference for easier interpretability:(i)
The geometric-mean scorer (G-mean) consists of the geometric mean of specificity and recall [54]. Both metrics are calculated in a multi-class context considering a one-versus-all approach. For multiclass problems, the G-mean scorer is calculated as its average per class values as follows:(1)G−mean=recall¯×specificity¯.(ii)
The F-score metric consists of the harmonic mean of precision and recall. The two metrics are also calculated considering a one-versus-all approach. The F-score for the multiclass case can be calculated using its average per class values as follows [52]:(2)F−score=2×precision¯×recall¯precision¯+recall¯.(iii)
The OA consists of the number of TP divided by the total amount of observations. Consideringc as the label for the different classes present in a target class, OA is given by the following formula:(3)OA=∑cTPc∑cTPc+FPc.The comparison of the performance of AL frameworks is based on its data selection and augmentation efficacy. Specifically, an efficient data selection/generation policy allows the production of classifiers with high performance on unseen data while using as least nonartificial training data as possible. We follow the recommendations found in [55]. To measure the performance of the different AL setups, the performance of an AL setup will be compared using two AL-specific performance metrics.(iv)
Area under the Learning Curve (AULC). It is the sum of the classification performance over a validation/test set of the classifiers trained of all AL iterations. The resulting AULC scores are fixed within the range 0,1 by dividing the AULC scores by the total amount of iterations (i.e., the maximum performance area) to facilitate the interpretability of this metric.(v)
Data Utilization Rate (DUR) [56]. It measures the percentage of training data required to reach a given performance threshold, as a ratio of the percentage of training data required by the baseline framework. This metric is also presented as a percentage of the total amount of training data, without making it relative to the baseline framework. The DUR metric is measured at 45 different performance thresholds, ranging between 0.10,1.00 at a 0.02 step.


## 4.4. Experimental Procedure

The evaluation of different active learners in a live setting is generally expensive, time-consuming, and prone to human error. Instead, a common practice is to compare them in an offline environment using labeled datasets [57]. Since the dataset is already labeled, the annotation process is performed at zero cost in this scenario. Figure 5 depicts the experiment designed for one dataset over a single run.Figure 5
Experimental procedure flowchart. The preprocessed datasets are split into five folds. One of the folds is used to test the best-found classifiers using AL and the classifiers trained using the entire training dataset (containing the remaining folds). The training set is used to run both the AL simulations as well as train the normal classifiers. The validation set is used to measure AL-specific performance metrics over each iteration. We use different subsets for overall classification performance and AL-specific performance to avoid data leakage.A single run starts with the splitting of a preprocessed dataset into five different partitions, stratified according to the class frequencies of the target variable using the K-fold cross-validation method. During this run, an active learner or classifier is trained five times using a different partition as the test set each time. For each training process, a validation set containing 25% of the subset is created and is used to measure the data selection efficiency (i.e., AULC and DUR using the classification performance metrics, specific to AL). Therefore, for a single training procedure, 20% of the original dataset is used as the validation set, 20% is used as the test set, and 60% is used as the training set. The AL simulations and the classifiers’ training occur within the training set. However, the classifiers used to find the maximum performance classification scores are trained over the full training set. The AL simulations are run over a maximum of 50 iterations (including the initialization step), adding 1.6% of the training set each time (i.e., all AL simulations use less than 80% of the training set). Once the training phase is completed, the test set classification scores are calculated using the trained classifiers. For the case of AL, the classifier with the optimal validation set score is used to estimate the AL’s optimal classification performance over unseen data.The process shown in Figure5 is repeated over three runs using different random seeds over the 15 different datasets collected. The final scores of each AL configuration and classifier correspond to the average of the three runs and 5-fold cross-validation estimations (i.e., the mean score of 15 fits, across 15 datasets).The hyperparameters defined for the AL frameworks, classifiers, and generators are shown in Table3. In the generators table, we distinguish the G-SMOTE algorithm working as a normal oversampling method from G-SMOTE-AUGM, which generates additional artificial data on top of the usual oversampling mechanism. Since the G-SMOTE-AUGM method is intended to be used with varying parameter values (via within-iteration parameter tuning), the parameters were defined as a list of various possible values. The remaining parameters were selected based on knowledge gathered in previous literature and typical default values for each of the algorithms. This choice was motivated by the impossibility of parameter tuning in a real-world setting when applying the benchmark AL methods. Although the proposed method addresses this limitation, we show that exclusively tuning the parameters on the augmentation policy is already sufficient to achieve superior, statistically significant performance.Table 3
Hyperparameter definition for the active learners, classifiers, and generators used in the experiment.
Active learnersHyperparametersInputsStandard# Initial obs1.6%# Additional obs. per iteration1.6%Max. Iterations + initialization50Evaluation metricsG-mean, F-score, OASelection strategyRandom, entropy, breaking tiesWithin-iteration param. tuningNoneGeneratorNoneClassifierDT, LR, KNN, RFOversamplingGeneratorG-SMOTEProposedGeneratorG-SMOTE-AUGMWithin-iteration param. tuningGrid search K-fold CVClassifierDTMin. samples split2CriterionGiniLRMaximum iterations100MulticlassOne-vs-allSolverLiblinearPenaltyL2 (ridge)KNN# Neighbors5WeightsUniformMetricEuclideanRFMin. samples split2# Estimators100CriterionGiniGeneratorG-SMOTE# Neighbors4Deformation factor0.5Truncation factor0.5G-SMOTE-AUGM# Neighbors3, 4, 5Deformation factor0.5Truncation factor0.5Augmentation factor1.1,2.0 at 0.1 step


## 5. Results and Discussion

In a multiple dataset experiment, the analysis of results should not rely upon the average performance scores across datasets uniquely. The domain of application and fluctuations of performance scores between datasets make the analysis of these averaged results less accurate. Instead, it is generally recommended to use the mean ranking scores to extend the analysis [58]. Since mean performance scores are still intuitive to interpret; we will present and discuss both results. The rank values are assigned based on the mean scores of three different 5-fold cross-validation runs (15 performance estimations per dataset) for each combination of dataset, AL configuration, classifier, and performance metric.


### 5.1. Results

The average rankings of the AL methods’ AULC estimations are shown in Table4. The proposed method almost always improves AL performance and ensures higher data selection efficiency.Table 4
Mean rankings of the AULC metric over the different datasets (15), folds (5), and runs (3) used in the experiment. The proposed method constantly improves the results of the original framework and, on average, almost always improves the results of the oversampling framework.
ClassifierMetricStandardOversamplingProposedDTAccuracy2.13± 0.962.40± 0.491.47 ± 0.62DTF-score2.47± 0.812.20± 0.401.33 ± 0.70DTG-mean2.73± 0.571.93± 0.441.33 ± 0.70KNNAccuracy2.07± 0.932.07± 0.681.87 ± 0.81KNNF-score2.47± 0.811.87± 0.501.67 ± 0.87KNNG-mean2.87± 0.341.47 ± 0.501.67± 0.70LRAccuracy2.13± 0.882.20± 0.651.67 ± 0.79LRF-score2.80± 0.401.87± 0.501.33 ± 0.70LRG-mean2.80± 0.401.80± 0.541.40 ± 0.71RFAccuracy2.27± 0.851.87 ± 0.501.87 ± 0.96RFF-score2.73± 0.571.80± 0.541.47 ± 0.72RFG-mean2.87± 0.341.53 ± 0.501.60± 0.71Table5 shows the average AULC scores, grouped by the classifier, evaluation metric, and AL framework. The performance of the proposed method is almost always superior when considering the F-score and G-mean. On some occasions, the average AULC score is significantly improved when compared with the oversampling AL method.Table 5
Average AULC of each AL configuration tested. Each AULC score is calculated using the performance scores of each iteration in the validation set. By the end of the iterative process, each AL configuration used a maximum of 80% instances of the 60% instances that compose the training sets (i.e., 48% of the entire preprocessed dataset).
ClassifierMetricStandardOversamplingProposedDTAccuracy0.663±0.1490.658±0.1530.664±0.155DTF-score0.610±0.1760.612±0.1790.618±0.181DTG-mean0.744±0.1290.751±0.1270.755±0.129KNNAccuracy0.741±0.1600.730±0.1780.734±0.179KNNF-score0.678±0.2080.684±0.2110.687±0.213KNNG-mean0.786±0.1520.804±0.1390.804±0.141LRAccuracy0.736±0.1520.723±0.1850.731±0.184LRF-score0.644±0.2280.673±0.2200.682±0.221LRG-mean0.767±0.1620.811±0.1340.814±0.136RFAccuracy0.789±0.1480.786±0.1530.785±0.156RFF-score0.724±0.2140.735±0.2040.735±0.205RFG-mean0.818±0.1500.834±0.1350.833±0.135The average DUR scores were calculated for various G-mean thresholds, varying between 0.1 and 1.0 at a 0.02 step (45 different thresholds in total). Table6 shows the results obtained for these scores starting from a G-mean score of 0.6 and was filtered to show the thresholds ending with 0 or 6 only. In most cases, the proposed method reduces the amount of data annotation required to reach each G-mean score threshold.Table 6
AL algorithms’ mean data utilization as a percentage of the training set.
G-meanClassifierStandard (%)Oversampling (%)Proposed (%)0.60DT19.818.919.30.60KNN18.411.812.80.60LR23.09.79.70.60RF14.17.77.80.66DT23.123.322.90.66KNN23.921.721.90.66LR25.620.520.50.66RF22.017.617.50.70DT25.525.024.80.70KNN26.824.123.90.70LR29.923.623.40.70RF23.822.122.30.76DT33.430.530.10.76KNN34.027.727.30.76LR38.027.626.20.76RF28.224.524.70.80DT48.243.841.20.80KNN38.834.434.60.80LR43.732.631.30.80RF32.427.227.70.86DT69.666.564.80.86KNN53.952.052.50.86LR48.745.345.00.86RF43.940.040.00.90DT81.279.476.60.90KNN60.961.160.40.90LR62.162.959.90.90RF57.155.756.20.96DT100.099.7100.00.96KNN82.479.777.10.96LR86.584.081.80.96RF70.871.170.3The DUR scores relative to the standard AL method are shown in Figure6. A DUR score below 1 means that the proposed/oversampling method requires less data than the standard AL method to reach the same performance threshold. For example, running an AL simulation using the KNN classifier requires 80.7% of the amount of data required by the standard AL method using the same classifier to reach an F-Score of 0.62 (i.e., requires 19.3% less data).Figure 6
Mean data utilization rates. They-axis shows the percentage of data (relative to the baseline AL framework) required to reach the different performance thresholds.The comparison of mean optimal classification scores of AL methods with classifiers (using the entire training set, without AL) is shown in Table7. Aside from the case of overall accuracy, the proposed AL method produces classifiers that almost consistently outperform classifiers using the whole training set (i.e., the ones labeled as MP).Table 7
Optimal classification scores. The maximum performance (MP) classification scores are calculated using classifiers trained using the entire training set.
ClassifierMetricMPStandardOversamplingProposedDTAccuracy0.732±0.1550.726±0.1570.721±0.1670.727±0.168DTF-score0.682±0.1940.679±0.1930.679±0.1970.684±0.200DTG-mean0.792±0.1380.791±0.1360.797±0.1340.800±0.137KNNAccuracy0.801±0.1640.799±0.1680.784±0.1830.789±0.183KNNF-score0.742±0.2240.744±0.2230.741±0.2230.746±0.224KNNG-mean0.827±0.1600.829±0.1580.839±0.1460.840±0.147LRAccuracy0.778±0.1570.791±0.1580.764±0.1840.773±0.185LRF-score0.693±0.2430.717±0.2410.718±0.2220.727±0.226LRG-mean0.796±0.1710.814±0.1650.839±0.1300.842±0.137RFAccuracy0.827±0.1450.832±0.1480.827±0.1540.829±0.153RFF-score0.767±0.2150.775±0.2160.781±0.2040.784±0.204RFG-mean0.844±0.1480.849±0.1490.863±0.1310.865±0.131


### 5.2. Statistical Analysis

When checking for statistical significance in a multiple dataset context, it is critical to account for the multiple comparison problem. Consequently, our statistical analysis focuses on the recommendations found in [58]. Overall, we performed three statistical tests. The Friedman test [59] is used to understand whether there is a statistically significant difference in performance between the three AL frameworks. As post hoc analysis, the Wilcoxon signed-rank test [60] was utilized to check for statistical significance between the performance of the proposed AL method and the oversampling AL method across datasets. As a second post hoc analysis, the Holm [61] method was employed to check for statistical significance between the methods using data generators and the standard AL framework across classifiers and evaluation metrics.Table8 displays the p values obtained with the Friedman test. The difference in performance across AL frameworks is statistically significant at a level of α=0.05 regardless of the classifier or evaluation metric being considered.Table 8
Friedman test results. Statistical significance is tested at a level ofα=0.05. The null hypothesis is that there is no difference in the classification outcome across oversamplers.
ClassifierEvaluation metricp valueSignificanceDTAccuracy1.1e − 15TrueDTF-score2.4e − 31TrueDTG-mean2.3e − 23TrueKNNAccuracy5.9e − 20TrueKNNF-score8.8e − 69TrueKNNG-mean8.8e − 52TrueLRAccuracy1.1e − 30TrueLRF-score4.0e − 98TrueLRG-mean2.3e − 83TrueRFAccuracy2.8e − 26TrueRFF-score1.8e − 88TrueRFG-mean1.8e − 61TrueTable9 contains the p values obtained with the Wilcoxon signed-rank test. The proposed method was able to outperform both the standard AL framework, as well as the AL framework using a typical oversampling policy with statistical significance in 14 and 12 out of 15 datasets, respectively.Table 9
Adjustedp values using the Wilcoxon signed-rank method. Bold values are statistically significant at a level of α=0.05. The null hypothesis is that the performance of the proposed framework is similar to that of the oversampling or standard framework.
DatasetOversamplingStandardBaseball5.0e − 013.4e − 01Gas drift3.7e − 264.6e − 57Gesture segmentation1.3e − 028.7e − 04Image segmentation9.6e − 182.1e − 44Japanese vowels2.4e − 091.6e − 32Mfeat zernike1.2e − 129.5e − 40Mice protein6.5e − 321.5e − 61Pendigits5.0e − 182.3e − 45Steel plates3.4e − 041.3e − 08Texture1.5e − 226.7e − 57Usps3.8e − 012.1e − 29Vehicle7.4e − 117.9e − 13Volkert2.5e − 011.3e − 02Waveform8.9e − 082.6e − 02Wine quality3.8e − 056.1e − 03Thep values shown in Table 10 refer to the results of the Holm-Bonferroni test. The proposed method’s superior performance was statistically significant in 9 out of 12 cases.Table 10
Adjustedp values using the Holm-Bonferroni method. Bold values are statistically significant at a level of α=0.05. The null hypothesis is that the oversampling or proposed method does not perform better than the control method (standard AL framework).
ClassifierEvaluation metricOversamplingProposedDTAccuracy7.7e − 011.1e − 04DTF-score6.3e − 022.0e − 06DTG-mean1.0e − 082.9e − 12KNNAccuracy1.0e − 028.5e − 01KNNF-score7.1e − 078.3e − 13KNNG-mean1.9e − 111.0e − 12LRAccuracy3.2e − 028.3e − 01LRF-score1.5e − 095.8e − 17LRG-mean1.9e − 135.6e − 16RFAccuracy4.3e − 014.3e − 01RFF-score1.4e − 111.1e − 12RFG-mean1.5e − 101.2e − 10


### 5.3. Discussion

In this paper, we study the application of data augmentation methods through the modification of the standard AL framework. This is performed to further reduce the amount of labeled data required to produce a reliable classifier, at the expense of artificial data generation. Overall, the proposed method achieves better and more consistent performance when compared to the remaining benchmark approaches. It was implemented to focus on the optimization of the data augmentation policy, as well as the introduction of a more informed AL-based data augmentation approach. The proposed method could be further extended and achieve an even higher performance by optimizing parameters of the ML classification using the hyperparameter optimizer. In addition, this framework could be further generalized by searching, within AL iterations, for the optimal ML classifier as well; at different stages of the data collection procedure, some ML classifiers might be more useful than others. Although the proposed framework significantly improves the flexibility of AL implementations, we found that even a superficial parameter search is sufficient to ensure a superior performance when compared to related approaches.In Table4, we found that the proposed method was able to outperform the standard AL framework in all scenarios. Except for the overall accuracy metric, the mean rankings are consistent with the mean AULC scores found in Table 5, while showing performance improvements between the proposed method and both the standard and oversampling methods. The Friedman test in Table 8 showed that the difference in the performance of these AL frameworks are statistically significant, regardless of the classifier or performance metric being used.The proposed method evidenced more consistent data utilization requirements in most of the assessed G-mean score thresholds when compared to the remaining AL methods, as seen in Table6. For example, to reach a G-mean score of 0.9 using the KNN and LR classifiers, the average amount of data required with the oversampling AL approach increased when compared to the standard approach. However, the proposed method was able to decrease the amount of data required in both situations. The robustness of the proposed method is clearer in Figure 6. In most cases, this method was able to outperform the oversampling method. At the same time, the proposed method also addresses inconsistencies in situations where the oversampling method was unable to outperform the standard method.The statistical analyses found in Tables9 and 10 revealed that the proposed method’s superiority was statistically significant in all datasets except three (Baseball, Usps, and Volkert) and established statistical significance when compared to the standard AL method for all combinations of classifier and performance metric, except for three cases regarding the use of the overall accuracy metric. These results show that the proposed method increased the reliability of the new AL framework and improved the quality of the final classifier while using fewer data.Even though it was not the core purpose of this study, we found that the proposed AL method consistently outperformed the maximum performance threshold. Specifically, in Table7, the performance of the classifiers originating from the proposed method was able to outperform classifiers trained using the full training dataset in 9 out of 12 scenarios. This outcome suggests that the selection of a meaningful training subset training dataset paired with data augmentation not only matches the classification performance of ML algorithms but also improves them. Even in a setting with fully labeled training data, the proposed method may be used as a preprocessing technique to further optimize classification performance.This study discussed the effect of data augmentation within the AL framework, along with the exploration of optimal augmentation methods within AL iterations. However, the conceptual nature of this study implies some limitations. Specifically, the large number of experiments required to test the method’s efficacy, along with the limited computational power available, led to a limited exploration of the grid search’s potential. Future work should focus on understanding how the usage of a more comprehensive parameter tuning approach improves the quality of the AL method. In addition, the proposed method was not able to outperform the standard AL method at 100% of the scenarios. The exploration of other, more complex data augmentation techniques might further improve its performance by producing more meaningful training observations. Specifically, in this study, we assume that all datasets used follow a manifold, allowing the usage of G-SMOTE as a data augmentation approach. However, this method cannot be used in more complex, non-Euclidean spaces. In this scenario, the usage of G-SMOTE is not valid and might lead to the production of noisy data. Deep learning-based data augmentation techniques are able to address this limitation and improve the overall quality of the artificial data being generated. We also encountered significant standard errors throughout our experimental results (see Subsection5.1), consistent with the findings in [20, 55]. This facet suggests that the usage of more robust generators did not decrease the standard error of AL performance. Instead, AL’s performance variability is likely dependent on the quality of its initialization.


## 5.1. Results

The average rankings of the AL methods’ AULC estimations are shown in Table4. The proposed method almost always improves AL performance and ensures higher data selection efficiency.Table 4
Mean rankings of the AULC metric over the different datasets (15), folds (5), and runs (3) used in the experiment. The proposed method constantly improves the results of the original framework and, on average, almost always improves the results of the oversampling framework.
ClassifierMetricStandardOversamplingProposedDTAccuracy2.13± 0.962.40± 0.491.47 ± 0.62DTF-score2.47± 0.812.20± 0.401.33 ± 0.70DTG-mean2.73± 0.571.93± 0.441.33 ± 0.70KNNAccuracy2.07± 0.932.07± 0.681.87 ± 0.81KNNF-score2.47± 0.811.87± 0.501.67 ± 0.87KNNG-mean2.87± 0.341.47 ± 0.501.67± 0.70LRAccuracy2.13± 0.882.20± 0.651.67 ± 0.79LRF-score2.80± 0.401.87± 0.501.33 ± 0.70LRG-mean2.80± 0.401.80± 0.541.40 ± 0.71RFAccuracy2.27± 0.851.87 ± 0.501.87 ± 0.96RFF-score2.73± 0.571.80± 0.541.47 ± 0.72RFG-mean2.87± 0.341.53 ± 0.501.60± 0.71Table5 shows the average AULC scores, grouped by the classifier, evaluation metric, and AL framework. The performance of the proposed method is almost always superior when considering the F-score and G-mean. On some occasions, the average AULC score is significantly improved when compared with the oversampling AL method.Table 5
Average AULC of each AL configuration tested. Each AULC score is calculated using the performance scores of each iteration in the validation set. By the end of the iterative process, each AL configuration used a maximum of 80% instances of the 60% instances that compose the training sets (i.e., 48% of the entire preprocessed dataset).
ClassifierMetricStandardOversamplingProposedDTAccuracy0.663±0.1490.658±0.1530.664±0.155DTF-score0.610±0.1760.612±0.1790.618±0.181DTG-mean0.744±0.1290.751±0.1270.755±0.129KNNAccuracy0.741±0.1600.730±0.1780.734±0.179KNNF-score0.678±0.2080.684±0.2110.687±0.213KNNG-mean0.786±0.1520.804±0.1390.804±0.141LRAccuracy0.736±0.1520.723±0.1850.731±0.184LRF-score0.644±0.2280.673±0.2200.682±0.221LRG-mean0.767±0.1620.811±0.1340.814±0.136RFAccuracy0.789±0.1480.786±0.1530.785±0.156RFF-score0.724±0.2140.735±0.2040.735±0.205RFG-mean0.818±0.1500.834±0.1350.833±0.135The average DUR scores were calculated for various G-mean thresholds, varying between 0.1 and 1.0 at a 0.02 step (45 different thresholds in total). Table6 shows the results obtained for these scores starting from a G-mean score of 0.6 and was filtered to show the thresholds ending with 0 or 6 only. In most cases, the proposed method reduces the amount of data annotation required to reach each G-mean score threshold.Table 6
AL algorithms’ mean data utilization as a percentage of the training set.
G-meanClassifierStandard (%)Oversampling (%)Proposed (%)0.60DT19.818.919.30.60KNN18.411.812.80.60LR23.09.79.70.60RF14.17.77.80.66DT23.123.322.90.66KNN23.921.721.90.66LR25.620.520.50.66RF22.017.617.50.70DT25.525.024.80.70KNN26.824.123.90.70LR29.923.623.40.70RF23.822.122.30.76DT33.430.530.10.76KNN34.027.727.30.76LR38.027.626.20.76RF28.224.524.70.80DT48.243.841.20.80KNN38.834.434.60.80LR43.732.631.30.80RF32.427.227.70.86DT69.666.564.80.86KNN53.952.052.50.86LR48.745.345.00.86RF43.940.040.00.90DT81.279.476.60.90KNN60.961.160.40.90LR62.162.959.90.90RF57.155.756.20.96DT100.099.7100.00.96KNN82.479.777.10.96LR86.584.081.80.96RF70.871.170.3The DUR scores relative to the standard AL method are shown in Figure6. A DUR score below 1 means that the proposed/oversampling method requires less data than the standard AL method to reach the same performance threshold. For example, running an AL simulation using the KNN classifier requires 80.7% of the amount of data required by the standard AL method using the same classifier to reach an F-Score of 0.62 (i.e., requires 19.3% less data).Figure 6
Mean data utilization rates. They-axis shows the percentage of data (relative to the baseline AL framework) required to reach the different performance thresholds.The comparison of mean optimal classification scores of AL methods with classifiers (using the entire training set, without AL) is shown in Table7. Aside from the case of overall accuracy, the proposed AL method produces classifiers that almost consistently outperform classifiers using the whole training set (i.e., the ones labeled as MP).Table 7
Optimal classification scores. The maximum performance (MP) classification scores are calculated using classifiers trained using the entire training set.
ClassifierMetricMPStandardOversamplingProposedDTAccuracy0.732±0.1550.726±0.1570.721±0.1670.727±0.168DTF-score0.682±0.1940.679±0.1930.679±0.1970.684±0.200DTG-mean0.792±0.1380.791±0.1360.797±0.1340.800±0.137KNNAccuracy0.801±0.1640.799±0.1680.784±0.1830.789±0.183KNNF-score0.742±0.2240.744±0.2230.741±0.2230.746±0.224KNNG-mean0.827±0.1600.829±0.1580.839±0.1460.840±0.147LRAccuracy0.778±0.1570.791±0.1580.764±0.1840.773±0.185LRF-score0.693±0.2430.717±0.2410.718±0.2220.727±0.226LRG-mean0.796±0.1710.814±0.1650.839±0.1300.842±0.137RFAccuracy0.827±0.1450.832±0.1480.827±0.1540.829±0.153RFF-score0.767±0.2150.775±0.2160.781±0.2040.784±0.204RFG-mean0.844±0.1480.849±0.1490.863±0.1310.865±0.131


## 5.2. Statistical Analysis

When checking for statistical significance in a multiple dataset context, it is critical to account for the multiple comparison problem. Consequently, our statistical analysis focuses on the recommendations found in [58]. Overall, we performed three statistical tests. The Friedman test [59] is used to understand whether there is a statistically significant difference in performance between the three AL frameworks. As post hoc analysis, the Wilcoxon signed-rank test [60] was utilized to check for statistical significance between the performance of the proposed AL method and the oversampling AL method across datasets. As a second post hoc analysis, the Holm [61] method was employed to check for statistical significance between the methods using data generators and the standard AL framework across classifiers and evaluation metrics.Table8 displays the p values obtained with the Friedman test. The difference in performance across AL frameworks is statistically significant at a level of α=0.05 regardless of the classifier or evaluation metric being considered.Table 8
Friedman test results. Statistical significance is tested at a level ofα=0.05. The null hypothesis is that there is no difference in the classification outcome across oversamplers.
ClassifierEvaluation metricp valueSignificanceDTAccuracy1.1e − 15TrueDTF-score2.4e − 31TrueDTG-mean2.3e − 23TrueKNNAccuracy5.9e − 20TrueKNNF-score8.8e − 69TrueKNNG-mean8.8e − 52TrueLRAccuracy1.1e − 30TrueLRF-score4.0e − 98TrueLRG-mean2.3e − 83TrueRFAccuracy2.8e − 26TrueRFF-score1.8e − 88TrueRFG-mean1.8e − 61TrueTable9 contains the p values obtained with the Wilcoxon signed-rank test. The proposed method was able to outperform both the standard AL framework, as well as the AL framework using a typical oversampling policy with statistical significance in 14 and 12 out of 15 datasets, respectively.Table 9
Adjustedp values using the Wilcoxon signed-rank method. Bold values are statistically significant at a level of α=0.05. The null hypothesis is that the performance of the proposed framework is similar to that of the oversampling or standard framework.
DatasetOversamplingStandardBaseball5.0e − 013.4e − 01Gas drift3.7e − 264.6e − 57Gesture segmentation1.3e − 028.7e − 04Image segmentation9.6e − 182.1e − 44Japanese vowels2.4e − 091.6e − 32Mfeat zernike1.2e − 129.5e − 40Mice protein6.5e − 321.5e − 61Pendigits5.0e − 182.3e − 45Steel plates3.4e − 041.3e − 08Texture1.5e − 226.7e − 57Usps3.8e − 012.1e − 29Vehicle7.4e − 117.9e − 13Volkert2.5e − 011.3e − 02Waveform8.9e − 082.6e − 02Wine quality3.8e − 056.1e − 03Thep values shown in Table 10 refer to the results of the Holm-Bonferroni test. The proposed method’s superior performance was statistically significant in 9 out of 12 cases.Table 10
Adjustedp values using the Holm-Bonferroni method. Bold values are statistically significant at a level of α=0.05. The null hypothesis is that the oversampling or proposed method does not perform better than the control method (standard AL framework).
ClassifierEvaluation metricOversamplingProposedDTAccuracy7.7e − 011.1e − 04DTF-score6.3e − 022.0e − 06DTG-mean1.0e − 082.9e − 12KNNAccuracy1.0e − 028.5e − 01KNNF-score7.1e − 078.3e − 13KNNG-mean1.9e − 111.0e − 12LRAccuracy3.2e − 028.3e − 01LRF-score1.5e − 095.8e − 17LRG-mean1.9e − 135.6e − 16RFAccuracy4.3e − 014.3e − 01RFF-score1.4e − 111.1e − 12RFG-mean1.5e − 101.2e − 10


## 5.3. Discussion

In this paper, we study the application of data augmentation methods through the modification of the standard AL framework. This is performed to further reduce the amount of labeled data required to produce a reliable classifier, at the expense of artificial data generation. Overall, the proposed method achieves better and more consistent performance when compared to the remaining benchmark approaches. It was implemented to focus on the optimization of the data augmentation policy, as well as the introduction of a more informed AL-based data augmentation approach. The proposed method could be further extended and achieve an even higher performance by optimizing parameters of the ML classification using the hyperparameter optimizer. In addition, this framework could be further generalized by searching, within AL iterations, for the optimal ML classifier as well; at different stages of the data collection procedure, some ML classifiers might be more useful than others. Although the proposed framework significantly improves the flexibility of AL implementations, we found that even a superficial parameter search is sufficient to ensure a superior performance when compared to related approaches.In Table4, we found that the proposed method was able to outperform the standard AL framework in all scenarios. Except for the overall accuracy metric, the mean rankings are consistent with the mean AULC scores found in Table 5, while showing performance improvements between the proposed method and both the standard and oversampling methods. The Friedman test in Table 8 showed that the difference in the performance of these AL frameworks are statistically significant, regardless of the classifier or performance metric being used.The proposed method evidenced more consistent data utilization requirements in most of the assessed G-mean score thresholds when compared to the remaining AL methods, as seen in Table6. For example, to reach a G-mean score of 0.9 using the KNN and LR classifiers, the average amount of data required with the oversampling AL approach increased when compared to the standard approach. However, the proposed method was able to decrease the amount of data required in both situations. The robustness of the proposed method is clearer in Figure 6. In most cases, this method was able to outperform the oversampling method. At the same time, the proposed method also addresses inconsistencies in situations where the oversampling method was unable to outperform the standard method.The statistical analyses found in Tables9 and 10 revealed that the proposed method’s superiority was statistically significant in all datasets except three (Baseball, Usps, and Volkert) and established statistical significance when compared to the standard AL method for all combinations of classifier and performance metric, except for three cases regarding the use of the overall accuracy metric. These results show that the proposed method increased the reliability of the new AL framework and improved the quality of the final classifier while using fewer data.Even though it was not the core purpose of this study, we found that the proposed AL method consistently outperformed the maximum performance threshold. Specifically, in Table7, the performance of the classifiers originating from the proposed method was able to outperform classifiers trained using the full training dataset in 9 out of 12 scenarios. This outcome suggests that the selection of a meaningful training subset training dataset paired with data augmentation not only matches the classification performance of ML algorithms but also improves them. Even in a setting with fully labeled training data, the proposed method may be used as a preprocessing technique to further optimize classification performance.This study discussed the effect of data augmentation within the AL framework, along with the exploration of optimal augmentation methods within AL iterations. However, the conceptual nature of this study implies some limitations. Specifically, the large number of experiments required to test the method’s efficacy, along with the limited computational power available, led to a limited exploration of the grid search’s potential. Future work should focus on understanding how the usage of a more comprehensive parameter tuning approach improves the quality of the AL method. In addition, the proposed method was not able to outperform the standard AL method at 100% of the scenarios. The exploration of other, more complex data augmentation techniques might further improve its performance by producing more meaningful training observations. Specifically, in this study, we assume that all datasets used follow a manifold, allowing the usage of G-SMOTE as a data augmentation approach. However, this method cannot be used in more complex, non-Euclidean spaces. In this scenario, the usage of G-SMOTE is not valid and might lead to the production of noisy data. Deep learning-based data augmentation techniques are able to address this limitation and improve the overall quality of the artificial data being generated. We also encountered significant standard errors throughout our experimental results (see Subsection5.1), consistent with the findings in [20, 55]. This facet suggests that the usage of more robust generators did not decrease the standard error of AL performance. Instead, AL’s performance variability is likely dependent on the quality of its initialization.


## 6. Conclusion and Future Directions

The ability to train ML classifiers is usually limited to the availability of labeled data. However, manually labeling data is often expensive, which makes the usage of AL particularly appealing for selecting the most informative observations and reducing the amount of required labeled data. On the other hand, the introduction of data variability in the training dataset can also be conducted via data augmentation. However, most, if not all, AL configurations that use some form of data augmentation are domain and/or task-specific. These methods typically apply deep learning approaches to both classification and data augmentation. Consequently, they may not apply to other classification tasks or when the available computational power is insufficient.In this paper, we proposed a domain-agnostic AL framework that implements data augmentation and hyperparameter tuning. We found that a heuristic data augmentation algorithm is sufficient to improve the data selection efficiency in AL. Specifically, the data augmentation method used almost always increased AL performance, regardless of the target goal (i.e., optimizing classification or data selection efficiency). The usage of data augmentation reduced the number of iterations required to train a classifier with a performance as good as (or better than) classifiers trained with the entire training dataset (i.e., without using AL). In addition, the proposed method reduced the size of the training dataset, which is expanded with artificial data.With this revised AL configuration, data selection in AL iterations aims towards observations that optimize the quality of the artificial data produced. The substitution of less informative labeled data with artificial data is especially useful in this context since it reduces some of the user interaction necessary to reach a sufficiently informative dataset. In order to further improve the proposed method, future work should (1) focus on the development of methods with varying data augmentation policies depending on the different input space regions, (2) develop augmentation-sensitive query functions capable of avoiding the unnecessary selection of similar observations from the unlabeled dataset, (3) understand the gap between randomized data augmentation techniques and neural network/feature space data augmentation techniques in an AL context better, (4) explore more efficient ways to leverage the information collected in AL queries for better augmentation strategies, and (5) expand the current framework to integrate alternative learning strategies using unlabeled data, such as self and semisupervised learning techniques.Finally, the proposed method may be applied to any classification problem where labeled data is not readily available and an easily accessible unlabeled data pool. For more complex data structures, the application of this framework will require the learning of a manifold space as an additional preprocessing step. After that, this AL framework may be used as it is.



---

*Source: 7941878-2023-02-20.xml*