# An Active Learning Method Based on Variational Autoencoder and DBSCAN Clustering

Authors: Fang Chen; Tao Zhang; Ruilin Liu
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

Active learning is aimed to sample the most informative data from the unlabeled pool, and diverse clustering methods have been applied to it. However, the distance-based clustering methods usually cannot perform well in high dimensions and even begin to fail. In this paper, we propose a new active learning method combined with variational autoencoder (VAE) and density-based spatial clustering of applications with noise (DBSCAN). It overcomes the difficulty of distance representation in high dimensions and prevents the distance concentration phenomenon from occurring in the computational learning literature with respect to high-dimensional p-norms. Finally, we compare our method with four common active learning methods and two other clustering algorithms combined with VAE on three datasets. The results demonstrate that our approach achieves competitive performance, and it is a new batch mode active learning algorithm designed for neural networks with a relatively small query batch size.


---

## Body

## 1. Introduction

In practical issues, the amount of labeled data is relatively small and the vast majority is unlabeled. It is often unrealistic to consume a lot of human resources and expensive cost to annotate unlabeled data for the annotation budget is limited. Active learning [1] is born out of this problem, which is one field trying to address the difficulties in data labeling. It assumes that the collection of data is relatively easy, but the labeling process is costly. It tackles the problem of which samples we should label to result in the highest improvement in test accuracy under a fixed labeling budget. Existing active learning algorithms are mainly divided into two categories, query-synthesizing [2–4] and query-acquiring [5]. Query-synthesizing approaches use generative models to generate informative samples whereas query-acquiring algorithms use different sampling strategies to determine how to select the most informative samples.In this paper, we mainly pay attention to query-acquiring methods. One of the query-acquiring algorithms is uncertainty-based method [6, 7]. Settles [8] mentioned that the classifier gives every unlabeled data a probability score which represents the uncertainty belonging to its class, and then we choose the data with the highest uncertainty. Lewis and Gale [9] argued that uncertainty-based methods perform well on a large and diverse set of datasets. Yarin et al. [10] also proposed a Bayesian active learning framework, in which Bayesian neural networks [11] are used to estimate uncertainty. Later, Gissin and Shai proposed a discriminative active learning method [12] in which an uncertainty idea is also used to sample the unlabeled data with the top-K highest possibility when the batch size is relatively large. The other type of query-acquiring methods is representation-based approach, which relies on selecting few examples by increasing diversity in a given batch. Ozan and Savarese [13] and Suyog and Grauman [14] adopted this idea in their experiments. However, it seems to be ineffective to use distance-based representation methods like Core Set for high-dimensional data because the distance concentration phenomenon appears in the computational learning literature with respect to p-norms in high dimensions [15].To address the difficulty of high-dimensional distance representation, Sinha proposed a variational adversarial method [5] by learning a latent space using a VAE [16, 17]. VAE is called the perfect representation learning method of high-dimensional data, which has been proved effective. Therefore, we adopt VAE in our approach to solve the difficulty of distance representation especially for digital images [18]. VAE plays a great role in our method, which perfectly represents high-dimensional data into the low-dimensional representation in latent space, preventing the occurrence of the phenomenon ofp-norm distance concentration of high-dimensional data. And in common clustering approaches [19–22], DBSCAN [23] is good at identifying the noise and discovering arbitrarily shaped clusters without knowing the number of cluster classes to be formed in advance. Therefore, we propose a new active learning strategy combined with VAE and DBSCAN clustering for considering their advantages. What is more, excellent active learning models are usually based on large query batch sizes. But they may ignore the performance of a relatively small query batch size based on small sample problems. So we conduct our experiment on a relatively small query batch size to see the performance.The rest of this paper is organized as follows. Section2 illustrates the problem setting and the brief definition of related methods. Then Section 3 introduces our proposed model and the algorithms. Subsequently, the experimental results based on a relatively small query batch size are provided in Section 4. In the end, we draw a conclusion in Section 5.


## 2. Related Notion

### 2.1. Problem Definition

We define the problem of active learning formally as follows. Given a labeled poolXL,YL and a much larger unlabeled pool XU, we are aimed to sample the most informative unlabeled data xU from XU by iteratively querying a fixed sampling budget in order to train the most label-efficient model in an active learning task. During this progress, n number of unlabeled data will be sampled by an acquisition function and be annotated by the oracle. This is an iterative process which is done until a certain stopping criterion is reached, such as the desired amount of samples or test accuracy.


### 2.2. Related Methods

With respect to VAE, the parameters are trained by two loss functions ifN training samples are given. One is the reconstruction loss L1, which forces the reconstructed sample xi∧ to match the original input sample xi. Here, we use the cross entropy for measurement:(1)L1=−∑i=1Nxi⋅logxi∧+1−xi⋅log1−xi∧,wherei=1,2,…,N.In addition, we are aimed to makexi∧ as close as possible to xi itself. That is to say, it can retrieve as much of the original information as possible through decoding. The other one is regularization loss L2, which helps to learn latent space with good structure and reduces overfitting on the training data. The formula is shown as follows:(2)L2=1N∑i=1N1+logσi2−μi2−explogσi2,wherei=1,2,…,N.In the formula above, the meanμi and the variance σi2 of every input data are computed by an encoder. Then the encoder learns a low-dimensional latent space for the underlying distribution Nμi,σi2 using a Gaussian prior. Therefore, we obtain the total objective function for VAE as follows:(3)LVAE=L1+L2.Then, we minimize the total lossLVAE and a high efficient variational autoencoder model will be obtained after rounds of optimization. It is worth mentioning that the reparameterization trick is adopted by sampling a data ε from a standard normal distribution N0,1. Then, we could sample the low-dimensional data zi randomly in the latent space:(4)zi=μi+σi⊙ε,in which ε is a very small random tensor and ⊙ is the element-wise product.In our approach, clusters in DBSCAN are defined as the largest set of points connected by density. A region with sufficient density can be divided into a cluster, and points that are not clustered are called noise. The basic process of DBSCAN is as shown in Figure1.Figure 1
DBSCAN clustering if MinPts = 5.


## 2.1. Problem Definition

We define the problem of active learning formally as follows. Given a labeled poolXL,YL and a much larger unlabeled pool XU, we are aimed to sample the most informative unlabeled data xU from XU by iteratively querying a fixed sampling budget in order to train the most label-efficient model in an active learning task. During this progress, n number of unlabeled data will be sampled by an acquisition function and be annotated by the oracle. This is an iterative process which is done until a certain stopping criterion is reached, such as the desired amount of samples or test accuracy.


## 2.2. Related Methods

With respect to VAE, the parameters are trained by two loss functions ifN training samples are given. One is the reconstruction loss L1, which forces the reconstructed sample xi∧ to match the original input sample xi. Here, we use the cross entropy for measurement:(1)L1=−∑i=1Nxi⋅logxi∧+1−xi⋅log1−xi∧,wherei=1,2,…,N.In addition, we are aimed to makexi∧ as close as possible to xi itself. That is to say, it can retrieve as much of the original information as possible through decoding. The other one is regularization loss L2, which helps to learn latent space with good structure and reduces overfitting on the training data. The formula is shown as follows:(2)L2=1N∑i=1N1+logσi2−μi2−explogσi2,wherei=1,2,…,N.In the formula above, the meanμi and the variance σi2 of every input data are computed by an encoder. Then the encoder learns a low-dimensional latent space for the underlying distribution Nμi,σi2 using a Gaussian prior. Therefore, we obtain the total objective function for VAE as follows:(3)LVAE=L1+L2.Then, we minimize the total lossLVAE and a high efficient variational autoencoder model will be obtained after rounds of optimization. It is worth mentioning that the reparameterization trick is adopted by sampling a data ε from a standard normal distribution N0,1. Then, we could sample the low-dimensional data zi randomly in the latent space:(4)zi=μi+σi⊙ε,in which ε is a very small random tensor and ⊙ is the element-wise product.In our approach, clusters in DBSCAN are defined as the largest set of points connected by density. A region with sufficient density can be divided into a cluster, and points that are not clustered are called noise. The basic process of DBSCAN is as shown in Figure1.Figure 1
DBSCAN clustering if MinPts = 5.


## 3. Proposed Model

### 3.1. The Combination of VAE and DBSCAN Clustering

In this paper, we suggested a new active learning method based on VAE and DBSCAN clustering on a relatively small query batch size. We motivate our method with a simple idea. First, VAE learns a valid low-dimensional latent feature space for the underlying distribution using a Gaussian prior from the labeled and unlabeled pool, whose space is a mixture of the latent features. Then we adopt density-reachable clustering DBSCAN to remove noise in the initial clusters, and we sample the most valuable unlabeled data in high density also in different clusters from the latent space. The framework of our model is shown in Figure2.Figure 2
An active learning method based on VAE and DBSCAN clustering.In detail, the VAE model in our experiment is combined by a convolutional neural network and a deconvolutional neural network. The convolutional one named encoder consists of four convolutional layers, one flatten layer, and three fully connected layers in order. And the deconvolutional one named decoder is relatively simple, consisting of a fully connected layer and two convolutional layers. We update VAE by descending stochastic gradients. After iteration and updating, we obtain the trained parametersθVAE also the trained model VAE. In the end, VAE learned a two-dimensional latent feature space efficiently.The low-dimensional feature vectors learned by VAE serve as the input data for the following clustering methods. In DBSCAN, we start sampling unlabeled points in the clustered classes after an initial unlabeled core object is computed. Here, we note that we remove the noise at the first step. Suppose the needed amount of unlabeled data isC, the total number of all unlabeled density-reachable points in all clusters by DBSCAN is required to be as close to C as possible. The purpose is to ensure that different types of high-density unlabeled data can be retrieved as much as possible. After that, C number of corresponding original unlabeled data will be sampled by an acquisition function and to be annotated by Oracle. Then put them into the original labeled pool and use the task learner to test the mean accuracy of our proposed model.


### 3.2. Algorithms

For clarity, we describe our method in Algorithms1 and 2.Algorithm 1: Variational autoencoder for active learning.
Input: labeled poolXL,YL, unlabeled pool XU, and initialized model θVAEInput: hyperparameters: epochs and step sizeα(1)
fore = 1 to epochs do(2)
SamplexL,yL∼XL,YL(3)
SamplexU∼XU(4)
Compute the reconstruction loss of VAEL1 by using equation (1)(5)
Compute the regularization loss of VAEL2 by using equation (2)(6)
LVAE←L1+L2(7)
Update VAE by descending stochastic gradients:θVAE′←θVAE−α∇LVAE(8)
end for(9)
return trainedθVAE.Algorithm 2: Sampling strategy by DBSCAN in the proposed model.
Input:XL,YL, XU, query batch size: k, and parameters: Eps and MinPts, N: total budget, and n: the amount of miniquery(1)
Sampleε∼N0,1(2)
Samplezii=1N from the underlying distribution by using equation (4)(3)
Samplez1,z2,…,zk as P randomly from zii=1N and shuffle, k<N(4)
ClusterP by adjusting Eps and MinPts(5)
Remove noise(6)
Sample all density-reachable unlabeled setC in all the clusters(7)
forj=1,2,…,N/ndo(8)
Sample the needed amount ofx1,x2,…,xj∈Xj′ from C randomly, and find the corresponding original high-dimensional samplesXj(9)
Yo←ORACLEXj(10)
XL,YL←XL,YL∪Xj,Yo(11)
XU←XU−Xj(12)
end forOutput:XL,YL,XU


## 3.1. The Combination of VAE and DBSCAN Clustering

In this paper, we suggested a new active learning method based on VAE and DBSCAN clustering on a relatively small query batch size. We motivate our method with a simple idea. First, VAE learns a valid low-dimensional latent feature space for the underlying distribution using a Gaussian prior from the labeled and unlabeled pool, whose space is a mixture of the latent features. Then we adopt density-reachable clustering DBSCAN to remove noise in the initial clusters, and we sample the most valuable unlabeled data in high density also in different clusters from the latent space. The framework of our model is shown in Figure2.Figure 2
An active learning method based on VAE and DBSCAN clustering.In detail, the VAE model in our experiment is combined by a convolutional neural network and a deconvolutional neural network. The convolutional one named encoder consists of four convolutional layers, one flatten layer, and three fully connected layers in order. And the deconvolutional one named decoder is relatively simple, consisting of a fully connected layer and two convolutional layers. We update VAE by descending stochastic gradients. After iteration and updating, we obtain the trained parametersθVAE also the trained model VAE. In the end, VAE learned a two-dimensional latent feature space efficiently.The low-dimensional feature vectors learned by VAE serve as the input data for the following clustering methods. In DBSCAN, we start sampling unlabeled points in the clustered classes after an initial unlabeled core object is computed. Here, we note that we remove the noise at the first step. Suppose the needed amount of unlabeled data isC, the total number of all unlabeled density-reachable points in all clusters by DBSCAN is required to be as close to C as possible. The purpose is to ensure that different types of high-density unlabeled data can be retrieved as much as possible. After that, C number of corresponding original unlabeled data will be sampled by an acquisition function and to be annotated by Oracle. Then put them into the original labeled pool and use the task learner to test the mean accuracy of our proposed model.


## 3.2. Algorithms

For clarity, we describe our method in Algorithms1 and 2.Algorithm 1: Variational autoencoder for active learning.
Input: labeled poolXL,YL, unlabeled pool XU, and initialized model θVAEInput: hyperparameters: epochs and step sizeα(1)
fore = 1 to epochs do(2)
SamplexL,yL∼XL,YL(3)
SamplexU∼XU(4)
Compute the reconstruction loss of VAEL1 by using equation (1)(5)
Compute the regularization loss of VAEL2 by using equation (2)(6)
LVAE←L1+L2(7)
Update VAE by descending stochastic gradients:θVAE′←θVAE−α∇LVAE(8)
end for(9)
return trainedθVAE.Algorithm 2: Sampling strategy by DBSCAN in the proposed model.
Input:XL,YL, XU, query batch size: k, and parameters: Eps and MinPts, N: total budget, and n: the amount of miniquery(1)
Sampleε∼N0,1(2)
Samplezii=1N from the underlying distribution by using equation (4)(3)
Samplez1,z2,…,zk as P randomly from zii=1N and shuffle, k<N(4)
ClusterP by adjusting Eps and MinPts(5)
Remove noise(6)
Sample all density-reachable unlabeled setC in all the clusters(7)
forj=1,2,…,N/ndo(8)
Sample the needed amount ofx1,x2,…,xj∈Xj′ from C randomly, and find the corresponding original high-dimensional samplesXj(9)
Yo←ORACLEXj(10)
XL,YL←XL,YL∪Xj,Yo(11)
XU←XU−Xj(12)
end forOutput:XL,YL,XU


## 4. Experiments

### 4.1. Dataset and Task Module

We evaluated our method on three typical image datasets MNIST [24, 25], Fashion-MNIST, and CIFAR-10. The first one MNIST contains 60,000/10,000 (train/test) handwritten digit images (0∼9) with the resolution of 28×28. And Fashion-MNIST is an alternative version of MNIST which has the same size as MNIST with frontal images of different items from 10 categories such as shirts, pants, sandals, and bags. They are also the single channel images with the same size of 28×28. CIFAR-10 contains 10 categories of images such as airplanes, automobiles, birds, and cats. Different from the two datasets above, images in CIFAR-10 are 3-channel color RGB images with the size of 32×32. Not only the noise is very large, but also the proportion and characteristics of the object are not the same, which brings great difficulties for recognition.We used the classic LeNet architecture [26] as our task module for MNIST and Fashion-MNIST, while for CIFAR-10 task we used VGG-16 (Simonyan and Zisserman, 2014) as our architecture. And we took a simple convolutional network just similar to LeNet architecture as the second task module when comparing the different clustering active learning algorithms. This module consists of three convolutional layers, two maximum pooling layers, a flatten layer, and two fully connected layers in order.


### 4.2. Baseline Algorithms

To demonstrate the effectiveness of our method, we took the following four common active learning methods and also the random sampling as the baseline algorithms. They are briefly described as follows:(i)
Random: the query batch is chosen uniformly at random(ii)
Uncertainty: uncertainty sampling with minimal top confidence(iii)
Core Set: in every round of query, sample the unlabeled data which are of the largest distance from the labeled set and then add to the labeled pool(iv)
EGL: estimated gradient length(v)
Bayesian: Bayesian uncertainty sampling with minimal top confidenceFurthermore, in order to comparing the performance of different clustering algorithms [27] in active learning task, we replaced DBSCAN by two classic and widely used clustering algorithms: K-means [28] and Mean shift [29]. And they are also compared with random sampling. These clustering methods in our experiment were based on a two-dimensional feature dataset learned by VAE which are extracted from the three datasets listed above. Their brief description is as follows:(i)
K-means with VAE: after VAE, we sample the two-dimensional data around the cluster centers according to K-means clustering rule and a given radius.(ii)
Mean shift with VAE: after VAE, solve for a vector that moves the center of the circle in the direction with the highest density of the dataset, and the average position of the points inside the circle is found as the new center position in each iteration. We choose the points around these centers.(iii)
DBSCAN with VAE: our method, detailed above.


### 4.3. Implementation Details

In our VAE model, the learning rate was selected by default asα=1×10−3. What is more, we chose Sigmoid as our optimizer. Based on a relative small query batch size, it shows that when epochs reach about 400, the learning loss almost does not decrease and the distribution of latent feature space learned by VAE comes to be stable. In other words, we set the number of training epochs to 400 to prevent overfitting. Figure 3 shows the two-dimensional feature points learned by VAE in different epochs. After VAE is trained, a certain number of low-dimensional data samples were extracted by its well-trained encoder. We checked some corresponding original images as shown in Figure 4 (taking Fashion-MNIST as an example).Figure 3
The two-dimensional feature points in latent space in different epochs: (a) 20 epochs, (b) 200 epochs, and (c) 2000 epochs.
(a)(b)(c)Figure 4
Some original images corresponding to the data points in latent space.In order to observe the performance of active learning algorithms on a relatively small query batch size, we tried to sample 100 low-dimension data points randomly from the latent space learned by encoder in VAE as our clustering dataset for DBSCAN, including 10 labeled data and 90 unlabeled data, and our query batch size is set to 5 which is relatively small. Finally, we obtained the needed amount of unlabeled density-reachable points in different clusters after removing the noise. It is particularly noted that we adjust the parameters to make the total number of unlabeled points in all density-reachable clusters to be as close as possible to the needed amount of data for annotations.


### 4.4. Results

In our experiment, we used the test accuracy as our metric to evaluate the performance. The results are averaged from 20 runs in order to ensure statistical validity. By the results in Table1, we plot the test accuracy of experiments using four different active learning methods on MNIST in Figure 5. The results demonstrate that, in MNIST, our method, DBSCAN with VAE, performs better on small query batch size than the active learning methods listed above. Figure 5 also shows that some methods perform on par or worse than random sampling. This may be the reason that the sample size is relatively small and the randomness is strong, which reduces the gap between each sampling method and random sampling. It may also be because some methods are more suitable for large query batch sizes.Table 1
Mean accuracy (%) of different methods on MNIST from 20 runs.
Number of labeled examples1520253035404550DBSCAN with VAE49.7658.5159.6760.9064.3869.0269.6871.28Random30.1943.0552.3451.6857.3064.0360.7564.96Uncertainty37.7448.2944.1559.3561.4167.6961.7068.59Core Set49.0348.0354.5154.9956.3458.8953.9760.13EGL44.7346.8041.2752.3749.8951.4957.0559.73Bayesian32.3331.5035.3037.5939.5245.1354.4156.21Figure 5
Mean accuracy (%) of different active learning methods on MNIST.We also notice that, except for EGL performing well than random sampling on very small batch sizes below 20, most algorithms listed above including random sampling consistently outperform EGL. As for EGL, a possible explanation for the discrepancy between these results may be the architecture used for the different tasks because EGL uses the gradient relative to the model parameters as the score function [12]. Among all the methods above, Bayesian performs the poorest and it may be because Bayesian is more suitable for large training datasets.Next, we did a comparison test on CIFAR-10. The result is in Table2 and we plot it in Figure 6.Table 2
Mean accuracy (%) of different methods on CIFAR-10 from 20 runs.
Number of labeled examples1520253035404550DBSCAN with VAE11.1911.8413.4817.2518.8518.8120.7520.98Random11.1011.5010.1711.6012.6312.4016.6418.27Uncertainty10.2211.8712.8712.9213.9113.9515.5620.24Core Set9.7010.4311.1710.3210.8213.2812.3613.65EGL11.219.4113.7115.0916.0915.0119.0218.31Bayesian8.6110.3412.6613.0113.1514.9117.6118.12Figure 6
Mean accuracy (%) of different active learning methods on CIFAR-10.As can be seen from Figure6 on CIFAR-10, Core Set often performs worse than random sampling, and it is also possibly because the random advantage is greater than Core Set when the sample size is small. And when the size is below 25, our approach shows little advantage than other methods, but our approach leads all the way obviously when the number of labeled samples is over 25.Furthermore, we replaced DBSCAN by K-means and Mean shift in order to study the different effects caused by different clustering methods. We also combined them with VAE and did experiment on MNIST and Fashion-MNIST. Detailed experimental results are listed in Tables3 and 4, respectively. And we plot their performance in Figures 7 and 8.Table 3
Mean test accuracy (%) of different clustering methods on MNIST.
Number of labeled examplesRandom (%)K-means with VAE (%)Mean shift with VAE (%)DBSCAN with VAE (%)1544.648.3648.3449.762047.257.7749.9758.512554.5658.7756.9959.673055.4261.6857.3860.903560.2764.0963.3464.384065.1765.5064.9569.024567.5366.5768.8369.685068.0469.2469.1871.28Table 4
Mean test accuracy (%) of different clustering methods on Fashion-MNIST.
Number of labeled examplesRandom (%)K-means with VAE (%)Mean shift with VAE (%)DBSCAN with VAE (%)159.4910.569.5211.19209.8010.7410.3211.842510.4912.2710.6713.483011.6115.1013.5817.253513.5817.4813.2718.854014.6217.1215.9818.814514.4316.2218.3720.755015.2618.3518.4620.98Figure 7
Mean test accuracy (%) of different clustering methods on MNIST.Figure 8
Mean test accuracy (%) of different clustering methods on Fashion-MNIST.However, what we may notice that, in Figure7, the result of K-means with VAE is better than our method when the number of labeled data reaches 30 on MNIST. DBSCAN and K-means perform similarly until size 35 and DBSCAN performs clearly better after size 40. The reason may be that, in this quantity case, a great part of samples sampled by K-means method are closer to the potential characteristics of the labeled data than the data sampled by DBSCAN. In other side, this situation is probably caused by the uneven distribution of the samples when mapping the high-dimensional image data into the two-dimensional latent space. After size 40, DBSCAN may be more sensitive to detect the outlier points in latent space and prevent them being clustered. So the performance of DBSCAN is better than that of K-means when size is greater than 40. Of course, this is a hypothetical explanation. Probably, it is difficult to know for sure why K-means starts performing worse after size 40. However, on CIFAR-10 in Figure 8, compared with random sampling and the other two clustering methods, DBSCAN with VAE behaves better and the performance of increasing accuracy is more stable. It is somewhat clear from the results that our proposed method performs well on the whole on MNIST and Fashion-MNIST.In order to show our experimental results more significantly, the improvement percentage of different clustering methods over random sampling is listed in Tables5 and 6, and we plot their performance in Figures 9 and 10. On MNIST, our method DBSCAN with VAE, achieves the best results when the size of labeled samples reaches 20, increasing the accuracy by 11.30% over random sampling. Generally speaking, our approach performs even better in the other labeled data quantities. However, on Fashion-MNIST, our approach performs clearly better than the other two clustering methods at every labeled size in our experiment. And it achieves the best improvement of 6.32% when the labeled samples size reaches 45.Table 5
Accuracy improvement percentage over random sampling on MNIST.
Number of labeled examplesK-means (%)Mean shift (%)DBSCAN with VAE (%)153.723.705.122010.562.7611.30254.212.435.11306.421.965.48353.823.074.11400.33−0.223.8545−0.961.302.15501.201.773.24Table 6
Accuracy improvement percentage over random sampling on Fashion-MNIST.
Number of labeled examplesK-means (%)Mean shift (%)DBSCAN with VAE (%)151.070.031.70200.940.522.04251.780.182.99303.491.975.64353.90-0.315.27402.501.364.19451.793.946.32503.093.205.72Figure 9
Accuracy improvement percentage over random sampling on MNIST.Figure 10
Accuracy improvement percentage over random sampling on Fashion-MNIST.In the end, we computed the average accuracy improvement of each clustering method combined with VAE to see their overall improvement over random sampling, and we plot them in Figure11. On MNIST, K-means and Mean shift achieve the improvement of 3.26% and 1.86%, respectively. And our method achieves the best performance of 4.48%. And on Fashion-MNIST, the three clustering active learning methods achieve the improvement over random sampling by 3.23%, 1.36%, and 4.23% in order. In a word, our proposed algorithm, DBSCAN with VAE, finally shows its superiority than K-means and Mean shift when combined with VAE on a relatively small query batch size on MNIST and Fashion-MNIST.Figure 11
Mean accuracy improvement percentage on MNIST and Fashion-MNIST.


## 4.1. Dataset and Task Module

We evaluated our method on three typical image datasets MNIST [24, 25], Fashion-MNIST, and CIFAR-10. The first one MNIST contains 60,000/10,000 (train/test) handwritten digit images (0∼9) with the resolution of 28×28. And Fashion-MNIST is an alternative version of MNIST which has the same size as MNIST with frontal images of different items from 10 categories such as shirts, pants, sandals, and bags. They are also the single channel images with the same size of 28×28. CIFAR-10 contains 10 categories of images such as airplanes, automobiles, birds, and cats. Different from the two datasets above, images in CIFAR-10 are 3-channel color RGB images with the size of 32×32. Not only the noise is very large, but also the proportion and characteristics of the object are not the same, which brings great difficulties for recognition.We used the classic LeNet architecture [26] as our task module for MNIST and Fashion-MNIST, while for CIFAR-10 task we used VGG-16 (Simonyan and Zisserman, 2014) as our architecture. And we took a simple convolutional network just similar to LeNet architecture as the second task module when comparing the different clustering active learning algorithms. This module consists of three convolutional layers, two maximum pooling layers, a flatten layer, and two fully connected layers in order.


## 4.2. Baseline Algorithms

To demonstrate the effectiveness of our method, we took the following four common active learning methods and also the random sampling as the baseline algorithms. They are briefly described as follows:(i)
Random: the query batch is chosen uniformly at random(ii)
Uncertainty: uncertainty sampling with minimal top confidence(iii)
Core Set: in every round of query, sample the unlabeled data which are of the largest distance from the labeled set and then add to the labeled pool(iv)
EGL: estimated gradient length(v)
Bayesian: Bayesian uncertainty sampling with minimal top confidenceFurthermore, in order to comparing the performance of different clustering algorithms [27] in active learning task, we replaced DBSCAN by two classic and widely used clustering algorithms: K-means [28] and Mean shift [29]. And they are also compared with random sampling. These clustering methods in our experiment were based on a two-dimensional feature dataset learned by VAE which are extracted from the three datasets listed above. Their brief description is as follows:(i)
K-means with VAE: after VAE, we sample the two-dimensional data around the cluster centers according to K-means clustering rule and a given radius.(ii)
Mean shift with VAE: after VAE, solve for a vector that moves the center of the circle in the direction with the highest density of the dataset, and the average position of the points inside the circle is found as the new center position in each iteration. We choose the points around these centers.(iii)
DBSCAN with VAE: our method, detailed above.


## 4.3. Implementation Details

In our VAE model, the learning rate was selected by default asα=1×10−3. What is more, we chose Sigmoid as our optimizer. Based on a relative small query batch size, it shows that when epochs reach about 400, the learning loss almost does not decrease and the distribution of latent feature space learned by VAE comes to be stable. In other words, we set the number of training epochs to 400 to prevent overfitting. Figure 3 shows the two-dimensional feature points learned by VAE in different epochs. After VAE is trained, a certain number of low-dimensional data samples were extracted by its well-trained encoder. We checked some corresponding original images as shown in Figure 4 (taking Fashion-MNIST as an example).Figure 3
The two-dimensional feature points in latent space in different epochs: (a) 20 epochs, (b) 200 epochs, and (c) 2000 epochs.
(a)(b)(c)Figure 4
Some original images corresponding to the data points in latent space.In order to observe the performance of active learning algorithms on a relatively small query batch size, we tried to sample 100 low-dimension data points randomly from the latent space learned by encoder in VAE as our clustering dataset for DBSCAN, including 10 labeled data and 90 unlabeled data, and our query batch size is set to 5 which is relatively small. Finally, we obtained the needed amount of unlabeled density-reachable points in different clusters after removing the noise. It is particularly noted that we adjust the parameters to make the total number of unlabeled points in all density-reachable clusters to be as close as possible to the needed amount of data for annotations.


## 4.4. Results

In our experiment, we used the test accuracy as our metric to evaluate the performance. The results are averaged from 20 runs in order to ensure statistical validity. By the results in Table1, we plot the test accuracy of experiments using four different active learning methods on MNIST in Figure 5. The results demonstrate that, in MNIST, our method, DBSCAN with VAE, performs better on small query batch size than the active learning methods listed above. Figure 5 also shows that some methods perform on par or worse than random sampling. This may be the reason that the sample size is relatively small and the randomness is strong, which reduces the gap between each sampling method and random sampling. It may also be because some methods are more suitable for large query batch sizes.Table 1
Mean accuracy (%) of different methods on MNIST from 20 runs.
Number of labeled examples1520253035404550DBSCAN with VAE49.7658.5159.6760.9064.3869.0269.6871.28Random30.1943.0552.3451.6857.3064.0360.7564.96Uncertainty37.7448.2944.1559.3561.4167.6961.7068.59Core Set49.0348.0354.5154.9956.3458.8953.9760.13EGL44.7346.8041.2752.3749.8951.4957.0559.73Bayesian32.3331.5035.3037.5939.5245.1354.4156.21Figure 5
Mean accuracy (%) of different active learning methods on MNIST.We also notice that, except for EGL performing well than random sampling on very small batch sizes below 20, most algorithms listed above including random sampling consistently outperform EGL. As for EGL, a possible explanation for the discrepancy between these results may be the architecture used for the different tasks because EGL uses the gradient relative to the model parameters as the score function [12]. Among all the methods above, Bayesian performs the poorest and it may be because Bayesian is more suitable for large training datasets.Next, we did a comparison test on CIFAR-10. The result is in Table2 and we plot it in Figure 6.Table 2
Mean accuracy (%) of different methods on CIFAR-10 from 20 runs.
Number of labeled examples1520253035404550DBSCAN with VAE11.1911.8413.4817.2518.8518.8120.7520.98Random11.1011.5010.1711.6012.6312.4016.6418.27Uncertainty10.2211.8712.8712.9213.9113.9515.5620.24Core Set9.7010.4311.1710.3210.8213.2812.3613.65EGL11.219.4113.7115.0916.0915.0119.0218.31Bayesian8.6110.3412.6613.0113.1514.9117.6118.12Figure 6
Mean accuracy (%) of different active learning methods on CIFAR-10.As can be seen from Figure6 on CIFAR-10, Core Set often performs worse than random sampling, and it is also possibly because the random advantage is greater than Core Set when the sample size is small. And when the size is below 25, our approach shows little advantage than other methods, but our approach leads all the way obviously when the number of labeled samples is over 25.Furthermore, we replaced DBSCAN by K-means and Mean shift in order to study the different effects caused by different clustering methods. We also combined them with VAE and did experiment on MNIST and Fashion-MNIST. Detailed experimental results are listed in Tables3 and 4, respectively. And we plot their performance in Figures 7 and 8.Table 3
Mean test accuracy (%) of different clustering methods on MNIST.
Number of labeled examplesRandom (%)K-means with VAE (%)Mean shift with VAE (%)DBSCAN with VAE (%)1544.648.3648.3449.762047.257.7749.9758.512554.5658.7756.9959.673055.4261.6857.3860.903560.2764.0963.3464.384065.1765.5064.9569.024567.5366.5768.8369.685068.0469.2469.1871.28Table 4
Mean test accuracy (%) of different clustering methods on Fashion-MNIST.
Number of labeled examplesRandom (%)K-means with VAE (%)Mean shift with VAE (%)DBSCAN with VAE (%)159.4910.569.5211.19209.8010.7410.3211.842510.4912.2710.6713.483011.6115.1013.5817.253513.5817.4813.2718.854014.6217.1215.9818.814514.4316.2218.3720.755015.2618.3518.4620.98Figure 7
Mean test accuracy (%) of different clustering methods on MNIST.Figure 8
Mean test accuracy (%) of different clustering methods on Fashion-MNIST.However, what we may notice that, in Figure7, the result of K-means with VAE is better than our method when the number of labeled data reaches 30 on MNIST. DBSCAN and K-means perform similarly until size 35 and DBSCAN performs clearly better after size 40. The reason may be that, in this quantity case, a great part of samples sampled by K-means method are closer to the potential characteristics of the labeled data than the data sampled by DBSCAN. In other side, this situation is probably caused by the uneven distribution of the samples when mapping the high-dimensional image data into the two-dimensional latent space. After size 40, DBSCAN may be more sensitive to detect the outlier points in latent space and prevent them being clustered. So the performance of DBSCAN is better than that of K-means when size is greater than 40. Of course, this is a hypothetical explanation. Probably, it is difficult to know for sure why K-means starts performing worse after size 40. However, on CIFAR-10 in Figure 8, compared with random sampling and the other two clustering methods, DBSCAN with VAE behaves better and the performance of increasing accuracy is more stable. It is somewhat clear from the results that our proposed method performs well on the whole on MNIST and Fashion-MNIST.In order to show our experimental results more significantly, the improvement percentage of different clustering methods over random sampling is listed in Tables5 and 6, and we plot their performance in Figures 9 and 10. On MNIST, our method DBSCAN with VAE, achieves the best results when the size of labeled samples reaches 20, increasing the accuracy by 11.30% over random sampling. Generally speaking, our approach performs even better in the other labeled data quantities. However, on Fashion-MNIST, our approach performs clearly better than the other two clustering methods at every labeled size in our experiment. And it achieves the best improvement of 6.32% when the labeled samples size reaches 45.Table 5
Accuracy improvement percentage over random sampling on MNIST.
Number of labeled examplesK-means (%)Mean shift (%)DBSCAN with VAE (%)153.723.705.122010.562.7611.30254.212.435.11306.421.965.48353.823.074.11400.33−0.223.8545−0.961.302.15501.201.773.24Table 6
Accuracy improvement percentage over random sampling on Fashion-MNIST.
Number of labeled examplesK-means (%)Mean shift (%)DBSCAN with VAE (%)151.070.031.70200.940.522.04251.780.182.99303.491.975.64353.90-0.315.27402.501.364.19451.793.946.32503.093.205.72Figure 9
Accuracy improvement percentage over random sampling on MNIST.Figure 10
Accuracy improvement percentage over random sampling on Fashion-MNIST.In the end, we computed the average accuracy improvement of each clustering method combined with VAE to see their overall improvement over random sampling, and we plot them in Figure11. On MNIST, K-means and Mean shift achieve the improvement of 3.26% and 1.86%, respectively. And our method achieves the best performance of 4.48%. And on Fashion-MNIST, the three clustering active learning methods achieve the improvement over random sampling by 3.23%, 1.36%, and 4.23% in order. In a word, our proposed algorithm, DBSCAN with VAE, finally shows its superiority than K-means and Mean shift when combined with VAE on a relatively small query batch size on MNIST and Fashion-MNIST.Figure 11
Mean accuracy improvement percentage on MNIST and Fashion-MNIST.


## 5. Conclusion and Discussion

In this paper, we proposed a new active learning method based on VAE and DBSCAN designed for neural networks with a relatively small query batch size. It overcomes the difficulty of distance representation in high dimensions and prevents the distance concentration phenomenon from occurring in the computational learning literature with respect to high-dimensional p-norms. Based on the results on MNIST, Fashion-MNIST, and CIFAR-10, we empirically show that our method achieves competitive performance compared with four common active learning methods listed above, and it is also superior to the other two clustering active learning methods for image classification when the query batch size is relatively small. For active learning models are usually based on a relatively large query batch sizes, our approach of small query batch sizes can be regarded as a supplement to the previous research studies. In addition, our method is simple to implement and can conceptually be extended to other domains, and we see it as a welcome addition to the arsenal of methods in use today. Furthermore, the inadequacy of our method is that manual parameter adjustment is required to make the total number of unlabeled points in all density-reachable clusters to be as close as possible to the needed amount of data for annotations in DBSCAN. And we will still thoroughly study the related work presented in this paper in the future.



---

*Source: 9952596-2021-08-02.xml*