# Multiview Active Learning for Scene Classification with High-Level Semantic-Based Hypothesis Generation

Authors: Tuozhong Yao; Wenfeng Wang; Yuhong Gu; Qiuguo Zhu
Journal: Scientific Programming
 (2020)

---

## Abstract

Multiview active learning (MVAL) is a technique which can result in a large decrease in the size of the version space than traditional active learning and has great potential applications in large-scale data analysis. This paper made research on MVAL-based scene classification for helping the computer accurately understand diverse and complex environments macroscopically, which has been widely used in many fields such as image retrieval and autonomous driving. The main contribution of this paper is that different high-level image semantics are used for replacing the traditional low-level features to generate more independent and diverse hypotheses in MVAL. First, our algorithm uses different object detectors to achieve local object responses in the scenes. Furthermore, we design a cascaded online LDA model for mining the theme semantic of an image. The experimental results demonstrate that our proposed theme modeling strategy fits the large-scale data learning, and our MVAL algorithm with both high-level semantic views can achieve significant improvement in the scene classification than traditional active learning-based algorithms.


---

## Body

## 1. Introduction

Scene classification is defined as using a computer to understand the class of an image scene. The related research studies can be roughly divided into two branches: some focus on fast holistic scene perception based on visual psychology and physiology [1, 2], while others build the statistical models through local image analysis to understand the scene, which is also the main developing tendency [3–5]. There have been many methods for image representation in the past two decades, which is a key step for scene classification. Low-level features such as color, texture, and edge have been widely used to represent the local regions of an image. Some researchers trained object detectors to achieve high-level semantics such as object’s class, size, and shape for more accurate image representation [6, 7]. Prevailing statistical models are bag-of-words (BoW) and related theme statistical models. These models reduce the gap between the low-level features and high-level semantics by mining the hidden themes from local image regions such as pLSA [8] and LDA [9]. Other new scene statistical models [10–12] were proposed for more accurate object recognition in the scene. However, these mentioned models above mainly focus on the occurrence of the image semantics, and the spatial semantic correlations between different image regions are usually ignored.For mining the spatial context information from an image, some researchers considered the information interaction between different spatial pyramid levels [13–15], and how to build reasonable attention mechanisms also can lead to significant improvement for scene classification. These methods used deep neural networks, and their large-scale network parameter estimation tasks usually lead to much higher computational complexity than nondeep learning based methods.Active learning ranks the unlabeled samples iteratively and only selects the samples with high uncertainty or which cause great ambiguity for the classifier. In PAC learning theory, compared with traditional passive learning, it can exponentially reduce its sample complexity toOlog1/ε in the feature space for learning a classifier with expectation classification error ε [16–18], which has good potential of wide application in large-scale data leaning. However, most of the traditional active learning algorithms’ lack of diversity of the hypotheses is generated usually by low-level image features, which affects their performances. This paper proposed a MVAL-based scene classification algorithm, which uses different high-level semantics as its views and can realize a decrease in more than a half size of the version space, and it is more efficient than both single-hypothesis-based and committee-based active learning [19].


## 2. Materials and Methods

### 2.1. Proposed Algorithm

The flowchart of our proposed algorithm is illustrated in Figure1. Our algorithm uses different high-level semantics as its views to generate the corresponding hypotheses. First, object detectors are trained to achieve the responses of different object classes in image regions. Furthermore, we design a cascaded online LDA (CO-LDA) as a secondary view for achieving more accurate image representation. Finally, a fine-tuned MVAL algorithm is utilized with both two high-level image semantics as its views for classifying the scene of an image.Figure 1
The flowchart of our scene classification algorithm.


### 2.2. Object Semantic-Based Image Representation

Our object semantic-based image representation is illustrated in Figure2.Figure 2
The flowchart of object semantic-based image representation.First, multiple object objectors are used to achieve the local object response maps. Second, these maps are decomposed into three spatial pyramid levels, and the maximal object responses are computed in image blocks in each spatial level, which is annotated as red blocks in Figure2. Finally, an object response histogram is computed, which can effectively reduce the influence of object response error in the whole image. For generating the object response, a latent SVM-based detector [7] is applied for recognizing the object classes with bulk type such as car and pedestrian. Another geometric context-based detector [6] is utilized for recognizing the object classes with different textures such as tree, sky, and building.


### 2.3. Theme Semantic-Based Image Representation

For satisfying the dynamic update of an active learning training set, an online LDA model [20] based on stochastic gradient descent strategy is used. It adds new samples sequentially, and old samples have been no longer stored, which can achieve efficient and accurate parameter estimation in large-scale data training.Online LDA computes the posterior probability distributionpθ,z,w,βα,η of the hidden nodes based on observed samples. It actually uses variational inference to estimate the maximum likelihood of pwα,η based on α and η. Three variational parameters ϕ, γ, and λ follow the distributions: ϕ∼multinomialε, γ∼Dirichletε, and λ∼Dirichletε. The variational distribution follows(1)qβ1:k,θ1:M,z1:Mλ,γ,ϕ=∏i=1kDirβiλi⋅∏d=1Mqdθd,zdϕd,γd.The optimalγ,ϕ is solved by maximizing the lower bound in the following equation:(2)logpwα,η≥Lw,ϕ,γ,λ=Eqlogpθ,z,w,βα,η−Eqlogqθ,z,β,where Eq denotes the conditional mathematical expectation. Maximizing the lower bound Lw,ϕ,γ,λ is equivalent to minimizing KL divergence of qθ,z,βγ,ϕ and pθ,z,βw,α,η:(3)logpwα,η=Lw,ϕ,γ,λ+KLqθ,z,βpθ,z,βw,α,η,where Lw,ϕ,γ,λ is factorized as follows:(4)Lw,ϕ,γ,λ=∑dEqlogpwdθd,zd,β+Eqlogpzdθd−Eqlogqzd+Eqlogpθdα−Eqlogqθd+Eqlogpβη−EqlogqβM.Equation (4) can be transformed into formula (5). In equation (5), ndw denotes the frequency that word w occurs in text d. lnd,ϕd,γd,λ reflects the contribution of d for the lower bound, which is iteratively optimized by a coordinate ascent algorithm:(5)Lw,ϕ,γ,λ=∑d∑wndw∑kϕdwkEqlogθdk+Eqlogβkw−logϕdwk−logΓ∑kγdk+∑kα−γdkEqlogθdk+logΓγdk+∑k−logΓ∑kλkw+∑wη−λkwEqlogβkw+logΓλkwM+logΓKα−KlogΓα+logΓNη−WlogΓηM,=∑dlnd,ϕd,γd,λ.ϕdwkin equation (5) is iteratively solved:(6)ϕdwk∝expEqlogθdk+Eqlogβkw,Eqlogθdk=Ψγdk−Ψ∑i=1Kγdi,Eqlogβkw=Ψλkw−Ψ∑i=1Nλki,where digamma functionΨ is the first-order derivative of function Γ. γdk and λkw are iteratively solved in the following way: γdk=α+∑wndwϕdwk,λkw=η+∑dndwϕdwk.Whentth vector of word frequency nt is observed, we keep λ unchanged and update the local optimal solution of γt and ϕt in E step. In M step, ϕt and λ from last iteration are both used to update λ:(7)λ=1−ρtλ+ρtλ˜,ρt=τ+t−k.λ˜ in formula (7) is solved as follows:(8)λ˜kw=η+MS∑sntskϕtskw,where nts is sth text in each batch text set, M is the number of the training text set, and Sis the size of each batch text set. Hyperparameters α and η are updated by the Newton–Rapson method: α⟵α−ρtα˜γtandη⟵η−ρtη˜λ. Here, α˜γt is the product of Hessian matrix and gradient ∇αl of the objective function lnd,ϕd,γd,λ. η˜λ is the product of Hessian matrix and gradient ∇ηL of the objective function Lw,ϕ,γ,λ.Based on online LDA, we proposed the CO-LDA theme model, which is similar with the classic SP-pLSA model in structure for enhancing the spatial correlation between different image regions. The framework of CO-LDA is illustrated in Figure3. The main difference between CO-LDA and SP-pLSA is that different online LDAs (LDA1, LDA2, and LDA3) are applied in different spatial levels to jointly mine the theme of an image. The main advantage of CO-LDA is that it integrates the spatial correlation of objects in different image resolutions, which further improves the holistic scene understanding. The visual histogram computation in online LDA is the same as the way of object response histogram in Section 2.2, and the theme feature of each spatial block is represented by variational parameter γ of the online LDA model.Figure 3
The framework of the CO-LDA theme model.Finally, the theme featureγ of the whole image is achieved by concatenating the theme features of different blocks of different spatial pyramid levels:(9)γ=w1⋅γL0⊕w2⋅γL1⊕w3⋅γL2,where γLi denotes the theme feature of the corresponding block in Lith pyramid level, ⊕ denotes the linear concatenation between feature vectors, and the weights of different spatial levels are configured as follows: w1=1/2,w2=1/2,andw3=1/4.


### 2.4. Multiview Active Learning

The MVAL referred in this paper is our previous work [21], which has two improvements in both hypothesis generation and selective sampling. First, boosting-like technique is integrated into MVAL, which uses a similar way of iterative weak classifier optimization, and the current hypothesis is boosted by weighted voting of all the hypotheses from the past queries. Furthermore, an adaptive hierarchical competition sampling is presented. In this sampling strategy, if the number of the contention samples is large, an unsupervised spectral clustering is activated to obtain the coarse spatial distribution of these contention samples in the high-dimensional feature space, and then, a multiview-based batch mode selective sampling is run based on two measures: sample uncertainty and redundancy by solving quadratic programming to determine the queried samples in each cluster.


#### 2.4.1. Hypothesis Generation

If an active learning can select enough number of contention samples, which could improve the hypothesis in each query, the number of unlabeled samples, which are incorrectly classified, will decrease. It is quite similar with boosting technique in weak classifier optimization. The MVAL incorporates the AdaBoost algorithm into our framework to boost the generated hypothesis in each query, and the main flowchart is described in Figure4.Figure 4
The framework of hypothesis generation in our MAVL [21].In Figure4, a support vector machine (SVM) is used as a base classifier to construct a multiview classifier, which replaces the single-view classifier in AdaBoost, and this multiview classifier in each query can be considered as a weak classifier in each iteration in AdaBoost. The hypothesis of multiview classifier hix is computed by weighted voting of n SVM base classifiers v1,v2,…,vn whose weights are ω1,ω2,…,ωn. Unlike traditional query by boosting, we update the weight of each base classifier in each query and obtain the boosted hypothesis Hix by weighting all the hypotheses from the past queries and not from the current query only.The detailed process of the MAVL’s hypothesis generation based on AdaBoost is as follows:(a)
In iterationhtxj=∑f∈f1t,f2t,…fntωitfitxj, weighted voting is used to generate the initial multiview-based hypothesis:(10)htxj=∑f∈f1t,f2t,…,fntωitfitxj,wherefitxj is the classification confidence of sample xj by view i, and ωit denotes the contribution of view i for classification which is determined by the soft classification error rate εit, which defines how correctly a sample is classified:(11)εit=1∑x∈L,y=1fitx−∑x∈L,y=−1fitx,where∑x∈L,y=1fitx and ∑x∈L,y=−1fitx denote the sum of classification confidence of unlabeled samples, which are labeled as y=1 and y=−1, respectively. For a “positive/negative” sample, the distance of it to the decision boundary in the “positive/negative” side reflects the degree of how correctly it is classified, and this information is utilized to calculate the error degree εit here instead of the traditional classification error calculated by the decision hypothesis in AdaBoost. Also, ωit is updated through the following way: ωit=1/Z1tln1−εit/εit, where Z1t is the normalized weight. Then, the classification confidence δt of the multiview classifier can be computed by the following equation:(12)δt=∑j=1Nβjthtxj−yj.(b)
After iterationt, the size of the labeled sample set is increased as follows: Jt=Jt−1∪Lt. Jt denotes the labeled sample set in iteration t, and Lt denotes the newly added samples after query. As we know, the size of the labeled samples set Jt is increased during iteration in active learning. Thus, if the size of the initial labeled training set is small, the influence of Jt should be considered when updating the weight ηt of the multiview classifier, which is illustrated by the following equation:(13)ηt=1Z2tln1−δtδt+λJt.Then, the weight of each sample is updated through the following way:ωjt+1=ωjtβt1−et, where βt=δt/1−δt, if xj is correctly classified, ej=0, otherwise, ej=1.(c)
The final boosted hypothesisHtx of the queried sample xi is equivalent to the weighted sum of all the hypotheses from the past K queries, which is defined by(14)Htx=1,∑t=1Kηthtx≥12∑t=1Kηt,0,else.


#### 2.4.2. Sampling Strategy

The MVAL uses a new hierarchical competition-based sampling strategy in order to query the contention samples with high probability in different sample distributions, which is illustrated in Figure5.Figure 5
The framework of sampling strategy in our MAVL [21].(1)Intercluster Sampling Competition. In the MVAL, a fast approximate spectral clustering algorithm is designed to reduce the computational complexity significantly to OKNT+OK3, where T is the iteration number of K mean clustering, and N is the total number of contention samples. The detailed process is illustrated as follows: (a) perform traditional K mean clustering on the contention unlabeled samples x1,x2,…,xN, compute the centroid of each cluster y1,y2,…,yK as K representative points, and build a correspondence table to associate each xi with the nearest cluster centroid yiyi; (b) run the normalized cut algorithm on y1,y2,…,yK to obtain a m-way cluster membership for each of yi; and (c) recover the cluster membership for each xi by looking up the cluster membership of the corresponding centroid yi in the corresponding table.After fast spectral clustering, two intercluster sampling measures are defined: the number of samples in the cluster and its information entropy. Both measures are weighted to obtain the number of selected samplesNCS in cluster C in the following equation:(15)NCS=NTZγNumC+1−γEntC,C=1,2,…,K,where NumC is proportional to the total number of samples NC in cluster C, and computing EntC is equivalent to kernel density estimation of x in cluster C. Weight γ=0.5 reflects the impact of both measures in intercluster sampling competition, Z is the normalized factor, NT is the total number of selected samples in the current query, and ⋅ is rounding operation.(2)Intracluster Sampling Competition. In the MVAL, an efficient quadratic programming based-method [22] is utilized, which dynamically estimates the weights of the redundancy and uncertainty of an unlabeled sample in each query. It is used for intracluster selective sampling and solved by minimizing the following object function:(16)minp∈Rn−lpTf˜v+12pTKu,ups.t.pTu=k,0≤p≤1.Equation (16) aims to estimate the normalized parameter pi∈0,1, which reflects how probable the unlabeled sample is selected. f^v=fvx1,…,fvxlT is the classification confidence of sample x in vth view. x1,…,xl are the queried unlabeled samples, u is a unit vector, and k=NCS is the number of unlabeled samples in batch mode. The first part denotes the sample uncertainty in vth view, and the sampling strategy tends to select the contention sample near the classification hyperplane of vth view by minimizing pTf˜v. The second part denotes the sample redundancy in vth view, and the similar samples are selected by minimizing pTKu,up. The sampling probability p is calculated by a convex quadratic programming, and finally, p1v,p2v,…,plv, which corresponds to x1,…,xl in vth view, is obtained. For selective sampling in each cluster, the conservative sampling strategy is utilized in a classic co-testing algorithm [23].


## 2.1. Proposed Algorithm

The flowchart of our proposed algorithm is illustrated in Figure1. Our algorithm uses different high-level semantics as its views to generate the corresponding hypotheses. First, object detectors are trained to achieve the responses of different object classes in image regions. Furthermore, we design a cascaded online LDA (CO-LDA) as a secondary view for achieving more accurate image representation. Finally, a fine-tuned MVAL algorithm is utilized with both two high-level image semantics as its views for classifying the scene of an image.Figure 1
The flowchart of our scene classification algorithm.


## 2.2. Object Semantic-Based Image Representation

Our object semantic-based image representation is illustrated in Figure2.Figure 2
The flowchart of object semantic-based image representation.First, multiple object objectors are used to achieve the local object response maps. Second, these maps are decomposed into three spatial pyramid levels, and the maximal object responses are computed in image blocks in each spatial level, which is annotated as red blocks in Figure2. Finally, an object response histogram is computed, which can effectively reduce the influence of object response error in the whole image. For generating the object response, a latent SVM-based detector [7] is applied for recognizing the object classes with bulk type such as car and pedestrian. Another geometric context-based detector [6] is utilized for recognizing the object classes with different textures such as tree, sky, and building.


## 2.3. Theme Semantic-Based Image Representation

For satisfying the dynamic update of an active learning training set, an online LDA model [20] based on stochastic gradient descent strategy is used. It adds new samples sequentially, and old samples have been no longer stored, which can achieve efficient and accurate parameter estimation in large-scale data training.Online LDA computes the posterior probability distributionpθ,z,w,βα,η of the hidden nodes based on observed samples. It actually uses variational inference to estimate the maximum likelihood of pwα,η based on α and η. Three variational parameters ϕ, γ, and λ follow the distributions: ϕ∼multinomialε, γ∼Dirichletε, and λ∼Dirichletε. The variational distribution follows(1)qβ1:k,θ1:M,z1:Mλ,γ,ϕ=∏i=1kDirβiλi⋅∏d=1Mqdθd,zdϕd,γd.The optimalγ,ϕ is solved by maximizing the lower bound in the following equation:(2)logpwα,η≥Lw,ϕ,γ,λ=Eqlogpθ,z,w,βα,η−Eqlogqθ,z,β,where Eq denotes the conditional mathematical expectation. Maximizing the lower bound Lw,ϕ,γ,λ is equivalent to minimizing KL divergence of qθ,z,βγ,ϕ and pθ,z,βw,α,η:(3)logpwα,η=Lw,ϕ,γ,λ+KLqθ,z,βpθ,z,βw,α,η,where Lw,ϕ,γ,λ is factorized as follows:(4)Lw,ϕ,γ,λ=∑dEqlogpwdθd,zd,β+Eqlogpzdθd−Eqlogqzd+Eqlogpθdα−Eqlogqθd+Eqlogpβη−EqlogqβM.Equation (4) can be transformed into formula (5). In equation (5), ndw denotes the frequency that word w occurs in text d. lnd,ϕd,γd,λ reflects the contribution of d for the lower bound, which is iteratively optimized by a coordinate ascent algorithm:(5)Lw,ϕ,γ,λ=∑d∑wndw∑kϕdwkEqlogθdk+Eqlogβkw−logϕdwk−logΓ∑kγdk+∑kα−γdkEqlogθdk+logΓγdk+∑k−logΓ∑kλkw+∑wη−λkwEqlogβkw+logΓλkwM+logΓKα−KlogΓα+logΓNη−WlogΓηM,=∑dlnd,ϕd,γd,λ.ϕdwkin equation (5) is iteratively solved:(6)ϕdwk∝expEqlogθdk+Eqlogβkw,Eqlogθdk=Ψγdk−Ψ∑i=1Kγdi,Eqlogβkw=Ψλkw−Ψ∑i=1Nλki,where digamma functionΨ is the first-order derivative of function Γ. γdk and λkw are iteratively solved in the following way: γdk=α+∑wndwϕdwk,λkw=η+∑dndwϕdwk.Whentth vector of word frequency nt is observed, we keep λ unchanged and update the local optimal solution of γt and ϕt in E step. In M step, ϕt and λ from last iteration are both used to update λ:(7)λ=1−ρtλ+ρtλ˜,ρt=τ+t−k.λ˜ in formula (7) is solved as follows:(8)λ˜kw=η+MS∑sntskϕtskw,where nts is sth text in each batch text set, M is the number of the training text set, and Sis the size of each batch text set. Hyperparameters α and η are updated by the Newton–Rapson method: α⟵α−ρtα˜γtandη⟵η−ρtη˜λ. Here, α˜γt is the product of Hessian matrix and gradient ∇αl of the objective function lnd,ϕd,γd,λ. η˜λ is the product of Hessian matrix and gradient ∇ηL of the objective function Lw,ϕ,γ,λ.Based on online LDA, we proposed the CO-LDA theme model, which is similar with the classic SP-pLSA model in structure for enhancing the spatial correlation between different image regions. The framework of CO-LDA is illustrated in Figure3. The main difference between CO-LDA and SP-pLSA is that different online LDAs (LDA1, LDA2, and LDA3) are applied in different spatial levels to jointly mine the theme of an image. The main advantage of CO-LDA is that it integrates the spatial correlation of objects in different image resolutions, which further improves the holistic scene understanding. The visual histogram computation in online LDA is the same as the way of object response histogram in Section 2.2, and the theme feature of each spatial block is represented by variational parameter γ of the online LDA model.Figure 3
The framework of the CO-LDA theme model.Finally, the theme featureγ of the whole image is achieved by concatenating the theme features of different blocks of different spatial pyramid levels:(9)γ=w1⋅γL0⊕w2⋅γL1⊕w3⋅γL2,where γLi denotes the theme feature of the corresponding block in Lith pyramid level, ⊕ denotes the linear concatenation between feature vectors, and the weights of different spatial levels are configured as follows: w1=1/2,w2=1/2,andw3=1/4.


## 2.4. Multiview Active Learning

The MVAL referred in this paper is our previous work [21], which has two improvements in both hypothesis generation and selective sampling. First, boosting-like technique is integrated into MVAL, which uses a similar way of iterative weak classifier optimization, and the current hypothesis is boosted by weighted voting of all the hypotheses from the past queries. Furthermore, an adaptive hierarchical competition sampling is presented. In this sampling strategy, if the number of the contention samples is large, an unsupervised spectral clustering is activated to obtain the coarse spatial distribution of these contention samples in the high-dimensional feature space, and then, a multiview-based batch mode selective sampling is run based on two measures: sample uncertainty and redundancy by solving quadratic programming to determine the queried samples in each cluster.


### 2.4.1. Hypothesis Generation

If an active learning can select enough number of contention samples, which could improve the hypothesis in each query, the number of unlabeled samples, which are incorrectly classified, will decrease. It is quite similar with boosting technique in weak classifier optimization. The MVAL incorporates the AdaBoost algorithm into our framework to boost the generated hypothesis in each query, and the main flowchart is described in Figure4.Figure 4
The framework of hypothesis generation in our MAVL [21].In Figure4, a support vector machine (SVM) is used as a base classifier to construct a multiview classifier, which replaces the single-view classifier in AdaBoost, and this multiview classifier in each query can be considered as a weak classifier in each iteration in AdaBoost. The hypothesis of multiview classifier hix is computed by weighted voting of n SVM base classifiers v1,v2,…,vn whose weights are ω1,ω2,…,ωn. Unlike traditional query by boosting, we update the weight of each base classifier in each query and obtain the boosted hypothesis Hix by weighting all the hypotheses from the past queries and not from the current query only.The detailed process of the MAVL’s hypothesis generation based on AdaBoost is as follows:(a)
In iterationhtxj=∑f∈f1t,f2t,…fntωitfitxj, weighted voting is used to generate the initial multiview-based hypothesis:(10)htxj=∑f∈f1t,f2t,…,fntωitfitxj,wherefitxj is the classification confidence of sample xj by view i, and ωit denotes the contribution of view i for classification which is determined by the soft classification error rate εit, which defines how correctly a sample is classified:(11)εit=1∑x∈L,y=1fitx−∑x∈L,y=−1fitx,where∑x∈L,y=1fitx and ∑x∈L,y=−1fitx denote the sum of classification confidence of unlabeled samples, which are labeled as y=1 and y=−1, respectively. For a “positive/negative” sample, the distance of it to the decision boundary in the “positive/negative” side reflects the degree of how correctly it is classified, and this information is utilized to calculate the error degree εit here instead of the traditional classification error calculated by the decision hypothesis in AdaBoost. Also, ωit is updated through the following way: ωit=1/Z1tln1−εit/εit, where Z1t is the normalized weight. Then, the classification confidence δt of the multiview classifier can be computed by the following equation:(12)δt=∑j=1Nβjthtxj−yj.(b)
After iterationt, the size of the labeled sample set is increased as follows: Jt=Jt−1∪Lt. Jt denotes the labeled sample set in iteration t, and Lt denotes the newly added samples after query. As we know, the size of the labeled samples set Jt is increased during iteration in active learning. Thus, if the size of the initial labeled training set is small, the influence of Jt should be considered when updating the weight ηt of the multiview classifier, which is illustrated by the following equation:(13)ηt=1Z2tln1−δtδt+λJt.Then, the weight of each sample is updated through the following way:ωjt+1=ωjtβt1−et, where βt=δt/1−δt, if xj is correctly classified, ej=0, otherwise, ej=1.(c)
The final boosted hypothesisHtx of the queried sample xi is equivalent to the weighted sum of all the hypotheses from the past K queries, which is defined by(14)Htx=1,∑t=1Kηthtx≥12∑t=1Kηt,0,else.


### 2.4.2. Sampling Strategy

The MVAL uses a new hierarchical competition-based sampling strategy in order to query the contention samples with high probability in different sample distributions, which is illustrated in Figure5.Figure 5
The framework of sampling strategy in our MAVL [21].(1)Intercluster Sampling Competition. In the MVAL, a fast approximate spectral clustering algorithm is designed to reduce the computational complexity significantly to OKNT+OK3, where T is the iteration number of K mean clustering, and N is the total number of contention samples. The detailed process is illustrated as follows: (a) perform traditional K mean clustering on the contention unlabeled samples x1,x2,…,xN, compute the centroid of each cluster y1,y2,…,yK as K representative points, and build a correspondence table to associate each xi with the nearest cluster centroid yiyi; (b) run the normalized cut algorithm on y1,y2,…,yK to obtain a m-way cluster membership for each of yi; and (c) recover the cluster membership for each xi by looking up the cluster membership of the corresponding centroid yi in the corresponding table.After fast spectral clustering, two intercluster sampling measures are defined: the number of samples in the cluster and its information entropy. Both measures are weighted to obtain the number of selected samplesNCS in cluster C in the following equation:(15)NCS=NTZγNumC+1−γEntC,C=1,2,…,K,where NumC is proportional to the total number of samples NC in cluster C, and computing EntC is equivalent to kernel density estimation of x in cluster C. Weight γ=0.5 reflects the impact of both measures in intercluster sampling competition, Z is the normalized factor, NT is the total number of selected samples in the current query, and ⋅ is rounding operation.(2)Intracluster Sampling Competition. In the MVAL, an efficient quadratic programming based-method [22] is utilized, which dynamically estimates the weights of the redundancy and uncertainty of an unlabeled sample in each query. It is used for intracluster selective sampling and solved by minimizing the following object function:(16)minp∈Rn−lpTf˜v+12pTKu,ups.t.pTu=k,0≤p≤1.Equation (16) aims to estimate the normalized parameter pi∈0,1, which reflects how probable the unlabeled sample is selected. f^v=fvx1,…,fvxlT is the classification confidence of sample x in vth view. x1,…,xl are the queried unlabeled samples, u is a unit vector, and k=NCS is the number of unlabeled samples in batch mode. The first part denotes the sample uncertainty in vth view, and the sampling strategy tends to select the contention sample near the classification hyperplane of vth view by minimizing pTf˜v. The second part denotes the sample redundancy in vth view, and the similar samples are selected by minimizing pTKu,up. The sampling probability p is calculated by a convex quadratic programming, and finally, p1v,p2v,…,plv, which corresponds to x1,…,xl in vth view, is obtained. For selective sampling in each cluster, the conservative sampling strategy is utilized in a classic co-testing algorithm [23].


## 2.4.1. Hypothesis Generation

If an active learning can select enough number of contention samples, which could improve the hypothesis in each query, the number of unlabeled samples, which are incorrectly classified, will decrease. It is quite similar with boosting technique in weak classifier optimization. The MVAL incorporates the AdaBoost algorithm into our framework to boost the generated hypothesis in each query, and the main flowchart is described in Figure4.Figure 4
The framework of hypothesis generation in our MAVL [21].In Figure4, a support vector machine (SVM) is used as a base classifier to construct a multiview classifier, which replaces the single-view classifier in AdaBoost, and this multiview classifier in each query can be considered as a weak classifier in each iteration in AdaBoost. The hypothesis of multiview classifier hix is computed by weighted voting of n SVM base classifiers v1,v2,…,vn whose weights are ω1,ω2,…,ωn. Unlike traditional query by boosting, we update the weight of each base classifier in each query and obtain the boosted hypothesis Hix by weighting all the hypotheses from the past queries and not from the current query only.The detailed process of the MAVL’s hypothesis generation based on AdaBoost is as follows:(a)
In iterationhtxj=∑f∈f1t,f2t,…fntωitfitxj, weighted voting is used to generate the initial multiview-based hypothesis:(10)htxj=∑f∈f1t,f2t,…,fntωitfitxj,wherefitxj is the classification confidence of sample xj by view i, and ωit denotes the contribution of view i for classification which is determined by the soft classification error rate εit, which defines how correctly a sample is classified:(11)εit=1∑x∈L,y=1fitx−∑x∈L,y=−1fitx,where∑x∈L,y=1fitx and ∑x∈L,y=−1fitx denote the sum of classification confidence of unlabeled samples, which are labeled as y=1 and y=−1, respectively. For a “positive/negative” sample, the distance of it to the decision boundary in the “positive/negative” side reflects the degree of how correctly it is classified, and this information is utilized to calculate the error degree εit here instead of the traditional classification error calculated by the decision hypothesis in AdaBoost. Also, ωit is updated through the following way: ωit=1/Z1tln1−εit/εit, where Z1t is the normalized weight. Then, the classification confidence δt of the multiview classifier can be computed by the following equation:(12)δt=∑j=1Nβjthtxj−yj.(b)
After iterationt, the size of the labeled sample set is increased as follows: Jt=Jt−1∪Lt. Jt denotes the labeled sample set in iteration t, and Lt denotes the newly added samples after query. As we know, the size of the labeled samples set Jt is increased during iteration in active learning. Thus, if the size of the initial labeled training set is small, the influence of Jt should be considered when updating the weight ηt of the multiview classifier, which is illustrated by the following equation:(13)ηt=1Z2tln1−δtδt+λJt.Then, the weight of each sample is updated through the following way:ωjt+1=ωjtβt1−et, where βt=δt/1−δt, if xj is correctly classified, ej=0, otherwise, ej=1.(c)
The final boosted hypothesisHtx of the queried sample xi is equivalent to the weighted sum of all the hypotheses from the past K queries, which is defined by(14)Htx=1,∑t=1Kηthtx≥12∑t=1Kηt,0,else.


## 2.4.2. Sampling Strategy

The MVAL uses a new hierarchical competition-based sampling strategy in order to query the contention samples with high probability in different sample distributions, which is illustrated in Figure5.Figure 5
The framework of sampling strategy in our MAVL [21].(1)Intercluster Sampling Competition. In the MVAL, a fast approximate spectral clustering algorithm is designed to reduce the computational complexity significantly to OKNT+OK3, where T is the iteration number of K mean clustering, and N is the total number of contention samples. The detailed process is illustrated as follows: (a) perform traditional K mean clustering on the contention unlabeled samples x1,x2,…,xN, compute the centroid of each cluster y1,y2,…,yK as K representative points, and build a correspondence table to associate each xi with the nearest cluster centroid yiyi; (b) run the normalized cut algorithm on y1,y2,…,yK to obtain a m-way cluster membership for each of yi; and (c) recover the cluster membership for each xi by looking up the cluster membership of the corresponding centroid yi in the corresponding table.After fast spectral clustering, two intercluster sampling measures are defined: the number of samples in the cluster and its information entropy. Both measures are weighted to obtain the number of selected samplesNCS in cluster C in the following equation:(15)NCS=NTZγNumC+1−γEntC,C=1,2,…,K,where NumC is proportional to the total number of samples NC in cluster C, and computing EntC is equivalent to kernel density estimation of x in cluster C. Weight γ=0.5 reflects the impact of both measures in intercluster sampling competition, Z is the normalized factor, NT is the total number of selected samples in the current query, and ⋅ is rounding operation.(2)Intracluster Sampling Competition. In the MVAL, an efficient quadratic programming based-method [22] is utilized, which dynamically estimates the weights of the redundancy and uncertainty of an unlabeled sample in each query. It is used for intracluster selective sampling and solved by minimizing the following object function:(16)minp∈Rn−lpTf˜v+12pTKu,ups.t.pTu=k,0≤p≤1.Equation (16) aims to estimate the normalized parameter pi∈0,1, which reflects how probable the unlabeled sample is selected. f^v=fvx1,…,fvxlT is the classification confidence of sample x in vth view. x1,…,xl are the queried unlabeled samples, u is a unit vector, and k=NCS is the number of unlabeled samples in batch mode. The first part denotes the sample uncertainty in vth view, and the sampling strategy tends to select the contention sample near the classification hyperplane of vth view by minimizing pTf˜v. The second part denotes the sample redundancy in vth view, and the similar samples are selected by minimizing pTKu,up. The sampling probability p is calculated by a convex quadratic programming, and finally, p1v,p2v,…,plv, which corresponds to x1,…,xl in vth view, is obtained. For selective sampling in each cluster, the conservative sampling strategy is utilized in a classic co-testing algorithm [23].


## 3. Results and Discussion

In our experiment, two classic image sets (OT image set from MIT [9] and UIUC sports event image set from UIUC [24]) are used for algorithm comparison. Average classification precision (ACP) and mean of average classification precision (MACP) are both used for evaluating the performance of both CO-LDA models and multiview active learning algorithms.


### 3.1. Evaluation of Theme Semantic

The first experiment is designed for evaluating the performance of our proposed theme semantic. In OT and UIUC Sports datasets, the parameter configuration of the CO-LDA model is as follows: (1)kOT=0.5,τOT=256 and kUIUC=0.8,τUIUC=1024 in formula (7). (2) The batch sizes of sampled images in MVAL are SOT=SUIUC=512.We observe MACP variation of the CO-LDA model by changing the numbers of both theme and visual word:T=20,30,40,50 and W=200,500,800,1200,1500, and a total of twenty groups of (T, W) are obtained. In Figure 6, we find that (T, W) curves for both datasets show the similar trends that MASP increase first and then decrease. Thus, in our CO-LDA model, we set TOT=30,WOT=500 and TUIUC=40,WUIUC=1200.Figure 6
MACP curves with different groups of the theme and visual word in OT and UIUC Sports datasets.Figures7(a)-7(b) and 8(a)-8(b) show the probability distributions of different themes by CO-LDA in OT and UIUC Sports image datasets.Figure 7
The theme probability distribution in different scene classes (OT image dataset).
(a)(b)Figure 8
The theme probability distribution in different scene classes (UIUC image dataset).
(a)(b)In the OT image set, we can see that there are significant differences between four scene classes “Highway,” “Forest,” “Mountain,” and “Tall building” in theme probability distributions, and multiview SVM classifier works well in scene classification. In the UIUC Sports image set, the theme probability distributions are very similar in four scene classes “Bocce,” “Croquet,” “Polo,” and “Snowboarding,” which significantly increases the difficulty of scene classification.Furthermore, we compare the CO-LDA model with traditional LDA [9] and SP-pLSA [8] models, and the performance comparison of three theme models is shown in Table 1. N1 ∼ N8 denote the following eight natural scene classes: “Coast,” “Forest,” “Mountain,” “Open country,” “Highway,” “Inside city,” “Tall building,” and “Street.” S1 ∼ S8 denote the following eight event scene classes: ”Badminton,” “Bocce,” “Croquet,” “Polo,” “Rock Climbing,” “Rowing,” “Sailing,” and “Snowboarding.”Table 1
Performance comparison of different theme models.
ACPN1N2N3N4N5N6N7N8MACP(a) OT image datasetLDA73.24±1.1686.69±0.5574.83±0.7264.42±1.4768.13±1.8580.75±1.6082.10±0.5084.74±1.3176.56±1.15SP-pLSA82.40±0.5891.59±0.3487.74±1.7973.05±1.0479.66±0.9889.27±0.5989.68±1.4279.06±1.5583.80±1.04CO-LDA86.32±1.3994.22±1.5385.17±0.2173.70±0.3083.53±0.5287.39±1.2690.74±1.3882.47±0.4186.26±0.88ACPS1S2S3S4S5S6S7S8MACP(b) UIUC Sports image datasetLDA79.84±2.2863.86±0.9459.74±1.5365.49±0.5973.01±0.2961.65±0.9884.92±1.3858.22±0.0368.30±1.00SP-pLSA75.25±1.4469.14±0.5167.19±1.1162.70±1.7386.32±1.5072.54±0.1380.75±0.7675.34±1.0273.64±1.13CO-LDA77.62±0.1667.48±0.7468.73±0.8671.48±2.1382.75±1.0477.81±0.0783.43±0.4272.89±0.5575.27±0.75Note. Bold values represent the best performance of the algorithms corresponding to each class.In the LDA model, each image is divided into11×11 blocks, and 5 pixels are overlapped between neighbored blocks. For feature representation, gray-scale SIFT descriptors are sparsely sampled, and means of three color channels are calculated. The numbers of the theme and visual word are TOT=30,WOT=200 and TOT=50,WOT=800 by cross validation. In the SP-pLSA model, the ways of image division and feature representation are the same as the LDA model. The numbers of the theme and visual word are TOT=25,WOT=1200 and TOT=50,WOT=1500 by cross validation.In the OT image set, CO-LDA achieves both higher ACP and MACP than SP-pLSA in six scene classes except “Mountain” and “Inside city.” LDA performs the worst in all of scene classes except “Street.” It is easy to conclude that CO-LDA can achieve more accurate scene semantics than other two classic methods. In the UIUC Sports image set, CO-LDA achieves the highest ACP in the following three event classes: “Croquet,” “Polo,” and “Rowing,” and SP-pLSA achieves the highest ACP in the following three event classes: “Bocce,” “Rock Climbing,” and “Snowboarding.” But in the event classes “Badminton” and “Sailing,” in which LDA has the highest ACP, CO-LDA still performs better than SP-pSLA. Thus, we can conclude that our proposed CO-LDA also have slightly better performance in theme mining than the two classic image representation methods.


### 3.2. Evaluation of MVAL

In the second experiment, we compare our algorithm with other single-view active learning algorithm with both high-level semantics and low-level features for scene classification. In our initial labeled training set, label size = 150, batch size = 20, and iteration = 10.Our proposed algorithm MVALHS (MVAL and HS denote MVAL [21] and two proposed high-level semantics, respectively) is compared with the following four algorithms: (1) MVALLS (LS denotes low-level image features): in MVAL, both means of three color channels and densely sampled color-SIFT descriptors are concatenated as a feature vector for image representation. (2) ALQP [22]: a single-view SVM active learning by QP-based selective sampling, which relies on the sample uncertainty and redundancy. (3) DiffWS [25–30]: a disagreement-based active learning from weak and strong labelers. (4) GraphGP [23]: a graphical model-based active learning with robust Gaussian process. The feature representations of ALQP, DiffWS, and GraphGP are the same as MVALLS. The performance comparison of the five active learning algorithms is shown in Table 2.Table 2
Performance comparison of different active learning algorithms.
DSAlgorithmN1N2N3N4N5N6N7N8OT image datasetALQP70.52±1.8085.65±1.5074.94±1.3353.52±2.1477.34±07480.04±01779.17±1.3475.00±0.85DiffWS72.43±1.6784.19±1.5976.53±1.7355.14±0.7472.88±0.9878.58±1.1281.40±0.8874.30±1.00GraphGP71.52±1.4486.32±0.9974.30±0.1552.05±1.5475.85±1.1779.53±2.0378.76±1.8976.75±0.44MVALLS73.23±0.5585.57±1.7677.53±1.5953.85±0.9577.34±1.2281.40±2.0081.87±1.3677.52±0.53MVALHS77.25±0.8890.41±1.0180.37±1.5560.49±0.2982.14±0.7287.38±1.1385.91±0.1481.73±0.66DSAlgorithmS1S2S3S4S5S6S7S8UIUC-Sports image datasetALQP68.54±1.4149.42±0.3245.90±1.4055.17±0.6267.19±0.8860.20±1.5275.19±1.0751.09±1.21DiffWS72.22±1.5050.98±1.2248.79±0.5957.33±1.3167.00±0.5557.53±0.7871.01±0.1654.58±1.50GraphGP69.18±1.3754.10±0.7647.52±2.1353.75±0.8969.04±0.3462.55±1.0474.10±1.0055.22±0.28MVALLS71.18±1.3953.88±0.5549.62±0.5457.93±1.6571.05±1.7562.98±1.7075.59±1.0457.18±1.24MVALHS76.75±1.3955.44±1.6053.96±1.2359.78±0.7770.66±0.5266.40±1.0878.05±1.5160.68±0.41Note. Bold values represent the best performance of the algorithms corresponding to each class.From Table2, it is easily found that our algorithm MVALHS has the highest MACP in almost all scene classes than the other four algorithms in both image sets, which demonstrates that high-level semantics can achieve more significant improvement in holistic scene understanding than traditional low-level image features. Furthermore, we can see that MVALLS performs better in most cases than other three single-view algorithms, which also means that multiple view setting can successfully result in larger decrease in the size of the version space than traditional single-view active learnings due to its independent and diverse views.


## 3.1. Evaluation of Theme Semantic

The first experiment is designed for evaluating the performance of our proposed theme semantic. In OT and UIUC Sports datasets, the parameter configuration of the CO-LDA model is as follows: (1)kOT=0.5,τOT=256 and kUIUC=0.8,τUIUC=1024 in formula (7). (2) The batch sizes of sampled images in MVAL are SOT=SUIUC=512.We observe MACP variation of the CO-LDA model by changing the numbers of both theme and visual word:T=20,30,40,50 and W=200,500,800,1200,1500, and a total of twenty groups of (T, W) are obtained. In Figure 6, we find that (T, W) curves for both datasets show the similar trends that MASP increase first and then decrease. Thus, in our CO-LDA model, we set TOT=30,WOT=500 and TUIUC=40,WUIUC=1200.Figure 6
MACP curves with different groups of the theme and visual word in OT and UIUC Sports datasets.Figures7(a)-7(b) and 8(a)-8(b) show the probability distributions of different themes by CO-LDA in OT and UIUC Sports image datasets.Figure 7
The theme probability distribution in different scene classes (OT image dataset).
(a)(b)Figure 8
The theme probability distribution in different scene classes (UIUC image dataset).
(a)(b)In the OT image set, we can see that there are significant differences between four scene classes “Highway,” “Forest,” “Mountain,” and “Tall building” in theme probability distributions, and multiview SVM classifier works well in scene classification. In the UIUC Sports image set, the theme probability distributions are very similar in four scene classes “Bocce,” “Croquet,” “Polo,” and “Snowboarding,” which significantly increases the difficulty of scene classification.Furthermore, we compare the CO-LDA model with traditional LDA [9] and SP-pLSA [8] models, and the performance comparison of three theme models is shown in Table 1. N1 ∼ N8 denote the following eight natural scene classes: “Coast,” “Forest,” “Mountain,” “Open country,” “Highway,” “Inside city,” “Tall building,” and “Street.” S1 ∼ S8 denote the following eight event scene classes: ”Badminton,” “Bocce,” “Croquet,” “Polo,” “Rock Climbing,” “Rowing,” “Sailing,” and “Snowboarding.”Table 1
Performance comparison of different theme models.
ACPN1N2N3N4N5N6N7N8MACP(a) OT image datasetLDA73.24±1.1686.69±0.5574.83±0.7264.42±1.4768.13±1.8580.75±1.6082.10±0.5084.74±1.3176.56±1.15SP-pLSA82.40±0.5891.59±0.3487.74±1.7973.05±1.0479.66±0.9889.27±0.5989.68±1.4279.06±1.5583.80±1.04CO-LDA86.32±1.3994.22±1.5385.17±0.2173.70±0.3083.53±0.5287.39±1.2690.74±1.3882.47±0.4186.26±0.88ACPS1S2S3S4S5S6S7S8MACP(b) UIUC Sports image datasetLDA79.84±2.2863.86±0.9459.74±1.5365.49±0.5973.01±0.2961.65±0.9884.92±1.3858.22±0.0368.30±1.00SP-pLSA75.25±1.4469.14±0.5167.19±1.1162.70±1.7386.32±1.5072.54±0.1380.75±0.7675.34±1.0273.64±1.13CO-LDA77.62±0.1667.48±0.7468.73±0.8671.48±2.1382.75±1.0477.81±0.0783.43±0.4272.89±0.5575.27±0.75Note. Bold values represent the best performance of the algorithms corresponding to each class.In the LDA model, each image is divided into11×11 blocks, and 5 pixels are overlapped between neighbored blocks. For feature representation, gray-scale SIFT descriptors are sparsely sampled, and means of three color channels are calculated. The numbers of the theme and visual word are TOT=30,WOT=200 and TOT=50,WOT=800 by cross validation. In the SP-pLSA model, the ways of image division and feature representation are the same as the LDA model. The numbers of the theme and visual word are TOT=25,WOT=1200 and TOT=50,WOT=1500 by cross validation.In the OT image set, CO-LDA achieves both higher ACP and MACP than SP-pLSA in six scene classes except “Mountain” and “Inside city.” LDA performs the worst in all of scene classes except “Street.” It is easy to conclude that CO-LDA can achieve more accurate scene semantics than other two classic methods. In the UIUC Sports image set, CO-LDA achieves the highest ACP in the following three event classes: “Croquet,” “Polo,” and “Rowing,” and SP-pLSA achieves the highest ACP in the following three event classes: “Bocce,” “Rock Climbing,” and “Snowboarding.” But in the event classes “Badminton” and “Sailing,” in which LDA has the highest ACP, CO-LDA still performs better than SP-pSLA. Thus, we can conclude that our proposed CO-LDA also have slightly better performance in theme mining than the two classic image representation methods.


## 3.2. Evaluation of MVAL

In the second experiment, we compare our algorithm with other single-view active learning algorithm with both high-level semantics and low-level features for scene classification. In our initial labeled training set, label size = 150, batch size = 20, and iteration = 10.Our proposed algorithm MVALHS (MVAL and HS denote MVAL [21] and two proposed high-level semantics, respectively) is compared with the following four algorithms: (1) MVALLS (LS denotes low-level image features): in MVAL, both means of three color channels and densely sampled color-SIFT descriptors are concatenated as a feature vector for image representation. (2) ALQP [22]: a single-view SVM active learning by QP-based selective sampling, which relies on the sample uncertainty and redundancy. (3) DiffWS [25–30]: a disagreement-based active learning from weak and strong labelers. (4) GraphGP [23]: a graphical model-based active learning with robust Gaussian process. The feature representations of ALQP, DiffWS, and GraphGP are the same as MVALLS. The performance comparison of the five active learning algorithms is shown in Table 2.Table 2
Performance comparison of different active learning algorithms.
DSAlgorithmN1N2N3N4N5N6N7N8OT image datasetALQP70.52±1.8085.65±1.5074.94±1.3353.52±2.1477.34±07480.04±01779.17±1.3475.00±0.85DiffWS72.43±1.6784.19±1.5976.53±1.7355.14±0.7472.88±0.9878.58±1.1281.40±0.8874.30±1.00GraphGP71.52±1.4486.32±0.9974.30±0.1552.05±1.5475.85±1.1779.53±2.0378.76±1.8976.75±0.44MVALLS73.23±0.5585.57±1.7677.53±1.5953.85±0.9577.34±1.2281.40±2.0081.87±1.3677.52±0.53MVALHS77.25±0.8890.41±1.0180.37±1.5560.49±0.2982.14±0.7287.38±1.1385.91±0.1481.73±0.66DSAlgorithmS1S2S3S4S5S6S7S8UIUC-Sports image datasetALQP68.54±1.4149.42±0.3245.90±1.4055.17±0.6267.19±0.8860.20±1.5275.19±1.0751.09±1.21DiffWS72.22±1.5050.98±1.2248.79±0.5957.33±1.3167.00±0.5557.53±0.7871.01±0.1654.58±1.50GraphGP69.18±1.3754.10±0.7647.52±2.1353.75±0.8969.04±0.3462.55±1.0474.10±1.0055.22±0.28MVALLS71.18±1.3953.88±0.5549.62±0.5457.93±1.6571.05±1.7562.98±1.7075.59±1.0457.18±1.24MVALHS76.75±1.3955.44±1.6053.96±1.2359.78±0.7770.66±0.5266.40±1.0878.05±1.5160.68±0.41Note. Bold values represent the best performance of the algorithms corresponding to each class.From Table2, it is easily found that our algorithm MVALHS has the highest MACP in almost all scene classes than the other four algorithms in both image sets, which demonstrates that high-level semantics can achieve more significant improvement in holistic scene understanding than traditional low-level image features. Furthermore, we can see that MVALLS performs better in most cases than other three single-view algorithms, which also means that multiple view setting can successfully result in larger decrease in the size of the version space than traditional single-view active learnings due to its independent and diverse views.


## 4. Conclusion

This paper proposed a MVAL-based scene classification algorithm, which applies two different high-level image semantics to generate the corresponding hypotheses. Different object detectors are first trained to achieve the responses of different object classes as object semantic. Furthermore, a CO-LDA model is proposed for achieving more accurate theme semantic by integrating the spatial correlation of objects in different image resolutions, which improves the holistic scene understanding. With the help of both two independent views, our MVAL algorithm has potential to not only handle large-scale data training but also improve the performance of scene classification.



---

*Source: 3878153-2020-09-01.xml*