Feature encoding with hybrid heterogeneous structure model for image classification

IET Image ProcessingVolume 14, Issue 10 p. 2166-2174 Research ArticleFree Access Feature encoding with hybrid heterogeneous structure model for image classification Zhihang Ji, Zhihang Ji School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian, 116024 People's Republic of China School of Information Engineering, Henan University of Science and Technology, No.263 Kaiyuan Avenue, Luoyang, 471023 People's Republic of ChinaSearch for more papers by this authorYan Yang, Yan Yang School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian, 116024 People's Republic of China Liaoning Normal University, No.850 Huanghe Road, Dalian, 116029 People's Republic of ChinaSearch for more papers by this authorFan Wang, Fan Wang School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian, 116024 People's Republic of ChinaSearch for more papers by this authorLijuan Xu, Lijuan Xu School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian, 116024 People's Republic of ChinaSearch for more papers by this authorXiaopeng Hu, Corresponding Author Xiaopeng Hu xphu@dlut.edu.cn School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian, 116024 People's Republic of ChinaSearch for more papers by this author Zhihang Ji, Zhihang Ji School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian, 116024 People's Republic of China School of Information Engineering, Henan University of Science and Technology, No.263 Kaiyuan Avenue, Luoyang, 471023 People's Republic of ChinaSearch for more papers by this authorYan Yang, Yan Yang School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian, 116024 People's Republic of China Liaoning Normal University, No.850 Huanghe Road, Dalian, 116029 People's Republic of ChinaSearch for more papers by this authorFan Wang, Fan Wang School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian, 116024 People's Republic of ChinaSearch for more papers by this authorLijuan Xu, Lijuan Xu School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian, 116024 People's Republic of ChinaSearch for more papers by this authorXiaopeng Hu, Corresponding Author Xiaopeng Hu xphu@dlut.edu.cn School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian, 116024 People's Republic of ChinaSearch for more papers by this author First published: 08 July 2020 https://doi.org/10.1049/iet-ipr.2019.0719Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In the standard bag-of-visual-words model, the relationship between visual words and geometric structure information embedding in Voronoi cells is important for expressing the topology of the feature space. However, this information is usually ignored by recent works. To overcome it, the authors proposed a hybrid heterogeneous structure model (HHSM), where local hyperspheres and local structure subspaces are applied to simulate the intrinsic structure of the feature space. Firstly, the local hypersphere is formed by choosing some links between parts of visual words, with the use of a proposed decision strategy derived from k-dense neighbour algorithm. In order to capture the geometric structure information around the visual word, they then construct the local structure subspace with the transformed PCA principal vectors of the visual features within a Voronoi cell. Finally, this study introduces a novel feature encoding method based on the HHSM. Experiments are conducted on 15-Scenes, Pascal VOC2007, Caltech101, Caltech256 and MIT Indoor 67 datasets, which include 4485, 9963, 9146, 30607 and 15620 images, respectively. The results demonstrate the effectiveness of the proposed method in improving the accuracy of the classification. In addition, the proposed method achieves comparable performance when combined with CNN local features. 1 Introduction Image classification, which aims to categorise a set of unlabelled images into several predefined classes according to their visual content, is a challenging topic in computer vision. During the past few years, researchers have devoted their effort to design a discriminative representation for images, which is a crucial step in image classification. Thus, different models have been proposed, such as BoVW model (bag-of-visual-words) [1, 2] and deep learning [3–5]. Recently, deep convolutional neural network (DCNN) has attracted a lot of attention due to its excellent performance in image classification [6–8]. Many researches [9–11] have shown that a DCNN model trained for image classification on a large dataset, such as ImageNet [12], can be used as a feature extractor and employed in many other recognition tasks. Motivated by these researches, a lot of novel methods based on CNN local features are proposed and some of them [11, 13, 14] make use of the BoVW representation. Before the emergency of deep learning, BoVW model and its variations play an important role in image classification. In these models, an image is decomposed into lots of local features. Then a codebook is learned to encode those features. Finally, the encoded results are aggregated together to form an image representation which can be used to various applications. In the BoVW framework, codebook generation and corresponding encoding strategy are two important operations. For a well-designed codebook, it should be as far as possible to approximate the manifold structure of the visual feature space. Usually, k-means method is adopted to build a codebook, which partitions the feature space into several of Voronoi cells, where each cell is a cluster and the cluster centre is the centroid of the cell. Then each cluster centre is taken as a visual word. However, the codebook build by this way bears the following two drawbacks: (i) pay little attention to the relationship between the visual words, which is important for describing the connection between the local topology areas of the feature space; (ii) the geometric structure information within each Voronoi cell is ignored, which is necessary for describing the detailed information inside a local Voronoi cell [15], as shown in Fig. 1a. Fig. 1Open in figure viewerPowerPoint Illustration of four structure models in feature space (a) Illustration of the clustering result by k-means. Generally, in BoVW model, only the centroids of cluster (shown by the stars) are chosen as the visual words without considering the relationship (indicated by the link) among them. In addition, it ignores the geometric structure information (indicated by the stars with arrows around) within each Voronoi cell, (b) Hypersphere is illustrated, which is formed by the links between a central visual word with its KNN counterparts. The hypersphere can be considered as one visual word in a high-order codebook, (c) Enhanced local hypersphere is shown, which is formed by the links between a central visual word with its KNN counterparts and ignores the links failing to satisfy the constrains of the intersect angle, (d) HHSM proposed by us is shown, which is formed by several enhanced local hypersphere (shown by the links between the visual words) and the geometric structure within each Voronoi cell (indicated by the arrows with arrows around) For the first drawback, some studies [16–18] employ local patch annotation or class label to conduct a high-level codebook by different grouping strategy, where each word is a visual phrase formed by some pre-trained visual words. However, these methods are computationally expensive due to a large number of candidate combinations. Compared with the supervised methods, some researches [19–21] explore the topology relationship of visual words in unsupervised way. In [19, 20], the authors construct a graph codebook and encode each local feature using the linked pair-wise visual words. In [22], the authors take the links, between each visual word and its KNN counterpart, to form a hypersphere, which can be considered as a high-order visual word, as shown in Fig. 1b. Recently, the studies [19, 21, 23–25] point out not only the relative distance but also the orientation of the link should be considered. In another word, the intersection angle between the links should be larger than a threshold, as shown in Fig. 1c. The authors think this constrain can enhance the discrimination of a codebook further. In order to address the second shortcoming, Hideki Nakayama [26] build a codebook via k-means clustering first and then fit each Voronoi cell with a Gaussian independently. In the encoding procedure, descriptors of an image are encoded by the Fisher vector with respect to the nearest neighbouring Gaussian. In this way, the high-order information around visual word can be exploited. Recently, Li et al. [27] propose a feature coding method called locality-constrained affine subspace coding (LASC). It employs the k-means to build a codebook first, and then characterises each Voronoi cell as a multiple-order subspace, which provides a fine approximation of the Voronoi cell. Finally, the descriptors of the images are mapped to the subspaces and weighted to form the representation vector. Although, the two approaches utilise the geometric structure of the Voronoi cell and achieve good result, they suffer from the large memory consumption. According to these observations, we put forward a new representation scheme which is based on a hybrid heterogeneous structure model (HHSM) embedding in the feature space, as shown in Fig. 1d. This model is composed of two sub-structures: the local hyperspheres formed by some neighbour visual words and the local structure subspaces embedding in the Voronoi cells. For the first issue, motivated by the work [19, 22], an enhanced local hypersphere, meeting the constraints for the orientation of the links, is constructed for each original visual word. To this end, a building method based on a combinatorial optimisation strategy is proposed which chooses optimal links as the base vectors of the local hypersphere. This combinatorial optimisation problem can be solved by the k-dense neighbour (KDN) algorithm [28]. After that, taking these hyperspheres as a high-order codebook, an efficient image encoding method is proposed, which is named as enhanced local hypersphere coding (ELHC). Meanwhile, for the second problem, we adopt the LASC method in [27] but only consider the first-order statistical information of the affine subspace (first-order LASC). By this way, we can describe the geometric structure inside the Voronoi cell with only half storage than the original one. Lastly, the final representation is formed by fusing the two results from each sub-model. Fig. 2 shows the framework of the method. Fig. 2Open in figure viewerPowerPoint Overview of the proposed framework Compared with the previous approaches, the main contribution of our work can be summarised as follows: (i) Based on the KDN algorithm [28], a novel approach for building the enhanced hypersphere is proposed. In our method, the constructing process can be modelled as a combinatorial optimisation problem. By solving it, we can choose the optimal link combination to form the hypersphere. Then, a high-order codebook is constructed with the local hyperspheres. (ii) Based on these local hyperspheres, a novel encoding method, named ELHC, is proposed. In the encoding process, each hypersphere is considered as a local coordinate system, and the image features are mapped and weighted to form the representation vector. Specifically, the objective function of the encoding step can be formulated as Ridge regression and a closed-form result can be achieved efficiently. (iii) Fusing the result of ELHC and first-order LASC by multiple kernel learning (MKL) technology [29], an image representation is achieved. It takes advantage of the HHSM, and characterises the feature distribution of a single image from multiple viewpoints. Like the traditional pipeline of BoVW model, feature descriptors are extracted first. Then a codebook based on the proposed HHSM is constructed. After two corresponding encoding representations achieved, MKL technology is adopted to accomplish the classification task. The rest of the paper is organised as follows. Section 2 briefly reviews the related work of the codebook generation and image encoding. Section 3 describes the image representation method proposed by us in detail and Section 4 gives a comprehensive performance evaluation of our method in five benchmark datasets. Finally, the conclusions are summarised in Section 5. 2 Related work In this section, the related works are reviewed briefly. First, we introduce the studies about codebook construction. Then, the classical feature encoding methods are involved. 2.1 Research on building representative codebook Usually, k-means algorithm is employed to build the codebook. However, some works [30–32] show that this kind of codebook is not enough powerful. Since only considering the appearance similarity, the visual words are not as discriminative as they are expected to be. In order to overcome this shortcoming, some attempts have been made to fuse label information. In [33], the authors use the mutual information between the features and class labels to create the compact codebook. Brian Fulkerson et al. [34] adopt the agglomerative information bottleneck method to compress the original codebook by merging visual word. With the similar strategy, Kim and Kweon [35] refine the large codebook but with the entropy-based minimum description length criterion. In addition, some authors [36–39] propose to construct category-specific codebook which clusters the features from images only in a single class. Although these methods improve the classification performance, they need lots of label information which are time consuming and not suitable for large-scale datasets. Recently, the local manifold structure of a feature space is often exploited to improve codebook representative ability. Tian and Lu [32] introduce the subspace learning method to construct codebook and a contextual local descriptor subspace is found for embedding the discriminative information. Liu et al. [40] make use of diffusion maps to construct codebook which can capture the local geometric relations between visual words. Although these methods achieve encouraging result, they suffer from a high computational cost. In addition, the studies [24, 25] point out not only the relative distance between features but also the orientation play an important role in feature space analysis. In [24], the authors propose a criterion to select useful features for compressing neural networks, in which the orientation information between features is an important selection basis. In [25], the authors design a self-weighted multiview clustering method to integrate the points’ correlations from both the orientation and context views. The method proposed in [23] shares the similar intentions with us but constructs the hypersphere in a different way (we name this method HOTM for simplicity). In order to confirm a proper intersect angle between the links, HOTM test different values within a predefined range in the exhaustive way and then select the optimal one. This approach is time consuming and susceptible to the predefined selection range. In contrast to HOTM, our method choses the links of the hypersphere by solving a combinatorial optimisation problem. In addition, we design a HHSM. This model captures more structure information about the feature space. 2.2 Classical image encoding methods The BoVW model is widely used for image representation and has been proved to perform well in many visual recognition applications. For this model, feature encoding plays an important role and lots of encoding methods have been developed. The simplest encoding method is hard coding, which is a kind of classical vector quantisation (VQ) method [1]. This method exclusively quantises each local feature to only one visual word according to the nearest neighbour criteria. Consequently, such a hard assignment would produce large quantisation error and lead to the loss of discriminative information. The soft coding [41, 42] is a more precise VQ method, which assigns each local feature to multiple visual words and uses the Gaussian weights to describe the quantisation responses. In addition, some extensive methods have been developed. Instead of adopting the traditional VQ approach, Yang et al. [43] proposes ScSPM method which encodes the input feature by selecting the less basis while not sacrificing the quantisation accuracy, accompanying with spatial-pyramid trick. Further, fusion with the locality and sparsity, Wang et al. [44] proposes the locality-constrained linear coding method (LLC), in which the locality rather than the global sparsity is emphasised. Different with BoVW scheme, an alternative method is proposed to aggregate the local descriptors. In [45], the visual codebook is modelled by a Gaussian mixture model (GMM) then the visual features are encoded according to the Fisher kernel principle, namely, computing the gradient of the log-likelihood with respect to some parameters. It takes advantage of a generative model of data and discriminative properties at the same time. Later, Jegou et al. [46] proposed a simplified version of FV called VLAD. In practice, it accumulates the residual between the features and the visual words of the codebook. Although these encoding methods achieve state-of-the-art performance on many image classification benchmark datasets in specific period, most of them treat the visual words individually and disregard the high-level information around the visual words. The approach proposed by us encodes each image basing on the two sub-models of the HHSM separately and forms the image representation by fusing the encoding results. The final representation introduces high-order information and keeps a strong discriminative ability. 3 Image representation based on hybrid heterogeneous structure model In this section, the proposed representation model will be introduced elaborately. First, we provide an overall description of the model. For each module unit included in the model, the corresponding algorithm and constructing scheme is given, respectively. In Section 3.2, the first-order LASC is presented briefly. After that the constructing process of the local hyperspheres structure will be described in detail. Finally, the corresponding encoding method and related data fusion method will be presented. 3.1 Overall description of the proposed image representation model In this section, an overall description of the proposed model is provided, which presents the generating procedure of an image representation, as shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint Diagram of the procedure that an image presentation is generated by utilising the proposed HHSM (a) Flowchart of building HHSM, (b) Flowchart of constructing image representation based on the HHSM Here, two key module units of this model are introduced: (i) constructing the HHSM, (ii) encoding and representing an image based on HHSM. Figs. 3a and b illustrate specific steps, respectively, which are listed as follows: (i) Constructing the HHSM (a) Constructing basic codebook: Given an image dataset, following the steps of the traditional BoVW model, a basic codebook is constructed by K-means clustering. (b) Constructing affine subspace sub-model: With the basic codebook, the affine subspace sub-model is built by the PCA method proposed in [27]. (c) Constructing enhanced local hypersphere sub-model: Based on the basic codebook, the enhanced local hypersphere sub-model is built by the Algorithm 1 introduced in Section 3.3.1. (d) Constructing the HHSM: Combining the two obtained sub-models to constitute the final HHSM. (ii) Generating image representation (a) Feature extraction: Given an image, DenseSIFTs or CNN local features are extracted. Specially, when extracting CNN local features, we adopt the same mean as in [47]. For each local patch, we take its corresponding activation, extracting from the last full-connected layer of a pre-trained AlexNet model [3], as the CNN local feature. The extracted CNN local feature plays a similar role as DenseSIFT to provide local region description. (b) Generating the first-order LASC representation: Encoding image by using the first-order LASC method introduced in Section 3.2 to generate first-order LASC representation. (c) Generating the ELHC representation: Encoding image by using the ELHC method proposed in Section 3.3.2 to generate ELHC representation. (d) Generating the final HHSM representation: Combining the obtained first-order LASC and ELHC representation to generate the final HHSM representation by using the MKL technology or direct-concatenation way. 3.2 First-order LASC image representation Different with the traditional BoVW representation, which only chooses the cluster centroids as the visual words and ignores the geometric structure information inside the Voronoi cell, Li et al. [27] propose a novel encoding method called LASC. In this method, the dictionary is an ensemble of low-dimensional linear subspace attached to each cluster centroid, which provides a piecewise linear approximation of the manifold structure within the Voronoi cells, as shown in Fig. 1a. For each Voronoi cell, the authors assume there exists a low-dimensional subspace attached to each cluster centroid, which can be formulated as (1)where indicates the centroid of cluster and is an n × p matrix whose columns form a basis of a linear subspace. Indeed, defines a local coordinate system and all of them approximately depict a holistic topology of the manifold. More specifically, suppose that is the mean vectorand is its covariance matrix obtained from the features inside each cell. Then, the PCA is adopted to approximate the linear subspace and the most significant principal directions is chosen as the basis, i.e. (2)where the is the principal directions of the ith Voronoi cell. In the encoding step, given a feature vector , it can be represented by its top-R nearest subspace. Thus, the objective function of first-order LASC can be formulated as (3)where is a regularisation parameter, is the R closet subspaces of y and indicates the distance between y and subspace . This function can be decoupled into independent Ridge regression problems about , which has closed-form solution (4)when , otherwise is equal to zero. Thus, the first-order LASC vector of the feature y can be written as (5)Further, average pooling is applied to form the final representation of an image. For more details, please refer to the work [27]. 3.3 Image representation based on enhanced local hyperspheres structure 3.3.1 Constructing the enhanced local hyperspheres codebook In this paper, inspired by the method in [22], a novel high-order codebook is constructed, which is formed by several of enhanced local hyperspheres. Let be a pre-trained codebook with N visual words. In our hypersphere model, for each word , we assume that there exists a corresponding local hypersphere, denoted as , whose centre locates at . So, the proposed hypersphere codebook can be defined as . Fig. 1c gives an example to illustrate the geometric structure of a hypersphere. As shown in the figure, the hypersphere is composed of several of ‘valid vectors’, which start from the centre to the corresponding ‘valid words’. Definition 1.Let be a hypersphere located at . Assuming and are two arbitrary visual words, we call them as valid words for hypersphere if and only if they satisfy the following locality and intersection angle constraints: (a) , i.e. and are in the KNN visual words of ; (b) , i.e. the intersection angle between the vector and should larger than certain threshold. In another words, the vectors and should keep enough difference. The vectors that start from the visual word to the corresponding ‘valid words’ are called valid vectors. Next, we introduce some other notations and definitions that will be used throughout this paper. First, let the set denote the KNN visual words of and represent the ‘valid words’ of the hypersphere . According to the properties of the valid words, we know that all the valid words should be the subset of the , i.e. . Correspondingly, we denote the as the candidate vector and the as the ‘valid vector’ set of the hypersphere . In addition, the notation represents the nearest visual word of and denotes the nearest vector that start from to its nearest visual word . Following the analysis above, we can know that how to determine the ‘valid vector’ set of each enhanced local hypersphere is the key to the problem. Taking the hypersphere as an example, in our methods, the determining process includes the following two steps: First, for the locality constraint, candidate vector set should be specified. Then, select the ‘valid vector’ set from . The details can be described as follows. Given the visual word and the corresponding candidate vector set , select a subset with M elements from so that the accumulation of intersection angle of the vectors in the reaches its maximum, i.e. (6)Here the function measures the intersection angle of the vectors in the . If one subset satisfies the demand, it is just the ‘valid vector’ set that we are seeking, i.e. . Let (7)and we define a affinity matrix , where (8)Meanwhile, we assume the nearest vector must belong to the . As a matter of convenience, we assign . Suppose that y is a K-dimension indicator vector and satisfies (9)Thus, for the candidate vector set , the should satisfy (10)So, the objective function for finding the subset can be represented as (11)The constraint means the solution y indicates M objects, and implies the solution must contain the vector . The objective function (11) is a question about KDN problem, which can be solved by the method in [28]. Once the y is obtained, the index of non-zero elements can be used to choose the ‘valid vector’ from . Then the local hypersphere can be determined. Doing the same operation on each visual words , we can gain an enhanced-local-hypersphere codebook . Algorithm 1 shows the building procedure in detail(Fig. 4). Fig. 4Open in figure viewerPowerPoint Algorithm 1: building the enhanced local hypersphere 3.3.2 Enhanced local hypersphere coding Given the pre-trained codebook and the corresponding high-order codebook . For any descriptor y in an image, it can be represented by its top-R nearest hyperspheres. Thus, the encoding procedure can be formulated as (12)where is a regularisation parameter, is the R nearest hypersphere of y and indicates the distance between y and hypersphere . This function can be decoupled into independent Ridge regression problems about , which has closed-form solution (13)when , otherwise is equal to zero. Thus, the ELHC vector of the feature y can be written as (14)Last, average pooling is applied to form the final representation of an image. 3.4 Distance of feature to sub-model In the first-order LASC and ELHC, we need to define the distance between the feature descriptors with the corresponding sub-model. Here, we choose the measure metric as (15)where is the ith visual words. 3.5 Fusing the two sub-representations In this paper, we use MKL technology to integrate the two representation vector. Based on this technology, we can train a SVM with a combined kernel, which can be employed to fuse the different kinds of features with adaptive weight. The combined kernel can be expressed as follows: (16)where is the weight to combine sub-kernel . Here, we need to choose the types of kernels for representation, depending on their discriminative power and computational cost. The weight can be estimated automatically by MKL technology from the training data. By this way, we can fuse the ELHC vector and first-order LASC vector to obtain the optimal classifier. 4 Experiments and results In order to demonstrate the effectiveness of our approach, we carry out evaluation experiment on five typical datasets with different scale. First, we introduce the experiment setting and the datasets used. Then, some important factors that might impact the classification accuracy are assessed. Finally, we compare our method with several state-of-the-art methods to demonstrate the effectiveness. 4.1 Experimental setting In the experiment, we adopt dense SIFT local features and CNN local feature, respectively, to validate our method. All the algorithms are implemented by the VLFeat [48] and Matconvnet toolbo