Vehicle classification approach based on the combined texture and shape features with a compressive DL

IET Intelligent Transport SystemsVolume 13, Issue 7 p. 1069-1077 Research ArticleFree Access Vehicle classification approach based on the combined texture and shape features with a compressive DL Wei Sun, Corresponding Author Wei Sun sunw0125@163.com orcid.org/0000-0002-4870-2971 School of Automation, Nanjing University of Information Science & Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorXiaorui Zhang, Xiaorui Zhang School of Computer and Software, Nanjing University of Information Science & Technology, Nanjing, People's Republic of China Jiangsu Collaborative Innovation Centre on Atmospheric Environment and Equipment Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorShunshun Shi, Shunshun Shi School of Automation, Nanjing University of Information Science & Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorXiaozheng He, Xiaozheng He School of Computer and Software, Department of Civil and Environmental Engineering, Rensselaer Polytechnic Institute, Troy, NY, USASearch for more papers by this author Wei Sun, Corresponding Author Wei Sun sunw0125@163.com orcid.org/0000-0002-4870-2971 School of Automation, Nanjing University of Information Science & Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorXiaorui Zhang, Xiaorui Zhang School of Computer and Software, Nanjing University of Information Science & Technology, Nanjing, People's Republic of China Jiangsu Collaborative Innovation Centre on Atmospheric Environment and Equipment Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorShunshun Shi, Shunshun Shi School of Automation, Nanjing University of Information Science & Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorXiaozheng He, Xiaozheng He School of Computer and Software, Department of Civil and Environmental Engineering, Rensselaer Polytechnic Institute, Troy, NY, USASearch for more papers by this author First published: 12 March 2019 https://doi.org/10.1049/iet-its.2018.5316Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Automatic vehicle classification is a fundamental task in intelligent transportation systems. Image-based vehicle classification is challenging due to occlusion, low-illumination, and scale change. This study proposes an innovative approach by combining texture and shape features into a complementary feature and using the complementary feature to train a compressive dictionary to improve accuracy and efficiency. In the feature combination, the scale-invariant feature transform descriptor is applied to extract the texture and shape features from the original vehicle images and their edge images, respectively. In the dictionary training, a compressive dictionary learning (DL) algorithm, called compressive K singular value decomposition (CKSVD) algorithm, is proposed to improve the dictionary training efficiency. The CKSVD algorithm divides the feature dictionary into several same-sized data blocks and then performs the DL in each data block based on a very sparse random projection matrix. On the feature combination and DL, the proposed approach employs the kernel sparse representation method to classify vehicles into four types: buses, trucks, vans, and sedans. The kernel sparse representation method enables the linearly inseparable classification in the combined feature space to be linearly separable. Experimental results show that the proposed approach can improve the accuracy and efficiency of vehicle classification. 1 Introduction Automatic vehicle classification is a fundamental task in intelligent transportation systems. The accurate vehicle classification plays a vital role in various traffic monitoring and management systems such as traffic surveillance system, electronic toll collection system, and an intelligent parking system [1, 2]. Owing to its importance, the vehicle classification has been an active research topic over several decades, and its performance has been improved through various studies. These studies mainly focus on the vehicle feature extraction and classifier design, which are the two critical steps affecting the performance of vehicle classification methods. There are two main features used in vehicle classification methods, i.e. shape feature and texture feature. The vehicle shape-feature characterises vehicle attribute using the global contour of a vehicle image. Several shape-feature-based extraction methods have been proposed; for example, Lin et al. [3] and Fabrizia and Renata [4] suggested to extract vehicle shape feature via edge-detection operators such as Canny and Sobel operators. Applying these edge-detection operators to the vehicle classification can effectively classify vehicles in real time when the vehicle images can be acquired clearly and efficiently. However, the accuracy of these methods reduces significantly with image quality, for example, when the illumination is poor. Different from the edge-detection-based techniques, Fu et al. [5] proposed a shape-feature extraction method based on Hu invariant moments. The method can partly enhance the robustness against vehicle scale and orientation change in images. However, when the texture of the vehicle image is complex, extracting vehicle shape feature is difficult. In contrast to the shape-feature extraction method, the texture feature extraction method extracts textural details of vehicle image, and it is robust to illumination change or scale variation [6, 7]. Therefore, to better adapt to travel environment variations (including illumination and background objects changes), texture feature is integrated into vehicle classification methods. For example, Wang and Cai [6] proposed to use the histogram of oriented gradients (HOG) descriptor to express the texture features of the vehicle, which generates the HOG feature according to a weighted voting result of gradient magnitude of every pixel in each given direction. The HOG features are invariant to the geometric and optical deformation of the image. However, extracting these features is time-consuming, which will reversely affect the accuracy and efficiency of vehicle classification. To improve the accuracy and efficiency of texture extraction, Adi and Arnida [7] and Zhang et al. [8] applied gabor filter to extract the texture features of the vehicle from multiple scales and orientations. This method can not only represent textural details of vehicle image from multiple scales and orientations but also provide robust results against illumination changes or occlusion. In addition, the gabor filter has the invariance property in spatial locality and orientation which facilitates the robustness of classification results. However, as the gabor filter involves multiple kernel functions with different scales and orientations, the texture feature extraction method based on gabor filter derives a significant amount of auxiliary information, which increases the computational complexity of the vehicle classification. Compared to the gabor filter, scale-invariant feature transform (SIFT) [9] is a robust feature detector. Shi et al. [10] proposed an SIFT descriptor to classify vehicle. Since the SIFT descriptor is robust to illumination variation, affine transform, and scale change, the vehicle classification methods based on SIFT descriptor can achieve good classification accuracy under the real-world traffic scene. The SIFT descriptor can also extract a shape feature of an object in its edge image [11]. Intuitively, multiple feature descriptors with different feature description ability can extract more descriptive and discriminative feature information for object classification. Although multiple feature descriptors may result in massive computation burden, feature dimension reduction methods such as principal component analysis (PCA) [12] or sparsity preserving projection [13] can be introduced to improve computational speed by reducing the dimensionality of extracted features. This motivates the need to extract texture and shape features based on the SIFT descriptor and combine both features for efficient vehicle classification. In addition to vehicle feature extraction, classifier design is vital for vehicle classification. Inspired by John and Yang [14], where the sparse representation-based classification is proved to be the most effective for face recognition even under occlusion and illumination variation, this paper applies the sparse representation-based method to classify vehicles. In sparse representation classification, learning a discriminative dictionary is crucial. The most popular dictionary learning (DL) method for image processing is the K singular value decomposition (KSVD) [15] algorithm. On the basis of the KSVD algorithm, a discriminative reconstruction constraint [16] is added into the DL to improve the discrepancy of the classifier. However, the discriminative KSVD (D-KSVD) cannot efficiently deal with the sparse coding coefficient. The label consistent KSVD (LC-KSVD) [17, 18] is an extension of the D-KSVD. This method jointly incorporates the discriminative sparse coding and classifier parameter into the objective function, which can prevent learning a suboptimal or local optimal solution in the DL. Fisher discriminative DL (FDDL) [19, 20] combines the Fisher discriminative criterion into the process of DL for good classification performance. However, the DL is time-consuming because of the high-dimension data. For the classification tasks involving data with unbalanced variations in different classes, the classification accuracy of the LC-KSVD and FDDL is unsatisfactory. Although the aforementioned feature extraction and DL methods can be applied to vehicle classification, texture features or shape features for vehicle classification do not provide enough discriminative information, which will affect the classification accuracy and reliability. Also, the high-dimension data in the aforementioned DL methods affects the efficiency of vehicle classification. In recent years, with the rise of deep learning, vehicle classification methods based on convolutional neural network (CNN) [21-23] have been proposed. Although these methods can learn more discriminative features and achieve fine-grained classification, they have complex network structure and need a large number of training samples to train the network for obtaining satisfactory network parameters. Furthermore, it is extremely difficult to collect so many training samples in real-world and training network parameters of CNN is also time-consuming. To address these limitations, this paper proposes a new vehicle classification approach that applies a compressive DL to the combined texture and shape features. First, the proposed approach applies the SIFT descriptor to extract texture and shape features and combines them into a complementary feature to produce a more distinctive representation. After achieving the complementary feature, this paper constructs a codebook by bag-of-features (BoFs) [11] model to normalise the complementary feature into a low and equal-dimension feature vector. Then, the proposed approach applies a high-efficiency DL algorithm, called compressive K SVD (CKSVD) DL algorithm, to train the dictionary. This paper first divides feature dictionary into several same-sized blocks and then construct the very sparse random projections (VSRPs) [24] matrices to train dictionary in the divided blocks. We use the batch-orthogonal matching pursuit (batch-OMP) algorithm [25] to compute the coefficient matrix. Finally, the proposed approach applies kernel sparse representation classification (KSRC) [26] method to classify vehicles into four types by minimising the reconstruction residuals between the test object and the feature dictionary. The main contributions of this paper are three-fold. First, in contrast to the vehicle classification methods based on a single feature, the complementary feature used in the proposed approach combines both texture and shape features to enhance the discriminative ability of feature. The same SIFT algorithm is used to extract the texture feature and shape feature from original vehicle image and its edge image, respectively, which not only improves the computational efficiency of feature extraction but also enhances discrepancy of the combined features. Furthermore, a codebook is constructed to normalise the complementary feature into a low- and equal-dimension feature vector based on the BoF model. It enables the construction of feature dictionary in following DL. Second, we propose a compressive DL algorithm based on dictionary division and VSRP to balance accuracy and efficiency of DL. The algorithm divides the original dictionary into several same-sized blocks and trains dictionary based on the divided blocks. Such consideration can improve the sparsity of learned dictionary, yielding an accurate dictionary reconstruction. Employing the VSRP, the proposed algorithm can reduce the data dimension by projecting original high-dimension data onto a few low-dimension sparse random vectors. Finally, this paper proposes a KSRC algorithm to classify vehicle. This method can improve the accuracy of vehicle classification by making the linearly inseparable samples in the original feature space to be linearly separable in the new kernel space. The rest of this paper is organised as follows. Section 2 presents the texture and shape-feature extraction methods as well as the combination of texture and shape features. Section 3 describes the proposed compressive DL algorithm and recognition method based on KSRC. Experiments and analysis are shown in Section 4 to illustrate the efficacy of the proposed vehicle classification method. The final section summarises this paper. 2 Feature extraction and combination Only using a single feature in vehicle classification will reduce classification accuracy. To solve the problem, this paper proposes to use combined complementary features for vehicle classification. First, we use the SIFT algorithm to extract the texture and shape features from the original vehicle images and their edge images, respectively. Then, we combine the texture and shape features into a complementary feature. The complementary feature can illustrate the texture and shape characteristics of the vehicle image at the same time, which enables a more discriminative recognition. Finally, we construct a codebook by BoF model based on K -means clustering algorithm [11] to normalise the complementary features into a low- and equal-dimension feature vector. 2.1 Texture feature and shape-feature extraction Texture features can describe the structure information of vehicle which is an important discriminative characteristic in vehicle classification. SIFT algorithm has good rotation-invariance and anti-interference ability. The texture feature extraction method based on the SIFT algorithm is robust against camera shooting angle change, light change, noise, and occlusion. Thereby, it is suitable for feature extraction under real-world traffic environment [10]. In this paper, we choose the SIFT algorithm to extract the descriptors to describe the texture features of the vehicle image. For an original vehicle image I with pixels, we extract their SIFT descriptors as the texture features of the vehicle image. Denote the set of SIFT descriptors as (1) where the subscript T stands for the texture feature; denotes the description vector; and n is the number of SIFT descriptors. Although SIFT descriptors are effective in describing texture features, it is difficult to extract the shape information of a vehicle on its original image. To resolve this problem, we introduce new shape descriptors named Edge-SIFT [11] to obtain shape information of the vehicle. For applying the Edge-SIFT descriptors, we need to extract the edge image of the original vehicle image in advance. The compass operator [27] has powerful abilities to detect edges in the colour image. Using distributions of pixel colours rather than the mean, the operator can leverage the orientation information of edge pixel to extract a continuous edge. Additionally, the use of histogram analysis can also facilitate the extraction of complete vehicle contour even under low-illumination traffic environment. Therefore, in this paper, we use the compass operator to detect the edge of the vehicle image. On the basis of the compass operator, the edge image of the original vehicle image I can be obtained (2) where is the binarised value of every pixel in . After extracting edge information using compass operator, texture details of the detected objects are filtered out. As the object contour is reserved, the object edge becomes distinct, which enhances the shape features of the edge image. Therefore, this paper applies the Edge-SIFT descriptors to the edge image to describe the shape features of the vehicle. Denote the set of Edge-SIFT descriptors as (3) where the subscript S represents the shape feature; denotes the description vector; and m is the number of Edge-SIFT descriptors. 2.2 Feature combination The features extracted from the original image and the edge image are used to construct a set of texture feature descriptors and a set of shape-feature descriptors . Note that, after constructing the feature descriptors based on the SIFT, typical methods adopt histogram to estimate gradient modulus values of eight gradient directions () in a subdomain [9]. Although SIFT and Edge-SIFT descriptors are calculated on different images, they have the same property, i.e. both are based on the histogram estimation of gradients. Therefore, we can directly combine them to generate complementary descriptors. The complementary descriptors A, called complementary feature, is a union of SIFT and Edge-SIFT descriptors, represented by (4) where the number of descriptors A is p, which satisfies . 2.3 Feature dictionary construction The generated complementary feature has a rather high dimension due to combining both the texture and shape features, which results in a heavy computational burden. Additionally, they are also unequal in the dimension, which causes the difficulty in following feature dictionary construction. To solve the problem, we proposed to construct a codebook [11] by BoF model using K -means clustering algorithm [28] based on the complementary feature from the whole vehicle image dataset to normalise the feature vector into a low- and equal-dimension feature vector. The trained codebook consists of B D -dimension vectors, every vector is called a code word [29]. In this paper . After achieving the codebook, the complementary features are projected onto the codebook based on the Euclidean distance to generate a normalised feature vector with dimension B, which is used to construct the feature dictionary for image classification. The feature dictionary can be expressed as , where is the feature vector generated by the i th sample image and n is the total number of sample images. The detailed procedure of feature combination and feature dictionary construction can be summarised in Fig. 1. The first step uses the SIFT algorithm to extract texture and shape features. Then, the second step combines the texture and shape features to generate a complementary feature. The third step constructs a codebook by BoF model using the K -means clustering algorithm, and these complementary features are normalised into the feature vectors with equal dimension. The last step uses these feature vectors to construct the feature dictionary by class for the following vehicle classification. Fig. 1Open in figure viewerPowerPoint Feature dictionary construction process 3 Classification based on CKSVD and KSRC Past studies [15, 16] found that the employed dictionary plays an important role in sparse representation-based vehicle classification. Although learning a dictionary from training samples has led to state-of-the-art results, most of the methods still consume long computation time [18, 19], affecting their practical applicability. Therefore, to improve the efficiency of DL, this paper proposes a new compressive DL algorithm called CKSVD, which divides the feature dictionary into several same-sized blocks and exploits several very sparse random matrices to project the data onto a few sparse random vectors. In addition, the proposed DL process applies batch-OMP to obtain the coefficient matrix, which reduces computational complexity. The vehicle classification process is shown in Fig. 2. Fig. 2Open in figure viewerPowerPoint Process of vehicle classification The vehicle classification process, illustrated by Fig. 2, consists of a training stage and recognition stage. In the training stage, the feature dictionary consisting of these equal-dimension feature vectors is trained by the proposed compressive DL algorithm to generate the compressive dictionary. The recognition stage calculates the feature vectors of the test image using the same methods as in the training stage. Then, the generated compressive dictionary is used to construct the KSRC classifier. The vehicle classification is performed based on the constructed KSRC classifier by minimising the residual. 3.1 DL based on dictionary division and VSRP As mentioned in [30], if based only on one very sparse random matrix, it will become difficult to recover the true dictionary because of its low sparsity. Increasing the number of random matrices can improve accuracy; however, will induce additional computational burden. Therefore, we need to balance the computational accuracy and efficiency when considering the number of very sparse random matrices. Inspired by Anaraki and Hughes [30], when a set of indices of compressive measurements increases, the random matrix can converge to the identity matrix. Therefore, we can choose a reasonable number of the very sparse random projections (VSRP) matrices to increase the efficiency of DL. Assume that the set of complementary features is divided into L blocks , where represents the l th block. Let represents the random matrix used for the l th block. Then, we have (5) where is the sketch of . Each entry of is distributed on with probabilities [29]. Here, the parameter s controls the sparsity of random matrix such that each column of has non-zero entries on average. The following optimisation problem provides a solution for DL: (6) The CKSVD algorithm in [30] solves the optimisation problem (6) via an iterative approach that involves sparse coding and dictionary update steps. 3.1.1 Sparse coding In the sparse coding step, (6) is minimised with respect to a fixed D to find the coefficient matrix under the strict sparsity constraint. This can be written as (7) where represents the i th sample in the l th block of the coefficient matrix . is the training dictionary consisting of the complementary feature. counts the number of non-zero entries of the coefficient matrix . Applying batch-OMP algorithm [25] to each block yields the coefficient matrix, which is faster than applying OMP to each separately. 3.1.2 Dictionary update The goal of the dictionary update step is to identify the k th dictionary atom for , assuming that is fixed. Equation (6) can be rewritten as (8) , where is the k th element of , and is the representation error for the compressive measurement when the k th dictionary atom is removed. The objective function in (8) is a quadratic function of and the minimum can be identified based on the first-order optimality condition. First, we define as a set of indices of compressive measurements in the l th block. It yields the following expression: (9) where is the square sum of all the coefficients related to the k th dictionary atom in the l th block, . Note that for computing efficiently, we can concatenate in a matrix , and define the diagonal matrix as (10) where represents a square diagonal matrix with the elements of the vector on the main diagonal. Then, we have . Given the updated , for all , the optimal is given by least squares as , . The proposed compressive DL algorithm is summarised into Algorithm 1 (see Fig. 3). The time complexity of the compressive DL algorithm for dictionary update step is , where K is the number of dictionary atom, L is the number of all sparse coefficients of the sparse coefficient matrix corresponding to every column in the dictionary, and M is the iteration number of dictionary update. Fig. 3Open in figure viewerPowerPoint Algorithm 1: Compressive DL algorithm 3.2 Vehicle classification based on KSRC The SRC method needs to train the overcomplete dictionary based on the target features, and reconstruct the samples using the trained dictionary. Let be the set of training samples, where represents the subset of training samples from class i, , is the descriptor of the j th sample for the i th class. Given a test sample , the sparse representation of y in terms of A can be identified by solving the following problem: (11) where is the norm of and is a sparsity parameter. The class identity of y is based on the minimisation of residuals [14] (12) where is a function that selects the coefficients corresponding to the c th class and makes the rest equal zero. For K classes, there are K functions of , which generates K residuals. Thus, the SRC algorithm labels y to the class that has the minimum residual. Kernel sparse representation is a non-linear extension of basic sparse representation. Through the kernel trick, samples are mapped into a new kernel feature space, and then the SRC algorithm is applied to the new feature space. This non-linear mapping can change the samples’ distribution, e.g. some samples which are linearly inseparable in the original space become linearly separable in the new space, which can improve the performance of object classification [31]. Therefore, we propose to use the KSRC to classify the vehicle. Considering the Lasso problem [32] of (11), we rewrite (11) as follows: (13) where is the Lasso penalty and is a sparsity parameter. In this paper, we focus on the Lasso problem of (13) in kernel space (14) where is an implicit mapping that projects a feature vector to the kernel space. Assume that satisfies . We introduce a Kernel coordinate descent (KCD) [33] algorithm to solve (14) considering its simplicity and efficiency. Assume that all are fixed, except . Minimising (14) is equivalent to minimising the problem below: (15) where is the partial residual [33] for fitting . Considering that , (15) is equivalent to (16) where is the residual correlation [26]. If , letting in (16), we can obtain the minimisation values when , i.e. for , and for . Otherwise, if , we have (17) The equality holds only if . The coordinate-wise update of follows the conditions: (18) This update, denoted by , is the well-known soft-thresholding shrinkage operation [34]. Note that (19) It can be rewritten as (20) where is the kernel function and . Integrated with the kernel function K, the KCD algorithm [33] updates iteratively in the kernel space using (18) and (20). For classification, the corresponding kernel SRC criterion is as follows: (21) where is the training kernel matrix, contains correlation coefficients in the kernel space, and is the class label of the i th sample. Thus, KSRC algorithm labels y as the class that has the minimum residual. 4 Experiments We applied the proposed algorithm to a dataset consisting of 4000 vehicle images including 1000 bus images, 1000 truck images, 1000 van images, and 1000 sedan images. In the dataset, 10% of the images represent the challenging cases that were taken on rainy and snow days, partly occluded by other vehicles, or at night. The size of every image is px2. Fig. 4 shows some sample images of the dataset under various conditions. Fig. 4Open in figure viewerPowerPoint Vehicle images under various conditions (a) On sunny or cloudy days, (b) On rainy and snowy days, (c) At night, (d) Partly occluded All the experiments were conducted on a personal computer with 3 GHz i7-2600 central processing unit and 16 GB random access memory, and all programme codes were implemented in MATLAB 2014b. 4.1 Comparison of complementary feature and single feature To validate the efficacy of complementary features for vehicle classification, we randomly select 3600 samples as training samples and 400 samples as test samples from the constructed dataset to conduct the experiment. Fig. 5 shows the average accuracy of vehicle classification in different data dimensions (e.g. 8, 16, 32, 64, and 128 dimensions) of complementary features, texture features, and shape features under various conditions including normal conditions and challenging conditions. Illustrated by Fig. 5, the average accuracy using SIFT descriptors or Edge-SIFT descriptors is consistently lower than the average accuracy using complementar