Classification method based on the deep structure and least squares support vector machine

Electronics LettersVolume 56, Issue 11 p. 538-541 Control engineering Free Access Classification method based on the deep structure and least squares support vector machine Wenlu Ma, School of Automation and Information Engineering, Xian University of Technology, Xian, 710048 People's Republic of ChinaSearch for more papers by this authorHan Liu, Corresponding Author liuhan@xaut.edu.cn School of Automation and Information Engineering, Xian University of Technology, Xian, 710048 People's Republic of ChinaSearch for more papers by this author Wenlu Ma, School of Automation and Information Engineering, Xian University of Technology, Xian, 710048 People's Republic of ChinaSearch for more papers by this authorHan Liu, Corresponding Author liuhan@xaut.edu.cn School of Automation and Information Engineering, Xian University of Technology, Xian, 710048 People's Republic of ChinaSearch for more papers by this author First published: 01 May 2020 https://doi.org/10.1049/el.2019.3776Citations: 2 AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Support vector machines (SVMs) are one of the most representative shallow network models and have good generalisation abilities in small data sets. In this Letter, a new classification method based on the deep structure and least squares SVM (LSSVM) is proposed. For large-scale data sets, the method builds the structures of a multi-layer SVM. Using edge detection and the K-means algorithm, the sample set is compressed into a smaller sample set, which is used to train the LSSVM model of each layer and the discriminant classification function is obtained. Finally, this method is applied to UCI data sets and compared with several density-dependent quantised LSSVM methods and other methods. The experimental results show that the method has good performance in solving the large-scale data set classification problem. Introduction Support vector machines (SVMs) are one of the most important algorithms in machine learning [1]. SVM was proposed by Vapnik and Corinna Cortes as a typical intelligent classification model. To simplify the solution process, Suykens et al. proposed an improved SVM algorithm named the least squares SVM (LSSVM) [2]. However, a drawback of LSSVM models is the lack of sparse solutions and the dimension of the linear equations increases as the number of samples increases, thereby consuming more computer memory. In the last ten years, in order to solve the sparse solution problem, a large number of scholars have conducted related research and proposed many new algorithms and improved algorithms to solve the training sample set sparsity problem. Suykens et al. have made substantial efforts using large-scale data sets; these authors have developed many excellent algorithms [3-8] and combined the LSSVM with other algorithms for machine learning [9-11]. In recent years, due to the emergence of deep learning, deep structures have been proven to be more effective in many cases and have been successfully applied in various fields. Traditional machine learning methods, including the LSSVM, have been greatly affected. A SVM classification algorithm based on depth kernel theory was proposed that can be applied to large-scale data sets with a large number of possible classes [12-14]. Suykens proposed a theory of deeply restricted kernel machines using conjugate feature duality [15]. A deep learning model based on SVMs and a probability output network has also been proposed [16, 17]. Qi et al. [18] and Li et al. [19, 20] mainly studied deep learning classification based on SVMs. Based on the above analysis, as the amount of data information continues to increase, the simple LSSVM is not able to deal with such problems. The combination of the traditional LSSVM algorithm with deep structures is a new way to solve this problem. Therefore, it is necessary to improve and study related algorithms. Based on the excellent performance of the LSSVM model in classification problems, this Letter proposes a new deep structure and LSSVM (DSLSSVM) classification algorithm. For large-scale sample data sets, this method constructs a multi-layer SVM structure and introduces neighbouring edge detection and the K-means algorithm to reduce the original big data sample set to a small data sample set. In each layer, the reduced LSSVM model is used to train the LSSVM model. After the layer training, the accuracy is improved, and finally, a better classification effect is obtained while maintaining a good generalisation ability. Neighbour edge detection Edge detection is the foundation of image processing technologies, such as digital image measurement, image segmentation, image compression, and pattern recognition, and is one of the important research topics in digital image processing. The basic idea of edge detection is to detect the edge of each object in the image and to use the edge line to represent the information in the image to simplify, compress, and recognise the image information. In our edge detection model, we change this rule to detect the changes of the class labels between samples and their neighbours (the Euclidean distance is used to measure the similarity between samples in this Letter) and determine whether it is an edge sample. In Fig. 1, in the neighbourhood scanning edge detection, the nearest neighbour samples of the target are selected as training data samples (where ). If at least one of the category labels in the adjacent samples is different, the selection sample is retained as the edge sample. An important issue with this approach is how to determine the number of adjacent samples. Therefore, we can set the number of neighbouring samples as a constant, but it cannot be too small. As to how to select this constant, this problem is discussed in [10, 11] and is not restated here. Algorithm 1 describes in detail the specific steps of edge detection for nearest neighbour scanning. It is as follows: (1)where (1) is the formula for calculating the similarity value, and it will be applied to the following algorithm, represents the similarity value of samples and , and represents the pth component of the ith sample. Algorithm 1.Steps of edge detection for nearest neighbour scanning Input: Data set: ,where X is the sample set, Y is the label, and the neighbour parameter is kOutput: Training subset: . 1: Randomly take a sample c to be tested using data set D, and calculate the similarity value between each sample to be tested and the remaining samples in the training set. 2: For do For each sample to be tested, the first samples with high similarity are selected to form a detection set and a corresponding label set . 3: If all elements in are consistent edge sample else non-edge sample endif endfor 4: Repeat steps 2 and 3 until all samples in the training set are judged and the algorithm is over. Fig 1Open in figure viewerPowerPoint Edge detection in classification problem Multi-layer LSSVM The LSSVM primal problem is formulated as follows: (2) Finally, the classification function is obtained (3)where b is the deviation term of the classification, is a Lagrange multiplier, and is the kernel function. The network structure of the deep LSSVM (DLSSVM) is shown in Fig. 2. The structure consists of several layers, each of which is a standard LSSVM. After the training sample trains the first layer, using the support vector that is obtained by the training as a new training sample, the connection with the next layer is established, and a new layer of the training model is constructed. This process repeats until the optimal result is output. Fig 2Open in figure viewerPowerPoint DLSSVM model Deep structure and LSSVM (DSLSSVM) By using the edge detection method from 'Neighbour edge detection' section, a set of samples can be extracted from a given training data set, which preserves most of the training sample set information and filters out the information that may not be very relevant, which greatly reduces the amount of training data. In addition, in order to obtain higher accuracy and better robustness, we use a clustering method to find k centres from the remaining edge samples and train the LSSVM. The flow chart of the method is as follows. The dotted line in the graph represents the process of reducing the training set by using adjacent edge detection and clustering methods. In this Letter, we introduce the K-means algorithm for the following research. To solve the sparsity problem, based on deep support vector machine (DSVM) theory, DSLSSVM based on the edge detection method is proposed in this Letter. The DSLSSVM is applied to the network structure of the DSVM, and each layer of the DSVM is replaced by the LSSVM to achieve the superposition of the multi-layer LSSVM. The training process is the same as that of the DSVM. As shown in the flow chart of Fig. 3, this is a model training layer. After each layer of the DSLSSVM model is trained, the support vector is obtained as the input sample of the next layer, the new input is placed in the second layer of the DSLSSVM to be trained, and this repeats layer by layer until the optimal result is output. In the multi-layer structure of the DSLSSVM, the new training samples are used to train each layer to achieve deep learning. Compared with the DSVM, the method ensures good classification precision, reduces the calculation complexity and the operating time, increases the sparsity of the training sample, and has good robustness. The specific steps of the DSLSSVM algorithm are as follows (Algorithm 2): Algorithm 2.Steps of edge detection for nearest neighbour scanning Initialisation: . Test set X, labelY, , the neighbour parameter is k.Training set , test set , andInput: Data set: .Output: Training subsample set: . 1: Using the neighbouring edge detection algorithm (Algorithm 1), obtain the edge sample set to form a first part training set. 2: For training sets , use the k-means method to find k clustering centre points, denote as . 3: Get the final training set , and use this sample set to train the LSSVM model to obtain the final classification surface. 4: Test the performance of classifier f on the test set and get the test results. Fig 3Open in figure viewerPowerPoint Single layer LSSVM flow chart Numerical examples All results of experiments were obtained using a PC with an Intel Core i7 CPU, 4 GB RAM and MATLAB 2013a. To properly measure the performance of each algorithm, all data sets were tested using a ten-fold cross-validation method. Figs. 4 and 5 show the classifier output settings based on the Ripley data. Fig. 4 shows the visualisation result of the DSLSSVM algorithm that is proposed in this Letter and shows the training sample set of the LSSVM model combined with the edge detection and the K-means clustering algorithms, which is a subsampled set of the original data. Fig. 5 shows the results of training the LSSVM model without pre-processing the original sample set. Both methods can achieve better classification results. Fig 4Open in figure viewerPowerPoint Single layer DSLSSVM flow chart Fig 5Open in figure viewerPowerPoint Single layer LSSVM flow chart Table 1 compares the test results of several algorithms on the UCI data sets. Each algorithm evaluates the performance of the algorithm in terms of its accuracy (standard deviation), number of support vectors (SVs), and time. From the table, we can see that the number of SVs in the DSLSSVM, the random sample LSSVM (RSLSSVM), the low-rank linearisation SVM (LSSVM) and LSSVM are relatively large, while the numbers of SVs in the density-dependent quantised LSSVM (DQLSSVM) and the fixed-size LSSVM (FSLSSVM) algorithms are relatively small. In terms of time, the other five algorithms run faster, while the DSLSSVM algorithm runs for a long time. This phenomenon is because our algorithm model is not shallow and most of the others are at least two layers, resulting in a relatively long running time for this algorithm. However, in terms of accuracy, this algorithm and several other algorithms can achieve better classification results, and the accuracy of our algorithm is higher than those of other algorithms, which shows the effectiveness of the algorithm that is proposed in this Letter. Table 1. Comparison of the result of several algorithms on UCI data sets DSLSSVM DQLSSVM Data set Acc. (Std.) M Time Acc. (Std.) M Time Liver 0.7289 (4.47) 246 0.39 0.7193 (4.39) 20 0.01 Ionosphere 0.9960 (0.83) 172 0.66 0.9430 (2.50) 49.5 0.03 Credit 0.8849 (4.02) 130 0.34 0.8628 (3.30) 45.5 0.05 BCW 0.9685 (1.81) 43 0.65 0.9662 (2.32) 29.5 0.02 Pima 0.7911 (3.82) 188 0.47 0.7460 (2.79) 82.5 0.07 Mushroom 0.9950 (0.04) 138.6 0.40 0.9969 (0.10) 90.6 0.65 EEG Eye 0.8475 (0.09) 798.3 17.05 0.8299 (0.40) 118.5 1.89 BM 0.8887 (0.03) 839.4 104.26 0.8880 (0.05) 126.4 5.94 FSLSSVM RSLSSVM Data set Acc. (Std.) M Time Acc. (Std.) M Time Liver 0.6418 (4.42) 20 0.26 0.6791 (4.64) 100 0.03 Ionosphere 0.9369 (2.46) 50 0.65 0.9249 (2.71) 100 0.03 Credit 0.8330 (2.57) 50 1.19 0.7877 (4.06) 150 0.06 BCW 0.9540 (1.71) 50 1.21 0.9580 (1.27) 300 0.14 Pima 0.6462 (3.02) 100 4.10 0.7260 (3.06) 100 0.04 Mushroom 0.9893 (0.13) 90 31.85 0.9886 (0.54) 90 0.08 EEG Eye 0.8024 (0.26) 100 89.51 0.6687 (1.42) 100 0.29 BM 0.8870 (0.06) 100 307.60 0.8862 (0.50) 100 0.64 LSSVM LLSVM Data set Acc. (Std.) M Time Acc. (Std.) M Time Liver 0.7193 (4.39) 258 0.02 0.5711 (4.88) 100 0.04 Ionosphere 0.9409 (2.37) 263 0.02 0.8518 (3.46) 100 0.06 Credit 0.8613 (2.28) 489 0.07 0.8421 (2.85) 150 0.12 BCW 0.9608 (1.37) 512 0.09 0.9568 (1.43) 300 0.48 Pima 0.7548 (3.21) 576 0.10 0.7386 (0.90) 100 0.07 Mushroom 0.9848 (0.78) 90 0.49 EEG Eye 0.7227 (1.30) 120 1.71 BM 0.8706 (0.89) 130 8.63 Fig. 6 depicts the accuracy of each layer of the DSLSSVM and LSSVM on the UCI data sets. The dark blue bar represents the precision that is obtained without directly pre-processing the training set using the LSSVM. The light blue bar represents the results of running the first layer of the algorithm (DSLSSVM). The yellow bar represents the results of running the second layer of the DSLSSVM. The red bar represents the results of running the third layer of the DSLSSVM. Obviously, the accuracy of the first two layers of the LSSVM algorithm for the Liver and Ionosphere data sets is higher than that of the LSSVM, and the third layer is less accurate. The accuracy of the first three layers of the DSLSSVM algorithm for the Credit, BCW and Pima data sets is higher than that of the LSSVM. In the DSLSSVM algorithm, the other four data sets, except Credit, have the highest accuracy in the first layer, and the second layer has higher accuracy than the third layer, indicating that in these four data sets, only the DSLSSVM method of one layer can be run, and the Credit data set must run the second layer of the DSLSSVM to achieve higher accuracy. Fig 6Open in figure viewerPowerPoint Accuracy of DSLSSVM and LSSVM with per layer on UCI data sets Conclusion As an improved SVM algorithm, the LSSVM has a good learning ability for small sample data, but it simultaneously loses sparsity. To solve the LSSVM's sparsity problem, a new deep structure SVM classification algorithm is proposed in this Letter. Using edge detection and the K-means algorithm, the sample set is compressed into a smaller sample set, which is used to train the LSSVM model of each layer and the discriminant classification function is obtained. The data containing more information in data sets near the edges are found, and they are reduced to a subset of the original data set. Then, the LSSVM model is trained, and finally, the best classification is obtained. Compared with other algorithms, our algorithm runs for a longer time, but it is more accurate, which ensures the better classification effect of the algorithm. The experimental results show the effectiveness of the proposed algorithm. Acknowledgments This work was supported by the Chinese National Natural Science Foundation under grant nos. 61973248, 61833013, and 61533014, the Key Project of Shaanxi Key Research and Development Program under grant no. 2018ZDXM-GY-089 and the Research Program of Shaanxi Collaborative Innovation Center of Modern Equipment Green Manufacturing under grant no. 304-210891704. References 1Vapnik, V.N.: ' The nature of statistical learning theory' ( Springer, New York, NY, USA, 1995) Crossref Web of Science®Google Scholar 2Suykens, J.A.K., Lukas, L., Vandewalle, J.: ' Least squares support vector machine classifiers', Neural Process. Lett., 1999, 9, (3), pp. 293– 300 (https://doi/org/10.1023/A:1018628609742) Crossref Web of Science®Google Scholar 3Zhou, S.S.: ' Sparse LSSVM in primal using cholesky factorization for large-scale problems', IEEE Trans.Neural Netw. Learn. Syst., 2016, 27, (4), pp. 783– 795 (https://doi/org/10.1109/TNNLS.2015.2424684) Crossref PubMed Web of Science®Google Scholar 4Nan, S.Y., Sun, L., Chen, B.D. et. al.,: ' Density dependent quantized least squares support vector machine for large data sets', IEEE Trans. Neural Netw. Learn. Syst.., 2017, 28, (1), pp. 94– 106 (https://doi/org/10.1109/TNNLS.2015.2504382) Crossref PubMed Web of Science®Google Scholar 5De Brabanter, K., De Brabanter, J., Suykens, J.A.K. et. al.,: ' Optimized fixed-size kernel models for large data sets', Comput. Statist. Data Anal., 2010, 54, (6), pp. 1484– 1504 (https://doi/org/10.1016/j.csda.2010.01.024) Crossref Web of Science®Google Scholar 6Wang, P., Yan, A.J.: ' Improved pruning algorithm using quadratic Renyi entropy for LS-SVM modeling'. Proc. 24th Chinese Control and Decision Conf. (CCDC), Taiyuan, China, May 2012, pp. 3471– 3474 Google Scholar 7Mall, R., Suykens, J.A.K.: ' Sparse reductions for fixed-size least squares support vector machines on large scale data'. Pacific-Asia Conference on Knowledge Discovery and Data Mining, Gold Coast, Australia, April 2013, pp. 161– 173 Google Scholar 8Mall, R., Suykens, J.A.K.: ' Very sparse LSSVM reductions for large-scale data', IEEE Trans. Neural Netw. Learn. Syst.., 2015, 26, (5), pp. 1086– 1097 (https://doi/org/10.1109/TNNLS.2014.2333879) Crossref PubMed Web of Science®Google Scholar 9Liu, B., Xiang, H.: ' An approximate linear solver in least square support vector machine using randomized singular value decomposition', WuHan Univ. J. Nat. Sci., 2015, 20, (4), pp. 283– 290 (https://doi/org/10.1007/s11859-015-1094-9) Crossref Google Scholar 10Li, B., Wang, Q., Hu, J.: ' Fast SVM training using edge detection on very large datasets', IEEE Trans. Electr. Electron. Eng., 2013, 8, (3), pp. 229– 237 (https://doi/org/10.1002/tee.21844) Wiley Online Library Web of Science®Google Scholar 11El-tawel, G.S., Helmy, A.K.: ' An edge detection scheme based on least squares support vector machine in a contourlet HMT domain', Appl. Soft Comput., 2015, 26, (1), pp. 418– 427 (https://doi/org/10.1016/j.asoc.2014.10.025) Crossref Web of Science®Google Scholar 12Rebai, I., BenAyed, Y., Mahdi, W.: ' Deep kernel-SVM network'. 2016 Int. Joint Conf. on Neural Networks (IJCNN), Vancouver, BC, Canada, July 2016, pp. 1995– 1960 Google Scholar 13Zareapoor, M., Shamsolmoali, P., Jain, D.K. et. al.,: ' Kernelized support vector machine with deep learning: an efficient approach for extreme multiclass dataset', Pattern Recognit. Lett., 2018, 115, pp. 4– 13 (https://doi/org/10.1016/j.patrec.2017.09.018) Crossref Web of Science®Google Scholar 14Li, Y.J., Zhang, T.: ' Deep neural mapping support vector machines', Neural Netw., 2017, 93, pp. 185– 194 (https://doi/org/10.1016/j.neunet.2017.05.010) Crossref PubMed Web of Science®Google Scholar 15Suykens, J.A.K.: ' Deep restricted kernel machines using conjugate feature duality', Neural Comput., 2017, 29, (8), pp. 2123– 2163 (https://doi/org/10.1162/neco_a_00984) Crossref PubMed Web of Science®Google Scholar 16Kim, S., Chol, Y., Lee, M.: ' Deep learning with support vector data description', Neurocomputing, 2015, 165, pp. 111– 117 (https://doi/org/10.1016/j.neucom.2014.09.086) Crossref Web of Science®Google Scholar 17Kim, S., Yu, Z., Kil, R.M et. al.,: ' Deep learning of support vector machines with class probability output networks', Neural Netw., 2015, 64, pp. 19– 28 (https://doi/org/10.1016/j.neunet.2014.09.007) Crossref PubMed Web of Science®Google Scholar 18Qi, Z.Q., Wang, B., Tian, Y.J. et. al.,: ' When ensemble learning meets deep learning: a new deep support vector machine for classification', Knowl.-Based Syst., 2016, 107, pp. 54– 60 (https://doi/org/10.1016/j.knosys.2016.05.055) Crossref Web of Science®Google Scholar 19Li, K., Zhang, R., Li, F.C. et. al.,: ' A new rotation machinery fault diagnosis method based on deep structure and sparse least squares support vector machine', IEEE Access, 2019, 7, pp. 26571– 26580 (https://doi/org/10.1109/ACCESS.2019.2901363) Crossref Web of Science®Google Scholar 20Li, X., Yang, Y., Pan, H.Y. et. al.,: ' A novel deep stacking least squares support vector machine for rolling bearing fault diagnosis', Comput. Ind., 2019, 110, pp. 36– 47 (https://doi/org/10.1016/j.compind.2019.05.005) Crossref Web of Science®Google Scholar Citing Literature Volume56, Issue11May 2020Pages 538-541 FiguresReferencesRelatedInformation