SVMnet: Non-parametric image classification based on convolutional SVM ensembles for small training sets

1 INTRODUCTION . Deep convolutional neural networks ( DCNNs ) are powerful tools for multiple tasks of automatic image analysis , demonstrating paramount success and consequently gaining substantial popularity over the past decade . By analyzing the pixels directly , CNNs can be applied to various types of image content without the need to develop task-specific algorithms , and can easily be applied to a broad range of domains with excellent performance ( Khan et al. , 2020 ) . One of the major weaknesses of modern DCNNs is their dependence on a large set of examples for training . Cutting-edge DCNNs can have hundreds of layers , each with thousands of trainable parameters . For instance , the common ResNet-50 ( He et al. , 2016 ) contains over 2·106 artificial neurons . Therefore , to achieve meaningful performance and avoid underfitting , DCNNs normally rely on relatively large training sets . Training DCNNs normally requires large datasets of labeled ground truth images . Commonly used datasets include benchmarks such as the Modified National Institute of Standards and Technology ( MNIST ) or ImageNet . These benchmark datasets provide tens of thousands of images with highquality annotations for training deep CNNs and are commonly used for testing their performance . However , in many cases of real-world image classification problems , large datasets of clean , labeled ground truth are not available . A common solution to increasing the size of the training set is data augmentation , in which different modifications of the images in the original dataset can create more training samples . However , that strategy can also lead to biases by overusing the same examples . In some cases transfer learning can be used to fine-tune neural networks using pre-trained models , but for domains with very small datasets for fine-tuning , the pre-trained models may remain too sensitive to their original task . In many real-world cases large datasets are not available . For instance , in the biomedical domain the acquisition and annotation of each image requires the use of costly medical instrumentation , technicians , and medical staff who can annotate each sample manually . Additionally , human protection procedures and protocols are required for the acquisition of each sample , making the preparation of large datasets less practical . Therefore , biomedical image datasets are normally far smaller than the modern datasets commonly used to train DCNNs . Rare cases can also make it difficult to acquire a suitable training set ( Zhang et al. , 2019 ) . The need for a large number of training samples is a practical downside of DCNNs , making it difficult to use optimally in many real-world cases . The problem of small training sets has been addressed in the past by using previous knowledge for a few shot training ( Zhang et al. , 2019 ) and even one-shot training ( Fei-Fei et al. , 2006 ; Wolf et al. , 2009 ; Koch et al. , 2015 ; Vinyals et al. , 2016 ; Bender et al. , 2018 ) . This paper explores a new form of image classification in cases when the number of samples is limited . Based on an ensemble composition of support vector machines ( SVMs ) , the method can work with no prior knowledge , in a similar manner to “ standard ” supervised machine learning . Inspired by CNN architecture , SVMnet utilizes a large number of small SVMs to quickly analyze image patches , structured in layers that allow for stacking or custom ensemble techniques . An SVM ( Cortes & Vapnik , 1995 ) is less sensitive to high-dimensionality feature spaces ( Chen et al. , 2005 ; Mukkamala et al. , 2002 ) , and has the advantage of learning from a relatively small number of training samples ( Shin et al. , 2005 ; Byvatov et al. , 2003 ; Paiva et al. , 2018 ) compared to other supervised machine learning approaches . 2 ARCHITECTURE OF SVMNET . The proposed SVMnet1 architecture is designed as a stacked ensemble of numerous simple SVM classifiers organized into one or more layers . Each layer is an array of SVMs which functions similarly to a convolutional layer in a CNN . Each SVM in a layer is independent and all are assigned an equal-sized patch of the layer ’ s input , referred to as a window . Variable stride length and padding , as described in Chapter 2 of Dumoulin & Visin ( 2018 ) , are specified as hyperparameters . Each input to the following layer is the output of one SVM . When a layer is evaluated , each SVM in the layer is trained on ground truth labels . The input to the SVM is the flattened portion of each input image that is within the SVM ’ s window . Each pixel channel within the window is essentially treated as one input feature . For instance , a 5 × 5 window would create a 25-feature SVM for grayscale input and a 75-feature SVM for 3-channel RGB input . During this step , the SVMs may be given weights based on the accuracy of the fit , used for ensemble classification . Each SVM then predicts a class label or a vector of class probabilities for its window of each input , creating an input tensor for the next layer . Figure 1 shows a simple layer in SVMnet . Each node in the layer is one SVM , trained using the ground truth labels for the input samples . The weights are determined based on the classification accuracy of the SVM compared to the ground truth of the training set . The weight function is configurable and will be described later in this Section . To produce one class label for each input , SVMnet may perform a weighted vote after the final layer . This vote combines the results of the final layer by treating each value as a vote for that class label . If the final layer is weighted , these are used to weigh the votes in favor of SVMs with higher accuracy . 1SVMnet is not a ” network ” in the same sense as CNNs and are named as such mostly due to analogy . Sc = Σiη ( Ai ) [ Pi = c ] ( 1 ) The total voting score Sc of each class c is calculated by Equation 1 , where Ai is the accuracy score of SVM i in the final layer , η is the weight function , and Pi is the class label predicted by SVM i . That is , if the predicted label Pi of SVM i is class c , the weighted score η ( Ai ) is added to the vote for that class . The weight function emphasizes the predictions of the SVMs with higher accuracy during training . The class that has the highest score Sc is chosen as the predicted label by the model for the given sample . The weight function η is configurable and in our experiments is defined as η ( x ) = x2 , where x is the classification accuracy of the SVM determined during training . While the layers support arbitrary estimators , here we use only support vector machines ( SVM ) , hence the name SVMnet . The SVMs are trained with a Radial Basis Function ( RBF ) kernel ( Chang & Lin , 2011 ) and scaling gamma value , and they continue to iterate until convergence with a 0.001 tolerance . The ability to choose different estimators in each layer can be compared to the ability to use different activation functions in the layers of neural networks . Figure 2 illustrates one possible two-layer SVMnet architecture . Each SVM in the first layer analyzes a specific patch of each image and is fitted independently against ground truth labels . These SVMs then produce a vector of class probabilities for the same pixel region which forms the input matrix for the following layer . The SVMs in the second layer are fitted on a region of these probabilities and predict a class label for the image . These labels are then tallied in a final vote to produce one label for the input . The motivation for multiple layers is that secondary layers can in essence learn which of the SVMs in their window are more accurate or ” trustworthy ” , as their predictions are being compared to ground truth labels in each layer . 2.1 DROPOUT . Not every patch is expected to produce a well-informed SVM . Some regions of the images , particularly towards the edge , often lack the details necessary to distinguish samples from each other . This can cause the outputs of these SVMs to act as noise in a vote tally . Even with the expected low accuracy score of the SVM depressing the weight of its vote , if the low-information regions are large then enough inaccurate votes may overwhelm the more accurate votes . To help prevent this , a dropout system is implemented for the vote tally . When using dropout , which SVMs to drop are calculated when fitting SVMnet . First , the SVMs are ordered from the highest weight to the lowest . Votes are then cumulatively tallied one SVM at a time with the accuracy of the votes measured between each tally . SVMnet then finds the global maximum accuracy of the cumulative tally . This marks the point where including the votes of the less-accurate SVMs lowers the overall accuracy of the tally , so those SVMs are marked for dropout and are not included in the final vote . 3 EXPERIMENTAL RESULTS . To test the efficacy of SVMnet compared to a “ conventional ” CNN , several experiments were performed using common , relatively small datasets . The purpose of SVMnet is not to outperform CNNs in the general case , but to achieve higher accuracy when the number of labeled training images is limited . Therefore , the experiments were made with different sizes of training sets to compare the classification accuracy as the training set increases . The performance of the SVMnet was compared to the performance of residual network , or ResNet , models with 18 , 34 , and 50 layers ( He et al. , 2016 ) . ResNet is a powerful architecture that was designed to reduce the number of required training samples for deep learning tasks and has demonstrated excellent efficacy in image classification . Each ResNet model was compared when trained from scratch and when fine-tuned using pretrained ImageNet weights . Following the practice in He et al . ( 2016 ) , the final convolutional layer is followed by a global average pooling layer , then by a single fully-connected layer with softmax activation and as many units as class labels in the respective task . Models were trained using stochastic gradient descent ( SGD ) optimization with a linearly decaying learning rate ( given by 0.999 ( 1 − s/2 ) + 0.001 where s is the training step ) and Nesterov momentum of 0.9 . The models were trained for a maximum of 200 epochs but were stopped early if the loss on the validation dataset did not improve by at least 0.01 over 20 epochs . The number of epochs is limited in order to keep the ResNet training times comparable to SVMnet . The resulting accuracy and training time for each model was averaged over 5 repetitions of each experiment . While the height and width of inputs can be adjusted for ResNet , the architecture always expects 3-channel RGB color images . Grayscale images were modified for use by ResNet by duplicating the pixel values into three equal channels . This approach was used in Section 3.3 and Section 3.4 . Before training and classification by ResNet , images were also passed through a preprocessing filter provided by the Keras library to prepare the data for ResNet models . All inputs were normalized by dividing by the mean and subtracting the variance before being used to train SVMnet . For RGB color inputs , the images were normalized per-channel . All experiments and analysis presented in this section used the same hardware environment . SVMnet was parallelized across 16 cores of Intel Xeon Gold 6130 CPUs , and ResNet models were trained on an nVidia GeForce GTX 2080 GPU .

Summary: One of the weakness of traditional DCNNs is it needs large clean-labeled dataset. In this paper, the author proposes SVMNET, a deep learning architecture which includes a layered structure of Support Vector Machine (SVM) ensembles. The result shows that the SVMNET outperforms other deep convolutional neural networks such as ResNet-50 with less training time for cases in which the number of labeled training samples is small.