SVMnet: Non-parametric image classification based on convolutional SVM ensembles for small training sets

1 INTRODUCTION . Deep convolutional neural networks ( DCNNs ) are powerful tools for multiple tasks of automatic image analysis , demonstrating paramount success and consequently gaining substantial popularity over the past decade . By analyzing the pixels directly , CNNs can be applied to various types of image content without the need to develop task-specific algorithms , and can easily be applied to a broad range of domains with excellent performance ( Khan et al. , 2020 ) . One of the major weaknesses of modern DCNNs is their dependence on a large set of examples for training . Cutting-edge DCNNs can have hundreds of layers , each with thousands of trainable parameters . For instance , the common ResNet-50 ( He et al. , 2016 ) contains over 2·106 artificial neurons . Therefore , to achieve meaningful performance and avoid underfitting , DCNNs normally rely on relatively large training sets . Training DCNNs normally requires large datasets of labeled ground truth images . Commonly used datasets include benchmarks such as the Modified National Institute of Standards and Technology ( MNIST ) or ImageNet . These benchmark datasets provide tens of thousands of images with highquality annotations for training deep CNNs and are commonly used for testing their performance . However , in many cases of real-world image classification problems , large datasets of clean , labeled ground truth are not available . A common solution to increasing the size of the training set is data augmentation , in which different modifications of the images in the original dataset can create more training samples . However , that strategy can also lead to biases by overusing the same examples . In some cases transfer learning can be used to fine-tune neural networks using pre-trained models , but for domains with very small datasets for fine-tuning , the pre-trained models may remain too sensitive to their original task . In many real-world cases large datasets are not available . For instance , in the biomedical domain the acquisition and annotation of each image requires the use of costly medical instrumentation , technicians , and medical staff who can annotate each sample manually . Additionally , human protection procedures and protocols are required for the acquisition of each sample , making the preparation of large datasets less practical . Therefore , biomedical image datasets are normally far smaller than the modern datasets commonly used to train DCNNs . Rare cases can also make it difficult to acquire a suitable training set ( Zhang et al. , 2019 ) . The need for a large number of training samples is a practical downside of DCNNs , making it difficult to use optimally in many real-world cases . The problem of small training sets has been addressed in the past by using previous knowledge for a few shot training ( Zhang et al. , 2019 ) and even one-shot training ( Fei-Fei et al. , 2006 ; Wolf et al. , 2009 ; Koch et al. , 2015 ; Vinyals et al. , 2016 ; Bender et al. , 2018 ) . This paper explores a new form of image classification in cases when the number of samples is limited . Based on an ensemble composition of support vector machines ( SVMs ) , the method can work with no prior knowledge , in a similar manner to “ standard ” supervised machine learning . Inspired by CNN architecture , SVMnet utilizes a large number of small SVMs to quickly analyze image patches , structured in layers that allow for stacking or custom ensemble techniques . An SVM ( Cortes & Vapnik , 1995 ) is less sensitive to high-dimensionality feature spaces ( Chen et al. , 2005 ; Mukkamala et al. , 2002 ) , and has the advantage of learning from a relatively small number of training samples ( Shin et al. , 2005 ; Byvatov et al. , 2003 ; Paiva et al. , 2018 ) compared to other supervised machine learning approaches . 2 ARCHITECTURE OF SVMNET . The proposed SVMnet1 architecture is designed as a stacked ensemble of numerous simple SVM classifiers organized into one or more layers . Each layer is an array of SVMs which functions similarly to a convolutional layer in a CNN . Each SVM in a layer is independent and all are assigned an equal-sized patch of the layer ’ s input , referred to as a window . Variable stride length and padding , as described in Chapter 2 of Dumoulin & Visin ( 2018 ) , are specified as hyperparameters . Each input to the following layer is the output of one SVM . When a layer is evaluated , each SVM in the layer is trained on ground truth labels . The input to the SVM is the flattened portion of each input image that is within the SVM ’ s window . Each pixel channel within the window is essentially treated as one input feature . For instance , a 5 × 5 window would create a 25-feature SVM for grayscale input and a 75-feature SVM for 3-channel RGB input . During this step , the SVMs may be given weights based on the accuracy of the fit , used for ensemble classification . Each SVM then predicts a class label or a vector of class probabilities for its window of each input , creating an input tensor for the next layer . Figure 1 shows a simple layer in SVMnet . Each node in the layer is one SVM , trained using the ground truth labels for the input samples . The weights are determined based on the classification accuracy of the SVM compared to the ground truth of the training set . The weight function is configurable and will be described later in this Section . To produce one class label for each input , SVMnet may perform a weighted vote after the final layer . This vote combines the results of the final layer by treating each value as a vote for that class label . If the final layer is weighted , these are used to weigh the votes in favor of SVMs with higher accuracy . 1SVMnet is not a ” network ” in the same sense as CNNs and are named as such mostly due to analogy . Sc = Σiη ( Ai ) [ Pi = c ] ( 1 ) The total voting score Sc of each class c is calculated by Equation 1 , where Ai is the accuracy score of SVM i in the final layer , η is the weight function , and Pi is the class label predicted by SVM i . That is , if the predicted label Pi of SVM i is class c , the weighted score η ( Ai ) is added to the vote for that class . The weight function emphasizes the predictions of the SVMs with higher accuracy during training . The class that has the highest score Sc is chosen as the predicted label by the model for the given sample . The weight function η is configurable and in our experiments is defined as η ( x ) = x2 , where x is the classification accuracy of the SVM determined during training . While the layers support arbitrary estimators , here we use only support vector machines ( SVM ) , hence the name SVMnet . The SVMs are trained with a Radial Basis Function ( RBF ) kernel ( Chang & Lin , 2011 ) and scaling gamma value , and they continue to iterate until convergence with a 0.001 tolerance . The ability to choose different estimators in each layer can be compared to the ability to use different activation functions in the layers of neural networks . Figure 2 illustrates one possible two-layer SVMnet architecture . Each SVM in the first layer analyzes a specific patch of each image and is fitted independently against ground truth labels . These SVMs then produce a vector of class probabilities for the same pixel region which forms the input matrix for the following layer . The SVMs in the second layer are fitted on a region of these probabilities and predict a class label for the image . These labels are then tallied in a final vote to produce one label for the input . The motivation for multiple layers is that secondary layers can in essence learn which of the SVMs in their window are more accurate or ” trustworthy ” , as their predictions are being compared to ground truth labels in each layer . 2.1 DROPOUT . Not every patch is expected to produce a well-informed SVM . Some regions of the images , particularly towards the edge , often lack the details necessary to distinguish samples from each other . This can cause the outputs of these SVMs to act as noise in a vote tally . Even with the expected low accuracy score of the SVM depressing the weight of its vote , if the low-information regions are large then enough inaccurate votes may overwhelm the more accurate votes . To help prevent this , a dropout system is implemented for the vote tally . When using dropout , which SVMs to drop are calculated when fitting SVMnet . First , the SVMs are ordered from the highest weight to the lowest . Votes are then cumulatively tallied one SVM at a time with the accuracy of the votes measured between each tally . SVMnet then finds the global maximum accuracy of the cumulative tally . This marks the point where including the votes of the less-accurate SVMs lowers the overall accuracy of the tally , so those SVMs are marked for dropout and are not included in the final vote . 3 EXPERIMENTAL RESULTS . To test the efficacy of SVMnet compared to a “ conventional ” CNN , several experiments were performed using common , relatively small datasets . The purpose of SVMnet is not to outperform CNNs in the general case , but to achieve higher accuracy when the number of labeled training images is limited . Therefore , the experiments were made with different sizes of training sets to compare the classification accuracy as the training set increases . The performance of the SVMnet was compared to the performance of residual network , or ResNet , models with 18 , 34 , and 50 layers ( He et al. , 2016 ) . ResNet is a powerful architecture that was designed to reduce the number of required training samples for deep learning tasks and has demonstrated excellent efficacy in image classification . Each ResNet model was compared when trained from scratch and when fine-tuned using pretrained ImageNet weights . Following the practice in He et al . ( 2016 ) , the final convolutional layer is followed by a global average pooling layer , then by a single fully-connected layer with softmax activation and as many units as class labels in the respective task . Models were trained using stochastic gradient descent ( SGD ) optimization with a linearly decaying learning rate ( given by 0.999 ( 1 − s/2 ) + 0.001 where s is the training step ) and Nesterov momentum of 0.9 . The models were trained for a maximum of 200 epochs but were stopped early if the loss on the validation dataset did not improve by at least 0.01 over 20 epochs . The number of epochs is limited in order to keep the ResNet training times comparable to SVMnet . The resulting accuracy and training time for each model was averaged over 5 repetitions of each experiment . While the height and width of inputs can be adjusted for ResNet , the architecture always expects 3-channel RGB color images . Grayscale images were modified for use by ResNet by duplicating the pixel values into three equal channels . This approach was used in Section 3.3 and Section 3.4 . Before training and classification by ResNet , images were also passed through a preprocessing filter provided by the Keras library to prepare the data for ResNet models . All inputs were normalized by dividing by the mean and subtracting the variance before being used to train SVMnet . For RGB color inputs , the images were normalized per-channel . All experiments and analysis presented in this section used the same hardware environment . SVMnet was parallelized across 16 cores of Intel Xeon Gold 6130 CPUs , and ResNet models were trained on an nVidia GeForce GTX 2080 GPU .

Summary: The paper argues that one of the major drawbacks for deep convolutional neural networks (DCNNs) is the need for large annotated training sets. To address this drawback, the paper proposes a new architecture, SVMnet, which is designed to achieve relatively high accuracy  (compared to DCNNs) in settings of small training sets.  The SVMnet architecture is composed of one or more stacked "SVM layers". Each SVM layer is composed of a set of independent svm classifiers, where the input to each svm is a patch in the image and the output is a probability estimate for the image class. For example, given a grayscale h X w input image, an SVMnet with 5x5 kernels and stride of 5 would have a 2d “array” of h/5 X w/5 svms, each one trains on a patch of 5x5 pixels whichis flattened to a vector of 25 features. Each svm i is then trained directly to predict the class of the image, based on the 5x5 window, thus resulting in a probability vector for the 5x5 patch. From the validation step, we also obtain the average accuracy of each svm i, denoted by Ai, from which we can compute its weight when aggravating the predictions from all svms. In general each svm layer will have k X l svms, and thh output of the svm layer can be formulated either as a 2d array of k X l predictions or a 3d array of k X l X c of probabilities for each class (where c is the number of classes). When stacking a second SVM layer, the input to that layer is then the k X l X c probability map which is treated as a feature map. Finally, the predictions from the last SVM layer are tallied to produce a majority vote for the image. When training on images with multiple channels, for example RGB images, the channels are flattened to one vector, for example: a 3x5x5 patch would be flattened to a 75 dimensional feature vector.   To demonstrate the applicability of the proposed method, the paper compares SVMnet to ResNet on 3 publicly available datasets and claims to achieve superior performance to resnet in settings of limited training data as well as faster training time.