SVMnet: Non-parametric image classification based on convolutional SVM ensembles for small training sets
Authors: Hunter Goddard
Lior Shamir
Venue: ICLR 2022 Submitted (2022)
Keywords: machine learning
support vector machine
convolutional neural network

Abstract: Deep convolutional neural networks (DCNNs) have demonstrated superior power in their ability to classify image data. However, one of the downsides of DCNNs for supervised learning of image data is that their training normally requires large sets of labeled "ground truth" images. Since in many real-world problems large sets of pre-labeled images are not always available, DCNNs might not perform in an optimal manner in all real-world cases. Here we propose SVMnet -- a method based on a layered structure of Support Vector Machine (SVM) ensembles for non-parametric image classification. By utilizing the quick learning of SVMs compared to neural networks, the proposed method can reach higher accuracy than DCNNs when the training set is small. Experimental results show that while "conventional" DCNN architectures such as ResNet-50 outperform SVMnet when the size of the training set is large, SVMnet provides a much higher accuracy when the number of "ground truth" training samples is small.

Under review as a conference paper at ICLR 2022
SVMNET: NON-PARAMETRIC IMAGE CLASSIFICATION BASED ON CONVOLUTIONAL SVM ENSEMBLES FOR SMALL TRAINING SETS
Anonymous authors Paper under double-bline review
ABSTRACT
Deep convolutional neural networks (DCNNs) have demonstrated superior power in their ability to classify image data. However, one of the downsides of DCNNs for supervised learning of image data is that their training normally requires large sets of labeled "ground truth" images. Since in many real-world problems large sets of pre-labeled images are not always available, DCNNs might not perform in optimal manner in all real-worle cases. Here propose SVMnet method based on a layered structure of Support Vector Machine (SVM) ensembles for non-paraman con-parametric image classification. By utilizing the quick learning of SVMs compared to neural networks, the proposed method can reach higher accuracy than DCNNs when the training set small. Experimental results show that while "conventional" DCNN architectures such as ResNer-50 outperform SVMnet when the size of the training set is large, SVMnet provides much higher accuracy when the number of "ground truth" training samples is small.
INTRODUCTION
Deep convolutional neural networks (DCNNs) are powerful tools for multiple tasks of automatic image analysis, demonstrating paramount success and consequently gaining substantial popularity over the past decade. By analyzing the pixels directly, CNNs can be applied to various types of image content without the need to develop task-specific algorithms, and can easily be applied to a broad range of domains with excellent performance (Khan et al. 2020)..
One of the major weaknesses of modern DCNNs is their dependence on a large set of examples for training. Cutting-edge DCNNs have hundreds of layers, each with thousands of trainable parameters. For instance, the common ResNet-50 (He et al. 20166 contains over 2-106 artificial neurons. Therefore, to achieve meaningful performance and avoid underfitting, DCNNs normally rely on relatively large training sets.
Training DCNNs normally requires large datasets of labeled ground truth images. Commonly used datasets include benchmarks such as the Modified National Institute of Standards and Technology (MNIST) or ImageNet. These benchmark datasets provide tens of thousands of images with high- quality annotations for training deep CNNs and are commonly used for testing their performance. However, many cases real-world image classification problems, large datasets of clean, labeled ground truth are not available.
common solution to increasing the size of the training set data augmentation. in which different modifications of the images in the original dataset can create more training samples. However, that strategy can also lead to biases by overusing the same examples. In some cases transfer learning can be used to fine-tune neural networks using pre-trained models, but for domains with very small datasets for fine-tuning, the pre-trained models may remain too sensitive to their original task.
In many real-world cases large datasets are not available. For instance, in the biomedical domain the acquisition and anmotation of each image requires the use of costly medical instrumentation, tech- nicians, and medical staff who can annotate each sample manually. Additionally, human protection procedures and protocols are required for the acquisition each sample, making the preparation of large datasets less practical. Therefore, biomedical image datasets normally far smaller than the

---

Under review as a conference paper at ICLR 2022
modern datasets commonly used to train DCNNs. Rare cases also make it difficult to acquire a suitable training set (Zhang et al. 2019). The need for a large number of training samples is a practical downside of DCNNs, making it difficult to optimally in many real-world cases.
The problem of small training sets has been addressed in the past by using previous knowledge for a few shot training Zhang al. 2019) and even one-shot training (Fei-Fei al.) 2006 Wolf et al. 2009 Koch et al.) 2015: Vinyals et al. 2016; Bender et al.) 2018). This paper explores a new form of image classification cases when the number of samples is limited. Based on an ensemble composition of support vector machines (SVMs), the method can work with no prior knowledge, in a similar manner "standard" supervised machine learning. Inspired by CNN architecture, SVMnet utilizes a large number of small SVMs to quickly analyze image patches, structured in layers that allow for stacking or custom ensemble techniques. An SVM (Cortes & Vapnik] 1995) is less sensitive to high-dimensionality feature spaces (Chen et al.) 2005 Mukkamala et 20.2). and has the advantage learning from a relatively small number of training samples (Shin et al. [2055]
Byvatov et al. 2003 Paiva et al. 2018 compared to other supervised machine learning approaches.
ARCHITECTURE OF SVMNET
The proposed SVMne architecture is designed as a stacked ensemble of numerous simple SVM classifiers organized into one or more layers. Each layer is an array of SVMs which functions similarly to convolutional layer a CNN. Each SVM in layer is independent and all are assigned an equal-sized patch of the layer's input, referred to as window. Variable stride length and padding, as described in Chapter 2 Dumoulin & Visin 2018). , are specified as hyperparameters. Each input to the following layer is the output of one SVM.
When a layer is evaluated, each SVM in the layer trained on ground truth labels. The input to the SVM is the flattened portion of each input image that is within the SVM's window. Each pixel channel within the window is essentially treated as one input feature. For instance, 5 x window would create 25-feature SVM for grayscale input and 75-feature SVM for 3-channel RGB input. During this step, the SVMs may be given weights based on the accuracy of the fit, used for ensemble classification. Each SVM then predicts a class label or a vector of class probabilities for its window of each input, creating an input tensor for the next layer.
Figure [ shows simple layer in SVMnet. Each node in the layer is one SVM, trained using the ground truth labels for the input samples. The weights are determined based on the classification accuracy of the SVM compared the ground truth of the training set. The weight function is configurable and will be described later in this Section.
P1
SVM
SVM
W1
SVM
Inputs
SVM Layer
Predictions and Weights
Figure 1: Example of a simple weighted layer of SVMnet. Each node in the layer an SVM, trained with a subset of the inputs (pixels). Weight outputs are optional for given layer.
To produce one class label for each input. SVMnet may perform a weighted vote after the final layer. This vote combines the results of the final layer by treating each value as vote for that class label. If the final layer is weighted, these are used to weigh the votes in favor of SVMs with higher accuracy.
VVMet is not "network" in the same sense as CNNs and named as such mostly due analogy.
2

---

Under review as a conference paper at ICLR 2022
S. = =d
(I)
The total voting score S. of each class is calculated by Equation where A, is the accuracy score of SVM in the final layer, 77 the weight function, and P. the class label predicted by SVM That is, if the predicted label P; of SVM i is class the weighted score (A,)) is added to the vote for that class. The weight function emphasizes the predictions of the SVMs with higher accuracy during training. The class that has the highest score S. is chosen as the predicted label by the model for the given sample. The weight function 1j is configurable and our experiments is defined as m(x) x² where is the classification accuracy of the SVM determined during training.
While the layers support arbitrary estimators, here we use only support vector machines (SVM), hence the name SVMnet. The SVMs are trained with Radial Basis Function (RBF) kernel (Chang) & Lin 201 and scaling gamma value, and they continue to iterate until convergence with 0.001 tolerance. The ability choose different estimators in each layer can be compared to the ability to use different activation functions in the layers of neural networks.
Figure illustrates one possible two-layer SVMnet architecture. Each SVM in the first layer lyzes a specific patch each image and is fitted independently against ground truth labels. These SVMs then produce a vector of class probabilities for the same pixel region which forms the input matrix for the following layer. The SVMs in the second layer are fitted on a region of these proba- bilities and predict class label for the image. These labels are then tallied in final vote to produce one label for the input. The motivation for multiple layers is that secondary layers can in essence learn which of the SVMs in their window are more accurate or "trustworthy", as their predictions are being compared to ground truth labels in each layer.
3-channel Input Image
Predicted Class Probabiittes
Predicted Class Labels
Trained SVM
Trained SVM
Vole
Figure 2: Example SVMnet architecture containing two SVM layers and a class label vote.
2.1 DROPOUT
Not every patch is expected produce well-informed SVM. Some regions of the images, par- ticularly towards the edge, often lack the details necessary to distinguish samples from each other. This can cause the outputs of these SVMs low accuracy score of the SVM depressing the weight of its vote, if the low-information regions are large then enough inaccurate votes may overwhelm the more accurate votes. To help prevent this, dropout system is implemented for the vote tally.
noise in vote tally. Even with the expected
When using dropout, which SVMs drop are calculated when fitting SVMnet. First, the SVMs are ordered from the highest weight to the lowest. Votes are then cumulatively tallied one SVM at time with the accuracy of the votes measured between each tally. SVMnet then finds the global maximum accuracy of the cumulative tally. This marks the point where including the votes of the less-accurate SVMs lowers the overall accuracy of the tally, so those SVMs are marked for dropout and are not included in the final vote.
3 EXPERIMENTAL RESULTS
To test the efficacy of SVMnet compared to a "conventional" CNN, several experiments were per- formed using common, relatively small datasets. The purpose of SVMnet is not outperform CNNs
3

---

Under review as a conference paper at ICLR 2022
in the general case, but to achieve higher accuracy when the number of labeled training images is limited. Therefore, the experiments were made with different sizes of training sets to compare the classification accuracy the training set increases.
The performance of the SVMnet was compared to the performance of residual network, or ResNet, models with 18, 34, and 50 layers (He al. 2016). ResNet is powerful architecture that was designed to reduce the number of required training samples for deep learning tasks and has demon- strated excellent efficacy in image classification. Each ResNet model was compared when trained from scratch and when fine-tuned using pretrained ImageNet weights. Following the practice in He al.] (2016), the final convolutional layer is followed by a global average pooling layer, then by a single fully-connected layer with softmax activation and as many units as class labels in the respec- tive task. Models were trained using stochastic gradient descent (SGD) optimization with a linearly decaying learning rate (given by 0.999(1 s/2) + 0.001 where is the training step) and Nesterov momentum of 0.9. The models were trained for a maximum of 200 epochs but were stopped early if the loss the validation dataset did not improve by at least 0.01 over 20 epochs. The number of epochs is limited in order to keep the ResNet training times comparable to SVMnet. The resulting accuracy and training time for each model was averaged over 5 repetitions of each experiment.
While the height and width of inputs can be adjusted for ResNet, the architecture always expects 3 3-channel RGB color images. Grayscale images were modified for use by ResNet by duplicating the pixel values into three equal channels. This approach was used in Section 3.3 and Section 3.4 Before training and classification by ResNet, images were also passed through preprocessing filter provided by the Keras library to prepare the data for ResNet models. All inputs were normalized by dividing by the mean and subtracting the variance before being used to train SVMnet. For RGB color inputs, the images were normalized per-channel.
All experiments and analysis presented in this section used the same hardware enviromment. SVM- net was parallelized across 16 cores of Intel Xeon Gold 6130 CPUs, and ResNet models were trained on an Vidia GeForce GTX 2080 GPU.
3.1 COIL-100 OBJECT RECOGNITION
Columbia Object Image Library (COIL-100) is a common dataset used for basic object recognition (Nene et al. 1996). contains RGB color images of 100 different objects, each photographed 72 times at deg increments about the vertical axis. Background details were removed in all images and the objects are centered and enlarged to fill the frame. Some objects contained in this dataset include coffee mugs, small toy cars, and various fruits and vegetables.
The SVMnet in this experiment used one layer with a 25 x 25 window (giving each SVM 1875 input features) and a stride length of 7, followed by weighted vote with dropout. The SVMnet and ResNet models were fitted with 100-500 training images in increments of 100, each controlled to have an equal number of samples from each object. A separate subset of 200 images was used as validation data for ResNet models.
Figure 3 shows the results of this experiment. When fitted the smallest training set, containing only one example per object, SVMnet correctly predicted labels for over 60% of the remaining images. With the same training set, ResNet-50 showed about the same accuracy and only pretrained ResNet-34 exceeded SVMntt however, SVMnet was significantly faster to train in all cases.
3.2 IMAGENETTE
Imagenette fairly small, 10-class subset of the ImageNet dataset (Howard). Several versions of this dataset exist; here we use version 2 of the 160 px dataset with noiseless labels. Many of these images are rectangular with their shortest side scaled to 160 px. In this experiment, we symmet- rically zero-pad each image along its shorter axis to make it square, then downscale the images to have the same dimensions of 160 x 160 px.
The SVMnet used here contains one layer with a window size of 22 and stride length 7, followed by a weighted vote with no dropout. Imagenette is pre-divided into training and testing subsets containing 9,469 and 3,925 images, respectively. Models were trained using 20, 40, 80, 160, and
4

---

Under review as a conference paper at ICLR 2022
200
Ikairing Time (Socconid
0.8
150
Accuracy
0.7
Restves-88 Peetrained Aeskee
0.6
Restect 24
50
Pretrained
0.5
100
300
403
500
100
300
400
500
Training Samples
Training Samples
Figure 3: Test-set accuracy (left) and training time (right) of SVMnet and ResNet on COIL-100 images when fitted with different training set sizes.
320 images from the provided training set and evaluated using the provided testing set. An additional 100 images were selected from the training set validation data for the ResNet models.
Figure shows the results of this experiment. SVMnet achieved higher accuracy than all ResNet models for all training sets except the largest, where the ResNet-50 model pretrained with ImageNet weights improved drastically. The generally low accuracy of these models could be explained by the method used to conform each image to the same dimensions, which introduces a significant amount of empty space in many images. However, even under these conditions, SVMneet attained the highest accuracy in the least time for the smaller training sets.
0.6
Training Time (seconds)
100
0.5
Accuracy
Pretrained Reskvet-3
NES
0.3
Prerained Reshet-50 30
0.2
25
40
80
160
320
20
40
160
320
Training Samples
Training Samp es
Figure 4: Test-set accuracy (left) and training time (right) of SVMnet and ResNet on Imagenette when fitted with different training set sizes.
3.3 COVID-19 RADIOGRAPHY
During the COVID-19 pandemic, machine learning techniques have been applied to various kinds of data assist the medical community making accurate diagnoses (El-Din Hemdan al. 2020)] Gangloff et al. 2021 Li et al., 2020). During the early stages of disease outbreak, diagnostic data is expected be limited or sparse, making difficult train most kinds of machine learning models. A type of model capable of learning from small number of samples would be the most effective this time frame.
Here we apply SVMnet to a database of chest x-ray images from healthy patients and patients diagnosed with COVID-19 (Chowdhury et al.) 2020 Rahman et al. (02T). this experiment, only the images labeled as "Normal" and "COVID" are used. Images were downscaled to 128 x 128 pixels (approx. 43% the original size). An equal number of images were selected from each class, totaling 7232 samples. Models were fitted with 10, 20, 50, 100, and 200 training samples, with 50 separate images used as validation data for the ResNet models. The SVMnet uses two layers: the first with window size 19, stride 7, and class probability outputs; the second with window size and stride 5, followed by an unweighted vote. During the architecture experiments described in Section 3.7 the 2-layer SVMnet shown outperform the l-layer models for this dataset.
5

---

Under review as a conference paper at ICLR 2022
Figure 5 shows the results of this experiment. SVMnet was able correctly label between 64% and 78% of unseen x-rays depending on the number of training samples, but most ResNet models failed to make significantly accurate predictions. Only the 18- and 34-layer ResNet models trained from scratch approached the accuracy of SVMttt Additionally, S VMeeet was several times faster to train.
0.80
50
SVifnet
RewNen-11
Training Time (seconds)
40
Res
0.70
Pretrolhed ResNet 34
Accuracy
30
Pretraiteed Seatel--0
ResSer:50
0.65
0.60
20
0.55
10
0.50
100
200
10
50
100
200
Training Samples
Training Samples
Figure 5: Test-set accuracy (left) and training time (left) of SVMnet and ResNet COVID-19 chest x-ray images when fitted with different training set sizes. The accuracy of the ResNet models displays considerable overlap.
3.4 ASTRONOMICAL IMAGE DATA
To test the performance of SVMnet current real-world image classification problem, dataset of galaxy images from the Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) was used. The dataset is made of galaxies separated into elliptical and spiral morphology. The galaxy images were taken from the catalog of Pan-STARRS galaxies classified by their morphological type (Goddard & Shamir 2020).
An equal number of images were selected of each morphological type, totaling 26,732 samples. Each image is grayscale and has dimension of 120 x 120 px. SVMnet and ResNet models were fitted with 10, 20, 40, 80, 160, and 320 training samples, with 200 separate images used as validation data for the ResNet models. The SVMnet uses layer with window size of 22 and stride 5, followed by weighted vote with dropout.
0.9
0.8
0.6
SWWeet
Training
0.5
10-
Aestee-10
Pretrained Recitee-
10
160
40
160
320
Trining Samples
Training Samples
Figure 6: Test-set accuracy (left) and training time (right) of SVMnet and ResNet on Pan-STARRS galaxy images when fitted with different training set sizes.
Figure shows the results of this experiment. As the graph shows, SVMnet outperformed almost every ResNet model when trained with relatively small dataset. The models generally improve as the training set grows, with several ResNets slightly overtaking SVMnet with the largest training set. In all cases, SVMnet finished training many times faster than all ResNet models.

---

Under review as a conference paper at ICLR 2022
3.5 WND-CHARM
To test "traditional" approach of using SVM after extracting image features, we used the WND- CHARM open source feature set (Shamir et al. 2008) combined with an SVM with linear kernel implemented through SVMLib. Table compares the test set accuracy of net using the experimental datasets described earlier this Section. WND-CHARM was trained on equal sized training subsets and consistently showed lower classification accuracy than SVMnet under the same conditions.
and SVM-
Imagenette
COIL-100
WND-CHARM
SVMsel
WND-CHARM
20
11%
16%
100
54%
62%
40
13%
19%
200
59%
70%
80
16%
24%
300
61%
75%
160
18%
26%
400
64%
78%
320
21%
31%
Pun-STARRS WND CHARM
COVID-19
SVMaet
WND-CHARM
SVMnet
10
52%
71%
10
53%
64%
20
56%
79%
20
55%
69%
40
61%
77%
50
60%
71%
80
63%
90%
100
64%
77%
160
72%
91%
200
66%
78%
320
88%
91%
Table Comparison of the classification accuracy WND-CHARM and SVMnet when trained on small number of samples from four datasets.
3.6 COMPUTATIONAL COMPLEXITY
The complexity fitting an SVM is asymptotic and polynomial. For a training set containing n samples, the algorithm is dominated by either an n² term or an n² term based on the formulation of the problem (Bottou & Lin 2007). Therefore, training large number SVMs can a computa- tionally demanding task, and can lead to substantial computational complexity during training.
The number of SVMs N a layer receiving rectangular input with width Lg and height 4y is given by Equation 2 The window size W (equivalent the kernel size in other CNN literature), stride length S, and padding amount P in their respective dimensions follow from standard convolutional arithmetic.
I + 2P. -W +1) (Ty + 2P, +2, S
W
+ +1)
S
Fitting a layer in SVMnet requires fitting N SVMs polynomial time operation. If the layer includes weights, then the SVMs must predict class label for each input during the fit step, which scales linearly with the number of samples n. When using dropout as described in Section 2.] SVMnet performs an additional step during training that scales linearly with n. Thus, fitting SVMnet dominated by the polynomial fit time the SVMs.
CNNs thcoretically be trained infinitely, but there is definitive point at which the SVMs within SVMnet converge. This places a soft upper bound on the training time of SVMnet based on the tolerance parameter of the SVMs. Additionally, firm upper bound may be placed on the number of iterations the SVM algorithm, allowing for a shortened training time at the expense of some accuracy.
SVMnet trains multiple SVMs simultancously using process-base parallelism and shared memory, greatly increasing its speed typical multicore computers with minimal overhead. While this

---

Under review as a conference paper at ICLR 2022
allows SVMnet quite easily relatively inexpensive systems, the potential performance gain from extra hardware minimal compared to the extreme optimization of CNNs for GPU devices.
3.6.1 INFERENCE TIME OF IMAGE CLASSIFICATION
Predicting single class label of image using SVMnet typically requires large number of indi- vidual SVMs to predict label followed by vote tally. Despite its affinity for parallelization, this process is expected to take longer than the highly optimized matrix operations of CNN. Table compares the inference time of SVMnet and ResNet on images in the COIL-100 dataset.
10
100
1000
SVMnet
2.36
2.66
3.81
24.2
ResNet- 18
0.054
0.056
0.082
0.296
ResNet-33
0.060
0.060
0.098
0.410
ResNet-50
0.061
0.064
0.
0.515
Table 2: Comparison the response time (in seconds) of SVMnet and ResNet to predict class labels for 1, 10, 100, and 1000 samples of the COIL-100 dataset.
The comparison shows that SVMnet is significantly slower than ResNet for classifying samples, but the speed of classification is still practical for many real-world systems. The parallelization of SVMnet greatly reduces the time needed to make predictions, but the overhead of shared memory operations is significant in the case of few samples.
3.7 ARCHITECTURE COMPARISON
As with CNNs, SVMnet can be configured into a variety of architectures which are expected differ in performance depending the classification task. Due to the high number of possible models, determining which the most effective for a single task is non-trivial. In this section we show how variety of SVMnet configurations were tested on the COIL-100 dataset to inform the choice of model used Section 3.1 Similar methods were used to select the models for other datasets. SV models with multiple layers were tested in the same manner.
Unweighted Vote
Weighted Vote
Weighted Vote with Dropout
0.8
0.7
hpp
0.6
4g
1
0.5
0.4
7
10
13
16
19
22
25
7
10
13
16
19
22
25
10
13
16
19
22
25
Window Size
Window Size
Window Size
Prediction accuracy of one-layer SVMnet architectures fitted to COIL-100. Each group of three box plots represents the same window size with stride length 3, 5, and 7, respectively. Each box plot shows the distribution in model accuracy when using five training sets of 200-1000 examples.
Figure 7:
Figure 7 shows how the performance of a one-layer SVMnet changes with the window size, stride length, voting method, and number training samples when fitted to COIL-100, Prediction accu- racy improves in all cases the window size increases but with diminishing returns. Increasing the stride length tends to lower accuracy when the window small but incurs little to no penalty when the window is large. When the vote of an SVM is weighted, model accuracy improves in all cases compared to an unweighted vote; performance increases further when using dropout as described in Section 2.1 This effect is significant when the window size is small.
8

---

Under review as a conference paper at ICLR 2022
4 CONCLUSION
Deep convolutional neural networks provide excellent performance in automatic classification of image data while eliminating the need to develop and tailor algorithms for specific image classifica- tion problems. With the availability of libraries, DCNNs have become the de facto first solution to image classification.
Here we explore one of the primary weaknesses of DCNNs, which is the need of a relatively high number of labeled "ground truth" samples for effective training of the network. While DCNNs are often tested on relatively large datasets such as MNIST or ImageNet, in many real-world problems a very large number of clean labeled samples that can be used for training is not available.
In many other cases labeled training samples are not available. For instance, when analyzing archae- ological artifacts, the number of training samples are limited by the number available artifacts, which is hard limit that cannot be easily changed. Using computer vision to analyze art (Khan jet al. 2014) is limited by the number of paintings each artists created, which can be a firm limit, especially when the painter no longer alive.
SVMnet aims at providing an effective solution for the numerous real-world situations in which the number of labeled image samples that can be used for training is limited. SVMnet utilizes the ability of an SVM to learn from a smaller number of samples compared to other machine learning approaches. The flexible structure of SVMnet allows it to learn directly from the pixel values, and to utilize different layers that correspond to the convolutional and fully connected layers in "conventional" deep neural networks. Like DCNNs, SVMnet does not require the design of specific algorithm for a specific image classification problem.
The proposed approach structured as network take advantage of the stronger signal from neighboring pixels, similar to the core idea in the basis of CNNs. SVMs are known for their ability to learn quickly from relatively few training samples. By training many SVMs on small pixel regions across an image, this quick learning can be leveraged to extract much information from small sets of images in less time than would take fully train a deep neural network.
Complexity analysis shows that the training time for SVMnet scales more quickly with the number of input samples than DCNNs, suggesting that SVMnet might take substantial computational re- sources when trained using large datasets. However, SVMnet is designed for situations in which the labeled training set is relatively small. As shown in our experiments, the training time might not be practical obstacle in many real-world situations in which SVMnet can be used. While computing is an available resource, and training SVMnet with a few hundred training samples scales within reasonable response time, annotated clean or rare training samples might in many cased be much more difficult to obtain.
The underlying structure used to create SVMnet is very flexible, allowing other kinds of machine learning algorithms to be used rather than solely SVMs. Constructing the layers with classifiers such as random forests or logistic regression may result in better performance for some datasets. These layers can be mixed in the same model as well, i.e. using one layer of SVMs followed by a layer of random forests. These possibilities present a promising avenue for future related work.
SVMnet is not designed to become general solution that can outperform deep convolutional neural networks such as ResNet-50, but experimental results show that it is an effective solution for cases in which the number of labeled training samples is small. Since such cases are not rare, SVMnet can complement "conventional" deep neural networks by providing image classification in the cases where not many labeled training samples are available.
ACKNOWLEDGEMENTS
The research was funded by NSF grant number XXXXXX.
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. In International Conference Machine Learn-

---

Under review as a conference paper at ICLR 2022
ing. pp- 530-559. PMLR, 2018.
Léon Bottou and Chih-Jen Lin. Support vector machine solvers. Large scale kernel machines, 3(1): 301-320, 2007.
Evgeny Byvatov, Uli Fechner, Jens Sadowski, and Gisbert Schneider. Comparison of support vector machine and artificial neural network systems for drug/nondrug classification. Journal of chemi- cal information and computer sciences, (816):1882-1888 2003.
Chih-Chung Chang and Chih-Jen Lin. Libsvm: library for support vector machines. ACM trans-
actions on intelligent systems and technology (TIST), 2(3):1-27, 2011.
Wun-Hwa Chen, Sheng-Hsun Hsu, and Hwang-Pin Shen. Application of svm and ann for intrusion detection. Computers Operations Research, 32(10):2617-2634, 2005.
Muhammad E. H. Chowdhury. Tawsifur Rahman, Amith Khandakar, Rashid Mazhar, Muham- mad Abdul Kadir, 7aid Bin Mahbub, Khandakar Reajul Islam, Muhammad Salman Khan, Atif Iqbal, Nasser Al Emadi, Mamun Bin Ibne Reaz, and Mohammad Tariqul Islam. Can ai help in screening viral and covid-19 pneumonia? IEEE Access, 8: 8:132665-1326777 2020. doi: 10. 109//CS/SSSSSS... TTT.///AAEESSS20.....O.... 2020.3010287.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-297, 1995.
Vincent Dumoulin and Francesco Visin. A guide to convolution arithmetic for deep learning, 2018.
Ezz El-Din Hemdan, Marwa A. Shouman, and Mohamed Esmail Karar. COVIDX-Net, A Frame- work of Deep Learning Classifiers to Diagnose COVID-19 in X-Ray Images. arXiv e-prints, art. arXiv:2003.I 1055, March 2020.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions pultern analysis and machine intelligence, 544)),994611, 2006.
Cedric Gangloff, Sonia Rafi, Guillaume Bouzillé, Louis Soulat, and Mare Cuggia. Machine learning the key diagnose covid- 19: a proof oo-oncet-- study. Scientific Reports, 11(1):716,, Mar 2021. ISSN 2045-2322. doi: 10.
URL btctps://doi.org//
1038/ 1038/s41596 021 86735-9
Hunter Goddard and Lior Shamir. catalog of broad morphology of pan-starrs galaxies based on deep learning. The Astrophysical Journal Supplement Series, 251(2):28, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Jeremy Howard. Imagenette. URL hrRJl/ttgs/ggtttu..ooo////aa//as
Asifullah Khan, Anabia Sohail, Umme Zahoora, and Aqsa Saeed Qureshi. survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review, 53(8):5455- 5516, 2020.
Fahad Shahbaz Khan, Shida Beigpour, Joost Van de Weijer, and Michael Felsberg. Painting-91 a large scale database for computational painting categorization. Machine Vision and Applications, 1385) 2014.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
Wei Tse Li, Jiayan Ma, Neil Shende, Grant Castaneda, Jaideep Chakladar, Joseph C. Tsai, Lauren Apostol, Christine O. Honda, Jingyue Xu, Lindsay M. Wong, Tianyi Zhang, Abby Lee, Aditi Gnanasekar, Thomas K. Honda, Selena Z. Kuo, Michael Andrew Yu, Eric Y. Chang, Mahadevan Raj" Rajasekaran, and Weg M. Ongkeko. Using machine learning of clinical data to diag- covid-19 19; systematic review and meta-analysis. BMC Medical Informatics and Deci- sion Making, 20(1):247, Sep 2020. ISSN 1472-6947, doi: 10. 1111///12911020--11111 https: / .
URL
0166--
10

---

Under review as a conference paper at ICLR 2022
Srinivas Mukkamala, Guadalupe Janoski, and Andrew Sung. Intrusion detection using neural net- works and support vector machines. Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 02 (Cat. No. 02CH37290), volume 2, pp. 1702-1707. IEEE, 2002.
Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-100). 1996.
Joana S Paiva, João Cardoso, and Tânia Pereira. Supervised learning methods for pathological arterial pulse wave differentiation: svm and neural networks approach. International Journal of Medical Informatics, 109:30-38, 2018.
Tawsifur Rahman, Amith Khandakar, Yazan Qiblawey, Anas Tahir, Serkan Kiranyaz, Saad Bin Abul Kashem, Mohammad Tariqui Islam, Somaya Al Maadeed, Susu M. Zughaier, Muhammad Salman Khan, and Muhammad E.H. Chowdhury. Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images. Computers in Biology and Medicine, 132: 2021. ISSN 0010-4825. doi:
http:////w..oooooooooooooooooom/ www www /twp://ww.acccaccccccoc/cooo/ooa
URL https:
Lior Shamir, Nikita Orlov, D Mark Eckley, Tomasz Macura, Josiah Johnston, and Ilya G Goldberg. Wndchrm-an open source utility for biological image analysis. Source Code for Biology and Medicine, 3(1):1-13, 2008.
2005. Kyung-Shik Shin, Taik Soo Lee, and Hyun-jung Kim. An application of support vector machines in bankruptcy prediction model. Expert Systems with Applications,
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29:3630-3638, 2016.
Lior Wolf, Tal Hassner, and Yaniv Taigman. The one-shot similarity kernel. 2009 IEEE 12th International Conference Computer Vision, pp. 897-902. IEFE, 2009.
Ansi Zhang, Shaobo Li, Yuxin Cui, Wanli Yang, Rongzhi Dong, and Jianjun Hu. Limited data rolling bearing fault diagnosis with sho-shot learning. IEEE Access, 7:1 10895-1111111 10004, 2019.


Reviews:
[{"review_id": "wLdrXaY3xs", "reviewer": "Reviewer_SXM2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenghts:\n- the paper presents an ambitious idea by designing SVM classifiers that operate locally. The class probabilities are organized in tensors in a \"second layer\" which implements ideas from classifier combination (stacked classifiers)\n\nWeaknesses:\n- the SVMNet is not strictly a \"multi-layer\" network since each layer and each local classifier is trained independently. Although the outputs are \"connected\" in some way there is no joint training or any \"chained\" learning. Therefore, claiming it is a network it is a bit farfetched. \n- the baselines are not good: training from scratch is clearly not the way to go when it comes to ResNets. I don't think the authors can just ignore the existence of pretrained weights. Another baseline would be to train a 2-layer Dense network (with the actual pixels) and a 2-layer CNN network, which would resemble much more the proposal.\n- the choice of the datasets is also debatable: why not using MNIST, Fashion-MNIST, ImageNette, or others, which have many results reported in previous papers and would allow for a better comparison?\n- training time is emphasized in the first part of the paper, but in fact this is not critical: inference time would be much better as a comparison. In fact, Table 1 confirms this drawback", "summary_of_the_paper": "The paper describes the use of SVM classifiers trained independently using local receptive fields of images, outputting class probabilities for each pixel that are organized in channels. New independent SVM classifiers are trained over the class probabilities and their results are combined using voting to perform the final prediction. The authors report experiments on two datasets, comparing with versions of ResNet trained from scratch, in which SVMNet shows better results than ResNet when using fewer training examples, For one of the datasets, the ResNet could not converge, while SVMNet obtained up to 80%.", "summary_of_the_review": "The paper presents an idea of classifier combination much more than an actual \"network\". The concepts borrowed from multiple-classifier systems and related fields are not novel. Also, the baselines were not adequate, and the choice of datasets to compare was limited. Therefore, I cannot recommend this paper to be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "pyPEPXiQnEk", "reviewer": "Reviewer_8kHE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will divide my review into three sections, addressing different aspects in the paper: motivation, method and experiments.\n\nMotivation: The paper argues that in order to achieve high performance and avoid overfitting, DCNNs generally require very large labeled training sets. While this claim is true in general, there is a wide array of methods (e.g.: pre-training, self-supervised learning, training on artificially generated samples) previously proposed to address this drawback and I would expect at least some discussion (and experiments) illustrating why the proposed method is important given previous work and how it differs. The paper does mention data augmentation and claims that data augmentation can cause overfitting to overused samples, but this is not true if augmations is done and evaluated correctly (nor is there an experiment demonstrating the claim). The paper also mentioned transfer learning as a solution, but doesn’t discuss it further. Moreover, the paper simply states that the need for large training sets makes DCNNs difficult to use in an optical manner in many real world cases, which is simply not true given the wide applicability of DCNNs in real world applications across numerous fields including medical imaging, autonomous driving, retail, etc. Overall, the motivation for the proposed method and its importance compared to existing work was not well founded.\n\nMethod:\nIn my opinion, there are a few inherent drawbacks to the proposed SVMnet method:\nEach SVM corresponds to a specific fixed location in the image. This means that the architecture is sensitive to translation changes or intra class variability (of the location of the object in the image).\nThe basic building blocks of the architecture are svms which are fed with image patches and instead of being modeled as feature extractors (as in DCNNS), are models as class level classifiers. The result of the first SVM layer are patch level class probabilities, meaning the raw low level visual information of the image is lost (or more precisely encoded only by the output probabilities). Thus, it’s not clear how the svm layers can be stacked in a meaningful way, other than weighing local patch classifiers (and there are no experiments that demonstrate the effectiveness of stacking more than 2 svm layers).\n\nHowever, it’s worth mentioning that trying to model a network as an ensemble of weak classifiers does make sense, but in my opinion the method has inherent drawbacks and it’s superiority is not demonstrated by the experiments presented. \n\n\nExperiments:\nThe authors compare the proposed method against ResNet and claim to achieve competitive performance. However, there are a few major issues in the way the comparison and the experiments were done.\n\nFirst, the authors state that when training resnet the learning rates for each experiment were set differently and the network was trained for 50 epochs. This value, which dramatically affects the ResNet experiment results, seemed to be set arbitrarily and no method for setting this value (using a validation set used for early stopping, training until convergence, treating the number of epochs as a hyperparameter and optimizing it) is mentioned or explained.\n\nSecond, important implementation details for training the ResNet are not mentioned and it seems that no hyperparameter optimization was done, again hindering ResNet performance (which the proposed SVMnet is compared against). \n\nThird, regarding training times - the authors claim to obtain faster running times using SVMNet. However, with having the number of epochs arbitrary set to 50 in ResNet (where SVMNet is not trained in epochs), not mentioning the batch size, checking simply for 50 epochs without considering convergence, early stopping or any meaningful criteria, it’s hard to make sense of the comparison in training times. Also, it’s not clear how the CPU hardware used to train SVMnet compares against the GPU used to train ResNet. \n\nFourth, the authors compare against ResNet 18, 34 and 50 in settings where the number of training samples is low. However, in such situations, due to risk of overfitting, it makes much more sense to consider architectures with fewer parameters, i.e: fewer layers. Furthermore, in the COIL-100 experiments, it seems that ResNet 18 performs best of the ResNet variants where the performance degrades as the number of layers increases which makes sense since the number of samples is small and ResNets with more layers have more parameters, thus are more prone to overfitting. From this experiment, it seems that a better comparison would be against a ResNet (or any other DCNNs) with fewer parameters (e.g.: fewer layers).\n\nFifth, COIL 100 is a very constrained dataset, not depicting ‘in the wild’ variability. Specifically, the dataset does not depict large translation variability or large variability in the location of the object in the image, which seems to be the drawback of the proposed architecture. In order to properly evaluate the method, I would recommend using a more challenging benchmark. \n\nSixth, in the COVID-19 Radiography, all resnet variants receive 50% accuracy (in a binary problem), regardless of the number of samples used for training ,since they predicted the same class for all test images, meaning they are no better than random. This clearly looks like a bug, especially since the very paper cited which presents the benchmark demonstrates almost perfect accuracy for this benchmark with ResNet 18 (given of course the entire training set, but this is evidence that the model does work). The same problem occurs in the Astronomical Image Dataset where when training with 10-80 samples the models achieve random 50% accuracy, but when training with 120 images the performance suddenly increases to 70%-90%.  \n", "summary_of_the_paper": "The paper argues that one of the major drawbacks for deep convolutional neural networks (DCNNs) is the need for large annotated training sets. To address this drawback, the paper proposes a new architecture, SVMnet, which is designed to achieve relatively high accuracy  (compared to DCNNs) in settings of small training sets.\n\nThe SVMnet architecture is composed of one or more stacked \"SVM layers\". Each SVM layer is composed of a set of independent svm classifiers, where the input to each svm is a patch in the image and the output is a probability estimate for the image class. For example, given a grayscale h X w input image, an SVMnet with 5x5 kernels and stride of 5 would have a 2d “array” of h/5 X w/5 svms, each one trains on a patch of 5x5 pixels whichis flattened to a vector of 25 features. Each svm i is then trained directly to predict the class of the image, based on the 5x5 window, thus resulting in a probability vector for the 5x5 patch. From the validation step, we also obtain the average accuracy of each svm i, denoted by Ai, from which we can compute its weight when aggravating the predictions from all svms. In general each svm layer will have k X l svms, and thh output of the svm layer can be formulated either as a 2d array of k X l predictions or a 3d array of k X l X c of probabilities for each class (where c is the number of classes). When stacking a second SVM layer, the input to that layer is then the k X l X c probability map which is treated as a feature map. Finally, the predictions from the last SVM layer are tallied to produce a majority vote for the image. When training on images with multiple channels, for example RGB images, the channels are flattened to one vector, for example: a 3x5x5 patch would be flattened to a 75 dimensional feature vector. \n\nTo demonstrate the applicability of the proposed method, the paper compares SVMnet to ResNet on 3 publicly available datasets and claims to achieve superior performance to resnet in settings of limited training data as well as faster training time. ", "summary_of_the_review": "In my opinion, the experiments presented in the paper fail to demonstrate or give any meaningful evidence to the claims made by the authors regarding the proposed method. While in my opinion the method has inherent drawbacks, the true flaw is the lack of experimental evidence which is a major issue. Also, I believe the claims could have been better supported with a wider discussion demonstrating how the proposed method relates to other previously published work on training deep models in settings with limited training data. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "QSUkS2hDKVI", "reviewer": "Reviewer_ycpg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The problem studied is relevant and importnat to solve. For many applications the amount of labeled data available is small.\n- The studied idea is interesting and result are encouraging.\n- Sensitivity analysis was performed for the window/patch size parameter.\n- Authors tested the approach in multiple datasets.\n- The paper is easy to follow.\n\nWeaknesses:\n- The evaluation framework is inadequate for the proposed apporach. It is expected for neural networks trained with very small data to quickly overfit. More so the larger the network architecture is, and it is unfair to compare that setup to an SVM ensemble. How is the performance of the neural network when you do finetuning of the last few layers from imagenet pretrained weights, or under stronger regularization? All those are standard approaches used by practitioners. \n- Simple baselines like having an svm classifier or an ensemble of SVMs trained on a feature representation of the imagery are missing\n- SVMnet's Inference time is much higher compared to standad neural networks.\n- There is a large line of work on few shot learning, N shot learning, and even zero shot learning that focuses on this specific problems that are not mentioned in this work. Comparing with those would approaches provide a better evaluation framework. \n- The reason to set up multiple layers of of SVM trained only on local patches is not explained beyond emulating neural networks. ", "summary_of_the_paper": "In this work auhors proposed an ensemble of multiple SVM classifiers setup in a hierarchical way (somewhat similar to neural networks). Each SVM is trained is a small patch or window from the input imagery and some classifiers can be eliminated from contributing to the predictions. Results show better performance by the model in the small data regime compared with larger convolutional neural networks trained from scratch.", "summary_of_the_review": "I think the idea proposed is interesting and tackles an important problem, however the it lacks a proper evaluation setup. I do not think the paper is ready for publication and I recommend the authors to retrink the evaluation framework and add relevant baselines and method for comparison.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2cCYp2tjrw", "reviewer": "Reviewer_ZeYm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the author illustrates the following progress:\n1. They propose a novel deep learning architecture that uses ensemble of numerous simple SVM classifiers as network layers.\n2. They design a voting system for classification along with the SVMNET.\n3. In the experiment, they compare the performance of SVMNET with ResNet models with 18, 34, and 50 layers on 3 different datasets. They compare the training time, accuracy versus the number of training samples between these methods and prove the main benefit of the SVMNET.\n\nQuestions remain as follows:\n1. From the result, generally while the training samples is larger than 200, SVMNET doesn't outperforms the ResNet models except for the \nCOVID dataset. For most cases, such amount of training data is not so difficult to generate.  So the advantage of SVMNET might be limited.\n2. According to the paper, all ResNet models were trained from scratch rather than starting with pre-trained weights derived from ImageNet. In some ways this might be unnecessary since the pre-trained model is available in practice. It would be better if the author can show the comparison result with pre-trained ResNet models.", "summary_of_the_paper": "One of the weakness of traditional DCNNs is it needs large clean-labeled dataset. In this paper, the author proposes SVMNET, a deep learning architecture which includes a layered structure of Support Vector Machine (SVM) ensembles. The result shows that the SVMNET outperforms other deep convolutional neural networks such as ResNet-50 with less training time for cases in which the number of labeled training samples is small. \n", "summary_of_the_review": "In this paper, the author proposes a novel deep learning architecture that uses ensemble of numerous simple SVM classifiers as network layers. Their result shows that the SVMNET outperforms other deep convolutional neural networks such as ResNet-50 with less training time for cases in which the number of labeled training samples is small. \nAccording to the experiment result, it supported the statement to some extent. For two of the three datasets, SVMNET has better performance when the training samples are less than 200 compared with ResNet models, but the ResNet catch up to the SVMNET quickly after the sample increases. Also it would be better if they can do comparison with pre-trained ResNet and show the results in their paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to overcome the challenge of annotating datasets to train convolutional networks by considering instead an architecture that is composed of stacked support vector machine layers. Each support vector machine is trained on a small patch from the input image. A voting mechanism is used to aggregate the predictions. Results show better performance by the model in the small data regime compared with larger convolutional neural networks trained from scratch.

The reviewers appreciated the relevance of the problem and the originality of the approach. The reviewers also appreciated several parts of the experimental evaluation that were carefully conducted in particular the sensitivity of the analysis with respect to the patch size and the multiple datasets considered for the experimental evaluation. The reviewers also expressed concerns about the adequacy of the evaluation (unfair comparisons), the completeness of the baselines (missing baselines), and the significance of the improvements. In particular, the experimental evaluation was considered too limited given the problem considered.

The authors submitted responses to the reviewers' comments. After reading the response, updating the reviews, and discussion, the reviewers considered that ‘the experimental evaluation improved a bit', that several concerns were satisfactorily addressed, and yet that the updated results 'do not show a significant improvement of the proposed method over existing works, simple baselines or pre-trained ResNet'. We encourage the paper to pursue their approach further taking into account the reviewers' comments, encouragements, and suggestions. The revision of the paper will generate a stronger submission to a future venue.

Reject.
Decision: Reject