Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
Authors: Sanjeev Arora
Simon S. Du
Zhiyuan Li
Ruslan Salakhutdinov
Ruosong Wang
Dingli Yu
Venue:  (2020)
Keywords: small data
neural tangent kernel
UCI database
few-shot learning
kernel SVMs
deep learning theory
kernel design

Abstract: Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under l2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018). An efficient algorithm to compute the NTK, as well as its convolutional counterparts, appears in Arora et al. (2019a), which allowed studying performance of infinitely wide nets on datasets like CIFAR-10. However, super-quadratic running time of kernel methods makes them best suited for small-data tasks. We report results suggesting neural tangent kernels perform strongly on low-data tasks.
1. On a standard testbed of classification/regression tasks from the UCI database, NTK SVM beats the previous gold standard, Random Forests (RF), and also the corresponding finite nets.
2. On CIFAR-10 with 10 – 640 training samples, Convolutional NTK consistently beats ResNet-34 by 1% - 3%.
3. On VOC07 testbed for few-shot image classification tasks on ImageNet with transfer learning (Goyal et al., 2019), replacing the linear SVM currently used with a Convolutional NTK SVM consistently improves performance.
4. Comparing the performance of NTK with the finite-width net it was derived from, NTK behavior starts at lower net widths than suggested by theoretical analysis(Arora et al., 2019a). NTK’s efficacy may trace to lower variance of output.

Published as a conference paper at ICLR 2020
HARNESSING THE POWER OF INFINITELY WIDE DEEP NETS ON SMALL-DATA TASKS
Sanjeev Arora Princeton University arora0cs. wrw.:esss..innnttoo.eedeee
Simon S. Du Institute for Advanced Study esdu@ias edd
Zhiyuan Li Princeton University
Ruslan Salakhutdinoy
Carnegie Mellon University rsalakhu@cs cmu. .ed.
tbiyyanni@ee.pprncettto edu
Ruosong Wang
Dingli Yu
Carnegie Mellon University
Princeton University
ruosongwaandres cmu. edu
dingliyecs princeton.edd
ABSTRACT
Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under & loss by gradient descent with in- finitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al. 2018). An efficient algorithm compute the NTK, as well as its convolutional counterparts, appears in Arooro et al.) (2019a), which allowed studying performance of infinitely wide nets on datasets like CIFAR-10. 10. However, super-quadratic running time of kernel methods makes them best suited for small-data tasks. We report results suggesting neural tangent kernels perform strongly on low-data tasks.
1. On standard testbed of classffection/regrrssinn tasks from the UCI database, NTK SVM beats the previous gold standard, Random Forests (RF), and also the corresponding finite nets.
2. On CIFAR- 10 with 10- 640 training samples, Convolutional NTK consistently beats ResNet-34 by 1 -%%.
3. On VOCO7 testbed for few-shot image classification tasks on ImageNet with transfer learning (Goyal et al. 2019). replacing the linear SVM currently used with Convolutional NTK SVM consistently improves performance.
4. Comparing the performance of NTK with the finite width net was derived from, NTK behavior starts at lower net widths than suggested by theoretical malysis(Arore et al. 2019a). NTK's efficacy may trace to lower variance output.
INTRODUCTION
Modern neural networks (NNs) have way more parameters than training data points, which allow them to achieve near-zero training error while simultaneously - for some reason yet to be under- stood have low generalization error (Zhang al.) 2016). This motivated formal study of highly overparametrized networks, including networks whose width (i.c., number of nodes in layers, or number of channels in convolutional layers) goes to infinity. A recent line of theoretical results shows that with & loss and infinitesimal learning rate, in the limit of infinite width the trajectory of training converges to kernel regression with particular kernel, neural tangent kernel (NTK) (Jacol et al. 2018). For convolutional networks, the kernel is CNTK. See Section 2 for more discussions. Arora et al. (2019)) gave an algorithm to exactly compute the kernel corresponding to the infinite limit of varous realistic NN architectures with convolutions and pooling layers, allowing them to compute performance on CIFAR-10 10, which revealed that the infinite networks have 6 to 8% higher enror than their finite counterparts. This is still fairly good performance for fixed kernel.

---

Published as a conference paper at ICLR 2020
Ironically, while the plicated model - deep networks - to an older, simpler model - kernel regression - in practice the simpler model is computationally less efficient because running time of kernel regression can be quadratic in the number of data points! (1 Thus computing using CNTK kernel on large datasets like ImageNet currently appears infeasible. Even on CIFAR-10, it seems infeasible incorporate data augmentation.
analysis, at first sight, appears reduce the study of com-
However, kernel classifiers are very efficient on small datasets. Here NTKs could conceivably be practical while at the same time bringing some of the power of deep networks to these settings. We recall that recently Olson et al. (2018) showed that multilayer neural networks can be reasonably effective on small datasets, specifically on UCI testbed of tasks with as few as dozens of training examples. Of course, this required some hyperparameter tuning, although they noted that such tuning is also needed for the champion method, Random Forests (RF), which multilayer neural networks could not beat.
It is thus natural to check if NTK - corresponding infinitely wide fully-connected networks - performs well in such small-data task Convex objectives arising from kernels have stable solvers with minimal hyperparameter tuning. Furthermore, random initialization in deep network training seems to lead to higher variance in the output, which can hurt performance in small- -ata settings. Can NTK's do better? Below will see that in the setup of Olson et al. (2018), NTK predictors indeed outperforms corresponding finite deep networks, and also slightly beats the earlier gold standard, Random Forests. This suggests NTK predictors should belong in any list of off-the- shelf machine learning methods.
Following are low-data settings where we used NTKs and CNTKs:
In the testhed of 90 classification tasks from UCT database, NTK predictor achieves superior, and arguably the strongest classification performance. This is verified via several standard statistical tests, including Friedman Rank, Average Accuracy, Percentage of the Maximum Accuracy (PMA) and probability of achieving 90%/95% maximum accuracy (P90 and P95), performed compare performances of different classifiers on 90 datasets from UCI database. (The authors plan to release the code, allow off- off-the-shelf use of this method. It does not require GPUs.)
We find the performance of NN is close to that of NTK. On every dataset from UCI database, the difference between the classification accuracy of NN and that of NTK is within 5%. On the other hand, on some datasets, the difference between classification accuracy of NN (or NTK) and that of other classifiers like RF can be as high as 20%. This indicates in low-data settings, NTK is find NTK is more stable (smaller variance),
indeed good description of NN. Furthermore, which seems to help it achieve better accuracy on small datasets (cf. Figure
CNTK is useful in computer vision tasks with small-data. On CIFAR-10, we compare CNTK with ResNet using 10- 640 training samples and find CNTK can beat ResNet by 1% - 3%. We further study few-shot image classification task on VOCO7 dataset. The standard method to first use a pre-trained network, e.g., ResNet-50 trained on ImageNet, to extract features and then directly apply a linear classifier on the extracted features (Goyal et al., (219). Here we replace the linear classifier with CNTK and obtain better classification accuracy in various setups.
Paper organization. Section 2 discusses related work. Section 3 reviews the derivation of NTK. Section presents experiments using NN and NTK on UCI datasets. Jection&ppresen experiments using CNN and CNTK on small CIFAR-10 datasets. Section 6 presentss experiments using CNTK for the few-shot learning setting. Additional technical details are presented in appendix.
2 RELATED WORK
Our paper is inspired by [Fernández-eelgado et al. (2014) which conducted extensive experiments on UCI dataset. Their conclusion is random forest performs the best, which is followed by the SVM with Gaussian kernel. Therefore, RF may be considered as a reference "good-standard") to compare
'The bottleneck constructing the kernel, which scales quadratically with the number of data points Arora 2019a). The regression can take cubic time in the number of data points.
ctal. al.
Note that NTKs can also be used in kernel SVMs, which are not known to be equivalent to training infinitely wide networks. Currently, equivalence is only known for ridge regression. We tried both.
2

---

Published as a conference paper at ICLR 2020
with classifiers. Olson et al. (2018) followed this testing strategy to evaluate the performance of modern neural networks concluding that modern neural networks, even though being highly over- parameterized, still give reasonable performances these small datasets, though not strong as RFs. Our paper follows the same testing strategy to evaluate the performance of NTK.
The focus of this paper, neural tangent kernel is induced from neural network architecture. The connection between infinitely wide neural networks and kernel methods is not new (Neal, 1996]
Williams 1997. Roux & Bengio 2007; Hazan & Jaakkola, 2015 Lee et al. 2018 Matthews al.
2018 Novak et al. 2019] Marriga-Alonss et al. 2019) Cho & Saul 2009 Daniely et al. 2016
Daniely 2017 However, these kermels correspond to neural network where only the last layer is traind.. Neural tangent kernel, first proposed by Jacot et al. (2018). is fundamentally different as NTKs correspond to infinitely wide NNs with all layer being trained. Theoretically, a line of work study the optimization and generalization behavior of ultra-wide NNs (Allen-Zhu et al. 2018b a 2019b Du et al., 2018b a Li Liang 2018 Zou al.) [018]] Yang 2019). Recently, Arora et al. 2019a) gave non-asympioti perturbation bound between the NN predictor trained by gradient descent and the NTK predictor. Empirically, Lee et al. (2019) verified on small scale data,
Arora et
NTK is good approximation NN. However, Arora al. (2019a) showed on large scale dataset, NN can outperform NTK which may due to the effect of finite-width and/or optimization procedure.
Generalization to architectures other than fully-connected NN and CNN are recently pro- posed (Yang) 2019 Du et al.] [2119] Bietti & Mairal 2019). Du et al. (2019) showed graph neural tangent kernel (GNTK) can achieve better performance than its counter part, graph neural network (GNN), on datasets with up to 5000 samples.
NEURAL NETWORK AND NEURAL TANGENT KERNEL
Since NTK induced by a NN architecture, we first define NN formally. Let x E Rd be the input, and denote (z)) = x and do d for notational convenience. We define L-hidden-layer fully-connected neural network recursively:
(x) (h R (x) - -V (FW)) E Rd
h = 1.2......4
where (h)) E Rd, - is the weight matrix in the h-th layer (h E [L]) IR + R is coordinate- wise activation function, and e, a scaling factor In this paper, for NN we will consider a being ReLU ELU (Clevert et al. 2013) and for NTK we will only consider kernel functions induced by NNs with ReLU activation. The last layer of the neural network
f(w, x) = (()+)) = w(L+1). g14)(x)
= w(L+1)
Co Vd
(WL)).
/ C CO (((--).... V d2-1
V V (w())z) )
di
where ((++1) E Rixdz is the weights in the final layer, and we let w = (W())... ,..., W((+1)) be all parameters in the network. All the weights are initialized to be i..d. /o 1) random variables. From now on, by NTK initialization mean neural network with parameterization defined in Equation 2 with all weighted being initialized to be i..dd. (0,
When the hidden widths d2, d2....,dyy dd co, certain limiting behavior emerges along the gradient trajectory. Let x, x' E Rd be two data points, the covariance kernel of the h-th layer's 's outputs, (, x') = p(h (1)((x)) can be recursively defined an analytical form:
( (z. x²) xx TTa'.
(1)) (z, x²) (2)))))(,,z) (2,z) (x,
E 2x2
(x',
(0h (z, x²) co (M.) la (w) o (v)].
PPtttin an explicit scaling factor in the definition of NN is typically called NTK parameterization (Jacot et al. 2018 Park et al.] 2019). Standard parameterization scheme does not have the explicit scaling factor. The derivation of NTK requires NTK parameterization. In our experiments on NNs, try both parameterization
schemes.
3

---

Published as a conference paper at ICLR 2020
Classifier
Friedman Rank
Average Accuracy
P90
P95
PMA
NTK
28.34
81.95%++14.1 %
88.89
72.22% %
95.72% +5.17%
NN (He init)
40.97
10.88%%+:144969
81.11%
65.56%
94.34% +7.22%
NN (NTK init)
38,06
81.02%::114.4%%
85.56%
60.00%
94.55%
15.89%
RF
33.51
81.56% 13.90%
85.56%
67.78%
95.25%
15.30%
Gaussian Kernel
35.76
81.03% + 15.09%
85.56%
72.22% %
94.56%
18.22%
Polynomial Kernel
38.44
78.21% + 20.30%
80.00%
62.22%
91.29% + 18.05%
Table I: Comparisons of different classifiers on 90 UCI datasets. P90/P95: the number of datasets a classifier achieves 90%/95% or more of the maximum accuracy, divided by the total number of datasets. PMA: average percentage of the maximum accuracy.
for h E [Z]. Crucially, this analytical form holds not only the initialization, but also holds during the training (when gradient descent with small learning rate is used as the optimization routine).
Formally, NTK is defined the limiting gradient kernel
+ (I,2') & [f(w,)) ( x) 0f(w,x))
L+1 Σ
0f(m,, x)) 0f(w,x')
dw
Ow
(W aW())
Again, one can obtain recursive formula
5000 (x,x') = Ca B, (u.)) (o))))/)__0((()(())))))))))))))) (o(u) I L+1
(4)
1++1
Iot1
(xx,
11- 13 (x,
II (x, l'=h
(5)
where we let (x(+1) (2, for convenience. It is easy to check if we fix the first L' lay- ers and only train the remaining (L + L') layers, then the resulting NTK is O(2, x')
Eht1 En-1/41 (2, T ((K) (x, Note when L'
then the resulting NTK is
E(M) (x)) which is the NNGP kernel (ee et al. 2018). L' can be viewed as a hyperparameter of NTK classifier, and in UCI experiment tune L'. Given kernel function, one directly it for downstream classification tasks (Scholkopt & Smola 2001).
EXPERIMENTS ON UCI DATASETS
In this section, we present our experimental results on UCI datasets which follow the setup of Fernánddez-Delaado et al. (2014) with extensive comparisons of classifiers, including random for- est, kernel SVM, multilayer neural networks, etc. Section 4.2 discusses the performance through detailed comparisons with other classifiers tested by [eernnndee-eelgado et al. (2014). Sec- tion 0.0 NTK classifier and the corresponding NN classifier and verifies how similar their predictions are. See Table in Appendix(A]] A summary of datasets used. The detailed experi- ment setup, including the choices the datasets, training / test splitting and ranges of hyperparameters, described in Appendix A We note that usual methods of obtaining confidence bounds in these low-data settings are somewhat heuristic.
NTK
4.1 OVERALL PERFORMANCE COMPARISONS
Table lists the performance of 6 classifiers under various metrics: the three top classifiers identified in Fernández- -ellaad et al. (2014), namely, RF, Gaussian kernel and polynomial kernel, along with new methods NTK. NN with He initialization and NN with NTK initialization. Table shows NTK is the best classifier under all metrics, followed by RF, the best classifier identified in Bernández-Delgado et al. (014). Now we interpret each metric in more details.

---

Published as a conference paper at ICLR 2020
Friedman Ranking and Average Accuracy. NTK is the best (Friedman Rank 28.34, Average Accuracy 81. 81.95%), followed by RF (Friedman Rank 33.51, Average Accuracy 81.56%) and then followed by SVM with Gaussian kernel (Friedman Rank 35.76, Average Accuracy 81.03%). The difference between NTK and RF is significant (-5.17 in Friedman Rank and +0.39% in Average Accuracy), just as the superiority of RF is significant compared to other classifiers as claimed in Fernández-Delgaado et al.] (2014). NN (with either He initialization or NTK initialization) per- forms significantly better than the polynomial kernel in terms of the Average Accuracy (80.88% and 81.02% vs. 78.21%) 1%). but in terms of Friedman Rank, NN with He initialization is worse than polynomial kernel (40.97 vs. 38.44) and NN with NTK initialization is slightly better than polyno- mial kernel (38.06 vs. 38.44). On many datasets where most classifiers have similar performances, NN's rank high as well, whereas on other datasets, NN is significantly better than most classifiers, including SVM with polynomial kernel. Therefore, NN enjoys higher Average Accuracy but suffers higher Friedman Rank. For example, on OZONE dataset, NN with NTK initialization is only 0.25% worse than polynomial kernel but their difference in terms of rank is 56. It is also interesting to see that NN with NTK initialization performs better than NN with He initialization (38.06 vs. 40.97 in Friedman Rank and 81.02% vs. 80.88% in Average Accuracy).
P90/P95 and PMA. This measures, for a given classifier, the fraction of datasets on which it achieves more than 90%/95% of the maximum accuracy among all classifiers. NTK is one of the best classifiers (ties with Gaussian kernel on P95), which shows NTK can consistently achieve superior classification performance across a broad range of datasets. Lastly, we consider the Percentage of the Maximum Accuracy (PMA). NTK achieves the best average PMA followed by RF whose PMA is 0.47% below that of NTK and other classifiers are all below 94.6%. An interesting observation is that NTK, NN with NTK initialization and RF have small standard deviation (5.17%, 5.89% and 5.30%) whereas all other classifiers have much larger standard deviation.
4.2 PAIRWISE COMPARISONS
NTK vs. RE. In Figure Ia| we compare NTK with RF. There are 42 datasets that NTK outperforms RF and 40 datasets that RF outperforms NTK | The mean difference is 2.51% % which is statistically significant by Wilcoxon signed rank test. We see NTK and RF perform similarly when the Bayes егror rate is low with NTK being slightly better. There are some exceptions. For example, on the BALANCE-SCALE dataset, NTK achieves 98% accuracy whereas RF only achieves 84. 1 When the Bayes error rate is high, either classifier can be significantly better than the other.
NTK vs. Gaussian Kernel. The gap between NTK and Gaussian kernel is more significant. As shown in Figure Ib NTK generally performs better than Gaussian kernel no matter Bayes error rate is low or high. There are 43 datasets that NTK outperforms Gaussian kernel and there are 34 datasets that Gaussian kernel outperforms NTK. The mean difference is 2.22%, which is also statistically significant by a Wilcoxon signed rank test. On BALLOONS, HEART-SWITZERLANT TEACHING, TRAINS datasets, NTK better than Gaussian kernel by least 11% in terms of accuracy. These five datasets all have less than 200 samples, which shows that NTK can perform much better than Gaussian kernel when the number of samples is small.
PITTSBURG-
BRIDGES-MATERIA
NTK vs. NN In this section we compare NTK with NN. The goals are (i) comparing the perfor- mance and (ii) verifying NTK is good approximation to NN.
In Figure 2a we compare the performance between NTK and NN with He initialization. For most datasets, these two classifiers perform similarly. However, there are a few datasets that NTK per- forms significantly better. There are 50 datasets that NTK outperforms NN with He initialization and there are 27 datasets that NN with He initialization outperforms NTK. The mean difference is 1.96%, which is also statistically significant by Wilcoxon signed rank test.
In Figure 2b we compare the performance between NTK and NN with NTK initialization. Recall that for NN with NTK initialization, when the width goes to infinity, the predictor is just NTK.
2 The average rank of classifier over 90 datasets.
A 'We consider classifier A outperform classifier B if the classification accuracy of A is at least 0.1% better than B. The number 0.1% 5% is chosen because results in Fernnnddee--eeeeeee Delgado et al. (2014) only kept the precision up to 0.1 %.
5

---

Published as a conference paper at ICLR 2020
100
100
80
80
NTK
NTK
60
60
40
40
40
60
80
100
40
60
80
100
Random Forest
SVM with Gaussian Kernel
(a) RF vs. NTK
(b) Gaussian Kernel vs.
Figure 1: Performance comparisons between NTK and other classifiers.
100
100
NTK
NTK
60
60
40
40
40
60
80
100
40
60
80
100
Neural Networks
Neural Networks
(a) NN He init NTK
(b) NN NTK init vs. NTK
Figure 2: Performance Comparisons between NTK and NN.
Therefore, we expect these two predictors give similar performance. Figure 2b verifies our conjec- ture. There is no dataset the one classifier is significantly better than the other. We do have the same observation in Figure La lb or 2a Nevertheless, NTK often performs better than NN. There are 52 datasets that NTK outperforms NN with He initialization and there are 25 datasets NN with He initialization outperforms NTK. The mean difference is 1.54%, which is also statistically significant by Wilcoxon signed rank test.
5 EXPERIMENTS ON SMALL CIFAR- 10 DATASET
In this section, study the performance of CNTK on subsampled CIFAR-10 dataset. We randomly choose n samples from CIFAR-10 10 training set, use them to train CNTK and ResNet-34, and test both classifiers the whole test set. In experiments, convolutional layers of CNTK varies from to 14. See Appendix for detailed experiment setup.
vary from 10 1280, and the number of
The results reported in Table It can be observed that in this setting CNTK consistently out- performs ResNet-34. The largest gap occurs at n 320 where 14-layer CNTK achieves 36.57% accuracy and ResNet achieves 33. 15%. The smallest improvement occurs 10: 15.33%

---

Published as a conference paper at ICLR 2020
ResNet
5-layer CNTK
8-layer CNTK
11-layer CNTK
14-layer CNTK
10
14.59% + 1.99%
15.08% 2.43%
15.24% 2.44%
15.31% = 2.38%
15.33% 2.43%
20
17.50% 1 2.47%
18.03% 1 1.91%
18.50% 1 2.03%
18.69% = 2.07%
18.79%
2.13%
40
19.52% + 1.39%
20.83% + 1.68%
21.07% + 1.80%
21.23% = 1.86%
21.34%
+ 11.91 %
80
23.32%
+ 1.61%
24.82% + 1.75%
25.18% + 1.80%
25.40% = 1.84%
25.48
+ 1.91%
160
28.30%
+ 1.38%
29.63%
1.13%
30.17%
1.11%
30.46%
1.15%
30.48%
1.17%
320
33.15%
1: 1.20%
35.26%
0.97%
36.05%
0.92%
36.44%
= 0.91%
36.57%
0.88%
640
41.66%
+ 1.09%
41.24%
+ 0.78%
42.10% + 0.74%
42.44%
= 0.72%
42.63% % + 0.68%
1280
49.14%
+ 1.31%
47.21%
+ 0.49%
48.22% + 0.49%
48.67% = 0.57%
48.86% + 0.68%
Table 2: Performance of ResNet-34 and CNTK on small CIFAR-10 Dataset.
lineur SVM
1-layer CNTK
2-layer CNTK
3-layer CNTK
4-layer CNTK
5-layer CNTK
6-layer CNTK
49 4 4.71
50.04 4.28
18,94=4.11
49.66 4.19
19.24
4.20
66 +: 4.26
48,00 4.41
64.39 +.402
65.81 5.28
65.89 5.23
65.65 +.17
65.30
5.10
64.76
5.07
64.12 5.00
70.38 3.28
71.60
3.51
71.56 3.65
71.32 4 3.74
70.96
3.84
70.51
4: 3.92
69.98 3.95
72.43 3.21
73.65 3.29
73.59 = 3.28
73.35 3
13.00
3.20
72.56 + 3.20
12.09 3.20
5
74.49 + 3.08
75.87 + 2.88
75.69 2.89
75.45
+ 2.89
75.13
2.85
74.68 + 2.88
74.24 2.85
76.59 2..15
77.64 2 2.37 2.37
T7.57 = 2.43
77.39 2.50
17.13 = 2.56
76.87
1: 6.
76.52 = 2.63
77.11 1.25
78.64
1.35
78.57 = 1.39
78.39
1.41
12.12
1.42
77.80
1.41
77.43 1.41
79.57 +0.87
80.42
0.88
80.38
0.86
80.23
1:3::000 0.86
79.99
0.88
79.72
+ 0.89
79.41 0.92
Table 3: Performance of linear SVM and CNTK with different number of convolutional layers on
the 11 11-20th classes in VOC07. The cost value is tuned on the first 10 classes. Feature extracted from conv5 in ResNet-50.
14.59%. When the number of training data is large, i.e., n = 1280, ResNet can outperform CNTK. It is also interesting to that 14-layer CNTK is the best performing CNTK for all values of n, suggesting depth has a significant effect on this task.
EXPERIMENTS ON FEW-SHOT LEARNING
In this section, we test the ability of NTK as a drop-in classifier to replace the linear classifier in the few-shot learning setting. Using linear SVM top of extracted features is arguably the most widely used strategy in few-shot learning as linear S SVM is easy and fast to train whereas more complicated strategies like fine-tuning randomness and may overfit due to the small size of the training set. NTK has the same benefits (easy and fast train, no randomness) as the linear classifier but also allows some non-linearity in the design. Note since we consider image classification tasks, we use CNTK in experiments in this section.
training neural network
top of the extracted feature have more
Experiment Setup. We mostly follow the settings of Goyal et al. (0119). Features are extracted
from layer convl, conv2, conv3, conv4, conv5 in ResNet-50 (He et al. 2016) trained on ImageNet (Deng et al.) 2009), and we use these features for VOCO7 classification task. The number of positive examples varies from to For each value of k. for each class the VOCO7 dataset, we choose l positive examples and 19k negative examples. For each A 21, 8} and for each class, we randomly choose 10 independent sets with 20k training samples in each set. We
lincar SVM
Hlayer CNTK
2-layer CNTK
3-layer CNTK
4-layer
CNTK
5-layer
CNTK
6-layer CNTK
23.25 + 2.89
24.54 + 3.39
24.57 + 3.46
24.57 + 3.50
24.52 3.51
24.43 + 3.50
24.34 3.49
3.85
33.81 = 4.25
33.77 = 4.33
33.08 1 4.40
33,53
4.46
33.31 + 4.51
33.04 4.55
38.1 2.774
2.70
40.10 2.47
22411
30.87
2.37
39.65 2.35
39.36 2.29
39.88 2.08
42.20
1.78
41.80
2.13
42.64 1.641.2000
42.51
2.02
42.32
1: 2.03
42,08
2.03
42.46
22770
44.71 3.12
41,69
= 2,98
44.72
3.001
44.69
= 3.04
44.57
3.07
41.40
3.10
45.71
4 3.27
49.02
2.63
48.63
2.69
48.97
4: 2.68
48.92
2.71
48.79
4: 2.74
48.63
2.77
7
47.97
1 3.25
50.89
3.50
50.43
3.48
50.50
1 3.47
50.48
3.44
50.30
1 3.44
50.24
3.41
49.81
+2188
52.32
2.65
52.06
2.32
52.23
+ 2.38
52.31
2.40
52.26
+ 2.45
52.14
2.50
Table 4: Performance of linear SVM and CNTK with different number of convolutional layers on the 11-20th classes in VOCO7. The cost value is tuned on the first 10 classes. Feature extracted from conv4 in ResNet-50.

---

Published as a conference paper at ICLR 2020
linear SVM
1-layer CNTK
2-layer CNTK
3-layer CNTK
4aayer CNTK
5-layer CNTK
6-layer CNTK
14.38 + 1.98
15.42 +: 1.90
15.36 = 1.93
15.30 +: 1.97
15.24 2.02
15.17 + 2.05
15.10 2.08
2
16.67 + 1.50
18.48
1.58
18.39 = 1,53
18.28 +: 1.52
18.18 = 1.49
18.00 + 1.47
17.99 = 1.45
19.56 + 1.04
21.79
1.69
21.73 1.63
21.67 + 1.60
21.61 1.55
21.54 + 1.50
21.47 1.46
20.53
1.62
23.39
2.13
23.36
2.18
23.29 1: 2.22
23.20
2.24
23.11 1: 2.26
23.03 2.27
21.51
1.08
25.13
2.36
25.09
2.37
25.03
2.300
24.96
2.36
24.87 : 2.30
24.18
2.37
23.51
2.39
26.
2.39
26.26
2.43
26.10 2.47
25.97
2.47
25.87 +: 2.47
25.78
2.46
24.33 4 1.59
27.98
+ 2.46
27.75
= 2.52
28.24 1 2.22
28.20
2.22
28.14
2.21
28.08 2.21
8
25.31
2.07
27.76 + 2.87
28.52 1 2.65
28.52
2.69
28.49
2.70
28.44 1: 2.72
28.38
2.74
Table 5: Performance of linear SVM and CNTK with different number of convolutional layers on the 11-20th classes in VOCO7. The cost value C' is tuned on the first 10 classes. Feature extracted from conv3. in ResNet-50
report the mean and standard deviation of mAP on the test split of VOCO7 dataset. This setting has been used in numbers of previous few-shot learning papers (Goyal et al. 2019 Zhang et al. (017).
We take the extracted features as the input to CNTK. We tried CNTK with 0-6 convolution lay- ers, a global average pooling layer and a fully-connected layer. We normalize the data in the fea- ture space, and finally use SVM train the classifiers. Note without the convolution layer, it is equivalent to directly applying linear SVM after global average pooling and normalization. We use sklearn. svm. Linearsve to train linear SVMs, and sklearn svm SVC train kernel SVMs (for CNTK). To train SVM, we choose the cost value C' from 2 19, U U100 101-7.2] and set the wlagh ratio be for positive/negative classes as in Goyal al. (2019).
Since the number of given samples is usually small, Goyal et al. (2019) chooses to report the per- formance of the best cost value C. In our experiments, we use a more standard method to perform cross-validation. We use the first 10 classes of VOCO7 to tune C, and report the performance of selected C' in the other 10 classes in Table 3 We also report the performance of the best C as in Goyal et al. (2019), in Tables 7 in the appendix for completeness.
Discussions. First, we find CNTK is a strong drop-in classifier to replace the linear classifier in few-shot learning. Tables 9 clearly demonstrate that CNTK is consistently better than linear clas- sifier. Note Tables 3 only show the prediction accuracy in an average sense. In fact, we find that on every randomly sampled training set, CNTK always gives better performance. We conjecture that CNTK gives better performance because the non-linearity in CNTK helps prediction. This is verified by looking at the performance gain of CNTK for different feature extractors. Note Conv5 is often considered to be most useful for linear classification. There CNTK only outperforms the linear classifier by about 1%. On the other hand, with Conv3 and Conv4 features, CNTK can outperform linear classifier by around 2% and sometimes by 3%-4% (last two rows in Table 5 We believe this happens because Conv3 and Conv4 correspond to middle-level features, and thus non-linearity is indeed beneficial for better accuracy.
We also observe that CNTK often performs the best with a single convolutional layer. This is ex- pected since features extracted by ResNet-50 already produce a good representation. Nevertheless, we find for middle-level features Conv3 with & or 8, CNTK with 3 convolutional layers give the best performance. We believe this is because with more data, utilizing the non-linearity induced by CNTK can further boost the performance.
7 CONCLUSION
The Neural Tangent Kernel, discovered by mathematical curiosity about deep networks in the limit of infinite width, is found yield superb performance on low-data tasks, beating extensively tuned versions of classic methods such as random forests. The (fully-connected) NTK classifiers are easy to compute (no GPU required) and thus should be good off- off-the-shelf classifier in many settings. We plan to release our code to allow drop-in replacement for SVMs and linear regression.
Many theoretical questions arise. Do NTK SVMs correspond to some infinite net architecture (as NTK ridge regression does)? What explains generalization in small-data settings? (This under- standing is imperfect even for random forests.) Finally one can derive NTK corresponding other
8

---

Published as a conference paper at ICLR 2020
architectures, e.g., recurrent neural tangent kemel (RNTK) induced by recurrent neural networks. It would be an interesting future research direction to test their performance on benchmark tasks.
ACKNOWLEDGMENTS
S. Arora, Z. Li and D. Yu are supported by NSF, ONR, Simons Foundation, Schmidt Foundation, Amazon Research. DARPA and SRC. S. S. Du is supported by National Science Foundation (Grant No. DMS-1638352) and the Infosys Membership. R. Salakhutdinov and R. Wang are supported in part by NSF IIS-1763562, AFRL CogDeCON 1A875018C0014 and DARPA SAGAMORE HR001 1990016. Part of this work was done while S. S. Du was visiting Google Brain Prince- ton and R. Wang was visiting Princeton University. The authors would like thank Amazon Web Services for providing compute time for the experiments in this paper, and NVIDIA for GPU sup- port. We thank Priya Goyal for providing experiment details of Goyal al. (0019). We thank Xiaolong Wang for discussing the sew-shot learning task.
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameter- ized neural networks, going beyond two layers. arXiv preprini arXiv: 1.04918, 2018a.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. arXiv preprint 17Xiv:811.00396. 2018b.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. arXiv preprint arXiv: 11955, 2019a.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint 1wXiv:1901.0858: 2019b.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. arXiv preprint /20. 2019.
Youngmin Cho and Lawrence K Saul. Kernel methods for deep learning. In Advances in neural information processing systems, pp. 342-350, 2009.
Amó Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.0728) 2015.
Amit Daniely. SGD learns the conjugate kernel class of the network. arXiv preprint 1r02v:/70..085500 2017.
Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and dual view on expressivity. In Advances In Neural Information 2016.
Processing Systems, pp.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 2482255. leee, 2009.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. arXiv preprint 1tfi::8811.08804 2018a.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arx1v:1810.02054, 2018b.
Simon S. Du, Kangcheng Hou, Barnabás P6czos, Ruslan Salakhutdinov, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. ArXiv, abs/1905. 2019.
Manuel Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim. Do need hun- dreds of classifiers to solve real world classification problems? The Journal of Machine Learning Research, 15(1333133-3181, 2014.

---

Published as a conference paper at ICLR 2020
Adrià Garriga-Alono, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional net- works as shallow gaussian processes. International Conference on Learning Representations, 2019. /ttpss//oppneereee...ttt/// http://ettttrrrmmmaaaa11111ooo..
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. arXiv preprint arXhv:1905.012355 2019.
Tamir Hazan and Tommi Jaakkola. Steps toward deep kernel methods from infinite neural networks. arXiv preprint arXiv: 150..0500551 133, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, pp. 8571- -- 8580, 2018.
Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as gaussian processes. In International Confer- ence on Learning Representations, 2018. URL https:://penrevieww net net./orumfffddd
BlEA-M- 02
Jachoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Jascha Sohl-Dicksterr Dil--iiiittein, and Jef- frey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. arXiv preprint 1902.00670 2019.
Yuanzhi Li and Yingyu Liang. Learning overparameterrized neural networks via stochastic gradient descent on structured data. arXiv preprint arXiv: / /10.//0..///// 0/204, 2018.
Alexander G de Matthews, Mark Rowland, Jiri Hron, Richard Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv: / 2018.
Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pp. 29-53. Springer, 1996.
Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Daniel A. Abolafia, Jeffrey Pennington, and Jascha dick-dickstein. Bayesian deep convolutional networks with many chan- nels are gaussian processes. In International Conference Learning Representations, 2019. URL
net /tte:/frrmmmdddll3000000 http:///wwriiii.eeeeeeooooooaaaa
Matthew Olson, Abraham Wyner, and Richard Berk. Modern neural networks generalize on small data sets. In Advances Neural Information Processing Systems, pp. 3619-3628, 2018.
Daniel S Park, Jascha Sohl-Dickstein, Quoc V Le, and Samuel L Smith. The effect of network width on stochastic gradient descent and generalization: an empirical study. arXiv preprint arXiv:1905.03776. 2019.
Nicolas Le Roux and Yoshua Bengio. Continuous neural networks. In Proceedings of the Eleventh International Conference Artificial Intelligence and Statistics, volume 2 of Proceedings of Machine Learning Research, pp. 404-411, San Juan, Puerto Rico, 2007.
Bernhard Scholkopf and Alexander J Smola. Learning with kemnels: support vector machines, regularization, optimization, and beyond. MIT press, 2001.
Christopher KI Williams. Computing with infinite networks. In Advances in neural information processing systems, pp. 295-301, 1997.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint 1902: 2019.
10

---

Published as a conference paper at ICLR 2020
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1rXiv:1611.03530, 2016.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep ReLU networks. arXiv preprint arXiv: 2018.

---

Published as a conference paper at ICLR 2020
A ADDITIONAL EXPERIMENTAL DETAILS ON UCI
In this section, we describe our experiment setup.
Dataset Selection Since we only wish test on small datasets, we only select UCI datasets with number of samples smaller than 5000. Furthermore, we only use datasets without explicit training / testing splitting from the link: httt::::tp::seeeee cat.rsslll.iiiu... use. es as/manuel. fernandez,
delgado/ papers/
dataaaaaa....................... tar gz, which are the pre-processed datasets by Fernández- Delgado et al. (2014). These datasets are originally from UCI datasets (including most of the datasets
before March, 2013) and 4 real-world datasets not included in the UCI repository (see Fernández. Delgado al. (2014) for details). The datasets are pre-processed to be classification problems and all categorical features are turned into numerical features and normalized along the samples for each feature. The reason to discard datasets with explicit splitting is that we found there is an obvious distributional shift between training and testing data. The pre-proeee pre-peeeesee data provided from the link shows that training and testing data are normalized with different mean and standard deviation. For example, the AUDIOLOGY-S in training and (-1.2, 0.8) in testing; on the ANNEALING dataset, the support of the first feature is {-0.670274, 0.223425, 3.7922 in training and (-0.111111 0, in testing. See Table 6 for summary of the datasets.
dataset, the support of the first feature is (-0.761526, 1 1.30547)
# samples
features
# classes
min
10
3
2
25%
178
8
2
50%
583
16
3
75%
1022
32
6
max
5000
262
100
Table 6: Dataset Summary
Performance Comparison Details We follow the comparison setup in Fernández-Delgado et al. (2014) that we report 4-fold cross-validation. For hyperparameters, we tune them with the same validation methodology in Fernándezz
et al. (2014): all available training samples are ran- domly split into one training and one set, while imposing that each class has the same number of training and test samples. Then the parameter with best validation accuracy is selected. It is possible give confidence bounds for this parameter tuning scheme, but they are worse than standard ones for separated training/validationn wawiivaaiootttstiigggnnn data. For NTK and NN classifiers we train them on these 90 datasets, For other classifiers, we use the results from Fernández- Senannde--Deladdo al. (2014).
NTK Specification We calculate NTK induced fully-connected neural networks with layers where L' bottom layers are fixed, and then use C'support vector classification implemented by sklearn. svm. We tune hyperparameters L from 1 to 5, L' from 0 to L 1, and cost value C as powers of ten from 2 to The number of kernels used is 15, so the total number of parameter combinations is 105. Note this number is much less than the number of hyperparameter of Gaus- sian Kernel reported in Mradandez-Delgadee et al. (2014) where they tune hyperparameters with 500 combinations.
NN Specification We use fully-connected NN with T. layers, 512 number of hidden nodes per layer and use gradient descent to train the neural network. We tune hyperparameters L from 1 to 5, with / without batch normalization and learning rate 0.1 or 1. We run gradient descent for 2000 epochs [ We treat NN with He initialization and NTK initialization as two classifiers and report their results separately.
'ee found NN can achieve 0 training loss after less than 100 epochs. This is consistent with the observation Olson et al. 2018).
12

---

Published as a conference paper at ICLR 2020
B ADDITIONAL EXPERIMENTAL DETAILS ON SMALL CIFAR- 10 DATASETS
We randomly choose n/ 10 samples from each class of CIFAR-10 training set and test classifiers on the whole testing set. varies from 10 to 1280. For each n, we repeat 20 times and report the mean accuracy and its standard deviation for each classifier.
The number of convolution layers of CNTK ranges from 5-14. After convolutional layers. we apply a global pooling layer and fully connected layer. We refer readers to Arora et al. (2019a) for exact formulas of CNTK. We normalize the kernel such that each sample has unit length feature space.
We use ResNet-34 with width 64,128,256 and default hyperparameters: learning rate 0.1, momen- tum 0.9, weight decay 0.0005. We decay the learning rate by 10 at the epoch of 80 and 120, with 160 training epochs in total. The training batch size is the minimum of the size of the whole training dataset and 160, We report the best testing accuracy among epochs.
C ADDITIONAL RESULTS IN FEW-SHOT LEARNING
Tables show the performance the best as has been done in Goyal et al. (2019).
lineur SVM
I-layer CNTK
2-layer CNTK
3-layer CNTK
4-Jayer CNTK
S-layer CNTK
6-layer CNTK
52.63 + 5.35
53.16 4.96
53.08 = 4.89
52.83 + 4.85
52.44 4.85
51.88 + 4.85
51.21 4.88
2
66.08 + 2.69
66.22 + 3.09
66.29 + 3.09
66.11 + 3.07
65.79
3.04
65.33 + 3.00
64.77 2.99
3
71.78 1.85
72.76 1.76
72.72 = 1.73
72.49 1.71
72.14
1.73
71.71 1.76
71.17 1.78
74.40 1.86
75.37
1.95
15.33 = 1.95
75.14 196
14.85 = 1.96
74.48 = 1.99
74.05 = 1.97
75.94 2.511
77.01
: 2.39
76,91 = 2.39
76.72
2.30
76.47
2.37
76.13
2.30
75.76 2,39
76.39 1.27
77.15
1.47
77.14 1.50
77.02
1.55
76.83
1.58
76.62
1.60
76.35
1.61
78.18
79. 12 0.85
79.36
0.87
79.22 -: 0.88
79.00
0.91
78.75 1: 0.93
78.50 0.94
79.78 0.66
80.47
0.59
80.45
0.61
80.33 + 0.63
80.14
0.65
79.92 + 0.67
79.66 0.70
Table 7: Performance of linear SVM and CNTK with different number of convolutional layers on all classes in VOCO7 with the best C. Feature extracted from conv5 in ResNet-50
linear SVM
-laaeer CNTK
2-layer CNTK
3-layer CNTK
4-layer CNTK
5-layer CNTK
6-layer CNTK
24.18 2.64
25.
2.93
25.13 = 3.00
25.06 1 3.05
24.95
3.10
24.80 +: 3.13
24.60 3.14
2
31.59 2.64
32.88 2.91
32,81 = 2.94
32.67 2.96
32.48 = 2.98
32.24 2.99
31.94 1 3,01
37.34 2.18
38.69
2.39
38.64
2.26
38.53 2.28
38.36
2.31
38.12
+ 2:33
37.83 2.35
30.67 2.31
41.34
2.33
41.39
2.28
41.33
2.24
41.21
2.22
41.02 1: 2:20
40.77
2.18
41.74 1.200
43.48
1.40
43.48 + 1.24
43.44
43.32
1.27
43.12
42.89
I.33
44.68
1.84
46.77
1.82
46.42
1.91
46.44
1.90
46.39
1.91
46.25 + 1.93
46.08
1.94
47.71 + 1.84
50.07
1.97
49.74
1.95
49.63 + 1.81
49.36
1.81
49.41 = 1.82
49.19
1.81
8
48.91
+ 1.90
51.32
1.97
50.98
= 1.87
51.09
+: 1.84
51.09
1811
51.00
+ 1.78
50.84 1.76
Table 8: Performance of linear SVM and CNTK with different number of convolutional layers on all classes in VOCO7 with the best C. Feature extracted from conv4 in ResNet-50
hnear SVM
1-layer CNTK
2-layer CNTK
3-layer CNTK
4 4ayer CNTK
5-layer
CNTK
6-layer CNTK
+ 1.49
15.01 = 1.27
14.96 = 1.28
14.92 + 130
1487 = 1.32
14.81 # 1.34
14,75 1.36
16.67 1.04
17.52 1.05
17.45 1.03
17.36
1.02
17.28
1.00
17.20 + 0.99
17.11 0.97
19.21 1.31
20.65 1.68
20.63 = 1.65
20.58
1.63
20.33 = L.61
20.47 + 1.59
20.41 L.57
20.87
1.49
22.28
22.27 = 1.81
22
1.83
22.16
L88
22.09 :: 1.83
22.03
1.82
21.98 + 1.17
23.45
1.14
23.45 1 1.12
23.43
1.08
23.38
1.07
23.32 +: 1.05
23.25
1.04
23.02 1.56
24.44
= 161
24.29 = 1.66
24.27
+ 1.82
24.25
1.81
24.21
+ 1.80
28.18
1.78
24.38
1.52
25.96
+ 1.80
25.99
1.70
25.98
+: 1.70
25.96
1.69
25.92
4: 1.67
25.88
1.66
25.28 1
1.30
27.06
1 1.61
27.15
1.58
27.18
1 1.56
27.17
1.33
27.13
1.51
27.09
1.50
Table 9: Performance of linear SVM and CNTK with different number of convolutional layers all classes in VOCO7 with the best C. Feature extracted from conv3 in ResNet-50
13


Reviews:
[{"review_id": "r1eG5udtKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThis paper performs an extensive empirical evaluation of Neural Tangent Kernel (NTK) classifiers---kernel methods that theoretically characterize infinitely wide neural nets---on small-data tasks. Experiments show that NTK classifiers (1) strongly resemble the performance of neural nets on small-data tasks, (2) can beat prior benchmark methods such as Random Forests (RF) on classification tasks in the UCI dataset, and (3) can also outperform standard linear SVM on a few-shot learning task.\n\n[Pros]\nThe question considered in this paper is well motivated, and a very natural extension of Lee et al. (2019) and Arora et al. (2019a). These papers show that NTK performs well on (relatively) large benchmark tasks such as CIFAR-10 but is still a bit inferior to fully trained neural nets. On the other hand, for small-data tasks, the relationship is reversed --- neural nets are slightly inferior to more traditional methods such as random forests (e.g. from Fernandez-Delgado et al. 2014) and Gaussian kernel SVMs. As the NTK gives a limiting characterization for wide neural nets, it is a sensible question to test the performance of NTK on these small datasets, and see if they can improve over neural nets and compare more favorably against the traditional methods.\n\nThe experimental results, in my perspective, is a reasonably convincing evidence that the resemblance between NTK and NN on small-data tasks is stronger than on larger tasks such as CIFAR-10, which agrees with the NTK theory. In addition to the UCI datasets, the paper also tries out NTK in a few-shot learning task and show that SVM with the convolutional NTK does better than the linear SVM as the few-shot learner. I am less familiar with few-shot learning though so am not entirely sure about the strength of this part.\n\nThe paper is well-written and delivers its messages clearly. The results and discussions are easy to follow.\n\n[Cons, and suggestions]\nThe message that “NTK beats RF” seems a bit delicate to me, specifically considering the fact that the average accuracies of (NTK, NN, RF) are all pretty close but the Friedman rank comparison says NTK > RF > NN (somewhat more significantly). This implies the difference between all these methods has to be small and it’s only that NTK happens to win on more tasks. In addition, NTK tunes one more parameter (L’) than NNs, so I guess perhaps NNs can also be tuned to outperform RF in the rank sense if we also tune L’ (by fixing the bottom L’ layers to be not trained) in NNs?\n\nAlso, it would be better if the authors could provide a bit more background on the metrics used in the UCI experiments -- for example, the Friedman rank is not defined in the paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Sklj47KpFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper evaluates the empirical power of neural tangent kernel (NTK) on small-data tasks. The authors demonstrate the superior performance of NTK for classification/regression tasks on UCI database, small CIFAR-10 dataset and VOC07 testbed.\n\nOverall, this paper is well written and organized. The experimental results are also quite interesting. Besides, some questions and comments are as follows:\n\nOne of the baseline algorithms in Table 1 is NN with NTK initialization. However, this paper does not give the formal definition of NTK initialization.\n\nIn Figures 1-2, it can be observed that NTK cannot universally outperform baselines on all dataset. For some dataset, NTK can be worse than baselines but for some other dataset, NTK can be significantly better than baselines. Therefore, I would like the authors to briefly discuss which kind of data can be more efficiently learned through NTK or other training algorithms.\n\nIn Tables 2-5, it can be observed that for CIFAR10 dataset, increasing the number of layers leads to higher test accuracy. But for VOC07, one can observe the opposite thing. Is there any explanation for this phenomenon?\n\nThe authors should provide a clear description of the experimental setting. For example, do you use batch normalization/weight decay in ResNets? For training NN, which optimization algorithms do you use? Do you use learning rate decay? \n\n======================\nAfter reading authors' response:\n\nThanks for your response, I would like to keep my score.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJx5zKqaKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper conducts very interesting and meaningful study of kernels induced by infinitely wide neural networks on small data tasks. They show that on a variety of tasks performance of these kernels are superior to both finite neural networks and Random Forest methods.\n\nWhile neural tangent kernel (NTK) [1] is motivated for studying training dynamics of neural networks, it is also important to ask to find utility of these new powerful kernels that captures functional priors of neural networks. This paper conducted important study on small dataset regime and on a wide range of tasks (90 UCI datasets, small subset of CIFAR-10, few shot image classification task on VOC07. \n\nAuthors introduce a family of generalized NTK kernels interpolating between NNGP kernels [2] to original NTK[1] by fixing first L’ layers and allowing to train remaining layers. Treating L’ as a hyperparameter, the authors try both NNGP/NTK and kernels in between as well. \n\nAnother contribution I observe is applying kernel SVM where one utilizes NTK and shows that it can work well. This paper shows that kernels induced by infinitely wide networks could become useful for real world applications where data size is not so large. \n\nThere are few small concerns regarding experiments which are discussed in detailed comments. Overall I think the message of the paper is clear and well supported therefore I recommend accepting the paper. \n \nDetailed comments\n\t\n1) From reading the paper it was not easy to grasp where point 4 of the abstract was based on. \n2) In the first footnote, small nit is that, in practice one should not invert matrix but just do a linear solve for better numerical stability and efficiency (still O(N^3) but with better constant)\n3) In section 3, there seems to be no bias. Are NTK and NNs considered in this work contain no bias? Or is bias ignored for ease of presentation? \n4) Nit p4 first paragraph in section 4 : multiplayer -> multilayer\n5) Regards to NTK initialization performing better than standard He initialization: It was observed in [3] that for multilayer perceptron both parameterization is on-par but for CNN or WideResNet case standard parameterization performed significantly better.\n6) Note that similar to analysis in section 5,  for CIFAR-10 with fully connected model [1] shows that for all dataset size(100-45k) NNGP performs better than trained neural networks.\n7) One may worry that ResNet-34 is not properly tuned as most hyperparameters were fixed for large dataset. \n8) Regards to hyperparameters for NTK, is there a consistent trend one could find regards to L’? What percentage of tasks that NTK performed well actually have a high L’?\n9) To help the readers, I would suggest adding a little more description on statistics used for comparison as well as what VOC07 task entails. \n\n[1] Jacot et al., Neural Tangent Kernel: Convergence and Generalization in Neural Networks, NeurIPS 2018\n[2] Lee et al., Deep Neural Networks as Gaussian Processes, ICLR 2018\n[3] Park et al., The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study, ICML 2019\n\nEDIT AFTER AUTHOR RESPONSE:\nI have read the response from authors. I appreciate all the efforts to improve the paper. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper carries out extensive experiments on Neural Tangent Kernel (NTK) --kernel methods based on infinitely wide neural nets on small-data tasks. I recommend acceptance.
Decision: Accept (Spotlight)