On the Equivalence between Neural Network and Support Vector Machine

1 Introduction . Recent research has made some progress towards deep learning theory from the perspective of infinitewidth NN . For a fully-trained neural network , it follows kernel gradient descent in the function space with respect to NTK [ 27 ] . Under this linear regime and squared loss , it is rigorously proved that the fully-trained net is equivalent to kernel regression with NTK [ 4 ] , which gives the generalization ability of such a model [ 5 ] . NTK helps us understand the optimization [ 27 , 18 ] and generalization [ 5 , 12 ] of NN through the perspective of kernels . However , existing theories about NTK [ 27 , 30 , 4 , 13 ] usually assume the loss is a function of the model output , which does not include the case of regularization . Besides , they usually consider the squared loss which corresponds to a kernel regression , which may have limited insights to understand classification problems since squared loss and kernel regression are usually used for regression problems . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . On the other hand , another popular machine learning paradigm with solid theoretical foundation before the prevalence of deep neural networks is the support vector machine ( SVM ) [ 10 , 15 ] , which allows learning linear classifiers in high dimensional feature spaces . SVM tackles the sample complexity challenge by searching for large margin separators and tackles the computational complexity challenge using the idea of kernels [ 43 ] . To learn an SVM model , it usually involves solving a dual problem which is cast as a convex quadratic programming problem . Recently , there are some algorithms using subgradient descent [ 44 ] and coordinate descent [ 23 ] to further scale the SVM models to large datasets and high dimensional feature spaces . We noticed that existing theoretical analysis mostly focused on connecting NN with kernel regression [ 27 , 4 , 30 ] but the connections between NN and SVM have not yet been explored . In this work , we establish the equivalence between NN and SVM for the first time to our best knowledge . More broadly , we show that our analysis can connect NNs with a family of ` 2 regularized KMs , including kernel ridge regression ( KRR ) , support vector regression ( SVR ) and ` 2 regularized logistic regression , where previous results [ 27 , 4 , 30 ] can not handle . These are the equivalences beyond ridge regression for the first time . Importantly , the equivalence between infinite-width NN and these ` 2 regularized KMs may shed light on the understanding of NN from these new equivalent KMs [ 16 , 45 , 42 , 48 ] , especially towards understanding the training , generalization , and robustness of NN for classification problems . Besides , regularization plays an important role in machine learning to restrict the complexity of models . This equivalence may shed light on the understanding of the regularization for NN . We highlight our contributions as follows : • We derive the continuous ( gradient flow ) and discrete dynamics of SVM trained by subgradient descent and the dynamics of NN trained by soft margin loss . We show the dynamics of SVM with NTK and NN are exactly the same in the infinite width limit because of the constancy of the tangent kernel and thus establish the equivalence . We show same linear convergence rate of SVM and NN under reasonable assumption . We verify the equivalence by experiments of subgradient descent and stochastic subgradient descent on MNIST dataset [ 28 ] . • We generalize our theory to general loss functions with ` 2 regularization and establish the equivalence between NN and a family of ` 2 regularized KMs as summarized in Table 1 . We prove the difference between the outputs of SVM and NN sacles as O ( lnm/ p m ) , where is the coefficient of the regularization and m is the width of the NN . Additionally , we show every finite-width neural network trained by a ` 2 regularized loss function is approximately a KM . • We show that our theory offers three practical benefits : ( i ) computing non-vacuous generalization bound of NN via the corresponding KM ; ( ii ) we can deliver nontrivial robustness certificate for the over-parameterized NN ( with width m ! 1 ) while existing robustness verification methods would give trivial robustness certificate due to bound propagation [ 22 , 52 , 55 ] . In particular , the certificate decreases at a rate of O ( 1/ p m ) as the width of NN increases ; ( iii ) we show that the equivalent infinite-width NNs trained from our ` 2 regularized KMs are more robust than the equivalent NN trained from previous kernel regression [ 27 , 4 ] ( see Table 3 ) , which is perhaps not too surprising as the regularization has a strong connection to robust machine learning . 2 Related Works and Background . 2.1 Related Works . Neural Tangent Kernel and dynamics of neural networks . NTK was first introduced in [ 27 ] and extended to Convolutional NTK [ 4 ] and Graph NTK [ 20 ] . [ 26 ] studied the NTK of orthogonal initialization . [ 6 ] reported strong performance of NTK on small-data tasks both for kernel regression and kernel SVM . However , the equivalence is only known for ridge regression currently , but not for SVM and other KMs . A line of recent work [ 19 , 1 ] proved the convergence of ( convolutional ) neural networks with large but finite width in a non-asymptotic way by showing the weights do not move far away from initialization in the optimization dynamics ( trajectory ) . [ 30 ] showed the dynamics of wide neural networks are governed by a linear model of first-order Taylor expansion around its initial parameters . However , existing theory about NTK [ 27 , 30 , 4 ] usually assume the loss is a function of the model output , which does not include the case of regularization . Besides , they usually consider the squared loss which corresponds to a kernel regression , which may have limited insights to understand classification problems since squared loss and kernel regression are usually used for regression problems . In this paper , we study the regularized loss functions and establish the equivalence with KMs beyond kernel regression and regression problems . Besides , we studied the robustness of NTK models . [ 24 ] studied the label noise ( the labels are generated by a ground truth function plus a Gaussian noise ) while we consider the robustness of input perturbation . They study the convergence rate of NN trained by ` 2 regularized squared loss to an underlying true function , while we give explicit robustness certificates for NNs . Our robustness certificate enables us to compare different models and show the equivalent infinite-width NNs trained from our ` 2 regularized KMs are more robust than the equivalent NN trained from previous kernel regression . Neural network and support vector machine . Prior works [ 50 , 49 , 34 , 47 , 31 ] have explored the benefits of encouraging large margin in the context of deep networks . [ 14 ] introduced a new family of positive-definite kernel functions that mimic the computation in multilayer neural nets and applied the kernels into SVM . [ 17 ] showed that neural networks trained by gradient flow are approximately KMs with a new conceptual kernel named path kernel . [ 44 ] proposed a subgradient algorithm to solve the primal problem of SVM , which can obtain a solution of accuracy ✏ in Õ ( 1/✏ ) iterations , where Õ omits the logarithmic factors . In this paper , we also consider the SVM trained by subgradient descent and connect it with NN trained by subgradient descent . [ 46 , 3 ] studied the connection between SVM and regularization neural network [ 41 ] , one-hidden layer NN that has very similar structures with that of KMs and is not widely used in practice . NNs used in practice now ( e.g . fully connected ReLU NN , CNN , ResNet ) do not have such structures . [ 40 ] analyzed NN trained by two-layer NN trained by hinge loss without regularization on linearly separable dataset . Note for SVM , it must have a regularization term such that it can achieve max-margin solution . 2.2 Neural Networks and Tangent Kernel . We consider a general form of deep neural network f with a linear output layer as [ 32 ] . Let [ L ] = { 1 , ... , L } , 8l 2 [ L ] , ↵ ( 0 ) ( w , x ) = x , ↵ ( l ) ( w , x ) = l ( w ( l ) , ↵ ( l 1 ) ) , f ( w , x ) = 1 p mL hw ( L+1 ) , ↵ ( L ) ( w , x ) i , ( 1 ) where each vector-valued function l ( w ( l ) , · ) : Rml 1 ! Rml , with parameter w ( l ) 2 Rpl ( pl is the number of parameters ) , is considered as a layer of the network . This definition includes the standard fully connected , convolutional ( CNN ) , and residual ( ResNet ) neural networks as special cases . For a fully connected ReLU NN , ↵ ( l ) ( w , x ) = ( 1pml 1w ( l ) ↵ ( l 1 ) ) with w ( l ) 2 Rml⇥ml 1 and ( z ) = max ( 0 , z ) . Initialization and parameterization . In this paper , we consider the NTK parameterization [ 27 ] , under which the constancy of the tangent kernel has been initially observed . Specifically , the parameters , w : = { w ( 1 ) ; w ( 2 ) ; · · · ; w ( L ) ; w ( L+1 ) } are drawn i.i.d . from a standard Gaussian , N ( 0 , 1 ) , at initialization , denoted as w0 . The factor 1/ p mL in the output layer is required by the NTK parameterization in order that the output f is of order O ( 1 ) . While we only consider NTK parameterization here , the results should be able to extend to general parameterization of kernel regime [ 53 ] . Definition 2.1 ( Tangent Kernel ) . The tangent kernel associated with function f ( w , x ) at some parameter w is ⇥̂ ( w ; x , x0 ) = hrwf ( w , x ) , rwf ( w , x0 ) i . Under certain conditions ( usually infinite width limit and NTK parameterization ) , the tangent kernel at initialization converges in probability to a deterministic limit and keeps constant during training , ⇥̂ ( w ; x , x0 ) ! ⇥1 ( x , x0 ) . This limiting kernel is called Neural Tangent Kernel ( NTK ) . 2.3 Kernel Machines . Kernel machine ( KM ) is a model of the form g ( , x ) = ' ( h , ( x ) i + b ) , where is the model parameter and is a mapping from input space to some feature space , : X ! F . ' is an optional nonlinear function , such as identity mapping for kernel regression and sign ( · ) for SVM and logistic regression . The kernel can be exploited whenever the weight vector can be expressed as a linear combination of the training points , = Pn i=1 ↵i ( xi ) for some value of ↵i , i 2 [ n ] , implying that we can express g as g ( x ) = ' ( Pn i=1 ↵iK ( x , xi ) +b ) , where K ( x , xi ) = h ( x ) , ( xi ) i is the kernel function . For a neural network in NTK regime , we have f ( wt , x ) ⇡ f ( w0 , x ) + hrwf ( w0 , x ) , wt w0i , which makes the neural network linear in the gradient feature mapping x ! rwf ( w0 , x ) . Under squared loss , it is equivalent to kernel regression with ( x ) = rwf ( w0 , x ) ( or equivalently using NTK as the kernel ) , = wt w0 and ' identity mapping [ 4 ] . As far as we know , there is no work establishing the equivalence between fully trained networks and SVM . [ 17 ] showed that neural networks trained by gradient flow are approximately KMs , but didn ’ t discuss any specific KM . In this work , we compare the dynamics of SVM and neural network trained by subgradient descent with soft margin loss and show the equivalence between them in the infinite width limit .

Summary: This paper theoretically analyzes the connection between Neural Network (NN) and Support Vector Machine (SVM). It shows the equivalence between SVM with Neural tangent Kernel (NTK) and infinite-width NN trained by soft margin loss with subgradient descent.  Besides, they extend their theory to general $\mathit{l}_2$ regularized loss functions and show finite-width NN trained by a $\mathit{l}_2$ regularized loss function is approximately a kernel machine.  It provides robustness certificates for infinite-width NN.