Learning Backpropagation-Free Deep Architectures with Kernels

One can substitute each neuron in any neural network with a kernel machine and obtain a counterpart powered by kernel machines . The new network inherits the expressive power and architecture of the original but works in a more intuitive way since each node enjoys the simple interpretation as a hyperplane ( in a reproducing kernel Hilbert space ) . Further , using the kernel multilayer perceptron as an example , we prove that in classification and under certain losses , an optimal representation that minimizes the risk of the network can be characterized for each hidden layer . This result removes the need of backpropagation in learning the model and can be generalized to any feedforward kernel network . Moreover , unlike backpropagation , which turns models into black boxes , the optimal hidden representation enjoys an intuitive geometric interpretation , making the dynamics of learning in a deep kernel network transparent . Empirical results are provided to complement our theory . 1 INTRODUCTION . Any neural network ( NN ) can be turned into a kernel network ( KN ) by replacing each artificial neuron ( McCulloch & Pitts , 1943 ) , i.e. , learning machine of the form f ( x ) = σ ( w > x + b ) , with a kernel machine , i.e. , learning machine of the form f ( x ) = 〈w , φ ( x ) 〉 + b with kernel function k ( x , y ) = 〈φ ( x ) , φ ( y ) 〉 . This combination of connectionism and kernel method enables the learning of hierarchical , distributed representations with kernels . In terms of training , similar to NN , KN can be trained with backpropagation ( BP ) ( Rumelhart et al. , 1986 ) . In the context of supervised learning , the need for BP in learning a deep architecture is caused by the fact that there is no explicit target information to tune the hidden layers ( Rumelhart et al. , 1986 ) . Moreover , BP is usually computationally intensive and can suffer from vanishing gradient . And most importantly , BP results in hidden representations that are notoriously difficult to interpret or assess , turning deep architectures into “ black boxes ” . The main theoretical contribution of this paper is the following : Employing the simplest feedforward , fully-connected KN as an example , we prove that in classification and under certain losses , the optimal representation for each hidden layer that minimizes the risk of the network can be explicitly characterized . This result removes the need for BP and makes it possible to train the network in a feedforward , layer-wise fashion . And the same idea can be generalized to other feedforward KNs . The layer-wise learning algorithm gives the same optimality guarantee as BP in the sense that it minimizes the risk . But the former is much faster and evidently less susceptible to vanishing gradient . Moreover , the quality of learning in the hidden layers can be directly assessed during or after training , providing more information about the model to the user . For practitioners , this enables completely new model selection paradigms . For example , the bad performance of the network can now be traced to a certain layer , allowing the user to debug the layers individually . Most importantly , the optimal representation for each hidden layer enjoys an intuitive geometric interpretation , making the learning dynamics in a deep KN more transparent than that in a deep NN . A simple acceleration method that utilizes the “ sparse ” nature of the optimal hidden representations is proposed to further reduce computational complexity . Empirical results on several computer vision benchmarks are provided to demonstrate the competence of the model and the effectiveness of the greedy learning algorithm . 2 FROM NEURAL TO KERNEL NETWORKS . In this section , we discuss how to build a KN using a given NN . First , the generic approach is described in Fig . 1a . Note that KN inherits the expressive power of the original NN since a kernel machine is a universal function approximator under mild conditions ( Park & Sandberg , 1991 ; Micchelli et al. , 2006 ) and the two models share the same architecture . However , KN works in a more intuitive way since each node is a simple linear model in a reproducing kernel Hilbert space ( RKHS ) . We now concretely define the KN equivalent of an l-layer Multilayer Perceptron ( MLP ) , which we shall refer to as the kernel MLP ( kMLP ) .1 Given a random sample ( xn , yn ) Nn=1 , where ( xn , yn ) ∈ X1 × Y ⊂ Rd0 × R , denote ( xn ) Nn=1 as S and ( yn ) Nn=1 as YS for convenience . For i = 1 , 2 , . . . , l , consider kernel k ( i ) : Xi ×Xi → R , Xi ⊂ Rdi−1 ( for i > 1 , di−1 is determined by the width of the i− 1th layer ) . k ( i ) ( x , y ) = 〈 φ ( i ) ( x ) , φ ( i ) ( y ) 〉 Hi , where φ ( i ) is a mapping into RKHS Hi . For i ≥ 1 , the ith layer in a kMLP , denoted F ( i ) , is an array of di kernel machines : F ( i ) : Xi → Rdi , F ( i ) = ( f ( i ) 1 , f ( i ) 2 , . . . , f ( i ) di ) , a di-tuple . Let F ( 0 ) be the identity map on Rd0 , each f ( i ) j : Xi → R is a hyperplane inHi : f ( i ) j ( x ) = 〈 w ( i ) j , φ ( i ) ( F ( i−1 ) ◦· · ·◦F ( 0 ) ( x ) ) 〉 Hi +b ( i ) j , w ( i ) j ∈ Hi , b ( i ) j ∈ R. In practice , f ( i ) j ( x ) is usually implemented as ∑N n=1 α ( i ) njk ( i ) ( F ( i−1 ) ◦ · · · ◦ F ( 0 ) ( x ) , F ( i−1 ) ◦ · · · ◦ F ( 0 ) ( xn ) ) + b ( i ) j , where the α ( i ) nj , b ( i ) j ∈ R are the learnable parameters.2 The set of mappings { F ( ` ) ◦ · · · ◦ F ( 1 ) : α ( i ) nj , b ( i ) j ∈ R for all admissible n , j , i } defines an l-layer kMLP . In the rest of this paper , we shall restrict our discussions to this kMLP . 3 ASSUMPTIONS AND NOTATIONS . We now specify the assumptions that we impose on all kernels considered in this paper . First , we consider real , continuous , symmetric kernels only and we call a kernel positive semidefinite ( PSD ) or positive definite ( PD ) if for any S , the kernel matrix defined as ( G ) mn = k ( xm , xn ) is PSD or PD , respectively . We shall always assume that any kernel considered is at least PSD and that 1A PyTorch-based ( Paszke et al. , 2017 ) library for implementing kMLP and the proposed layer-wise learning algorithm is available at : anonymized URL . 2The optimality of this expansion will be later justified in Section 4 under the layer-wise setting using representer theorem ( Schölkopf et al. , 2001 ) . k ( i ) ( x , x ) = c < +∞ for all x ∈ Xi and infx , y∈Xi k ( i ) ( x , y ) = a > −∞ . It is straightforward to check using Cauchy-Schwarz inequality that the first condition implies maxx , y∈Xi k ( i ) ( x , y ) = c. For each fixed x ∈ Xi , we assume that k ( i ) ( x , y ) , as a function of y , is L ( i ) x -Lipschitz with respect to the Euclidean metric on Xi . Let supx∈Xi L ( i ) x = L ( i ) , which we assume to be finite . The following notations will be used whenever convenient : We use the shorthand F ( 1 ) ( S ) for ( F ( 1 ) ( xn ) ) N n=1 . For i = 2 , 3 , . . . , l , F ( i ) ( S ) : = F ( i ) ◦ F ( i−1 ) ◦ · · · ◦ F ( 1 ) ( S ) and the same is true with S substituted by any x . Throughout this paper , notations such as F ( i ) can either be used to denote a set of functions or a specific function in some set depending on the context . Also , when there is no confusion , we shall suppress the dependency of any loss function on the example for brevity , i.e. , for a loss function ` , instead of writing ` ( f ( x ) , y ) , we shall write ` ( f ) . 4 A LAYER-WISE LEARNING ALGORITHM . To simplify discussion , we shall restrict ourselves to binary classification ( Y = { +1 , −1 } ) and directly give the result on classification with more than two classes in the end . A generalization to regression is left as future work . Again , we only focus on kMLP although the idea can be directly generalized to all feedforward KNs . We now discuss the layer-wise learning algorithm , beginning by addressing the difficulties with training a deep architecture layer-by-layer . 4.1 FUNDAMENTAL DIFFICULTIES . There are two fundamental difficulties with learning a deep architecture layer-wise . First , the hidden layers do not have supervision ( labels ) to learn from . And it depends on BP to propagate supervision from the output backward ( Rumelhart et al. , 1986 ) . We shall prove that for kMLP , one can characterize the optimal target representation for each hidden layer , which induces a risk for that layer . The target is optimal in the sense that it minimizes the risk of the subsequent layer and eventually that of the network if all layers succeed in learning their optimal representations . This optimal representation defines what we call “ layer-wise optimality ” . The other difficulty with layer-wise learning is that for any given hidden layer , when the upstream layers create error , layer-wise optimality may not coincide with “ network-wise optimality ” , i.e. , the solution of this layer that eventually leads to the composed solution that minimizes the risk of the network in this suboptimal case . Indeed , when a hidden layer creates error , the objective of any layer after it becomes learning a solution that is a compromise between one that is close to the layer-wise optimality and one that prevents the error from the “ bad ” layer before it from “ getting through ” easily . And the best compromise is the network-wise optimality . The two solutions may not coincide , as shown in the toy example in Fig . 1b . Clearly , we would like to always learn the network-wise optimality at each layer , but the learner is blind to it if it is only allowed to work on one layer at a time . By decomposing the overall error of the network into error at each layer , we prove that in fact , network-wise optimality is learnable for each hidden layer even in a purely layer-wise fashion and that the proposed layer-wise algorithm learns network-wise optimality at each layer .

Summary: The paper considers so-called kernel neural networks where the non-linear activation function at each neuron is replaced by a kernelized linear operation, and analyses a layer-wise training scheme to train such networks. The theoretical claims are that (i) the optimal representation at each hidden layer can be determined by getting the similarity between two kernel matrices and (ii) this procedure gives a more interpretable training procedure and can avoid the vanishing gradient problems. Some small-scale experiments are provided.