Deep Learning Is Composite Kernel Learning

1 INTRODUCTION . The success of deep learning is attributed to feature learning . The conventional view is that feature learning happens in the hidden layers of a deep network : in the initial layers simple low level features are learnt , and sophisticated high level features are learnt as one proceeds in depth . In this viewpoint , the penultimate layer output is the final hidden feature and the final layer learns a linear model with these hidden features . While this interpretation of feature learning is intuitive , beyond the first couple of layers it is hard make any meaningful interpretation of what happens in the intermediate layers . Recent works Jacot et al . ( 2018 ) ; Arora et al . ( 2019 ) ; Cao and Gu ( 2019 ) have provided a kernel learning interpretation for deep learning by showing that in the limit of infinite width deep learning becomes kernel learning . These works are based on neural tangents , wherein , the gradient of the network output with respect to the network parameters known as the neural tangent features ( NTFs ) are considered as the features . Arora et al . ( 2019 ) show that at randomised initialisation of weights , the kernel matrix associated with the NTFs , known as the neural tangent kernel ( NTK ) converges to a deterministic matrix and that optimisation and generalisation of infinite width deep neural networks is characterised by this deterministic kernel matrix . Cao and Gu ( 2019 ) provided generalisation bounds in terms of the NTK matrix . Arora et al . ( 2019 ) also proposed a pure-kernel method based on CNTK ( NTK of convolutional neural networks , i.e. , CNNs ) which significantly outperformed the previous state-of-the-art kernel methods . The NTK either as an interpretation or as a method in itself has been very successful . Nevertheless it has some open issues namely i ) non-interpretable : the kernel is the inner product of gradients and has no physical interpretation , ii ) no feature learning : the NTFs are random and fixed during training and iii ) performance gap : finite width CNN outperforms the infinite width CNTK , i.e. , NTK does not fully explain the success of deep learning . Recently , Lakshminarayanan and Singh ( 2020 ) developed a neural path ( NP ) framework to provide a kernel interpretation for deep learning that addresses the open issues in the current NTK framework . Here , DNNs with ReLU activations are considered , and the gates ( on/off state of ReLU ) are encoded in the so called neural path feature ( NPF ) and the weights in the network in the so called neural path value ( NPV ) . The key findings can be broken into the following steps . Step 1 : The NPFs and NPV are decoupled . Gates are treated as masks , which are held in a separate feature network and applied to the main network called the value network . This enables one to study the various kinds of gates ( i.e. , NPFs ) , such as random gates ( of a randomly initialised network ) , semi-learnt gates ( sampled at an intermediate epoch during training ) , and learnt gates ( sampled from a fully trained network ) . This addresses the feature learning issue . Step 2 : When the gates/masks are decoupled and applied externally it follows that NTK = const ⇥ NPK , at random initialisation of weights . For a pair of input examples , NPK is a similarity measure 1Introduced for the first time in the work of Lakshminarayanan and Singh ( 2020 ) . that depends on the size of the sub-network formed by the gates that are active simultaneously for examples . This addresses the interpretability issue . Step 3 : CNTK performs better than random gates/masks and gates/masks from fully trained networks perform better than CNTK . This explains the performance gap between CNN and CNTK . It was also observed ( on standard datasets ) that when learnt gates/masks are used , the weights of the value network can be reset and re-trained from scratch without significant loss of performance . 1.1 CONTRIBUTIONS IN THIS WORK . We attribute the success of deep learning to the following two key ingredients : ( i ) a composite kernel with gates as fundamental building blocks and ( ii ) allowing the gates to learn/adapt during training . Formally , we extend the NP framework of Lakshminarayanan and Singh ( 2020 ) as explained below . • Composite Kernel : The NPK matrix has a composite structure ( architecture dependent ) . 1 . Fully-Connected networks : H fc is the Hadamard product of the input data Gram matrix , and the kernel matrices corresponding to the binary gating features of the individual layers . 2 . Residual networks ( ResNets ) with skip connections : H res assumes a sum of products form . In particular , consider a ResNet with ( b + 2 ) blocks and b skip connections . Within this ResNet there are i = 1 , . . . , 2b possible dense networks , and then H res = P2b i=1 CiH fc i , where Ci > 0 are positive constants based on normalisation layers . 3 . Convolutional neural networks ( CNN ) with pooling : Hcnn is rotation invariant . • Gate Learning : We show that learnt gates perform better than random gates . Starting with the setup of Lakshminarayanan and Singh ( 2020 ) , we build combinatorially many models by , 1. permuting the order of the layers when we apply them as external masks , 2. having two types of modes based on input provided to the value network namely i ) ‘ standard ’ : input is the actual image and ii ) ‘ all-ones ’ : input is a tensor with all entries as ‘ 1 ’ . We observe in our experiments that the performance is robust to such combinatorial variations . Message : This work along with that of Lakshminarayanan and Singh ( 2020 ) provides a paradigm shift in understanding deep learning . Here , gates play a central role . Each gate is related to a hyperplane , and gates together form layer level binary features whose kernels are the base kernels . Laying out these binary features depth-wise gives rise to a product of the base kernels . The skip connections gives a ‘ sum of product ’ structure , and convolution with pooling gives rotation invariance . Organisation : Section 2 describes the network architectures namely fully-connected , convolutional and residual , which we take up for theoretical analysis . Section 3 extends the neural path framework to CNN and ResNet . Section 4 explains the composite kernel . Section 5 connects the NTK and NPK for CNN and ResNet . Section 6 consists of numerical experiments . 2 ARCHITECTURES : FULLY CONNECTED , CONVOLUTIONAL AND RESIDUAL . In this section , we present the three architectures that we take up for theoretical analysis . These are i ) fully connected ( FC or FC-DNN ) , ii ) convolutional ( CNN ) and iii ) residual ( ResNets ) . In what follows , [ n ] is the set { 1 , . . . , n } , and the dataset is given by ( xs , ys ) ns=1 2 Rdin ⇥ R. Fully Connected : We consider fully connected networks with width ‘ w ’ and depth ‘ d ’ . CNN : We consider a 1-dimensional convolutional neural network with circular convolutions ( see Table 2 ) , with dcv convolutional layers ( l = 1 , . . . , dcv ) , followed by a global-average/max-pooling layer ( l = dcv + 1 ) and dfc ( l = dcv + 2 , . . . , dcv + dfc + 1 ) FC layers . The convolutional window size is wcv < din , the number of filters per convolutional layer is w , and the width of the FC is also w. Definition 2.1 ( Circular Convolution ) . For x 2 Rdin , i 2 [ din ] and r 2 { 0 , . . . , din 1 } , define : ( i ) i r = i+ r , for i+ r din and i r = i+ r din , for i+ r > din . ( ii ) rot ( x , r ) ( i ) = x ( i r ) , i 2 [ din ] . ( iii ) qx , ⇥ ( ifout , iout , l ) = P icv , iin ⇥ ( icv , iin , iout , l ) · zx , ⇥ ( ifout ( icv 1 ) , iin , l 1 ) , where iin/iout are the indices ( taking values in [ w ] ) of the input/output filters . icv denotes the indices of the convolutional window ( taking values in [ wcv ] ) between input and output filters iin and iout . ifout denotes the indices ( taking values in [ din ] , the dimension of input features ) of individual nodes in a given output filter . Definition 2.2 ( Pooling ) . Let Gpool x , ⇥ ( ifout , iout , dcv + 1 ) denote the pooling mask , then we have zx , ⇥ ( iout , dcv + 1 ) = X ifout zx , ⇥ ( ifout , iout , dcv ) ·Gpoolx , ⇥ ( ifout , iout , dcv + 1 ) , where in the case of global-average-pooling Gpool x , ⇥ ( ifout , iout , dcv + 1 ) = 1 din , 8iout 2 [ w ] , ifout 2 [ din ] , and in the case of max-pooling , for a given iout 2 [ w ] , Gpoolx , ⇥ ( imax , iout , dcv + 1 ) = 1 where imax = argmaxifout zx , ⇥ ( ifout , iout , dcv ) , and G pool x , ⇥ ( ifout , iout , dcv + 1 ) = 0 , 8ifout 6= imax . ResNet : We consider ResNets with ‘ ( b + 2 ) ’ blocks and ‘ b ’ skip connections between the blocks ( Figure 1 ) . Each block is a FC-DNN of depth ‘ dblk ’ and width ‘ w ’ . Here , pre i , post i , i 2 [ b ] are normalisation variables . Definition 2.3 ( Sub FC-DNNs ) . Let 2 [ b ] denote the power set of [ b ] and let J 2 2 [ b ] denote any subset of [ b ] . Define the ‘ J th ’ sub-FC-DNN of the ResNet to be the fully connected network obtained by ignoring/removing ( see Figure 1 ) the skip connections skip j , 8j 2 J ( see Figure 1 ) . 3 NEURAL PATH FRAMEWORK . In this section , we extend the neural path framework developed by LS2020 , to CNN and ResNet architectures described in the previous section . The neural path framework exploits the gating property of ReLU activation , which can be thought of as gate/mask that blocks/allows its pre-activation input depending on its 0/1 state ( 0 if pre-activation is negative and 1 if pre-activation is positive ) . The key idea here is to break a DNN ( with ReLU ) into paths , and express its output as a summation of the contribution of the paths . The contribution of a path is the product of the signal in its input node , the weights in the path and the gates in the path . For a DNN with P paths , for an input x 2 Rdin , the gating information is encoded in a novel neural path feature ( NPF ) , x , ⇥ 2 RP and a novel neural path value ( NPV ) , v⇥ 2 RP encodes the weights . The output of the DNN is then the inner product of the NPFs and NPVs , i.e. , ŷ⇥ ( xs ) = h xs , ⇥ , v⇥i ( Proposition 3.4 ) . Definition 3.1 . A path starts from an input node , passes through weights , hidden nodes , and normal- isation constants and ends at the output node . Proposition 3.1 . The total number of paths in FC-DNN , CNN and ResNet are respectively given by P fc = dinw ( d 1 ) , P cnn = din ( wcvw ) dcvw ( dfc 1 ) and P res = din · P b i=0 b i w ( i+2 ) dblk 1 . Notation 3.1 ( Index Maps ) . The ranges of index maps I f l , Icv l , Il are [ din ] , [ wcv ] and [ w ] respectively . The index maps are used to identify the nodes through which a path p passes . Further , let IJ ( p ) : [ P res ] ! 2 [ b ] specify the indices of the skip connections ignored in path p. Also , we follow the convention that weights and gating values of layers corresponding to blocks skipped are 1 . Definition 3.2 ( Path Activity ) . The product of the gating values in a path p is its ‘ activity ’ denoted by A⇥ ( x , p ) . We define : ( a ) A⇥ ( x , p ) = ⇧ d 1 l=1 Gx , ⇥ ( Il ( p ) , l ) , for FC-DNN and ResNet . ( b ) A⇥ ( x , p ) = ⇧ dcv+1 l=1 Gx , ⇥ ( I f l ( p ) , Il ( p ) , l ) ·⇧ dcv+dfc+1 l=dcv+2 Gx , ⇥ ( Il ( p ) , l ) , for CNN . In CNN , the pooling layer is accounted by letting G = Gpool for l = dcv + 1 . Definition 3.3 ( Bundle Paths of Sharing Weights ) . Let P̂ cnn = P cnn din , and { B1 , . . . , BP̂ cnn } be a collection of sets such that 8i , j 2 [ P̂ cnn ] , i 6= j we have Bi \Bj = ; and [ P̂ cnn i=1Bi = [ P cnn ] . Further , if paths p , p 0 2 Bi , then Icvl ( p ) = Icvl ( p0 ) , 8l = 1 , . . . , dcv and Il ( p ) = Il ( p0 ) , 8l = 0 , . . . , dcv . Proposition 3.2 . There are exactly din paths in a bundle . Definition 3.4 ( Normalisation Factor ) . Define ( J ) = ⇧ j2J pre j · ⇧ j02 [ b ] post j0 Weight sharing is shown in the the cartoon in Figure 2 , which shows a CNN with din = 3 , w = 1 , wcv = 2 , dcv = 3 , dfc = 0 . Here , the red coloured paths all share the same weights ⇥ ( 1 , 1 , 1 , l ) , l = 1 , 2 , 3 and the blue coloured paths all share the same weights given by ⇥ ( 2 , 1 , 1 , l ) , l = 1 , 2 , 3 . Definition 3.5 ( Neural Path Value ) . The product of the weights and normalisation factors in a path p is its ‘ value ’ . The value of a path bundle is the value of any path in that bundle . The path/bundle values are denoted by v⇥ ( p ) /v⇥ ( Bp̂ ) and are defined as follows : ( a ) v⇥ ( p ) = ⇧dl=1⇥ ( Il 1 ( p ) , Il ( p ) , l ) . ( b ) v⇥ ( Bp̂ ) = ⇧ dcv l=1⇥ ( Icvl ( p ) , Il 1 ( p ) , Il ( p ) , l ) ·⇧ dcv+dfc+1 l=dcv+2 ⇥ ( Il 1 ( p ) , Il ( p ) , l ) , for any p 2 Bp̂ . ( c ) v⇥ ( p ) = ⇧dl=1⇥ ( Il 1 ( p ) , Il ( p ) , l ) · ( IJ ( p ) ) . The neural path value is defined as v⇥ = ( v⇥ ( p ) , p 2 [ P fc ] ) 2 RP fc , v⇥ = ( v⇥ ( Bp̂ ) , p̂ 2 [ P̂ cnn ] ) 2 RP̂ cnn , and v⇥ = ( v⇥ ( p ) , p 2 [ P res ] ) 2 RP res for FC-DNN , CNN and ResNet respectively . Proposition 3.3 ( Rotational Invariance ) . Internal variables in the convolutional layers are circularly symmetric , i.e. , for r 2 { 0 , . . . , din 1 } it follows that ( i ) zrot ( x , r ) , ⇥ ( ifout , · , · ) = zx , ⇥ ( ifout r , · , · ) , ( ii ) qrot ( x , r ) , ⇥ ( ifout , · , · ) = qx , ⇥ ( ifout r , · , · ) and ( iii ) Grot ( x , r ) , ⇥ ( ifout , · , · ) = Gx , ⇥ ( ifout r , · , · ) . Definition 3.6 . The neural path feature ( NPF ) corresponding to a path p is given by ( a ) x , ⇥ ( p ) = x ( I f0 ( p ) ) A⇥ ( xs , p ) for FC-DNN and ResNet . ( b ) x , ⇥ ( p̂ ) = P p̂2Bp̂ x ( I f 0 ( p ) ) A⇥ ( x , p ) for CNN . The NPF is defined as x , ⇥ = ( x , ⇥ ( p ) , p 2 [ P fc ] ) 2 RP fc , x , ⇥ = ( x , ⇥ ( Bp̂ ) , p̂ 2 [ P̂ cnn ] ) 2 RP̂ cnn , and x , ⇥ = ( x , ⇥ ( p ) , p 2 [ P res ] ) 2 RP res for FC-DNN , CNN and ResNet respectively . Proposition 3.4 ( Output=hNPF , NPVi ) . The output of the network can be written as an inner product of the NPF and NPV , i.e. , ŷ⇥ ( x ) = h x , ⇥ , v⇥i .

Summary: This paper builds on recent work characterising deep neural networks in terms of Neural Tangent Kernels and Neural Path Features. Over the past few years, a number of papers have developed the theory of Neural Tangent Kernels, which can be used to interpret infinite width deep neural networks in the context of a particular type of kernel. A recent paper (Lakshminarayanan and Singh, NeurIPS 2020) provided a new perspective on Neural Tangent Kernels for Gated Neural Networks, by decomposing the network into independent paths. For a fixed set of network weights, we can consider each path to give rise to a feature, corresponding to whether this path is active (i.e., is not switched off by one of the gates on the path). Then, the output of the neural network can be viewed as a weighted sum of active paths, equivalently the dot product of the neural path feature vector and a neural path value vector. Lakshminarayanan and Singh showed that under certain assumptions, a kernel defined in terms of the neural path feature is approximately equal to the neural tangent kernel (up to a constant). Specifically, they show that the value of the neural tangent kernel matrix tends to a constant multiple of the neural path kernel matrix as the width of the network goes to infinity. This suggests that the key component in a deep neural network with RELU activations is the gating structure, which defines active subnetworks, as opposed to the values.