Adaptive Convolutional Neural Networks

1 INTRODUCTION . Convolutional Neural Networks ( CNN ) have demonstrated their capacity to achieve state of the art accuracy in image classification , semantic segmentation , and object detection . In order to increase the accuracy of recognition , most of these works rely on deeper and deeper architectures with millions of parameters . The reason behind this is that more complex features can be abstracted as we add more layers to a network . Unfortunately such models can not be used in embedded devices , cellphones or drones due to the size of the models . Recently there are many approaches like ShuffleNet Xiangyu Zhang ( 2017 ) , MobileNet Andrew G. Howard ( 2017 ) , HENet Qiuyu Zhu ( 2018 ) and SqueezeNet Forrest N. Iandola ( 2017 ) trying to generate small models with an small drop in accuracy . The proposed method generates the smallest model with comparable accuracy . In addition this method can be combined with most of the existent techniques like ResNet generating an improved version of it . Previous research has demonstrated that the response of visual cells is a non-linear function of their stimuli ( Szulborski & Palmer , 1990 ) . Thus , finding non-linear models that best represent data is imperative . Given that a typical convolutional layer is a linear system , its ability to express this response is limited by the layers and the number of neurons in the intermediate layers . The use of a non-linear neuron , as it was firstly explored to solve the XOR problem by Minsky & Papert ( 1969 ) , ( which can not be solved by a first order neuron , but it can by a second order neuron ) , seems an appropriate way to tackle this kind of non linearity issue . Ideally , such non-linear approaches should be able to provide similar accuracy as compared to traditional CNNs , albeit at a much lower computation and memory costs . In other hand , in computer vision the filter used to extract borders is different to the filter used to extract corners , etc . Our method uses the input image to define the filter that better fits in on the specific location . Motivated by this , we developed a convolutional kernel , that includes non-linear transformations obtaining similar results as the state of the art algorithms , while yielding a reduction in required memory up to 14x in the CIFAR10 ( Krizhevsky , 2009 ) classification , and up to 66x for the MNIST classification . The main contributions of this paper are : ( i ) we present the non-linear convolutions designed for high visual classification accuracy under memory constraints ; ( ii ) using the proposed convolutions , we present a network design that is partially pre-defined and is capable of completing self-definition during the pattern evaluation phase , including defining the convolutional kernels on the fly depending of the input pattern ; ( iii ) we present a method to tackle problems associated with higher order neural networks like the saturation of the activation due to the N-order of multiplications used ; ( iv ) a method to constrain every new dynamically generated weight to a pre-known range defined by the activation function , for instance if hyperbolic tangent function was used , the dynamically generated weights would be all in ( −1 , 1 ) range ; and ( v ) a pytorch-based implementation located in https : //github.com/adapconv/adaptive-cnn ( 2018 ) . This work is organized as follows : Section 2 describes previous approaches , Section 3 explains in detail our proposed method , Section 4 shows the results for MNIST , CIFAR10 and Navigation . We end with our conclusions in Section 5 . 2 RELATED WORK . Different ways of increasing the accuracy of neural networks have been addressed in the existing scientific literature , most of these rely on the use CNNs , as they are generalized linear models , and their level of abstraction is low ( Lin et al. , 2013 ) . Some approaches have dealt with this low abstraction by having additional layers ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ) , resulting in a considerable increase in the accuracy on different datasets , e.g . CIFAR10 ( Krizhevsky , 2009 ) and ImageNet ( Deng et al. , 2009 ) . However , although the network depth has shown a crucial importance in neural network performance , the difficulty to train the network also increases , and moreover , the accuracy of the network drops ( He et al. , 2015 ) . A proposal to tackle this problem is the use of ResNet blocks He et al . ( 2015 ) , which given a set of inputs X , with an associated label Y and a function H ( x ) that maps X to Y , the networks define a building block y = F ( x ) + x , where F ( x ) represents the residual mapping to be learned . This residual network allows to train a deeper network without being affected by the degradation problem , using a larger amount of layers and parameters . In the work presented in Lin et al . ( 2013 ) a nonlinear function approximator is proposed as a solution to increase the level of abstraction . The typical convolutional kernel is replaced with a micro network , i.e . a non linear approximator . A multilayer perceptron is used as the instantiation of this micro network , and by sliding the micro network over the input in a similar manner as convolutional neural networks the feature maps are obtained . In the approach of Brabandere et al . ( 2016 ) , a dynamic filter module is introduced , where the filters used for the convolution are generated dynamically depending on the input . This module consists of two parts : a filter generating network , which generates sample specific parameters given an input ; and the dynamic filtering layer , which applies the parameters to another input . Although the mentioned work presents similar components to our proposal , it is important to notice the differences between these : in Brabandere et al . ( 2016 ) a CNN is trained to get the convolutional kernels , and then another network is defined to generate such kernels , where this second network is trained independently . In our work , we define a second order convolutional kernel trained using a novel training rule , which is explained in detail in the next section . 3 METHOD . An Adaptive Kernel is a dynamic kernel that changes its weights by itself depending on the input image . An Adaptive Kernel can be viewed as an array of traditional kernels . For instance , each element in a 3x3 adaptive kernel is a 3x3 linear kernel , as shown in the Figure 1 . The convolution of each kernel element Qu , v with the window X of the input image generates a component of the new kernel Ku , v after applying the activation function ( Figure 1 ) . This new dynamically generated kernel K is convolved again with the same window X of the input image ( Figure 2 ) . As result of this second order convolution and the activation function an output pixel value is obtained . By sliding the window through all the input image , we generate the final filtered image . 3.1 FEED FORWARD . Convolving the kernel Qu , v in the Figure 1 and using tanh as activation function the weights Ku , v are generated Ku , v ( σ ) = tanh ( σu , v ) , ( 1 ) where σu , v is the convolution of each linear kernel Qu , v with the input image σu , v = N−1∑ i=0 N−1∑ j=0 Q ( u , v ) i , jxi , j , ( 2 ) This new kernel Ku , v is now convolved with the input image to compute S in this location S = ∑ u , v xu , vK ∑ i , j Qu , vi , jxi , j ( 3 ) and finally the output pixel is computed using hyperbolic tangent as activation function like f = tanh ( S ) . Another activation functions can be used , like sigmoid or Relu . We use hyperbolic tangent to generate weights in the range of ( −1 , 1 ) . 3.2 TRAINING . For the Adaptive kernel a new training rule is obtained using gradient descent technique in order to adjust the weights of adaptive kernel Q . The kernel can be seen as an array of NxN traditional kernels , using Qu , v to refer to each linear kernel and using ( i , j ) to refer the elements ( weights ) of each Qu , v kernel . It means the element Q ( u , v ) ( i , j ) is a scalar value and represents a weight . The kernel output f ( S ) k , l , where f is the activation function and S represents sum of input xu , v weighted by σu , v as shown in Equation ( 3 ) . The error for output pixel ( k , l ) denoted as Ek , l is given by : Ek , l = 1 2 ( dk , l − f ( s ) k , l ) 2 , ( 4 ) where dk , l represents the expected output at coordinates ( k , l ) . The training rule for the t − th iteration ( Qt ) is given by the error derivative per component ( u , v ) i , j : ∂Ek , l ∂Qt ( u , v ) ( i , j ) = ( dk , l − f ( s ) k , l ) ∂f ( s ) k , l ∂sk , l ∂Sk , l ∂Qt ( u , v ) ( i , j ) ( 5 ) The gradient descent updates of the weights Qt ( u , v ) i , j are given by : Qt+1 ( u , v ) ( i , j ) = Qt ( u , v ) ( i , j ) + γ ∑ k , l ( dk , l − f ( s ) k , l ) ∂f ( s ) k , l ∂Qt ( u , v ) ( i , j ) ( 6 ) where γ represents the learning rate . Using the hyperbolic tangent as the activation function f ( S ) k , l = tanh ( Sk , l ) : ∂f ( S ) k , l ∂Sk , l ∂Sk , l ∂Qt ( u , v ) ( i , j ) = ( 1− f ( S ) 2k , l ) ∂S ∂Qt ( u , v ) ( i , j ) , ( 7 ) where Sk , l is the weighted sum computed as : Sk , l = N−1∑ u=0 N−1∑ v=0 ( Kk , lu , v ) ( xu+k , v+l ) . ( 8 ) Here , N is the size of the kernel , and Kk , lu , v ( σ ) = tanh ( σ k , l u , v ) , with : σk , lu , v = N−1∑ i=0 N−1∑ j=0 Q ( u , v ) i , jxi+k , j+l , ( 9 ) Since the hyperbolic tangent was used as the activation function , the gradient is : ∂f ( s ) k , l ∂Qt ( u , v ) ( i , j ) = ( 1− f ( S ) 2k , l ) N−1∑ u=0 N−1∑ v=0 ( 1− ( Kk , lu , v ( σ ) ) 2 ) ( xu+k , v+l ) ( xi+k , j+l ) ( 10 ) by replacing ( 10 ) in ( 6 ) , the final training rule is defined by : Qt+1 ( u , v ) ( i , j ) = Qt ( u , v ) ( i , j ) + γ ∑ k , l ( dk , l − f ( S ) k , l ) ( 1− f ( S ) 2 k , l ) δk , l ( 11 ) where γ is the learning rate , dk , l is the desired value at the window position ( k , l ) , f is the activation function and δk , l is δk , l = ∑ u , v ( 1− ( Kk , lu , v ( σ ) ) 2 ) ( xu+k , v+l ) ( xi+k , j+l ) ( 12 ) For this training rule tanh was used as activation function to generate the kernel weights , it ensures all weight are in the range of ( −1 , 1 ) . Additionally , this layer allows the reduction of filters in the subsequent layers without affecting the performance of the network , on the experimental results different experiments will be used to compared against state of the art for MNIST and CIFAR10 to show a significant memory compression using the proposed models .

Summary: The paper introduces adaptive kernels (that adapts its weights as a function of image content) to the framework of CNN. The benefit of adaptive kernels is the reduction of memory usage (at training and at the inference time) as well as training speedups (up to 2x). The kernels are evaluated on two datasets MNIST and CIFAR10