Convolutional Neural Networks combined with Runge-Kutta Methods

1 INTRODUCTION . Residual Networks ( ResNets ) which are feed-forward network models with skip connections have achieved great success on several vision benchmarks ( He et al. , 2016a ) . Recently , researchers have studied the relation between ResNets and dynamical systems ( Liao & Poggio , 2016 ; E , 2017 ; Haber et al. , 2017 ; Chang et al. , 2018a ; b ; Lu et al. , 2018 ) . Forward Euler method , a first-order RK method , has been employed to explain ResNets with full pre-activation ( He et al. , 2016b ) from the dynamical systems view ( Haber et al. , 2017 ; Chang et al. , 2018b ) . Nevertheless , there is no firm evidence to prove that the residual block is just forward Euler method but not any other RK method . We regard the residual mapping as an approximation to the increment in a time-step . The accuracy of the approximation is determined by the structure of the convolutional network . Wide residual network ( WRN ) ( Zagoruyko & Komodakis , 2016 ) has been proposed to improve the ability of the convolutional subnetwork . However , it is not very efficient only to widen the subnetwork . The new explanation of pre-activation ResNet and its variants which focus on improving residual mapping is one of our contributions . In addition , some improvements on network architecture based on ordinary differential equations ( ODEs ) are proposed ( Chang et al. , 2018a ; Lu et al. , 2018 ; Chen et al. , 2018 ) . Under the assumption that pre-activation ResNet is forward Euler method , Chang et al . ( 2018a ) ; Lu et al . ( 2018 ) use special linear multi-step methods ( LM methods ) with low order to construct the network . Chen et al . ( 2018 ) utilize a third-party package which offers numerical ODE methods to replace residual block . There is no efficient network architecture for systematic generalization to high order till now . Nevertheless , a higher-order method can achieve a lower truncation error . Since a lower truncation error likely leads to a high accuracy , it is necessary to study an efficient network architecture with a high order . If the process of image classification is deemed a sequence of time-dependent dynamical systems , there should be a series of ODEs to describe these systems . RK methods are widely-used procedures to solve ODEs in numerical analysis ( Butcher , 2008 ) . They are also the building blocks of highorder LM methods . Consequently , these methods can be used to build network models for visual processing . The neural network community has long been aware of the numerical methods for dynamical systems . Runge-Kutta Neural Network ( RKNN ) is proposed for identification of unknown dynamical systems in high accuracy ( Wang & Lin , 1998 ) , but it has not been used to model the visual system nor been extended to convolutional networks . Moreover , RKNNs adopt a specific RK methods by indicating every coefficient for the RK methods . Thus , it is hard to apply high order RK methods in RKNNs . In addition , the time-step size need to be prespecified . Hence , RKNN can not be used in tasks where the total time is unknown such as image classification . In contrast , we learn all the coefficients and time-step sizes implicitly by training in order to avoid these difficulties . As a result , one of the major contributions of the paper is a novel and effective neural network architecture inspired by the RK methods . In order to apply RK methods to the image classification problem , the following assumptions are made throughout the paper . Firstly , the image classification procedure is multi-period and there are transitions between adjacent periods . Secondly , each period is modeled by a time-dependent firstorder dynamical system . Based on these assumptions , a novel network model called the RKNet is proposed . In an RKNet , a period is composed of iterations of time-steps . A particular RK method is adopted throughout the time-steps in a period to approximate the increment in each step . The increment in each step is broken down to the increments in several stages according to the adopted RK method . Each stage is approximated by a convolutional subnetwork due to the versatility of neural networks on approximation . Another contribution of this paper is a theoretical interpretation of DenseNets and CliqueNets from the dynamical systems view . The dense connections in DenseNet resemble the relationship among increments in the stages in explicit RK methods ( ERK methods ) . Similarly , the clique blocks in CliqueNets resemble the relationship among increments in the stages in implicit RK methods ( IRK methods ) . Under some conditions , DenseNets and CliqueNets can be formulated as approximating dynamical systems using multi-stage RK methods . We also propose a method to convert a DenseNet to an explicit RKNet ( ERKNet ) and a method convert a CliqueNet to an implicit RKNet ( IRKNet ) . Furthermore , DenseNets and CliqueNets have only one time-step in each period , whereas RKNets are more general and can have multiple time-steps in each period . We evaluate the performance of RKNets on benchmark datasets including CIFAR-10 , CIFAR100 ( Krizhevsky , 2009 ) , SVHN ( Netzer et al. , 2011 ) and ILSVRC2012 classification dataset ( Russakovsky et al. , 2015 ) . Experimental results show that both ERKNets and IRKNets conform to the mathematical properties . Additionally , RKNets achieve higher accuracy than the state-of-the-art network models on CIFAR-10 and comparable accuracy on CIFAR-100 , SVHN and ImageNet . The rest of the paper is organized as follows . The related work is reviewed in Section 2 . The architecture of RKNets , the dynamical systems interpretation of DenseNets and CliqueNets , and the conversion from them to RKNets are described in Section 3 . The performance of RKNets is evaluated in Section 4 . The conclusion and future work is described in Section 5 . 2 RELATED WORK . ResNets have gained much attention over the past few years since they have obtained impressive performance on many challenging image tasks , such as ImageNet ( Russakovsky et al. , 2015 ) and COCO object detection ( Lin et al. , 2014 ) . ResNets are deep feed-forward networks with the shortcuts as identity mappings . ResNets with pre-activation can be regarded as an unfolded shallow RNN , which implements a discrete dynamical system ( Liao & Poggio , 2016 ) . It provides a novel point of view for explaining pre-activation ResNets from dynamical systems view . Recently , more work has emerged to connect dynamical systems with deep neural networks ( E , 2017 ) or ResNets in particular ( Haber et al. , 2017 ; Chang et al. , 2018a ; b ; Li et al. , 2018 ; Long et al. , 2018 ; Lu et al. , 2018 ; Wang et al. , 2018 ; Chen et al. , 2018 ) . E ( 2017 ) proposes to use continuous dynamical systems as a tool for machine learning . Chang et al . ( 2018a ) propose three reversible architectures based on ResNets and ODE systems . Chang et al . ( 2018b ) propose a novel method for accelerating ResNets training based on the interpretation of ResNets from dynamical systems view ( Haber et al. , 2017 ) . Li et al . ( 2018 ) present a training algorithm which can be used in the context of ResNets . Lu et al . ( 2018 ) propose a 2-step architecture based on ResNets . In addition , research combining dynamical system identification and RK methods with neural networks for scientific computing has emerged recently ( Raissi et al. , 2017a ; b ; Raissi , 2018 ) , introducing physics informed neural networks with automatic differentiation . Chen et al . ( 2018 ) utilize a third-party package which offers some numerical methods to compute the numerical solution in each time-step . DenseNets ( Huang et al. , 2017 ) are the state-of-the-art network models after ResNets . The dense connection is the main difference from the previous models . There are direct connections from a layer to all subsequent layers in a dense block in order to allow better information and gradient flow . There is no interpretation of DenseNets from dynamical systems view yet . CliqueNets ( Yang et al. , 2018 ) are the state-of-the-art network models based on DenseNets . They adopt the alternately updated clique blocks to incorporate both forward and backward connections between any two layers in the same block . However , there is no interpretation of CliqueNets from dynamical systems view yet . Given that the process of image classification is regarded as a sequence of time-dependent dynamical systems , there should be a set of ODEs that describes these systems . Consequently , mathematical tools could be employed to construct network models . RK methods are commonly used to solve ODEs in numerical analysis ( Butcher , 2008 ) . Higher order RK methods can achieve lower truncation error . Moreover , these methods are usually the building blocks of high-order LM methods . Therefore , RK methods are ideal tools to construct network models from dynamical systems view . RK methods have been adopted to construct neural networks , which are known as RKNN , for identification of unknown dynamical systems described by ODEs ( Wang & Lin , 1998 ) . In that paper , neural networks are classified into two categories : ( 1 ) a network that directly learns the state trajectory of a dynamical system is called a direct-mapping neural network ( DMNN ) ; ( 2 ) a network that learns the rate of change of system states is called a RKNN . Hence , AlexNet ( Krizhevsky et al. , 2012 ) , VGGNet ( Simonyan & Zisserman , 2015 ) , GoogLeNet ( Szegedy et al. , 2015 ) and ResNet ( He et al. , 2016a ) all belong to DMNNs . Specifically , the original ResNet ( He et al. , 2016a ) is a DMNN because of the ReLU layer after the addition operation . As a result , the ResNet building block learns the state trajectory directly , not the rate of change of the system states . On the contrary , a ResNet with pre-activation ( He et al. , 2016b ) is an RKNN . RKNNs are proposed to eliminate several drawbacks of DMNNs , such as the difficulty in obtaining high accuracy for the multi-step prediction of state trajectories . It has been shown theoretically and experimentally that the RKNN has higher prediction accuracy and better generalization capability than the conventional DMNN ( Wang & Lin , 1998 ) . Therefore , it is reasonable to believe that RK methods can be adopted to design effective network architectures for image classification problems . Additionally , the RK methods might improve the performance of image classification since the convolutional subnetworks are able to approximate the rate of change of the dynamical system states more precisely . 3 RKNETS . The introduction to RK methods are in Section 3.1 . We describe the overall structure of RKNet in Section 3.2 . The structure of subnetwork for increment in each time-step is elaborated on in Section 3.3 . 3.1 RUNGE-KUTTA METHODS . An initial value problem for a time-dependent first-order dynamical system can be described by the following ODE ( Butcher , 2008 ) : dy dt = f ( t , y ( t ) ) , y ( t0 ) = y0 . ( 1 ) where y is a vector representing the system state . The dimension of y should be equal to the dimension of the dynamical system . The ODE represents the rate of change of the system states . The rate of change is a function of time and the current system state . RK methods utilize the rate of change calculated from the ODE to approximate the increment in each time-step , and then obtain the predicted final state at the end of each step . RK methods are numerical methods originated from Euler method . There are two types of RK methods : explicit and implicit . Both of them are employed in the RKNet . The family of RK methods is given by the following equations ( Sli & Mayers , 2003 ) : yn+1 = yn + h s∑ i=1 bizi , tn+1 = tn + h , ( 2 ) where zi = f tn + cih , yn + h s∑ j=1 aijzj , 1 ≤ i ≤ s. ( 3 ) In equation 2 , yn+1 is an approximation of the solution to equation 1 at time tn+1 , i.e . y ( tn+1 ) ; y0 is the input initial value ; h ∑s i=1 bizi is the increment of system state y from tn to tn+1 ; ∑s i=1 bizi is the estimated slope which is the weighted average of the slopes zi computed in different stages . The positive integer s is the number of zi , i.e . the number of stages of the RK method . The equation 3 is the general formula of zi . h is the time-step size which can be adaptive for different time-steps but must be fixed across stages within a time-step . In numerical analysis , s , aij , bi and ci in equation 2 and equation 3 need to be prespecified for a particular RK method . These coefficients are displayed in a Butcher tableau . The ERK methods are those methods with aij = 0 when 1 ≤ i ≤ j ≤ s. All the RK methods other than ERK methods are IRK methods . The algebraic relationships of the coefficients have to meet the order conditions to reach the highest possible order . Different RK methods have different truncation errors which are denoted by the order : an order p indicates that the local truncation error is O ( hp+1 ) . If a s-stage ERK method has order p , then s ≥ p ; if p ≥ 5 , then s > p ( Butcher , 2008 ) . Furthermore , a s-stage IRK method can has order p = 2s when its coefficients are chosen under some conditions ( Butcher , 2008 ) . Therefore , more stages may achieve higher orders , i.e . lower truncation errors . The Euler method is a one-stage first-order RK method with b1 = 1 and c1 = 0 . In other words , high-order RK methods can be expected to achieve lower truncation errors than Euler method . Thus , the goal of our proposed RKNets is to improve the classification accuracy by taking advantage of high-order RK methods . It is necessary to specify h in order to control the error of approximation in common numerical analysis . The varying time-step size can be adaptive to the regions with different rates of change . The truncation error is lower when the h is smaller .

Summary: This paper is based on an interpretation of so-called Residual Deep Networks as a discrete realization of some dynamical system. So far, work in this area used simple, Euler-forward like discretization schemes, while the paper at hand proposes to apply higher order Runge-Kutta Methods. This approach allows also to give a different interpretation of DenseNets and CliqueNets.