A framework of deep neural networks via the solution operator of partial differential equations

1 INTRODUCTION . Deep neural networks ( DNN ) have achieved success in tasks such as image classification ( Simonyan & Zisserman , 2015 ) , speech recognition ( Dahl et al. , 2012 ) , video analysis ( Bo et al. , 2011 ) , and action recognition ( Wang et al. , 2016 ) . Residual learning ( He et al. , 2016a ; b ) has revolutionized DNN architecture design , making it practical to train ultra-deep DNN , and has ability to avoid gradient vanishing . The idea of residual learning has motivated the development of powerful DNN such as WideResNet ( Zagoruyko & Komodakis , 2016 ) , ResNeXt ( Xie et al. , 2017 ) , and DenseNet ( Huang et al. , 2017 ) . So how to understand Residual learning from a theoretical perspective ? In ( Weinan , 2017 ) , the author formulated the forward propagation of ResNet as a dynamic system and in ( Li & Shi , 2017 ; Haber et al. , 2018 ; Li et al. , 2017 ) bridge the connection between partial differential equations ( PDE ) and ResNet . In fact , ResNet is composed of a series of residual mapping . Each residual block is realized by adding a shortcut connection to connect the input and output of the original mapping ( F ) . Mathematically , the n-th residual mapping can be formulated as xn+1 = xn + F ( xn , wn ) ( 1 ) with x0 ∈ Rd be a data point . xn is the input to the n-th residual mapping and F ( xn , wn ) is a non-linear function parameterized bywn which can be learned by back propagating the training error . The prediction of the data point x0 can be expressed by ŷ = softmax ( wFCxL ) . By introducing a temporal partition : tn = nT/L , for n = 0 , 1 , .. , L with the time interval ∆t = T/L and let x ( tn ) = xn and w ( tn ) = wn . Without considering the dimensional consistency , ResNet can be formulated as : x ( tn+1 ) = x ( tn ) + F ( x ( tn ) , w ( tn ) ) ∆t ∆t , n = 0 , 1 , · · · , L , which is the forward Euler discretization of the following equations with time step ∆t , that is , dx ( t ) = v ( x ( t ) , t ) dt , x ( 0 ) = x , t ∈ [ 0 , T ] , ( 2 ) where v ( x ( tn ) , tn ) = F ( x ( tn ) , w ( tn ) ) /∆t and L is the number of resdiual mapping in a ResNet architecture . The prediction can be expressed by ŷ = softmax ( wFCx ( T ) ) . The equation 2 defines the characteristic curves of the following TE ∂u ∂t ( x , t ) = v ( x , t ) ∇u ( x , t ) , x ∈ Rd . ( 3 ) So the prediction ŷ of ResNet can be seen as the solution u ( x , T ) of following Transport equation ( TE ) : { ∂u ∂t ( x , t ) = v ( x , t ) ∇u ( x , t ) , x ∈ R d. u ( x , 0 ) = softmax ( wFCx ) . ( 4 ) In fact , not only ResNet , similar to the process above , many ResNet-like DNN can correspond to PDEs . For example , it has been shown ( Lu et al. , 2018 ) that PolyNet ( Zhang et al. , 2017 ) and FractalNet ( Larsson et al. , 2016 ) can be respectively considered numerical schemes to solve a PDEs system . Based on the link of PDEs and ResNet-like DNN , many theoretical results to understand ResNet-like DNN were proved . In ( Chen et al. , 2018 ) , a continuous form for ResNet called Neural ODE was proposed . After that , the property of Neural ODE has been theoretically analyzed ( Thorpe & van Gennip , 2018 ; Avelin & Nyström , 2021 ; Zhang et al. , 2019a ) . Based on the stability theory of differential equations , many architectures has been designed to build more robust deep networks ( Zhang et al. , 2019b ; Yan et al. , 2019 ; Reshniak & Webster , 2021 ) . Not only stability theory , inspired by the numerical scheme of differential equations , in ( Zhu et al. , 2018 ; Tao et al. , 2018 ) , various deep network architectures to improve testing accuracy have been proposed in success . Despite the success of the application of PDEs in residual learning , these neural network design space in the literature mentioned above is restricted due to the specific form of PDEs , which prevents the design of more effective neural network structures . Existed works can not meet these requirements . So , a natural question is that can we find the general form of the PDEs corresponding to the ResNetlike DNN which can unify existed model as well as guide new effective DNN architecture design ? To achieve this goal , in this paper , we follow the setting of paragraph 2 , the softmax model and ResNet are the initial value and the terminal value of PDEs equation 4 respectively , so we can view ResNet as an evolution from a softmax model . Without loss of generality , we indicate this evolutive relationship by an adjustment operator denoted by Tt , then we have Tt : softmax ( w FCx ) → ResNet ( x ) . If we denote the softmax model by f , then the ResNet-like DNN can be expressed as Tt ( f ) and Tt is the solution operator of PDEs equation 4 . To obtain the general form of the PDEs corresponding to these ResNet-like DNN , we list a series of formal properties likely to be satisfied by the operator Tt , and we deduce an explicit formulation from them . We see that under reasonable assumptions , the new model u ( x , t ) = ( Ttf ) ( x ) is the solution of convection diffusion equations . Then , motivated by the theory of PDEs and the framework of our general form of PDEs , we design a training method for ResNet-like DNN models for better robustness and less overfitting risk . Our training method is summarized as follows . First , we train a ResNet-like DNN model using natural training . Because calculating the real solution Tt ( f ) ( x ) is difficult , we next train a new ResNet-like DNN denoted by g ( x , t ) to approximate Tt ( f ) ( x ) . At last , we choose g ( x , T ) as our final classifier . We make the following contributions : • Under reasonable assumptions , we build a PDEs framework for ResNet-like DNN with the help of operator Tt and show some effective models that can be modeled by our framework . • We design a training method for DNN based on the theory of PDEs . This is an essential contribution of this paper due to certified adversarial robustness and theoretical analysis of generalization ability for our training method . • We verify our training method on a synthetic dataset and the CIFAR-10/CIFAR-100 datasets , then demonstrate that a ResNet trained by our method is more resistant to multistep ` 2 and ` ∞ adversarial attacks than natural trained ResNet . 2 MAIN RESULTS . We show under several reasonable assumptions that the adjustment operator Tt is the solution operator of convection diffusion equations . Based on the regularity theory of PDEs , we design a training method for DNN models and theoretically illustrate its benefits for improving the robustness of models and reducing the generalization gap . 2.1 GENERAL FORM OF PDES . We denote the base DNN model by fw ( x ) , which is uniformly continuous . We assume data points are distributed inD ⊂ Rd , and label function l ( x ) is defined inD but known only in training set SN ∈ D. Our method formulates the DNN as a continuous dynamic system , namely u ( x , t ) = Tt ( f ) , t ∈ [ 0 , T ] . To get the expression of the adjustment operator Tt , we assume it has some fundamental properties . [ Locality ] ( Tt ( f ) − Tt ( g ) ) ( x ) = o ( t ) as t→ 0+ , for all f , g such that Dαf ( x ) = Dαg ( x ) for all |α| ≥ 0 . [ Comparison Principle ] Tt ( f ) ≤ Tt ( g ) for all t ≥ 0 and f , g such that f ≤ g. [ Markov Property ] Tt+s = Tt+s , s ◦Ts , for all s , t ≥ 0 and t+ s ≤ T . Tt+s , sdenotes the flow from time s to time t+ s. [ Spatial Regularity ] There exist a positive constant C such that ‖Tt ( τhf ) − τh ( Ttf ) ‖L∞ ≤ Ch for all h in Rd , t ≥ 0 , where ( τhf ) ( x ) = f ( x+ h ) and ‖h‖2 = h. [ Temporal Regularity ] The following two properties hold : 1 ) For all t , s , t+ s ∈ [ 0 , T ] and all g ∈ Q , there exist a constant C ≥ 0 depending only on Q such that ‖Tt+s , s ( g ) − g‖L∞ ≤ Ct , where Q is the subset of C∞b defined by Q = { f ∈ C∞b , ∀n ≥ 0 , ‖Dαf‖L∞ ≤ C for all |α| = n } . 2 ) For all t , s , t+ s ∈ [ 0 , T ] , there exist a constant C ≥ 0 such that ‖Tt+s , t ( f ) −Ts,0 ( f ) ‖L∞ ≤ Cst uniformly in s ∈ ( 0 , T ] . [ Ensemble Additivity ] Tt ( f + g ) = Tt ( f ) + Tt ( g ) , and if C is a constant , then Tt ( C ) = C. To illustrate the reasonableness of these assumptions above , we give the following explanation . We first require that Tt can be expressed as a solution operator of PDEs and for small enough t , the value of Tt ( f ) at any point x is determined by the behavior of the base classifier f near x , which implies [ Locality ] . We then require the operator Tt is provided with an order-preserving property which means that no enhancement is made . Thus if the confidence level for an event of one base classifier g is higher than another base classifier f , this ordering is preserved , which implies [ Comparison Principle ] . Next we require Tt+s can be computed from Tt for any s ≤ t , and T0 is of course the identity . This is natural , since the future behaviour of the adjustment process at terminal time t+ s is likely to be deduced from the state at intermediate time t without any dependence upon the original model f , which implies [ Markov Property ] . We need function Tt ( f ) to be spatial stable . If we add a perturbation h to data point x , the output Tt ( f ) ( x+ h ) will not change a lot , which implies [ Spatial Regularity ] . We also need an additional property for giving some temporal stability to the operator Tt . Namely , in any time interval , the adjustment will not be rapid , which implies [ Temporal Regularity ] . Assume an ensemble classifier F consists of base classifiers f and g , and another ensemble classifier consists of base classifiers Tt ( f ) and Tt ( g ) . We require the prediction Tt ( F ) to equal Tt ( f ) + Tt ( g ) , which implies [ Ensemble Additivity ] . Theorem 1 . Under the above conditions , there exists Lipschitz continuous function v : Rd× [ 0 , T ] → Rd and Lipschitz continuous positive function σ : Rd × [ 0 , T ] → Rd×d such that for any bounded and uniformly continuous base model fw ( x ) , u ( x , t ) = Tt ( fw ) ( x ) is the unique viscosity solution of the following convection diffusion equation : { ∂u ( x , t ) ∂t = v ( x , t ) · ∇u ( x , t ) + 1 2 ∑ i , j σi , j ∂2u ∂xi∂xj ( x , t ) , u ( x , 0 ) = fw ( x ) , ( 5 ) where x ∈ Rd , t ∈ [ 0 , T ] . Here σi , j is the i , j-th element of matrix function σ ( x , t ) . We will provide the proof of Theorem 1 in Appendix A . In this subsection , we introduce a PDEs framework for DNN models . The framework is quite general , many existing models with residual connections can be modeled by our framework . As examples , we briefly list some of them in the next subsection .

Summary: This paper considers DNNs as transformations of the input data which can be seen as discrete approximations to the solutions of PDEs. It reformulates a point of view which has been trending for a few years now. More precisely, this paper represents the network as an operator, which is actually solution to a PDE, which acts on a "base classifier". Using facts from PDE theory, they then propose improvements for standard neural architectures.