ResNet After All: Neural ODEs and Their Numerical Solution

1 INTRODUCTION . The choice of neural network architecture is an important consideration in the deep learning community . Among a plethora of options , Residual Neural Networks ( ResNets ) ( He et al. , 2016 ) have emerged as an important subclass of models , as they mitigate the gradient issues ( Balduzzi et al. , 2017 ) arising with training deep neural networks by adding skip connections between the successive layers . Besides the architectural advancements inspired from the original scheme ( Zagoruyko & Komodakis , 2016 ; Xie et al. , 2017 ) , recently Neural Ordinary Differential Equation ( Neural ODE ) models ( Chen et al. , 2018 ; E , 2017 ; Lu et al. , 2018 ; Haber & Ruthotto , 2017 ) have been proposed as an analog of continuous-depth ResNets . While Neural ODEs do not necessarily improve upon the sheer predictive performance of ResNets , they offer the vast knowledge of ODE theory to be applied to deep learning research . For instance , the authors in Yan et al . ( 2020 ) discovered that for specific perturbations , Neural ODEs are more robust than convolutional neural networks . Moreover , inspired by the theoretical properties of the solution curves , they propose a regularizer which improved the robustness of Neural ODE models even further . However , if Neural ODEs are chosen for their theoretical advantages , it is essential that the effective model—the combination of ODE problem and its solution via a particular numerical method—is a close approximation of the true analytical , but practically inaccessible ODE solution . Code : https : //github.com/boschresearch/numerics_independent_neural_odes In this work , we study the empirical risk minimization ( ERM ) problem LD = 1 |D| ∑ ( x , y ) ∈D l ( f ( x ; w ) , y ) ( 1 ) where D = { ( xn , yn ) | xn ∈ RDx , yn ∈ RDy , n = 1 , . . . , N } is a set of training data , l : RDy × RDy → R is a ( non-negative ) loss function and f is a Neural ODE model with weights w , i.e. , f = fd ◦ ϕfvT ◦ fu ( 2 ) where fx , x ∈ { d , v , u } are neural networks and u and d denote the upstream and downstream layers respectively . ϕ is defined to be the ( analytical ) flow of the dynamical system dz dt = fv ( z ; wv ) , z ( t ) = ϕ fv t ( z ( 0 ) ) . ( 3 ) As the vector field fv of the dynamical system is itself defined by a neural network , evaluating ϕ fv T is intractable and we have to resort to a numerical scheme Ψt to compute ϕt . Ψ belongs either to a class of fixed step methods or is an adaptive step size solver as proposed in Chen et al . ( 2018 ) . For fixed step solvers with step size h one can directly compute the number of steps taken by the solver # steps = Th−1 . We set the final time T = 1 for all our experiments . The global numerical error etrain of the model is the difference between the true , ( unknown ) , analytical solution of the model and the numerical solution etrain = ||ϕT ( z ( 0 ) ) −ΨT ( z ( 0 ) ) || at time T . The global numerical error for a given problem can be controlled by adjusting either the step size or the local error tolerance . Since the numerical solvers play an essential role in the approximation of the solutions of an ODE , it is intuitive to ask : how does the choice of the numerical method affect the training of a Neural ODE model ? Specifically , does the discretization of the numerical solver impact the resulting flow of the ODE ? To test the effect of the numerical solver on a Neural ODE model , we first train a Neural ODE on a synthetic classification task consisting of three concentric spheres , where the outer and inner sphere correspond to the same class ( for more information see Section 2.4 ) . For this problem there are no true underlying dynamics and therefore , the model only has to find some dynamics which solve the problem . We train the Neural ODE model using a fixed step solver with a small step size and a solver with a large step size ( see Figure 1 ( a ) and ( b ) respectively ) . If the model is trained with a large step size , then the numerically computed trajectories for the individual Initial Value Problems ( IVPs ) cross in phase space ( see Figure 1 ( b ) ) . Specifically , we observe that trajectories of IVPs belonging to different classes cross . This crossing behavior contradicts the expected behavior of autonomous ODE solutions , as according to the Picard-Lindelöf theorem we expect unique solutions to the IVPs . We observe crossing trajectories because the discretization error of the solver is so large that the resulting numerical solutions no longer maintain the properties of ODE solutions . We observe that both , the model trained with the small step size and the model trained with the large step size , achieve very high accuracy . This leads us to the conclusion that the step size parameter is not like any other hyperparameter , as its chosen value often does not affect the performance of the model . Instead , the step size affects whether the trained model has a valid ODE interpretation . Crossing trajectories are not bad per se if the performance is all we are interested . If , however , we are interested in applying algorithms whose success is motivated from ODE theory to , for example , increase model robustness ( Yan et al. , 2020 ) , then the trajectories must not cross . We argue that if any discretization with similar or lesser discretization error yields the same prediction , the trained model corresponds to an ODE that is qualitatively well approximated by the applied discretization . Therefore , in our experiments we evaluate each Neural ODE model with smaller and larger step sizes than the training step size . We notice that the model trained with the small step size achieves the same level of performance when using a solver with smaller discretization error for testing ( Figure 1 ( c ) ) . For the model trained with the large step size , we observe a significant drop in performance if the model is evaluated using a solver with a smaller discretization error ( see Figure 1 ( d ) ) . The reason for the drop in model performance is that the decision boundary of the classifier has adapted to the global numerical error etrain in the computed solution . For this specific example , correct classification relies on crossing trajectories as a feature . Therefore , the solutions of solvers with a smaller discretization error are no longer assigned the right class by the classifier and the Neural ODE model is a ResNet model without ODE interpretation . If we are interested in extending ODE theory to Neural ODE models , we have to ensure that the trained Neural ODE model indeed maintains the properties of ODE solutions . In this work we show that the training process of a Neural ODE yields a discrete ResNet without valid ODE interpretation if the discretization is chosen too coarse . With our rigorous Neural ODE experiments on a synthetic dataset as well as CIFAR10 using both fixed step and adaptive step size methods , we show that if the precision of the solver used for training is high enough , the model does not depend on the solver used for testing as long as the test solver has a small enough discretization error . Therefore , such a model allows for a valid ODE interpretation . Based on this observation we propose an algorithm to find the coarsest discretization for which the model is independent of the solver . 2 INTERACTION OF NEURAL ODE AND ODE SOLVER CAN LEAD TO DISCRETE DYNAMICS . We want to study how the Neural ODE is affected by the specific solver configuration used for training . To this end , in our experiments we first train each model with a specific step size htrain ( or a specific tolerance toltrain in the case of adaptive step size methods ) . For the remainder of this section we will only consider fixed step solvers , but all points made equally hold for adaptive step methods , as shown by our experiments . Post-training , we evaluate the trained models using different step sizes htest and note how using smaller steps sizes htest < htrain affects the model performance . We expect that if the model yields good performance in the limiting behavior using smaller and smaller step sizes htest → 0 for testing , then model should correspond to a valid ODE . For a model trained with a small step size , we find that the numerical solutions do not change drastically if the testing step size htest is decreased ( see Figure 1 ( c ) ) . But if the step size htrain is beyond a critical value , the model accumulates a large global numerical error etrain . The decision layer may use these drastically altered solutions as a signal/feature in the downstream computations . In this case , the model is tied to a specific , discrete flow and the model remains no longer valid in the limit of using smaller and smaller step sizes htest → 0 . 2.1 THE TRAJECTORY CROSSING PROBLEM . In this sub-section , we examine the trajectory crossing effect which causes the ODE interpretation to break down . First , we look at the numerically computed trajectories in phase space of a Neural ODE model trained with a very large step size of htrain = 1/2 ( see Figure 1 ( b ) ) . A key observation is that the trajectories cross in phase space . This crossing happens because the step size htrain is much bigger than the length scale at which the vector field changes , thus missing “ the curvature ” of the true solution . Specifically , we observe that the model exploits this trajectory crossing behavior as a feature to separate observations from different classes . This is a clear indication that these trajectories do not approximate the true analytical solution of an ODE , as according to the Picard-Lindelöf theorem ( Hairer et al. , 1993 , § 1.8 ) , solutions of first order autonomous ODEs do not cross in phase space . Since the numerical solutions using smaller step sizes htest < htrain no longer maintain the crossing trajectory feature , the classifier can not separate the data with the learned vector field ( see Figure 2 ) . 2.2 LADY WINDERMERE ’ S FAN . In cases where trajectory crossings do not occur , other , more subtle effects can also lead to a drop in performance in the limit of using smaller and smaller test step sizes htest → 0 . The compound effect of local numerical error leads to a biased global error which is sometimes exploited as a feature in downstream blocks . This effect how the local error gets accumulated into global error was coined as Lady Windermere ’ s Fan in Hairer et al . ( 1993 , § 1.7 ) . To understand these effects we introduce an example based on the XOR problem D = { ( ( 0 , 0 ) 7→ 0 ) , ( ( 1 , 1 ) 7→ 0 ) , ( ( 0 , 1 ) 7→ 1 ) , ( ( 1 , 0 ) 7→ 1 ) } . This dataset can not be classified correctly in 2D with a linear decision boundary ( Goodfellow et al. , 2016 , § 1.2 ) . Therefore , we consider the ODE z′ ( t ) = ( α 1 −γ||z||δ β ) z . ( 4 ) The qualitative behavior of the analytical flow are increasing ellipsoids with ever increasing rotational speed . We chose this problem as an example based on the knowledge that the precision of a solver influences how the rotational speed of the ellipsoids is resolved . Therefore , this problem is useful in illustrating how the numerical accuracy of the solver can affect the final solution . Fig . 3 depicts the numerical solution of this flow with one set of fixed parameters and different step sizes h = 10−2.5 , 10−3.5 . For both step sizes we do not observe crossing trajectories , but the final solutions differ greatly . For h = 10−2.5 the numerical flow produces a transformation in which the data points can be separated linearly . But for the smaller step size of h = 10−3.5 , the numerical solution is no longer linearly separable . The problem here is that the numerical solution using the larger step size is not accurate enough to resolve the rotational velocity . For each step the local error gets accumulated into the global error . In Figure 3 ( a ) , the accumulation of error in the numerical solution results in a valid feature for a linear decision ( classification ) layer . The reason for this is that the global numerical errors etrain are biased . We define as the fingerprint of the method the structure in the global numerical error . The decision layer then adapts to this method specific fingerprint . How this fingerprint affects the performance of the model when using smaller step size htest is dependent on two aspects . First , does the data remain separable when using smaller step sizes htest < htrain ? If not , we will observe a significant drop in performance . Second , how sensitive is the decision layer to changes in the solutions and how much do the numerical solutions change when decreasing the test step size htest → 0 ? Essentially , the input sensitivity of the downstream layer should be less than the output sensitivity of htest < htrain . For the decision layer , there should exist a sensitivity threshold d such that fd ( z ( T ) + δ ) = fd ( z ( T ) ) ∀||δ|| < d. Thus , if two solvers compute the same solution up to δ , the classifier identifies these solutions as the same class and the result of the model is not affected by the interchanging these solvers .

Summary: This paper empirically studies whether Neural ODEs have a valid ODE interpretation. The authors show that a Neural ODE model does not necessarily represent a continuous dynamical system if the discretization of the numerical method is too coarse. Indeed, this is a widely overlooked issue that has been largely ignored in the Neural ODE community. To address this issue, the authors propose a novel adaptive step size scheme.