ResNet After All: Neural ODEs and Their Numerical Solution

1 INTRODUCTION . The choice of neural network architecture is an important consideration in the deep learning community . Among a plethora of options , Residual Neural Networks ( ResNets ) ( He et al. , 2016 ) have emerged as an important subclass of models , as they mitigate the gradient issues ( Balduzzi et al. , 2017 ) arising with training deep neural networks by adding skip connections between the successive layers . Besides the architectural advancements inspired from the original scheme ( Zagoruyko & Komodakis , 2016 ; Xie et al. , 2017 ) , recently Neural Ordinary Differential Equation ( Neural ODE ) models ( Chen et al. , 2018 ; E , 2017 ; Lu et al. , 2018 ; Haber & Ruthotto , 2017 ) have been proposed as an analog of continuous-depth ResNets . While Neural ODEs do not necessarily improve upon the sheer predictive performance of ResNets , they offer the vast knowledge of ODE theory to be applied to deep learning research . For instance , the authors in Yan et al . ( 2020 ) discovered that for specific perturbations , Neural ODEs are more robust than convolutional neural networks . Moreover , inspired by the theoretical properties of the solution curves , they propose a regularizer which improved the robustness of Neural ODE models even further . However , if Neural ODEs are chosen for their theoretical advantages , it is essential that the effective model—the combination of ODE problem and its solution via a particular numerical method—is a close approximation of the true analytical , but practically inaccessible ODE solution . Code : https : //github.com/boschresearch/numerics_independent_neural_odes In this work , we study the empirical risk minimization ( ERM ) problem LD = 1 |D| ∑ ( x , y ) ∈D l ( f ( x ; w ) , y ) ( 1 ) where D = { ( xn , yn ) | xn ∈ RDx , yn ∈ RDy , n = 1 , . . . , N } is a set of training data , l : RDy × RDy → R is a ( non-negative ) loss function and f is a Neural ODE model with weights w , i.e. , f = fd ◦ ϕfvT ◦ fu ( 2 ) where fx , x ∈ { d , v , u } are neural networks and u and d denote the upstream and downstream layers respectively . ϕ is defined to be the ( analytical ) flow of the dynamical system dz dt = fv ( z ; wv ) , z ( t ) = ϕ fv t ( z ( 0 ) ) . ( 3 ) As the vector field fv of the dynamical system is itself defined by a neural network , evaluating ϕ fv T is intractable and we have to resort to a numerical scheme Ψt to compute ϕt . Ψ belongs either to a class of fixed step methods or is an adaptive step size solver as proposed in Chen et al . ( 2018 ) . For fixed step solvers with step size h one can directly compute the number of steps taken by the solver # steps = Th−1 . We set the final time T = 1 for all our experiments . The global numerical error etrain of the model is the difference between the true , ( unknown ) , analytical solution of the model and the numerical solution etrain = ||ϕT ( z ( 0 ) ) −ΨT ( z ( 0 ) ) || at time T . The global numerical error for a given problem can be controlled by adjusting either the step size or the local error tolerance . Since the numerical solvers play an essential role in the approximation of the solutions of an ODE , it is intuitive to ask : how does the choice of the numerical method affect the training of a Neural ODE model ? Specifically , does the discretization of the numerical solver impact the resulting flow of the ODE ? To test the effect of the numerical solver on a Neural ODE model , we first train a Neural ODE on a synthetic classification task consisting of three concentric spheres , where the outer and inner sphere correspond to the same class ( for more information see Section 2.4 ) . For this problem there are no true underlying dynamics and therefore , the model only has to find some dynamics which solve the problem . We train the Neural ODE model using a fixed step solver with a small step size and a solver with a large step size ( see Figure 1 ( a ) and ( b ) respectively ) . If the model is trained with a large step size , then the numerically computed trajectories for the individual Initial Value Problems ( IVPs ) cross in phase space ( see Figure 1 ( b ) ) . Specifically , we observe that trajectories of IVPs belonging to different classes cross . This crossing behavior contradicts the expected behavior of autonomous ODE solutions , as according to the Picard-Lindelöf theorem we expect unique solutions to the IVPs . We observe crossing trajectories because the discretization error of the solver is so large that the resulting numerical solutions no longer maintain the properties of ODE solutions . We observe that both , the model trained with the small step size and the model trained with the large step size , achieve very high accuracy . This leads us to the conclusion that the step size parameter is not like any other hyperparameter , as its chosen value often does not affect the performance of the model . Instead , the step size affects whether the trained model has a valid ODE interpretation . Crossing trajectories are not bad per se if the performance is all we are interested . If , however , we are interested in applying algorithms whose success is motivated from ODE theory to , for example , increase model robustness ( Yan et al. , 2020 ) , then the trajectories must not cross . We argue that if any discretization with similar or lesser discretization error yields the same prediction , the trained model corresponds to an ODE that is qualitatively well approximated by the applied discretization . Therefore , in our experiments we evaluate each Neural ODE model with smaller and larger step sizes than the training step size . We notice that the model trained with the small step size achieves the same level of performance when using a solver with smaller discretization error for testing ( Figure 1 ( c ) ) . For the model trained with the large step size , we observe a significant drop in performance if the model is evaluated using a solver with a smaller discretization error ( see Figure 1 ( d ) ) . The reason for the drop in model performance is that the decision boundary of the classifier has adapted to the global numerical error etrain in the computed solution . For this specific example , correct classification relies on crossing trajectories as a feature . Therefore , the solutions of solvers with a smaller discretization error are no longer assigned the right class by the classifier and the Neural ODE model is a ResNet model without ODE interpretation . If we are interested in extending ODE theory to Neural ODE models , we have to ensure that the trained Neural ODE model indeed maintains the properties of ODE solutions . In this work we show that the training process of a Neural ODE yields a discrete ResNet without valid ODE interpretation if the discretization is chosen too coarse . With our rigorous Neural ODE experiments on a synthetic dataset as well as CIFAR10 using both fixed step and adaptive step size methods , we show that if the precision of the solver used for training is high enough , the model does not depend on the solver used for testing as long as the test solver has a small enough discretization error . Therefore , such a model allows for a valid ODE interpretation . Based on this observation we propose an algorithm to find the coarsest discretization for which the model is independent of the solver . 2 INTERACTION OF NEURAL ODE AND ODE SOLVER CAN LEAD TO DISCRETE DYNAMICS . We want to study how the Neural ODE is affected by the specific solver configuration used for training . To this end , in our experiments we first train each model with a specific step size htrain ( or a specific tolerance toltrain in the case of adaptive step size methods ) . For the remainder of this section we will only consider fixed step solvers , but all points made equally hold for adaptive step methods , as shown by our experiments . Post-training , we evaluate the trained models using different step sizes htest and note how using smaller steps sizes htest < htrain affects the model performance . We expect that if the model yields good performance in the limiting behavior using smaller and smaller step sizes htest → 0 for testing , then model should correspond to a valid ODE . For a model trained with a small step size , we find that the numerical solutions do not change drastically if the testing step size htest is decreased ( see Figure 1 ( c ) ) . But if the step size htrain is beyond a critical value , the model accumulates a large global numerical error etrain . The decision layer may use these drastically altered solutions as a signal/feature in the downstream computations . In this case , the model is tied to a specific , discrete flow and the model remains no longer valid in the limit of using smaller and smaller step sizes htest → 0 . 2.1 THE TRAJECTORY CROSSING PROBLEM . In this sub-section , we examine the trajectory crossing effect which causes the ODE interpretation to break down . First , we look at the numerically computed trajectories in phase space of a Neural ODE model trained with a very large step size of htrain = 1/2 ( see Figure 1 ( b ) ) . A key observation is that the trajectories cross in phase space . This crossing happens because the step size htrain is much bigger than the length scale at which the vector field changes , thus missing “ the curvature ” of the true solution . Specifically , we observe that the model exploits this trajectory crossing behavior as a feature to separate observations from different classes . This is a clear indication that these trajectories do not approximate the true analytical solution of an ODE , as according to the Picard-Lindelöf theorem ( Hairer et al. , 1993 , § 1.8 ) , solutions of first order autonomous ODEs do not cross in phase space . Since the numerical solutions using smaller step sizes htest < htrain no longer maintain the crossing trajectory feature , the classifier can not separate the data with the learned vector field ( see Figure 2 ) . 2.2 LADY WINDERMERE ’ S FAN . In cases where trajectory crossings do not occur , other , more subtle effects can also lead to a drop in performance in the limit of using smaller and smaller test step sizes htest → 0 . The compound effect of local numerical error leads to a biased global error which is sometimes exploited as a feature in downstream blocks . This effect how the local error gets accumulated into global error was coined as Lady Windermere ’ s Fan in Hairer et al . ( 1993 , § 1.7 ) . To understand these effects we introduce an example based on the XOR problem D = { ( ( 0 , 0 ) 7→ 0 ) , ( ( 1 , 1 ) 7→ 0 ) , ( ( 0 , 1 ) 7→ 1 ) , ( ( 1 , 0 ) 7→ 1 ) } . This dataset can not be classified correctly in 2D with a linear decision boundary ( Goodfellow et al. , 2016 , § 1.2 ) . Therefore , we consider the ODE z′ ( t ) = ( α 1 −γ||z||δ β ) z . ( 4 ) The qualitative behavior of the analytical flow are increasing ellipsoids with ever increasing rotational speed . We chose this problem as an example based on the knowledge that the precision of a solver influences how the rotational speed of the ellipsoids is resolved . Therefore , this problem is useful in illustrating how the numerical accuracy of the solver can affect the final solution . Fig . 3 depicts the numerical solution of this flow with one set of fixed parameters and different step sizes h = 10−2.5 , 10−3.5 . For both step sizes we do not observe crossing trajectories , but the final solutions differ greatly . For h = 10−2.5 the numerical flow produces a transformation in which the data points can be separated linearly . But for the smaller step size of h = 10−3.5 , the numerical solution is no longer linearly separable . The problem here is that the numerical solution using the larger step size is not accurate enough to resolve the rotational velocity . For each step the local error gets accumulated into the global error . In Figure 3 ( a ) , the accumulation of error in the numerical solution results in a valid feature for a linear decision ( classification ) layer . The reason for this is that the global numerical errors etrain are biased . We define as the fingerprint of the method the structure in the global numerical error . The decision layer then adapts to this method specific fingerprint . How this fingerprint affects the performance of the model when using smaller step size htest is dependent on two aspects . First , does the data remain separable when using smaller step sizes htest < htrain ? If not , we will observe a significant drop in performance . Second , how sensitive is the decision layer to changes in the solutions and how much do the numerical solutions change when decreasing the test step size htest → 0 ? Essentially , the input sensitivity of the downstream layer should be less than the output sensitivity of htest < htrain . For the decision layer , there should exist a sensitivity threshold d such that fd ( z ( T ) + δ ) = fd ( z ( T ) ) ∀||δ|| < d. Thus , if two solvers compute the same solution up to δ , the classifier identifies these solutions as the same class and the result of the model is not affected by the interchanging these solvers .

Summary: The paper demonstrates how neural ODE models generating features for downstream tasks (or simply modelling trajectories) may rely on the discreteness of integration methods to generate features and thus fail in the exact ODE limit of integration step-size going to zero. The paper highlights particular failure modes, such as the discreteness of integration methods allowing for qualitative differences like overlapping trajectories (impossible for the exact solution of an autonomous ODE) compared to exact solutions, or quantitative differences like the accumulated error of a numerically integrated ODE resulting in useful features for downstream tasks. The paper empirically demonstrates the phenomenon that low training losses can be achieved for a range of integration methods and integration step-sizes, but that, of these models, the ones robust to changes in integration method and decreases in integration step-sizes at test time are those trained below a certain (empirically determined) integration step-size threshold. This is attributable to models trained with lower integration step-sizes maintaining features that are qualitatively the same as or quantitatively close to those features produced by the same model with smaller integration step-sizes. The paper proposes an algorithm for adapting integration step-size during training so that the resulting neural ODE model is robust to changes in integration method and integration step-size at test time. The algorithm is empirically demonstrated to achieve the same performance as grid search (for similar numbers of function evaluations).