Imbedding Deep Neural Networks

1 UNPACKING CONTINUOUS-DEPTH NEURAL NETWORKS . The long-standing enigma surrounding machine learning still remains paramount today : What is it that machines are learning and how may we extract meaningful knowledge from trained algorithms ? Deep Neural Networks ( DNNs ) , whilst undeniably successful , are notorious black-box secret keepers . To solve a supervised learning process , mapping vector inputs x to their targets y , parameters are stored and updated in ever deepening layers with no facility to access the physical significance of the internal function approximating the global mapping x 7→ y . We propose a new class of DNNs obtained by imbedding multiple networks of varying depth whilst keeping the inputs , x , invariant ; we call these ‘ Invariant Imbedding Networks ’ ( InImNets ) . To illustrate the concept , Figure 1 depicts a system of projectiles fired from a range of positions p1 < p2 < · · · < pn with the same initial velocity conditions x . The red curve ( initiated at p1 ) is fit to a sample ( circles ) along a single trajectory , representing a traditional regression problem . InImNet architectures are trained on the output values y = y ( pi , x ) at pn ( the diamonds ) as the depth pi of the system varies . This analogy applies to DNN classifiers where increasing the depth from pi to pi−1 outputs a classification decision for each of the i-steps . As a machine learning tool , the use of deep hidden layers , whilst successful , was first considered ad hoc in implementations , such as in multilayer ∗Equal Contribution . perceptrons . But following the advent of residual neural networks ( He et al. , 2015 ) which use ‘ Euler-step ’ internal updates between layers , DNN evolution is seen to emulate a continuous dynamical system ( Lu et al. , 2018 ; Ruthotto & Haber , 2020 ) . Thus was formed the notion of a ‘ Neural ODE ’ ( Chen et al. , 2018 ) in which the hidden network state vector z ( t ) ∈ RN , instead of being defined at fixed layers t ∈ N , is allowed to vary continuously over an interval [ p , q ] ⊂ R for a system of dimension N ≥ 1 . Its evolution is governed by an Ordinary Differential Equation ( ODE ) ż ( t ) = f ( t , z ( t ) , θ ( t ) ) ; z ( p ) = x ( 1 ) where the training function f is controlled by a parameter vector θ ( t ) ∈ RM for t ∈ [ p , q ] of length M ≥ 1 . Network outputs are retrieved at z ( q ) = y after fixing the endpoint t = q . As such , the enigmatic ‘ depth ’ of a Neural ODE is controlled by varying t = p , at which point we insert the initial condition z ( p ) = x . This dynamical description has given the theory of DNNs a new home : the mathematical framework of optimal control ( Massaroli et al. , 2020 ; Bensoussan et al. , 2020 ) . In this framework , whether discrete or continuous , solution networks are sought after that : ( A ) satisfying their update law ( 1 ) over a fixed ‘ depth ’ interval [ p , q ] ; ( B ) minimise a loss function subject to terminal success and internal regulation ( see §2.1 ) . As initiated by Euler and Lagrange ( Euler , 1766 ) , this mathematical framework determines networks given by ( A ) satisfying condition ( B ) using , amongst other techniques , the adjoint method : here a new state , which we call λ ( t ) or the ‘ Lagrange multiplier ’ , is introduced containing the system losses with respect to both t and the parameters θ ( t ) ; see §2.3 . The connection between the traditional method of ‘ backpropagation-by-chain rule ’ and the rigorous ‘ adjoint method ’ is quite brilliantly explicated in proof by Chen et al . ( 2018 ) , in that they directly deduce the adjoint method using infinitesimal backpropagation–far from the train of thought of Lagrange ’ s multiplier method ( Liberzon , 2012 , Ch . 2 ) ; see also §D and §E.2 in the appendix for extended discussion and derivation . Even within the above theory , the initial condition , z ( p ) = x , and location , t = p , remain implicit constraints ; a clear understanding of network depth remains illusive . Our new class of InImNet architectures may be obtained by imbedding networks of varying depth p whilst keeping the inputs , x , invariant . Explicating these two variables throughout the network , writing z ( t ) = z ( t ; p , x ) , has exciting conceptual consequences : 1 . Forward pass to construct multiple networks : InImNet state vectors z ( t ; p , x ) are computed with respect to the depth variable p rather than t ∈ [ p , q ] , which is considered fixed ( in practice at t = q ) . We build from the bottom up : initiate at p = q with the trivial network z ( q ; q , x ) = x and unwind the p-varying dynamics , as described in Theorem 1 , by integrating ∇pz ( q ; p , x ) = −∇xz ( q ; p , x ) · f ( p , x , θ ( p ) ) ( 2 ) from p = q to a greater depth p. Note that at depth p an InImNet returns an external output z ( q ; p , x ) ∼ y , subject to training . This contrasts with convention , where one would obtain z ( q ; p , x ) by integrating from t = p to t = q , where t < q states are considered internal . A general algorithm to implement the forward pass is described in Algorithm 1 . The gradient operator ∇ denotes the usual vector , or Jacobian , of partial derivatives . 2 . Backpropagate independently from the forward pass : We generalise the adjoint method of Chen et al . ( 2018 ) , who was able to do away with the backpropagation-by-chain rule method in favour of a continuous approach with at most bounded memory demand . With our bottom-up formulation , we are able to go one step further and do away with the initial forward pass altogether by initiating our ‘ imbedded ’ adjoint Λ ( p , x ) , generalising λ ( t ) , with loss gradients for the trivial network z ( q ; q , x ) = x and computing to depth p via ∇pΛ ( p , x ) = − [ ∇xΛ ( p , x ) · f ( p , x , θ ( p ) ) +∇xf ( p , x , θ ( p ) ) T ·Λ ( p , x ) ] . ( 3 ) See Theorem 2 for a precise , more general explication . Backward passes may be made independently of forward passing altogether ; see Theorem 2 and Algorithm 1 . 3 . Pre-imposed optimality : Working in the framework of optimal control theory , we consider both running and terminal losses–a general ‘ Bolza problem ’ –see §2.1 . We give a necessary first-order criterion for optimal control ( Theorem 3 ) . In this way , we account for t-varying parameter controls θ ( t ) = θ ( t ; p , x ) , omitted from the original Neural ODEs , permitting future compatibility with the recent particle-shooting models of Vialard et al . ( 2020 ) . Our contribution . We prove that it is possible to reduce the non-linear , vector-valued optimal control problem to a system of forward-facing initial value problems . We introduce a new theoretical leap for the understanding of depth in neural networks , in particular with respect to viewing DNNs as dynamical systems ( Chen et al. , 2018 ; Massaroli et al. , 2020 ; Vialard et al. , 2020 ) . We demonstrate that this approach may be used in creating discrete and continuous numerical DNN schemes . We verify such algorithms by successfully applying our method to benchmark experiments , which perform well on a range of complex tasks ( high-dimensional rotating MNIST and bouncing balls ) in comparison to state-of-the-art techniques . Architecture . In §3 we give novel ‘ InImNet ’ architectures implementing the above results . These may be applied to regression and classification problems using discrete or continuous algorithms . Experimental performance . In §4 we present some promising results for their practical functionality . These are designed to support the theoretical development of InImNets in §2 & 3 and support the proof of concept in application . We derive various successful discrete implementations , based on computing the state evolutions ( 2 ) and ( 3 ) with an Euler-step method . We also identify operational aspects in InImNets which we expect to encourage ongoing research and development . Crucially , our architectures are compatible with the various technical advancements since made to Neural ODEs , including those by Dupont et al . ( 2019 ) ; Davis et al . ( 2020 ) ; Yıldız et al . ( 2019 ) ; see also a study of effective ODE solvers for the continuous models ( Hopkins & Furber , 2015 ) . Broader impact for optimal control theory . Further afield , our result applies more generally as a new tool in the theory of optimal control . The mathematical technique that we apply to ( 1 ) , deriving ( 2 ) and ( 3 ) , is known as the Invariant Imbedding Method . The key output of the method is the reformulation of a two-point boundary value problem as a system of initial value problems , given as functions of initial value x and input location p alone ( Meyer , 1973 ) . This stands in the literature as an alternative to applying the calculus of variations ( Liberzon , 2012 ; Vialard et al. , 2020 ) . For linear systems , the technique was first developed by Ambarzumian ( 1943 ) to study deep stellar atmospheres . It has since found widespread applications in engineering ( Bellman & Wing , 1975 ) , optical oceanography ( Mobley , 1994 ) , PDEs ( Maynard & Scott , 1971 ) , ODEs ( Agarwal & Saraf , 1979 ) and control theory ( Bellman et al. , 1966 ; Kalaba & Sridhar , 1969 ) , to name a few . The non-linear case is only touched on in the literature for scalar systems of zero terminal loss ( Bellman et al. , 1966 ; Kalaba & Sridhar , 1969 ) –including some numerical computations to support its efficiency ( Spingarn , 1972 ) . In this work we derive a complete invariant imbedding solution , fit for applications such as those mentioned above , for a non-linear , vector-valued Bolza problem . Overview of the paper . In §2 we give the main theorems used for InImNet architectures and more widely in the field of optimal control . Detailed derivations and proofs may be found in the appendix , §E . In §3 we put forward various architectures to illustrate how InImNets may be utilised in learning paradigms . In §4 we describe our supporting experimental work for such architectures .

Summary: This paper proposes a new type of continuous depth neural network architecture, named Invariant Imbedding Networks (InImNets). InImNets can imbed a given $\bf x$ input to different time steps (time positions) and efficiently calculate the solutions of the dynamic system (starting from different time positions) simultaneously. The authors also give a practical implementation for training the InImNets architecture. The proposed architecture is then applied to several benchmark problems for continuous neural networks for verifying its effectiveness.