Imbedding Deep Neural Networks
Authors: Andrew Corbett
Dmitry Kangin
Venue: ICLR 2022 Spotlight (2022)
Keywords: Neural ODEs
Optimal Control
Deep Neural Networks
Invariant Imbedding

Abstract: Continuous-depth neural networks, such as Neural ODEs, have refashioned the understanding of residual neural networks in terms of non-linear vector-valued optimal control problems. The common solution is to use the adjoint sensitivity method to replicate a forward-backward pass optimisation problem. We propose a new approach which explicates the network's `depth' as a fundamental variable, thus reducing the problem to a system of forward-facing initial value problems. This new method is based on the principal of `Invariant Imbedding' for which we prove a general solution, applicable to all non-linear, vector-valued optimal control problems with both running and terminal loss.
Our new architectures provide a tangible tool for inspecting the theoretical--and to a great extent unexplained--properties of network depth. They also constitute a resource of discrete implementations of Neural ODEs comparable to classes of imbedded residual neural networks. Through a series of experiments, we show the competitive performance of the proposed architectures for supervised learning and time series prediction. 

IMBEDDING DEEP NEURAL NETWORKS
Andrew Corbett*
Dmitry Kangin"
University of Exeter a. corbett wew.eer.eeee.....e c...betttteetttt..cc...
Eteembly Ltd.
dkang i gmailmmlll com
ABSTRACT
Continuous-depth neural networks, such Neural ODEs, have refashioned the understanding of residual neural networks in terms of non-linear vector-valued optimal control problems. The common solution is to use the adjoint sensitivity method to replicate a hewaardbbckwwar pass optimisation problem. We propose new approach which explicates the network's 'depth' fundamental variable, thus reducing the problem to a system of forward-facing initial value problems. This new method is based the principle of 'Invariant Imbedding" for which we prove a general solution, applicable to all non-linear, vector-valued optimal con- trol problems with both running and terminal loss. Our new architectures provide a tangible tool for inspecting the theoretical-ane properties of network depth. They also constitute a resource of discrete implemen- tations of Neural ODEs comparable to classes of imbedded residual neural net- works. Through series of experiments, we show the competitive performance of the proposed architectures for supervised learning and time series prediction. Ac- companying code made available at filgithu.... com/ www.wwdww0000. 1a1.000///iimteeee
to great extent unexplained-
UNPACKING CONTINUOUS-DEPTH NEURAL NETWORKS
The long-standing enigma surrounding machine learning still remains paramount today: What is that machines learning and how may extract meaningful knowledge from trained algo- rithms? Deep Neural Networks (DNNs), whilst undeniably successful, are notorious black-box secret keepers. To solve supervised learning process, mapping vector inputs to their tar- gets y, parameters are stored and updated in ever deepening layers with no facility to access the physical significance of the internal function approximating the global mapping x + y.
We propose a new class of DNNs obtained by imbedding multiple networks of varying depth whilst keeping the inputs, x, invariant; we call these 'Invariant Imbedding Networks' (InImNets). To illustrate the concept, Figure [ depicts a sys- tem of projectiles fired from range of positions P1 <2 < < Pn. with the same initial velocity conditions The red curve (initiated at pi) is fit to a sample (circles) along a single trajectory, rep- resenting traditional regression problem. InIm- Net architectures are trained on the output values y y(p, x) Pn. (the diamonds) as the depth of the system varies. This analogy applies to DNN classifiers where increasing the depth from pa to Pi-1 outputs classification decision for each of i-steps.
htt.
P1
P
Pa
range of Figure 1: Plotted are heights h(t:p, lengths t of projectile curves initiated with identical initial velocities points p; along the l-axis. The red curve depicts a regression fit to t-varying sam- ple set; contrast with the blue InImNet train- ing paradigm which learns the endpoints (dia- monds) from varying the input position p;-
As a machine learning tool, the use of deep hid- den layers, whilst successful, was first considered ad hoc in implementations, such in multilayer
Equul Contribution.

---

perceptrons. But following the advent of residual neural networks (He al. 2015) which use 'Euler-step' internal updates between layers, DNN evolution is seen to emulate a continuous dy- namical system (Lu et al. 2018 Ruthotto & Haber 2020). Thus was formed the notion of a 'Neural ODE' (Chen et al. 2018) in which the hidden network state vector x(t) E R instead of being de- fined at fixed layers E N, is allowed to vary continuously over interval [p. C R for system of dimension N 2 1. Its evolution is governed by an Ordinary Differential Equation (ODE)
((t) ftt, ztt), O(t)); z(p) =x
(I)
where the training function f is controlled by parameter vector ((t) E RM for t E [p. g] of length M 2 1. Network outputs are retrieved at z(4) = y after fixing the endpoint = q. As such, the enigmatic 'depth' of a Neural ODE is controlled by varying / = p. at which point we insert the initial condition x(p) = x. This dynamical description has given the theory of DNNs a new home: the mathematical framework of optimal control (Massaroli et al.) [020 Bensoussan et al., 2020). In this framework, whether discrete or continuous, solution networks are sought after that: (A) satisfying their update law (T) over fixed 'depth' interval [p. g]: (B) minimise a loss function subject to terminal success and internal regulation (see $2.1). As initiated by Euler and Lagrange (Euler 1766). this mathematical framework determines networks given by (A) satisfying condition (B) using, amongst other techniques, the adjoint method: here a new state, which we call X(t) or the 'Lagrange multiplier', is introduced containing the system losses with respect to both and the parameters f(t); see $2.3
The connection between the traditional method of "backpropagation-byy 'adjoint method' is quite brilliantly explicated in proof by [Cenn et al. (2018). in that they directly deduce the adjoint method using infinitesimal backpropagation--in Lagrange's s multiplier method (Liberzon [012] Ch. 2); see also $ and $ $E.2 in the appendix for extended discussion and derivation.
rule' and the rigorous
from the train of thought of
Even within the above theory, the initial condition, z(p) and location, ( <= p, remain implicit constraints; a clear understanding of network depth remains illusive.
Our new class of InImNet architectures may be obtained by imbedding networks of varying depth p whilst keeping the inputs, x, invariant. Explicating these two variables throughout the network, writing z(t) = z(t; p, x). has exciting conceptual consequences:
1. Forward pass to construct multiple networks: InImNet state vectors z(t; p, x) are com- puted with respect to the depth variable p rather than & E [p, 9]. which is considered fixed (in practice at f = ②). We build from the bottom up: initiate at q with the trivial network z(4 q, x) and unwind the integrating
p- p-varying dynamics, as described in Theorem 1 by
V,z(g;,,)) Vxz(g;p,,) f(p, x, 0(p))
from p q to a greater depth Note that at depth p an InImNet returns an external output z(q; p, x) y, subject to training. This contrasts with convention, where would obtain ((F:,, x) by integrating from 1 = p to = q. where ( < states are considered internal. A general algorithm to implement the forward pass is described in Algorithm The gradient operator V denotes the usual vector, or Jacobian, of partial derivatives.
2. Backpropagate independently from the forward pass: We generalise the adjoint method of Chen et al. (2018), who was able to do away with the backpropagation-by--hee method in favour of a continuous approach with at most bounded memory demand. With our bottom-up formulation, we are able to go one step further and do away with the initial forward pass altogether by initiating our 'imbedded" adjoint A(p, x), generalising A(t), with loss gradients for the trivial network 2(q: g, x) and computing depth p via
rule
V v,A(p, x) - AV x) f(p. x, 0(p)) + V x, 0(p)) A(p, x)].
(3)
See Theorem 2 for a precise, more general explication. Backward passes may be made independently of forward passing altogether; see Theorem 2 and Algorithm [
3. Pre-imposed optimality: Working in the framework of optimal control theory, we consider both running and terminal losses-a general 'Bolza problem' see $2.1 We give a necessary first-order criterion for optimal control (Theorem 3). In this way, account for t-varying parameter controls (t) (tt;p, x), omitted from the original Neural ODEs, permitting future compatibility with the recent particle-shooting models of Vialard al. (2020).
2

---

X
2(t) = f( z, 0)
z(g)
A(p)
( V,f(t, X
A(4) (V+7]
+
x.g,p.x)
[al(np,)) - -vz((tp,)) -(pxx)
z(g.
V,A(p.x) - - 0(p)),((p))
A(p A,x)
A(g,)) [VxT(xx
k(p)),x0(p))² A(p.xj]
Figure 2: Top: Neural ODE constitutes two-point boundary value problem over [p. g]- Bottom: An InImNet separates the forward and backward passes into separate initial value problems along the depth variable p.
Our contribution. We prove that it possible to reduce the non-linear, rector-valued optimal con- trol problem to system of cerward-facing initial value problems. We introduce new theoretical laap for the understanding of depth in neural networks, in particular with respect to viewing DNNs dynamical systems (Chen al. 2018 Massaroli al., 2020; Vialard et al., 2020). We demon- strate that this approach may be used in creating discrete and continuous numerical DNN schemes. We verify such algorithms by successfully applying our method to benchmark experiments, which perform well on a range of complex tasks (high-dimensional rotating MNIST and bouncing balls) comparison to state-of-the-an of-the-art techniques.
Architecture. In 3 we give novel 'InImNet' architectures implementing the above results. These may be applied to regression and classification problems using discrete or continuous algorithms.
Experimental performance. In $ we present some promising results for their practical func- tionality. These are designed support the theoretical development of InImNets $ and support the proof of concept in application. We derive various successful discrete implementations, based on computing the state evolutions (2) and (3) with an Euler-step method. We also identify operational aspects in InImNets which we expect to encourage ongoing research and development. Crucially, our architectures are compatible with various technical advancements since made to Neural ODEs, including those by Dupont et al. (2019); Davis et al. (2020); Yıldiz et al. (2019); see also a study of effective ODE solvers for the continuous models (Hopkins Furber 2015).
Broader impact for optimal control theory. Further afield, result applies more generally as a new tool in the theory of optimal control. The mathematical technique that we apply to ( deriving (2) and 33) is known the Invariant Imbedding Method. The key output of the method the reformulation
a two-point boundary value problem as a system of initial value problems, given as functions of initial value and input location p alone (Meyer 1973). This stands in the literature as an alternative to applying the calculus of variations (Liberzon) 2012 Vialard et al.,
2020 For linear systems, the technique was first developed by Ambarzumian (1943) study deep
stellar atmospheres. It has since found widespread applications in engineering (Bellman & Wing,
1975). optical oceanography (Mobley) 1994). PDEs (Maynard & Scott 197I). ODEs (Agarwal & Saraf 1979) and control theory (Bellman et al. 1966] Kalaba & Sridhar, 1969), to name a few. The non-lineat case is only touched on in the literature for scalar systems of zero terminal loss Bellman al.) 1966 Kalaba & Sridhar 1969 inccluingg some numerical computations to support its efficiency (Spingarn 1972). In this work we derive a complete invariant imbedding solution, fit for applications such those mentioned above, for a non-linear, vector-valued Bolza problem.
Overview of the paper. In 2 we give the main theorems used for InImNet architectures and more widely in the field of optimal control. Detailed derivations and proofs may be found in the appendix, { In 33 we put forward various architectures to illustrate how InImNets may be utilised in learning paradigms. In $ we describe supporting experimental work for such architectures.
3

---

2 OPTIMISATION VIA INVARIANT IMBEDDING
Solutions to (T) depend implicitly on both am input datum x and the input position at which the input is cast. The (p. x) relationship is at the heart of the invariant imbedding method which plicates these arguments, written into the notation as zt) z(t; p, p)). In this section we begin by introducing the optimisation problem before giving invariant imbedding solution. Fix inte- gers M, N 2 1 to denote the dimension of the instantaneous parameter space (()) E IR ad and the dynamical system z(1), [11, 2, 0) R
2.1 THE BOLZA OPTIMISATION PROBLEM
The key advantage of studying continuous DNNs is to redress the learning problem as an optimal control problem. Here we consider the most general control problem, a Bolza problem, which subject to two forms of loss: terminal loss, the discrepancy between the system outputs z(g) and the true outputs y measured by loss function T on RR%; and a running loss, which regulates both the state vector x(() itself as it evolves over &. E [p, g) but also the control (() over [p, q) by square-intedd integrable functional R on (p. q) x RN x RM Together, the minimisation problem is to find control ((t) and solution ztt) to I whilst minimising the total loss
J(0: D. x) = |" R(t, pl O(t; x)) T(z(Tp,,)).
for each known datum pair y). Applying the calculus of variations, one considers small pertur- bations about an optimal control O which minimise J(O: p. x) whilst determining a solution z to []]. The well known first-order Laler-Lagrange optimality equations (see $E.3) thus derived con- stitute a constrained, two-point boundary value problem as depicted in Figure 2 By contrast, the invariant imbedding method restructures the first-order optimal system an initial value problem. The t-dependence brushed implicitly under the rug, with numerical integration performed instead on the depth variable p.
2.2 THE INVARIANT IMBEDDING SOLUTION
The fundamental principle we follow is to observe the imbedding of intervals [p + A. C [p. gl. for 0 < A < p- q. which carry solutions z(4; + A, x) (I), whilst keeping the input, x, each invariant. In limiting terms as A + 0, the partial rate of change in depth V, 0/0p is directly related the vector gradient Vx for the initial value x at p. This controlled by the coefficient
(p. == f(p, x, 0p)).
(5)
Theorem 1. Let O(t) be an admissible control, by which we mean O is piecewise continuous on t E p..g). Suppose the dynamics function f: [p. q) x RNN x TR T is continuous (1, 0) E
[p. gl x TRM and continuously differentiable on R.N Let E [p.q] and suppose z(t: Pt x) and
((t: x) satisfy ( for each E NN Then we have the invariant imbedding relation
V vpz(tip,x)= = -V (p,
(6)
The assumptions in Theorem could be relaxed further. For instance, one could expand the class of admissible controls (t) to those which are measurable and locally bounded. The dynamics function could relax the existence assumption on V.f in replace of Lipschitz property. The assumptions stated are permissible for wide range of applications. One may read more about possible weaker hypotheses in (Liberzon 2012 $3.3.1).
We use this result given by 66 as model to address the following learning problem. Consider collection of input values x R corresponding to known output values y E R. We seek to extend approximation of + y to larger subsets of g.V. We proceed by choosing an interval [p, ql C R of arbitrary depth (fixing and varying p) and postulate a state vector z(t: p. x), subject to that approximates y at q given the input 2(p; x) x.
The parameter control, whilst commonly restricted in applications, is a priori subject to the same dependencies O(t) 0(t:p, x). We denote second coefficient related to its endpoint by
T (p, x) 0(p:p,
4

---

which, for S & 9, also satisfies the invariant imbedding relation in Theorem
V pt,(t;p,x) = - - - (p,
(8)
Whilst this observation may be made of the underlying dynamical system alone, the consequences extend to the control problem in 2.1 and form the basis of our solution.
2.3 BACKWARD LOSS PROPAGATION
The calculus of variations, introduced by Euler and Lagrange (Euler 1766). provides a mechanism by which to find an optimal control 0 (see Liberzon 2012 Ch. 2). The key trick to invoke function called the Lagrange multiplier (tt) X(t: x). also known as the "adjoint state" (Chen et al. 2018) or "Hamiltonian momentum" (Vialard et al. 2020), which encodes the backward- propagated losses. Indeed, p(;;,, the initial value at the endpoint i = q. is defined to be the gradient of the terminal loss T (,( with respect to z(q: P. x). To optimise the network, whether directly using (12) X(t;p, x) back to t = p. To this end we introduce the 'imbedded adjoint' coefficient
stochastic gradient descent on the weights, one must propagate
App X,p:,,
which is the subject of main backward result.
Theorem 2. Suppose that 0, and f satisfy the hypotheses of Theorem 7 for each S Suppose too that the terminal loss, T, and running loss, R. are defined as in Equation (4) subject to the hypotheses of $2.1 Then the imbedded adjoint Mp, x) satisfies the reverse initial value problem
AV x) XxA(p.x) (V
((px)))
(p, (10)
initiated at p q by the value (g,,x) = (x).
The bracketed use of [v,f] etc. is to stress that the differential operator acts before evaluation.
We contrast our approach of fixed et al. (2018), who fix p and vary p I t s q. Our derivation provides a new proof of the standard Euler-Lagranee Lagrange equations which give the adjoint method, manifesting in our account as
q and varying depth p with the adjoint method of Chen
/ (1, A(t:p.x)
x(p:p.x) (V
2, (11)
with initial value A(1; = V (2(q: p, x)) at p. Observe that Theorem the initial loss (x), for the trivial network. Back-integrating this
at y is given by [ x]) term thus does not require a forward pass of the z-state at the cost of computing the derivatives with respect to AxA(p,x). x). Optimising this latter process opens new window of efficient DNNs.
2.4 A FIRST ORDER OPTIMALITY CONDITION
With the insights gleaned from optimal control theory, Vialard et al. (2000) take different approach facilitating f-varying parameter controls O i; p, x). This is based on assuming that 0 is optimal from the outset. This is achieved through specifying 0 by the t-varying constraint
2, 0) + Vo (1, о)т x(I:p,x)= 0.
(12)
We obtain a corresponding condition for the coefficients that constitute an InImNet.
Theorem 3. Suppose that 0. z and f satisfy the hypotheses of Theorem for each p S q. Suppose the losses T. R and I are defined as in (4) subject to the hypotheses of $2.1 Then the first-order optimality condition for the total loss J (6;:1 x) to be minimised is given by
[oR] (p.x)) + [of (p.))) A(p,,)=
(13)
Making the (p, x) deeeendnny explicit for optimal controls, this identity provides a mechanism by which the depth p itself is accessible to optimise. In practice, (p. x). and hence ( x),
derived from A (p. x); these are connected by and (13 constitute the invariant imbedding solution to the general Bolza problem advertised.
Altogether, Equations (6) (). (10
5

---

3 INIMNET ARCHITECTURES
The architectures presented here are based upon the results of Theorems 2 &
Whilst the pro-
cesses for obtaining the p-th network state z(;; p, x) and backward adjoint A p. x) processes, we nevertheless describe their general form together in Algorithm
are independent
Algorithm Independent forward and backward pass with InImNet
Require: Training set of input/output pairs (x,y): evaluation points p1 <... Pni loss function T (see 2..
: training function f(1, , z, 0).
Ensure: Track x-operations for auto-differentiation, substitute numerical derivative (see $3.2
Inputs: z(g; Ps, x) = x; A (q, x) = TxT(x)
for
do
Z(q Pi, (q: Pi+1, x) + fpt vpz(F:.x) (pt, x) = A (pi+1, x) Ipi (p,
D Use Theorem
D Use Theorem
end for Returns: Tuple of outputs z(qi Pcs x),) corresponding networks of varying depths pt-
In the remainder of this section describe various discrete models to implement variants of Algo- rithm []] Continuous architectures using black-box auto-differentibble ODE solvers, such as those considered by Chen al. 2018). may be readily implemented. This approach poses interesting new avenues of research based on implementing accurate numerical alternatives to the computation of nested Jacobians. Simultaneously, the question of stability of DNN dynamics functions becomes another crucial question to address.
For our experiments we seek show first-principle implementation of InImNets, and we do so by describing discrete architecture, executing the minimum computation time whilst demonstrating high performance; see $3.1
Finally, time-series data, or running losses, are not considered by Algorithm but may be solved by InImNet, their dynamical structure natural formulation for InImNets. We consider such an architecture in $C of the appendix well as its application to regression problems in F22
3.1 EULER-METIIOD EXPERIMENTAL ARCHITECTURE
For implementation, we pass through the underlying ODEs with a proposed architecture based on simple forward-Euler solution to the integrals in Algorithm [ This is comparable the original ResNet architectures (He et al. 2015). To do this we divide up the interval into collection of layers [p, g] = Ui= [P: Pi+1] and rewrite the invariant imbedding equation of Theorem 1
(t:: Pi, x) = (t: - (p: V Pi+1. + (p;, x),
(14)
subject to 2Pn: Pn, x) = x. Backpropagation may then executed by either differentiating through the system, the standard approach, or by implementing Theorem) through the forward-Fuler formula
A x) A(p++1.x)
- (p:-Pi4il V xp), + xxf(p,,x00pp)) A(p(11,)) (15)
with the initial condition A(p., x) Tx)(x). For our experimental applications, we also apply technique to approximate the inherent Jacobians within the InImNet architecture; see Equation (2T)
in $32
3.2
NUMERICAL APPROXIMATION OF INPUT GRADIENTS
An implicit computational speed bump is the computation of the gradients Vx in (2). 6 and 10) The immediate solution is to track the gradient graphs of these terms with respect to x and implement automatic differentiation. Indeed, this approach does yield successful models-if if one has time their hands but the drawback that a high memory cost incurred for deep or high dimensional networks.

---

We offer some surrogate numerical solutions. For the sake of example, suppose wish compute p. E I. N by integrating
[pz( = -V Vx T (pp x)
(16)
with respect to p. To compute the derivatives Vx consider perturbations of the input vector RN of the form x + A,e; for appropriately small A > 0 and e; := (ō.)) for L....,.. N.
We then solve for the 2.V + states z p. x e; by simultaneously integrating (16) alongside
V V (T P, X + e, = - V (T + A,e.) (bp, + A,
(17)
where the gradients V Z 4 Xo) are modelled by
<(q:P, x + A,e;) A0::, A,e;)
V z q: Xo)
(18)
24;
for X x, , respectively. This method is known as the symmetric difference quotient. Other approximations may also be applied on a bespoke basis, such as Newton's 's difference quotient. This uses similar construction but the negative shifts are forgotten, resulting tracking N equations along (16) and (17) where we estimate
z(gip. e,,,) p.
V Vxz(g;;,))
(19)
A
and
(rd(px++++e))) -V xx
(20)
Alternatively, and more directly, may tackle computing the successive Jacobians x p, x), incurring a high memory cost storing gradient graphs, by approximating such terms through crop- ping the higher order gradients:
Pxz(t:p(.x) = V ((t1; Vx V2ttttp x) ((p..)
V x P1+1, xp)
(21)
Whilst theoretically losses are easily quantifiable, we show experimentally that for this approxima- tion an increasing the number of layers still improves the performance of the model.
4 EXPERIMENTAL RESULTS
In this section we demonstrate the practical ability of the proposed architectures in solving bench- mark problems for deep neural networks. All experiments use backpropagation to differentiate through the system as outlined in exceptt for the projectile motion study the appendix, $12 where training is executed via the update rule stated by Equation (15). Without loss of generality, we arbitrarily choose 9 = 0 and assume p to be in the range [Pmin- 0] where Pmin is a negative value whose modulus represents 'depth' of the network.
4.1 BENCHMARK VALIDATION
4.1.1 ROTATING MNIST
We replicate the experimental setting used by Yildız et al. (2019) and Vialard et al. (2020). This gives a point of comparison between our proposed InImNet, for various depth architectures, as described in $3.
In the 'Rotating MNIST' experiment, we solve the task of learning to generate the handwritten digit '3' for a sequence of sixteen equally spaced rotation angles between [0. 2nT] given only the first example of the sequence. To match the experimental setting with Yíldız al. (2019) and Vialard et al. (2020), we train the model on layer Pmni using backpropagation by minimising the objective of mean squared error for all angles, except for one fixed angle (in the fifth frame) as well as three random angles for each sample. We report the Mean Squared Error (MSE) at the fifth frame and the standard deviation over ten different random initialisations.
The results of this experiment are given in Table [ The details of the experimental set-up and the hyperparameters are given in &G.I of the appendix. The results show comparable performance to that achieved by (Vialard et al. (020) while using more computationally efficient discrete invariant imbedding formulation (see $4.1.2). We implement the InImNet dynamics function (as in Theorem 1 as Multilayer Perceptron (MLP) with either two three layers.
7

---

Existing Models
InImNet
Time
Method
MSE 1
Pmin
MSE to
[s/epoch]
GPPVAE-DIS
0.0309 + 0.00002
0.0156 + 0.0008
3.3459
GPPVAE-JOINT
0.0288 + 0.00005
-2
0.0130 + 0.0005
4.1624
VAE
0.0194 0.00006
-3
0.0126 + 0.0007
5.0060
VAE-VAE
0.0184 + 0.0003
-4
0.0125 + 0.0004
5.5806
Vialard et al. 2020
0.0122 + 0.0064***
-1
0.0176 0.0010
3.1504
(*) 8.63633/epoch
2
0.0129 = 0.0008
4.0412
-3
3
0.0125 + 0.0003
4.886
-4 0.0126 + 0.0004
5.8521
Table 1: Rotating MNIST: Reported MSE for the proposed InImNet (where n is the number of MLP layers) and state-of-the-art methods, results from other methods taken from Yildiz al. (209))
and Vialard et al. 2220)) (**) The standard deviation given is more than half the mean value as stated in Vialard al. (2020)
GT
p
p 3
4
experiment with Pmin 4 and two-layer MLP. Right: samples from 'ouuncing Balls' experiment with Pinin = -3 and a three-layer MI.P. See G.1 and sf.2 of the appendix for architectural details.
Figure 3: Left: samples from 'Rotating MNIST
4.1.2 BOUNCING BALLS
in the previous section, we replicate the experimental setting Yildiz et al. (2019) and Vialard al. 2020). We use the experimental architecture given in $3.1 and list hyperparameters used of the appendix.
S..2
Time
ODEVAE,
Model
Pmin
MSE
[s/epoch]
0.12
DOPAL meam: 0,0242
D. mean: 0.0732
InImNet
0.0234
91
0.10
proposed, pminm-1, pm3, mear:: 0.0234
InImNet
2
0.0204
121
MSS
proposed, pmcpl:mmpp-22p00 n-3 mean: 0.0204
pmin=33,
InImNet
3
0.0192
153
Vialard
0.0146
516
0.00
Frame
Figure 4: Bouncing balls experiment. Left: reported MSE for the proposed InImNet and state-of- art methods, results from other methods are taken from Yildiz et al. (2019) and Vialard et al.) 2020). Right: average time consumption per epoch.
8

---

For this 'Bouncing Balls' experiment note that the MSE of the proposed InImNet and the state-of- the-art methods are comparable while using a more computationally efficient model. We measured the time per epoch whilst using public configuration of Google Colab for the (best-performing) up-down method of Vialard et al. (020)) against InImNet (with Punin = -3; three-layer MI.P). We set the batch size to the same value of 25 given in the configuration of the official implementation in Vialard et al. (2020). While the proposed InImNet requires 153 seconds per epoch, the method as described by Vialard al. (2020) took 516 seconds to finish one epoch.
4.2 EXTRAPOLATION BEYOND OPTIMISED MODELS
The key insight provided by InImNets is the simulation of multiple networks (for each p) from single network pass. We give an ex- panded account of performance of this nature $F. of the appendix, alongside the experi- mental set-up. To demonstrate such observa- tions, we exhibit prototype loss-plot for the Rotating MNIST experiment (in the appendix, see Figure 77.. The horizontal axis varies with the frame of the rotation-the dip in loss corre- sponds to a neutral frame, as expected. Each line, p -1 -2..... corresoonds to output at a different p-layer. The network is tuned (or trained) on its outputs layer p Pmin -4. However, by extrapolating either side of Pmin we can make quantitative observations on the rate of loss degradation; for example, the total loss remains lower shifting to p -5 rather than 3.
0.14
pa-2 2, MSE 0.0467)
0.12
MSE 0.0209)
MSE 0.0157)
MSE 0.0191)
MSR
p MSE 0.0720)
MSE 0.0834)
0.06
0.04
Framele Id
Figure 5: Extrapolation beyond and before chosen training depth Pmin -4.
5 DISCUSSION AND CONCLUSIONS
We have shown that it is possible to reduce the non-linear, vector-valued optimal control problem to a system of forward- facing initial value problems. We have demonstrated that this approach may be used in creating discrete and continuous numerical schemes. In our experiments, we show that (1) for a range of complex tasks (high-dimensional rotating MNIST and bouncing balls), the discrete model exhibits promising results, competitive with the current state-of-the-art methods while being more computationally efficient; and (2) the continuous model, via the Euler method, shows promising results on a projectile motion task. They demonstrate the potential for inference in continuous neural networks by using the invariant imbedding method to vary the initial conditions of the network rather than the well-known forward-backward backward pass optimisation.
We have outlined a class of DNNs which provide a new conceptual leap within the understanding of DNNs dynamical systems. In particular, the explication of the depth variable leads new handle for the assessment of stacking multiple layers in DNNs. This also fits within the framework of Explainable AI, whereby an InImNet model able to depict a valid output at every model layer.
Of course, nothing comes for free. The expense we incur is the presence of nested Jacobian terms; for example, Vxz(1;p, p, x). We show experimentally that our models perform well with elemen- tary approximations for the purpose of functionality. But understanding these terms truly is deeply
related to the stability of Neural ODEs over a training cycle.
In this article we do not explore the ramifications of the optimality condition of Theorem 3 With the work of (Vialard et al. 2020), in which systems are considered optimal from the outset via Theorem 3 propose study the variability of depth of such optimal systems.
We end where we started with the image of Figure I In the appendix, in C Candd F2 we implement time-series architecture and apply this modelling projectile motion curves. We discuss the diffi- culties faced by InImNets for high-depth data in (F.Tand suggest a promising avenue for extended research.

---

ACKNOWLEDGMENTS
The first author would like extend sincere thanks to Prof. Jacq Christmas and Prof. François- Xavier Vialard for their engagement on this topic.
REFERENCES
A. Agarwal and S. Saraf. Invariant embedding: A new method of solving a system of nonlin- car boundary-value differential equations. J. Math, Anal. Appl.. 72(2):524 532, 1979. ISSN 0022-247X. doi: 10.10160022--477850 244170222-2777799999255222 (79)
URL https: wttp://i......0.........0000....
V. Ambarzumian. Diffuse reflection of light by a foggy medium. C. R. (Doklady) Acad. Sci. URSS (N.S.), 18:229-2333 1943.
R. Bellman and G. Wing. An Introduction to Invariant Imbedding, volume Society for Industrial and Applied Mathematics (SIAM), 1975. ISBN 0-89871-3044 doi: 10. 10.11377/.9781111197233 URL mattts:://oll. htt.:::://ooo.ogg11.... 11w1/111111111 911.111177...911111999711..
R. Bellman, H. Kagiwada, R. Kalaba, and R. Sridhar. Invariant Imbedding and Nonlinar Filtering Theory. J. Astronaut. Sci., 13:110-115, 1966. ISSN 0021-9142.
A. Bensoussan, Y. Li, D. C. Nguyen, M.-B. Tran, C. Yam, and X. Zhou. Machine Learning and Control Theory. arXiv preprint, 2020. URLhttp://ann www.xiiioorrr.oooooomoooo /l///bb//20 taab:/200..0. 05604
Q. Chen, Y. Rubanova, nary Differential Equations. Advances in Neural Information Processing Syy- tems 31, pp. 6571-6583, 2018. URL http://papers nips cc/paper/
Bettencourt, and D. Duvenaud.
Neural Ordi-
J. Davis, K. Choromanski, J. Varley, H. Lee, J.-J. Slotine, Likhosherstov, A. Weller, A. Makadia, and V. Sindhwani. Time Dependence in Non-Autonomous Neural ODEs. arXiv preprint, 2020.
URL /tttp://arxiv..rtt
abs/ 01 906
E. Dupont, A. Doucet, and Y. W. Teh. Augmented Neural ODEs. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d' Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 3140-3150. Curran Associates, Inc., 2019. URL //ppers.
n
pdf
Fuler. Elementa Caleuli Variationum. Novi Comment, Acad. Sei. Imp. Petropol, 10:51-93, 1766. URL httpp:::/scchlaalyycoaaac/a httpa//aee..cccooooppppppeceeeea
workse 296/
K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015.
M. Hopkins and S. Furber. Accuracy and Efficiency in Fixed-Point Neural ODE Solvers. Neural Comput., 27(10):2148 2182, 2015. doi: 10.1162/NEC....o00 _00722. URL httts::///ttt orgg
10.1162/NECO. a -0772
R. Kalaba and Sridhar. Invariant Imbedding and Optimal Control Theory, Optim. Theory Appl., 4:343-351, 1969. ISSN 0022-3239, doi: 10. 10.1007/BF00927666
URL httttp::///oto org/
10.1007/BP00922661
D. Liberzon. Calculus of Variations and Optimal Control Theory: A Concise Introduction. Princeton University Press, 2012. ISBN 9780691 1151778.
Y. Lu, A. Zhong, Q. Li. and B. Dong. Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations. In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3285. PMLR, 10-15 Jul 2018. URL http: http:://proceedings. mlr. mlr.preas/v8) uu 8d. html
10

---

A. Lyapunov. The general problem of the stability of motion. International Journal of Control, 55(3):331-533 534, 1992. doi: 10. 001108000022011192999933333 URL https: http:///oo....//./ 1080/00201179208322
Massaroli, M. Poli, J. Park, A. Yamashita, and H. Asama. Dissecting Neu- ral ODEs. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, pp. 3952-3963. Curran Asso- ciates, Inc., 2020. URL /tpps///pp/ppccce......s http://errrssss.ccaaaeee//////// hc/pp//er///22///iiee///////// 23935222220336611197788133553--3
pdf
C. Maynard and M. Scott. Invariant imbedding of linear partial differential equations via generalized Riccati transformations. J. Math. Anal. Appl. 36(2):432 459, 1971. ISSN 0022-247X. doi: 10.1016/0022 1111166002222477()1))00000 90011-4]
(71)
G. Meyer. Initial Value Methods for Boundary Value Problems: Theory and Application of Invariant Imbedding, volume 100 of Mathematics in Science and Engineering. Elsevier, 1973. doi: 10. 101655007-532220869988888 62980-X
URL 04
(08)
C. Mobley. Light and Water. Academic Press, 1994. UBL/http:/ww..ccaangllcccbb Info/view/ http://ttooootooooeoeeeeeeee
L. Ruthotto and Haber. Deep Neural Networks Motivated by Partial Differential Equations. J. Math. Imaging Vis, 62:352 364, 2020. doi: 00.10077/10851-019-000000 URL https:
//doi. 1007 00000://10011011111 011-0000---0-00-0-11
K. Spingam. Some numerical results using kalaba's approach to optimal control and filtering.
IEEE Trans. Automat. Contr. 17(5):133-715, 1972. ISSN 0018-9286. doi: 10.1109/TAC.197. 1100124. URL https:/ /oi.. /10 110. TAC. 1972 1100124
F-X. Vialard, R. Kwitt, S. Wei, and M. Niethammer. A Shooting Formulation of Deep Learning. Advances in Neural Information Processing Systems, 33, 2020.
C. Yıldız, M. Heinonen, and H. Lähdesmäki. VDE Deep generative second order ODEs with Bayesian neural networks. Advances in Neural Information Processing Systems, 32:13412- 13421, 2019. |tthttts:: wttp://aavvv//////e/ wwtp://ggpppp///////oe 10994
A OVERVIEW OF NOTATION AND CONVENTIONS
We consider all R-vectors column vectors, to be denoted in bold font. Let (v, w) denote the Euclidean product between two vectors v and w. Matrix multiplication between A and B shall be denoted A B and transposition by AT For the row vector V. = (0/0u;); which operates on scalar fields as usual but also on vector fields F(v) = (v)()))
(v)); introduce the gradient operator by
give the Jacobian matrix F (OF,/au,)i Then, if v v(w), the chain F.F V. which could be expressed as 00/Ow = vV dv /dw) if w were both scalar valued. Moreover we extend the gradient notation by using it for scalars p E IR to denote the single partial derivative V = -/Opp. Define the second order gradient operator by the matrix of second order partial derivatives V2 = V V. = (82 (0/00r0rr ) with respect to v = (wa); and w = (w:);- We use the Dirac delta symbol given by 8.z = if and only otherwise. The N N identity matrix then given by 1N == (&tz) siJ.N.
rule reads V V F (vw))
= é and
if = j and 6,g Lastly, for a (possibly vector valued) function on IR we write F(x) = o(z), implicitly with respect + 0, to denote that lim F(x)/x -.
B THE IMBEDDING RULE
InImNet architectures are based on the principle of invariant imbedding: the description of a system of given length in terms of its imbedded sister systems. For example, consider two lengths p2

---

P1 <. such that the solution z(t: p1,x) I is known for P1. The solution z(t: (2; x) over
[22. ql restricted to 2 Pt. with x imput at p2, may be expressed using by
z P2, = (1;
(zz(t:p.))
(22)
Pi
In this way, two separate intervals may be adjoined by observing the imbedding rule
(t: P2, <t; P1, 2 P2: P2.x0 x))
(23)
where the new input P(pi:22, x) is evaluated using 22 at t = P1. In the case that is a linear
function of z, the identity in (22) is sometimes known as the Ricatti transformation (Bellman &
Wing 1975)). ). In practical computations, one should devise models, such as step-linearisation, to
solve 22 by simulating the gradients Vxn(p1;,, x) for p < p2-
C LEARNING LATENT DYNAMICS FROM END-POINT OBSERVATIONS
Here we propose an architechture for the application of InImNets to time-series regression problems, such as we implement in $F2
Typical regression problems ask for model, given by a state z(t), to approximate t-varying process yt) sampled at finite number of training points y(t,) in a fixed region of interest i; E pp. 9). Assuming that y satisfies well behaved ODE, the learning paradigm of a Neural ODE 2 = f. see (I) is well suited to learn such continuous dynamics. Inherently, the initial condition (p) = x is assumed fixed.
InImNets are naturally structured to learn a variant of this problem: consider a I-varying process modelled by the dynamical system
(t:pp. =
- g(t,)); y(P:,,))= =x
(24)
for which we are able vary the location p S q of an invariant initial condition y(p) x. Suppose the sampled data is only available at the fixed output = q of the process, meaning that the training data given by a set of the form
{u = y S 4):
(25)
In this way, an InImNet learns the dynamical system itself, as the initial conditions vary, from the outputs alone. Apply this to the running loss in Theorem 2 by, for a differentiable cost function C: RRNN x ERN.N R. defining
R(t, 2, 2, 0)
R(t,2) = Σ (le
(26)
Then the p-th invariant imbedding coefficient is
(p. Σ 6p.p. (VC_(((((p::x).))) Vxz(g,pxx).
(27)
The term 6p.p. features intrinsically in Neural ODE architectures as only single depth is consid-
ered in a given system. We implement (26) recursively (cf. Chen et al. 2018 Figure 2) so that the discrete architecture for the adjoint reads
A x) = A(pi+ + V R x)
- (p: - P++ V (p;, ( + x x)], (28)
The forward pass is executed exactly as in
3.1
D THE ADJOINT METHOD IN APPLICATION
D.1 BACKPROPAGATION AND THE ADJOINT STATE
As touched on in the introduction, the adjoint method, in optimal control, is natural conse- quence of the Fuler-Lagraage equations (which explicitly derive in (E.) To outline the
12

---

method, postulates multiplier function X(t: p.). the 'Lagrange multiplier' or 'adjoint', as in SE.2 One derives the initial value of the adjoint, at the endpoint = q, to be the loss gradient X. V ( q: x)). On the other hand, the t-varying dynamics is seen to follow the dy- namical system given by (47) Integrating (47) backward from t = q to = p may be interpreted as the backpropagation of losses throughout the system. Indeed, using routines such as those describe below in $D2 the loss gradients with respect to parameters may be explicitly derived.
The interpretation of the adjoint as backpropagated losses was illuminated, in the context of deep neural networks by Chen et al. (2018). They derive the backward dynamical system (47) for À using the chain rule over infinitesimal steps. This illumination makes clear that the adjoint method the direct generalisation of the traditional backpropagation method.
In another twist in the tale, the derivation of the adjoint method also leads natural first-order op- timality condition; we deduce this in (43) This theoretical tool gives an option for single-sweep op- timisation by explicitly solving (43) for the parameters via the initial value problem (47). Moreover, this equation permits one to determine continuously varying parameters (t). Such an investigation conducted by Vialard et al. (2020).
In the next section we describe the formulas required to determine loss gradients with respect to parameters the case when the parameter vector 0 = (()) not dependent on l.
D.2 AUGMENTED STATE VARIABLES FOR CONSTANT PARAMETERS
Updates to varying parameter controls (t:: x) hot subject of research (see Vialard et al. 2020 Massaroli et al. 2020 for example). The original Neural ODEs (Chen et al. 2018) operated by assuming that 0= O(t: p. x) E R was constant. It was then possible recover loss gradients for the components of A by augmenting the system and running the adjoint method (Chen et al.
2018 BB.2). A similar trick works in our context, with some quite remarkable consequences.
Consider an augmented state vector composed of z, and 0, whereby the dynamics in becomes
2 dt
ftt z, 0)
=
(29)
0
0
with z(p) x, = p and 0 = 0 as the initial condition. Working through the invariant imbedding procedure, the forward process (Theorem for z is preserved. But the backward process (Theorem
2) is revised by replacing A with the (new) components A A and A ( The first, A satisfies
Theorem 2 independently of (!) and A (0), Moreover, the assumption that = O()) is constant in
t implies that [V, x, L (p, x)) Vx + (p, x) so we deduce the new backward equation
(x) (p,x) ( A
p. x) $ x)
V x x (p,
V (2 V (p, ))
(30)
The term A (0) (p.x), corresponding to the gradient of the terminal loss with respect to the constant parameters 0, assumes the initial value
A (q, x) =
(31)
for whom the p-evolution (10) becomes v.A (0) (p.x) A (p.)) Po (p, (T,,))xTAA (A
())))) (32)
dependant on the simultaneous solution of A (z) (p.
For the (-term let us additionally assume that f(1, z, 0) f(z, and R(1, z, 0) = R(z, 0). This
implies that vyf(p, x, (p, x)) = &x x) o x). Then A (0) (p. x). the gradient of the termi-
nal loss with respect to = p, assumes the initial value
A (0) xg.x) = (A (z) (
(33)
and the p-evolution follows
V A = - AAVAA (p.x) ( + (V + (p, Φ x)) (x)
+ V. x, (p, (pp,))), (34)
13

---

dependant the simultaneous solution of A (p. x). This last identity is particularly exciting. It enables one to compute the loss with respect to p. the depth, itself. Given the explication this may thus plot this loss function of p and
p-variable in our Invariant Imbedding Networks, isolate the minima for any given training run. Observing the flow of such minima provides great insight to the optimal depth that network should take.
E DERIVING THE INVARIANT IMBEDDING SOLUTION
Here we give proofs of Theorems I and 3 which constitute an invariant imbedding solution for non-linear, vector-valued Bolza problem. Consider the minimisation problem described in $2.1 for the ODE system ①
We maintain the hypotheses stated in Theorem throughout.
E.1 VARIATIONS IN THE CONTROL
We explore the consequences of the calculus variations (Liberzon 2012 Ch. 2). This provides mechanism to find an optimal control 0; that is, provides a global minimum for J(0) < J
over all piecewise continuous controls 0. For E R considered close to 0, we consider controls perturhed from the optimal control 0 by
O(I:,,)) + +x((p;
(35)
for an admissible perturbation Following the system (I. the control in 35 determines the state (or trajectory)
(l; x) + (t: + o(s)
(36)
where 7) is corresponding perturbation subject to
(p; p, x) =
(37)
so that 36 agrees with z at t = p. To minimise the cost, we consider the Taylor expansion of J( EV) with respect to ε. We call the linear term the first variation J at O which may be defined by the Gateaux derivative:
J(0 + =v) - ((0)
= lim c-0
(38)
for admissible perturbations v. For the cost function given in ④4. the first variation is equal to
= f (VoR] 0), + S
8Tlo(u) =
(t:p,, x))
[V m7:: p, p,)).
(39)
The first-order, necessary condition for optimality (Liberzon, 2012 $1.3.2) is satisfied whenever
8J (v)) 0.
(40)
An auxiliary identity for the manipulation of (39 is obtained by evaluating vj. Differentiating (36) with respect to about 0, both directly and via (.. implies that
(t: x) = |V (t, 0) (t:pp. x) + [tf vt:p,,)).
(41)
E.2 LAGRANGE MULTIPLIERS
Introduce A(t) E R.V, an arbitrary vector-valued function for E [p.g) to specified forthwith. This is the Lagrange multiplier, called its namesake used it to multiply Equation (41 and insert it into the first variation (39). We subsequently obtain
8Tlo(v) 1 (Vo z, (Vo A(t), p(:
+
(A(t), (V)) 2, 0) 7t:p,,x ri(:: xt):,,)))dl
+
([V.R] (
x(:.p) (42)
14

---

We obtain optimality condition by making choice of A(t) p() such that
RV z, 0) + (t, 2, 0)" X(t;p;x)=
(43)
for all L r]: ef. (Vialad et al. 2020 (3.3)) where their "P:"" is equal to our - A. The first
variation (39 simplifies accordingly:
8Tlo(v) [ (A(t:p,x), V (t, 0) n(t:,,)) - p(t;p,x))dl
+ ff (V. 0) ( x) + (V & ( p.)
(44)
At this point our method of invariant imbedding diverges from the classical method of Euler and Lagrange. We give brief aside here to review the Fuler-Lagrange equations as the comparison is illuminating; see
$E.3
We also direct the reader to (Liberzon, 2012] $3.3-3.4) in which the classical
route is explored.
E.3 THE E LAGRANGE EQUATIONS
The LalernL equations reveal how the adjoint method of et al. (018), see also (Vialard et al. 2020 $3.3), will correspond to our invariant imbedding formulation. The derivation follows from integrating (44) by parts with respect to rj. One derives
8Jlo(()) = (VVfl(t 0)T x):p,x)= +
+ A(t;p, ((;p,,))
([V27(z(())))) x.)) - P( m(q: p,)). (45)
noting that the initial value of the perturbation is (p:p,, 0 by assumption. We simplify the constant term by imposing the initial condition on A:
P. = [2,7
(46)
Since (45) equates to 0 by the first order optimality condition (40) for all admissible perturbations 77. the fundamental lemma of the calculus of variations implies
{(t:p,
[[V z, O)T
A(t;p, + ( (t,
(47)
Equations (46) and 47) constitute the continuous analogy backpropagation known as the "adjoint method" by [Cen et al. (2018). Its formulation is thus encoded in (44) and so the forthcoming
method of invariant imbedding provides direct alternative to this backpropagation approach.
E.4 THE INVARIANT IMBEDDING METHOD
For each 0 < A 9 consider the family of subintervals [p+ A, q] of [p. g). Over these intervals, we imbed the systems ((:p++, x) into (1; p. x) whilst keeping x, the input, invariant. This results
in the partial differential equations 51) and 52 in which changes in p are equated to changes with respect to x.
To derive these equations, first note that over each interval the systems follow I which may be expressed as a finite difference equation:
2(1:p. x) = 2(1 + A; x) - =((,,((;p.x,
o(A)
(48)
for S t -
A. A. We then formally extend z(t; A to
48 so that
2(p + A,
f(p, <p: p + A, x), pp: (
o(D).
(49)
As such, the imbedding of systems over ip + A, 9] C (p. q is explicitly defined by
z(t:p + x) = 2tt;p, f(p, zpp + x), (pp x)) + o())
(50)
for all < t < q. To compute the partial derivative v,z = Oz/Op. consider the difference 2(1;p p- + A, x) z(I;p, p, x) in which x remains invariant. Substituting (50). dividing by A and taking
the limit A - 0+ one obtains the invariant imbedding identity for the trajectory.
V v,z(t:p, pe(t:,,xx x) -V x)- f(p,x, x, (p: x))).
(51)
Mutatis mutandis, derives the invariant imbedding identity for the control:
(,0(t:p, p, = (t: (t::p.. x,)) x). f(p. x, (x:;p)
(52)
These equations express a fundamental translational invariance property of both the optimal control and the corresponding trajectory. This proves the basic identity in Theorem T
15

---

E.5 INVARIANT IMBEDDING OF THE LAGRANGE MULTIPLIER
We extend the invariant imbedding relations (51) and (52) to the Lagrange multiplier A(t; P, x). The
extension is obtained by taking the gradient of (43) with respect Vp 0/0p and V Adding
the resulting expressions together and using (51) and (52) to replace the terms ppz(tip: x) and
V vy x) we derive the identity
[of]((, 2, OT (V,A(Lip.x) VxA(tipx)) f(p, x, ()):,
(53)
For non-trivial dependence of f on the control we assume that [1) 2, 0) is not identically implying the invariant imbedding identity for the Lagrange multiplier, A:
V A(: p, -x>((t;p,x) f(p, x, p,);,
(54)
These equations express fundamental translational invariance property of both the optimal control and the corresponding trajectory.
The coefficients f(p, P (,x) alone, independently is thus sensible to introduce the terminology
x)) and P(p,p, x) in 51 and (52) depend on the variables (p. x)
(p,, f(p,x, 0 p, E IRN;
(55)
(p, x) = Opp;, x) ERM
(56)
E.6 FIRST ORDER OPTIMALITY WITH INVARIANT IMBEDDING
Here derive auxiliary optimality condition from the first variation 44
subject the invariant
imbedding relations 51 (52) and (54). Recall that (44) is equivalent to the adjoint-state equation
of Chen et al. 2018 seen by application of the Euler-Lagrange equations: $E.3 The theory we
develop here should thus also be seen as analogue to the standard backpropagation method.
The first variation using Leibniz' rule and apply 7pp: p. xj O; moreover, each occurrence of V,z, vp0 and v
is equal to by 44). One may take its gradient with respect v,= 0/0p
may be substituted with their respective invariant imbedding relation. Then take the sum of the resulting equation with the gradient of (44) with respect to Vx scaled by the term ( x). One thus obtains the general form of the inst-variation auxiliary equation:
(V vn + Vxn(g:p.x) Dpp, + (A(p;p,x), ((::,,
(o.f(( 2, T¹T A(t:,, + V.R 0), V,n(t:p.x)) Vxn(t:p.x)
(pp x))
+ + (A(t:p,x), V,n(t:, + Vxn(t:,,)) @ (p,x) = 0. (57)
The derivation of 57 requires the multiplication of higher order tensors. This is readily completed using Einstein's summation convention. Note that before specifying choice of n there is no stipulation that it should adhere to an invariant imbedding rule of the form Vn = Vxn d.
E.7 SPECIFYING THE VARIATIONS TO SOLVE FOR THE LAGRANGE MULTIPLIER
The auxiliary optimality condition 57 holds for admissible perturbations (v. We now make specific choices to derive an initial condition for A p. x) at = For 1 < N define the column vector
n, (1: x) := ((1 -0)-)))))
(58)
This perturbation satisfies ny (p: po x) = O, as required, and has derivatives given by vi; = -V - =
(diy)s and V 0; whilst all gradients with respect to Vx satisfy Vx Vx 0. By
substitution into 57 we show that
- [. ([V.)](1))))))" A(:., + ( (59)
X.p:p.))
(.7((((()))))) (
V
Note that by the fundamental theorem of calculus, the initial condition in (59 is equivalent to the adjoint equation of Chen al. (2018).
16

---

The remaining results then derived as follows. Theorem 2 follows directly from (59). Take the derivatives VpA(p, x) and (p x) from the explicit formula in (59) and substitute the Vp terms with the invariant imbedding relations (5T). (52) and (54). Letting V (p. x) operate on
(pp, x) Theorem 2 follows by summing the two equations. The initial condition at = q follows
immediately from (59) Similarly, Theorem follows by specifying (40) at the endpoint 9-
F FURTHER EXPERIMENTATION
F.1 INIMNETS AND LEARNING DYNAMICAL SYSTEMS
A significant speed bump in the training of InImNet architectures is the computation of nested Jaco- bians Vx. This problem increases exponentially with depth. We are able demonstrate successful results with low depth implementations. We also provide elementary mechanisms to circumvent the issue; see $3.2
Time series problems require vastly deep InImNets given the training data is spread along series of times (or rather depths p). This the case when applying InImNets to model ODE dynamics directly. But it is the immediate alternative replace the Jacobians with numerical gradients as in t3.2 draws out a more interesting problem. The stability due to the initial divergence of the numerical gradients is related to 'Lyapunov exponents' (Lyapunov) 1992) that measure the growth rates of generic perturbations. The simulation and optimisation ODES by neural networks with perturbed initial conditions, in particular InImNets, is deep one and the subject of immediate ongoing research.
We nevertheless are able to construct rigorous architecture to handle time series data-see and demonstrate its functionality on a sufficiently small experiment.
F.2 A PROJECTILE MOTION REGRESSION PROBLEM
InImNets can be trained to model unique brand of time series observations. We depict this with the example of projectile curves, see Figure The data provided is not sample along the l-varying shape of given curve, but the single output of many curves initialised at different positions with the same velocity.
We simulate this via the discrete running cost training paradigm in $ The system we consider is parameterised by state variable z [h, u). denoting vertical height and velocity, such that l-axis is proportional to horizontal position. The ODE system dynamics is then the form z(t) = [u(t), -9] where 9, gravity, is positive scalar constant (which we take to equal 9.81). We choose to integrate p (backward) over the interval |pmin q| = [0, 1].
The architecture training data by the time-series algorithm in $ We implement a constant-in-f training function and further make use of the augmented adjoint routine described in $D.2 to update the parameters of f. We run the experiment with learning rate of 0.001 over 10 training epochs.
use is the backward loss propagation described in Theorem 2 adapted to the
We operate the experiment resolution of points p 0, 0.25, 0.5, 0.75, 1 with sample size of just four. Using the automatic differentiation method for computation of Jacobians as per Equation (15), this is the maximum resolution that we are able run without RAM availability being exhausted by growing Jacobian calculations-tie is prototypical of the difficulties described in $F.1 As such, the InImNet performs poorly on this task in lieu of sufficient p-resolution, and hence training data. Nevertheless, this experiments suffices to prove the concept that InImNets may be used to solve different genre of their nested derivatives.
problems, with further research demanded on the implementation of
We implement this experiment in a Google Colab notebook which provided as supplement to this article.
F.3 EXTRAPOLATION OF OUTPUTS BEYOND OPTIMISED DEPTHS
To further demonstrate the ability of InImNets to produce meaningful representations at different layers, we have conducted experiments that extrapolate results for depths p beyond Punin- This
17

---

Fixed p 0.000 varying initial velocity
Fixed inilial velocity V. varying
10
13.01
==.90
v= 11.55
v 13.90
hit: x.10
%
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
t
Figure 6: Plotted curves are given as training samples, where the InImNet's task, trained only on end points, is to identify the curves' height h(t; x) t - q The InImNet outputs are represented by correspondingly coloured triangular markers.
considers networks of greater depth, with training occurring at lower layers only. our experimental architecture of the weights are optimised independently: each i-th layer is parameterised by distinct (p,, x). Therefore we cannot perform extrapolation we have introduced variant InImNet architecture for both the rotating MNIST and the bouncing balls tasks where & identical for all layers; that is, for each we have 2 (p,, x) = ((x). This is comparable to the t-invariant parameters used by Chen al. (2018) In this architecture
compensate, such setting.
take Punin = -3 for bouncing balls and Pmin = -4 for the rotating MNIST. Otherwise, the rest of the parameters have not been changed from the original model hyperperameters (see G
With this architecture, we observe competitive performance for rotating MNIST-albeis worse than the original architecture varying & over the layers in 44.1.JJand |G.TTaa well the impact of going beyond Pmin- Figure 5 shows the average performance depending on the frame sequence number. One can see that the accuracy decreases for higher layers. However, we can make quantitative observations about the rate of this loss change in different directions. For example, the loss is lost at a slower rate moving deeper into the network (p 5) rather than shallower (p -3).
The examples of extrapolated sequences from the testing set are given in Figure 7 It can be observed that the extrapolation in higher layers (such -8) accompanied by decrease in sample diversity across the sequence.
For the bouncing balls task, we see that coupling the weights makes more challenging predict the balls' dynamics with the same hyperparameters in G.2 as expected. However, we demon- strate here that we can quantify and explain this discrepancy as the model gives us the insight into what it learnt. In Figure 7 while not successfully modelling the dynamics, the model converges to the 'best' strategy of erasing the sequence. Once we increase the layers, the same effect results in the reversal of the original sequence.
F.4 EXPERIMENTS WITH CONVOLUTIONAL ARCHITECTURES
We demonstrate the ability of the InImNet architectures to be composed of larger convolutional layers, allowing for end-to-end training of an InImNet model. With the autoencoder architecture as in Vialard et al. (2020) to define an InImNet layer and repeat the experiments modelling bouncing balls outlined $4.1.2 For explicit details on the architecture of this model, see GG.3
We report on performance results for Pmiin
(-1, -2 -3} in Figure g and give sample demonstra-
tions for different layer numbers in Figure
18

---

3
3
5
5
GT
p=
p -1
3
5
3
== 2
3
3
3
3
3
3
3
3
3
P -4
3
3
3
3
3
3
4
-5
3
5
3
3
p 6
3
3
3
4
p
GT
P=0
p == p=-1
p -
p == 3
-4
P 5
== 6
p
8
Figure 7: Extrapolated ouputs for the Rotating MNIST (above) and Bouncing Balls experiments. The models are optimised dependant on their output at level Pmin = -4 and Pmin = -3, respec- tively. One able to assess performance at extrapolated depths p + Pmin around these.
19

---

Pmin
MSE
ODEVAE moars 0.0093
Valard at (2020), maan. 0.0166
InImNet, Pmin = =-1 0.0140
prosoued prame-S, med, mman: 0.0192
0.230
pronesed, prone-1, comn, mears 0.0140
InImNet, Pmin =-2 0.0123
prososed, prine 2, comy, moars 0.0123
0.025
poooscd. pminn 3, comy, moant 0.0122
InImNet, Pmin = -3 0.0122
Vialard et al (2020)
0.0146
0.110
0.005
Fromc
Figure 8: Bouncing balls experiment. Left: reported MSE for proposed InImNet and state-of- the-art methods, results from other methods are taken from Yıldız al. (2019) and Vialard et al. (2020). Right: average time consumption per epoch.
Testing (omin
Training g Pmin
GT
0
p 11
p
p
Testing (Pmin
Training Pmin
GT
p =
p =
2
Testing (pmin
Training g (rmin
GT
= 0
p= -1
Figure 9: Bouncing balls experiment using a convolutional architecture; details of the architecture are given in Section G.3
20

---

F.5 ON THE APPROXIMATION OF THE INPUT GRADIENTS
Equation (21 outlines the approximation we implement allowing us to avoid high memory costs during training whilst needing to evaluate a growing number of nested Jacobians. The aforemen- tioned costs are associated with the recurrent computation of Vx Pt, x) which would require the computation of an i-th order gradient tree; this hold up is also discussed in $F.. While it is still possible use the exact solution-via automatic differentiation and gradient graphs-for some of the smaller problems with small Pinin. we use the approximation from Equation (2T) in all discrete architecture experiments with high performance success and with acceptable memory costs.
G MODEL HYPERPARAMETERS
G.1 ROTATING MNIST
We reproduce the experimental setting from Vialard et al. (2020); the experiments have been con- ducted in Tensorflow 2.6.0. The autoencoder's architecture, reimplemented in Tensorflow from the official pytorch code of Vialard et al. (2020), and parameters, including the batch size, replicate the same experimental set-up. We follow the discrete InImNet architecture described in Section 3.1 and we minimise the mean squared error loss between the ground truth and the outputs decoded at the layer Pmin using backpropagation and gradient descent based optimisation. To enable the time series prediction, we append the time value to the latent representation of the autoencoder. The multilayer perceptron models use dropout regularisation for every layer except the last. The hyperparameters of the rotating MNIST model are listed in Table All experiments have been run in a Google Colab GPU environment. Similar to Vialard et al. (2020), we denote as the inflation factor the size ratio between the intermediate and input MLP layers.
Hyperparameter
Value
Optimiser
Adam
Batch size
Initial learning rate
0.001
Epochs
500
Dropout value
0.3
Inflation factor Dimension of latent space
2
20
Learning rate schedule
0.5 [decaying*]
Table 2: Hyperparameters for the rotating MNIST experiment. (+) The learning rate decays expo- nentially steps of 30 epochs.
G.2 BOUNCING BALLS
The architecture of the autoencoder and experimental setting reimplements in Tensorflow 2.6.0 the one from the official code of Vialard et al. (2020), the rest of the considerations are identical to the ones given G. As in the previous section, we concatenate the time parameter with the latent representation of the first three images in the autoencoder to obtain time series imbedding. The hyperparameters of the bouncing balls model are given in Table 3
G.3 CONVOLUTIONAL EXPERIMENTS
In our Bouncing- Bouncing-BBall-wiiie plements the fully-convolutionaal autoencoder from Vialard et al. (2020) Tensorflow 2.6.0, with the exception of omitting the final layer's sigmoid (see the autoencoder description below). The dimensionality reduction is omitted as the input x and the outputs z(q: p, x) are 32 x 32 images. Following Vialard et al. (2020), autoencoder input, and add the fourth channel filled with the time value shaped 32 x 32. We obtain the prediction by taking the first channel of the autoencoder output. As in the previous experiments,
experiment, $F.. every InImNet layer reim-
concatenate the first three images into the three channels of the
21

---

Hyperparameter
Value
Optimiser
Adam
Batch size
Initial learning rate
0.001
Epochs
100
Dropout value
0.3
Inflation factor
Dimension of latent space 50
Learning rate schedule
0.5 [decaying*]
Table Hyperparameters for the bouncing balls experiment. (*) The learning rate decays exponen-
tially at steps of 30 epochs.
we optimise the mean squared error loss at the layer Pmin and use backpropagation to optimise the parameters of all InImNet layers.
We calculate the Jacobians by flattening the values of x and z(q; p. x) into vector of 32 32 x values. The overall structure of the autoencoder as implemented in Tensorflow Keras follows.
Encoder:
Conv2D (16, (5, 5), strides-2, padding- same' same"
2. Batcchooralizaaionn))
3. ()
4. Conv2D (32, (5, 5), strides=2, padding=' same"
5. Batchhorrallzzationn
6. ()LU()
7. Conv2D (64, (5, 5), strides-2, padding=' same')
8. ()
Decoder:
Conv2DTranspose (128, (3, 3), strides=2, padding=' same')
2. aatchNormalizatioon
3. ()LU()
4. Conv2DTranspose (64, (5, 5), strides-2, padding='sane same')
5. Batchhoormalizationn))
6. ()LU()
7. Conv2DTranspose (32, (5, 5), strides=2, padding=' same')
8. BatchNormallsationn
9. ()LU()
10. Conv2DTranspose (4, (5, 5), padding='same same') )
The hyperparameters for the described convolutional model are listed in Table
Hyperparameter
Value
Optimiser
Adam
Batch size
2
Learning rate
0.001
Epochs
20
Learning rate schedule
Constant
Table 4: Hyperparameters for the convolutional Bouncing balls experiment.
22


Reviews:
[{"review_id": "fS6zO5ccdZx", "reviewer": "Reviewer_Ynkf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper aims to tackle the problem of modeling a continuous time dynamic system, using a category of neural networks. This idea is part of a novel field of study, starting with Neural ODEs (Chen et al, 2018). Thus, this paper studies a very interesting and novel problem, in a way that is in tune with prior work on the same topic.\n\nStrengths:\n+ The mathematical formulation of the proposed model is solid. The continuous time behavior of InImNets is adequately analyzed, and derivations for the forward and backward steps of the model are included.\n+ The motivation behind the proposed model is also very interesting. Instead of solving a two-point boundary conditions problem (as in regular Neural ODEs), the authors solve two initial value problems for the same task. This allows the implementation of a simpler and more efficient scheme to train the network.\n+ The formulation of the problem also leads to links with optimal control theory. This is an exciting direction which, while not extensively explored in the paper, can lead to interesting future works.\n+ The experiments considered are adequate in order to provide a proof of concept for this architecture, when compared to prior work on the field.\n\nWeaknesses:\n-\tI believe that there are parts of the paper that could be reformulated to become simpler and easier for the reader to understand. In particular:\n  -\tSection 2.1 should appear before the introduction of the section right above (since it describes the overall form of the optimization problem the model tries to solve).\n  -\tEquations 21 and 22 describe an alternative scheme which is not actually used, and I believe they may be omitted without hampering the flow of that section.\n  -\tEquations 16 and 17 are more closely related to Section 3.2 rather than the one they are placed right now.\n-\tThe experimental section is not as clear as the rest of the paper. Firstly, Section 4.1 does not include any results, other than Figure 1 (I understand that the goal is primarily to provide a toy example as a motivation, but its placement makes the reader think that there are associated numerical results). Secondly, while Section 4 seems to imply that the model used was the discretized form of Algorithm 1, as described in Section 3, in the conclusion the authors state that they provide results for two models, a discrete and a continuous one. These two models should be better explained, both structurally and in which experiments they are used precisely.\n\nMinor comments/Typos:\n-\tIn the figures/tables in the experimental section, it would be nice to explicitly state the source of each architecture, in a way that matches the figure/caption (i.e. using the name of the architecture directly).\n-\tI believe the notation would be better if $z(t;p,x)$ for the InImNets was $z(p,x)$ (or something similar, given that while they do depend on the variable $t$, it is not considered to be the primary variable of the model.\n-\tTypos:\n  -\tLine after equation 2: “a external” -> “an external”.\n  -\tPage 3, overview of the paper: “InImnets” -> “InImNets”.\n  -\tPage 3, at the very end: “satisfies” -> “satisfy”.\n  -\tPage 5, after equation 13: “accessible optimize” -> “accessible to optimize”.\n  -\tPage 5, second to last paragraph before 3.1: missing word after “demonstrating high”.\n\nQuestions:\n-\tAs mentioned above, I would be grateful if the authors could provide some additional explanations on the models used in their experiments.\n", "summary_of_the_paper": "This paper proposes a class of networks called Invariant Imbedding Networks (InImNets). This type of networks aims to model an ODE as a pair of initial value problems, with the variable being the starting point of the problem. This allows for a more efficient implementation of a model of a dynamic system. The proposed model is analyzed both theoretically and experimentally.", "summary_of_the_review": "Overall, this is a good paper which provides an exciting novel architecture, with good theory and some simple experiments to justify the quality of this architecture, with comparisons to related work. My current issues lie with the clarity of writing and the explanations of the models used in the experiments. Given that these issues can be easily resolved during the rebuttal process, I lean towards accepting this work, pending some extra explanation provided by the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZJy4jIPwEM", "reviewer": "Reviewer_W69g", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of the paper is that it introduces an alternative view to the neural ODE, by reducing the non-linear, vector-valued optimal control problem to a system of forward-facing initial value problems. They introduce the invariant imbedding method from mathematics community to achieve this. They also show that the resulting algorithm can perform well compared to state-of-the-art methods.\n\nThere are two weaknesses of the paper. First, the presentation of the paper can be improved. I believe the paper should be written in a self-contained manner such that for the readers who have not seen the Neural ODE and optimal control view can still easily follow the paper, which is not the case in the current form. For example, the authors introduce and then start to derive the properties about the adjoint $\\Lambda(p,x)$ without ever defining or explaining its role. Because of this, Theorem 2 is hard to follow without knowing the exact definition of $\\Lambda(p,x)$. One possibility is to provide some sort of background review in the Appendix. Second, I think another weakness of the paper is that it does not seem to provide a rigorous treatment. Even though there are Theorem 1-3, I could not find any technical assumptions. It is weird to see Theorems stated without any assumptions on the model, and that makes the readers to wonder the rigor of the paper.", "summary_of_the_paper": "In recent years, continuous depth neural networks, such as Neural ODEs have helped understanding of ResNet in terms of non-linear vector-valued optimal control problems. The authors of the paper show that it is possible to reduce the non-linear, vector-valued optimal control problem to a system of forward-facing initial value problems, providing an alternative perspective on the problem. They demonstrate that this approach can be used in creating discrete and continuous numerical DNN schemes. ", "summary_of_the_review": "(1) I suggest the authors to include a section discussing the background of Neural ODE, adjoint method etc. in the Appendix.\n\n(2) Technical assumptions should be added, and the proofs should be made rigorous (if possible) throughout the paper. Otherwise, the paper stands as a non-rigorous treatment of the subject.\n\n(3) On page 7, '3' and on page 9 'Invariant Imbedding'  are typos.\n\n(4) In the caption of Table 1, the stop somehow appears in the next line.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NaZHr7pEdkq", "reviewer": "Reviewer_nSYs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies a very important and interesting topic, continuous depth neural networks. It may be one way to reach the framework of Explainable AI. I appreciate the authors providing a very detailed derivation and implementation for the InImNet architecture, which enables readers to easily reproduce the proposed method.\n\nHowever, I still have some concerns, which are listed as below:\n\n1. According to the current experiment results, I am not sure what is the superiority of the proposed InImNet compared to existing approaches. For example, in Table 1, both the InImNet and other existing models can achieve similar performance. The authors may want to give a detailed explanation about such superiority.\n\n2. For the \"rotating MNIST\" experiments, only the MLP model is used for implementing InImNet. I would be happy to see InImNet being implemented with other types of models, for example, a small CNN. Besides, can the authors apply InImNet to some large-scale datasets like CIFAR-10?\n\n3. According to the implementation of InImNet, it seems would take more calculation consumption for training the InImNet. Therefore, it is necessary to compare the time used for training InImNet and other models. Further, I would like to see the comparison of performance for different models under the same training time cost.", "summary_of_the_paper": "This paper proposes a new type of continuous depth neural network architecture, named Invariant Imbedding Networks (InImNets). InImNets can imbed a given $\\bf x$ input to different time steps (time positions) and efficiently calculate the solutions of the dynamic system (starting from different time positions) simultaneously. The authors also give a practical implementation for training the InImNets architecture. The proposed architecture is then applied to several benchmark problems for continuous neural networks for verifying its effectiveness.", "summary_of_the_review": "Based on the previous comments, I tend to recommend accepting this paper. But I would like the authors to resolve my concerns first.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CdiH8v5Rgnb", "reviewer": "Reviewer_4Mn5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points:**\nThe idea is interesting and well motivated. The paper provides the derivations for both the forward pass and backward pass for the gradients w.r.t. a new parameter p. The methods section builds the model from the ground-up and it is easy to follow.\n\n**Weak points:**\n\nThe results section is a bit underwhelming. The results demonstrate the bare minimum that a) the approach works, matches the performance of other models b) how the performance changes when we vary p_max. \n\nIn the introduction, the paper puts  a lot of emphasis on understanding and interpreting the network. With this motivation, I expected to see more analysis on the learned ODE and the relation between the learned trajectories z(…) and the trajectories in the data space (for example. similarly to figure 1); or the analysis what depth of the network is required to learn the bounce off the walls in the bouncing ball experiment. Perhaps instead network interpretation, the authors could highlight and experiment with other benefits of the model: 1) we can take the output of the network after every “depth” and decode it into a meaningful output 2) we can do smooth interpolation between the input and the output. \n\n**Additional questions:**\nWhat happens if we go beyond p_max in the experiments?\n\nDo you actually see high memory cost to justify using the finite difference method that is as accurate as the chosen delta? All experiments use MLP with a few layers\n\nCan we just vary the t_max rather than conditioning on p?\n\nMore details on the experiments need to be provided in the results section as well. Specifically, 1) How do you choose q? 2) In the results section, what does p_max represent? Why do you vary p_max, even though the model integrates on the interval [p, q]. Should it be p_min?\n", "summary_of_the_paper": "The paper extends the Neural ODE model by introducing the dependency on the extra parameter p that can be interpreted as a control parameter or the “depth” of the network. The models allows to decode the network output by varying this parameter. \n", "summary_of_the_review": "The paper has an interesting idea to condition the Neural ODE on the parameter p that can be interpreted as the “depth” of the network. The Results section does not explore the new architecture enough and provides only the basic results to demonstrate that the model works. I suggest the acceptance of the paper, but recommend the authors to expand the results section. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors presents an alternative view of Neural ODEs, offering a novel understanding of depth in neural networks. The reviewers were overall impressed by the novelty and potential for insights this work brings. There was some disappointment that the empirical results were not stronger (both in terms of pure performance and computational cost) and that it wasn't clear how the theoretical insights into depth actually translated into a practical insight. Nevertheless, I agree with the reviewers that this is a good submission and would I think make for an interesting addition to the conference programme.
Decision: Accept (Spotlight)