Go with the flow: Adaptive control for Neural ODEs
Authors: Mathieu Chalvidal
Matthew Ricci
Rufin VanRullen
Thomas Serre
Venue: ICLR 2021 Poster (2021)
Keywords: Neural ODEs
Optimal Control Theory
Hypernetworks
Normalizing flows

Abstract: Despite their elegant formulation and lightweight memory cost, neural ordinary differential equations (NODEs) suffer from known representational limitations. In particular, the single flow learned by NODEs cannot express all homeomorphisms from a given data space to itself, and their static weight parameterization restricts the type of functions they can learn compared to discrete architectures with layer-dependent weights. Here, we describe a new module called neurally-controlled ODE (N-CODE) designed to improve the expressivity of NODEs. The parameters of N-CODE modules are dynamic variables governed by a trainable map from initial or current activation state, resulting in forms of open-loop and closed-loop control, respectively. A single module is sufficient for learning a distribution on non-autonomous flows that adaptively drive neural representations. We provide theoretical and empirical evidence that N-CODE circumvents limitations of previous NODEs models and show how increased model expressivity manifests in several supervised and unsupervised learning problems. These favorable empirical results indicate the potential of using data- and activity-dependent plasticity in neural networks across numerous domains.

Published as a conference paper at ICLR 2021
Go WITH THE FLOW:
ADAPTIVE CONTROL FOR NEURAL ODES
Mathieu Chalvidal Matthew Ricci', Rufin VanRullen 13, Thomas Serre
AAttiiiiil and Natural Intelligence Toulouse Institute, Universite de Toulouse, France CCamnyy Institute for Brain Science, Dpt. of Cognitive Linguistic & Psychological Sciences Brown University, Providence, RI 02912
"Centre de Recherche Cerveau & Cognition CNRS, Université de Toulouse "Data Science Initiative, Brown University, Providence, RI 02912 (mathieu chalvid, d, mgr, thomas serre berre)@brood @erown..owweeooo.. edu rufin. vanrul manrullen@cnrs.f
ABSTRACT
Despite their elegant formulation and lightweight memory cost, neural ordinary differential equations (NODEs) suffer from known representational limitations. In particular, the single flow learned by NODEs cannot express all homeomor- phisms from a given data space to itself, and their static weight parameterization restricts the type of functions they can learn compared to discrete architectures with layer-dependent weights. Here, we describe a new module called neurally- controlled ODE (N-CODE) designed to improve the expressivity of NODEs. The parameters of N-CODE modules are dynamic variables governed by a trainable map from initial or current activation state, resulting in forms of open-loop and closed-loop control, respectively. A single module is sufficient for learning dis- tribution non-autonomous flows that adaptively drive neural representations. We provide theoretical and empirical evidence that N-CODE circumvents limi- tations of previous NODEs models and show how increased model expressivity manifests in several supervised and unsupervised learning problems. These favor- able empirical results indicate the potential of using data- and deeedty- plasticity in neural networks across numerous domains.
INTRODUCTION
The interpretation of artificial neural networks as continuous-time dynamical systems has led to both theoretical and practical advances in representation learning. According to this interpretation, the separate layers of a deep neural network are understood to discretization of a continuous-ting operator so that, in effect, the net is infinitely deep. One important class of continuous aatin models, neural ordinary differential equations (NODEs) (Chen et al., 2018), have found natural applications in generative variational inference (Grathwohl et al., 2019) and physical modeling (Köhler et al., 2019: Ruthotto et al., 2020) because of their ability to take advantage of black-box differential equation solvers and correspondence to dynamical systems in nature.
Nevertheless, NODES suffer from known representational limitations, which researchers have tried to alleviate either by lifting the NODE activation space to higher dimensions or by allowing the transition operator to change in time, making the system non- non-autonomous (Dupont et al., 2019). For example, Zhang et al. (2020) showed that NODEs can arbitrarily approximate maps from Rd to R NODE dynamics operate with an additional time dimension in Rd+1 and the system is affixed with additional linear layer. The same authors showed that NODEs could approximate homeo- morphisms from Rd to itself if the dynamics were lifted to 2d Yet, the set of homeomorphisms from 4d to itself is in fact quite conservative function space from the perspective of represen- tation learning, since these mappings preserve topological invariants of the data space, preventing them from "disentangling" data classes like those of the annulus data in Fig. 1 (lower left panel). In general, much remains to be understood about the continuous-ting framework and its expressive capabilities.

---

Published as a conference paper at ICLR 2021
In this paper, propose new approach that ODEs (N-CODE) designed
call neurally-controll to increase the expressivity of continuous-time neu- ral nets by using tools from control theory. Whereas previous continuous-time methods learn single, time-varying vector field for the whole input space, our system learns a family of vector fields parameter- ized by data. We do so by mapping the input space collection of control weights which interact with neural activity to optimally steer model dynamics. The implications of this new formulation are critical for model expressivity.
NODE
NCODE
In partieular, the transformation of the input space no longer constrained to be homeomorphism, since the flows associated with each datum are specifi- cally adapted to that point. Consequently, tem can easily "tear" apart the two annulus classes in Fig. (lower right panel) without directly lift- ing the data space to higher dimension. Moreover, when control weights are allowed to vary in time, they can play the role of fast, plastic synapses which can adapt to dynamic model states and inputs.
sys-
Figure I: Vector fields for continuous- time neu- ral networks. Integral curves with arrows show the trajectories of data points under the influence of the network. Top left: Standard NODEs learn a single time-independent flow (in black) that must account for the whole data space. Top right: N- CODE learns family of vector fields (red vs yel- low blue), enabling the system to flexibly ad- just the trajectory for every data point. Bottom left: Trained NODE trajectories of initial values in data set of concentric annuli, colored green and yellow. The NODE transformation is home- omorphism
The rest of the paper proceeds as follows. First, we will lay out the background for N-CODE and its technical formulation. Then, we will demon- strate its efficacy for supervised and unsupervised learning. In the supervised case, we show how N- CODE can classify data by learning to bifurcate its dynamics along class boundaries as well as memo- rize high-dimensional patterns in real-time using fast synapses. Then, show how the flows learned by N-CODE can be used as latent representations in an unsupervised autoencoder, improving image genera- tion over base model.
the data space and cannot separate the classes as a result. Colored points are the ini- tial state of the dynamics, and black points are the final state. Bottom right: Corresponding flows for N-CODE which easily separate the classes. Transiting from the inner to outer annulus effects bifurcation which lincarly separates the data.
2 BACKGROUND
Neural ODEs (NODEs) (Chen et al., 2018) are dynamical systems of the form
dæ [ax 0,1),
where X' is a space of features, 0 is collection of learnable parameters, and f: X'Ox + X is an equation of motion which we take to be differentiable on its whole domain. f defines flow, ie. a triple (X', 2o) with & X + X' defined by
T
2 = x + / [(x(()) 0,t)dt
(2)
which relates an initial point =0)) to an orbit of points (zx(1) = (o(x()), (), IR.. For fixed T, the map x + (2 T) a homeomorphism from to itelf parametrized by 0.
Several properties of such flows make them appealing for machine learning. For example, the ODEs that govern such such flows can be solved with off-the shelf solvers and they can potentially model data irregularly sampled in time. Moreover, such flows are reversible maps by construction whose inverse is just the system integrated backward in time, (o(., t) = ( -t). This property en- ables depth-constant memory cost of training thanks to the adjoint sensitivity method (Pontryagin Lev Semyonovich Boltyanskii V G & F, 1962) and the modeling of continuous-tin generative normalizing flow algorithms (?).
2

---

Published as a conference paper at ICLR 2021
Interestingly, discretizing Eq. yields the recursive formulation of residual network (He et al., 2015) with a single residual operator fo:
T (a(
Resser (ers))e((=(0))77))))))) (z)(0,,)) (0) +
In this sense, NODEs with time-independent (autonomous) equation of motion, f. are the infinitely-deep limit of weight-tied residual networks. Relying on the fact that every non- autonomous dynamical system with state E is equivalent to autonomous system the extended state (æ: t) E Rd-1, Eq. 2 can also be used to model general, weight-untied residual net- works. However it remains unclear how dependence of f in time should be modeled in practice and how their dynamics relate to their discrete counterparts with weights evolving freely across blocks through gradient descent.
3 N-CODE: LEARNING TO CONTROL
FLOWSS
General formulation The main idea of N-CODE is to consider the parameters, ((t), in Eq. as control variables for the dynamical state, xt)). Model dynamics are then governed by coupled system of equations on the extended state xtt) = (a<t), (t)). The initial value of the control weights, 0(0), is given by mapping 7 X O Throughout,
assume that the initial time point is 0. The full trajectory of control weights, ((t), is then output by a controller, g. given by another differentiable equation of motion g θ X' x E - θ with initial condition ((z)). Given an initial point, x (0), we can solve the initial value problem (IVP)
dz dt
{
{
dæ d0 di di (x()), (())) = (#o. (xx))
h(z,t)
= (f(æ, 0, 0,1),((0,2,)
(4)
=(0) Z0
where h = (). We may think of g and y as a controller of the dynamical system with equa- tion of motion, f. We model g and y as neural networks parameterized by μ E R and use gradient descent techniques where the gradient can be computed by solving an adjoint prob- lem (Pontryagin Lev Semyonovich : Boltyan- skii V G & F, 1962) that we describe in the next section. Our goal here is to use the meta- parameterization in the space θ to create richer dynamic behavior for a given f than directly optimizing fixed weights O.
N-CODE module
300
(xx,0tt)
(x())
((L)
7(20)
g(0, xt,)
n(1)
Gradient descent
-ooe-posss If f and are continuously and 0 and con-
Hett), (1)) 1)
differentiable with respect tinuous with respect to i, then, for all initial conditions (a=))), (()),, there exists unique solution z for Eq. 4 by the Cauchy-Lipschitz theorem. This result leads to the existence and uniqueness of the augmented flow (X x O, R. p with T (X' x +) x RR X' x Moreover, considering the restriction of such flow on X. we are now endowed with univer- sal approximator for at least the set of homeo- morphisms on X' given that this restriction con- stitutes aonaautoomoous system. We discuss now how and y affect the evolution the variable x (t), exhibiting two forms of control and noting how they relate previous extensions of NODEs.
Figure 2: Diagram of a general N-CODE module: Given initial state æ0)), the module consists an augmented dynamical system that couples activity state and weights 0 over time (red arrows). A mapping infers initial control weights Os defining initial flow (open-loop control). This flow can potentially evolve time as might be driven by a feedhack signal from æ (closed-loop. dotted line). This meta- meta- patttcaaaaaatzatiion of f can be trained with gradient descent by solving an augmented adjoint sensitivity system (blue arrows).
3

---

Published as a conference paper at ICLR 2021
3.1 OPEN AND CLOSED-LOOP CONTROLLERS
If the controller outputs control weights function of the current state, a<t), then we say it is closed-loop controller. Otherwise, it is an open-loop controller.
First, we consider the effect of using only the mapping ? in Eq. 4 as a con- troller. Here, maps the input space A' to θ so that f is conditioned on x(0) but not necessarily on x(t) for > In other words, each initial value (0) evolves according to its own learned flow (X, R, (=(=()))) This allows for trajectories to evolve more freely than within single flow that must account for the whole data distribution and resolves the problem
Open-loop control:
non-intersecting orbits (see Figure 4). Recently, (Massaroli et al., 2020b) proposed similar form of data-conditioned open- loop control with extended state (t), x0))). This a version of our method which y is of the form ((x) (0) [C id] with C' constant vector, id is the identity function, and denotes concatenation. Our open-loop formulation makes an architectural distinction between controller and dynamics and is consequently generalizable to the following closed-loop formulation.
Closed-loop control: Defining a differentiable mapping, g. which outputs the time-dependent control weights ((1) given the state the vari- able x(t) yields non-autonomous system on N' (see Fig. 3). This can be seen as a specific dimension augmentation technique where addi- tional variables correspond to the parameters O. However, contrary appending extra dimen- sions which does not change the autonomous property of the system, this augmentation re- sults in module describing transformation L (zo. Note that this formulation generalizes the functional parame- terization proposed in recent non-autonomous NODES systems (Choromanski et al., 2020; Massaroli et al., 2020b; ?), since the evolution of O(t) depends case of classical control theory, we hypothe- sized that the use of dynamic control weights would be of particular use in reacting to a non- stationary stimulus. We evaluate this hypothe- sis in section. 5.3.
ab [ubett - 1/351t 6t00
Phasse plot
vanatie 1S1 TRaNctioN
time-varying
d2
dx
11(1
x.
x.
d.t.
dt
=(t). Much like in the
④:
1 xr 0) 3
0
Figure 3: Examplc of dynamical control augmenta- tion: The Van der Pol oscillator: As second order differential equation, the dynamics cannot be approx- imated by a single I-dimensional NODE with a con- stant control 0 (degenerate solution in red). However, if the dynamics are decomposed into planar system with a dynamic control variable (t). then the parameter μ can adapted to fit a particular oscillatory regime (in black). This is particular form of augmentation dis- cussed in (Dupont al. (2019), 9); Norcliffe al. (2020)) that showcases the henefit of using additional variables as evolving parameters of the dynamical system.
The expressivity N-CODE compared to other continuous-time neural networks is encapsu- lated the following proposition. The re- sult, proven in appendix, shows that both open- loop and closed-loop control systems overcome NODEs' expressivity constraint with two dis- tinct strategies, cata-conditioning and state- space augmentation.
Proposition 1 Theree exists a transformation R Rd which can be expressed by N-CODE but not by NODEs. In particular, homeomorphism.
3.2 TRAINING
Loss function:
Dynamics are evaluated according to a generalized loss function that integrates a cost over some interval [0,T]:
1(x) = ((=(1), = / (((x(1)))0)) ()
T
(5)
4

---

Published as a conference paper at ICLR 2021
NODE
N-CODE (opas-loop)
N-CODE (cleec--cop)
Figure 4: Trajectories over time for three types of continuous-time neural networks learning the 1-dimensional reflection map (0(=)) z. Left: (NODE) Irrepecttvve of the form of f. NODE module cannot learn such function trajectories cannot intersect (Dupont al., 2019). Middle: (Open-loop N-CODE). The model able to learn family of vector fields by controlling single parameter of conditioned on z(0). Right: (Closed-low loop N-CODE) With a fixed initialization, z0)), the controller model still learns a deformation of the vector field to learn Note that the vector field is time-varying in this case, contrary to the two others versions.
The loss in Eq. is more general than in Chen et al. (2018) since can be any Lebesgue-measuren function of both states, æ, and control parameters, 9. In particular, this includes penalties discrete time points or over the entire trajectory (Massaroli et al., 2020b) but also regularizations on the weights activations over the entire trajectory rather than the final state 2T)) of the system.
In order to estimate a control function ((t) that is optimal with respect to Eq. 5, we invoke Pon- tryagin's 's maximum principle (PMP) (Pontryagin Lev Semyonovich Boltyanskii V G & F, 1962), which only requires mild assumptions on the functional control space θ and applies to functions f that are non-smooth in O. The PMP gives necessary conditions on (tt) optimality via the aug- mented adjoint variable a(t). This quantity is the Jacobian of l with respect to both æ(t) and O(t). In the case of (tt) being differentiable with respect the meta petamearssssers μt, solving for the aug- mented adjoint state (t) as in Chen et al. (2018) allows us to compute the gradient of the loss with respect to l thanks to Theorem 1.
Theorem 1 Augmented adjoint method: Given the IVP of equation 4 and for e defined in equa- tion 5, we have:
da
Oh de
Øl 0p
Oh at))
-aa"
dt, such that satisfies
dl
Oz
(6)
0 0p
a(T)
af
0f
where dz
Oh
is the Jacobian of h with respect to z:
ah
00
8z
0g
80
In practice we compute the Jacobian for this augmented dynamics with open source automatic differentiation libraries using Pytorch (Paszke al., 2019), enabling seamless integration of N- CODE modules in bigger architectures. We show an example of such modules in section B of Appendix.
EXPERIMENTS
4.1 REFLECTION AND CONCENTRIC ANNULI
We introduce our approach with the 1- and 2-dimensional problems discussed in (Dupont et al., 2019; Massaroli et al., 2020b), consisting of learning either the reflection map ((x)) -2 linear classification boundary on a data-set of concentric annuli. Earlier results have shown that vanilla NODEs cannot learn these functions since NODEs preserve the topology of the data space, notably its linking number, leading to unstable and complex flows for entangled classes. However, both the open and closed loop formulations of N-CODE easily fit these functions, as shown in Figs 4 and 5. The adaptive parameterization allows N-CODE to learn separate vector fields for each input, allowing for simpler paths that ease model convergence. Informally, we may think of the classification decisions as being mediated by bifurcation parameterized by the data space. Transiting across the true classification boundary switches the vector field from one pushing points to one side of the classification boundary the other. Moreover, the construction of y and neural network guarantees a smooth variation of the flow with respect to the system state, which can potentially provide interesting regularization properties the family learned by the model.
5

---

Published as a conference paper at ICLR 2021
0.6
000
N-CODE (c) N-CODE (o)
0.5
0.4
NODE
0.3
0.2
0.1
0.0
1000
1500 2000
Figure 5: The concentric annuli problem. classifier must separate two classes R a central disk and an annulus that encircles it Left: Soft decision boundaries (blue to white) for NODE (First). N-CODE open-loop (Second) and closed-loop (First) models. Right: Training curve for the three models.
4.2 SUPERVISED IMAGE CLASSIFICATION
We now explore different data-dependent controls for continuous-time models classification task. We tested the open-loop version of N-CODE against augmented NODEs (Dupont al., 2019) and the data-control model of (Massaroli et al., 2020b). We per- form this experiment on the MNIST and CIFAR-10 image datasets. Our model consists of the dynamical system with equation of motion f expressed
image
single convolutional block (1 x 1 + 3 x 3 + 1 x 1) with 50 channels. The control weights O E R real-valued vector of K concatenated parameters instantiating the convolution kernels.
The 10-class prediction vector is obtained as linear transformation of the final state x(T") of the dynam- ies with softmax activation. We aimed at simplicity by defining the mapping y as a two-layer perceptron with 10 hidden units. We varied how y affects 0 by defining different linear heads for each convolution the block. (see C.2 of Appendix). We did not perform any fine-tuning, although we did augment the state dimension, akin to ANODE, by increasing the channel dimension by 10, which helped stabilize training. We show that the open-loop control outperforms competing models. Results are shown in Table. I with further details in Section C.2 of Appendix.
Model
MNIST
CIFAR-I0 10
ANODE
98.2%
61.2%
NODE DC
98.5%
62.0%
N-CODE (open)
99.2%
74.1% 1
Table 1: Classification accuracy MNIST and CIFAR1O test sets for NODE, NODE with data- dependant flow (NODE DC) and open-loop N- CODE (ours)
4.3 REAL-TIME PATTERN MEMORIZATION
To validate our intuition that closed-loop control is particularly valuable on problems involving flex- ible adaptation to non-stationary stimuli and to distinguish our system from related data-conditioned methods, we trained N-CODE on a dynamic, few-shot memorization task (Miconi et al., 2018). Here, the model is cued to quickly memorize sets of sequentially presented n-dimensiinal binary patterns and to reconstruct one of these patterns when exposed to degraded version. For each presentation episode, we exposed the model sequentially to m = 2, 5, or 10 randomly generated n-length (n 100, 1000) bit patterns with 1-1, 1) -valued bits. Then, we presented degraded ver- sion of one of the previous patterns in which half of the bits were zeroed, and we tasked the system with reconstructing this degraded pattern via the dynamic state, ar()). This is difficult task since it requires the memorization of several high-dimensional data in real-time. Miconi et al. (2018) have shown that, although recurrent networks are theoretically able to solve this task, they struggle dur- ing learning, while models storing transient information in dynamic synapses tend to perform better. Following this result, we evaluated the performance of a closed-loop N-CODE model which the output of the control function takes the form of dynamic synapses (Eq. 7). Here, the control pa- rameters are a matrix of dynamic weights, (tt E Rm which are governed by the outer product @ xx.
da: de
dt dt = (p (0x) μ Q xx²) (x) (())))((0) = (a Ov)
where a is matrix of learned parameters, O, is a learned initial condition, p is an element-wise sigmoidal non-linearity, @ is element-wise multiplication, and Istm is the presented stimulus. We

---

Published as a conference paper at ICLR 2021
N-bis pettern memorization
Input
Input
Input
Query
Error rate
to
z(t)
SOOE 3000
dl g(0.z)
GOERAN 1500
(6,00)
((t)
Episodes
Figure 6: Left: Diagram an episode the few-shot memorization task Miconi al. (2018). Three, 5-Jength -1, 1-value hit patterns (n 5 here for visualization) are presented to the system at regular intervals. At query time, tg.. a degraded version of one of the patterns is presented and the task of the model is to complete this pattern. Right: Performance of closed-loop N-CODE, LSTM, continuous-tima ODE-RNN Rubanova al. (2019) and the data-conditioned NODE of Massaroli et al. (2020b) averaged over 3 runs. All models besides N-CODE plateau high values (chance .25). N-CODE, on the other hand, learns immediately and with extremely high accuracy across all v., (see Appendix)
have suppressed for concision. We ran this experiment on the closed-loop version of N-CODE as well as an LSTM, continuous-time ODE-RNN (Rubanova et al., 2019) and the data-controlled model of Massaroli et (2020b). All systems had the same number of learnable parameters, except for the LSTM which was much larger. We found that all considered models, except closed-loop N- CODE, struggled to learn this task (Fig. 6). Note that chance is .25 since half the bits are degraded. A discrete LSTM endowed with a much larger hidden state of 5000 neurons learns marginally better than chance level. ODE-RNN and data control versions platcau at 17.5% error rate. For both n tested, N-CODE learned the problem not only much better, but strikingly faster.
4.4 UNSUPERVISED LEA.NING IMAGE AUTOENCODING WITH CONTROLLED FLOW
Finally, we use our control approach in an autoencoding model for low-dimensional representation learning in order idea is to use the solution to system of linear ODEs to encode the model's latent representations. First, images from L E I. are mapped by an encoder to a space of couplings, 0, by
if merely adding N-CODE to simpler model can improve performance. Our
7i
7 (u) 0 Rmxm.
The latent code is taken as the vector x x(')) acquired by solving the autonomous differential system presented in Eq. 9 from T' starting from the initial condition (Do. (u)) with æo (0,
dæ d0 (0xz 0)
dt dt (z))))
(Do. 7/p (u))
This model, which we call autoN-CODE, can be interpreted as a hybrid version of autoencoders and generative normalizing flows, where the latent generative flow is data-dependent and parameterized by the encoder output.
We found that a simple linear system was already more expressive than a discrete linear layer in shaping the latent representation. Since the output dimension of the encoder grows quadratically with the desired latent code dimension of the bottleneck, we adopted a sparse prediction strategy, where as few as two elements of each row of were non-7ero, making our model match exactly the number of parameters of a VAE with same architecture. We believe that the increase in representa- tional power over a vanilla autoencoder, as shown by improvements in Fréchet Inception Distance (FID; Heusel et al. (2017)) and more disentangled latent representations (see A.7), despite these simplifying assumptions, shows the potential this approach for representation learning (Fig. 2). Training details can be found in Sec. C.4.
Here we measure the marginal improvement acquired by adding latent flows to a baseline VAE. In order endow our deterministic model with the ability generate image samples, perform,
7

---

Published as a conference paper at ICLR 2021
similarly to (Ghosh et al., 2020), a post hoc density estimation of the distribution of the latent tra- jectories' final state. Namely, we employ gaussian mixture model fit by pnpectation-maximizee and explore the effect of increasing the number of components in the mixture. We report in Table 2, contrary to Ghosh et al. (2020), substantial decrease in FID of our sampled images to the image test set with increasing number of components in the mixture (see Fig. 7 for latent evolution and Fig. 8 for sampled examples).
4.5
IMAGE GENERATION
FID (L)
CIFAR-10
CelebA-(64)
Considering the identical ar- chitecture of the encoder and decoder to a vanilla model, these results suggest that the optimal encoding control for- mulation produced structural change in the latent manifold or- ganization. (See section C.7 in Appendix). Finally, we note that this model offers several original ficients, or adding a regularization on the dynamic evolution as in (Finlay et al., 2020), which we leave for future work.
VAET (Kingma & Welling, 2013)
105.45
68.07
Auto-encoder (100 components)
73.24
63.11
Auto-encoder (1000 components)
55.55
55.60
AutoNCODE (100 components)
34.90
55.27
AutoNCODE (1000 components) 24.19
52.47
Table 2:
Frechet Inception Distance (FID) for several recent architec- tures (lower is better). "Components" are the numhers of components used in the gaussian mixture estimation of the latent distribution.
possibilities explore, such as sampling from the control coef-
Figure 7: Left to right: Temporal evolutions of an image reconstruction along controlled orbits of AutoN- CODE for representative CIFAR-10 and CelabA test images. Last column: ground truth image.
5 RELATED WORK
Neural ODEs This work builds on the recent development of Neural ODEs, which have shown great promise for building generative normalizing flows (Grathwohl et al., 2019) used for modeling molecular interactions (Köhler al., 2019) in mean field games (Ruthotto et al., 2019). More recent work has focused on the topological properties of these models (Dupont et al., 2019; Zhang al.. 2020), introducing time-dependent parameterization (Zhang et al., 2019b; Massaroli et al., 2020b; Choromanski et al., 2020), developing novel regularization methods using optimal trans- port or stochastic perturbations (Finlay al., 2020; Oganesyan et al., 2020) and adapting them to stochastic differential equations (Tzen& & Raginsky, 2019).
Optimal Control Several authors have interpreted deep learning optimization as an optimal con- trol problem (Li et al., 2017; Benning et al., 2019; Liu & Theodorou, 2019; Zhang et al., 2019a; Seidman et al., 2020) providing strong error control and yielding convergence results for alternative algorithms to standard baaeed learning methods. Of particular interest is the mean-field opti- mal control formulation of (E et al., 2018), which notes the dependence of a unique control variable governing the network dynamics on the whole data population.
8

---

Published as a conference paper at ICLR 2021
Hypernetworks- Finally, system is related network architectures with adaptive weights such as hypereetwrks (Ha et al., 2017), dynamic filters (Jia et al., 2016) and spatial transformers (Jader- berg et al., 2015). Though not explicitly formulated neural optimal control, these approaches effectively implement form of modulation of neural network activity as a function of input data and activation state, resulting in compact and expressive models. These methods demonstrate, in our opinion, the significant untapped potential value in developing dynamically controlled modules for deep learning.
RECONSTRUCTIONS
SAMPLES
VAE
VAB
AE
AE
Figure Left Reconstructions of random test images from CelebA for different autoencoding models. Right: Random samples from the latent space. Deterministic models are fit with 100- components gaussian mixture model.
6 CONCLUSION
In this work, we have presented an original control formulation for continuous matinnuuus-tim neural feature transformations. We have shown that is possible dynamically shape the trajectories of the trans- formation module applied to the data by augmenting the network with a trained control mechanism, and further demonstrated that this can be applied the context of supervised and unsupervised representation learning. In future work, we would like to investigate the robustness and generaliza- tion properties such controlled models as well as their similarities with fast-synaptic modulation systems observed in neuroscience, and test this natural applications such as recurrent neural net- works and robotics. An additional avenue for further research is the connection between our system and the theory of bifurcations in dynamical systems and neuroscience.
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
This work was funded by the ANR-3IA 31A Artificial and Natural Intelligence Toulouse Institute (ANR- 19-PI3A-000).
Additional support provided by ONR (N00014-19-1-2029) NSF (IIS-1912280, OSCI-DEEP ANR grant WW-R-19-NEUC-OON and the Center for Computation and Visualization (CCV). We ac- knowledge the Cloud TPU hardware resources that Google made available via the TensorFlow Re- search Cloud (TFRC) program as well as computing hardware supported by NIH Office of the Director (S100D025181).

---

Published as a conference paper at ICLR 2021
REFERENCES
Martin Benning, Elena Celledoni, Matthias J. Ehrhardt, Brynjulf Owren, and Carola-Bibiane Schönlieh. Deep learning as optimal control problems: models and numerical methods, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In European Conference Computer Vision, 2018.
Chang, L.. Wang, Meng, Xiang, and C. Pan. Deep adaptive image clustering. In 2017 IEEE International Conference Computer Vision (ICCV), 2017.
Ricky Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, 2018.
Krzysztof Choromanski, Jared Quincy Davis, Valerii Likhosherstov, Xingyou Song, Jean-Jacques Slotine, Jacob Varley, Honglak Lee, Adrian Weller, and Vikas Sindhwani. An ode to an ode, 2020.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Sociery. Series B (Methodological), 1977.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In Advances in Neural Information Processing Systems, 2019.
Weinan E, Jiequn Han, and Qianxiao Li. A mean-field optimal control formulation of deep learning, 2018.
Chris Finlay, Jömn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ode: the world of jacobian and kinetic regularization, 2020.
Ghosh, M. S. M. Sajjadi*, A. Vergari, M. Black, and Schölkopf. From variational to de- terministic autoencoders. In 8th International Conference on Learning Representations (ICLR), 2020.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. International Conference on Learning Representations, 2019.
David Ha, Andrew Dai, and Quoe V. Le. Hypernetorrss. 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial transformer networks. In Advances in Neural Information Processing Systems, 2015.
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Gool. Dynamic filter networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. 2013.
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: sampling configurations for multi- body systems with symmetric energies, 2019.
10

---

Published as a conference paper at ICLR 2021
Qianxiao Li, Long Chen, Cheng Tai, and Weinan E. Maximum principle based algorithms for deep learning, 2017.
Guan-Homg Liu and Evangelos A. Theodorou. Deep learning theory review: An optimal control and dynamical systems perspective, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Stuart P. Lloyd. Least squares quantization in pem. IEEE Transactions on Information Theory, 1982.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008.
Stefano Massaroli, Michael Poli, Michelangelo Bin, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Stable neural flows, 2020a.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama, Dissecting neural odes, 2020b.
Thomas Miconi, Kenneth Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural networks with backpropagation. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Andrew Ng. Sparse autoencoders, 2011.
Alexander Norcliffe, Cristian Bodnar, Ben Day. Nikola Simidjievski, and Pietro Liò. On second order behaviour in augmented neural odes, 2020.
Viktor Oganesyan, Alexandra Volokhova, and Dmitry Vetrov. Stochasticity in neural odes: An empirical study, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, 7ach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
Gamkrelidze R V Pontryagin Lev Semyonovich, Boltyanskii G and Mishchenko E F. The math- ematical theory of optimal processes. 1962.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud. Latent odes for iregularly-sampled time series, 2019.
Lars Ruthotto, Stanley Osher, Wuchen Li, Levon Nurbekyan, and Samy Wu Fung. A machine learn- ing framework for solving high-dimensional mean field game and mean field control problems, 2019.
Lars Ruthotto, Stanley J. Osher, Wuchen Li, Levon Nurbekyan, and Samy Wu Fung. A machine learning framework for solving high-dimensional mean field game and mean field control prob- lems. Proceedings of the National Academy of Sciences, 2020.
Jacob H. Seidman, Mahyar Fazlyab, Victor M. Preciado, and George J. Pappas. Robust deep learn- ing as optimal control: Insights and convergence guarantees. Proceedings of Machine Learning Research, 2020.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. Technical report, 2000.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders, 2017.
11

---

Published as a conference paper at ICLR 2021
Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit, 2019.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. Journal in Machine Learning Res., 2010.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. In Advances Neural Information Processing Systems 32. 2019a.
Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural odes and invertible residual networks, 2020.
Tianjun Zhang, Zhewei Yao, Amir Gholami, Joseph E Gonzalez, Kurt Keutzer, Michael W Ma- honey, and George Biros. Anodev2: A coupled neural ode framework. In Advances in Neural Information Processing Systems. 2019b.
12

---

Published as a conference paper at ICLR 2021
APPENDIX
A THEORETICAL RESULTS
A.1 PROOF OF PROPOSITION 1
In order to show that the set functions defined by N-CODE are not necessarily homeomorphisms generally injective.
on X, show that they
Open-loop: Consider the one-dimensional N-CODE system & (x + X' with equation of motion (xx 0) -0 where 0 ((=(0)) ==0). This system has solution
(ax(0), 7) ar(0) / (x,,0,1) di
x0)) x0)T.
(10)
However, this implies
(0 1) = (11, 1) = 0,
(11)
giving the result.
Closed-loop: Similarly, we can show that the following 2D oscillatory system:
{ (t) = (t))
= xx)) cos(t) + + sin
=xt) = =x0) cos(t) if ((0) = 0 (12)
(0(1) x(()
But ((x, 2/2) = 0, so x + ( t) = (t) is not injective.
This shows that the two control forms, open and closed, are not restricted learn homemorphisms on the space X.
A.2 PROOF OF THEOREM 1
Proof. The proof is inspired from Massaroli et al. (2020a). We place analysis in the Banach space RR. Since & and h are continuously differentiable, let us form the Lagrangian functional L
where a is an element of the dual space of
x
θ
/ je (a,
Oz
Clz,a,t)
h)]dt
(13)
Dt
The constraint (a(t), Oz D1 M2z 1)) is always active on the admissible set by construction of z in
OC
Øl
equation 4, such that we have Vz, a, part of the integral in equation 13 gives:
(2, a,
(z, a, 1). Moreover, integrating the left
Oye
aa Oz dt (a, z) 11
da
z)
(14)
01."
da Now, given that 8 ( 2) is differentiable μ for any t E [0, 2}. Leibniz integral rule allows to write
01 =
le + da dt'
dz(T), ; (a(0), Oz(0)
2) + (a. + (a(T).
(15)
0u
0u
Ou
0l dz
da dz
+ (a, Oh
adl + (a(T), 0z(T))
+ dt 0μ'
(16)
0p
Oz 0μ
0p
13

---

Published as a conference paper at ICLR 2021
T
0I
de 8z
da Oz
Oh
Oh ar
Oh Oz
di + (a(T), 0z(T)
+
+ (a.
+
(17)
0p
0z
dt dμ
dp it 0p
Oz dp
dp
The last equation can be reordered as:
I
01
da
8h
di
dz
Ōh
dt + (a(T), Üz(T))
dt
+
(a,
(18)
Dp
dt
Oz
Oz
0u
0u
0u
0
da
Oh
O0
0z
Posing that (T)) = 11 and
0, the result follows.
dt
Oz
dz
0p
Remark 1: This result allows to compute gradient estimate assuming that μ directly parametrizes straightforward to compute. However, in the case of optimizing the initial mapping y that infer the variable 0 this result can be combined with usual chain rule to estimate the gradient:
Öh
h such that the gradient 0y
00 01
(19)
0p,, Or 00
B IMPLEMENTATION
Generic N-CODE module We provide here a commented generic PyTorcl implementation for the N-CODE module in the open-loop setting.
class NCODE NCODE_func (torch. nn Module)
def Linit_(seff, *args)
super SrcODE_func, self) init
#Definition of the CDE system self.f self.g ...
def forward t, 2):
#Unpack variables (theta_t is potentially tuple of tensors) x_t, *theta_t
#Compute evolution od the system xxt f d d_theta_t self. (theta_t,dxxttt
return td_ *d thett___))
class NCODE block (torch. nn Module)
def Unit__seeff *args):
super (NCODE self) (alf)_initt.. init
#Definition of module and controller self.gamma =
self.NCODE_fune
def (orw,, x
#Set initial conditions self.theta_0 self. gamma (x_0)
14

---

Published as a conference paper at ICLR 2021
2_0 (x_0, *se.f.
#Run the system forward with defined settings
x_t, *theta_t = odeint_adjoine (self.odefunc, z_0, integration rtol, at
return (x_t, *theta_t)
C EXPERIMENTAL RESULTS
Resources - Our experiments were run on a 12GB NVIDIA@ Titan Xp GPUs cluster equipped with CUDA 10.1 driver. Neural ODEs were trained using the corchdiffeg (Chen et al., 2018) PyTorch package.
C.1 FLOWS FOR ANNULI DATASET
2
0
0
-2
2
-2
2
2
2
2
-2
2
2
-2
2
0
2
-2
2
2
-2
2
-2
-2
Figure A.1: Time-varying flows for two points in different classes (inner: top row; outer: bottom row) of the annuli dataset. Time advances from left to right.
C.2 SUPERVISED IMAGE CLASSIFICATION
For MNIST and CIFAR- 10 classification, we defined the equation of motion f as sequence con- volutional filters replicating ResNet block architecture with padding conserve dimensionality and non-linear ReLU activation. The base architecture takes one the forms
1, & filters, stride, padding
3 3, & filters, stride, padding
x1x 1, c filters, 1 stride, 0 padding
with & the number of filters, e the number of channels (1 for MNIST and for CIFAR10) and a the augmentation channel (Dupont et al., 2019). The open-loop controller is a very simple linear transformation of the data into 10 hidden units followed by multiple heads outputting the vector of weights for each convolution kernel. In order to examine the effect of the size of dimensionality of the equation of motion on learnability, we tried different parameterizations for f:
NCODE The three kernels are conditioned by the controller
NCODE 3 x 3: The inner 3x3 kernels are dynamically conditioned.
NODE DC : This corresponds to the form of data-control proposed by (Massaroli et al., 2020b) where the image data is concatenated with state xxt) at each evaluation of the first Ixl convolution kernel.
All parameters not adaptively controlled are simply learned as fixed parameters. For all models we used the adaptative solver Dornund-Prince with tolerance of 1e-3.
15

---

Published as a conference paper at ICLR 2021
NCDE DC NCOE NCOOE
NODE DC
NODE
Accuraty
Thaining
Eerations
Remetions
Figure A.2:
Accuracy, Train and test losses over training interations and Number of function evaluations (NFE) vs loss for several versions of NCODE CIFAR-16 10.
The results show that N-CODE models achieve higher accuracy in fewer epochs and overall lower loss (Fig. A.2, top panels). However, the simplicity of the mapping 7 in this experiment and the absence of regularization of the induced map Φ. (Finlay et al., 2020) yields complex flow requiring higher total number of function evaluations (Fig. A.2, lower panels). This potentially prevents NCODE from achieving even better accuracy. Interestingly, the results for the partially controlled systems suggest that gradient descent accommodates the learning of hybrid modules f with both static and dynamic convolution parameters. These results advocate for the exploration of specific closed-loop formulations regularization of control for convolution layers which we leave for future work.
C.3 PATTERN MEMORIZATION
This experiment is update of Miconi et al. (2018), which showed that using simple Hebbian update rule between weights endowed recurrent networks with fast memory properties. The base model is a single-cell recurrent network with all-to-all coupling, (()). The considered models
Open-loop control: We adopt the implementation of (Massaroli et al., 2020b) by appending the pre- sented pattern Estio to the dynamic variable at) such that (t) is linear layer T( R.).
Closed-loop control: The weights are fully dynamic and the influence of the plastic evolution is tuned by learning the components μ that apply an element-wise gain for every weight velocity, according to Eq. 7.
ODE-RNN: Here, statht- NODE models the evolution of continuous-time single-cell recur- rent network whose hidden state is carried over from one presentation to the next. A supplementary linear read-out L(R RVV) outputs the model's guess and balances the number parameters with other models.
LSTM:
vanilla LSTM cell (Hochreiter & Schmidhuber, 1997). In coherence with (Miconi al., 2018) results, the hidden state dimension need to be greatly increased for the module to start mem- orizing. We tested 5000-dimensional hidden state for the reported results.
For each model, we learn the model weights with gradient descent using an Adam optimizer with learning rate 3e and use the La distance between the reconstruction and objective pattern as our objective function. For continuous models, each episode consists of sequential presentation of each pattern for 0.5 sec followed by a query time of 0.5 sec. For the LSTM, we adopted the setting of (Miconi et al., 2018) where the sequence of presentation is discretized into 5 time-steps of presentations. We tested for episodes of 3.5 and 10 patterns with presentations in random order and different proportions of degradation (0.5.0.7 7 and 0.9). Static- Static-weight model performance rapidly deteriorated in more challenging settings, whereas N-CODE reconstruction converged to a residual enror below 1 % in all cases.
16

---

Published as a conference paper at ICLR 2021
Error rate for varied deqradation proportion (%)
Error rate for varied number of patterms (%)
0-0.5
0.25
5 patterns
g-0.7
10 pattems
8 0.3
p-0.9
0.20
patterms
tro
0.2
0.10
0.05
0.0
100
200
300
400
500
100
200
300
400
500
600
700
Episodes
Episodes
Figure A.3: Error rate N-CODE on the 1000-bit reconstruction task as function of episodes for (Left) different proportions of degraded bits and (Right) different number of patterns presented in one episode. In all cases, the model learns in several hundreds episodes.
MNIST
CIFAR-10
CelebA
X R 23X28228
X E R 33x3233
X R33642661
Encoder:
Flatten
Conv25e + BN Relu
Conv266 F> BN + Relu
FC400 + Relu
Conv512 + BN + Relu
Conv512 + BN + Relu
FC25 + Norm
ConV + BN + Relu
ConV1024 + BN + Relu
FC128+2 + Norm
FC Norm
Latent dynamies:
ODE[0,10]
ODE[0,10]
ODE[0,10]
Decoder:
FC + Relu
F02444 + Norm
FC102448+8 + Norm
FC784 Sigmoid
ConvTs12 + BN + Relu
ConvTs12 + BN H Relu
Conv T256 + BN + Relu
ConvT + BN H Relu
ConvT3
ConvT3
Table A.1: Model architectures for the different data sets tested. FC, and Conv,, represent, respec- tively, fully connected and convolutional layers with output/filters. We apply component- -ise normalisation the control components which proved crucial for good performance the model. The dynamic is the time segment [0,10] which empirically yields good results.
C.4 AUTON-CODE ARCHITECTURES
Adapting the models used by Tolstikhin et al. (2017) and Ghosh et al. (2020), we use latent space dimension of 25 for MNIST, 128 for CIFAR- and 64 for CelebA. All convolutions and transposed convolutions have filter size of 4x4 for MNIST and CIFAR-10 10 and 5x5 for CELEBA. We apply batch normalization to all layers. They all have a stride of size except for the last convolutional layer in the decoder. We use Relu non-linear activation and batch normalisation the end of every convolution filter. Official train and test splits are used for the three datasets. For training, we use mini-batch size of 64 in MNIST and CIFAR and 16 for CelebA in AutoN-CODE. (64 for control models.) All models are trained for a maximum of 50 epochs on MNIST and CIFAR and 40 epochs on CelebA. We make no additional change the decoder. We train the parameters of the encoder and decoder module for minimizing the mean-soo -qcarrs eтror (MSE) on CIFAR-10 and CelebA (Liu al., 2015) or alternatively the Kullback-Leeibl divergence between the data distribution and the output of the decoder for MNIST (formulas in Appendix.) Gradient descent is performed for 50 epochs with the Adam optimizer (Kingma Ba, 2014) with learning rate À 1 reduced by half every time the loss plateaus. All experiments are run on single GPU GeForce Titan X with 12 GB of memory.
C.5 VISUALIZATION OF LATENT CODE DYNAMICAL EVOLUTION
C.6 EXPLORING SAMPLING DISTRIBUTIONS
For random sampling, train the VAE with a. aN I) prior. For the deterministic models, samples are drawn from a mixture of multivariate gaussian distributions fit using the testing set embeddings. TTe distribution is obtained through expecration-maxiizzatii Dempster al. (1977) with sin-
17

---

Published as a conference paper at ICLR 2021
5
5
0
4
2
4
2
2
2
2
222
2
3
3
3
3
3
3
33332
H
H
H
4
4
5
5 5
6 6
7
7
77 888
7
7
8
8
8
8
8
8
Figure A.4: Reconstructions of the image along the controlled orbits of AutoN-CODE for MNIST.
The temporal dimension reads from left to right. Last column: ground truth image
Figure A.5: Reconstructions of the image along the controlled orbits of AutoN-CODE for CIFAR- The temporal dimension reads from left to right. Last column: ground truth image.
gle k-means initialization, tolerance 1 and run for at most 100 iterations. We compute the FID using 10k generated samples evaluated against the whole test set for all FID evaluations, using the standard 2048-dimensional final feature vector of an Inception V3 following Heusel al. (2017) implementation.
The results show that 10K components (matching the number of data points in the test sets) natu- rally overfits on the data distribution. Generated images display marginal changes compared to test images. However, 1000 components does not, showing that our AutoN-CODE sampling strategy mediates trade-off between sample quality and generalization of images. We alternatively tested non nonpparametric kernel density estimation with varying kernel variance replace our initial sampling strategy. We report similar results to the gaussian mixture experiment with overall lower FID of AutoN-CODE for small variance kernels. As fitting distribution becomes very rough (o 5), the generated image quality is highly deteriorated. Figure A6).
18

---

Published as a conference paper at ICLR 2021
Figure A.6: Reconstructions of the image along the controlled orbits of AutoN-CODE for CelebA
The temporal dimension reads from left to right. Last column: ground truth image
5000
Figure A.7: Left: Evolution of FID as a function of the number of components in the Gaussian mixture used to fit the latent distribution. We also tested vanilla autoencoder with twice the capacity (33-ddimensiona), exactly matching the number of parameters of our model. Interestingly, this model performs worst in low interpret this ample population becomes less dense in this augmented space, making the naive gaussian mixture fitting less accurate for fitting the latent distribution. Right: Samples of generated images for different number of mixture components. The shape and details becomes clearer with incrcasing components.
component regime.
manifestation of the "curse of dimensionality", as latent ex-
19

---

Published as a conference paper at ICLR 2021
Figure A.8: Nearest neighbors search of random samples from the gaussian mixture with (Left) 1000 components (Left) and (Right) 10K components in the testing set of CIFAR-10. 10. The sampled latents show overfiting on test set in the highest case.
ODE
100
Autoencoder_ 236
128
80
Figure A.9: (Left) Evolution of FID between test set and generated samples as a function of the gaus- sian kernel variance used perform kernel estimation of the the latent code distribution. (Right) Nearest neighbor search of random samples from the latent distribution fit by gaussian kerne esti- mation with variance in the testing set.
20

---

Published as a conference paper at ICLR 2021
C.7 AUTON-CODE LATENT CLUSTERING
We further investigate the representations learned by our model by measuring its clustering accu- racy against several other techniques in Fig. A.10. Similarly to the image generation experiment, we consider the final state of the dynamical system as the latent representation that we cluster using a 10 component gaussian mixture with different initializations. We did not perform any supplemen- tary fine-tuning or data augmentation to train our model, but we also tested a version with further dimensionality reduction using t-SNE (Maaten & Hinton, 2008). The results, although inferior to recent deep clustering techniques, show better clustering accuracy than other autoencoding models, suggesting different organization of the latent code compared to the vanilla linear projection of the autoencoder.
Clustering accuracy
Model
CIFAR-I0
MNIST
K-means (Lloyd, 1982)
22.9
57.2
Spectral clustering (Shi & Malik. 2000)
24.7
69.6
Variational Bayes AE! (Kingma & Welling. 2013)
29.1
83.2
Sparse AE (Ng. 2011)
29.7
82.7
Denoising AE (Vincent et al., 2010)
29.7
83.2
AE (GMM) GAN (2015) (Radford et al., 2015)
31.4
81.2
31.5
DeepCluster (Caron al., 2018)
DAC (Chang al., 2017)
52.2
97.8
11C (Ji et al., 2019)
61.7
99.2
AutocN-CODE (GMM))
33.31
86.02
AutoeN-CODE (I-SNE GMM)
27.00
97.26
Figure A.10:
Unsupervised image clustering accuracy on CIFAR-10 and MNIST against recent models. Results obtained with the authors original code are noted with f.
9.
25
26
-90
-35
Figure A.II: (SNE embeddings of the latent code at f1 for MNIST test set colored with 10 component gaussian mixture model.
21

---

Published as a conference paper at ICLR 2021
C.8 LATENT CODE INTERPOLATION
Figure A.12: Interpolation: We further explore the latent space of AutoN-CODE by interpolating between the latent vectors of the CelebA test set. (Upper panel) Linear interpolation between dom samples reconstructed with AutoN-CODE. (Middle panel) Interpolation comparison between AutoN-CODE and vanilla autoencoder for single pair of vectors. (Lower panel) 2d interpolation with three anchor reconstructions corresponding to the three corners of the square (A:up-left,) B:up- right and C:down-left. Left square corresponds AutoN-CODE reconstructions and right vanilla autoencoder.
22


Reviews:
[{"review_id": "b8AibOatHxS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n**Paper summary**\nThe paper proposes a new class of neural ODE based models, called neurally-controlled ODEs (N-CODE). Instead of directly learning the weights of the neural network parameterizing the vector field f of the ODE, the paper instead proposes to learn a controller that takes as input the initial state and outputs the initial weights of f and optionally also updates the weights of f as the ODE is solved. This allows the model to parameterize a family of (potentially time varying) vector fields instead of a single vector field. The authors go on to show that their model can be trained using an augmented adjoint method (similar to the adjoint method used in the original neural ODE paper).\n\nThe authors test the performance of the proposed model across a variety of tasks. First, they test their model on some of the toy problems proposed in the augmented neural ODEs paper. They show that their model can successfully learn mappings to represent the 1D reflection and 2D concentric annuli functions. For example, in the case where the model only learns a controller for the initial state, the model learns to map +1 and -1 to different vector fields, effectively allowing the model trajectories to cross. Similarly, in the case where the vector field is additionally controlled as time evolves, the model also allows states to cross and represent both functions. Second, the authors evaluate their model on a supervised learning task, showing that the proposed method improves performance on both MNIST and CIFAR10.\n\nFinally, the authors build an interesting new form of autoencoder based on their method. The autoencoder takes as input an image and outputs the initial weights of the vector field parameterizing an ODE. The authors then allow a fixed initial state to evolve in this vector field. The final state is then used to decode the model into the reconstructed image. Therefore, the latent information about the image is contained in the weights of the vector field as opposed to a latent vector as is usually the case. The authors evaluate their method on both CIFAR10 and CelebA using the methods from the deterministic autoencoder paper to endow their model with a latent space they can sample from. The results on both datasets are generally good.\n\n\n**Positives**\n- The paper is clearly written and the model is well explained. The figures are also nice.\n- The autoencoder experiments are interesting and novel.\n- The model performs well across a wide variety of tasks from toy problems to image classification and autoencoder like unsupervised learning.\n- The motivation for the paper is clear and fairly important: alleviating some of the representational weaknesses of neural ODEs and generally improving and extending neural ODE models by using time varying weights.\n\n**Negatives**\n- The paper is closely related to other works that have also focused on making the weights of neural ODEs time dependent. Data controlled neural ODEs from the dissecting neural ODEs paper are, as far as I can tell, exactly the same as the open loop model proposed in this paper. Further, the dissecting neural ODEs paper also proposes using time varying weights (as do some other papers), although not in exactly the same way as this paper. As such the novelty of this paper is quite limited (although the autoencoder experiments are, to the best of my knowledge novel). It would be good to have a more thorough discussion of the differences between these models to better understand where the novelty/contribution comes in.\n- The closed loop model (which is the main innovation) is thoroughly discussed in the paper. However, it seems like most experiments are actually performed with the open loop model (which is very similar to other models already proposed in the literature). The image classification experiments and the autoencoder experiments both use (as far as I can tell), the open loop model. The only place where the authors seem to use the closed loop model is in the toy experiments, where the model performs worse than the simpler open loop model. This does put into question whether the full closed loop model is actually useful in practice.\n- There is no evaluation of the number of functions evaluations in the paper (as far as I can see). According to the abstract, the model trains faster so it seems important to include this information in the paper.\n- The authors mention in the abstract that the model achieves state of the art reconstruction on CIFAR10. However, reconstruction doesn’t really make sense as a task (since the identity function would be optimal in this case). It is interesting if the latent space learned is very low dimensional (i.e. the model compresses the dataset well), however there seems to be no discussion of this. So I feel like this claim is questionable and should be removed.\n\n**Recommendation**\nWhile the paper is well written and has thorough experiments, the proposed model is very closely related to several models that have already been proposed in the literature. Further, the full model seems to not be used in most of the experiments and appears to perform worse than the simplest version of the model. The autoencoder experiments however are quite compelling and novel and would likely be of interest to the ICLR community. I therefore recommend a weak accept.\n\n**Typos and small comments**\n- In the first sentence of the paper the word “opens” seems like it shouldn’t be there\n- The word “such” is repeated on bottom of page 2\n- There are no labels on the Figure A.5 axes\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UtZ8zBUxnrx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a novel approach N-CODE, based on Neural Ordinary Differential Equations (NODE), that increases the expressivity of  continuous-time neural nets by using approaches from Control theory. N-CODE, in contrast to NODE, can learn a family of vector fields and is therefore able to flexibly adjust the flow for every data point. Authors provide theoretical evidence and compute several simulations with different problems (including Reflection, concentric annuli, supervised image classification and image auto encoding) that confirm that the proposed method greatly improves or outperforms state-of-the-art methods.\n\nThe paper is well written and clear. The proposed approach is original, and the results suggest that the performance of the approach leads to significant improvements compared to the state-of-the-art. \n\nThe performance of the method was assessed by applying N-CODE to different problems. While evidence in terms of performance is presented, no theoretical or empirical evidence is presented that confirms that the training speed is significantly lower compared to NODE. Also, while performance seems generally higher, the limitations of the proposed method are not clear. Better performance in less time is a strong claim, that is not fully supported by the results.\n\nIn summary, however, this is a very  interesting approach with potential.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GKNZKml_Wqq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update:\n\nI am happy with the authors' rebuttal and have increased my rating accordingly.\n\n--------------------------------\n\nSummary\n\nThe paper proposes to predict the required weights of a Neural ODE function from the input data. The paper has two parts - supervised classification, and unsupervised image reconstruction and generation. In the supervised part, has good visualizations of the decision boundaries induced. The unsupervised part shows good comparisons with previous methods, and good visualizations of the reconstructed and generated images.\n\nStrengths\n\nThe paper is well organized. The images and figures are explained well. The graphs showing the trajectories of the Neural CODE are great for visualization. The paper provides enough details about training its networks.\n\nComments\n\nThe paper is split into two parts - supervised classification, and unsupervised modeling. The supervised classification part is highly related to the “Data Controlled Neural ODEs” section in Massaroli et al (2020b), a paper that has been cited in the related work section but not addressed in the main content. In fact, the problems tackled (-x, concentric annuli) and the results are highly related to those of Massaroli et al (2020b). This issue needs to be addressed sufficiently.\n\nThe unsupervised section needs a lot more work. The experiments and tables can be described more effectively. For example, it would be preferable to clearly explain which row section 5.3 refers to in Figure 7, and which row corresponds to section 5.4.\n\nThe fact that the replacement of a linear layer with a Neural CODE improves image reconstruction quality should mean that all layers in the encoder can be replaced to give better encoding, taking care of dimensionality (such as in Normalizing Flows). In fact, generative modelling papers such as FFJORD, or Normalizing Flows, do replace all layers with a Neural ODE and map to a latent space of noise. However, this paper uses a typical neural network for image reconstruction and generation. This perhaps means that the majority of heavy-lifting is done by the decoder, hence the Neural CODE is more amenable to warping the latent space suitable for the decoder. For image generation as well, the latent space has been designed so that the decoder can produce nice images, this is not necessarily a win for the Neural CODE.\n\nTo make the case for Neural CODE, especially for images, higher resolution images need to be tackled, since it is in higher dimensions that the success of the advancements in the methods listed in Figure 7 lies.\n\nMassaroli et al. (2020b) : Dissecting Neural ODEs; NeurIPS 2020;  https://arxiv.org/abs/2002.08071\nGrathwohl et al. : FFJORD : FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models\n\nMinor\n\nMany variables needs to be bold at multiple places.\nFigure 4 needs to say “Third” for the closed loop figure.\nZhang et al., 2019b is cited in related work, but needs to be cited along with the respective experiment for Figure 5.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1f8na4AVhG", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Comment: Summary: This paper presents a technique for more expressive neural ordinary differential equations (NODE) flows. Instead of learning a fixed set of parameters $\\theta$ that governs the ODE dynamic, the proposed approach learns dynamic parameters that evolve over time. The authors propose two variants of the methods: $\\textit{open-loop}$ and $\\textit{closed-loop}$. The former only maps the initial observation as the controller whereas $\\theta$ in the later model follows another NODE $g$. Model performance is demonstrated on several tasks. The model is shown to solve the well-known \"crossing curves\" problem, on which NODE fails. Also, it's demonstrated that the presented technique yields sharper images compared to VAEs and also better at classification and interpolation. \n\nOverall Score: I recommend an accept. \n- First of all, the presented N-CODE method solves one of the most crucial limitations of NODEs in a principled manner. The model is also shown to outperform the vanilla Augmented NODE method.\n- The results are very impressive. Both the tables and generated images/interpolations are of high-quality, especially given VAEs don't excel at this task.\n- The method has certain overlaps with the control theory and maximum principle, deep generative models, and neural nets with adaptive weights. So I believe such connections would open new research avenues.\n\nCons: I would be happy if the below are addressed:\n- Did you investigate how the model performance changes as f grows? I speculate that learning the parameters of a neural network via another neural network(s) is a very challenging problem, and would like to see this is verified or not. Also, I cannot see the architecture of f in your experiments (looking at Table A.1).\n- Connection with control theory can be made clear. As such, there is very little reference to Pontryagin's maximum principle and the link is not visible (at least to me).\n- Did you test vanilla NODE on experiments 5.3 and 5.4? The virtue of N-CODE is obvious on the toy problem (as expected) and somewhat significant on the classification task. I'm wondering when NODE is latent (as in 5.3-5.4), is the improvement significant?\n\nAdditional comment: Is Figure-4 caption correct?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a few variants of neural ODE architectures to improve their expressivity.  The motivation and method make sense, but are fairly incremental.  The tasks are also fairly low dimensional and as one reviewer pointed out, reconstruction isn't a good benchmark task.

However, the paper seems well-executed, and the rebuttals answered the expert rewiewers' concerns.
Decision: Accept (Poster)