Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper introduces a novel approach N-CODE, based on Neural Ordinary Differential Equations (NODE), that increases the expressivity of  continuous-time neural nets by using approaches from Control theory. N-CODE, in contrast to NODE, can learn a family of vector fields and is therefore able to flexibly adjust the flow for every data point. Authors provide theoretical evidence and compute several simulations with different problems (including Reflection, concentric annuli, supervised image classification and image auto encoding) that confirm that the proposed method greatly improves or outperforms state-of-the-art methods.

The paper is well written and clear. The proposed approach is original, and the results suggest that the performance of the approach leads to significant improvements compared to the state-of-the-art. 

The performance of the method was assessed by applying N-CODE to different problems. While evidence in terms of performance is presented, no theoretical or empirical evidence is presented that confirms that the training speed is significantly lower compared to NODE. Also, while performance seems generally higher, the limitations of the proposed method are not clear. Better performance in less time is a strong claim, that is not fully supported by the results.

In summary, however, this is a very  interesting approach with potential.

review_title: The flow is strong with you N-CODE

final_score: rating: 7: Good paper, accept
review_content: Update:

I am happy with the authors' rebuttal and have increased my rating accordingly.

--------------------------------

Summary

The paper proposes to predict the required weights of a Neural ODE function from the input data. The paper has two parts - supervised classification, and unsupervised image reconstruction and generation. In the supervised part, has good visualizations of the decision boundaries induced. The unsupervised part shows good comparisons with previous methods, and good visualizations of the reconstructed and generated images.

Strengths

The paper is well organized. The images and figures are explained well. The graphs showing the trajectories of the Neural CODE are great for visualization. The paper provides enough details about training its networks.

Comments

The paper is split into two parts - supervised classification, and unsupervised modeling. The supervised classification part is highly related to the “Data Controlled Neural ODEs” section in Massaroli et al (2020b), a paper that has been cited in the related work section but not addressed in the main content. In fact, the problems tackled (-x, concentric annuli) and the results are highly related to those of Massaroli et al (2020b). This issue needs to be addressed sufficiently.

The unsupervised section needs a lot more work. The experiments and tables can be described more effectively. For example, it would be preferable to clearly explain which row section 5.3 refers to in Figure 7, and which row corresponds to section 5.4.

The fact that the replacement of a linear layer with a Neural CODE improves image reconstruction quality should mean that all layers in the encoder can be replaced to give better encoding, taking care of dimensionality (such as in Normalizing Flows). In fact, generative modelling papers such as FFJORD, or Normalizing Flows, do replace all layers with a Neural ODE and map to a latent space of noise. However, this paper uses a typical neural network for image reconstruction and generation. This perhaps means that the majority of heavy-lifting is done by the decoder, hence the Neural CODE is more amenable to warping the latent space suitable for the decoder. For image generation as well, the latent space has been designed so that the decoder can produce nice images, this is not necessarily a win for the Neural CODE.

To make the case for Neural CODE, especially for images, higher resolution images need to be tackled, since it is in higher dimensions that the success of the advancements in the methods listed in Figure 7 lies.

Massaroli et al. (2020b) : Dissecting Neural ODEs; NeurIPS 2020;  https://arxiv.org/abs/2002.08071
Grathwohl et al. : FFJORD : FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

Minor

Many variables needs to be bold at multiple places.
Figure 4 needs to say “Third” for the closed loop figure.
Zhang et al., 2019b is cited in related work, but needs to be cited along with the respective experiment for Figure 5.

review_title: Needs better motivation and results, and clarity on relation to previous work

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Comment: Summary: This paper presents a technique for more expressive neural ordinary differential equations (NODE) flows. Instead of learning a fixed set of parameters $\theta$ that governs the ODE dynamic, the proposed approach learns dynamic parameters that evolve over time. The authors propose two variants of the methods: $\textit{open-loop}$ and $\textit{closed-loop}$. The former only maps the initial observation as the controller whereas $\theta$ in the later model follows another NODE $g$. Model performance is demonstrated on several tasks. The model is shown to solve the well-known "crossing curves" problem, on which NODE fails. Also, it's demonstrated that the presented technique yields sharper images compared to VAEs and also better at classification and interpolation. 

Overall Score: I recommend an accept. 
- First of all, the presented N-CODE method solves one of the most crucial limitations of NODEs in a principled manner. The model is also shown to outperform the vanilla Augmented NODE method.
- The results are very impressive. Both the tables and generated images/interpolations are of high-quality, especially given VAEs don't excel at this task.
- The method has certain overlaps with the control theory and maximum principle, deep generative models, and neural nets with adaptive weights. So I believe such connections would open new research avenues.

Cons: I would be happy if the below are addressed:
- Did you investigate how the model performance changes as f grows? I speculate that learning the parameters of a neural network via another neural network(s) is a very challenging problem, and would like to see this is verified or not. Also, I cannot see the architecture of f in your experiments (looking at Table A.1).
- Connection with control theory can be made clear. As such, there is very little reference to Pontryagin's maximum principle and the link is not visible (at least to me).
- Did you test vanilla NODE on experiments 5.3 and 5.4? The virtue of N-CODE is obvious on the toy problem (as expected) and somewhat significant on the classification task. I'm wondering when NODE is latent (as in 5.3-5.4), is the improvement significant?

Additional comment: Is Figure-4 caption correct?
review_title: Recommendation to Accept

final_score: rating: 7: Good paper, accept
review_content: 
**Paper summary**
The paper proposes a new class of neural ODE based models, called neurally-controlled ODEs (N-CODE). Instead of directly learning the weights of the neural network parameterizing the vector field f of the ODE, the paper instead proposes to learn a controller that takes as input the initial state and outputs the initial weights of f and optionally also updates the weights of f as the ODE is solved. This allows the model to parameterize a family of (potentially time varying) vector fields instead of a single vector field. The authors go on to show that their model can be trained using an augmented adjoint method (similar to the adjoint method used in the original neural ODE paper).

The authors test the performance of the proposed model across a variety of tasks. First, they test their model on some of the toy problems proposed in the augmented neural ODEs paper. They show that their model can successfully learn mappings to represent the 1D reflection and 2D concentric annuli functions. For example, in the case where the model only learns a controller for the initial state, the model learns to map +1 and -1 to different vector fields, effectively allowing the model trajectories to cross. Similarly, in the case where the vector field is additionally controlled as time evolves, the model also allows states to cross and represent both functions. Second, the authors evaluate their model on a supervised learning task, showing that the proposed method improves performance on both MNIST and CIFAR10.

Finally, the authors build an interesting new form of autoencoder based on their method. The autoencoder takes as input an image and outputs the initial weights of the vector field parameterizing an ODE. The authors then allow a fixed initial state to evolve in this vector field. The final state is then used to decode the model into the reconstructed image. Therefore, the latent information about the image is contained in the weights of the vector field as opposed to a latent vector as is usually the case. The authors evaluate their method on both CIFAR10 and CelebA using the methods from the deterministic autoencoder paper to endow their model with a latent space they can sample from. The results on both datasets are generally good.


**Positives**
- The paper is clearly written and the model is well explained. The figures are also nice.
- The autoencoder experiments are interesting and novel.
- The model performs well across a wide variety of tasks from toy problems to image classification and autoencoder like unsupervised learning.
- The motivation for the paper is clear and fairly important: alleviating some of the representational weaknesses of neural ODEs and generally improving and extending neural ODE models by using time varying weights.

**Negatives**
- The paper is closely related to other works that have also focused on making the weights of neural ODEs time dependent. Data controlled neural ODEs from the dissecting neural ODEs paper are, as far as I can tell, exactly the same as the open loop model proposed in this paper. Further, the dissecting neural ODEs paper also proposes using time varying weights (as do some other papers), although not in exactly the same way as this paper. As such the novelty of this paper is quite limited (although the autoencoder experiments are, to the best of my knowledge novel). It would be good to have a more thorough discussion of the differences between these models to better understand where the novelty/contribution comes in.
- The closed loop model (which is the main innovation) is thoroughly discussed in the paper. However, it seems like most experiments are actually performed with the open loop model (which is very similar to other models already proposed in the literature). The image classification experiments and the autoencoder experiments both use (as far as I can tell), the open loop model. The only place where the authors seem to use the closed loop model is in the toy experiments, where the model performs worse than the simpler open loop model. This does put into question whether the full closed loop model is actually useful in practice.
- There is no evaluation of the number of functions evaluations in the paper (as far as I can see). According to the abstract, the model trains faster so it seems important to include this information in the paper.
- The authors mention in the abstract that the model achieves state of the art reconstruction on CIFAR10. However, reconstruction doesn’t really make sense as a task (since the identity function would be optimal in this case). It is interesting if the latent space learned is very low dimensional (i.e. the model compresses the dataset well), however there seems to be no discussion of this. So I feel like this claim is questionable and should be removed.

**Recommendation**
While the paper is well written and has thorough experiments, the proposed model is very closely related to several models that have already been proposed in the literature. Further, the full model seems to not be used in most of the experiments and appears to perform worse than the simplest version of the model. The autoencoder experiments however are quite compelling and novel and would likely be of interest to the ICLR community. I therefore recommend a weak accept.

**Typos and small comments**
- In the first sentence of the paper the word “opens” seems like it shouldn’t be there
- The word “such” is repeated on bottom of page 2
- There are no labels on the Figure A.5 axes

review_title: Improving neural ODEs with time varying weights and data dependent vector fields

Meta-review: This paper introduces a few variants of neural ODE architectures to improve their expressivity.  The motivation and method make sense, but are fairly incremental.  The tasks are also fairly low dimensional and as one reviewer pointed out, reconstruction isn't a good benchmark task.

However, the paper seems well-executed, and the rebuttals answered the expert rewiewers' concerns.
Decision: Accept (Poster)