Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new type of continuous depth neural network architecture, named Invariant Imbedding Networks (InImNets). InImNets can imbed a given $\bf x$ input to different time steps (time positions) and efficiently calculate the solutions of the dynamic system (starting from different time positions) simultaneously. The authors also give a practical implementation for training the InImNets architecture. The proposed architecture is then applied to several benchmark problems for continuous neural networks for verifying its effectiveness.
main_review: This paper studies a very important and interesting topic, continuous depth neural networks. It may be one way to reach the framework of Explainable AI. I appreciate the authors providing a very detailed derivation and implementation for the InImNet architecture, which enables readers to easily reproduce the proposed method.

However, I still have some concerns, which are listed as below:

1. According to the current experiment results, I am not sure what is the superiority of the proposed InImNet compared to existing approaches. For example, in Table 1, both the InImNet and other existing models can achieve similar performance. The authors may want to give a detailed explanation about such superiority.

2. For the "rotating MNIST" experiments, only the MLP model is used for implementing InImNet. I would be happy to see InImNet being implemented with other types of models, for example, a small CNN. Besides, can the authors apply InImNet to some large-scale datasets like CIFAR-10?

3. According to the implementation of InImNet, it seems would take more calculation consumption for training the InImNet. Therefore, it is necessary to compare the time used for training InImNet and other models. Further, I would like to see the comparison of performance for different models under the same training time cost.
summary_of_the_review: Based on the previous comments, I tend to recommend accepting this paper. But I would like the authors to resolve my concerns first.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In recent years, continuous depth neural networks, such as Neural ODEs have helped understanding of ResNet in terms of non-linear vector-valued optimal control problems. The authors of the paper show that it is possible to reduce the non-linear, vector-valued optimal control problem to a system of forward-facing initial value problems, providing an alternative perspective on the problem. They demonstrate that this approach can be used in creating discrete and continuous numerical DNN schemes. 
main_review: The strength of the paper is that it introduces an alternative view to the neural ODE, by reducing the non-linear, vector-valued optimal control problem to a system of forward-facing initial value problems. They introduce the invariant imbedding method from mathematics community to achieve this. They also show that the resulting algorithm can perform well compared to state-of-the-art methods.

There are two weaknesses of the paper. First, the presentation of the paper can be improved. I believe the paper should be written in a self-contained manner such that for the readers who have not seen the Neural ODE and optimal control view can still easily follow the paper, which is not the case in the current form. For example, the authors introduce and then start to derive the properties about the adjoint $\Lambda(p,x)$ without ever defining or explaining its role. Because of this, Theorem 2 is hard to follow without knowing the exact definition of $\Lambda(p,x)$. One possibility is to provide some sort of background review in the Appendix. Second, I think another weakness of the paper is that it does not seem to provide a rigorous treatment. Even though there are Theorem 1-3, I could not find any technical assumptions. It is weird to see Theorems stated without any assumptions on the model, and that makes the readers to wonder the rigor of the paper.
summary_of_the_review: (1) I suggest the authors to include a section discussing the background of Neural ODE, adjoint method etc. in the Appendix.

(2) Technical assumptions should be added, and the proofs should be made rigorous (if possible) throughout the paper. Otherwise, the paper stands as a non-rigorous treatment of the subject.

(3) On page 7, '3' and on page 9 'Invariant Imbedding'  are typos.

(4) In the caption of Table 1, the stop somehow appears in the next line.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a class of networks called Invariant Imbedding Networks (InImNets). This type of networks aims to model an ODE as a pair of initial value problems, with the variable being the starting point of the problem. This allows for a more efficient implementation of a model of a dynamic system. The proposed model is analyzed both theoretically and experimentally.
main_review: The paper aims to tackle the problem of modeling a continuous time dynamic system, using a category of neural networks. This idea is part of a novel field of study, starting with Neural ODEs (Chen et al, 2018). Thus, this paper studies a very interesting and novel problem, in a way that is in tune with prior work on the same topic.

Strengths:
+ The mathematical formulation of the proposed model is solid. The continuous time behavior of InImNets is adequately analyzed, and derivations for the forward and backward steps of the model are included.
+ The motivation behind the proposed model is also very interesting. Instead of solving a two-point boundary conditions problem (as in regular Neural ODEs), the authors solve two initial value problems for the same task. This allows the implementation of a simpler and more efficient scheme to train the network.
+ The formulation of the problem also leads to links with optimal control theory. This is an exciting direction which, while not extensively explored in the paper, can lead to interesting future works.
+ The experiments considered are adequate in order to provide a proof of concept for this architecture, when compared to prior work on the field.

Weaknesses:
-	I believe that there are parts of the paper that could be reformulated to become simpler and easier for the reader to understand. In particular:
  -	Section 2.1 should appear before the introduction of the section right above (since it describes the overall form of the optimization problem the model tries to solve).
  -	Equations 21 and 22 describe an alternative scheme which is not actually used, and I believe they may be omitted without hampering the flow of that section.
  -	Equations 16 and 17 are more closely related to Section 3.2 rather than the one they are placed right now.
-	The experimental section is not as clear as the rest of the paper. Firstly, Section 4.1 does not include any results, other than Figure 1 (I understand that the goal is primarily to provide a toy example as a motivation, but its placement makes the reader think that there are associated numerical results). Secondly, while Section 4 seems to imply that the model used was the discretized form of Algorithm 1, as described in Section 3, in the conclusion the authors state that they provide results for two models, a discrete and a continuous one. These two models should be better explained, both structurally and in which experiments they are used precisely.

Minor comments/Typos:
-	In the figures/tables in the experimental section, it would be nice to explicitly state the source of each architecture, in a way that matches the figure/caption (i.e. using the name of the architecture directly).
-	I believe the notation would be better if $z(t;p,x)$ for the InImNets was $z(p,x)$ (or something similar, given that while they do depend on the variable $t$, it is not considered to be the primary variable of the model.
-	Typos:
  -	Line after equation 2: “a external” -> “an external”.
  -	Page 3, overview of the paper: “InImnets” -> “InImNets”.
  -	Page 3, at the very end: “satisfies” -> “satisfy”.
  -	Page 5, after equation 13: “accessible optimize” -> “accessible to optimize”.
  -	Page 5, second to last paragraph before 3.1: missing word after “demonstrating high”.

Questions:
-	As mentioned above, I would be grateful if the authors could provide some additional explanations on the models used in their experiments.

summary_of_the_review: Overall, this is a good paper which provides an exciting novel architecture, with good theory and some simple experiments to justify the quality of this architecture, with comparisons to related work. My current issues lie with the clarity of writing and the explanations of the models used in the experiments. Given that these issues can be easily resolved during the rebuttal process, I lean towards accepting this work, pending some extra explanation provided by the authors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper extends the Neural ODE model by introducing the dependency on the extra parameter p that can be interpreted as a control parameter or the “depth” of the network. The models allows to decode the network output by varying this parameter. 

main_review: **Strong points:**
The idea is interesting and well motivated. The paper provides the derivations for both the forward pass and backward pass for the gradients w.r.t. a new parameter p. The methods section builds the model from the ground-up and it is easy to follow.

**Weak points:**

The results section is a bit underwhelming. The results demonstrate the bare minimum that a) the approach works, matches the performance of other models b) how the performance changes when we vary p_max. 

In the introduction, the paper puts  a lot of emphasis on understanding and interpreting the network. With this motivation, I expected to see more analysis on the learned ODE and the relation between the learned trajectories z(…) and the trajectories in the data space (for example. similarly to figure 1); or the analysis what depth of the network is required to learn the bounce off the walls in the bouncing ball experiment. Perhaps instead network interpretation, the authors could highlight and experiment with other benefits of the model: 1) we can take the output of the network after every “depth” and decode it into a meaningful output 2) we can do smooth interpolation between the input and the output. 

**Additional questions:**
What happens if we go beyond p_max in the experiments?

Do you actually see high memory cost to justify using the finite difference method that is as accurate as the chosen delta? All experiments use MLP with a few layers

Can we just vary the t_max rather than conditioning on p?

More details on the experiments need to be provided in the results section as well. Specifically, 1) How do you choose q? 2) In the results section, what does p_max represent? Why do you vary p_max, even though the model integrates on the interval [p, q]. Should it be p_min?

summary_of_the_review: The paper has an interesting idea to condition the Neural ODE on the parameter p that can be interpreted as the “depth” of the network. The Results section does not explore the new architecture enough and provides only the basic results to demonstrate that the model works. I suggest the acceptance of the paper, but recommend the authors to expand the results section.
review_title: null

Meta-review: The authors presents an alternative view of Neural ODEs, offering a novel understanding of depth in neural networks. The reviewers were overall impressed by the novelty and potential for insights this work brings. There was some disappointment that the empirical results were not stronger (both in terms of pure performance and computational cost) and that it wasn't clear how the theoretical insights into depth actually translated into a practical insight. Nevertheless, I agree with the reviewers that this is a good submission and would I think make for an interesting addition to the conference programme.
Decision: Accept (Spotlight)