Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper solves the problem of decoupling dynamical complex systems in a data-driven manner. The authors propose an improved version of the projected differential equations. The authors conduct experiments on various important problems with synthetic or real-world datasets. The results show that the proposed DNS can achieve the best performance.
strength_and_weaknesses: Strong points.
1. The proposed method and the basic idea make sense to me. The design of the meta-system is novel.
2. The method is described well, and the paper presentation is nice.
3. The results show the best performance of the proposed DNS.

Weak points.
1. The experimental settings require improvement. There is no sufficient hyper-parameter study of the baselines. Thus, the comparisons among the models may not be fair, which leads to unreliable conclusions.
2. There is no study about the complexity. Compared with other baselines, which also work in a data-driven manner, the complexity of the proposed method is not clear. It seems the parameter size is far larger compared with baselines, and thus the computation time is a concern.
3. The performance improvement of the proposed method seems to be not steady. From the results, we can observe that the performance is very sensitive to the parameter size (influenced by the hyperparameter). Therefore, it is also concerned about the generalization ability of the proposed method.
clarity,_quality,_novelty_and_reproducibility: The originality is good, but the experiments do not support it well. The presentation is overall good.
summary_of_the_review: The experimental part requires a lot of improvements, although the idea makes sense.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims at modeling complex dynamical systems into smaller subsystems. It does so by combining neural controlled dynamics(Kidger et al.) with self-attention (Vaswani et al.). Empirical evaluations are performed in multiple regression and link prediction environments from 3-body gravitation, springs, charged particles and human actions.
strength_and_weaknesses: - Strengths
    - The problem of modeling modular dynamical systems at irregular time intervals is compelling.
    - The paper does a good job introducing neural ODEs, neural controlled dynamics and self-attention.
    - The paper evaluates in a wide diversity of scenarios, capturing different applications of the proposed approach.
- Weaknesses
    - Experimental results are not strong and some better results are missing from the table of comparisons. In particular, at least two methods achieve much better results in the charged and springs datasets. The paper obtains 68% link prediction accuracy in charged and 95.4% on springs. However, AFAIK, this is on the same datasets that Neural Relational Inference with Fast Modular Meta-learning (NeurIPS '19) obtained 88.4% and 99.9%, respectively. Furthermore, the paper where these datasets come from (Neural Relational Inference for Interacting Subsystems, ICML '18) is not shown in the table of comparisons, and that also obtains significantly better results: 82.1% and 99.9%.
    - The proposed architecture is missing justification, in the sense that it is proposed as something we can do, but the reason why the added complexity is necessary is missing. Better ablations in the experimental section, improved explanations in the method section and a better interplay between the two sections would greatly help in this direction.
    - Although good for many other conferences, the level of novelty (mainly combining attention-based mechanisms with neural controlled dynamics), is IMO not enough for the ICLR bar.
clarity,_quality,_novelty_and_reproducibility: - Quality and novelty
    - The paper is a combination of Neural Controlled Dynamics and Transformers and, in this view, is not very novel. Furthermore, stronger results and explanations would be necessary to convince me that this combination is better than other complex architectures one may propose.
    - As mentioned under weaknesses, experimental results were sometimes much lower than published algorithms.
    - Most experimental results are missing confidence intervals and probably have too many significant digits. Furthermore, I believe most people now do not show the size of the architecture as different architectures and instead usually use the larger version (briefly explaining if validation data favored a smaller version). Not all, but most cases, shown in the tables of this paper corroborates that the bigger model tends to be better.
    - The paper says that "-" means both not training reliably or not tried. It would be important to separate these two cases, particularly as DNS has many "-" in table 3, pointing to instabilities, which would also have to be explained.
- Clarity
    - The introduction of related concepts was good, the motivation and explanation of the proposed architecture needs to be worked on in my opinion.
    - When talking about the decoupling of related papers "their decoupling seem shallow" is itself a shallow explanation. I think it would be better to explain in technical terms the difference between their decoupling and the decoupling of this paper.
    - Minor
        - I would potentially edit the title to "complex systems" or "a complex system"
        - end of page 1 (e.g., ) empty two times
        - just after equation 3 it says dx(t) = ... yet dx(t) doesn't seem to appear in the equation.
summary_of_the_review: I like the goal of this paper and believe it has many practical applications. At the same time, I feel that, at the moment, the methodology and experimental results do not support the novelty and quality of the proposed approach.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new neural ODE-based framework to model irregularly-sampled sequential data. The approach is based on neural control differential equations, and on the observation that attention models can be viewed as integrating a differential equation path. The authors evaluate their approach on a few toy examples, and it is compared to several other baselines (ncde included). The reported results show certain benefits to using the proposed method.

strength_and_weaknesses: The main strength of this paper is the incorporation of two seemingly unrelated concepts: controlled differential equations and attention maps into a unified framework that models irregularly sampled time series data and their interactions. While that involved math is somewhat challenging, the authors did a good job in laying out the dependencies and relations between the various equations that appear in Sec. 3.

The main weakness of this paper is the evaluation of the proposed approach. First, the tasks chosen for evaluation seem somewhat arbitrary, and not necessarily related to the underlying assumptions. For instance, the motion of three bodies is independent as long as no interactions occur. However, the main motivation beind the coupled PDEs introduced in Sec. 3 was to model several highly independent sub-systems. In addition, the authors state that the three body problem was simulated with many interactions, so the amount of independent motion is further reduced. Second, the description of the evaluation protocol and reported results are somewhat alarming. Was there a validation set? The text suggests there was no validation set. How was grid search performed then? By reporting the best results on the (5 last epochs of) test set? If so, it might be that the results are biased towards overfitting the test sets. Moreover, the text suggest that only a single run was performed. Given that the differences in error measures are not significant, I would like to see a more extensive evaluation where testing is performed 5 or more times with different seeds, and average error metrics and standard deviations are reported. Also, are the results reported in Tabs. 1-4 reported elsewhere (for the baselines)? If not, how did achieve these results? training and testing the various methods? Was there an extensive grid search performed for the other methods? If so, how did you choose the parameter ranges? As a note, your framework uses more supervision information in comparison to other works, as you use e.g., 3 sub-systems in the three body example. Given the similarity between the proposed method and NCDE, I would like to see an extensive ablation study between these approaches. The graphs in Fig. 2 do not show any advantage to the proposed method in terms of gradient magnitudes. The issue around gradient magnitude is specifically related to exploding and vanishing gradients. Fig. 2 shows neither, and in particular, there is not even an order of magnitude difference between red and blue curves.

The second weakness is related to the modeling. Specifically, the text is dense and hard to follow at times. No proper motivation or description was provided regarding Eq. (11). Why model $dz_i(t)$ in this way? The interaction differential equation was better motivated using attention mechanisms, but still I do not necessarily see why this particular model satisfies the underlying assumptions. For instance, the interaction DE does not encourage the sub-systems to be independent. How $g$ is modeled, and what is its purpose? What is the loss function? What do you do with $z, A$ to get the output?


clarity,_quality,_novelty_and_reproducibility: The text is reasonably clear, although significant improvements can be performed. The work is of high quality deadling with challenging problems and approaches. To the best of my knowledge, this work is novel. Given the discussion above regarding evaluation, I believe that reproducibility is nearly impossible.

summary_of_the_review: This paper starts with an intuitive requirement: one wants to model complex systems using several simple and independent sub-systems. This motivates from a higher-level the method design, which turns out to be a coupled system of differential equations, with another equation dealing with interactions. While the intuition and modeling is generally reasonable, the evaluation is lacking in terms of chosen tasks, and testing protocol. I believe the paper could be significantly stronger if the authors can improve on these points.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper focuses on the problem of modeling data coming from a complex system with an underlying continuous dynamics. The key assumption/claim of the paper is that such a complex system is best modeled as an interaction between multiple coupled sub-systems (that has its own dynamics too). Using ideas from differential equation literature and modern attention mechanism, the paper comes up with a compelling approach towards modeling such systems. Advantages of the approach are the clear in the experimental results where the approach is compared against a variety of baselines.
strength_and_weaknesses: **Strengths**

The problem of simplifying a complex system into appropriate simpler sub-systems is one of the most important things scientists and engineers do in their daily life. Figuring out ways to automate this process directly from data can be extremely impactful. The paper very clearly formalizes the problems and identifies connections and bridges to theory of differential equations as well as continuous time interpretations of extremely successful attention mechanisms. Insights from these connections result in a very natural way of decoupling the sub-systems as well as the meta-interaction system. Results are validated on a variety of problems including human-action recognition with quite a few relevant baselines which are all appropriately modeling the task with an underlying continuous dynamics. 

**Weaknesses**

The paper doesn't mention this but training neural diffeq approaches can be quite inefficient, especially on modern heterogenous architectures.
Would be great to have limitations explicated somewhere.
There are a lot of typos in the paper. There are also quite a few w.r.t `\citep` vs `\citet` issues too. Reading with fresh eyes would help fix a lot of these issues.
clarity,_quality,_novelty_and_reproducibility: Barring the fair number of typos, the paper is really well articulated and has a lot of interesting ideas to build future work on. The code was fairly easy to follow as well and would be super useful to reproducing the experiments.
summary_of_the_review: This is a fantastic paper and likely has a wider real world impact capability especially when modeling real world man-made physical systems which are often modular. It would be great if the paper could discuss its own limitations more carefully (especially neural DiffEq can be hard to train with increasing dimensions), any effect of diffeq solver vs Euler as used here, GPUs vs CPUs, etc. W.r.t video classification, event camera data might be ideal demonstration for irregular data. For a lot of real world applications the modular structure is partially known as well and incorporating that information would be fairly impactful future work. Similarly a good test of decoupling could be if the components could be transferred to a similar but different system.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper focuses on the problem of modeling complex dynamical systems by multiple, independent ones.  The approach is incremental, and is supported by an eclectic experimental section.

Despite the good rebuttal phase, the experimental section still can be improved, for instance by adding confidence intervals to get better insight in the significance.  Also, in some cases the method is not superior or seems not to be tested with sufficient fairness (e.g., increased size of CT-GRU may improve).

All in all, the paper needs strengthening to be beyond incremental.
justification_for_why_not_higher_score: The decision is based on agreeing with the majority of the reviewers.  The experimental section was improved, but not at a level which merits acceptance, when compared to other papers.  Then, the paper is just incremental.
justification_for_why_not_lower_score: n/a
Decision: Reject