Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This conference paper aims at providing a computationally efficient algorithm to identify the underlying dynamics of nonlinear physical systems using Continuous-time (CT) neural network models that consider the presence of challenging features such as external inputs, noise in the measurement process, latent states, and robustness during training. Recent models have explored the ability of general models parametrized by CT neural networks to learn unknown nonlinear dynamics from partial observations. However, these models typically seek to include only one challenging feature – thereby ignoring the others – and are computationally intensive, especially for long time series. Building on these previous works, the authors propose to break the time series into shorter overlapping chunks, each one approximated by the same neural networks and linked by an encoder function, also approximated by a neural network. The authors convincingly show analytical evidence that this approach increases the smoothness of the cost function with respect to the parameters, and hence the robustness of the learning process, and that it can reduce the computational costs by partially parallelizing operations. The performance of the proposed model is tested on two standard benchmark problems and compared to other methods from machine learning. The novel method appears to perform on par or better than state-of-the-art models such as neural Ordinary Differential Equations (ODEs).
strength_and_weaknesses: Strengths
The problem of inferring the dynamics underlying physical, biological, and social systems is an important one, and this manuscript introduces promising results with respect to i) the robustness of the training algorithm when the problem is broken down into overlapping shorter subsections, and ii) the intriguing possibility of introducing an encoder to infer the initial state of the system at the beginning of each subsection. Particularly interesting is the analytical proof shown in the first part of the appendix, in which they derive the temporal scaling of the Lipschitz constant in the parameters space. Also significant are the insights given on the topic of the normalization terms required to avoid pathological behavior during training. The performance of the model on the two chosen benchmarks seems impressive compared to other methods. 

Weaknesses
The main claims of the paper are, in my opinion, not sufficiently explored in the results section. Specifically, a proper evaluation of the advantage in terms of computational costs compared to the other methods in Table 1 is missing. This is particularly important for other methods, like Forgione & Piga 2021 that also employ subsections. Furthermore, the authors cite Ayed et al 2019 as a similar work that also employ an encoder function, but do not compare it to their method. Along the same lines, the authors do not investigate the choice of hyperparameters such as na, nb or the length of a subsection Tdt. Exploring these hyperparameters and their impact of performance vs computing costs would greatly enhance the impact of the paper. Finally, the second benchmark only contains 500 samples in total, which seems modest if contrasted with the claims made by the authors about “accurate long-term predictions”. Perhaps a different benchmark – e.g. the EMPS dataset employed in Forgione & Piga 2021 – could be used to highlight the potential of the method, specifically the use of subsections. There’s one last aspect that I think is not particularly explored in the paper despite the initial claims: the detection of latent states. The benchmark used in the paper are not particularly well suited to evaluate the ability of the model to infer the presence of latent states. Perhaps a better benchmark could be used since this is one of the main claims of the paper.
clarity,_quality,_novelty_and_reproducibility: Clarity
While the first two sections do a good job in familiarizing the reader with the subject of the study, with previous relevant work, and by stating the desired properties of the model, the manuscript lacks clarity in later sections. Particularly, it not clear how the subsection approach solves the issue of gradient-descent requiring a full forward pass. The “pipe” notation in the Proposed method section is a bit abstruse, and the role of the subsection hyperparameters na and nb should be discussed in more detail. Similarly, the role of the encoder could be examined further, for example by clarifying the relationship between the transient error, the initial state error, and the encoder loss. The manuscript also contains a few typos, the most evident being the word “reconstructability” written as “reconstrubability” in a few instances.
Additional comments on clarity:
-	"However, they only consider a fixed output function and solve their optimization problem as an optimal control problem whereas our formulation alters the simulation loss function to obtain the computationally desirable form."
This needs to be discussed further. Is it always better to alter the simulation loss? What is the "however" referring to? What is the computationally desirable form?
-	The benchmark problems are introduced very briefly. As Figure 3 is practically uninformative, it could be replaced by the equations of the underlying dynamical systems that can be easily found in other articles.
-	It is not clear in the first part of the paper that the encoder is also implemented as a neural network. One has to wait for the first line of the Results section to learn that. It would be useful to the reader to mention that earlier in the paper since the encoder is a main feature of the proposed model.

Novelty
I have concerns regarding the originality of this work compared to [Forgione & Piga 2021 and Ayed et al 2019]. Subsections and the inclusion of external inputs that render the system non-autonomous were already introduced in Forgione & Piga 2021, but do not employ an encoder to estimate the initial states. The method of Ayed 2019, instead, does incorporate an encoder to estimate the initial states but do not include inputs. By combining these methods together, the present manuscript does not appear to introduce any especially novel feature. Nevertheless, the method does improve performance and the theoretical insights estimating the effect of introducing subsections and normalization on training robustness, are surely interesting. I believe that improving the clarity of the manuscript would also emphasize the innovative aspects of the model and help the reader identifying the differences with respect of previous works.

Reproducibility
The provided jupyter notebooks do reproduce the results of the paper. However, the training implementation is contained in a python package that I could not directly inspect.

summary_of_the_review: Summary of the review
This work addresses the important problem of reconstructing the underlying dynamics of a nonlinear system. Overall, the manuscript contains elements of interest, and the performance of the model are impressive. However, it somewhat lacks clarity and depth when discussing some key features of the model, such as the concrete advantages in computing costs, the novel aspects compared to previous works, and the advantage of utilizing the encoder to infer the initial state. Novelty is also a concern per se, since all the features of the model where already present in previous works, albeit not at simultaneously. 
The authors may want to clarify the unclear aspects and expand the discussion before considering the paper for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors consider the problem of learning models of continuous time dynamical systems in the presence of noise, latent variables and driving input. They present a method for doing this, which reduces the computational complexity of model fitting by breaking up longer observation sequences into shorter chunks.  They introduce the interesting idea of learning a neural network to estimate latent system state at the start of each of these chunks.   In addition, they provide theoretical results underlying their method and empirical results on benchmark applications.   
strength_and_weaknesses: The paper is well written and, as the authors do a very good job of motivating, the problem of fitting continuous models to the behavior of dynamical systems is important.  The authors also do a good job of contextualizing their work with regards to the existing literature.   While the idea of using Euler integration followed by back propagation to learn neural network descriptions of differential equations is not novel, 
I believe the idea of approaching this through (1) breaking up longer sequences into shorter ones and (2) using a neural network to predict initial state at the beginning of each shorter sequence is novel and very interesting.  This introduces the benefit that many of the computations associated with each sequence can be performed in parallel.   

The authors also provide theoretical arguments about the potential benefits of their approach.  However, there are multiple areas the presentation of these results should be clarified and improved.  First, it is not clear how surprising theorem 1 is: given that the derivative has a bounded Lipschitz constant is it surprising that when we integrate over shorter sequences, the Lipschitz constant of the integral, also decreases?  To this point, it would be helpful to know how tight the bound is.  If I understand the notation correctly to be big-O, this is an upper bound.  If the tightest upper bound possible is indeed exponential with T, then this result is indeed interesting, but we would need assurance (or at least a reason to conjecture) that this is indeed the case.   Second, while formal theoretical results are not needed, it would be helpful for the authors to discuss what, if anything, might be lost by breaking up longer observation sequences into shorter chunks.  For example, does this hinder the ability to learn dynamical systems with longer time scales?  Perhaps not if the network inferring initial state is accurate enough, but having an explicit discussion about this would help.   Third, the paragraph following theorem 2 appeals to it in a way that the theorem itself is not currently worded to support.  In particular, theorem 2 is currently worded as an existence proof: it is possible to find two constants such that both the state and derivative are normalized.  However, the paragraph immediately after this theorem (beginning "Hence, considering only...") appeals to this theorem as if it were a proof of necessity.  Following, the authors math shown in the proof of the theorem, I believe such a claim could be made, but the theorem would need to be re-worded (or the logic of the following paragraph made more clear).   Finally, and perhaps more importantly, I am a bit confused about how theorem 2 applies to model fitting.  In particular, I agree given a ground truth x(t) and f(t), two constants can be found such that the transformed descriptions are regularized as the authors claim.  However, when fitting models, there is flexibility in the f we chose, so that it would seem for any f and any value of 1/\tau in equation (3), can't we always find a f' = \tau f.  In other words, don't we have to restrict the flexibility of the function class we consider for f in some way for \tau to have any affect?  I acknowledge the empirical results the authors show in Fig. 5 supporting their theoretical arguments, but in those results there seems to be a large range of where RMSE of the simulation error plateaus and for which the RMS of x and f are not 1. 

Finally, the authors provide empirical results comparing their method to others in two benchmark tests.  They claim to achieve state of the art results in both.  However, as the authors commendably make clear, part of their test sets are also used for validation.  They state this is fair because many of the existing methods also do the same.  However, to fully evaluate the claim that a new state of the art has been achieved, it would be helpful for the authors to show their existing results as well as results obtained when validation and test sets are fully distinct.   When comparing to previous methods which did in fact have overlapping test and validation sets, they could use the current set of results, but when comparing to methods that had disjoint test and validation sets, this second set of results could then be used to ensure a fair comparison. If their method still outperforms all existing results when comparing each on this more equal footing,  I believe they could then rigorously claim a new state of the art has been achieved. 
clarity,_quality,_novelty_and_reproducibility: The work is very clearly written and the quality of the writing is high.   As noted above, the idea of breaking up longer sequences into shorter ones and inferring initial state for each shorter sequences is, to my knowledge, novel and an important contribution.  I have no concerns about the reproducibility of the results. 
summary_of_the_review: Summarizing all of the above, I feel the core idea of the paper is novel and important.  However, the theoretical claims need further discussion and clarification to make their full importance more apparent.  In addition, I find it likely that the authors have indeed achieved new state of the art results on the two benchmarks they provide, but additional analyses showing how their methods work when the test and validation sets are totally disjoint are needed to confirm this.   I believe addressing these points would take an already very interesting core idea and make the paper substantially stronger.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 1. The authors try to infer the underlying dynamics of some unknown dynamical system based on sparsely sampled observations and external inputs.
2. The authors build a framework to infer such a dynamical system using 1) neural networks as function approximators for a generic continuous-time dynamical system, and 2) gradient-based optimization.
3. The framework can account for a dynamical system that demands 1) handling noisy measurements, 2) doing long term prediction, 3) compatible with hidden states, and 4) compatible with external inputs.
4. The quality of the inferred model is evaluated by how good the observed trajectories are reconstructed in two benchmark problems.
strength_and_weaknesses: **Strength:**

Having the original joint optimization problem broken down to 1) optimizing initial hidden state and 2) optimizing forward dynamics have the potential to increase optimization stability.

**Weaknesses:**

Since the framework uses three neural networks as function approximators for encoder, iterative dynamics, and output projection, the full model and optimization scheme look very similar to training a standard recurrent neural network. From this comparison, it is less clear about the benefits of having such elaborate framework in terms of increasing performance and reducing computational complexity.
clarity,_quality,_novelty_and_reproducibility: **Novelty**: As the authors claim, the novel aspects of the framework includes:

1. generating initial conditions for each sequence section using an encoder function as another neural network independently optimized using past observations;
2. introducing a normalization term tau for better performance;

**Clarity & Quality:**

Unfortunately, the benefits as a result of both novel aspects are not clearly explained.

1. Specifically, while it is reasonable to expect that having such an encoder function for separately optimizing initial hidden states can increase stability of the full optimization, it is not clear at all how this invention can help to reduce overall computational complexity.
2. Supposedly, the normalization term tau plays a role of regularization for better training stability, but how it helps is not clear.
summary_of_the_review: 1. I will recommend this paper if the author address and clarify the above issues.
2. Without such clarification, it is hard to judge whether this work add to the existing effective approach: e.g., training RNN with backprop through time, or doing FORCE learning with chaotic or balanced neural network.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed a new scheme for the continuous-time nonlinear state-space system identification based on overlapping short subsections rather than the whole length of time, with the help of a subspace encoder and state-derivative normalization. The paper proved that, both algorithmically and empirically, the proposed scheme will reduce computational complexity and improve cost function smoothness. 
strength_and_weaknesses: Strength: The proposed work provides a solid foundation to improve the system identification modeling by the derivation of the necessary condition for encoding the initial state when performing system identification on overlapping short subsections and proving that the smoothness of the encoder cost function will be improved by doing so. Model performance shown in the experiment results, especially the comparison with neural ODE, further validates the effectiveness of the proposed model. 

Weakness: The structure of the SUBNET is unclear from Figure 2 and the corresponding text. As SUBNET is the key contribution of this work, more details are welcomed. The reviewer is also curious about how the discrete-time subspace encoder, as listed in Table 1, is implemented. 

clarity,_quality,_novelty_and_reproducibility: Clarity: Beyond the comments above, the methodology and experiment description of the manuscript is clear to read and easy to follow.  

Quality: The quality of the manuscript is good, with sufficient algorithm analysis and insights. Experiment results, especially the comparison with state-of-art methods, also validate the author’s claims.

Reproducibility: Based on the reproducibility statement provided by the manuscript, the proposed model and the experiment shall be reproducible. 

summary_of_the_review: A solid and novel scheme for the continuous-time nonlinear state-space system identification.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a novel approach for the estimation of continuous time state space dynamical models. 
This field has gained significant attraction since 2018 with the Neural ODE.
The proposed approach is based on the evaluations on short subsections, and a novel state-derivative normalization.
strength_and_weaknesses: Strengths:
1.	The authors tested their approach on two different datasets.
2.	The authors have developed proof of convergence of their technique.
3.	The authors have provided source code for better reproducibility purpose.
Weaknesses:

1.	The results in table 1 seems to suggest that for both problems the neural ODE approach with normalization seems to be performing as well or even better than the proposed technique. Do the authors suggest that the main advantage of their approach lies in the lower estimation time (lower computation complexity).
2.	I fail to understand how the proposed approach works, especially the joint training of 3 fully-connected neural networks.
3.	Figure 5 caption could be expanded for better understanding.

Additional questions:
Have the authors analyzed the effect of the subsections size on the performance of the technique?
Why did the authors compare their technique with different approaches on the two datasets (for example BLA (Relan et al. 2017) and Volterra model only assessed on the CCT benchmark)?
Could the authors comment on how the proposed technique compare to the Neural Laplace (“Neural Laplace: Learning diverse classes of differential equations in the Laplace domain”, Samuel Holt et al. ICML 2022) ?


clarity,_quality,_novelty_and_reproducibility: 
The paper lacks a bit of clarity, especially the description of the methodology and the training of the neural networks. The results of the proposed approach seem identical to the neural ODE with normalization, but at a much lower computational cost.
The authors intend to open source the code for reproducibility purpose.

summary_of_the_review: The paper introduces a novel approach for the identification of continuous dynamical state-space model parameters. The method is interesting, but I fail to understand the training process. The results seem to indicate performance on par with “state-of-the-art” approach but a lower computational cost.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new method for continuous time state space modeling. It has two components: a method for training on short subsequences of the training set which is shown to improve the smoothness of the cost function; and a new normalization strategy which improves the conditioning of the model. The method is shown to improve upon SOTA, especially in computational complexity and reliability, and theoretical guarantees of the subset training procedure are also provided.
justification_for_why_not_higher_score: While the reviewers found the method to be novel, they thought more could be done to improve the clarity of the paper and in the evaluation of the method.
justification_for_why_not_lower_score: The proposed method is clearly novel and holds promise.
Decision: Accept: poster