Neural graphical modelling in continuous-time: consistency guarantees and algorithms

Abstract: The discovery of structure from time series data is a key problem in fields of study working with complex systems. Most identifiability results and learning algorithms assume the underlying dynamics to be discrete in time. Comparatively few, in contrast, explicitly define dependencies in infinitesimal intervals of time, independently of the scale of observation and of the regularity of sampling. In this paper, we consider score-based structure learning for the study of dynamical systems. We prove that for vector fields parameterized in a large class of neural networks, least squares optimization with adaptive regularization schemes consistently recovers directed graphs of local independencies in systems of stochastic differential equations. Using this insight, we propose a score-based learning algorithm based on penalized Neural Ordinary Differential Equations (modelling the mean process) that we show to be applicable to the general setting of irregularly-sampled multivariate time series and to outperform the state of the art across a range of dynamical systems.

Reviews and author responses:

This paper introduces a brand new graphical modeling framework from the perspective of neural ODEs. Traditionally structure learning involves using sampled data to learn the structure of graphs. This paper, however, looks at the graph structure learning problem from a different viewpoint, using continuous-time dynamics inspired from neural ODEs. Theoretical guarantees on parameter estimation are provided. Some experiments on benchmark time-series datasets are also conducted.  Strengths:

-- This is a very well written paper that introduces a new framework for graphical modelling. Reading the paper was a real pleasure. Because I was asked to review the paper as an emergency reviewer, I couldn't afford time to check the proofs. 

-- The framework is very pleasant and novel and paves the way for other researchers to build on. 

Weaknesses:

-- Unfortunately, my enthusiasm for this paper is somewhat tempered by the theoretical results, which do have potential, but also limitations. In traditional graphical modeling, one is interested in the recoverability of the underlying graph. Here, unless I missed something, in Lemma 3, the authors are proving some results concerning the parameters of the model. Why would we be interested in the parameters? It seems like learning the structure would be able to provide practitioners with more useful qualitative information about the relationships between the processes $x_j$. 

-- It's also not clear to me why one would be interested in the generalization bound as stated in Lemma 2. Generalization bounds are useful in supervised learning settings, but here one is interested (as I mentioned), the quality of the graph that is recovered from the observations. 

-- Even if we accept that the risk and difference between the estimated parameter and the true parameter are important quantities, it is unclear what the results imply. Can the authors comment about tightness of the upper bounds on the generalization error (8) or the consistency gap in (9)? I realize that a single paper cannot contain too many results, but I fail to appreciate how the bounds in (8) and (9) depend on some hardness parameter of the learning task. I am familiar with graphical models, and usually the results depend on the minimum strength $\beta$ in the exponential family parameterization. Here, this dependence seems to be completely absent.  See main review above. The paper does have a lot of potential, but perhaps it is not fully mature at this point. 

This paper proposes a penalized neural ODE method for learning graphical models for multivariate dynamic systems.  The proposed method is a straightforward application of the neural ODE and group Lasso method. The novelty is low. In addition, it is unclear how the proposed method can be used to handle heterogeneous dynamic systems, which are often the case of practical systems and might make the proposed method less robust.   This paper provides a reasonable method for graphical modeling of dynamic systems, although the technical novelty is limited.   

The paper proposes to learn Jacobian-sparse neural network ODEs from irregular trajectories of a dynamical system. The main contribution is the sparsity of the ODE Jacobian, which results in learning of differential covariate causalities. Learning the differential structure is an important real-world problem. The proposed method is elegant, simple and effective, although incremental.
 The paper is masterfully written: the presentation is rigorous and yet reads effortlessly. I enjoyed reading the paper throughout. 

The presented method is straightforward extension of neural ODEs with adaptive group lasso with proximal methods. The main contribution is the theoretical consistency and convergence analyses of such a system. The paper discusses learning bounds in terms of data fit, and in terms of sparsity pattern. The main insight is to show that one only needs to sparsify the first layer weight of the neural ODE. 

The proposed method has outstanding performance in the experiments that cover observation irregularity and sparsity, and a biological and chaotic system. The method is compared against lots of competing methods. 

Technical comments:
* Lemma 1: If xk appears in differential of xj, then surely one expects to see non-zero jacobian between them, while here it is implied to be zero instead. 
 This is a fantastically written paper with an effective model for an important problem with solid theory and outstanding empirical performance. 


Meta-review: This main focus of this paper is graph modeling. Specifically, this paper considers a setting in which data is generated under continuous time dynamics based on neural ODE. Theoretical results regarding parameter estimation are provided. The results are also supported by experiments.

The reviewers appreciate a thorough response to their questions and think that this paper would be of interest to ICLR and ML community. Please address reviewers comments in your final version.

Decision: Accept (Poster)