A method for identifying causality in the response of nonlinear dynamical systems

Authors: Joseph Massingham, Ole Mattis Nielsen, T Butlin
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Nonlinear dynamical systems, causality, application to physical sciences, deep learning, noise estimation

Abstract:
Predicting the response of nonlinear dynamical systems subject to random, broadband excitation is important across a range of scientific disciplines, such as structural dynamics and neuroscience. Building data-driven models requires experimental measurements of the system input and output, but it can be difficult to determine whether inaccuracies in the model stem from modelling errors or noise. Therefore there is a need to determine the maximum component of the output that could theoretically be predicted using the input, if an improved model was to be developed through the investment of resources. This paper presents a novel method to identify the component of the output that could potentially be modelled, and quantify the level of noise in the output, as a function of frequency. The method uses input-output measurements and an available, but approximate, model of the system. A trainable, frequency dependent parameter balances an output prediction generated by the model with noisy measurements of the output to predict the input to the system. This parameter is utilised to estimate the noise level and then calculate a nonlinear coherence metric as a measure of causality or predictability from the input. There are currently no solutions to this problem in the absence of an accurate benchmark model.

Decision: reject
Reviewer scores: 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1729847253342

thread_type:
official_review

<thread_object>
creator: reviewer_wwmi

<content>
summary:
The authors of the manuscript ‘A METHOD FOR IDENTIFYING CAUSALITY IN THE RESPONSE OF NONLINEAR DYNAMICAL SYSTEMS’ develop a method for the calculation of the nonlinear coherence between input and output data for a noisy ODE class given random input. The approach allows to learn an optimal combination of an output prediction with noisy measurements. The nonlinear coherence that is calculated is then used as a measure of causality. The method is shown to perform well on both simulated dynamical datasets as well as experimental data.

strengths:
- The approach is well characterized mathematically in the manuscript.
- The demonstration of the approach over different types of simulated and experimental data is interesting.
- There are comparisons to alternative approaches over the same task.

weaknesses:
- The manuscript could be improved in terms of readability. It is in many cases very difficult to follow.
- While the method was evaluated over real experimental data, the noise was added synthetically.

questions:
Comments:

- The description of the problem and approach in the abstract is not clear enough.
- Could you provide a more thorough statistical analysis of the performance of the approach for the simulated datasets over different parameter regimes of the underlying model, to show its robustness?
- The figures are missing captions.
- It would be informative, for the simulated and experimental results, to explicitly show how a better approximation of the response (as presented in the figures) helps in the interpretation of the original systems.

Minor comments:
- ‘but has recently begun’ - the ‘but’ part of the sentence should probably be rephrased
- it should be mentioned in the abstract that this work is restricted to a specific class of ODEs.
- ‘A completely new strategy’ - rephrase?
- The related work section sounds too defensive.. e.g. ‘The presented method .. is very different to work in the literature.’
- ‘In this paper, the primary aim is to estimate the noise level and the model error, but a similar principle is utilised.’ - unclear.
- ‘The presented method.. provides an excellent estimation of the nonlinear coherence for each noise case’ - rephrase in a more quantitative way?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709985020

thread_type:
official_review

<thread_object>
creator: reviewer_1yvb

<content>
summary:
The manuscript presents a novel technique for separate measurement noise and modeling errors without accessing the actual input-output relationship. The calculation was performed in frequency domain and there's one weighting parameter per frequency controlling the contribution of modeling error and measurement error to achieve minimum reconstruction error. The performance, measured by the coherence between actual signal and recovered signal is high.

strengths:
- The presented method learns the a nonlinear functional mapping through a neural network. It works under a wide range of nonlinearities form as long as y can be mapped back to x (Eq.3).  
- It presented a novel way to balance modeling / observation error and contributed to maximize signal reconstruction performance in highly nonlinear systems. 
- Even in high noise cases, the recovered coherence was still high.

weaknesses:
- Only applies to the specific ODE described in Eq.3. Maybe the authors could extend it further as long as there's a one-to-one mapping from y to x? 
- the method only works from 1D signal to 1D signal? Is there any possibility to extend to high dimensional nonlinear systems.

questions:
- What is the relationship between K and \lambda? Is K a hyper parameter during network training? How did you find the optimal K. 
- Why the modeling error / measurement error affects the signal in an additive way (Eq.4)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710278444

thread_type:
official_review

<thread_object>
creator: reviewer_3sze

<content>
summary:
The paper introduces a method for estimating the coherence $Co(y, y_n)$ between the observed noisy data $y_n$, and the “true” underlying system $y$. This estimate, termed $\hat{y}$ is obtained by combining, in frequency domain, a model/input based estimate $y_z$, with the noisy data $y_n$ (similar to a Kalman filter). The estimated $Co(y, y_n)$ is closer to $Co(y, y_n)$, than an only model/input-based estimate $Co(y_z, y_n)$.

strengths:
- The method uses a sensible way to combine model/input based data with observed data.

- The resulting method outperforms the chosen baselines in terms of the coherence metric.

weaknesses:
1. I found the paper hard to read, I think this can be improved by the authors more clearly explaining or motivating their motivation for the different steps (which might be obvious to them, but might not be to the reader), e.g.,  
    - 1.1 “Causality” is in the title and a key point of the paper, however I am totally missing some explicit motivation for why measuring coherence is the best way to estimate causality. The one citation for coherency (Silva et al., 2016), seems an odd choice.
    - 1.2 ANR introduced, but it, and the cited papers seem only tangentially related. This could be scrapped to make space for motivating the actual method, i.e., the use of coherency.
     - 1.3 No motivation is given for combining a model-based estimate with data in the frequency domain.
     - 1.4 Altough $\mathcal{F}$ of Eq. 1 is called a function, it is not a map from $x(t)$ to $y(t)$ (then there would be no dynamical system). Relatedly, it might also be better to use a different characters for $\mathcal{F}$ and $\mathcal{F}_\theta$, which are two very different things.

2. I find it very confusing that the paper introduces, and keeps referring to non-linear coherence, when the paper really seems to be after estimating the “standard”, linear coherence between $y$ and $y_n$. 

3. Page 9, has half its space dedicated to a Figure of an experimental setup, but this is just another benchmark, I would suggest to make this Figure smaller (or move it to appendix), and use the remaining space to improve readability.

4. The only evaluation here is $Co(\hat{y},y)$, against other coherences, which is not immediately intuitive. Could the authors not simply also plot and/or quantify how close $\hat{y}$ is to $y$?

5. Fig. 1 seems to lack a $\hat{Y}$ before, and $\hat{y}$ after the IFFT box. $X$ and a box with $\mathcal{F}$ (with some parameters $\theta$?) feeding into $y_z$ would also be helpful.

6. I find the motivation for needing $\mathcal{L}y$ and $\lambda$ in general a bit lacking in theoretical foundations. A possible perspective could potentially be that Eq. 5 gives one the variational estimate of the true distribution of $y$ as $q(\hat{y}|y_n, x)$,  $F_\theta$ gives one $p(x|\hat{y})$ as $\mathcal{N}(x; F_\theta(\hat{y}), \sigma_x^2)$. If one adds a prior on $\hat{y}$ as p$(\hat{y})=\mathcal{N}(y_n, \sigma_y^2)$ and writes out the typical ELBO loss on $p(x)$, both terms in Eq. 7 drop out (just the entropy of $q$ is missing) - although I am sure other perspectives are possible!

questions:
1. Why is Wavenet a good baseline? If one wants to distinguish what combining the forward model with a data inferred state brings you, can one not simply take the forward model from section 3.3?
2. Are the $\dot{x}$ and $\ddot{x}$ terms in Eq. 3 needed (they seem to be absent in all benchmarks)?
3. Eq. 7, is there an implicit Gaussian assumption?
4. Line 197 Why would $\lambda=0$ yield the optimal $K$, this is not obvious to me?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732372268300

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to reviews

comment:
We thank the reviewers for their helpful, constructive comments. We have significantly revised the description of the method in order to improve clarity to a wider field. We have also emphasised the novelty and impact across a range of disciplines. We respond to your individual points below.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors aim in this work to provide a measure by which to judge the best possible dynamical system model that can fit experimentally measured inputs and outputs of a system. The measure focuses on a frequency based approximation of coherence,  and applies the metric to a number of systems. 

There were a number of issues raised, primarily in the clarity and impact of the work. Multiple reviewers found the work difficult to parse, which clearly led to a number of confusions including in notation and in properly understanding the underlying assumptions the govern the work. Despite extended conversations with the reviewers, it seems that as it stands the assumptions tend to be limiting and the manuscript can be much more clearly honed to describe how the metric presented connects back to model goodness and interpretability. These seem to be significant drawbacks and a revised manuscript might fare better.