Neural Symbolic Regression of Complex Network Dynamics

Authors: Haiquan Qiu, Shuzhi Liu, Yong Li, Quanming Yao
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: network dynamics, symbolic regression, complex network

Abstract:
Complex networks describe important structures in nature and society, composed of nodes and the edges that connect them. The evolution of these networks is typically described by dynamics, which are labor-intensive and require expert knowledge to derive. However, because the complex network involves noisy observations from multiple trajectories of nodes, existing symbolic regression methods are either not applicable or ineffective on its dynamics. In this paper, we propose Physically Inspired Neural Dynamics Symbolic Regression (PI-NDSR), a method based on neural networks and genetic programming to automatically learn the symbolic expression of dynamics. Our method consists of two key components: a Physically Inspired Neural Dynamics (PIND) to augment and denoise trajectories through observed trajectory interpolation; and a coordinated genetic search algorithm to derive symbolic expressions. This algorithm leverages references of node dynamics and edge dynamics from neural dynamics to avoid overfitted expressions in symbolic space. We evaluate our method on synthetic datasets generated by various dynamics and real datasets on disease spreading. The results demonstrate that PI-NDSR outperforms the existing method in terms of both recovery probability and error.

Decision: reject
Reviewer scores: 6, 6, 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730449270661

thread_type:
official_review

<thread_object>
creator: reviewer_kbls

<content>
summary:
This paper introduces Physically Inspired Neural Dynamics Symbolic Regression (PI-NDSR), a technique that combines neural networks with genetic programming to autonomously discover symbolic representations of dynamics. PI-NDSR includes two main components: a Physically Inspired Neural Dynamics (PIND) module, which enhances and denoises trajectories 
through observed trajectory interpolation; and a coordinated genetic search algorithm that extracts symbolic expressions. This algorithm uses references from both node and edge dynamics provided by neural dynamics to prevent overfitting in symbolic space.

strengths:
1. The author proposes a network dynamics symbolic regression method that uses neural networks to complete data, which can help mitigate the problem of noisy data trajectories.  
2. By applying genetic programming for symbolic regression, this method avoids the need for a pre-designed knowledge base.

weaknesses:
1. The innovation of the method in the article is limited. First, the use of neural networks to fit data in order to enhance symbolic regression performance has been widely adopted in many works. For instance, in AI-Feynman[1], a neural network is first used to fit the data, and then the fitted data is analyzed to infer the nature of the equation, which facilitates subsequent symbolic regression. Similarly, this paper also uses a neural network to fit data to avoid inaccuracies in derivative estimation caused by sparse or noisy observational data. Essentially, both approaches leverage the generalization capability of neural networks to mitigate the negative impact of sparse data points. Second, using surrogate models for symbolic regression is also a common approach. For example, SymDL [2], like this paper, uses an MLP locally as a surrogate model for node dynamics G and edge dynamics F.  
2. The experiments in the article are not sufficiently comprehensive; they are limited to relatively simple dynamical scenarios and have not been applied to more complex scenarios such as neuronal, genetic, social, and coupled oscillator dynamics. Moreover, the experiments are restricted to one-dimensional data, while many other works, like TP-SINDy, can handle higher-dimensional data. For example, the FHN dynamics is a higher-order network equation, and this paper does not include experimental discussion on this equation.  


References:  
[1] Silviu-Marian Udrescu, Max Tegmark ,AI Feynman: A physics-inspired method for symbolic regression.Sci. Adv.6,eaay2631(2020).DOI:10.1126/sciadv.aay2631.   
[2] Cranmer M, Sanchez Gonzalez A, Battaglia P, et al. Discovering symbolic models from deep learning with inductive biases[J]. Advances in neural information processing systems, 2020, 33: 17429-17442.  
[3] Gao, TT., Yan, G. Autonomous inference of complex network dynamics from incomplete and noisy data. Nat Comput Sci 2, 160–168 (2022). https://doi.org/10.1038/s43588-022-00217-0.

questions:
1. How does the computational efficiency of GA compare to TP-SINDy in the symbolic regression component? Please include an appropriate discussion on this part. If the method in this paper takes too long, using a predefined library, as in TP-SINDY, remains a good option. Determining which approach is better requires evaluating the time consumption of each method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730516870979

thread_type:
official_review

<thread_object>
creator: reviewer_fzqz

<content>
summary:
This paper proposes a physically-inspired neural dynamics symbolic regression (PI-NDSR) to attempt to address the limitations of sparse regression methods for nonlinear network dynamics that require finite differences and expert design function libraries. It comprises two principal components: a physically-inspired neural dynamic model that enhances trajectories and reduces noise by observing trajectory interpolation; and a coordinated genetic search algorithm to derive symbolic expressions.

strengths:
(1)The proposed method can handle observation data with noise for inferring interpretable network dynamics.
(2)The proposed method circumvents the need for expert knowledge elaboration, necessitating only the basic operator regression symbolic model.

weaknesses:
(1)The rationality of the method design needs to be further clarified. The design and assumptions of the encoder and decoder especially need to be explained. When training the neural network, the input of the Dec in Eq. 2 is \phi^n + \phi^e, but when inferring equations, the input of the Dec becomes \phi^n or \phi^e. This means that the Dec needs to satisfy Dec(\phi^n + \phi^e)=Dec(\phi^n)+Dec(\phi^e), which is a strong assumption for the design of the decoder, thus affecting the corresponding inverse operation, i.e. the encoder, which also satisfies the corresponding assumption. There is reason to doubt the effectiveness and rationality of the encoder and decoder.
(2)The contribution of the proposed method lacks clear clarification.
(3)The experimental analysis is not comprehensive. For example, there is a lack of time cost analysis while time consumption is an important metric for algorithm performance evaluation and the states of all network dynamics evaluated in the paper are one-dimensional.

questions:
-Please clarify the design rationale for the encoder and decoder, and explain how to ensure the decoder maintains consistency between training and inference. I suggest that the authors provide empirical evidence or theoretical justification for the effectiveness of the encoder-decoder design.

-Can the method proposed in the paper recover the symbolic expressions of network dynamics when the state of nodes has multiple features? Please either provide results for multi-dimensional node states, or discuss the potential challenges and necessary modifications for extending the method to such cases.

-As described in line 749 of the appendix, Rec.Prob is computed by “simplify,” which suggests that Rec.Prob takes the value 0 or 1. Why is there a float number in Table 3? Please clarify how to calculate the Rec.Prob values in Table 3, and explain why some values are not binary if the calculation method described in the appendix is correct.

-What is the specific meaning of the "error" in line 215 and how is it calculated?

-Lack of discussion on highly relevant work. DNND [1] employs the GNN proxy model to decompose the dynamics and then uses the SINDy to obtain the symbolic expression. Please explain the difference between the method in the paper and DNND and suggest adding comparisons. Please include a comparison with DNND in experiments, or at minimum, to provide a detailed discussion of how the proposed method differs from and potentially improves upon DNND.

-Due to the method's design in this paper being similar to the graph ODE models, the proposed method appears to be able to handle irregularly-sampled data, which could have been considered an advantage but was not mentioned and validated in this paper. I suggest that the authors conduct additional experiments with irregularly-sampled data to validate this capability, or discuss why this potential advantage was not explored in the current work.

-There are some typos and unclear points as follows: 
    The inputs in Eq. (2) and Eq. (4) are inconsistent. 
    In Eq. (6) and (7), too many G's appear, and these G's seem to have different meanings.

[1] Liu, B., Luo, W., Li, G., Huang, J., Yang, B.: Do we need an encoder-decoder to model dynamical systems on networks? In: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 2178–2186 (2023)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730587330031

thread_type:
official_review

<thread_object>
creator: reviewer_cabm

<content>
summary:
The paper introduces Physically Inspired Neural Dynamics Symbolic Regression, a method for automatically discovering symbolic expressions that describe complex network dynamics. The authors present a two-part solution: a Physically Inspired Neural Dynamics component that augments and denoises trajectory data through interpolation, and a coordinated genetic search algorithm that derives symbolic expressions using references from the neural dynamics. The method addresses key limitations of existing approaches by handling noisy observations from multiple trajectories without requiring pre-defined function libraries or estimated time derivatives. The authors demonstrate PI-NDSR's effectiveness through evaluation on both synthetic datasets with various dynamics and real datasets on disease spreading, showing improvements in recovery probability and error rates compared to existing methods.

strengths:
- The model introduced in this paper is groundbreaking, offering a method designed to overcome limitations of other approaches, such as high sensitivity to noise.

- Comprehensive experiments are provided, comparing the proposed model with established methods across synthetic and real-world datasets.

- The paper is generally well-written (except for the conclusion) with a clear and linear structure that makes it easy to follow the proposed methodology, experiments, and results. The explanations of the Physically Inspired Neural Dynamics Symbolic Regression framework and the challenges it addresses are presented in a logical sequence, enhancing reader comprehension.

weaknesses:
- The evaluation of the model is somewhat lacking, particularly in the choice of metrics for synthetic and real datasets, which is not fully explained. For example, filtering out models that don’t fully recover the skeleton in synthetic datasets may overlook valuable insights into how different methods handle structural errors. Often, a method’s ability to approximate the skeleton, even imperfectly, can still yield good performance on downstream tasks.

- The conclusion is brief, as it mainly summarizes the process without emphasizing key findings. Improvements over existing methods are not highlighted, and it lacks an impact statement or discussion of broader implications. The future work section is vague; it’s unclear what is meant by “more complex systems” and “more real-world applications.” Although limitations are mentioned, it’s not evident why the method fails for highly complex formulations.

questions:
- What could be the potential issues when using the model for highly complex formulations?

- For simulated datasets, would it be better to report both filtered and unfiltered MSE, or to use a separate metric for skeleton recovery?

- It would be beneficial to expand the discussion on the limitations of the model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730641547178

thread_type:
official_review

<thread_object>
creator: reviewer_6enn

<content>
summary:
This paper introduces a Physically Inspired Neural Dynamics Symbolic Regression (PI-NDSR) method for symbolic regression of complex network dynamics. The motivation behind this research is to address the challenges of symbolic regression in complex networks, particularly the issues of noisy, sparse observations and the need to decouple node dynamics from edge dynamics. PI-NDSR features two main innovations: (1) a neural dynamics module (PIND) that interpolates and denoises the observed data to generate dense, low-noise trajectories, thereby avoiding the instability of directly modeling derivatives; and (2) a coordinated genetic search algorithm that uses neural dynamics as a reference to guide symbolic regression of node and edge dynamics separately, preventing overfitting and improving search efficiency. Experiments on synthetic and real-world epidemic data demonstrate that PI-NDSR outperforms traditional methods in recovery accuracy and error rates, showing enhanced robustness in high-noise and sparse-sampling scenarios.

strengths:
**Addresses a Novel Challenge of Data Sparsity and Noise in Symbolic Regression**: The PI-NDSR method effectively tackles the issue of sparse and noisy data in symbolic regression by introducing a neural dynamics module that performs interpolation and denoising, making it suitable for complex network data with inconsistent quality.

 **Improves Genetic Algorithm for Symbolic Regression**: By incorporating a coordinated genetic search that uses neural dynamics as a reference, PI-NDSR refines the traditional genetic algorithm approach, allowing for more accurate and efficient search of symbolic expressions while avoiding overfitting.

**Demonstrates Promising Performance**: The method shows competitive performance in both synthetic and real-world datasets, achieving higher recovery accuracy and lower error rates than baseline methods, indicating its potential as a robust solution for complex network dynamics modeling.

weaknesses:
**Lack of Novelty in Using Neural Networks for Differential Equation Interpolation**: Modeling differential equations with neural networks to achieve interpolation is not particularly novel and has been explored in previous works. This may limit the perceived innovation in the proposed approach’s foundational methodology.

**Potential Oversmoothing from Interpolation without Fully Addressing Data Sparsity**: While the method employs interpolation to handle data sparsity, it does not adequately address how to mitigate potential oversmoothing effects in the interpolated results. This could mean that the sparsity issue remains partially unresolved, as the interpolation might overlook important variations in sparse data.

**Unclear Motivation for Decoupling Node and Edge Dynamics**: The rationale for decoupling node and edge dynamics is not clearly explained. The claim that "traditional symbolic regression methods struggle to find high-quality symbolic expressions without decoupling node and edge dynamics, leading to overfitting in symbolic space" is somewhat vague and lacks a concrete justification, potentially weakening the motivation behind this design choice.

questions:
1. **Can Interpolation Truly Address Data Sparsity?**
   - The paper uses neural network-based interpolation and denoising to handle issues related to data sparsity and noise in complex network dynamics. However, interpolation fundamentally relies on existing observed data, meaning that any missing information between observation points could still lead to deviations from the true dynamics. Particularly in cases of high data sparsity, interpolation methods may produce overly smooth results, potentially missing critical dynamic variations. Thus, interpolation alone may not fully resolve the information loss caused by sparsity, as its effectiveness remains dependent on the quality and density of the original data. This could be further addressed by exploring additional complementary data sources or methods that can better capture system dynamics under sparse conditions.

2. **Unclear Motivation in the Paper’s Methodology**
   - The paper highlights challenges in symbolic regression for complex network dynamics, such as noise, data sparsity, and the need for decoupling node and edge dynamics. However, it does not fully explain the specific reasons and impacts of these challenges. In particular, the motivation for decoupling node and edge dynamics lacks sufficient detail, making it difficult for readers to understand why this decoupling is critical to obtaining high-quality symbolic expressions. Simply mentioning that "traditional methods tend to lead to overfitting in symbolic space" is somewhat vague and lacks specific examples or theoretical support. Clarifying the distinct roles of node and edge dynamics in symbolic expressions, and why decoupling is necessary for accurate representation, would enhance the paper's persuasiveness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650671457

thread_type:
official_review

<thread_object>
creator: reviewer_1ew2

<content>
summary:
This paper presents PI-NDSR, a novel method for symbolic regression of complex network dynamics. It leverages neural networks for trajectory interpolation and denoising, and combines it with a coordinated genetic search algorithm that utilizes references from the neural dynamics.

strengths:
**Novel Approach** The use of genetic programming for symbolic regression of complex network dynamics is innovative as it constrains the symbolic search space, thereby helping to mitigate overfitting. Moreover, this approach does not rely on a predefined function library.

**Improved Performance** PI-NDSR outperforms existing methods in terms of recovery probability and error, as demonstrated by experiments on various datasets.

**Robustness** By incorporating neural ODEs, the method demonstrates strong robustness to noise and can effectively handle large time intervals between observations.

weaknesses:
**Limited Novelty in Applying NeuralODE for Interpolation** The use of neural networks to model differential equations for interpolation is not particularly novel and has been addressed in prior research. With respect to the interpolation method, the paper does not introduce any methodological innovations.

questions:
(1)  Have the authors attempted to explore solutions for the missing data problem?

(2) The paper mentions that the computational cost of PI-NDSR is relatively high. Could the authors provide some suggestions for reducing this computational cost?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680627221

thread_type:
official_review

<thread_object>
creator: reviewer_m6m5

<content>
summary:
The paper introduces Physically Inspired Neural Dynamics Symbolic Regression (PI-NDSR), a new method for symbolic regression of complex network dynamics. The approach has two main components: neural dynamics, which denoises and augments observed data, and a coordinated genetic search, which leverages the previously build neural references to optimize the search for symbolic expressions. Evaluated on synthetic an real-world datasets, PI-NDSR outperforms previous methods, offering more accurate symbolic representations and robustness against noise and sampling irregularities.

strengths:
The results on these datasets show the good performance of the method, with some proof of robustness too. This applies to the real world dataset too, where the extracted dynamics of this method don't have "un-physical" properties. The ablation studies show that both parts of the algorithm are necessary for the best performance

weaknesses:
It is not clear if this method could scale to real world datasets that have more complex dynamics, since there is only a single real-world dataset.

questions:
Did I understand correctly that to train PIND you use four networks? An Encoder, a Decoder, $\phi^e$, $\phi^n$? I believe section 4.1 could be slightly expanded to show how the neural dynamics is trained, maybe supported with a figure.

I also found these typos:
In section 5.1, in "baseline": metods should be methods. "can not be aplied directly" should be "cannot be applied directly".
In Appendix A2, "NerualODE" should be "NeuralODE"
</content>
</thread_object>

</thread>
<thread>
creation date:
1732532339364

thread_type:
official_comment

<thread_object>
creator: area_chair_cfyw

<content>
comment:
Dear reviewers,

A reminder that **November, 26** is the last day to interact with the authors, before the private discussion with the area chairs. At the very least, please acknowledge having read the rebuttal (if present). If the rebuttal was satisfying, please improve your score accordingly. Finally, if you have concerns that might be solved in time, this is the last chance before moving on to the next phase.

Thanks,
The AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a framework for symbolic regression of network dynamics using graph neural networks and genetic algorithms (for extracting a symbolic representation).

The paper had a large amount of reviews (6), which scores ranging from 3 to 6. After the rebuttal phase, the most negative score improved to 5, with 2 reviewer negative on the current status of the paper. They both raise a few concerns that should be addressed. In general, there was limited interaction in the rebuttal stage, and as a result some concerns remained open.

There is a consensus that the paper's novelty is unclear, and that the design of the method requires more details (especially concerning the assumptions that are made). The authors have commented that the use of a genetic algorithm conditioned on the specific neural formulation is novel enough. All reviewers who have interacted during the rebuttal have remained mostly negative.

Due to the unaddressed concerns and the generally unsatisfactory rebuttal, I have leaned towards a negative evaluation of the paper, although I mention this is a borderline case which could have benefitted from a stronger discussion.