Structural Inference of Dynamical Systems with Conjoined State Space Models
Authors: Aoran Wang
Jun Pang
Venue: NeurIPS 2024 poster (2024)
Keywords: Structural Inference
AI4Science
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: This paper introduces SICSM, a novel structural inference framework that integrates Selective State Space Models (selective SSMs) with Generative Flow Networks (GFNs) to handle the challenges posed by dynamical systems with irregularly sampled trajectories and partial observations. 
By utilizing the robust temporal modeling capabilities of selective SSMs, our approach learns input-dependent transition functions that adapt to non-uniform time intervals, thereby enhancing the accuracy of structural inference. 
By aggregating dynamics across diverse temporal dependencies and channeling them into the GFN, the SICSM adeptly approximates the posterior distribution of the system's structure. 
This process not only enables precise inference of complex interactions within partially observed systems but also ensures the seamless integration of prior knowledge, enhancing the model’s accuracy and robustness.
Extensive evaluations on sixteen diverse datasets demonstrate that SICSM outperforms existing methods, particularly in scenarios characterized by irregular sampling and incomplete observations, which highlight its potential as a reliable tool for scientific discovery and system diagnostics in disciplines that demand precise modeling of complex interactions.

Structural Inference of Dynamical Systems with Conjoined State Space Models
Aoran Wang' & Jun Pang 1.2
Faculty of Science, Technology and Medicine, University of Luxembourg Institute for Advanced Studies, University of Luxembourg {aoran wang, jun. Bann.aang)eud @uniilu
Abstract
This paper introduces SICSM, a novel structural inference framework that integrates Selective State Space Models (selective SSMs) with Generative Flow Networks (GFNs) to handle the challenges posed by dynamical systems with irregularly sampled trajectories and partial observations. By utilizing the robust temporal modeling capabilities of selective SSMs, our approach learns input-dependent transition functions that adapt to non-uniform time intervals, thereby enhancing the accuracy structural inference. By aggregating dynamics across diverse temporal dependencies and channeling them into the GFN, the SICSM adeptly approximates the posterior distribution of the system's 's structure. This process not only enables precise inference of complex interactions within partially observed systems but also ensures the seamless integration of prior knowledge, enhancing the model's accuracy and robustness. Extensive evaluations sixteen diverse datasets demonstrate that SICSM outperforms existing methods, particularly in scenarios characterized by irregular sampling and incomplete observations, which highlight its potential as a reliable tool for scientific discovery and system diagnostics in disciplines that demand precise modeling complex interactions.
Introduction
In the complex real-world phenomena, many dynamical systems manifest as networks of interacting entities. These systems are effectively modeled as graphs where nodes represent the agents, edges depict the interactions, and the adjacency matrix captures the structural essence of these interactions. Such representations are crucial across various domains, from intricate physical systems [32 23 [5]
and multi-agen, systems [10 34] to complex biological architectures 449 44]. Unveiling the hidden structures within these networks is not only academically enriching but also essential for enhancing understanding of the systems' intrinsic mechanisms and improving manage their behaviors. However, this task becomes challenging when the observable data, often limited features of agents within specific time frames, conceals the underlying structural dynamics. This limitation necessitates robust structural inference methodologies capable of discerning the latent structures from the trajectories-the observable features of all agents over given period.
ability to predict and
As the field scientific discovery advances, particularly with the integration of neural network technologies, structural inference has emerged as key method for decoding the complex interactions
[6 54] 64). This process is essential for understanding and predicting system behaviors but encounters significant challenges, particularly when addressing irregularly sampled data and partially observed some nodes of a dynamical system, which are typified by unequal sampling time intervals and the presence of unobserved nodes. Conventional methods, including those based on Variational Autoencoders (VAEs) 30), have pioneered some paths but often struggle with datasets characterized by non-
within dynamical systems from trajectories [I] 2 57 12 39 51
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

uniform sampling rates and incomplete observations. These methods typically require uniform data and have difficulty managing indirect or obscured node interactions [31 highlighting a critical need for more adaptable and resilient inference models.
57
12
39 51 [4]
To overcome these limitations, this paper introduces novel framework, Structural Inference with Conjoined State Space Models (SICSM), which conjoins Selective State Space Models (selective
SSMs) 20 with Generative Flow Network (GFN) g This innovative approach leverages
the robust temporal modeling capabilities of selective SSMs alongside the flexible, data-driven structural inference provided by the GFN. SICSM is specifically designed to address the challenges of irregular sampling and partial observability, enhancing inference accuracy and robustness through the sophisticated integration of prior knowledge and adaptive learning mechanisms. Central to SICSM is its ability to learn seputddeenddent transition functions that dynamically adjust to the timing of data points, crucial for managing datasets with irregular intervals. Moreover, by aggregating outputs from multiple Residual Blocks containing an selective SSM in each, SICSM offers rich representation of system dynamics, enabling more precise reconstruction of node interactions within partially observed systems. Our comprehensive evaluations across variety of datasets mechanical systems like spring simulations to biological networks depicting gene expressions-demonssei SICSM's superior performance over existing methods. Its robustness shines particularly in its ability to maintain high structural inference accuracy under diverse and challenging conditions, affirming its potential an essential tool for scientific discovery and system diagnostics in disciplines that demand intricate, accurate modeling of complex systems. In essence, SICSM not only redefines approaches to structural inference in complex systems but also paves new research avenues previously limited by data sampling and observability constraints. With further refinement, this approach is poised to transform our understanding of interactions with dynamical systems across multiple scientific fields. Our contributions encompass the following aspects:
We develop novel framework, SICSM, that integrates Selective State Space Models with Generative Flow Network to enhance structural inference.
We introduce adaptive mechanisms within SICSM that effectively handle irregular sampling and partial observability, significantly enhancing the model's 's applicability to real-world datasets.
SICSM employs a novel approach by aggregating outputs from multiple Residual Blocks, which enables it to capture deeper and more detailed representation of dynamic system interactions. Extensive validation demonstrates its superiority over baselines in reconstructing complex struc- tures, especially under challenging conditions of irregular sampling and incomplete observations.
2 Related Work
Structural Inference. Structural inference aims to uncover the hidden structure of complex sys- tems using observed trajectories. A pivotal contribution in this area is Neural Relational Inference (NRI) [31], which leverages a VAE within fixed. fully connected graph framework. Building on NRI, subsequent research has expanded the domain of structural inference. Recent advancements include handling multi-interaction systems [57]. integrating efficient message-passing [12], incorpo- rating modular meta-learning [2]. iteratively pruning indirect connections [5I], developing structural inference with reservoir computing [54]. and applying deep active learning to complex systems [52]
Other techniques involve reconstructing trajectories by minimizing relation potentials [16], comput- ing partial correlation coefficients based on node embeddings [53], and with diffusion process [64]. Existing methods frequently employ the Variational Information Bottleneck principle [1] which often requires regularly sampled trajectories and complete observation of all nodes. Incorporating known interactions into VAEs necessitates complex adjustments, combining unsupervised and supervised learning in the latent space. This complexity complicates generalization to unlabeled edges.
State Space Models. State space models (SSMs) update sequences through recurrent hidden states. The selective state space architecture known as Mamba [20], recently emerged as an efficient and flexible design, using recurrent scans and a selection mechanism to control sequence flow into hidden
states. Mamba shows promise across time-series tasks [56 38 43 video analysis [35 61 37
and healthcare applications 440 45 60 In addition, several studies explore graph modeling with Mamba [55] 36, (), but none of them apply it for structural inference from observational trajectories.
In this work, we employ the fundamental operating mechanism, the selective SSM module, in the form of stacking blocks, to model observational trajectories and deal with the challenge of irregular sampling well as partial observation.

---

Generative Flow Networks. Generative flow networks (GFNs) excel in generating and sampling Recent research has explored topics such as
discrete states from high- nigh-dimnssional distributions 78
amortized inference 33 Bayesian structure learning 17 combinatorial optimization [3] biological
sequence design (77,, and broader scientific discovery 28 Work network inference 17 18
focuses on Bayesian inference to maintain system structure within state spaces. Our approach leverages Residual Blocks to learn one-dimensional embeddings from multi-dimensional features, enabling GFN to effectively learn while preserving dynamic complexity within the embeddings.
Preliminaries
3.1 Notations and Problem Formulation
We model dynamical system as a directed graph G E), representing agents nodes and interactions as directed edges. The graph consists of node feature set with n nodes and an edge set E. Node features evolve over time, forming trajectories V {V} {Vo, V1 ..... VT-1] across T time steps, where V represents the feature set at time t. Each node feature v E IR is d-dimensional. For irregularly sampled trajectories, the T time steps may have different intervals. And for partial observation of the dynamical system, we expect the count of observed nodes is smaller than the total count of nodes nhot in the system. We observe a set of M trajectories: (V1), Vigi.... Vina assuming static edge set E. An asymmetrie adjacency matrix Adj is derived from E, where Adj.j E {0. 1} indicates the presence or absence of a directed edge. In dynamical systems, the dynamics of node i at time + 1 are influenced by Adj follows:
oft1 uf
+ A Σ
f(le,,,e/ JEM;
where A is the time interval, U denotes the set of nodes connected to node i which is derived from Adj. f(-) represents the state-transition function, and -la is the a-distance.
For an illustrative example, we may consider a dynamical system comprising n = 10 balls connected by springs, representing n 10 nodes V and directed edges E, respectively. Initially, we set the positions and velocities of each ball, so that each node feature of E IR is d-dimensional where d = 4 in this example. We then let them move under the influence of spring forces, which arise from the structural connections (edges) between the balls (nodes). Over the observation period, these balls change their positions and velocities. And we record the trajectories as the collection of the evolving features of all nodes: V = {V} = {V') V VT-1 }} across T time steps, where V represents the feature set at time t. In total we observe set of M trajectories: {V1), V22.................. VINJJ, assuming a static edge set E. Suppose we initially lack knowledge of which balls are connected, i.c., E is unknown; the task of structural inference in this scenario would involve deducing the connectivity between the balls based on their observed trajectories.eepreeeneed by either the edge set E or the adjacency matrix Adj.
In this work, we introduce SICSM, a novel structural inference method with conjoined state modeling, designed to handle both irregularly sampled trajectories and systems with partial observations.
3.2 State Space Models
SSMs capture the behavior of dynamical systems by modeling the internal state and relationships between latent states ht E R-W, input sequences x. E RD and output sequences y E R. h = Ah' + Bx¹ y² = Ch², where A E RR and B, C NxD are learnable matrices. Due to the complexity of solving the above differential equation in deep learning settings, discrete state space models 2 discretize this system using time-scale parameter A: h² Aht-1 Bxx, y¹ Ch², where A- = exp AA, and B
A (AA- - I)) AB. Note that A in discrete Discrete-time SSMs are also shown be equivalent
SSMs operates similarly to that in Eqn. to the following convolution: y I K. where K = (CB, CAB.... !. BB). Therefore, the continuous form (A, A, A, C)C) transitions to the discrete form (A, C), allowing efficient computation using linear recursive approach (22. Furthermore, structured state space sequence models (S4) structures the state matrix A based on HIPPO matrices, significantly improving efficiency and performance.
I
3

---

Recently, a selective structured state space architecture named Mamba 120 was introduced. It leverages recurrent scans and a selection mechanism to control which part of the sequence flows into the hidden states. Mamba's & selection mechanism can be interpreted as using data-dependent state transition mechanisms, meaning A, B and C are functions of the input x². In this work, we utilize Mamba's selection mechanism to handle irregularly sampled trajectories by learning A from node feature set V*, enhancing the method's s ability to model system dynamics. By stacking Residual Blocks containing selective SSM modules, aggregate embedded dynamics from the outputs of different blocks, allowing us to reconstruct structures with dynamics from various temporal dependencies and address incomplete node observation.
3.3
Generative Flow Networks
are generative models operating over structured sample space X' characterized by directed acyclic graph 9 with state space S, where X' S S. It is crucial to distinguish this from the interaction graph G' of the dynamical system, and the input sequence z' in SSMs. A sample x E X' is constructed by traversing 9 from an initial state So to terminal state s,, the latter being a special state indicating the end of the sequence. Terminal states in X' are those connected by a directed edge r - &,, representing valid samples of the distribution induced by GFN. A terminal state trajectory in G is represented by a path ~ 8y, distinct from the observational trajectories used in structural inference. Each terminal state is associated with a reward R(X) 2 O, representing its unnormalized probability. The distribution over terminal states is proportional to the reward, which is governed by a flow function Pa(s + 8') 2 0 and satisfies the now-matching conditions:
GFNs 7
Σ
Fa(s +')
E
Fals' + -²) = R(s').
(2)
Pag (sr)
Psg(s') and Chg (s") represent the preceding state and the subsequent state of s', respectively. The forward transition probability is P(sk_1|s,) x Fn(ss - SK+1), leading to a marginal probability for terminating in E X proportional to R(x). Consequently, there is also a backward transition usually set to some fixed distribution (e-g. the uniform distribution over probability (.), but
the parent states) to reduce the search space, making the forward transition probability the only quantity to learn 17 8 [7 Starting from the initial state sp, we sample a terminal state trajectory (so, $1...., 5K,,, 8y) following the forward transition probability, defined as:
P(sk+1 18k) & FA(Ss 8k+1),
where SK = x and SK+1 sf, then the likelihood of state trajectory ending in state x X is directly proportional to the reward R(x). The flow function Fn - s'). often parameterized by a neural network, is optimized to minimize discrepancies in the flow-matching conditions. This optimization results in transition model capable of approximately sampling from the distribution over X' in proportion to R. In the subsequent sections, we elucidate the construction of state spaces using graphs and detail the approximation of dynamical systems" structures through posterior estimation with a GFN. We demonstrate how GFN enhances SICSM by utilizing the embedded dynamics from Residual Blocks to reconstruct the structure of dynamical systems effectively. Moreover, SICSM facilitates the seamless integration of prior knowledge about existing connections into its training process, thereby improving the model's accuracy and efficacy in structural inference.
4 Structural Inference with Conjoined State Space Models
SICSM utilizes selective SSM modules, embedding in Residual Blocks, to adaptively manage input-dependent time intervals A, thus effectively handling irregularly sampled trajectories. This model architecture further aggregates dynamic embedding from multiple Residual Blocks, obtain dynamics from various temporal dependencies, enhancing our ability to process partial observations. These dynamics are subsequently input into GFN approximate and sample the graph structures representing the system's 's structure. Figure [ provides a schematic overview of the SICSM pipeline.
4.1 Aggregation of Learned Dynamics
As illustrated in the upper row of Figure [] the primary function of the upper branch of the SICSM is time-series forecasting. These models dynamically adapt the state space parameters (A, C)

---

Overview of the Whole Pipeline
Eusnder: Ex Rosihaal Blocks
Desoder: 2/x Residaal Blocks
Residual Block
Residual Block
Residual Block
based Embedding
Block
Projection
Residual Block
Adl
Ge
Convil
SOM
12
All
Figure 1:
(Upper) System architecture. (Lower Left) Detail of a Residual Block. (Lower Right) Structure of the Generative Flow Network for approximating the joint posterior distribution.
based on the node-specific feature embeddings. Inspired by recent advancements in dimensionality reduction and noise reduction in mixed-node feature datasets [56), we introduce a feature-based embedding network designed to compress the feature dimension from d to 1:
h Lemtee (v]), for E 20
where fembed is multi-layer perceptron. h sequentially organized per node to form H. [h. h ...; h.- The node dynamics are modeled by a series of Residual Blocks configured in an encoder-decoder structure. Each Residual Block incorporates a selective SSM module that dynamically learns the SSM parameters (A, B, C) based on the embeddings H; for each node:
(5) where /SSMproj a linear projection layer specific to each selective SSM, as described 20]. This allows each node's input-dependent step-size A adjust dynamically, enhancing the model's ability to handle irregularly sampled trajectories and reflect flexible time intervals as specified in Equ. Furthermore, the matrices B and C are tailored for each node, updating node features over time and accommodating the unique dynamics of each node. More details on the selection SSM in this work can be found in Appendix A
(A,, B, C,) fSSMdproj (H:),
To enhance the architectural sophistication of our model, we arrange Residual Blocks a sequential configuration (to build a residual model), with the output of each block feeding directly into the next. This design significantly improves the model's ability to discern and interpret diverse features and aspects of the trajectory, facilitating the capture of broad spectrum of temporal dependencies. The input to the first Residual Block is obtained by processing the concatenated node embeddings Hau H, for all nodes]:
URB_I= = S (Hal),
(6)
where fRB_I denotes the first Residual Block function. The 1-th Residual Block has: URBLL (iB_((UR___1)) This setup is particularly vital in systems with partial observability, where direct connections may not be visible. In such scenarios, observable nodes may appear isolated but frequently connected through hidden intermediaries, converting straightforward interactions into intricate multi-hop relationships. SICSM accommodates this complexity by integrating dynamics across a spectrum of temporal dependencies: shorter dependencies help reconstruct direct interactions, while longer dependencies are erucial for mapping multi-hop relationships. Unlike traditional methods that operate under fixed time intervals and predetermined direct interactions [31] 57 2
39 [1] [54),
which struggle with variable conditions, our SICSM's flexibility in adapting to different hop distances is essential. This adaptability enables it to accurately delincate potential indirect interactions and therefore to deal with incomplete observation.
Further enhancing our model, we implement an encoder-decoder structure composed of an additional L' Residual Blocks. This configuration not only maintains the model's symmetry but also boosts its accuracy. The outputs from these blocks undergo transformation via a projection network, which restores the features to their original d-dimensional state, preparing estimated node features for the subsequent time step, of Based on the system is fully observed not, we have:
5

---

if we observe all nodes, |E URB_ if observe partial nodes,
URBLL
UAu
(7)
URB_I 1 represents outputs from 1-th block. The aggregation of outputs from multiple blocks crucial, especially in scenarios with partial observations typically caused by non-visible intermediate nodes. This aggregation ensures that dynamics of various temporal dependencies are comprehensively captured, enabling SICSM to deal with the partial observation.
4.2 Approximation of Posterior with a Generative Flow Network
Upon acquiring the aggregated dynamics UAn from either the output of the final Residual Block the summation of outputs from all Residual Blocks, we feed these dynamics to the following GFN. We utilize GFN to model the posterior distribution P(Adj U UAu), where Adj delineates the structure of the dynamical system under study. While any GFN capable of modeling the structure of graphs could be employed, we specifically choose the Joint Structure- Partumeers GFN (ISP-GFN) [8 for its ability to capture the diverse dynamics of each node influenced by their interactions. This model effectively addresses the joint posterior distribution P(Adj. X UAu), where À {AI.... .a...... .n represents the parameters for conditional probability distributions associated with each node i, enhancing the accuracy and depth of structural inference by accommodating the unique characteristics and relationships of each node.
As depicted in the lower right of Fig. the state construction process begins with initial state containing a graph with an empty adjacency matrix Go = (UAu, Adjo), which progressively evolves by systematically adding edges to Adj based on the forward transition probability Pa(G"IG), G)), where G" is the resultant graph state. This iterative addition continues until a 'stop' action is chosen, signifying the completion of the graph construction phase. Once the graph G is established, we proceed to generate the parameter set X, conditioned on G, utilizing the forward transition probabilities (AG)). Each terminal state (G. X) thus encapsulates a potential configuration of the system, with the construction process forming tree structure rooted at Go-
To approximate the joint posterior distribution P(Adj, A(UAu) which is proportional to P(Adj, UAu)
18 we define a reward function for each terminal state:
(8) where P(UAu|A. Adj) represents the likelihood model implemented via a neural network that operates on each node, Ad)) denotes the prior over the parameter set, and P(Adj) constitutes the general prior over structures. This reward function integrates the likelihood of the observational data with the parameter and graph priors, directing the learning towards accurate structural inference. To derive the adjacency matrix for the system, we approximate the marginal posterior P(Adj]UAn An) by collecting samples { [Adj Adjz.... Adj } from the posterior distribution and estimate the marginal probability of edge from node to node j as:
R((G, = Adj P(AA))())))))
B 1ii Adj.),
Pa(i + j(UAA) ~ B
edjj),
where 1(.) is the indicator function. The collected value Pn is the approximation of structure of the dynamical system. This methodology ensures robust inference of the structure of the dynamical system. For details on the specific GFN in SICSM, please refer to Appendix B
This approach of modeling dynamical systems using a conjoined state space model framework integrates observational trajectory modeling via selective SSMs with posterior distribution modeling via a GFN, providing a comprehensive method for structural inference in complex systems.
4.3 Reward Function
The reward function in SICSM, as defined in Eqn. 8 comprises three key components: the likelihood model A\ G), the prior over parameters P G). and the prior over graphs P(G). Consistent with standard practices in GFNs 87, we utilize a logarithmic transformation of the reward function:
log R(, A)) log Au A, Adj) + log P()[Adj) log ((dj)),
(10)

---

with component being implemented distinctly:
Likelihood Model. The first term of the reward function log-likelihood model that estimates prediction errors for future node features, considering the current graph structure Adj in each state. This approach aligns naturally with the evolution of dynamical systems:
T-2 Σ log P Adj, U t=0
log P(UAulA, Adj)
(11)
where U. is the learned embeddings of node at time 1 and is obtained from UAu- log IA, Adj, UT) is modeled with neural network to enable accurate predictions of future node features. This setup integrates both node features and graph structure into the computation, ensuring their collective influence the reward function.
The second term of the reward function represents the prior over the parameters X. We utilize unit Normal distribution for each parameter Asy. correlating with the sender and receiver nodes indexed by i and j: P():g) Adj) (O, 1). This choice of prior contributes a balanced modeling of node interactions within the graph.
Parameter Prior.
Graph Prior.
The component of the graph prior comprises uniform prior alongside regularization terms designed to enhance the graph's structural smoothness. In reference to the current graph structure Adj. is formulated as:
P(Adj) = (Adj) + exp (D(Adj,U An) + La(Adj) + (oddi))
(12)
where Pr-(Aj) is the uniform prior. The regularization terms include Dirichlet energy D(Adj, Uau) to measure smoothness between adjacent node features, a connectivity term (A(d)) to penalize unconnected structures, and a sparsity term (Adj) regulate graph density:
D(Adj, UAu)
Σ Adi is ll - U,||3, n
1
(13)
Ca(Adj) 1 1 log(Adj1), and (Adj) 1 n
|djil
(14)
n
These terms, adapted for the reward function in SICSM, emphasize the influence of the graph's properties the state space, enriching the model's structural inference capability.
4.4 Learning Objectives
The learning objectives of SICSM are bifurcated into two main components: (1) time-series forecast- ing using Residual Blocks, and (2) modeling dynamics with the GFNs, focusing the accuracy of transition probabilities. For the Residual Blocks, the primary learning objective is the minimization of the Mean Squared Error between predicted and actual node features across all time steps and nodes:
T-2
CRB
-
1 TXn 1=0
1 - - 11
(15)
where of.t1 and o/ represent the predicted and actual features of node i at time + 1, respectively. This objective ensures that the Residual Blocks effectively capture and forecast the dynamics. Similar to [17 188. the objective for the GFN involves optimizing the squared error of the logarithmic ratio between forward and backward transition probabilities to ensure accurate modeling of the graph structure:
LGFN = E, [1OE (10 AYPPA 27
RRCCC,A)RRRCCCCCCAAAAAAA )
=GEN=EEr
(16)
where (n(() indicates the backward transition probability, and X' denotes the parameters generated conditional on graph G'. The sampling distribution covers pairs (G, and (G', with the 'stop-gradient" operation (A) critical for halting backpropagation through the parameters and X', thus preventing potential feedback loops during training. Please refer to Appendix B for the
7

---

Springs Simutations
NetSime
LI
LL
CY
BP
TF
BV-CV
30
7imm 5ape
Time Aeps
Thec 3429
Time Stps
Tiae Serg
Thac 5ACH
Time 5tge
VN SP
VN. SP
VN. SP
VN. SP. 100
VN. 15
VN.
VS.NS. 50
100
31
30 31
30
T3mh 512e
515T6
Stge
Figure 2: AUROC values (expressed in percentage) for various methods as function of the number of irregularly sampled time steps. Results are averaged across ten trials, with time steps varying from 49 to 10. All subplots share common x-axis and y-axis for uniform comparison.
parameterization of these terms. The final learning objective of SICSM combines these terms:
CRB + LGFN-
(17)
This combined objective facilitates concurrent optimization both time-series prediction accuracy and the fidelity of the inferred graph structure.
4.5 Integration of Prior Knowledge
In SICSM, the integration of prior knowledge concerning existing network connections is executed more seamlessly and effectively compared to traditional VAE-based methods [31 39 51 54 53 To incorporate this prior knowledge, we initialize the graph Go in the GFN's initial state with edges that represent the known connections based on prior knowledge: Go = (V. Adjo U F&), where Eş contains the of known edges based on prior knowledge. This setup ensures that the learning and sampling processes are continually influenced by this integrated knowledge, enhancing the model's accuracy and effectiveness in predicting and understanding the underlying dynamics of the system.
Experimental Results
This study systematically evaluates the performance of SICSM across an extensive array of datasets, which encompass both one-dimensional and multi-dimensional trajectories. The investigation specifi- cally concentrates on challenging scenarios of irregularly sampled trajectories and partial observations. More results and methodological specifics further elaborated in Appendix E
5.1 General Settings
Datasets. Our study first evaluates the SICSM model two established structural inference datasets: the Spring Simulations dataset [31], which simulates dynamic interactions balls connected by springs within symmetric setting, and the NetSim dataset [47 which consists of simulated blood- lrygen-level-deeendeee imaging data from various brain regions in an asymmetric network. Both datasets include 10 nodes, with Spring Simulations offering four-dimensional features and NetSim one-dimensional features at each timestep, initially sampled at 49 regular intervals.
Additionally, examined six directed synthetic biological networks (Linear, Linear Long, Cycle, Bifurcating, Trifurcating, and Bifurcating Converging) as outlined in [4), with abbreviations LI, LL, CY, BF, TF and BF-CV, respectively. These networks simulate developmental trajectories in differentiating cells using BoolODE 44), capturing oie-dimensi mRNA expression levels over 49 timesteps with irregular intervals tailored to experimental setups.
We also incorporated data from the StructInfer Benchmark [3] focusing on 'Vascular Networks' (VN) with node counts ranging from 15 to 100. These datasets, named under the categories Springs (SP) and NetSims (NS), were selected for their complex and varying underlying graph structures, providing robust platform to validate the efficacy of the SICSM model.

---

Baselines and metrics. To evaluate the performance of SICSM. we compared it against a suite of
ofate-of-th--ar models: NRI [T. MPM 12] ACD 39 iSIDG [TT RCSI [54) JSP-GFN [18].
CUTS [3]. and SIDEC 53 The comparative effectiveness of these methods was quantitatively assessed using the area under the receiver operating characteristic (AUROC) eurve, focusing on the accuracy of the inferred adjacency matrix relative to the ground truth.
Experimental settings. All experiments were conducted on single NVIDIA Ampere 40GB HBM graphics card, paired with 2 AMD Rome CPUs (32 cores @ 2.35 GHz). Detailed configurations and additional results are elaborated Appendix D and Appendix
5.2 Experimental Results with Irregularly Sampled Trajectories
This section examines the performance of the evaluated methods to irregular sampling of input trajectories. Detailed in Section [51 our datasets undergo randomized reduction in time steps to [40, 20, 10] from an original count of 49. The baselines, alongside SICSM, are then trained and evaluated on these irregularly sampled trajectories, with the average AUROC results of 10 runs depicted in Figure] 2 Besides, we report the AUPRC results, SHD values and Fl Fi-scores in Figues 7 Appendix [.. It should be noted that JSP-GEN, being limited to one-dimensional feature analysis, not applicable multi- Iulti-dimennional datasets such as Springs Simulations and VN SP.
SICSM exhibits exceptional consistency in its performance despite the decrease in time steps. which is critical indicator of robustness within structural inference models. In datasets like Spring Simulations and NetSim, while the baseline models show significant declines in performance from 49 to 10 time steps, SICSM maintains AUROC scores above 85%. This underscores its potent capability to effectively leverage essential structural information, even when data availability is constrained. Moreover, when faced with irregular sampling, traditional VAE-based methods such as NRI, ACD, MPM, iSIDG, and RCSI struggle significantly, often performing no better than random guessing. This decline is primarily attributed to their dependency on fixed time intervals between observations-an assumption not held our experimental conditions. In contrast, in challenging synthetic biological networks like BF and TF, SICSM consistently surpasses baseline models by margins of 5-10 across all sampling levels, affirming its sophisticated understanding of complex, directional interactions which are crucial in genomics and systems biology.
SICSM's robustness to irregular sampling intervals particularly notable in datasets such as VN_SP NN_S_30 and VN_SP_50. Unlike conventional models that falter under variable data availabil- ity, SICSM's architecture, equipped with adaptive time-interval handling, adeptly navigates these challenges, preserving its predictive accuracy. These findings validate the effectiveness of SICSM in managing complex, temporally variant structural inference challenges across a spectrum of demand- ing datasets. The model not only demonstrates resilience to data scarcity and irregular sampling, but also excels in capturing intricate systemic interactions.
5.3 Experimental Results with Incomplete Observation of Systems
This section explores the resilience of evaluated methods to scenarios where only a subset of the system's 's nodes is observable. As outlined in Sec- tion 5.1 the datasets undergo reduc- tion in node count by sampling from all nodes, scaled proportions of [100%, 90%, 80%, 70%, 60%], with rounding up ensure integer counts. This experimental setup was applied particularly to datasets with more than 10 sadds--secifically the LL and all VN datasets to facilitate com- prehensive investigation. The perfor- mance of the baselines, alongside our SICSM model, is quantified using the average AUROC from 20 runs, depicted in Figure 3
VX KS.
38 25
VL 35
25 100
Figure 3: AUROC values (expressed in percentage) for var- ious methods as a function of the proportion of observed nodes, averaged over 20 trials. Node sampling proportions are set at [100%, 90%, 80%, 70,, 60%).

---

SICSM demonstrates remarkable stability in AUROC scores across different levels of node sampling, particularly excelling in environments with partial observations. In the VN SP 30 dataset, it main- tains an AUROC above 80%, even with only 60% of nodes observed, significantly outperforming methods like NRI and ACD, whose performance dips below 75%. This highlights SICSM's ability to effectively utilize essential structural relationships under partial observations. In complex network structures like the VN_NS series, SICSM's strong performance underscores its proficiency in inferring critical interactions, despite considerable reductions in observable nodes. This robustness showcases the model's s ability for managing data sparsity and leveraging available information effectively.
SIDEC also shows competitive performance, underscoring the benefits of dynamics-encoding models in handling incomplete observations. However, without an adaptive transition dynamic function, SIDEC generally underperiorms compared to SICSM, especially when fewer nodes are observed. In comparisons, SICSM consistently outshines ISP-GEN at lower node sampling percentages, illustrating its superior capability in managing partial observations and effectively using structural information even with limited data visibility. These results confirm SICSM's robustness in structural inference, particularly in scenarios with incomplete observations. Its resilience to node sparsity and ability to discern complex interactions make a valuable tool for applications that require reliable, accurate structural predictions in data-constrained environments.
5.4
Why Do We Need All Residual Outputs?
Semple Caac
Raxu from SSCM
Reauta fro BGMM
Legand
All Nodks
Figure 4: (1st Column) Ground truth structure with all nodes. (2nd Column) One example of 12-node sampling. (3rd, 4th Columns) Structural inference results from SICSM-one and SICSM.
As discussed in Section 3.2 systems with partial observations can benefit from aggregating dynamies from multiple Residual Blocks for GFN input. We explored this by analyzing the use of only the dynamics from the final Residual Block of encoder in case study using the VN_SP _P___ 15 dataset, where 12 nodes (80% of the total) are sampled. We evaluated two configurations: the comprehensive SICSM, integrating outputs from all Blocks in encoder, and SICSM-one, which relies solely on the final Block's 's output. Figure 4 shows that SICSM-one often inaccurately classified two-hop interactions as three-hop connections, leading to increased false positives due to its dependence on the output from the larger, final Residual Block, which tends to blur hop distinctions. In contrast, SICSM's approach of using outputs from multiple layers provided a rich dynamics representation that effectively managed both shorter and longer connections, reducing wrong results. Despite these strengths, some inaccuracies point to the potential need for an adaptive weighting mechanism that adjusts the influence of dynamics based the graph's size and longest paths, potentially improving accuracy across different scenarios. These results, along with an additional case in Appendix E.4 highlight the effectiveness of SICSM's 's multi-layer dynamic integration in handling partial observations and its capability for precise structural inference in complex settings.
Conclusion
This paper presents SICSM, a novel structural inference approach that merges Selective State Space Models with Generative Flow Networks. By embedding dynamics with Residual Blocks, our method learns nepudddepnneent transition parameters, effectively handling irregularly sampled trajectories. Aggregating outputs from multiple blocks enriches the dynamics captured, addressing the significant challenge of incomplete node observations. The downstream Generative Flow Network, leveraging these dynamics, achieves precise structural inference and seamlessly incorporates prior knowledge. Empirical evidence demonstrates SICSM's effectiveness, particularly in settings with irregular sampling and partial observations. Future research will explore specific adaptations of conjoined state space models for dynamic systems with mutable structural elements, such as evolving connections and emerging nodes. Additionally, we aim explore the development of a comprehensive model capable of pioneering new paths in general scientific discovery.
10

---

References
[1] A. A. Alemi, I. Fischer, V. Dillon, and K. Murphy. Deep variational information bottleneck. In Proceedings of the 5th International Conference on Learning Representations (ICIR), 2017.
[2] F. Alet, F. Weng. T. Lozano-Pérez, and L. P. Kaelbling. Neural relational inference with fast modular meta-learning. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019.
[3] A. Anonymous. Benchmarking structural inference methods for dynamical interacting systems. https://structiinfe geierrgggttuu github. 1 2023.
[4] L. Atanackovic, A. Tong, B. WANG, L. J. Lee, Y. Bengio, and J. Hartford. DynGFN: Towards bayesian inference of gene regulatory networks with GFlownets. In Advances in Neural Information Processing Systems 36 (NeurlPS), 2023.
[5] W. Battaglia, J. Hamrick, V. Bapst, A.
V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1806,008000.11 2018.
[6] A. Behrouz and F. Hashemi. Graph mamba: Towards learning graphs with state space models. arXiv preprint wwXiv:2402.086777 2024.
[7] E. Bengio, M. Jain, M. Korablyov, D. Precup, and Y. Bengio. Flow network based generative models for non-iterative diverse candidate generation. In Advances in Neural Information Processing Systems 34 (NeurIPS), pages 27381-27394, 2021.
[8] Y. Bengio, S. Lahlou, T. Deleu, E. J. Hu, M. Tiwari, and E. Bengio. Gflownet foundations. Journal of Machine Learning Research, 24(210):1555 2023.
[9] Bradbury, R. Frostig, P. Hawkins, J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke,
J. VanderPlas, S. Wanderman-Milne and Q. Zhang. JAX: composable transformations of Python+NummPy programs, 2018. //tt:::::::tiiie com. google/ Version 0.3.13.
[10] G. Brasó and L. Leal-Taixé. Learning neural solver for multiple object tracking. In Proceedings of the IEFE/CVV Conference on Computer Vision and Pattern Recognition (CVPR), pages 6247- 6257, 2020.
[11] C. Chen, K. Petty, A. Skabardonis, P. Varaiya, and Z. Jia. Freeway performance measurement system: mining loop detector data. Transportation Research Record, 1748(1):96-11)) 2001.
[12] S. Chen, J. Wang, and G. Li. Neural relational inference with efficient message passing mechanisms, In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI). pages 7055-7063, 2021.
[13] Y. Cheng, R. Yang. T. Xiao, Z. Li, J. Suo, K. He, and Q. Dai. CUTS: Neural causal discovery from irregular time-series data. In Proceedings of the 1 11th International Conference Learning Representations (ICLR), 2023.
[14] W. Chiang, X. Liu, S. Si, Li, S. Bengio, and C. Hsich. Cluster-gen: gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD, pages
257-266. ACM, 2019.
[15] K. Cho. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1409. 1259, 2014.
[16] A. Comas, Y. Du, C. F. Lopez, S. Ghimire, M. Sznaier, J. B. Tenenbaum, and O. Camps. Inferring relational potentials in interacting systems. In Proceedings of the 40th International Conference on Machine Learning (ICML), pages 6364-6383. PMLR, 2023.
[17] T. Deleu, A. Góis, C. Emezue, M. Rankawat, Lacoste-Julien, Bauer, and Y. Bengio. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelli- gence, pages 518-528. PMLR, 2022.
[18] T. Deleu, M. Nishikawa- Toomey, J. Subramanian, N. Malkin, L. Charlin, and Y. Bengio. Joint bayesian inference of graphical structure and parameters with single generative flow network. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.
[19] J. Godwin, T. Keck, P. Battaglia, V. Bapst, T. Kipf, Y. Li, K. Stachenfeld, P. Veličković, and A. Sanchez- Jraph: A library for graph neural networks in jax., 2020. URL. http://githii com/ dwwpmindd jraph Version L.dev.
11

---

[20] A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00772, 2023.
[21] A. Gu, T.. Dao, S. Ermon, A. Rudra, and C. Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems 33 (NeurIPS),
pages 1474-1487, 2020.
[22] A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. In Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022.
[23] S. Ha and H. Jeong. Unraveling hidden interactions in complex systems with deep learning. Scientific Reports, 11(1):1-13, 2021.
[24] T. Hennigan, T. Cai, T. Norman, L. Martens, and I. Babuschkin. Haiku: Sonnet for JAX, 2020. dm- http://p://githtu http:hhbbcooooeeooooooooooeeooeo hww-pmiiiinddd-dakii Version 0.0.10.
[25] S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):
1735-1780, 11 1997. ISSN 0899-7667. doi: 10. w166/neco.1997.999 199..9.8.777.33.
[26] W. Huang, G. Wan, M. Ye, and B. Du. Federated graph semantic and structural learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI), pages 3830-3838, 2023.
[27] Jain, E. Bengio, A. Hernández- Gareia,, J. Rector-Brooks, B. F. P. Dossou, C. A. Ekbote, J. Fu, T. Zhang, M. Kilgour, D. Zhang, L. Simine, P. Das, and Y. Bengio. Biological sequence design with gflownets. In Proceedings of the 39th International Conference on Machine Learning (ICML). pages 9786-9801. PMLR, 2022.
[28] M. Jain, T. Deleu, Hartford, C.-H. Liu, A. Hernandez-Garcia, and Y. Bengio. Gflownets for ai-driven scientific discovery. Digital Discovery, 2(3):557-577, 2023.
[29] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint
14Xiv:1412.690 129.0,0.,. 2014.
[30] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:
2013.
[31] T. Kipf, E. Fetaya, K.-C. Wang, M. Welling, and R. Zemel. Neural relational inference for interacting systems. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 2688-2697, PMLR, 2018.
[32] J. Kwapień and S. Droadž. Physical approach to complex systems. Physics Reports, 515(3):
115 115-226,220 2012.
[33] S. Lahlou, T. Deleu, P. Lemos, D. Zhang, A. Volokhova, A. Hernández-Garee L. N. Ezzine, Y. Bengio, and N. Malkin. A theory of continuous generative flow networks. In Proceedings of the 40th International Conference on Machine Learning (ICML), pages 18269-18300, PMLR, 2023.
[34] J. Li, H. Ma, 7. Zhang, J. Li, and M. Tomizuka. Spatio-temporal graph dual-attention network for multi-agent prediction and tracking. IEEE Transactions on Intelligent Transportation Systems, 1055)1005561100 10569, 2022.
[35] K. Li, X. Li, Y. Wang, Y. He, Wang, L. Wang, and Y. Qiao. Vidcomamba: State space model for efficient video understanding. arXiv preprint 1rXiv:2403300697, 2024.
[36] L. Li, H. Wang, W. Zhang, and A. Coster. Stg-mamba: Spatial-temporal graph learning via selective state space model. arXiv preprint arXiv:2403.1244 2024.
[37] W. Li, X. Hong, and X. Fan. Spikemba: Multi-modal spiking saliency mamba for temporal video grounding. arXiv preprint arXiv:2404.0/ 2024.
[38] A. Liang, X. Jiang, Y. Sun, and C. Lu. Bi-mamba4ts: Bidirectional mamba for time series forecasting, arXiv preprint wxXiv:244411777 2024.
[39] S. Löwe, D. Madras, R. Z. Shilling, and M. Welling. Amortized causal discovery: Learning to infer causal graphs from time-series data. In Proceedings of the 1st Conference on Causal Learning and Reasoning (CleaR), pages 509-525. PMIR, 2022.
[40] Ma, F. Li, and B. Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arX/vv2401.04722, 2024.
12

---

[41] K. Madan, J. Rector-Brooks M. Korablyov, E. Bengio, M. Jain, A. C. Nica, T. Bose, Y. Bengio, and N. Malkin. Learning gflownets from partial episodes for improved convergence and stability. In Proceedings of the 40th International Conference Machine Learning (ICML), pages 23467-23483, PMLR, 2023.
[42] N. Malkin, M. Jain, E. Bengio, C. Sun, and Y. Bengio. Trajectory balance: Improved credit
assignment in gflownets. In Advances in Neural Information Processing Systems 35 (NeurIPS),
pages 5955-5967, 2022.
[43] B. N. Patro and V. S. Agneeswaran. Simba: Simplified mamba-based architecture for vision and multivariate time series. arXiv preprint arXiv:2403. 15360, 2024.
[44] A. Pratapa, A. P. Jalihal, J. N. Law, A. Bharadwaj, and T. Murali. Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data. Nature Methods, 17(2): 147-154, 2020.
[45] J. Ruan and S. Xiang. Vm-unet: Vision mamba unet for medical image segmentation. arXiv preprint (xXi::2402.0249), 2024.
[46] S. M. Smith, K. L. Miller, G. Salimi-Khorshidi, M. Webster, C. F. Beckmann, T. E. Nichols,
J. D. Ramsey, and M. W. Woolrich. Network modelling methods for FMRI. Neuroimage, 54(2):
875-891, 2011.
[47] S. M. Smith, K. L. Miller, G. Salimi- Khorshidd, M. Webster, C. F. Beckmann, T. E. Nichols,
J. D. Ramsey, and M. W. Woolrich. Network modelling methods for FMRI. Neuroimage, 54(2):
875-891, 2011.
[48] C. Song, Y. Lin, S. Guo, and H. Wan. Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 34, pages 914-921, 2020.
[49] M. Tsubaki, K. Tomii, and J. Sese. Compound proteen interaction prediction with end-to-end learning of neural networks for graphs and sequences. Bioinformatics. 35(2):309-311 -18, 2019.
[50] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems
30 (NIPS). pages 5998-666 6008. 2017.
[51] A. Wang and J. Pang. Iterative structural inference of directed graphs. In Advances in Neural Information Processing Systems 35 (NeurIPS), 2022.
[52] A. Wang and J. Pang. Active learning based structural inference. In Proceedings of the 40th International Conference on Machine Learning (ICML), pages 36224-36245. PMLR, 2023.
[53] A. Wang and J. Pang- Structural inference with dynamics encoding and partial correlation coefficients. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024.
[54] A. Wang, T. Tong, and J. Pang. Effective and efficient structural inference with reservoir computing. In Proceedings of the 40th International Conference on Machine Learning (ICML), pages 36391-36410, PMLR, 2023.
[55] C. Wang, O. Tsepa, J. Ma. and B. Wang. Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprint arXiv:2402.00 00000.0.......... 2024.
[56] Z. Wang, F. Kong, S. Feng, M. Wang. H. Zhao, D. Wang, and Y. Zhang. Is mamba effecttve for time series forecasting? arXiv preprint 2rxiv:2403.11144 1144 2024.
[57] E. Webb, B. Day, H. Andres-Terre, and P. Lió. Factorised neural relational inference for multi-interaction systems. arXiv preprints
[58] E. Webb, Day, H. Andres-Terre, and Lió. Factorised neural relational inference for multi-interaction systems. arXiv preprint
2019.
[59] H. Wu, Y. Liang, W. Xiong, Z. Zhou, W. Huang, S. Wang, and K. Wang. Earthfarsser: Versatile spatio-temporal dynamical systems modeling one model. In Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI), pages 15906-15914, 2024.
[60] Z. Xing, T. Ye, Y. Yang, G. Liu, and L. Zhu. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint 24X1 2024. [61] Y. Yang, Z.. Xing, and L.. Zhu. Vivim: video vision mamba for medical video object segmenta-
tion. arXiv preprint ar Xxxi:22401.14166 2024.
13

---

[62] H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. K. Prasanna. Graphsaint: Graph sampling based inductive learning method. In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
[63] D. W. Zhang, C. Rainone, M. Peschl, and R. Bondesan. Robust scheduling with gflownets. In Proceedings of the 11th International Conference Learning Reprexentations, (ICLR), 2023.
[64] S. Zheng, Z. Li, K. Fujiwara, and G. Tanaka. Diffusion model for relational inference. arXiv preprint 17X1v 2024.
14

---

Appendix of Structural Inference of Dynamical Systems with Conjoined State Space Models
A More Details on Selective SSM in SICSM
Selective SSM
B
C
%
DeckbN
Project
Selecton Machasriim
Figure 5: The overview of Selective SSM in Residual Blocks.
In this section, delve deeper into the capabilities of the Selective SSM, which is central to SICSM framework, enabling it to learn input-dependent time intervals, denoted as A, with enhanced adaptability. As illustrated in Figure [ the selective SSM processes time-series data for each node, represented by x;. For each node and at each time step f, the projection layer of the selective SSM dynamically learns the transition parameters B C and A This capability not only provides the flexibility needed to adapt to varying time intervals but also significantly enriches the model's ability capture the nuanced dynamics of each node
time.
Contrasting with previous models that often rely on static less adaptable transition parameters [1 58 39| 51]. our selective SSM design allows SICSM adjust its learning mechanism based on the input data's temporal characteristics. This ficxibility crucial for effcctively modeling the transitions in dynamics, particularly when dealing with irregularly sampled trajectories. By enabling the selective SSM to adapt its parameters dynamically, SICSM can more accurately reflect the evolving dynamics inherent in complex systems, thus providing robust framework for predicting changes and interactions within these systems under varying observational conditions.
B More Details on GEN in SICSM
The discussion in this section is addition to the description of GFN used in SICSM (in Section 4.2 with an illustration of the GFN shown in Figure
B.1 More Details on Flow-matching Conditions
We first provide more introduction on general GFNs. GFNs, originally conceptualized through the flow- now-matching conditions proposed by Bengio et al. [7], have seen the development of alternative
15

---

GFlowNet
G₁
Add 5-0
Adj
Aggregating posterior
Go
approximation
A
PaPalfal
REG Ap))
Figure 6: Structure of GFN in SICSM. Each state 8 consists of graph structure G of the underlying interaction graph of the dynamical system, and generated parameter A. The initial state so the completely disconnected graph. Each state is complete and connected to terminal state s and associated to reward R( R[G A)). Transitioning from one state another corresponds to adding directed edge to the graph.
conditions that ensure equivalent guarantees. These conditions ensure that GFN satisfying them would sample complete states in proportion to their associated rewards.
One such alternative is the detailed balance conditions (DB), derived from Markov chain theory, as discussed by Bengio et 8 These conditions are defined for any transition - within the GFN as:
(18)
where F(s) represents flow function, which can be parameterized by neural network. Bengio et al. 8 demonstrated that adherence to these detailed balance conditions across transitions s - s' in the GFN ensures that the resulting distribution is proportional to the reward R(s). In situations where all states in the GFN are complete, Deleu et 17 adapted these conditions eliminate the need for separate flow function.
An alternative of conditions, known as trajectory balance conditions (TB), introduced by Malkin et al. [2). These conditions apply at the level of complete trajectories rather than individual transitions. For complete trajectory (Su, $1, 8T,8)). 84), the trajectory balance condition is formulated as:
T Z II PF (8 = R(ST) II P3(s118441, t=1
T-1
(19)
1=1
with the convention 8T+1 8y. and where 7 is the partition function of the distribution (i.e., = E.ex R(x)); practice, Z learnable parameter the model that is being learned alongside the forward and backward transition probabilities. Compliance with the trajectory balance conditions across all complete trajectories ensures that the induced distribution by the GFN is proportional to the reward R(s).
B.2 Learning Objective of GFN
The GFN of the SICSM learning framework adopts the Subtrajectory Balance conditions (SubTB) [2), critical concept for ensuring the balance of flow in GFNs, which is more general and relaxed condition than DB. Instead of enforcing balance at each state transition, SubTB focuses on balancing the probability mass over entire subtrajectories of the generative process. A detailed overview of SubTB is provided Appendix B.3 The GFN in SICSM characterized by a dual struc- ture. comprising both the interaction graph G and the node-specific parameters X. To accommodate
16

---

this unique structure, we adopt modified form of SubTB as proposed in [8] R((G", (T)G",")))P(((GGG))R
= R((G,NN)P((G)G))RNNN
(20)
where (-( represents the backward transition probability, and X denotes the parameters generated conditional on graph G'. This reformulation ensures that both the graph G (including the aggregated node dynamics embeddings UAu well as the structure Adj) and the parameters À are integral to the reward function, thereby reinforcing their importance in structural inference tasks. We show in Appendix E.2 with experimental results that this set up greatly encourages the successful integration of prior knowledge on existing edges. Additional details on this reformulation are available in Appendix B.4
For SICSM, we define the learning objective as the squared error of the log ratio between forward and backward transitions, in line with the approach in [18]. Specifically, the learning objective is given by:
LGFN ET [(LR
RIGG', (O)AACGGG) 2 ARCCC,A)AA
(21)
where π is sampling distribution over pairs (G, A) and (G', X') and A denotes the 'stop-gradient" operation. This operation is crucial to prevent backpropagation through À and X', thereby avoiding potential infinite loops.
B.3 More details about SubTB
The concept of subtrajectory balance conditions, introduced by Malkin et al. 42). serves generalization of both detailed balance and trajectory balance conditions, extending their application to partial trajectories of varying lengths. These conditions defined for partial state trajectory
т (Sm; +m 8n) as follows:
n-1 F(sm) II Pr(s441(st) F(s.) II PB(Se(s441);
n-
(22)
where (s) denotes a flow function. This framework effectively encapsulates both conditions outlined in Appendix B.T. by accommodating for partial state trajectories of single-step transitions (as in
Eqn. 18), and for complete trajectories (as in Eqn 19p. with F(5o) = Z as per Bengio et al. (8). Furthermore, Madan et al. [41] proposed novel objective, SubTB(A), which synergizes subtrajectory balance conditions for partial trajectories of differing lengths, drawing inspiration from the TD(A)
approach in reinforcement learning.
These subtrajectory balance conditions are also adaptable to undirected paths, allowing for "back and forth" movements between states [42]. For an undirected path between 8m and 8n, this (generalized) subtrajectory balance condition can written as
n-1
m-
F(sm) II (s( II PP = F(s.) II Po(s(sse))) II Pp(st+i)s)) tokk
(23)
t=k
t=k
C=R
where &% is common ancestor of both 8m and 8,, These conditions, whether generalized specific, offer greater flexibility in their application. However, to guarantee that a GFN induces a distribution proportional to R(s), it is essential that these conditions are satisfied for all partial trajectories of any length. In this paper, we specifically focus on scenarios where these conditions are met for partial state trajectories of fixed length. Although this approach may deviate from the general guarantees, we follow the implementation discussed by Deleu et al. [18) and expound in Appendix B.4 how our GFN still induces a distribution & R(s) in our context.
B.4 More details about Reformulation of SubTB
In this section, we strictly follow the implementation and proofs discussed in 18
Subtrajectory balance conditions for undirected paths of length 3. Consider an undirected path of length denoted as (G, < (G,-) + (G', ()',)>> (G', X²), where G' derived from the directed acyclic graph (DAG) G by the addition of a new edge.. Given that the state (G..) is
17

---

common ancestor to both complete states (G, A) and (G', we can apply the subtrajectory balance conditions as expressed in Eqn. 23 The conditions are reformulated as follows:
F(G, (C"G))
=
(PP((G")
(24)
where ( X) denotes Pn((G, (G, 2)), a notation simplification for clarity. As (G, X) E X' has single parent state (G,-), -), it follows that (B]G))) 1, and a similar rationale applies to (G', X) Building upon the insights from Deleu et al. [17], the flow function A(G,A) of a complete state
(G, A)can be expressed function of its associated reward:
R(GG, A))
F(G, A) Pr(s)|G, A))
(25)
In our GFN implementation, 8f is the sole child of the terminal state (G, E X, indicating an (infinitely wide) tree structure rooted at (G.-). Consequently, (s x)) = 1, leading to the
simplification F(G,X) = R((G, A)). With these simplifications, Eqn. 24 becomes
B(GG,N))P(GE ()
R(((", ()))((G
(26)
which is the subtrajectory balance condition in Eqn. 20 This formulation effectively captures the essence of the balance conditions, providing a clear and concise representation of the underlying principles in the GFN structure for structural inference.
Integrating undirected paths of length 2. Similar to the previous paragraph, we consider here an undirected path of length of the form (G, < (G,-) + (G, A). Since (G, :) a common ancestor (a common parent in this case) of both terminal states (G, X) and (G, Å), we can write the subtrajectory balance conditions (Eqn. 23 as:
([G:A)PE(GIMPPA(GG)) = P(G,)))
(27)
Using the same simplifications as in the previous paragraph (PB(G)A) PG(())) = 1), we get the following subtrajectory balance conditions for the undirected paths of length 2:
A)) = (((
(28)
Note that these conditions are effectively redundant if the SubTB conditions over undirected paths of length 3 are satisfied for all possible pairs of terminal states (G, and (G', X') Indeed, if we write these conditions between (G. A) and (G) on the one hand, and between (G, X) and (G', X) on the other hand (with a fixed G' and X:
R(GG', ; X))PN(CI)))PP(GGG) R(G.A)))
(29)
B(GG', X (G)
(30)
we get the same subtrajectory balance conditions over undirected paths of length 2 as in Eqn. 28
B( x)) R[(G', (Y)PP(G)))))) R((
(31)
( P=(GII)):((NIG)
PF(A))
However, since the SubTB conditions are only satisfied approximately in practice, it might be advantageous to also satisfy Eqn. 28 The equation above provides an alternative way to express Eqn. 28 Indeed, Eqn. 31 shows that the function
fa(A) A log R(GG, x)) log Pp(X))
(32)
is constant, albeit with a constant that depends on the graph G. Since this function is differentiable, this is equivalent to Vxía(A) = 0, and therefore we get the differential form of the subtrajectory balance conditions:
V log PF|G) VA log R(G A))-
(33)
As shown by 18 one way to enforce the SubTB conditions over undirected paths of length 3 is to create a learning objective that encourages these conditions to be satisfied, and optimizing
using gradient methods. The learning objective has the form CCFN = ()))) where ((2) is a non-linear residual term
Á = log XI(G", (GGG))))G)G)))))) B[(GGNN)P((CCO)))AAYYYYYY)
(34)
18

---

Suppose that the parameters 92 of the GFN are such that the subtrajectory balance conditions in Eqn. [33 are satisfied for any (G, A). Although this assumption unlikely to be satisfied in practice, they will eventually be approximately satisfied over the course of optimization, given the discussion above about the relation between Eqn. 28 and Eqn. 26 Since and X depend on Ω (via the
reparametrization trick since they are sampled on-policy [8] ), taking the derivative of (1)). we
get:
d -3())= ds
d क
-())
log (((G))))) + log (a())) log R( A)) log Pa
log Pa())
(35)
Using the law of total derivatives, have
d llg (a|G) - log RC(() A)) =
D log Pa(A|G) - πx
dx
log R A))
(36)
dm
02
dn
log Pa(AIG)
(37)
an
0
log Pa(AG)
(38)
0n
and similarly for the terms in (G) The derivative of the objective then becomes d (9) = ((2)
a
a
d
log Pa(AG) an log Po(())") an
log Pa(G)))
(39)
dl
An alternative way obtain the same derivative in Eqn. 39 as the objective in Eqn. 33 take dA/dft = 0 instead, meaning that we would not differentiate through À (and X). Using the stop- gradient operation A, this shows the following objective
E [(DOE P(CC TRICCAA))P
27
LGFN -E,
(40)
takes the same value and has the same gradient (Eqn. 39) as the objective in Eqn. 34 when the subtrajectory balance conditions (in differential form) over undirected paths of length are satisfied.
While optimizing Eqn. 40aalone leads to eventually satisfying the subtrajectory balance conditions over undirected paths of length 2, it may be advantageous to explicitly encourage this behavior, especially in cases for non- non-linear models. We can incorporate some penalty to the loss function, such
$ B VA log Pa(AGG) VA log R(G |l Vx log Pa(X'G') EGEN- LOEN +
2
Vx log A(G' (41)
B,5 Forward Transition Probabilities
As delineated in Section 4.2 SICSM generates the pair (G, A) through two-phase process: (a) constructing the graph G = TU Au, Adj) by sequentially adding edges in Adj until a 'stop' action is triggered, followed by (b) sampling the parameters À conditional on G. These actions are governed by the forward transition probabilities (a(G'| in the first phase and Pit(A)G)) in the second phase.
To parameterize these terms, adopt hierarchical model strategy [18]. This model first deter- mines whether to halt the first phase using the probability Pat Based on this decision, the process either continues by adding an edge to Adj, forming G' = (UAn, Adj') with probability (G sstop), or transitions to the second phase by sampling À with probability Pa stop):
Pa = (1 Ph (p -stop),
(42)
Pa(A)) Po(stopiGGPPP(AGG. stop.
(43)
To accurately parameterize Po (t(stop]G), ( -stop), and Pa stop), we utilize combina-
tion of graph neural networks (GNNs) (5) and self-attention mechanisms [50]. This fusion of GNNs
19

---

and self-attention blocks ensures robust and flexible modeling of the transitions between states in SICSM. Further details regarding this parameterization approach are elaborated in Section B.6
B.6 Parameterization with Neural Networks
In the GFN of SICSM, various components are parameterized using neural networks, Specifically, we focus on parameterizing the following: (a) (stopllG), (b) Pn -stop,, (c) Po (A stop), and (d) the log-likelihood term log Plu X, Adj, U.). We combine GNN with self-attention mechanisms for parameterizing (a) (stotop) and (b) Pn[G"|G [,, stop)) This process generates a graph-level attribute g and node-level attributes (u; Vi, w.) for each node i in G:
g. [u, V. w. w SelfAttentiong (GNN,(G)).
(44)
The "stop" action probability is computed as Pa(stop G) fa(g), where fa is a neural network with sigmoid output layer. The probability of transitioning from G' to G' in the absence of a 'stop' action is defined as:
Pa -stop) x my exp(u,v,))
(45)
where my is a binary mask that excludes already explored graph structures. (c) For sampling parameters X for each node we define:
Pa(A/ stop) N (w:)
(46)
with bn and of being neural networks. This formulation effectively approximates the posterior distribution G UAu) upon full training. To get the adjacency matrix for the dynamical system, we need to approximate the marginal posterior P P(Adj(UAu). We follow the phases to generate G until 'stop' action. By aggregating { [Adj,, Adj2; ..., Adj from the posterior approximation, we estimate the marginal probability of directed edge from node to node j as:
B Eii+-E
Pa + ((UAA)
(47)
where 1-) is the indicator function. This process enables the inference of the underlying interaction graph's 'sssu structure, which is critical for evaluating the accuracy of SICSM in structural inference tasks. (d) For the log- leliikeiihodd term, employ a pressage-passingg neural network to compute future node features:
AL,
(48)
where MLP(.) is neural network that outputs the parameters of the probability distribution for the future state oi 1+1
MLP(UAU, A - US Uu f -CUst
-
(E F
Σ
(49)
A-HiE
where fe and fa are multilayer perceptions, and the operation performed for all nodes. Then we have likelihoddo for each node i at time t can be computed as:
log P(c Adj. U log (v 1n o 1
(50)
If the node features are multi cinlii-dimessoonal, we set up multiple readout heads in Eqn. 48 This approach effectively handles multi-dimensiona node features and incorporates both node features and graph structure, thereby reinforcing the model's 's predictive accuracy.
C More Details about Datasets
In this section, we provide more details about the datasets used in this work apart the description in
Section 5
20

---

C.1 Springs Simulations
To generate these Springs Simulations datasets, we follow the description of the data in [31] but with fixed connections and with 10 nodes, in order to simulate spring-connecteds connected particles' motion in a 2D box using the Springs simulation. In this setup, nodes represent particles, and edges correspond to springs governd by Hooke's ' law. The Springs simulation's dynamics
described by second-order ordinary differential equation: m, x (t) = ΣjeN; -k (z+(t) - zz())- Here, m, represents particle mass (assumed as 1), & is the fixed spring constant (set to 1), and Ng is the set of neighboring nodes with directed connections to node i, which is sub-sampled from the graphs generated in the StructInfer in previous steps. We integrate this equation to compute x (t) and subsequently zt(t) for each time step I. The resulting values of (1) and (:((1) create 4D node features each time step. To be specific, at the beginning of the data generation for each springs dataset, we randomly generate the same ground truth graph, but with
ground truth graph and then simulate 12000 trajectories
different initial conditions. The rest settings are the same as that mentioned in [31]. We collect the trajectories and randomly group them into three sets for training, validation and testing with the ratio of 8: 2: 2, respectively.
C.2 NetSims
It is firstly mentioned in [6], which offers simulations of imaging data in various human brain regions. Nodes in the dataset represent spatial regions of interest from brain atlases or functional tasks. Interaction graphs from the previous section determine connections between these regions. Dynamics are governed by a first-order ODE model: (1) = xza + C us, where a controls temporal smoothing and neural lag (set to 0.1 based on [46], and C' regulates external input interactions (set to zero to minimize external input noise) 46] ID node features at each time step are obtained from the sampled (t()).
(BOLD)
jeN; (1)
C.3 Synthetic Biological Networks
The six directed Boolean networks (LI, LL, CY, BF, TF, BF-CV) are the most often observed fragments in many gene regulatory networks, each has 7, 18, 6, 7, 8 and 10 nodes, respectively. Thus by carrying out experiments on these networks, we can acknowledge the performance of the chosen methods on the structural inference of weal-world biological networks. We collect the six ground- truth directed Boolean networks from 44 and simulate the single-cell evolving trajectories with BoolODE (44) /isttp://githube com /tt///Mraliiee grwup/ group/Bool0DD Boo1 with default settings mentioned in that paper for every network. We first sample a total number of 10000 raw trajectories. We then sample different numbers of trajectories from raw trajectories and randomly group them into three datasets: for training, for validation, and for testing, with ratio of 8 2: 2. After that, we sample different numbers of snapshots according the requirements of experiments in Section 5. with equal time intervals in every trajectory and save them as .npy' files for data loading.
C.4 StructInfer Benchmark
The StructInfer benchmark [3] evaluated 12 structural inference methods in a comprehensive way on synthetic dataset. The dataset covers 11 types of different underlying interaction graphs and two types of dynamical simulations. (https: /tttst/ttsttutifee..mmmmm gi titub...11 io/) As there are so many trajectories, we chose the ones under the name 'Vascular Networks', or in short 'VN', whose underlying interaction graphs approximate the real-world vascular networks in biology systems. As the data is already split into three sets: for training, for validation, and for testing, we keep this setting. In the following paragraphs, we describe more details about the Springs and NetSims simulations utilized by the StructInfer benchmark.
For Springs simulation, it follows the approach by Kipf et al. 31 to simulate spring-conneett particles' motion 2D box using the Springs simulation. In this setup, nodes represent particles, and edges correspond to springs governed by Hooke's 's law. But different from Springs Simulations mentioned above, StructInfer generates ground-truft interaction graphs with the graph properties of the real-world graphs or network. The ground-truth interaction graphs are used to determine the connectivity between the nodes. The Springs simulation's 's dynamics are described by a second-order ordinary differential equation: m, x ΣjEN, -k (x,(t) (t)). Here, m; represents
21

---

particle mass (assumed as 1), k is the fixed spring constant (set to 1), and N; is the set of neighboring nodes with directed connections node i, which sub-sampled from the graphs generated in the StructInfer in previous steps. We integrate this equation to compute (1) and subsequently æx(1) for each time step The resulting values of (t) and (t( create 4D node features each time step.
For NetSims simulation, it is firstly mentioned in NetSim dataset [46., which offers simulations of blood-oxygen-lever -eeeeeeeelldeee (BOLD) imaging data in various human brain regions. Nodes in the dataset represent spatial regions of interest from brain atlases or functional tasks. But different from NeiSim mentioned above, StructInfer generates ground-trul truth interaction graphs with the graph properties of the real-world graphs or network. The ground ground-truth truth interaction graphs are used to determine the connectivity between the nodes. Dynamics are governed by first-order ODE model: 2t(t) + C-u,, ug, where a controls temporal smoothing and neural lag (set to 0.1 based on [4]. and C' regulates external input interactions (set to zero to minimize external input noise) 46). ID node features each time step are obtained from the sampled x; 2t()).
(t) = a. EjeN, Ij (t)
C.5 PEMS Datasets
These datasets, derived from the California Caltrans Performance Measurement System (PeMS) [1] comprise data aggregated into 5-minute intervals. The adjacency matrix of the nodes is constructed by road network distance with a thresholded Gaussian kernel [48] Table summarizes these datasets.
Table 1: Statistics of PEMS datasets.
Dataset
Nodes
Edges
# Time Steps
Missing Ratio
PEMS03
358
547
26, 208
0.672%
PEMS04
307
340
16, 992
3.182%
PEMS07
883
866
28, 224
0.452%
We resampled the data such that constructing 49 time steps of points for each trajectory, and obtained 1 12000 trajectories for each with overlapping snapshots. It's important to note that these datasets' adjacency matrices only connect sensors on the same road, omitting alternative connecting paths, which could impact results.
D Implementation of Baselines
For the experiments without prior knowledge, we follow the official implementation of the baselines. As for the integrating of the prior knowledge, leverage different strategies. For the methods based on VAES, (e.g. NRI, MPM, ACD, iSIDG, RCSI), we directly perform supervised learning on the latent space with known edges, while keep the rest following the original implementation. For JSP-GFN, we set the graph structure in the initial state and reset states the same as prior knowledge.
D.1 NRI
NRI 31 is VAE-based model for unsupervised relational inference. We use the official imple- mentation code by the author from /estps://gt thub. com/ ethanf SaaanneettaaNRAA with a customized data loader for our chosen datasets. We add metrie evaluation in the 'test" function, after the calculation of accuracy in the original code.
D.2 MPM
employs VAE framework with a relational interaction mechanism and spatio-tempora message passing. We use the official implementation code by the author from lhttps://st ghubbb com
MPM 12
with a customized data loader for our chosen datasets. We add our metric evaluation for AUROC in the 'evaluate()" function of class 'XNRIDECIns' in the original code.
22

---

We run the code with a customized data loader D.3 ACD
ACD 39 utilizes shared dynamics to infer causal relations within datasets. We follow the official implementation code by the author as the framework for ACD (https://githu. com//oeweX/ AmoortizedCausalDiicooeer
We run the code with a customized data loader for the datasets in this work. We implement the metric-calculation pipeline in the 'forward_pass and_ and_eval()) function.
D.4 ISIDG
iSIDG
51
iteratively refines adjacency matrices We follow the official implementation code by the au-
enhance directional
inference. thor
the framework for iSIDG (https:/ wwwptauu..oo.aaaaaaaaaeaae www.waagn22003011// Bonchmarrknng-Stuccttrrr www.:ruuurrar-nneeeeeee-------oo axree/mainssr//eeeee
for- Interacting Dynamical: Sasaall-ystemss (ttaaaan//ccmmmoeel/////0/) We disable the metric evaluations for the AUPRC and Jaceard
index in the original implementation of iSIDG for faster computation.
D.5 RCSI
RCSI [54] integrates reservoir computing for efficient structural inference. We would like to thank the authors of RCSI for the code. Same iSIDG, we disable the metric evaluations for AUPRC and Jaccard index in the original implementation of iSIDG for faster computation.
D.6 JSP-GEN
JSP-GFN 18] applies Generative Flow Networks for Bayesian inference of graphical structures. We follow the official implementation code by the author the framework for JSP-GFN (https //githu.. httpl//cco/tttttttieeeeeee wwristandeeee/aaa-jaaapa. for the datasets in this work.
D.7 SIDEC
SIDEC 5 encodes node dynamics exploit partial correlations for structural inference. We follow the official implementation code by the author as the framework for SIDEC (https://githm Lang422003/SIDCC torch). We the code with customized data loader for the datasets in this work. By incorporating the prior knowledge, we did not figure out a feasible way to do so. Thus we omit the implementation for integrating prior knowledge.
com/
D.8 Implementation details of SICSM
The general training pipeline of SICSM is presented in Algorithm
SICSM is implemented with JAX g, including following packages: dm-haiku 24 and jraph [9 The implementation of SICSM model consists of two parts: (1) the implementation of Residual Blocks with Mamba, and (2) the implementation of GFN. The implementation of Residual Blocks follows the script of imamma-minimall-ee... tree/main). We would like to thank the contributors of this repository for all the efforts they have done. And the implementation of GFN of SICSM follows the implementation of JSP- GFN [8), and the authors' implementation can be found at attttss ://::t//gtttt jax-jsp-gfi We would like thank the authors for code and hints. The modifications were made to integrate various regularization terms in the graph prior, the liglliil,,ood, and the data-loading pipelines. Please refer to the link provided in the supplementary document for the exact implementa- tion SICSM. SICSM is trained with Adam 29 optimizer, with the learning rate as 0.00001 and for 1000 epochs. Implementation can be found at: https: //tp://ggttttttt thub........ com/ www.woow.///eeeeeco httmm//wng22200///1/m/ SSIICM/SSSSSMMMSSS JAX/
(https: : //ggttttub.... com/ http///ddaFuudn/mnaaea mamba- -immmiii wwmmimmmmmaaa...amaaaaaaaa-aaaia
coom/rristnndelee.
Among all, the most important hyperparameter of SICSM would be number of Residual Blocks in encoder and decoder, I and L,,, respectively. As we expect symmetric structure of both, so I/' is set as equal to I.. The exact number of layers actually depends on the number of nodes in the graph, and we report the values of L in Table 2
23

---

Algorithm The training procedure of SICSM
1: Input: trajectory V of nodes
2: Parameters: number of steps of prefilling Ç. learning rate
3: Parameters: number of Residual Blocks in encoder L, number of Residual Blocks in decoder L'
4: Output: Structure of the dynamical system Adj
5: Decompose the input trajectory form present feature V0:T~2 and forecasting feature
VI:T-1
6: Initialize the State trajectory Go unconnected graph with nodes
7: repeat
8: Get feature-based embedding h for every node and for every time step: hę fembed (ot)
9: Compose H h h²
h1 for every node
10: Compose Hall = H,, for all nodes)
for I S (L. L') do
12:
if F thenirst Residual Block
Get the output of the block: Un [nB, (Hait)
14:
else
15:
Get the output of the block: URB, = SRB, (URB:-1)
16:
end if
17:
end for
18:
if C thenomplete observation of all nodes
19:
Get the embeddings from the last block in encoder: UAu = URB.
20:
else
21:
Aggregate all embeddings from the blocks in encoder: UAu = Et= URB end if
22:
23:
Project back to input dimension pe+11 = fyroj (Uroj(URB,L+U))
24:
for & steps do
25:
Sample the stop action probability: Pat(stop(Gr) if a is the 'stop' action then
26:
27:
Reset the State trajectory: Gk+1 Go else
28:
Sample Gk+1 Pn(Gx+1| Gk: -stop) Store the transition G3 + Gk+1
30:
31:
end if
32:
end for
Sample (a stop)
34:
Sample X' ~ (X' stop)
35:
Evaluate the rewards R( (G, A)) and R((G',))
36:
Evaluate the loss L = LRn + LAFN
37:
Update the parameters of the branch of Residual Blocks and GFN
38: until Convergence criterion
39: Sample the approximation of posteriors: Pa(i + AlUAu) ~ 1 Σ 1(i + Adja)
40: Output the sampled Pa as the structure of the investigated dynamical system
Table 2: Number of Residual Blocks in the encoder of SICSM.
n S 10
10 m S 30 < n S 50
50 < n S 100
>> 100
L
5
E More Experimental Results
E.1
Supplementary Experimental Results on Other Metries
E.2
Experimental Results with Prior Knowledge
The experimental analysis focuses on the impact of integrating varying percentages of prior knowledge (0%, 10%, 20%, and 30%) into the training process of different structural inference models, including NRL MPM, ACD, iSIDG, RCSI, JSP-GFN, and our proposed SICSM. The results, depicted in the

---

Spring Semelaisnss
NatSinn
11L
BF
7F
BF-V
22
Team Skys
Tea 3kp
TrauS
Tia Saps
80 23
10
50
20
20
12
Figure 7: AUPRC values (expressed in percentage) for various methods as function of the number of irregularly sampled time steps. Results averaged across ten trials, with time steps varying from 49 to 10. The shadings show the standard deviation of each data point.
Springs
10
VN.
VN NS
2011
Time
3623
34794
15E.
$8.55
Figure 8: SHD values (expressed in percentage) for various methods as function of the number of irregularly sampled time steps. Results are averaged across ten trials, with time steps varying from 49 10. The shadings show the standard deviation of each data point.
Springs Simelations
NatSime
1I
CV
TF
BF-CV
20
Tran-
Sep
Sem
Tran
The
Tom
Team
VN $5
SP 30
VN SP SP
VN_SP_I0 100
IS
VNUNS_30
V0
VN 100
50
30
Skrge
Tiam Deys
Tea
Figure 9: Fl scores (expressed in percentage) for various methods as function of the number of irregularly sampled time steps. Results are averaged across ten trials, with time steps varying from 49 to 10. The shadings show the standard deviation of each data point.
25

---

YN SP 50
VN KP 100)
VN NS 50
VN NS 109
AUBOX
20
Peocestages Poko Knoslolge
Prrcentages Pric Knowlolge
Percertages Prix Knowlolge
Percenzaees of Prioe Keowlelge
Figure 10: Average AUROC results (in %) of SICSM and baselines with different percentages of prior knowledge on VN datasets.
provided line plots, evaluated across four datasets: VN_ AN_SP_50, VN VN_SP_100, VN_NS_ vS_50, and VN_NS_ 100. As for SIDEC, we did not figure out a feasible way of integrating prior knowledge into it. The results are shown in Figure TO and can that SICSM consistently outperforms other baseline models across all datasets and percentages of prior knowledge integrated. Notably, SICSM shows significant improvement in AUROC as the percentage prior knowledge increases, underscoring its capability to effectively utilize additional information to enhance structural inference accuracy. Models like NRI, MPM, and ACD show moderate improvements with increased prior knowledge but remain less effective compared to SICSM. This suggests that while these models benefit from prior knowledge, their overall adaptability and learning mechanisms might not fully capitalize on the information provided. JSP-GFN and RCSI display variable trends, for instance, JSP-GFN shows notable improvements in the VN VN_SP_ SP_50 and VN_SP_100 datasets but less so in VN_NS datasets, indicating potential dataset-specific sensitivities.
The enhancement in performance with increased prior knowledge is most pronounced in the VN_SP IN_SP_I00 and VN_ 50 datasets for SICSM. This pattern illustrates the model's 's robustness in leveraging prior knowledge, particularly in scenarios with larger and possibly more complex network structures. In contrast, the increments in AUROC scores for baselines like iSIDG and RCSI are less steep. suggesting these models, while benefiting from prior knowledge, do not adapt as effectively as SICSM.
To conclude, the integration of prior knowledge markedly benefits the performance of structural infer- ence models, with our SICSM model demonstrating superior capability to utilize such information to enhance prediction accuracy. These results validate the effectiveness of SICSM's design in adapting to additional contextual information, setting a benchmark for future developments in the field. Further investigations could explore optimizing the integration process of prior knowledge to maximize the performance benefits across diverse structural inference scenarios.
E.3 Experimental Results on PEMS
Table 3: Average AUROC results (%) PEMS datasets.
PEMS03
PEMS04
PEMS07
JSP-GFN SIDEC
60.04 L01
60.5 0.63
61.2a
73.5 013
70.0=
SICSM
71.24
74.7
71.2 0.47
This section presents the performance evaluation of two baseline methods, JSP-GEN and SIDEC, alongside proposed SICSM on three real-world datasets: PEMS03, PEMS04, and PEMS07. These datasets are instrumental in assessing the robustness and effectiveness of structural inference methods in real-world scenarios. The results summarized in Table Other baselines fail to work on large graphs and encountered OOM errors on these datasets.
As we can see from the table, SICSM consistently exhibits the highest AUROC across all three
datasets, with scores of 71.2% on PEMS03, 74.7% on PEMS04, and 71.2% on PEMS07. These results underscore SICSM's superior performance in capturing and predicting complex network
26

---

Sample Case
Results from SicSM-one
Results from SICSM
Structure
Sumpling
Rosuta
Sample
SICSM-one
from
Groundrruth
Olterned
Incevoredde rock
Infinc soge
8
Figure 11: (1st Column) Ground truth structure with all nodes. (2nd Column) Two examples of 12-node sampling. (3rd, 4th Columns) Structural inference results from SICSM-one and SICSM.
dynamics in traffic systems, which is real-world scenario. SIDEC also performs robustly, especially on the PEMS04 dataset where it achieves an AUROC of 73.5%. However, slightly trails behind the PEMS03 and PEMS07 datasets. JSP-GFN shows the lowest performance
SICSM, particularly among the evaluated methods, with its highest AUROC at 61.2% on PEMS07, indicating a lesser adaptability the dynamics of these specific traffic datasets.
The standard deviations reported alongside the AUROC scores indicate the stability of each method's performance across different runs. SICSM demonstrates moderate stability with standard deviation of approximately 0.44% to 0.47%, suggesting consistent performance despite the inherent variability in real-world data. SIDEC shows the highest stability, particularly standard deviation of 0.1.%. This suggests that SIDEC is reliably effective in scenarios represented by this dataset. JSP-GFN, while the least effective in terms of AUROC, maintains relatively consistent performance as indicated by its standard deviations, which range from 0.63% to 1.01%.
PEMS04, with minimal
The evaluation on the PEMS datasets validates the effectiveness of SICSM, particularly in comparison established baseline methods like ISP-GEN and SIDEC. SICSM's ability to consistently outperform other methods underlines its advanced structural inference capabilities, making it promising solution for complex real-world applications in system dynamics and network analysis.
E.4
Why Do We Need All Residual Outputs?
As discussed in Section 3.2 in systems with partial observation, it is advantagcous combine learned dynamics from multiple Residual Blocks to enrich the dynamics available for the GFN, However, an intriguing question arises: What the impact when only the dynamics from the final Residual Block in the encoder are used, similar to approaches used in fully observed systems? This section delves into this query through detailed case study on a specific dataset. We selected the VN_SP_Is dataset for this examination, focusing a scenario where 12 nodes (80% of the total) are sampled. We contrasted two configur configurations of our proposed structural inference method: the comprehensive SICSM, which integrates outputs from all Residual Blocks in the encoder, and simplified version, SICSM-one, which relics solcly on the output from the last Residual Block in the encoder. This comparison aimed to assess the impact of multi-layer output integration the accuracy and robustness of the inferred network structures, with results illustrated in Figure [T
Both configurations were evaluated on the same dataset, comprising full graph and 12-node sampled version, to appraise their performance across varying degrees of system completeness. Ob- servations from the results indicated that SICSM-one frequently misinterpreted two-hop interactions as three-hop connections, leading to an increased incidence of false positives. This issue arises be- cause SICSM-one relies solely on the output from the final, potentially larger, Residual Block, which can blur the distinctions between one-hop, two-hop, and three-hop dynamics. In contrast, SICSM leverages outputs from multiple layers, enabling comprehensive representation of dynamics across
27

---

Table 4: Average counts of multi-hop negative edges and true positive edges reconstructed upon VN SP_ P_15 dataset with 12 nodes are sampled with different residual blocks. The average performed based on 10 runs. For reference, L = 7. Each residual block is numbered as their closeness to the input side. For example, Residual Block [1] is the first one, [1 3] refers to the integrating outputs from to 3 Residual Blocks.
Residual Blocks
Count of multi-hop negatives
Count of true positives
7.5
10.2
7.3
9.8
6.9
9.2
6.8
8.5
6.6
8.0
6.5
7.1
6.5
7.0
1 2
6.5
7.2
3
6.1
8.6
5.3
9.4
1.2
10.0
3.1
11.1
[1 7] (Full SICSM)
2.1
12.5
different temporal dependencies. This multi-layer aggregation particularly effective emphasizing shorter connections while still accounting for longer pathways. The integration of shallow blocks plays a crucial role in this configuration, offering detailed insights into shorter dependencies and significantly reducing the likelihood of misidentifying longer connection paths as false positives. Despite these improvements, some inaccuracies remain, suggesting areas for further refinement. Future developments might include implementing an adaptive weighting mechanism that adjusts the influence of dynamics from different Residual Blocks. Such mechanism would be tailored according to the size of the graph and the longest potential paths within the network, optimizing the model's 's accuracy in diverse operating conditions.
Moreover, as shown in Table 4 the occurrence of negative multi-hop edges is notably higher when only a single Residual Block is used. This number decreases to 6.5 when only the first block is used, but at the cost of reducing true positive predictions. The best configuration, as highlighted in the table, is the concatenation of outputs from all blocks. This approach not only reduces the occurrence of negative multi-hop edges but also increases the count of true positives, providing more balanced and accurate representation of the underlying structure.
These findings underscore the effectiveness of multi-layer dynamic integration in SICSM, particularly in settings with partial node observability. They highlight the model's capacity to maintain structural integrity and provide accurate predictions, affirming its potential for broad application in complex, dynamically varying systems.
E.5 Ablation Study on the Choice of Neural Networks the Blocks
We conducted additional experiments comparing Transformer [50] LSTM 25 and GRU models on irregularly sampled trajectories with 30 time steps and partial observations with 12 nodes in the VN /N_SP_ dataset. The average results from 10 runs are presented in Table For all models, we adjusted the parameters to accommodate the trajectory lengths and performed hyperparameter tuning using Bayesian optimization. As shown in the table, the Transformer model outperforms LSTM and GRU by small margin, but all are notably inferior to SSSM, as they struggle to effectively handle multi-hop interactions. Additionally, these models perform poorly on irregularly sampled trajectories, as they lack the ability to learn adaptively.
E.6 Training Time Comparison
The training time analysis for SICSM and baseline methods on the VN_NS datasets, summarized in Table provides valuable insights into the computational efficiency of these models. The reported times are averaged over ten runs and are presented in hours.
28

---

Table 5: Average AUROC results of SICSM with different neural networks in each block. The networks under consideration are Transformer, LSTM and GRU. The experiments are irregularly sampled time steps and partially observed nodes on VN_SP_ 15 dataset.
Neural Network
Results on Irre. Sampled
Results on Par. Obser.
Transformer
7.5
10.2
LSTM
7.3
9.8
GRU
6.9
9.2
SSSM
6.8
8.5
Table 6: Training time (hours) of SICSM and baseline methods on VN_NS datasets.
Methods
VN_NS_IB 15
VN_NS_30
VN_NS_
VN_NS_1O 100
NRI
24.6
33.5
40.5
47.1
MPM
45.3
60.4
79.2
83.6
ACD
40.2
53.1
67.6
81.7
iSIDG
43.9
56.2
88.5
98.0
RCSI
44.6
58.0
91.6
103.4
JSP-GFN
45.0
55.1
72.0
97.3
SIDEC
26.8
32.9
39.5
45.0
SICSM
59.2
70.1
96.3
120.5
From the table, it's evident that SICSM, while providing advanced capabilities in structural inference demonstrated in previous sections, exhibits longer training times compared to both traditional and other state-of- ootc-----hhearr bascline methods. SICSM consistently shows higher training times across all dataset sizes compared to other methods. For instance, at VN NS 15, SICSM takes approximately 59.2 hours, which is about 35 hours longer than NRI and nearly 33 hours more than SIDEC, the method with the shortest training time for this dataset size. As the size of the dataset increases, the training time for SICSM also increases substantially, from 59.2 hours for VN VN_NS _N_SS____ 15 to 120.5 hours for VN_ VN_NS_ 100. This scaling trend consistent with other methods but more pronounced in SICSM, suggesting that its complexity scales significantly with larger networks. The reason is that we implement selective SSM with JAX, which lacks the cuda package boost the selective process that is designed in (20). and the search over all possible state spaces in GFN is come-consuming. The data highlights crucial area for future development optimizing the computational efficiency of SICSM. Enhancements might focus on the implementation of JAX-suited cuda package for boosting selective SSM or integrating more efficient learning algorithms reduce training times without compromising the model's performance.
VN_NS.
E.7 How Good is the Approximation?
In this work, we approximating P(Adj, X(Uai) instead of the Adj. Thus, is necessary evaluate how well ((Adj. X(Uan) is approximated with distributional metric.
Similar the experiments in JSP-GFN [1), we consider here models d variables, with linear Gaussian CPDs. We generate 20 different datasets of = 100 observations from randomly generated Bayesian Networks. The quality of the joint posterior approximations is evaluated separately for Adj and A. For Adj, we compare the approximation and the exact posterior on different marginals of interest, also called features in JSP-GFN [8]. e.g., the edge feature corresponds to the marginal probability of a specific edge being in the graph. Fig. shows comparison between the edge features computed with the exact posterior and with SICSM, proving that it can accurately approximate the edge features of the exact posterior. To evaluate the performance of the different methods as an approximation of the posterior over X, we also estimate the cross-entropy between the sampling distribution of À given G and the exact posterior P() Adj, Uau). The results are shown in Table 7 We observe that again SICSM samples parameters that are significantly more probable under exact posterior compared to other methods.
JICSM,proving
29

---

SCCMM
0.4
0.2
0.99988 9983
0.0
posterior
Figure 12: Comparison of the edge features computed with the exact posterior (x-axis) and the approximation given by GFN in SICSM.
Table 7: Comparison with the exact posterior distribution, on small graphs with n nodes. Quantitative evaluation of different methods for joint posterior approximation, both in terms of edge features and cross-entropy of sampling distribution and true posterior P()[Adj, Unx). All values correspond to the mean and 95% confidence interval across the 10 experiments.
Edge features
E log P(A|Adj, Unu)]
10
< 30
JSP-GEN
0.0019 0.005
0.998 0.001
-4.95 + 0.51 X 10"
SICSM
0.0018 + 0.007
0.998 + 0.001
-4.97 + 0.52 X 10
F
Limitations
While SICSM marks significant step forward in structural inference, it imperative to acknowledge its potential limitations for a comprehensive understanding and to guide future research:
Reliance Prior Knowledge Accuracy: SICSM's enhanced performance through prior knowl- edge integration is contingent on the accuracy this information. Mislcading or incorrect prior knowledge could adversely impact the model's inference accuracy, leading to potentially flawed conclusions.
Prior Knowledge of Edge Existence: Currently, SICSM leverages prior knowledge about the existence of edges in the graph. However, it is not equipped to incorporate prior knowledge about the non-existence of specific edges, limiting its ability to exclude certain connections during the inference process.
Scalability to Very Large Graphs: The scalability of SIGEN to graphs with an extremely large number of nodes remains untested (e.g., with more than 1,000 nodes). Training and inference in such large-scale graphs may demand significant computational resources and time, which could be practical constraint. We acknowledged this limitation of SICSM, and currently working the variant with sub-graph ensemble methods inspired by Cluster-GCN [4] and GraphS AINT [62] Some methods from federated graph learning may also solve the challenge of scalability of structural inference [26].
Evaluation on Synthetic Data: Due to challenges in obtaining reliable real-world datasets for structural inference, SICSM has primarily been evaluated on synthetic data in this study. We recognize the potential discrepancies between synthetic and real-world data and plan address this limitation in future research by exploring real-world applications.
30

---

Dynamic Graphs Handling: Currently, SIGFN is formulated for static graphs. However, many real-world graphs are dynamic, with structures that evolve over time. Adapting SICSM accom- modate such dynamic graphs is an essential area for future development.
Long Run Time: As detailed in Appendix [E66 SICSM exhibits the longest running time among all evaluated methods. This extended duration primarily results from the selective SSM in JAX lacking optimized CUDA integration, which is critical for enhancing computational efficiency. Additionally, the time-intensive process of constructing all possible state spaces within the GFN significantly contributes to the overall duration.
Future enhancements to SICSM could involve strategies for validating and correcting prior knowledge, improving scalability and efficiency for handling larger graphs, extending the model's capabilities to dynamic graphs, and implementing efficient selection SSM with JAX well as boosting the speed of GFN. These advancements will be vital in ensuring SICSM's applicability and reliability across various practical scenarios.
G Broader Impact
Much like NRI, MPM, ACD, iSIDG, RCSI, SIDEC, and other structural inference methodologies, SICSM extends its utility diverse range researchers across the realms of physics, chemistry, sociology, and biology, where the uncovering of underlying interaction graph structure is becoming more and more popular. our investigations, we have demonstrated SISICSM's proficiency in reconstructing graph structures and display robustness to variations in the irregular samplings and incomplete observations, underscoring its versatility and broad applicability. There may be potential societal consequences of our work, none which we feel must be specifically highlighted here.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The discussion of limitations can be found in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification:
All the formulas in the paper are numbered and cross cross-referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the link to our implementation in the supplementary document.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification: While this paper does not include new datasets, we provide links to our implementation and the references the data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Baaiii//guidee/CdeSSuuiii conooeooliiissiioooocccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssiicyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix D.8
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The plots in the main content come with shading showing the standard deviations at each data point.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
34

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: As we discussed in Appendix G did not recognize any direct path to negative applications or negative social impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets (e.g., code, data, models), used the paper, properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "jvXKp4xWKK", "reviewer": "Reviewer_2uCG", "summary": "Processes of scientific interest which are representable as graphs, in biology, chemistry, material sciences, mechanics, are an important application for machine learning. Nodes often represent physical objects, some of which influence each other. Nodes exhibit a set of features which can be observed over time. Prior knowledge about the process stems from a mechanistic understanding and can often be represented as the presence or absence of edges between nodes. Node feature observations may be irregularly spaced through time; not all nodes may be observed with every observation. \nThis paper develops a statistical model for this application with support for irregularly sampled and partial observations of node features, as well as prior knowledge incorporation. Prior knowledge is restricted to the indication of presence, but not absence, of edges. Partially observable nodes are assumed to be from a static node set throughout all observations (i.e., nodes are either always observable or always unobservable). Observations are not assume to contain a timestamp indication (as in mobile phone accelerometer readings, which may be irregularly sampled but whose timestamp is read at input).\nThe model's architecture is relatively sophisticated and is based on generative flow networks to represent and learn the structural aspects of the graph, and state space models to represent the evolution of node features over time. \nThe paper presents experiments on 16 datasets stemming from 4 physical models, and compares to 7 other models, showing superiority in scenarios where observations are irregularly spaced or nodes partially observable.", "questions": "* Eq.2: since s' is terminal, isn't any $s'' = s_f$ ?\n* Eq.5: in your contemplated application scenario, the interval between sampling times, or equivalently the timesamp of each sample, allowing to calculate $\\Delta$, is not given with the samples, correct? I've worked on mobile accelerometer/GPS data where sampling is irregular but the timestamps are given, which is why I'd like to make sure. Can you clarify whether a posterior over $\\Delta$ can be pratically recovered?\n* It might be useful to show a concrete, simple example of training data to clarify the scenario described in abstract terms sec3.1. \n* Annex 1 fig5: shouldn't all variables be indexed with $i$, the node id? I'm asking because $A, h^t$ aren't. But if so, how is the interdependence between nodes modelled? \n* fig6: do I have it right that GFlowNet only adds edges, but doesn't remove any, moving from start to end? Does that have as a consequence that any prior knowledge can only formulated as known-to-be-present edges, but not known-to-be-absent edges (impacts Annex F l.978)?\n* Annex C.5: what is the physical model? What is a node, an edge of the model?\n* Annex l.872: how is the % of prior knowledge defined?\n* Annex l.863: link is referred to but missing", "limitations": "* Checklist point 4 and 5: implementation link is claimed to be provided here and in Annex l.863, but I don't see it. It should be provided in the main paper since the supplementary can't be assumed to be reviewed.\n* Limitations: a few more assumptions on the usage scenario should be spelt out, as mentioned in this review\n* Checklist point 7: It is certainly possible to report error bars on plots through shading, provided they are not as tiny as you make them here. In addition, error bars could be reported, without lengthening the main paper, in the Annexes -- but they aren't, with the only exception of Table 3 on an experiment which is not reported in the main paper.\n* Checklist point 6: experimental settings are not as detailed as that they would allow reproduction. Several details are missing for this, e.g. batch size, data splits.\n* Checklist point 12: the claim, and requirement of the checklist that assets have their license mentioned is not complied with regarding either datasets or existing code for competing methods. Despite the claim, this point mostly is not complied with.\n* Checklist point 13: does this imply the code is not intended to be released as an asset?", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper takes an established problem class (graph systems) with its known challenges (irregular sampling, partial observations), which is not original. However it goes to great lengths to make use of two strong methods, GFN and SSM, with a resulting combination that seems reasonable, strong and of useful application.\nThe paper is generally clear, notations are coherent and legible, several diagrams support the explanation. To improve the writing, a running example might help bridge the abstractions (node, edge, state...) to physical reality, illuminating and motivating the implementation. The same goes to comment on the connection between the model and the applied datasets (some of this is covered in Annex C, with the exception of C.5 which leaves the physical counterparts of modelled data undescribed).", "weaknesses": "Experimental validation is moderately convincing. Baseline implementations seem strong, with care taken to recover implementations of competing methods, as documented in Annex D. However, all datasets are synthetic. The only real dataset, PEMS, presented in Annex C.5, with results in Annex E.2. In addition, experimental validation seems unconcerned with performance outside the specific cases of partial observations or irregular sampling -- reducing the paper's claim to \"this model is better for these two scenarios only\".\nThere seem to be a duplication in the presentation of datasets (both in sec5.1, between the paragraphs starting l.279 and l.290, and again between Annex C.1 and C.2 vs C.4) -- this is confusing. Also, sec3.3 seems to be internally redundant with duplicated points (e.g. l.151 vs eq3, and l.148 vs l.157, which again is confusing. Numerous sentences have incorrect English syntax which obscures their meaning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the inspiring review. And here are our answers to your concerns.\n\n> To improve the writing, a running example might help bridge the abstractions (node, edge, state...) to physical reality [...] \n\nWe would like to sincerely thank the reviewer for this advice. We added the following paragraph to our revision in Section 3.1 and it will appear with the camera-ready as we cannot upload the revision during rebuttal: \n\nWe may image a dynamical system of balls and strings, in which the balls are randomly connected with the springs. Then set initial positions and velocities of each ball and then let them go. Because of the existence of the spring forces, arising from the structural connections between the balls, the balls will change their positions and velocities in the observational period. Then suppose we have no idea on which balls are connected, so the task of structural inference would be, the inference of the connectivity between the balls based on the observational trajcetories of them.\n\n> However, all datasets are synthetic. The only real dataset, PEMS, presented in Annex C.5 [...]\n\nMany thanks! As acknowledges by the research in this field, the acquisition of trajectories with reliable understanding of the underlying structure from real-world, is of huge cost and time-expensive. We do acknowledge this urgent call of the reliable real-world data, and we are now working on it.\n\n> There seem to be a duplication in the presentation of datasets [...]. Also, sec3.3 seems to be internally redundant with duplicated points (e.g. l.151 vs eq3, and l.148 vs l.157, which again is confusing. \n\nMany thanks for the comments, we revised our paper to reduce the duplications and tries our best to keep most relevant information to understanding of our paper in the main body.\n\n> Eq.2: since s' is terminal, isn't any $s'' = s_f$?\n\nActually, no. $s''$ here in Eq. 2 is used to mark the child of the state $s'$. So only if $s'$ pointing to a terminal, then $s'' = s_f$.\n\n> Eq.5: in your contemplated application scenario, the interval between sampling times, or equivalently the timesamp of each sample, allowing to calculate Δ, is not given with the samples, correct? I've worked on mobile accelerometer/GPS data where sampling is irregular but the timestamps are given, which is why I'd like to make sure. Can you clarify whether a posterior over Δ can be pratically recovered?\n\nNo, $\\Delta$ is not given with the samples.  Many thanks for the inspiring question. Technically, we think the posterior over $\\Delta$ can be partially recovered, but we have to change the modeling of $\\Delta$ in the SSSM module to be with two vectors. The two vectors work similar to those in variational autoencoders, and we can recover the posterior over $\\Delta$ partially from them. \n\n> Annex 1 fig5: shouldn't all variables be indexed with $i$, the node id? [...]\n\nNo, the nodes share $A$ and $h^t$.\n\n> fig6: do I have it right that GFlowNet only adds edges, but doesn't remove any, moving from start to end? Does that have as a consequence that any prior knowledge can only formulated as known-to-be-present edges, but not known-to-be-absent edges?\n\nYes, it does not remove any edge. Yes, currently we can only incorporate known-to-be-present edges to be the prior knowledge, as it fullfills the scenarios in some fields like the GRN inference, where several connections are validated by experiments, so we are more sure about the existence of some edges than the absence of certain edges.\n\n> Annex C.5: what is the physical model? What is a node, an edge of the model?\n\nThe model in Annex C.5 is that: many sensors are used to count the traffic in a roadmap, and these sensors may be connected by the road they align with. In this case, suppose we have no idea on the map, but try to reconstruct the map from the traffic-counting from the sensors, as in most cases, continuous traffic flows happen between adjacent sensors.\n\n> Annex l.872: how is the % of prior knowledge defined?\n\nIt is defined as the proportion to the count of all positive edges in the groundtruth graph.\n\n> Annex l.863: link is referred to but missing AND Checklist point 4 and 5: implementation link is claimed to be provided here and in Annex l.863, but I don't see it [...]\n\nMany thanks! We revised our paper with the link. As you may interested in our implementation, we included them in our supplementary materials.\n\n> Limitations: a few more assumptions on the usage scenario should be spelt out\n\nMany thanks! We will include the assumption on prior knowledge: sure-to-to-be existed in the limitations among other suggested. And also the experiments in this work mainly covers synthetic data.\n\n> Checklist point 7: It is certainly possible to report error bars on plots through shading, provided they are not as tiny as you make them here. [...]\n\nMany thanks for the advice! We revised the figures to make them with shadings. Please refer to the images in the pdf file attached in the general rebuttal.\n\n> Checklist point 6: experimental settings are not as detailed [...]\n\nMany thanks! We revised this section and include the batch size: 32, data splits into training, validation and test: 8: 2: 2 for all of the datasets. We also include more such as number of epochs and so on.\n\n> Checklist point 12: the claim, and requirement of the checklist that assets have their license mentioned is not complied with regarding either datasets or existing code for competing methods. [...]\n\nMany thanks! We will include the license in our repository. As current we have to obey the double-blinded rule, we removed them from the repo.\n\n> Checklist point 13: does this imply the code is not intended to be released as an asset?\n\nSorry for our mistake, We correct it to be: We will release the code in our github repo upon acceptance. But we do not include the data, as they are from other work and are public available.\n\nWe hope our answers addressed you concerns correctly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QPBG5OKxD2", "reviewer": "Reviewer_hoUX", "summary": "The authors consider the problem of structure learning of dynamical systems from irregularly sampled trajectories and partially observed systems. They propose Structural Inference with Conjoined State Space Models (SICSM), a method based on selective state space models (SSMs) and generative flow network (GFNs). The central idea of this work is to use a SSM for modelling the behaviour of dynamical systems while using a GFN to learn the interacting graph structure between the variables of the system. The authors evaluate their proposed approach on a comprehensive set of datasets for various tasks and compare against a numerous baselines.", "questions": "- For the reward defined in Equation 8, what is the explicit form of $R(<G, \\lambda>)$? The authors state that $P(U_{all} | \\lambda, \\mathbf{Adj})$ represents the likelihood model implemented via a neural network. Is this model trained beforehand? Or is the reward being simultaneously learned throughout training with the GFN?\n- A central advantage to using a GFN (or specifically JSP-GFN) to model structure is the ability so approximate the distribution/uncertainty over this structure (and in this case also over the parameters) -- i.e. approximating $P(\\mathbf{Adj}, \\lambda | U_{all})$ instead of just $\\mathbf{Adj}$. In the results, only one deterministic metrics is considered (AUROC). Why not consider a distributional metric to evaluate how well $P(\\mathbf{Adj}, \\lambda | U_{all})$ is approximated, especially given you are comparing to JSP-GFN?\n- What is the motivation of also learning the parameters $\\lambda$ if the primary objective is to learn $\\mathbf{Adj}$? Moreover, there is no evaluation of $P(\\lambda | G)$. If this is an important aspect of the approach, why not include a distributional metrics (as stated in my previous comment), or possibly including evaluation of the negative log-likelihood?\n- What is not entirely clear to me is the use of the state space model (SSM) architecture -- specifically, is $\\mathbf{Adj}$ embedded in the SSM of each residual block? Is the approximated graph structure being used by the SSM or is this an independent output?", "limitations": "The authors discuss limitations and broader impacts in the appendix.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The authors present a method that addresses a challenging problem in the domain structure learning of dynamical systems -- i.e. learning system structure from irregularly sampled trajectories and partially observed systems. The use of SSMs to approximate system dynamics while using GFNs to learn the graph structure of the system is unique and novel approach to this problem. The authors provide a comprehensive evaluation of their method over variety of systems for irregularly sampled trajectories and partially observed systems, demonstrating SICSM consistently outperforms counterpart approaches.", "weaknesses": "- The method has 3 key components: state space model, embedding residual blocks, and a GFN to approximate the graph structure of the system. It is not entirely clear how these individual components interact and the explicit need for the GFN (see questions below). \n- The authors consider a comprehensive set of datasets and baselines, but only one evaluation metrics (AUROC). For example, some other metrics to consider for this task are: structural hamming distance (SHD), F1-score, area under the precision-recall curve (AUPRC). Only considering one evaluation metrics makes it difficult to assess the robustness of the approach.\n- Another method that seems relevant to this work which address an similar problems is CUTS (Cheng et al. 2023). It appears that majority of the baselines considered in this work are or not necessarily methods explicitly tailored to handle irregular time-series. Including a method like CUTS in this evaluation may be important to create a fairer comparison of SICSM. \n\nReferences:\nCheng, Yuxiao, et al. \"Cuts: Neural causal discovery from irregular time-series data.\" International Conference on Learning Representations (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer hoUX for the thoughtful comments. Here are our answers to the questions:\n\n> The method has 3 key components: [...]. It is not entirely clear how these individual components interact and the explicit need for the GFN.\n\nThe state space model in our approach handles the dynamics between sampled features at each time step and adapts to irregular sampling intervals through SSSMs. Residual blocks help extract multi-resolution dynamics, enabling the model to learn node dynamics across varying time resolutions. The GFN then creates a space of hidden states based on these dynamics to reconstruct the underlying structure. This layered approach ensures comprehensive learning and reconstruction capabilities.\n\n> The authors consider a comprehensive set of datasets and baselines, but only one evaluation metrics (AUROC) [...]\n\nMany thanks! We have expanded our evaluation metrics beyond AUROC, as detailed in the PDF attached to the general rebuttal. Figures 4-6 include results for AUPRC, SHD, and F1 score, where our method, SICSM, consistently outperforms baselines across most datasets, highlighting its robustness even in complex structures like those found in the TF dataset.\n\n>  Including a method like CUTS (Cheng et al. 2023) in this evaluation may be important to create a fairer comparison of SICSM.\n\nMany thanks for the suggestion. We investigated into the CUTS  (Cheng et al. 2023) and figured out that they suppose having regular time steps, but some of them are marked out by ZOH placeholders. So actually CUTS work on the regularly sampled trajetories with equal time intervals, but some of them were masked out, which is different from the problem setting of SICSM. Moreover, CUTS can only work on trajectories with one-feature for every node at each time step. In contrast, SICSM is supposed to have irregularly sampled trajectories without any ZOH placeholders, and can work on multi-dimensional features. We performed experiments of CUTS on all of the one-dimenstional features trajectories mentioned in this paper, and the results are shown in Fig. 1, 2, and 4 -6 in the attached PDF. As shown in the figures, CUTS has the ability to match with the best baselines, but still outperformed by SICSM. We included the new results in the revision.\n\n> For the reward defined in Equation 8, what is the explicit form of $𝑅(<𝐺,𝜆>)$? The authors state that $𝑃(𝑈_{𝑎𝑙𝑙}|𝜆,Adj)$ represents the likelihood model implemented via a neural network. Is this model trained beforehand?\n\nThe reward function $R(<G, \\lambda>)$ is transformed logarithmically (see Eq. 37 in Appendix B.5) for implementation purposes. The model $P(U_{all}|\\lambda,Adj)$, implemented via a neural network, is learned simultaneously with the GFN, ensuring integrated optimization and learning.\n\n> A central advantage to using a GFN to model structure is the ability so approximate the distribution over this structure (and in this case also over the parameters). [...] Why not consider a distributional metric to evaluate how well $𝑃(Adj,𝜆|𝑈_{𝑎𝑙𝑙})$ is approximated?\n\nMany thanks for the comment! We evaluate the approximation returned by SICSM by comparing with the exact joint posterior distribution $P(Adj, \\lambda | U_{all})$. Similar to the experiments in JSP-GFN, we consider here models over d = 5 variables, with linear Gaussian CPDs. We generate 20 different datasets of N = 100 observations from randomly generated Bayesian Networks. The quality of the joint posterior approximations is evaluated separately for $Adj$ and $\\lambda$. For $Adj$, we compare the approximation and the exact posterior on different marginals of interest, also called features in JSP-GFN (Deleu et al, 2023), e.g., the edge feature corresponds to the marginal probability of a specific edge being in the graph. Fig. 3 in the attached PDF shows a comparison between the edge features computed with the exact posterior and with SICSM, proving that it can accurately approximate the edge features of the exact posterior. To evaluate the performance of the different methods as an approximation of the posterior over $\\lambda$, we also estimate the cross-entropy between the sampling distribution of $\\lambda$ given G and the exact posterior $P(\\lambda | Adj, U_{all})$. The results are shown in Table 1 in the PDF. We observe that again SICSM samples parameters $\\lambda$ that are significantly more probable under the exact posterior compared to other methods. \n\n> What is the motivation of also learning the parameters $𝜆$ if the primary objective is to learn $Adj$? \n\nThe diverse nature of real-world graph connections, such as those differing significantly in physical properties (e.g., springs vs. electric forces), necessitates modeling each connection with unique parameters $\\lambda$. This individualized approach enhances the accuracy and applicability of our model in complex scenarios.\n\n> Is $Adj$ embedded in the SSM of each residual block? Is the approximated graph structure being used by the SSM or is this an independent output?\n\nThe graph structure $Adj$ is not explicitly integrated into the SSM of each residual block; it is an independent output of our model. We are exploring methodologies to potentially link $Adj$ more closely with SSM operations in future work.\n\nWe hope these clarifications meet your concerns and illustrate the depth and rigor of our study. We are committed to further enhancing our model based on the feedback received and appreciate your guidance in this process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "20GlwIgOYM", "reviewer": "Reviewer_3YzK", "summary": "This paper proposes to combine State Space Models and Generative Flow Networks to perform structural inference in an irregular time series context. The proposed method is evaluated on a series of different tasks where it performs well, and compared to a number of baselines. The method's robustness to short time series and missing observations is evaluated.", "questions": "I'm a bit put off by the framing of the method. The SSSM is the parameterization, the GFN is the optimization method, the structural inference is the task. The ingredients aren't individually novel (e.g. Mamba, ContiFormer), and some of those combos have been tried before (I'm thinking in paricular here of DAG-GFN/JSP-GFN). I don't really see how \"SICSM [..] redefines approaches to structural inference in complex systems\". Maybe what bothers me is that this kind of language obscures the actual contributions of the paper. Many design choices are close to ones taken in [14-15]. I'd encourage the authors to be more careful here. I understand this may come from the authors' lack of familiarity with English, but it creates an unfortunate ambiguity in deciphering what's a contribution and what is just using prior work.\n\nI'm not sure what an $\\alpha$-distance is, is it meant to be a placeholder for any norm?\n\nSection 3.3 introduces the flow-matching condition, but more modern conditions exist, and this work in particular seems to be using DB (**and not SubTB!** as suggested by the appendix text). Why is it only introduced in the appendix if it is the chosen objective? Why not just directly present the DB condition used? This is an example of the similarity to DAG-GFN being somewhat downplayed; I encourage the authors to exercise caution.\n\n\"To enhance the architectural sophistication of our model, we arrange L Residual Blocks in a sequential configuration, with the output of each block feeding directly into the next.\" This is a good example of an off putting phrasing. This describes a standard residual model, but the phrasing in this paragraph (and others in this section) suggests this is somehow a new way to do things. For example, unless I'm missing something, what the authors as \"intricate multi-hop relationships\" is simply a natural and normal consequence of depth in _deep_ neural networks. Either that or the text is not appropriately explaining the uniqueness of the method, which might be even more concerning.\n\nThe trick presented in (9) is neat, but it does imply spending $B$ times more compute. Are baselines also allowed to use this trick? If not the comparisons may be unfair.\n\nIn section 4.3, the objective is taken from [14-15]. Please use proper attribution.\n\nSection 5.4 poses an interesting hypothesis, but it's unfortunate that it is only qualitatively evaluated. Why not run proper experiments and measure the effect of residual depth? \n\nAnother issue more generally is that the design choice of using a residual SSSM model doesn't seem compared to alternatives. What about a deep transformer with exactly the same tricks? What choices matter? It's nice that the effect of the method is analyzed wrt to for example missing observations and compared to baselines, but what about the method with itself, i.e. ablations?", "limitations": "Adequate.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper proposes an interesting architecture and solves problems that have the potential to be very relevant in real world contexts, such as biological time series. The empirical evaluation is fairly thorough about testing on many different tasks.", "weaknesses": "My main concerns for the paper are its low novelty and its low number of ablations, which make it hard to understand how specific pieces contribute to the performance of the method.\n\nGenerally I'm uncomfortable with the way many things are presented in the paper, it's not always clear what's a novel contribution and what's not. I encourage the authors to be clear and exercise an abundance of caution. \n\n/!\\\\ In my humble opinion this paper uncomfortably downplays its similarity to DAG-GFN [14] and JSP-GFN [15] in several places, and I'm not even an author of these papers. This is especially concerning considering that in many instances JSP-GFN is the closest performing baseline to the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer 3YzK for the detailed and thoughtful comments. Here are our answers to the questions:\n\n> My main concerns for the paper are its novelty and its low number of ablations, which make it hard to understand how specific pieces contribute to the performance of the method.\n\nMany thanks for the comment. The core innovation of our work, SICSM, addresses structural inference with irregularly sampled trajectories and partial observation, a novel approach not previously explored in structural or relational inference fields. SICSM effectively integrates SSM and GFN to tackle these challenges, significantly enhancing accuracy in complex dynamical systems.\n\nAdditional ablation studies are detailed in the revised sections of our manuscript. These studies critically evaluate the contributions of individual components within SICSM, underscoring their collective impact on performance enhancement. We show in the other answers on the ablation studies.\n\n> [...] It's not always clear what's a novel contribution and what's not. I encourage the authors to be clear and exercise an abundance of caution. [...]\n\nMany thanks! We have clarified the modifications and novel contributions of the GFN in our system, particularly concerning the reward function adaptations detailed in Appendix B.5, now moved to the main text for greater visibility. Actually, JSP-GFN learns the parameters $\\lambda$ for each node and therefore restore individuity for each edges. This setup is highly align with the real-world: the connections may be isomorphic and we found it to be really helpful for the reconstruction of graph structure for dynamical systems. We have also revised our paper entirely to have clearer reference to DAG-GFN [14] and JSP-GFN [15] to omit misunderstanding of the contribution of this work. Such as on the objective mentioned in section 4.3, and so on.\n\n> I'm not sure what an $𝛼$-distance is, is it meant to be a placeholder for any norm?\n\nYes. The $\\alpha$-distance is indeed a placeholder for any norm, adaptable based on the dynamics of the specific system under study.\n\n> Section 3.3 introduces the flow-matching condition, but more modern conditions exist, and this work in particular seems to be using DB. Why is it only introduced in the appendix if it is the chosen objective? Why not just directly present the DB condition used? \n\nMany thanks! We revised Section 3.3 to better articulate the use of the DB condition, which we initially included in the appendix. It is now prominently discussed in the main text, aligning with its significance in our methodology.\n\n> [...] This describes a standard residual model, [...]\n\nMany thanks for the comment. We just wanted to have more detailed discussion on the advantages of using residual setup here in the text. We revised them to be more precise.\n\n> What the authors as \"intricate multi-hop relationships\" is simply a natural and normal consequence of depth in *deep* neural networks [...]\n\nWe sincerely disagree with this comment. We tried to handle this with regards to the multi-hop relationships illustrated in the Fig. 4 in the submission. As showcase here, on the third column, the red link from node 1 to 4 is a multi-hop connection, while the direct connections would be 1 -> 3 -> 4. So we are trying to state that a finer temporal resolution on the trajectories can help to decrease the possibility of getting multi-hop connections. So with more residual blocks, the learned dynamics will be finer in the temporal aspect.\n\n> The trick presented in (9) is neat, but it does imply spending $𝐵$ times more compute. Are baselines also allowed to use this trick? \n\nYes, the computational trick in Eq. 9 is also employed by baselines. This technique has been verified across all implementations, confirming its efficacy without disproportionately increasing computational demands.\n\n> Section 5.4 poses an interesting hypothesis, but it's unfortunate that it is only qualitatively evaluated. Why not run proper experiments and measure the effect of residual depth?\n\nThe observations in Section 5.4 are backed by rigorous experiments, with detailed quantitative results now included in Table 2 of the attached PDF.  As shown in the table, the negative multi-hop edges are commonly observed among setups with just one residual block, and decreases to 6.5 if we use just one first block, but at this moment, the true positives decrease as well. As shown in the table, the best setup is the concatenation the outputs of all blocks, which on one hand decrease the count of negative multi-hop edges and on the other hand increase the count of true positives.\n\n> Another issue more generally is that the design choice of using a residual SSSM model doesn't seem compared to alternatives [..] i.e. ablations?\n\nWe have expanded our ablation studies to include comparisons with Transformer, LSTM, and GRU models on both irregularly sampled trajectories with 30 time steps and partial observation with 12 nodes of VN\\_SP\\_15 dataset. We show the average results below of 10 runs. The parameters of all of the modules are set to match the length of the trajectories and we performed hyperparamter search with Bayesian optimization. As shown in the table, transformers perform slightly better then LSTM and GRU, but all of them are inferior to SSSM, as they cannot deal with multi-hops effectively. Moreover, they fell short on irregulaly sampled trajectories, as they can not learn $\\Delta$ adaptively. We included these results in revision.\n\n| Module      | Irre. Sampled | Par. Obser. |\n| ----------- | ------------- | ----------- |\n| Transformer | $70.5$        | $60.2$      |\n| LSTM        | $68.1$        | $59.8$      |\n| GRU         | $69.5$        | $59.2$      |\n| SSSM        | $89.4$        | $80.8$      |\n\nWe hope these clarifications and additions address your concerns effectively. We are grateful for your insights, which have significantly contributed to enhancing the rigor and clarity of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ukKArtvxJe", "reviewer": "Reviewer_uuWM", "summary": "The paper introduces the SICSM framework, integrating Selective State Space Models (SSMs) with Generative Flow Networks (GFNs) to tackle challenges in dynamical systems characterized by irregularly sampled trajectories and partial observations. SICSM leverages the adaptive temporal modeling capabilities of SSMs to learn input-dependent transition functions, enhancing structural inference accuracy. It aggregates diverse temporal dependencies and channels them into a GFN to approximate the posterior distribution of the system’s structure. Extensive evaluations across multiple datasets demonstrate SICSM's good performance in accurately inferring complex interactions in partially observed systems.", "questions": "1. How does SICSM handle situations where the interaction structures of the dynamical systems change over time? Are there plans to extend the framework to support dynamic graphs?\n\n2. Can the authors provide more details on the computational resources required for training? Are there any strategies to optimize resource usage?\n\n3. What specific real-world applications do the authors envision for SICSM? Are there particular domains where it has shown exceptional promise?", "limitations": "The authors have adequately addressed the limitations of their work, including the challenges posed by irregular sampling and partial observations. They propose future research directions to explore dynamic systems with mutable structural elements, indicating a proactive approach to potential limitations. The discussion on incorporating prior knowledge and adapting to different hop distances further strengthens the framework’s applicability.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The integration of Selective SSMs with GFNs is a novel approach that addresses significant challenges in structural inference for dynamical systems. The adaptive mechanisms for handling irregular sampling and partial observations are particularly innovative.\n- The research is thorough and well-documented, with extensive evaluations across a variety of datasets. The methodological rigor and comprehensive experimental validation enhance the reliability of the findings.\n- The paper is well-organized and clearly written, with detailed explanations of the methodologies and experimental setups. Figures and diagrams effectively illustrate the concepts and results.\n- The proposed SICSM framework has broad applicability in scientific discovery and system diagnostics across multiple disciplines. Its ability to handle real-world complexities such as irregular sampling and partial observations makes it a valuable tool for researchers.", "weaknesses": "- The implementation of SICSM is computationally intensive, requiring significant resources and expertise. This complexity may limit its accessibility and widespread adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer uuWM for the motivating review! Here are our answers to the concerns:\n\n> The implementation of SICSM is computationally intensive, requiring significant resources and expertise. This complexity may limit its accessibility and widespread adoption.\n\nMany thanks! We acknowledge that the complexity of SICSM, with its dual components of SSSM and GFN, may initially appear daunting. To enhance understanding and facilitate easier adoption, we will include a comprehensive tutorial in the code repository that outlines each step and its purpose within the framework.\n\n> How does SICSM handle situations where the interaction structures of the dynamical systems change over time? Are there plans to extend the framework to support dynamic graphs?\n\nMany thanks for the good question! Addressing changing interaction structures in dynamical systems is indeed challenging, particularly in distinguishing between existing and emerging connections which could blur the reconstructed adjacency matrix. We are exploring the potential use of latent variables to capture these dynamics more accurately. This approach is in its early stages and will require further research and validation.\n\n> Can the authors provide more details on the computational resources required for training? Are there any strategies to optimize resource usage?\n\nWe included the training time of each method in Table 4 in the appendix. We trained all of the methods with a single NVIDIA Ampere 40GB HBM graphics card, paired with 2 AMD Rome CPUs (32 cores@2.35 GHz), as mentioned in Section 5.1 of our submission. As we were trying to bridge the Generative Flow Networks with the SSSM, yet only JAX implementations were available, but it comes without the GPU acceleration of SSSM, which was included in its PyTorch implementation. We will work on the way of either implementing the GPU acceleration of SSSM with JAX, or trying to rebuild the GFN with PyTorch.\n\n> What specific real-world applications do the authors envision for SICSM? Are there particular domains where it has shown exceptional promise?\n\nYes, SICSM is particularly suited for complex domains such as single-cell biology, where it can infer gene regulatory networks, and other scientific fields requiring discovery of latent connectivity among variables. Its ability to handle irregularly sampled data and partial observations makes it a valuable tool for scientific discovery across various disciplines.\n\nWe hope these clarifications address your concerns, and we are committed to further refining SICSM to enhance its accessibility and applicability in real-world scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We are deeply grateful for the detailed reviews and constructive feedback provided by Reviewers 2uCG, hoUX, 3YzK, and uuWM. We appreciate the recognition of the novelty and applicability of our work in addressing the complex challenges associated with dynamical systems through the Structural Inference with Conjoined State Space Models (SICSM).

The innovative integration of Selective State Space Models (SSMs) with Generative Flow Networks (GFNs) has been recognized for effectively handling complex challenges associated with dynamical systems. Reviewer hoUX noted our "unique and novel approach to the problem of learning system structure from irregularly sampled trajectories and partially observed systems." This underscores the innovative nature of our methodology, particularly in adapting to the irregularities and complexities of real-world data.

The extensive evaluation across multiple datasets and comparison against numerous baselines has demonstrated the superiority of SICSM, particularly in scenarios involving irregularly sampled or partially observable nodes. Reviewer 2uCG mentioned, "The paper presents experiments on 16 datasets... and compares to 7 other models, showing superiority in scenarios where observations are irregularly spaced or nodes partially observable." This highlights the robustness and applicability of our approach in varied real-world settings.

The clarity of our paper's presentation, including its coherent notations and supportive diagrams, has been positively highlighted. Reviewer uuWM appreciated that "The paper is well-organized and clearly written, with detailed explanations of the methodologies and experimental setups." This feedback validates the effort put into ensuring that the sophisticated architecture of SICSM is accessible and understandable.

The potential applicability of SICSM in practical scenarios, such as biological time series analysis and system diagnostics, has been emphasized as a significant contribution. Reviewer 3YzK stated, "The paper proposes an interesting architecture and solves problems that have the potential to be very relevant in real world contexts, such as biological time series." This comment supports our claim that SICSM can be a valuable tool for scientific discovery across multiple disciplines.

We acknowledge the concerns regarding the computational intensity of implementing SICSM, as noted by Reviewer uuWM: "The implementation of SICSM is computationally intensive, requiring significant resources and expertise." We are committed to addressing this by developing tutorials and exploring optimizations to make SICSM more accessible and user-friendly.

In response to the feedback, we will enhance the manuscript to better clarify our novel contributions, particularly distinguishing them from related works like DAG-GFN and JSP-GFN. Reviewer 3YzK's advice to "exercise an abundance of caution" in presenting our contributions will guide our revisions to ensure clarity and accuracy. We will also revise the paper to include the new plots with shadings showing the standard deviations, as well as experimental results with CUTS (Cheng et al. 2023) and ablation studies.

As for the concerns and questions raised by each reviewer, we answered them individually under each review. We hope we have addressed the concerns and answered the questions correctly.

We are encouraged by the constructive critiques and the positive comments on the potential and performance of SICSM. Our team is committed to continuous improvement and is excited about the future contributions our work can make to the field of structural inference and beyond.

Thank you once again for your invaluable input and the opportunity to contribute to this esteemed conference.

Please check the attached PDF for revised figures and more experimental results.

Warm regards,

Authors of SICSM

Meta-review: The proposed SICSM framework presents a novel approach by integrating State Space Models (SSMs) and Generative Flow Networks (GFNs) to address the complex problem of structure learning in dynamical systems with irregularly sampled trajectories and partially observed data. The authors have demonstrated the model's superiority over multiple baselines on synthetic datasets and have effectively responded to reviewers' concerns by providing additional experiments and clarifications regarding the model's components and contributions. While the reliance on synthetic datasets and limited real-world data were identified as limitations, the paper's contributions and its potential applications in diverse domains underscore its value to the field and support its acceptance at NeurIPS.
Decision: Accept (poster)