Learning Spatiotemporal Dynamical Systems from Point Process Observations

Authors: Valerii Iakovlev, Harri Lähdesmäki
Primary area: generative models
Keywords: dynamics, spatiotemporal, neural, PDE, ODE

Abstract:
Spatiotemporal dynamics models are fundamental for various domains, from heat propagation in materials to oceanic and atmospheric flows. However, currently available neural network-based spatiotemporal modeling approaches fall short when faced with data that is collected randomly over time and space, as is often the case with sensor networks in real-world applications like crowdsourced earthquake detection or pollution monitoring. In response, we developed a new method that can effectively learn spatiotemporal dynamics from such point process observations. Our model integrates techniques from neural differential equations, neural point processes, implicit neural representations and amortized variational inference to model both the dynamics of the system and the probabilistic locations and timings of observations. It outperforms existing methods on challenging spatiotemporal datasets by offering substantial improvements in predictive accuracy and computational efficiency, making it a useful tool for modeling and understanding complex dynamical systems observed under realistic, unconstrained conditions.

Decision: accept (spotlight)
Reviewer scores: 8, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730217328186

thread_type:
official_review

<thread_object>
creator: reviewer_nyvc

<content>
summary:
This paper presents a novel method for modeling spatiotemporal dynamical systems from point process observations. The model integrates techniques from neural differential equations, neural point processes, implicit neural representations, and amortized variational inference. The authors also introduce a technique to speed training by addressing a computational bottleneck in latent state evaluation. The experimental results demonstrate the effectiveness of the model on challenging spatiotemporal datasets.

strengths:
- The paper is well-written and technically sound. The methodology is clearly presented, and the experimental setup is detailed
- The proposed model is technically sound. It effectively combines techniques from various fields, including neural differential equations, neural point processes and amortized variational inference
- experiments and "ablations studies" are comprehensive, showing the impact of many parameters of the model

weaknesses:
- While focusing on predictive capability and computational efficiency, discussing the interpretability of the model would enhance its value. Can something be said about the dynamical system?
- A little more discussion around the limitation of the Poisson process, and potential solution would have been welcome.

questions:
Questions are related to the weaknesses:
Could you address the issue of interpretability and the Poisson process a bit more
</content>
</thread_object>

</thread>
<thread>
creation date:
1730387040411

thread_type:
official_review

<thread_object>
creator: reviewer_bch2

<content>
summary:
A composition of different ML methods is presented to simulate spatiotemporal processes from point observations without access to the whole spatial field. The proposed approach encodes sparse context point observations into a latent state. In this latent state, the dynamic process evolution is integrated over large time steps, while a fine temporal resolution is obtained via interpolation in the latent state. A decoder projects the predicted latent states back to the observation space.

strengths:
_Originality:_ The combination of various different branches from machine learning is original. They are composed in an elegant and versatile way to solve spatiotemporal problems efficiently. The use of latent states enforces abstractions that push for generalizability. Intriguingly, the introduced method does not rely on complex architectures, but mostly emerges out of various MLPs that are well placed and wired.

_Quality:_ Claims are well supported with experimental results, which are contrasted against several recent  and competitive ML architectures. Figures are well designed and support the message conveyed in the manuscript.

_Clarity:_ The manuscript is well organized, structured, and written. A rich appendix provides details about the model design, yet a supplementary material to validate the results is missing.

_Significance:_ Results appear significant in terms of how sparse the data is sampled. Three synthetic problems of varying difficulty, as well as a real-world problem demonstrate the applicability of the method. Results are reported in two metrics along with standard deviations, which helps assessing the quality of the forecasts.

weaknesses:
1. Observation function is constrained to a normal distribution with fixed variance. It would be helpful to add arguments of this design choice, to what extend it limits the expressivity of the model, as well as for what problems this formulation is sufficient.
2. Ablations showing the performance under different spatial and temporal sparsities would be highly informative to understand the quality and limitations of the model at different tasks. Presumably, e.g., Navier-Stokes likely depends on more dense samples compared to Shallow Water. Extending this ablation to the other benchmarked methods would also provide valuable insights about the models' data efficacy.
3. Limitations are not reported. It would be valuable to understand the limits of the method, its computational cost, and the time this architecture needs to train. Also, it is unclear to which extend the method can directly be applied to a task at hand or how much fine tuning is involved.
4. No runtime comparison of the different models provided. If I'm not mistaken, the model must be called for each spatial position of interest in each time step, which amounts to a large number of model calls. Thus, to extend on Table 1, please provide information about the runtime of the entire model when generating a rollout of a spatiotemporal sequence of frames.
5. More details about the differences between the introduced method and AutoSTPP would be valuable, given that these two approaches perform almost equally well. For what reason is your method superior to AutoSTPP?

_Minor Comments_
- Typo in line 306, "withing"
- $N_{\text{ctx}}$ is unclear in Figure 4. What value does the variable take? Would be good to have the actual value. EDIT: C.1 provides this information; I thus suggest to refer to C.1 in the Caption of Figure 4.

questions:
1. For what reason is the distribution of the next sensor signal's location predicted? What is the benefit of such a prediction and what computational cost does it impose? If I understand correctly, Table 2 suggests removing the point process model (which simulates the next sensor signal position and time, if I'm correct). At least according to a minimal model when following Occams Razor.
2. The interpolation ablation is very illustrative. Have you tried higher-order interpolations to infer $\hat{z}(t_i)$, i.e., quadratic, cubic? What is the error that incurs from the interpolation compared to modeling the full temporal grid $t_1, \dots, t_N$? Table 1 demonstrates the time improvement when using interpolations; it would be great to see the error associated with the two techniques (Interp. vs Seq.).
3. Have you explored other solvers beyond dopri5, such as Euler, which is much cheaper? Or does the method depend on an adaptive solver to account for potentially different deltas between time steps? Figure 2 somehow suggest that the effectively processed time steps $\tau_m$ are separated by a constant time delta. Is this a requirement of the architecture?
4. How does the latent space dimensionality $d_z$ affect the runtime? Might be interesting to report along with its effect on the parameter count around line 375.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695807623

thread_type:
official_review

<thread_object>
creator: reviewer_f2uy

<content>
summary:
This article is devoted to the problem of learning spatiotemporal dynamics from randomly sampled points in space and time. This problem is particularly well suited for the situation where we have sensors that record a system, and we have to predict also the behavior of the sensors during the dynamics (e.g. meteorological sensors that are carried by currents). The method proposed in this article is based on the use of neural ODEs in a learned latent space.

strengths:
-- The problem of learning spatiotemporal point processes is rather important, and any contribution to this problem should be well welcomed by the scientific community.
-- The overall idea of the article is meaningful. 
-- The numerical results are rather good.

weaknesses:
-- Some explanations are not properly given. For instance, I assume that they are using an ODE solver in a latent space because a direct approach would immediately incur into stiffness problems. Why not using a neural PDE solver? Why is it better to learn a latent space and use an ODE solver for a problem that is formulated as a PDE (as in Eqn 5 of the paper)? This is unclear.
-- The latent approach makes the approach less clear, and more out of the control of the user. I suppose the authors have no idea why the encoder creates a certain latent space rather than another. A theoretical approach seems very complicated, in fact the authors limit themselves mostly to empirical results. 
-- It is unclear if a general system can be learned in this way. In a sense, we might think of the encoded latent space as a low-degree approximation of the system, but it might be that certain PDE models stemming from Eqn 5 might not be suitably tackled by such approach. 
-- One of the main claims is that the model is continuous. An interpolation task should be performed in this case to show that they can handle continuity well. They use interpolation in the method, but it is unclear if in an experiment where portions of the trajectories are completely hidden during training, could be recovered during evaluation.

questions:
My main questions relate the points raised in the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720906233

thread_type:
official_review

<thread_object>
creator: reviewer_ueak

<content>
summary:
This is an engineering oriented work that model STPP with intensity driven by a continuous latent states governed by a Neural-ODE, with initial states generated by a transformer encoder. The formulation sounds valid and the proposed benefits are for sparsely distributed data. The main contributions are the new formulation and the interpolation-based speedup technique.

strengths:
- The challenge seems well grounded as the sparse data over a large spatial domain is common for many types of problems, e.g., few agent trajectories over a large geographical domain.
- The method looks (possibly) scalable with low-resolution linear interpolation.
- The math formulation is clear and the empirical results are fair. 
- A lot of ablation study, accounting for context size, resolution, removal of components

weaknesses:
* I don't think the paper really answer the question of why it work on sparse data. There is no theoretical analysis / visualization of how the low-dimensional latent space captures the full dynamics from sparse observations. No discussion of information-theoretic bounds on what can be learned from sparse observations. It is reasonable to expect normalizing-flow based method (like Neural-STPP) not working well because the distribution is too localized, but I don't see why your method have an theoretical advantage over SOTA with kernel-based or closed spatial distribution.

questions:
* Can you give me a possible explanation of why it works?
* There is no ablation study on why transformer are used for generating the initial states. Or do you have evidence the initial state part is robust to architecture choice?
* Despite the proposed speedup method, I believe neural-ODE is still untolerably slow and does not scale well. Do you have actual training/ inference time comparison?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a method for learning to model spatio-temporal processes from data that is irregularly sampled in both the spatial and temporal dimensions. It employs an encode-process-decode framework and integrates the following components: point processes (non-homogeneous Poisson processes) for capturing irregular information and dynamics in a latent space, a neural ODE for time-stepping in the latent parameter space, variational inference for learning the parameters of the posterior distribution for the initial latent state, and Implicit neural representations for decoding. The model is evaluated on four datasets and compared against various baselines.

All reviewers acknowledge the novelty of the contribution, which combines existing components in an innovative way. They consider that the authors' claims are well-supported by the experiments and ablation studies. I recommend acceptance.