Robust Latent Neural Operators for a Family of Systems with Sparse Observations

Authors: Xin Li, Chengli Zhao, Xue Zhang, Xiaojun Duan
Primary area: learning on time series and dynamical systems
Keywords: Neural operator, complex system, variational autoencoder, recurrent neural network

Abstract:
Neural operator methods have achieved significant success in the efficient simulation and inverse problems of complex systems by learning a mapping between two infinite-dimensional Banach spaces. However, existing methods still exhibit room for optimization in terms of robustness and modeling accuracy. Specifically, existing methods are characterized by sensitivity to noise and a tendency to overlook the importance of sparse observations. Therefore, we propose a robust latent neural operator based on the variational autoencoder framework. In this method, an encoder based on recurrent neural networks effectively extracts sequential information and dynamical characteristics embedded in sparse observations. Subsequently, a neural operator in latent space and a decoder facilitate the modelling of the original system. Additionally, for certain higher-dimensional systems, opting for a lower-dimensional latent space can reduce task complexity while still maintaining satisfactory modeling performance. We conduct experiments across several representative systems, and the results validate that our method achieves superior modeling accuracy and enhanced robustness compared to the state of the art baseline approaches.

Decision: reject
Reviewer scores: 5, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1729138710706

thread_type:
official_review

<thread_object>
creator: reviewer_n6rd

<content>
summary:
This paper introduces a latent RNN-based neural operator designed to handle noise and sparse observations in solving partial differential equations (PDEs). The method is trained using a variational auto-encoder framework. While similar frameworks have been explored in literature on irregularly-sampled time series, this work proposes a novel encoder based on DeepONet to address the limitations in effectiveness and efficiency seen in existing approaches. The proposed method is validated on two 1D PDEs (the Diffusion-Reaction equation and the Kuramoto-Sivashinsky equation) and one 2D PDE (the Navier-Stokes equation).

strengths:
This paper addresses a critical limitation of existing neural operators in handling noise and sparse data. The proposed method shows promising results in overcoming these challenges. The experimental setup and data preparation are clearly outlined, and the authors have also provided the implementation code in the supplementary materials.

weaknesses:
The primary contributions of this paper are twofold: 1) the introduction of the latent RNN framework to address noise and sparse data in neural operators, and 2) the proposal of a new encoder based on DeepONet. However, several key issues remain to be addressed:

1. **The rationale behind the latent RNN framework in handling noise and sparse data is insufficiently explained**. I recommend that the authors conduct a more in-depth analysis of the differences between RNN-based methods and existing neural operators to better explain the performance gain of the proposed method. It would be helpful to provide additional insights to further justify the motivation behind the proposed method.

2. **While the DeepONet-based encoder introduces some modifications, its novelty appears limited**. The proposed framework (Equation 2) closely resembles that of [Rubanova et al., 2019] (Equations 5-7), with the primary distinction being the introduction of OPERATORsolve in place of ODESolve. This corresponds to the replacement of existing encoders (RNN-Decay and ODE-RNN) with OPERATOR-RNN. This DeepONet-based encoder is designed to enhance the effectiveness of RNN-Decay in handling irregularly sampled time series and to improve the computational efficiency of ODE-RNN. However, according to the ablation study, the performance improvement of the proposed encoder compared to RNN-Decay is not significant (Ab2 in Tables 2 and 3), especially considering the standard deviation. Additionally, there is no comparison of computational complexity with ODE-RNN to further substantiate the authors’ claims.

3. **The concept of neural operators utilizing latent space has been explored in previous works [Yin et al., 2022; Serrano et al., 2023]**. These studies employed an auto-decoder framework with implicit neural representations to learn the mapping between data space and latent space, as well as to model latent dynamics using Neural ODEs. While their primary focus is on spatially sparse data and temporal extrapolation, their methods appear applicable to the noise and temporally sparse scenarios considered in this paper. The contribution of this work could be further enhanced by comparing the proposed method with these approaches in both spatially sparse and temporally sparse settings.

I am willing to raise the score if the aforementioned issues are effectively addressed.

**Reference**

[Rubanova et al., 2019] "Latent ordinary differential equations for irregularly-sampled time series." NeurIPS 2019.

[Yin et al., 2022] "Continuous pde dynamics forecasting with implicit neural representations." ICLR 2022.

[Serrano et al., 2023] "Operator learning with neural fields: Tackling pdes on general geometries." NeurIPS 2023.

questions:
1. The parenthetical and textual citations are incorrectly used throughout the paper. 

2. To provide a more comprehensive ablation study, consider replacing the OPERATOR-RNN encoder with ODE-RNN in an additional experiment. This configuration would be equivalent to Latent ODE [Rubanova et al., 2019]. While the authors have compared their model to Latent ODE in Table 2, the settings used in that comparison differ from those in the ablation study. By including this additional experiment, a direct comparison between OPERATOR-RNN, RNN-decay, and ODE-RNN can be conducted, offering a clearer understanding of their relative performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729408765437

thread_type:
official_review

<thread_object>
creator: reviewer_ctqp

<content>
summary:
In this paper, the authors focus on the challenges of robustness to noise and sparse observation in neural operators. To tackle these issues, they introduce a latent neural operator that builds upon a VAE framework, leveraging recurrent neural networks to extract information from sparse data. The authors employ a neural operator to infer dynamics in the latent space, followed by a decoder to reconstruct the original data. The extensive experiments conducted across various systems demonstrate the superiority of the proposed method compared to existing baselines.

strengths:
1. The focus on robustness to noise and the capacity to handle sparse observations are critical considerations for neural operators, thus it is appreciated that this work focuses on these two important issues.
2. The proposed method, named RLNO, is somewhat new by extending the DeepONet within a VAE framework.

weaknesses:
1. The inference for the posterior distribution of $z_0$ is somewhat unclear. Why do we need to model $ z_0 $ based on so many observations? Should it not be sufficient for $ z_0 $ to depend solely on $ s_0 $? If so, might it be possible to directly mitigate the issue of irregular sampling in the latent space using an operator solver?
2. The preservation of the functional properties of the proposed method needs demonstration. It is uncertain whether the overall framework retains its status as an operator after the incorporation of multiple non-local operators.
3. In Eq. (7), a more detailed description of $p_{\theta,\theta_{dec}}(s_0, s_1, \dots, s_T|z_0)$ would enhance comprehensibility. Additionally, I am confused by the use of $\theta$ and $\tilde{\theta}$, are they same?
4. The manuscript requires improvements in its written presentation. Some writing suggestions are given below.

questions:
Several writing suggestions:
1. Line 129, if Fig.3 should be Fig. 1?
2. Line 236, repeated ``equation’’
</content>
</thread_object>

</thread>
<thread>
creation date:
1730290507195

thread_type:
official_review

<thread_object>
creator: reviewer_wuxe

<content>
summary:
This paper proposes the Robust Latent Neural Operator (RLNO) that utilizes a recurrent neural network (RNN) encoder and the variational autoencoder (VAE) framework within a latent neural operator learning. The authors claim that the proposed model is more robust to the observation noise and learns the dynamics of a system more accurately.

strengths:
This work introduces mainly two extensions of an existing work, Latent Neural ODE (LNODE) solver. First, it extends the LNODE solver to the operator learning framework. Second, the ODE-RNN encoder of LNODE is replaced with OPEARATOR-RNN in which the neural operator is used instead of an ODE solver. The idea is straightforward, and the proposed integration of each component looks natural and reasonable. The experiments show that the improvement of the proposed model is significant.

---

**After the discussion**, the contributions of the work were getting better recognizable; thus, I raised my score to 6 from 5.

weaknesses:
The extensions rely on existing ideas (such as neural operator/DeepONet, latent space neural ODE, VAE/Evidence Lower Bound loss); thus, the manuscript needs more thorough revision for a clearer presentation of its technical novelty. For example, Algorithm 1 is almost identical to the work of [Rubanova et al. 2019]; thus, it would be worth highlighting the distinctive novelty.

The experiments were evaluated mostly for the "interpolation" task. The "extrapolation" evaluation should be further discussed as shown in the related work (e.g., LNODE).

questions:
For OPERATORSolve, why are the two different neural operators (trained for different parameters $\theta$ and $\tilde{\theta}$) used? Does the proposed method train and use them independently for forward and backward evolution? If so, why?

The extrapolation evaluation should be further clarified. As described, the DR and KS experiments used the sparse observation data sampled from 100 steps of ground truth data and added noise to them. The predictions are evaluated and analyzed for 100 steps. What would the performance be if the prediction happens longer for, e.g., 200 steps and 400 steps? Alternatively, what if the sparse observation data is sampled only at the beginning of the ground truth data?

Minor questions should be clarified:
- L129: The text refers to Fig.3. Is this correct? Maybe Fig. 1?
- L194: Shouldn't it be "encoder" instead of "decoder"?
- L259: What is the RCDI method?
- L329: The text refers to Figure 3(c). Is this correct? Maybe Figure 3(b)?

Minor corrections are required:
- L57: typo "adetly"
- L99: missing space "structuresHao"
- L236: duplicated word "Equation equation 3"
- L309: typo "parse"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706494575

thread_type:
official_review

<thread_object>
creator: reviewer_vdlf

<content>
summary:
In this work, the authors introduce RLNO, a robust latent neural operator grounded in the VAE framework, to address the drawbacks of neural operator methods that lack robustness and accuracy due to their sensitivity to noise and inability to handle sparse data. RLNO comprises an RNN-based encoder to capture sequential and dynamical information from sparse observations, a latent space neural operator for modeling, and a decoder to reconstruct the original system. Experimental results across ODE and PDE systems verify RLNO's superiority in both accuracy and robustness.

strengths:
The work presents a derivation of an evidence lower bound that transforms the optimization problem from maximizing observation probability to maximizing the bound, which provides a different approach to the optimization process. The study includes experimental evaluations across multiple test cases.

weaknesses:
- The major limitation of this work lies in its insufficient justification for latent space dynamics inference, particularly since DeepONet already provides established methods for iterative dynamics inference. This design choice lacks clear comparative advantages over existing approaches.

- Though the encoder-decoder architecture is introduced to address sample complexity and sparse observations, the paper falls short in demonstrating how these components preserve DeepONet's fundamental functional properties. This gap raises concerns about the theoretical soundness of the proposed framework.

- The current manuscript structure and presentation hinder a clear understanding of the methodology and contributions.

questions:
please see above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731745807168

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
**General response:**

We would like to thank the reviewers for your time, efforts, and valuable comments and suggestions, which do help us to significantly improve the quality of this work. Due to potential shortcomings in readability of our manuscript, we have found that the reviewers **may have some misunderstandings about our method**. Hence, we succinctly summarize the **novelty** and **contributions** of our work as follows. 

(1) **Compared to traditional neural operator approaches, our method can utilize more domain-specific information.** Current neural operator methods, including DeepONet, often only input sampled functions (such as parameter functions, initial-boundary conditions, etc.) when tested in new domains. However, in many practical applications, there may exist prior observational data at several moments in this new domain, which is often overlooked in previous methods. Our research has found that fully leveraging these sparse prior observational states has a very significant effect on promoting the robustness and accuracy of neural operator modeling. This benefit arises from domain-specific observations, which not only provide more state information but also extract specific dynamic information from these sequential observations.

(2) **Embedding sparse observations from new domains into neural operators requires a sophisticated framework design.** To fully excavate and utilize the dynamic information within sparse observations, we develop a novel learning framework based on a variational autoencoder and adopt an RNN-based encoder. This method fully leverages the advantages of RNNs and neural operators to better model a family of dynamical systems.

(3) **Design an OPERATOR-RNN encoder to better handle unevenly spaced sparse observation data.** Here, we draw inspiration from the design of the ODE-RNN encoder and design an OPERATOR-RNN encoder, which can more robustly and efficiently extract domain-specific dynamic information from sparse observational data. It should be noted that in terms of computational efficiency, our method inherits the advantages of neural operators and has a very significant advantage over methods such as Latent ODE.

(4) **Our method can select a smaller latent space dimension, thereby reducing the complexity of the operator modeling task.** Results indicate that lower-dimensional latent states achieve impressive modeling accuracy. This dimensionality reduction in the latent space decreases the task's complexity, facilitating the neural operator's ability to capture essential features with limited training data. Consequently, this approach can be extended to modeling tasks of higher-dimensional complex systems. In addition, our approach exhibits higher robustness and accuracy compared to previous latent neural operator methods, attributable to point (1).

In summary, every component within our method is essential and valuable. To substantiate this claim, we select a variety of baseline methods (including approaches based on RNNs, classical neural ODEs, classical neural operators and latent neural operators), and conducted numerous ablation studies to affirm the significant contribution of each component in our methodology. Additionally, to enhance the reliability and reproducibility of our experimental results, we also provide all the experiment codes in the supplementary materials. Moreover, to prevent misunderstandings and further enhance the readability of the manuscript, we have carefully revised and refined the expressions throughout the paper and enriched the details in Fig. 1.


**General response 2:**

We have meticulously addressed all the issues and suggestions of the reviewers and have successfully **submitted a revised manuscript**. Specifically, **we have made the following enhancements:**

* 1. We have supplemented and refined the introduction section, succinctly summarizing the innovative aspects of our work.
* 2. We have corrected typographical errors as pointed out by the reviewers and conducted a thorough revision to enhance the clarity and expressiveness of the entire manuscript.
* 3. We have revised Figure 1 and its associated description, thereby increasing the readability of the method in the paper.
* 4. We have refined the details of our methodology in the main text and augmented Appendix A.2 with the pseudocode for the proposed RLNO method to enhance the readability of the paper.
* 5. We have supplemented our paper with ablation experiments on the ODE-RNN encoder to more effectively validate the robustness and accuracy of our approach.
* 6. We have supplemented the Universal Approximation Theorem for RLNO, as well as elucidated the significant role of the Variational Autoencoder framework, detailed in Appendix A.3.

Finally, we thank all the reviewers again for your valuable and insightful comments. We hope that our General response as well as the individual responses for each reviewer adequately addresses the reviewers’ concerns.
</content>
</thread_object>

</thread>


Metareview: