Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work presents an extension to the 4D-Var approach to data assimilation geared towards application on climate data based on neural operator learning. To do so the authors split the neural operator into three distinct operators, the flow-operator, the perturbation-operator, and the observation operator. With a checker board-style training algorithm the individual operators are trained and then tested on two Lorenz-benchmarks.
strength_and_weaknesses: Strengths:
- Clear algorithm derivation, with a very clear exposition of its individual components which make the contribution of the authors very clear.
- Strong relation of the algorithm design to the roots in the physical process itself, hence making this a true physical process-driven design.
- Highly intriguing approach to keep the neural operator's dynamics faithful to the dynamics of the physical process. An approach which should also carry over to similar problems in scientific machine learning
- Strong performance of the algorithm across the presented benchmarks, clearly outperforming previous results.

Weaknesses:
- Missing discussion of the training costs of the proposed algorithm, which make it hard to gauge the extensibility of the algorithm to larger, more complex problems.
- No clear ablations of the influence that each of the 3 operators has on the general performance of the approach. While not immediately clear to the reviewer if the algorithm could be decomposed as such, it would greatly improve the technical depth of the paper if the influences of each operator on the performance could be discerned.
- Evaluations are restricted to the Lorenz-63, and the Lorenz-96 system. Evaluation on more difficult and/or non-Lorenz systems would be of great help to evaluate its applicability to other chaotic systems.
clarity,_quality,_novelty_and_reproducibility: The work is presented with a lot of clarity and focus on its scientific story. As such the writing is of consistently high quality with minor typos at times (see below), and has a significant amount of originality in the design of the operator learning approach. While the individual components, and their combination together are novel, the depth of the paper could be enhanced by relating the flow operator learning approach back to the iterative refinement used in e.g. diffusion models. The checker board-style training approach could also be further rooted in preceding literature. While not immediately comparable, highly similar approaches have appeared before and further references could help to underline the intellectual thread this training approach builds upon.

While the choice of using the U-Net architecture is widespread, and has shown good performance in literature, I would encourage the authors to include discussion of the work of Wang et al. [1], which showed that U-Nets are not always the best choice of network for chaotic dynamic / fluid dynamics, and might present a limitation of the presented approach when being extended to other more difficult chaotic systems.

[1] Wang, Rui, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. "Towards physics-informed deep learning for turbulent flow prediction." In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1457-1466. 2020.

Minor typos in the text:
- pg 1: Four-dimensional variational assimilation (..) -> The four-dimensional (..)
- pg 2: The aim of learning observation operator is (..) -> The aim of learning observation operators (..)
- pg 2: to use neural network to construct a map (..) -> to use neural networks to (..)
- pg 3: the bias of prior model (..) -> the bias of the prior model (..)
- pg 4: To tackle with this difference, we carry out dimensionality reductions in U-Net (..) -> we carry out a dimensionality reduction in the U-Net (..)
- pg 9: The findings above also holds for the (..) -> The findings above also hold for (..)

summary_of_the_review: The authors present a compelling extension to existing 4-D Var data assimilation approaches with a neural operator learning framework, which splits its operators into three distinct operators fulfilling different complementary functions. Overall a strong paper with a very clear exposition, and presenting a novel approach to operator learning for data assimilation, and showing strong performance on Lorenz-systems, the paper would benefit from more ablation analyses, examination across a wider spectrum of benchmarks, as well as at times more references to adjacent literature such as iterative refinement, and similar training algorithms.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This study proposes a new method for data assimilation, i.e., for learning model-constrained dynamics from empirical data and estimating the state of the system of study. The method is composed of three types of networks (the observation operator, the flow operator and the perturbator), allowing it to flexibly balance the problem of reconstruction of the system latent states and the conforming to the specified model dynamics. On two classical problems in weather sciences, the Lorenz-63 and Lorenz-96, the method is shown to achieve higher performance than the state-of-the-art method 4D-Var (Fablet et al. 2021) in terms of reconstruction, dynamics loss, and computational cost.
strength_and_weaknesses: The paper is technically sound, previous work is cited and discussed, and claims are for the most part well supported by empirical evaluation. However, I have a few concerns I would appreciate the authors to address:

-Table 1 shows the average results over 10 independent tests. It would be important to also report the respective standard deviations and assess the empirical performance in light of these standard deviations (e.g., performing statistical tests);

-Table 1 reports test times. What is considered test times? On one hand, this should be clarified in the manuscript, and on the other hand, training times should also be reported (if reporting training times does not make sense, then this should be justified);

-Figure 5 quantifies the performance of the new method under different degrees of bias in the dynamics. While this is a useful experiment, it would be equally important to show how the state-of-the-art method performs under these same circumstances.
clarity,_quality,_novelty_and_reproducibility: To my knowledge, the proposed method is novel. The novelty resides in the fact that the method is composed of three types of networks, allowing to flexibly balance the different problem constraints.

However, the quality of the paper could be improved (see section on weaknesses). In addition, the lack of clarity impedes the full appreciation of the quality and significance of the work. In particular, the paper contains a substantial amount of typos and imprecisions that I would appreciate the authors to address:

-"and much more novel and bolder". This wording is slightly subjective, so I would refrain from writing this;
-several instances where either the article "a" is missing or the plural form should be used: "The aim of learning observation operator is to use neural network", "and another group of works uses neural network", "first work that directly uses neural network to";
-"exists a constant C" rather than "exits a constant C";
-Figure 2 legend. After "denote", "for" should be dropped, so "We denote K the spatial..., T the total...T_o the sequence..., T_h the intermediate...";
-"To tackle this difference", instead of "To tackle with this difference";
-page 5, "The major function of the flow operator is to restore the reconstructed sequence of states with a relatively high dynamic loss". Shouldn't it be "low dynamic loss"?
-"We train the flow operator with the dynamic loss, and the training process is self-supervised because no additional label is required". The authors should be explicit about why the training is self-supervised, and remind the reader what is the data the network is being trained on;
-the authors should mathematically define what L_{dyn-pr(\hat x)} is;
-"As long as inequality 9 is not satisfied, we end the current training phase and move on to the next one". Up until this point in the manuscript, it is unclear that the method is composed of several training phases, so I would suggest the authors to clarify this early in the manuscript. Perhaps the authors could also add an algorithm box early in the manuscript, so that the reader can follow how each step fits into the whole algorithm;
-typo in "Specifically, The perturbator";
-in Table 5, the abbreviation "GENN aG-LSTM" should be introduced;
-"Lorenz-63 and Lorenz-96 ... Both systems are one-dimensional." The systems are not one-dimensional neither in terms of state variables nor in terms of parameters. What do the authors mean by "one-dimensional"?
-typo in "ground truth dynamic";
-in Figure 4, bottom left panel, please describe what the dashed blue line corresponds to and discuss it in the text;
-Figure 5,"The left panels show" and "The right panels show".
summary_of_the_review: Overall, the paper is technically sound and has a novel methodological contribution with somewhat convincing empirical results. However, some of the empirical results need to be further expanded and the clarity of the manuscript could be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: ## Summary

The authors propose a new end-to-end fully data-driven method for data assimilation in which the different components of the system are fully parameterized using neural networks. Their model enjoys good accuracy and across different dynamical system parameterizations, specifically on the classical Lorenz systems.

## Review Summary

 Promising work but requires significant improvements.
 
1. Please clarify the role of prior dynamics in the model / training procedure. 
2. The claims regarding state-of-the-art, especially regarding 4DVar should be weakened (1) there is no explicit comparison to 4DVar, (2) there are no experiments approaching the scale or real-world applications in which 4DVar is used. The first pages of the paper read as if the authors recommend this as a substitute.

## High-level Feedback

- "Complicated modeling and solving process of 4DVar..."
	- If this is an important model, please remind the read of the basic outlines of how this model works. Ideally, within the mathematical framework that you use to define your model.
- The definition of prior dynamics need to be defined early so that the paper makes sense. It is very clearly defined, but only in the results section of the paper. Essentially, it is the parameters of the dynamical system in question.
	- The experimental setup regarding generalization across prior dynamics needs to be clarified. Either this or the training setup. It's hard to know without answering the question: How do the prior dynamics enter your model / training setup? This key question should be clarified multiple times, e.g. in text as well as in diagrams and figures.
- Please make this explicit: Your argument is that you have superior accuracy and generalization across different systems (e.g. prior dynamics). I believe you are missing stability, an important property of numerical simulations. You don't have to tackle this, but please mention it. Ideally, include an analysis of the stability of your learned system over long unrolls, even if brief.

## Detailed Feedback

- Equation 2 is sloppy because it omits time. All numerical simulations of chaotic systems diverge for large enough $t$. So what exactly does $C$ measure?
- "Our main goal is to construct an inverse map, implemented in the form of a neural network [...]"
	- Yes, great. Please provide citations for this general idea. I was first exposed to it in the work of Max Welling and his collaborators who used this idea for Bayesian inference to map observations to latent variables. You may need to dig deeper to find the right sources.
		- https://arxiv.org/pdf/1906.02691.pdf
- Equation 6: What is the utility of this notation if ultimately $\Phi$ is simply a neural network? 
	- How exactly do the prior dynamics come into play here? 
- Equation 7: Are you saying here that you want $\mathcal F(x)$ to match the dynamics? 
	- Why do you want to do this if you already have an implementation of the dynamics? 
	- What if the function of the other terms in this equation? 
	- Are you somehow trying to beat the numerical errors produced by the solver? Or this is where you get a mismatch between prior dynamics and your model (to be minimised during training).
- Equation 8: This is a neural ODE model. Maybe it is isn't exactly the same as the original paper, but it is in that spirit. Please make mention of this.
- Equation 9: Where is $\mathcal L_{dyn-pr}$ defined?
	- This entire paragraph is strange. Please compare and contrast with a simpler classical neural network training.
- Function of the Perturbator:
	- What does perturbation have to do with mapping states to observations? Or is it vice-versa? Is this a classical notion in physics? Is there any previous work / grounding of this idea? If the Perturbator is simply a network, how do you get multiple states out of single input (without adding noise somewhere in the process).
- Numerical Experiments
	- Time-to-solution: what is this? Training time? Unroll time?
	- Replace GENN aG-Conv with the citation, e.g. Fablet et al. 2021.
	- You seem to use a few tricks: stopping condition, pretraining, and perhaps more. Please clarify the improvements gained by each trick using ablation experiments. Knowing what did not work is very useful to the community and provides insight into the decisions that you made for this work.
	- If you did 10 independent tests, please provide std for all values.
	- Undefined terms: assimilation window, random half of the components, a variance of 2.0 is involved (what does this mean??)
	- Please move the basic properties of the dataset in the appendix to the main paper: number of examples, train / test / validation setup, etc.
	- Please move all figures after where they are first mentioned in the text. It is very frustrating to see a figure with undefined features only to discover that it is described in some random section below.
	- Figure 4:
		- Don't use "3 blocks," "4 blocks," etc. if what you are actually testing here is the role (e.g. smoothing / unsmoothing) of flow and perturbation operators.
		- The legend is laughably small.
		- This blue arrow seems to be guiding the reader about how to read the figure. Would be easier to just make a simpler figure which is more legible. 
		- The MSE plot should be separate. 
			- Label the y-axis "mean-squared error"
			- Remove the title which has too many words
			- Add legend
	- Figure 5
		- R-score means something else, not reconstruction loss. 
			- https://en.wikipedia.org/wiki/R_score
		- You have already define $\mathcal L_{rec}$ , just use that.
- Appendix
	- Figure 6
		- Is this the dataset? I'm confused. A simple description, e.g. 70 / 20 / 10 % train / validation / test split would suffice.
			- Anyway, as mentioned above, this should go into the main paper.
	- Figure 7
		- What is happening when you switch to fine-tuning? Why this blow up?
	- Figure 9
		- From 6 to 10 blocks there is little change in error. What is happening here? Shouldn't you be overfitting as model complexity grows?
	- Does not belong in this part of my review but: What do you mean by self-supervised loss? Please clarify this since it probably does not match up with the classical ML usage of this term.
strength_and_weaknesses: See above
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: See above
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A network architecture and the corresponding training strategy for solving data assimilation problems are proposed.
strength_and_weaknesses: ### Strengths

- Method is simple.
- Improvement is clearly shown, compared to at least one baseline.

### Weaknesses

1. The motivation behind the design choice of the proposed method is not overly clear. In the caption of Figure 3, the authors explain that the proposed learning procedure is necessary for making the models perform as they expect. Though I agree with it per se, I do not think it explains the fundamental motivation. In particular, why do you want to iteratively apply the $\mathcal{F}$ and $\mathcal{P}$? How is the finetune-based procedure beneficial compared to learning all the networks at once?

2. The experiments lack comparison to an obvious baseline, that is, a network comprising UNet blocks trained altogether. This lack of comparison makes it difficult to assess the particular benefit of the proposed learning procedure.

3. Solving state estimation problems just like data augmentation using neural nets has been an active research area; e.g.:

- Krishnan+, Structured inference networks for nonlinear state space models, AAAI 2017
- Karl+, Deep variational Bayes filters: Unsupervised learning of state space models from raw data, ICLR 2017
- Marino+, A general method for amortizing variational filtering, NeurIPS 2018

among many others. However, the current paper does not address the potential relation to these kinds of studies. Maybe the authors' interest does not lie in Bayesian inference, but references from the outside of the data augmentation community would be highly valuable to make the paper more complete.

---

### Minor points

- Naming a network an "observation operator" without cautionary statements may be confusing especially for machine learning researchers. A natural interpretation is that such an operator takes the hidden state $x$ as input and gives the observation $y$ as output. But, what could be read from the paper is the opposite, that is, it takes $y$ as input and spits out an estimation of $x$.
- In the "Modified U-Net Structure" part, how you do the dimensionality reduction is unclear. 
clarity,_quality,_novelty_and_reproducibility: Basically the writing is not bad, while some points are unclear as I commented above.

The quality of the paper looks to meet a minimum standard but could be improved much by a more intensive empirical study.

The specific learning procedure of the proposed method might be somewhat novel, if not overly impressive, but the current empirical materials provided in the paper are not sufficient to assess its utility. In particular, it lacks ablation studies to analyze the specific benefit of the proposed learning procedure.

I cannot really assess the reproducibility. Although the codes are not provided yet, it does not look difficult to implement all the things described in the paper from scratch.
summary_of_the_review: While the direction of the paper looks nice, the current empirical study does not tell much about the specific benefit of the proposed learning procedure, which makes the paper incomplete.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
The paper proposes a deep learning framework for data assimilation technique for physics and earth science, such as meteorological dynamical processes. The framework is based on an estimation of a coarse state of the system from noisy and sparse observations, associated to a refinement using dynamical priors. Do do so, different U-net based architectures are developed: an observation operator, a flow operator, and a perturbator. Numerical experiments were performed on Lorenz-63 and Lorenz-96 one-dimentional dynamical systems, having 3 state variables. Both models were also tested with biased versions of the prior dynamics, which reproduces more accurately realistic situations.
strength_and_weaknesses: Strenghs:
	- the problem tackled is of importance for a large community
	- the proposed method is end-to-end and able to provide a solution from both observation and prior dynamics 
	- the results are promising, reducing considerably the loss dynamics and the computational time from previous study
	- the pipeline is efficient and the different U-net blocks are assembled in an intelligent way
	- the prior dynamic bias version seems to indeed work, which is one of the main novelties and assets.
	
Weaknesses:
	- The technique seems to be in its early stages and it seems as if tuning needs to be performed for every new dynamical system and setting. First, the different training phases are difficult to follow and makes me wondering if the pipeline is robust enough. Also, please explain how was chosen to use a 4-layer U-Net for the Lorenz-63 and only a 3-layer U-Net for the Lorenz-96. 
	- Network structure: it is not clear which losses are minimized at the different phases, in particular, the perturbator+flow operator is not clear. From eq. 11, it looks as if the dynamics and the data fidelity losses were applied on the same current state x_hat, while my understanding was that L_rec was calculated on the perturbator's output while L_dyn was calculated on the flow operator's output. If the hybrid loss is calculated on the final output of the 2 blocks, I don't se how we can enforce the decoupling of the two goals as stated.
	- It is difficult to understand the size of the input data and latent data. In particular, please give axis labels on Figure 1 images, as it first looks like a 2D spatial problem, while later it is explained that the input is of size time and location. Please also clarify what does T represent in Figure 2: I guess the time dimension, and in this case where is the location dimension?
	- Please explain best the following sentence: 'For the case in which the prior dynamics are unbiased': in a real setting, how can we know if the prior dynamics are biased or not?
	- 'espilon(t) represents the white Gaussian noise.' --> isn't it reductive? 
	- Why stopping at 8 and 10 blocks for the experiments, while it looks as if the results are always improving with more blocks?

Typos:
Specifically, The perturbator

The perturbator uses the observations and labels it has learned to perturb the reconstructed states to make it deviate from the original flow --> please rephrase

Figure 3: the color do not match Figure 1, as here the colors are also linked to training/no training. Maybe use another sign to indicate training/no training, as a red line surrounding the box.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and on the most part easy to read. The novelty seems to be interesting, but a clearer explanation on the differences with Fablet et al. 2021 would be important to see in the contributions section. I did not see a code link, did I miss it?

summary_of_the_review: This paper is interesting and proposes a new end-to-end deep learning framework for a difficult yet important application. In particular, the authors tackled the prior dynamics bias problem. The method seems to be in its early stages, but I think it would be beneficial for the community. I have to add that while I have some knowledge in the field, I am not a data assimilation expert.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a machine learning method for the problem of data assimilation with the objective of reducing the computational cost compared to classical approaches like 4D-Var. Data assimilation is the reconstruction of the sequence of the hidden states of a dynamical system given a series of observations that provide incomplete information of the states via an observation operator.   It is widely used e.g. in numerical weather prediction and for many other problems. The authors start from a variational formulation of the problem involving two loss terms: a reconstruction term measures how well the states are estimated and a dynamical term measures how close are the empirical and the theoretical prior dynamics. They consider a supervised setting were the system state at some space-time points is available. This is a different assumption than the classical 4D-Var algorithm that opens the way to a different and simpler optimization strategy. The proposed method relies on three components, each implemented through a neural network that respectively inverse the observation operator, learn the dynamics and the state reconstruction.  Training is performed by first learning the inverse observation operator for computing a first guess of the state vector and then progressively training the elements responsible for the state reconstruction and dynamics that will refine this first guess. Experiments and comparisons with a baseline are performed on two Lorenz systems.

The authors tackle an important problem since assimilation is a key component for the incorporation of observations in the modeling of earth systems. This work follows a recent trend exploring the use of data-based methods for improving or complementing classical methods. The proposed method performs well compared to the baseline used in the paper. Thanks to the supervised setting, it allows the model to directly optimize the reconstruction term and to perform assimilation on a whole assimilation window at once, thus providing a speedup w.r.t traditional techniques. The authors answered with extensive responses to the reviewers’ questions and added new experiments. However, I have two reservations about accepting this article. One concerns the positioning w.r.t. the classical setting of assimilation. The supervised assumption used here greatly simplifies the assimilation problem and makes the proposed solution relatively straightforward. This also implies that the solution cannot be compared with traditional approaches which do not assume the availability of such a supervision. The second one concerns the technical description that despite clear improvements compared with the first version of the paper remains too vague with several assumptions, concepts and technical details not clearly specified. There are even some notation problems that remain. Note also that the presentation to an ML audience would benefit from a short but precise introduction to variational data assimilation. In its present form the paper does not allow to fully appreciate the significance of the contribution and will not benefit as it probably could to the audience.

justification_for_why_not_higher_score: Many inaccuracies and missing information in the document.
Lack of positioning in relation to conventional methods.
Decision: Reject