Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The overarching goal of this paper is to improve assimilation of (chaotic) dynamical systems. The authors introduced a self-supervised framework, amortized assimilation, which uses methods from Ensemble filters and supervised denoising. 
 In Section 3, they develop their own method, first focusing on amortized ensemble filters and then deriving their self-supervised assimilation method. In Section 4, their method is compared with standard data assimilation methods on chaotic benchmarks (such as the Lorenz 96). 
main_review: 1) I cannot really comment on the Originality of this paper. The methods they use seem common in the dynamical system literature (which they do not review), but maybe there is a crucial difference when moving to assimilation I overlook. Thus I don’t feel confident enough to make a statement.

2) I found the paper to be hard to understand. Some points are definitely lack of knowledge on my part, but other parts could probably be better explained

- The state that the problem they want to solve is a high-quality estimate of the initial state (see abstract). They actually never explain how they estimate the initial state with their model, why their model should be better suited to estimate the initial value nor do they compare a measure of quality for the initial state in their empirical section. 

- In the abstract they claim that they use "independent neural networks to assimilate specific observation types”(l. 8f), but later they claim that they didn’t do that, but instead handled them as follows: 
(Line 226 )“Coordinates which are not observed can be masked and an additional channel is appending indicating whether a particular coordinate was observed during the given assimilation cycle. This is depicted by the shading in the observation channels of Figure 3. More details on the masking process can be found in the supplementary material" 

Either I misunderstand the authors, or their abstract and later explanations do not match …

- They claim:” 
(Line 251)  All methods tested have explicit knowledge of the system dynamics. Due to the challenge of learning chaotic dynamics, we found that methods which rely on learning the dynamics were uncompetitive”
I am not sure that is true ... 

- Their core section about self-supervised assimilation could be a lot clearer. At one point the authors write "letting us use the framework from [16] to analyze the problem” without further explaining the framework … 
This makes it very arduous for the reader to understand what they do 

- Figure 6 should have a legend or a more elaborate caption. The plot is not understandable without reading the text and even in the text the colors of the lines plotted is not explained …

3) I should point out some comments that need to be addressed (please see below):

- First, there are some grammatical mistakes as well as missing words and typos …

- Figure labels especially in Figure 7 should be a lot larger. I had to zoom a lot to see the figures

- They should explain what they mean by “independent neural networks to assimilate specific observation types” in the abstract and how that compares to line 226ff (see above)

- If possible they should further explain, why they focus so much on the error in the estimate of the initial state of the system (abstract & introduction) as opposed to other model error and why they do not compare the quality of initial state estimation in their experimental section, if that estimation is so essential 

- Lastly, in their subsection Training objective, the authors should consider more training-runs. They find an “intriguing” result but because they only consider one training-run (why???) it’s entirely unclear whether this is to chance or an significant effect. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents, amortized assimilation, way to perform data assimilation through deep learning utilizing self-supervised training to obtain an unbiased estimate of the true state. 
main_review: The abstract of the paper is overwhelmed with technical jargons. While the need for technical jargons maybe be essential at most points in a paper, I would expect the authors to try and write a more easy-to-understand abstract so that the work may appeal to both the deep learning and the DA community.

"Line 44" what is oft-repeated in numerical simulations of Earth system ? I do not think that there are any oft-repeated processes, else the authors should cite something that explains this. 

Lines 51-60 are extremely hard to read. In fact, throughout the paper, the authors might want to keep in mind that their application is useful for both ML and DA audiences and can rephrase the end of their introduction to read a little more easily to a vaster range of audience, say from geoscience, DA, dynamical systems, and physics along with ML. 

"Lines 102-106" At small ensemble sizes DA has been successfully used for even the most complex NWPs through localization and inflation.  Generating large ensembles has also been successfully performed with deep learning methods as well. The statements in between these lines are not enough to showcase a need for the authors' method. A clear motivation is either  absent in this introduction or might get hidden owing to the overly technical writing . 

"Lines 112-113" Sequential filters do not respect conservation laws ? When making claims like these, the authors should cite papers or show evidence. 

"that we view sequential filtering as a particular recurrent prediction model which can be unrolled into a multi-step computation graph allowing us the train using backpropogation through time" -- What is this computation graph? While the authors present a very well thought out method, with enough novelty, the writing makes it very hard to follow. This is not helping the authors make their case for the importance or relevance of this method

The results are interesting but does not show the advantage of this method over other traditional DA methods. What happens with small ensemble sizes and localization and inflation? In the absence of fair comparisons , the results are not complete.

References: No journal names in the references

[39] Julien Brajard, Alberto Carassi, Marc Bocquet, and Laurent Bertino. Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the lorenz 96 model, 2020. -- incomplete reference, where's the journal name? https://gmd.copernicus.org/preprints/gmd-2019-136/

 [42] Ashesh Chattopadhyay, Mustafa Mustafa, Pedram Hassanzadeh, Eviatar Bach, and Karthik
 Kashinath. Towards physically consistent data-driven weather forecasting: Integrating data  assimilation with equivariance-preserving deep spatial transformers, 2021. --- same problem. Here's the paper: https://gmd.copernicus.org/preprints/gmd-2021-71/

With an improvement in the writing, and more experiments with comparisons with DA+localization and inflation, this will be a very good contribution.






limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce amortized assimilation or AmEnF, a novel method for inferring dynamics from sampled states, and predicting initial and future states from it.
main_review: The key idea of AmEnF is that we can run the model from several different timepoint and compare the different predictions we get from the model to one another, results in a self-supervised loss. They prove that the self supervised loss is bounded by the standard supervised loss. I think this is a really neat idea, and an interesting new direction for self-supervised learning for dynamical system. 

The authors show that their model performs well on standard data assimilation tasks such as Lorenz 96, Vissio-Lucarini 20 and Kuramoto-Shivashinsk, especially when observed data is sparse and noise is high, which are usually the instances where self-supervised are especially helpful. 

Overall, the paper does a good job at explaining the background, previous works, and their innovation and contributions. 

Minor remarks: All figures are to small and a quite hard to read, especially figure Figure 1.

I recommend acceptance.
limitations_and_societal_impact: I see no negative societal impact from this work.
review_title: null

Meta-review: From the SAC. This is an instance where the rebuttal and the discussion worked. While the original decision for this paper was to not accept, it is being raised to a recommended accept. The primary reason is the quality of the rebuttal, and the useful technical discussion between authors and reviewers that ensued that seems to have been revealing (in particular, reviewer kG8H). To the authors: I trust that you will take all reviewer feedback into account and most importantly, that all of the things in your rebuttal and discussion that were promised will be done in the next version of the paper.
Decision: Accept (Poster)