Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This study proposes a method that aims at efficiently solving the inverse problem of stochastic simulators. The new methodology uses neural networks to learn both an embedding of the simulator output and an emulator of the simulator -- i.e., a mapping from the parameters to the lower dimensional projection of the simulator output. The study provides empirical results on the multiscale Lorenz-96 (L96) model and shows that the new method leads to parameters that are closer to the ground-truth parameters, and with lower computational cost, compared to previous approaches.

strengths_and_weaknesses: ### Originality

The problem of efficiently and reliably inferring the parameters of stochastic simulators given empirical data is an important problem across fields of science. And, as far as I can tell, the solution provided by the study is technically novel. However, I have a few concerns:

-there is a large body of work relevant to this study that is currently not referenced, discussed and empirically compared. Indeed, methods that go under the name of Approximate Bayesian Computation, likelihood-free inference, or/and simulation-based inference would be very relevant to discuss and empirically compare. All these methods are precisely concerned with performing Bayesian inference of stochastic simulators (for a review, see https://www.pnas.org/doi/10.1073/pnas.1912789117). In particular, and most directly relevant to this study, several of these methods use neural density estimators and aim at estimating the likelihood function (i.e., an emulator - e.g., Papamakarios et al. 2019 http://proceedings.mlr.press/v89/papamakarios19a.html; Lueckmann et al. 2019 https://proceedings.mlr.press/v96/lueckmann19a.html), extracting informative summary statistics from the data for posterior inference (e.g. https://openreview.net/forum?id=SRDuJssQud), or directly estimating the posterior distribution (https://proceedings.neurips.cc/paper/2016/hash/6aca97005c68f1206823815f66102863-Abstract.html, https://proceedings.mlr.press/v97/greenberg19a.html). I believe it would be crucial to discuss and empirical compare the approach developed to at least a few of those previous approaches. I should note that there is code available to perform such comparisons (e.g. https://sbi-benchmark.github.io/);

-related to the point above, I believe the baseline approach provided, which the authors use to validate their approach, could have been better chosen. In particular, regarding the baseline approach, the authors write "However, it does not offer uncertainty estimates". A potentially stronger alternative would have been Neural Posterior Estimation (NPE), for instance with normalising flows (https://proceedings.mlr.press/v97/greenberg19a.html). NPE would likely require more simulations, but would provide a full posterior distribution and have the added advantage of allowing to perform amortised inference after network training.


Therefore, as it stands, the contribution of this study is not sufficiently put in the context of past work on this topic, and I would strongly recommend the authors to compare their method with some of these past methods.


### Quality

The paper is technically sound, and claims are backed by empirical evidence. 


### Clarity

The manuscript is for the most part clearly written, providing enough information to understand the technical contribution and empirical results. A few small comments and typos:

-page 1, line 28, "$\eta$ is observation noise". Unless the expression in line 27 is incorrectly written, $\eta$ would be more appropriately described as "dynamics noise" rather than "observation noise", since it impacts the dynamics update equation;

-page 3, lines 113-114, "Cleary et al. [2021] is designed for inferring..." This formulation requires a small rewriting for clarity;

-page 3, line 139, "when the model H is bijective". As it is written, it suggests that many models are bijective. However, this is rather an exception, so I would suggest the authors to rephrase this;

-page 4, line 174, "the numerical algorithm used to simulate H will generally produce some errors". It would be beneficial if the authors provided a quantification (an appendix figure?) of such errors;

-page 5, line 197, "variant of InfoNCE". For clarity, it would be good to introduce the abbreviation;

-page 6, caption Figure 1, "intra-trajecotry";

-page 6, caption Figure 1, a few typos in "diagonals are corresponds to dot product".


### Significance

The technical development and empirical results will be of interest to the ML community. 

questions: This study will substantially benefit from a thorough discussion and empirical comparison with previous methods for likelihood-free inference.
limitations: The authors have adequately addressed the limitations of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a parameter estimation approach for (non-linear) dynamical systems with multidimensional outputs. The key novelty involves jointly learning a low-dimensional embedding of the observations (Z) and an emulator mapping from parameters to a representation in the latent space while ensuring that the two representations are aligned and using contrastive learning. The learned representations can now be used in a classical parameter estimation setup such as EnKI. The method is evaluated on the L96 system, and the results are compared to a standard parameters estimation technique (EnKI with numerical solvers). The proposed method compares favorably in terms of both MAPE and CRPS to the chosen baselines on the L96 problem (and the additional problem in the appendix). Further experiments are included to demonstrate the uncertainty quantification properties and importance of specific elements of the method. 
strengths_and_weaknesses: 
General comments/suggestion/questions:

- A mostly clear and well-written paper on a timely problem. The resulting method is rather complex with many parts, but I think the paper does a relatively good job of guiding the reader through the various elements.
 
- A seemingly clear limitation is the inclusions of only one empirical study, namely based on the L96 Lorentz system; however, the experiment appears well-executed and well-documented, so I do not see any mainly issues with only a single case study given the nature of the problem (assuming the below question is addressed). I note the additional experiment in the appendix also favors the proposed method over the considered alternatives, and the authors may want to refer to the additional experiment in the main paper. 
 
- Complexity, practicality, and generality is my main concern with the proposed method. The rationale used to motivate the need for the E&E method focuses (initially) on the need to make choices about moments in the cost functions; however, the proposed method is very complex by any standard. Consequently, I would argue that there are even more design choices involved in the E&E method ranging from the backbone, dimensionality of latent embedding, metrics in the latent space, hyperparameters, etc. Could the authors clarify how general the specific method (really) is and to which degree it needs to be tailored to each specific problem (domain)? 
questions: Minor questions/comments/suggestions:

- Choice of EnKI: Early on the paper makes a choice to use EnKI, and while this appears sensible there are several other derivative free optimization methods. I’d suggest strengthening the justification for the EnKI even further and perhaps clarify (/speculate) how sensitive the proposed E&E approach is to using EnKI. 
 
- Resnet-32 structure: I would suggest clarifying why a convolutional resnet is a suitable backbone for the particular problem.
  
- l197: For completeness, please provide the reference for InfoNCE and explain the modifications. 
 
- l199: $p$ appears to be undefined (?)
  
- l201: Please clarify which metric is actually used (i.e. $\delta$)? Also, how sensitive is the method to the choice of $\delta$ ?
 
- l211 “…due to the dimensions of the dynamics to be emulated (i.e. dimensions of Zs)”. I appreciate that the dimension of Z is challenging but does it make sense to talk about it as the “dimensions of the dynamics”…?
 
- Optimization problem (l237-238): could the authors comment on the difficult of optimizing this objective (in terms of robustness, sensitivity to hyperparameters of the optimization algorithms etc.) and perhaps describe/show some optimization traces.
 
- l260: “latent” seems more appropriate to me than “hidden” to me.
 
- Eq 3: \langle \rangle are used for an inner product (I presume); later used as empirical average l253; perhaps clarify c.f. eq. 3 to avoid confusion.
 
- Discussion: The current method requires quite a lot of training data which is expensive to generate; do the authors see the current method being used in a sequential experimental design (aka active learning) setup?

limitations: I think the paper could do a slightly better job explaining the complexity and generalizability of the approach (see comments elsewhere).
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper deals with the problem of Bayesian parameter estimation for high dimensional systems, where generating a large number of particles for ensemble Kalman inversion (EnKI) may be prohibitively expensive. The authors approach this by jointly learning a lower dimensional latent representation of the physical space as well as an emulator on this low dimensional latent space so that EnKI can be applied without significant computational cost (called _embed & emulate_). Ideas from contrasive learning are used to design the loss function for training, so that the trajectories that are close in the physical space gets pulled together in the latent space. The method is benchmarked on the multiscale Lorenz-96 model, where the _embed & emulate_ approach is compared against traditional EnKI and supervised regression in terms of runtime and several performance metrics.
strengths_and_weaknesses: Strengths:
- The paper presents an interesting and novel approach to learning useful low dimensional representations of high dimensional models. It is a nice idea to use this emulator for parameter estimation.
- The experiments demonstrate that the method captures good parameter estimates and uncertainties, outperforming EnKI with significantly less compute time. Moreover the approach is robust to noise.
- Experimental procedure/setup is clear and detailed.
- The paper is overall well-written

Weaknesses:
- Labels in the figures 2a and 3 can be larger. There is also an error in the caption of Figure 3:

> ''Embed & Emulate method _(top row)_ produces posterior estimates...''

There is no 'top row'.

- There are some passages in the text where more explanation is desired. More details below in "Questions".


questions: - Currently, the algorithm seems to only deal with state-space models where the observation operator is the identity map. However, in practice, more often than not the observations are not the identity map (e.g. observes only some components of $Z$, or maybe it is a nonlinear function of the state $Z$). Is it possible to handle more general observation operators to make it more useful in practice?
- How robust is this method to model mismatch? i.e., when the algorithm is trained on a physical model that differs from the underlying truth (e.g., when the algorithm is trained on a single-scale L96 model with unknown parameters $\phi = \\{h, c, F\\}$, but the underlying truth is the multiscale L96).
- Why are the same values for $\tau$ used in the losses $\ell_{ZZ}$ and $\ell_{\phi \phi}$ but a different value $\tau'$ is used for $\ell_{Z\phi}$?
- When computing $\hat{g}_\theta(\phi)$, do we not need to specify the initial condition $Z_0$? Since the trajectory $Z = H(\phi)$ depends on it.
- Not sure where the moments $m(Z)$ are used in the algorithm, even though it is mentioned.
- Can this inversion method be used in tandem with online filtering/smoothing (e.g. 4DVar, EnKF)?
limitations: - As with most deep learning applications, the approach is limited to the parameter distribution used in the training data; if the true parameter lie outside of this distribution, the method will likely struggle. Thus, one needs to cover a wide range of parameters, which may require a very large number of training data. This may be challenging for very high dimensional systems used in NWP.

- If the method cannot deal with general observation operators or not robust to model mismatch (see "Questions" above), this may be quite limiting for practical usage.
review_title: null

Meta-review: metareview: All reviewers agree that the paper proposes an interesting approach that aims at efficiently solving the inverse problem of stochastic simulators. Although some reviewers have some technical concerns at their first reviews, basically those have been resolved by the authors' responses. Thus, although there are some points that should be modified from the current form, I think we can expect the authors modify the paper in the camera-ready by reflecting the discussion. Based on these, I recommend acceptance for this paper.
confidence: Certain
Decision: Accept