Embed and Emulate: Learning to estimate parameters of dynamical systems with uncertainty quantification
Authors: Ruoxi Jiang
Rebecca Willett
Venue: NeurIPS 2022 Accept (2022)
Keywords: Learned emulators
contrastive learning
parameter estimation
inverse problems
climate forecasting

Abstract: This paper explores learning emulators for parameter estimation with uncertainty estimation of high-dimensional dynamical systems. We assume access to a computationally complex simulator that inputs a candidate parameter and outputs a corresponding multi-channel time series. Our task is to accurately estimate a range of likely values of the underlying parameters. Standard iterative approaches necessitate running the simulator many times, which is computationally prohibitive. This paper describes a novel framework for learning feature embeddings of observed dynamics jointly with an emulator that can replace high-cost simulators. Leveraging a contrastive learning approach, our method exploits intrinsic data properties within and across parameter and trajectory domains. On a coupled 396-dimensional multiscale Lorenz 96 system, our method significantly outperforms a typical parameter estimation method based on predefined metrics and a classical numerical simulator, and with only 1.19% of the baseline's computation time. Ablation studies highlight the potential of explicitly designing learned emulators for parameter estimation by leveraging contrastive learning.

Embed and Emulate: Learning to estimate parameters of dynamical systems with uncertainty quantification
Ruoxi Jiang
Rebecca Willett
Department of Computer Science University of Chicago Chicago, IL 60637
Department of Statistics and Computer Science
University of Chicago Chicago, IL 60637 wil willett@uuciicago. edu
roxie628uchicago-.
Abstract
This paper explores learning emulators for parameter estimation with uncertainty estimation of high-dimensional dynamical systems. We assume access to a com- putationally complex simulator that inputs a candidate parameter and outputs a corresponding multichannel time series. Our task is to accurately estimate range of likely values of the underlying parameters. Standard iterative approaches neces- sitate running the simulator many times, which is computationally prohibitive. This paper describes a novel framework for learning feature embeddings of observed dynamics jointly with an emulator that replace high-cost simulators for param- eter estimation. Leveraging a contrastive learning approach, our method exploits intrinsic data properties within and across parameter and trajectory domains. On a coupled 396-dimensional multiscale Lorenz 96 system, our method significantly outperforms typical parameter estimation method based on predefined metrics and a classical numerical simulator, and with only 1.19% of the baseline's 's compu- tation time, Ablation studies highlight the potential of explicitly designing learned emulators for parameter estimation by leveraging contrastive learning.
Introduction
Physics-basee simulators play a vital role in many domains of science and engineering, from energy infrastructure to atmospheric sciences. They frequently critical for assessing risk and exploring "wwaa if" scenarios, which require running models many times [Snyder et al., 2019, Beusch et al., 2020, Helgeson et al., 2021, Zhao al., 2021]. However, the increasing complexity of operational computer models makes running such simulators a major challenge. Emulators (also known as surrogate models) are models trained mimic numerical simulations a much lower computational cost, particularly for parameters or inputs that have not been simulated.
This paper considers the problem of estimating the parameters of physies simulation that provide the best fit to data. The estimation problem can generally be thought of as a nonlinear inverse problem in which
R from noisy multichannel time series Z R = H(Φ: o Zo) + 17. where H(:: Zo) represents running is a physics simulator with parameters &p and initial condition Zo for T time steps, and n A/ T) is observation noise. To ease the notation, we drop the initial condition Zo in H(;; Zo). ). We are particularly interested in complex simulators for which do not have analytic expressions for H, evaluating H (() ) is computational bottleneck and we cannot readily compute its gradients.
goal is estimate parameters
One important application of this problem arises in climate science, where climate scientists have spent decades developing sophisticated physics-base models corresponding to H, often implemented using large-scale software systems that solve complex systems of differential equations [Kay et al., 2015]. In this case, evaluating Н can be computationally demanding. Furthermore, tools like inffrre- neural networks [Raissi et al., 2019] are inapplicable because cannot compute
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

losses that depend upon knowing the form of H. Estimating the parameters of such models based on observational data essential for accurate climate forecasting, and we typically seek not only point estimates of parameters, but also quantifiable uncertainty measures that allow us to forecast the full range of possible future outcomes. Parameter uncertainty quantification is vital here [Cleary et al., 2021, Souza et al., 2020, Hansen, 2022], especially when dealing with noisy observations where small error in Φ might lead to dramatically different forecasts of the dynamics; uncertainty quantification also plays a vital role in the decision- decision-making making process to prevent hazardous loss under tail events [Hansen, 2022].
We explore an alternative framework in which we use our simulator to generate training samples of the form (#;: Z;) for i = 1,...,P, n. and train a machine learning model for parameter estimation. Our approach, which we call EMBED & EMULATE, jointly trains two neural networks. The first, fo. maps an observation Z to low-dimensional embedding. The second, 9o. emulates the map ge : fa o H. and so maps a parameter vector & to the same low-dimensional embedding space. The learned ßo may be used in place of H within an optimization-base framework such as Ensemble Kalman Inversion (EnKI, $2).
More specifically, our method uses the learned low-dimensional embedding to specify an objective function well-suited to parameter estimation without expert knowledge and the learned emulator as surrogate for classical numerical methods used to compute H. Furthermore, inspired by empirical Bayes methods, our learned network suggests a natural mechanism for specifying the EnKl prior based on the trained network.
Related work
Brief parameter estimation background: A widely-used approach for parameter estimation is En- semble Kalman Inversion (EnKI; 1A.1) [Iglesias et al., 2013]. EnKI is derivative-free optimization framework allows a user to specify a prior distribution over the parameter vector Φ and results in samples from the corresponding posterior distribution. As a result, this method provides users with important information about the uncertainty of estimates of Φb. The clear and paper of Schneider et al. [2017] highlighted the use of the EnKI in earth system modeling.
Despite its success in the perfect-model setting (i.c., where (Φ)) can be computed exactly and perfectly represents the dynamics in Z for some Φ), using such methods in practice presents several challenges. To use the EnKI for parameter estimation, we must specify three key elements: (a) the objective function that EnKI is attempting mapping H (or a surrogate for H); and (c) a prior pş. For instance, Schneider et al. [2017] uses Gaussian prior Pa. computes H using Runge-Kute Kutta methods [Dormand and Prince. 1980], and attempts to minimize the Mahalanobis distance
minimize; (b) the tool used to compute the forward
Jmoment (Ø) = [m((Z)) m m(H()))) ((m())))) (
where mZ.) is vector of first and second moments of different spatial channels of Z. and Σ, is diagonal matrix with the j-th diagonal entry Σij. [ar[m[H (o))) Using a loss based on moments within the EnKI framework provides critical robustness to the chaotic nature of Z,
This framework presents two central challenges. First, choosing the objective for EnKI to minimize requires non-trivial domain knowledge, and a poor choice may lead to biased parameter estimates or unpredictable sensitivities to certain features in Z. Second, high-resolution simulations used to evaluate H for given & (e.g. Runge-Kun Kutta or other classical numerical solvers) can be computation- ally demanding. In particular, the EnKI method uses a collection of particles to represent samples of the posterior, and the forward model H must be calculated for each particle at each iteration. For complex optimization landscapes or in high-dimensional settings, the number of particles must be large. This is further complicated by the fact that often the parameter estimation task is conducted repeatedly, e.g., each time new climate data is acquired.
Simulation-base inference (SBI): A large body of work in SBI focuses on parameter estimation for physics- physics-based simulators [Beaumont et al., 2009, Papamakarios et al., 2019, Cranmer et al., 2020, Lueckmann et al., 2019, Chen et al., 2020b, Alsing et al., 2019]. Advanced SBI methods focus on adaptive generation of new training data to approximate the posterior and could be classified into different categories based on how they adaptively choose informative simulations [Lueckmann et al., 2021]. One approach, likelihood estimation, proposes to approximate an intractable likelihood function [Drovandi et al., 2018]. For instance, Sequential Neural Likelihood estimation (SNL., Papamakarios et al. [2019]) trains conditional neural density estimator (c.g., Masked Autoregressive
2

---

Flow (MAF)) which models the conditional distribution of data given parameters [Papamakarios et al., 2017I. While SNL does not readily scale to high-dimensional data, a recent variant called SNL+ [Chen et al., 2020b] addresses this limitation by learning sufficient statistics (embeddings) of the data based on the infomax principle, and it iteratively updates the network used to compute sufficient statistics and the neural density estimators during sequential sampling. An alternative approach, (sequential) neural posterior estimation ((S)NP)), aims to approximate directly the target posterior (SNPE-A in Papamakarios and Murray [2016], SNPE-B in Lueckmann et al. [2017]. SNPE-C in Greenberg et al. (2019]), where SNPE-C forces fewer restrictions on the form of prior and posterior by leveraging neural conditional density estimation.
However, in the context of this manuscript's is setting, i.e., estimating multiple Φ; for multiple different observations Z, at test time, the key technique of SBI, sequential sampling. can require substantial computational investments. This idea discussed further with our experimental results.
Learned emulators:
Physics models and simulations are pervasive in weather and climate science, astrophysics, high-energy and accelerator physics, and the study of dynamical systems. These models and simulations are used, for example, to infer the underlying physical processes and equations that govern our observations, design new sensors or facilities, estimate errors, understand experi- mental behavior, and estimate unknown parameters within models. Many such physics simulators require expensive computational resources, and are difficult to fully leverage. "L "Learned emulators," " approximate models that use
"surrogate models," or "approximants" are computationallly-ffficoi numerically simulated data to train machine learning system to mimic these numerical simulations at much smaller computational cost. There have been many recent successes in the development of surrogate model foundations [Brockherde et al., 2017, Raissi al., 2019, Chattopadhyay et al., 2019, 2020, Raissi et al., 2017, Vlachas et al., 2018, 2020, Brajard et al., 2020, Gagne et al., 2020] and applications [Goel et al., 2008, Mengistu and Ghaly, 2008, Brigham and Aquino, 2007, Kim et al., 2015, Papadopoulos et al., 2018, White et al., 2019, Gentine et al., 2018, Rasp et al., 2018, Cohen et al., 2020, Yuval and O' O'Gorman, 2020, Xue et al., 2021, Wang et al., 2019].
Typically, one would directly try to emulate H, and the efficacy of the learned emulator would be evaluated on how well (()) matches (Φ) (often using squared error) across a range of Φ. This formulation, however, may be suboptimal when the emulator will be used for parameter estimation. First, as we will detail later, emulating H when Z is very high-dimensional is quite challenging. (In our experiments, dim (z) = 396, 000.) Furthermore, in many climate settings, H/ represents a chaotic process in which very small changes to the initial conditions can result in very large differences later in the process. In this setting, training an emulator with a loss akin to (1/n) -== IZ. H())) | may be overly sensitive to noise and initial conditions and not preserve statistical features of Z, that may be essential to parameter estimation.
Inspired by the objective in (1), Cleary et al. [2021] trained a model فe to emulate the moments of parameters using the loss function Emoment (0) := (1/n) (1/ [l
- (o,)))1 E2 to ease the burden of rumning H. However, Emoment (0) is not always the best choice. First, estimating E, for each training sample an enormous computational burden that far exceeds the cost of generating the original training data (unless the range of candidate Φ is very small). Second, choosing the moment function m (essentially fixing a particular low-dimensional embedding) is critical design element that requires domain expertise and often must be tuned in practice. Finally, even if one is willing to generate accurate estimates of Σg, one could face additional challenges. Cleary et al. [2021] is designed for inferring parameters given one fixed test observation, and they train their emulator for a small domain of parameter space dependent on the test observation at hand. Extending this framework to a larger domain of parameters & results in large variability among the collection of corresponding E,s generated for training, making the loss landscape challenging to navigate.
With the above challenges in mind, we present novel and generic framework for parameter estimation by jointly optimizing (a) an embedding of the multichannel dynamics used to evaluate the accuracy of candidate parameter & and (b) an emulator of the dynamics projected into the embedding space. We leverage a contrastive framework to learn discriminative representations for high-dimensional spatio-temporal data.
Contrastive learning:
Contrastive representation learning has exploded in popularity recently for self-supervised visual feature learning that has achieved comparable performance with its supervised counterparts [Caron et al., 2020, Chen al., 2020a, He et al., 2020, Grill et al., 2020, Zhang and Maire, 2020, Caron et al., 2021]. The majority of these frameworks operate under the push-pull principle for instance-wise
3

---

discrimination: images generated from different forms of data augmentation (e.g., cropping, color jittering) are pulled together while other images are pushed away. Apart from its common setup in unsupervised representation learning, Khosla et al. [2020] also demonstrated selecting positíve and negative pairs in a supervised way leveraging label information can boost the performance. Many efforts focus on understanding the mechanism of contrastive learning [Oord et al., 2018, Wei et al., 2020, Arora et al., 2019, Wang and Isola, 2020, Zimmermann et al., 2021]. Among them, Zimmermann et al. [2021] show that contrastive loss can be interpreted as the cross-entropy between the latent conditional distribution and ground truth distribution.
3 Contributions
In work, follow the contrastive framework and learn embedding network capture structural information for high-dimensional spatio-temporal data and an emulator that can be used to compute parameter estimates with uncertainty estimates. This is effective for several reasons. First, when the model H is bijective, representations learned using contrastive losses are highly correlated with their underlying parameters [Zimmermann al., 2021]. Second, most existing emulating methods focus on approximating dynamical models for a fixed parameter Φ and lack generalization capacity IKrishnapriyan et al., 2021]. In part this is due to the difficulties of emulating dynamics with high spatio- temporal resolution. We cireumvent this bottleneck by training an emulator to output the learned latent representations of the dynamics, which are much lower-dimensionl and easier to learn with fewer training samples. Finally, our method is inspired by the Contrastive Language- Image Pre- training (CIIP) [Radford et al., 2021] framework, which designed for contrastive learning of aligned representations of images and language. Our approach, by analogy, learns aligned representations of dynamics and parameters, allowing us to jointly learn the embedding function and emulator for high-resolution data.
More specifically, this paper makes the following key contributions:
Incorporation of a contrastive loss function for learning an embedding of the simulation outputs (Z;) instead of relying a known "good" moment function m, leveraging ideas from CLIP [Radford et al., 2021];
Development of an emulator designed to facilitate parameter estimation with uncertainty quan- tification. We demonstrate that the proposed emulator can be used effectively within an EnKI framework to generate parameter estimates with accurate posterior distribution estimates.
An empirical evaluation of different methods (EnKI [Schneider et al., 2017], supervised learning. NPE-C Greenberg et al. [2019], SNL+ [Chen et al., 2020b], and our proposed enulator-based approach) in terms of robustness to noise errors in training and testing data for range of sample sizes as well as computational costs. Compared to using numerical solvers, we show that our method achieves higher accuracy overall in 1.08% of the computation time, even accounting for the computational costs of generating training data.
Evaluation of the quality of the uncertainty estimates using the Continuous Ranked Probability Score (CRPS) for varying numbers of training samples.
Our empirical results highlight both the improved computational and empirical accuracy of our proposed emulator-centric approach relative to competing methods and, more broadly, the benefits of designing emulators specifically for parameter estimation tasks.
4 Proposed method
Our goal is to learn a mapping from observations of multichannel dynamical system, Z E RRxxd to the underlying system parameters & E TR* where Z H ((Φ) + 17 for some noise vector 7) R'xd We assume we do not have access to an analytical expression for H, but can compute H (Ø) for any & using a computationally complex simulator. We further assume prior P& over parameter space.
Using this prior and the simulator, we generate n training samples as follows: for i = 1,..., n, draw & ~ Ps. and use the simulator to compute Z; ((),) + 77 We do not explicitly generate noise 7; rather, the numerical algorithm used to simulate H will generally produce some errors, with faster implementations of H being more error-prone. The distribution p& coupled with the noise distribution over induces the joint distribution Ps.z-

---

4.1 Baseline approach supervised learning
Given training samples (Ø. Z,) io = ,....,,,,, n. we train a neural network represented by he using a ResNet [He et al., 2016] as the backbone (see details in $B.2). 0 denotes the learned network parameters and our prediction is Φ; ho (Z,). We train using the Mean Absolute Percentage Error (MAPE) loss
n1 Pij (ha (Z.))
EAAEE (0) = n2
10 +
As we will see in the experimental results, this approach yiclds point estimates of Φ on holdout test data with reasonable accuracy and offers significant computational advantage over the EnKI method. However, it does not offer uncertainty estimates, and its test accuracy is lower than that of proposed approach ($4.2).
4.2 Our approach: joint embedding and emulation via contrastive feature representation
Contrastive feature representation: Measuring the distance between pair of dynamics, a necessary task for constructing a loss function used for training. is particularly challenging when the dynamics chaotic. As mentioned above ($2), one alternative in the literature is to use custom loss function based on summary statistics of Z: this requires expert knowledge to determine which statistics are relevant, and has demonstrated efficacy only for narrow ranges of parameters Φ
We present an alternative framework in which we learn an embedding of Z, denoted [o(),, that preserves statistical and structural characteristics of Z most relevant to the parameter estimation task and hence can be used to train an emulator. Here f denotes all the parameters of the neural network defining the embedding function. Inspired by recent advances in contrastive representation learning, from a collection of parameters and trajectories pairs { , Z. J==1 we propose trajectory encoder fe which learns a distinguishable representation by minimizing the following variant of the Info Noise Contrastive Estimation (InfoNCE) loss [Oord et al., 2018]:
exp ((fe fe
Ezz r) == Σ log
(3)
Σ (M)(U)()),,,)((())
Here (Z) E SP-1 is normalized and lives in a unit hypersphere. 2, is selected through data augmentation (see $B.1) 1) or in a supervised way by measuring the distance in the parameter space: 77, zz, 8(z,, Z.), where is metric used in calculating distance in parameter space,
Z; : arg min and T is a temperature hyperparameter balancing the impact of similar pairs and negative examples [Zhang et al., 2018, Ravula et al., 2021]. In this way, we drive the model to embed data with similar values of the parameter & in similar locations while repulsing data with unrelated parameters, resulting in an embedding that respects the latent structure of the parameter and trajectory pairs.
Emulating the dynamics of a simulator by training a deep neural network ga (-) is natural approach easing the computational burden of parameter estimation. However, recent works in emulating dynamics focus on the fixed-parameter setting [Krishnapriyan et al., 2021] and cannot be easily generalized to multi murtt-p settings (i.e. the emulator only approximates H (Φ)) for single fixed & instead of range of Φs, as we need for parameter estimation).
Emulators:
Learning emulators for range of parameter values is challenging in part due to the dimension of the dynamics to be emulated (i.e. the dimension of the Zs). To address this problem, we propose to learn an emulator ğo of go fo H. which represents the mapping from the parameter space (Ø) the latent representation space fo (Z). which is much lower-dimensional than Z and hence easier to learn with limited training samples.
We say that the trajectory encoder and the emulator are perfectly aligned when Go(Φ) = [o(Z), A (Ø, Z) Po.z. Under perfect alignment, given a test Z, we may seek to minimize
J(: Z, 9o) = (()+ (e (z)
instead of (1) with much faster objective function evaluations.
Pretraining an embedding So and then learning an emulator go with an Lo loss is hard to optimize; the different architectures, necessitated by the very different dimensions of &f and Z, make feature alignment challenging and the optimization prone to poor local
Learning the emulator via CLIP:
5

---

(0:1)
Encocerr
kabhl Rada
Embedding network fa
7
K.66
Z
Tragectory
KA6
Regression
hgg
MLP
x))
p
(Mapp(())
2.
6
7.
2.2. 2, 22
2.4.
Emulator
(2)(0;r)
6
Residnal
3.2,
3.2,
7.3
2,
7, 2,
2.6
2,
Cap(0:
(2((0,)
(22)
124
(2)00:))
(a) Network setup
(b) Contrastive losses
Figure I: The EMBED EMULATE framework. (a) Components of our network and loss function. (b) We use three contrastive learning schemes. The bottom left block denotes the intra-trajecotry domain contrastive mechanism, the upper right denotes the intra-parameter domain contrastive learning, and the bottom right block shows inter- crrajectory-parameee
domains contrastive learning. Within each block, diagonals correspond the dot product between the representations of positive trajectories pairs (Z,; Z:), positive parameter pairs (Ø:, Φz). and matched trajectory parameter pairs (Z,, 4,). which we aim to maximize. Off Off-diagonal pale pink blocks show similarities between the anchor point and the negatives examples, which we aim to minimize.
minimizers. Alternatively, inspired by the recent success of CLIP [Radford et al., 2021, Ravula et al., 2021] in cross-domain representation learning, we use the following variant of InfoNCE loss to align the representation space between two networks:
exp )x)
exp
(2)(0;7)) =
log
log
72
Σ
exp (Lf (0
Σ exp (Z,) (Zoo
"
aatra-parameter domain contrastive loss: In order to fully exploit data potential within parameter space and better guide the learning of the representation and preserve similarities within the parameter space, we add an intra-parameter contrastive loss and find empirically that it helps accelerate the training process:
exp (0 9o (x.)))
( (0; =
log
6
Σ exp ((00 (0 (0(
where Ø is selected through data augmentation by perturbing & with small amount of noise (see $B.1) or by finding nearest neighbors the training set.
Regression head: Zimmermann et al. [2021] shows that under certain assumptions (see $A.2), the feature encoder fo implicitly learns invert the data generating process H up transformation, implying we can estimate di via ((fe)Z)) for some linear function h. Inspired by their analysis, we add a regression head he (a single fully-connected linear layer) in our framework. and we ensure the affine relationship between he and fe by letting them share the same backbone up the last layers before the output.
affine
Unified training procedure: To this end, we formulate the final loss of our EMBED & EMULATE method
E(O) = Azzlzz (0; r) (0; + Azalzo + AMAPE(MAPE (0), where Azz, >dpep Azo and AMAPE control relative loss weights. We show our basie setup in Fig. 1(a) and loss framework in Fig. 1(b).

---

5 Experiments
We conduct a numerical case study on the multiscale Lorenz-96 (L96) model [Lorenz, 1996]. which common test model for climate models with both "fast" (igh-freuuenc) and "slow" (low- frequency) components and other geophysical applications [Majda and Harlim, 2012, Law and Stuart, 2012, Law et al., 2016, Brajard et al., 2020]. is a prototypical turbulent dynamical system "designed to mimic baroclinic turbulence in the midlatitude atmosphere" [Majda et al., 2010]. Its dynamics exhibit strong energy-conserving non-linearities, and for some settings of o, it can exhibit strong chaotic turbulence. Code is available at: https: //tt:////ttbb.. com/roxie62/Enlee Embed- and-mmuate
The governing equations of the L96 system are defined as follows:
d dt
x
(X
X x
X F -ccY
d YY
j.k
-Y. + Y!- Y
h x J
c dt
where X, E RK denotes the slow variable at the t-th time stamp and Y, E KJ denotes the fast variable. We use Z: == [X., Y.l E (J++)) to denote the system state at the t-th time stamp. We choose K = 36 and J = 10 throughout experiments in this section, as in Schneider et al. [2017].
We set our baseline following Schneider et al. [2017]. Within the EnKT framework, they use Runge-Kutta Dormand and Prince [1980] to compute the forward model H and optimize (1). After analyzing the physics information of the L96 system, they define the moment function as: m(Z) := (X)x, (Y)T, (X")T, (XY)T, (Y²)r), where (+)T denotes an empirical average over ' time steps, Y denotes an average across the J Y channels, (Y)r (1/JT) E E/= Y.S RK and other quantities are computed similarly (see details in $B.1). m m(Z)) E RK For the SBI methods, to suit the scenario we care about, i.e., estimating multiple &; for multiple different observations Z, at test time, we compare with the non-adaptive counterpart of the algorithms referenced in these papers. For instance, set the round number of SNL+ [Chen et al., 2020b] be one. And use the fixed training dataset for SNL+ and NPE-C [Greenberg et al., 2019].
We conduct experiments and demonstrate the efficacy when measuring (1) the accuracy of averaged point estimates, (2) the accuracy of uncertainty quantification, (3) the empirical computational complexity. For all of our experiments (including SNL+ and NPE-C), we use ResNet-34 [He et al., 2016] as the backbone of the trajectory encoder. We apply average pooling at the last layer to generate 512-d hidden vector. For contrastive learning, we project the 512-d hidden vector into a 128-d feature vector as the output of the trajectory encoder. For the regression head, we map the 512-d hidden vector to a 4-d parameter vector. It is important to note that activation functions are not used in the regression head to ensure an affine mapping between two projections. We parameterize the emulator network with ResNet backbone, and to enhance the representation power of the network, we use five residual connection blocks (see $B.2). We find this increases the alignment between the two representation spaces. Within each skip block, there is a residual connection between the input layer and the output. The implementation details for training are shown in B
5.1 Higher quality estimates with lower sample cost
In this section, we evaluate our method with different training sizes in the perfect-mode model setting where no noise is injected in observations. These experiments demonstrate the usefulness of our method in realistic scenarios where forward models are prohibitively expensive and only limited quantities of training data are available.
We train our EMBED & EMULATE method and the supervised regression baselines when the training size equals 500 and 1, 000. The training samples are sampled uniformly from Φ Lmnin
[Fmin: Mmin: Cmin: bmin] = [-5,0, 0.1, 0) to Φmax = [Fmax, Amaax Cmaxx buax] = [20, 5, 25, 25), and each simulation is of length 100, with dt = 0.1. We then sample 200 testing samples uniformly from a narrower range of -3, 0.5, 2, to [18, 4.5, 23, 23). As assumed in Schneider et al. [2017]. each testing sample is of length 1, 000, with dt- 0.1. The EnKI prior used for baselines (i.c. minimizing (1)) is a Gaussian distribution with means at the middle of the range of the testing instances, diagonal covariance matrix, and variances broad enough that all test samples denoted fixf (see $B.1). Within the context of the EMBED & EMULATE approach, we adopt an empirical Bayes approach: for each test instance, we compute & using the regression head and use
within 2x of the mean,
7

---

these values as the prior means when using EnKI to minimize (4); we denote this prior Po.empB. The impact of the empirical Bayes approach explored $5.3.
F
h
c.4
b4
0
EnKT w/ Learning
15.48 (3.77)
0.86 (0.20)
40.45 (13.39)
4.60 (0.60)
ho w/ EMBED & EMULATE
11.19 (3.65)
3.18 (1.60)
15.52 (6.24)
8.57 (2.17)
EnKI w/ EMBED & EMULATE
10.94 (4.07)
3.74 (2.26)
16.09 (6.41)
8.84 (2.93)
500
Supervised Regression
11.97 (3.10)
4.07 (2.24)
17.04 (5.88)
9.07 (2.74)
NPE-C SNL+
15.51 (4.52)
5.94 (2.59)
23.54 (8.16)
9.42 (3.70)
36.88(19.93)
48.71 (30.69)
57.45 (28.59)
23.45 (17.62)
he w/ EMBED & EMULATE
6.30 (1.86)
2.07 (1.31)
9.34 (3.71)
5,51 (1.74)
EnKT w/ EMBED EMULATE
6.59 (2.14)
2.36 (1.54)
9.38 (4.02)
5.54 (2.35)
1000 Supervised Regression
7.57 (1.78)
3.08 (1.54)
11,46 (3.29)
6.64 (2.19)
NPE-C SNL+
11.29 (3.54)
5.62 (2.29)
16.32 (6.53)
6.81 (2.38)
33.29(19.68)
49.05 (27.75)
48.17 (23.90)
25,40 (16.03)
Table 1:
Averaged MAPE (MdAPE, median absolute percentage error) for varying training size of different methods for 200 test samples. We compare EMBED & EMULATE w/ w//egression head (ha) and EMBED & EMULATE plugged into EnKI to supervised regression, NPE-C [Greenberg et al., 2019], SNL+ [Chen et al., 2020b), and a classical numerical solver (Runge-Kuun Kutta) plugged into EnKI minimize (1). The example illustrates that EMBED & EMULATE is able to achieve lower error than both the EnK1 approach based on a fixed moment vector objective and classical numerical solver [Schneider al., 2017] and straightforward supervised regression approach that is unable to produce uncertainty estimates.
Training data
Training
200
test
Total time (train + test)
generation
time
runs
EnKI w/o Learning
0.0
0.0
8,000.0
8,000 (5.5 d)
he w/ EMBED & EMULATE
21.0
72.0
1.0
94.0 (1.57 h)
EnKI w/ EMBED & EMULATE
21.0
72.0
2.0
95.0 (1.58 h)
Supervised Regression
21.0
59.0
1.0
81.0 (1.35 h)
NPE-C
21.0
72.0
2.0
95.0 (1.58 h)
SNL+
21.0
73.0
400.0
494.0 (8.23 h)
Table 2: Empirical computational time for different stages of different methods (n = 1, 000). Reported in minutes, total time for EMBED & EMULATE, the supervised regression, or NPE-C are 1.19% of EnKI with Runge-Kutte Kutta. All neural approaches are trained with 4 GPUs and tested with 1 GPU. Both training data generation and EnKI w/o Learning are run with 32 CPU Cores.
We first evaluate the accuracy of point estimates using the mean and median absolute percentage error (MAPE and MdAPE), see $B.1. In Table and Table 2, we see that EMBED & EMULATE evaluated with the regression head (h₂) and EnKI guarantees similar performance in terms of averaged accuracy. However, the regression head (h₂) is slightly better, which may be explained by our empirical observation that when estimates from he with EMBED EMULATE are far from the true parameter values, then using this estimate as the prior mean for EnKI can worsen the estimate. Despite this challenge, the EnKI has the advantage of providing uncertainty estimates, which are discussed below and reflected in Table 3.
Compared with other methods, is clear that EMBED & EMULATE yields a significant improvement in accuracy, especially for the parameter affecting
dynamics (e.g.. c). In particular, NPE-C [Greenberg et al., 2019] performs worse with a smaller training set size SNL+ [Chen et al., 2020b] using MCMC or rejection sampling suffers from increasing evaluation time as the number of test samples increases. Moreover, when compared to the classical method of running EnKl using a predefined moment function and expensive numerical solvers, EMBED & EMULATE yields better performance terms of accuracy and computation time. Last, our EMBED & EMULATE framework also performs well relative to supervised regression for smaller training sets.
8

---

We then evaluate our results based on the continuous ranked probability score (CRPS) [Hersbach, 2000. Zamo and Naveau, 2018, Pappenberger et al., 2015]. CRPS is an important metric used in quantifying uncertainty (e.g., in weather forecasts). It measures the accuracy of estimated posterior distributions and is defined as
CRPS(C, [-- (C(w)) - UVú +.,)))
where Φ. represents the j-th component of the estimate parameter vector, Φ., represents the th component of the true parameter vector, C' is the cumulative density function of the ensemble estimates with Cly) P(o.; S y), and U is the Heaviside step function. For deterministic estimate from supervised regression, CRPS is equal to the mean absolute error (MAE). Table 3 shows that EMBED & EMULATE achieves high-accuracy estimates of uncertainty.
F +
h
c
b 4
0
EnKI w/ Learning
0.910
0.019
2.443
0.393
hg w/ EMBED & EMULATE
0.698
0.076
1.715
0.824
EnKI w/ EMBED & EMULATE
0.615
0.073
1.561
0.720
500
Supervised Regression
0.707
0.104
1.785
0.917
NPE-C
0.844
0.106
2.117
0.853
SNL+
1.399
0.616
2.426
1.822
hg w/ EMBED & EMULATE
0.412
0,049
0.992
0.453
EnKI w/ EMBED EMULATE
0.360
0.042
0.829
0.394
1000
Supervised Regression NPE-C SNL+
0.478
0.078
1.242
0.650
0.593
0.096
1.389
0.564
1.304
0.595
2.010
1.763
Table 3: Continuous Ranked Probability Score (CRPS) evaluated on 200 test samples. The errors of the uncertainty estimates almost always lower for EMBED & EMULATE than for EnKI method using a classical numerical solver. And compared to neural methods, EnKI with EMBED & EMULATE yields significant improvements over supervised regression which cannot quantify uncertainty, defeats the simulation-base inference models NPE-C [Greenberg et al., 2019] and SNL+ [Chen al., 2020b] which relies on sequential sampling. Moreover, it's clear that EMBED & EMULATE evaluated with EnKI provides more accurate uncertainty estimates than woint-wise estimates from the regression head (he) trained with EMBED & EMULATE.
5.2 Visualizing uncertainty with noisy observations
In this subsection, we go beyond the perfect-model setting and evaluate our method in the realistic scenarios where observations are noisy and obtained through: Z H(Φ) + where NoO,, rT), where I is the temporal covariance of the trajectory Z, and is scaling value. Specifically, we set & [10, 1, 10, 10] following Schneider et al. [2017] to ensure we are in the chaotic regime. We use model trained with n = 4, 000 data samples and compare the results obtained in both the noiseless and noisy cases. For this visualization, both methods use the fixed prior from $5.1 based on the range of test values, so any differences observed are not caused by differences in the prior, but rather differences in the choice of objective and the method of computing the forward model.
As shown Fig. 2, EMBED & EMULATE is more robust noise than the baseline method with numerical solvers and predefined moments. Reconstructed posterior distributions learned with EMBED & EMULATE in Fig. 2 are more consistent with increasing noise levels, especially for the parameters F and b.
5.3 Ablation study: role of the regression head
In this section, we empirically verify the utility of the regression head ha in our EMBED & EMULATE framework. We use n 4000 training samples in two settings: first, we use the full EMBED & EMULATE model of $5.1; second, we discard the regression head (i.e. fix ha = 0) while keeping intra- and inter-domain contrastive losses. For both trained models, we run the EnKI. We run EnKl using components from the EMBED & EMULATE framework for two choices of prior: first, we use Poxfixed, and second, we use Po.empB- Note the empirical Bayes prior only possible with the regression head. These priors are detailed in $5.1 and I.

---

200
200
Moments and
20
3--
23
0.
10-
0.75
1.5
0.50
1.0
3.0
0.0
1.0
23
7.5
12 7/5
12.5
Figure 2: Impact of observation noise. Reconstructed posterior distributions, comparing a classical numerical solver (Runge-Kutta) Kutta) plugged into EnKI minimize (1) (left) with EMBED & EMULATE (right), Both variants of EnKI are run for 70 iterations with 100 particles. The green line shows the noise-free case, the blue shows the noisy case when r 0.1, and the red shows the noisy case when r = 0.2. We see here that our proposed EMBED & EMULATE method (right column) produces posterior estimates that are consistent over range of noise levels, while the baseline EnKI approach using pre-defined embedding corresponding to moments vector is much more sensitive to variations r.
Table 4 shows that having the regression component of the loss complement the contrastive losses yields a substantial improvement in parameter estimation accuracy using EMBED & EMULATE. In other words, explicitly training the embedding function and emulator for the parameter estimation task yields an emulator that is far more accurate during parameter estimation than training generic embedding and generic emulator. Furthermore, this table illustrates the efficacy of our empirical Bayes procedure- i.e., using our learned regressor to alter the prior used by EnKI. The medians are slightly improved while the means are strongly improved, suggesting that the empirical Bayes procedure is particularly helpful in the tails.
F +
h +
c
b +
(a) No regression head (Pobiiicod) fixed)
16.75 (2.62)
6.02 (1.63)
22.16 (6.35)
6.60 (1.86)
(b) EMBED & EMULATE (poefixed)
8.39 (1.12)
1.77 (0.82)
15.82 (1.61)
3.85 (0.91)
(c) EMBED & EMULATE (po.empa)
3.39 (1.03)
1.21 (0.76)
4.53 (1.52)
3.02 (0.90)
Table 4: Ablation study of regression head: Average MAPE (MdAPE, median absolute percentage error) over 200 test instances for estimating & by minimizing (4) in three different settings: (a) fe and do correspond to a generic emulator trained without the regression loss kung and the original prior Po.fxed; (b) fo and îo correspond to the emulator learned with our EMBED & EMULLATE framework and the original prior Polfxxed; and (c) So and go correspond to the emulator learned with our EMBED EMULATE framework and the empirical Bayes prior Po.empB- This experiment
shows that having the regression component of the loss complement the contrastive losses yields substantial improvement in parameter estimation accuracy using EMBED & EMULATE.
Conclusions
The proposed EMBED & EMULATE framework trains an emulator of a complex simulation to facilitate parameter estimation. Unlike generic emulation methods, which can lead to poor parameter estimates and require expert domain knowledge to construct, our method (a) leverages a contrastive learning framework coupled with a regression head to jointly learn a oiw-dineniional embedding of simulator outputs that can he used to construct an objective function for parameter estimation and (b) yields an emulator that can be used within an optimization framework such as the EnKI to produce accurate parameter estimates in 1.19% of the computation time of an approach using classical numerical methods, even accounting for the time required to generate training samples. We explore our approach in the context of earth system modeling as described by Schneider et al. [2017] and Cleary et al. [2021] and hypothesize that these tools can facilitate improved climate forecasts that account for uncertainties and cover the full range of possible outcomes. The social impacts of improved climate forecasting are positive if acted upon. While learned emulators are gaining in popularity, we as a community still know little about which systems are more or less challenging to emulate or how to design task-specific emulators more generally. There are further opportunities for exploring emulators for parameter estimation using optimization frameworks beside the EnKI.
10

---

Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers and area chair for their helpful comments. We thank Owen Melia and Xiao Zhang for helpful discussion and proofreading our paper. RJ and RW are partially supported by DOE MEE-AC2-00CH1122 13575, DE-SC0022232, AFOSR FA9550- 18- 1-0-0166, NSF OAC-1934637, NSF and NSF DGF-2022023.

---

References
Justin Alsing, Tom Charnock, Stephen Feeney, and Benjamin Wandelt. Fast likelihood- freee cosmology with neural density estimators and active learning. Monthly Notices of the Royal Astronomical Sociery, 488(3):44404445 4458, 2019.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
Mark A Beaumont, Jean-Marie Cornuet, Jean-Michel Marin, and Christian P Robert. Adaptive approximate bayesian computation. Biometrika, 96(4):983 990. 2009.
Lea Beusch, Lukas Gudmundsson, and Sonia Seneviratne. Emulating earth system model tempera- tures with mesmer: from global mean temperature trajectories to grid-point-lvel realizations on land. Earth System Dynamics, 11(1):139-15)) 2020.
Julien Brajard, Alberto Carrassi, Mare Bocquet, and Laurent Bertino. Combining data assimilation and machine learning to emulate dynamical model from sparse and noisy observations: A case study with the lorenz, 96 model. Journal of Computational Science, 44:101171, 2020.
John C Brigham and Wilkins Aquino. Surrogate-moue model accelerated random search algorithm for global optimization with applications to inverse material identification. Computer Methods in Applied Mechanics and Engineering, 196(45-48):4561 -4776, 2007.
Felix Brockherde, Leslie Vogt, Li Li, Mark E Tuckerman, Kieron Burke, and Klaus-Robert Müller. Bypassing the kohn-sham equations with machine learning. Nature communications, 8(1):1-10, 2017.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9992 9924,, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Fmerging properties in self-supervised vision transformers. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
Neil K Chada, Yuming Chen, and Daniel Sanz-Alonso. Iterative ensemble kalman methods: A unified perspective with some new variants. arXiv preprint arXiv:2010.. 13299, 2020.
Ashesh Chattopadhyay, Pedram Hassanzadeh, Krishna Palem, and Devika Subramanian. Data-driven prediction of a multi-scale lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ann, and rnn-lstm. arXiv preprint arXiv: 1eXiv::900...m 190..0829, 2019.
Ashesh Chattopadhyay, Pedram Hassanzadeh, and Devika Subramanian. Data-driven predictions of a multiscale lorenz 96 chaotic system using machine-learning methods: reservoir computing. artificial neural network, and long short-term memory network. Nonlinear Processes in Geophysics, 27(3):73-389, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020a.
Yanzhi Chen, Dinghuai Zhang, Michael Gutmann, Aaron Courville, and Zhanxing Zhu. Neural approximate sufficient statistics for implicit models. arXiv preprint arXiv:2010. 10079, 2020b.
Emmet Cleary, Alfredo Garbuno-Inigo, Shiwei Lan, Tapio Schneider, and Andrew M Stuart. Calibrate, emulate, sample. Journal of Computational Physics, 424:109716 2021.
Yair Cohen, Ignacio Lopez-Gomez, Anna Jaruga, Jia He, Colleen M Kaul, and Tapio Schneider. Unified entrainment and detrainment closures for extended eddy-diffusivity mass-flux schemes. Journal of Advances in Modeling Earth Systems, 1200)22020MS002212,, 162, 2020.
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation- -bsed inference. Proceedings of the National Academy of Sciences, 1 117(49):30555-3066 2020.
12

---

John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. computational and applied mathematics, 6(1):19-26, 1980.
Journal of
Christopher C Drovandi, Clara Grazian, Kerrie Mengersen, and Christian Robert. Approximating the
likelihood in abc. Handbook of approximate bayesian computation, pages 321-368, 2018.
David John Gagne, Hannah M Christensen, Aneesh C Subramanian, and Adam H Monahan. Machine learning for stochastic parameterization: Generative adversarial networks in the lorenz 96 model. Journal of Advances in Modeling Earth Systems, (2(3)=e201 9MS001 (000)0001M000011 196, 2020.
Pierre Gentine, Mike Pritchard, Stephan Rasp, Gael Reinaudi, and Galen Yacalis, Could machine learning break the convection parameterization deadlock? Geophysical Research Letters, 45(11);
1742-5751, 2018.
Tushar Goel, Siddharth Thakur, Raphael T Haftka, Wei Shyy, and Jinhui Zhao. Surrogate model- based strategy for cryogenic cavitation model validation and sensitivity evaluation. International journal for numerical methods in fluids, 58(9):969- 2008
David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood-ffree inference. In International Conference on Machine Learning, pages 2404-2414. PMLR, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to seff-super learning. Advances in Neural Information Processing Systems, 13:21271-211288 2020.
Lars Peter Hansen. Central banking challenges posed by uncertain climate change and natural disasters. Journal of Monetary Economics, 125:1-15, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
Casey Helgeson, Vivek Srikrishnan, Klaus Keller, and Nancy Tuana. Why simpler computer simulation models can be epistemically better for informing decisions. Philosophy of Science, 88 (2):213-233, 2021.
Hans Hersbach. Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather and Forecasting, 15(5):539-570, 2000.
James M Hyman and Basil Nicolaenko. The kuramoto-sivashinnky equation: a bridge between pde's and dynamical systems. Physica D: Nonlinear Phenomena, 18(113): 1986.
Marco A Iglesias, Kody J H Law, and Andrew M Stuart. Ensemble kalman methods for inverse problems. Inverse Problems, 29(4):045001, mar 2013. doi: 10. (18800266-56111/9944455555 URL https:///oi.
Jennifer E Kay, Clara Deser, A Phillips, A Mai, Cecile Hannay, Gary Strand, Julie Michelle Arblaster, SC Bates, Gokhan Danabasoglu, James Edwards, et al. The community earth system model (cesm) large ensemble project: A community resource for studying climate change in the presence of internal climate variability. Bulletin of the American Meteorological Society, 963)):1333-1349 2015.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33: 33:18661-18673 2020.
Seung- Goo Kim, Jeffrey A Melby, Norberto C. Nadal- -Crraballo,, and Jay Ratcliff. A time-dependent surrogate model for storm surge prediction based on an artificial neural network using high-fidelity synthetic hurricane modeling. Natural Hazards, 76(1):565-588 2015.
13

---

Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes physics-informed neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Kody JH Law and Andrew M Stuart. Evaluating data assimilation algorithms. Monthly Weather Review, (44((1):3757--333 2012.
Kody JH Law, Daniel Sanz-Alonso Abhishek Shukla, and Andrew M Stuart. Filter accuracy for the Lorenz 96 model: Fixed versus adaptive observation operators. Physica D: Nonlinear Phenomena, 325:1-13, 2016.
Edward N Lorenz. Predictability: A problem partly solved. In Proc. Seminar on Predictability 1996.
Jan-Mathhis Lueckmann, Pedro J Goncalves, Giacomo Bassetto, Kaan Öcal, Marcel Nonnenmacher, and Jakob H Macke. Flexible statistical inference for mechanistic models of neural dynamics. Advances in neural information processing systems, 30, 2017.
Jan-Matthis Lueckmann, Giacomo Bassetto, Theofanis Karaletsos, and Jakob H Macke. Likelihood- free inference with emulator networks. In Symposium on Advances in Approximate Bayesian Inference, pages 32-53. PMLR, 2019.
Jan-Matthis Lueckmann, Jan Boelts, David Greenberg, Pedro Goncalves, and Jakob Macke. Bench- marking simulation-based inference. In International Conference on Artificial Intelligence and Statistics, pages 443-351. PMLR, 2021.
Andrew Majda and John Harlim. Filtering Complex Turbulent Systems. Cambridge University Press, 2012.
Andrew J Majda, John Harlim, and Boris Gershgorin. Mathematical strategies for filtering turbulent dynamical systems. Discrete & Continuous Dynamical Systems, 27(2):441, 2010.
Temesgen Mengistu and Wahid Ghaly. Acrodynamic optimization of turbomachinery blades using evolutionary methods and ann-based surrogate models. Optimization and Engineering, 9()): 239-255, 2008
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint /807.0/80..0333 2018
Benjamin Pachev, Jared P Whitchead, and Shane A McQuarrie. Concurrent multi- parameter learning demonstrated on the kuramoto-sivashinsky equation. arXiv preprint arXiv:2106. XXV:22006.0666, 2021.
Papadopoulos, G Soimiris, DG Giovanis, and M Papadrakakis. A neural network- surro- gate model for carbon nanotubes with geometric nonlinearities. Computer Methods in Applied Mechanics and Engineering, 328:411-493 2018
George Papamakarios and lain Murray. Fast e-free inference of simulation models with bayesian conditional density estimation. Advances in neural information processing systems, 29, 2016.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
George Papamakarios, David Sterratt, and lain Murray. Sequential neural likelihood: Fast likelihood- free inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statisties, pages 837-848. PMLR, 2019.
Florian Pappenberger, Maria- Ramos, Hannah L Cloke, F Wetterhall, L Alfieri, Konrad Bogner, Anna Mueller, and Peter Salamon. How do know my forecasts are better? using benchmarks in hydrological ensemble prediction. Journal of Hydrology, 522:697-713, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763 PMLR, 2021.
14

---

Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part I): Data-driven solutions of nonlinear partial differential equations. arXiv preprint 171.v.1711.10561. 2017.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Stephan Rasp, Michael S Pritchard, and Pierre Gentine. Deep learning to represent subgrid processes in climate models. Proceedings of the National Academy of Sciences, 115(39))9884-9666) 2018.
Sriram Ravula, Georgios Smymiss, Matt Jordan, and Alexandros G Dimakis. Inverse problems leveraging pre-trained contrastive representations. Advances in Neural Information Processing Systems, 34, 2021.
Tapio Schneider, Shiwei Lan, Andrew Stuart, and Joao Teixeira. Earth system modeling 2.0: A blueprint for models that learn from observations and targeted high-resolution simulations. Geophysical Research Letters, 44(24):12-366, 2017.
Abigail Snyder, Robert Link, Kalyn Dorheim, Ben Kravitz, Ben Bond-Lamberty, and Corinne preciuuttppriiiittatiinnn realizations with internal
Hartin. Joint emulation of earth system model variability and space-time and cross- cross-variable variable correlation: fidgen v2. 0 software description. Plos one, 14(10):c0233542, 2019
Andre Nogueira Souza, GL Wagner, Ali Ramadhan, B Allen, V Churavy, James Schloss, J Campin, Chris Hill, Alan Edelman, John Marshall, et al. Uncertainty quantification of ocean parameter- for penetrative convection. Journal of Advances in Modeling Earth Systems, 12(1 (212))22020MSS00206 2020.
izations: Application to the
Pantelis R Vlachas, Wonmin Byeon, Zhong Y Wan, Themistoklis P Sapsis, and Petros Koumout- sakos. Data-driven forecasting of high- dighniinnnsooal chaotic systems with long short-term memory networks. Proceedingy of the Royal Society A: Mathematical, Physical and Engineering Sciences, (74(2213):01170444 2018.
Pantelis Vlachas, Jaideep Pathak, Brian R Hunt, Themistoklis P Sapsis, Michelle Girvan, Edward Ott, and Petros Koumoutsakos. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Networks, 126: 191-217, 2020.
Jiali Wang, Prasanna Balaprakash, and Rao Kotamarthi. Fast domain neural network emulation of planetary boundary layer parameterization in numerical weather forecast model. Geoscientific Model Development, 12(10), 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. International Conference on Machine Learning, pages PMLR, 2020.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self self-training with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622. 2020.
Daniel A White, William Arrighi, Jun Kudo, and Seth E Watts. Multiscale topology optimization using neural network surrogate models. Computer Methods in Applied Mechanics and Engineering, 346: 1135, 2019.
Jayye Xue, Zhongming Xiang, and Ge Ou Predicting single freestanding transmission tower time history response during complex wind input through convolutional neural network based surrogate model. Engineering Structures, 233:111859) 2021.
Janni Yuval and Paul O'Gorman. Stable machine-leariing parameterization of subgrid processes for climate modeling at range of resolutions. Nature communications, 11(1):1-10, 2020.
Michaël Zamo and Philippe Naveau. Estimation of the continuous ranked probability score with limited information and applications to ensemble weather forecasts. Mathematical Geosciences, 50(2):209-25 234, 2018.
15

---

Xiao Zhang and Michael Maire. Self Self-supervised visual representation learning from hierarchical grouping Advances in Neural Information Processing Systems, 13:16579-16590, 2020.
Xu Zhang, Felix Xinnan Yu, Svebor Karaman, Wei Zhang, and Shih-Fu Chang. Heated-up softmax embedding, arXiv preprint arX1v:1889.04157, 2018.
Lei Zhao, Keith Oleson, Elie Bou-Zeid, E. Scott Krayenhoff, Andrew Bray, Qing Zhu, Zhonghua Zheng, Chen Chen, and Michael Oppenheimer. Global multi-model projections of local urban climates. Nature Climate Change, 11(2):12-117,,2) 202..
Roland Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990. PMLR, 2021.
16


Reviews:
[{"review_id": "iIubL3baYIA", "reviewer": "Reviewer_25cv", "summary": "The paper deals with the problem of Bayesian parameter estimation for high dimensional systems, where generating a large number of particles for ensemble Kalman inversion (EnKI) may be prohibitively expensive. The authors approach this by jointly learning a lower dimensional latent representation of the physical space as well as an emulator on this low dimensional latent space so that EnKI can be applied without significant computational cost (called _embed & emulate_). Ideas from contrasive learning are used to design the loss function for training, so that the trajectories that are close in the physical space gets pulled together in the latent space. The method is benchmarked on the multiscale Lorenz-96 model, where the _embed & emulate_ approach is compared against traditional EnKI and supervised regression in terms of runtime and several performance metrics.", "questions": "- Currently, the algorithm seems to only deal with state-space models where the observation operator is the identity map. However, in practice, more often than not the observations are not the identity map (e.g. observes only some components of $Z$, or maybe it is a nonlinear function of the state $Z$). Is it possible to handle more general observation operators to make it more useful in practice?\n- How robust is this method to model mismatch? i.e., when the algorithm is trained on a physical model that differs from the underlying truth (e.g., when the algorithm is trained on a single-scale L96 model with unknown parameters $\\phi = \\\\{h, c, F\\\\}$, but the underlying truth is the multiscale L96).\n- Why are the same values for $\\tau$ used in the losses $\\ell_{ZZ}$ and $\\ell_{\\phi \\phi}$ but a different value $\\tau'$ is used for $\\ell_{Z\\phi}$?\n- When computing $\\hat{g}_\\theta(\\phi)$, do we not need to specify the initial condition $Z_0$? Since the trajectory $Z = H(\\phi)$ depends on it.\n- Not sure where the moments $m(Z)$ are used in the algorithm, even though it is mentioned.\n- Can this inversion method be used in tandem with online filtering/smoothing (e.g. 4DVar, EnKF)?", "limitations": "- As with most deep learning applications, the approach is limited to the parameter distribution used in the training data; if the true parameter lie outside of this distribution, the method will likely struggle. Thus, one needs to cover a wide range of parameters, which may require a very large number of training data. This may be challenging for very high dimensional systems used in NWP.\n\n- If the method cannot deal with general observation operators or not robust to model mismatch (see \"Questions\" above), this may be quite limiting for practical usage.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper presents an interesting and novel approach to learning useful low dimensional representations of high dimensional models. It is a nice idea to use this emulator for parameter estimation.\n- The experiments demonstrate that the method captures good parameter estimates and uncertainties, outperforming EnKI with significantly less compute time. Moreover the approach is robust to noise.\n- Experimental procedure/setup is clear and detailed.\n- The paper is overall well-written\n\nWeaknesses:\n- Labels in the figures 2a and 3 can be larger. There is also an error in the caption of Figure 3:\n\n> ''Embed & Emulate method _(top row)_ produces posterior estimates...''\n\nThere is no 'top row'.\n\n- There are some passages in the text where more explanation is desired. More details below in \"Questions\".\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dxTXa-bo6Dz", "reviewer": "Reviewer_QCnJ", "summary": "This study proposes a method that aims at efficiently solving the inverse problem of stochastic simulators. The new methodology uses neural networks to learn both an embedding of the simulator output and an emulator of the simulator -- i.e., a mapping from the parameters to the lower dimensional projection of the simulator output. The study provides empirical results on the multiscale Lorenz-96 (L96) model and shows that the new method leads to parameters that are closer to the ground-truth parameters, and with lower computational cost, compared to previous approaches.\n", "questions": "This study will substantially benefit from a thorough discussion and empirical comparison with previous methods for likelihood-free inference.", "limitations": "The authors have adequately addressed the limitations of their work.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\n\nThe problem of efficiently and reliably inferring the parameters of stochastic simulators given empirical data is an important problem across fields of science. And, as far as I can tell, the solution provided by the study is technically novel. However, I have a few concerns:\n\n-there is a large body of work relevant to this study that is currently not referenced, discussed and empirically compared. Indeed, methods that go under the name of Approximate Bayesian Computation, likelihood-free inference, or/and simulation-based inference would be very relevant to discuss and empirically compare. All these methods are precisely concerned with performing Bayesian inference of stochastic simulators (for a review, see https://www.pnas.org/doi/10.1073/pnas.1912789117). In particular, and most directly relevant to this study, several of these methods use neural density estimators and aim at estimating the likelihood function (i.e., an emulator - e.g., Papamakarios et al. 2019 http://proceedings.mlr.press/v89/papamakarios19a.html; Lueckmann et al. 2019 https://proceedings.mlr.press/v96/lueckmann19a.html), extracting informative summary statistics from the data for posterior inference (e.g. https://openreview.net/forum?id=SRDuJssQud), or directly estimating the posterior distribution (https://proceedings.neurips.cc/paper/2016/hash/6aca97005c68f1206823815f66102863-Abstract.html, https://proceedings.mlr.press/v97/greenberg19a.html). I believe it would be crucial to discuss and empirical compare the approach developed to at least a few of those previous approaches. I should note that there is code available to perform such comparisons (e.g. https://sbi-benchmark.github.io/);\n\n-related to the point above, I believe the baseline approach provided, which the authors use to validate their approach, could have been better chosen. In particular, regarding the baseline approach, the authors write \"However, it does not offer uncertainty estimates\". A potentially stronger alternative would have been Neural Posterior Estimation (NPE), for instance with normalising flows (https://proceedings.mlr.press/v97/greenberg19a.html). NPE would likely require more simulations, but would provide a full posterior distribution and have the added advantage of allowing to perform amortised inference after network training.\n\n\nTherefore, as it stands, the contribution of this study is not sufficiently put in the context of past work on this topic, and I would strongly recommend the authors to compare their method with some of these past methods.\n\n\n### Quality\n\nThe paper is technically sound, and claims are backed by empirical evidence. \n\n\n### Clarity\n\nThe manuscript is for the most part clearly written, providing enough information to understand the technical contribution and empirical results. A few small comments and typos:\n\n-page 1, line 28, \"$\\eta$ is observation noise\". Unless the expression in line 27 is incorrectly written, $\\eta$ would be more appropriately described as \"dynamics noise\" rather than \"observation noise\", since it impacts the dynamics update equation;\n\n-page 3, lines 113-114, \"Cleary et al. [2021] is designed for inferring...\" This formulation requires a small rewriting for clarity;\n\n-page 3, line 139, \"when the model H is bijective\". As it is written, it suggests that many models are bijective. However, this is rather an exception, so I would suggest the authors to rephrase this;\n\n-page 4, line 174, \"the numerical algorithm used to simulate H will generally produce some errors\". It would be beneficial if the authors provided a quantification (an appendix figure?) of such errors;\n\n-page 5, line 197, \"variant of InfoNCE\". For clarity, it would be good to introduce the abbreviation;\n\n-page 6, caption Figure 1, \"intra-trajecotry\";\n\n-page 6, caption Figure 1, a few typos in \"diagonals are corresponds to dot product\".\n\n\n### Significance\n\nThe technical development and empirical results will be of interest to the ML community. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-ej0qfSMxhk", "reviewer": "Reviewer_fsU3", "summary": "The paper proposes a parameter estimation approach for (non-linear) dynamical systems with multidimensional outputs. The key novelty involves jointly learning a low-dimensional embedding of the observations (Z) and an emulator mapping from parameters to a representation in the latent space while ensuring that the two representations are aligned and using contrastive learning. The learned representations can now be used in a classical parameter estimation setup such as EnKI. The method is evaluated on the L96 system, and the results are compared to a standard parameters estimation technique (EnKI with numerical solvers). The proposed method compares favorably in terms of both MAPE and CRPS to the chosen baselines on the L96 problem (and the additional problem in the appendix). Further experiments are included to demonstrate the uncertainty quantification properties and importance of specific elements of the method. ", "questions": "Minor questions/comments/suggestions:\n\n- Choice of EnKI: Early on the paper makes a choice to use EnKI, and while this appears sensible there are several other derivative free optimization methods. I’d suggest strengthening the justification for the EnKI even further and perhaps clarify (/speculate) how sensitive the proposed E&E approach is to using EnKI. \n \n- Resnet-32 structure: I would suggest clarifying why a convolutional resnet is a suitable backbone for the particular problem.\n  \n- l197: For completeness, please provide the reference for InfoNCE and explain the modifications. \n \n- l199: $p$ appears to be undefined (?)\n  \n- l201: Please clarify which metric is actually used (i.e. $\\delta$)? Also, how sensitive is the method to the choice of $\\delta$ ?\n \n- l211 “…due to the dimensions of the dynamics to be emulated (i.e. dimensions of Zs)”. I appreciate that the dimension of Z is challenging but does it make sense to talk about it as the “dimensions of the dynamics”…?\n \n- Optimization problem (l237-238): could the authors comment on the difficult of optimizing this objective (in terms of robustness, sensitivity to hyperparameters of the optimization algorithms etc.) and perhaps describe/show some optimization traces.\n \n- l260: “latent” seems more appropriate to me than “hidden” to me.\n \n- Eq 3: \\langle \\rangle are used for an inner product (I presume); later used as empirical average l253; perhaps clarify c.f. eq. 3 to avoid confusion.\n \n- Discussion: The current method requires quite a lot of training data which is expensive to generate; do the authors see the current method being used in a sequential experimental design (aka active learning) setup?\n", "limitations": "I think the paper could do a slightly better job explaining the complexity and generalizability of the approach (see comments elsewhere).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nGeneral comments/suggestion/questions:\n\n- A mostly clear and well-written paper on a timely problem. The resulting method is rather complex with many parts, but I think the paper does a relatively good job of guiding the reader through the various elements.\n \n- A seemingly clear limitation is the inclusions of only one empirical study, namely based on the L96 Lorentz system; however, the experiment appears well-executed and well-documented, so I do not see any mainly issues with only a single case study given the nature of the problem (assuming the below question is addressed). I note the additional experiment in the appendix also favors the proposed method over the considered alternatives, and the authors may want to refer to the additional experiment in the main paper. \n \n- Complexity, practicality, and generality is my main concern with the proposed method. The rationale used to motivate the need for the E&E method focuses (initially) on the need to make choices about moments in the cost functions; however, the proposed method is very complex by any standard. Consequently, I would argue that there are even more design choices involved in the E&E method ranging from the backbone, dimensionality of latent embedding, metrics in the latent space, hyperparameters, etc. Could the authors clarify how general the specific method (really) is and to which degree it needs to be tailored to each specific problem (domain)? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept