SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use a quadratic cost function  $c(\mathbf{s},\hat{\mathbf{s}}) = \frac{1}{2}\| \mathbf{s} - \hat{\mathbf{s}}\| ^2$  (i.e.,  $W$  is the 2-Wasserstein distance), and the map  $\pi \in \Pi (\mu_{\mathbf{s}},\mu_{\hat{\mathbf{s}}})$  must be a valid transport map between  $\mu_{\mathbf{s}}$  and  $\mu_{\hat{\mathbf{s}}}$  (i.e., a joint distribution for  $\mathbf{s}$ ,  $\hat{\mathbf{s}}$  with marginals that match  $\mu_{\mathbf{s}}$  and  $\mu_{\hat{\mathbf{s}}}$ ) [38].
In the discrete setting with distributions represented by samples  $\mathbf{S} = \{\mathbf{s}_i\}_{i=1}^L$  and  $\hat{\mathbf{S}} = \{\hat{\mathbf{s}}_j\}_{j=1}^L$ , the Wasserstein distance is given by
$$
\frac {1}{2} W (\mathbf {S}, \hat {\mathbf {S}}) ^ {2} := \min  _ {T \in \Pi} \sum_ {i, j} T _ {i j} C _ {i j}, \tag {11}
$$
where the cost matrix  $C_{ij} = \frac{1}{2}\| \mathbf{s}_i - \hat{\mathbf{s}}_j\|^2$ . The set of valid discrete transport maps  $\Pi$  consists of all matrices  $T$  such that  $\forall i,j, T_{ij} \geq 0, \sum_j T_{ij} = 1$ , and  $\sum_i T_{ij} = 1$ .
Entropy-regularized optimal transport—Sinkhorn divergence. Exactly solving the optimal transport problem associated with computing the Wasserstein distance is computationally prohibitive,
especially in higher dimensions. A common approximation made to speed up computation is to introduce an entropy regularization term, resulting in a convex relaxation of the original optimal transport problem:
$$
\frac {1}{2} W ^ {\gamma} (\mathbf {S}, \hat {\mathbf {S}}) ^ {2} := \min  _ {T \in \Pi} \sum_ {i, j} T _ {i j} C _ {i j} - \gamma h (T), \tag {12}
$$
where  $h(T) = -\sum_{i,j} T_{ij} \log T_{ij}$  is the entropy of the transport map. This entropy-regularized optimal transport problem can be solved efficiently using the Sinkhorn algorithm [31].
As  $\gamma \rightarrow 0$ , the entropy-regularized Wasserstein distance  $W^{\gamma} \to W^{0} = W$  reduces to the exact Wasserstein distance (11). For  $\gamma > 0$ , we can further correct for an entropic bias to obtain the Sinkhorn divergence [39, 40]
$$
\ell_ {\mathrm {O T}} (\mathbf {S}, \hat {\mathbf {S}}) = \frac {1}{2} \overline {{W}} ^ {\gamma} (\mathbf {S}, \hat {\mathbf {S}}) ^ {2} := \frac {1}{2} \left(W ^ {\gamma} (\mathbf {S}, \hat {\mathbf {S}}) ^ {2} - \frac {W ^ {\gamma} (\mathbf {S} , \mathbf {S}) ^ {2} + W ^ {\gamma} (\hat {\mathbf {S}} , \hat {\mathbf {S}}) ^ {2}}{2}\right), \tag {13}
$$
which gives us our optimal transport loss. Combined with relative root mean squared error (RMSE)
$$
\ell_ {\mathrm {R M S E}} (\mathbf {U}, \hat {\mathbf {U}}) := \frac {1}{K + 1} \sum_ {\mathbf {u} _ {t}, \hat {\mathbf {u}} _ {t} \in \mathbf {U}, \hat {\mathbf {U}}} \frac {\left\| \mathbf {u} _ {t} - \hat {\mathbf {u}} _ {t} \right\| _ {2}}{\left\| \mathbf {u} _ {t} \right\| _ {2}} \tag {14}
$$
for short-term prediction consistency [2, 16, 18], our final loss function is
$$
\ell (\theta) = \underset { \begin{array}{c} n \in \{1, \dots , N \} \\ I \in \{0, \dots , L - K \} \end{array} } {\mathbb {E}} \left[ \alpha \ell_ {\mathrm {O T}} \left(\mathbf {S} _ {I: I + K} ^ {(n)}, \hat {\mathbf {S}} _ {I: h: I + K} ^ {(n)}\right) + \ell_ {\mathrm {R M S E}} \left(\mathbf {U} _ {I: I + K} ^ {(n)}, \hat {\mathbf {U}} _ {I: h _ {\mathrm {R M S E}}: I + K} ^ {(n)}\right) \right], \tag {15}
$$
where  $\mathbf{S}_{I:I + K}^{(n)}\coloneqq \{\mathbf{s}(\mathbf{u})\mid \mathbf{u}\in \mathbf{U}_{I,I + K}^{(n)}\}$ ,  $\hat{\mathbf{S}}_{I:h:I + K}^{(n)}\coloneqq \{\mathbf{s}(\hat{\mathbf{u}})\mid \hat{\mathbf{u}}\in \hat{\mathbf{U}}_{I:h:I + K}^{(n)}\}$ , and  $\alpha >0$  is a hyperparameter. Note that  $\hat{\mathbf{S}}_{I:h:I + K}^{(n)}$  and  $\hat{\mathbf{U}}_{I:h:\mathrm{RMSE}:I + K}^{(n)}$  implicitly depend on weights  $\theta$ .
3.2 Contrastive feature learning
When there is an absence of prior knowledge pertaining to the underlying dynamical system, or when the statistical attributes are not easily differentiable, we propose an alternative contrastive learning-based approach to learn the relevant invariant statistics directly from the data. We first use contrastive learning to train an encoder to capture invariant statistics of the dynamics in the multi-environment setting. We then leverage the feature map derived from this encoder to construct a feature loss that preserves the learned invariant statistics during neural operator training.
Contrastive learning. The objective of self-supervised learning is to train an encoder  $f_{\psi}(\mathbf{U})$  (with parameters  $\psi$ ) to compute relevant invariant statistics of the dynamics from sequences  $\mathbf{U}$  with fixed length  $K + 1$ . We do not explicitly train on the environment parameters  $\phi$  but rather use a general-purpose contrastive learning approach that encourages the encoder  $f_{\psi}$  to learn a variety of time-invariant features that are able to distinguish between sequences from different trajectories (and therefore different  $\phi$ ).
A contrastive learning framework using the Noise Contrastive Estimation (InfoNCE) loss has been shown to preserve context-aware information by training to match sets of positive pairs while treating all other combinations as negative pairs [32]. The selection of positive pairs is pivotal to the success of contrastive learning. In our approach, the key premise is that two sequences  $\mathbf{U}_{I:I + K}^{(n)}$ ,  $\mathbf{U}_{J:J + K}^{(n)}$  from the same trajectory  $\mathbf{U}_{0:L}^{(n)}$  both sample the same chaotic attractor, i.e. their statistics should be similar, so we treat any such pair of sequences as positive pairs. Two sequences  $\mathbf{U}_{I:I + K}^{(n)}$ ,  $\mathbf{U}_{H:H + K}^{(m)}$  from different trajectories are treated as negative pairs. This allows us to formulate the InfoNCE loss as:
$$
\begin{array}{l} \operatorname {l} _ {\operatorname {I n f o N C E}} (\psi ; \tau) := \\ \underset { \begin{array}{c} n \in \{1, \dots , N \} \\ I, J \in \{0, \dots , L - K \} \end{array} } {\mathbb {E}} \left[ - \log \left(\frac {\exp \left(\left\langle f _ {\psi} \left(\mathbf {U} _ {I : I + K} ^ {(n)}\right) , f _ {\psi} \left(\mathbf {U} _ {J : J + K} ^ {(n)}\right) \right\rangle / \tau\right)}{\underset {m \neq n} {\mathbb {E}} \left[ \exp \left(\left\langle f _ {\psi} \left(\mathbf {U} _ {I : I + K} ^ {(n)}\right) , f _ {\psi} \left(\mathbf {U} _ {H : H + K} ^ {(m)}\right) \right\rangle / \tau\right) \right]}\right) \right]. \tag {16} \\ \end{array}
$$
The term in the numerator enforces alignment of the positive pairs which ensures we obtain time-invariant statistics, while the term in the numerator encourages uniformity, i.e. maximizing mutual information between the data and the embedding, which ensures we can distinguish between negative pairs from different trajectories [41]. This provides intuition for why our learned encoder  $f_{\psi}$  identifies relevant time-invariant statistics that can distinguish different chaotic attractors.
Contrastive feature loss. To construct our feature loss, we use the cosine distance between a series of features of the encoder network  $f_{\psi}$  [42]:
$$
\ell_ {\mathrm {C L}} \left(\mathbf {U}, \hat {\mathbf {U}}; f _ {\psi}\right) := \sum_ {l} \cos \left(f _ {\psi} ^ {l} (\mathbf {U}), f _ {\psi} ^ {l} (\hat {\mathbf {U}})\right), \tag {17}
$$
where  $f_{\psi}^{l}$  gives the output the  $l$ -th layer of the neural network. The combined loss that we use for training the neural operator is given by
$$
\ell (\theta) = \underset { \begin{array}{c} n \in \{1, \dots , N \} \\ I \in \{0, \dots , L - K \} \end{array} } {\mathbb {E}} \left[ \lambda \ell_ {\mathrm {C L}} \left(\mathbf {U} _ {I: I + K} ^ {(n)}, \hat {\mathbf {U}} _ {I: h: I + K} ^ {(n)}; f _ {\psi}\right) + \ell_ {\mathrm {R M S E}} \left(\mathbf {U} _ {I: I + K} ^ {(n)}, \hat {\mathbf {U}} _ {I: h _ {\mathrm {R M S E}}: I + K} ^ {(n)}\right) \right], \tag {18}
$$
We evaluate our approach on the 1D chaotic Kuramoto-Sivanshinsky (KS) system and a finite-dimensional Lorenz 96 system. In all cases, we ensure that the systems under investigation remain in chaotic regimes. We demonstrate the effectiveness of our approach in preserving key statistics in these unpredictable systems, showcasing our ability to handle the complex nature of chaotic systems. The code is available at: https://github.com/roxie62/neuraloperators_for Chaos.
Experimental setup. Our data consists of noisy observations  $\mathbf{u}(t)$  with noise  $\eta \sim \mathcal{N}(0, r^2 \sigma^2 I)$ , where  $\sigma^2$  is the temporal variance of the trajectory and  $r$  is a scaling factor. Baselines. We primarily consider the baseline as training with RMSE [2]. We have additional baselines in Appendix B, including Gaussian denoising and a Sobolev norm loss with dissipative regularization [16]. Backbones. We use the Fourier neural operator (FNO, [18]). Evaluation metrics. We use a variety of statistics-based evaluation metrics and other measures that characterize the chaotic attractor. See Appendix C.1 for details.
4.1 Lorenz-96
As is a common test model for climate systems, data assimilation, and other geophysical applications [43-45], the Lorenz-96 system is a key tool for studying chaos theory, turbulence, and nonlinear dynamical systems. It is described by the differential equation
$$
\frac {d u _ {i}}{d t} = \left(u _ {i + 1} - u _ {i - 2}\right) u _ {i - 1} - u _ {i} + F \tag {19}
$$
Its dynamics exhibit strong energy-conserving non-linearity, and for a large  $F \geq 10$ , it can exhibit strong chaotic turbulence and symbolizes the inherent unpredictability of the Earth's climate.
Experimental setup. When using optimal transport loss, we assume that expert knowledge is derived from the underlying equation. For Lorenz-96, we define the relevant statistics as  $\mathbf{s}(\mathbf{u}) \coloneqq \left\{\frac{du_i}{dt}, (u_{i+1} - u_{i-2})u_{i-1}, u_i\right\}$ . We generate 2000 training data points with each  $\phi^{(n)}$  randomly sampled from a uniform distribution with the range [10.0, 18.0]. We vary the noise level  $r$  from 0.1 to 0.3 and show consistent improvement in the relevant statistics. Results. The results are presented in Table 1, and predictions and invariant statistics are shown in Fig. 3 (refer to C.4 for more visualizations).
4.2 Kuramoto-Sivashinsky
Known as a model for spatiotemporal chaos, Kuramoto-Sivashinsky (KS) has been widely used to describe various physical phenomena, including fluid flows in pipes, plasma physics, and dynamics of certain chemical reactions [46]. It captures wave steepening via the nonlinear term  $u\frac{\partial u}{\partial x}$ , models dispersion effects through  $\frac{\partial^2u}{\partial x^2}$ , and manages discontinuities by introducing hyper-viscosity via  $\frac{\partial^4u}{\partial x^4}$ :
$$
\frac {\partial u}{\partial t} = - u \frac {\partial u}{\partial x} - \phi \frac {\partial^ {2} u}{\partial x ^ {2}} - \frac {\partial^ {4} u}{\partial x ^ {4}}. \tag {20}
$$
Figure 3: Sampled emulator dynamics and summary statistic distributions. We evaluate our proposed approaches by comparing them to a baseline model that is trained solely using relative RMSE loss. We conduct this comparison on two dynamical systems: (a) Lorenz-96 and (b) Kuramoto-Sivashinsky (KS). For each system, we show a visual comparison of the predicted dynamics (left) and two-dimensional histograms of relevant statistics (middle and right). We observe that training the neural operator with our proposed optimal transport (OT) or contrastive learning (CL) loss significantly enhances the long-term statistical properties of the emulator, as seen in the raw emulator dynamics and summary statistic distributions. The performance of the CL loss, which uses no prior knowledge, is comparable to that of the OT loss, which requires an explicit choice of summary statistics.
Experimental setup. For the KS system, we define  $\mathbf{s}(\mathbf{u})\coloneqq \left\{\frac{\partial u}{\partial t},\frac{\partial u}{\partial x},\frac{\partial^2u}{\partial x^2}\right\}$  [47]. We generate 2000 training data points, with each  $\phi^{(n)}$  being randomly selected from a uniform distribution within the range of [1.0, 2.6]. Results. We report our results over 200 test instances in Table 2 and visualize the predictions and invariant statistics in Fig. 3.
We have demonstrated two approaches for training neural operators on chaotic dynamics by preserving the invariant measures of the chaotic attractors. The optimal transport approach uses knowledge of the underlying system to construct a physics-informed objective that matches relevant summary statistics, while the contrastive learning approach uses a multi-environment setting to directly learn relevant invariant statistics. In both cases, we see a significant improvement in the ability of the emulator to reproduce the invariant statistics of chaotic attractors from noisy data when compared with traditional RMSE-only training that focuses on short-term forecasting.
We also find, in high-noise settings, that both of our approaches give similar or lower leading LE errors than an emulator trained on RMSE loss alone, despite the fact that our method is only encouraged to match invariant statistics of the final attractor rather than dynamical quantities. We also see evidence that the fractal dimension of the contrastive learning approach is closer to the true attractor. However, we note that the fractal dimension is difficult to reliably estimate for high-dimensional chaotic attractors [48].
r	Training	Histogram Error ↓	Energy Spec. Error ↓	Leading LE Error ↓	FD Error ↓
0.1	lRMSE	0.056 (0.051, 0.062)	0.083 (0.078, 0.090)	0.013 (0.006, 0.021)	1.566 (0.797, 2.309)
lOT + lRMSE	0.029 (0.027, 0.032)	0.058 (0.052, 0.064)	0.050 (0.040, 0.059)	1.424 (0.646, 2.315)
lCL + lRMSE	0.033 (0.029, 0.037)	0.058 (0.049, 0.065)	0.065 (0.058, 0.073)	1.042 (0.522, 1.685)
0.2	lRMSE	0.130 (0.118, 0.142)	0.182 (0.172, 0.188)	0.170 (0.156, 0.191)	2.481 (1.428, 3.807)
lOT + lRMSE	0.039 (0.035, 0.042)	0.086 (0.079, 0.095)	0.016 (0.006, 0.030)	2.403 (1.433, 3.768)
lCL + lRMSE	0.073 (0.066, 0.080)	0.131 (0.117, 0.149)	0.012 (0.006, 0.018)	1.681 (0.656, 2.682)
0.3	lRMSE	0.215 (0.204, 0.234)	0.291 (0.280, 0.305)	0.440 (0.425, 0.463)	3.580 (2.333, 4.866)
lOT + lRMSE	0.057 (0.052, 0.064)	0.123 (0.116, 0.135)	0.084 (0.062, 0.134)	3.453 (2.457, 4.782)
lCL + lRMSE	0.132 (0.111, 0.151)	0.241 (0.208, 0.285)	0.064 (0.045, 0.091)	1.894 (0.942, 3.108)
Table 1: Emulator performance on Lorenz-96 data with varying noise scale  $r = 0.1, 0.2, 0.3$ . The median (25th, 75th percentile) of the evaluation metrics (Appendix C.1) are computed on 200 Lorenz-96 test instances (each with 1500 time steps) for the neural operator trained with (1) only RMSE loss  $\ell_{\mathrm{RMSE}}$ ; (2) optimal transport (OT) and RMSE loss  $\ell_{\mathrm{OT}} + \ell_{\mathrm{RMSE}}$  (using prior knowledge to choose summary statistics); and (3) contrastive learning (CL) and RMSE loss  $\ell_{\mathrm{CL}} + \ell_{\mathrm{RMSE}}$  (without prior knowledge). We show significant improvements on the long-term statistical metrics including  $L_1$  histogram error of the chosen statistics  $\mathbf{S}(\mathbf{u}) := \{\frac{du_i}{dt}, (u_{i+1} - u_{i-2})u_{i-1}, u_i\}$ ; relative error of Fourier energy spectrum; and absolute error of estimated fractal dimension (FD). For high noise, OT and CL training also improve the leading Lyapunov exponent (LE) of the neural operator.
Training	Histogram Error ↓	Energy Spec. Error ↓	Leading LE Error ↓
\( \ell_{\text{RMSE}} \)	0.390 (0.325, 0.556)	0.290 (0.225, 0.402)	0.101 (0.069, 0.122)
\( \ell_{\text{OT}} + \ell_{\text{RMSE}} \)	0.172 (0.146, 0.197)	0.211 (0.188, 0.250)	0.094 (0.041, 0.127)
\( \ell_{\text{CL}} + \ell_{\text{RMSE}} \)	0.193 (0.148, 0.247)	0.176 (0.130, 0.245)	0.108 (0.068, 0.132)
Table 2: Emulator performance on Kuramoto-Sivashinsky data with noise scale  $r = 0.3$ . The median (25th, 75th percentile) of the evaluation metrics (Appendix C.1) are computed on 200 Kuramoto-Sivashinsky test instances (each with 1000 time steps) for the neural operator trained with (1) only RMSE loss  $\ell_{\mathrm{RMSE}}$ ; (2) optimal transport (OT) and RMSE loss  $\ell_{\mathrm{OT}} + \ell_{\mathrm{RMSE}}$  (using prior knowledge to choose summary statistics); and (3) contrastive learning (CL) and RMSE loss  $\ell_{\mathrm{CL}} + \ell_{\mathrm{RMSE}}$  (without prior knowledge). We again show significant improvements in the long-term statistical metrics including  $L_1$  histogram error of the chosen statistics  $\mathbf{S}(\mathbf{u}) \coloneqq \{\frac{\partial u}{\partial t}, \frac{\partial u}{\partial x}, \frac{\partial^2 u}{\partial x^2}\}$  and relative error of Fourier energy spectrum. The fractal dimension (FD) is highly unstable in high dimensions [48] and could not be estimated for this dataset.
Limitations. Because we rely on invariant measures, our current approach is limited to trajectory data from attractors, i.e. we assume that the dynamics have reached an attractor and are not in a transient phase. We also cannot handle explicit time dependence, including time-dependent forcing or control parameters. For the optimal transport approach, choosing informative summary statistics based on prior knowledge is key to good performance (Appendix B.3). For the contrastive learning approach, the quality of the learned invariant statistics also depends on the diversity of the environments present in the multi-environment setting, although our additional experiments show that we can still obtain good performance even with minimal environment diversity (Appendix B.4).
Future work. In the future, we may be able to adapt our approaches to allow for mild time dependence by restricting the time range over which we compute statistics and select positive pairs. This would allow us to study slowly varying dynamics as well as sharp discrete transitions such as tipping points. We can also improve the diversity of the data for contrastive learning by designing new augmentations or using the training trajectory of the neural operator to generate more diverse negative pairs. We will investigate generalizing our approaches to other difficult systems, such as stochastic differential equations or stochastic PDEs, and we would like to further study the trade-offs and synergies between focusing on short-term forecasting (RMSE) and capturing long-term behavior (invariant statistics). In addition, we would like to investigate and compare training methods [22-24] across different architectures.
Broader impacts. While better emulators for chaotic dynamics may be used in a wide range of applications, we foresee no direct negative societal impacts.
The authors gratefully acknowledge the support of DOE grant DE-SC0022232, AFOSR grant FA9550-18-1-0166, and NSF grants DMS-2023109 and DMS-1925101. In addition, Peter Y. Lu gratefully acknowledges the support of the Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship, a Schmidt Futures program.
[43] Dirk Leendert van Kekem. Dynamics of the lorenz-96 model: Bifurcations, symmetries and waves, 2018. https://research.rug.nl/en/publications/dynamics-of-the-lorenz-96-model-bifurcations-symmetries-and-waves.
[44] Kody JH Law, Daniel Sanz-Alonso, Abhishek Shukla, and Andrew M Stuart. Filter accuracy for the Lorenz 96 model: Fixed versus adaptive observation operators. Physica D: Nonlinear Phenomena, 325:1-13, 2016.
[45] Andrew J Majda and John Harlim. Filtering Complex Turbulent Systems. Cambridge University Press, 2012.
[46] James M Hyman and Basil Nicolaenko. The kuramoto-sivashinsky equation: a bridge between pde's and dynamical systems. Physica D: Nonlinear Phenomena, 18(1-3):113-126, 1986.
[47] Pantelis R Vlachas, Georgios Arampatzis, Caroline Uhler, and Petros Koumoutsakos. Multiscale simulations of complex systems by learning their effective dynamics. Nature Machine Intelligence, 4(4):359-366, 2022.
[48] H. S. Greenside, A. Wolf, J. Swift, and T. Pignataro. Impracticality of a box-counting algorithm for calculating the dimensionality of strange attractors. Phys. Rev. A, 25:3453-3456, Jun 1982. doi: 10.1103/PhysRevA.25.3453. URL https://link.aps.org/doi/10.1103/PhysRevA.25.3453.
[49] Bowen Zhu, Jingwei Hu, Yifei Lou, and Yunan Yang. Implicit regularization effects of the sobolev norms in image processing. arXiv preprint arXiv:2109.06255, 2021.
[50] Chuanfu Wang, Chunlei Fan, and Qun Ding. Constructing discrete chaotic systems with positive Lyapunov exponents. International Journal of Bifurcation and Chaos, 28(07):1850084, 2018.
[51] Pavel Dvurechensky, Alexander Gasnikov, and Alexey Kroshnin. Computational optimal transport: Complexity by accelerated gradient descent is better than by sinkhorn's algorithm. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1367-1376. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/dvurechensky18a.html.
[52] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
A Additional Discussion
A.1 Contrastive feature learning vs. physics-informed optimal transport
Our two proposed approaches both aim to train neural operators to preserve the invariant measures of chaotic attractors. Ultimately, both methods have strong performance according to a variety of metrics, but the contrastive learning approach requires no prior physical knowledge and is often faster. Both approaches have significant advantages over the standard approach of minimizing RMSE, which fails to preserve important statistical characteristics of the system.
The primary conceptual difference between our approaches is that the optimal transport approach relies on prior domain knowledge, while the contrastive learning approach learns from the data alone. Specifically, the physics-informed optimal transport approach requires a choice of summary statistics, so it is much more dependent on our prior knowledge about the system and its chaotic attractor. Contrastive feature learning does not require a choice of statistics and instead learns useful invariant statistics on its own—although it may still be useful to have known relevant statistics to use as evaluation metrics for hyperparameter tuning.
The neural operators trained using the optimal transport loss perform the best on the  $L_{1}$  histogram distance of the summary statistics S (Table 1 and 2), which is unsurprising given that the optimal transport loss specifically matches the distribution of S. If we look at a different evaluation metric, e.g. the energy spectrum error or leading LE, the contrastive learning loss, in some cases, performs better even without prior knowledge. This suggests that contrastive feature learning may be capturing a wider range or different set of invariant statistics. We also see that the quality of the emulator degrades if we choose less informative statistics (Appendix B.3).
A.2 Interactions and trade-offs between short-term prediction and long-term statistics