SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the case of the optimal transport loss $^2$ , described in Section 3.1, the blur parameter  $\gamma$  governs the equilibrium between faster convergence speed and the precision of the Wasserstein distance approximation. We determined the value of  $\gamma$  through a grid search, examining values ranging from  $\{0.01, 0.02, \dots, 0.20\}$ . Similarly, we decided the weights of the optimal transport loss  $\alpha$  through a grid search, exploring values from  $\{0.5, 1, 1.5, \dots, 3.0\}$ . For the experiments conducted using Lorenz-96, we selected  $\alpha = 3$  and  $\gamma = 0.02$ .
In the context of the feature loss, the primary hyperparameter we need to consider is its weights, represented as  $\lambda$ . Given that we do not have knowledge of the invariant statistics  $\mathbf{S}$  during the validation phase, we adjust  $\lambda$  according to a specific principle: our aim is to reduce the feature loss  $\ell_{\mathrm{feature}}$  (as defined in Eqn. 17) as long as it does not adversely affect the RMSE beyond a predetermined level (for instance,  $10\%$ ) when compared with the baseline model, which is exclusively trained on RMSE. As illustrated in Figure 4, we adjust the values of  $\lambda$  in a systematic grid search from  $\{0.2, 0.4, 0.6, 0.8, 1.0, 1.2\}$ . From our observations during the validation phase, we noted that the feature loss was at its lowest with an acceptable RMSE (which is lower than  $110\%$  compared to the baseline) when  $\lambda = 0.8$ . Therefore, we have reported our results with  $\lambda$  set at 0.8.
Results visualization. We present more results with varying noise levels in Figures 5, 6, 7.
$^{2}$ https://www.kernel-operations.io/geomloss/
Figure 4: The trend of feature loss with its weight  $\lambda$  when the scale of noise is  $r = 0.3$ . The solid lines in the figure represent the evaluation metrics during the validation phase, comparing the outputs of the neural operator to the noisy data. In contrast, the dashed lines represent the actual metrics we are interested in, comparing the outputs of the neural operator to the clean data and calculating the error of the invariant statistics. In addition, the horizontal solid dashed line corresponds to the bar we set for the RMSE, i.e.,  $110\%$  of the RMSE when  $\lambda = 0$ . We observe that, (1) with the increase of  $\lambda$  from 0, the feature loss decreases until  $\lambda$  reaches 1.0. (2) The RMSE generally increases with the increase of  $\lambda$ . (3) The unseen statistical error generally decreases with the increase of  $\lambda$ . We reported the results when  $\lambda = 0.8$  as our final result, since the further increase of  $\lambda$  does not bring further benefit in decreasing the feature loss, and the result remains in an acceptable range in terms of RMSE.
C.5 Kuramoto-Sivashinsky
Data generation. In order to ascertain that we are operating within a chaotic regime, we set the domain size  $L = 50$  and the spatial discretization grids number  $d = 256$ . We select initial conditions randomly from a uniform range between  $[-\pi, \pi]$ . A fourth-order Runge-Kutta method was utilized to perform all simulations. We generate 2000 trajectories for training, with each  $\phi^{(n)}$  being randomly chosen from a uniform distribution within the interval [1.0, 2.6].
Training hyperparameters. In the case of the optimal transport loss, similar to the discussion in C.4, we search the blur value  $\gamma$  and the weights of loss  $\alpha$  from a grid search. And in our experiment, we set  $\alpha = 3$  and  $\gamma = 0.05$ . In the case of the feature loss, we again adopt the same rule for deciding the value of  $\lambda$ , where we compare the trend of feature loss with the relative change of RMSE of the baseline and choose to report the results with the lowest feature loss and acceptable RMSE increase (110% compared to the baseline when  $\lambda = 0$ ).
Figure 5: Visualization of the predictions when the noise Level  $r = 0.1$ . We evaluate our method by comparing them to the baseline that is trained solely using RMSE. For two different instances (a) and (b), we visualize the visual comparison of the predicted dynamics (left), two-dimensional histograms of relevant statistics (middle and right). We notice that, with the minimal noise, the predictions obtained from all methods look statistically consistent to the true dynamics.
Figure 6: Visualization of the predictions when the noise level  $r = 0.2$ . We evaluate our method by comparing them to the baseline that is trained solely using RMSE. For two different instances (a) and (b), we visualize the visual comparison of the predicted dynamics (left), two-dimensional histograms of relevant statistics (middle and right). We observe that as the noise level escalates, the degradation of performance in the results of RMSE is more rapid compared to our method, which employs optimal transport and feature loss during training.
Figure 7: Visualization of the predictions when the noise level  $r = 0.3$ . We evaluate our method by comparing them to the baseline that is trained solely using RMSE. For two different instances (a) and (b), we visualize the visual comparison of the predicted dynamics (left), two-dimensional histograms of relevant statistics (middle and right). We find that, under higher levels of noise, the RMSE results exhibit fewer negative values, as indicated by the blue stripes in the predicted dynamics. This is further confirmed by the energy spectrum, which clearly shows that the RMSE results are significantly deficient in capturing high-frequency signals.