SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Besides the number of refinement step, PDE-Refiner has as a second hyperparameter the minimum noise variance  $\sigma_{\mathrm{min}}^2$ , i.e., the variance of the added noise in the last refinement step. The noise variance determines the different amplitude levels at which PDE-Refiner improves the prediction. To show how sensitive PDE-Refiner is to different values of  $\sigma_{\mathrm{min}}^2$ , we repeat the experiments of PDE-Refiner on the KS equation while varying  $\sigma_{\mathrm{min}}^2$ . The results in Figure 25 show that PDE-Refiner is robust to small changes of  $\sigma_{\mathrm{min}}^2$  and there exist a larger band of values where it performs equally well. When increasing the variance further, the performance starts to decrease since the noise is too high to model the lowest amplitude information. Note that the results on Figure 25 show the performance on the test set, while the hyperparameter selection, in which we selected  $\sigma_{\mathrm{min}}^2 = 2\mathrm{e} - 7$ , was done on the validation set.
(a) 1 step
(b) 125 steps
(c) 500 steps
(d) 1000 steps
(e) 1 step
(f) 125 steps
(g) 500 steps
Figure 26: Evaluating PDE solver stability over very long rollouts (800 seconds, corresponding to 1000 autoregressive prediction steps). (a-d) The frequency spectrum of predictions of an MSE-trained model and PDE-Refiner. Over time, the MSE baseline's overestimation of the high frequencies accumulates. In comparison, PDE-Refiner shows to have an increase of extremely high frequencies, which is likely caused by the continuous adding of Gaussian noise. (e-h) When we apply the error correction [58] on our models by setting all frequencies above 60 to zero, PDE-Refiner remains stable even for 1000 steps and does not diverge from the ground truth frequency spectrum.
(h) 1000 steps
In combination with the hyperparameter of the number of refinement steps, to which PDE-Refiner showed to also be robust if more than 3 steps is chosen, PDE-Refiner is not very sensitive to the newly introduced hyperparameters and values in a larger range can be considered.
E.8 Stability of Very Long Rollouts
Besides accurate rollouts, another important aspect of neural PDE solvers is their stability. This refers to the solvers staying in the solution domain and not generating physically unrealistic results. To evaluate whether our solvers remain stable for a long time, we roll out an MSE-trained baseline and PDE-Refiner for 1000 autoregressive prediction steps, which corresponds to 800 seconds simulation time. We then perform a frequency analysis and plot the spectra in Figure 26. We compare the spectra to the ground truth initial condition, to have a reference point of common frequency spectra of solutions on the KS equation.
For the MSE-trained baseline, we find that the high frequencies, that are generally overestimated by the model, accumulate over time. Still, the model maintains a frequency spectrum close to the ground truth for wavenumbers below 40. PDE-Refiner maintains an accurate frequency spectrum for more than 500 steps, but suffers from overestimating the very high frequencies in very long rollouts. This is likely due to the iterative adding of Gaussian noise, that accumulates high-frequency errors. Further, the U-Net has a limited receptive field such that the model cannot estimate the highest frequencies properly. With larger architectures, this may be preventable.
However, a simpler alternative is to correct the predictions for known invariances, as done in McGreivy et al. [58]. We use the same setup as for Figure 3 by setting the highest frequencies to zero. This stabilizes PDE-Refiner, maintaining a very accurate estimation of the frequency spectrum even at 800 seconds. The MSE-trained model yet suffers from an overestimation of the high-frequencies.
In summary, the models we consider here are stable for much longer than they remain accurate to the ground truth. Further, with a simple error correction, PDE-Refiner can keep up stable predictions for more than 1000 autoregressive rollout steps.