SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Initially, we suspected that ConvLSTM outperforms the other models since the loss curves looked like it is able to learn "faster" due to its smaller amount of learnable parameters. However, this was only true when training on a small subset of the years. Further experiments covering the whole time-length showed that all ML models are early-stopping around 20 epochs as their performance gets worse after that. It should be noted, that CimaX's performance is plummeting suddenly after 20 epochs. Further experiments could be done to try to stabilize the training and investigate if CimaX performance improves over a longer period of training time. Future work could investigate different training regime settings for all models. The experiments we conducted to date indicate to us that all ML models need around the same amount of epochs to learn to super emulate.
We think that the determining factor of the super-emulator performance is the models generalization capability. The better it can generalize onto different climate models, the better it can perform as a super-emulator. To investigate this, we conducted finetuning experiments (see M.3. They show that the smaller models are better at generalizing across different climate models, with ConvLSTM clearly outperforming all the other models. The ML-model ranking of the finetuning results aligns with the ranking for super-emulation. These results support our hypothesis that smaller models are better at generalizing and, for this reason, might be the better super-emulators for now.
We are convinced that adding challenging tasks - such as the super-emulation task presented here - will be helpful to benchmark ML models on climate relevant tasks.
M.3 Generalization Capabilities of ML Models
The results for the finetuning experiments of the single-emulator are shown in Table 7. The first column lists the choice of the pre-training dataset where "None" means that the model is trained directly on NorESM2-LM (single-emulator experiment). ConvLSTM seems to outperform all other ML models for these experiments. One possible explanation for this is that ConvLSTM is indeed the ML model that can generalize best from one climate model to another climate model. Another reasonable explanation however is, that ConvLSTM is just in general the best ML model to emulate this specific climate model, NorESM2-LM. Hence, it is more informative to look at the performance differences between single-emulator and finetuned model which is shown in Table 8.
Here, we can see that the ConvLSTM achieves the largest performance gain through the finetuning setting,  $\mathrm{ClimaX}_{\text{Frozen}}$  gains slightly, U-Net neither gains nor looses, while  $\mathrm{ClimaX}$  drops in performance. There are some exceptions, e.g.  $\mathrm{ClimaX}$  gets the highest performance gain for FGOALS-f3-L. The slight performance gain of  $\mathrm{ClimaX}_{\text{Frozen}}$  could be explained by the fact that it had the poorest overall performance on NorESM2-LM before, i.e. there was more space for improvement compared to e.g. the U-Net that was already performing very well on NorESM2-LM. However, the pretty consistent performance gain of the ConvLSTM cannot be explained by that, indicating that the ConvLSTM could actually be inherently better at generalizing to other climate models. While ConvLSTM seems to benefit from pre-training, it makes less of a difference for U-Net, and seems to have little benefit for  $\mathrm{ClimaX}$  when evaluating on the NorESM2-LM dataset. The benefit for  $\mathrm{ClimaX}$  might be limited as compared to other models as it has already been pre-trained on a collection of datasets as described in Nguyen et al. (2023).
Overall, Table 8 shows that finetuning on one climate model can improve the performance on another model. This indicates, that climate models indeed share patterns and information and that patterns learned from one climate model can be transferred to another.
Pre-training Dataset	U-Net	ConvLSTM	ClimaX	ClimaXFrozen
TAS	PR	TAS	PR	TAS	PR	TAS	PR
None	0.291	0.309	0.278	0.292	0.311	0.325	0.353	0.370
AWI-CM-1-1-MR	0.302	0.325	0.208	0.220	0.351	0.365	0.335	0.346
BCC-CSM2-MR	0.327	0.253	0.277	0.291	0.354	0.368	0.326	0.344
EC-Earth3	0.302	0.319	0.247	0.262	0.385	0.399	0.329	0.343
FGOALS-f3-L	0.283	0.306	0.242	0.256	0.327	0.341	0.320	0.342
MPI-ESM1-2-HR	0.266	0.284	0.218	0.229	0.412	0.427	0.304	0.317
Table 7: RMSE values for ML models that were finetuned on NorESM2-LM and pre-trained on different climate models. We report the RMSE for two variables, TAS (surface air temperature) and PR (precipitation). The best performing models are emboldened. The pre-training dataset column shows which dataset the model was initially trained on before being finetuned on the NorESM2-LM dataset. The first row with "None" for the pre-training dataset represents the results from training on NorESM2-LM from scratch. ConvLSTM performs best across all climate model datasets here.
Pre-training Dataset	U-Net	ConvLSTM	ClimaX	\( ClimaX_{Frozen} \)
TAS	PR	TAS	PR	TAS	PR	TAS	PR
AWI-CM-1-1-MR	-0.011	-0.016	0.070	0.072	-0.004	0.004	0.089	0.024
BCC-CSM2-MR	-0.036	0.056	0.001	0.001	-0.043	-0.043	0.027	0.027
EC-Earth3	-0.011	-0.016	0.031	0.030	-0.074	-0.074	0.024	0.027
FGOALS-f3-L	0.019	0.013	0.005	0.006	0.058	0.058	0.009	0.001
MPI-ESM1-2-HR	0.025	0.025	0.060	0.063	-0.101	-0.102	0.049	0.053
Table 8: RMSE delta values for ML models that were finetuned on NorESM2-LM and pre-trained on different climate models. Here, we report the RMSE differences between pre-training and finetuning for TAS and PR. The pre-training dataset column shows which dataset the model was initially trained on before being finetuned on the NorESM2-LM dataset. Positive digits indicate that pre-training on this climate model improved performance compared to the single-emulator experiment, negative digits that performance dropped. ConvLSTM is the model that can generate the largest performance gain by finetuning.