SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first describe the dataset which has been used for the construction of the pre-trained GNN feature map (OC20), and then present the system-specific MD datasets where our method is fine-tuned on  $(\mathrm{Cu / formate},\mathrm{Fe} / \mathrm{N}_2)$ .
OC20 The Open Catalyst (OC) 20 is a large dataset of ab initio calculations aimed at estimating adsorption energies on catalytic surfaces. It comprises  $\sim 250$  millions of DFT calculations, generated from over 1.2 million relaxations trajectories of different combinations of molecules and surfaces. In each relaxation, the positions of the molecule and of the surface upper layers are optimized via gradient descent in order to compute the adsorption energy. The adsorbate is selected out of 82 molecules relevant to environmental applications, while, for each of them, up to  $55^{3}$  surfaces are selected, including binary and ternary compounds. We underline that, for each adsorbate-surface pair, the configurational space sampled is very limited, and especially it does not cover out-of-equilibrium and reactive (e.g. bond forming or breaking) events.
We fine-tune the method and test it on two datasets that are representative of reactive catalytic events, obtained by means of molecular dynamics simulations coupled with enhanced sampling methods [64, 65] to avoid mode collapse into the local minima of the potential energy landscape. Indeed, whereas the OC20 dataset contains short, correlated relaxations toward the nearest equilibrium state, typical catalytic reaction datasets require sampling all local minima (adsorption states of the molecule) and especially reactive events, in which, due to interaction with the surface, bonds can be broken or formed. For this reason, these applications are challenging as they relate to realistic datasets containing mostly reactive events that are outside the distribution of the OC20 dataset. We split all the below datasets into a train, validation, and test set using random splitting of  $60/20/20$ .
Cu/formate The first dataset is a collection of molecular dynamics simulations of the dehydrogenation reaction of formate on a copper  $(\mathrm{Cu}) < 110>$  surface [15], initialized along the reaction path (obtained with the Nudged Elastic Band method [65]), in which the molecule loses its hydrogen atom upon interaction with the surface.
$\mathrm{Fe} / \mathrm{N}_2$  ( $D_i$ ) The second dataset consists of molecular dynamics simulations of a nitrogen molecule adsorbing on an iron (Fe)  $< 111>$  surface at high temperature ( $T = 700\mathrm{K}$ ) and breaking in two nitrogen atoms [66]. A peculiarity of this dataset is that it contains data from different sources (e.g. standard and biased molecular dynamics) and system sizes, allowing us to also assess the transferability of the methods across different conditions. For this reason, we divide it into 4 subsets, denoted with  $D_i$ :
$D_{1}$ : AI-MD Ab initio molecular dynamics simulations. The resulting configurations are highly correlated and cover a small portion of the configurational space related to the adsorption process, thus being the closest dataset to the OC20 one.
$D_{2}$ : AI-METAD Here the ab initio MD simulation is accelerated with the help of the metadynamics [64] technique. This is an importance sampling method that allows rare events to be observed, and thus it has been employed for collecting reactive configurations in the training set [67]. Due to the metadynamics approach, a larger region of configurational space is sampled with respect to  $D_{1}$ , allowing one to sample one bond-breaking event.
$D_{3}$  : AL-METAD Dataset built from an active learning procedure using an ensemble of NNs combined with metadynamics. In this simulation, multiple reaction events are observed, covering a wider region of the configurational space and providing a large number of
uncorrelated samples. Hence, these configurations are far from those used to pre-train the feature map.
$D_{4}$ : AL-METAD-72 Same as  $D_{3}$  but the surface is composed of 72 atoms (8 layers) to test the transferability across systems of different sizes.
5.3 Interpolating between shared and independent weights
The  $\alpha$  parameter in the  $K_{\alpha}$  kernel can vary in the range [0, 1], which are the limiting cases between a shared or independent set of weights for each chemical species. We use cross-validation to set this parameter in practice. To initially fit the regularization parameter  $\lambda$  we set  $\alpha = 0$  and cross-validate  $\lambda \in \{10^{-3}, \dots, 10^{-9}\}$  using the same datasets. Despite this simple heuristic cross-validation scheme, as we will see, the scheme is effective, which we believe is a strength as it shows that the MEKRR method is simple to tune while still having the strongest performance among the competitors.
In Fig. 2 we show the cross-validation curves for MEKRR-(SchNet), related to the two datasets  $\mathrm{Cu}$  /formate and  $\mathrm{Fe} / \mathrm{N}_2$ . In the latter case, we perform the cross-validation only on  $(D_2)$ , which is representative of the family of datasets, and then use the found parameters also for the other datasets. The two plots show different behavior with the optimal  $\alpha$  for the  $\mathrm{Cu}$  /formate dataset occurring around  $10^{-2}$ . This means that the potential energy can be well described with shared weights across chemical species together with a small perturbation. Instead, in the  $\mathrm{Fe} / \mathrm{N}_2$  dataset the optimal  $\alpha$  occurs at the boundary leading to a kernel in which the weights for the two species are learned independently.
Figure 2: Validation error (RMSE / MAE) of MEKRR-(SchNet) on the Cu/formate and  $\mathrm{Fe} / \mathrm{N}_2$  ( $D_2$ ) datasets as a function of  $\alpha$  geometrically spaced on a grid from 0 to 1 with optimal  $\alpha$  and error given by a bold orange point. The optimal  $\alpha$  for the Cu/formate dataset is positive but close to zero while the optimal  $\alpha$  for the  $\mathrm{Fe} / \mathrm{N}_2$  is found at the boundary at 1.0 leading to a hard multi-embedding kernel. We see that tuning the  $\alpha$  allows for improved performance in practice and that the multi-weight formulation (5) is practically beneficial.
5.4 Potential energy regression
In this section, we consider the setting of predicting the potential energy surface. We first evaluate the performance of the models in predicting the energy for each of the datasets and then assess their generalization performance through the transfer learning setting, where we train and test on similar but distinct datasets.
Same-dataset energy prediction From Table 1 we see that MEKRR achieves the best performance in all datasets, both when the input features are extracted from SchNet and SCN. We note that, in general, the transfer learning algorithms (SchNet-FT and the two MEKRR variants) outperform the ones trained from scratch, with MEKRR being significantly faster (see Appendix B). Furthermore, it is worth highlighting that our method performs better than the baselines even when it is applied to datasets that are out-of-distribution for the pre-trained feature map. This is particularly evident for  $D_{3}$  and  $D_{4}$  which contain multiple reactive events (bond-breaking) that were never seen in the relaxations composing the OC20 dataset. This demonstrates the ability of the GNN trained on large and heterogeneous datasets to effectively represent chemical environments.
Table 1: Same-dataset energy prediction, metric being RMSE. The errors are in units of meV/atom. Best performance given by bold number in gray cell.
Group	Algorithm	Fe/N2	Cu/formate
D1	D2	D3	D4
Supervised	GAP	0.4	2.1	3.9	4.9	2.8
SchNet	0.5	4.1	5.1	6.2	6.0
SCN	0.3	5.1	7.5	7.3	2.5
Fine-tune	SchNet-FT	0.1	2.0	2.5	3.2	1.9
Ours	MEKRR-(SchNet)	0.1	1.3	2.4	3.3	1.2
MEKRR-(SCN)	0.2	0.9	1.9	2.7	1.7
Across-dataset energy prediction Here we evaluate the performance of the algorithms and MEKRR on transferring from different systems in the  $\mathrm{Fe / N_2}$  family of datasets. To do this we consider the task of zero-shot transfer learning (see e.g. [18] and references therein) where we evaluate a model trained on a source dataset  $D_{\mathrm{source}}$  on a target dataset  $D_{\mathrm{target}}$ . While the two datasets  $D_{\mathrm{source}}$  and  $D_{\mathrm{target}}$  may be sampled from arbitrary systems, we consider here systems that share some characteristics as we are evaluating the transfer capability of the models [68, 69]. Due to the ordering of the datasets  $D_{1}, \ldots, D_{4}$  in increasing complexity on several axes (size, ab initio vs. active sampling, standard vs. biased dynamics, etc.) we consider a transfer from simpler to more complicated systems. Successfully transferring from simpler to more complicated systems has real-world impact as it can alleviate the high computational cost required for labeling via DFT calculations by reducing the number of points. From Table 2 we see that MEKRR-(SCN) has the lowest error in four out of five tasks, while MEKRR-(SchNet) has the lowest error in the remaining task. Furthermore, the relative transferability of MEKRR compared to the other methods even improves as the task becomes harder. To this respect, it is worth noting that  $D_{1}, D_{2}, D_{3}$  are qualitatively similar, being all composed of 5 layers of Fe and differing for the sampling method used. Instead, the atomic environments contained in  $D_{4}$  are different as they refer to a slab with a different number of layers. This explains the different order of magnitudes in the last two columns. Despite this, MEKRR still performs very well compared to the baselines.
Table 2: Transfer evaluation of algorithms on source to target:  $D_{\mathrm{source}} \rightarrow D_{\mathrm{target}}$ , metric being RMSE. The errors are in units of meV/atom. Best performance is given by **bold** number in gray cell.
Group	Algorithm	D1→D2	D1→D3	D2→D3	D2→D4	D3→D4
Supervised	GAP	24.9	59.1	5.8	830	888
SchNet	13.2	15.4	6.2	93	107
SCN	22.1	29.3	9.7	139	131
Fine-tune	SchNet-FT	17.6	27.3	3.7	121	116
Ours	MEKRR-(SchNet)	8.0	9.3	2.9	27	55
MEKRR-(SCN)	7.0	6.3	2.0	40	42
5.5 Leveraging MEKRR beyond supervised learning
In the previous sections, we have shown how MEKRR performs very well on both supervised and transfer learning tasks. The effectiveness comes from the combination of a pre-trained feature map together with the  $K_{\alpha}$  kernel. However, this idea is not restricted to supervised learning. We can indeed leverage the similarity measure provided by the kernel for tasks beyond potential energy regression. As a simple example, in Fig. 3 we plot the kernel matrix, when using SchNet as the feature map, of a part of the trajectory of  $D_{2}$  containing an N-N bond breaking event in the cases  $\alpha = 0$  and  $\alpha = 1$ . In both images we can see a clear structure that highlights at least two distinct states, but with the second heatmap having more signal. We can then use the kernel to perform spectral clustering with two classes, the result is visualized on the top margin of the heatmap along with the time evolution of a physical quantity that signals the N-N bond-breaking. This facilitates a
Figure 3: Heatmaps of the  $K_{\alpha}$ -SchNet kernel applied to a part of the trajectory of  $D_{2}$  (where a reactive event occurs) and time series of the distance between nitrogen atoms over time  $t$ . The cases with  $\alpha = 0$  and  $\alpha = 1$  are reported on the left and right, respectively. Using spectral clustering with the two kernels as inputs we label each time-index with one of two classes, with the background color showing the class. Spectral clustering with the multi-weight kernel picks out the reactive event perfectly.
physical interpretation, as the two classes correspond to configurations containing the reactants (the  $\mathrm{N}_2$  molecule) and products (two N atoms) of the chemical reaction. Interestingly, the  $\alpha = 1$  case correlates more closely with the handpicked physical quantity, shown in the top panel. The reason for this is the fact that learning the weights independently gives more weight to chemical species that are under-represented, which typically correspond to adsorbed atoms in surfaces. This allows us to give more weight to the most important actors in catalytic applications.
In this work, we introduced an approach to model the potential energy surface of atomistic systems. Our method employs GNN representations trained on the large OC20 dataset along with kernel ridge regression, which we tested on two catalytic processes that are outside the distribution of the pre-training dataset. We devised a kernel function incorporating GNN features, blending kernel mean embedding with information related to the atom's chemical species. Our approach outperforms standalone GNN or kernel methods, demonstrating impressive transferability. This suggests promising avenues for transfer learning application in computational chemistry. However, we recognize certain limitations. Firstly, our method is based upon KRR which scales poorly to large scale datasets, although potential ways around this such as random features [70] or Nyström approximations [71] can overcome this limitation. Secondly, although we tested MEKRR on out-of-distribution datasets for representations pre-trained on OC20, we still focused our analysis on catalytic systems similar to those in the dataset. In this regard, it would be interesting to understand the extent to which MEKRR can predict well the chemical properties of generic systems. In addition, it would be important to incorporate forces into the loss function in order to use it in molecular dynamics applications. We believe that addressing these aspects will further improve the impact of this framework in computational chemistry.
CRediT author statement J. I. Falk: Conceptualization, Methodology, Software, Investigation, Formal analysis, Writing - Original Draft; L. Bonati: Conceptualization, Methodology, Formal analysis, Visualization, Writing - Original Draft; P. Novelli: Conceptualization, Methodology, Writing - Original Draft; M. Parrinello: Conceptualization, Writing - Review & Editing, Supervision; M. Pontil: Conceptualization, Methodology, Writing - Review & Editing, Supervision.
Acknowledgements We acknowledge the financial support from the PNRR MUR Project PE000013 CUP J53C22003010006 "Future Artificial Intelligence Research (FAIR)", funded by the European Union - NextGenerationEU, EU Project ELIAS under grant agreement No. 101120237, and the "Joint project TransHyDE_FP3: Reforming ammonia - transport of  $H_{2}$  via derivatives", funded from the German Federal Ministry of Research (BMBF), funding code: 03HY203A-F.
[63] Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic, 2018.
[64] Alessandro Laio and Michele Parrinello. Escaping free-energy minima. Proceedings of the national academy of sciences, 99(20):12562-12566, 2002.
[65] Graeme Henkelman, Blas P Uberuaga, and Hannes Jonsson. A climbing image nudged elastic band method for finding saddle points and minimum energy paths. The Journal of chemical physics, 113(22):9901-9904, 2000.
[66] Luigi Bonati, Daniela Polino, Cristina Pizzolitto, Pierdomenico Biasi, Rene Eckert, Stephan Reitmeier, Robert Schlögl, and Michele Parrinello. The role of dynamics in heterogeneous catalysis: Surface diffusivity and n2 decomposition on fe (111). Proceedings of the National Academy of Sciences, 120(50):e2313023120, 2023.
[67] Manyi Yang, Luigi Bonati, Daniela Polino, and Michele Parrinello. Using metadynamics to build neural network potentials for reactive events: the case of urea decomposition in water. Catalysis Today, 387:143-149, 2022.
[68] Karl Weiss, Taghi M. Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big Data, 3(1):9, 2016.
[69] Justin S. Smith, Benjamin T. Nebgen, Roman Zubatyuk, Nicholas Lubbers, Christian Devereux, Kipton Barros, Sergei Tretiak, Alexandr Isayev, and Adrian E. Roitberg. Approaching coupled cluster accuracy with a general-purpose neural network potential through transfer learning. Nature Communications, 10(1):2903, 2019.
[70] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Proceedings of the 20th International Conference on Neural Information Processing Systems, NIPS'07, page 1177-1184, Red Hook, NY, USA, 2007. Curran Associates Inc.
[71] Giacomo Meanti, Luigi Carratino, Lorenzo Rosasco, and Alessandro Rudi. Kernel methods through the roof: Handling billions of points efficiently. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.