A data-efficient foundation model for porous materials based on expert-guided supervised learning

Domain: physics
Published: 2026-02-11
Editorial outcome: major_revision

Abstract
Foundation models require massive amounts of training data, which are often costly to obtain in materials science. Meanwhile, long-established physical knowledge such as molecular force fields and geometric analysis provides direct guidance for material behavior, but remains insufficiently leveraged. Here we demonstrate that expert knowledge can directly supervise pre-training and substantially reduce data requirements. A set of potential energy surface (PES) basis functions, which encode guest-host interaction energetics, is developed as unified descriptors for different guest molecules. A multi-modal architecture is designed to fuse information from both material structure and PES. Pre-training is achieved by learning comprehensive geometric features spanning different spatial scales. Consequently, a foundation model for porous materials is developed under limited data regimes, named SpbNet. SpbNet is evaluated on over 50 downstream tasks, including adsorption, separation, and intrinsic properties, etc. SpbNet consistently outperforms models pre-trained on datasets nearly 20 times larger, reducing the relative errors by over 20%. In addition, SpbNet demonstrates strong generalization capabilities across both in-distribution and out-of-distribution materials, such as Metal Organic Frameworks, Covalent Organic Frameworks, and zeolites.

Introduction
Foundation models have demonstrated remarkable potential across diverse domains, including chemistry1, medicine2–4, and meteorology5. Trained on large-scale datasets, these parameter-rich architectures exhibit exceptional versatility in addressing a wide range of downstream tasks. Most importantly, compared with conventional machine learning approaches, foundation models show superior data efficiency, enabling rapid adaptation to downstream tasks with limited training samples6. In the field of material science, porous materials, such as metal-organic frameworks (MOFs), covalent organic frameworks (COFs), porous polymer networks (PPNs), and zeolites, possess high specific surface areas7 and tunable structural features8, conferring substantial agricultural and industrial value9–15. Therefore, developing a foundation model tailored for porous materials is of considerable scientific and practical importance.

However, pre-training a foundation model typically requires an extremely large amount of data, which is often prohibitively expensive or even impractical in material science. For instance, despite decades of research, fewer than 1000 experimental zeolite structures have been reported16. To address this challenge of data scarcity, several studies have incorporated domain expertise to enhance data efficiency, either through model architecture design17 or task formulation during training18. Nevertheless, in terms of constructing larger-scale pre-trained foundation models as well as improving data efficiency on pre-trained data, the potential of expert knowledge remains largely unexplored. Here, we present a framework for expert-guided pre-training to enhance data efficiency. The framework comprises three key components: (1) input descriptors19,20, which encode raw material structures using unbiased molecular force fields; (2) a dual-stream multi-modal architecture designed to jointly capture structural and potential energy surface (PES) information; and (3) pre-training tasks that enable the model to learn comprehensive, multi-scale geometric representations18,20. For the descriptors, in addition to the raw material structure, we introduce a set of PES basis functions. These basis functions can be linearly combined to represent the interaction energies between the material and various guest molecules, thereby mitigating bias toward specific adsorbates. We further design a dual-stream multi-modal model architecture comprising three key components: a graph neural network (GNN) that processes raw material structures, a vision transformer that encodes PES representations, and a cross-attention fusion module that integrates structural and energetic information. This dual-stream design effectively leverages the strengths of modern GNNs and the expressive power of PES basis functions. Finally, we construct multi-scale pre-training tasks covering probes of different sizes and material structures of varying granularity. We refer to the resulting foundation model as the Structure-informed Potential energy surface Basis function Network (SpbNet). SpbNet is pre-trained on 0.1 million MOF structures. For in-distribution downstream tasks, we evaluate SpbNet across more than 40 MOF-related benchmarks, including gas adsorption, separation, and intrinsic properties such as elastic modulus. For out-of-distribution (OOD) evaluations, we assess SpbNet on eight tasks involving COFs, PPNs, and zeolites. SpbNet consistently outperforms other pre-trained models trained on over 2 million porous materials20,21 and 120 million molecular and material samples22. These results highlight that the potential of incorporating expert knowledge in the construction of foundation models remains far from fully realized, particularly in enhancing data efficiency.

Results
The overall architecture and training pipeline of SpbNet are illustrated in Fig. 1. The design of SpbNet comprises four major components: (1) Descriptors, encompassing both the raw material structure and the proposed PES basis functions; (2) Model architecture, corresponding to the multi-modal input design; (3) Pre-training tasks, which facilitate multi-scale geometric feature learning; and (4) Fine-tuning process, which adopts a simple multi-head-based fine-tuning strategy.

Descriptors. Descriptors play a pivotal role in material informatics. Although recent studies have increasingly employed raw structural information (i.e., atomic numbers and positions) as input, expert-designed descriptors continue to demonstrate clear advantages in specific tasks19,20,23. In particular, PES are highly suitable for tasks involving guest molecules—the primary application domain of porous materials19,20. This suitability arises because PES directly characterizes the interaction energy between a material and a guest molecule, which governs the latter’s behavior. However, PES profiles differ substantially across guest species (see Supplementary Fig. 1 for details), rendering molecule-specific PES representations unsuitable as universal inputs for a foundation model. To construct a generalizable interaction energy descriptor, we propose a set of PES basis functions that describe the interaction energy between porous materials and various guest molecules. The potential energy of any guest molecule \phi at position r can be expressed as a linear combination of 21 basis function terms with molecule-specific coefficients Coe( \phi ), comprising 13 terms of Pauli exchange-repulsion energy, 7 terms of London dispersion energy, and 1 term of Coulombic energy:

E \phi (r) = \sum_{i=1}^{13} E_{pauli}^i (r) Coe_i^{pauli} ( \phi ) + \sum_{i=1}^{7} E_{london}^i (r) Coe_i^{london} ( \phi ) + E_{coulomb} (r) Coe_{coulomb} ( \phi ) (1)

Detailed formulations and derivations are provided in Section “Derivation of potential energy surface basis functions”. The 21 terms (E_{pauli}, E_{london}, and E_{coulomb}) serve as the input features of SpbNet. These 21 basis functions are evaluated at predefined sampling points across the lattice, resulting in a 21-channel three-dimensional image that spans the entire periodic cell. The 21 channels are subsequently linearly combined through a linear layer to generate a single-channel image representing a universal PES, which can then be processed using vision-based methods. Further implementation details are described in Sections “Representing PES basis functions via multi-channel 3D image” and “Linearly combining PES basis functions via residual connection”. Importantly, this set of basis functions is free from bias toward any specific guest molecule, making it a suitable and transferable input representation for a universal foundation model.

Model architecture. Model architecture represents another critical component of a foundation model. It must be capable of capturing essential structural features from the input, such as bond lengths, bond angles, and other local geometrical descriptors. Given the multi-modal nature of SpbNet’s inputs, the architecture must effectively integrate heterogeneous representations. As illustrated in Fig. 1, the main architecture of SpbNet consists of a GNN for processing material structures24 and a vision transformer25 for encoding the linearly combined PES. To fuse information from the structural and energetic modalities, we employ a cross-attention mechanism commonly adopted in multi-modal architectures26,27. Following the design in ref. 25, an additional learnable [CLS] token is appended to the PES patch sequence. The feature extracted from the [CLS] token is subsequently passed through a prediction head (see Section “Training strategy” for details) to infer the target property.

To enable simultaneous prediction of multiple material properties during pre-training, we construct separate task-specific heads (multi-head design). During fine-tuning, only the backbone parameters are retained, while all pre-training heads are discarded and replaced with randomly initialized heads for downstream task. Compared with single-stream architectures, where the same layers and weights are used to process features from different modalities, the dual-stream design allows distinct sets of layers (self-attention and cross-attention layers28) to specialize in modality-specific representations26,27. This separation not only preserves the diversity of structural and energetic features but also facilitates flexible substitution of the structural encoder, thereby leveraging advances in modern GNN architectures.

Pre-training. Pre-training strategy constitutes one of the most critical components of a foundation model. In this work, we focus on training the network to learn well-established geometric analysis of porous materials. A series of multi-scale pre-training tasks are designed. At the material level, the model is pre-trained to predict (1) topology type (e.g., pcu, fcu, etc. The model is trained to classify the material into a specific topology in the pre-defined topology type set20), (2) pore limiting diameter (PLD), (3) largest cavity diameter (LCD), (4) accessible surface area (ASA), and (5) accessible volume fraction (VF). To mitigate bias towards specific guest molecules, ASA and VF are computed using probes of varying sizes29. At the atomic level, the model is trained to predict (1) the number of atoms within each PES patch, and (2) the signed distance function (SDF) at the patch center. The SDF is also computed with probes of different sizes. Calculation details and the differences in geometric features obtained using various probe sizes are provided in Supplementary Text 1 and Supplementary Fig. 2. These pre-training tasks comprehensively leverage the accumulated domain knowledge of geometric characterization in porous materials, thereby providing a robust and domain-informed pre-training process. The correlation between these geometric features and downstream tasks can be found in Supplementary Figs. 3–9.

Pre-training results. Figure 2 presents the pre-training performance of SpbNet. As shown, for macroscopic geometric features such as VF and LCD, SpbNet achieves an R2 score of 0.9 and a Pearson correlation coefficient of 0.95. When the probe size decreases from 2.0 to 0.5, the accuracy of VF remains largely unchanged, whereas the accuracy of ASA decreases. This observation is consistent with Supplementary Fig. 7, which shows that correlations among ASAs computed using different probe sizes are more complex than those of VF. The visualization of the topology type classification task can be found in Supplementary Fig. 10, with the accuracy of 94.9%.

For the fine-grained SDF and atom-number prediction tasks, the t-SNE visualizations are shown in Fig. 2c, d. The PES patch embeddings exhibit distinct trends as the SDF and atom numbers vary, suggesting that the model effectively learns to predict both properties simultaneously. The R2 scores for atom-number and SDF prediction tasks are 0.911 and 0.957, respectively (see Supplementary Fig. 11 for visualization). As shown in Fig. 2c, most PES patches contain only a few atoms, reflecting SpbNet’s high spatial resolution in decomposing PES into fine patches. In Fig. 2d, most patch centers display positive SDF values, indicating that the porous regions occupy the majority of the MOF volume.

Collectively, these findings demonstrate that SpbNet effectively learns well-defined geometric features through pre-training. The multi-head architecture further allows the model to capture diverse types of features simultaneously without mutual interference, even when identical properties are calculated by different scales of probes. Through such comprehensive pre-training objectives, SpbNet develops generalizable geometric representations that are transferable to downstream applications such as adsorption, facilitating rapid and data-efficient adaptation. Although predicting these properties is relatively straightforward, the model does not inherently acquire such intermediate features when trained solely on downstream task labels. Incorporating these properties as auxiliary objectives in multi-task training also enhances downstream accuracy (Supplementary Table 2), underscoring their value in guiding the model toward more informative structural representations.

Performance on in-distribution metal-organic-frameworks. Gas adsorption stands as one of the most critical application scenarios for porous materials9,30. We conduct experiments on two representative datasets: hMOF31–34 (comprising ~130,000 hypothetical MOFs) and CoREMOF35,36 (containing ~14,000 experimentally synthesized MOFs). The Mean Absolute Error (MAE) is adopted as the performance metric. SpbNet is compared with CGCNN37, MOFormer38, and MOFTransformer20,21. Detailed architectures of the baseline models can be found in Supplementary Text 2. Beyond typical industrial gases such as CO2 and CH4, we also incorporate (1) noble gases (including Kr, Xe) and (2) other hydrocarbons (including propylene, propyne, and para-xylene (p-xylene)).

SpbNet demonstrates superior performance across different gases (see Fig. 3a and Supplementary Data 1). Specifically, for CO2 adsorption, SpbNet achieves a substantial reduction in MAE by 29.1%. Notably, even when predicting CH4 adsorption (the probe used to calculate the PES), SpbNet still maintains a remarkable improvement, reducing the MAE from 0.248 to 0.175, an improvement of 29.4%. These results unequivocally demonstrate that the enhanced generalization capabilities do not compromise accuracy for specific guest molecules. For other hydrocarbons, SpbNet reduces the relative error for C3H6 and p-xylene by 23.9% and 9.6%, respectively (as shown in the right panel of Fig. 3a). Furthermore, SpbNet exhibits promising performance for noble gases such as Kr and Xe, with MAE decreases of 11.0% and 17.8%, respectively. Additional quantitative results are provided in Supplementary Data 1.

Beyond gas adsorption, SpbNet is versatile and applicable to three distinct categories of tasks: (1) multi-guest molecule-related tasks (e.g., gas separation); (2) single-guest molecule-related tasks (e.g., adsorption heat and Henry’s constant); and (3) material intrinsic properties (e.g., thermal decomposition temperature).

For gas separation, as illustrated in Fig. 3b, SpbNet significantly improves accuracy for CH4/N2 and N2/O2 separation, reducing the MAE by 32.3% and 21.0%, respectively. Even for chemically similar gases like propane and propylene, SpbNet achieves an 8.03% improvement. For single-guest molecule-related tasks (Fig. 3c, d), we assessed: (1) heat of adsorption; (2) Henry’s constant; and (3) gas diffusion. In the context of CO2 and CH4 adsorption heat, SpbNet reduces the relative error by 0.75% and 20.9%. Regarding the Henry’s constant for propylene and CH4, SpbNet demonstrates substantial enhancements, reducing the MAE by 17.2% and 41.0%. Additionally, we evaluated SpbNet’s performance in gas diffusion. Further quantitative results are detailed in Supplementary Data 1.

Aside from guest molecule-related properties, SpbNet is also capable of predicting intrinsic material properties (Fig. 3e). Four different intrinsic properties are examined: (1) mechanical stability (including elastic modulus and shear modulus); (2) thermal stability; and (3) heat capacity. For mechanical properties, tasks are conducted to predict the bulk elastic modulus (K_{VRH}) and shear modulus (G_{VRH}), where VRH represents Voigt–Reuss–Hill techniques39. SpbNet achieves higher accuracy, reducing the MAE by 11.1% and 13.7%, respectively. Regarding thermally relevant properties, SpbNet demonstrates comparable performance in the prediction of thermal decomposition temperature, reducing the MAE by 2.34%. Although the heat capacity data is sparse, consisting of only a few hundred data points, the four compared models perform well, maintaining an R2 score above 0.7 (see Supplementary Data 1).

Furthermore, we compare SpbNet with other universal models designed for molecules and general materials in Supplementary Table 1 and Supplementary Text 3. Notably, even when benchmarked against modern foundation models pre-trained on 209 million data points40 and 120 million data points22, SpbNet achieves the best performance. Nevertheless, it must be emphasized that SpbNet is primarily trained to capture the macroscopic geometric structure of the material, which is closely related to the behavior of guest molecules within the material pores. Consequently, SpbNet primarily leverages its structure encoder to infer other properties, such as electronic structure, which may not be its main focus.

Out-of-distribution generalization. Except for performance on in-distribution downstream datasets, another critical evaluation metric for a foundation model is its transferability to OOD datasets. In this section, we evaluate SpbNet’s performance on other porous material systems, specifically COFs, PPNs, and zeolites. The results are presented in Fig. 4. As shown in Fig. 4a, b, although the model has never encountered COF or zeolite structures during the pre-training stage, SpbNet accurately predicts the geometric properties of both material classes. This confirms that the pre-trained SpbNet possesses inherent generalization capability across different porous systems. The model exhibits substantially better generalization on COFs than on zeolites, owing to the closer structural similarity between MOFs and COFs (see Supplementary Figs. 12 and 13 for the visualization of the features). Interestingly, despite MOFs and zeolites showing comparable statistical properties, such as pore volume and atomic density (Supplementary Fig. 14), the model generalizes poorly to zeolites. This is primarily because zeolites are composed predominantly of Si, O, and Al atoms, leading to distinct chemical environments. In contrast, the model retains strong generalization on COFs even when there exist notable differences from MOFs in both atomic density and pore volume.

Further OOD evaluation is conducted on downstream tasks. For two gas adsorption tasks (CH4, 298K, 65 bar and 5.8 bar) on COFs41 (Fig. 4c left), SpbNet reduces the MAE by 27.6% and 34.4%, respectively. For PPNs42 (CH4, 298K, 65 bar and 1 bar) (Fig. 4d), the improvements reach 32.5% and 74.7%. Furthermore, for COFs’ adsorption heat and Henry’s constant43 (CO2, 298K), SpbNet reduces the MAE by 17.7% and 13.2%. On the zeolite dataset44 (CH4, 298K), the corresponding improvements reach 39.7% and 40.8% (Fig. 4e). These findings demonstrate that transferring SpbNet to new porous material systems does not lead to a significant degradation of its accuracy advantage, confirming the generalization ability of the pre-trained model to support fine-tuning of downstream tasks across diverse material classes.

Label efficiency. Foundation models possess broad application prospects due to their ability to achieve strong performance with minimal training data, significantly expanding their application scope compared to models trained from scratch. We evaluate the label efficiency of SpbNet on the gas adsorption and diffusion tasks, varying the number of training samples from 5000 to 20,000. The results are illustrated in Fig. 5. With the same pre-training data volume (100K data), SpbNet consistently outperforms MOFTransformer20,21. Furthermore, even when compared to MOFTransformer pre-trained on 1.9 million data (~19 times the amount used for SpbNet), SpbNet still maintains superior performance on the gas adsorption task. Only for the gas diffusion task, when MOFTransformer is fine-tuned on approximately three times the amount of downstream task data (15K data), the performance of SpbNet and MOFTransformer becomes comparable. For gas adsorption, both SpbNet and MOFTransformer significantly benefit from increasing the size of the training dataset. However, for gas diffusion, the effect of increasing the data volume is less pronounced, indicating that diffusion is an inherently more challenging task than adsorption. These results underscore the high data efficiency of SpbNet. SpbNet not only obtains sufficient pre-training capabilities from a relatively small dataset (100K) but also demonstrates rapid adaptability to downstream tasks.

Discussion. To fundamentally understand the underlying mechanism of the pre-training tasks, we performed ablation studies. We considered four different versions of SpbNet: (1) Trained from scratch (i.e., without any pre-training process and with randomly initialized weights). (2) Pre-training utilizing only global geometric features. (3) Pre-training using global geometric features and the atom number prediction task. (4) Pre-training using global geometric features, the atom number prediction task, and the SDF prediction task.

Feature space analysis. We first evaluate the feature space extracted by SpbNet. Figure 6a demonstrates the averaged cosine similarity between the global [CLS] token and other local PES tokens. When the model is trained from scratch, the feature of the [CLS] token rapidly aligns with other tokens as the network deepens. This is primarily due to the absence of a token-scale loss constraint, where the inherent low-pass filtering property of the attention mechanism causes the features of all tokens to become overly similar, a phenomenon known as over-smoothing45. In contrast, when fine-tuned from pre-trained weights, the average similarity remains at a relatively low level, indicating that each energy patch maintains more localized information compared to the global feature. We further discovered that one crucial factor controlling this effect is the size of the pre-training data (see Supplementary Fig. 15 for the comparison between large and small pre-training dataset). When the data volume is sufficiently large, even with only material-level supervision signals, each token can still retain local information; conversely, over-smoothing is prone to occur when the data volume is small.

These results indicate that the behavior of the [CLS] token in fine-tuning and training-from-scratch is essentially different. After the model is pre-trained, the model already learns to compress the structure information into a low-rank feature space. In this fine-tuning phase, the [CLS] token learns to aggregate this information to make prediction. When the model is trained from scratch, the lower layers still maintain local information of each token. However, the global feature of the [CLS] token quickly spreads to other tokens and erases almost all local information of PES tokens. At this point, the network degenerates to a simple MLP.

Effect of task combination on feature rank. To observe the impact of different pre-training tasks, Fig. 6b illustrates the effective ranks under various pre-training task combinations. Similar to the Principal Component Analysis method, the effective rank quantifies the actual dimensionality that significantly contributes to the high-dimensional feature space46. From the lower to the higher layers, the effective rank gradually decreases, suggesting that the feature of each PES token is compressed into a low-dimensional semantic space. During the model’s fine-tuning process, these extracted semantics are leveraged for making robust predictions. Both the atom number prediction task and the SDF prediction task enhance the effective rank throughout this process, enabling the model to utilize more layers to extract richer local information. The local information is also maintained after fine-tuning, as evidenced by the linear probing experiments shown in Supplementary Figs. 16 and 17.

Task-specific performance. Figure 6c illustrates the ablation study results on six selected downstream tasks. The inclusion of token-level pre-training tasks yields performance improvements. Notably, the SDF prediction task is particularly beneficial for gas separation, which may be attributed to its role in capturing the molecular sieving effect. The detailed surface shape information provided by the SDF assists the model in accurately determining whether a guest molecule can easily pass through or be retained by the material’s pores. Conversely, the atom number prediction task produced a certain negative effect on the zeolite tasks. This negative transfer may be due to the significant structural differences between zeolites and MOFs, as illustrated by the t-SNE plots of different materials in Supplementary Figs. 12 and 13. These combined results decisively demonstrate the effectiveness and mechanism of the atom number and SDF prediction tasks in enhancing SpbNet’s representation learning capability. In addition, removing basis functions also reduces the model accuracy. The effect of basis functions is relatively limited on CH4-related tasks, which can be attributed to that the CH4 molecule is adopted as the probe to calculate the PES without basis functions.

Limitations. In this work, we present SpbNet, an expertise-guided supervised pre-training framework for foundation models of porous materials. It achieves high data efficiency and strong performance across a range of guest-molecule and geometry-related downstream tasks. The model leverages unbiased descriptors derived from molecular force fields and adopts a supervised pre-training paradigm centered on geometric representations. Crucially, SpbNet integrates a broad and physically grounded set of pre-training tasks designed to improve generalization beyond any single downstream objective. This approach is particularly suited to porous materials. The key functions of porous materials, such as adsorption, separation, and storage, are governed by host-guest interactions dictated by pore geometry. SpbNet combines three components within a unified framework: the PES as an input descriptor that encodes guest-host interaction energetics; a multi-modal architecture that couples PES-based and structural representations; and a geometry-oriented pre-training objective targeting features correlated with macroscopic metrics like adsorption capacity and separation ratio. Together, these components provide both local energetic and global geometric perspectives, forming a comprehensive pre-training scheme that enhances generalization and data efficiency under limited data availability.

Nevertheless, the strong task specificity of this supervised strategy may constrain its generalization compared with broader self-supervised learning (SSL) approaches. For tasks in which the underlying physical principles differ from the pre-training tasks, the benefits of PES-based features, such as electronic structure, may diminish. In this case, the model instead relies on latent representations from its structural encoder and the adaptability of its multi-modal design. Overall, these findings suggest that under data-scarce conditions, carefully designed input descriptors, tailored architectures, and broad supervised pre-training tasks explicitly aligned with downstream objectives can substantially enhance generalization and achieve superior data efficiency compared with mainstream unsupervised approaches.

Methods
Derivation of potential energy surface basis functions
The first crucial step in our approach is to establish a universal PES representation suitable for diverse guest molecules. A feasible solution is to define a set of PES basis functions, {E_i(r)}, based on the position vector r. The potential energy E of any guest molecule \phi at position r can then be expressed as a linear combination of these basis functions:

E(r, \phi) = \sum_i E_i(r) Coe_i( \phi ) (2)

Here, E_i(r) are the basis functions, and Coe_i( \phi ) represents the coefficient unique to the guest molecule \phi. For simplicity, we initially consider the case of a single atomic probe. Additionally, we focus only on the major non-bonded interactions.

Molecular force field and PES components. We assume that the non-bonded interaction potential energy primarily consists of three components: the Pauli exchange mutual repulsion potential (E_{pauli}), the London dispersion potential (E_{london}), and the Coulombic potential (E_{coulomb}). The potential energy between a material atom i and a guest atom j at distance r_{ij} is defined as:

Pauli: E_{pauli}(i,j) = 4\epsilon_{ij} (\sigma_{ij} / r_{ij})^{12}
London: E_{london}(i,j) = -4\epsilon_{ij} (\sigma_{ij} / r_{ij})^6 (3)
Coulomb: E_{coulomb}(i,j) = k_e q_i q_j / r_{ij}

The parameters \epsilon and \sigma denote the well depth and the van der Waals radius, respectively, which depend on the specific (pseudo) atoms. q represents the partial charge, and k_e = 1 / (4\pi\epsilon_0) is the electrostatic force constant.

For any atom in the material, V(r_{ij}) = E_{pauli}(i,j) + E_{london}(i,j), simplifies to:

V(r_{ij}) = 4\epsilon_{ij} [(\sigma_{ij} / r_{ij})^{12} - (\sigma_{ij} / r_{ij})^6] (4)

The cross-parameters \sigma_{ij} and \epsilon_{ij} are calculated using the Lorentz–Berthelot mixing rules:

\sigma_{ij} = (\sigma_i + \sigma_j) / 2
\epsilon_{ij} = \sqrt{\epsilon_i \epsilon_j} (5)

Let j be the reference probe used for calculation and \phi be the new probe atom in the actual task. Assuming the distance r_{i\phi} = r_{ij} = r, and substituting the mixing rules into the LJ potential for both probes:

V_\phi(r) = 4 \sqrt{\epsilon_i \epsilon_\phi} [(\sigma_i + \sigma_\phi) / (2r)]^{12} - 4 \sqrt{\epsilon_i \epsilon_\phi} [(\sigma_i + \sigma_\phi) / (2r)]^6 (6)

V_j(r) = 4 \sqrt{\epsilon_i \epsilon_j} [(\sigma_i + \sigma_j) / (2r)]^{12} - 4 \sqrt{\epsilon_i \epsilon_j} [(\sigma_i + \sigma_j) / (2r)]^6 (7)

If we were to calculate the ratio V_\phi / V_j, the result would be a complex function f(\sigma_\phi, \sigma_i, r) dependent on the distance r and the specific material atom i. Since the total potential energy for a material M is the summation \sum_{i \in M} V, the complex dependence of V_\phi / V_j on i and r prevents a simple linear correlation between different probes (\sum V_\phi and \sum V_j). This necessitates a decomposition into universal basis functions.

Construction of universal basis functions. Our goal is to find a set of basis functions E such that the ratio of the total potential energy of two probes satisfies \forall \phi, j: E_\phi / E_j = g( \phi, j), where the coefficient g( \phi, j) is independent of the material atom \sigma_i and the position r. This proportionality must hold even after summing over all atoms in the material structure (\sum E_\phi / \sum E_j = g( \phi, j)), which is possible only if E_\phi / E_j is independent of i and r.

The required basis functions can be constructed by applying the binomial theorem to the Pauli repulsion and London dispersion terms, leveraging the Lorentz–Berthelot mixing rules. We analyze the London dispersion potential L as an example (r_{i\phi} = r_{ij} = r):

L_{i\phi} = -4 \sqrt{\epsilon_i \epsilon_\phi} [(\sigma_i + \sigma_\phi) / (2r)]^6 (8)

The term inside the first set of parentheses depends on the material atom i and position r, defining the basis function. The term inside the second set of parentheses depends only on the probes \phi and j, defining the coefficient. Since the coefficient is independent of i and r, we can sum over all atoms i \in M, and the total potential energy remains a linear combination with the same coefficients.

Thus, for a specified exponent k_l, the London basis function (summed over material atoms) is:

E_{london}^{k_l}(r) = \sum_{i \in M} -4 \sqrt{\epsilon_i \epsilon_j} (1 / (2r))^6 C_l^{k_l} (\sigma_i)^{6-k_l} (\sigma_j)^{k_l} (10)

The Pauli repulsion term is derived similarly with an exponent of 12, and the form of the electrostatic potential (which is E_{coulomb} \propto r^{-1} for a single charge q_j = 1) is already a simple basis. Using the UFF molecular force field parameters47, the complete set of PES basis functions is:

E_{pauli}^{k_p}(r) = \sum_{i \in M} 4 \sqrt{\epsilon_i \epsilon_j} (1 / (2r))^{12} C_p^{k_p} (\sigma_i)^{12-k_p} (\sigma_j)^{k_p}, 0 \le k_p \le 12 (11)

E_{london}^{k_l}(r) = \sum_{i \in M} -4 \sqrt{\epsilon_i \epsilon_j} (1 / (2r))^6 C_l^{k_l} (\sigma_i)^{6-k_l} (\sigma_j)^{k_l}, 0 \le k_l \le 6 (12)

E_{coulomb}^{k_q}(r) = \sum_{i \in M} (q_i / r_{ij}) (13)

where \sigma_\phi = \sigma_j + \Delta\sigma. Any (pseudo) atom j can serve as the reference probe, yielding a universally applicable set of basis functions. In our implementation, we specifically use the methane (a pseudo-atom representing methane) probe to calculate these basis functions. It should be noted that these functions describe the PES of a single (pseudo) atom. Further refinement for more complex guest molecules containing multiple atoms is reserved for future work.

Additionally, considering that many common guest molecules (e.g., CH4, N2, H2) are non-polar, their interactions are less affected by the Coulomb term. Introducing partial charges can potentially introduce a bias against specific guest molecules during the pre-training stage. Therefore, the partial charges and the corresponding E_{coulomb} potential are not considered in practice, leaving a total of 13 + 7 = 20 basis terms.

Representing PES basis functions via multi-channel 3D image. For a specific material, the crystal structure is first expanded into a supercell to ensure that the minimum length of the lattice vector exceeds 8 \AA. Coordinates are then uniformly sampled within the supercell to construct a 24 \times 24 \times 24 PES grid (13,824 sampling points in total). For each coordinate r, the corresponding n terms of the PES basis functions are computed based on the formulas in Section “Derivation of potential energy surface basis functions”. The resulting data structure is a tensor of shape 24 \times 24 \times 24 \times n, where n is the number basis functions. This data structure is naturally interpreted as a three-dimensional (3D) n-channel image with a resolution of 24 \times 24 \times 24, where each sampling point serves as a voxel and each of the basis function terms corresponds to a distinct channel.

Linearly combining PES basis functions via residual connection. In Eq. (9), while a direct polynomial expansion of (\sigma_i + \sigma_\phi)^6 is mathematically feasible, we note the potential benefit of employing the difference substitution \sigma_\phi = \sigma_j + \Delta\sigma. The term in the basis expansion that is independent of \Delta\sigma is the one corresponding to the original London dispersion potential energy of the reference probe j (similarly for the Pauli term). This decomposition allows the final PES to be conceptually viewed as the original potential energy of probe j (specifically, j = CH4 is used to calculate the basis functions) plus a series of correction terms. This is analogous to the residual connection paradigm in deep learning48, which is known to enhance training stability and facilitate the training of deeper or more complex networks. SpbNet adopts this practice by fixing the coefficient of the reference PES (probe j) to 1, while allowing the coefficients of the other basis functions to be learnable.

Consequently, the PES basis functions are categorized into: (1) Original PES Term: A single-channel 3D image, shaped 24 \times 24 \times 24 \times 1, which represent the PES of the reference probe j (CH4) and acts as the identity mapping. (2) Correction Terms: Composed of the remaining 18 channels (Pauli and London terms excluding the reference term), shaped 24 \times 24 \times 24 \times 18. These correction terms are transformed into a single channel via a learnable linear layer (with weight shape (18, 1) and no bias) and subsequently added to the Original PES Term to obtain the final universal PES. This specific implementation of the residual PES was found to be critical for performance. Direct implementation where all basis function coefficients (including that of the original CH4 PES) were learnable resulted in significant performance loss, performing even worse than simply using the original CH4 PES without any basis function expansion.

Model architecture. The architecture of SpbNet is designed around a cross-attention mechanism28 integrated with a structure encoder. Cross-attention is widely used in multi-modal learning to enable effective fusion of heterogeneous information26,27. In SpbNet, this mechanism facilitates the interaction between structural features of porous materials and the PES representations.

SpbNet consists of two main components: a structure encoder and a PES decoder. For the structure encoder, we adopt the PaiNN architecture24 to generate atom-level embeddings following the implementation in ref. 20. All hyperparameters remain consistent with the original PaiNN settings. Since each atom possesses an individual feature vector, the representation of an entire structure can be viewed as an unordered set of atomic embeddings.

For the PES decoder, we employ a Vision Transformer (ViT)25. The PES grid of size 24 \times 24 \times 24 \times 1 (see Section “Linearly combining PES basis functions via residual connection”) is divided into cubic patches with a patch size of 3, yielding (24/3)^3 = 512 patches in total. Each 3 \times 3 \times 3 \times 1 patch is treated as a token, transforming the PES into an ordered sequence of patch tokens. This token sequence is processed by a 36-layer Transformer decoder28, where cross-attention layers inject the structural embeddings from the encoder into the PES token stream. The resulting feature of each patch is used to predict atomic quantities within that region.

A learnable [CLS] token49 is prepended to the PES token sequence, and its final feature is passed through two linear layers followed by a tanh activation to predict both pre-training objectives (e.g., void fraction and topology) and downstream properties (e.g., gas adsorption). The hidden feature dimension is set to 512, and the model employs eight attention heads.

Training strategy. We follow the standard training paradigm commonly adopted in Transformer-based models. During the pre-training stage, the framework consists of (1) a linear layer that combines the PES basis functions (see Section “Linearly combining PES basis functions via residual connection”), (2) the structure encoder, and (3) the PES Transformer decoder as the backbone, with a single linear layer serving as the prediction head. An additional [CLS] token is appended to the sequence of PES patch tokens. The extracted feature of the [CLS] token is subsequently processed by a pooling layer composed of a single linear layer followed by a tanh activation function. Each pre-training task is associated with an independent prediction head, implemented as a single linear layer to predict the corresponding target property (e.g., topology type, PLD, LCD, etc.). All trainable components share the same training schedule. The learning rate increases from 10^{-5} to 10^{-4} over the first 15 epochs and then decays to 10^{-5} following a cosine schedule. The AdamW optimizer is used with a weight decay of 0.25. The model is pre-trained for 40 epochs, and the checkpoint with the lowest validation loss is selected for fine-tuning.

In the fine-tuning phase, all backbone weights are retained, while the prediction head (including the pooling layer and task-specific linear layer) is replaced. The pooling layer is re-initialized, and a new downstream task-specific head is added. The same learning rate schedule is used as in the pre-training phase: the learning rate increases linearly from 0 to 10^{-4} during the first 2 epochs and then decays to 0 following the cosine function. AdamW is again employed as the optimizer, with the weight decay set to 0.1. The model is fine-tuned for 60 epochs, and the checkpoint achieving the lowest validation loss is used for inference. Notably, a smaller weight decay is adopted in fine-tuning, as it is empirically found to yield better performance.

Datasets. SpbNet uses different datasets as benchmarks. Except for bandgap, thermal stability, heat capacity, mechanical properties and datasets for COFs, zeolites and PPNs, other benchmarks use three public datasets: (1) hMOF31–34, (2) CoREMOF36, and (3) Tobacco50. hMOF31–34 is a commonly used dataset, containing over 130,000 hypothetical MOF materials. The hMOF dataset includes data on the adsorption capacity of seven gases (CO2, CH4, N2, H2, Kr and Xe) and adsorption heat data of four gases (CO2, CH4, N2, H2). It should be noted that the structural topologies in the hMOF dataset are relatively limited. Another, more challenging and reliable dataset is CoREMOF35,36. The CoREMOF dataset includes about 14,000 MOF structures, coming from experimental data, Cambridge Structural Database, and Web of Science searches. Thus, the CoREMOF dataset mainly includes experimental structures (instead of hypothetical structures as in hMOF and Tobacco). Compared with hMOF, CoREMOF has richer topologies and more complex structures. A large part of the experiments in this paper uses the CoREMOF dataset, such as adsorption, separation, etc. (see Supplementary Data 2). In addition, Tobacco50 is also a hypothetical dataset. Unlike hMOF, Tobacco uses a topology-guided method, thus the topologies are richer. The mechanical stability benchmark39 uses a part of the Tobacco dataset. In addition to these datasets, there are also some datasets used for specific works, such as heat capacity. For all datasets, we try to select ~5000 for training and 1000 for test and validation. For smaller datasets with less than 7000 samples, the training-validation-test data are randomly split into 8:1:1. All datasets, sources, training data volume, and tasks are listed in Supplementary Data 2. For pre-training, SpbNet is pre-trained on 95,370 structures, which are randomly sampled from the data provided by MOFTransformer20.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_2": "The revised manuscript has improved significantly in terms of clarity. In particular, the authors have now clearly outlined how the individual components of their proposed model interact, and they have updated Figure 1 as previously requested to better illustrate the model architecture and workflow. These updates make the manuscript more accessible and easier to follow.\n\nThe model, now named CT-FFKL, uses a transformer architecture that is pre-trained on a theoretical potential energy surface (PES), with graph neural network (GNN) embeddings (e.g., from CGCNN) as input features. During the pre-training phase, the model is trained on three prediction tasks: number of atoms per energy patch, void fraction, and a \"topological prediction\" task. However, these tasks appear relatively trivial given that the full MOF structure is already provided as input. As such, it remains unclear how much meaningful generalization or representation learning occurs during pre-training.\n\nThe manuscript explores multiple variations of the FFKL model, including versions using SchNet instead of CGCNN. However, SchNet is an older architecture originally developed for molecular systems and may not be well-suited for MOFs. The rationale for including it is not entirely clear, although this variation does not appear central to the model’s overall performance or conclusions.\n\nRegarding the authors' response to my earlier review, I would like to clarify a key point. In their rebuttal, the authors write:\n“The reviewer criticizes that our model should not be called a ‘foundation model’ because it needs to be fine-tuned for specific tasks rather than being directly applied.”\nThis is a misinterpretation of my original comment. My concern was not about the need for fine-tuning per se, but rather that the original manuscript did not sufficiently describe the model’s structure or justify its designation as a foundation model. Now that the structure is more clearly presented, one can assess this claim more meaningfully.\n\nThat said, the designation of CT-FFKL as a foundation model remains questionable. To qualify as such, it should be demonstrated that the model benefits substantially from pre-training and can generalize across multiple downstream tasks with minimal task-specific tuning. Currently, the manuscript does not provide strong evidence in this regard. The pre-training tasks, as mentioned, seem relatively easy given the model inputs, and no detailed analysis is presented to show how pre-training improves downstream performance. Additionally, the manuscript lacks a description of any specific fine-tuning strategies (e.g., LoRA, multi-head, etc.) commonly used in other foundation models such as MACE-MP-0 or large language models.\n\nFurthermore, it is unclear why large datasets such as hMOF (130k structures) are not used during pre-training rather than fine-tuning. If the goal is to create a transferable representation, training on the broadest dataset available would seem more appropriate.\n\nIn summary, the results look very promising. However, for such a high-profile journal, I believe these points should be addressed.\n\nMajor Comments\n- Pre-training Tasks: The manuscript does not clearly describe the nature of the \"topological prediction\" task. What precisely is being predicted, and how does this relate to generalizable chemical or structural features?\n- Fine-tuning Procedure: The manuscript outlines fine-tuning for MOFormer but not for CT-FFKL. How is fine-tuning implemented in practice? Are techniques such as weight decay, early stopping, or learning rate scheduling used? How is catastrophic forgetting mitigated?\n- Pre-training Relevance: The manuscript should include experiments or ablations demonstrating whether and how pre-training benefits downstream performance. This is critical if the authors wish to frame CT-FFKL as a foundation model.\n\nMinor Comments\n- The Methods section contains a number of spelling and grammatical errors that should be carefully corrected.\n\n(Remarks on code availability)\nThe github repository seems well structured and the code usable."}, "author_response": "Point-to-Point Response to Reviews\nResponses to Reviewer #1\nThe revised manuscript has improved significantly in terms of clarity. In particular,\nthe authors have now clearly outlined how the individual components of their\nproposed model interact, and they have updated Figure 1 as previously requested to\nbetter illustrate the model architecture and workflow. These updates make the\nmanuscript more accessible and easier to follow.\nResponse: We sincerely thank the reviewer very much for comments and useful\nsuggestions. In the revised manuscript, the work is further split into four parts:\ndescriptor, model architecture, pre-training, and fine-tuning. Fig. 1 is modified to\nclarify the training pipeline. Based on the original model, we optimized the architecture,\nincreased the parameter scale and added additional pre-training tasks. These\nmodifications further enhance the model’s performance and improve pre-training data\nefficiency. In addition, additional experiments are conducted to illustrate how pre-\ntraining works, how fine-tuned models essentially differ from models trained from\nscratch, and the effectiveness of different pre-training tasks in this process.\nSpecifically, the input descriptors remain the same as before. For the model\narchitecture, the structure encoder has been simplified to a Graph Neural Network\n(GNN) without the following Transformer architecture. The model architecture of the\nPES part, which is a Vision Transformer decoder, has been enlarged to 36 layers from\nthe previous 6 layers. For the pre-training process, besides the previous tasks of\ntopology type classification, void fraction prediction, and atom number prediction, we\nhave added: (1) Other global geometric feature prediction task, including prediction of\nPore Limited Diameter (PLD), Largest Cavity Diameter (LCD), and Accessible Surface\nArea (ASA); (2) The Signed Distance Function (SDF) prediction.\nCorrespondingly, we have updated the Figure 1.\nDescriptors. SpbNet adopts raw material structure and PES basis functions as input.\nPES basis functions are regarded as a multi-channel three-dimensional (3D) image and\nlinearly combined through a linear layer to obtain a single-channel 3D image. (2) Model\narchitecture. A GNN extracts material structure features from the raw material structure.\n“i, j, k” represent the atoms in the material. The combined PES representation is\nprocessed with a Vision Transformer (ViT). Structural features are incorporated into the\nViT via the cross-attention mechanism. (3) Pre-training. The model undergoes pre-\ntraining with two objectives: prediction of global geometric features and prediction of\natom-level properties including atom number and SDF. A [CLS] token is appended to\nthe PES patches to predict the global features via dedicated prediction heads. Features\nof PES patches are used to predict SDF and number of atoms through separate output\nheads. (4) Fine-tuning. After pre-training, the model retains the backbone weights while\ndiscarding all pre-training heads. A new task-specific prediction head is added to predict\nthe target property. Downstream tasks cover guest molecule-related properties, intrinsic\nmaterial properties, and generalization to out-of-distribution material systems . All trainable\nparts are marked in blue background.\nAdditionally, we have added the following description in the revised manuscript.\n(Section 2.1, paragraph 1, page 2)\nThe overall architecture and training pipeline of SpbNet are illustrated in Fig. 1. The\ndesign of SpbNet comprises four major components: (1) Descriptors, encompassing\nboth the raw material structure and the proposed PES basis functions; (2) Model\narchitecture, corresponding to the multi-modal input design; (3) Pre-training tasks,\nwhich facilitate multi-scale geometric feature learning; and (4) Fine-tuning process,\nwhich adopts a simple multihead-based fine-tuning strategy.\n(Section 2.1.2, paragraph 2, page 4)\nwe construct separate task-specific heads (multi-head design). During fine-tuning, only\nthe backbone parameters are retained, while all pre-training heads are discarded and\nreplaced with randomly initialized heads for downstream task. Compared with single-\nstream architectures, where the same layers and weights are used to process features\nfrom different modalities, the dual-stream design allows distinct sets of layers (self-\nattention and cross-attention layers) to specialize in modality-specific representations\n[26, 27]. This separation not only preserves the diversity of structural and energetic\nfeatures but also facilitates flexible substitution of the structural encoder, thereby\nleveraging advances in modern GNN architectures.\n(Section 2.1.3, paragraph 1, page 4)\nPre-training strategy constitutes one of the most critical components of a foundation\nmodel. In this work, we focus on training the network to learn well-established\ngeometric analysis of porous materials. A series of multi-scale pre-training tasks are\ndesigned. At the material level, the model is pre-trained to predict (1) topology type\n(e.g., pcu, fcu, etc. The model is trained to classify the material into a specific topology\nin the pre-defined topology type set), (2) pore limiting diameter (PLD), (3) largest\ncavity diameter (LCD), (4) accessible surface area (ASA), and (5) accessible volume\nfraction (VF). To mitigate bias toward specific guest molecules, ASA and VF are\ncomputed using probes of varying sizes. At the atomic level, the model is trained to\npredict (1) the number of atoms within each PES patch, and (2) the signed distance\nfunction (SDF) at the patch center. The SDF is also computed with probes of different\nsizes. Calculation details and the differences in geometric features obtained using\nvarious probe sizes are provided in Supplementary Information Text 1 and Fig. 2. These\npre-training tasks comprehensively leverage the accumulated domain knowledge of\ngeometric characterization in porous materials, thereby providing a robust and domain-\ninformed pre-training process. The correlation between these geometric features and\ndownstream tasks can be found in Supplementary Information Fig. 3-9.\nComment #1. The model, now named CT-FFKL, uses a transformer architecture that\nis pre-trained on a theoretical potential energy surface (PES), with graph neural\nnetwork (GNN) embeddings (e.g., from CGCNN) as input features. During the pre-\ntraining phase, the model is trained on three prediction tasks: number of atoms per\nenergy patch, void fraction, and a \"topological prediction\" task. However, these tasks\nappear relatively trivial given that the full MOF structure is already provided as input.\nAs such, it remains unclear how much meaningful generalization or representation\nlearning occurs during pre-training.\nThat said, the designation of CT-FFKL as a foundation model remains questionable.\nTo qualify as such, it should be demonstrated that the model benefits substantially\nfrom pre-training and can generalize across multiple downstream tasks with minimal\ntask-specific tuning. Currently, the manuscript does not provide strong evidence in\nthis regard. The pre-training tasks, as mentioned, seem relatively easy given the\n3\nmodel inputs, and no detailed analysis is presented to show how pre-training\nimproves downstream performance.\nPre-training Relevance: The manuscript should include experiments or ablations\ndemonstrating whether and how pre-training benefits downstream performance.\nThis is critical if the authors wish to frame CT-FFKL as a foundation model.\nResponse #1: Thanks for the useful suggestion from the reviewer. We greatly\nappreciate the suggestion on clarifying the effect of pre-training. In the revised\nmanuscript, we have added discussions about the role of pre-training in the Discussion\nsection. Here, four points have been emphasized: (1) Geometric features are correlated\nwith downstream tasks and can help the model adapt quickly. (2) The fine-tuned model\ncombines learned local information while the model trained from scratch tends to lose\nit. (3) The fine-grained number of atoms and SDF prediction task helps alleviate over-\nsmoothing, improving the effective rank of the low-dimensional semantic feature space.\nThis prevents model degradation and improves performance. (4) Additional ablation\nexperiments have been supplemented to quantify the effect of pre-training .\nIn the following, we elaborate on the four points and describe the corresponding\nmodifications made to the manuscript.\nPoint 1. Geometric features are correlated with downstream tasks and can help\nthe model adapt quickly.\nIn the revised manuscript, SpbNet is pre-trained to predict five global geometric\nfeatures: (1) Topology type, Topo, (2) Void Fraction, VF, (3) Accesible Surface Area,\nASA, (4) Pore Limited Diameter, PLD, and (5) Largest Cavity Diameter, LCD.\nWe first illustrate the correlation between the used geometric features with the\ndownstream task. Since the model has learned to accurately predict these features\nduring the pre-training phase, it is expected to be able to quickly adapt to the relevant\ndownstream tasks during the fine-tuning phase. (Supplementary Information Fig. 3,\n6a, 9)\nFurthermore, these features are not captured well when the model is trained from\nscratch. When we use these tasks as auxiliary training tasks, the model’s performance\nimproves over a single downstream task (i.e. using multi-task training). The model\nneeds to predict both the target property and these geometric properties. Compared with\nthe case where only the target property is predicted, the model’s performance is\nimproved. (Supplementary Information Table 2)\nFor SpbNet, the designed network architecture, due to directly taking the raw\nmaterial structure as input, does have the ability to predict these geometric properties\nsuch as VF and PLD. However, when the model is trained directly on a downstream\ntask without pre-training, it does not spontaneously capture these intermediate features.\nPre-training makes the network to capture these relatively general geometric features,\nthus assisting the model to quickly adapt to downstream tasks.\nCorrespondingly, we have added the following results in the revised manuscript.\nScatter plots of adsorption capacity versus geometric features. The scatter plots\nare drawn using 5000 training samples in the CoREMOF dataset (using the probe\nradius of 2.0 angstrom). The adsorption capacity of N (77K, 100kPa) and the four\n2\ngeometric properties show a significant linear correlation.\nSupplementary Information Fig. 6a.\nRandom forests regression results on CoREMOF dataset. The model takes PLD,\nLCD, ASA and VF as input, and predict the adsorption capacity of N (77K, 100kPa).\n2\n(a) The prediction results of the model on training and test set. MAE and R2 Score\nhave been annotated. As a comparison, the MAE and R2 Score of SpbNet in the test\nset is about 13.24 cm3(STP)/g and 0.9876.\nBox plot between topology type and adsorption capacity. About 15,000 MOFs are\nrandomly selected from the benchmark dataset. The horizontal axis shows the top 10\ntopology types with the most occurrences. The vertical axis plots the distribution of gas\nadsorption amounts for different topological types of MOF.\n(Section 2.1.3, paragraph 1, page 4)\nThese pre-training tasks comprehensively leverage the accumulated domain knowledge\nof geometric characterization in porous materials, thereby providing a robust and\ndomain-informed pre-training process. The correlation between these geometric\nfeatures and downstream tasks can be found in Supplementary Information Fig. 3-9.\n(Section 2.2, paragraph 3, page 4)\nCollectively, these findings demonstrate that SpbNet effectively learns well-defined\ngeometric features through pre-training. The multi-head architecture further allows the\nmodel to capture diverse types of features simultaneously without mutual interference,\neven when identical properties are calculated by different scales probes. Through such\ncomprehensive pre-training objectives, SpbNet develops generalizable geometric\nrepresentations that are transferable to downstream applications such as adsorption,\nfacilitating rapid and data-efficient adaptation. Although predicting these properties is\nrelatively straightforward, the model does not inherently acquire such intermediate\nfeatures when trained solely on downstream task labels. Incorporating these properties\nas auxiliary objectives in multi-task training also enhances downstream accuracy\n(Supplementary Information Table. 2), underscoring their value in guiding the model\ntoward more informative structural representations.\nCH\n2 3 6 3 6 3 8 4 2 4\nHeat of\nModel Adsorption Adsorption Separation Separation Uptake\nadsorption\n(MOF) (MOF) (MOF) (MOF) (COF)\n(zeolite)\nSpbNet\n25.7 0.451 0.195 0.403 13.4 1.10\n(scratch)\nSpbNet\n(scratch) 26.7 0.430 0.172 0.402 12.9 1.08\n+MT\nUsing the same benchmark dataset, the geometric features are calculated. The VF,\nASA, PLD, and LCD are calculated using the probe of 1.0 angstrom radius. The ASA\nis normalized via ASA =ASA/1000. The PLD and LCD are normalized via\nnorm\nPLD =PLD/8, LCD =LCD/8. In addition, we use an additional Pooler for all\nnorm norm\ngeometric features and different heads for each geometric features, like in the pre-\ntraining phase. The overall loss = MSE(property) + MSE(VF) + MSE(ASA) +\nMSE(PLD) + MSE(LCD). All training strategy remains the same. Simply adding the\ntasks of predicting geometric features prediction enhances the model’s performance.\nPoint 2. The fine-tuned model combines learned local information while the model\ntrained from scratch tends to lose it.\nSpecifically, when fine-tuning, the local features retained by the token do not\nchange significantly, and the [CLS] token is mainly responsible for aggregating these\nfeatures to predict downstream tasks. When the model is trained from scratch, the\nfeature of the [CLS] token propagates quickly and cover all other tokens, thus almost\neliminating all the original local features maintained in tokens.\nThe cosine similarity between the [CLS] token and the PES tokens is showed in\nFig. 6a to illustrate this point. When the model is trained from scratch, the average\ncosine similarity between the [CLS] token and other PES tokens increases to nearly 1.0\nas the layer becomes deeper. Considering that the [CLS] token represents the global\nfeature and is used to predict the material-level target property. This phenomenon\nindicates that when the model is trained from scratch, the [CLS] token is able to capture\nglobal information due to the explicit supervised loss. But due to the lack of token-level\nloss and the over-smoothing nature inherent in Transformer's attention mechanism1, the\nfeature of the [CLS] token quickly spreads and overwrites the features of all other\ntokens. Almost all PES tokens lose their own local information and instead preserve\nglobal features. This causes the deep layers of the model degenerate to a simple MLP.\nHowever, when the model is pre-trained, the situation is totally different. The\naverage cosine similarity between the [CLS] token and other PES tokens remains low,\nindicating that PES tokens maintain their own local feature. These features are learned\nin the pre-training phase, as will be shown in the next point.\nCorrespondingly, we have added the following figure and description in the\nrevised manuscript.\nEffect of pre-training. (a) The average cosine similarity between the [CLS] token\nand PES tokens. 100 samples are randomly selected in the pre-training dataset for\ncalculation. For “scratch”, the model is directly trained on the CoREMOF dataset to\npredict N adsorption capacity. For “geo”, “geo+anp”, and “geo+anp+sdf”, the model\n2\nis pre-trained using different tasks and fine-tuned on CoREMOF. Cosine similarities\nare calculated between the added [CLS] token and all PES patch tokens and averaged.\n(Section 3.1, paragraph 1 and 2, page 9)\nFig. 6a demonstrates the averaged cosine similarity between the global [CLS] token\nand other local PES tokens. When the model is trained from scratch, the feature of the\n[CLS] token rapidly aligns with other tokens as the network deepens. This is primarily\ndue to the absence of a token-scale loss constraint, where the inherent low-pass filtering\nproperty of the attention mechanism causes the features of all tokens to become overly\nsimilar, a phenomenon known as over-smoothing [47]. In contrast, when fine-tuned\nfrom pre-trained weights, the average similarity remains at a relatively low level,\nindicating that each energy patch maintains more localized information compared to\nthe global feature.\nThese results indicate that, the behavior of the [CLS] token in fine-tuning and\ntraining-from-scratch is essentially different. After the model is pre-trained, the model\nalready learns to compress the structure information into a low-rank feature space.\nIn this fine-tuning phase, the [CLS] token learns to aggregate these information to make\nprediction. When the model is trained from scratch, the lower layers still maintain local\ninformation of each token. However, the global feature of the [CLS] token quickly\nspreads to other tokens and erases almost all local information of PES tokens. At this\npoint, the network degenerates to a simple MLP.\nPoint 3. Token-level tasks enhance the dimension of the semantic feature space,\n8\nwhich is maintained in the fine-tuning phase.\nTo understand the token features in high-level networks, linear probing evaluation\ncan be used. This is a method commonly used in self-supervised learning and can be\nused to detect specific information held in features2. In summary, this approach uses a\nnetwork with fixed parameters to extract features, and then trains a simple linear layer\nto predict a certain attribute based on the extracted features. Thus the accuracy can\nreflect how much information of the target attribute is maintained in the features.\nAs mentioned before, SpbNet is trained to predict the number of atoms and SDF in\nthe pre-training phase. We investigated whether these features are retained after fine-\ntuning. Here we use linear probing evaluation to clarify this. Results are shown in Reply\nFig. 1.\nReply Fig. 1.\nLinear probing evaluation of the feature in high layer of the model. (a) Scatter plot\nof the predicted atom number in the validation set. (a) Scatter plot of the predicted\nSDF in the validation set. Eighteen randomly selected MOFs are used as the training\nset. Two MOFs are used as the test set. Eighteen MOFs contain totally 18*8*8*8=9216\npatches for training. Two MOFs contain totally 2*8*8*8=1024 patches for test. The\nlearning rate is fixed to 0.001 and the linear layer is trained for 50000 epochs with batch\nsize of 1024.\nSpbNet is finetuned on the H gas adsorption task. Next, we extract the features of\natoms) is maintained in these features. Specifically, we use a small neural network with\na single linear layer to predict the SDF and number of atoms. In this case, each PES\npatch is a sample, and the label is the SDF and atom number of this patch. The R2 score\nof predicting SDF by the features of PES tokens reaches 0.722 in the training set and\n0.393 in the validation set. The R2 score of predicting number of atoms by the features\nof PES tokens reaches 0.785 in the training set and 0.485 in the validation set. As a\ncomparison, the R2 score of SpbNet in SDF and atom number prediction in the pre-\ntraining phase are 0.957 and 0.911 (Supplementary Information Fig. 14). These results\ndemonstrate that the SDF and atom number information is maintained in the fine-tuning\n9\nphase.\nIn addition, due to the existence of these additional local information, the effective\nrank of the local feature in the high layer is increased. Effective rank represents the\nactual working dimension in the high dimensional feature space3. In the common case,\nduring training, the model spontaneously compress the input token features into a low-\ndimensional semantic feature space to make predictions for downstream tasks4,5. Thus,\nthe effective rank will decrease as the layer get deeper. As shown in Fig. 6b, both the\natom number and SDF prediction task increase the effective rank. These results indicate\nthat these tasks increase the dimension of the semantic feature. This may prevent the\nmodel from getting over-smoothing and degenerate too early.\nCorrespondingly, we have added the following figure and description in the\nrevised manuscript.\nFig. 6b.\nEffect of pre-training. (b) The Effective rank of PES tokens. Effective rank3\nrepresents the actual working dimension in the feature space. From lower layers to\nhigher, the effective rank decreases, indicating that the model learns to compress\ncomplex material structure information into low-dimensional semantic\nrepresentations4,5. Both anp and sdf task increase the effective rank, indicating that\nthese tasks make the model pay more attention to the local feature learning.\n(Section 3.2, paragraph 1, page 9)\nTo observe the impact of different pre-training tasks, Fig. 6b illustrates the effective\nranks under various pre-training task combinations. Similar to the Principal Component\nAnalysis (PCA) method, the effective rank quantifies the actual dimensionality that\nsignificantly contributes to the high-dimensional feature space [44]. From the lower to\nthe higher layers, the effective rank gradually decreases, suggesting that the feature of\neach PES token is compressed into a low-dimensional semantic space. During the\nmodel’s fine-tuning process, these extracted semantics are leveraged for making robust\npredictions. Both the atom number prediction task and the SDF prediction task enhance\n10\nthe effective rank throughout this process, enabling the model to utilize more layers to\nextract richer local information.\nPoint 4. Ablation studies.\nWe have conducted ablation studies on model’s performance in Fig. 6c and\nadded the following description in the revised manuscript.\n(Section 3.3, pages 9-11)\nFig. 6c.\nEffect of pre-training. (c) Ablation studies on global geometric property predict\n(geo), atom number prediction (anp), and SDF prediction (sdf). Global geometric\nproperty prediction task significantly improve the model performance. The local\ngeometric feature learning (anp and sdf) further enhances the accuracy. Removing the\nbasis functions decreases the accuracy.\nFig. 6c illustrates the ablation study results on six selected downstream tasks. The\ninclusion of token-level pretraining tasks yields performance improvements. Notably,\nthe SDF prediction task is particularly beneficial for gas separation, which may be\nattributed to its role in capturing the molecular sieving effect. The detailed surface shape\ninformation provided by the SDF assists the model in accurately determining whether\na guest molecule can easily pass through or be retained by the material’s pores.\nConversely, the atom number prediction task produced a certain negative effect on the\nzeolite tasks. This negative transfer may be due to the significant structural differences\nbetween zeolites and MOFs, as illustrated by the t-SNE plots of different materials in\nSupplementary Information Fig. 12 and 13. These combined results decisively\ndemonstrate the effectiveness and mechanism of the atom number and SDF prediction\ntasks in enhancing SpbNet’s representation learning capability.\nincluding versions using SchNet instead of CGCNN. However, SchNet is an older\narchitecture originally developed for molecular systems and may not be well-suited\nfor MOFs. The rationale for including it is not entirely clear, although this variation\ndoes not appear central to the model’s overall performance or conclusions.\nResponse #2: Thanks for the valuable comment. In the previous manuscript, the\nCGCNN-Transformer architecture is replaced with SchNet mainly to evaluate the\nadaptability of the model architecture to different structure encoders. Both SchNet\nand CGCNN are graph convolutional networks (GCNs) and their messages (in the\nmessage passing proess6) are constructed in a similar manner. They both generate\ndistance-based filters and aggregate information from neighbors accordingly. The main\ndistinction between the two lies in how they handle periodicity. CGCNN explicitly\nincorporates periodic boundary conditions when constructing the atomic topology\ngraph, whereas SchNet does not. As a result, the graph built by SchNet is one-sided,\nwhile that of CGCNN is multi-sided, since different periodic images of the same\natom—as well as neighboring atoms—can be adjacent in CGCNN. In our approach, we\ninprove SchNet’s graph construction by introducing periodicity in the same way as\nCGCNN. This modification leads to a significant improvement in model accuracy.\nAs shown in Reply Table 1, when the periodicity is taken into account, the\nperformance of SchNet is significantly improved. It should be noted that SchNet\nperforms better than CGCNN. This is mainly due to that we trade larger GPU memory\nrequirements for higher accuracy by adopting different hyperparameters. Specifically,\nin CGCNN, the crystal graph is constructed using a cutoff distance of 8 Å, with each\natom connected to at most 20 neighboring atoms. However, for large molecular crystals\nsuch as MOFs, this neighbor limit may not be sufficient to fully capture the local atomic\nenvironment. In our experiments, SchNet employs a larger cutoff distance of 10 Å and\nincreases the maximum number of neighbors to 30, enabling a more comprehensive\nrepresentation of the structural context and consequently improving model performance.\nThe structure encoder is one of the most critical components in SpbNet. The selection\nof structural encoders suitable for crystals and capable of handling porous material\nsystems is crucial to the final performance.\nReply Table 1. The effect of encoding of periodic crystal graphs (MAE).\nCH\n2 3 6 3 6 3 8 4 2 4\nHeat of\nModel Adsorption Adsorption Separation Separation Uptake\nadsorption\n(MOF) (MOF) (MOF) (MOF) (COF)\n(zeolite)\nCGCNN 71.8 0.653 0.216 0.616 19.7 1.88\nSchNet 50.7 0.777 0.211 0.639 13.1 1.97\nSchNet+\n33.1 0.498 0.192 0.424 14.9 1.39\nperiodicity\ntuning strategies (e.g., LoRA, multi-head, etc.) commonly used in other foundation\nmodels such as MACE-MP-0 or large language models.\nFine-tuning Procedure: The manuscript outlines fine-tuning for MOFormer but not\nfor CT-FFKL. How is fine-tuning implemented in practice? Are techniques such as\nweight decay, early stopping, or learning rate scheduling used? How is catastrophic\nforgetting mitigated?\nResponse #3: Thanks for the valuable comment. In the revised manuscript, the detailed\ndescription on the fine-tuning phase has been added in the Main Text. In summary, we\nuse two specific fine-tuning strategies: (1) The multi-head strategy is used in the pre-\ntraining phase, where each head corresponds to a task. In the fine-tuning phase, all\nheads are discarded and a new downstream head is added for training (Reply Fig. 2).\n(2) We reduce the weight decay from 0.25 (in the pre-training phase) to 0.1 (in the fine-\ntuning phase) for better performance. the learning rate increases linearly from 0 to\n10−4 during the first 2 epochs and then decays to 0 following the cosine function. In\naddition, we quantify the catastrophic forgetting problem, showing that parts of the\nfeatures learned by the model during the pre-training phase are retained in the model\nafter fine-tuning, as shown in Reply Fig. 1.\nReply Fig. 1.\nLinear probing evaluation of the feature in high layer of the model. (a) Scatter plot\nof the predicted atom number in the validation set. (a) Scatter plot of the predicted\nSDF in the validation set. Eighteen randomly selected MOFs are used as the training\nset. Two MOFs are used as the test set. Eighteen MOFs contain totally 18*8*8*8=9216\npatches for training. Two MOFs contain totally 2*8*8*8=1024 patches for test. The\nlearning rate is fixed to 0.001 and the linear layer is trained for 50000 epochs with batch\nsize of 1024.\nMulti-head strategy used in SpbNet. All trainable parts are marked in blue. All input,\nintermediate feature and output are marked in gray. In the pre-training phase, the global\nproperty prediction is based on a shared pooler and multiple heads, each one of which\ncorresponds to a pre-training task. The local feature prediction is based on a single\nlinear layer (SDF head and ANP head). After pre-training, in the fine-tuning phase all\ntask heads are discarded. A new pooler and downstream task head is added for fine-\ntuning.\nCorrespondingly, we have added the following description in the revised\nmanuscript on the manuscript.\n(Section 5.5, paragraph 1 and 2, page 14)\nWe follow the standard training paradigm commonly adopted in Transformer-based\nmodels. During the pre-training stage, the framework consists of (1) a linear layer that\ncombines the PES basis functions (see Sec. 5.3), (2) the structure encoder, and (3) the\nPES Transformer decoder as the backbone, with a single linear layer serving as the\nprediction head. An additional [CLS] token is appended to the sequence of PES patch\ntokens. The extracted feature of the [CLS] token is subsequently processed by a pooling\nlayer composed of a single linear layer followed by a tanh activation function. Each\npre-training task is associated with an independent prediction head, implemented as a\nsingle linear layer to predict the corresponding target property (e.g., topology type, PLD,\nLCD, etc.). All trainable components share the same training schedule. The learning\nrate increases from 10−5 to 10−4 over the first 15 epochs and then decays to 10−5\nfollowing a cosine schedule. The AdamW optimizer is used with a weight decay of 0.25.\nThe model is pre-trained for 40 epochs, and the checkpoint with the lowest validation\nloss is selected for fine-tuning.\nIn the fine-tuning phase, all backbone weights are retained, while the prediction\nhead (including the pooling layer and task-specific linear layer) is replaced. The pooling\n14\nlayer is re-initialized, and a new downstream task-specific head is added. The same\nlearning rate schedule is used as in the pre-training phase: the learning rate increases\nlinearly from 0 to 10−4 during the first 2 epochs and then decays to 0 following the\ncosine function. AdamW is again employed as the optimizer, with the weight decay set\nto 0.1. The model is fine-tuned for 60 epochs, and the checkpoint achieving the lowest\nvalidation loss is used for inference. Notably, a smaller weight decay is adopted in fine-\ntuning, as it is empirically found to yield better performance.\nComment #4. Furthermore, it is unclear why large datasets such as hMOF (130k\nstructures) are not used during pre-training rather than fine-tuning. If the goal is to\ncreate a transferable representation, training on the broadest dataset available would\nseem more appropriate.\nResponse #4: Thanks for the valuable comment. In addition to constructing the\nfoundation model, the primary goal of SpbNet is to enhance the pre-training data\nefficiency. To ensure a fair comparison with state-of-the-art (SOTA) models such as\nMOFTransformer7, our dataset is entirely a subset of the MOFTransformer dataset.\nFurthermore, Reply Fig. 3 displays the feature distributions of the pre-trained\nMOFs and hMOFs. The overall feature distribution of hMOFs is a subset of that of the\npre-trained MOFs. The pre-trained MOFs exhibit a broader and sparser distribution,\nwhereas the hMOF dataset is relatively concentrated. This difference may arise from\nthe relatively homogeneous topology of the hMOF dataset, in which MOFs with the\npcu (primitive cubic unit) topology account for approximately 90% of the database8.\nTherefore, to ensure both a fair comparison and sufficient sample diversity, and to avoid\nbias toward a specific topology, we did not adopt hMOFs as the pretraining dataset.\nReply Fig. 3.\nUMAP (Uniform Manifold Approximation and Projection for Dimension\nReduction) visualization of pre-trained MOF and hMOF dataset.\nComment #5. Pre-training Tasks: The manuscript does not clearly describe the\nnature of the \"topological prediction\" task. What precisely is being predicted, and\nhow does this relate to generalizable chemical or structural features?\nclarified that the topological prediction is the classification on material topology types.\nSince the pre-training dataset is obtained by combining different metal clusters and\nlinker according to a specific topological type. The label of the topological type can be\nobtained when constructing the dataset7. Specifically, the whole pre-training dataset\ncontains 1030 different types of topologies, thus on average each topology contains\nabout 100 material structures in the pre-training dataset. Given the raw material\nstructure, the model is trained to output the topology type such as pcu, fcu, etc. The\ncorrelation between topology and downstream tasks is shown in Supplementary\nInformation Fig. 9.\nSupplementary Fig. 9.\nBox plot between Topology type and adsorption capacity. About 15,000 MOFs are\nrandomly selected from the benchmark dataset. The horizontal axis shows the top 10\ntopology typies with the most occurrences. The vertical axis plots the distribution of\ngas adsorption amounts for different topological types of MOF.\nCorrespondingly, we have added the following description in the revised\nmanuscript on the manuscript.\n(Section 2.1.3, paragraph 1, page 4)\nThese pre-training tasks comprehensively leverage the accumulated domain knowledge\nof geometric characterization in porous materials, thereby providing a robust and\ndomain-informed pre-training process. The correlation between these geometric\nfeatures and downstream tasks can be found in Supplementary Information Fig. 3-9.\nReferences\ntransformers via the fourier domain analysis: From theory to practice. arXiv\n16\npreprint arXiv:2203.05962 (2022).\non machine learning. 1597-1607 (PMLR).\n606-610 (IEEE).\nInformation Processing Systems 35, 33054-33065 (2022).\ncompression in deep neural networks. arXiv preprint arXiv:2410.07687 (2024).\n34th International Conference on Machine Learning. (Jmlr-Journal Machine\nLearning Research, 2017).\ntransformer for universal transfer learning in metal-organic frameworks. Nat\nMach Intell 5, 309-318 (2023). https://doi.org/10.1038/s42256-023-00628-2\nframeworks for thermal conductivity. npj Computational Materials 9, 11 (2023).\n17"}, {"round": 2, "reviewer_comments": {"reviewer_2": "The authors have addressed all issues."}, "author_response": "Point-to-PointResponsetoReviews(NCOMMS-25-37906A)\nReviewer#2(RemarkstotheAuthor):\nTheauthorshaveaddressedallissues.\nResponse:Wethanktherevieweragainforthethoughtfulcommentsandpositivefeedbackonour\nrevisions."}]

Decision letter