An equivariant pretrained transformer for unified 3D molecular representation learning

Domain: chemistry
Published: 2026-02-10
Editorial outcome: major_revision

Abstract
Pretraining on a large number of unlabeled 3D molecules has showcased superiority in various scientific applications. However, prior efforts typically focus on pretraining models in a specific domain, missing the opportunity to leverage cross-domain knowledge. To mitigate this gap, we introduce Equivariant Pretrained Transformer, an all-atom foundation model that can be pretrained from multiple domain 3D molecules. Built upon an E(3)-equivariant transformer, the model learns both atom-level interactions and graph-level structural features (e.g. residuals in proteins), allowing it to generalize across diverse tasks. The model achieves strong gains in ligand binding affinity prediction, while also performing competitively in predicting properties of proteins and small molecules. We further show that the model can help identify potential antiviral compounds against the main protease of the COVID-19 virus, and validate promising candidates through computational and experimental studies.

Introduction
Understanding and accurately representing the 3D geometric structure of molecular systems is critically important in numerous scientific domains, such as life sciences, drug discovery, and material design. This significance arises because 3D structures largely determine molecular properties in downstream tasks, which cannot be effectively captured by simpler 1D representations like SMILES for chemical molecules or amino acid sequences for proteins. Recent advances in geometric Graph Neural Networks (GNNs) have enabled the accurate modeling of physical interactions between atoms while respecting the E(3) symmetry inherent in molecular systems. These models incorporate 3D coordinates for each node, ensuring that scalar attributes and dynamic processes remain invariant or equivariant under E(3) transformations. Strategies such as irreducible representations, frame averaging, and scalarization mechanisms have been employed to preserve the E(3) symmetry. Geometric GNN models have demonstrated remarkable success in various applications, including molecular property prediction, protein generation, and antibody design.

Despite remarkable successes, a central challenge remains: the scarcity of labeled data, which significantly limits the applicability of deep learning methods in scientific research. To address this challenge, researchers have drawn inspiration from self-supervised pretraining techniques in natural language processing (NLP), as exemplified by BERT and GPT. Pretraining establishes a general-purpose 3D representation that captures atom-level interactions under E(3) symmetry. Downstream tasks such as property prediction or binding affinity estimation rely on accurate modeling of such interactions, which are implicitly learned during pretraining. Experimental results confirm that models pretrained on large-scale, unlabeled 3D molecular datasets exhibit substantial performance gains on downstream tasks after finetuning with limited labeled data. Specifically, for small molecules, approaches such as GraphMVP and 3D Infomax apply contrastive learning from 2D to 3D pairs, while MoleBlend aligns 2D and 3D features through a multimodal pretraining framework. For proteins, GearNet uses contrastive learning on sequential and structural representations. Other works, such as Uni-Mol, have explored cross-domain interactions, by pretraining separate models for small molecules and protein pockets before finetuning on binding datasets. In addition, inspired by score-based generative models, denoising tasks have emerged as powerful pretraining methods for learning force fields, with approaches like NERE focusing on translation and rotation denoising at the ligand level.

Existing pretraining methods predominantly focus on domain-specific models, such as those tailored to small molecules, proteins, or dual-tower frameworks that separately handle small molecules and proteins. However, these approaches are limited in their ability to generalize across multiple domains. We believe that developing a foundation model for diverse atomic systems is indispensable for advancing various scientific applications. On the one hand, recent breakthroughs in AI, such as the GPT series, Gato, highlight the advantages of foundation models that unify learning across tasks and domains. These benefits include reduced need for handcrafted inductive biases, increased data diversity, and scalability with larger datasets and models. Extending this paradigm to molecular sciences could yield transformative results. On the other hand, from a physical perspective, all atomic systems, regardless of scale, are governed by the same fundamental principles, such as the Schrödinger Equation. Developing a single neural network capable of learning these universal principles represents a highly compelling pursuit.

Undoubtedly, building such a foundation model involves significant challenges that require addressing a range of technical and conceptual complexities. First, cross-domain data formats vary substantially. For small molecules, representations are typically at the atomic level, with each atom in a molecule corresponding to a specific set of features, such as atomic number, charge, and bond type. On the other hand, proteins are much more complex, requiring hierarchical representations where residues are composed of multiple atoms, and the structural context, including secondary structures, plays a critical role in determining function. This disparity in representation demands the development of a unified method capable of seamlessly integrating these different data formats and scales. Second, to capture the complex interactions between atoms and molecules, models must respect the fundamental symmetries that govern atomic systems. In particular, molecular interactions must be modeled while preserving E(3) symmetry, which accounts for the invariance of the system under rotations and translations in three-dimensional space. Finally, current self-supervised objectives used for pretraining are often task- or domain-specific. For example, objectives that work well for small molecules may not be directly applicable to proteins or other biological systems due to differences in their structural and functional properties.

In this paper, we propose an equivariant pretrained transformer (EPT), an all-atom foundation model that is pretrained from multiple-domain 3D molecules, including small molecules, proteins, and complexes. To our knowledge, we are one of the earliest attempts to develop a unified, all-atom representation model for handling diverse atomic systems across small molecules, proteins, and complexes in a self-supervised manner. Recent work, such as AlphaFold3, has also explored unified modeling of multiple molecular types, with a focus on accurate structure prediction across proteins, RNAs, and complexes. Firstly, by leveraging a unified molecular modeling component, EPT is able to not only process all-atom information but also incorporate block-level features that attend to a broader context of each atom, such as the atom-level surroundings for small molecules and the residue-level belongings for proteins. Secondly, EPT conforms to E(3) symmetry and is thoroughly designed upon a generic transformer. It derives the embedding layer with a one-layer equivariant GNN to reflect the graph geometry, and then updates the atom-level scalar and vector features via the equivariant self-attention and feed-forward mechanisms in each layer. Finally, the training objective requires the model to learn E(3)-equivariant denoising terms for reconstructing original conformations. This approach explicitly enables the model to not only understand the geometry of input systems, but also learn to stabilize noisy systems through rotation and translation. Notably, the pretraining objectives are applied to entire blocks, rather than individual atoms, allowing the model to better capture hierarchical and contextual relationships.

To pretrain EPT, we construct a large-scale pretraining dataset of over 5.89M entries by integrating existing datasets from different sources. We then demonstrate the efficacy of EPT across diverse downstream tasks and datasets, achieving state-of-the-art results in several cases. Specifically, for ligand binding affinity (LBA) prediction, EPT outperforms existing methods on benchmarks with varying protein sequence similarities (30 and 60%). Furthermore, for the Mutation Stability Prediction (MSP) task, EPT showcases robust generalizability, significantly outperforming existing domain-specific models. For the molecular property prediction (MPP) task, EPT achieves competitive results compared to leading approaches, with significant gains when using block-level denoising strategies. These results validate EPT’s ability to unify geometric learning across domains, providing a robust and generalizable framework for 3D molecular modeling. To further validate the utility of our EPT model, we apply EPT to rank FDA-approved drugs for their binding potential to the 3CL protease, a critical target in SARS-CoV-2 replication. The model successfully identifies known anti-COVID-19 drugs as top candidates, with further t-SNE analysis revealing that these drugs clustered closely with other high-affinity candidates predicted by EPT. From these clusters, we select 12 promising candidates for further validation through Molecular Dynamics (MD) simulations. Seven of these molecules exhibited higher binding affinities than the top-ranked known anti-COVID-19 drug. These results demonstrate the predictive power of EPT and its potential to accelerate drug discovery.

Results
Overview of EPT
EPT is an all-atom foundation model for multiple domains, including small molecules, proteins, and complexes. EPT creates a unified molecular representation across domains by defining “blocks” of atoms. For small molecules, each block is a collection of heavy atoms (non-hydrogen) and directly bonded hydrogens. For proteins, blocks are defined as amino acids. To enhance the model’s ability to capture the block-level information, EPT utilizes a block-level pretraining strategy based on denoising. Specifically, each block is perturbed by a random translation of its center-of-mass (CoM) and a random rotation around the CoM. The model is trained to predict both the translational forces and angular momentum needed to restore the original structure, strengthening its ability to understand hierarchical geometries and maintain the physical integrity of molecular structures.

To effectively capture the geometry of molecular structures, EPT employs an improved transformer architecture that integrates E(3) symmetry as its backbone model. The process begins with a graph embedding layer that encodes both atom- and block-level information while initializing vector features through a message-passing layer. Subsequently, EPT alternates between equivariant self-attention and feed-forward layers to model interatomic relationships, producing rich scalar and vector representations. In total, EPT contains parameters exceeding 30M. Finally, the pretrained model is finetuned for various downstream tasks at the molecular, protein, and complex levels. Remarkably, EPT showcases strong effectiveness and interpretability in screening potential anti-COVID-19 drugs.

Multi-domain pretraining dataset
We construct a large-scale pretraining dataset of over 5.89M entries by integrating existing 3D molecules from GEOM, PCQM4Mv2, PDB, and PDBBind. For small molecules, we filter conformations with the top-5 Boltzmann weights for each molecule in GEOM, yielding 1.89M conformations. These are combined with 3.38M structures from PCQM4Mv2 to form the pretraining dataset. For proteins, we source experimental structures from the Protein Data Bank (PDB), comprising 77,814 unique sequences along with 600k structural entries. Additionally, we incorporate 22,295 binding pockets from PDBBind to enrich the dataset with information on protein–protein and protein–molecule interactions. During pretraining, we deliberately exclude all property labels to prevent data leakage and ensure unbiased performance evaluation in downstream tasks.

Evaluation of model performance on downstream tasks
To evaluate the general performance of EPT, we select typical downstream tasks across two levels. At the binding level, we focus on ligand binding affinity prediction (LBA) and mutation stability prediction (MSP), while at the molecular level, we choose the molecule property prediction (MPP) task, with details as follows.

Ligand binding affinity prediction (LBA). We address the LBA task using the Atom3D dataset, which involves predicting the binding affinity between a protein pocket and its corresponding ligand. Following the setup in ref. 35, each sample in the dataset comprises a protein–molecule complex paired with its binding affinity value. The dataset offers two split configurations based on protein sequence similarity thresholds: ID30 (sequence identity capped at 30%) and ID60 (capped at 60%). The results evaluate our EPT model under four different training conditions: trained from scratch, pretrained on the small-molecule subset, pretrained on the protein subset, and pretrained on the entire multi-domain dataset. We have the following observations: (1) Structure-based models generally surpass sequence-based counterparts, underscoring the significance of 3D geometry in capturing interactive information. (2) Pretraining on each individual subset is capable of enhancing performance. Remarkably, EPT-Molecule, which is pretrained without exposure to protein or complex structures, still outshines the scratch-trained model, suggesting the presence of cross-domain transferable knowledge. (3) Interestingly, even EPT-Scratch outperforms prior pretraining-based baselines such as Uni-Mol. This can be explained by architectural differences: Uni-Mol adopts a dual-tower structure that processes proteins and ligands separately, whereas EPT uses a unified model that jointly encodes the entire protein-ligand complex. These results actually align with our motivation of developing a unified model, which better capture the geometric nature of molecular complexes and enhances downstream performance. (4) EPT-MultiDomain, benefited from the entire dataset containing diverse domains, outperform previous methods and achieves state-of-the-art performance on both of the splits. This implies that the breadth of pretraining data correlates positively with the model’s performance, and enables a more generalizable understanding of biological interactions.

Mutation stability prediction (MSP). Additionally, we evaluate the model on the MSP task to assess its generalization capabilities. This task involves predicting whether a point mutation at the interface of protein complexes improves binding affinity, framed as a binary classification problem. The MSP dataset is derived from the SKEMPI 2.0 database, containing experimentally measured binding affinities of protein–protein complexes before and after single-point mutations. We compare the scratch-trained EPT with Atom3D-CNN, GNN, and ENN, as well as GVP and GearNet-Edge. Additionally, we evaluate the multi-domain pretrained EPT against prior structure-based pretraining strategies using GearNet-Edge, including Multiview Contrast, DiffPreT, and SiamDiff. The results demonstrate that EPT outperforms all baselines, both when trained from scratch and when pretrained on external dataset, highlighting its superior foundational expressiveness and exceptional knowledge transferability in modeling macromolecular systems.

Molecule property prediction (MPP). For the MPP task, we select QM9 to evaluate the performance of EPT on small molecules. In detail, QM9 serves as a quantum chemistry benchmark that offers 12 chemical properties for each 3D molecule composed of C, H, O, N, and F elements. To benchmark EPT, we compare it against a variety of 3D geometric models and pretraining approaches designed for small molecules. In Table 1, our EPT outperforms or matches the performance of existing denoising-based methods, underscoring the effectiveness of multi-domain block-level pretraining. Additionally, we evaluate an augmented model, EPT-10, which incorporates ten layers as opposed to the original six-layer configuration. The improved results, presented in the last row of Table 1, demonstrate that performance improves as the model complexity is scaled up.

Analyses of the core components in EPT. We provide a series of ablation studies on the MPP task (QM9 dataset) to elucidate the contribution of each component to the performance of our backbone model. Specifically, we explore the following aspects. 1. We first substitute the input embedding delineated in Eq. (5) with a straightforward atom-level embedding. The findings suggest that enriching atom features with block-level information slightly improves model performance. 2. In Eq. (10), we integrate the distance matrix D and edge features R into the attention mechanism. Eliminating either or both of these elements leads to a decline in performance, thereby underscoring their collective significance in effectively capturing the varying interatomic relations. 3. In Eq. ((13–15)), we employ the FFN layers to amalgamate scalar and vector features. The resultant sharp drop after removing the FFN layers underscore the critical role of feature fusion in our model.

In addition, we evaluate the influence of pretraining datasets and denoising strategies on the LBA and MPP tasks. As an extension of Fig. 2b, we observe a consistent trend where pretraining on one domain confers benefits to downstream tasks in another domain. Specifically, the model pretrained on small molecules demonstrates enhanced performance on the LBA task, while the model pretrained on proteins exhibits improved results on the QM9 benchmark. In general, the model pretrained on the multi-domain dataset shows superior performance across all evaluated downstream tasks. We further compare the three kinds of denoising strategies. The L_atom strategy focuses on atom-level denoising, showing superior results for small molecular structures. However, its benefits are less pronounced when applied to large, complex systems such as the protein–molecule complexes in LBA. On the contrary, L_block-T adopts a more macroscopic approach by only considering the translations of each block’s center-of-mass. This coarse-grained strategy improves performance for larger systems but tends to struggle in smaller molecules. Finally, our model, which accounts for both block-level rotations and translations, provides comprehensive supervision for predicted forces, resulting in the optimal performance across all strategies.

EPT-accelerated discovery of anti-COVID-19 candidates
We conduct virtual screening experiments to find potential anti-COVID-19 small-molecule drugs. We adopt the pretrained EPT and finetune it on the training complexes. To benchmark EPT, we compare its performance against two typical baselines: the classical method, Glide based on energy computation, and the dual-tower model ESM+Uni-Mol, which applies the pretrained protein model ESM for pocket modeling and Uni-Mol for representing small molecules. As shown in Fig. 4a, our model achieves significantly better performance than both Glide and the dual-tower models in the affinity prediction tasks. For ranking tasks, EPT outperforms the dual-tower models and performs on par with Glide. These results underscore the effectiveness of our unified representation model in capturing the complex interactions between ligands and their target protein pockets.

Based on these findings, we apply our finetuned model to rank 1978 FDA-approved drugs (including 26 antiviral drugs and eight marketed anti-COVID-19 drugs) in order to evaluate their binding potential to 3CL protease, a key target in SARS-CoV-2 replication. The rankings of 8 anti-COVID-19 drugs are shown in Fig. 4b, where we observe that nearly all drugs are ranked within the top 200. We visualize the representations of all ligand-pocket complexes using t-SNE in Fig. 4b. The results reveal that, with the exception of the structurally distinct outlier Xocova, which features a non-peptidic triazine scaffold, anti-COVID-19 drugs generally form a concentrated cluster surrounded by other drugs with high EPT ranking scores. Additionally, we select the top six antiviral drugs based on their rankings and further identify the top six non-antiviral, non-anti-COVID-19 drugs as extra potential anti-COVID-19 candidates. These 12 top-ranking candidates are located close to seven marketed anti-COVID-19 drugs in the t-SNE embeddings, indicating their potential high affinity for the 3CL protease. To quantitatively assess the binding potential of the 12 selected candidates, we perform molecular docking and Molecular Dynamics (MD) simulations, identifying two hits, Ac-Leu-Leu-Nle-CHO and Saquinavir, with EPT-predicted dissociation constants (Kd) of 87.4 and 24.5 nM, respectively. Of the 12 selected candidates, 9 demonstrate binding potential to the target with ΔG < 0, and 7 deliver lower ΔG than Leritrelvir, the top-ranking anti-COVID-19 drug in Fig. 4b. These results not only emphasize the predictive power of EPT but also highlight its potential to accelerate the identification of promising drug candidates. Among the two strong hits, Saquinavir is reported by the literature to inhibit 3CL protease with an IC50 of 9.92 μM. We further tested Ac-Leu-Leu-Nle-CHO, which showed an IC50 of 5.47 μM. These results confirm their inhibitory potency.

Methods
Datasets
We curate a comprehensive structural dataset comprising 5.89 million samples and 261 million blocks, spanning small molecules, proteins, and complexes. Among these, small-molecule datasets, including GEOM and PCQM4Mv2, contribute 35.8% of the total blocks. For protein structures, experimental data are collected from the Protein Data Bank (PDB), which encompasses 600,000 structural entries and accounts for 59.4% of the blocks. The remaining 4.8% of blocks are derived from PDBBind, including protein–protein complexes (PP), protein–molecule complexes from the refined set, and other protein–molecule complexes collected before version 2020, denoted as v2020-other-PL. Each sample is stored as a sequence of blocks, with each block defined by four attributes: the types of atoms within the block, the type of the block itself, the position index of each atom, and the 3D coordinate matrix of the block.

We employ virtual screening as a typical application of the pretrained EPT model. To finetune EPT, we first prepare the Docked-PDBBind dataset. Specifically, we obtain 10,000 protein-small-molecule complexes from PDBBind and re-dock them using the Glide software to generate 10,000 positive samples. Additionally, for each complex, we randomly selected ten non-binding small molecules and docked them with the protein using Glide, yielding 100,000 negative samples. The re-docking process ensures consistency with the inference phase, where only individual protein and small-molecule structures are available, rather than pre-formed complexes. This alignment is crucial to maintain distributional similarity between the training data and the docking results used during inference. For inference, we dock the target 3CL protease with 1978 FDA-approved drugs and rank them based on the predicted binding probabilities from the finetuned model.

Model architecture
We represent a molecule with N atoms as a fully connected graph G, where atoms are depicted as nodes, and their interactions are depicted as edges E. To capture the high-level structure within molecules, atoms are grouped into M predefined blocks to enrich the node features. Let m_i denote the index of the block containing atom i, the feature set for an atom is extended to (a_i, b_{m_i}, p_i, z_i), where a_i in A specifying the atom type, b_{m_i} indicates the block type of m_i, p_i denotes the atom’s positional index within its block, and z_i in R^3 represents the atom’s 3D coordinate. The positional index within an amino acid is determined using Greek letters, ordered based on their distance from the Cα atom. For small molecules, the positional index is represented by a designated token, [Mol]. While G is fully connected in the sense that all atom pairs are processed by the self-attention mechanism, we assign discrete edge types based on geometric distance and topology to enhance local interaction modeling. Detailedly, local interactions between atoms are categorized into three distinct edge types to reflect both intra-block and inter-block relationships. Mathematically,

e_{ij} = 0, m_i = m_j
1, m_i != m_j, d(m_i, m_j) <= delta_topo
2, m_i != m_j, delta_topo < d(m_i, m_j) <= delta_max

where delta_topo and delta_max are predefined thresholds that represent topological and maximum allowable distances, respectively. The function d(m_i, m_j) = min_{p in m_i, q in m_j} ||z_p - z_q||_2 calculates the minimum Euclidean distance between any two atoms belonging to blocks m_i and m_j.

In the pursuit of efficiently capturing the nuanced interactions of atoms within molecules, we present the equivariant full-atom transformer. It utilizes the transformer-based backbone to model the complex interactions among atoms, while updating the scalar and vector features to capture the rich geometric information inherent in molecular structures.

Our model first acquires the input features from the Graph Embedding layer, and iteratively updates the features at each layer l. Let H^{(l)} = [h_1^{(l)}, h_2^{(l)}, ..., h_N^{(l)}] in R^{h x N} denote the scalar, and V^{(l)} = [v_1^{(l)}, v_2^{(l)}, ..., v_N^{(l)}] in R^{3 x h x N} denote the vector. The model is constructed in this way:

[H^{(0)}, V^{(0)}] = Embedding(A, B, P, Z)
[H^{(l-0.5)}, V^{(l-0.5)}] = [H^{(l-1)}, V^{(l-1)}] + Self-Attn(LN(H^{(l-1)}), V^{(l-1)})
[H^{(l)}, V^{(l)}] = [H^{(l-0.5)}, V^{(l-0.5)}] + FFN(LN(H^{(l-0.5)}), V^{(l-0.5)})

After the embedding layer, self-attention (Self-Attn) and feed-forward networks (FFN) are applied alternately, with pre-layer normalization (LN) and residual connections preceding each operation. We modify the Self-Attn and FFN layers to be E(3)-equivariant, preserving the geometrical symmetry of molecular structures. These layers are detailed as follows. For conciseness, we omit the layer subscript l unless otherwise specified.

For the graph embedding layer, the input features are obtained as:

f_i = f_b(b_{m_i}) + f_a(a_i) + f_p(p_i)
e_{ij}^0 = [f_i, f_j, e_{ij}, RBF(||z_i - z_j||_2)]
h_i^{(0)} = phi_h(f_i, sum_{j in N(i)} phi_s(e_{ij}^0) * f_j)
v_i^{(0)} = sum_{j in N(i)} phi_v(e_{ij}^0) * (z_i - z_j)

where f_b, f_a, f_p separately embed the block types, atom types and atom orders, RBF(⋅) denote the radial basis functions, and phi_h, phi_s, phi_v are MLPs to aggregate neighbor information to enrich the 0th layer features. It is necessary to initialize V^{(0)} with SE(3)-equivariant non-zero values via Eq. (8), otherwise the vector features will remain zeros in subsequent layers.

The self-attention layer plays a crucial role in modeling interatomic interactions. For each layer, query Q_s, key K_s, and value V_s are computed as:

Q_s = H W_Q, K_s = H W_K, V_s = [H W_V^h, V W_V^v]

where W_Q, W_K, W_V^h, W_V^v in R^{h x h_s} are trainable parameters that map the features to the appropriate query, key, and values spaces. And the attention mechanism is given by

[H_s, V_s] = Softmax(Q_s K_s^T / sqrt(d)) * (D + R) V_s

where D = [d_{ij}]_{N x N} = [||z_i - z_j||_2]_{N x N} is the distance matrix, and R = [r_{ij}]_{N x N} encodes the edge interactions and geometric relations as

r_{ij} = phi_r(e_{ij}, RBF(||z_i - z_j||_2)), (i,j) in E
0, (i,j) not in E

Here phi_r is an MLP. The outputs of the self-attention layer combine the contributions of all heads:

Delta H = sum_s H_s W_O^h, Delta V = sum_s V_s W_O^v

where W_O^h, W_O^v in R^{h_s x h} are head-specific trainable parameters.

Building upon the Geometric Vector Perceptron (GVP) concept, the equivariant feed-forward layer is where the scalar and vector features are fused and updated simultaneously:

V_1, V_2 = V W_1, V W_2
Delta H, U = phi_FFN(H, ||V_1||)
Delta V = LN(U) * V_2

where W_1, W_2 in R^{h x h} are trainable parameters, phi_FFN is an MLP as an auxiliary, and * denotes element-wise multiplication. The intermediate matrix U is layer-normalized to conserve the scale of the updated vectors.

Block-level denoised pretraining
Based on the backbone model above, an additional force prediction head is required for denoised pretraining. In practice, we apply an additional FFN-like layer over layer-L to fuse the output scalars and vectors as

[H_out, V_out] = [H^{(L)}, V^{(L)}]
F^0 = phi_F(H_out, ||V_out W^0||) * V_out W^0

Perturbations are applied by introducing translational and rotational noise to each block, modeled as:

Z_b^0 = C(g_t(Z_b + sigma_t epsilon_b)) + Q_b Z_b^r

where epsilon_b ~ N(0, I_{3M}) represents translational noise, Q_b is the rotation matrix derived from rotational noise omega sampled from the isotropic Gaussian distribution IG(sigma_r), and Z_b^r = Z_b - g_t(Z_b) represent the relative atom positions to block centers. The centering operator C(⋅) ensures translational neutrality.

The pretraining objective incorporates two components. First, a translation objective minimizes the discrepancy between predicted resultant forces and translational noises averaged over blocks:

L_block-T = E_{epsilon_b ~ N(0, I_{3M})} ||mu_b(F^0) - mu_b(Z_b^0 - Z_b)||_2^2

where mu_b(⋅) computes block-level averages. Second, a rotation objective aligns angular accelerations derived from predicted torques with gradients of the rotational noise distribution.

L_block-R = E_{omega ~ IG(sigma_r)} ||alpha_b - nabla_omega p(omega)||_2^2

where alpha_b = I_b^{-1} M_b^0 is the predicted angular acceleration, M_b^0 = sum_{j in b} (z_j - g_t(Z_b)) * F_j^0 represent the torque, and I_b is the inertia matrix of each block. The combined block-level denoising loss integrates these two objectives:

L = L_block-T + L_block-R

Training regimen
We first pretrain the backbone model on the collected multi-domain dataset based on the objective L_block-C, leveraging 8 NVIDIA Tesla A800 GPUs. The pretraining process spans 50 epochs, incorporating translational and rotational noise scales of sigma_t = 0.04 and sigma_r = 0.1, respectively. A dynamic batching strategy is applied, capping each mini-batch at a maximum of 10,000 atoms per GPU. After pretraining, the force prediction head is omitted, and the backbone model is further finetuned with task-specific heads.

For MPP, we finetune the model using the objective L = L_MAE + lambda L_block-C, where lambda = 0.1 balances the weight of each term. For LBA, we apply a local mask where only attentions from neighbors within delta_max of each node are calculated, and we consider three types of output heads based on an MLP phi_E as follows:

phi_atom(H^{(l)}) = sum_i phi_E(h_i^{(l)})
phi_block(H^{(l)}) = sum_{m_i} phi_E(sum_{j in m_i} h_j^{(l)})
phi_graph(H^{(l)}) = phi_E(sum_i h_i^{(l)})

For MSP, we use the split by sequence identity over 30% provided by Atom3D, and extract all residues within 6Å distance to the mutation point as the local view for input, where the distance between two residues is measured by the minimum distance between atom pairs.

For the virtual screening benchmark, we design two tasks. For the ranking task, we use a negative rate of 0.6 to adjust the fraction of negative data pairs during training. Additionally, for the prediction task, we use the same local mask as the LBA task. We list all hyperparameters in Supplementary Note 3 Implementation details.

Evaluation metrics
Following Atom3D benchmark, we use mean absolute error (MAE) on the MPP task by measuring the average deviation between predicted properties and ground-truth labels. For the LBA task, we report RMSE, Pearson, and Spearman correlation coefficients to comprehensively evaluate model performance. RMSE (root mean square error) measures the absolute difference between predicted and ground-truth binding affinities, directly reflecting the accuracy of the predicted magnitudes. Pearson correlation coefficient (Pearson or PCC) assesses the linear relationship between predicted and actual values, focusing on the consistency of relative magnitudes, and thus is insensitive to systematic offsets or scale mismatches. Spearman correlation coefficient (Spearman or SPCC), on the other hand, captures the monotonic rank-order agreement between predictions and ground truth, evaluating how well the predicted affinities preserve the correct ordering. Together, these metrics offer complementary perspectives, respectively emphasizing numerical precision, linear alignment, and ranking accuracy. For the MSP task, we compute the area under the receiver operating characteristic curve (AUROC) to measure binary classification performance.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The Equivariant Pretrained Transformer (EPT) is a cross-domain foundation model for 3D molecules, leveraging E(3)-\nequivariant transformers and block-level denoising. Pretrained on 5.89M entries, it outperforms previous SOTA in ligand\nbinding affinity, protein, and molecular property prediction. EPT also shows results in virtual screening, identifying anti-\nCOVID-19 drugs and novel compounds with higher binding affinities.\nThe noteworthy results:\n1. The manuscript includes a comprehensive list of baselines for comparision and evaluates their model's performance\nacross three major downstream tasks.\n2. A detailed ablation study is conducted, clearly highlighting the imporance of each component in the proposed method.\n3. The methods section is well-formulated and easy to follow, providing sufficient detail for reproducibility.\n4. Beyond benchmarking baselines and proposing their method, the authors also perform virtual screening (3CL-protease) to\naddress a real-world problem, demonstrating practical applicability.\nMajor issues that need addressing to improve clarity and depth:\n1.Pre-training Mechanism: What is the purpose of pre-training? Is there explicit evidence to explain how pre-training\nenhances performance on downstream tasks? What is the relationship between pre-training and downstream tasks? Why\nwere translation and rotation denoising chosen as the pre-training objectives?\n2.Baseline Implementation: In Figure 2b, were all comparable methods re-trained by the authors? The manuscript does not\nmention this. How were the baseline methods implemented?\n3.EPT-Scratch Performance: In Figure 2b, the EPT-Scratch variant already outperforms all other methods. Can the authors\nexplain why? Since some baselines (e.g., Uni-Mol) also use transformer architectures, were the comparisons conducted\nfairly?\n4.Atom3D Dataset Details: In line 151, the authors mention using the Atom3D dataset for the MSP task. Can more details\nabout the dataset be provided? What are the labels, and how much data does it contain? Given the difficulty of obtaining\ncrystal structures for single protein mutations, how does the model perform when predicting on less accurate or predicted\nstructures?\n5.Figure 3b Clarification: In Figure 3b, what does \"Base\" refer to? Does it indicate no pre-training?\n6.Ranking Task and Virtual Screening:\n6.1 In Section 2.4, why were only 10 additional ligands sampled as negatives? In virtual screening scenarios, it is common\nto rank inhibitors among millions of molecules.\n6.2 The manuscript does not explain how negative samples were sampled until Figure 5, which mentions docking. What\ndocking program was used?\n6.3 How were the Glide program and ESM+Uni-Mol implemented? Details of their re-implementation are lacking, which\ncould impact performance if not used properly.\n6.4 In line 248, how were docking and MD simulation performed? How was the delta G of ~-30 kcal/mol calculated? The\nprocess is unclear. While the supplementary material includes MD details, this should be referenced in the main manuscript,\nalong with the total simulation time (e.g., how many nanoseconds were run for each simulation?). More details on baseline\nimplementation, especially for Glide, are needed.\n6.5 For greater impact (Advice), the top candidates from the virtual screening should be validated through wet lab\nexperiments to confirm their inhibitory potency.\nThe manuscript has several issues related to typos and clarity that need addressing:\n1. In Figure 2b, the colors are difficult to distinguish, making comparisons challenging. Using a consistent color scheme for\neach method would improve readability.\nThe authors need to address the above points to strengthen the paper's contributions and ensure its impact on the field. By\nproviding more details on the pre-training mechanism, baselines implementation, dataset characteristics and experimental\nprocedures.\n(Remarks on code availability)\nI reviewed the source code on private GitHub and found the guide to be promising. It's structured in a way that allows me to\nexecute the code step by step.\nThe source code includes the training component only, the author should also provide an inference module to enable others\nto replicate the results reported in the paper. This would significantly enhance the reproducibility and utility of the work.", "reviewer_2": "This manuscript presents Equivariant Pretrained Transformer (EPT), a novel all-atom foundation model designed for unified\n3D molecular representation learning across multiple domains, including small molecules, proteins, and complexes. The\nauthors leverage an E(3)-equivariant transformer architecture, which is capable of modelling atom-level information while\nincorporating block-level features. The model is pretrained using a block-level denoising task and evaluated on various\ndownstream tasks, showcasing remarkable performance. Additionally, the model is applied to identify small-molecule drug\ncandidates targeting 3CL protease, a critical target in SARS-CoV-2 replication.\nOverall, the manuscript is well-structured and presents a significant advancement in the field of 3D molecular representation\nlearning. EPT addresses a key gap in the literature by providing a unified framework for handling diverse molecular systems.\nThe E(3)-equivariant transformer backbone is innovative and aligns well with the physics principle of molecular interactions.\nNotably, the block-level denoising task contributes to the model’s ability to learn robust and generalizable representations.\nHowever, there are several aspects of the study that could benefit from further clarification and discussion:\n1. In Eq. (1), the edges are defined within a range of $\\delta_{max}$. How does this definition reconcile with the fully-\nconnected graph described in line 301? Is there a potential conflict between the two representations?\n2. While the model demonstrates impressive performance on many tasks, its performance on the <R^2> task in the QM9\ndataset appears to be less satisfactory. Could the authors provide an explanation for this result?\n3. In the virtual screening experiments, the fine-tuned model effectively identified known anti-COVID-19 drugs as top\ncandidates. Could the authors further elaborate on the fine-tuning process?\n4. Following question 3, it is worth noting that Xocova, one of the marketed anti-COVID-19 drugs, was ranked relatively low\nin the list. Could the authors provide a possible explanation for this outlier result?\nMinor editing suggestions:\nThe title number in line 345 is redundant.\n(Remarks on code availability)", "reviewer_3": "Summary\nThe paper introduces Equivariant Pretrained Transformer (EPT), a novel all-atom foundation model designed to leverage 3D\nmolecular data across multiple scientific domains, including proteins, small molecules, and their complexes. Unlike previous\napproaches limited to single domains, EPT utilizes an E(3)-equivariant transformer architecture to capture both atom-level\nand block-level structural features. A key innovation is the use of a block-level denoising pretraining task, enabling better\ngeneralization.\nEPT is trained on a large-scale dataset of 5.89 million 3D molecular structures, spanning diverse biological and chemical\nentities. Experimental results demonstrate EPT’s strong performance on various downstream tasks, with state-of-the-art\nresults in ligand binding affinity prediction and competitive outcomes in protein and molecular property predictions.\nEPT also shows promise in drug discovery, notably ranking 7 out of 8 known anti-COVID-19 drugs among the top 200\ncandidates from 1,978 FDA-approved drugs targeting SARS-CoV-2’s 3CL protease. Moreover, Molecular Dynamics\nsimulations validate 7 newly identified compounds with superior binding affinities, highlighting EPT’s potential for impactful\ntherapeutic development.\nStrengths\n* The empirical performance of EPT is robust. The manuscript selects several key downstream tasks highly relevant to drug\ndiscovery (LBA, MSP, MPP, affinity prediction, ranking) and demonstrates that EPT consistently outperforms existing strong\nbaseline methods. This indicates that EPT holds significant promise for enhancing efficiency in drug design workflows.\n*\n* The manuscript is well-structured and easy to follow, clearly presenting motivation and technical details. Figures are well-\ndesigned, clear, and visually appealing.\n* The concept of unifying diverse 3D molecular representations within a single framework is valuable. This approach\nenhances data utilization efficiency and contributes a foundation model beneficial to the community, potentially reducing the\nnecessity for extensive model selection.\nWeaknesses\n* Several critical discussions are missing, affecting the manuscript's rigor:\n* EPT underperforms Frad on MPP tasks, and the paper claims that EPT-10 (enlarging the backbone from 6 to 10 layers)\nachieves state-of-the-art results. However, this comparison might be unfair since the network sizes for other models are fixed\nor unknown. What if the network sizes of other models are also increased? It would be beneficial to include an explanation\nfor EPT-10 explicitly in the caption of Table 1.\n* The claim that EPT is the first general model capable of handling atomic systems of various types (lines 59-60) may be\noverstated. AlphaFold3, for instance, also manages multiple types of 3D molecules. It is advisable to moderate this claim or\nprovide a detailed comparison with relevant related work.\n* The manuscript contains minor typographical errors:\n* Line 13 is missing a space between \"property prediction\" and \"[56]\".\n* Panel (d) appears to be missing in Figure 1.\n* Several questions remain open for clarification:\n* Could the authors discuss the key differences between various correlation metrics (RMSE, Pearson, Spearman) used in\nLBA evaluation? Clarifying the specific strengths and focus areas of these metrics would better contextualize the model's\nperformance.\n* Can the current block definitions be universally applied across general molecular systems? Currently, blocks are defined\nspecifically for proteins and molecules—can this be extended to other molecular systems such as nucleic acids or crystals?\n* Technically, this paper does not propose an entirely novel method. The E(3) Transformer architecture and pre-training\nstrategy are pre-existing techniques. The primary novel contribution is the block-level representation for proteins and\nmolecules, yet alternative block definitions are also possible (e.g., defining blocks as amino acids for proteins and heavy\natoms for molecules is not a fundamentally new approach).\n(Remarks on code availability)"}, "author_response": "Point-by-Point Response to Reviewer Comments\nWe thank all the reviewers for their insightful feedback and constructive comments during\nthis round of review. Here we provide detailed responses to address the proposed concerns\nand revise our paper accordingly. The reviewers’ comments are written in blue , our\nresponses are in black, and the changes to the manuscripts are in red .\nReferee #1\nMajor issues that need addressing to improve clarity and depth:\nQ1: Pretraining Mechanism: What is the purpose of pretraining? Is there explicit evidence to\nexplain how pretraining enhances performance on downstream tasks? What is the\nrelationship between pretraining and downstream tasks? Why were translation and rotation\ndenoising chosen as the pretraining objectives?\nWe thank the reviewer for this insightful question regarding the motivation and impact of our\npretraining strategy. We address your concerns point by point below.\nA1.1: The purpose of pretraining.\nAs already presented in Lines 15-28, pretraining has become a widely successful technique\nin modern AI systems (e.g., ChatGPT), enabling learning from large-scale unlabeled data. In\nour framework, pretraining primarily aims to equip the model with a generalizable\nunderstanding of 3D molecular geometries across diverse domains, including small\nmolecules, proteins, and their complexes. Given the scarcity of labeled data in molecular\nscience, such as protein-ligand affinity scores, pretraining helps the model learn rich\nstructural priors from abundant unlabeled 3D data.\nA1.2: Empirical evidence on the efficacy of pretraining.\nIn Figure 3b/c, we demonstrate that all EPT variants pretrained on different domains (small\nmolecules or proteins) consistently outperform the Base model without pretraining, on both\nthe Atom3D and QM9 datasets. Notably, multi-domain pretraining yields stronger\nperformance than single-domain pretraining models (i.e. Molecule-Only and Protein &\nComplex Only) across all downstream tasks, demonstrating effective cross-domain\nknowledge transfer.\nWe have revised the caption of Figure 3 by adding: “Base” refers to the EPT model without\npretraining. This clarification emphasizes that the experiments in Figure 3 demonstrate the\nefficacy of pretraining.\nA1.3: The relationship between pretraining and downstream tasks.\nPretraining establishes a general-purpose 3D representation that captures atom- and\nblock-level interactions under E(3) symmetry. Downstream tasks such as property prediction\nor binding affinity estimation rely on accurate modeling of such interactions, which are\nimplicitly learned during pretraining. Our use of equivariant architectures ensures that the\nlearned features are physically consistent and transferable.\nA1.4: The use of translation and rotation denoising.\nWe design our pretraining task to match the physical nature of molecular systems.\nSpecifically, we apply random translation and rotation perturbations to blocks, and train the\nmodel to recover them. This choice is motivated by several considerations:\n(1) Geometric Awareness: The training objective requires the model to learn\nE(3)-equivariant denoising terms for reconstructing original conformations. This\napproach explicitly enables the model to not only understand the geometry of input\nsystems, but also learn to stabilize noisy systems through rotation and translation.\n(2) Hierarchical Nature: The pretraining objectives are applied to entire blocks, rather\nthan individual atoms, allowing the model to better capture hierarchical and\ncontextual relationships.\n(3) Empirical Superiority: As shown in Figure 3c, the combined block-level translation\nand rotation denoising strategy outperforms both atom-level and translation-only\ndenoising strategies, indicating that our chosen objective is optimal across molecular\nscales.\nWe have revised Lines 18-21 and 71-76 to reflect the above points.\nQ2: Baseline Implementation: In Figure 2b, were all comparable methods re-trained by the\nauthors? The manuscript does not mention this. How were the baseline methods\nimplemented?\nWe sincerely thank the reviewer for raising this important point regarding the baseline\nimplementations, which is crucial for ensuring the validity of our comparisons.\nIn Figure 2b, we did not re-train the baseline models but utilized the performance metrics\nreported in the original publications of ProtNet [1] and ProFSA [2], following the standard\npractice for the well-established LBA benchmark. This approach ensures consistency and\navoids potential discrepancies from reimplementation/re-training, which can be highly\nresource-intensive and non-trivial due to dependencies and version-specific implementation\ndetails, potentially introducing unintended variations. The LBA benchmark, along with its\nstandard data splits (ID30/ID60), is widely recognized and used in the field. Following the\nprecedent set by prior works, including [1] and [2] themselves, which also report results\nrelative to established baselines using published numbers, our setting allows for direct and\nfair comparison.\nWe acknowledge the reviewer's concern for thoroughness and want to assure them that this\nmethodology is chosen specifically to uphold the integrity and comparability of our results.\nShould the reviewer deem it essential, we would be willing to discuss the feasibility of\nre-running specific baselines as an additional analysis, though we believe the current\napproach provides the most consistent and reliable comparison foundation.\nWe have mentioned this in Lines 145-147 of the main paper.\n[1] Wang, Limei, et al. \"Learning Hierarchical Protein Representations via Complete 3D\nGraph Networks.\" The Eleventh International Conference on Learning Representations.\n[2] Gao, Bowen, et al. \"Self-supervised Pocket Pretraining via Protein\nFragment-Surroundings Alignment.\" The Twelfth International Conference on Learning\nRepresentations.\nQ3: EPT-Scratch Performance: In Figure 2b, the EPT-Scratch variant already outperforms all\nother methods. Can the authors explain why? Since some baselines (e.g., Uni-Mol) also use\ntransformer architectures, were the comparisons conducted fairly?\nWe thank the reviewer for this insightful observation. The superior performance of\nEPT-Scratch compared to prior methods like Uni-Mol is largely attributed to architectural\ndifferences: Uni-Mol adopts a dual-tower model to encode the ligand and pocket separately,\nwhereas EPT uses a unified model that jointly encodes the entire protein-ligand complex.\nThis unified model enables more effective learning of cross-molecular interactions, even\nwithout pretraining. These results actually align with our motivation of developing a unified\nmodel, which better captures the geometric nature of molecular complexes and enhances\ndownstream performance.\nWe have provided more explanations in Lines 155-160.\nQ4: Atom3D Dataset Details: In line 151, the authors mention using the Atom3D dataset for\nthe MSP task. Can more details about the dataset be provided? What are the labels, and\nhow much data does it contain? Given the difficulty of obtaining crystal structures for single\nprotein mutations, how does the model perform when predicting on less accurate or\npredicted structures?\nWe thank the reviewer for this question and sorry for the insufficient details. The MSP\ndataset in Atom3D is based on SKEMPI 2.0 [1], which contains experimentally measured\nbinding affinities (Kd) of protein-protein complexes before and after single-point mutations.\nEach mutation is labeled as 1 if it improves binding (mutant Kd < wild-type Kd), and 0\notherwise. The dataset includes 893 positive and 3255 negative examples, and is split by\n30% sequence identity for evaluation.\nTo generate 3D mutant structures, the authors of Atom3D applied in silico mutagenesis\nusing PyRosetta , where local sidechain repacking was performed within 10 Å of the mutated\nresidue. This modeling protocol introduces predicted structures based on experimentally\nresolved wild-type PDB entries, rather than experimentally determined mutant structures. As\nsuch, while the protocol is standardized and widely used, it may introduce some structural\ninaccuracies. Nevertheless, our model demonstrates robustness in this setting.\nWe added the details of MSP data construction in Lines 168-172.\n[1] Jankauskaitė, Justina, et al. \"SKEMPI 2.0: an updated benchmark of changes in\nprotein–protein binding energy, kinetics and thermodynamics upon mutation.\" Bioinformatics\n35.3 (2019): 462-469.\nQ5: Figure 3b Clarification: In Figure 3b, what does \"Base\" refer to? Does it indicate no\npretraining?\nWe thank the reviewer for pointing this out. In Figure 3b, “Base” refers to the EPT model\nwithout any pretraining, i.e., trained from scratch using only the downstream task data.\nWe refined the caption of Figure 3b with additional details.\nQ6: Ranking Task and Virtual Screening:\nQ6.1: In Section 2.4, why were only 10 additional ligands sampled as negatives? In virtual\nscreening scenarios, it is common to rank inhibitors among millions of molecules.\nWe appreciate this thoughtful question and apologize for any confusion. In Section 2.4, we\nused just 10 additional ligands as negatives to efficiently train EPT to discriminate true\nligands from decoys. Importantly, once trained, EPT can in principle rank inhibitors among\nmillions of molecules.\nIn the evaluation process, we limited the negative ligands to 10 per target across all 500\ntargets. While we recognize that virtual screening typically involves ranking binders from\nmuch larger compound libraries, scaling to millions of molecules across 500 targets would\nbe computationally prohibitive. Crucially, our further analysis here shows that the evaluation\nmetric remains stable whether using 10 or more decoys per target ( Supplementary 1.10 ),\ndemonstrating that this reduced setting still enables robust performance comparisons across\ntargets.\nSupplementary Fig. 7 : avg_rank(n) and 𝝙avg_rank(n)=avg_rank(n)-avg_rank(n+1) over the\nnumber of negative samples $n$ on the id30 split. The Oracle method indicates positive\nsamples are always ranked as top-1.\nTo complement our large-scale benchmark, we conducted a real-world virtual screening\nexperiment targeting SARS-CoV-2 3CL protease. In this evaluation, we ranked true binders\namong thousands of FDA-approved drugs, and our model successfully ranked 3 out of 8\nknown positive drugs within the top 10 candidates. Notably, two additional top-10 candidates\n- Saquinavir and Conivaptan HCl - have been independently reported as 3CL protease\ninhibitors in prior studies [1,2]. As detailed in our response to Q6.5, we further validated\nthese findings by synthesizing and testing Ac-Leu-Leu-Nle-CHO (ranked 7th), which showed\nthe most favorable binding energy (MM/PBSA ΔG) among the top candidates. Experimental\nvalidation confirmed its inhibitory activity with an IC50 of 5.47 μM. These results collectively\ndemonstrate EPT's practical utility in real-world drug screening applications involving large\ncompound libraries.\nWe have revised Lines 249-254 to address the reviewers’ concern.\n[1] Chiou, Wei-Chung, et al. \"Repurposing existing drugs: identification of SARS-CoV-2\n3C-like protease inhibitors.\" Journal of enzyme inhibition and medicinal chemistry 36.1\n(2021): 147-153.\n[2] Drayman, Nir, et al. \"Drug repurposing screen identifies masitinib as a 3CLpro inhibitor\nthat blocks replication of SARS-CoV-2 in vitro.\" BioRxiv (2020): 2020-08.\nQ6.2: The manuscript does not explain how negative samples were sampled until Figure 5,\nwhich mentions docking. What docking program was used?\nSorry for the confusion. We use the Glide program for docking negative samples. To address\nyour concerns, we have explained this point in Line 237.\nQ6.3: How were the Glide program and ESM+Uni-Mol implemented? Details of their\nre-implementation are lacking, which could impact performance if not used properly.\nThank you for this valuable suggestion. We provide more details as follows:\n(1) Glide:\nWe used Schrödinger Suite 2021-2 for docking. Protein structures were preprocessed with\nPrepWizard by adding hydrogen atoms and optimizing with the OPLS3 force field at pH 7.4.\nLigands were prepared using LigPrep, preserving the original chirality, while Epik was used\nto predict pKa values and generate protonation states at pH 7.0. Ligand conformations were\noptimized with the S-OPLS force field, and a single conformation was retained for docking.\nThe receptor grid was generated with an innerbox size of 10 × 10 × 10 Å and an outerbox\ndefined as (x − x + 20) × (y − y + 20) × (z − z + 20), with the force field set to\nmax min max min max min\nOPLS3 and other parameters left at default values. Docking was then carried out using Glide\nSP (standard precision).\n(2) ESM+Uni-Mol:\nWe appreciate the reviewer’s concern regarding the implementation details of the\nESM+Uni-Mol baseline. In our experiments, we used the docked-PDBBind dataset to ensure\nconsistency across all methods. For the protein representation, we adopted the pretrained\nmodel esm2_t33_650M_UR50D [1], generating an embedding of shape (R, 1280), where R\ndenotes the number of amino acid residues in the protein. We then computed the\nprotein-level embedding by averaging the residue-wise embeddings. For the ligand\nrepresentation, we employed the UniMolRepr function from the unimol_tools [2,3] package\nto obtain a 512-dimensional ligand embedding.\nTo combine the two modalities, we utilized a standard dual-tower architecture. Specifically,\nfor each protein-ligand pair, we concatenated the protein and ligand embeddings as the joint\ninput. This concatenated vector was passed through a Feed Forward Network (FFN) with 4\nhidden layers and 1 output layer, where the hidden size is 512 and the activation function is\nSiLU. The model was trained with a learning rate of 5e-4, and all remaining setups, including\nthe training objective and negative rates, were kept identical to those used in our EPT\nmodel.\nThe above details have been explained in Supplementary Information Section 1.3.1. We\nhope these clarifications adequately address the reviewer’s concern.\n[1] Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., ... & Rives, A. (2022). Language\nmodels of protein sequences at the scale of evolution enable accurate structure prediction.\nBioRxiv , 2022 , 500902.\n[2] Zhou, G., Gao, Z., Ding, Q., Zheng, H., Xu, H., Wei, Z., ... & Ke, G. Uni-Mol: A Universal\n3D Molecular Representation Learning Framework. In The Eleventh International\nConference on Learning Representations .\n[3] Gao, Z., Ji, X., Zhao, G., Wang, H., Zheng, H., Ke, G., & Zhang, L. (2023). Uni-qsar: an\nauto-ml tool for molecular property prediction. arXiv preprint arXiv:2304.12239 .\nQ6.4: In line 248, how were docking and MD simulation performed? How was the delta G of\n~-30 kcal/mol calculated? The process is unclear. While the supplementary material includes\nMD details, this should be referenced in the main manuscript, along with the total simulation\ntime (e.g., how many nanoseconds were run for each simulation?). More details on baseline\nimplementation, especially for Glide, are needed.\nWe thank the reviewer for pointing out the need for additional details regarding the docking\nand MD simulation procedures.\n(1) Docking and MD simulation:\nMolecular docking was performed using Glide. The detailed protocol is provided in our\nresponse to Q6.3 and has also been incorporated into Section 1.3.1 of the Supplementary\nInformation.\nAs for MD simulation, we have carefully reviewed Section 1.4.2 in the Supplementary\nInformation and added important implementation details as: This was followed by a\nrestrained constant number of particles, volume, and temperature (NVT) ensemble\nequilibration for 100 ps, and a constant number of particles, pressure, and temperature\n(NPT) ensemble equilibration for an additional 100 ps.\nIn addition, we included a summary of the MD simulation procedure, including the total\nsimulation time, in Lines 269-276 as: Detailed protocols for molecular docking, MD\nsimulations, and ΔG calculation are provided in the Supplementary Information Section 1.4.2\nand 1.4.3. The SARS-CoV-2 3CLpro-ligand complexes were simulated using GROMACS\n2024.2 [1]. Energy minimization was performed using the steepest descent algorithm,\nfollowed by 100 ps of restrained equilibration under the constant number of particles,\nvolume, and temperature (NVT) ensemble and 100 ps under the constant number of\nparticles, pressure, and temperature (NPT) ensemble. Subsequently, 100 ns of unrestrained\nproduction MD simulations were carried out at 310 K and 1 bar.\n(2) Binding free energy calculation:\nBinding free energy (ΔG, e.g., ~-30 kcal/mol) was estimated using the MM/PBSA approach\nbased on explicit-solvent MD trajectories, considering the complex, receptor, and ligand\ncomponents [1-2]. Further details regarding the calculation of ΔG are provided in Section\n1.4.3 of the Supplementary Information.\n[1] Valdés-Tresanco, Mario S., et al. \"gmx_MMPBSA: a new tool to perform end-state free\nenergy calculations with GROMACS.\" Journal of chemical theory and computation 17.10\n(2021): 6281-6291.\n[2] King, Edward, et al. \"Recent developments in free energy calculations for drug\ndiscovery.\" Frontiers in molecular biosciences 8 (2021): 712085.\nQ6.5: For greater impact (Advice), the top candidates from the virtual screening should be\nvalidated through wet lab experiments to confirm their inhibitory potency.\nWe thank the reviewer for the valuable suggestion regarding the experimental validation. In\nthe development pipeline of SARS-CoV-2 3CL protease inhibitors, we first performed\nEPT-based virtual screening, and identified 12 candidate compounds. Subsequent MD\nsimulations combined with MM/PBSA analysis revealed two hits: Ac-Leu-Leu-Nle-CHO (ΔG\n= −29.34 kcal/mol) and Saquinavir (ΔG = −31.08 kcal/mol), both exhibiting significantly more\nfavorable binding free energies than the positive control Leritrelvir (ΔG = −6.16 kcal/mol).\nNotably, Saquinavir was reported to inhibit 3CL protease with an IC50 of approximately 9.92\nμM [1], providing additional support for its potential. Accordingly, we further synthesized and\nexperimentally evaluated Ac-Leu-Leu-Nle-CHO, which demonstrated an IC50 of\napproximately 5.47 μM. These wet lab experiments confirm the inhibitory potency of the top\ncandidates selected by our EPT. The experimental details are as follows.\n(1) Compound information:\nThe compound Ac-Leu-Leu-Nle-CHO (also known as MG-101; CAS No. 110044-82-1) was\nsynthesized and purchased from MedChemExpress (MCE, Cat# HY-18964). The compound\nwas supplied with a reported purity of ≥98% and was used without further purification. Its\nidentity and purity were confirmed by MCE based on Mass Spectrometry (MS) and Nuclear\nMagnetic Resonance (NMR) analysis ( s ee Supplementary Fig. 3 ), as documented in the\nCertificate of Analysis (CoA). The CoA and additional quality control documents are\navailable upon request.\nSupplementary Fig. 3 : Analytical data for Ac-Leu-Leu-Nle-CHO. a , ESI-MS spectrum\nshowing the [M+H]⁺ peak at m/z 384.2, consistent with the expected molecular weight\n(383.28). b , ¹H NMR spectrum (DMSO-d₆) confirming the chemical structure and purity of\nthe compound. Source: Certificate of Analysis provided by MedChemExpress.\n(2) SARS-CoV-2 3CL protease inhibition assay:\nThe inhibitory activity of Ac-Leu-Leu-Nle-CHO against SARS-CoV-2 3CL protease was\nevaluated by TaoShu Bioscience (Shanghai, China). The compound was prepared as a\n10 mM stock in DMSO and serially diluted using a 10-point, 3-fold dilution series starting\nfrom 5 mM. Diluted compound (80 nL per well, transferred by Echo) was dispensed into\n384-well plates (Corning 4514), with each well adjusted to a final volume of 8 μL (top dose\n50 μM, final DMSO 1%).\nRecombinant SARS-CoV-2 3CLpro was diluted in assay buffer (50 mM HEPES pH 7.5, 2 mM\nDTT, 0.01% Triton X-100, 0.01% BSA) to 20 nM, and 4 μL enzyme was pre-incubated with\ncompound at room temperature for 30 minutes. The reaction was initiated by addition of 4 μL\nof a fluorogenic peptide substrate (HiLyte Fluor 488-ESATLQSGLRKAK(QXL520)-NH2, final\n2 μM).\nAfter 60 minutes at room temperature, fluorescence (Ex/Em 480/540 nm) was measured\nusing an EnVision plate reader (PerkinElmer). Percent inhibition was calculated relative to\npositive and negative controls. IC50 values were determined by four-parameter logistic fitting\n(4PL, XLfit Model 205, Microsoft Excel), and dose-response curves were plotted in Python\n(matplotlib) using the fitted parameters. All experiments were performed in duplicate (n = 2).\nSupplementary Fig. 4 : Dose-response curve of Ac-Leu-Leu-Nle-CHO in the SARS-CoV-2\n3CL protease inhibition assay. The absolute IC50 value was determined to be 5.47 μM using\na 4PL model. The curve was plotted in Python (matplotlib) with fitted parameters provided in\nthe experimental report.\nOverall, the content related to the Ac-Leu-Leu-Nle-CHO assay has been added to Lines\n286–289 of our paper and to Section 1.5 of the Supplementary Information.\n[1] Chiou, Wei-Chung, et al. \"Repurposing existing drugs: identification of SARS-CoV-2\n3C-like protease inhibitors.\" Journal of enzyme inhibition and medicinal chemistry 36.1\n(2021): 147-153.\nQ7. In Figure 2b, the colors are difficult to distinguish, making comparisons challenging.\nUsing a consistent color scheme for each method would improve readability.\nWe thank the reviewer for the suggestion. We revised Figure 2b to use a consistent color\nscheme for each method, thereby improving readability. Inspired by this comment, we also\nrefined the molecular diagrams in Figures 1d and 4b to further enhance the quality of\npresentation.\nQ8. The source code includes the training component only, the author should also provide\nan inference module to enable others to replicate the results reported in the paper.\nWe thank the reviewer for this suggestion. The evaluation codes are provided in the\n“/evaluation” folder of our codebase. We also list the commands required to run the\nevaluation, which are integrated into the training pipelines for each downstream task.\nReferee #2\nThere are several aspects of the study that could benefit from further clarification and\ndiscussion:\nQ1: In Eq. (1), the edges are defined within a range of $\\delta_{max}$. How does this\ndefinition reconcile with the fully-connected graph described in line 301? Is there a potential\nconflict between the two representations?\nWe thank the reviewer for the insightful observation. The term \"fully-connected graph\" refers\nto the fact that all atom pairs are considered in the self-attention mechanism, ensuring that\nlong-range dependencies can be modeled across the entire molecular structure.\nIn contrast, the edge types defined in Eq. (1) with distance thresholds are specifically used to\nintroduce localized edge-aware attention, allowing the model to pay extra attention to local\nand near-local interactions. These edge encodings are incorporated as part of the edge\nattention term (denoted R_{ij}) in the attention computation (see Eq. (10–11)), alongside the\nself-attention and distance-based attention components.\nTherefore, there is no contradiction: our architecture supports global communication via a\nfully-connected attention graph, while explicit edge features are used to encode and\nemphasize local interactions to further improve expressiveness.\nWe have provided more explanations of the setting in Lines 357-360.\nQ2: While the model demonstrates impressive performance on many tasks, its performance\non the <R^2> task in the QM9 dataset appears to be less satisfactory. Could the authors\nprovide an explanation for this result?\nWe appreciate the reviewer’s attention to the ⟨R²⟩ results on the QM9 dataset. We\nacknowledge that the performance of EPT on this particular target is less competitive\ncompared to other properties.\nThis observation aligns with prior studies [1,2], which have highlighted that ⟨R²⟩—the\nelectronic spatial extent—can suffer from negative transfer in pretrained models. In addition,\nwe note that some previous works adopt a specific projection head tailored for ⟨R²⟩, based\non the formulation R^2 = \\sum_{i=1}^N e_i \\|x_i - x_m\\|_2^2, where e_i is a predicted scalar\nand x_m is the molecular center-of-mass (CoM). Inspired by this finding, we also apply an\nadditional projection head to EPT, achieving a test MAE of 0.037, which is significantly better\nthan the result obtained with a standard regression head (0.122).\n[1] Jiao, Rui, et al. \"Energy-motivated equivariant pretraining for 3d molecular graphs.\"\nProceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 7. 2023.\n[2] Feng, Shikun, et al. \"Fractional denoising for 3d molecular pretraining.\" International\nConference on Machine Learning. PMLR, 2023.\nQ3: In the virtual screening experiments, the fine-tuned model effectively identified known\nanti-COVID-19 drugs as top candidates. Could the authors further elaborate on the\nfine-tuning process?\nWe thank the reviewer for requiring more details regarding the fine-tuning process used in\nthe virtual screening experiments. The pretrained EPT model was fine-tuned on the\nDocked-PDBBind dataset (described in Methods – Datasets ) , where each protein receptor is\nassociated with 1 positive ligand and 10 negative decoys. We formulated two tasks on this\ndataset—affinity prediction and ranking (see Section 2.4). For virtual screening on the 3CL\nprotease, we employed the model weights fine-tuned under the ranking task.\nDuring training, each epoch iterates over all protein receptors in the dataset. For each\nreceptor, a protein-ligand pair is sampled based on a hyperparameter negative_rate ∈\n[0,1]. In particular, a negative ligand is uniformly selected with probability negative_rate ,\nwhile a positive ligand is selected with probability 1-negative_rate . In the affinity\nprediction task, negative_rate is fixed to 0 to train only on positive complexes. For the\nranking task, the negative_rate is set as described in Supplementary Table 8 .\nArchitecturally, we treated EPT as an encoder that generates a graph-level representation\nfor each protein-ligand complex. Different output heads are tailored for different tasks. For\nthe affinity prediction task, we employ the same head as in Eq. (24), which outputs a single\nscalar representing the predicted binding affinity. For the ranking task, we appended a\n2-layer Feedforward Neural Network (FFN) as a task-specific output head, where the hidden\nsize is the same as that of EPT and the activation function is ReLU. It outputs a probability\nvector [ 𝑦 , 𝑦 ] , where 𝑦 , 𝑦 indicate the probabilities of the ligand being positive or negative,\n𝑝 𝑛 𝑝 𝑛\nrespectively. If the ligand is a negative sample, the target label is [0, 1], otherwise the label is\n0 . 183\ndefined as [ 𝑦 , 1 − 𝑦] , where 𝑦 = ( 𝑝𝐾 / 16. 0) . Here, pK is the ground-truth binding score,\n16.0 is the maximum pK observed in the training set (used for normalization), and 0.183 is\nan empirical exponent reflecting the prior that a complex with pK = 9.0 corresponds to a 0.9\npositive probability. The cross-entropy loss is leveraged as the training objective.\nWe performed finetuning under three data splits: id30, id60, and scaffold. The virtual\nscreening against the 3CL protease specifically used the model finetuned under the id60\nsplit.\nWe hope this clarifies the fine-tuning procedure used in our study.\nQ4: Following question 3, it is worth noting that Xocova, one of the marketed anti-COVID-19\ndrugs, was ranked relatively low in the list. Could the authors provide a possible explanation\nfor this outlier result?\nWe thank the reviewer for bringing this phenomenon to our attention. Xocova adopts a\nnon-peptidic triazine scaffold, which is markedly different from the peptidic or peptidomimetic\nstructures of mainstream 3CL protease inhibitors such as Leritrelvir, Rupintrivir, 3CLpro-1,\nGC376, Simnotrelvir, Nirmatrelvir, and Lufotrelvir ( s ee Supplementary Fig. 1 ) . This\nobservation indicates that, although the model exhibits strong discriminative power for\nmainstream structures, its performance still has room for improvement when it comes to\nidentifying and evaluating compounds with novel scaffolds.\nSupplementary Fig. 1 : EPT rankings of marketed anti-COVID-19 drugs. Global rankings of\neight marketed anti-COVID-19 drugs based on EPT scores, along with their 2D chemical\nstructures.\nWe have added the explanation regarding the outlier Xocova to Lines 260-261 of our paper\nand to Section 1.4.1 of the Supplementary Information.\nMinor editing suggestions:\nQ5: The title number in line 345 is redundant.\nWe thank the reviewer for pointing this out. We have removed the redundant title number.\nReferee #3\nSeveral critical discussions are missing, affecting the manuscript's rigor:\nQ1: EPT underperforms Frad on MPP tasks, and the paper claims that EPT-10 (enlarging\nthe backbone from 6 to 10 layers) achieves state-of-the-art results. However, this\ncomparison might be unfair since the network sizes for other models are fixed or unknown.\nWhat if the network sizes of other models are also increased? It would be beneficial to\ninclude an explanation for EPT-10 explicitly in the caption of Table 1.\nWe thank the reviewer for this helpful suggestion. To assess the impact of model size, we\nhave conducted additional experiments by scaling both Frad and EPT from 6 to 12 layers.\nResults on HOMO, LUMO, and Gap Prediction show that Frad does not benefit consistently\nfrom larger network size, while EPT continues to improve up to 10–12 layers, indicating\nbetter scalability and stability.\nTo clarify the use of EPT-10 in Table 1, we have modified the caption of Table 1 as follows:\nWe also include the performance for both Frad and EPT when calling the networks from 6 to\n12 layers in the Supplementary Fig. 5. Frad does not benefit consistently from larger network\nsize, while EPT continues to improve up to 10–12 layers, indicating better scalability and\nstability.\nSupplementary Fig. 5: Comparison of performance of EPT and Frad under increasing layers.\nMean absolute errors (MAE) on HOMO, LUMO, and Gap prediction are reported for both\nEPT and Frad with varying numbers of layers (6, 8, 10, and 12). EPT demonstrates\nconsistent performance improvements as the model depth increases. In contrast, Frad does\nnot consistently benefit from deeper architectures.\nQ2: The claim that EPT is the first general model capable of handling atomic systems of\nvarious types (lines 59-60) may be overstated. AlphaFold3, for instance, also manages\nmultiple types of 3D molecules. It is advisable to moderate this claim or provide a detailed\ncomparison with relevant related work.\nWe thank the reviewer for pointing this out. We agree that the original wording may overstate\nthe novelty of our approach. We provide a detailed comparison between AlphaFold3 and our\nmethod here.\nWhile AlphaFold3 and EPT share the common motivation of modeling diverse molecular\ndomains, they differ in scope and application. AlphaFold3 focuses on structure prediction,\nwhereas EPT is designed as a general-purpose representation learning framework. Through\nself-supervised pretraining, EPT learns transferable representations that support a broader\nrange of downstream tasks beyond structural modeling alone, including molecular property\nprediction and protein-ligand affinity scoring.\nWe have moderated the claim and revised the wording in Lines 61–65 and Lines 305-310.\nThe manuscript contains minor typographical errors:\nQ3: Line 13 is missing a space between \"property prediction\" and \"[56]\".\nQ4: Panel (d) appears to be missing in Figure 1.\nWe thank the reviewer for pointing these out, and have fixed the typos and missing panel\nnumbers in the revised manuscript.\nSeveral questions remain open for clarification:\nQ5: Could the authors discuss the key differences between various correlation metrics\n(RMSE, Pearson, Spearman) used in LBA evaluation? Clarifying the specific strengths and\nfocus areas of these metrics would better contextualize the model's performance.\nWe thank the reviewer for the suggestion. In the LBA task, we report RMSE, Pearson, and\nSpearman to provide a comprehensive evaluation from different perspectives. The specific\nstrengths and focus areas of these metrics are as follows.\nRMSE (Root Mean Square Error) measures the absolute error between predicted and\nground-truth binding affinities. It is sensitive to large errors and directly reflects how close the\nmodel’s predictions are with respect to the ground-truth pKds.\nPearson Correlation Coefficient captures the linear correlation between predicted and\nground-truth values. It is scale-invariant and focuses on whether high-affinity complexes are\nconsistently predicted with higher scores. In practice, a high Pearson correlation is valuable\nwhen the relative magnitude of predicted affinities is more important than the precise values.\nSpearman Correlation Coefficient evaluates the rank-order alignment between predictions\nand labels, focusing only on the order rather than the values themselves.\nWe have extended the details of evaluation metrics in Lines 433-442.\nQ6: Can the current block definitions be universally applied across general molecular\nsystems? Currently, blocks are defined specifically for proteins and molecules—can this be\nextended to other molecular systems such as nucleic acids or crystals?\nWe thank the reviewer for this thoughtful question. In our current work, blocks are defined\nbased on domain knowledge: amino acid residues for proteins and heavy-atom groups for\nsmall molecules, reflecting domain-specific structural and chemical units.\nHowever, the underlying framework of EPT is flexible and extensible. Once domain-specific\nblock definitions are provided, the same architecture can be applied to other molecular\nsystems. To directly prove this, we extended the EPT block vocabulary to nucleic acids and\nevaluated its zero-shot generalization on Nucleic acid-ligand binding prediction. Specifically,\nwe introduced four deoxyribonucleic acid and four ribonucleic acid building blocks into the\nvocabulary while keeping the model architecture unchanged. The model was finetuned on\nProtein-Protein, Protein-Nucleic acid, and Protein-Ligand complexes from the PDBBind\ndataset and then evaluated on novel data type: Nucleic acid–Ligand complexes. As shown in\nTable 1 below, the pretrained model achieves significantly better performance compared to\ntraining from scratch, demonstrating that EPT generalizes effectively to nucleic acid systems\nwith simple vocabulary extensions. These results are added as Section 1.6 in the revised\nSupplementary Information.\nTable 2: The mean and standard deviations of 3 runs on the PDBBind NL complexes.\nModel Pearson↑ Spearman↑\nEPT-pretrain 0.402 ± 0.037 0.315 ± 0.058\nEPT-scratch 0.359 ± 0.057 0.263 ± 0.061\nQ7: Technically, this paper does not propose an entirely novel method. The E(3) Transformer\narchitecture and pretraining strategy are pre-existing techniques. The primary novel\ncontribution is the block-level representation for proteins and molecules, yet alternative block\ndefinitions are also possible (e.g., defining blocks as amino acids for proteins and heavy\natoms for molecules is not a fundamentally new approach).\nWe sincerely thank the reviewer for this insightful observation, which helps clarify the scope\nof our contributions. We agree that the foundational concepts of E(3)-equivariance and\ntransformer architectures are established previously, and our work builds thoughtfully upon\nthese shoulders. The core novelty of EPT lies not in inventing wholly new technical\ncomponents, but in their purposeful integration into the first unified framework that achieves\nhigh-performance, all-atom representation learning across diverse molecular domains (small\nmolecules, proteins, complexes) at scale. We highlight three synergistic innovations enabling\nthis:\n(1) Unified Block-Level Representation: We abstract both proteins and molecules into a\nconsistent format, enabling a single transformer model to operate across different\nmolecular modalities including small molecules, proteins, and complexes. While\nblock definitions like residues or heavy-atom groups are domain-informed, our\ncontribution is to integrate them seamlessly into a unified representation framework.\n(2) Model Architecture Enhancements: The EPT backbone is not a direct reuse of\nexisting E(3)-equivariant transformers (e.g. TorchMD-Net and Equiformer). We\nintroduce several key modifications to make it suitable for all-atom, multi-domain\nrepresentation learning, including block-aware embedding (Eqs. (5-8)), hybrid\nscalar-vector updations (Eqs. (3-4)), and distance-aware attention bias (Eqs.\n(10-11)). Moreover, unlike conventional E(3)-equivariant transformers that rely on\nGNN-based edge updates, EPT employs a pure attention mechanism that maintains\ncompatibility with standard transformer optimizations (e.g., xFormers for memory\nefficiency) while preserving equivariance. This design allows computationally\ntractable integration of geometric biases. Implementation details are provided\nthrough PyTorch-style pseudocode in Appendix 1.6.1. As demonstrated in Appendix\n1.6.2, EPT achieves superior memory efficiency compared to TorchMD-Net and\nEquiformer across varying system sizes (up to 2048 nodes). This scalability enables\nmodeling of large molecular systems without sacrificing representational capacity.\n(3) Block-Level Pretraining Objective: We design a block-level translation and rotation\ndenoising task that leverages rigid-body perturbations to capture local geometry and\ncontext. This task is specifically aligned with the physics of molecular systems, and\nwe show in Figure 3c that it outperforms the atom-level denoising counterparts.\n(4) All-Atom Multi-Domain Pretraining: EPT is pretrained on large-scale datasets across\nthree domains (molecules, proteins, complexes) and shows strong performance on\ndiverse downstream tasks. This supports our claim that EPT is a general-purpose 3D\nmolecular representation model.\nWe hope the above explanations sufficiently address the reviewer’s concern."}, {"round": 2, "reviewer_comments": {"reviewer_1": "I would like to thank the aurthos for their thorough responses to my concerns and questions, as they have fully addressed all\nthe issues I raised. In the revision manuscript, the author have included extensive additional experimental details. Moreover,\nthey validated the affinity of the top-ranked candidate through web-lab testing, which is particularly impressive to see in an\nAI-forcused paper. Given these improvments, I believe the manuscript is now suitable for acceptance.\n(Remarks on code availability)", "reviewer_3": "Q1: Regarding Supplementary Fig. 5, I am quite confused about why adding layers to Frad results in a significant\nperformance drop on MPP. Please provide a reasonable discussion to explain this phenomenon. Typically, simply\nincreasing model parameters without tuning the default training recipe leads to poor performance. To ensure a fair\ncomparison, a reasonable hyperparameter search should be conducted for Frad. Otherwise, the current results are not\nconvincing.\nQ2: Regarding AF3, I agree that the objectives of EPT and AF3 diverge. However, my intention in raising this concern was\nto encourage a more comprehensive literature review on the unification of tasks for atomic systems—not just in\nrepresentation learning, but also in broader understanding and generation tasks (e.g., [1,2,3]). Please include a discussion\nbased on a more thorough literature review to better contextualize your contribution (not just AF3).\n[1] Xia, Yingce, et al. \"Nature Language Model: Deciphering the Language of Nature for Scientific Discovery.\" arXiv preprint\narXiv:2502.07527 (2025).\n[2] Zhang, Gongbo, et al. \"Unigenx: Unified generation of sequence and structure with autoregressive diffusion.\" arXiv\npreprint arXiv:2503.06687 (2025).\n[3] Lu, Shuqi, et al. “Uni-3dar: Unified 3D generation and understanding via autoregression on compressed spatial tokens.”\narXiv preprint arXiv:2503.16278 (2025).\nQ3 & Q4: Thanks for your revision.\nQ5: While I understand the distinction between Spearman and Pearson, a more detailed\ndiscussion on the difference between RMSE and Pearson should be provided. Both metrics evaluate the error between predicted and ground-truth\nbinding affinities. So why use the two metrics for this task? Are there specific benefits, or is this choice simply based on\nprecedent in existing work?\nQ6: Thank you for including the experiment demonstrating extensibility to nucleic acids. However, since the block definition\nis based on domain knowledge, this should be noted as a limitation in the paper. EPT is not a “free lunch” when it comes to\ngeneralizing to new atomic systems. This manual effort should be acknowledged, as some systems may not have easily\ndefinable blocks—this constitutes a limitation of the method.\nQ7: I agree that this paper makes meaningful technical contributions. However, in the\nresponse to Q7, it was stated: “We highlight three synergistic innovations…” but four\ninnovations were listed. This appears to be a typo and should be corrected.\nOverall Comments: Thank you to the authors for their hard work in addressing many of my concerns. Some issues have\nbeen satisfactorily resolved. However, several important concerns remain unaddressed, as outlined above.\n(Remarks on code availability)", "reviewer_4": "The manuscript entitled “An Equivariant Pretrained Transformer for Unified 3D Molecular Representation Learning” written\nby Jiao et al. reported the development of Equivariant Pretrained Transformer (EPT), an atom-based pre-trained model\ncapable of predicting ligand binding affinity with a performance reportedly better than previous state-of-the-art methods. To\ndemonstrate the potential of EPT, they screened FDA-approved drugs against SARS-CoV-2 3CL protease, where they\nidentified both known 3CL protease inhibitors and other novel candidates as their top-ranking hits. For validation, in the\nlatest revision of this manuscript, they acquired top candidate Ac-Leu-Leu-Nle-CHO and experimentally evaluated it in a\nbiochemical assay against SARS-CoV-2 3CL protease.\nRegarding their wet-lab experiments, the identity and purity of the Ac-Leu-Leu-Nle-CHO compound that they purchased from\na commercial vendor (MedChemExpress) was convincingly demonstrated by the mass spec and NMR spectra. The\nbiochemical assay was then conducted by a separate CRO (Taoshu BioScience) using recombinant 3CL protease and a\nfluorogenic substrate, which is standard and typical for evaluating compounds as 3CL protease inhibitor. But I would\nrecommend the authors to further strengthen their manuscript by addressing the following points as in a minor revision:\n1. A known 3CL protease inhibitor (for example nirmatrelvir) should be included in the same assay as a positive inhibitor\ncontrol to validate the assay setup and also benchmark the relative potency of Ac-Leu-Leu-Nle-CHO, as IC50 values may\nvary with the assay conditions. The CRO should already have data in this regard as they establish and optimize the assay.\nAlternatively, the authors can cite IC50 values of known 3CL inhibitors from paper that also used this CRO for the same\nbiochemical assay.\n2. 3CL protease inhibitor Nirmatrelvir exhibited an IC50 value of 10 nM in a similar biochemical assay against SARS-CoV-2\n3CL protease. But it was only ranked 173th by the EPT. In contrast, IC50 value of their top-ranking candidate Ac-Leu-Leu-\nNle-CHO is 5.47 µM. So Nirmatrelvir is >500-fold more potent than Ac-Leu-Leu-Nle-CHO. The authors should provide\ndiscussion to explain this mismatch to help readers understand what the true capability of this model is, as well as its\npotential limitations.\nA minor typo is in the abstract that IC50 is 5.47 M.\n(Remarks on code availability)"}, "author_response": "Point-by-Point Response to Reviewer Comments\nWe thank all the reviewers for their insightful feedback and constructive comments during\nthis round of review. Here we provide detailed responses to address the proposed concerns\nand revise our paper accordingly. The reviewers’ comments are written in blue, our\nresponses are in black, and the changes to the manuscripts are in red.\nReferee #1\nI would like to thank the authors for their thorough responses to my concerns and questions,\nas they have fully addressed all the issues I raised. In the revision manuscript, the author\nhave included extensive additional experimental details. Moreover, they validated the affinity\nof the top-ranked candidate through web-lab testing, which is particularly impressive to see\nin an AI-forcused paper. Given these improvements, I believe the manuscript is now suitable\nfor acceptance.\nWe sincerely thank the reviewer for the thoughtful and encouraging comments. We\nappreciate the recognition of the expanded experimental details and the wet-lab validations.\nWe are grateful for the time and constructive feedback that helped strengthen the\nmanuscript.\nReferee #3\nQ1: Regarding Supplementary Fig. 5, I am quite confused about why adding layers to Frad\nresults in a significant performance drop on MPP. Please provide a reasonable discussion to\nexplain this phenomenon. Typically, simply increasing model parameters without tuning the\ndefault training recipe leads to poor performance. To ensure a fair comparison, a reasonable\nhyperparameter search should be conducted for Frad. Otherwise, the current results are not\nconvincing.\nWe thank the reviewer for raising this concern. To address it, we additionally performed a\ntargeted hyperparameter search for Frad by sweeping learning rates across\n{1e-4, 5e-5, 1e-5} and reporting the best result. We included more clarification on this setting\nin Supplementary 1.7. This alleviated, but did not eliminate the performance drop observed\non MPP as depth increases. The results are updated in Supplementary Fig. 5 and also\nprovided as below. We believe this degradation is tied to the TorchMD-Net backbone used\nby Frad. For TorchMD-Net, vector features are initialized to zero and gradually updated via\ngated interactions with scalar filters and edge directions. Such zero initialization could\npromote the collapse of vector features across nodes as depth increases, owing to the\nover-smoothing issue in message passing neural networks [1]. In contrast, our backbone\nexplicitly initializes non-zero equivariant vectors at the embedding stage via\nneighbor-weighted positional differences (Eq. 8), so anisotropic cues exist from the first\nlayer. We also fuse scalar and vector features with a GVP-style FFN and a layer-normalized\ngate (Eqs. 13–15), which stabilizes vector norms and curbs accumulation or bleed-through\nof low-contrast signals across many layers. Consistent with this mechanism, EPT scales\nmore robustly with the number of layers.\n[1] Rusch, T. Konstantin, Michael M. Bronstein, and Siddhartha Mishra. \"A Survey on\nOversmoothing in Graph Neural Networks.\" SAM Research Report 2023 (2023).\nQ2: Regarding AF3, I agree that the objectives of EPT and AF3 diverge. However, my\nintention in raising this concern was to encourage a more comprehensive literature review\non the unification of tasks for atomic systems—not just in representation learning, but also in\nbroader understanding and generation tasks (e.g., [1,2,3]). Please include a discussion\nbased on a more thorough literature review to better contextualize your contribution (not just\nAF3).\n[1] Xia, Yingce, et al. \"Nature Language Model: Deciphering the Language of Nature for Scientific Discovery.\" arXiv preprint\narXiv:2502.07527 (2025).\n[2] Zhang, Gongbo, et al. \"Unigenx: Unified generation of sequence and structure with autoregressive diffusion.\" arXiv\npreprint arXiv:2503.06687 (2025).\n[3] Lu, Shuqi, et al. “Uni-3dar: Unified 3D generation and understanding via autoregression on compressed spatial tokens.”\narXiv preprint arXiv:2503.16278 (2025).\nWe appreciate the reviewer’s suggestion to broaden the contextualization of unified\nmodeling for atomic systems beyond AF3. In the revision, we added a dedicated discussion\nin the Supplementary 1.11 to survey three complementary directions, including\nstructure-aware generalist encoders, unified 3D generative frameworks, and sequence-level\nfoundation models. We also rewrote Line 308-316 in the discussion section with a concise\nsummary positioning EPT within this broader landscape.\nQ5: While I understand the distinction between Spearman and Pearson, a more detailed\ndiscussion on the difference between RMSE and Pearson should be provided. Both metrics\nevaluate the error between predicted and ground-truth binding affinities. So why use the two\nmetrics for this task? Are there specific benefits, or is this choice simply based on\nprecedent in existing work?\nWe thank the reviewer for this valuable suggestion. The two metrics capture complementary\naspects of performance. RMSE measures absolute calibration error in affinity and thus\ndirectly reflects how accurate the predicted magnitudes are. In contrast, Pearson\n(Correlation Coefficient) quantifies linear association and is invariant to affine\ntransformations of the predictions. Therefore, it is insensitive to systematic offsets or scale\nmismatches. Reporting these two metrics allows us to measure both linear ranking\n(Pearson) and scale fidelity (RMSE), which are both relevant for binding-affinity applications.\nPlus, our choice also follows established practice in existing benchmarks and prior work [1],\nso we keep both metrics for comparability. We have highlighted the reference in Line 432.\n[1] Townshend, Raphael John Lamarre, et al. \"ATOM3D: Tasks on Molecules in Three\nDimensions.\" Thirty-fifth Conference on Neural Information Processing Systems Datasets\nand Benchmarks Track.\nQ6: Thank you for including the experiment demonstrating extensibility to nucleic acids. However, since the block definition\nis based on domain knowledge, this should be noted as a limitation in the paper. EPT is not a “free lunch” when it comes to\ngeneralizing to new atomic systems. This manual effort should be acknowledged, as some systems may not have easily\ndefinable blocks—this constitutes a limitation of the method.\nWe appreciate the reviewer’s constructive point and agree that EPT is currently not a “free\nlunch” when extending to new atomic systems. Our block-level denoising assumes an\nexternally pre-defined decomposition of atoms into blocks, such as residues for proteins or\nphosphate–sugar–base units for nucleic acids, which depends on domain knowledge. We\nhave made this limitation explicit in Line 319-322 and also outlined future directions such as\ndata-driven block discovery that could reduce manual effort in domains where block\ndefinitions are ambiguous.\nQ7: I agree that this paper makes meaningful technical contributions. However, in the\nresponse to Q7, it was stated: “We highlight three synergistic innovations...” but four\ninnovations were listed. This appears to be a typo and should be corrected.\nWe thank the reviewer for catching the inconsistency in our previous response. This was a\ntypo that we meant to state “four synergistic innovations”, and the list indeed contained four\nitems. We apologize for the confusion.\nReferee #4\n1. A known 3CL protease inhibitor (for example nirmatrelvir) should be included in the same\nassay as a positive inhibitor control to validate the assay setup and also benchmark the\nrelative potency of Ac-Leu-Leu-Nle-CHO, as IC50 values may vary with the assay\nconditions. The CRO should already have data in this regard as they establish and optimize\nthe assay. Alternatively, the authors can cite IC50 values of known 3CL inhibitors from paper\nthat also used this CRO for the same biochemical assay.\nWe thank the reviewer for this important suggestion, and apologize for missing the report of\nthe positive control in our previous response. The CRO included GC376 (ranked #82 by\nEPT) as a positive inhibitor control under the same assay conditions. Using a 4-parameter\nlogistic (4PL) fit to the dose–response curves, the absolute IC50 values were 9.54 nM for\nGC376, aligning well with previous reported value of 30 nM [1] and thus showcasing the\nreliability of the experimental settings. These control data validate the assay setup and\nprovide a potency benchmark for our candidate. We have added the curves to the\nSupplementary Fig. 4, which is also shown as below.\n[1] Ma, Chunlong, et al. \"Boceprevir, GC-376, and calpain inhibitors II, XII inhibit\nSARS-CoV-2 viral replication by targeting the viral main protease.\" Cell research 30.8\n(2020): 678-692.\n2. 3CL protease inhibitor Nirmatrelvir exhibited an IC50 value of 10 nM in a similar\nbiochemical assay against SARS-CoV-2 3CL protease. But it was only ranked 173th by the\nEPT. In contrast, IC50 value of their top-ranking candidate Ac-Leu-Leu- Nle-CHO is 5.47 μM.\nSo Nirmatrelvir is >500-fold more potent than Ac-Leu-Leu-Nle-CHO. The authors should\nprovide discussion to explain this mismatch to help readers understand what the true\ncapability of this model is, as well as its potential limitations.\nWe appreciate the reviewer’s comment. Our screening pipeline relies on docked complexes.\nTarget pockets and candidate ligands are first docked with Glide, and EPT then re-scores\nthese complexes. Empirically, EPT ranks most truly active compounds near the top.\nNevertheless, the docking software is not always accurate, while high-fidelity affinity\nprediction requires precise structures. Consequently, the ranking cannot guarantee that the\nmost potent ligand will always appear at the very top. We further evaluated undervalued\nmolecules in Supplementary Fig. 1, and found that Nirmatrelvir (EPT rank #173) and\nLufotrelvir (EPT rank #176) have Glide ranks #218 and #203, respectively. EPT partially\ncorrects Glide’s ordering but cannot fully recover from suboptimal inputs. We present this as\na pipeline-based limitation, and note that improvements in the upstream docking software\nare expected to enhance the final performance. We thank the reviewer for this valuable\nsuggestion once again, and have included the above discussion in Supplementary 1.4.1.\nA minor typo is in the abstract that IC50 is 5.47 M.\nThanks for pointing this out! We have corrected this typo."}, {"round": 3, "reviewer_comments": {"reviewer_3": "I would recommend accept after very minor edits. The technical clarifications and new analyses directly address my earlier\nconcerns. If the authors can incorporate the (optional) numeric ablation deltas for Frad’s LR sweep in Supp. §1.7, that would\nfurther strengthen transparency (currently only the best results are recorded), but I would not hold the paper for this. Also in\nthe Discussion, consider fixing a small grammatical glitch (“*it block-aware, E(3)-equivariance nature…*”) to “**its** block-\naware, E(3)-equivariant nature…”.\n(Remarks on code availability)", "reviewer_4": "I appreciate the efforts that the authors put together to further improve their manuscript. They have fully addressed all the\nminor points that I raised and have strengthened their manuscript accordingly. Therefore, I am more than glad to recommend\npublication of this manuscript. Congratulations!\n(Remarks on code availability)"}, "author_response": ""}]

Decision letter