Large Language-Geometry Model: When LLM meets Equivariance
Authors: Zongzhao Li
Jiacheng Cen
Bing Su
Tingyang Xu
Yu Rong
Deli Zhao
Wenbing Huang
Venue: ICML 2025 poster (2025)
Keywords: Equivariance
Graph Neural Networks
Large Language Models
Primary area: deep_learning->graph_neural_networks

Abstract: Accurately predicting 3D structures and dynamics of physical systems is crucial in scientific applications. Existing approaches that rely on geometric Graph Neural Networks (GNNs) effectively enforce $\mathrm{E}(3)$-equivariance, but they often fail in leveraging extensive broader information. While direct application of Large Language Models (LLMs) can incorporate external knowledge, they lack the capability for spatial reasoning with guaranteed equivariance. In this paper, we propose EquiLLM, a novel framework for representing 3D physical systems that seamlessly integrates $\mathrm{E}(3)$-equivariance with LLM capabilities. Specifically, EquiLLM comprises four key components: geometry-aware prompting, an equivariant encoder, an LLM, and an equivariant adapter. Essentially, the LLM guided by the instructive prompt serves as a sophisticated invariant feature processor, while 3D directional information is exclusively handled by the equivariant encoder and adapter modules. Experimental results demonstrate that EquiLLM delivers significant improvements over previous methods across molecular dynamics simulation, human motion simulation, and antibody design, highlighting its promising generalizability.

Large Leomuay Model: When LLM meets Equivariance
Zongzhao Li Jiacheng Cen 123 Bing Su 123
Tingyang Xu Yu Rong 45 Deli Zhao 45
Wenbing Huang 123
Abstract
sign (Tiller & Tessier, 2015) require not only deep un- derstanding of complex spatial geometry but also the preser- vation of E(3)--quivariaace ensuring predictions trans- form correspondingly with input rotations, reflections and translations (Batzner et al., 2022; Huang et al., 2022). From the machine learning perspective, Equuvvaaraii models are more powerful than their equivariiin counterparts, as they are inherently generalizable across arbitrary coordi- nate systems when modeling physical systems. To achieve equivariance, current approaches primarily rely on geomet- ric Graph Neural Networks (GNNs) (Wu et al., 2024; Kong et al., 2022; Li et al., 2025a). Despite their fruitful progress, these models often lack the ability to leverage external do- main knowledge and broader contextual information, such as task-specific instructions and expert-curated guidance. hindering further performance enhancement.
Accurately predicting 3D structures and dynam- ies of physical systems is crucial in scientific applications. Existing approaches that rely on geometric Graph Neural Networks (GNNs) ef-
fectively enforce E(3) equuivarrva,,,,,,, but they of- ten fail in leveraging extensive broader informa- tion. While direct application of Large Language Models (LLMs) can incorporate external knowl- edge, they lack the capability for spatial reasoning with guaranteed equivariance. In this paper, we propose EquiLLM, novel framework for repre- senting 3D physical systems that seamlessly inte- grates F((3)--quivariance with LLM capabilities. Specifically, EquiLLM comprises four key com- ponents: geometry-avee aware prompting, an equivari- ant encoder, an LLM, and an equivariant adapter. Essentially, the LLM guided by the instructive prompt serves as a sophisticated invariant feature processor, while 3D directional information is ex- clusively handled by the equivariant encoder and adapter modules. Experimental results demon- strate that EquiL.L delivers significant improve- ments over previous methods across molecular dynamics simulation, human motion simulation, and antibody design, highlighting its promising generalizability.
Recently, Large Language Models (LLMs) have demon- strated remarkable success across a wide range of applica- tions, owing to their large-scale pretraining on extensive datasets and their substantial model size (Sun et al., 2025a; Li et al., 2025c; Yang et al., 2025a; Sun et al., 2025b; Bian et al., 2025; Liu al., 2025b; Li et al., 2025b; Xu al., 2025; Sun et al., 2025e; Liang et al., 2025; Yang al., 2025b). It is well known that L.MMs can not only understand and generate text but also excel at integrating and leverag- ing scientific knowledge (Liu et al., 2025a; Jablonka et al., 2024; Wang et al., 2023b). For instance, LLMs can compre- hend fundamental chemical concepts and molecular struc- tural characteristics (Guo et al., 2023). More significantly, based on our results, speculate that L.MMs flexibility in prompt engineering enables the development of tailored instructions that better leverage their capabilities, producing outputs more precisely suited to the task.
1 Introduction
Accurately predicting 3D structures/ynnamis of physical systems remains a fundamental challenge in physics and biology. Typical tasks such as molecular dynamics sim- ulation (Hollingsworth & Dror, 2018) and antibody de-
A natural idea is to directly employ LLMs for modeling 3D physical systems. However, this approach fails to yield satisfactory results in practice. A key limitation is that LLMS trained process ordered and discrete text tokens, restricting their ability to directly comprehend unordered and continuous data in 3D space. One possible solution is to adapt existing multimodal L.MM architectures, such as LLaVA (Liu et al., 2024b), by treating 3D structures as a sep- arate modality and simply replacing the image encoder with geometric GNN. However, this naive adaptation fails to satisfy the -q3uvaauaae requirement. Since geometric
"aooling School of Artificial Intelligence, Renmin University of China 'Beijing Key Laboratory of Research on Large Mod- els and Intelligent Governance 'Engineering Research Center of Bext-Generatio Intelligent Search and Recommendation, MOE DDAMO Academy, Alibaba Group, Hangzhou, China Hupan Lab, Hangzhou, China. Correspondence to: Wenbing Huang <hwen- bing@1 126.com>, Yu Rong <yu.rong @ hom..
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Large Geomery Model: When LLM meets Equivariance
GNNs produce both invariant features and equivariant coor- dinates, passing these outputs through an L.M inevitably compromises equivariance. Although the canonicalization approach (Puny et al., 2022; Mondal et al., 2023) enables the application of non-equivariant LLM to equivariant tasks, our experiments empirically show that the resulting perfor- mance is still suboptimal. Therefore, it is non-trivial to integrate the strengths of both LLMs and geometric GNNs while maintaining essential geometric properties and ensur- ing strong performance.
2 Related Work
Geometric GNNs have achieved sig- nificant success across wide range of scientific applica- tions by leveraging physical symmetries in 3D space (Han et al., 2024; Xu et al., 2024; Wang et al., 2025a; Li et al., 2024d;e;c; Yan et al., 2025; Wu et al., 2025; Lin et al., 2025; Han et al., 2025; Liu et al., 2025c; Wang et al., 2025b), Among them, tensor-product-basee models (Thomas et al., 2018: Fuchs et al., 2020; Brandstetter et al., 2021; Bata- tia et al., 2022; An et al., 2025; Xie et al., 2025) excel at capturing interactions between steerable features of differ- ent degrees but are computationally expensive. In contrast, eeclarization-basee models (Satorras et al., 2021; Schütt et al., 2021; Huang et al., 2022; Aykent & Xia, 2023; Zhang et al., 2024; 2025) focus on constructing invariant scalars (e.g. norms and inner products), which offer both effi- ciency and expressiveness. This approach has been further extended by spherical scalaiizaanal methods (Frank et al., 2024; Cen et al., 2024; Aykent & Xia, 2025). Well-designed scalar features have also proven effective in improving per- formance in applications (Wang et al., 2023a; Zhou et al., 2023; Wang et al., 2024; Battiloro et al., 2024; Yue et al., 2024; 2025). For dynamics simulations, ESTAG (Wu et al., 2024) enhances trajectory prediction using frequency cross- correlations, while SEGNO (Liu al., 2024c) reduces roll- out errors by integrating sealars into neural operator learning. In biological modeling, MEAN (Kong al., 2022) improves antibody representation via a multi-channel scalar attention mechanism, and GeoAB (Lin et al., 2024) generalizes this to capture higher-order atomic interactions. Motivated by these successes, we investigate whether incorporating LLMs into scalar design can further enhance geometric models.
Geometrie GNNs.
To this end, this paper introduces EquiLLM, a novel frame- work for representing 3D physical systems that seamlessly integrates E(3) equivariance with LLM capabilities. Equi- LLM is carefully designed and comprises four core modules (see Figure I): geometry-aware prompting, an equivariant encoder, LLM, and an equivariant adapter. A key insight of Eq in maintaining equivariance lies in its inno- vative design: the LLM guided by the instructive prompt serves as sophisticated invariant feature processor, while 3D directional information is exclusively handled by the equivariant encoder and adapter modules. Specifically, to fully activate the spatial reasoning capabilities of LLMs, introduce the geometry-aware prompts containing in- variant geometric information, including task description, input feature description and statistical information. Then, EquiLLM employs a geometric GNN as a domain-specific encoder to effectively model and extract 3D representa- tions of input systems, Subsequently, to satisfy the equiv- ariance constraint, the input to the LLM are strictly limited to the prompt and invariant features derived from the equiv- ariant encoder's ' output. The LLM-generated outputs are subsequently combined with the 3D equivariant vectors produced by the equivariant encoder and fed into an equiv- ariant adapter for information fusion. EquiLLM ultimately produces both invariant labels and equivariant coordinates required by downstream applications.
LLM+ GNN. LLMs with rich knowledge are being widely transferred and applied across multiple domains to enhance model capabilities (Singhal et al., 2023; Song et al., 2025; Wang et al., 2025c; Singhal et al., 2025). Numerous excel- lent works have emerged in combining GNNs with LLMs for scientific applications. ChemL.L (Guo et al., 2023) tests LLM''s understanding, reasoning, and explain- ing capabilities on various chemical tasks using in-context learning. Prot2Text (Abdine et al., 2024) integrates protein sequence, structure, and textual annotations into an encoder- decoder framework composed of GNN and LLM to predict protein functions. MoleculeSTy, (Liu et al., 2023a) uses a contrastive learning paradigm to align molecular graphs and textual descriptions in the semantic space, thereby learning better feature representations. MolCA (Liu et al., 2023b) em- ploys Q-Former (Li et al., 2023) as a cross-modal projector to align the feature spaces of graph encoder and language encoder, enhancing performance in molecule captioning tasks. The aforementioned methods enhance interactions between GNNs and LLMs through various paradigms and yield promising results. However, exploring such LLM-
To sum up, our main contributions threefold:
To the best of our knowledge, we present the first inves- tigation into modeling 3D physical systems by integrat- ing LLMs with geometric GNNs, aiming to combine the strengths of both approaches.
We present EquiLLM, novel framework that is metic- ulously designed to permit Equivaurveee and in- still 3D spatial reasoning into LLMs' powerful capa- bilities.
We conduct extensive experiments on diverse tasks of molecular dynamics simulation, human motion sim- ulation and antibody design. The results show that our method achieves superior performance, attaining state-of-the-an results in nearly all metrics.
2

---

Large Geomery Model: When LLM meets Equivariance
GNN integration paradigms for 3D structural data tasks, such as 3D structure generation and dynamic trajectory simulation, remains relatively unexplored frontier. The EquiLLM framework we propose in this paper integrates LLMMs with Geometric GNNs that embed spatial symmetry constraints and has been effectively validated across datasets from both physical and biological domains.
and H E RRXx to denote the sets of node coordinates and invariant features of the geometric graph g.
Task Formulation. Here, we provide a general form of our task and will elaborate specific applications including dynamic simulation and antibody design in $ 3.3 and 3.4. Given the input geometric graph 9 our goal is to find a function o to predict the output gour This process can be formally delineated as:
3D Structure Tokenization. As a critical step in methods employing language models for structural tasks, the tok- enization of 3D structures has become an actively advanc- ing research frontier, with numerous studies demonstrating its successful implementation. ESM3 (Hayes et al., 2025) encodes 3D atomic structures into discrete tokens, enabling seamless integration with sequences and functions that are similarly tokenized. This unified representation allows for coherent structural reasoning within shared latent space. ProSST (Li et al., 2024a) tokenizes 3D structures by first se- rializing them into local structures at the residue level before encoding them into dense vector space. BindGPT (Zholus et al., 2025) employs an XYZ representation for structures, where the 3D coordinates of each atom are expressed as tex- tual entries per line. Geo2Seq (Li al., 2024b) converts 3D structures into ID discrete sequences through canonical la- beling and invariant spherical representations. CHEAP (Lu et al., 2024) utilizes FSQ (Mentzer et al., 2024) to encode 3D structures into a quantized latent space. However, such tokenization approaches may inherently carry risks of in- formation loss. To address this, EquiLLM employs a dual- pathway architecture where the Geometric GNN processes 3D structure while the L.M handles discrete textual tokens, thereby preserving information integrity.
g = "G"").
(I)
Meanwhile, since we introduce LLMs into our framework, we will further construct task-specific prompts to guide the extraction of relevant domain knowledge from L.LMs, recasting our task as:
Gour = *(gin ,),
where P denotes the prompt.
Equivariance. It is crucial to emphasize that in the tasks above, the function e must satisfy E(3) symmetries of phys- ical laws (Han et al., 2024). Specifically, if arbitrary trans- lations, reflections, or rotations coordinate matrix X the output coordinate matrix X should undergo the corresponding transformation.
applied to the input
3.2 Large Saanguagg-Geometry Model
In this section, we provide meticulous description of our model EquiLLM, which consists of three main components: Equivariant Encoder, LLM, and Equivariant Adapter. Un- like existing works (Gruver et al., 2024) that applys an LLM to predict the 3D coordinates directly, EquiLLM leverages an L.M to acquire broader scientific domain knowledge while employing geometric GNNs for precise modeling of 3D structures. These two components seamlessly integrated through an equivariant adapter, achieving supe- rior predictive performance without compromising E(3)- equivariance.
3 Method
In this section, we first introduce the preliminaries related to geometric modeling in $ 3.1. Next, in $ 3.2, we present the proposed framework Eq Finally, in $ 3.3 and 3.4, we describe how EquiLLM is applied to two representative tasks (e.g. dynamic simulation and antibody design). The overview of our EquiLLM is illustrated in Fig. 1.
Equivariant Encoder. The Equivariant Encoder is a domain specific equivariant model, which can be any suit- able equivariant model from the relevant field. The model takes the graph Gin (>²T g input, performing ini- tial encoding and embedding of geometric information, and outputs a processed geometric graph g' = =V",'", This process can be formally defined as:
3.1 Preliminaries, Notations and Definitions
Physical systems (such as molecules) can be naturally mod- eled with geometric graphs. We represent each static system as a geometric graph G = EV,E), where each node U2 in V is associated with an invariant feature h; E R (e.g. atom type) and an 3D equivariant vector H. E R3 (e.g. atom coordinates); each edge (e.g. chemical bonds) denotes the connectivity between nodes. Apart from modeling static systems, we explore dynamic systems, focusing on con- structing geometric graphs across different time steps. The details will be thoroughly discussed in $ 3.3 and 3.4. In the following sections, we use the matrices X E Nx3
G' = (G
(3)
where de can be any equivariant model, used jointly model the geometric relationships between Xin and Hi features across different nodes, resulting in processed fea- tures X and H'
Since LLMs are not naturally equivariant, directly feed- ing X' into an LLM would likely undermine the intrinsic
3

---

Large Language-Geometry Model: When LLM meets Equivariance
Geometric Graph SIV.E)
EquiLLM
Applications on Physics and Biology
Ma X
1D Sequence
3D Structure X .
X*
Equivariant Adapter
H
H
Geometric-aware Prompt
Commoognn the
Large Language Model
Antihady Design
Human Motion Simalation
Task description: Predict 1D seq and 3D strc...
P
Tokenizer & Embedder
Projector
Object feature description:
Heavy chain of...
Object statistical information: Dist between centroids of...
Equivariant Uncoder
Gioumetric-wware Prompt
9t
Sr
Molecular Dynamies Simmalation
Figure 1: The overall framework of EquiLLM. Given geometric graph G = (V,E) input, EquiLLM initially employs an Equivariant Encoder derive processed features X and H' The features H are first projected through a projector, then concatenated with prompt features P in task-specific manner. This concatenated vector is subsequently fed into an LLM. The output features Hilm from the LLM, alongside the previously obtained processed features X and H' are then passed into an Equivariant Adapter. The Equivariant Adapter then generates the final outputs, including the vector Xour for cquivariant tasks and the feature H for invariant tasks. The blue module means the invariant module, while the purple module means the equivariant module.
equivariance of the overall architecture. Thus, in contrast to existing works, we convey the invariant features H' to the L.M,, but pass the equivariant matrix X' to the subsequent Equivariant Adapter via a skip connection. Before feeding H' to the LLM, we first conduct projector on H' align its dimension with the input space of the LLM. This process can be formally characterized as:
of the task by the LLM.
D Object feature description. The feature description of the input object begins with <Object> and primarily out- lines the composition information as well as the structural characteristics of the input object.
Object statistical information. This components starts with <Statistics>, encapsulating detailed metrics pertaining to the distribution of the object's coordinates in 3D space, including the maximum, minimum, and mean values. is crucial to note that, unlike conventional tasks, directly incor- porating absolute coordinate values into the prompt is not recommended in 3D spatial modeling tasks. This due the fact that transformations such translation, reflection, or rotation applied to the input object will invariably alter the corresponding coordinate distribution, thereby violating the principle that the prompting process must remain E(3)- invariant. Consequently, we represent the coordinate distri- bution of the input object indirectly by computing statistical metrics related distances. Several viable alternatives exist (e.g. Principal Component Analysis (PCA) on the original coordinates), long as the computed statistical information preserves E(3)-invariance. The specific contents of the task prompts will be elucidated upon in $ 3.3 and 3.4.
Pro pproj (H',
(4)
where Oproj is implemented linear layer in EquiLLM.
Geometric-aware Prompt. One may directly input the aligned features H into the LLM to make the final predictions. However, this approach overlooks the piv- otal role of the prompt, as it does not utilize the linguis- tic form of the prompt to effectively harness the LLM's comprehension and articulation of the specific task hand. Therefore, in the EquiLLM framework, we carefully design task-specific prompts for different tasks to unleash domain- specific knowledge.
The prompt content for all tasks can be broadly divided into three key components: (1) task description, (2) object feature description, and (3) object statistical information.
The task description consists of two parts: <Task> and <Requirement>. <Task> appears at the beginning of the prompt, providing a succinct description of the task to help the LLM quickly identify the task's 's objective. <Requirement> is located in the main body the prompt and elaborates on the ouputt requirements and con- straints of the task, ensuring a comprehensive understanding
Task description.
Large Language Model (LLM), After designing the prompt, we employ the tokenizer and embedding layer of the LLM obtain the corresponding word embedding features, denoted as P. Subsequently, depending the specific task, concatenate P with the invariant features H in an appropriate way. The concatenation strategies for different

---

Large Geomery Model: When LLM meets Equivariance
tasks will be discussed in detail in 3.3 and 3.4. Next, the concatenated features are input into the LLMM with the aim of unlocking and leveraging the scientific knowledge embed- ded within the LLM to enhance the model's 's understanding and reasoning capabilities for the relevant tasks. Unlike previous works like CrystalLLM (Gruver et al., 2024) and UniST (Yuan et al., 2024) that require fine-tuning some layers within the LLM, resulting in significant computa- tional costs and time expenditure, the EquiLLM framework freezes all parameters of the LLM, eliminating the need for additional training. This process can be roughly represented as follows:
shown in Fig. 1. First, the equivariant GNN encoder extracts both equivariant and invariant features, but only the invari- ant features are fed into the LLM, unlike LLaVA where the LLM receives all encoder outputs. Then, after LLM processing, the output is concatenated with the encoder's equivariant features via a skip connection and passed to the equivariant adapter module to generate both equivariant and invariant predictions.
Our EquiLLM framework guarantees that the overall archi- tecture preserves the critical property of -equiaauie in 3D space while also avoiding the introduction of lengthy text context due to direct 3D coordinate input, which could severely affect the efficiency of training and inference. The rigorous mathematical proof of the framework's equivari- ance properties can be found in Appendix A. Moreover, compared with domain-specinn Equivariant Encoder, Equi- LLM introduces only two projection layers and a one-layer EGNN network, significantly reducing the additional train- ing parameters in comparison to the existing literature (Jin et al., 2024; Yuan et al., 2024). Finally, our EquiLLM framework demonstrates exceptional flexibility and can be applied to various geometric modeling tasks, showcasing its robustness and generalizability.
li pllm] = LLM(Concat (HFPm P)):
(5)
Equivariant Adapter: Upon obtaining the output from the LLM, we extract the part corresponding to the invariant features, denoted as Hilm While directly utilizing it for final predictions may be viable for invariant tasks (e.g. pre- dicting the energy of a molecular system), it is inadequate for equivariant tasks, where the core objective is to predict the 3D coordinates of objects. To address this challenge, we propose the Equivariant Adapter, which leverages one- layer EGNN (Satorras et al., 2021) to process H while minimizing the introduction of excessive additional parame- ters. We select EGNN for its simple architecture and robust performance, as it has been widely adopted in prior works. Specifically, we first employ a projection layer to re-project Hil back into the space corresponding to the invariant fea- tures H' and add with H', yielding the refined feature representation H" Then, both the equivariant coordinate features X' from Equivariant Encoder and the refined in- variant features H" are transmitted to the EGNN, yielding the output X and H The whole process is formally expressed as:
3.3 Applications on Dynamic Simulation
In this section, we will present a detailed discussion on the application of our EquiLLM in dynamic simulation.
While our model is applicable to the simulations of both molecular dynamics and human motions, we only illustrate molecular dynamics here and provide details on human motions in Appendix D. Given 3D coordinate trajectory of a physical system (e.g., molecules) over T frames X E X''XX33 along with the invariant features H E R xc. encoded by atomic numbers, the model aims to infer future trajectories X E R xxxx3 for F subsequent frames.
mij = PP (hJ, 155; || - =I,1),
h:u hτ + ph h EjEN()) maz))
(6)
Geometric-aware Prompt. Here, we will provide general overview of the contents encompassed within the prompt, with a more thorough exposition available in Appendix E.
xPth + + NO - EjEN(s) Pu (m.) (Zx -x))
where Pm Pas and 4hh denote Multi-Layer Perceptrons (MLPs), and (ii) refers to the set of neighboring nodes associated with the i-th node. Specifically, mij represents an E(3)- invvariat message transmitted from node j to node i, which is utilized to aggregate and refine the feature vector h via the function 4o.. Regarding the update of a',, the func- tion fo, is employed to compute a scalar yp,, (m..)) which subsequently multiplied by the difference , o to re- tain directional information, while incorporating residual connections to ensure translation equivariance.
Task description. The model is tasked with predicting the 3D coordinates (1r, 2) of heavy atoms for next F frames based on the information from the previous T frames.
Object feature description. For molecular systems, the emphasis is on compositional information and structural characteristics.
Object statistical information. Given that the intrin- sie properties of each element remain invariant throughout temporal evolution, performing frame-wise computation of coordinate statistics to construct the prompt and then con- catenating prompt features with invariant features at the node level would result in redundant information and in-
Comparison with LLaVA. Our method does not simply replace LLaVA's encoder with an equivariant GNN encoder, as that would compromise the framework's 's overall equivari- ance. Instead, EquiL.L introduces an innovative design, as
5

---

Large Geomery Model: When LLM meets Equivariance
troduce unnecessary computational complexity. Hence, we aggregate statistics across all T frames for all N elements. Specifically, we compute the centroid T -: No Et Et, from T X N 3D coordinates, followed by calculating the maximum, minimum, and mean distances from all T x Na coordinates to this central point. The final representation concatenates P and H along the temporal dimension.
complex consists of three chains containing hundreds of amino acids. Therefore, computing the statistical informa- tion for each amino acid individually may exceed the input token limit. To address this, we compute Ca atom statistics at both the chain-level and residue-level. At the chain-level, we first calculate the centroid coordinates for the light chain, heavy chain, and antigen chain and compute the distances between pairs of chains. Additionally, we calculate the distance between the two most distant amino acids within each chain, thereby constructing the corresponding prompt P.. At the residue-level, we calculate the distance from each amino acid to the centroid of its respective chain and compute the maximum, minimum, and mean distances, thus constructing the prompt P.. Finally, we concatenate P., Pr. and H along the amino acid sequence dimension.
Task Formulation and Training Objective. With GEr = {9. (X,, Hr, and Gr+1-++F == (G
(X,, 8) we provide the entire process as fol-
lows:
GT-IT+F P)
Let y denote the ground-truth 3D coordinates for the time period from T + 1 to T + F. we define the object function as L = + Σイ= meo Xour 821
T+t T+s) refers to
Task Formulation and Training Objective. With G (X, H)E,, where (X, H) == {(Z, h.) the entire process is delineated as follows:
the mean squared error (MSH).
3.4 Applications on Antibody Design
H X out = (r,
In this section, we will present detailed discussion on the application of our EquiLLM in antibody design.
(8)
you Softmax (H
Antibodies are Y-shaped proteins primarily responsible for recognizing and binding to specific antigens. Current re- search predominantly focuses on the variable region. The variable region is present in both the heavy and light chains of the antibody and can be further subdivided into the frame- Re-
where N denotes the number of residues in CDR-H3 re- gion; you and y denote the predicted distribution over all amino acid categories and the ground truth amino acid type; Xour and X denote the predicted 3D structure and the ground truth 3D structure of the CDR-H3 region, respec- tively. The loss function is defined as L Lueg XCstruct. where Lee = #e (3) .z²") denotes the cross entropy and Limber N Lmmber (X (Xo"tt ;x))) denotes the Huber loss (Huber, 1992); the is used to balance the two losses.
work region and three gions (CDRs). These six CDRs are critical in determin- ing the affinity between the antibody and antigen, with the CDR-H3 region on the heavy chain exhibiting the most pronounced variability. Consequently, the primary objec- tive of this paper is to predict the amino acid sequence and the 3D coordinates of the CDR-H3 region, given the antibody-antigen complexes excluding the CDR-H3 region. In antibody design task, each node in V associates with a trainable feature h; E R encoded by amino acid type and a matrix of 3D coordinates Z, E R4x3, We choose 4 backbone atoms {N, Ca., C, 03 to constitute Z,.
panglemenntrrty--eeemmiiin
4 Experiments
We validate the effectiveness of the proposed EquiLLM framework on two tasks from different domains: the dy- namic simulation in physics (6 4.1 and 4.2) and the antibody design in biology ($ 4.3). Furthermore, in $ 4.4, we conduct ablation studies and explore the contribution of each mod- ule. We also perform additional exploratory experiments in 4.5.
Geometric-aware Prompt. Here, we will provide general overview of the contents encompassed within the prompt, with a more thorough exposition to be presented in Ap- pendix E.2.
Datasets. In the dynamic simulation task, to demon- strate the broad applicability of our model across varying scales, we conduct experiments on two distinct datasets: the molecular-level MD17 (Chmiela et al., 2017) dataset and the macro-level Human Motion Capture (De la Torre et al., 2009) dataset. In order to expedite the dynamics simula- tions, implement sampling strategy based on previous research (Huang et al., 2022) to extract subset of trajecto- ries for the purposes of training, validation, and testing. This approach involves randomly selecting an initial point and then sampling 2 x T timestamps. The first T timestamps are utilized as input for the models, while the remaining T
> Task deseription. The model is tasked with predicting both the ID sequence and 3D coordinates of CDR-H3 re- gion.
Object feature description. The structural features of the light chain, heavy chain, and antigen, which are described individually.
b Object statistical information. In contrast to tasks in- volving dynamic simulation, where each physical object comprises only dozens of elements, the antibody-antigen
6

---

Large Language-Geomett Model:
When LLM meets Equivariance
from an input of 10 frames. EquiLLM (DST) and Equi- LLM (PCA) represent the utilization of distance-related and principal component analysis-related information as object statistical information, respectively. From the table, the following conclusions can be drawn: 1. The proposed EquiLLM framework achieves state-of-ther (SOTA) per- formance on all eight molecules, demonstrating its supe- riority; 2. Compared to our Equivariant Encoder model ESTAG, EquiLLM achieves a performance improvement of 5.41% % to 42.76% on six molecules, indicating that Equi- L.MM effectively leverages knowledge from L.MSs to en- hance the prediction of molecular dynamics trajectories; 3. We also tested the prediction capability of several leading and
(a) GPT-4o-mini (b) ESTAG
(c) EquiLLM
Figure 2: The visualization of the predicted structures across various methods on Toluene of the MD17 dataset, where the pink represents the ground-truth structure.
L.LMs, including GPT-4o-mini, 1e5mni-1.5-flash--lle DeepSeek- V3. Using the same prompt as EquiLLM, we provide the 3D coordinates of all atoms from the past 10 frames these LLMs, allowing it to predict the coordinates of all atoms in the following 10 frames. The result shows that these LLMs significantly underperforms most baseline methods in prediction accuracy, indicating its weaker capa- bility in directly predicting 3D coordinates. In contrast, our EquiLLM framework, by providing structured molecular descriptions and statistical constraints, enables the LLM to combine its pretrained knowledge with the specific task, thus significantly reduce the predicted MSE; 4. We fur- ther test pretrained models with canonicalization. On the MD17 dataset, we first subtract the mean from coordinates to ensure translational invariance, then perform SVD de- composition for rotational invariance, We directly feed this canonicalized data into GPT-4o-min mini,, with results shown in Row 8 of Table 1. The results demonstrate that while canon- icalization indeed improves model's 's predictive capability, there remains a remarkable performance gap compared to our Equil This suggests that direct prediction of 3D coordinates remains suboptimal for current LLMs.
timestamps represent the future states that the models need to predict.
Baselines and metrics. We evaluate EquiLLM with several baseline models, including traditional GNNs: ST_GNN (Gilmer et al., 2017) and STGCN (Yu et al., 2017), equivariant GNNs: ___TN (Thomas et al., 2018), (T_SE(3)-TH (Fuchs et al., 2020), ST EGNN (Satorras et al., 2021), and ESTAG (Wu et al., 2024). We also compare EquiLLM with existing LLMs, including GPT-4o- mini (OpenAL, 2024), Gemini-1.5 --tiii-----llah-attee (Team al., 2024), and DeepSeek-V? V3 (Liu al., 2024a). The models marked with "ST" are those we modified to handle multi- frame inputs by adding basic spatio-temporal aggregation, as done in (Wu et al., 2024). For evaluation, we calcu- late the Mean Squared Errors (MSEs) averaged across all predicted frames as the metric.
4.1 Molecular Dynamics
Implementation details. MD17 consists of time-evolving paths produced through molecular dynamics simulation for eight different small compounds (such as aspirin, benzene, and others). To ensure a fair comparison, all hyperparame- ters (e.g. learning rate, number of training epochs) are kept consistent across our model and all other baselines. Detailed information can be found in Appendix B. We utilize the ESTAG (Wu et al., 2024) as the Equivariant Encoder and GPT-2 (Radford et al., 2019) as the language model within our EquiLLM framework. ESTAG utilizes Equivariant Dis- crete Fourier Transform to extract periodic patterns, and further employs alternating equivariant spatial and temporal modules to enhance the modeling of physical dynamics. Un- less otherwise specified, in all experiments presented in this paper, the LLM module in EquiLLM remains frozen, while all other modules are learnable and trained from scratch.
4.2 Human Motion Simulation
Implementation details. The Human Motion Capture dataset contains human motion trajectory data across multi- ple scenes. We focus primarily on two sub-datasets: Subject #35 (Walk) and Subject #102 (Basketball). To ensure fair comparison, all hyperparameters (e.g. learning rate, number of training epochs) are kept consistent across our model and all other baselines. Detailed information is pro- vided in Appendix B.2. We utilize the ESTAG as the Equiv- ariant Encoder and GPT-2 as the language model within our EquiLLM framework.
Results. Table 2 presents a performance comparison of all models on the Walk and Baske tball datasets under settings requiring the prediction of 10, 15, and 20 frames, respectively. From the table, it is evident that EquiLLM framework achieves SOTA performance across all six set- tings, with a performance improvement ranging from 5.63%
Results. Table 1 presents the performance of all models on MD17 dataset under the setting of predicting 10 frames
A more powerful LLM may lead superior performance. Here we use GPT-2 for concept validation.
7

---

Large Lenouaye Model: When LLM meets Equivariance
Table 1: Predicted MSE (x10-9) on MD17 dataset.
Aspirin
Benzene
Ethanol
Malonaldehyde
Naphthalene
Salicylic
Toluene
Uracil
ST JNN
7.180
1.359200m
2.108
5.620=0018
2.397
2. 2.6464amm
2.233=0on
1.913+0012
ST TFN
+.389
1849
2.041
5.346-0000
3.555+0.110
5.728+om1s
2.979
4.272+auss
STGCN
21.08
654.7
7.102
32.87=00001
5,42
3.001
3.679
7.142
ST ST_SE())-Tr.
6.234
1.000
1.76510.001
5.277
3.256.10.01
4.737.10016
2.104_100
3.900
ST ST_EGNN
6.68220.300
1.48220
2.14511000
4.729=0.00
4.034:10.02
6.29610
2.88
20002
3.394
Equiformer
10.130
2.000
1.880
8.050
3.430
5.790
2,090
4.380
GPPT-4o-mini
13.070
9.581
5.011
9.910
35.155
10.627
8.132
9.762
GPT-44o-mii + canonicalization
11.783
3.055
4.512
8.916
8,263
9.751
6.364
8.989
17.347
7.586
8,871
15.495
15.188
17.978
15,426
16.935
DeepSeek-V3.
12.009
3.648
6.729
10,247
7.883
10.423
6,941
9.428
ESTAG
3.263110.66
0.09
1.090100000
2.046=00085
2.036110350
3.13410004
1.634
1.85221006
EquiLIM (DST)
2.391
1.031 0.732
20:00
1.671
1.453+0071
2.162
1.178=0:166
1.060+0
EquiLLM (PCA)
1.931
0.552
1.044++000
1.634=0.004
1.454
1.049 1.852
0.940+a.02
Table 2: Predicted MSE (x10-3) on Motion dataset.
Table 3: Results on RAbD benchmark.
Walk
Basketball
Method
AAR
TM 1
RMSD
Method
R=10
R=15
R=20
R=10
R=15
RosettaAD
22.50%
0,9435
5.52
ST_ WT_NNN
1.150amm
2.544
2.765
34.536ga11
278, 278.246_.19
LSTM
22.36%
ST TFN
9.584.015
20.667.1m
31.437 to
168
188.674
358.881 11
613.755_1116
C-LSTM
22.18%
ST_GCN
18.737
19.467
20.498=22s
275.
516.462e91se
+62
RefineGNN
29.79%
0.8303
7.55
ST_ ST(3)-TTT
5.248
10.869
20.999.are
77
390
C-RefineGNN
28.90%
0.8317
7.21
ST EGNN
2.867
g.
g.644
30.813gmmm
g2.963gaae
152,55 gg
GeoAB
36.43%
0.9836
1.79
ESTAG
0.709 sooes
1.877
g.
10.507
33.636
76.548
MEAN
36.77%
0,9812
1.81
EquiLLM
0.539
1.300
2.213
9,438
3
72.233
EquiLLM
38.97
0,9830
Table 4: Ablation studies (x10-3 3) on MD17 dataset.
Aspirin
Benzene
Ethanol
Malonaldehyde
Naphthalene
Salicylic
Toluene
Uracil
EE (ESTAG)
3.263
0.891
1.090
2.046
2.036
3.134
1.634
1.852
LLM (w/o Prompt) + EE
2.524
0.810
1.062
2.326
2.363
2.180
1.538
1.425
LLM + EE
3.083
0.773
1.128
2.100
2.899
2.581
1.756
1.418
w/o Prompt
3.671
0.860
1.092
2.479
2.837
2.193
1.941
1.542
w/o Object Feature
3.122
0.833
1.080
2.255
2.297
2.470
1.627
1.387
w/o Statistics
3.532
0.820
1.054
1.889
2.286
2.528
1.650
1.463
EquiLLM
2.391
0.732
1.031
1.671
1.453
2.162
1.178
1.060
to 36.11%. This demonstrates that EquiLLM effectively handles predictions over varying prediction lengths, exhibit- ing excellent robustness and generalization ability
MEAN. For test set, we selected 60 diverse complexes from the RAbD (Adolf-Bryfogle et al., 2018) dataset to evaluate the performance of different methods. Before starting the experiments, we remove samples from the training and val- idation sets that belong to the same cluster as the test set to prevent data leakage. More experimental results can be found in Appendix C.3.
4.3 Antibody Design
Following previous study MEAN (Kong et al., 2022), we selected complete selbody-antigen complexes from the SAbDab (Dunbar et al., 2014) dataset to construct the train- ing and validation sets. First, we performed clustering based on CDRs, grouping complexes with CDR sequence identity above 40% into the same cluster. Then, the training and validation sets were partitioned in the same manner as in
Baselines and metries. We compared our EquiLLM with seven methods, including RosettaAD (Adolf-Bryoogle et al., 2018), LSTM (Saka et al., 2021; Akbar et al., 2022), Re- fineGNN (Jin et al., 2022), MEAN (Kong et al., 2022), GeoAB (Lin et al., 2024), and two variants of LSTM and
8

---

Large Geomery Model: When LLM meets Equivariance
RefineGNN, C-LSTM and C-RefineGNN, which utilize the full contextual information. We use AAR and RMSD to reflect the recovery ratio of the CDR-H3 amino acid se- quence and the accuracy of the corresponding 3D structure prediction. Additionally, we employ the TM-ssore (Zhang & Skolnick, 2004; Xu & Zhang, 2010) to measure the global similarity between two protein structures. We utilize the MEAN (Kong et al.. 2022) as the Equivariant Encoder and GPT-2 the language model within our framework. De- tailed information can be found in Appendix B.3.
completely removing prompt or reducing its content leads to decline in performance. This observation reinforces our design philosophy: LLMs necessitate comprehensive information, including molecular deseriptions and statistical constraints, to fully utilize their knowledge integration and constraint reasoning capabilities. This, in turn, facilitates more accurate predictive guidance.
4.5 Further Exploratory Investigations
In this section, we conduct additional exploratory experi- ments to further investigate the EquiLLM framework. More exploratory investigations can be found in Appendix C.I.
Results. Table 3 presents the performance of all models on the RAbD dataset. It can be concluded from the ta- ble: 1. The proposed EquiLLM framework achieves the best performance in both AAR and RMSD metrics, with comparable results in the TM-score metric. The SOTA method GeoAB achieves superior performance in TM-score by incorporating more detailed geometric constraints, such as bond lengths, bond angles, and torsion angles, into the model, which enhances its overall structural prediction ca- pabilities; 2. Compared to the Equivariant Encoder model MEAN, Eq shows significant improvement across all metrics, demonstrating that EquiLLM successfully lever- ages LLM's knowledge integration and constraint-anddling abilities while effectively utilizing LLM's capacity for un- derstanding and reasoning ID sequences.
Following CrystalLLMM, we finetune Llama-7b (Touvron et al., 2023) on MD17. Due to token length limitations in our prediction task (predicting 10 frames), we select the smallest molecule, Ethanol (3 heavy atoms) for evaluation. We conduct three settings: (1) 500 samples (original setup in main experiments) trained for 10 epochs; (2) 30,000 samples trained for 1 epoch; (3) 30,000 samples trained for epoch with canonicalization. The results in Table 5 reveal that without canonicalization, 500-sample and 30.000-sample fine-tuned models perform poorly. lagging behind EquiLLM by two orders of magnitude. Remarkably, when we incorpo- rate canonicalization, the model's predictive performance improved by factor of 100, even surpassing GPT-4o-mini. This compelling result demonstrates that the combination of canonicalization with direct LLM fine-tuning is indeed promising and warrants further investigation.
4.4 Ablation Studies
In this section, we delve into the design of the EquiLLM framework, analyzing the impact of different architectural designs and prompt configurations on model performance. The experimental results are shown in Table 4, where EE represents the Equivariant Encoder. More ablation studies can be found in Appendix C.2.
Table 5: Results of fine-tuning Llama on MD17.
Setting 1
Setting 2
Setting 3
EquiLLM
Ethanol
460
457
4.446
1.031
Architecture Design. 1. The results in the second row indi- cate that processing raw features through the LLM before feeding them into the Equivariant Encoder, while omitting the Equivariant Adapter for a simpler architecture, yields a performance improvement. This finding validates that the LLM has a fundamental capability to process and integrate structured information effectively. 2. However, fully exploit the potential of the L.MM model, it is necessary to leverage prompts to capitalize on its strengths in text under- standing. Building upon the second-row model, we perform experiments by adding prompts, as the results shown in the third row. The results indicate a significant performance drop. We speculate that this is due to the large semantic space difference between the unprocessed raw feature H and the text features, which hampers the model's prediction capabilities. This suggests that LLM requires appropriate interface to harness its advantages. This led to the design of the current EquiLLM framework.
5 Conclusion
We present EquiLLM, a framework that synergizes the strengths of LLMs and geometric GNNs to address the dual challenges of E(3) -qqiivaiive and knowledge inte- gration in 3D physical system modeling. By introducing geometry-aware prompting and modular architecture that isolates invariant and equivariant processing, EquiLLM cir- cumvents inherent limitations of LLMs in spatial reasoning while enabling the infusion of domain-specific knowledge through flexible prompting strategies. The separation of roles-LLA LLMS as invariant feature processors and geomet- ric GNNs as directional information handlers- a principled approach to preserving symmetry constraints. In future work, we plan to explore optimal prompting strate- gies for better leveraging domain knowledge and extending this framework to broader scientific tasks. We hope the EquiLLM framework will serve as a valuable reference for applying L.LMs in scientific domains.
Prompt Design. 3. From Table 4, it is evident that either

---

Large Genmery Model: When LLM meets Equivariance
Acknowledgements
neural networks for fast and accurate force fields. Ad- vances in Neural Information Processing Systems, 35:
This work was jointly supported by the following projects: the National Natural Science Foundation of China (No. 62376276); Beijing Nova Program (No. 20230484278); the Fundamental Research Funds for the Central Universi- ties, and the Research Funds of Renmin University of China (23XNKJ19); Damo Academy (Hupan Laboratory) through Damo Academy (Hupan Laboratory) Innovative Research Program.
11423-11436, 2022.
Battiloro, C., Karaismailoglu, E., Tec, M., Dasoulas, G., Au- dirac, M., and Dominici, E (n) equivariant topologiaal neural networks. arXiv preprint arXiv:2405. 10.4405.5550 15429, 2024.
Batzner, S., Musaclian, A., Sun, L., Geiger, M., Mailoa, J. P., Kornbluth, M., Molinari, N., Smidt, T. E., and Kozinsky, B. (3)-eqiivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):2453, 2022.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Bian, T., Niu, Y., Yuan, C.,, Piao, C., Wu, B., Huang, L.-K., Rong. Y., Xu, T., Cheng, H., and Li, J. IBCireuit: Towards holistic circuit discovery with information bottleneck. In Forty-second International Conference Machine Learning, 2025.
References
Brandstetter, J., Hesselink, R., van der Pol, E., Bekkers, E. J., and Welling, M. Geometric and physical quantities im- prove (3) equivariant message passing. In International Conference on Learning Representations, 2021.
Abdine, H., Chatzianastasis, M., Bouyioukos, C., and Vazir- giannis, M. Prot2text: Multimodal protein's function generation with gnns and transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, number 10, pp. 10757-10765, 2024.
Cen, J., Li, A., Lin, N., Ren, Y., Wang, Z., and Huang, W. Are high-degree representations really unnecessary in equivariant graph neural networks? In Annual Conference Neural Information Processing Systems, 2024.
BoofflBryfogle, J., Kalyuzhniy, O., Kubitz, M., Weitzner, B. D., Hu, X., Adachi, Y., Schief, W. R., and Dumbrack Jr, R. L.. Rosettaantiboodyesigg (rabd): A general framework for computational antibody design. PLoS computational biology, 14(4):e 1006 12, 2018
Chmiela, S., Tkatchenko, A., Sauceda, H. E., Poltavsky, I., Schütt, K. T., and Müller, K.-R. Machine learning of ac- curate energy-conserving molecular force fields. Science 2017.
Akbar, R., Robert, P. A., Weber, C. R., Widrich, M., Frank, R., Pavlović, M., Scheffer, L.,, Chernigovskaya, M., Snap- kov, I., Slabodkin, A., et al. In silico proof of principle of machine learning-based antibody design unconstrained scale. In MAbs, volume 14, number pp. 2031482. Taylor & Francis, 2022.
advances,
De Torre, F., Hodgins, J., Bargteil, A., Martin, X., Macey, J., Collado, A., and Beltran, P. Guide to the carnegie mel- lon university multimodal activity (emu-mmac) database, 2009.
Dunbar, J., Krawczyk, K., Leem, J., Baker, T., Fuchs, A., Georges, G., Shi, J., and Deane, C. M. Sabdab: the structural antibody database. Nucleic acids research, 42 (D1)): 140-D1 46, 2014.
An, J., Lu, X., Qu, C., Shi, Y., Lin, P., Tang, Q., Xu, L., Cao, F, and Qi, Y. Equivariant spherical trans- former for efficient molecular modeling. arXiv preprint 2.Xi:25555.23080 2025.
Frank, J. T., Unke, O. T., Müller, K.-R., and Chmiela, S. A euclidean transformer for fast and stable machine learned force fields. Nature Communications, 15(1):6539, 2024.
Aykent, S. and Xia, T. Savenet: A scalable vector network for enhanced molecular representation learning. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Fuchs, F., Worrall, D., Fischer, V., and Welling, M. Se (3) transffsofo::::: 3d roto- translatiioon equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1981. 2020.
Aykent, S. and Xia, T. Gotennet: Rethinking efficient 3d equivariant graph neural networks. In The Thirteenth International Conference on Learning Representations, 2025.
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quantum chem- istry. In International Conference on Machine Learning, pp. 1263-1272, PMLR, 2017.
Batatia, I., Kovacs, D. P., Simm, G., Ortner, C., and Csányi, G. Mace: Higher order equivariant message passing
10

---

Large Genmery Model: When LLM meets Equivariance
Gruver, N., Sriram, A., Madotto, A., Wilson, A. G., Zitnick, C. I.., and Ulissi, 7. W. Fine-tuned language models generate stable inorganic materials text. In The Twelfth International Conference on Learning Representations, 2024. URL httph:://///....... http:///ppeeeeeeeeeeeeeeeeee ww..vieww.... forum? id-w99fffqPP..
sequence-structuue co-design. In International Confer- ence Learning Representations, 2022. URL https:
/ aeppenrersss.rrtttffooaaaeeeeeee htt.//rrmmmmddddi........
Kong, X., Huang, W., and Liu, Y. Conditional antibody de- sign 3d equivariant graph translation. In The Eleventh International Conference on Learning Representations, 2022.
Guo, T., Nan, B., Liang, Z., Guo, Z., Chawla, N., Wiest, O., Zhang, X., et al. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 36: 59662-59688, 2023.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730- 19742. PMLR, 2023.
Han, J., Cen, J., Wu, L., Li, Z., Kong, X., Jiao, R., Yu, Z., Xu, T., Wu, F., Wang. Z., et al. A survey of geomet- ric graph neural networks: Data structures, models and applications. arXiv preprint MxXiv:2403.00440 2024.
Li, M., Tan, Y., Ma, X., Zhong, B., Yu, H., Zhou, Z.,, Ouyang, W. Zhou, B., Tan, P., and Hong, L. ProSST: Pro- tein language modeling with quantized structure and dis- entangled attention. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024a.
Han, R., Huang, W., Luo, L., Han, X., Shen, J., Zhang, Z., Zhou, J., and Chen, T. Hemenet: Heterogeneous multichannel equivariant network for protein multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, number 1, pp. 237-245, 2025.
Li, X., Wang. L., Luo, Y., Edwards, C.,, Gui, S., Lin, Y., Ji, H., and Ji, S. Geometry informed tokenization of molecules for language model generation. arXiv preprint arXiv:2408. 10120, 2024b.
Li, Z., Wang, X., Huang, Y., and Zhang, M. Is distance matrix enough for geometric deep learning? Advances in Neural Information Processing Systems, 36, 2024c.
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. Simulating 500 million years of evolution with a language model. Science, pp. cads0018, 2025.
Li, Z., Wang, X., Kang, S., and Zhang, M. On the complete- ness of invariant geometric deep learning models. arXiv preprint arXiv:2402.044: 04836, 2024d.
Hollingsworth. S. A. and Dror, R. O. Molecular dynamics simulation for all. Neuron, 99(6): 129-119-111 1143, 2018
Li, Z., Zhou, C., Wang, X., Peng, X., and Zhang, M. Ge- ometric representation condition improves equivariant molecule generation. arXiv preprint arxtv:2410.03655, 2024e.
Huang. W., Han, J., Rong, Y., Xu, T., Sun, F., and Huang, J. Equivariant graph mechanies networks with constraints. In International Conference on Learning Representations, 2022.
Li, Z., Cen, J., Huang, W., Wang, T., and Song, L. Size- generalizable RNA structure evaluation by exploring hi- erarchical geometries. In The Thirteenth International Conference on Learning Representations, 2025a.
Huber, P. J. Robust estimation of location parameter. In Breakthroughs in statistics: Methodology and distribu- fion, pp. 492-518. Springer, 1992.
Li, Z., Ma, Z., Li, M., Li, S., Rong, Y., Xu, T., Zhang, Z., Zhao, D., and Huang, W. Star-rl: Spacial transformation reasoning by reinforcing multimodal llms. arXiv preprint arXiv:2505 15804, 2025b.
Jablonka, K. M., Schwaller, P., Ortega-Guerreet Guerrero. A., and Smit, B. Leveraging large language models for predictive chemistry. Nature Machine Intelligence, 6(2):161-169, 2024.
Li, 7.-7., Liang, X., Tang, 7.,, Ji, I.., Wang, P., Xu, H., W, X., Huang. H., Deng, W., Wu, Y. N., Gong, Y., Guo, Z.,
Jin, M., Wang, S., Ma, I.., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y.. Liang, Y., Li, Y.-F, Pan, S., and Wen, Q. Time-LLM, Time series forecasting by re- programming large language models. In The Twelfth International Conference on Learning Representations, forum?
Liu, X., Yin, F., and Liu, C.-L. Tl:dr: Too long, do re- weighting for efficient Ilm reasoning compression, 2025c. URL https: : / arxiv
http//awwwv..g./aaaa/ wow./brgg//............ /taaab///56... 2506.000000 02678.
2024. URL https::////. id=Unb5CVPtae.
Liang, X., Li, Z.-Z., Gong, Y., Wang. Y., Zhang, H., Shen, Y., Wu, Y. N., and Chen, W. Sws: Self-aware weakness- driven problem synthesis in reinforcement learning for Ilm reasoning, 2025. URL https: // //aaxiivvvoor/..////aa abs/2506. 009029906.0..9.
Jin, W., Wohlwend, J.. Barzilay, R., and Jaakkola, T. S. Iterative refinement graph neural network for antibody

---

Large Lenmury Model: When LLM meets Equivariance
Lin, H., Wu, L., Huang, Y., Liu, Y., Zhang. O., Zhou, Y., Sun, R., and Li, S. Z. GeoAB: Towards realistic anti- body design and reliable affinity maturation. In Forty- first International Conference on Machine Learning, 2024, URL wwt.s:tttt.//....m http:///pppevvviieeeee reperreview.. net/ forum? id 61H55
Lu, A. X., Yan, W., Yang, K. K., Gligorijevic, V., Cho, K., Abbeel, P., Bonneau, R., and Frey, N. Tokenized and continuous embedding compressions of protein sequence and structure. bioRxiv, pp. 2024-08, 2024.
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. In The Twelfth International Conference on Learning Represen- tations, 2024.
Lin, H., Zhang, O., Xu, J., Liu, Y. Cheng. Z., Wu, L., Huang, Y., Gao, Z., and Li, Z. Tokenizing elec- tron cloud in protein-ligant interaction learning. arXiv preprint arXiv:2505. 19014, 2025.
Mondal, A. K., Panigrahi, S. S., Kaba, O., Mudumba, S. R., and Ravanbakhsh, S. Equivariant adaptation of large pretrained models. Advances Neural Information Pro- cessing Systems, 36:50293-50309, 2023.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseck- v3 technical report. arXiv preprint 0rXiv:2412.19437 2024a.
OpenAL, G. 4o mini: Advancing cost-efficient intelli- gence, 2024, URL: https://openai com/indes//pt-4o-miii advancing- coft--fffciititttteeeeee-------e intfeiiinn-ittelligennee, 2024.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Puny, O., Atzmon, M., Smith, E. J., Misra, I,, Grover, A., Ben-Hamu, H., and Lipman, Y. Frame averag- ing for invariant and equivariant network design. In International Conference on Learning Representations, 2022. URL https: / /openrevieww net/ net/forun? forum? id=zIUyj55nxE
Liu, H., Yin, H., Luo, Z., and Wang. X. Integrating chem- istry knowledge large language models via prompt engincering. Synthetic and Systems Biotechnology. 10(I): 23-38, 2025a.
Liu, S., Nie, W., Wang, C., Lu, J., Qiao, Z., Liu, L., Tang. J., Xiao, C., and Anandkumar, A. Multi-modal molecule structure text model for text-based retrieval and editing. Nature Machine Intelligence, (11)):444-1457) 2023a.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Saka, K., Kakuzaki, T., Metsugi, S., Kashiwagi, D., Yoshida, K., Wada, M., Tsunoda, H., and Teramoto, R. Anti- body design using Istm based deep generative model from phage display library for affinity maturation. Scientific reports, 1 11(1):5852, 2021.
Liu, W., Lu, Z., Hu, X., Zhang, J., Li, D., Cen, J., Cao, H., Wang, H., Li, Y., Xie, K., Li, D., Zhang, P., Zhang, C., Ren, Y., Ma, Y., and Huang, X. STORM-BORN: A challenging mathematical derivations dataset curated via multi-agent framework. In The 63rd Annual Meeting of the Association for Computational Linguistics, 2025b.
Satorras, V. G., Hoogeboom, E., and Welling, M. E(n) equivariant graph neural networks. In International Con- ference on Machine Learning, pp. 2021.
PMLR,
Liu, Y., Cheng, J., Zhao, H., Xu, T., Zhao, P., Tsung, F., Li, J., and Rong, Y. SEGNO: Generalizing equivariant graph neural networks with physical inductive biases. In The Twelfth International Conference on Learning Represen- tations, 2024c. URL. https: /ttp:p:://ppppeeee wwwpr/vveeeeeeeeee///eeee. net./ Dorum?id==oTPPORR.
Schütt, K., Unke, O., and Gastegger, M. Equivariant mes- sage passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pp. 9377- PMLR, 2021.
Liu, Y., Chen, Jiao, R., Li, J., Huang, W. and Su, B. De- noisevae: Learning molecule-adaptin noise distributions for denoising- based 3d molecular pre-training. In The Thirteenth International Conference on Learning Repre- sentations, 2025c.
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. Large language models encode clinical knowledge. Nature, 620(7972):172-188) 2023.
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulezyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., al. Toward expert-level medical question answering with large language models. Nature Medicine, pp. 1-8, 2025.
Liu, Z., Li, S., Luo, Y., Fei, H., Cao, Y., Kawaguchi, K., Wang, X., and Chua, T.SS. Molca: Molecular graph- language modeling with cross modal projector and uni- modal adapter. arXiv preprint arXiv:2310. 12798, 2023b.
12

---

Large Geomery Model: When LLM meets Equivariance
Song, Z.,, Ouyang, G., Li, M., Ji, Y., Wang, C., Xu, Z., Zhang, Z.,, Zhang, X., Jiang. Q-, Chen, Z., Li, Z.,, Yan, R., and Chen, X. Maniplvm-rl: Reinforcement learning for reasoning in embodied manipulation with large vision- language models, 2025. URL /tttps://arxiv...t /atpiv.xxx.og//////.//../.. abs/ 0bb/25005.0 ..0....6510 16517.
Wang, E,, Guo, W., Ou, Q. Wang, H., Lin, H., Xu, H., and Gao, Z. Polyconf: Unlocking polymer conformation gen- eration through hierarchical generative models. In Forty- second International Conference on Machine Learning, 2025b, URL ht.ps:/ /ttptt::://ppeeeeeeeee /tpp/pepeeevveeeeeeeeeeeeeeeeeee ew net/ forum? Ad-BsTLUX38qV.
Sun, H.-I.., Sun, Z., Peng, H., and Ye, H.-J Mitigating visual forgetting via take-along visual conditioning for multi-modal long cot reasoning. In ACL, 2025a.
Wang. P., Yang, C., Li, Z.-Z., Yin, F., Ran, D., Tian, M., Ji, Z., Bai, J., and Liu, C.-L. Solidgeo: Mcasuring multi- modal spatial math reasoning in solid geometry, 2025c. URL https: / /aaxivivvvv.//////////aa////aata org/ /abs///////// 2555.221777
Sun, H.-L., Zhou, D.-W., Li, Y., Lu, S., Yi, C., Chen, Q-G., Xu, Z.. Luo, W., Zhang, K., Zhan, D.-C., et al. Parrot: Multilingual visual instruction tuning. In ICML., 2025b.
Wang, X., Hu, Z.. Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. Scibench: Evaluating college-level scientific problem- solving abilities of large language models. arXiv preprint arXiv:2307.10633 2023b.
Sun, Y., Qian, X., Xu, W., Zhang, H., Xiao, C., Li, I.., Rong, Y., Huang, W., Bai, Q. and Xu, T. Reasonmed: A 370k multi-agent generated dataset for advancing medical reasoning. arXiv preprint arXiv:2506. 09513, 2025c.
Wu, L.. Hou, Z., Yuan, J., Rong, Y., and Huang, W. Equivari- ant spatio-temporal attentive graph networks to simulate physical dynamics. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A.. Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint 1xXiv:2403.05530. 2024.
Wu, L., Huang, W., Jiao, R., Huang, J., Liu, L., Zhou, Y., Sun, H., Liu, Y., Sun, F,, Ren, Y., et al. Siamese foundation models for crystal structure prediction. arXiv preprint arXiv:2503.1044 2025.
Thomas, N., Smidt, T., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation-ant and translation -equivvarnnii neural networks for 3d point clouds. arXiv preprint 1rXiv:1802.082/9, 2018.
Xie, Y., Daigavane, A., Kotak, M., and Smidt, T. The Exploring expressivity and runtime tradeoff's in equivariant tensor products. arXiv preprint 2KKi::2506.135233 2025.
price of freedom:
Tiller, K. E. and Tessier, P. M. Advances in antibody design. Annual review of biomedicul engineering, 17(1)1191-216, 2015.
Xu, J. and Zhang, Y. How significant is protein structure similarity with tm-score= 0.5? Bioinformatics, 26(7): 889-895, 2010.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., al. Llama: Open and efficient foundation lan-
guage models. arXiv preprint 2rXiv:2302.13971, 2023.
Xu, M., Han, J., Lou, A., Kossaifi, J., Ramanathan, A., Azizzadenesheli, K., Leskovec, J., Ermon, S., and Anand- kumar, A. Equivariant graph neural operator for modeling 3d dynamics. arXiv preprint arXiv:2401.11033 2024.
Wang, F., Xu, H., Chen, X., Lu, S., Deng, Y., and Huang, W. Mperformer: An se (3) transformer- molecular per- ceptron. In Proceedings of the 32nd ACM International Conference Information and Knowledge Management, pp. 2512-2522, 2023a.
Xu, W., Chan, H. P., Li, L., Aljunied, M., Yuan, R., Wang, J., Xiao, C., Chen, G., Liu, C., Li, Z., et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv: 2025.
Wang, F., Guo, W., Cheng, M., Yuan, S., Xu, H., and Gao, Z. Mmpolymer: A multimodal multitask pretraining frame- work for polymer property prediction. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 2336-2346, 2024.
Yan, S., Li, Z., and Zhang, M. Georecon: Graph-level rep- resentation learning for 3d molecules via reconstruction- based pretraining. arXiv preprint arXiv:2506. 13174, 2025.
Wang, F., Cheng, M., and Xu, H. WGFormer: An SE(3)- transformer driven by wasserstein gradient flows for molecular ground-state conformation prediction. In Forty- second International Conference on Machine Learning, 2025a. URL https::///// http::::////ppeeeieeeeee www.ww.wwweeeeeeeeeeeeeeeeeeeeee net/forum? 1 1d=2wUgttials
Yang, W., Chen, J., Lin, Y., and Wen, J.RR. Deepcritic: Deliberate critique with large language models. arXiv preprint arXiv:2505.0066 2025a.
13

---

Large Lenguare Model: When LLM meets Equivariance
Yang, W. Ma, S., Lin, Y., and Wei, F. Towards thinking- optimal scaling of test-time compute for IIm reasoning. arXiv preprint arXiv:2502./8080 2025b.
Yu, B., Yin, H., and Zhu, Z. Spatio-temporal graph convo- lutional networks: A deep learning framework for traffic forecasting. arXiv preprint arXiv:
2017.
Yuan, Y., Ding. J., Feng, J., Jin, D., and Li, Unist: prompt-empowered universal model for urban spatio- temporal prediction. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4095-4106, 2024.
Yue, A., Luo, D., and Xu, A glug-and-plas quaternion puessage-passing module for molecular conformation rep- resentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 16633-16641 2024.
Yue, A., Wang, Z., and Xu, H. Reqflow: Rectified quater- nion flow for efficient and high-quality protein backbone generation. arXiv preprint arXiv:2502.14637. 2025.
Zhang. Y. and Skolnick, J. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57-4):702-710, 2004.
Zhang, Y., Cen, J., Han, J., Zhang, Z.,, Zhou, J., and Huang, W. Improving equivariant graph neural networks on large geometric graphs via virtual nodes learning. In Forty-first International Conference Machine Learning, 2024.
Zhang, Y., Cen, J., Han, J., and Huang, W. Fast and dis- tributed equivariant graph neural networks by virtual node learning. arXiv preprint arXiv::5066. 19482, 2025.
Zholus, A., Kuznetsov, M., Schutski, R., Shayakhmetov, R., Polykovskiy, D., Chandar, S., and Zhavoronkov, A. Bindgpt: A scalable framework for molecular design via language modeling and reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, number 24, pp. 26083-26091, 2025.
Zhou, G., Gao, Z., Ding, Q., Zheng, H., Xu, H., Wei, Z., Zhang, L.,, and Ke, G. Uni-mol: universal molecu- lar representation learning framework. In The Eleventh International Conference on Learning Representations, 2023.
14

---

Large Lenouty Model: When LLM meets Equivariance
A Proof of EquiLLM's Equivariance
now present the theoretical foundations for the symmetries exhibited by EquiLLM. Specifically, we analyze two key properties: the invariance of the node representation h and the equivariance of the node coordinate ".... To establish these properties, we break down the proof into components, demonstrating the equivariance of both the encoder and the adapter.
Proof. Consider the rotation/reflection matrix O E IR333 and the translation vector b E R
Equivariance of the Encoder. The equivariance of the encoder can be expressed as:
g' = delg glm),
(9)
where g is an element of the E(3) group. The proof this property detailed in the appendix of the original ESTAG paper (Wu et al., 2024). In essence, this ensures that the output node features H' are invariant under transformations, while the output coordinates X" are equivariant.
Equivariance of the Adapter. Next, prove the equivariance of the adapter. Recall that the node features from the encoder are first processed through projector and then passed through large language model, both of which are invariant operations. Consequently, the resulting node features H" remain invariant, while the coordinates X remain equivariant. These outputs serve as inputs to the adapter, and we analyze their transformation properties below:
(1) Invariance of pairwise interactions. The interaction term maj between nodes i and j is defined as:
mij Pm (h'', hrj. (Oz) + b) (O#)+ +)
Pm (h hj x - 1i,|))
mij+
where, may denotes the transformed variable. Since the Euclidean distance is invariant under rotations and translations, mij remains unchanged.
(2) Invariance of output node features he Its update rule is given by:
haut h² ph (hg. EsEN m.s)
Here, both hG and m.j invariant variables, and the multi-layer perceptron (MIP) ón and summation Ejer() invariant operations. Therefore, it follows that h.th is also invariant.
(3) Equivariance of output coordinates z. Thh transformation is formulated as:
Sout Oz + b
EjENa) Pa (m.)) ((O%) + b) - (Oz' + +b) ΣjEN(O 82 (m:)) (H) - F²))
of +b+
0 d [ol EjEN(y) Pn (mig): (H) d'() b
0
nou b. a
Thus, the output coordinates transform consistently with the input coordinates under rotations and translations, confirming their equivariance.
Conclusion. In summary, we have rigorously demonstrated that the node representations hTT are invariant, while the node coordinates ब are equivariant under E(3) transformations. These results align with our initial claims about the symmetries of Eq.iL.LM.
15

---

Large Lenmery Model: When LLM meets Equivariance
B
Dataset Details
B.1
Implementation Details on MD17 dataset
The first column in Table 6 delineates the standardized parameter configuration utilized throughout our experiments on the MD17 dataset. This uniform parameterization scheme has been implemented across both our EquiLLM architecture and all other baseline methodologies. All computational experiments, encompassing model training, validation and testing phases, are executed on a single NVIDIA A100-80G GPU. The training, validation, and testing sets consist of 500, 2000, and 2000 samples, respectively.
The MD17 dataset exhibits a characteristic graph structure with a maximum cardinality of 13 nodes. Regarding graph topology construction, establish interatomic connectivity through distance- lissence-basse criterion, where atomic pairs separated by less than the threshold parameter À are classified as primary neighbors. The graph connectivity framework extends to incorporate both primary and secondary neighboring relationships, thereby establishing comprehensive edge structure to support subsequent message passing.
Table 6: Hyper- parameters of EquiLLM and other methods. The previous length Tp denotes the length of input sequence, the future length y denotes the length of output sequence, the time lag A/. denotes the interval between two timestamps, the hidden size denotes the size of hidden states in all Multi--ayer Perceptrons (MLPs) within the EquiLLM framework, and the layer denotes the number of layers.
MD17
Motion Capture
Learning Rate Epochs
5e-3
5e-3
500
500
Previous Length T
10
10
Future Length T
10
10, 15, 20
Time Lag At
10
Hidden Size
16
16
Layer
2
B.2 Implementation Details on Motion Capture
The second column in Table 6 outlines the standardized configuration utilized for Human Motion Capture dataset evaluations. This consistent parameterization is implemented across our EquiLLM architecture and all other baselines. All computational experiments, encompassing model training, validation and testing phases, are executed on a single NVIDIA A 1000880G GPU. We maintain the experimental configurations and dataset partitions specified in (Wu al., 2024). For subject #35 (Walk), the dataset comprises 1100 training, 600 validation, and 600 testing trajectories, whereas subject #102 (Basketball) includes 600 training, 300 validation, and 300 testing trajectories.
The Motion Capture dataset comprises graphs with a maximum of 31 nodes per instance. Graph connectivity is established by defining directly adjacent joints as primary neighbors. Each joint forms edges with both primary and secondary adjacent nodes to enable efficient subsequent message passing. All joints share identical invariant feature representations, initialized as unit vectors.
B.3 Implementation Details on Antibody Design
We adhere to the experimental configurations established in MEAN (Kong et al., 2022) to ensure fair comparisons. Specifically, EquiLLM maintains identical hyper-parameters with MEAN: 64-dimensional trainable embedding for each amino acid type, 228-dimensional hidden states, 3 network layers, batch size of 16, and 20 training epochs. The Adam optimizer is employed with an initial learning rate of 0.001, which decays by 5% per epoch. We utilize the same dataset splits as MEAN, excluding one antigen-antibody complex from the training set due to missing light chain data. For GeoAB (Lin et al., 2024), we reproduce the results using the official implementation on our datasets.
To ensure fair comparisons, we adopt the same settings as MEAN (Kong et al., 2022). Specifically, EquiLLLM shares the same hyperparameters as MEAN: the trainable embedding size for each amino acid type is 64; the hidden state size is 128; the number of layers 3; the batch size 16; and the number of training epochs 20. We employ the Adam optimizer
16

---

Large Geomery Model: When LLM meets Equivariance
with learning rate Ir of 0.001, which is decayed by factor of 0.95 after each epoch. Furthermore, the same training, validation, and test datasets those in MEAN, except for one antigen-antibody complex in the training set that was excluded due to the absence of the light chain. For the GeoAB (Lin et al., 2024) method, we reproduce the results on our datasets using the official implementation.
C More Experiment Results
C.1 More Exploratory Experiments
Fine-tuning the LLM Layers. We set the LLM's parameters to be trainable and fine-tune the model on the SAbDab dataset. However, the experimental results ( Table 7, first row) show performance degradation, suggesting that fine-tuning may compromise the original information encoded the LLM, particularly since the dataset used for fnn-tuning is not large enough.
Table 7: EquiL_M with different backbones.
Method
AAR
TM-score
RMSD
Finetune LLM
38.57%
0.9819
1,77
Normal GNN encoder
32.32%
0.9308
4.14
Qwen2.5-3B
39.04%
0.9828
1.76
Original
38.97%
0.9830
1.73
Normal GNN Encoder. We additionally replace the original equivariant GNN encoder with normal GNN encoder. As shown in Row 2 of Table 7, the model exhibits significant performance degradation, demonstrating the importance of maintaining -quivivareee when modeling 3D structures. Furthermore, in our original equivariant GNN encoder, equivariant and invariant features interact through message passing and feature updating, with 3D spatial distances explicitly encoded. As established in Section 3.3 of PAINN (Schütt et al., 2021), incorporating distance information across stacked layers implicitly models angular relationships, enabling the output invariant features to inherently capture spatial geometric information.
Other LLM Backbone. We conduct additional evaluations using the Qwen2.5-3B model (see Table 7, Row 3). Although it shows a marginal improvement in AAR, we observe slight decreases in RMSD) and TM-score performance. We hypothesize that the language model's capability remains constrained by limited text-3D structure paired data; otherwise, upgrading the LLM component could yield significant gains. We leave this exploration for future work.
C.2 More A Ablation Studies
Table 8: Ablation studies on RAbD dataset.
Method
AAR
T'Mscore
RMSD
w/o object feature
38.32%
0.9826
1.76
w/o LLM
37.58%
0.9818
1.79
w/o prompt1
37.84%
0.9820
1.76
w/o prompt2
38.57%
0.9823
1.77
w/o prompt3
38.52%
0.9827
1.74
Equil.L
38.97% %
0.9830
1.73
We further conduct more ablations on the RAbD dataset. The experimental results are shown in Table 8.
Role of External Knowledge and LLM. We indirectly demonstrate, through ablation experiments, that the model's perfor- mance suffers without properly designed prompts to activate the LLM's knowledge. Specifically, when removing antigen, light chain, and heavy chain feature deseriptions from antibody design prompts (Table 8, Row 1), we observe performance
17

---

Large Lenouty Model: When LLM meets Equivariance
degradation, highlighting how domain-specific knowledge enhances EquiLLM's geometric modeling capabilities. Moreover, We also remove the L.M module, with results presented in Row 2 of Table 8. The model exhibits significant performance degradation, underscoring the critical role of LLM in our framework.
Impact of Prompt Components. We conduct more detailed prompt ablations on the RAbD dataset, to investigate the impact of different prompt components on model performance. For antibody design task, the object statistical information encompasses two hierarchical levels: 1. Chain-level features: Inter-chain centroid distances (prompt 1) and Maximum essidue-resddu distances within each chain(prompt 2); 2. Residue-level features: Statistics (max/min/mmean) of residue-to- centroid distances per chain(prompt 3).
As shown in Rows 3-5 of Table 8, the results demonstrate that chain-level features contribute more significantly to performance improvement compared to residue-level features. We hypothesize that this discrepancy arises because chain- level features provide macroscopic structural information that better facilitates global 3D structure understanding and modeling.
C.3 Inference Time
As shown in Table on antibody desgin task, our comparative anal- ysis of inference times reveals that EquiLLM requires slightly more computation than state-of-the-art methods (MEAN and GeoAB), but this modest overhead justified by its substantial accuracy gains.
Table 9; The inference time on RAbD.
GeoAB
MEAN
EquiLLM
Timels
0.0265
0.0139
0.0539
D Human Motion Simulation
Given the 3D coordinate trajectory of a physical system (e.g., human
bodies) over 7' frames, namely, X E Nxxx>xx x3 along with the invariant features H of all the joints (all 1s), the model aims to infer future trajectories X Nx3 for F subsequent frames.
Geometric-aware Prompt
b Task description.
Taak:
Predict human basketball-relater metions in 3D space.
>> Object feature description.
Action: Basketball movements, such as dribbling, shooting, passing, and defense. Motions involve
> Object statistical information.
Statistics for all joints in the human body over the past 10 frames:
Distance of each joint from the origin (0, 0, 8)
"Minimum distance": min_valuss "Maximum distance": max_value,
"Mean distance": mean_value.
Task Formulation and Training Objective. With G1r {9: (X, E,, and GT+1T+F (G,
(X,, H2, (1,8))7+4 T+F we provide the entire process as follows:
GTHIT4F= = P).
(10)
Let ge
T+t denote the ground-truth 3D coordinates for the time period from T 1 to T + F, we define the object function
C 1 E/= lmso (Xom
gt y refers the mean squared error (MSE).
F
18

---

Large Lenmetyy Model: When LLM meets Equivariance
E Detailed Geometric-awars Prompt
E.1
Molecular Dynamics Simulation
In Eq (DST), compute the distances from all atomic 3D coordinates to the origin (0, 0, 0) after coordinate centering and derive their statistical properties. The corresponding prompt is as follows:
Geometric-awenl aware Prompt
>> Task description.
Task:
Fredict molecular 3D coordinates (x, 2)
Object feature deseription.
Molecule: Aspirin (C91804) with 13 heavy atoms (9c, 40) structure includes aromatic ring with acetyl
> Object statistical information.
Statistics for all heavy atoms in one melecule over the past 10 frames:
Distance of each heavy atom from the crigin (0, 0, 0):
"Minimum distance"; min_value,
"Maximum distance"; max_value, "Mean distance": value.
In EquiLLM (PCA), we employ Principal Component Analysis (PCA) to process coordinate data, project the data onto the first principal components, and subsequently compute statistical information from the projected data. The corresponding prompt is as follows:
Geometric-aware Prompt
> Task description.
Task:
Predict molecular 3D coordinates (x, Y, 2)
Object feature description.
Molecule:
Aspirin (C9H804) with 13 heavy atoms (9C, 40). structure includes aromatic ring with acetyl ...
Object statistical information.
Statistics for all heavy atoms in one melecule over the past 10 frames: Shape of the original input tensor: criginal_shape After reshaping for PCA: new_shape
Statisties of the PCA projection the reshaped trajectory data:
Eigenvalues of the PCA components:
PC1: FCI_value
PC2: FC2_value
Dimension (PC1):
Minimum: min_value
Maximum: max_value
Mean: mean mean value
Median: median value
Standard Deviation: std_valus
Dimension (PC2):
19

---

Large Lenouryy Model: When LLM meets Equivariance
Minimum: min_value
Maximum mum: max_value
Mean: rean_valus
Median: median, aodian__alus
Standard Deviation: value
E.2
Antibody Design
Geomettric-wwrre Prompt
Task description.
Task: Predict the structure and aminc acid sequence of the CDR-H3 region in antibody-antigen complex.
Object feature description.
<Module: Antigen>
Defines binding constraints for the CDR-H3 backbone prediction
<Module: Heavy Chain
Provides structural context for the CDR-H3 region
<Module: Light Chain>
Offers spatial context to guide the CDR-H3
Object statistical information.
Statistics of Global Features:
Distances between centroids of different chains ....
"#eavy-Light"; heavy_light_walus,
"#eavy-Antigen": heavy_antigon_value,
"light-Antigen": light_antigen__allus
Distance between the two most di distant amino acids on one chain ...:
"Heavy": heavy_value,
"Light": light_value,
"Antigen"; antigen_value.
Statistics of Chain-Level Features:
Distances of each amino acid's alpha carbon atom to the chain centroid's ...:
"Minimum distance"; min_value,
"Maximum distance"; max_value,
"Xean distance"; value.
20


Reviews:
[{"review_id": "wzIsH5wba4", "reviewer": "Reviewer_c49n", "summary": "The authors propose EquiLLM – a framework designed to enhance spatial reasoning in 3D structure and dynamics by integrating geometry-aware prompting and equivariant Graph Neural Network layers. Experiments on molecular dynamics, human motion, and antibody design demonstrate are carried out, and show good performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you have devoted to providing detailed and constructive feedback. Your insightful comments have been invaluable in improving both the technical quality and clarity of our manuscript. We have carefully revised our paper to incorporate your suggestions. Below, we address each of your points individually.\n\n> **Q1: Method is built on the LLaVA approach.**\n\nThank you for your comment. There may be some misunderstandings here—our method does not simply replace LLaVA's encoder with an equivariant GNN encoder, as that would compromise the framework's overall equivariance. Instead, EquiLLM introduces an innovative design, as shown in Fig.1. First, the equivariant GNN encoder extracts both equivariant and invariant features, but only the invariant features are fed into the LLM, unlike LLaVA where the LLM receives all encoder outputs. Then, after LLM processing, the output is concatenated with the encoder's equivariant features via a skip connection and passed to the equivariant adapter module to generate both equivariant and invariant predictions. \n\n> **Q2: The impact of fine-tuning the LLM layers.**\n\nNice suggestion! We set the LLM's parameters to be trainable and fine-tune the model on the SAbDab dataset. However, the experimental results (Table A, first row) show performance degradation, suggesting that fine-tuning may compromise the original information encoded in the LLM, particularly since the dataset used for fine-tuning is not large enough.\n\n**Table A. EquiLLM with different backbones.**\n\n||AAR|TM-score|RMSD|\n|-|-|-|-|\n|Finetune LLM|38.57%|0.9819|1.77|\n|Normal GNN encoder|32.32%|0.9308|4.14|\n|Qwen2.5-3B|**39.04%**|0.9828|1.76|\n|Original|38.97%|**0.9830**|**1.73**|\n\n> **Q3: Different encoding layers & normal GNN encoders.**\n\nThank you for raising this point. We additionally replace the original equivariant GNN encoder with a normal GNN encoder. As shown in Row 2 of Table A, the model exhibits significant performance degradation, demonstrating the importance of maintaining E(3) equivariance when modeling 3D structures.\n\n> **Q4: Baseline models are relatively outdated and weak models.**\n\nThank you for this constructive comment. We would like to clarify that ESTAG (NeurIPS 2023) remains the SOTA model on MD17, while MEAN (ICLR 2023) and GeoAB (ICML 2024) are also leading methods on the RAbD dataset. That said, we agree that additional baselines could further validate our approach. To address this, we have included the results of Equiformer (ICLR 2023) on MD17 in Table B, where it still significantly underperforms our model.\n\n**Table B. The performance of Equiformer on MD17.**\n\n||Aspirin|Benzene|Ethanol|Malonaldehyde|Naphthalene|Salicylic|Toluene|Uracil|\n|-|-|-|-|-|-|-|-|-|\n|Equiformer|10.13|2.00|1.88|8.05|3.43|5.79|2.09|4.38|\n|EquiLLM|**2.391**|**0.732**|**1.031**|**1.671**|**1.453**|**2.162**|**1.178**|**1.060**|\n\n> **Q5: All hyperparameters are kept consistent.**\n\nThank you for your comment. Our experimental settings follow the ESTAG paper, using identical parameter configurations across all baseline models to ensure fair comparisons. We also explored various hyperparameter choices for the baselines on MD17, but the performance improvements remained marginal compared to our model's results.\n\n> **Q6: Discuss/acknowledge EGNN.**\n\nGreat suggestion! We will provide a more comprehensive discussion of EGNN in the revised version.\n\n> **Q7: Lack of more technical details.**\n\nThank you for this insightful suggestion. In our EquiLLM framework, the LLM module parameters remain frozen during training. In the revised version, we will provide more technical details and a brief introduction to the specific LLM employed in our work.\n\n> **Q8: The language model is very outdated.**\n\nThank you for your insightful observation. To address this point, we conducted additional evaluations using the Qwen2.5-3B model (see Table A, Row 3). While it shows marginal improvement in AAR, we observe slight decreases in RMSD and TM-score performance. We hypothesize that the language model's capability remains constrained by limited text-3D structure paired data; otherwise, upgrading the LLM component could yield significant gains. We leave this exploration for future work. \n\n> **Q9: The training size**\n\nNice question! Our experimental setup primarily follows established conventions in the field. For the MD17 and Human Motion datasets, we adopt the same configurations as the EGMN and ESTAG papers, while for the SAbDab dataset, we maintain the settings used in MEAN and GeoAB.", "claims_and_evidence": "Most of the claims are well-supported. However, some arguments are not as convincing.\nFor instance, the authors “One possible solution is to adapt existing multimodal LLM architectures, such as LLaVA (Liu et al., 2024b), by treating 3D structures as a separate modality and simply replacing the image encoder with a geometric GNN. However, this naive adaptation fails to satisfy the E(3)-equivariance requirement.” It seems that what the authors do is simply replacing the encoder with an equivariant GNN encoder. I would say that the authors’ method is built on the LLaVA approach.", "methods_and_evaluation_criteria": "Appropriate evaluation methods are used in the article. one concern is that the authors only trained the GNN encoders and adaptors. It would be interesting to explore the impact of fine-tuning the LLM layers as well.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Model comparison: it would be great to compare different encoding layers to verify the significance of the equivariance as claimed by the authors. Meanwhile, the baseline models are relatively outdated and weak models.\n\nIn line 361, “To ensure a fair comparison, all hyperparameters (e.g. learning rate, number of training epochs) are kept consistent across our model and all other baselines”. This is not a valid approach since different models would need different hyperparameters to make them function well.", "relation_to_broader_scientific_literature": "Compared with existing Large Language Models for Sciences, incorporating equivariance GNN encoders is a meaningful attempt. And the empirical results support the motivation.", "essential_references_not_discussed": "Speaking about equivariant GNN, key references such as [1] should be better discussed/acknowledged in the article.\n\n1. Satorras, Vıctor Garcia, Emiel Hoogeboom, and Max Welling. \"E (n) equivariant graph neural networks.\" International conference on machine learning. PMLR, 2021.", "other_strengths_and_weaknesses": "Strengths: The authors have shown good empirical results, demonstrating efficacy of the proposed method.\n\nWeaknesses: Lack of more technical details such as how is the language model trained, \nThe language model (GPT-2) used in the experiments is very outdated.", "other_comments_or_suggestions": "A comparison with LLMs trained with normal GNN encoders would be great to see the effect of equivariance here.", "questions_for_authors": "Why is the training size so small?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0X1bWu3IDt", "reviewer": "Reviewer_VBCb", "summary": "This paper puts forward EquiLLM, a strategy to merge large language models (LLMs) with geometric (E(3)-equivariant) graph neural networks (GNNs). The motivation is straightforward: GNNs with built-in physical symmetry can handle 3D data in a rotation-, reflection-, and translation-consistent way, but they typically lack the broader domain insights or contextual knowledge that LLMs are good at capturing. Conversely, LLMs excel at analyzing text and general knowledge, yet they struggle when asked to directly process 3D coordinates or enforce geometric symmetries.\n\nEquiLLM bridges that gap by clearly splitting the workloads:\n1. Equivariant Encoder (GNN) – Handles spatial structure, ensuring that rotating or shifting inputs leads to the correct rotation or shift in outputs.\n2. Prompted LLM – Receives only “invariant” features and carefully prepared textual prompts (e.g., sequence data, summary statistics, or domain context). This way, the LLM can apply its pretrained knowledge without worrying about coordinate transformations.\n3. Equivariant Adapter – Recombines the LLM’s outputs with the GNN’s spatial embeddings. Because the adapter is itself an equivariant module, any coordinate transformations flow through properly.\n\nBy cleanly separating invariant and equivariant representations, EquiLLM is able to inject the LLM’s knowledge about, for instance, chemical or biological concepts, while still guaranteeing correctness in handling 3D geometry. Experiments on molecular dynamics, human motion, and antibody design suggest that this setup can outperform using purely geometric networks or purely language-based models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our work! We are deeply grateful for the time and thoughtful effort you have dedicated to offering such detailed and constructive feedback. Your valuable suggestions have significantly enhanced both the scholarly rigor and presentation of our manuscript. Having carefully revised the paper to reflect your comments, we now address each point in detail below.\n\n> **Q1: Role of  \"External Knowledge \"& how LLM knowledge influences outcomes1 Role of \"External Knowledge.**\n\nThank you for raising this point. Directly explaining how the LLM's knowledge affects the results is a challenging task. Here, we indirectly demonstrate, through ablation experiments, that the model's performance suffers significantly without properly designed prompts to activate the LLM's knowledge. Specifically, when removing antigen, light chain, and heavy chain feature descriptions from antibody design prompts (Table C, Row 1), we observe clear performance degradation, highlighting how domain-specific knowledge enhances EquiLLM's geometric modeling capabilities.\n\n> **Q2: Independence of Added Model Capacity.**\n\nWe sincerely appreciate your thoughtful review. We have removed the LLM module for further ablation study as you suggested, with results presented in Row 2 of Table C. The model exhibits significant performance degradation, underscoring the critical role of LLM in our framework. We have included these ablation results in the revised manuscript.\n\n**Table C. Further ablation studies on RAbD.**\n\n||AAR|TM-score|RMSD|\n|-|-|-|-|\n|w/o object feature|38.32%|0.9826|1.76|\n|w/o LLM|37.58%|0.9818|1.79|\n|w/o prompt1|37.84%|0.9820|1.76|\n|w/o prompt2|38.57%|0.9823|1.77|\n|w/o prompt3|38.52%|0.9827|1.74|\n|EquiLLM|**38.97%**|**0.9830**|**1.73**|\n\n> **Q3: Formal Proofs of Equivariance & How is E(3)-equivariance assured once data flows through the LLM?**\n\nWe apologize for any lack of clarity in the current manuscript. To clarify, since the LLM exclusively processes invariant features, its outputs remain strictly invariant. These invariant outputs are then concatenated with the original equivariant features from the encoder through a skip connection, and subsequently processed by the equivariant adaptor. Throughout this data flow, we rigorously maintain E(3)-equivariance. In the revised version, we will add the following contents: (1) a rigorous mathematical proof of the framework's equivariance properties, and (2) a detailed analysis of how data flow maintains E(3)-equivariance throughout the architecture.\n\n> **Q4: Which aspects of prompt engineering bring the greatest advantages & How sensitive is the method to changes in prompts.**\n\nThank you for this valuable comment! We have conducted more detailed prompt ablations on the RAbD dataset, to investigate the impact of different prompt components on model performance. For antibody design task, the object statistical information encompasses two hierarchical levels:\n\n1. **Chain-level features**:\n    1. Inter-chain centroid distances (prompt 1)\n    2. Maximum residue-residue distances within each chain(prompt 2)\n2. **Residue-level features**:\n    1. Statistics (max/min/mean) of residue-to-centroid distances per chain(prompt 3)\n\nAs shown in Rows 3-5 of Table C, the results demonstrate that chain-level features contribute more significantly to performance improvement compared to residue-level features. We hypothesize that this discrepancy arises because chain-level features provide macroscopic structural information that better facilitates global 3D structure understanding and modeling. These comprehensive ablation results will be included in the revised manuscript.\n\n> **Q5: How  \"geometry-aware \" prompts are formulated to remain invariant.**\n\nNice question! To guarantee the invariance of geometry-aware prompts input to the LLM, we exclusively employ **distance-based** statistical measures, which are inherently rotation- and translation-invariant. The three prompt types mentioned above are deliberately designed as distance metrics to preserve E(3)-invariance.", "claims_and_evidence": "Overall, the key quantitative claims—namely that EquiLLM maintains E(3)-equivariance and achieves higher accuracy than both GNN-only and LLM-only baselines—are reasonably backed by the results on molecular dynamics, human motion, and antibody design. The authors provide side-by-side performance tables, ablation studies, and comparisons across multiple datasets. These empirical tests support the conclusion that adding a language model to a geometric GNN can improve 3D prediction accuracy under symmetry constraints.\n\nHowever, there are a few areas where the evidence leaves some questions:\n\n1 Role of “External Knowledge.”  \n   The paper attributes improvements partly to leveraging LLM “domain knowledge,” but how that knowledge is used is demonstrated only indirectly. While ablations show that prompts help, they do not isolate whether the gains come specifically from knowledge embedded in LLM pretraining or simply from a new trainable pathway (even though the LLM is frozen). A direct test—e.g., tasks that require specialized domain facts only learned from large-scale text—could clarify how much of the improvement is truly “knowledge-driven” rather than architectural flexibility.\n\n2 Independence of Added Model Capacity.  \n   While the LLM’s parameters are frozen, the approach still involves an additional module (the LLM plus the adapter) beyond the geometric encoder. This extra capacity might explain some of the improvement. The authors conduct ablations highlighting the importance of prompting, but comparisons controlling for parameter count could strengthen the argument about how much improvement comes from bridging textual knowledge and 3D GNNs.\n\n3 Formal Proofs of Equivariance.  \n   The paper relies largely on references to prior geometric GNN proofs and the stated separation of invariant vs. equivariant pathways. While that is common in similar research, readers not well-versed in geometric GNNs might want a more explicit derivation.\n\nOn the whole, the main findings about quantitative improvements under 3D symmetry constraints are well supported by experiments. Claims regarding “injecting domain knowledge” and attributing gains mainly to that knowledge are plausible but would benefit from more targeted evidence isolating the effect of LLM pretraining.", "methods_and_evaluation_criteria": "Yes, the paper’s choices of tasks and datasets are sensible for testing a framework that combines LLMs with 3D-equivariant GNNs. The molecular dynamics, human motion, and antibody design settings all demand careful handling of 3D structures under symmetry transformations, which aligns with the paper’s claim about preserving E(3)-equivariance. Plus, each of these tasks benefits from the richer contextual or domain-level reasoning that an LLM can contribute.\n\nThe benchmarks—MD17 for small-molecule dynamics, a motion-capture dataset for human skeletal movement, and a standard antibody-design dataset—are representative of real-world scenarios where both spatial invariances and contextual knowledge are key.", "theoretical_claims": "The paper primarily cites existing formulations of geometric GNNs that have already established E(3)-equivariance, rather than presenting a fully standalone proof for its combined EquiLLM framework. The core argument is that by strictly separating invariant features (handled by the LLM) from equivariant features (handled by the GNN), the architecture inherits the GNN’s established symmetry properties. Since the paper does not include a detailed, from-scratch proof of how these components integrate to preserve equivariance, there is no step-by-step proof to check in the manuscript itself. Instead, the authors rely on references to standard results in geometric deep learning. \n\nConceptually, the design appears consistent with known proofs for equivariant GNNs, and nothing in the method obviously breaks those symmetries. However, for a rigorous guarantee, readers would need to rely on both the cited prior proofs and a clear statement about how the LLM’s outputs (restricted to invariant inputs and outputs) blend with the GNN pipeline.\n\nI would suggest the authors do a better job at convincing the reader that their work is theoretically sound and more self-contained.", "experimental_designs_or_analyses": "The experimental setup for each of the three application domains—molecular dynamics, human motion prediction, and antibody design—largely follows established practices (e.g., using MD17 for small molecule simulations, a motion capture dataset, and standard antibody-design benchmarks). The baseline models are fairly chosen, and the evaluation metrics (root-mean-squared error for 3D positions, cross-entropy for sequences, etc.) are standard. The inclusion of ablation studies also helps show the effect of prompting and how the LLM interacts with the GNN. \n\nOne minor point is that the authors rely on a fixed set of hyperparameters across different architectures; while this is aimed at fairness, some baselines might not be fully optimized. Another is that although the ablation results highlight the model’s different components, they do not completely isolate the effect of LLM “knowledge” versus just having additional trainable modules. However, none of these issues seem to undermine the core claims, and overall the experiments appear consistent and well controlled.", "relation_to_broader_scientific_literature": "This paper takes ideas from two active research areas—3D-equivariant graph neural networks (GNNs) and large language models (LLMs)—and combines them in a single method called EquiLLM. Existing geometric GNNs preserve important symmetry constraints for molecules or other 3D structures but lack broad domain understanding. Meanwhile, LLMs are trained on large amounts of text-based knowledge but do not naturally handle 3D transformations. EquiLLM addresses this by giving the LLM only invariant or “directionless” information (such as molecular descriptors or statistical summaries) while the GNN processes raw 3D coordinates. The two parts share information through a small adapter that maintains the desired geometric symmetries. Experimental results on molecular dynamics, human motion, and antibody design show that EquiLLM outperforms both purely geometric models and purely language-based models, offering a middle ground that applies each technique where it works best.", "essential_references_not_discussed": "Some recently published approaches combine large language models with graph-based molecular or protein modeling but do not necessarily enforce strict 3D symmetries. Methods such as MoleculeSTM, MolCA, or Prot2Text show how text-based knowledge can be integrated with structural data. Also, newer “text-to-structure” techniques generate or modify 3D configurations directly from prompts, which might offer additional context. Finally, broader libraries such as e3nn for building E(3)-equivariant networks could situate EquiLLM among related tools for geometric learning. A brief discussion of these works would help readers place EquiLLM in the wider landscape of combining language models with 3D-aware architectures.", "other_strengths_and_weaknesses": "Strengths:\n• The paper’s main contribution—combining an equivariant GNN with an LLM by carefully separating invariant and directional information—feels fresh, especially given the clear synergy with 3D tasks.  \n• The authors demonstrate the technique on several real-world applications (molecular dynamics, human motion, and antibody design), indicating practical significance.  \n• The writing, while sometimes concise on certain technical points, is reasonably clear for readers with a background in geometric deep learning, and the experimental setup is straightforward to follow.  \n\nWeaknesses:\n• The manuscript relies heavily on referencing established proofs for equivariant GNNs. A more direct or step-by-step argument that the combined system remains equivariant would improve clarity for a broader audience.  \n• While the experimental results are comprehensive, the paper could devote more attention to precisely how LLM knowledge influences outcomes—especially to separate general architectural effects from true “knowledge injection.”  \n• The ablations show the benefits of prompting but do not fully dissect which aspects of prompt engineering bring the greatest advantages.  \n\nOverall, the idea of letting GNNs handle 3D geometry and LLMs handle higher-level contextual information is an interesting hybrid design that could be valuable in a range of domains involving complex spatial structures and domain knowledge.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. How does pretrained knowledge from the LLM specifically boost performance?  \n2. How is E(3)-equivariance assured once data flows through the LLM?  \n3. Can you illustrate how “geometry-aware” prompts are formulated to remain invariant?  \n4. How sensitive is the method to changes in prompts or hyperparameters, especially for the LLM component?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UWqXnNvrIy", "reviewer": "Reviewer_M747", "summary": "This paper presents a method for solving equivariant tasks by combining a pre-trained large language model (LLM) with a trained, geometric graph network. The large language model is prompted only with invariant quantities, which come from both a natural language prompt and learned invariant features from the graph network. It outputs invariant quantities, which are fed back into a new equivariant graph network. Only the equivariant networks are trained, while the LLM weights are frozen. They evaluate their method on a molecular dynamics dataset, a human motion capture dataset, and an antibody design dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for the time and effort you have dedicated to offering valuable feedback. We have revised the paper to address all your comments. Below, we respond to each of your points in detail.\n\n**Questions in Claims And Evidence**\n\n> Q1:\n\nIn our experiments, we evaluated multiple LLMs (GPT, Gemini, and DeepSeek) by directly inputting 3D systems, all of which performed significantly worse than our method. This finding aligns with CrystalLLM's [1] conclusion that direct fine-tuning of LLMs for 3D structure modeling leads to suboptimal performance across 5/8 evaluation metrics.\n\n> Q2\n\nWhile our work focuses on equivariant 3D tasks requiring directional vector outputs, the works you mentioned primarily address invariant 3D tasks. That said, we will expand our discussion in the revised manuscript to incorporate recent advances in 3D structure tokenization.\n\n> Q3\n\nWe have expanded our discussion of canonicalization in the manuscript and moderate the original claim accordingly.\n\n> Q4\n\nWe have revised the claim in our paper in the format as you suggested.\n\n> Q5\n\nRegarding the claim in the Related Work section, our intention was primarily to contrast our EquiLLM with the LLM+GNN approaches mentioned in the same paragraph. In the revised version, we will add a detailed discussion of ESM3 and removed the original claim to ensure a more precise and rigorous presentation.\n\n**Questions in other sections:**\n\n> Q6: Sanity check.\n\nWe have computed the equivariance error for each module, and the overall model indeed satisfies equivariance.\n\n> Q7: Ablation  study.\n\nWe have replaced the equivariant GNN with a normal GNN, with the results shown in Row 2 of Table A in our response to Reviewer c49n. The model exhibits performance degradation, indicating the importance of maintaining E(3) equivariance\n\n> Q8: Lack of other, stronger baselines.\n\nAs requested, we have conducted the following experiments:\n\n1. Using pretrained models with canonicalization: On the MD17 dataset, we first subtract the mean from coordinates to ensure translational invariance, then perform SVD decomposition for rotational invariance. We directly feed this canonicalized data into GPT-4o-mini, with results shown in Row 2 of Table D. The results demonstrate that while canonicalization indeed improves model's predictive capability, there remains a remarkable performance gap compared to our EquiLLM. This suggests that direct prediction of 3D coordinates remains suboptimal for current LLMs.\n\nTable D. Results of the pretrained models with canonicalization on MD17.\n\n||Aspirin|Benzene|Ethanol|Malonaldehyde|Naphthalene|Salicylic|Toluene|Uracil|\n|-|-|-|-|-|-|-|-|-|\n|GPT-4o-mini|13.070|9.581|5.011|9.910|35.155|10.627|8.132|9.762|\n|GPT-4o-mini+canonicalization|11.783|3.055|4.512|8.916|8.263|9.751|6.364|8.989|\n|EquiLLM|**2.391**|**0.732**|**1.031**|**1.671**|**1.453**|**2.162**|**1.178**|**1.060**|\n\n\n1. Fine-tuning the LLM: Following CrystalLLM, we finetune the llama-7b model on MD17. Due to token length limitations in our prediction task (predicting 10 frames), we select the smallest molecule, Ethanol (3 heavy atoms) for evaluation. We have investigated three experimental settings: (1) 500 samples (the original paper's setup) trained for 10 epochs; (2) 30,000 samples trained for 1 epoch; (3) 30,000 samples trained for 1 epoch with canonicalization. The results in Table E reveal that without canonicalization, 500-sample and 30,000-sample fine-tuned models perform poorly, lagging behind EquiLLM by two orders of magnitude. Remarkably, when we incorporate canonicalization as suggested, the model's predictive performance improved by a factor of 100, even surpassing GPT-4o-mini. This compelling result demonstrates that the combination of canonicalization with direct LLM fine-tuning is indeed promising and warrants further investigation.\n\nTable E. Results of fine-tuning LLM on MD17.\n\n||Ethanol|\n|-|-|\n|Setting 1|460|\n|Setting 2|457|\n|Setting 3|4.446|\n|EquiLLM|**1.031**|\n\n> Q9: The clarity of the paper.\n\nIn our current implementation, the LLM module remains pretrained (and fixed), while all other module parameters are learnable and trained from scratch.\n\n> Q10: The related work also needs work.\n\nIn the revised manuscript, we will modify the Related Work section by providing a clearer organization, including a comprehensive discussion of ESM3 and other language model-based methods for structural tasks. We will also provide detailed descriptions of ESTAG in the experimental section.\n\n> Q11: Typos & Contextualize their experimental results.\n\nWe have implemented all revisions in accordance with your suggestions in the revised manuscript.\n\n> Q12: Ensure wasn’t data leakage\n\nOur study utilizes exclusively 3D structural data, whereas all comparative LLMs were pretrained on textual corpora alone. This fundamental modality difference potentially effectively eliminates the risk of data leakage.\n\n[1] Fine-Tuned Language Models Generate Stable Inorganic Materials as Text, ICLR24", "claims_and_evidence": "The claims made regarding experimental performance, relative to the chosen baselines (more on that later), are clearly supported by the reported numbers. However, I found several of the motivating claims to be made without evidence/citation. For example:\n\n“A natural idea is to directly employ LLMs for modeling 3D physical systems. However, this approach fails to yield\nsatisfactory results in practice.” Are there citations to support this? \n\n“A key limitation is that LLMs are trained to process ordered and discrete text tokens, restricting their ability to directly comprehend unordered and continuous data in 3D space.” Actually, tokenizing 3D structures is an active area of research, and has been deployed successfully in recent papers e.g. ESM3, ProSST, BindGPT, CHEAP, Geo2Seq, etc.\n\n“Therefore, it is non-trivial to integrate the strengths of both LLMs and geometric GNNs while maintaining essential geometric properties.” Canonicalization is a very natural way of achieving this, which has been used together with LLMs for certain applications. Thus, I find this claim too strong. This should also be added as a baseline. \n\n“More significantly, LLMs’ flexibility in prompt engineering enables the development of tailored instructions that better leverage their capabilities, producing outputs more precisely suited to the task.” I believe that this is probably true, but for good scholarship, statements like this should either be phrased as “We speculate, based on our results, that…” or with explicit citations to back up the claim.\n\n\"Although the aforementioned methods promote interactions between GNNs and LLMs through various paradigms and yield promising results, they have yet to explore tasks involving 3D structural data, such as 3D structure generation and dynamic trajectory simulation in 3D space.” I believe that this is simply not true; consider e.g. ESM3. Although it uses equivariant attention instead of an equivariant graph network, I think this is tangential to the claim. The authors should tone down the strength of this claim, e.g. perhaps if you restrict to works which freeze pre-trained LLMs this is true (although, with the sheer volume of LLM literature, it is hard to say for absolutely sure).", "methods_and_evaluation_criteria": "The benchmark datasets do make sense, and they cover a diverse range of tasks.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "It seems to me that the lack of other, stronger baselines are the biggest flaw in the experimental design. For example, natural ones include fine-tuning the LLM (without a geometric module), canonicalizing the inputs to the LLM, etc. The authors claim that things like fine-tuning are “too expensive”, but it is also not fair to compare their method (which involves some training computation as well as a pretrained LLM) to methods which have no training compute (pretrained LLM alone without fine-tuning) or to methods which have no access to a pretrained LLM (such as geometric models trained from scratch).\n\nAs a useful sanity check, I would recommend computing the equivariance error for each model, as a way of ensuring that there are no implementation bugs in the proposed method.\n\nOne ablation I would like to see is, replacing the equivariant graph network with a non-equivariant network (eg a transformer).", "relation_to_broader_scientific_literature": "The lines of work on equivariant architectures, vs language model approaches, are mostly disparate; this paper unifies them in a way that’s conceptually easy to understand.", "essential_references_not_discussed": "There are several papers that combine language models with equivariant layers that are not discussed. For example, the ESM3 paper trains (from scratch) a masked language model that includes equivariant modules for the 3D structure channel, as well as other channels containing non-structural information (similar to this paper’s task prompt).", "other_strengths_and_weaknesses": "Strengths:\nThe proposed method is easy to understand, and it does not seem too hard to implement since only the geometric graph network is trained (not the LLM). It can be adapted to a variety of domains, as shown in the experiments. Also, the performance is quite a bit better than the chosen baselines. I think this merging of LLMs with geometric methods is a valuable direction.\n\nWeaknesses:\nThe idea itself is simple, and feels like an incremental change relative to the literature — yet the paper is framed as a methods paper, not as an application paper. The baselines/ablations are not a very strong comparison: the pure graph models do not get to benefit from a pretrained language model in any way, whereas the language models are not fine-tuned at all on the task (which is not the case for e.g. Gruver et al 2024, which fine-tunes a pretrained language model for materials generation). It is of course good to compare EquiLLM to graph networks trained from scratch, but stronger baselines are necessary to validate the authors’ specific method. Also, the authors make very strong claims about the novelty of their work, which ignores existing, more complicated methods that train combination language models and geometric modules from scratch, together (eg ESM3); the related work and contextualization of this work is lacking.", "other_comments_or_suggestions": "It is straightforward to use pertained, non-equivariant models for equivariant tasks with canonicalization and/or frame-averaging (see e.g. “Equivariant Adaptation of Large Pretrained Models” by Mondal et al 2023). The use of a trained geometric module is strictly more general, so it might perform better, but the authors should check this by comparing their method to canonicalization (perhaps allowing the same computation budget to fine-tune as was used to train the graph networks).\n\nThe clarity of the paper could be strongly improved along certain dimensions. For example, it is not made clear which parts of the learning pipeline are actively trained vs pretrained (and fixed) vs fine-tuned, except for a very brief aside on Line 177 that the LLM weights are frozen. This is a very important part of the proposed method and should be made clear from the start.\n\nThe related work also needs work. I believe that “Geometric GNNs” is too broad of a category to properly summarize in one paragraph; the authors cite a seemingly random selection of specific papers instead of summarizing overall categories of approaches (and then citing multiple papers for each paradigm). Related work should summarize the state of a field to the extent relevant for the contribution, which the related work currently does not do. Papers such as ESM3 and others, which use language models for structural tasks, are also not adequately cited. Also, ESTAG is one of the main comparison methods, so it should be described in greater detail in the experiments section.\n\nOverall, I think the idea is neat and intuitive, and I'd like to see a more polished version of this paper (with more thorough baselines), published eventually. \n\nSome typos:\n*L16, “fall” -> “fail”\n*L292: “definite”", "questions_for_authors": "1. How can you ensure that there wasn’t data leakage, where the chosen datasets were used to train the LLMs (both the one that is used in your method, and the ones you compare against)?\n\n2. Can the authors please contextualize their experimental results (specifically, the evaluation metrics) by citing the SOTA numbers for each task, with references?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EkPQjydzX2", "reviewer": "Reviewer_whe6", "summary": "EquiLLM integrates Large Language Models (LLMs) with geometric Graph Neural Networks (GNNs) to improve 3D structure and dynamics prediction. It uses an LLM for invariant feature processing, a GNN for equivariant encoding, and an adapter to ensure equivariance while leveraging external knowledge. Experiments show significant improvements in molecular dynamics, human motion, and antibody design, demonstrating strong generalizability.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the time and careful consideration you have given to providing detailed and constructive feedback. Your valuable insights have greatly improved both the technical accuracy and clarity of our manuscript. We have meticulously revised the paper to incorporate your suggestions. Below, we respond to each of your comments point by point.\n\n## Responses to Weaknesses\n\n> **Q1: The explanation for this choice.**\n\nIn our experiments, we follow the mainstream settings on three different domains. For MD17 and Human Motion Capture datasets, future frame prediction is a common setting that used in many studies, such as Eqmotion[1] and ESTAG. This approach offers a more direct method to assess model capabilities without relying on external solvers. For the antibody design task, we benchmarked against MEAN (ICLR 2023) and GeoAB (ICML 2024), which represent the current mainstream and state-of-the-art approaches in this domain.\n\n> **Q2： Including additional baselines.**\n\nThank you for your suggestion. Due to the tight time limitation, we have included additional experimental results for only one recent model, Equiformer, on the MD17 dataset. The results in Table F indicate that our EquiLLM still outperforms Equiformer on by a large margin. We'll include more experiments across three tasks to further strengthen credibility. \n\n**Table F. The performance of Equiformer on MD17.**\n\n||Aspirin|Benzene|Ethanol|Malonaldehyde|Naphthalene|Salicylic|Toluene|Uracil|\n|-|-|-|-|-|-|-|-|-|\n|Equiformer|10.13|2.00|1.88|8.05|3.43|5.79|2.09|4.38|\n|EquiLLM|**2.391**|**0.732**|**1.031**|**1.671**|**1.453**|**2.162**|**1.178**|**1.060**|\n\n> **Q3: Computational cost & Include the Uni-Mol, AlphaFold, etc.**\n\nWe appreciate your comments. As shown in Table G on antibody desgin task, our comparative analysis of inference times reveals that EquiLLM requires slightly more computation than state-of-the-art  methods (MEAN and GeoAB), but this modest overhead is justified by its substantial accuracy gains. However, it is worth to note that the primary contribution of this paper is not the optimization of computational cost, but rather the integration of Pretained Large Language Models into geometric learning. We excluded large pretrained models (Uni-Mol, AlphaFold) due to their extensive domain-specific pretraining and significantly higher computational costs, which would result in an unfair comparison.\n\n**Table G. The inference time on RAbD.**\n\n||Inference time/s|\n|-|-|\n|GeoAB|0.0265|\n|MEAN|0.0139|\n|EquiLLM|0.0539|\n\n## Responses to Questions For Authors:\n\n> **Q1: More details**\n\nGreat suggestion! We will include detailed descriptions of the MLP dimensions in the manuscript's appendix and will open-source the code upon paper acceptance.\n\n> **Q2: Do Invariant features capture \"spatial information\"**\n\nThank you for your question. In our Equivariant Encoder, equivariant and invariant features interact through message passing and feature updating, with 3D spatial distances explicitly encoded. As established in PAINN (Section 3.3), incorporating distance information across stacked layers implicitly models angular relationships, enabling the output invariant features to inherently capture spatial geometric information. \n\nTo validate our design, we have conducted ablation studies by replacing the Equivariant Encoder with two types of invariant encoders: (1) a standard GNN (see Table A, Response to Reviewer c49n) and (2) a canonicalization approach converting equivariant vectors to invariant forms (see Tables D&E, Response to Reviewer M747). Both variants underperformed our original model, confirming that the advantage of equivariant encoder against the invariant counterparts in effectively capturing spatial information.\n\n[1] EqMotion: Equivariant Multi-agent Motion Prediction  with Invariant Interaction Reasoning, CVPR2023.", "claims_and_evidence": "See subsequent subsections.", "methods_and_evaluation_criteria": "See subsequent subsections.", "theoretical_claims": "See subsequent subsections.", "experimental_designs_or_analyses": "See subsequent subsections.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:  \n\n1. The paper presents an innovative attempt by cleverly integrating pre-trained large language models with equivariant networks and demonstrates the feasibility of the proposed approach.  \n2. The paper conducts experiments on multiple tasks related to equivariant graph neural networks, proving the applicability of the method across various tasks.  \n3. Compared to the baseline models provided in the paper, the proposed method demonstrates superior performance.\n\nWeaknesses:\n\n1. The paper selects tasks from three different domains, but it is unclear whether the chosen methods are mainstream approaches within their respective fields. This raises concerns about whether the proposed method has been fairly compared with widely recognized and robust baselines. For example, in molecular dynamics (MD) simulations, the paper employs a temporal equivariant graph network to predict atomic positions in future frames. While this is a valid mathematical modeling choice, a more common approach in MD simulations is to first predict atomic forces at each frame and then compute the next frame’s positions accordingly to better preserve physical consistency. If the paper does not intend to directly predict forces like machine learning force fields in MD, it should provide an explanation for this choice.  \n\n2. The choice of baselines may not be comprehensive enough, particularly for domain-specific models (typically listed above the first horizontal divider in tables). For instance, in Table 1, EGNN represents work from 2021—should more recent methods from the past two years, such as Equiformer v2, MACE, etc., be included to strengthen the credibility of the results? Similar concerns apply to other tasks as well.  \n\n3. The integration of a large language model inevitably leads to a significant increase in inference time. However, the results section does not provide any data regarding model parameters, training time, or inference time, making it difficult to assess the computational cost of this combination. Given this, in the selection of baselines, it is reasonable for the proposed method to outperform large language models alone (typically listed between the first and second horizontal dividers in tables) since additional information and parameters are introduced. However, in domain-specific tasks such as molecular dynamics prediction and protein structure prediction, there are already many large-scale pretrained models, such as Uni-Mol, AlphaFold, etc. If inference time is not considered a primary factor, should these methods also be included in the comparison?\n\n[1]Liao Y L, Smidt T. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs[J]. arXiv preprint arXiv:2206.11990, 2022.\n\n[2]Liao Y L, Wood B, Das A, et al. Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations[J]. arXiv preprint arXiv:2306.12059, 2023.\n\n[3]Batatia I, Kovacs D P, Simm G, et al. MACE: Higher order equivariant message passing neural networks for fast and accurate force fields[J]. Advances in neural information processing systems, 2022, 35: 11423-11436.\n\n[4]Ji X, Wang Z, Gao Z, et al. Uni-Mol2: Exploring Molecular Pretraining Model at Scale[J]. arXiv preprint arXiv:2406.14969, 2024.\n\n[5]Jumper J, Evans R, Pritzel A, et al. Highly accurate protein structure prediction with AlphaFold[J]. nature, 2021, 596(7873): 583-589.", "other_comments_or_suggestions": "NO.", "questions_for_authors": "1. In the Equivariant Adapter section, the paper describes \\( e_{\\phi_m}, \\phi_x, \\) and \\( \\phi_h \\) in Equation (6) as Multi-Layer Perceptrons (MLPs), but their specific dimensions do not appear to be provided in the main text or appendix. Have the authors considered adding these details in the appendix or releasing the source code to enhance reproducibility?  \n\n2. In the ablation study, the paper analyzes the impact of including the Equivariant Encoder, but since the LLM itself is not equivariant, only the invariant part of the output can be fed into the LLM. Can we demonstrate that the invariant features obtained through the Equivariant Encoder truly capture \"spatial information\" or similar properties that improve LLM predictions, compared to a potential Invariant Encoder? Given that invariant computations are generally more computationally efficient than equivariant ones.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a novel approach combining Large Language Models with equivariant Graph Neural Networks for 3D structure prediction tasks. The core idea of separating invariant and equivariant processing is conceptually sound. After careful consideration of all reviews and the authors' thorough response to concerns, I've decided to accept this paper. The authors have adequately addressed the initial concerns about baseline comparisons through additional experiments with canonicalization and fine-tuned LLMs, and have committed to improving the related work section. While some limitations remain, the originality of the approach, strong empirical results, and the paper's contribution to bridging geometric deep learning with language models justify acceptance. The authors should incorporate their rebuttal improvements into the final manuscript.
Decision: Accept (poster)