ELoRA: Low-Rank Adaptation for Equivariant GNNs
Authors: Chen Wang
Siyu Hu
Guangming Tan
Weile Jia
Venue: ICML 2025 poster (2025)
Keywords: graph neural network
equivariance
parameter-efficient fine-tuning
low-rank adaptation
interatomic potential
Primary area: applications->chemistry_physics_and_earth_sciences

Abstract: Pre-trained interatomic potentials have become a new paradigm for atomistic materials simulations, enabling accurate and efficient predictions across diverse chemical systems. Despite their promise, fine-tuning is often required for complex tasks to achieve high accuracy. Traditional parameter-efficient fine-tuning approaches are effective in NLP and CV. However, when applied to SO(3) equivariant pre-trained interatomic potentials, these methods will inevitably break equivariance—a critical property for preserving physical symmetries. In this paper, we introduce ELoRA (Equivariant Low-Rank Adaptation), a novel fine-tuning method designed specifically for SO(3) equivariant Graph Neural Networks (GNNs), the backbones in multiple pre-trained interatomic potentials. ELoRA adopts a path-dependent decomposition for weights updating which offers two key advantages: (1) it preserves SO(3) equivariance throughout the fine-tuning process, ensuring physically consistent predictions, and (2) it leverages low-rank adaptations to significantly improve data efficiency. We prove that ELoRA maintains equivariance and demonstrate its effectiveness through comprehensive experiments. On the rMD17 organic dataset, ELoRA achieves a 25.5\% improvement in energy prediction accuracy and a 23.7\% improvement in force prediction accuracy compared to full-parameter fine-tuning. Similarly, across 10 inorganic datasets, ELoRA achieves average improvements of 12.3\% and 14.4\% in energy and force predictions, respectively. Code will be made publicly available at https://github.com/hyjwpk/ELoRA.

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Chen Wang" 12
Siyu Hu"
Guangming Tan 12
Weile Jia
Abstract
1. Introduction
Deep learning-based potentials hold significant promise in addressing the accuracy- -fffraccnef trade-off in DFT calcu- lations and expand the spatial and temporal scales of molecular dynamics simulations (Gong al., 2023). A vari- ety of deep learning potentials (Behler & Parrinello, 2007; Smith et al., 2017; Schütt et al., 2017; Gilmer et al., 2017; Zhang et al., 2018; Unke & Meuwly, 2019; Unke et al., 2021; Klicpera et al., 2021) have demonstrated the ability to predict total energies and atomic forces for given mate- rial structures. Recently with the continuous generation of ab initio datasets and deeper understanding of symmetry, pre-trained potentials have emerged in large numbers, such as MACE (Batatia et al., 2022b), M3GNet (Chen & Ong, 2022), Equiformerv2 (Liao et al., 2023), Uni-Mol (Zhou et al., 2023), CHGNet (Deng et al., 2023), SevenNet (Park et al., 2024), GPTFF (Xie et al., 2024), and Orb (Neumann et al., 2024). Among these, equivariant Graph Neural Net- works (GNNs) stand out as particularly effective for mod- cling physical systems. By leveraging equivariance, these models achieve high performance, even with limited first- principles data (Satorras et al., 2021; Batzner et al., 2022; Batatia et al., 2022b; Pelacz et al., 2024). This is particularly valuable because generating first-principles training data often extremely resource intensive. Equivariant preitrained models have been increasingly employed in research in finding symmetry-breaking order parameters (Smidt et al., 2021) and predicting molecular properties (Miller et al., 2020), phonon density of states (Chen et al., 2021), etc.
Pre-trained interatomic potentials have become a new paradigm for atomistic materials simula- tions, enabling accurate and efficient predictions across diverse chemical systems. Despite their promise, fine-tuning is often required for com- plex tasks to achieve high accuracy. Traditional parameter-efficient fine-tuning approaches are ef- fective in NLP and CV. However, when applied to SO(3) equivariant pre-trained interatomic poten- tials, these methods will inevitably break equivari- ance-a a critical property for preserving physical symmetries. In this paper, we introduce ELoRA A (Equivariant Low-Rank Adaptation). a novel fine- tuning method designed specifically for SO(3) equivariant Graph Neural Networks (GNNs), the backhones in multiple pre-trained interatomic po- tentials. ELoRA adopts path-dependent decom- position for weights updating which offers two key advantages: (1) preserves SO(3) equivari- ance throughout the fine-tuning process, ensur- ing physically consistent predictions, and (2) it leverages low-rank adaptations to significantly improve data efficiency. We prove that ELORA maintains equivariance and demonstrate its effec- tiveness through comprehensive experiments. On the rMD17 organic dataset, ELoRA achieves a 25.5% improvement in energy prediction accu- racy and a 23.7% improvement in force predic- tion accuracy compared full-parameter fine- tuning. Similarly, across 10 inorganic datasets, ELoRA achieves average improvements of 12.3% and 14.4% in energy and force predictions, re- spectively. Code will made publicly available at https://gttubb..m//yywwww
However, due to the diversity of material structures and the significant variations of atomic interactions across different materials, pre-trained interatomic potentials may not achieve the desired accuracy when dealing with material structures that are not well learned during training. Moreover, obtain- ing accurate labels for such configurations typically requires costly first frrtt-prnncipls calculations, which further limits the diversity of data available for training. These potentials can only be applied to general downstream tasks without fine- tuning. For example, the accuracy of pre-trained MACE- MP (Batatia et al., 2023) is insufficient study molecular crystal polymorphs (Kaur et al., 2025).
State Key Lab of Processors, Institute of Computing Tech- nology, Chinese Academy of Sciences Uniiversity of Chi- nese Academy of Sciences. Correspondence to: Siyu Hu <husiyu@ict.ac.cn3 Guangming Tan <tgm@ict.ac.ens. Weile Jia <jiaweile@ict.se ie...
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Fine-tuning the pre-trained model with downstream data helps refine the interatomic potential to better predictions.

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Fine-tuning approaches can be broadly categorized into full- parameter fine-tuning and parameter-efficient fine-tuning (PEFT) (Lialin et al., 2023). Full-parameter fine-tuning has been explored in pre-trained interatomic potentials like MACE (Kaur et al., 2025). However, as pre-trained mod- els continue to grow in size, full-parameter fine-tuning be- comes increasingly computationally expensive and memory intensive. Moreover, it may cause catastrophic forgetting of pre-trained knowledge during fine-tuning (Kirkpatrick et al., 2017). In contrast, PEFT methods are designed to modify only a small subset of the model's parameters and keep most parameters fixed. This approach reduces com- putational overhead and memory usage compared to full- parameter fine-tuning. Furthermore, PEFT methods help mitigate catastrophic forgetting, improving model transfer- ability and generalization (Ding et al., 2023). PEFT meth- ods, such as adapter- adapter-based fine-tuning (Houlsby al., 2019) and low-rank adaptation (LoRA) (Hu et al., 2022), have been developed and demonstrated superior advantages in terms of resource efficiency and scalability. However, these meth- ods are developed primarily for transformer- based models. For GNNs, specialized methods such as AdapterGNN (Li et al., 2024) have been introduced to adapt pre-trained GNN models to downstream tasks. Unfortunately, these methods cannot be directly applied to SO(3) equivariant GNNs, as they will destroy the equivariance during fine-tuning. To the best of our knowledge, no PEFT methods have been specifically designed for SO(3) equivariant GNNs.
2. Preliminary
LoRA (Low-Rank Adaptation); Low-Rank Adaptation (LoRA) (Hu et al., 2022) is a PEFT technique for Large Language Model (LLM). The core idea behind LoRA is that the updates to the pre-trained model weights, denoted as Wo E [ during fine-tuning exhibit a low intrinsic rank. As a result, the update AW can be approximated as the product of two low-rank matrices B E IR and A E RTx*, where r is the rank and r < min(d, k). The forward pass h = Wor then be modified as:
h = Wo + AWr = Wax + @ BAx, r
(1)
where a is a scaling constant, A is initialized with random Gaussian distribution and B is zero-initialized, so AW is zero at the beginning of training. LoRA reduces the weights that need to be updated during training. During inference, AW can be merged into Wo. ensuring there is no additional latency compared to full-parameter fine-tuned model.
SO(3) EMPNNS (Equivariant Message Passing Neu- ral Networks): Equivariance is introduced by (Cohen & Welling, 2016) to maintain physical symmetries of the in- put data. They use geometric tensors as node embeddings and ensure equívariance by imposing constraints on the operations that can be performed (Kondor et al., 2018). For- mally, a mapping
X + Y is equivariant for vector spaces X and Y to a group G. if f Dx Dy (g) o is satisfied for any in G. where Dx and Dy represent the transformation of group G on vector spaces X and Y, respectively. More detailed information on group theory and equivariance is displayed in Appendix A. SO(3) equiv- ariant GNNs leverage the inherent symmetry of their do- mains with respect to SO(3), the group of 3D rotations. They transmit equivariant messages between nodes (repre- senting atoms) through edges (representing bonds or in- teractions between atoms) in the graph. This approach is commonly referred to as SO(3) Equivariant Message Passing Neural Networks (EMPNNs). Tensor field net- works (Thomas et al., 2018), NequlP (Batzner et al., 2022), MACE (Batatia et al., 2022b), Allegro (Musaelian al., 2023) and Equiformerv2 (Liao et al., 2023) belong to EMPNNs. Among them, MACE (Batatia et al., 2022b) is representative SO(3) EMPNN model that uses higher body order messages.
This highlights the need for SO(3) equivariant based PEFT methods, as SO(3) equivariant GNN pre-trained mod- els present promising path for modeling universal inter- atomic potentials and play a crucial role in material and chemistry simulations.
Our contributions can be summarized as follows:
We explore the difference between two domi- nant interatomic potential training paradigms (From- Scratch-Training Paradigm and Pre-training-Fine tuning Paradigm) from the perspective of chemical space. We point out that Pre-training-Fine Fine-tuning Paradigm exhibits stronger generalization capabilities.
We develop a PEFT strategy tailored for SO(3) equiv- ariant GNN models, called ELoRA (Equivariant Low- Rank Adaptation). We prove that ELoRA can preserve the equivariance during fine-tuning.
SH (Spherical Harmonics) and TP (Tensor Product): In SO(3) EMPNNs, node features are represented using the coefficients of spherical harmonies, ensuring SO(3) equiv- ariance. Spherical harmonics, denoted as Y are functions that map points on the sphere to real or complex numbers, where / is a non non-negative integer and m is an integer ranging from -1 to I. The features of a node i are represented as h, where is the rotation order with 0 S [ < L. m ranges from -1 to 1, and indicates the channel index with
The experiments show that ELoRA works both on or- ganic and inorganic pre-trained models. For example, on the rMD17 organic dataset, ELORA achieves 25.5% and 23.7% improvement in energy and force predic- tions compared to full full-parameter fine-tuning. On 10 inorganic real cases, the average improvements are 12.3% and 14.4%.
2

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Even though the pre-trained models demonstrate superior generalization capabilities, they may not be sufficiently ac- curate to make predictions on certain out of domain (OOD) data in complex downstream tasks. Pre-training datasets at first-principles accuracy are often sparse across the full chemical space. This is primarily due to the high computa- tional cost of generating such data using density functional theory (DFT), well as the combinatorial complexity of chemical systems. This inherent sparsity presents chal- lenge to the accuracy of pre-trained interatomic potentials.
pre-train
(b)
from scratch
(c)
fine-tune
Fine-tuning is a commonly used technique to improve the performance pre- pre-trained models on specific tasks. Fine- tuning transfers knowledge from a large pre-trained model to more specific task. Through fine-tuning, the performance of the model in specific tasks or compounds enhanced, while its ability generalize across broader chemical space is also retained (depicted in Figure (c)).. As ob- served in Section 5, when trained on the same task-specific dataset, the fine-tuned pre-trained model often achieves bet- ter performance on the test set compared to a model trained from scratch, indicating improved generalization and data efficiency. Moreover, generating training data with ab ini- lio accuracy is computationally costly due the need for density functional theory (DFT) calculations, which makes the data efficiency of fine-tuning particularly valuable in practical applications.
0
0
0
pre-train data
downstream task data
Figure 1. Comparison of chemical space covered by pre-trained (a), from scratch (b), and fine-tuned (c) model. The dark blue part of the figure represents the chemical space covered by the model. The fine-tuned model covers more chemical space than model trained from scratch, as it retains knowledge inherited from the pre-trained model.
4. Equivariant PEFT
s < Kj. The generalized tensor product is an equiv- ariant operation defined using Tebbsch-Gordaa coefficients C' (Griffiths & Schroeter, 2019) as:
EMPNNs can be broken into three main modules: input bedding, interaction blocks, and readout layer. Interaction blocks update the node features iteratively which are the main part of EMPNNS. Notably, the majority of the train- able parameters in EMPNNS are concentrated in the interac- tion blocks. For example, the pre-trained model, MACE- MP (Batatia et al., 2023), approximately 99.7% of the model weights reside in the interaction blocks. Each interaction block is composed of three fundamental operations: point convolution (Thomas et al., 2018), self-interaction (Thomas et al., 2018), and residual connection (He et al., 2016). We prove that these key operations can be induced into a uni- fied tensor prroorrpruucttee (TP) formalism, which enables a consis- tent PEFT method for implementing equivariance. The TP transformation introduced in 4.1. The implementation of ELoRA is described in 4.2.
(u & v)tmm
Σ
clom Lama my Ulymng
This tensor product produces non-zero values only for [4 l1 < lg < (11 + (m, ranges from -ly to (z). Due to this restriction, the tensor product can be divided into multiple different paths by (4, 12, 1a), only the path (4, l2, (a) that satisfies 1/1 - 12 < Is I (11 + (2) is non-zero.
3. Fine-tuning Covers a Larger Chemical Space
Pre-trained models tend to achieve better generalization than task-specific models because they are trained on larger and more diverse datasets. For pre-trained interatomic poten- tials, their generalization ability reflected in covering wider range of chemical space, Figure 1(a) shows. In contrast, task-specifer models trained from scratch tend to have limited generalization capabilities and cover a smaller range of chemical space, as shown in Figure 1(b). These models learn exclusively from the dedicated training set of the downstream task.
4.1. TP transformation
We first make statement about the three key operations and prove that they can be induced into unified TP formalism.
Message passing between adjacent nodes is performed using point convolution. This operation is applied to each node based on its neighboring nodes. To
Point Convolution:
3

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
ensure that the point convolution remains equivariant, the edges are embedded in the following form derive the mi (77,
special cases of the fully connected tensor product. There- fore, the three key operations be induced into a unified TP formalism. In Section 4.2, we focus introducing ELoRA for the fully connected tensor product.
rotation equivariant filter: plats mi (7) 22 (r)
where 22 (r) is learnable function and YM, (7) is the spherical harmonic. When computing the point convolution centered at node i, the contribution of node is:
4.2. ELoRA: Equivariant LoRA
Σ
Flata klima (Tja) X hig.lama
We propose novel path-dependent low-rank adaptation in SO(3) EMPNNs' fine-tuning process. Then we prove that ELoRA can project equivariant messages into low- dimensional space while preserving the inherent equivari- ance.
Mij.klams
12m1.12m2
pagla where "kimm (134) and Mil are considered as tensors and @ denotes the tensor product. Mij.klama is the message passing between nodes and j. By summing Mij.klyms over all the neighboring nodes N (i), we can obtain the messages of node i:
A path-dependent decomposition: The equivariance of the tensor product stems from its constraints on paths. To preserve equivariance, we employ path-dependent decom- position for weight updates, as opposed to the global de- composition used in LoRA. This approach assumes that the weights along each path exhibit a lower intrinsic dimension during adaptation. Formally, for a pre-trained weight matrix W utilized in a fully connected tensor product, we con- strain its updates by decomposing the updates along each path into low-rank representation:
Mi klymyy
Σ (EM(
Mij.klama
(4)
Self-interaction: The self-interaction layer mixes the chan- nels of features at each point which is analogous to 1 x 1
convolutions:
Σ W Rimm K
(5)
W 1shh + AW
Lala = Wighh + Biglah Atglal (8)
where & is the learnable weight.
where W:D
Islah E R x K;), Bight E " xR
Residual connection: The residual connection layer per- forms point convolution on the initial feature of the node (chemical species z:) and the current node feature h,:
Atgh2, E RK and the rank R < min(K], K? K(, ). K is the number of channels, and the superscript of K
indicates whether it is associated with the first (1) or second (2) input tensor or the output tensor (3).
Σ W he
(6)
We illustrate our approach in Figure 2. Taking tensors u and with rotation order 0 <IS as an example, Figure 2(a) demonstrates the fully connected tensor product of and using pre-trained weights W In this process, and first perform a generalized tensor product, but only compute the results for each path without calculating the output tensor, after which W combines the results from different paths into the output tensor. Figure 2(b) shows the corresponding computation process using ELoRA weights A and B. In contrast to the fully connected tensor product, this approach replaces the fully connected operation with two-step process: first, project along the paths using A, and then fully connect the results to the output tensor using B. The result of the fully connected tensor product of u and u using ELoRA fine-tuned weights is the sum of the results from Figure 2(a) and Figure 2(b).
where W k.k is the learnable weight, and the result is added to the residual of the current layer as the feature update.
For tensors u and v with multiple channels, the fully connected tensor product defined to mix different channels. Specifically, (u @ halamy defined as:
Fully Connected Tensor Product:
Σ (nlymy,kkaam
Waskeh Cammetame Wkimm Ukalaman
(7)
where Wkykeh Ashah is the learnable weight corresponding to the path (1,, lz).
Proposition 4.1. The self-interaction of the tensor h is equivalent to the fully connected tensor product of h and 1.
Proposition 4.2. The residual connection of the tensor hy is equivalent the fully connected tensor product of h4 and 2, where 2; is the one-hot encoding of the chemical species 24.
We give the following propositions to explain how ELORA projects equivariant messages into low-dimensional space while preserving the inherent equivariance. The proofs are detailed in Appendix C.
The proofs of the propositions are given in Appendix C. Proposition 4.1 and Proposition 4.2 point out that the for- malization of the self-interaction and residual connection are
Proposition 4.3. For the pre-trained weight matrix Wo of the fully connected tensor product and its corresponding ELORA matrices A and B, the fully connected tensor prod-
4

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Input Tonsor
Tensor product
Fully connect
Outpui tensor
Iaput lansor
Temser product
Downward project
Fully connect
Ourput tensor
Auu
Da
Avp
Bus
Kisk]
P-1
Bo
War
K7
Aue
Bgg
Wi
Aril
R
B2
W.
Kj*K
Kj+K
Figure overview ELoRA. (a) the fully connected tensor product of tensors and e using pre-traines weights W (b) the corresponding computation process using EL.RA weights A and B. The final result is the sum of the results of (a) and (b).
uct of the tensors u and e with ELoRA is SO(3) equivariant.
matrices A and B, Atatal projects the equivariant message the path (11, 12, 13) in the fully connected tensor product of the tensors u and with ELoRA into low-dimensional space of dimension R.
Proposition 4.3 demonstrates that the overall rotational cquivariance is preserved even after incorporating ELoRA. We then present the following proposition to illustrate how the
decomposition employed by ELoRA projects the equivariant messages, depicted in Figure 2. Proposition 4.4. For the pre-trained weight matrix Wo of the fully connected tensor product and its correspond- ing ELoRA matrices and B, the fully connected tensor product of the tensors u. and with ELoRA satisfies (u & U)kkkmmg (u & v) U)kalsms En, N2 E, Aglrd (u,, @ Mi (lggg where (u 6 o)balamg is the fully connected tensor kalama
5. Experiments
5.1. Experimental Setup
5.1.1. PRE-TRAINED MODELS
MACE (Batatia et al., 2022b) is a fundamental EMPNN that utilizes high body order messages and achieves state- of-the-art prediction performance. We adopt the MACE architecture the backbone to evaluate the effectiveness of EL Details of the MACE architecture are provided in Appendix B. Due to the distinct properties of organic and inorganic systems, the pre-trained models are divided into two classes, the organic pre-trained model, MACE- OFF (Kováes et al., 2023) and the inorganic pre-trained model, MACE-MP (Batatia al., 2023). MACE-OFF is pre- trained on SPICE (Eastman et al., 2023), while MACE-MP is pre-trained on MPTrj (Deng et al., 2023). More details about the pre-trained datasets can be found in Appendix E.
product of tensors u and using w without ELoRA and (m,, (ig)tggm is the fully connected tensor product of tensors ut, and Ui, using ELoRA matrix A.
Proposition 4.4 states that, after applying ELoRA, the fully connected tensor product of the tensors u and v can be decomposed into components. The first component is the fully connected tensor product computed using W The second component involves using Aiglsh to project the message along the path (41,4, 43) onto (w, @ Vla rrlams where the projected (u, 8 has its channels reduced to This projection then followed by fully connected operation applied to the projected message using Biglals which produces the output tensor. Therefore, have the following corollary.
1.2. DATASETS
For organic downstream tasks, we use the revised MD17 (rMD17) (Christensen & Von Lilienfeld, 2020). 3BPA (Kovács et al., 2021), and AcAc (Batatia al., 2022a) datasets, which representative benchmarks for organic systems. For inorganic downstream tasks, we employ 10
Corollary 4.5. For the pre-trained weight matrix w of the fully connected tensor product and its corresponding ELORA
5

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
datasets that reflect variety of real-world scenarios. These datasets are based on simulations of diverse inorganic com- pounds under complex temperature and pressure conditions, encompassing a wide range of chemical and configura- tional spaces. They include datasets for metal elements (Cu (Zhang et al., 2020), Sn (Chen et al., 2023), Ti (Wen et al., 2021), V (Wang et al., 2022a), W (Wang et al., 2022b), AgAu (Wang et al., 2021), AlMgCu (Jiang et al., 2021)), water/ice systems (Zhang et al., 2021), oxides (HfO2 (Wu et al., 2021)), and solid-state electrolytes (SSE-PBE (Huang et al., 2021)). These datasets are generated using the DP- GEN method (Zhang et al., 2020). Details of these datasets are given in Appendix E.2.
Table 1. MAE of energy (E, me and force (F, meV/A) on the rMD17 dataset. Each model is trained using only 50 molecules. The best results are in bold.
ACB
NaqulP
PACE
MACE
MACE
MACE
Method
From scratch
Full Dall-parmeeerr
ELORA
Aspirin
E
63.8
26.2
52.0
19.5
37.4
15.7
43.9
17.0
23.6
10.3
7.3
Azobenzene
F
E
28.8
9.0
20.0
6.0
17.5
6.7
17.7
5.4
15.1
4.7
12.4
4.0
Benzznne
2.7
0.2
2.9
0.6
3.3
0.7
2.7
2,4
Bihnn]
E
43.0
8.6
40.2
8.7
25.4
6,3
32.6
6.7
13,9
2.7
2.1
Malooaláchyde
12.8
12.7
11.5
10.0
7.3
6.5
F
63.5
$2.5
57.3
43.3
25.3
21.7
5.1.3. BASELINE
Naphthalene
19.7
3.8
10.0
2.1
2.1
9.7
9.2
2.1
1.8
We compare the accuracy of models trained from scratch, pre-trained models, pre-trained models with full-parameter fine-tuning, and pre-trained models with ELoRA fine-tuning.
Paracetamol
E 13.6
14.3
10.1
9.7
6.9
4.8
45.7
39.7
29.3
31.5
20.4
14.8
Salicylic acid
41.7
8.9
35.0
29.2
7.0
28.4
6.5
17.8
14.2
The baseline for models trained from scratch includes ACE (Kovács et al., 2021), NequlP (Batzner et al., 2022), BOTNet (Batatia et al., 2022a), Allegro (Musaelian et al., 2023), DPA2 (Zhang et al., 2023), PACE (Xu et al., 2024), and MACE (Batatia al., 2022b), all of which have demon- strated strong performance across various downstream tasks. The baseline for pre-trained models consists of the open- source organic MACE-OFF and inorganic MACE-MP. Both full-parameter fine-tuning and ELoRA fine-tuning are ap- plied to these same pre-trained models (MACE-OFF and MACE-MP) under identical hyperparameter settings. More details of these models are provided in Appendix E.3.
Toluene
E
5.3
3,3
2.7
3.1
1.9
1,3
27.1
12.0
12.1
Unsiil
E
6.5
7.3
5.9
4.4
2.6
2.1
F
36.
40.1
26.8
25.9
14.7
11.6
K. The results (in Table 2) of MACE without fine-tuning are significantly less accurate than those obtained with fine- tuning, highlighting the necessity of fine-tuning pre-trained models. Furthermore, ELoRA fine-tuning outperforms train- ing from scratch and achieves a new state-of-the-arr. As the test set temperature increases from 300 K to 1200 K, the accuracy gap between the ELoRA fine-tuned model and the full- aall-aramettr fine-tuned model grows from 0.3 (m to 2.7 (me for energy and from 0.3 (me V/Ā) to 6.7 (meV/A) for forces. This demonstrates that the ELORA fine-tuned model exhibits superior out-of-domain generalization capa- bilities.
5.2. Main Experimental Results
1. RESULTS ON THE ORGANIC DATASETS
Revised MD17: To verify whether fine-tuning pre-trained models offers an advantage on downstream tasks with lim- ited data, follow the previous setting (Batatia et al., 2022b), using only 50 configurations for each organic molecule in the rMD17 dataset for training. As shown in Table 1, fine-tuning on MACE-OFF achieves higher accu- racy than training from scratch. This is consistent with the analysis in Section 3, which indicates that fine-tuning will cover a broader chemical space in real cases. Furthermore, ELORA fine-tuning outperforms full-pan parameerrrtrr fine-tuning across all organic molecules listed in Table 1, achieving new stateoof-the-ar results. Compared to full-parameter fine-tuning, ELoRA improves energy prediction accuracy by 25.5% and force prediction accuracy by 23.7%.
In Figure 3, we compare the predictions of models trained on the 3BPA dataset for three dihedral slices. Overall, the results show that the ELoRA fine-tuned model aligns more closely with the ground truth. Notably, in the middle panel, which includes geometries farthest from the training set, the ELoRA fine-tuned model significantly outperforms the model trained from scratch, demonstrating remarkable ex- trapolation capabilities.
The AcAc dataset is designed to evaluate a model's ability to extrapolate on the acetylacetone molecule under high temperatures. bond breaking, and bond torsion. As shown in Table 3, the pre-trained model MACE-OFF, when used without fine-tuning, performs significantly worse than the model trained from scratch. ELORA fine-tuning outper- forms training from scratch and full- hull-parameter fine-tuning in all cases, achieving improvements of 26.4% and 14.4% in energy and force accuracy, respectively. After fine-tuning with ELORA, the fine-tuned parameters can be merged into
AcAc:
3BPA: Compared to the rMD17 task with limited data, the 3BPA dataset is used to evaluate the model's extrapolation capabilities under varying temperatures and dihedral slices. The models are trained using datasets collected at 300 K, and the temperatures of the test set range from 300 K to 1200
6

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Table 2. RMSE of energy (E, m) and foree (E, meV/ on the 3BPA dataset. The training set is collected at 300 K. Standard deviations are computed over three runs shown brackets. The results are in bold.
Allegro
NequlP
BOTNet
PACE
MACE
MACE
MACE
MACE
Method
From scratch
Full-parameter
ELoRA
MACE-OFP
3.84 (0.08)
3.3 (0.1)
3.1 (0.13) 11.0 (0.14)
2,4 (0.1)
(.0)
3.3 (0.03)
3.0 (0.05)
17340.0
300 K
12.98 (0.17)
10.8 (0.2)
9.1 (0.1)
8.8 (0.3)
7.8 (0.01)
7.5 (0.05)
124.7
12.07 (0.45)
11.2 (0.1)
11.5 (0.6)
7.9 (0.2)
9.7 (0.5)
7.3 (0.04)
6.5 (0.10)
17369.8
600 K
29.17 (0.22)
26.4 (0.1)
26.7 (0.29)
21.4 (0.3)
21.8 (0.6)
16.6 (0.05)
15.5 (0.12)
136.
E
42.57 (1.46)
(1.6)
(1.1)
29.6 (0.4)
29.8 (1.0)
20.3 (0.17)
17.6 (0.11)
17412.8
1200
F
82.96 (1.77)
76.2 (1.1)
81.1 (1.5)
60.7 (2.0)
62.0 (0.7)
48.7 (0.56)
42.0 (0.51)
157.3
16.3 (1.5)
7.6 (0.4)
7.8 (0.6)
7.3 (0.28)
(0.28)
17309.4
Dihedral Slices
20.0 (1.2)
16.0 (0.5)
16.5 (1.7)
12.3 (0.10)
11.4 (0.17)
116.2
71.0%, 120.0
67.0", 8 150.0"
151.0% 180.0
Prozs sculch
Prou sealed
ELoRA
ELoRA
AB (meV)
JB (mx+V
30
30
20-
Proas woulds
-10
KLaRA
01
10
100
-50
2001
200
100
50
100
150
100
150
300
30
100
150
700
200 300
(*)
Figure Energy predictions for the three cuts the potential energy surface the 3BPA molecule by models trained from scratch and fine-tuned with ELoRA. The energy (DFT) is shown in black. For each cut, the curves have been vertically shifted so that the minimum true energy is zero. The shaded area represents the standard deviation.
the original model parameters during inference, adding additional computational overhead. The reduction in the number of parameters in the fine-tuned model results from removing the weights associated with those elements of the pre-trained model that are absent from the downstream task.
ELoRA fine-tuned pre-trained models outperform other models in force prediction on 9 out of 10 datasets. For energy prediction, the fine-tuned pre-trained models exceed the performance of the models trained from scratch using the same backbone, but are slightly behind state-of-the-art, likely due to differences model architecture. Compared to full- full-parameter fine-tuning, ELoRA achieves average improvement of 12.3% in energy prediction and 14.4% in force prediction across the 10 inorganic datasets.
5.2.2. RESULTS ON INORGANIC DATASETS
In this section, we evaluate the performance of ELoRA fine-tuning inorganic datasets. Inorganic crystals are much more diverse than small organic molecules, and the 10 datasets used in this section are generated under com- plex temperature and pressure conditions, requiring models to demonstrate strong generalization capabilities. Model performance is evaluated using RMSE, with the results sum- marized in Table 4.
5.3. Other Experimental Results
5.3.1. DATA EFFICIENCY
Data efficiency can be measured by the size of the required training data. Taking rMDI7-aspirin as example, Figure describes the comparison of full-parameter fine-tuning and ELoRA on the pre-trained model. ELoRA obtains lower error than full parameetr fine-tuning when the training data size varies from 50 to 1000. ELoRA shows lower error than
The results show that pre-trained models MACE-MP, when used without fine-tume fineituning, significantly underperform mod- els trained from scratch in inorganic datasets. However,

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Table 3. RMSE of energy (E, meV) and force (F, m/ on the acctylacetone dataset. The training set is collected at 300 K. Standard deviations are computed over three runs and shown brackets. The best results are in bold.
BOTNet
NequiP
MACE
MACE
MACE
MACE
Method
From scratch
Full-pane Full-parameter
ELoRA
MACE-OK OFF
E
0.89 (0.0)
0.81 (0.04)
0.9 (0.03)
1.0 (0.02)
0.8 (0.03)
24183.8
300 K
F
6.3 (0.0)
5.90 (0.38)
5.1 (0.10)
5.1 (0.07)
4.5 (0.06)
463.8
E
6.2 (1.1)
6.04 (1.26)
4.6 (0.3)
5.8 (0.28)
3.9 (0.33)
24160.1
600 K
29.8 (1.0)
27.8 (3.29)
22.4 (0.9)
16.4 (0.70)
13.6 (0.26)
474.2
No Parameters
2,756,416
3,190,488
2,803,984
751,896
751,896
1,428,368
full-parameter fine-tuning when the size of the downstream data is smaller. In cases of small data size, the amount of data required for fine-tuning can be reduced by 42% and 44% in reaching similar errors, thereby enhancing data efficiency. As the training data increases, the difference in error between full-parameter fine-tuning and ELoRA gradually decreases. Theoretically, their accuracy gap will diminish as the dataset size approaches infinity. Detailed theoretical analysis can be found in Appendix D.
7-appiin
Cu
1.1
120
0/131
Number panmmettees
L20
010-
Looe
0.033
0.025
0155
0.074
0.022
Rank
Figure 5. The validation set loss and ELoRA parameter propor- tion across different ranks. The results are obtained from tests conducted on the rMD17-aspirin and inorganic- datasets. The shaded area represents the standard deviation.
Full parameter
ELoRA
10
42% data reduction
44% data reduction
MAE E [mee]]
MAE F [meV)A]
findings suggest that setting rank = 16 achieves a favorable balance, with the ELoRA parameters accounting for 22.5% and 24.0% of the original model parameters in the respective datasets.
10 Training data size
Training data size
6. Related Work
Figure 4. Comparison of energy and force MAE trends between Full- call-parrmeter fine-tuning and ELoRA across different train- ing data sizes. The results obtained from experiments con- ducted on the rMDI7-aspirin dataset.
Neural Network Interatomic Potentials: The traditional approach to using machine learning for predicting molecu- lar energy and force combines hand-crafted representations of the atomie neighhorhood with neural networks (Behler & Parrinello, 2007). Recent advancements have shifted to- wards end-to-end learnable models based on GNNs, which learn representations directly from atom types and molec- ular coordinates without requiring hand-crafted features. Early GNN models, such as CGCNN (Xie & Grossman, 2018), DimeNet (Klicpera al., 2020), SphereNet (Liu et al., 2022), Ewald message passing (Kosmala et al., 2023). PotNet (Lin et al., 2023), and TGT (Hussain et al., 2024) fo- cused on scalar representations that achieve rotational equiv- ariance using invariant features. More recently, equivariant models like PaiNN (Schütt et al., 2021), SEGNN (Brandstet- ter et al., 202 NewtonNet (Haghighatlari et al., 2022), and GeoMFormer (Chen al., 2024) have been proposed, mark-
5.3.2. RANK SETTING
The hyperparameter rank in ELoRA directly influences the number of trainable parameters. To evaluate its impact on model performance, we conduct experiments using rMD17- aspirin and inorganic-Cu datasets. As shown in Figure 5, the results indicate that as rank increases, the validation loss ini- tially decreases but then starts to rise. When the rank is too small, the model tends to underfit the data, whereas larger rank may lead to overfitting. This aligns with theoretical analysis: as the number of trainable parameters increases, the model transitions from underfitting to overfitting. Spe- cific theoretical analyses can be found in Appendix D. Our
8

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Table RMSE of 10 different inorganic datasets. The best results are in bold.
Energy RMSE [iato))
Force RMSE [meVIA]
NoqulP
Allogro
DPA2
MACE
MACE
NequlP
Allegro
DPA2
MACE
MACE
MACE
Method
From scratch
Full Full-parancter
ELORA
MACE-MP
From scratch
Full-parameter
ELoRA
MACE-MP
SSE-PRI PBE-
1.6
1.4
1.8
0.3
0.3
243.5
41.1
50.3
29.9
16.9
14.2
167.4
H20-PD>
0.9
79.9
88
213.1
27.1
24.7
29.7
19.9
16.1
1924.9
AgUAu-PPEE
12.3
39.2
2.4
369.1
10.5
8.1
531.6
43.8
58.9
17.8
34.5
11.3
9.5
$30.6
AlUMgUCa
38.0
2.1
7.7
3.0
2.2
919.1
48.3
40.6
19.1
42.9
9.9
8.8
227.9
18.2
3.5
1
4.9
4.6
595.6
62.2
40.2
54.4
1
31.7
29.2
164.9
T1
27.6
6.9
5.0
8.3
5.9
5.8
2668.2 2
137,4
85.6
113.1
94.2
79.4
73.3
302.1
20.8
4.0
5.6
15.6
4.7
3025 2
160.4
101.6
108.1
181.2
87.2
78.7
893.6
HRO2
1.5
1.4
L.0
2.3
0.5
0.3
635.88
58.8
64.0
54.2
14.7
30.0
21.0
223.8
ing a significant advancement in leveraging equivariance for modeling interatomic potentials.
intrinsic equivariance can be preserved. ELoRA not only guarantees physically consistent predictions, but also en- hances data efficiency. Compared to full-parameter fine- tuning, ELoRA achieves SOTA performance on the organic rMD 17, 3BPA, and AcAc datasets, well as on 10 chal- lenging inorganic real cases. ELoRA transforms fine-tuning practices for equivariant models, paving the way for more accurate and efficient atomistic simulations across wide range of chemical systems.
Craameter-Efficieen Fine- Tuning: Parameter-efficient fine- tuning (PEFT) techniques aim to reduce the number of parameters required for fine-tuning, addressing the inef- ficiency of fully updating all model parameters the model size and task complexity grow (Ding et al., 2022). For instance, adapter tuning (Chen et al., 2022) introduces adapter modules with bottleneck architectures between lay- ers. BitFit (Zaken et al., 2022) focuses on updating only the bias terms while keeping the rest of the parameters fixed. LoRA (Hu et al., 2022) reduces the number of trainable parameters by decomposing weight matrices into low-rank matrices. Some theoretical research (Hao et al., 2024; Jang et al., 2024) and variants of LoRA (Berman & Peherstorfer, 2024; Feng et al., 2024; Hayou et al., 2024; Ostapenko et al., 2024; Qin et al., 2024; Zhang& & Pilanci, 2024; Zhou et al.,
While our method shows promising results, the following aspects still remain open for further works: (1) Identify- ing other more effective weight decomposition strategies. (2) Enhancing data efficiency and reducing the number of samples needed for fine-tuning achieve few-shot even one-shot learning. (3) Reducing the training epochs required for fine-tuning. (4) Establishing more rigorous proofs for the error generalization bounds of ELoRA.
2024) have also been proposed. In the field of GNNs, meth- ods such GPPT (Sun al., 2022), MolCPT (Diao al., 2022), GPF (Fang et al., 2024), and GraphPrompt (Liu al., 2023) have been introduced. However, these parameter- efficient fine-tuning methods are not applicable to equivari- ant models, as they will destroy the fundamental equivari- ance the model during the fine-tuning process.
Acknowledgments
This work is supported by the following funding: Na- tional Science Foundation of China (62032023, 92270206, 62372435, T2125013, 61972377, 61972380, T2293702) and China National Postdoctoral Program for Innovative Talents (BX20240383), and Huawei Technologies Co., Ltd.. The Al-driven experiments, simulations and model training performed on the robotic Al-Scientist platform of Chi- Academy of Science. We thank Prof. Lin- Hann Wang and Dr. Haibo Li for helpful discussions.
7. Conclusion
Fine-tuning pre-trained interatomic potentials is erucial to enhancing model accuracy on downstream tasks. Equiv- ariant pre-trained models play a pivotal role in machine learning-based interatomic potentials, maintaining the phys- ical symmetries, which essential for accurate and reli- able atomistic materials simulations. Although various parameter-efficient fine-tuning methods have been made successful in NLP and CV, they fall short when applied to equivariant models because they will not preserve equivari- ance. To address this challenge, we propose a novel PEFT method called ELORA. By leveraging path-dependent de- composition for weight updates, ELoRA ensures that the
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
References
molecular representation learning. In International Con- ference Machine Learning, pp. 7118-7142. PMLR, 2024.
Batatia, I., Batzner, S., Kovács, D. P., Musaclian, A., Simm, G. N., Drautz, R., Ortner, C., Kozinsky, B., and Csányi, G. The design space of (euequvrrr centecenteeed interatomic potentials. arXiv preprint arXiv:2205.0666 2022a.
Chen, Z., Andrejevic, N., Smidt, T., Ding, Z., Xu, Q., Chi, Y.-T., Nguyen, Q. T., Alatas, A., Kong, J., and Li, M. Direct prediction of phonon density of states with euclidean neural networks. Advanced Science, 8(12):
Batatia, I., Kovaes, D. P., Simm, G., Ortner, C., and Csányi, G. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. Ad- vances in Neural Information Processing Systems, 35: 11423-11436, 2022b.
2004214, 2021.
Chmiela, S., Tkatchenko, A., Sauceda, H. E.,, Poltavsky, I., Schütt, K. T., and Müller, K.-R. Machine learning of ac- curate
molecular foree fields. Science advances, 3e 1603015, 2017.
Batatia, I., Benner, P., Chiang, Y., Elena, A. M., Kovács, D. P. Riebesell, J., Advincula, X. R., Asta, M., Avaylon, M., Baldwin, W. J., et A foundation model for atomistic materials chemistry. arXiv preprint arXiv:2401.00096 2023.
Christensen, A. S. and Von Lilienfeld, O. A. On the role of gradients for machine learning of molecular energies and forces. Machine Learning: Science and Technology, 1(4): 045018, 2020.
Batzner, S., Musaelian, A., Sun, L., Geiger, M., Mailoa, J. P.. Kornbluth, M., Molinari, N., Smidt, T. E., and Kozinsky, B. E (3)-euuiva graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):2453, 2022.
Cohen, T. and Welling, M. Group equivariant convolutional networks. In International conference on machine learn- ing, pp. 2990-2999. PMLR, 2016
Deng, B., Zhong, P., Jun, K., Riebesell, J., Han, K., Bartel, C. J., and Ceder, G. Chgnet as a pretrained universal neural network potential for charge-informn atomistic modelling. Nature Machine Intelligence, 509):1031-,041, 2023.
Behler, J. and Parrinello, M. Generalized neural-netwon representation of high-dimensional potential-energy sur- faces. Physical review letters, (8(14):14640) 2007.
Berman, J. and Peherstorfer, B. Colora: Continuous low- rank adaptation for reduced implicit neural modeling of parameterized partial differential equations. In Interna- tional Conference on Machine Learning, pp. 3565-3583. PMLR. 2024.
Diao, C., Zhou, K., Liu, Z., Huang, X., and Hu, X. Mol- cpt: Molecule continuous prompt tuning general- ize molecular representation learning. arXiv preprint arXiv: 22..: 10614, 2022.
Ding, N., Qin. Y.. Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint N0w:22033.69044 2022.
Brandstetter, J., Hesselink, R., van der Pol, E., Bekkers, J., and Welling, M. Geometric and physical quantities improve (3) equivariant message passing. arXiv preprint arXiv:2110.02905. 2021.
Chen, C. and Ong, S. P. A universal graph deep learning interatomic potential for the periodic table. Nature Com- putational Science, 2(1 ((111))7882221 1788 2022.
Ding, N., Qin, Y., Yang, G., Wei, F., Yang. Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-22 2023.
Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transform- for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-166 16678, 2022.
Eastman, P., Behara, K., Dotson, L.,, Galvelis, R., Herr, J. E., Horton, J. T., Mao, Y., Chodera, J. D., Pritchard, B. P., Wang, Y., et al. Spice, a dataset of drug like molecules and peptides for training machine learning potentials. Scientific Data, 10(1):11, 2023.
Chen, T., Yuan, F., Liu, J., Geng, H., Zhang, L., Wang, H., and Chen, M. Modeling the high-pressure solid and liquid phases of tin from deep potentials with ab initio accuracy. Physical Review Materials, 755)0033033, 2023.
Fang, T., Zhang, Y., Yang, Y., Wang, C., and Chen, Uni- versal prompt tuning for graph neural networks. Advances Neural Information Processing Systems, 36, 2024.
Chen, T., Luo, S., He, D., Zheng, S., Liu, T.-Y., and Wang, L. Geomformer: general architecture for geometric
10

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Feng, W., Zhou, W., He, J., Zhang, Wei, T., Li, G., Zhang, T., Zhang, W., and Yu, N. Aqualora: Toward white-box protection for customized stable diffusion models via watermark lora. In International Conference on Machine Learning, pp. 13423-13444. PMLR, 2024.
Hussain, M. S., Zaki, M. J., and Subramanian, D. Triplet interaction improves graph transformers: accurate molec- ular graph learning with triplet graph transformers. In Proceedings of the 41st International Conference on Ma- chine Learning, pp. (0768-20792, 2024.
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quantum chem- istry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Isert. C., Atz, K., Jiménez-Luna, J., and Schneider, G. Qmugs, quantum mechanical properties of drug-like molecules. Scientific Data, 9())2273, 2022.
Jain, A., Ong, S. P., Hautier, G., Chen, W., Richards, W. D., Dacek, S., Cholia, S., Gunter, D., Skinner, D., Ceder, G., et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials, 1(1), 2013.
Gilmore, R. Lie groups, physics, and geometry: an introduc- tion for physicists, engineers and chemists. Cambridge University Press, 2008
Gong, X., Li, H., Zou, N., Xu, R., Duan, W., and Xu, Y. General framework for e (3) (3)-equivariant neural network representation of density functional theory hamiltonian. Nature Communications, 14(1):2848, 2023.
Jang, U., Lee, J. D., and Ryu, E. K. Lora training in the ntk regime has no spurious local minima. In Inter- national Conference on Machine Learning, pp. 21306- 21328. PMLR, 2024.
Griffiths, D. J. and Schroeter, D. F. Introduction to quantum mechanics. Cambridge university press, 2019.
Jiang, W., Zhang. Y., Zhang, L., and Wang. H. Accurate deep potential model for the al-cu-mg alloy in the full concentration space. Chinese Physics B, 30(5):050706, 2021.
Haghighatlari, M., Li, J., Guan, X., Zhang, O., Das, A., Stein, J., Heidar -Zadeh, F., Liu, M., Bertels, L.., et al. Newtonnet: a newtonian message pass- ing network for deep learning of interatomic potentials and forces. Digital Discovery, 1(3):333 343, 2022.
M.,
Ju, H., Li, D., Sharma, A., and Zhang, H. R. Generalization in graph neural networks: Improved pac- pac-bayesian bounds on graph diffusion. In International Conference on Artifi- cial Intelligence and Statistics, pp. 6314-6341. PMLR, 2023.
Hao, Y., Cao, Y., and Mou, L. Flora: Low-rank adapters are secretly gradient compressors. In International Con- ference on Machine Learning, pp. 17554-17571. PMLR, 2024.
Kaur, H., Della Pia, F., Batatia, I., Advincula, X. R., Shi, B. X., Lan. J., Csányi, G., Michaelides, A., and Kapil, V. Data-efficient fine-tuning of foundational models for first-principles quality sublimation enthalpies. Faraday Discussions, 2025.
Hayou, S., Ghosh, N., and Yu, B. Lora+ efficient low rank adaptation of large models. In Proceedings of the 41st International Conference on Machine Learning. pp- 17783-17806, 2024.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska- Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-3355
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
2017.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q.. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning. pp. 2790- 2799. PMLR, 2019.
Klicpera, J., GroB, J., Günnemann, S., et al. Directional message passing for molecular graphs, In ICLR, pp. 1-13, 2020.
Klicpera, J., Becker, F., and Günnemann. S. Gemnet: Uni- versal directional graph neural networks for molecules. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 6790-6802, 2021.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Huang, J., Zhang, L., Wang, H., Zhao, J., Cheng, J., et al.
Kondor, R., Lin, Z., and Trivedi, S. Clebsch-gordan nets: a fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems, 31, 2018.
Deep potential generation scheme and simulation proto- col for the li10gep2s12-yy 2-type superionic conductors. The Journal of Chemical Physics, 154(9), 2021.

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Kosmala, A., Gasteiger, J., Gao, N., and Günnemann, S. Ewald-based long-range message passing for molecular graphs. In International Conference on Machine Learn- ing, Pp. 11544-7563. PMLR, 2023.
McAllester, D. A. Pac-bayesian model averaging. In Pro- ceedings of the twelfth annual conference Computa- tional learning theory, pp. 164-170, 1999.
Miller, B. K., Geiger, M., Smidt, T. E., and Noé, F. Relevance of rotationally equivariant convolutions for predicting molecular properties. arXiv preprint AXiv:2008.08461 2020.
Kováes, D. P., Oord, C. v. d., Kucera, J., Allen, A. E., Cole, D. Ortner, C., and Csányi, G. Linear atomic cluster expansion force fields for organic molecules: beyond rmse. Journal of chemical theory and computation, 17 (12):7666-7711, 2021.
Musaclian, A., Batzner, S., Johansson, A., Sun, L.,, Owen, C. J., Kornbluth, M., and Kozinsky, B. Learning local equivariant representations for large-scale atomistic dy- namics. Nature Communications, 14(1):579, 2023.
Kovács, D. P., Moore, J. H., Browning, N. J., Batatia, I., Horton, J. T., Kapil, V., Witt, W. C., Magdäu, L.-B., Cole, D. and Csányi, G. Mace-off23: Transferable machine learning force fields for organic molecules, 2023.
Neumann, M., Gin, J., Rhodes, B., Bennett, S., Li, Z., Choubisa, H., Hussey, A., and Godwin, J. Orb: A fast, scalable neural network potential. arXiv preprint arXiv:2410.22570, 2024.
Kresse, G. and Furthmüller, Efficiency of ab-initio total energy calculations for metals and semiconductors using plve basis set. Computational materials science, 6(1):15-50, 1996a.
Neyshabur, B., Bhojanapalli,, S., and Srebro, N. A pac-bayesian approach to spectrally margin bounds for neural networks. In International Conference Learning Representations, 2018
Kresse, G. and Furthmülleer, J. Efficient iterative schemes for ab initio total-energy calculations using a plane- plane-wave wave basis set. Physicul review B, 54(16): 1169,, 169,, 1996b.
Ostapenko, O., Su, Z., Ponti, E., Charlin, L., Le Roux, N., Caccia, L., and Sordoni, A. Towards modular Ilms by building and reusing a library of loras. In Inter- national Conference on Machine Learning, pp. 38885- 38904. PMLR, 2024.
Li, S., Han, X., and Bai, J. Adaptergnn: Parameter-efficient fine-tuning improves generalization in gnns. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13600-13608, 2024.
Lialin, V., Deshpande, V., and Rumshisky, A. Scaling down to scale up: guide to parameter-efffiient fine-tuning. arXiv preprint ar arXi::203. 2023.
Park, Y., Kim, J., Hwang, S., and Han, S. Scalable parallel algorithm for graph neural network interatomic potentials in molecular dynamics simulations. Journal of Chemical Theory and Computation, 2024.
Liao, Y.I..,, Wood, B., Das, A., and Smidt, T. Equiformerv2: Improved equivariant transformer for scaling to higher- degree representations. arXiv preprint arXiv:2306. 12059 2023.
Pelaez, P., Simeon, G., Galvelis, R., Mirarchi, A., East- man, P., Doerr, S., Tholke, P., Markland, T. E., and De Fabritis, G. Torchmd net 2.0: Fast neural network potentials for molecular simulations. Journal of Chemical Theory and Computation, 2024.
Lin, Y., Yan, K., Luo, Y., Liu, Y., Qian, X., and Ji, S. Effi- cient approximations of complete interatomic potentials for crystal property prediction. In International Confer- ence on Machine Learning, pp. 11260-21288 PMLR, 2023.
Qin, H., Ma, X., Zheng, X., Li, X., Zhang, Y., Liu, S., Luo, J., Liu, X., and Magno, M. Accurate lora-finetuning quantization of Ilms via information retention. In Inter- national Conference Machine Learning, pp. 41498 41516. PMLR, 2024.
Liu, Y., Wang, L., Liu, M., Lin, Y., Zhang, X., Oztekin, B., and Ji, S. Spherical message passing for 3d molec- ular graphs. In International Conference on Learning Representations (ICLR), 2022.
Satorras, V. G., Hoogeboom, F., and Welling, M. E (n) equivariant graph neural networks. In International con- ference on machine learning, pp. 9323 9332.933. PMLR, 2021.
Liu, Z., Yu, X., Fang, Y., and Zhang, X. Graphprompt: Uni- fying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, pp. 417-428, 2023.
Schütt, K., Kindermans, P.-J., Sauceda Felix, H. E., Chmiela, S., Tkatchenko, A., and Müller, K.R.. Schnet: A continuous filter convolutional neural network for model- ing quantum interactions. Advances in neural information processing systems, 30, 2017.
McAllester, D. A. Some pac- noc-bayesia theorems. In Proceed- ings of the eleventh annual conference on Computational learning theory, pp. 230-234, 1998.
12

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Schütt, K., Unke, O., and Gastegger, M. Equivariant mes- sage passing for the prediction of tensorial properties and molecular spectra. In International Conference Machine Learning, pp. 9377- 9388. PMLR, 2021.
Wen, T., Wang, R., Zhu, L.., Zhang, L.,, Wang. H., Srolovitz, D. J., and Wu, Z. Specialising neural network potentials for accurate properties and application to the mechanical response of titanium. npj Computational Materials, 7(1):
206, 2021.
Shawe-Taylor, J. and Williamson, R. C. A pac analysis of bayesian estimator. In Proceedings of the tenth annual conference on Computational learning theory, pp. 2-9, 1997.
Wu, J., Zhang, Y., Zhang, L., and Liu, S. Deep learning of accurate force field of ferroelectric hfo 2. Physical Review B, 103(2):024108, 2021.
Smidt, T. E., Geiger, M., and Miller, B. K. Finding symme- try breaking order parameters with euclidean neural net- works. Physical Review Research, 3(1::001200,, 2021.
Xie, F., Lu, T., Meng, S., and Liu, M. Gptff: A high- accuracy out-of-the-box universal ai force field for arbi- trary inorganic materials. Science Bulletin, 69(22):3525- 3532, 2024.
Smith, J. S., Isayev, O., and Roitherg, A. E. Ani-1: an extensible neural network potential with dfft accuracy at force field computational cost. Chemical science, 8(4): 3192-3203, 2017.
Xie, T. and Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable predic- tion of material properties. Physical review letters, 120 (14):145301, 2018.
Sun, M. Zhou, K., He, X., Wang, Y., and Wang, X. Gppt: Graph pre-training and prompt tuning to gener- alize graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1717-1727,202
Xu, Z., Yu, H., Bohde, M., and Ji, S. Equivariant graph network approximations of high- polynomials for force field prediction. arXiv preprint arXiv:2411.04219, 2024.
Thomas, N., Smidt, T., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation-ant and translation cessaation-euivariaan neural networks for 3d point clouds. arXiv preprint arXiv:1802.082119 2018.
Zaken, E. B., Goldberg, Y., and Ravfogel, S. Bitfit: Sim- ple parameter-efficient fine-tuning for transformer-hased masked language- models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, 2022.
Unke, O. T. and Meuwly, M. Physnet: A neural network for predicting energies, forces, dipole moments, and partial charges. Journal of chemical theory and computation, 15 (6):3678-3693, 2019.
Zhang, D., Liu, X., Zhang, X., Zhang, C., Cai, C., Bi, H., Du, Y., Qin, X., Huang, J., Li, B., et al. Dpa-2: Towards a universal large atomic model for molecular and material simulation. arXiv preprint arXiv:2312. 2023.
Unke, O. T., Chmicla, S., Gastegger, M.. Schütt, K. T., Sauceda, H. E., and Müller, K.-R. Spookynet: Learn- ing force fields with electronic degrees of freedom and nonlocal effects. Nature communications, 12(1):7273, 2021.
Zhang, F. and Pilanci, M. Riemannian preconditioned lora for fine-tuning foundation models. In International Con- ference on Machine Learning, pp. 59641-59660 PMLR, 2024.
Wang, R., Ma, X., Zhang, L., Wang, H., Srolovitz, D. J., Wen, T., and Wu, Z. Classical and machine learning interatomic potentials for bec vanadium. Physical Review Materials, 6(1 1):1 13603, 2022a.
Zhang, L.,, Han, J., Wang, H., Car, R., and Weinan, E. Deep potential molecular dynamics: scalable model with the accuracy of quantum mechanics. Physical review letters, 220(14)143001,, 2018
Wang, X., Wang. Y., Zhang, L., Dai, F., and Wang. H. A tungsten deep neural-networt potential for simulating mechanical property degradation under fusion service environment. Nuclear Fusion, 62(12)) 2022b.
Zhang, L., Wang, H., Car, R., and E, W. Phase diagram of deep potential water model. Physical review letters, 126 (23):236001, 2021.
Wang, Y., Zhang, I.., Xu, B., Wang, X., and Wang, H. A gen- cralizable machine learning potential of ag-au nanoalloys and its application to surface reconstruction, segregation and diffusion. Modelling and Simulation in Materials Science and Engineering, 30(2):020003, 2021.
Zhang, Y., Wang, H., Chen, W., Zeng, J., Zhang, L., Wang, H., and Weinan, E. Dp-gen: A concurrent learning plat- form for the generation of reliable deep learning based potential energy models. Computer Physics Communica- tions, 253:107206, 2020.
13

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Zhou, G., Gao, Z., Ding, Q. Zheng, H., Xu, H., Wei, Z., Zhang, I.., and Ke, G. Uni-mol: A universal 3d molecu- lar representation learning framework. In International Conference on Learning Representations, 2023.
Zhou, Y., Zhao, Z., Du, S., Li, H., Yao, J., Zhang, Y., and Wang, Y. Exploring training heterogeneous data with mixture of low-rank adapters. In International Confer- ence on Machine Learning, pp. 62294 62306. PMLR, 2024.
14

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
A. Group representations and Equivariance
Group theory provides a formal framework for defining symmetries axiomatically, representation of group G is a function from G square matrices such that for all g. G.
= (()))
99)
Formally, mapping f X + Y is equivariant for vector spaces X' and Y to a group G, if Dx(0)= Dy is satisfied for any g in G, where Dx and Dy represent the transformation of group G on vector spaces X and Y, respectively. In this context, we focus on the group of symmetry operations that encompass both isometries of 3D space and permutations of points. When using vectors (e.g., atomic coordinates) in R& as input, both X and Y are in IR', and f can be written as:
j = f()).
(10)
We specifically examine the following three types of input transformation and their equivariance properties: Permutation: Let Pa represents the permutation of vector subscripts, then the permutation equivariance is:
fo P. Po
(11)
Translation: Let TH I denote the translation, then the translation equivariance is:
ToT=oToo
(12)
Rotation: SO(3) is the group of 3D rotations. For E SO(3), let R(g) be the representation of g in R3, then the rotational cquivariance is:
f R(g) R(g) f.
(13)
The group elements of SO(3) are represented by D called Wigner D-matrices (Gilmore, 2008), which map the elements of SO(3) to (21 + 1) x (2l + 1)-dimensional matrices. For scalars and vectors in R2, the (real) Wigner D-matrices are
(g) =1 and (') = R(g).
(14)
In Message Passing Neural Networks (MPNNs), since the ordering of nodes in the graph has effect the feature update and the distance between nodes is used for message passing, permutation equivariance and translation equivariance are naturally satisfied. Therefore, EMPNNs mainly focus on how to achieve rotational equivariance.
B. MACE Architecture
MACE (Batatia al., 2022b) is an EMPNN model that uses higher body order messages. The expressiveness of the model improved by using efficient multi-body messages instead of two-body messages. Multi-body messages reduce the number of network layers required to achieve the same expressiveness under two-body messages, resulting in fast and highly parallelizable model. The MACE architecture follows the general framework of MPNN and includes three parts: message construction, update, and readout.
Message passing: In message construction, MACE combines equivariant message passing with efficient many-body messages. The edges are embedded using learnalle radial basis (+)) Riglats and a set of spherical harmonics y, and the self-interaction is performed on the features h (1) j.klgmz
with learnable weights w-(t) (t)
kklp" A,kigma is the two-body feature
obtained by pooling neighbor atoms:
A i.klams
Σ
Clamy Chmisama
(0) Rightai (ra)) (ftr JEN(()
Σ
Σ w-()) (1) kklz j,klzmz?
(15)
Lymiilgma
where Cams (1miidggmg are the standard Tlebsch-Gordann coefficients (Griffiths & Schroeter, 2019). The key operation of MACE
is to construct a mudt feature Bdd (t)
LLL through the tensor product of the two-body features (1)
i.klmss
B
Σ
CLM
(1)
(16)
lodde t,klems
15

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
where the coupling coefficients Ch corresponding to the generalised Clebsch-Gordee coefficients and w (2) kkig
the learnable
weight for the self-interaction of A A Finally, the message m;
(A) be written linear expansion:
m ILLM
ΣΣ
w ( Sckh, L.n.
(17)
Update:
The update linear function of the message and the residual connection (He et al., 2016):
h (1+1) MiLL (o m (*)
Σ
w(l kL,
LKLM
Ew(( (c)
sikL, "KLMS
(18)
The readout is mapping from the invariant part of the node features to a hierarchical decomposition of site energies:
Readout:
E, = E + ES!) + (T)
where
E w
(c)
if KKT
readour, "0000
(19)
E(d = R2 (n!.0)
MLP
( (i,,00 J
if t=
readout
The readout only depends on invariant features 1,000 making the site energy contribution E(") invariant.
C. Proofs
Proposition C.1. The self-interaction of the tensor h is equivalent to the fully connected tensor product of h and 1.
Proof. Since 1 is a scalar, stm = 1 when = = m = 0, and otherwise.
(h 3 1)kalamm =
Σ
Wigkoks Clama damg Maymm Aofamm
kalama
Σ
Clams h h 1o00
kihm
Σ
Waaky Clama
tom hky baymm
kidyma
Σ h1
W kigke tyms Iymss
In the second equation, the subscripts hi and l2 in W are omitted because non-zero result is only possible when k2 = l1 m2 = 0. Similarly, in the last equation, the subscript 4 in W is omitted because 4ymi 000 = 1 when / ls and m. = ma, and it is zero otherwise. The final form is equivalent to Equation 5, with only a renaming of the subscripts. Proposition C.2. The residual connection of the tensor hy is equivalent to the fully connected tensor product of h, and &4. where &; the one-hot encoding of the chemical species 24.
Proof. The tensor the one-hot encoding of the chemical species z;, where ki,kim 1 when 24, m and otherwise.
(h4 2 keglams
Σ
Wkskek dalah, Clama dama hikiim 2s, Ralaama
W Wkyzeky dal, Clyms 00 hi,ki m 21,2,00
kylim
Σ kydmm
Clams W Wkyseks Ashi Clamo 00 hi kalm
W Wkeee dy hicksamm
16

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
In the second equation, the subscript in W is omitted because non-zero result is only possible when k2 =, and m2 O. Similarly, in the last equation, the subscript I in W is omitted because Cima, .00 when 11 = and m m3., and it is zero otherwise. The final form is equivalent to Equation 6, with only renaming and rearranging of the subscripts.
Proposition C.3. For the pre-trained weight matrix 8 of the fully connected tensor product and its corresponding ELoRA matrices A and B, the fully connected tensor product of the tensors u and with ELoRA SO(3) equivariant.
Proof. The Clebsch-Gordan coefficients and Wigner D-matrices satisfy the following properties. For any element in the SO(3) group:
Σ mi mi
D (im/damm my my (g) =
(9)
D' DD'n
(g) (g) d2mm"
mam'
my
Then:
E
Waskaks Aslats Clams
Lamadaam
D (g)us
D'a Di mamb (g)ukalams)
min my
ki aalimi Sulimikkalam kalama
mi
mn
Σ
Σ
D'a mamin (g)
Jam's
Wkakzki Jalzh
damg Wk Vkyl2ms
ki mi Kalami
ml
Σ
Σ
mamy (g) m/
D
Clam lymidomg 4k mm Viglamg
kih my Raatama
>'
D mam' (g)
Σ
W Waskak Aaaat
chami lamidama Uklmm Va222m2
mg
where the second equation rewrites m' and m² as my and m2, Therefore, the SO(3) equivariance of the fully connected tensor product holds for any matrix W. For the pre-trained model, W Waskska,daaal = WD dalgh For the fine-tuned model with ELoRA, W kikkkk = W dalls + (Bigizh, Aigiali)
Proposition C.4. For the pre-trained weight matrix wo of the fully connected tensor product and its corresponding ELORA matrices and B, the fully connected tensor product of the tensors u and v with ELoRA satisfies (u & k)kggmm (u & kgggmm + +Ln,, E, Bkgrdslzhi (m, @ (12)tsmm. where (u @ kggamg is the fully connected tensor product of tensors and using W without ELoRA and (ur, @ 012 1iggrg is the fully connected tensor product of tensors ut and Ut, using ELoRA matrix A.
Proof.
(u & o)kslsms
Σ
(Wh dakg + (Bigtsh Alytth Dkskak )l
Iima, Uk,lm, Ukalama
kylima Ralama
(u @ 1) balama
Σ
(Bigtsh, Atgists Daskyk Clams hamadams Uky my Ukalgmg
k2l2ma
(u @ Uakilmms
Σ
Σ
(Bigtals Atglol )akkkk Clama
tamp Wkilimg Ukglggg
h Kim, 22m2
(u @ v) V)kaiams
+
Σ
Σ
Σ
Bigrdstalss Arkzki Ommpy Ukilm Ukglgmm
h,d2 kimm, Kamma
(u @ U)katsms
+
>>> Σ
Bestishah
Arks Jalah damadama Malmm Ukylama
Cdama
1112 r
kymy Agmg
(u & Kagiama
+ Σ Σ
Bkgrdiglzt, (ui, & Drgaa
11
Corollary C.S. For the pre-trained weight matrix Wo of the fully connected tensor product and its corresponding ELoRA matrices A and B, Aiglsh projects the equivariant message on the path (l1, 1g, 13) in the fully connected tensor product of the tensors u and with FLoRA into a low-dimensional space of dimension R.
17

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
D. Generalization Theory
The PAC-Bayesian framework (Shawe-Tayle Taylor & Williamson, 1997; McAllester, 1998; 1999) is an analytical approach that combines Probably Approximately Correct (PAC) theory with Bayesian theory to study the generalization performance of randomized learning algorithms. By introducing Bayesian prior and posterior distributions into the analysis of generalization error in PAC theory, it establishes generalization bounds for arbitrary priors and provides probabilistic guarantees for the generalization error.
Let w represent the parameters of the neural network, and let [w) : X y denote the mapping from input r to output y as computed by the neural network. Assume that the data points (æ1, 31)s (22.82)..... .... (#N: MN) are independently drawn from a distribution D. Define & the loss function, where the loss on a sample (æ, y) given by ((Jw (x), y). The generalization error of the neural network on D is expressed [(fw) F ( (z). y)]. while the empirical error on (x)))
a dataset [(Jw) = &
Since the underlying distribution typically unknown, the generalization error can only be approximated through the empirical error computed on a test set. The PAC-Bayesian framework provides an estimation of the generalization bound L(fw) i(fw) ) for randomized predictors. Recent studies based on PAC-Bayesian theory have derived generalization bounds for models such Multi-Layer Perceptrons (MLPs) (Neyshabur al., 2018) and Message Passing Neural Networks (MPNNs) (Ju et al., 2023). These bounds are typically derived using the spectral norm of weight matrices, which is often proportional to the stable rank [|WI|2
Below, we present the propositions and their corollaries for MLPs and MPNNs.
Lemma D.1. (Neyshabur et al., 2018) For any B, B, 0, let fw XB,, - R be d-layer feedforward network with ReLU activations. Then, for any 8, y O, with probability 2 - 8 over training set of size m, for any w, have:
B&d In
W; 12 W.
In
dm
Lolsw) S [n([w) +0
7m
Corollary D.2. For d-layer feed-forward network, the generalization bound is related to the stable rank of the weight matrix as follows:
11d
| W
I
N
where N is the number of samples.
Lemma D.3. (Ju al., 2023) Suppose all of the nonlinear activations in (6t, Pt, the : V) and the loss fanction v) (for any fixed label y Y) are difeediffere Lipschitz-coninnuuu and their first-order and second-order derivatives are both contriizs. With probability at least 1 - 8 over the randomness of N independent samples from D, for any 6 > 0, and any > 0 close to zero, any model with weight matrices in the set H satisfies:
CBd,
max |X| | 2(11))
r
II
2
[log(0-))
(X.G.9)>D
L() S (1+))))))+++++ +
Σ
NT344
where an upper bound on the value of the loss function ((x, u) for any (a,y) ~ D, a fixed constant depending on the activation, and the loss function's n's Lipsehitz- casschitz-continuity
Corollary D.4. For l-layer message-paseeee passing neural network, the generalization bound is related to the stable rank of the weight matrix as follows:
W 11?
N
==1
where Ni is the number of samples.
18

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Corollaries D.2 and D.4 indicate that reducing the stable rank of the weight matrix leads tighter generalization bound. Therefore, we focus the upper bound of the stable rank. The upper bound of the stable rank of the fine-tuned weight matrix can be determined using the following propositions.
|| Lemma D.5. For any matrix A R its stable runk is defined as srank(A) = b where ||A|| the Frobenius norm of A and ||A||2 is the spectral norm of A, then srank( A) S rran((A)
Proof. Let 01, 02,...,00 Or be the non-zero singular values A from large to small, then r = rank(A) is the rank of A, ||A||e = / / 1=1 o, ||ll = 01. The definition of stable rank can written as srank(A) = - of Note that of 2 of
holds for = 1,2,... ...., ,,. s0 Ei of Sr Then srank(A) Eizy Sr= rank(A).
of
Proposition D.6. For the pre-trained weight matrix W Baxk and its update AW R srank( + AW) s rank(W/)) + rank(AW).
Proof. srank (Wo + AW) & rank(W" + AW) & rank + (a))
Proposition D.7. For the pre-trained weight matrix 8 E Rdxk and its update AW E Bdxk ife TW
[AW/,
< 1, then
Wso + +WW < - (v SrrnkkWWU te. srank(AWW)
Proof.
I + AW
srank(Wo AW)
| AW
| [| + ||WW|F
||Wo|12 |AWII 112
|
W IF
|WW|F
|WW|22
||
la
|W
|
|
(V/srank(WW)) + srank (AW))
From Propositions D.6 and D.7, we observe that the upper bound of the stable rank of the fine-tuned weight matrix is determined by the rank or stable rank of AW. Consequently, LoRA can achieve a tighter generalization bound compared to full- uull-parameter fine-tuning by reducing the rank AW. When AW has low rank, the empirical error of the LoRA fine-tuned model is relatively high, resulting in underfitting. As the rank AW increases, the empirical error decreases, the generalization performance worsens, causing the model to transition from underfitting to overfitting. At the balance point, where the empirical errors of full-parameter fine-tuning and LoRA are equal, LoRA expected to exhibit superior generalization performance. This process is consistent with the description in Section 5.3.2.
Furthermore, Corollary D.4 shows that the generalization bound is inversely proportional to the number of samples. This implies that as the number of samples increases, the difference between LoRA and full-parameter fine-tuning gradually diminishes, aligning with the observations described in Section 5.3.1.
E. Experimental Details
E.1. Pre-trained Datasets
SPICE (Eastman et al., 2023): SPICE contains small molecules of up 50 atoms and involves ten chemical elements: H, C, O, F, P. S. Cl, Br, and I. To facilitate the learning of intramolecular non-bonded interactions, also uses few larger molecules of 50-90 atoms randomly selected from the QMugs dataset (Isert et al., 2022).
MPTrj (Deng et al., 2023): MPTrj consists of a large collection of static calculations and structural optimization trajectories from the Materials Project (MP) (Jain et al., 2013), including approximately 1.5M configurations of 150k unique inorganic crystal structures.
E.2. Downstream Datasets
rMD17 (Christensen & Von Lilienfeld, 2020): The revised MD17 (rMD17) is recomputed version of the original MD17 dataset (Chmiela et al., 2017) higher numerical accuracy. This dataset contains long molecular dynamics trajectories of
19

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
small organic molecules, with 100,000 structures each. It recomputes energies and forces at the PBE/def2-SVP level of theory using very tight SCF convergence and a very dense DFT integration grid to reduce numerical noise.
3BPA (Kováes et al., 2021): The 3BPA dataset is sampled from molecular dynamics simulation of the large and flexible drug-like molecule 3- 3-(benzyloxy 1pyridin-2-ammiiii --amine. It is used to test the extrapolation ability of model. Its training set contains 500 structures sampled at 300 K, and three test sets contain structures sampled at 300 K, 600 K, and 1200 K to evaluate the accuracy both in and out of domain. The fourth test set consists optimized dihedral slices by rotating the dihedral angles to produce PES regions far away from the training data.
AcAc (Batatia et al., 2022a): The AcAc dataset is dataset for evaluating the extrapolation ability of the model to higher temperatures, bond breaking, and bond torsion for the acetylacetone molecule. The training set is sampled at 300 K, and the test set is sampled independently at 300 K and 600 K.
AgUAu-PBE (Wang et al., 2021), A1UMgUCa JC (Jiang et al., 2021): The AgUAu-PBE dataset consists of Ag and Au configurations generated using the DP-GEN scheme, while the AlUMgUCu dataset includes AI, Mg. and Cu configurations produced by the same method. All DFT calculations are performed with the VASP (Kresse & Furthmülllr, 1996a;b) software, applying the PBE exchange-correlation functional.
H2O-PD (Zhang et al., 2021): The water/ice dataset is used to train the model to caleulate the water phase diagram. This dataset is labeled by the VASP software using the SCAN xcchage-correlatto functional.
SSE-PBE (Huang et al., 2021): This dataset contains the solid electrolytes Li and Lipo 12 generated by the DP-GEN method. All DFT calculations are performed with the VASP software, applying the PBE exchange-correlation functional.
The Cu (Zhang et al., 2020), Sn (Chen al., 2023), Ti (Wen et al., 2021), V (Wang et al., 2022a), W (Wang et al., 2022b), and HfO2 (Wu et al., 2021) datasets generated by the DP-GEN method under complex temperature and pressure conditions.
Other datasets:
We list the sizes and links of these datasets in Table 5.
Table 5. Sizes and links of different datasets.
Dataset
Size
Link
rMD17
1,000,000
https//dx..oii.o /rg/ (0.6084/m9 www.saie..g2.5..0 12672038
3BPA
13,997
AcAc
6,263
SSE-PBE
15,774
H20-PD
48,419
AgUAu-PBE AU-PBE
17,508
AlUMgUCu
25,397
Cu
15,366
Sn
6,725
Ti
10,528
15,673
W
44,397
HIO2
28,577
E.3. Models
ACE (Kovács et al., 2021): The Atomic Cluster Expansion (ACE) a framework that linearly parameterizes potential energy surfaces using body-ordered symmetric polynomials. Its key innovation the density trick, which projects atomic neighborhoods onto invariant basis functions, allowing efficient computation of high-order interactions with costs scaling linearly with neighboring atoms, combining accuracy with efficiency.
NequIP (Batzner et al., 2022): NequlP is an E(3)-cquivariant graph neural network that leverages geometric tensor representations, including scalars, vectors, and higher-orde tensors, to model interatomic potentials. Its innovative use of equivariant convolutions and tensor features ensures that internal operations preserve symmetry transformations, enabling more accurate and data-efficient representations of atomic environments.
20

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
BOTNet (Batatia et al., 2022a): BOTNet (Body Ordered Equivariant Network) simplifies NequlP's design by adopting body-ordered structure, incrementally building interaction complexity through layers. It removes non-essential nonlinearities while retaining NequlP's equívariant framework, achieving similar accuracy with improved interpretability and efficiency.
Allegro (Musaelian et al., 2023): Allegro is an innovative deep learning architecture for molecular dynamics, which combines strict locality and equivariance in its design. Unlike message--assiiggg neural networks, it employs tensor product layers to directly model many-body interactions without iterative propagation, enabling efficient parallel computation and scalability to large systems.
DPA2 (Zhang al., 2023): DPA-2 is an advanced large atomic model (LAM) that leverages a multi-task pre-training framework to embed diverse chemical and configurational knowledge. It employs fine-tuning adapt the pre-trained model for specific downstream tasks, achieving high generalizability and efficiency across wide range of molecular and material systems.
PACE (Xu et al., 2024): PACE approximates a broader class of SE(3) x S, equivariant polynomial functions with higher degrees by combining spherical harmonics, tensor products, and an edge booster. extends ACE with symmetric contractions to capture many-body interactions, achieving strong accuracy and generalization in force field prediction.
E.4. Training Details
To facilitate reproducibility, Table summarizes the training hyperparameters for different MACE models. The settings for MACE-From-scratee (Batatia et al., 2022b). MACE-MP (Batatia et al., 2023), and MACE-OFF (Kovács al., 2023) are derived from previously published studies, while those for MACE-MP-Fing -ineteeeee and MACE-OFF-Finee fine-tuning hyperparameters used in this work. The MACE model code is modified from the main branch of the open source GitHub repository at https//gihubbcom/AAEsiiuu (commit hash: 346a829f).
reflect the
Table 6. Training hyperparameters for different MACE models.
MACE-From-seratchh
MACE-MP
MACE-OFF
MACE-MP-Fine-tune
MACE-OFF-Fine-tune
model
ScaleShinMAAE
ScaleShifMMAC
MACE
ScaleShiftMACI
MACE
max
5.0
6.0
5.0
6.0
5.0
num cammarrdallbsss
8
10
8
10
8
num_chume channels
256
128
128
128
128
max_L loss
2
universal
ef
ef
forces weight energy_weight Ir weight_decar scheduler_pp eeatiene
1000
1000
1000
1000
40
0.01
0.005
0.01
0.005
0.005
5c-7
le-8
Sc-10
1c-8
1e-8
50
5
20
5
5
ema. ema.decar decay clip grad
0.99
0.995
0.99
0.995
0.995
10
100
1
100
100
F. Other Experimental Results
F.1 Comparison of Different Fine-tuning Methods
We compare two other popular fine-tuning methods: Readout fine-tuning (freezing the previous layers and only fine-tuning the Readout layers) and Adapter fine-tuning. Table depicts the RMSE results of these fine-tuning methods on Cu and Sn datasets. Adapter fine-tuning performs worse because the equivariance is destroyed under the fine-tuning process. Readout fine-tuning is not as accurate as full-parameter fine-tuning because only the last few layers are fine-tuned, which lacks flexibility compared to full-parameter fine-tuning.
We also conduct experiments with ELoRA on SevenNet (Park et al., 2024), which is another representative SO(3) -quivvaana MPNN. As shown in Table 8, the results are consistent with those in Table 7.
21

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
Table 7. RMSE of energy (E, me m/tomm and foree (F, V/A) for different finetuning methods MACE. The best results are in bold.
MACE
MACE
MACE
MACE
Method
Full parammeerr
Adapter
Readout
ELoRA
0.6
4.4
0.4
Cu
5.4
22.9
28.2
4.4
4.9
27.6
30.8
4.6
Sn
31.7
67.5
74.8
29.2
Trainable parameters
723,866
169,296
2,192
176,666
Table RMSE energy (E, me/amom) and force (F, meV/A) for different finetuning methods on SevenNet. The best results are hold.
SevenNet
SevenNet
SevenNet
SevenNet
Method
Full-parann parameter
Adapter
Readout
ELoRA
E
0.9
2.5
10.2
0.8
Cu
12.8
32.1
153.2
12.2
E
3.4
6.8
16.4
3.0
Sn
74.1
117.0
190.5
73.4
F22 Stable Rank
In Appendix D, analyze how low-rank adaptation reduces the stable rank of the update the weight matrix, AW, during fine-tuning, Using the rMDI7-aspirin dataset as example, assess the stable rank of AW for each layer of the model under both full-parameter fine-tuning and ELoRA fine-tuning. The results, shown in Figure 6, indicate that ELoRA fine-tuning generally leads to lower stable rank for AW.
ELoRA
4-
8-
0.
0bbyrr])
MayyrrJ))
M.byrr.)
Figure 6. Comparison of stable rank for AW between full-paramett fine-tuning and ELoRA fine-tuning across different layers. Results are from models trained on rMDI7-aspirin dataset.
F.3. Hyperparameters
Since full-parameter fine-tuning and ELoRA update different numbers of parameters, they may require different learning rates to achieve optimal performance. We test various learning rates, and the results are shown the left panel of Figure 7. Overall, ELoRA outperforms full-parameter fine-tuning, with the best ELoRA model surpassing the best full full-parameter fine-tuning model. This eliminates the potential influence of learning rate settings on the performance comparison between
22

---

ELoRA: Low-Rank Adaptation for Equivariant GNNs
the approaches.
ELoRA improves generalization by reducing the number of parameters being trained, which helps prevent overfitting. We aim to test whether other regularization methods could achieve similar effects. Typically, weight decay and dropout are used for regularization. However, in SO(3) equivariant GNNs, applying dropout directly could disrupt the network's ' equivariance. Therefore, we only test weight decay, and the results of using different weight decay hyperparameters farl-parrmeter fine-tuning are shown in the right panel of Figure As we can see, weight decay do not provide significant benefits this case.
Full-paranetor
FloRA
10
1688
16a
10
30-1
10
10-
10
learning rate
Wrightt denuy
Figure The impact learning rates and weight decay on model loss. Results are from models trained on the rMD17-aspirin dataset.
23


Reviews:
[{"review_id": "szkm8oOtab", "reviewer": "Reviewer_iY6i", "summary": "The paper introduces a variant of LoRA for finetuning geometric graph neural networks that use spherical harmonics. The idea is to consider the main model parameters that appear in path-dependent tensor-product using Clebsch-Gordon in these SO(3) equivariant models, and provide path-dependent low-rank adaptation, which is shown to preserve equivariance. The proposed method is applied to fine tuning of pretrained MACE on organic and inorganic datasets for force-field prediction and it is compared against models trained from scratch or fully finetunned on the downstream dataset.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the professional and valuable comments.\n\n**Q1: The analysis on SVD decomposition of weight matrices.**\n\nA1: We apologize for the analysis of the SVD decomposition, which could be misleading. Section 3 serves as a transitional part, aiming to convey the necessity of fine-tuning on pre-trained models rather than training models from scratch. Then, we naturally introduce our innovative fine-tuning method, ELoRA.\n\nIn Section 3, Figure 1 provides a qualitative interpretation from the chemical space perspective, comparing the coverage function space among pre-trained models, fine-tuned models, and models trained from scratch. The SVD decomposition analysis intends to offer quantitative information, demonstrating that the weights of fine-tuned models exhibit higher similarity to the pre-trained models than those trained from scratch models. \n\nThe function space represented by deep neural networks is complex due to its high dimension and nonlinear nature. To our knowledge, we have not yet found an alternative theoretical explanation to better characterize the learned function spaces. As a result, we use spectral data in each layer to figure out the knowledge embedded in model weights. It may not be a rigorous analysis. If the reviewers consider the SVD analysis insufficiently compelling, we can move the SVD experiments to the Appendix and move the AcAc and 3BPA experiments to the main text.\n\nAs for spectral analysis on ELoRA-fine-tuned weights ($W_{\\text{ELoRA}}$), we add the comparison on the pre-trained weights $W_0$ and $W_{\\text{ELoRA}}$ (see Link: https://anonymous.4open.science/r/ELoRA/picture/spectra.png). The results show that $W_0$ and $W_{\\text{ELoRA}}$ maintain high similarity. The distribution of cosine similarity is close to that of $W_0$ and full fine-tuned weights ($W_{\\text{Full-parameter}}$). The added figure indicates that $W_{\\text{ELoRA}}$ and $W_{\\text{Full-parameter}}$ have high similarity to $W_0$.  We will add this figure in the revised manuscript.\n\n**Q2: The claim of \"equivariant MPNNs\"**\n\nA2: Our proposed fine-tuning method applies specifically to SO(3)-equivariant MPNNs. We will clarify the scope by specifying SO(3)-equivariant MPNNs in our revised paper.\n\n**Q3: Choice of decomposition.**\n\nA3: In Equation (8) $W^0_{l_3 l_2 l_1} + \\Delta W^0_{l_3 l_2 l_1} = W^0_{l_3 l_2 l_1} + B_{l_3 l_2 l_1} A_{l_3 l_2 l_1}$, the weight matrix $W_{l_3 l_2 l_1}$ has dimensions $K^3_{l_3}$, $K^2_{l_2}$, and $K^1_{l_1}$. We merge $K^2_{l_2}$ and $K^1_{l_1}$ because the computation involves a transformation from an intermediate tensor of dimension $K^2_{l_2} \\cdot K^1_{l_1}$ to an output tensor of dimension $K^3_{l_3}$, making this merging a natural design choice, as illustrated in Figure 3. There exist various weight decomposition method, identifying other more effective weight decomposition strategies (such as Tucker decomposition) will be one of our future works.", "claims_and_evidence": "The claims on effectiveness of the proposed method is well-supported by experiments. Theoretical claims also make sense. \n\nThe paper’s original claim on providing low-rank adaptation for equivariant MPNNs at large is too broad. It should clarify at the outset that it is suggesting a method for fine-tuning that applies to methods using parameterized tensor-product using CG. \n\nThere is a claim at the beginning of the paper about the closeness of fine-tuned model to the pretrained model when compared against a model trained from scratch. The claim generally make sense, but the conclusion drawn from experiments is not accurate, since it is comparing two deep networks as functions based on the closeness of their weights’ spectra. It makes sense to acknowledge that difference in the weight space does not imply difference in the function space.", "methods_and_evaluation_criteria": "Please see my question on a design choice in the method.\n\nI cannot comment on the specifics of experiments such as the choice of dataset for finetuning, as I am not familiar with those details, but the evaluation is in my view quite extensive and supportive.", "theoretical_claims": "Theoretical claims of the paper are statements that appear to be correct, even without considering a formal proof. I have not checked the proofs.", "experimental_designs_or_analyses": "To the extent of my familiarity with this domain the experiments and the supporting analysis make sense. I appreciated the ablation on the rank.", "relation_to_broader_scientific_literature": "The paper is nicely positioned in the context of broader literature. In particular, literature on finetuning in geometric deep learning and important related works on neural networks for interatomic potentials are reviewed.", "essential_references_not_discussed": "Nothing comes to mind.", "other_strengths_and_weaknesses": "Strengths:\n\n– The method is highly motivated given the extensive use of deep learning in molecules and materials and the need for finetuning on smaller datasets\n\n– presentation is quite polished and the organization helps with a smooth delivery\n\n– the paper shows perspective in discussing the problem and related literature\n\n– experimental results are extensive and supportive\n\n– the paper makes a good use of figures \n\nWeakness:\n\n– the scope of the contribution needs to be clarified early on, in the abstract and introduction.", "other_comments_or_suggestions": "Use of the term equivariant by the paper is confusing: vanilla message passing neural networks are also equivariant, but to symmetric group only. The paper is targeting the SO(3), and it makes sense to use better terminology to make the distinction. The term geometric GNN or MPNN, as opposed to equivariant MPNN, makes more sense.", "questions_for_authors": "While the method generally makes sense, one design choice appears rather suboptimal: Given W is third order (ignoring channels) why this particular choice of decomposition for the tensor W? For example, why not use Tucker decomposition?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "5loteYS2Bq", "reviewer": "Reviewer_qEUK", "summary": "Existing parameter-efficient fine-tuning (PEFT) methods are not suitable for equivariant GNNs. To that end, the authors propose a novel method for equivariant low-rank adaptation for finetuning equivariant GNNs. Specifically, the authors propose a path-dependent low-rank adaptation for the tensor product weights which preserves the equivariance of the operation. The authors evaluate their PEFT method compared to full-parameter fine-tuning and from-scratch training on several common organic and inorganic datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to your questions and suggestions.\n\n**Q1: The fairness in using only 50 samples in rMD17 experiments.**\n\nA1: In practical downstream applications, fine-tuning methods are often expected to perform well with limited first-principles training data, as DFT calculations  are computationally expensive. To simulate such realistic requirements, we construct low-data scenarios (e.g., extract only 50 training samples from each small dataset in rMD17) to demonstrate the accuracy performance of dedicated models (ACE, NequIP) and fine-tuned pretrained model such as MACE.\n\n**Q2: Full-parameter finetuning results of 3BPA and AcAc.**\n\nA2: Our experiments show that ELoRA based MACE is better than full-parameter fine-tuned MACE, especially in the out of domain test set. Table1&2 show the from-scratch-training and finetuning reults of 3BPA and AcAc.\n\n**Table1 3BPA results**\n\n||MACE (From scratch)|MACE (Full-parameter)|MACE (ELoRA)|\n|:-:|:-:|:-:|:-:|\n|300K,E|3.0 (0.2)|3.3 (0.03)|**3.0 (0.05)**|\n|300K,F|8.8 (0.3)|7.8 (0.01)|**7.5 (0.05)**|\n|600K,E|9.7 (0.5)|7.3 (0.04)|**6.5 (0.10)**|\n|600K,F|21.8 (0.6)|16.6 (0.05)|**15.5 (0.12)**|\n|1200K,E|29.8 (1.0)|20.3 (0.17)|**17.6 (0.11)**|\n|1200K,F|62.0 (0.7)|48.7 (0.56)|**42.0 (0.51)**|\n|dih,E|7.8 (0.6)|7.3 (0.28)|**5.9 (0.28)**|\n|dih,F|16.5 (1.7)|12.3 (0.10)|**11.4 (0.17)**|\n\n**Table2 AcAc results**\n\n||MACE (From scratch)|MACE (Full-parameter)|MACE (ELoRA)|\n|:-:|:-:|:-:|:-:|\n|300K,E|0.9 (0.03)|1.0 (0.02)|**0.8 (0.03)**|\n|300K,F|5.1 (0.10)|5.1 (0.07)|**4.5 (0.06)**|\n|600K,E|4.6 (0.3)|5.8 (0.28)|**3.9 (0.33)**|\n|600K,F|22.4 (0.9)|16.4 (0.70)|**13.6 (0.26)**|\n\n**Q3: The comparison with SOTA results PACE.**\n\nA3: We will cite the PACE paper and include the results of PACE in the organic experiments. PACE represents SOTA results among dedicated models that are individually trained for each small dataset. However, our work focuses on developing novel fine-tuning techniques for pretrained models. \n\nIn the case of rMD17, PACE achieves high accuracy when 1000 samples are used in training, as the PACE paper shows. In low-data scenarios, when training the PACE model with 50 samples per dataset, the accuracy cannot be reached, which is achieved with 1000 training samples. While, ELoRA makes it possible to obtain high-precision downstream models when a small amount of training data is provided.\n\nTable3 lists PACE and MACE results. The second and third columns show the MAE for PACE with 1000 and 50 training samples respectively. The fourth column reports the MAE results of MACE ELoRA. Table 3 reveals that PACE requires abundant training data to maintain its high accuracy. Its performance will drop at 50 training samples. ELoRA maintains prediction precision even with this minimal training set.\n\n**Table3 rMD17 results**\n\n||PACE (1000, From scratch)|PACE (50, From scratch)|MACE (50, ELoRA)|\n|:-|:-:|:-:|:-:|\n|Aspirin,E|1.7|15.7|7.3|\n|Aspirin,F|5.8|37.4|17.6|\n|Azobenzene,E|0.5|6.7|4.0|\n|Azobenzene,F|2.2|17.5|12.4|\n|Benzene,E|0.02|0.6|0.2|\n|Benzene,F|0.2|3.3|1.6|\n|Ethanol,E|0.3|6.3|2.1|\n|Ethanol,F|1.8|25.4|10.7|\n|Malonaldehyde,E|0.5|11.5|6.5|\n|Malonaldehyde,F|3.6|57.3|21.7|\n|Naphthalene,E|0.2|2.1|1.4|\n|Naphthalene,F|0.9|9.7|6.0|\n|Paracetamol,E|0.9|10.1|4.8|\n|Paracetamol,F|4.0|29.3|14.8|\n|Salicylicacid,E|0.5|7.0|3.2|\n|Salicylicacid,F|2.9|29.2|14.2|\n|Toluene,E|0.2|2.7|1.3|\n|Toluene,F|1.1|12.0|5.9|\n|Uracil,E|0.3|5.9|2.1|\n|Uracil,F|2.0|26.8|11.6|\n\n**Q4: The analysis on SVD decomposition of weight matrices.**\n\nA4: Please refer to Reviewer iY6i, Q1.\n\n**Q5: Reduction in the number of parameters.**\n\nA5: The pretrained model covers multiple elements spanning the periodic table. But for downstream tasks, we retain only weights relevant to the task's elements and prune all others. This causes the reduction of model parameters. We will clarify this issue in the revised paper.", "claims_and_evidence": "The authors first provide a study of the singular-value decomposition of weight matrices from the fine-tuned model and the model trained from scratch and compare with the pretrained model. It is not clear to me how this supports the rest of the paper. \n\nThe main theoretical claims in Section 4 are supported with proofs in the appendix, which appear to be correct and are convincing. \n\nHowever, I am not necessarily convinced by the empirical performance of ELoRA. Specifically, it is not clear that ELoRA consistently outperforms full-parameter finetuning based on the experiments provided. In fact, in Section 5.3.1, the authors actually show that given even a small amount of finetuning data (1000 samples), ELoRA and full-parameter fine-tuning perform the same. The inorganic dataset experiment provided provide support for the proposed method, but the organic dataset experiments do not.\n\nThe authors also make several claims about ELoRA reducing the number of parameters. In my view is a misleading claim, though I am less familiar with existing literature on parameter-efficient fine-tuning. After finetuning the model, the total number of parameters is the same.", "methods_and_evaluation_criteria": "The inorganic datasets used for evaluation are reasonable and provide a solid evaluation of ELoRA compared to full-parameter finetuning. Specifically, the authors demonstrate that ELoRA outperforms full-parameter finetuning even given ~10,000 finetuning samples, which provides strong support for the proposed method. \n\nHowever, there are several issues with the evaluations on organic datasets. The only organic dataset used in the main paper is rMD17, and the authors choose to only train on 50 molecules, which is an unfair limitation for baseline methods. The full rMD17 dataset contains 100,000 structures for each molecule, and it is common for works to train on only 1,000 structures per molecule, however, as demonstrated in 5.3.1,  ELoRA does not outperform full-parameter fine-tuning with 1,000 training structures per molecule. They authors additionally evaluate on 3BPA and AcAc datasets in the appendix, however, they do not provide results for full-parameter finetuning, which makes it impossible to evaluate ELoRA on these tasks. Additionally, for rMD17, 3BPA, and AcAc, the state-of-the-art method, PACE [1] is not included in the comparison. \n\n[1] Equivariant Graph Network Approximations of High-Degree Polynomials for Force Field Prediction, Xu et al, https://arxiv.org/abs/2411.04219", "theoretical_claims": "I briefly checked the proofs of theoretical claims in the appendix and they appear to be correct, however it is possible that I may have missed some details.", "experimental_designs_or_analyses": "As mentioned previously, there are several issues with the organic dataset evaluations. The authors only train on a subset of the rMD17 dataset, which does not provide a fair comparison for their method. They also do not provide results for full-parameter finetuning on the 3BPA and AcAc datasets, making it impossible to evaluate ELoRA.\n\nThe first analysis on SVD of weight matrices is also confusing to me. Clearly, we should expect the weights of a fine-tuned model to have some similarity to it's original pre-trained weights, but even for separate training runs of the same model on the same dataset, is there any reason to expect that there would be similarity in the weight matrices? In my view, this analysis is not convincing and does not support the rest of the paper.", "relation_to_broader_scientific_literature": "This work provides a novel method for parameter efficient finetuning of equivariant GNNs. Prior works have shown powerful results from finetuning pretrained equivariant foundation models, so this work is important for downstream applications.", "essential_references_not_discussed": "The paper is currently missing the state-of-the-art method on all of the organic datasets provided [1]. \n\n[1] Equivariant Graph Network Approximations of High-Degree Polynomials for Force Field Prediction, Xu et al, https://arxiv.org/abs/2411.04219", "other_strengths_and_weaknesses": "Strengths:\n- To my knowledge, ELoRA is the first work to enable parameter-efficient fine-tuning while preserving equivariance\n- ELoRA demonstrates consistent performance improvement compared to full-parameter fine-tuning on several inorganic datasets\n\nWeakness:\n- The evaluation on organic datasets does not show that ELoRA outperforms full-parameter fine-tuning, making the experimental results of the paper as a whole inconclusive. Specifically, the study provided in Section 5.3.1 actually weakens the authors claims. \n- The study on SVD of weight matrices does not support the rest of the paper.", "other_comments_or_suggestions": "Unless the authors can significantly revise the study on SVD to better support the story of the paper, I think it would be better off if this section was moved to the appendix to make room for other experimental results (preferably on 3BPA and AcAc) in the main paper. It may also be interesting to compare the similarity of ELoRA vs. full-parameter finetuning with the pre-trained weights instead of fine-tuning vs. from scratch.\n\n### Updates After Rebuttal\n\nWhile I still feel that the rMD17 experiments are not so strong or realistic, at least the are following what was done by previous works. The rest of the experiments are strong, and the ideas in this paper are a good contribution to the MLFF community. I am leaning towards acceptance on this paper given the authors' response to all reviewers during the rebuttal period.", "questions_for_authors": "1) Can the authors provide results for ELoRA training on the standard split of 1,000 structures for each molecule on rMD17?\n2) Can the authors provide results for full-parameter fine-tuning on 3BPA and AcAc? \n3) Can the authors clarify the SVD experiments and how these results support the rest of the paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xEsvYXshHD", "reviewer": "Reviewer_S1HM", "summary": "The paper presents *ELoRA*, a novel method for fine-tuning equivariant GNNs that preserves the essential equivariance property, addressing limitations of traditional fine-tuning approaches. ELoRA demonstrates significant improvements in model performance. The method employs a path-dependent weight update decomposition strategy and low-rank adaptation to enhance data efficiency while maintaining physical consistency, thereby advancing the understanding of pretraining-fine-tuning paradigms in the context of materials science and chemical simulations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and we will correct all the typos, improve the writing and enhance the readability according to your comments. We will polish the describtion of Figure1 and add reference link for the proved propositions.\n\n**Q1: The novelty of ELoRA.**\n\nA1: We propose a PEFT method for SO(3)-equivariant MPNNs, which is technically novel in its path-dependent low-rank adaptation. To the best of our knowledge, no existing PEFT methods preserve equivariance.\n\n**Q2: The comparative experiment of different finetuning methods.**\n\nA2: We will compare two other popular fine-tuning methods: Readout fine-tuning (freezing the previous layers and only fine-tuning the Readout layers) and Adapter fine-tuning. Table 1 depicts the RMSE results of these fine-tuning methods on Cu and Sn datasets. Adapter fine-tuning performs worse because the equivariance is destroyed under the fine-tuning process. Readout fine-tuning is not as accurate as full-parameter fine-tuning because only the last few layers are retuned, which lacks some flexibility compared to full-parameter fine-tuning. \n\n**Table 1 The comparasion of four different finetuning methods.**\n\n||MACE (Full-parameter)|MACE (Adapter)|MACE (Readout)|MACE (ELoRA)|\n|:-:|:-:|:-:|:-:|:-:|\n|Cu,E|0.6|4.4|5.6|**0.4**|\n|Cu,F|5.4|22.9|28.2|**4.4**|\n|Sn,E|4.9|27.6|30.8|**4.6**|\n|Sn,F|31.7|67.5|74.8|**29.2**|\n|Number of trainable parameters|723866|169296|2192|176666|\n\n**Q3: Number of parameters after using ELoRA.**\n\nA3: In ELoRA, the path-dependent weights can be merged into the model weights after training, just like in the original LoRA. Thus, it will not increase the number of the model's parameters. The last row of Table 1 records the number of trainable parameters of the four different fine-tuning methods.\n\n**Q4：The applicability on other SO(3)-equivarant MPNNs.**\n\nA4: We add experiments with ELoRA on SevenNet [1] (another representative SO(3)-equivariant MPNN), as shown in Table 2. The conclusions drawn from Table 2 are consistent with those from Table 1. Our proposed ELoRA can be adapted to other SO(3)-equivariant MPNNs in a user-friendly manner.\n\n**Table 2 The finetuning methods on SevenNet.**\n\n||SevenNet (Full-parameter)|SevenNet (Adapter)|SevenNet (Readout)|SevenNet (ELoRA)|\n|------|:-----------------------:|:----------------:|:----------------:|:--------------:|\n|Cu,E|0.9|2.5|10.2|**0.8**|\n|Cu,F|12.8|32.1|153.2|**12.2**|\n|Sn,E|3.4|6.8|16.4|**3.0**|\n|Sn,F|74.1|117.0|190.5|**73.4**|\n\n[1] Park, Yutack, et al. \"Scalable parallel algorithm for graph neural network interatomic potentials in molecular dynamics simulations.\"\n\n**Q5: Results analysis on inorganic dataset.**\n\nA5: The prediction errors of energy and forces should be considered jointly, as they are typically optimized together during training. We cannot evaluate based solely on either energy or forces. For a comprehensive analysis, we can equally combine energy and force RMSEs as a joint metric. Under this metric, ELoRA achieves the best accuracy on 9 out of 10 datasets, as Table 3 shows.\n\n**Table 3 Summed RMSE of Energy and Force.**\n\n||NequIP|Allegro|DPA2|MACE|MACE (Full-parameter)|MACE (ELoRA)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|SSE-PBE|42.7|48.8|51.7|31.7|17.2|**14.5**|\n|H2O-PD|28.0|OOM|25.2|109.6|20.5|**16.9**|\n|Ag$\\cup$Au-PBE|86.1|98.1|20.2|403.6|21.9|**17.6**|\n|Al$\\cup$Mg$\\cup$Cu|86.3|58.9|21.2|50.6|12.9|**11.0**|\n|Cu|22.9|10.2|10.1|52.4|6.0|**4.8**|\n|Sn|80.4|45.8|58.5|/|36.6|**33.8**|\n|Ti|165.0|92.5|118.1|102.5|85.3|**79.1**|\n|V|100.4|86.3|94.9|154.6|78.9|**72.9**|\n|W|181.2|105.6|113.7|196.8|93.3|**83.4**|\n|HfO2|60.3|65.4|55.2|**17.0**|30.5|21.3|\n\n**Q6: Clarify of preseved equivariance in ELoRA.**\n\nA6: ELoRA is a parameter-efficient fine-tuning method. Its parameters can be merged into the base model after tuned, the network structure remains the same. Thus, the equivariance is preserved. \n\nELoRA projects the equivariant features into a lower-dimensional space through equivariant operations. These projected features remain SO(3)-equivariant, as proved in Proposition 4.4.\n\n**Q7: The explanation of the pretraining dataset's sparsity and the materials' diversity.**\n\nA7: The diversity refers to the combinatorial diversity of material structures. For example, even small organic molecules composed of carbon（C）, hydrogen（H）, oxygen (O), and nitrogen (N) atoms can theoretically form up to $10^{60}$ possible structures.\n\nThe sparsity refers to the fact that labeled data with ab initio accuracy is in a small number. Computing structures' quantum properties (e.g., energy and atomic forces) require expensive Density Functional Theory (DFT) calculations. Currently, only a small fraction structures has high-quality DFT labels. The available labeled structures represent sparsity in the large structure space.\n\n**Q8: The organization of the paper.**\n\nA8: For a detailed explanation of the Section 3, please refer to Reviewer iY6i, Q1.", "claims_and_evidence": "Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "methods_and_evaluation_criteria": "I believe that the proposed methods and evaluation criteria make sense for the problem or application at hand. However, I think the author needs to give examples to show that the proposed method does maintain equivariance.", "theoretical_claims": "I double-checked the theory as well as the proof of the appendix and found no major errors.", "experimental_designs_or_analyses": "- For the comparison of fine-tuning methods, the authors only compared ELoRA and FFT, and other parameter fine-tuning methods should be added for comparison. \n- In Table 2, MACE-ELoRA performed best on only 3 datasets (under the same architecture) for Energy RMSE. The authors should add comparisons with fine-tuning methods such as ELoRA, downstream fine-tune in other architectures.\n- With regard to equivariance, the authors should add case studies to illustrate the effectiveness of the proposed method.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**:\n- The contribution is both original and strong.\n- Sufficient theoretical proof is provided.\n- Experiments have proven their excellent performance.\n- The main architecture is clear and provides detailed explanations.\n\n**Weaknesses**:  \n- This paper is ill-motivated. There is a contradiction: L160 \"The pre-trained training datasets at first-principles accuracy are often sparse...\", and the diversity of data in the Introduction \"due to the diversity of material structures...\" \n- The readability of the paper is poor, especially in Section 3. \n    - What is mean of L150-153: “They can only learn from the data (the dark blue circle in Figure 1(b)) in the specific downstream task they are trained on…” \n    - Some terms should be standardized for academic expressions, e.g. complex downstream task data in L158 should refer to OOD data. \n    - The organization of the paper is confusing, for example, the method mentioned in Section 3 (active learning/DP-GEN) does not seem to be relevant to this paper, and if it is, it should clarify the relationship with the existing works (limitation, improvement, etc.).\n- The comparative experiment is inadequate and unconvincing.\n- In general, the author proposes that the method is not technically novel.", "other_comments_or_suggestions": "- Reference error: GNNadapter in L69 should be AdapterGNN.\n- Figure 1 should give more captions for understanding, e.g. dark blue/light blue circles, the meaning of the dotted box.\n- For propositions with proof, a reference link can be provided.", "questions_for_authors": "- Does path-dependent weight increase the number of parameters? Please provide the number of parameters for the different fine-tuning methods.\n- please refer to the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X7mFN3x77I", "reviewer": "Reviewer_G2Qy", "summary": "The paper introduces Equivariant Low-Rank Adaptation, a parameter-efficient fine-tuning method for pre-trained equivariant Graph Neural Networks used in interatomic potential modeling. Unlike existing fine-tuning approaches that break equivariance, ELoRA employs a path-dependent low-rank decomposition to update weights while preserving equivariance, ensuring physically consistent predictions. Theoretical proofs confirm its equivariance preservation, and experiments on organic and inorganic datasets demonstrate that ELoRA significantly improves energy and force prediction accuracy over full-parameter fine-tuning, reducing data requirements while maintaining efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments and suggestions.\n\n**Q1: The meaning of $K$.**\n\nA1: $K$ means the number of channels. $K_0^1$ means the number of channels of the first tensor when $l=0$.\n\n**Q2: The meaning of the superscript of $K$.**\n\nA2: The superscript of $K$ does not represent a power. The $K$ in Figure 3 corresponds to the $K$ in the equations of Section 4.2. Here, the superscript of $K$ is used to identify different tensors, while in Equation (7), the subscript of $k$ is used for the same purpose. We will address this in our revised paper.\n\n**Q3: The ambiguity of the multiplication symbol.**\n\nA3: In line 254, it is used to denote the dimension of a matrix. In lines 256–257, it means $R \\ll \\min(K^3_{l_3}, K^2_{l_2} \\cdot K^1_{l_1})$. The use of the multiplication symbol here might cause misunderstanding, and we will revise it.\n\n**Q4: The choice of $K_3$.**\n\nA4: $K_3$ is a hyperparameter of the neural network and can be specified arbitrarily.\n\n**Q5: The connection between $K$ and datasets.**\n              \nA5: Here, $K_1$, $K_2$, and $K_3$ are not uniquely determined by the dataset; rather, they are hyperparameters of the network and can be set arbitrarily. This is similar to the hidden dimensions in an MLP, which can be larger than the input dimension. The number of channels used in our model is provided in Section E.4, which is typically set to 128.\n\n**Q6: Code availability.**\n\nA6: ELoRA is implemented based on e3nn. We have provided the code in the anonymous repository https://anonymous.4open.science/r/ELoRA/README.md.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidenc", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I checked the proofs. The most important one is the proof of equivariance of ELora, it's correct.", "experimental_designs_or_analyses": "The experiments are reasonable and sound.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n1. Preserves Equivariance in Fine-Tuning – Unlike traditional fine-tuning approaches that break equivariance, ELoRA ensures that equivariance is maintained throughout the adaptation process, which is crucial for physically consistent predictions in interatomic potential modeling.\n2. Improves Data Efficiency – By leveraging low-rank adaptations, ELoRA significantly reduces the amount of training data required for fine-tuning while maintaining or even improving predictive accuracy, making it highly practical for resource-intensive scientific applications.\n3. Strong Theoretical Foundation – The paper provides rigorous mathematical proofs demonstrating that ELoRA preserves equivariance and effectively projects equivariant messages into a lower-dimensional space.\n4. Superior Performance on Benchmarks – Experimental results on organic (rMD17) and inorganic datasets show that ELoRA achieves state-of-the-art accuracy, outperforming both full-parameter fine-tuning and models trained from scratch in energy and force prediction.\n\nWeaknesses & Questions:  \nFor Figure 3 and Eq. 8:  \n(1) What does $K$ mean? Does it mean the number of channels? And $K_0^1$ means the number of channels of the first tensor when $l=0$?  \n (2) The superscript of K is the index instead of the power, right? The notation makes me confused. In Eq.7, the $k$ uses subscript, but in Fig. 3, the $K$ uses superscript.  \n(3) I think the notation × has different meanings. In line 254, it is used to denote the dimension of a matrix. But in other places, does it mean product?  For example, in line 256-257, do you mean $R \\ll \\min (K_{l_3}^3, K_{l_2}^2 \\cdot K_{l_1}^1 )$?   \n (4) How to get $K^3$? Is it a hyperparameter or is it decided by $K^1$ and $K^2$?  \n\nFor the method and setting:  \n(5) So this papers assumes $R \\ll \\min (K_{l_3}^3, K_{l_2}^2 \\times K_{l_1}^1 )$. But in some real-world scenarios, the number of channels = 1. For example, many datasets only have the atomic number when l=0 and the atom position when l=1. In this case, is Elora still meaningful? Could the authors provide more information about the datasets used? For example, the datasets in E.1 and E.2, how many channels do they have? And could you provide the table of the number of parameters to tune with and without ELoRA?   \n\nFor the implementation:  \n(6) No code available.    \n(7) Did you implement ELoRA based on e3nn?", "other_comments_or_suggestions": "See Strengths And Weaknesses", "questions_for_authors": "See Weaknesses & Questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission proposes an equivariant Low-Rank Adaptation (ELoRA) method for efficiently finetuning tensor-product-based equivariant GNNs. This method fills an important gap for finetuning equivariant models, and makes a reasonable contribution to the field of equivariant foundation models for molecular systems. The empirical demonstrations are quite supportive in general.

A few issues and queries are also raised during the review. Reviewer qEUK noticed the results on rMD17 are not under a fully convincing setting, which turns out not a critical problem after authors' explanation that it follows previous settings (the advantage under this setting is still not as significant as in the inorganic case, though). The authors properly responded to Reviewer S1HM's request for an empirical verification of equivariance and presentation issues, and Reviewer G2Qy's request of additional ablation study and hyperparameter choices. I didn't find remaining major problems. Considering the timely contribution, I would like to recommend an accept.
Decision: Accept (poster)