3D Interaction Geometric Pre-training for Molecular Relational Learning
Authors: Namkyeong Lee
Yunhak Oh
Heewoong Noh
Gyoung S. Na
Minkai Xu
Hanchen
Tianfan Fu
Chanyoung Park
Venue: NeurIPS 2025 spotlight ()
Keywords: molecular science
molecular relational learning
Primary area: machine_learning_for_sciences

Abstract: Molecular Relational Learning (MRL) is a rapidly growing field that focuses on understanding the interaction dynamics between molecules, which is crucial for applications ranging from catalyst engineering to drug discovery. 
Despite recent progress, earlier MRL approaches are limited to using only the 2D topological structure of molecules, as obtaining the 3D interaction geometry remains prohibitively expensive.
This paper introduces a novel 3D geometric pre-training strategy for MRL (3DMRL) that incorporates a 3D virtual interaction environment, overcoming the limitations of costly traditional quantum mechanical calculation methods. 
With the constructed 3D virtual interaction environment, 3DMRL trains 2D MRL model to learn the global and local 3D geometric information of molecular interaction.
Extensive experiments on various tasks using real-world datasets, including out-of-distribution and extrapolation scenarios, demonstrate the effectiveness of 3DMRL, showing up to a 24.93% improvement in performance across 40 tasks.
Our code is publicly available at https://github.com/Namkyeong/3DMRL.

3D Interaction Geometric Pre-training for Molecular Relational Learning
Namkyeong Lee', Yunhak Oh', Heewoong Noh', Gyoung S. 1.2
Minkai Xu', Hanchen Wang Tianfan Fu², Chanyoung Park
1 KAIST 2 KRICT
Stanford University
Genentech Nanjing University
Abstract
Molecular Relational Learning (MRL) a rapidly growing field that focuses on understanding the interaction dynamics between molecules, which is crucial for applications ranging from catalyst engineering to drug discovery. Despite recent progress, earlier approaches are limited to using only the 2D topological struc- ture of molecules, as obtaining the 3D interaction geometry remains prohibitively expensive. This paper introduces a novel 3D geometric pre-training strategy for MRL (3DMRL) that incorporates 3D virtual interaction environment, overcoming the limitations of costly traditional quantum mechanical calculation methods. With the constructed 3D virtual interaction environment, 3DMRL trains 2D MRL model learn the global and local 3D geometric information of molecular interaction. Extensive experiments on various tasks using real- world datasets, including out-of- distribution and extrapolation scenarios, demonstrate the effectiveness of 3DMRL, showing up to 24.92 24.93% improvement performance across 40 tasks. Our code is publicly available at httpt::: /ttpt/////hhhtt/////ge ctmb/ccom///// Naw/yeongk////// 3DMRL
Introduction
Molecular relational learning (MRL) focuses on understanding the interaction dynamics between molecules and has gained significant attention from researchers thanks its diverse applica- tions 21 32]. Despite recent advancements in
MRL, previous works tend to ignore molecules' 3D geometric information and instead focus solely on their 2D topological structures. How- ever, in molecular science, the 3D geometric information of molecules (Figure (a)) is cru- cial for understanding and predicting molecular behavior across various contexts, ranging from
(a) Single Molecule
(b) Molecular Interaction Environment
Figure 1: 3D geometry of (a) individual molecule and (b) the molecular interaction envi- ronment.
physical properties [ to biological functions (10). This is particularly important in MRL, geomet- ric information plays a key role in molecular interactions by determining how molecules recognize, interact, and bind with another in their interaction environment [35]. In traditional molecular dynamics simulations, explicit solvent models, which directly consider the detailed environment of molecular interaction, have demonstrated superior performance compared to implicit solvent models, which simplify the solvent as a continuous medium, highlighting the significance of modeling the complex geometries of interaction environments [4]]
However, acquiring stereochemical structures molecules is often very costly, resulting in limited availability of such 3D geometric information for downstream tasks (24]. Consequently, in the
"Conresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

domain of molecular property prediction (MPP), there has been substantial progress in injecting 3D geometric information to 2D molecular graph encoders during the pre-training phase, while utilizing only the 2D molecular graph encoder for downstream tasks [36 25 In contrast, compared to the MPP, pre-training strategies for MRI. have been surprisingly underexplored, primarily due to the following two distinct challenges in modeling complex molecular interaction environments.
Firstly, interactions between molecules occur through complex geometry as they are chaotically distributed in space as shown in Figure T (b). Therefore, it is essential to consider not only each molecule's 's independent geometry but also their relative positions and orientations in space. This requirement further complicates the acquisition of geometric information, making it more challenging to obtain detailed 3D geometry of molecular interaction environments. Consequently, it is essential to model an interaction environment that can simulate molecular interactions based solely on the 3D geometry of the individual molecules.
Secondly, even after constructing the interaction environment, how to inject the geometry between molecules during interactions are not trivial. More specifically, while the global geometry of the interaction environment is essential for understanding overall interactions and system stability, the local geometry is also critical for examining localized interactions and precise molecular behaviors. Therefore, developing pre-training strategies that effectively capture the complementary global and local geometries between molecules and their interaction environment essential.
To address these challenges, we introduce a novel 3D geometric pre-training strategy that is appli- cable various MRL models by incorporating the 3D geometry of the interaction environment for molecules (3DMRL). Specifically, instead of relying on costly traditional quantum mechanical calcu- lation methods obtain interaction environments, involving multiple molecules designed to simulate real molecular interactions. Then, during the pre-training stage, a 2D MRL model is trained to produce representations that are globally aligned with those of the 3D virtual interaction environment via contrastive learning. Additionally, the 2D MRL model is trained to predict the localized relative geometry between molecules within this virtual interaction environment, allowing the model to effectively learn fine- ane-grained atom-level interactions between molecules. These two pre- pre-training strategies enable the 2D MRL model to be pre-trained to understand the nature of molecular interactions, facilitating positive transfer to a wide range of downstream MRI. tasks. In this paper, we make the following contributions:
first propose virtual interaction environment
Rather than relying on costly traditional quantum mechanical calculation methods to obtain interac- tion geometry, we propose virtual interaction geometry made up of multiple molecules to mimic the molecular interaction environment observed in real-world conditions.
We propose pre-training strategies that allow the 2D MRL model to learn representations of the 3D interaction environment, capturing both its global and local geometries.
We conduct extensive experiments across various MRL models pre-trained with 3DMRL on a range of MRL tasks, including out-af- -i----iiiitiibttion and extrapolation scenarios. These experiments demonstrate improvements of up to 24.93% compared to MRL methods trained from scratch, underscoring the versatility of 3DMRL (Section 5).
To the best of our knowledge, this is the first paper proposing pre-training strategies specifically designed for molecular relational learning.
2 Related Works
Molecular Relational Learning. Molecular Relational Learning (MRL) focuses on understanding the interaction dynamics between paired molecules. Delfos (23 employs recurrent neural networks
combined with attention mechanisms to predict solvation-free energy, key factor influencing the solubility of chemical substances, using SMILES string as input. Similarly, CIGIN (32 utilizes pesssae--aassigg neural networks [II] along with cross-attention mechanism to capture atomic
representations for solvation free energy prediction. In a different context, Joung et al. [I7 use graph convolutional networks [18] to generate representations of chromophores and solvents, which are then used to predict various optical and photophysical properties of chromophores, essential for developing new materials with vibrant colors. Meanwhile, MHCADDI 4 introduces a co-attentive message passing network (38] designed for predicting drug-drig interactions (DDI), which aggregates information from all atoms within pair of molecules, not just within individual molecules. Recently,

---

and CMRL 12 have introduced comprehensive framework for MRL tasks, such as predicting solvation-fiee energy, chromophore-soluun interactions, and drug-drug interactions. These models achieve this by identifying core functional groups involved in molecular interactions using information bottleneck and causal theory, respectively. However, prior studies have largely ignored molecules' 3D geometric information despite its well-established importance in comprehending various molecular properties.
CGIB 21
3D Pre-training for Molecular Property Prediction. Recently, the molecular science community has shown increasing interest pre-training machine learning models with unlabeled data, primarily due to the scarcity of labeled data for downstream tasks [22 37 44 A promising approach in this area leverages molecules' inherent nature, which can be effectively represented as both 2D topological graphs and 3D geometric graphs. For instance, 3D Infomax [36 aims to enhance mutual information between 2D and 3D molecular representations using contrastive learning- GraphMVP (4 extends this concept by introducing generative pre-training framework alongside contrastive learning. More recently, Noisy Nodes [46] and MoleculeSDE) [25] have introduced methods to learn the 3D geometric distribution of molecules using denoising framework, thereby uncovering the connection between the score function and the force field of molecules. Although the 3D structure molecules has been effectively leveraged in pre-training for predicting single molecular properties, it remains surprisingly underexplored in the context of molecular relational learning (MRL). We provide more detailed explanations with the figure in Appendix
3 Preliminaries
3.1 Problem Statement
Notations. Given a molecule g. we first consider a 2D molecular graph, denoted gan (X,A), where X XxF represents the atom attribute matrix, and E R N the adjacency matrix, with define 3D conformer as
A, 1 if covalent bond exists between atoms i and j. Additionally, _BD = (X. R), where R E R- x3 is the matrix of 3D coordinates, each row representing the spatial position of an individual atom.
Task Description. Given 2D molecular graph pair (gin: kip) and 3D conformer pair (gin, k our goal is to pre-train the 2D molecular encoders fhp and fan simultaneously with the virtual interaction geometry Ive, derived from the 3D conformer pair. Then, the pre-traie pre-trained 2D molecular encoders faD and 2 are utilized for various MRL downstream tasks.
3.2
2D MRL Model Architecture
In this paper, mainly focus 1) the construction of virtual interaction geometry, and 2) pre- training strategies for MRL. Therefore, we employ existing model architectures for 2D MRL, i.c., CIGIN (32) which provides straightforward yet effective framework for MRL as depicted in Figure (a). Specifically, for each pair of 2D molecular graphs, denoted as 920 and 92p- the graph neural networks (baNs) molecular encoders fhp and f20 initially produce an atom embedding matrix for each molecule, formulated as:
E
PDD (OLD), E 12D (g2D).
(1)
where E N and E R No xd the atom embedding matrices for gho and 02p- containing 2 and N2 atoms, respectively. Next, we capture the interactions between nodes in gho and gid using interaction matrix I e N xN2 defined by L sim E EE),, where sim(..) represents the cosine similarity measure. Subsequently, we derive new embedding matrices E and E² E N for each graph, reflecting their respective interactions. This is computed using
IR N xd E' II 2 and E I E', where denotes matrix multiplication. Here, E' represents the node embeddings of 92D that incorporates the interaction information with nodes in 92p, and similarly for É², To obtain the final node embeddings, we concatenate the original and interaction-based embeddings for each graph, resulting in H
(E E T x24 and H (E² (²) E IR x2d Finally, we apply the Set2Set function 40 to compute the graph-level embeddings z and z for graph dip and g2p. respectively.
3

---

[a] ZD MRL Model Architecture
(b) Virtual Interaction Geometry Construction
[c) SE(3)-Invariant Global Geometry Learning
kizu)
xio
Projection
Larger Moleade (ofe)
R PT
Virtual Interection (par
Contrastive Learning Ligish
Nariccle loigl
Node Interaationn
(d) SE(3)-Equivvrriat Local Relative Geometry Learning
Viriull Internction Endhrtemnnt id..)
X (IF (4E)
hu
(0 (BF)
MLP
ix 4o fx
L
g20
Projection
Cescol
Relative
fz
Si Stll-egpanntanttroomm Frore
Figure 2: Overall Framework: (a) MRL model architecture (Section 3.2). (b) Virtual interaction SF(3) Global Geometry Learning (Section 4.2.1
geometry construction (Section 1.1... (c)
(d) S -qqiivrauiaariant Local Relative Geometry Learning (Section 4.2.2
4 Methodology
In this section, we introduce our method, named 3DMRL, a novel pre-training framework for MRL utilizing 3D geometry information. Specifically, Section 4.1 we introduce how construct the virtual interaction geometry that can be utilized instead of expensive calculation of real interaction geometry of molecules. Then, in Section 4.2 we present two complementary geometric pre-training strategies for the 2D MRL model to acquire representations aligned with the constructed virtual interaction geometry in both global and local perspectives. The overall framework is depicted in Figure 2 and the pseudocode for the entire framework is provided in Appendix
4.1 Virtual Interaction Geometry Construction
While the 3D geometry of molecules plays a significant role in predicting molecular properties, acquiring this information involves a trade-off' between cost and accuracy. For example, RDKit's ETKDG algorithm 20 is fast but less accurate. In contrast, the widely adopted metadynamics method, CREST
12). achieves a more balanced compromise between speed and accuracy, yet still requires around hours process a drug-like molecule. This challenge even more pronounced in molecular interaction systems, which necessitates not just the geometry of individual molecules but also the relative spatial arrangements between multiple molecules (6. Moreover, an appropriate initial geometry of molecular interaction system is highly dependent on individual molecules in the system (27. For this reason, data-agnostic process for generating the initial molecular geometry crucial for flexible and robust representation learning on molecular interaction systems.
Drawing inspiration from the explicit solvent models used in traditional molecular dynamics simula- tions (9]. we propose one-to-many geometric configuration that involves a relatively larger molecule aep, determined based its radius, surrounded by multiple smaller molecules Shp as shown in Figure (b) (28). Specifically, for a given conformer pair (gip (x-, RRR A.p = (X* R²)) we create an
2.n around centrally placed environment by arranging the smaller molecules (g2) ..... 93D ................................ IAD
2.1 2,
larger molecule sip as follows:
[Step 1] Select Target Atoms in the Larger Molecule. We start by randomly selecting atoms from the larger molecule dip that are not part of any aromatic ring. This choice is based on the fact that aromatic rings are more stable and less likely engage chemical reactions.
[Step 2] Positioning the Smaller Molecules. Each smaller molecule in (0 ..... 93D .... 93D
2
2.n
is then placed close to one of the selected atoms in the larger molecule o.p. This positioning

---

is achieved by transiting and rotating the original 3D coordinates R of the smaller molecule GDD. following the method widely employed in computational chemistry [9]
[Step 2-1] Determine Transition Direction. For flexible and robust molecular relational learning, we follow a widely used strategy that samples initial geometries from parameterized stochastic processes 42). Specifically, we generate normalized random Gaussian noise vector (with norm of 1), which will be used to set the direction for the transition. We then scale this direction vector by the radius of the smaller molecule, x², to establish the transition distance.
[Step 2-2] Transit and Rotate to the New Position. The new 3D coordinates for each smaller molecule are determined using the formula R R + E; * r² + +R, where R! E TR3 represents the 3D position of the i-th selected atom in the larger molecule gap This operation is performed through broadcasting, meaning R and E; are added to each row of R², Additionally, we apply a random rotation matrix rotate the small molecule after its transition. This transition and rotation operations ensure that each smaller molecule is positioned close to its corresponding selected atom the larger molecule, simulating realistic interaction environment.
[Step 3] Constructing Virtual Interaction Geometry. After positioning each smaller molecule 24
3sD near the i-th selected atom in the larger molecule gbp. we compile all the 3D coordinates to form a unified virtual environment Jvr. This process involves combining the coordinate matrix R of the larger molecule 93p. with the transited coordinates (R2. ..... R R2. of the
smaller molecules (g)D .......8 IDD ................................ ............ IaD 2,),, resulting in R,, (R- | |R [[22** tnN*+nnN22>
Additionally, it involves concatenating all the atom attribute matrices to form X., - [X X 1.......'' IX2) E RR + thereby defining the virtual interaction geometry as Ivr = (X R..)) Note that multiple small molecules share the same attribute matrix X2, since we use the atom attribute irrelevant to the atomic coordinates.
Note that such randomized configurations of interaction environment is a well-established strategy in molecular simulations. For instance, protein-ligand docking protocols (e.g.. Rosetta) often initialize ligands in random orientations relative to the protein before searching for binding modes. Similarly. Monte Carlo insertion methods like Widom's 's test-particle approach randomly insert solvent molecules to explore solute-solvent configurations without bias. Moreover, while we construct the virtual interaction geometry (Step 1 to Step 3) at each epoch during the pre-training phase, the virtual environment can be generated in real time because transition and rotation are matrix operations. Therefore, we argue that our approach allows efficient sampling over a wide range of distances and orientations while remaining physically sound: in the limit of sufficient sampling, no unphysical configuration is favored, and the process mimics the early stages of solvation when solvent molecules approach from arbitrary directions. In Section and and Appendix E..4 we analyze the environment in various aspects, justifying the proposed approach for constructing virtual interaction environment.
4.2 Pre-training Strategies
Once the virtual interaction geometry is established, we pre-train the 2D MRL model using two com- plementary geometry learning strategies: (En3)-invriiant global geometry learning (Section 4.2.1 and (F)(S -qquivarraa local relative geometry learning (Section 4.2.2).
4.2.1 SEE33 Global Geometry Learning
Given paired 2D molecular graphs (g)p PiD) and its corresponding 3D virtual interaction geometry Ivr. we first encode them with 2D MRL model, and a geometric deep learning model, respectively. For 2D molecular graphs, we compute the molecule-level representations, znp and z for each molecule glp and 2 respectively, as outlined in the Section 3.2 Following this, we derive the 2D interaction representation ZZD. by concatenating these two representations, i.c., Zzp = (= | On the other hand, to encode the virtual interaction geometry gvr = (X,r, R2y), we use geometric given the coordinates of atoms R,, 34 i.e., ZaD = (R,r). Then, as shown in Figure 2 (c), we
GNNs Jan that output SFE(3 invariant
7 representations
in virtual interaction geometry
align the 2D interaction representation Z and the 3D geometry representation ZıD via Normalized
5

---

lemperature-scaled cross entropy loss 3 as follows:
Nusa
sin(2m,, Rap,.
Cglob
log
Murch g R.b,) + log Maa eain(zu. i km
Nbatch
Σ k=ll
where sim(:.) represents cosine similarity, T denotes the temperature hyperparameter, and Match refers to the number of pairs within a batch. By training the 2D MRL model to output interaction representations that align with the 3D interaction geometry, the model effectively learns the overall global geometry of molecular interactions during the pre-training phase.
4.2.2 (3 Equivariant Local Relative Geometry Learning
Beyond the overall global geometry of interaction, it is essential to learn about the intermolecular local relative geometry between molecules during molecular interactions, their localized relative geometry governs how molecules interact in various environments. To achieve this, we propose pre-training the 2D MRL model to learn local relative geometry by predicting the 3D geometry of the paired molecule, specifically by training a smaller 2D molecule encoder to predict the geometry of the larger molecule. However, predicting relative geometry from a 2D representation challenging because the prediction must adhere to the physical properties of the molecule, specifically being equivariant to rotations and transitions in 3D Euclidean space, also known as -Equ3---quvarrancce [ To address this, we propose predicting the relative geometry between molecules by utilizing local frame [5], which allows for flexible conversion between invariant and equivariant features.
2.1 More specifically, given the position 222 of the i-th small molecule 93D in the constructed virtual interaction geometry, we first define an orthogonal local frame Fk,t between atoms k and / within 2, molecule 9nD as follows:
FR.
r
X
||r 1 rill X
F2
F X rI ||L
(2)
||T ryll'
P& xill
rz|l
where r& E 3 and ri R33 indicate the position of atoms i and I in constructed virtual inter- action geometry, respectively. For simplicity, please note that we will omit the molecule index i in the notation from here. With the established local frame,
derive the invariant 3D fea- ture for the edge between atoms k and / by projecting their coordinates into the local frame, i.e., e3D = Projection,, (IK, Rd. Additionally, we obtain the 2D invariant edge feature be- tween atoms k and 1 by concatenating the respective features from the 2D molecular graph, i.e., k.l
k.l
eD M H Rd. Now that we have both invariant 2D and 3D features, can derive the final invariant edge feature end! by combining these invariant edge features follows:
k.l k.l en + eap
ex
We define the edge feature set E, which includes ex,1 for every possible pair of atoms.
With the invariant final edge feature set E, through GNNs to predict the geometry of the larger molecule. To achieve this, we first obtain the atom features specific the i-th small molecule by concatenating the i-th atom representation of the larger molecule (to which the i-th small molecule is assigned) with each atom representation of the small molecule, i.e., X = (H2IHH) E R using broadcasting. This approach allows the model to learn a more precise geometry by incorporating the features of the assigned atom in the larger molecule. Next, with the edge feature set & and the atom feature X. we derive the final edge representation h through multiple GNN layers, represented as h = GNN(N, E). Finally, we determine the relative geometry fü between the atom & of the small molecule and the central larger molecule by combining the final invariant edge representation h with our SE(3) -3euevvraaai frame FR.1 as follows:
further process the small molecule information
in = Σ hi O FE.
where @ indicates element-wise product. This approach guarantees our predicted relative geometry Then, we calculate the relative geometry prediction loss as follows:
fü to be
N
Liocal n
f.
(5)
N

---

where f represents the ground truth relative geometry between the larger molecule and the k-th atom of the i-th small molecule. We define the relative geometry f% as the direction between the k-th atom of the i-th small molecule and the i-th atom of the larger molecule to which the small molecule attached, i.e. fk (R /
R, l12. Note that Liocal is calculated for every molecule pair in the batch, although we have omitted this notation for simplicity.
Finally, pre-train the 21D MRL model by jointly optimizing two proposed losses, i.e., SF)) invariant global geometry loss and
local relative geometry loss, as follows:
Cpre-trin
Lglob + a Liocal:
(6)
where a is hyperparameter that determines the trade-off between the global geometry loss and the local geometry loss. After task-agnostic pre-training, the 2D) molecular encoders fhb and fiD
fine-tuned for specific downstream tasks where access to 3D geometric information is limited.
4.3 Discussion
While we define local relative geometry for learning fine- fine- sne-ggrined interactions between molecules. we can view local relative geometry as an interaction force between molecules. This provides physically motivated supervision signal rooted in classical intermolecular forces, many of which are central and act along the internuclear axis. For example, van der Waals interactions (described by the Lennard-Jonel potential) exhibit repulsive or attractive forces directed along this axis. At short distances, repulsion dominates and aligns directly outward between nuclei.
This supervision scheme serves as central-force approximation, consistent with classical force fields, and offers a lightweight surrogate for full force labels. which would require costly quantum chemistry or MD simulations. Notably, SchNet 34] demonstrated that even approximate force signals improve learning of molecular interactions. Our firection-base supervision enables the model learn geometric features like hydrogen bond alignment or steric repulsion trajectories in an SE(3) equuivrruu manner.
Since solvent atoms are placed near specific solute atoms, the dominant interaction direction aligns with the interatomic vector, making it reasonable proxy for the net force axis. Thus, the unit direction vector serves pseudo-force label, conveying the primary interaction axis and encouraging the model to encode directionality of intermolecular interactions.
5
Experiments
5.1 Experimental Setup
Downstream Tasks. Following a prior study 21.. we employ ten datasets to comprehensively
evaluate the performance of 3DMRL on two tasks: 1) molecular interaction prediction, and 2) drug-drug interaction (DDI) prediction. For the molecular interaction prediction task, we utilize the Chromophore dataset [16), which pertains to three optical properties of chromophores, along with five other datasets related to the solvation free energy of solutes: MNSol (26.. FreeSolv 22). CompSol [30], Abraham [3), and CombiSoly 39 In the Chromophore dataset, we focus on the maximum absorption wavelength (Absorption), maximum emission wavelength (Emission). and excited state lifetime (Lifetime) properties. For the DDI prediction task, we use two datasets: ZhangDDI 48 and ChChMiner [49], both of which contain labeled DDI data. provide further details on pre-training and downstream task datasets in Appendix B.
and B.2 respectively.
Baseline methods. We validate the effectiveness of 3DMRL by using enhance various recent oate-of-the-ar molecular relational learning methods, including MPNN [T]. AttentiveFP [43). CIGIN 32 CGIB (2T). and CGIB [21] Additionally, we compare our proposed pre-training framework, 3DMRL, with recent molecular pre- preiiiaining approaches that aim to learn 3D structure of individual molecules, such as 3D Infomax (36). GraphMVP 24 and MoleculeSDE (25). is
important to note that these approaches involve pre-training single encoder for molecular property prediction (MPP Pre-training in Table(2. whereas our work pioneering in training two separate encoders simultaneously during pre-training for molecular relational learning (MRL Pre-training in Table For the baseline methods, we use the original authors' code and conduct the experiments in the same environment 3DMRL ensure fair comparison. Moreover, while we choose to mainly
7

---

Table 1: Performance improvement in molecular interaction tasks across different models with our proposed pre-training strategy (RMSE) (L). We conduct 15 independent runs for each model and report their mean along with the standard deviation (in parentheses). Colors indicate the performance improvement compared to the models trained from scratch.
Chromophore
Model
MNSol
FreeSolv
CompSol
Abraham
CombiSoly
Absorption
Emission
Lifetime
MPNN
22. 033
26.31
0.789 such
0. 3.613 reme
1.127 1.
0.120 cums
0.640 moes)
0.611 (sen)
3DMRL
19.96 011))
25.21
0.753
0.609
1.068 sa
0.377
0.550
0.599 coucs)
Improvement
4.29%
4.56%
5.28%
5.24%
10.24%
14.06%
2.44%
AtientiveFP
22. m339
26. 70 -025
0.871 (0.000
O. 57570 juuct
1.019 0.000
0.350 (2
0.626 1004))
0:471 (0020)
3DMRL
28.54
jaso
0.901 aee
(03M0))
0.378 (ouzz))
0.448 (Sun)))
Improvement
0.26%
0.55%
9.99%
1.40%
11.57%
22.5798
11.26%
4.88%
CIGIN
19. mus9)
25.81.0025
0.821 (0.17))
1.582 (e/22)
0.958 (11110)
0. 0.369 (2005))
0.121 (6013)
0.461 (2022)
3DMRL
18:00 (137)
24.21 01340
0.729 (amn
0.528 saar
0.839 oa 1089
0.277 (CUMM)
0.371 (ous)))
0:435 (DUN))
Improvement
8.44%
6.30%
11.20%
9.28%
12.42%
24.93%
11,87%
6.25%
CGIB
18.37 0.35)
21.52 0015)
01888 (0.015)
0.562 (eu00s
0.876 (0187)
0.321 (2002)
0.104 (oua7)
0:418 (2008)
3DMRL
17.93 m.35)
92 00:39
0.733
0. 1.538 (6.00m
0.842 in 078)
0. 0.274 (5/002)
0.370 (6.27))
0.442 (8015)
Improvement
2.40%
5,90%
9.28%
4.27%
3,88%
8.42%
CGIB
18.39 0.24)
21.68 o0.4)
(1803 (0009)
0.561 (e,12))
0.897
(a (0.0
O1 13333 (2UM5)
0.404 (6429)
0.452 (2015)
3DMRL
17 17.90
23.94.00344
0.720
0.863 075)
2.284
0.372 (6/20))
0.441 (8/22))
Improvement
3.71%
3,00/2
10.33%
6.59%
3.79%
14.71%
7.92%
2.43%
compare 2D encoder pre-training approach, we also compare 3D encoder pre-training approaches
15
31
[ in Appendix E.3
We provide more details the compared methods in Appendix C
Evaluation protocol. Following Pathak et al. [32), for the molecular interaction prediction task, we evaluate the models under a 5-fold cross-validatio scheme. The dataset is randomly split into 5 subsets and one of the subsets is used as the test set, while the remaining subsets are used to train the model. A subset of the test set is selected as the validation set for hyperparameter selection and early stopping. We repeat 5-fold cross- validatiin three times (i.e., 15 runs in total) and report the accuracy and standard deviation of the repeats. For the DDI prediction task (21), we conduct experiments on two different out-o--diittiibution scenarios, namely molecule split and scaffold split. For the molecule split, the performance is evaluated when the models are presented with new molecules not included in the training dataset. In the scaffold split setting [4], just like in the molecule split, molecules corresponding to scaffolds that were not seen during training will be used for testing. For both splits, we repeat 5 independent experiments with different random seeds on split data, and report the accuracy and the standard deviation of the repeats. In both scenarios, we split the data into training, validation, and test sets with a ratio of 60/20/20% We provide details on evaluation
protocol, model implementation, and model training in Section D
5.2 Experimental Results
We begin by comparing each model architecture trained from scratch with the same architecture pre-trained using our proposed strategy, referred to as +3DMRL in Table We have the following observations: 1) 3DMRL obtains consistent improvements over the base graph neural networks in all 40 tasks (across various datasets and neural architectures), achieving up to 24.93% relative reduction in RMSE. While the paper is written based on CIGIN for better understanding in Section 3.2 we could observe performance improvements not only in CIGIN but also in various other model architectures. demonstrating the versatility of proposed pre-training strategies. We further demonstrate how our pre-training strategies are adopted to various model architectures in Appendix C
Additionally, we compare our pre-training strategies with recent molecular pre-training approaches proposed for molecular property prediction (MPP) of a single molccule. Table 2 (a) and (b) show the results for the molecular interaction prediction task, and the drug-drug interaction (DDI) task, respectively. As these approaches are originally designed for single molecules. we first pre-train the GNNs using each strategy, then incorporate the pre-trained GNNs into the CIGIN architecture and fine-tune them for various MRL downstream tasks. We have the following observations: 2) Although MPP pre- pre-training methods have demonstrated success in molecular property prediction in prior studies, they did not yield satisfactory results in molecular relational learning tasks and, in some cases, even resulted in negative transfer. This highlights the need for creating specialized pre-training strategies tailored to MRI. tasks. We further demonstrate the MPP pre-training strategy with a large- scale dataset still performs worse than 3DMRL in Appendix E.1 3) On the other hand, pre- pre-training
8

---

Table 2: Performance of CIGIN model on (a) molecular interaction tasks using different pre-training strategies (RMSE) (J) and (b) out-of- -it-o--dittibuttion DDI tasks using different pre-trate vee-trainnng strategies (AUROC) (t). For each dataset, we highlight the best method in bold.
Medocitar Interaction Taoks (RMSE
Uht Denp Deng-dingg Interaction thek (AUIOOG t>
Meraiegy
Chrvespharre
MNNe
Fressehr
Comptie
Abrahan
Combitot
Maleeli 9y4i
14: Me- split
No Pie suising
1106 osin
1L52: minn
0.207
0.066 san
0:511 sono
1/422 sct
1:454 coot
7L.76-20e
Th-21 cum
75,41 amo
candecake progats
Pre-iminies
16-73 cm
36.95 est
9.821 mAto
5-685 KaM
B4484 mona
21.0 cUm
Thulk cam
71
16.56 eou
24,95 oi
01266 (4:
9.971 9:30
0.086 sutt
1:635 sctu
0:454 anits
70007
01.48c..8 mM
T1.63 cm
lcanteg)
30MAL
18 with
24 ouss 0.720 a0e4
0.82
0.608
0.977 (ANOO
00371 aMt
0.436 A.RC
7400
75.90 8m
15.30 74:89:00
with 3DMRL consistently delivers significant performance improvements across downstream tasks. This validates the effectiveness of our approach, as it successfully integrates scientific knowledge into the pre- pre-training strategy, enhancing the model's 's overall performance. 4) Additionally, for the DDI task in (b), we observed that the performance improvement is more pronounced in challenging scenarios ((d) Scaffold split) compared to less difficult ones ((c) Molecule split). This highlights the enhanced generalization ability of 3DMRL in --diiiiiiiiittoo scenarios, demonstrating its potential for drug discovery applications where robust generalization across unknown molecules is essential. We explore the extrapolation capability of 3DMRL Appendix E.2
5.3 Model Analysis
To further understand our model, we conduct an ablation study to investigate the impact of two key components on the final performance. Specifically, as shown in Equation 66] the objective function contains two terms: (i) global geometry loss and (ii) intermolecular local geometry loss; we curate two variants that involve only (i) (denoted only glob.) and only (ii) (denoted only
Ablation Studies.
3DMRL
Only glob.
Only lcca
0.50
0.76
27
15.5
0.76
74.0
18.0
735
0.70
local) in Figure 3 As shown in Figure 3 learning the
17.0 Absorption
Lifetime Melesular Interaction (RM5F)
0.50
72.5
Zhangool
OcChiiner
global geometry plays particularly critical role. Remov- ing it from 3DMRL results in a significant performance drop, even falling below MPP pre-training strategies such as 3D Infomax and GraphMVP. This is because the global
DDI (AUBOC)
Figure 3: Ablation studies.
geometry loss allows the model to capture the overall interaction geometry at the molecular level, while the local geometry loss focuses on learning more fine-grained, atom-level interactions. How- ever, combining both losses, as in 3DMRL, yields the best results, demonstrating the importance of leveraging the strengths of both levels of granularity. We provide further detailed results of ablation studies in Appendix E.3
Molecule Collision Analysis. While the virtual environment is designed to carefully mimic the nature of molecular interactions, as discussed in Section 4.1 molecule collisions can still occur within the environment. To first examine how molecule collisions affect model performance, we created a "No Radius" model that does not take the radius into account during pre-training. Looking at the atomic overlap in Table 3 we observed that 3DMRL, which utilizes radius in- formation, significantly reduces the overlap ratio between molecules compared to the "No Radius" configuration. Moreover, we found that in the "No Radius" case, where the atomic overlap ratio is very high, the performance was much lower than that of the 3DMRL. To further investigate whether further reducing atomic overlap would be helpful, we experimented with several additional configurations. The "Fixed direction" configuration was designed to prevent overlap caused by random direction placement by positioning the solvent along the direction from the origin to the target atom. "Twice radius" refers to multiplying the radius in the equation in line 170. These methods reduce atomic overlap by decreasing randomness and increasing the distance between molecules, respectively; however, in terms of performance, they were either similar to or worse than 3DMRL.
Table 3:
Model performance in various 3D interac- tion environments with reduced collision.
Atomic
Time
Performance
Overlap
(minieppchh)
Absorption
Emitsion
Litetime
37
0.745
0.734
Twice Radiss
18:23
24.25
0.730
Regenerate
23.01
18.20
23.86
0.727
3DMAL
25.12%
18:00
24.21
0.729

---

The experimental results showed that both methods were able to reduce atomic overlap, but in terms of performance, they were either similar to or worse than 3DMRL. Lastly, we introduce "Regenerate." " which regenerates the 3ID virtual environment every time collision occurs between any molecules in a virtual environment. Although the collision between molecules can certainly be avoided in this case, this approach incurs high computational complexity. In Table []] we observe that the performance gain of "Regenerate" is minimal, despite its significantly higher computational requirements, Based on these results, we argue that 3DMRL strikes an appropriate balance between computation and performance.
Sensitivity analysis on n. Moreover, we conduct sensi- tivity analysis to explore the empirical effect of the number of target atoms n, which determines the number of small molecules in a virtual interaction geometry. Figure 4 (a), we observe that the model performs the best when using five small molecules to construct the virtual inter- action geometry. More specifically, using too few small molecules (n = 2) results in poorer performance, as it fails to adequately simulate real-world interaction environ- ments. On the other hand, the model performance also declines as the number of small molecules increases, likely due to the 3D geometry encoder overfitting to the small molecules with an excessive count. Furthermore, increases, more extensive computational resources are required to encode the 3D interaction geometry during pre-training. Hence, selecting an appropriate number of target atoms is crucial for both model performance and computational efficiency. We provide further analyses on virtual interaction geometry and other tasks in Appendix E.44
Assorstion
Compucation imirorspoch)
11.66
Absorptio (BSCE)
Absoe so (WNSE)
Lifertma IRM0E
28:3
6.60
12.0
0.77
3.76
3.5
17.0
0.70
37.4
0.0
011
3.70
[a] Target atoms (n)
(b) Hyperparametera
Figure 4: Sensitivity analysis.
observe that as the number of target atoms
Sensitivity analysis on a. We conduct sensitivity analysis on a, which controls the weight of local geometry loss, in Equation 6 In Figure 4 (b), the model's performance declines increases from 0.1, primarily because it overly emphasizes atom-level interactions between the molecules instead of considering the overall interaction geometry. Conversely, also notice drop in performance when local geometry loss is not utilized (a = 0.0), as this causes the model to lose ability in learning fine-grained atom-level interactions. It is important note that while during pre- pre-training, models pre-trained with varying n and a consistently outperform those trained from seratch, demonstrating 3DMRL's robustness.
5 and 0.1
6 Conclusion
In this work, we propose 3DMRL, a novel pre-training framework that effectively integrates 3D geometric information into MRL. By constructing virtual interaction geometry and utilizing lo- cal and global geometry prediction, our approach effectively incorporates complex 3D interaction geometry information into 2D MRL models. Experimental results demonstrate that 3DMRL signifi- cantly enhances the performance of 2D MRL models across various downstream tasks and neural architectures, validating the importance of incorporating 3D geometric data.
For future work, we intend to develop and train a virtual interaction geometry generator capable of mimicking MD trajectories of molecular interactions. We will then substitute this generator for the purely random generation method currently used Section providing a more physically informed signal.. Furthermore, we plan to broaden this research in drug- drug-target binding affinity prediction, core task in drug discovery that involves complex protein structures as the larger molecule.
Acknowledgement
This work was supported by the Institute of Information & Communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) RKS-2025-02304961 Al Star Fellowship (KAIST)). Additionally, this research received funding from the National Research Foundation of Korea (NRF) through two separate grants: RS-2024-00335099 (funded by the Korea government (MSIT)) and 22220222222772205 (funded by the Ministry of Science and ICT).
10

---

References
[1] Atkins, P. W., De Paula, J., and Keeler, J. Atkins' physical chemistry. Oxford university press, 2023.
[2] Axelrod, S. and
R. Geom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 9(1):185, 2022.
[3] Chen, T., Korbllith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-16077 PMLR, 2020.
[4] Deac, A., Huang. Y.-H., Veličković, P., Liò, P., and Tang. J. Drug-drug adverse effect prediction with graph co-attention. arXiv preprint arXiv: vrXiv:1905.000 1905.00544, 2019.
[5] Du, W., Zhang, H., Du, Y., Meng, Q. Chen, W., Zheng. N., Shao, B., and Liu, T.-Y. Se (3) equivariant graph neural networks with complete local frames. In International Conference on Machine Learning, pp. 5583-5608. PMLR, 2022.
[6] Durrant, J. D. and McCammon, J. A. Molecular dynamics simulations and drug discovery. BMC biology, 9:1-9, 2011.
[7] Duval, A., Mathis, S. V., Joshi, C. K., Schmidt, V., Miret, S., Malliaros, F. D., Cohen, T., Lio, P., Bengio, Y., and Bronstein, M. A hitchhiker's guide to geometric gnns for 3d atomic systems. arXiv preprint arXhv:2312.075111 2023.
[8] Feng. S., Ni, Y., Lan, Y., Ma, Z.-M., and Ma, W.-Y. Fractional denoising for 3d molecular pre-training. In International Conference on Machine Learning, pp.
PMLR, 2023.
[9] Frenkel, D. and Smit, B. Understanding molecudar simulation: from algorithms applications. Elsevier, 2023.
[10] Fu, Y., Lu. Y., Wang. Y., Zhang, B., Zhang, Z., Yu, G., Liu, C., Clarke, R., Herrington, D. M., and Wang, Y. Ddn3. O: Determining significant rewiring of biological network structure with differential dependency networks. Bioinformatics, pp. btae376, 2024.
[11] Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, E. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
[12) Grimme, S. Exploration of chemical compound, conformer, and reaction space with meta- dynamics simulations based on tight-binding quantum chemical calculations. Journal of chemical theory and computation, 15(5):2447-2822, 2019.
[13] Grubbs, L. M., Saifullah, M., Nohelli, E., Ye, S., Achi. S. S., Acree Jr. W. E., and Abraham, M. H. Mathematical correlations for describing solute transfer into functionalized alkane solvents containing hydroxyl, ether, ester or ketone solvents. Fluid phase equilibria, 298(1): 48-53, 2010.
[14] Huang, K., Fu, T., Gao, W., Zhao, Y., Roohani, Y., Leskovec, J., Coley, C. W., Xiao, C., Sun, J., and Zitnik, M. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. arXiv preprint 1XIv:2202.00558. 2021.
[15] Jiao, R., Han, J., Huang, W., Rong, Y., and Liu, Y. Energy-motivated equivariant pretraining for 3d molecular graphs. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 1004, 2023.
[16] Joung, J. F., Han, M., Jeong, M., and Park. S. Experimental database of optical properties of organic compounds. Scientific data, 7(1):1-6, 2020.
[17] Joung, J. F., Han, M., Hwang, J., Jeong, M., Choi, D. H., and Park, S. Deep learning optical spectroscopy based on experimental database: potential applications to molecular design. JACS Au, 1(4):427-438, 2021.

---

[18] Kipf. T. N. and Welling, M. Semi- Semi-superrvsed classification with graph convolutional networks. arXiv preprint arXiv: www.i66090.0000 00907, 2016.
[19] Kuroshima, D., Kilgour, M., Tuckerman, M. E., and Rogal, J. Machine learning classification of local environments in molecular crystals. Journal of chemical theory and computation, 20 1966197-6266, 2024.
[20] Landrum, G. Rdkit documentation. Release, 1(1-79):4, 2013.
[21] Lee, N., Hyun, D., Na, G. S., Kim, S., Lee, J., and Park, C. Conditional graph information bottleneck for molecular relational learning. In International Conference on Machine Learning. pp. 18852-18871, PMLR, 2023.
[22] Lee, N., Yoon, K., Na, G. S., Kim, S., and Park, C. Shift-robust molecular relational learning with causal substructure. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1200-1212, 2023.
[23] Lim, H. and Jung, Y. Delfos: deep learning model for prediction of solvation free energies in generic organic solvents. Chemical science, 10(36):8306 8315, 2019.
[24] Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., and Tang. J. Pre-training molecular graph representation with 3d geometry. arXiv preprini wrrXiv:2110007101 2021.
[25] Liu, S., Du, W., Ma, Z.-M., Guo, H., and Tang, J. A group symmetric stochastic differential equation model for molecule multi-modal pretraining. In International Conference on Machine Learning, pp. 21497-2155 21526. PMLR, 2023.
[26] Marenich, A. V., Kelly, C, P., Thompson, J. D.,, Hawkins, G. D., Chambers, C. C., Giesen, D. J., Winget, P. Cramer, C. J., and Truhlar, D. G. Minnesota solvation database (mnsol) version 2012. 2020.
[27] Martínez, I.,, Andrade, R., Birgin, E. G., and Martínez, J. M. Packmol: A package for building initial configurations for molecular dynamics simulations. Journal of computational chemistry, 30(13):2157-2164, 2009.
[28] Megyes, T., Bálint, S., Peter, E., Grósz, T., Bakó, I., Krienke, H., and Bellissent-Funel, M.-C. Solution structure of nano3 in water: Diffraction and molecular dynamics simulation study. The Journal of Physical Chemistry B, 13((1)):00444000 4064, 2009.
[29] Mobley, D. L. and Guthrie, J. P. Freesolv: a database of experimental and calculated hydration free energies, with input files. Journal of computer-aides molecular design, 28(7)7111720, 2014.
[30] Moine, E., Privat, R., Sirjean, B., and Jaubert, J.-N. Estimation of solvation quantities from experimental thermodynamic data: Development of the comprehensive compsol databank for pure and mixed solutes. Journal of Physical and Chemical Reference Data, 46(3):033102, 2017.
[31] Ni, Y., Feng, S., Ma, W.-Y Ma, Z.-M., and Lan, Y. Sliced demoising: A physics-informed molecular pre-training method. arXiv preprint arXiv:2311.02122. 2023.
[32] Pathak, Y., Laghuvarapu, S., Mchta, S., and Priyakumar, U. D. Chemically interpretable graph interaction network for prediction of pharmacokinetic properties of drug-like molecules. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 873-880, 2020.
[33] Ryu, J. Y., Kim, H. U., and Lee, S. Y. Deep learning improves prediction of dlrug-druu and drug- food interactions. Proceedings of the national academy of sciences, 1 115(18))E4304-E4) 2018.
[34] Schütt, K., Kindermans, P.-.. Sauceda Felix, H. E.,, Chmicla, S., Tkatchenko, A.. and Müller, K.- R. Schnet: A continuous-filte convolutional neural network for modeling quantum interactions. Advances neural information processing systems, 30, 2017.
[35] Silverman, R. and Holladay, M. W. The organic chemistry of drug design and drug action. Academic press, 2014.
12

---

[36] Stärk, H., Beaini, D., Corso, G., Tossou, P., Dallago, C., Günnemann, S., and Liò, P. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pp. (0479-205022 PMLR, 2022.
[37] Velez-Arce, A., Huang, K., Li, M. M., Lin, X., Gao, W., Fu, T., Kellis, M., Pentelute, B. L., and Zitmik, M. Tdc-2: Multimodal foundation for therapeutic science. bioRxiv, pp. 2024-06, 2024.
[38] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. Graph attention networks. arXiv preprint arXiv:1710.1090) 2017.
[39] Vermeire, F. H. and Green, W. Transfer learning for solvation free energies: From quantum chemistry to experiments. Chemical Engineering Journal, 418:129307, 2021.
[40] Vinyals, O., Bengio. S., and Kudlur, M. Order matters: Sequence to sequence for sets. arXiv preprint arXiv: 151 1.06991, 2015.
[41] Wang. Y., Min, Y., Chen, X., and Wu, J. Multi-view graph contrastive representation learning for drug-drug interaction prediction. In Proceedings of the Web Conference 2021, pp. 2921-2933, 2021.
[42] Wu, F. and Li, S. Z. Diffmd: a geometric diffusion model for molecular dynamics simulations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 5321-532222 2023.
[43] Xiong, Z., Wang, D., Liu, X., Zhong, F., Wan, X., Li, X., Li, Z., Luo, X., Chen, K., Jiang, H., al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 63(16):8749 04488799-760, 2019.
[44] Xu, B., Lu, Y., Li, C., Yue, L.., Wang, X., Hao, N., Fu, T., and Chen, J. Smiles-mamba: Chemical mamba foundation models for drug admet prediction. arXiv preprint wrXiv22408.05690 2024.
[45] Xu, K., Hu, W., Leskovec, J., and Jegelka, How powerful are graph neural networks? arXiv preprint arXiv: 00826, 2018
[46] Zaidi, S., Schaarschmidt, M., Martens, J., Kim, H., Teh, Y. W., Sanchez-Gonzalez, A., Battaglia, P., Pascanu, R., and Godwin, J. Pre-training via denoising for molecular property prediction. arXiv preprini 2tti::222600010 2022.
[47] Zhang, J., Zhang, H., Wu, T., Wang. Q., and Van Der Spoel, D. Comparison of implicit and explicit solvent models for the calculation of solvation free energy in organic solvents. Journal of chemical theory and computation, 13(3):1034-1000
2017.
[48] Zhang, W., Chen, Y., Liu, F., Luo, F., Tian, G., and Li, X. Predicting potential drug-drug inter- actions by integrating chemical, biological, phenotypic and network data. BMC bioinformatics, 18(1):1-12, 2017.
[49] Zitnik, M., Sosic, R., and Leskovec, Biosnap datasets: Stanford biomedical network dataset collection. Note: http::/ssnap. stanford. edu/biodata Cited by, 5(1), 2018.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Yes, abstract and introduction accurately reflect the paper's contributions and
scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification: In Section 5.3, we have discussed that our approach generates the configuration with collapsed molecules.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the source code the external URL along with the implementation details in the paper.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide access the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
iow.eeeSuuumissiooooooooiyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//ipss.cc/publi//guue
Codeeuumiiii wettsbbmmssiiioi ssisssolicyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided a detailed evaluation protocol and hyperparameters for model training in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provide statistical error for each table.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided computing resources in Appendix D.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://nerrpp:: http://wwwa.accoooo/oo///c/cccca
Answer: [Yes]
Justification:
We follow the NeurIPS code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Since our model is about science application, we believe there are no potential societal impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
18

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have adequately cited the relevant works and data sources.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo//aaa hom/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our source code is well documented in an external URL, and also describes the implementation details in the Appendix.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
19

---

Justification:
This paper does not involve crowd sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowd sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
We only use LLMs for editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy htte:s//Deerrrppcccc///fferereep for what should or should not be described.
20

---

Supplementary Material for 3D Interaction Geometric Pre-training for Molecular Relational Learning
Molecular Relational Learning
22
Datasets
22
B. Pre-Training Datasets
22
Downstream Task Datasets
23
Baselines Setup
24
Implementation Details
25
D.1
Evaluation Protocol
25
D.2
Model architecture
................................
26
D.3
Model training
26
Additional Experimental Results
26
E.
Molecular Property Prediction Pre-training with Scale Datasets
26
Extrapolation Molecular Interaction Task
26
Ablation Studies
28
4
Further Virtual Interaction Environment Analysis
28
3D Encoder Pre-training Approaches
29
Pseudocode
30
21

---

A Molecular Relational Learning
In this section, we provide further clarification on molecular relational learning by contrasting with conventional molecular property prediction tasks. As illustrated in Figure (a), conventional molecular property prediction focuses on learning the properties of a single molecule. Models like GVaphMVP, 3D Infomax, and MoleculeSDE utilize the 3D information of individual molecules during pre-training to improve performance downstream tasks aimed at predicting single molecular properties.
In contrast, as shown in Figure 5 (b). molecular relational learning focuses on learning the properties of molecules after their interactions. Our pre-training approach trains both encoders simultaneously to learn information from the virtual environment ger. What sets our approach apart from traditional molecular pretraining is its specific tailoring to our Molecular Relational Learning strategy. This allows the two encoders to learn how paired molecules interact in 3D space, which essential for various downstream tasks in Molecular Relational Learning.
[a] Molecule Property Prediction
Pre-training Stage
Downstream Tasks [Fing-tuning]
Single Molecule Properties Tostchly Homo tumo Gap
20 Molecale (Rzo)
3D Molcaule (gao)
Molecular Relational Learning
Pre trainriinng Stage
Downstream Tasks (Fine-tuning)
sio
Malecaler Interactions
Sollve-S Interaction Drug Interaction
20 Paired Molecules
30 Vituaal Environment (p..)
20 Pained Molecules
Figure 5: Difference between the conventional pre-training strategy for (a) molecular property prediction and our (b) molecular relational learning.
B Datasets
B.1
Pre-Training Datasets
We utilize three distinct datasets, i.e., Chromophore, CombiSoly, and DDI, pre-train 3DMRL for each downstream task as described in Section Specifically, we use the Chromophore dataset for downstream tasks involving the optical properties of chromophores, the CombiSolv dataset for tasks related to the solvation free energy of solutes, and the DDI dataset, which we created for the drug-drug interaction task.
The Chromophore dataset 16
consists of 20,236 combinations derived from 6,815 chro- mophores and 1,336 solvents, provided in SMILES string format. For pre-training, we initially convert chromophores and solvents into their respective 3D structures via rdkit, resulting in 6,524 3D structures for chromophores and 1,255 for solvents. These 6,524 unique chromophores are then randomly paired with the 1,255 solvents to generate a sufficient number of pairs. Out of the possible 8, 117,8260 chromophore-sollent combinations, we randomly sample 1%, which corresponds to 81,876 pairs, for pre-training.
The CombiSoly dataset [39 contains 10, 145 combinations derived from 1,368 solutes and 291 solvents, provided SMILES string format. Similar to our approach with the Chromophore dataset, we first convert solutes and solvents into their corresponding 3D structures, yiclding
22

---

1,368 3D structures for solutes and 290 for solvents. From the potential random combinations, we select 79,344 solute-solvent pairs, representing 20% of all possible pairs.
For the DDI dataset, we compile drug-drug pairs from the ZhangDDI 48), ChChMiner 449, and DeepDDI [33] datasets. From total of 235,547 positive pairs, we randomly sample 40% (i.e., 94,218 pairs) for use as the pre-training dataset. While chromophores and solutes act as the larger molecule g in molecular interaction tasks, in the DDI dataset, we designate the drug with the larger radius as the larger molecule.
B.2 Downstream Task Datasets
Molecular Interaction Prediction. For the molecular interaction prediction task, we transform the SMILES strings into graph structures using the CIGIN implementation available on GitHub Regarding the datasets related to solvation free energies, such as MNSol, FreeSolv, CompSol,
Abraham, and CombiSolv, we utilize SMILES- -baseedddddd--ee datasets from previous studies [39]. Following previous work [21] we specifically filter the data to include only solvation free energies measured at temperatures of 298 K (+: 2) and exclude any data involving ionic liquids and ionic solutes [39].
The Chromophore dataset [6 consists of 20,236 combinations derived from 6,815 chro- mophores and 1,336 solvents, provided in SMILES string format. This dataset includes optical properties sourced from scientific publications, with unreliable experimental results being ex- cluded after thorough examination of absorption and emission spectra. In our work, we assess model performance by predicting three key properties: maximum absorption wavelength (Ab- sorption), maximum emission wavelength (Emission), and excited state lifetime (Lifetime), which are crucial for designing chromophores for specific applications. To ensure the integrity of each dataset, we remove any NaN values that were not reported in the original publications. Additionally, following previous work [2T]. for the Lifetime data, we apply log normalization to the target values to mitigate skewness the dataset, thereby enhancing training stability.
The MNSol dataset (26 features 3,037 experimentally measured free energies of solvation or transfer for 790 distinct solutes and 92 solvents. For our study, focus on 2,275 pairs comprising 372 unique solutes and 86 solvents, in alignment with prior research [39].
The FreeSolv dataset 29 offers 643 hydration free energy values, both experimental and calcu- lated, for small molecules in water. In research, utilize 560 experimental measurements, consistent with the dataset selection criteria from previous studies [39
The CompSol dataset (30] has been designed to illustrate the impact of hydrogen- association effects on solvation energies. For our study, we analyze 3,548 solute-solvent pairs, encompassing 442 distinct solutes and 259 solvents, in accordance with prior research parameters
[39..
The Abraham dataset 13]
curated by the Abraham research group at University College London, provides extensive data on solvation. For this study, we focus 6,091 solute-solvent combinations, comprising 1,038 distinct solutes and 122 solvents, as outlined in previous research [39].
The CombiSoly dataset 33 integrates the data from MNSol, FreeSolv, CompSol, and Abraham, encompassing a total of 10,145 145 solute-solvent combinations. This dataset features 1,368 unique solutes and 291 distinct solvents.
Drug-Drug Interaction (DDI) Prediction. In the drug-drug interaction prediction task, we utilize the positive drug pairs provided in the MIRACLE GitHub repository which excludes data instances that cannot be represented as graphs from SMILES strings. To create negative samples, we generate a corresponding set by sampling from the complement of the positive drug pairs. This approach is applied to both datasets. Additionally, for the classification task, we adhere to the graph conversion process outlined by MIRACLE [41)
The ZhangDDI dataset 48 includes data 548 drugs and 48,548 pairwise interactions, along with various types of similarity information pertaining to these drug pairs.
http:://ggtaabb..camaaaaaaaaaa/t
23

---

Table 4: Statistics of datasets. G and G are defined Section
5.1
Task
Dataset
g²
ga
# #g
# #
# Pairs
Absorption
Chromophore
Solvent
6,416
725
17,276
Chromophore
Emission Lifetime
Chromophore
Solvent
6,412
1,021
18,141
Chromophore
Solvent
2.755
247
6,960
Molecular
MNSol
Solute
Solvent
372
86
2.275
Interaction
FreeSolv
Solute
Solvent
560
1
CompSol
Solute
Solvent
442
259
3,548
Abraham
Solute
Solvent
1,038
122
6,091
CombiSoly
Solute
Solvent
1,495
326
10,145
Drug-Drug
ZhangDDi
Small-molecule Drug
Small-molecule Drug
544
544
40,255
Interaction
ChChMiner
Small-molecule Drug
Small-molecule Drug
949
949
21,082
The ChChMiner dataset 49 comprises 1,322 drugs and 48,514 annotated DDIs, sourced from drug labels and scientific literature.
Despite the ChChMiner dataset containing significantly higher number of drug instances compared to the Z dataset, the number of labeled DDIs is nearly equivalent. This suggests that the ChChMiner dataset exhibits much sparser network of relationships between drugs.
C Baselines Setup
To validate the effectiveness of 3DMRL, primarily evaluate molecular relational learning model architectures trained from scratch for downstream tasks, as well as the same models that are first pre-trained with 3DMRL and then fine-tuned for various downstream tasks. We include the following molecular relational learning model architectures:
MPNN (Message Passing Neural Networks) I was originally proposed predict the various chemical properties of a single molecule. For molecular relational learning tasks, we indepen- dently encode each molecule a pair using MPNN and then concatenate their representations.
To apply 3DMRL for MPNN, we first obtain the atom representation matrices E* and E² using J and 2 which MPNNs. Then, H., which considers the interaction between two molecules in Section 3.2 That is, we obtain graph-level embeddings ZnD and z via E' and E² with Set2set readout function. Following contrastive learning is done with Zin and z and the edge representations 2D and and initial atom representations for relative geometry X is obtained through E and E². One can simply alternate H and H2 in Section 4 E and E².
directly use E' and E instead of the H² and
AttentiveFP 43 was also initially proposed predict various chemical properties of individual molecules by employing graph attention mechanism to gather more information from relevant molecular datasets. For molecular relational learning tasks, we independently encode each molecule in pair using MPNN and then concatenate their representations.
More specifically, AttentiveFP first obtain atom representation matrices H' and H² using Sh and fib which consist of GAT and GRU layers. Then, the model obtain initial molecule representation Z/p and Z2p which are further enhanced by considering other molecules batch through GAT layers. After passing multiple GAT layers, the model obtain final molecule representations 2/p and ž In our framework, contrastive learning is done with zip and zi
umn http:///ddacaoooooimmmmm/aaaatta
v2012.
zip
24

---

and the edge representations k.l e2D and and initial atom representations for relative geometry X is obtained through H and H².
CIGIN (Chemically Interpretable Graph Interaction Network) [2] proposes to model the interac- tion between the molecules through dot product between atoms in paired molecules. By doing so, they successfully predict the solubility of drug molecules. We provide detailed descriptions on how apply 3DMRL for CIGIN in Section 4
CGIB (Conditional Graph Information Bottleneck) and CGIB cont (Conditional Graph Information Bottleneck with Contrastive Learning) [21] aim to enhance generalization in molecular relational learning by identifying the core substructure of molecules during chemical reactions, based on the information bottleneck theory. While CIGIN is limited to predicting drug solubility, CGIB and CGIB cont extend molecular relational learning to predict the optical properties of chromophores in various solvents, molecule solubility various solvents, and drug-drug interactions.
CGIB and CGIB model architectures are highly similar to CIGIN, but they have another branch named compress module, which aims inject noise to the atoms that are not important during the model. Specifically, they obtain T that is node representation matrix with noise, and obtain Z0de from the noise injected matrix along with zg: and Zga which are obtained from H and H2, respectively. To apply 3DMRL for CGIB, we pre-train the model without noise injection module, thereby using H' H², zg,, and zga in CGIB as H', H² zyp= and z in Section 4 After pre-training staget, all the modules including noise injection module is trained for the downstream tasks.
In addition to the model architectures, we also compare the recent state-of-the-an molecular pre- training methods based on CIGIN architecture. Since molecular pre-training methods are specifically designed for a single molecule, we pre-train each molecule encoder in CIGIN architecture and adopted the pre-trained weights for molecular relational learning downstream tasks. In Section S we include following molecular pre-training approaches:
No pre-training does not involve pertaining process and fine-tune the model using labeled data
increase the mutual information between 2D and 3D molecular representations using contrastive learning
3D Infomax
[3
GraphMVP 24 incorporates a generative pre-training framework in addition to contrastive
learning
MoleculeSDE 25 designs denoising framework to capture the 3D geometric distribution of molecules, thereby revealing the relationship between the score function and the molecular force field.
To apply these approaches for MRL, we first pre-train the each encoder f&b and fip in Section 3.2 with the above approaches. Then, the pre-trained encoders f2D and f2p are utilized to output the representations E and E², following the remaining pipeline of the model outlined in Section 3.2 That is, each molecule encoder f1p and fi implicitly possesses knowledge about the 3D structure of individual molecules, but not the complex interaction geometry between multiple molecules.
D Implementation Details
D.1 Evaluation Protocol
Following Pathak et al. 32 for the molecular interaction prediction task, we evaluate the models under 5-fold volss-valiotionn scheme. The dataset randomly split into 5 subsets and one of the subsets is used as the test set, while the remaining subsets are used to train the model. A subset of the test set is selected as the validation set for hyperparameter selection and early stopping. We repeat 5-fold cross cross-validation three times (i.e.. 15 runs in total) and report the accuracy and standard deviation of the repeats. For the DDI prediction task (2I,, we conduct experiments on two different -iitiiiibbtittionn scenarios, namely molecule split and scaffold split. For the molecule split, the performance is evaluated when the models are presented with new molecules not included in the training dataset. Specifically, let G denote the total set of molecules in the dataset. Given G, we split G into Gold and Gnews so that Gold contains the set of molecules that have been seen in the training phase, and Gnew contains the set of molecules that have not been seen in the training phase.
25

---

Then, the new split of dataset consists of D Derain-
E E Gold A G2 E Gola and Drest {(G', (²) E DI(g'= G Gnew) V(g GG
Gnow Age V E Gaa AG e Gnew)) }. We use subset of Drest as the validation set in inductive setting. In the scaffold split setting [4], like in the molecule split, molecules corresponding to scaffolds that were not seen during training will be used for testing. For both splits, we repeat 5 independent experiments with different random seeds on split data, and report the accuracy and the standard deviation of the repeats. In both scenarios, we split the data into training, validation, and test sets with ratio of 60/20/20%
D.2 Model architecture
For the 2D MRL model, following a previous work [3], we use 3-layer MPNNS M as our backbone molecule encoder to learn the representation of solute and solvent for the molecular interaction prediction, while we use a GIN [45] to encode both drugs for the drug-drug interaction prediction task [21]. We utilize hidden dimension of 56 for molecular interaction tasks and 300 for drug-drug interaction tasks, employing the ReLU activation function for both. For the 3D virtual environment encoder faD. utilize SchNet (4), which guarantees an 3E(3)-imvrrian representation of the environment. For both molecular interaction and drug-drug interaction tasks, we configure SchNet with 128 hidden channels, 128 filters, 6 interaction layers, and cutoff distance of 5.0.
D.3
Model training
For model optimization during Pre-training stage, we employ the Adam optimizer with an initial learning rate of 0.0005 for the chromophore task, 0.0001 for the solvation free energy task, and 0.0005 for the DDI tasks. The model is optimized over 100 epochs during pre-training.
In the downstream tasks, the learning rate reduced by factor of 10- after 20 epochs of no improvement in model performance in validation set, following the approach in a previous work (2), with the initial learning rate of 0.005 for the chromophore task, 0.001 for the solvation free energy task, and 0.0005 for the DDI tasks.
Computational resources. We perform all pre-training on a 40GB NVIDIA A6000 GPU, whereas all downstream tasks are executed on a 24GB NVIDIA GeForce RTX 3090 GPU.
Software configuration. Our model is implemented using Python 3.7, PyTorch 1.9.1, RD- Kit 2020.09.1, and
2.0.3.
E
Additional Experimental Results
E.1
Molecular Property Prediction Pre-training with Large-Scale Datasets
Although MPP pre-training approaches demonstrate unsatisfactory performance in Section 5 a positive aspect is their ability to leverage large-scale datasets containing both 2D and 3D molecular information. Consequently, we further explore whether utilizing a large-scale pre-training dataset can enhance MPP pre-training strategies in MRL tasks. To do so, we pre-train the encoders with each strategy with randomly sampled 50K molecules in GEOM dataset (2), which consists of 2D topological information and 3D geometric information, following the previous work (24). In 5 we observe that a large-scale pre-training dataset does not consistently result in performance improvements for MRL downstream tasks and can still cause negative transfer in various tasks. On the other hand, we note that MoleculeSTM benefits the most from the sarge-scale dataset among the strategies, likely due to the complexity of its denoising framework, which necessitates a large-scale dataset to learn the data distribution effectively. Nevertheless, it still exhibits negative transfer in the FreeSolve dataset and performs worse than 3DMRL, highlighting the need for a pre-training strategy specifically tailored to molecular relational learning.
E.2 Extrapolation in Molecular Interaction Task
The model's generalization ability in outtt--distiibution (OOD) datasets crucial for its application in real-world scientific discovery processes. To this end, we further conduct experiments molecular interaction tasks by assuming out-of-distribution scenarios, as shown in Table(G] Specifically, we split the dataset based on molecular structure, i.c., molecule split and scaffold split, similar to the approach
26

---

Table 5: Performance comparison of CIGIN model on molecular interaction tasks using different pre- pre-training strategies and pre- wre-training dataset (RMSE) (J). The blue color signifies a positive transfer between the pre-training task and the downstream task, whereas the orange color denotes a negative transfer between the pre-training task and the downstream task. Pre-training Dataset indicates the pre-training datasets used during pre-training.
Strategy
Pre training
Chromophore
MNSol
FreeSoly
CompSol
A Abraham
CombiSSav
Dataset
Absorption
Emission
Lifetime
training
19,80 sus
(0:3)
(1007)
0.567 (014)
0.884 suaze
0:412 juzz)
0.458 suar))
MPP (molecular property prediction) Pre-training
3D Infomax
GROM
MRL.
18 92
18:71 aus
24:55
26. y
705 ore
0. 0,580 yuars
0.585 HERES
U. 880
U.875 on
0:3221 50351)
0.319 29-014
0.4 gantt
U-126 303000
0.464 2030s
0.466 (4017)
GraphMPP
MRL
18.20
28.73
0.797
0.501
10010
0:301
01:18
18.341
000130
30007
suar
MoleculeSIDE
GEOM
MRL
18.3 auee:
18.72 auto
21.91 10:30s
24.77 10 48
0.835 muse
0.772 (0/03)
0.504 aunto
0.560 suaae
0.971 KuIz::
0.9009 VL
01308 VUIDA
0,200 JUi0:
01:000 (UOSM)
01420 SuzaO
0.050 aunza
01.44t) auar
MRL (molecular relational leoraing) Pre-tralning
3DMRL
MRL
18.00
24.21
0.729
0.528
0.839
0:277
0.377
0:435
used in the DDI task in Section 5 It is important to note that this scenario is significantly more challenging than the out-of-distribution DDI task in Section because it involves a regression task, which can also be viewed approaches generally benefit model performance in extrapolation tasks, with the exception one case, namely 3D Infomax for the Lifetime dataset. Among the pre-training approaches, 3DMRL performs the best, underscoring the extrapolation capability of 3DMRL.
extrapolation task. As shown T we observe that pre-training
Table 6: Performance comparison of the CIGIN model on extrapolation in molecular interaction tasks using different pre-training strategies (RMSE) ()).
Molecule Split
Scaffold Split
Strategy
Absorption
Emission
Lifetime
Absorption
Emission
Lifetime
No Pre-training
27.51 (0.4))
37.0 04 (mm)
1.205 (0.03))
59.55 (1.55)
60.11 (1:88)
1.221 (0033)
MPP (molecular property prediction) Pre-training
3D Infomax
27.38 (1.19)
36.95 66.98.m (1.2)
1.257 (0.00)
58.34 (1.89)
58.67 (1.00)
1.207 (0041)
GraphMVP
26.93 (1.89
36.5 51 (0.2)
1.201 (0.04)
59.27 (1.57)
57.67 (1.14
1.199 (0.024)
MoleculeSDE
27.26 (1.19)
36 48
1.135 (007))
57.75 (074)
58.74 (1.02)
1.214 (1010)
MRL (molecular relational learning) Pre-training
3DMRL
25.01 (1.51) 34.66 (0.39) 1.033 (0027)
57.58 (1.63)
57.53 (1.13)
1.178 (0.010)
27

---

E.3
Ablation Studies
We provide further ablation studies on molecular interaction task and drug-dru interaction task in Table 7 and 8 respectively.
Table 7: Further results from ablation studies molecular interaction tasks.
Chromophore
Strategy
MISol
FreeSolv
CompSol
Abraham
CombiSolv
Absorption
Emission
Lifetime
Only Glob.
18.30
24.70
0.739 mosis
0.531
0.874
0.301
0.376
0.458
Only Local
19.34 0.50
21.80 0.06)
0.801 morr
0.587 (8009)
1.181 (0.73)
0.330 (0028)
0.391 (0020))
0.166 (0021)
3DMRL
18.00
(009)
(0:14)
0.528 (0.019)
(0.000
0.371 (.08))
0.435
Table 8: Further results from ablation studies on drug-drug interaction tasks.
(a) Molecule Split
(b) Scaffold Split
Strategy
ZhangDDI
ChChMiner
ZhangDDI
ChChMiner
Only Glub.
73.09 msn))
m.59)
73.18 (019)
76.79 (1:1)
Only Local
73.45
75.93
73.41 (2.00
74.29 (1.70)
3DMRL
74.00 (0.72)
78 93 (0.59)
74.85 0.50
78.56 (1.03)
E,4 Further Virtual Interaction Environment Analysis
Sensitivity Analysis on DDI Datasets. In Table 9 we provide sensitivity analysis results in drug-drug interaction tasks.
Table 9: Sensitivity analysis on n in drug-drug interaction tasks.
Molecule Split
Scaffold Split
ZhangDDi ChChMiner
ZhangDD1
ChChMiner
2
73.77
77.15
74.76
77.01
= 10
73.96
79.12
74.36
77.76
== 20
73.94
78.75
74.03
77.64
Further Environment Analysis. While we propose assigning a single small molecule to each target in Section 4.1 also investigate the impact of varying the number of assigned small molecules per atom in the larger molecule. As illustrated in Figure 6 we observe decline in model performance as the number of small molecules per atom increases, given a fixed number of target atoms n. This suggests that modeling interactions between multiple small molecules and a single atom in larger molecule can degrade model performance. This is consistent with the scientific understanding that, although hydrogen bonding can occasionally allow multiple molecules to interact with a single atom simultaneously, steric and electronic hindrances frequently impede such interactions. Thus, we contend that our proposed virtual interaction geometry appropriately reflects the real-world physics molecular interactions.
Absorption
Emission
Lifetime
Small molecule per atoms
430
2
molecule 2
Small molocule
430
P2
3
4
RMSSE
RMSE
5
RMAS
Target atoms
Target atoms (n)
Target atoms (n)
Figure 6: Further environment analysis results.
Number Larger Molecules. In Section we initially constructed the virtual geometry in one-to-many manner (one larger molecule and many smaller molecules) to effectively mimic the
28

---

explicit solvent model in traditional MD simulations. However, in this section, we explore the many-to-many configurations between larger molecules and smaller molecules. In Table [0] we observe that the best performance was achieved when there was only one larger molecule. However, since the performance differences were not significant, we can conclude that our model is robust across various configurations.
Table 10: Model performance on different number of larger molecules.
# Larger molecules
Absorption
Emission
Lifetime
(Ours)
18.00
24.21
0.729
2
18.28
24.43 24.35
0.738
3
18.37
0,749
E,5 3D Encoder Pre-training Approaches
Since the core concept of our paper is to inject 3D information into 2D encoder, we choose baseline approaches that pre-train 2D molecular encoder with 3D information. In this section, compare the approaches that pre-train 3D molecular encoder with 3D information. To do so, since the elaborately first calculate the 3D
calculated 3D structure of the molecules is not available for our datasets, structure of the molecules in the dataset using RDKit ETKDG algorithm. However, some of the molecules in the dataset were able obtain 3D structures through RDKit ETKDG algorithm. We excluded these molecules from the experiment, and the results are shown below.
Before Conversion: Absorption 17,276 pairs, Emission 18,141 pairs, and Lifetime 6.960 pairs.
After Conversion: Absorption 16,756 pairs, Emission 17,525 pairs, and Lifetime 6,740 pairs.
Table 11:
Performance of various 3D encoder pre-training strategies in RMSE (4). Note thut these results are not directly comparable since some of the molecules the dataset were not able to obtain 3D structures through RDKit ETKDG algorithm.
Absorption
Emission
Lifetime
EMGG
18.62
24.06
0.753
SliDe
21.96
28.87
0.859
Frad
19.58
28.43
0.781
3DMRL
18.00
24.21
0.729
29

---

Pseudocode
In this section, we provide pseudocode of 3DMRL in Algorithm !
Algorithm Overall framework 3DMRL.
1: Input:
2D molecular topology graphs sap. sig
3D molecular geometric graphs gip. Sān
graph encoders fi fãb
3D Virtual Interaction Geometry Encoder f3D
Pre- Training Stage:
3:
For epoch in epochs:
4: záp zip. H H 2D MRL. ENCODER (giD. giD)
5: ZíD (7:0] 12D
6:
Pvr VIRTUAL INTERACTION GEOMETRY CONSTRUCTION (g3D= g5D)
ZnD (gv))
1s Virtual Geometry Encoding via SchNet
8:
Syk SE(3) INVARIANT GLoBaL GEOMETRY LEARNING (Zm,, Znn)
Cixal Li=1 SE(3) EQUIVARIANT LoCAL RELATIVE GEOMETRY LEARNING (93D- H H²)
10:
Epessrain Lylah + a Lincal
11:
Update f20. fiD. and fau
12:
Function 2D MRL ENCODER (QD GiD)
13:
E' fib (gt)).
E
fiD (92))
I sim E. D3)
where sim( is cosine similarity
E'
E
16:
H' (E
H
EE
17:
zn Set2set(H) H
ZiD
(et2set(HH))
18:
return ziu zio. H
H
Function VIRTUAL INTERACTION GEOMETRY CONSTRUCTION (gin. 2iD)
20:
Randomly select n atoms larger molecule dip
21:
Copy small molecule Să small molccules 2:1 ..... 93D ..... 93D
2,n
22:
Generate normalized random Gaussian noise vector
23:
Create new 3D coordinates for each smaller molecule gap r² R!
R R²
1 Broadcasting operation
24:
Create virtual interaction geometry fvr
RM, (R'
R
[ R 2
X
[X]
X
x²)
9xr [Xm: Rwy
25:
return 9vr
26:
Function SF(3) INVARIANT GLOBAL GEOMETRY LEARNING (ZID. Z3D)
27:
return Lylob
Nounh Nosech
log
sin/egp, i #30.4)/7 Macch cin 119,,, R3/T
+ log Nauch simtesn, Fintesn.-..9.....
oix(#3p M/
28:
Function SE(3) EQUIVARIANT LOCAL RELATIVE GEOMETRY LEARNING (g HH H?)
29:
For all edges (I:, in IiD 2,
30:
Fx.)
FA F 2i
Construct Orthogonal Frame
[[rk P
where R indicates the position of atoms k. k/ = Projection (T)
31:
1 Convert to 3E(3)-Invarial) Feature #/
32:
ex MLP(HH HF)) k.l
33:
ex.1 exp + epD
k.
34:
X (H² (
1 Broadcasting operation */
35:
1 = GNN(X, E), where E indicates all edges in 2,1
/ Obtain Edge Features +/
36:
f Σ, hu. Fu.i
Convert to (3E(3)-qqivvaiin) Feature
37:
return Lincal NZ DN² ll
fa1l
30


Reviews:
[{"review_id": "89viOVIEw0", "reviewer": "Reviewer_6exS", "summary": "This paper proposes 3DMRL, a pre-training framework that teaches 2 D molecular-relational models to encode 3 D interaction geometry by (a) constructing a randomized “virtual interaction environment” and (b) optimizing global contrastive and local SE(3)-equivariant losses. The expriments show consistent improvements (up to ≈25 % RMSE reduction) over five MRL backbones on eight datasets, plus better OOD generalization than prior 3 D pre-training methods.", "questions": "1. How often do generated configurations violate typical inter-molecular distance/angle distributions seen in MD trajectories?\n2. As mentioned above about the efficiency concern, can the authors provide the efficiency experiment?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1. The expriments are thorough on 40 downstream tasks with vairous ablation and OOD tests.\n2. Method pipeline and algorithm pseudocode are easy to follow, the methods are rational.\n3. The proposed component virtual interaction geometry is interesting.\n4. The paper is well-written, with sound motivaitons.\n\nWeakness\n1. Virtual environment lacks physical validation, where random Gaussian placement may create unrealistic interactions despite collision checks.\n2. Comparisons against 3 D encoders rely on ETKDG-generated conformers, weakening those baselines.\n3. The pre-training strategies are not novel, global contrastive loss similar to Infomax, thus is very incremental.\n4. The efficiency of the method might be a concern.", "was_revised": true, "final_justification": "I would recommend this paper as the authors addressed my questions.", "rebuttal": "Thank you for acknowledging our effort in extensive experiments of 40 downstream tasks and OOD generalization ability of 3DMRL! We're happy to address each of the weaknesses and questions that the reviewer raised.\n\n---\n## W1 and Q1\n\nWe acknowledge the reviewer's concern that random Gaussian placement, even with collision checks, might lead to unrealistic interactions and that our virtual environment lacks physical validation. As we detailed in Section 5.3, we're aware that some molecule collisions can still occur.\n\nOur approach prioritizes an efficient balance between computational complexity and performance, as demonstrated in Table 3. Regenerating the environment every time a collision occurs would increase computation time by over four times compared to 3DMRL. Furthermore, as shown in Figure 4, computational complexity significantly increases with the number of solvent molecules ($n$). Considering all possible realistic inter-atomic distances between molecules for every generated configuration would incur an even greater, potentially prohibitive, computational cost. Therefore, while we fully agree on the importance of a physically realistic environment, this was a necessary design choice given the computational constraints and the practicality of our proposed framework.\n\nRegarding the question of how often generated configurations violate typical inter-molecular distance/angle distributions seen in MD trajectories, we couldn't quantitatively assess this as our datasets do not include MD trajectories. However, we truly appreciate you suggesting this promising future work. If we could obtain MD trajectories of molecular interactions, we would first train a virtual interaction geometry generator to mimic these trajectories. This generator could then be used in our pre-training strategy instead of purely random generation, providing a more physically informed signal. Your insightful question highlights a crucial direction for further enhancing our model's realism and capabilities.\n\n---\n## W2\n\nWe agree with the reviewer that relying on ETKDG-generated conformers, while fast, introduces some inaccuracy. As discussed in Section 4.1, this inherent limitation of computationally generated conformers often leads to the underperformance of previous 3D encoder-based models when compared to methods like 3DMRL, as further evidenced in Appendix E.5.\n\nHowever, it's crucial to recognize that this is a practical challenge faced by most 3D molecular modeling approaches: obtaining experimentally accurate 3D conformers for the larger molecules is prohibitively expensive and often unfeasible. Therefore, any evaluation of 3D encoder-based models resorts to the datasets that consist of small molecules such as MD17 and QM9.\n\nIn contrast, our approach explicitly addresses this challenge through its two-step design. While our pre-training leverages the 3D structural information, our fine-tuning step relies primarily on the more robust and readily available 2D molecular topology. This design makes our model inherently less sensitive to the imperfections of computationally generated 3D conformers during the critical downstream task, allowing for more generalizable and practical applications.\n\n---\n## W3\n\nWe agree with the reviewer that our global contrastive loss shares similarities with Infomax. However, Infomax is designed exclusively for single-molecule property prediction and cannot learn interaction geometry. To overcome this, we developed virtual interaction geometry, an approach that, to our knowledge, is entirely novel. Beyond global contrastive loss, we also introduce novel local relative geometry learning. This aims to capture localized geometry, which is crucial for understanding how molecules interact in diverse environments. We want to emphasize that the true novelty of our work lies in these innovative methods for modeling molecular interactions.\n\n---\n## W4 and Q2\n\nWe understand your concerns about the efficiency of our method due to its inherent complexity. However, we'd like to clarify that certain aspects of our approach are highly efficient, while others represent known bottlenecks, primarily from components borrowed from prior work.\n\nAs described in Lines 189-191, our virtual environment construction is fast because it relies solely on efficient matrix operations for transitions and rotations. This real-time generation capability is a key strength. Furthermore, as shown in Table 3, 3DMRL maintains a similar computational time across various virtual interaction environment construction variants while consistently demonstrating superior performance on downstream tasks.\n\nOn the other hand, the pre-training phase does see an increase in complexity as the size of the virtual environment grows. Specifically, Figure 4(a) illustrates that computational time increases with the number of smaller molecules ($n$) attached to the target atoms. This is primarily because the $f_{3D}$​ encoder model, which we utilize, requires more computation as the input size expands.\n\nTo further validate this, we performed time complexity analyses on ablated models. Our findings, presented in the table below, show that the \"Only Glob.\" model, which requires 3D encoding of virtual geometry, demands significantly more training time than the \"Only Local\" model. This indicates that the 3D encoder is indeed the primary computational bottleneck of our approach.\n\n| | 3DMRL (Full Model) | Only Glob. | Only Local |\n| --- | --- | --- | --- |\n| Time (min/epoch) | 5.32 | 4.30 | 2.30 |\n\nIn conclusion, our proposed components, such as virtual geometry construction and local geometry learning, do not introduce additional computational cost. The main efficiency challenge arises from the 3D encoder, a component integrated from previous works."}, {"review_id": "y1nPP4L9tm", "reviewer": "Reviewer_1qFd", "summary": "This paper proposes a training strategy for molecular relational learning that involves a 2D and a 3D molecule encoder, where the interaction geometry of two molecules is provided with a computationally efficient approximation to enhance training with notions of the 3D space that molecules live in, while avoiding computationally expensive simulations to obtain the data. The work presents improved performance over baselines, ablation studies, and sensitivity analyses.", "questions": "- The rationale for the local geometry learning is not entirely clear to me. If I understand correctly, the target relative geometry $f^i_k$ was constructed randomly in the first place (compare definitions of $R^2$ and $R^1$). Aren’t you learning random noise here?\n- What is the added computational expense for constructing the environment and how does it scale with the involved parameters?\n- Would you generally recommend to set the number of small molecules $n$ to 5? Does that depend on factors, such as the size of the molecules?\n\nThe following questions are speculative and more out of curiosity, I don’t require them to be answered:\n\n- Is this method intended for solute-solvent interactions only or do you expect it to be applicable to eg molecules of similar size or more than two interaction partners?\n- Do you expect your method to scale to large biomolecules such as proteins? (both conceptually and computationally)\n- I am curious whether the random virtual environment might already provide sufficient information to model real-world interactions (as you argue in 4.1) and what the performance delta to an expensive QM method would be. How do you think (qualitatively) would your method using the virtual environment compare to using QM-derived data?", "limitations": "The authors discuss a limitation where molecules are placed on top of each other. I would recommend to also touch on the \"physicality\" of the virtual environment and the limitations of this approximation, as well as computational complexity of its construction (including scaling with various parameters).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This is a very solid contribution to a relevant problem with (mostly) clear description and notation, convincing experimental setups, and good improvement over existing baselines. I especially appreciated the given rationale of the virtual environment in 4.1, the ablation studies, and the sensitivity analysis. I have a few clarifying questions, but would recommend to accept as this work provides an easily applicable improvement that equips molecular prediction models with general 3D geometric information.", "was_revised": true, "final_justification": "I am keeping my positive score, although the author discussion was more confusing than clarifying. I was under the impression that the algorithm can be used without ground truth data obtained from conformer generators like ETKDG and use only a simple synthetic interaction environment. This reduces the value of the contribution a bit in my eyes. The more important it is to then please provide quantification of how much complexity your algorithm adds compared to others, either with explicit runtime comparisons or with a complexity statement (Big O). Also, there was no mention of denoising in the method description. If your model is phrased within a denoising framework, please mention that in the revision. I also recommend to clarify the use of the word \"geometry\" as it seems to be used in multiple ways.", "rebuttal": "We appreciate your strong endorsement of our research! It's rewarding to know that our virtual interaction environment construction, ablation studies, and sensitivity analyses resonated positively with you.\n\n---\n## Q1\n\nAs detailed in Lines 189-191, our approach generates the target geometry anew in each epoch. Crucially, we utilize random Gaussian noise directly without any training on it. While learning this random noise could indeed be a promising research direction, it presents a significant challenge: we lack the ground truth geometry that would typically be obtained through methods like MD simulation. Without such a signal, the noise could easily degenerate to a zero vector, as our model is designed to predict this noise, making a zero vector an \"easy target\" for the local geometry learning described in Section 4.2.2.\n\nHowever, if we could obtain MD trajectories of molecular interactions, we could first train a virtual interaction geometry generator to mimic these trajectories. This generator could then be used during our pre-training strategy instead of purely random generation, providing a more informed signal. We're truly grateful for your insightful question; it highlights a crucial direction for further enhancing our model's realism and capabilities.\n\n---\n## Q2\n\nAs described in our answer to W1, we do not involve any training of noise during environment construction. This key feature means it scales effectively with all involved parameters, irrespective of their quantity.\n\nWhile pre-training can encounter scaling issues as molecule and environment sizes increase, this concern is isolated to the training phase, not the environment construction itself.\n\n---\n## Q3\n\nYes, we generally recommend setting the number of small molecules, $n$, to 5. As illustrated in Figure 4, we found that this value strikes an optimal balance between computational cost and model performance.\n\nYou're right that the optimal $n$ could depend on factors like the size of the molecules. We could dynamically control $n$ by calculating the size ratio between the larger and smaller molecules. However, doing so would introduce an additional hyperparameter that users would then need to tune, potentially adding complexity to the model's application.\n\nHowever, we believe that exploring how to dynamically learn the environment based on experimental conditions or specific solvent types would be a very promising direction for future work. This would allow our model to adapt to a wider range of realistic scenarios, moving beyond static configurations to more accurately capture the complexities of molecular interactions in diverse experimental settings.\n\n---\n## Q4\n\nAs detailed in Section 5.1, our model accounts for diverse interaction types, including solute-solvent, chromophore-solvent, and drug-drug interactions. Notably, 3DMRL also handles drug-drug interactions, where interacting molecules are of similar size. This suggests our method is well-suited for modeling interactions between molecules of comparable dimensions.\n\nCurrently, our experiments are limited to interactions between two partners. While we believe extending our virtual environment to accommodate more than two distinct interacting molecules (by arranging them without overlap) is straightforward, our 2D encoder parts primarily model pairwise interactions using cross-attention. To effectively model higher-order interactions involving more than two types of molecules, we would need to develop new methods. \n\nThank you for suggesting us highly promising future work, which is crucial for a deeper understanding of complex molecular systems!\n\n---\n## Q5\n\nWe believe our method can conceptually be extended to large biomolecules like proteins.\nHowever, scaling to such large molecules presents practical challenges. Since proteins comprise a significantly greater number of atoms, the resulting virtual environments would become prohibitively large. To address this, we would need to develop specialized methods for constructing virtual environments using larger components than individual atoms, such as amino acid residues. This would be a crucial step for applying our approach to proteins and other large biomolecules.\n\n---\n## Q6\n\nYour question regarding the sufficiency of random virtual environments for modeling real-world interactions, and the performance delta compared to expensive QM methods, is insightful. As we discuss in Section 4.1, our goal was to mimic real-world molecular interactions through our virtual environment. The results in Table 3 demonstrate that our virtual environment strikes an effective balance between computational cost and the realism of the generated environment.\n\nOn the other hand, we expect that integrating our method with QM-derived data would lead to significant performance improvements. For instance, instead of purely random generation, we could use MD snapshots as initial configurations for our virtual environments, providing a more physically accurate starting point. Furthermore, leveraging entire MD trajectories as pre-training data would allow our model to learn from much richer and dynamic interaction landscapes. While these QM/MD-driven approaches inherently involve a higher computational cost, we believe they offer a path to even more realistically model interaction environments, thereby substantially enhancing our method's predictive power.\n\n---\n## Limitations\n\nThank you for suggesting additional limitations of our work. We will make sure to incorporate the 1) physicality of the virtual environment and 2) computational complexity as additional limitations.\n\n---\n## Paper Formatting Concerns\n\nThank you for your detailed suggestions! We will ensure that all suggestions are included in our camera-ready version."}, {"review_id": "je7kif8o8G", "reviewer": "Reviewer_8DvQ", "summary": "The paper introduces 3DMRL, a novel 3D geometric pre-training strategy for Molecular Relational Learning (MRL). Traditional MRL approaches are limited to 2D topological structures due to the high cost of acquiring 3D interaction geometry. 3DMRL overcomes this by constructing a 3D virtual interaction environment, allowing 2D MRL models to learn global and local 3D geometric information without expensive quantum mechanical calculations.", "questions": "1. Incorporating molecular dynamics trajectories to simulate dynamic interactions beyond static configurations may be virtual environment realism.", "limitations": "Yes, In Section 5.3.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nProposes a cost-effective 3D virtual environment simulating real molecular interactions by arranging small molecules around a larger molecule, replacing expensive quantum calculations.\n\nDual Pre-training Strategies: SE(3)-Invariant Global Learning and SE(3)-Equivariant Local Learning.\n\nExtensive experiments (15 runs per model), ablation studies, and sensitivity analyses validate the approach's robustness and generalizability.\n\nWeakness:\n\nRelies on RDKit-generated 3D conformations, which may introduce errors for molecules without experimental structures.\n\nLacks formal mathematical proof of SE(3) equivariance's necessity, relying primarily on empirical results.", "was_revised": true, "final_justification": "I retained my score.", "rebuttal": "Thank you for acknowledging our main contributions to the paper: replacing expensive quantum calculations with cost-effective 3D virtual environment simulation!\n\n---\n## W1\n\nWe agree with the reviewer that relying on ETKDG-generated conformers, while computationally efficient, does introduce some inaccuracy. As you rightly noted, using experimental structures could indeed reduce this error; however, their acquisition comes with substantial expense and often limits the scalability of such methods.\n\nObtaining experimental 3D structures for molecules, especially for a large and diverse set, is a significant challenge. Techniques like X-ray crystallography or NMR spectroscopy are highly resource-intensive, requiring specialized equipment, significant time, and often large quantities of purified samples. Many molecules are difficult to crystallize or are unstable in conditions required for these experiments. This makes it impractical to generate experimental 3D structures for the vast number of molecules needed for large-scale machine learning model training and evaluation.\n\nA core strength of our work, as you pointed out, is our aim to replace costly operations in traditional QM approaches with AI, thereby reducing overall experimental costs. Introducing the requirement for expensive experimental structures would, unfortunately, create another bottleneck, thereby limiting the practical applicability of our approach to a broader range of molecules and scenarios.\n\nNevertheless, we concur that incorporating experimental structures, despite the cost and difficulty, could further enhance our model's accuracy. This remains a valuable direction for future work, particularly in contexts where such high-fidelity data is available.\n\n---\n## W2\n\nWe appreciate the reviewer's comment regarding the perceived lack of a formal mathematical proof for SE(3) equivariance's necessity. We want to clarify that the necessity of SE(3) equivariance in molecular modeling isn't about deriving a mathematical theorem, but rather about aligning our models with a fundamental and inherent property of the physical world.\n\nSpecifically, molecules exist in 3D space, and their intrinsic characteristics—such as energy, bond lengths, or interaction forces—are fundamentally independent of their arbitrary position or orientation. If we rotate or translate a molecule in space, its physical nature remains unchanged. For a computational model to accurately capture these true molecular properties, its internal representations and final predictions must transform predictably (or remain invariant, for scalar properties) when the input molecule undergoes such rigid transformations. This is precisely what SE(3) equivariance mathematically enforces and represents, as demonstrated in many previous works [1, 2].\n\n[1] UNKE, Oliver, et al. SE (3)-equivariant prediction of molecular wavefunctions and electronic densities. NeurIPS 2021.\n\n[2] HOOGEBOOM, Emiel, et al. Equivariant diffusion for molecule generation in 3d. ICML 2022.\n\n---\n## Q1\n\nYes, we fully agree that incorporating molecular dynamics (MD) trajectories into our pre-training data would significantly enhance the realism of our virtual environments and further improve our work. Beyond just enriching the environment, this would also open avenues for new pre-training strategies, such as predicting molecular movement rather than just relative geometry.\n\nWhile acknowledging that calculating extensive MD trajectories introduces a computational bottleneck (as we discussed in our response to W1), our immediate research interest is indeed focused on integrating this valuable MD data into our virtual environment framework.\nWe are truly grateful for your insightful suggestion, which highlights a crucial direction for further enhancing our model's realism and capabilities."}, {"review_id": "hejkYpHpFy", "reviewer": "Reviewer_x5n8", "summary": "The paper introduces 3DMRL, a novel pre-training framework for Molecular Relational Learning (MRL) that incorporates 3D geometric information into 2D models. It constructs a virtual interaction environment to simulate molecular interactions and uses two pre-training strategies: global geometry learning and local relative geometry learning. The approach significantly improves model performance, with up to 24.93% improvement in various molecular interaction tasks.", "questions": "See Weaknesses above.", "limitations": "No. The authors did not demonstrate the limitations of their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n1. Novel Contribution: This paper introduces an innovating method for incorporating 3D geometric information into MRL. And the concept of constructing a virtual interaction environment to simulate molecular interactions is an effective and interesting solution to overcome the high computational costs of traditional methods.\n\n2. Effective and rational framework: The use of two complementary pretraining strategies is a good approach for improving the representation of molecular interactions at both macro and micro levels.\n\n3. Comprehensive experimental validation: Extensive experiments across multiple real-world datasets and tasks, which demonstrate that the method consistently improves performance. And this manuscript also covers ablation study, efficiency study, and so on.\n\n4. This manuscript is well-written and logically organized.\n\n# Weaknesses\n\nIn my opinion, this manuscript has no significant flaws. The few shortcomings are: this manuscripts lacks of discussions of the proposed approach and theoretical justification.", "was_revised": true, "final_justification": "The paper proposes a novel and effective method for molecular representation learning by incorporating 3D geometric information through a virtual interaction environment, demonstrating strong performance across multiple tasks with comprehensive experiments. The authors addressed concerns regarding model design and validity in the rebuttal, further strengthening the empirical foundation. While the theoretical justification and in-depth discussion on 3D geometric modeling remain limited, the significant experimental results and methodological innovation outweigh this weakness. The paper is well-written and clearly presented. I recommend **acceptance**, with a suggestion to expand the discussion on design rationale and the implications of the geometric modeling in the final version.", "rebuttal": "Thank you for strongly championing our work! We are happy to further discuss the justification and other possible views for our method.\n\n---\n## Justification for Random Placement of Small Molecules\n\nRandomly placing solvent molecules around a solute is a well-established strategy in molecular simulations. For instance, protein–ligand docking protocols (e.g., Rosetta) often initialize ligands in random orientations relative to the protein before searching for binding modes. Similarly, Monte Carlo insertion methods like Widom’s test-particle approach randomly insert solvent molecules to explore solute–solvent configurations without bias. This stochastic initialization allows efficient sampling over a wide range of distances and orientations while remaining physically sound: in the limit of sufficient sampling, no unphysical configuration is favored, and the process mimics the early stages of solvation when solvent molecules approach from arbitrary directions.\n\nOur approach constructs a “virtual interaction environment” by generating many such random solute–solvent geometries. This approximates the diversity of encounters seen in explicit solvation, while avoiding the high cost of molecular dynamics. It also exposes the model to both favorable and unfavorable orientations, encouraging generalization across interaction types. The method is computationally efficient and theoretically grounded, supported by literature precedent.\n\n---\n## Local Relative Geometry as an Interaction Force\n\nWhile we define local relative geometry for learning fine-grained interactions between molecules, we can view local relative geometry as an interaction force between molecules.\nMore specifically, we define a local relative geometry using the normalized vector between a solute atom and its closest solvent atom. \n\nThis provides a physically motivated supervision signal rooted in classical intermolecular forces, many of which are central and act along the internuclear axis. For example, van der Waals interactions (described by the Lennard-Jones potential) exhibit repulsive or attractive forces directed along this axis. At short distances, repulsion dominates and aligns directly outward between nuclei.\n\nThis supervision scheme serves as a central-force approximation, consistent with classical force fields, and offers a lightweight surrogate for full force labels, which would require costly quantum chemistry or MD simulations. Notably, SchNet [1] demonstrated that even approximate force signals improve learning of molecular interactions. Our direction-based supervision enables the model to learn geometric features like hydrogen bond alignment or steric repulsion trajectories in an SE(3)-equivariant manner.\n\nSince solvent atoms are placed near specific solute atoms, the dominant interaction direction aligns with the interatomic vector, making it a reasonable proxy for the net force axis. Thus, the unit direction vector serves as a pseudo-force label, conveying the primary interaction axis and encouraging the model to encode directionality of intermolecular interactions.\n\n[1] Schütt, Kristof, et al. \"Schnet: A continuous-filter convolutional neural network for modeling quantum interactions.\" Advances in neural information processing systems 30 (2017).\n\n---\n## Limitations\n\nAlthough the limitations of our work are addressed in Section 5.3, we will add a separate section in the camera-ready version to specifically detail concerns regarding the physicality of our virtual environment"}]

Author rebuttal:

Meta-review: This paper proposes a 3D molecular relational learning framework with a novel 3D geometric pretraining strategy by incorporating a 3D virtual interaction environment, which overcomes the limitations traditional costly mechanisms. The loss is SE(3)-equivariant, and the effectiveness is significant, obtaining up to a 24.93% improvement across 40 tasks.

All the reviewers are existed about the paper with positive reviews. They all feel positive about the significance and novelty of the proposed method. The propose method is novel and shows consistent improvements on various settings such as different backbones and datasets. I agree with the reviewers, and consider that most of the concerns are on some technical and clarification issues, which have been well resolved in the rebuttal. I encourage the authors to incorporate all the discussions from the rebuttal into the final revision, which would further improve the paper.
Decision: Accept (spotlight)