Learning Over Molecular Conformer Ensembles: Datasets and Benchmarks
Authors: Yanqiao Zhu
Jeehyun Hwang
Keir Adams
Zhen Liu
Bozhao Nan
Brock Stenfors
Yuanqi Du
Jatin Chauhan
Olaf Wiest
Olexandr Isayev
Connor W. Coley
Yizhou Sun
Wei Wang
Venue: ICLR 2024 poster (2024)
Keywords: conformer ensembles
geometric learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We present the MoleculAR Conformer Ensemble Learning (MARCEL) benchmark that comprehensively evaluates the potential of learning on conformer ensembles across a diverse set of molecules, datasets, and models.

Abstract: Molecular Representation Learning (MRL) has proven impactful in numerous biochemical applications such as drug discovery and enzyme design. While Graph Neural Networks (GNNs) are effective at learning molecular representations from a 2D molecular graph or a single 3D structure, existing works often overlook the flexible nature of molecules, which continuously interconvert across conformations via chemical bond rotations and minor vibrational perturbations. To better account for molecular flexibility, some recent works formulate MRL as an ensemble learning problem, focusing on explicitly learning from a set of conformer structures. However, most of these studies have limited datasets, tasks, and models. In this work, we introduce the first MoleculAR Conformer Ensemble Learning (MARCEL) benchmark to thoroughly evaluate the potential of learning on con- former ensembles and suggest promising research directions. MARCEL includes four datasets covering diverse molecule- and reaction-level properties of chemically diverse molecules including organocatalysts and transition-metal catalysts, extending beyond the scope of common GNN benchmarks that are confined to drug-like molecules. In addition, we conduct a comprehensive empirical study, which benchmarks representative 1D, 2D, and 3D MRL models, along with two strategies that explicitly incorporate conformer ensembles into 3D models. Our findings reveal that direct learning from an accessible conformer space can improve performance on a variety of tasks and models.

Published as a conference paper at ICLR 2024
LEARNING OVER MOLECULAR CONFORMER ENSEM- BLES: DATASETS AND BENCHMARKS
Yanqiao Zhu Jeehyun Hwang x Keir Adams
Zhen Liu
Bozhao Nan
Brock Anton Stenfors $
Yuanqi Du E
Jatin Chauhan ~
Olaf Wiest $
Olexandr Isayev
Connor W. Coley
Yizhou Sun ~
Wei Wang
UCLA
MIT
III
CMU
Notre Dame
E
Cornell
Primary contact: yzhu@cs. ucla.
Project homepage: https: // hatp::///gthhll github. com/SXXDZ/MARCEL
ABSTRACT
Molecular Representation Learning (MRL) has proven impactful in numerous biochemical applications such drug discovery and enzyme design. While Graph Neural Networks (GNNs) are effective at learning molecular representations from 2D molecular graph or single 3D structure, existing works often overlook the flexible nature of molecules, which continuously interconvert across conforma- tions via chemical bond rotations and minor vibrational perturbations. To better account for molecular flexibility, some recent works formulate MRL semble learning problem. focusing on explicitly learning from a set of conformer structures. However, most of these studies have limited datasets, tasks, and models. In this work, we introduce the first MoleculAR Conformer Ensemble Learning (MARCEL) benchmark thoroughly evaluate the potential of learning on former ensembles and suggest promising research directions. MARCEL includes four datasets covering diverse molecule- and reaction-level properties of chemi- cally diverse molecules including organocatalysts and transition-men metall catalysts, extending beyond the scope of common GNN benchmarks that are confined drug-like molecules. In addition, we conduct comprehensive empirical study, which benchmarks representative ID, 2D, and 3D MRL models, along with two strategies that explicitly incorporate conformer ensembles into 3D models. Our findings reveal that direct learning from an accessible conformer space can improve performance variety of tasks and models.
INTRODUCTION
Recent years have seen the emergence Molecular Representation Learning (MRL) as a promising approach for modeling molecules with machine learning. In the typical formulation, MRL maps discrete molecular objects to continuous features a data-driven manner, encoding complex chemical structures into representation vectors that subsequently be utilized in different downstream tasks. In particular, MRL now underpins a variety of biochemical applications spanning molecular property prediction the design of novel drug candidates [1-3].
Traditional approaches often encode chemical compounds with fingerprints, such as extended- connectivity fingerprints [4, 5], which indicate the existence of certain substructures binary bits in fixed-length sequence. Such line-based representations are concise and efficient, but have limited ex- pressive power and have difficulty capturing 3D structural information such as bonding geometries and global shapes, which can be important for analyzing molecular properties and chemical reactiv- ity [6, 7]. Recently, Graph Neural Networks (GNNs) have become an increasingly popular method of learning molecular representations by treating molecules as graph-structured objects. Existing GNN models for MRL can be broadly classified into two categories: 2D topological models [8-11] and 3D geometric models [12-17]. 2D GNNs typically model the molecular connectivity graph with atoms as nodes and bonds as edges, learning representations of chemical environments by iteratively passing messages between neighboring atoms. Although powerful in the absence of structural information. 2D GNNs may fail to capture key conformational effects or stereochemical properties like chirality (18, 19], which is critical for modeling molecular interactions in areas such as
flat 2D

---

Published as a conference paper at ICLR 2024
Merko
Datasett
LDMM
ECC 001-11125- EX0-4000--2--00-0 30H0100-1110
1D string
20 topelagy graph
ilmo
MARCEL
Evalustion
MRL modells
Qtantiin
Clssorti
SD conformatio
Confeemer memile
Iobectind
Chetuical compoounda
Descriptora
Figure I: We present a MARCEL benchmark that comprehensively evaluates the potential of learning on conformer ensembles across diverse set of molecules, datasets, and models.
drug design chemical catalysis. Conversely, 3D GNNs designed model molecular conformers (confornations), which describe the structure of molecules in 3D space. Thus, these models have found widespread adoption for modeling electronic properties, predicting conformer energies and forces, and scoring interactions between ligands and proteins, amongst other applications.
In almost all applications, benchmarks, and demonstrations. 3D GNN models focus on encoding individual conformer structures. It is critical to recognize that in reality molecules are not rigid. static objects; rather, thermodynamically saaammcalll--rriisbbbbl rotations of chemical bonds, small vibrational motions, and dynamic intermolecular interactions cause molecules to continuously convert between different conformations [20]. As a consequence, many experimentally observable chemical properties conformers, For example, molecule
depend on the full distribution of hattmoddyaamiclllyaccssssss needs to be arranged into a particular pose to bind to a target protein, and this binding conformation changes depending often challenging to determine a priori the conformers that predominantly contribute to molecular properties without doing prohibitively expensive simulations. Therefore, a natural question arises: can we leverage the collective power of many different conformer structures lying on the local minima of the potential energy surface, also known as the conformer ensemble, to improve MRL models?
the dynamic interaction between the molecule and the target [21]. Also. it is
As shown by the empirical evidence from various studies, learning from an explicit conformer ensemble can prove be advantageous for many tasks, including property and energy prediction [22- 24]. key conformer pose identification [25], and RNA sequence design [26]. However, these studies have been mostly confined to small-scale datasets, limited set of tasks, and restricted set of model architectures. As result, it remains unclear (1) to what extent 2D GNNs can implicitly model molecular flexibility and (2) whether the explicil encoding of conformer ensembles can improve the performance of 3D models that traditionally encode only one single conformer.
In this paper, we present the first MoleculAR Conformer Ensemble Learning (MARCEL) benchmark. It covers diverse range of chemical space (Figure 1), which focuses four chemically-relese tasks for both molecules and reactions, with an emphasis on Boltzmann-averaged properties of conformer ensembles computed at the Pensity-Functionn of compounds with high-quality conformers, including organocatalysts and transition -etall catalysts, extending beyond the scope of conventional GNN benchmarks which are often restricted to drug-like molecules. Moreover, we implement a benchmark suite that enables extensive empirical studies across representative ID, 2D, and 3D models. We further explore the advantages of leveraging conformer ensembles through two straightforward strategies: (1) augmenting training samples by randomly selecting one conformer from the ensemble for each molecule and (2) applying explicit multi-instance ensemble learning layer, which aggregates individual conformer embeddings.
Theory (DFT) level. Our datasets encompass variety
Our experimental results confirm the potential effectiveness of incorporating conformer ensembles in MRL, highlighting the improvements over conventional single-con cogell-cnfoooation 3D networks. However, is important to understand the heterogeneity of outcomes based on different dataset characteristics, task objectives, and model choices. Our investigation yields three key findings: (1) Leveraging molecular conformers by incorporating explicit set encoders, as a part of conformer ensemble learning strategies, can improve single-conformet 3D MRL models performance. (2) Data augmentation through conformer sampling may offer potential benefits, evidenced by improved results in the BDE dataset, suggesting method to increase model robustness against imprecise structures. (3) Model selection for MRL depends on dataset sizes and tasks, with traditional ID fingerprints and 2D models preferred for smaller datasets and 3D models for larger reaction-focused tasks.
2

---

Published as a conference paper at ICLR 2024
2 PROBLEM FORMULATION
We represent a 2D molecular graph as a tuple G = 8, K.W. where V {1: 1/ is the node set with each node corresponding to atom, and & C x V is the edge set representing chemical bonds edges between nodes. Further, X E R x contains vector attributes for each node, and W Rd x|E] contains attributes for each edge. When modeling chemical reactions, we represent molccule-molccule complex as a pair of graphs G2). In this case, the conformation describes the combined structure of the interacting molecules. For given molecule molecular complex, assume that its geometry can be effectively characterized by representative set of discrete, sampled conformers from the thermodynamically accessible conformer distribution. Formally, this set be
denoted as C = {C,),= where C, E 1 represents one conformer structure in 3D space. In reality, the conformer distribution is continuous; C' in our study contains representative samples of the infinite set. Each conformer in the sampled ensemble associated with statistical weight given by
cxp
Agg
Pj
E, exp
which corresponds to its probability under experimental conditions. Here, e; is the energy of the conformer C, kB is the Boltzmann constant, and T is the temperature. Notably, p; is not prior information to the models analyzed in this benchmark. Rather, we use a discrete approximation of p; to compute the ground-true -ruth labels for our regression tasks.
DATASETS AND TASKS
MARCEL contains four small-to-large-sett datasets involving nine regression tasks with consider- ably diverse chemistry. Drugs-75K and Kraken focus on molecular properties, while EE and BDE focus on reaction-centrie properties. MARCEL includes molecules with high structural flexibility, evidenced by an average number of rotatable bonds exceeding 5. Table summarizes the datasets.
Drugs-75K is a subset of the GEOM-Drugs [27] dataset, which includes 75,099 molecules with at least 5 rotatable bonds. For each molecule, Auto3D [28] is used to generate and optimize the conformer ensembles and AIMNet-NSE [29] is used to calculate three important quantum chemical descriptors: ionization potential, electron affinity, and electronegativity [30]. Note that Auto3D and AIMNet-NSE achieve DFT-level accuracy but are much more efficient [21, 32].
lonization Potential (IP) is the minimum energy required to remove an electron from a neutral atom or molecule to form a positively charged ion (cation): IP Lection
Encutral
Electron Affinity (EA) denotes the energy change associated with the addition of an electron to neutral atom or molecule to form a negatively charged ion (anion): EA = Eneutral Enioon
Electronegativity (x) measures the tendency of an atom to attract a bonding pair of electrons:
OE àN
x
Ecation- Encutrals and Exnion are the electronic energy of the positively charged, neutral, and negatively charged molecules, respectively. E and N are the energy and the number of electrons, respectively.
The tasks are to predict the Voltzmann-averagge value of each property across the conformer ensemble (w) -cc Pats,, where y/ is conformer-specifie property. We given each C, and the goal to predict (V) km from the molecular graph G, a single conformer C; E C, or the set C.
Kraken [33] is a dataset of 1,552 monodentate organophosphorus (III) ligands along with their DFT-computed conformer ensembles. In this study, we consider four 3D ligand descriptors exhibiting significant variance among conformers: Sterimol Bs. Sterimol L., buried Sterimol Bs, and buried Sterimol L. These descriptors quantify the steric features each ligand in units of and are often employed for Quantitative prructure-Activiig Relationship (QSAR) modeling in catalyst design.
As in the Drugs-75K tasks, the goal is to predict the Boltzmann-averagee value of each property across the conformer ensemble from the molecular graph G, single conformer C₁ C, or the set C.
3

---

Published as a conference paper at ICLR 2024
Table 1: Statistics of the four datasets. bonds") are averaged per conformer.
The numbers of heavy atoms and rotatable bonds ("rot.
Datasel
# Molecules
# Conformers
# Heavy atoms
# Rot. bonds
# Targets
Atomic species
D55gs-75K
75,099
558,002
30.56
7.53
3
O.
H. S.
Kruken
1,552
21,287
23.70
9.05
4
Fe, Se, Sn,
Dataset
Reactions
# Conformers
# Heavy atoms
# Rot. bonds
# Targets
Atomic species
Pro-R: 14,807
872
59.32
18.57
H. C. N. O. Br. Rh
Pro-S: 13,999
Ligand: 73,834
29.62
6.99
H.C.N.O..RRINIICC. Cu, Pl.
BDE
5,915
Complex: 40,264
32.38
6.99
EE [34] is a dataset of 872 catalyst-substrate pairs involving 253 Rhodium (Rh)-bound atropisomeric catalysts derived from chiral bisphosphine, with 10 enamides as substrates. The dataset includes conformations of catalyst-substrate transition state complexes in two separate pro-S and pro-R configurations. The task is to predict the Enantiomeric Excess (EE) of the chemical reaction involving the substrate, defined as the absolute ratio between the concentration of each enantiomer in the product distribution. This dataset is generated with Q2MM, which automatically generates Transition State Force Fields (TSFFs) in order simulate the conformer ensembles of each prochiral transition state complex. EE can then be computed from the conformer ensembles by Boltzmann-averagie the activation energies for the competing transition states [34, 35].
Unlike properties in Drugs-75K and Kraken, EE depends on the conformer ensembles of each pro-R and pro-S complex. The goal is to predict EE from the graphs of the catalyst and substrate (Geat; Ganh), a conformer C(R) E ( and c.S)) E ((S) for each complex, or the ensembles (( and ((S)
BDE [36] is dataset containing 5,915 organometallic catalysts ML,L2 consisting of a metal center (M = Pd, Pt, Au, Ag, Cu, Ni) coordinated to two flexible organic ligands (L, and L2), each selected from a 91-membered ligand library. The data includes conformations of each unbound catalyst, as well as conformations of the catalyst when bound to ethylene and bromide after oxidative addition with vinyl bromide. Each catalyst has electronic binding energy, computed as the difference in the minimum energies of the bound-catalyst complex and unbound catalyst, following the DFF-optimization of their respective conformer ensembles. Although the binding energies are computed via DFT, the conformers provided for modeling are initially generated with Open Babel [37] followed by further geometry optimization, which ensures that the 3D structures are likely to be the global minimum energy conformers at the force field level [36]. Note that obtaining DFT- optimized conformers for BDE is not feasible given the sigmificant computational cost. Therefore, this realistically represents the setting in which precise conformer ensembles are unknown at inference.
The task is to predict the binding energy from the graphs of the unbound and bound catalyst, sampled conformers C (hunbound) Clunbound) and C((boual) E ((hound), or the ensembles Clunbound) and (houndd)
Dataset Preparation. We implement several preprocessing steps to ensure the quality and validity of our datasets and facilitate their integration into machine learning models.
Conformer deduplication. To eliminate redundant conformers in each ensemble C, we first align every pair of conformers using RDKit [38], accounting for symmetric atom permutations. Subsequently, we employ Butina clustering [39] based on the Root Mean Square Deviation (RMSD) values derived from conformer alignment. Within each cluster, we select the conformer with the lowest energy. Note that Boltzmann- -veeraged regression labels are computed before deduplication.
Selection of molecules. We focus on modeling flexible molecules, for which conformer ensemble learning may be relevant to capture their properties. Hence, we only retain molecules with more than 5 rotatable bonds. We also remove molecules with missing 3D geometries or 2D graphs.
4 BENCHMARKING MOLECULAR REPRESENTATION LEARNING MODELS
The representation of molecular data is crucial for applying machine learning models to problems in chemistry and biology. These representations typically include 1D strings, 2D topological graphs, and 3D geometric graphs. For a comprehensive benchmark for MRL models, our MARCEL includes diverse representative selection of models for each of the aforementioned molecular representations.
4

---

Published as a conference paper at ICLR 2024
In this section, provide overview of these models and describe how they are tailored to tasks. We also introduce two strategies of explicitly encoding conformer ensembles using 3D models.
4.1 ID MODELS
Our ID baselines include Random Forest [40] models operating molecular fingerprints [38, 41, 42]. Fingerprints convert a molecular graph into a bit array indicating the presence of chemical substructures and widely used for cheminformatics and QSAR modeling in the low-data regime. Additionally, we include Long Short-Term Memory (LSTM) [43] and Transformer [44] models, popular sequence-based neural network architectures, operating on SMILES strings. For the BDE and EE datasets, we concatenate the SMILES of each molecule or complex with a symbol and use single sequence encoder. Further details on model implementations can be found in Appendix B.
4.2 2D GRAPH NEURAL NETWORKS
We employ four widely-used GNN models as 2D baseline methods, including Graph Isomorphism Network (GIN) [45], GIN with Virtual Node (GIN-VN) [46], ChemProp [47], and GraphGPS [48]. GIN is a commonly-used model with strong representation ability. GIN-VN augments the vanilla GIN by incorporating a virtual node to aggregate the features of all nodes in the graph, thereby capturing global information more effectively. ChemProp is directed message passing GNN designed specifically for molecular property prediction. GraphGPS is a Transformer-like [44] model specifically tailored for graph-structured data, which is able capture long-range relationships.
Following OGB protocols [46], we employ a diverse set of atomic features such as aromaticity and hybridization for nodes, as well bond features like ring information for edges (Appendix B.2). For the EE and BDE datasets, we employ a two-tower architecture with two separate 2D GNN models: for EE, since both pro-S and pro-R complexes share the same 2D graph. we leverage two separate GNNs to encode the catalyst and substrate; for BDE, we also encode the unbound and bound catalysts separately. We then concatenate these together obtain the system-level embeddings.
4.3 3D GRAPH NEURAL NETWORKS
We include six representative 3D GNNs that encompass diverse modeling perspectives. SchNet [12], an F(3)-invariant network, models spatial interactions by encoding pairwise interatomic distance. DimeNet++ [13], another E(3)-invariant model, uses directional message passing that embeds angles between triplets of atoms order to enhance geometric expressivity. GemNet [14], an SE(3)-invariant model, utilizes unique attention mechanism and dihedral angles between four atoms to model atomic interactions. PaiNN [15], initially developed to predict tensorial properties and molecular spectra, incorporates rotational equivariance into its message passing framework. ClofNet [16], an SE(3)-equivariant model that improves the popular EGNN [49]. uses complete local frames for each atom, effectively capturing 3D atomistic structures while preserving invariance and equivariance. LEFTNet [17]. based on ClofNet, introduces Local Substructure Encoding (LSE) and Frame Transition Encoding (FTE) to enhance the model expressivity via scalarization and tensorization.
We use atom types as the sole atom features for the 3D models. For both training and inference on Drug-75K, Kraken, and EE datasets, all the single-conformer 3D models encode the lowest-enene conformer of each conformer ensemble, which has the largest Boltzmann weight and hence provides the strongest model. Since imprecise conformers are encoded for the BDE task, we use fixed, randomly sampled conformer for each unbound- and bound-catalyst during training and inference.
The 3D models also employ two-tower architecture for the EE and BDE datasets. Two separate 3D GNNs are used to encode representations for each pro-S and pro-R complex in EE, or for each catalyst and bound complex in BDE, which are then concatenated to form the final representations.
We note that although using the lowest-energy conformer will yield the strongest performance, this setting be unrealistic: it is often not possible identify the lowest energy conformer without searching the entire conformer space. The lowest energy conformer can also depend on the force field used for geometry optimization, which may neglect experimental conditions such as solvents.
5

---

Published as a conference paper at ICLR 2024
4.4 INCORPORATING CONFORMER ENSEMBLES INTO MOLECULAR REPRESENTATIONS
3D geometric models primarily focus on learning representations from individual 3D structures. Although some models preserve global symmetries such SE())-equie,riacce, these models do not learn representations that capture conformational flexibility which is caused by internal degrees of freedom such bond rotations. Here, we describe two straightforward strategies that model conformational flexibility by explicitly leveraging conformer ensembles.
4.4.1 STRATEGY 1: TRAINING- -IME DATA AUGMENTATION VIA CONFORMER SAMPLING
direct approach to modeling conformer flexibility is to simply enrich the training data by randomly sampling conformer from the ensemble during each training epoch. Formally, for given molecule and its conformer ensemble C, we randomly select a conformer with uniform probability p = 1/ie while using the same training label for each conformer. Note that during inference, the lowest-energy conformer is used to evaluate the model performance. This strategy aligns with learning represen- tations invariant to conformational changes, thus implicitly encoding the flexibility of molecular structures, and has been shown to be useful for learning chirality-sensitive 3D representations [19]. When conformer ensembles are available, the strategy computationally efficient it maintains the same complexity as the base 3D model. Unlike the other ensemble methods, this strategy can be used conformer ensembles are only available at training time. In Appendix C, we evaluate two alternative scenarios where conformer ensembles are also available during evaluation.
4.4.2 STRATEGY 2: ENSEMBLE LEARNING WITH EXPLICIT SET ENCODERS
The second strategy utilizes a set encoder to simultaneously encode the entire conformer ensemble at both training and inference time. Inspired by the strategy first employs 3D GNNs to generate individual conformer embeddings and then aggregates these embeddings using set encoder, illustrated in Figure 2.
learning framework [50-52], this
Formally, for each conformer C; E C, we obtain its corresponding embedding 2; -f(G, C:) E Rd. where is single-conformer 3D model and d is the embedding dimension. Note that the embedding z is a (3D) graph-level representation resulting from a pooling function over the node- level embeddings after message passing. To further aggregate these embeddings 2 == {2)) into single molecular representation, we consider the following three set encoders:
Mean pooling simply computes the mean of all the conformer embeddings:
MRAN
DeepSets [53] utilizes permutation-invarian function process set of inputs. first applies MultiLayer Perceptron (MLP) h to each conformer embedding and then aggregates the transformed embeddings using sum pooling followed by another MLP g:
DS 9 h(z))
This method retains more discernible information from individual embeddings compared to mean pooling at a cost two non-linear functions.
Self-attention [54] further computes a weighted sum of the embeddings, where the weights obtained by applying a softmax function to the dot product of the embeddings:
CC
CI
AFT
Σ
C..
where c; =
Σ
(ssh)(z))
a.j = cl (WNP(WW)))T)WWheee
i=1
Here, W E Bdxd is a learnable weight matrix. This approach can capture conformer interactions.
By employing these set encoders, conformer variations present in the ensemble. After obtaining the ensemble embeddings, we further apply linear projection head to generate the final prediction.
learn model that is more sensitive to the full range of

---

Published as a conference paper at ICLR 2024
Quazi um propecty
sn GNN emooder
Sis monler
Prodictio hoad
EE selectivity
Conformer
Binding, meergy
eusemuble
Conformer esubedkdings
Ensemble embedddings
Ensemble-level quautitiee
Figure 2: Conformer ensemble learning with explicit set encoders (Strategy 2). Individual conformer embeddings are first obtained via 3D GNN encoders. Then, a set encoder is employed to aggregate conformer embeddings. Finally, linear projection head is used to generate the prediction.
EXPERIMENTS
5.1 EXPERIMENTAL CONFIGURATIONS
Each dataset is partitioned randomly into three subsets: 70% for training, 10% for validation, and 20% for test. Each model is trained over 2,000 epochs using the Adam optimizer [55] with early stopping triggered if there no improvement on the training loss over 200 epochs. For all nine regression targets, experiments are repeated three times, and the results reported correspond to the model that performs best on the validation in terms of Mean Absolute Error (MAE).
The Boltzmann Boltzmann-averaged targets are computed over all available conformers. For ensemble learning models, we cap the number of encoded conformers per molecule to maximum of 20, which empirically improves training stability and leads to better performance. To ensure fair comparison, the hidden dimension size uniformly set to 128 for all models. Other settings mostly follow the original configurations described in the respective papers. We specify all hyperparameters and describe experimental environments in Appendix B.3.
5.2 RESULTS AND ANALYSIS
We summarize the performance of the ID, 2D, and 3D MRL models and the best results from ensemble learning strategies on 3D models in Table 2. Figure reports the performance changes in Mean Absolute Error (MAE) for each 3D model when applying the ensemble learning strategies. The raw performance data with standard deviation and the parameter size of each model can be found in Appendix D. summary, although performance varies across the datasets, tasks, models, the ensemble learning strategies improve upon 3D models that only encode conformer in 48 out of 54 experiments across tasks and 6 base models, demonstrating the effectiveness of conformer ensemble learning. Our analysis leads the following key observations.
Observation 1. The conformer ensemble learning strategy with explicit set encoders frequently yields improved performance.
Figure indicates that encoding conformer ensembles can substantially reduce test error, achieving improvements in 108 experiments across all 9 tasks, 6 base models, and 3 set encoders, most notably on the tasks in the smaller-sized Kraken dataset. This, however, does not always extend to larger datasets like Drugs-75K. We conjecture that for Drugs-75K, the computational burden of encoding all conformers in each ensemble alters the learning dynamics of the underlying model, making training more challenging. A similar finding was reported by Axelrod and K6mezzBoombaeell [23].
Among the three set encoders, DeepSets consistently demonstrates significant improvements in 42 out of 54 experiments across tasks and 6 base 3D models. We conjecture that this superior performance is due to its ability of effectively modeling set objects at relatively minor computational overhead of two non-linear transformations. On the other hand, the simple mean pooling approach loses discriminative power across the conformers in the ensemble, resulting in inferior performance. It is also noteworthy that the attention models exhibit mixed results compared to the vanilla 3D models, despite theoretically being the most powerful set encoders. This inconsistency might be attributable to the computational intricacy of the self-attention layer, which models the pairwise relationship among conformers in each ensemble and hence could require Future research should consider developing better neural architectures that are specifically designed more efficiently encode structural information from conformer ensembles.
sophisticated training strategies.
7

---

Published as a conference paper at ICLR 2024
Table 2: Performance of ID, 2D, and 3D baseline MRL models and the best results from ensemble learning strategies on 3D GNNs. The metric used is the Mean Absolute Error (MAE, 4). The bold indicates the rest-performing model, while underlined denotes the second-best.
Drugs-75K
Kraken
Category
Model
EE
BDE
IP
EA
BurB:
BurL
Random forest
0.4987
0.4747
0.2732
0.4760
0.4303
0.2758
0.1521
61.2963
3.0335
ID
LSTM
0,4788
0,4648
0.2505
0.4879
0,5142
0.2813
0.1924
64,0088
2.8279
Transformer
0.6617
0.5850
0.4073
0.9611
0.8389
0.4929
0.2781
62.0816
10.0771
GIN
0.4354
0.4
0.2260
0.3128
0.4003
0.1719
0.1200
62,3065
2.6368
GIN+VN
0.4361
0.4169
0.2267
0.3567
0.4344
0.2422
0.1741
62.3815
2.7417
2D
ChemProp 0.4595
0.4417
0.2441
0.4850
0,5452
0,3002
0.1948
61.0336
2.6616
GraphGPS
0.4351
0.4085
0.2212
0.3450
0.4363
0.2066
0.
61.6251
2,4827
SchNet
0.4394
0.4207
0.2243
0.3293
0.5458
0,2295
0.1861
17.7421
2.5488
DimeNet++
0.4441
0,4233
0.2436
0.3510
0.4174
0.2097
0.1526
14,6414
1.4503
GemNet 0.4069
0.3922
0.1970
0.2789
0.3754
0.1782
0.1635
18.0338
1.6530
3D
PaiNN
0,4505
0,4495
0.2324
0.3443
0,4471
0.2395
0,1673
20.2359
2.1261
ClofNet
0.4393
0.4251
0.2378
0.4873
0.6417
0.2884
0.2529
33.9473
2.6057
LEFTNet
0.4174
0.3964
0.2083
0.3072
0.4493
0.2176
0. 1416
19.7974
1.5328
SchNet
0.4452
0,4232
0.2243
0.2704
0.4322
0.2024
0.1443
14.2238
1.9737
DimeNet++
0.4126
0.3944
0.2267
0,2630
0,3468
0.1783
0.1585
12.0259
1.4741
Best Ensemble
GemNet
0.4066
0.3910
0.2027
0.2313
0.3386
0.1589
0.0947
11.6142
1.6059
Strategy
PaiNN
0,4466
0,4269
0.2294
0.2225
0,3619
0.1693
0.1324
13.5570
1.8744
ClofNet
0.4280
0.4033
0.2199
0.3228
0.4485
0.2178
0.1548
13.9647
2,0106
LEFTNet
0.4149
0.3953
0.2069
0.2644
0.3643
0.2017
0.1386
18.4189
1.5276
Observation 2. Sampling conformers at training time can improve performance, especially on imprecise conformer structures.
We observe that data augmentation improves performance on 34 experiments, especially on the challenging BDE dataset, where the other ensemble learning strategies often do not help. Note that the conformers in the BDE dataset originate from Open Babel, as opposed to the golden-standard DFT-level conformers present in other datasets. This suggests that training with randomly sampled conformers might offer robustness to noise in the imprecise structures. On other tasks, randomly sampling the conformers at each epoch may help the model learn an invariance to conformational changes, but does not always increase performance for all 3D models. This might be because the sampling probability is uniform across the entire conformer set, which does not respect the underlying Boltzmann weight of each conformer. In future work, it may be worthwhile to investigate whether more physics-informat sampling strategies could lead to more consistent performance gains.
Observation 3. No model consistently outperforms the rest, with substantial task dependencies.
The results Table 2 suggest that no single model outperforms the others across all tasks. Of the 1D models, LSTM outperforms Random Forest and Transformer models on Drugs-75K and BDE, demonstrating the effectiveness of SMILES- representations of molecules on large-scale datasets. For small datasets such as Kraken and EE, Random Forests outperform sequence models a lower computational cost, indicating that traditional models are superior in the low-data regime.
Amongst 2D models, GIN delivers the best performance on four tasks compared to all other models; GraphGPS also demonstrates strong performance on several tasks (Bs, L. and BurL). Surprisingly, the 2D models are also competitive with some 3D models on the large-scale Drugs-75K tasks. This is possibly due to the fact that the electronic properties in Drugs-75K are not as sensitive to conformational changes, thus explicitly modeling the structures in 3D may not be necessary. However, all 2D models perform worse as compared to the 3D models in the reaction datasets EE and BDE, indicating the important role of spatial interactions in determining reaction-related properties.
For 3D models, GemNet and LEFTNet excel in IP, EA, and X. The complexity of these two equivariant models may especially benefit from the large dataset size of Drugs-75K. For Kraken and the two reaction datasets, DimeNet++ - an invariant model achieves promising performance, suggesting that highly-comee complecc 3D models may be less useful for chemical applications with small-to -oll--uummmmm sized datasets. On EE, observe that 3D models remarkably outperform ID and 2D models, likely because enantiosclectivity depends on subtle spatial interactions. When predicting binding energies, using 3D models also leads to modest improvements.
8

---

Published as a conference paper at ICLR 2024
3D Sampling
Ensemble-Menn
Ensemble Tnsemble-DerpSens
CnsembbeeAAtcatti
SchNet
+
GemNet
PaINN
ClefNet
LEFTNet
#
-0.025
0.000
0.025 Tncreseed MAR
0.025 0.000
0.125 Introwed MAE
0.02
0.00
0.02 Increvsed MAR
Reduowl MAE
Rodhoe MAR
Reduod MAE
(a) Druge73K:
(b) Drigi-5K:; FA
(c) Drugs 75K: X
SchNet
+-meNet+++
GemNet
#
PaiNN
8O
CloiNet
LEFTNe
0.2
0.05 Encressed MAE
Rechaad MAE
Lacressedd MAB
Raducal MAR
Intrssni MAE
Reduadd MAE
(4) Kraken: 13,
(e) Krakm::
(E) Kraloem: Bur
SchNet
DimeNet 1-1
GomNet
do
PaiNN
4b
ClofNet
LEFTNNt
8+
Redured MAE
Excrensed MAE
Reduced MAE
Increased MAE
Reduced MAE
Encrowsed MAE
[g] Kraken: BurL
(b) EE
( BDE
Figure 3: Performance changes of four conformer ensemble learning strategies on the basis of six 3D graph models. Here, negative values (marked in hatch patterns denote reduced Mean Absolute Error (MAB), signifying a performance improvement due to the incorporation of conformer ensembles.
Overall, model performance varies substantially across tasks, within the same dataset, emphasiz- ing the diversity of the tasks in MARCEL. Generally. ID and 2D models perform well on scall-scale molecular datasets, while 3D models excel on large datasets and reaction- centrrc tasks. MARCEL also highlights the benefits of explicitly encoding multiple conformers to improve MRL.
6 DISCUSSIONS AND CONCLUSIONS
In this work, we present the first MoleculAR Conformer Ensemble Learning benchmark (MARCEL) evaluate the potential of learning from of conformer structures. Through conformer ensemble learning strategies, we discover performance improvements across various tasks. However, there are some limitations that require further consideration. First, our studied ensemble learning strategies do not universally improve performance across all tasks and datasets. This highlights the need for more tailored approaches that integrate with domain expertise better model specific tasks and datasets of practical interest. Second, the computational cost of encoding all conformers within the ensembles, especially for larger datasets, suggests the need further study the trade-offs between model complexity and efficiency. Finally, our datasets only contain regression tasks and do not cover all of the relevant chemical space, which might limit the generalization of our experimental findings.
Despite these challenges, we envision that our work will prompt further research in the geometric deep learning community on how to make use of conformer ensembles for molecular property prediction. For instance, future research could explore new model architectures that can efficiently encode ensemble- hope that work will stimulate collaborative research across the machine learning and chemistry fields, with the ultimate goal of pushing the boundaries predictive molecular modeling and aligning algorithmic advancements with the practical needs of the chemistry community.
information or more sophisticated conformer sampling strategies. We also

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work is supported by NSF Center for Computer Assisted Synthesis (2202693).
REFERENCES
[1] Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z. Li. A Systematic Survey of Chemical Pre- Pre-trained Models. 2023. 1
[2] Oliver Wieder, Stefan Kohlbacher, Mélaine Kuenemann, Arthur Garon, Pierre Ducrot, Thomas Seidel, and Thierry Langer. A Compact Review of Molecular Property Prediction with Graph Neural Networks. Drug Discov. Today Technol. 37:1- 12, 2020. 1
[3] W. Patrick Walters and Regina Barzilay. Applications of Deep I Learning in Molecule Generation and Molecular Property Prediction. Acc. Chem, Res., 54(2):263-270, 2021.
[4] H. L. Morgan. The Generation of Unique Machine Description for Chemical Structures A Technique Developed at Chemical Abstracts Service. Chem. Doc., 5(2):007 113, 1965,
[5] Robert C. Glem, Andreas Bender, Catrin H. Arbby, Lars Carlsson. Scott Boyer, and James Smith. Circular Fingerprints: Flexible Molecular Descriptors with Applications from Physical Chemistry to ADME. IDrugs, 9(3):199-204, 2006.
[6] G. Skoraczyúski, Dittwald, B. Miasojedow, S. Szymkué, E. P. Gajewska, Grzybowski, and A. Gambin. Predicting the Outcomes of Organic Reactions via Machine Learning: Are Current Deseriptors Sufficient? Sci. Rep., 7(1):1-9, 2017.
[7] Zhen Liu, Yurii S. Moroz, and Olexandr Isayev. The Challenge of Balancing Model Sensitivity and Robustness in Predicting Yields: Benchmarking Study of Amide Coupling Reactions, chemrxivorg, 2023.
[8] Thomas N. Kipf and Max Welling, Semi-Supervised Classification with Graph Convolutional Networks. In ICLR, 2017. 1
[9] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. In ICML, pages 1263-1272 2017.
[10] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefannie Jegelka. Representation Learning on Graphs with Jumping Knowledge Networks. ICML, pages 5453-5462, 2018.
[11] Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. In ICLR, 2018.
[12] Kristof Schütt, Picter-Jan Kindermans, Huziel Enoc Sauceda Folix, Stefan Chmicla, Alexandre Tkatchenko, and Klaus-Robert Müller. SchNet: A Continuous- Filter Convolutional Neural Network for Modeling Quantum Interactions. In NIPS, pages 991-1001, 2017. 1,5
[13] Johannes Klicpera, Janek GroB, and Stephan Günnemann. Directional Message Passing for Molecular Graphs. In ICLR, 2020. 5,5
[14] Johannes Gasteiger, Florian Becker, and Stephan Günnemann. GemNet: Universal Directional Graph Neural Networks for Molecules. NeurlPS, pages 6790-6802, 2021.
1,5
[15] Kristof Schütt, Oliver T. Unke, and Michael Gastegger. Equivariant Message Passing for the Prediction of Tensorial Properties and Molecular Spectra. In /CML, pages 9377-9388, 2021. 1,5
[16] Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie- Liu. SE(3) Equivariant Graph Neural Networks with Complete Local Frames. In ICML, pages 5583 5608, 2022. 1,5
[17] Weitao Du, Yuanqi Du, Limei Wang, Dicqiao Feng, Guifeng Wang, Shuiwang Ji, Carla Gomes, and Zhi-Ming Ma. A New Perspective on Building Efficient and Expressive 3D Equivariant Graph Neural Networks. arXiv.org. 2023. 1,5
[18] Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grobe. Weisfeiler and Leman Go Neural: Higher-Order Graph Neural Networks. In AAAI, pages 4602-4609, 2019.
10

---

Published as a conference paper at ICLR 2024
[19] Keir Adams, Lagnajit Pattanaik, and Connor W. Coley. Learning 3D Representations of Molecular Chirality with Invariance Bond Rotations. ICLR, 2022. 1,
[20] Bharath Ramsundar, Peter Eastman, Patrick Walters, and Vijay Pande. Deep Learning for the Life Sciences: Applying Deep Learning to Genomics, Microscopy, Drug Discovery, and More. O'Reilly Media, 2019. 2
[21] Emanuele Perola and Paul S. Charifson. Conformational Analysis of Drug-Liks Molecules Bound to Proteins: An Extensive Study of Ligand Reorganization upon Binding. Med. Chem., 47110):449--2100 2004. 3,3
[22] Andrew F. Zahrt, Jeremy J. Henle, Brennan T. Rose, Yang Wang, William T. Darrow, and Scott E. Denmark. Prediction of 363364244)eaau6611, 2019.
Catalysts by Computer- Computer-Driven Workflow and Machine Learning. Science,
[23] Simon Axelrod and Rafael Gómez-Bombarelli Molecular Machine Learning with Conformer Ensembles. arXiv.org, 2020. 2,
[24] Jan Weinreich, Nicholas J. Browning, and O. Anatole von Lilienfeld. Machine Learning of Free Energies in Chemical Compound Space Using Ensemble Representations: Reaching Experimental Uncertainty for Solvation. Chem. Phys., 154(13):134115 2021.
[25] Kangway V. Chuang and Michael J. Keiser. Attention-Based Learning on Molecular Ensembles. arXivorg, 2020. 2
[26] Chaitanya K. Joshi, Arian R. Jamasb, Ramon Viñas, Charles Harris, Simon Mathis, and Pietro Liò.
Multi-Stan State RNA Design with Geometrie Aulti-Graph Neural Networks. arXiv.org 2023. 2
[27] Simon Axelrod and Rafael Property Prediction and Molecular Generation. Sci. Data, 91): 2022. 3, 14
GEOM, Energy-Annotatedd Molecular Conformations for
[28] Zhen Liu, Tetiana Zubatiuk, Adrian Roitberg, and Olexandr Isayev. Auto3D: Automatic Generation of the Low-Energy 3D Structures with ANI Neural Network Potentials. J. Chem. Inf. Model., 62((2))5373-555 5382,
2022. 3, 14
[29] Roman Zubatyuk, Justin S. Smith, Benjamin T. Nebgen, Sergei Tretiak, and Olexandr Isayev. Teaching a Neural Network to Attach and Detach Electrons From Molecules. Nat. Commun., 12(1):1-11, 2021. 3, 14
[30] Andrzej M. Zurański, Jason Y. Wang, Benjamin J. Shields, and Abigail G. Doyle. Auto-QChem: An Automated Workflow for the Generation and Storage of DFT Calculations for Organic Molecules. React. Chem. Eng., 766)1276-128 2022. 3, 14
[31] Qiyuan Zhao, Sai Mahit Vaddadi, Michael Woulfe, Lawal A. Ogunfowora, Sanjay S. Garimella, Olexandr Isayev, and Brett M. Savoie. Comprehensive Exploration of Graphically Defined Reaction Spaces. Sci. Data, 10(1):1-10 2023. 3, 14
[32] Peikun Zheng, Roman Zubatyuk, Wei Wu, Olexandr Isayev, and Pavlo O. Dral. Artificial Intelligence- Enhanced Quantum Chemical Method with Broad Applicability. Nat. Commun., 12(1):7022, 2021. 3
[33] Tobias Gensch, Gabriel dos Passos Gomes, Pascal Friederich, Ellyn Peters, Théophile Gaudin, Robert Pollice, Kjell Jorner, AkshatKumar Nigam, Michael Lindner-D' Addario, Matthew Sigman, and Alán Aspuru-Guzik. A Comprehensive Discovery Platform for Organophosphorus Ligands for Catalysis. J. Am. Chem. Soc., 144:1205 - 2022. 3, 15
[34] Anthony R. Rosales, Jessica Wahlers, Elaine Limé, Rebecca E. Meadows, Kevin W. Leslie, Rhona Savin, Fiona Bell, Eric Hansen, Paul Helquist, Rachel Munday, Olaf Wiest, and Plr-Ola Norrby, Rapid Virtull Screening of Enantioselective Catalysts Using CatVS, Nat. Catal., 2(1):41-45: 2019. 4, 15, 16
[35] G. P. Moss. Basic Terminology of Stereochemistry (IUPAC Recommendations 1996). Pure Appl. Chem 68(12):21932222., 1996. 4, 16
[36] Benjamin Meyer, Boodsarin Sawatlon, Stefan Heinen, O. Anatole von Lilienfeld, and Clémence Cormin- bocuf. Machine Learning Meets Volcano Plots: Computational Discovery of Cross- cooss-Coupling Catalysts. Chem. Sci., 9:7069-7077, 2018. 4, 16
[37] Noel M. O'Boyle, Michael Banck, Craig A. James, Chris Morley, Tim Vandermeersch, and Geoffrey R. Hutchison. Open Babel: An Open Chemical Toolbox. J. Cheminformatics, 3(1):1-14, 2011. 4, 16
11

---

Published as a conference paper at ICLR 2024
[38] Greg Landrum, Paolo Tosco, Brian Kelley, Ric, sriniker, gedeck, Riceardo Vianello, NadineSchneiider, Eisuke Kawashima, Andrew Dalke, Dan N, David Cosgrove, Brian Cole, Matt Swain, Samo Turk, AlexanderSavelyev, Gareth Jones, Alain Vaucher, Maciej Wojcikowski, Ichiru Take, Daniel Probst, Kazuya Ujihara, Vincent F. Scalfani, guillaume godin, Axel Pahl, Francois Berenger, JLVaroo, strets JP, and DoliathGavid. rdkit/rdkit: 2022_02 03 2 (ql 2022) release, 2022. 4, 5, 16
[39] Darko Butina. Unsupervised Data Base Clustering Based on Daylight's Fingerprint and Tanimoto Similarity: Fast and Automated Way Cluster Small and Large Data Sets. Chem. Inf. Comput. Sci., 39(4): 747-750, 1999. 4
[40] Leo Breiman. Random Forests, Mach. Learn., 45(1):5-32, 2001. 5
[41] David Rogers and Mathew Hahn. Extenited-Commeccvvtty Fingerprints. Chem. Inf. Model., 50:742-754, 2010. 16
[42] Joseph L.. Durant, Burton A. Leland, Douglas R. Henry, and James G. Nourse, Reoptimization of MDL. Keys for Use in Drug Discovery. J. Chem. Inf. Comput. Sci., 121-1120, 2002. 5, 16
[43] Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Temm Memory. Neural Comp., 983:1735- 1997. 5
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Uszkoreit Kaiser, and Illia Polosukhin. Attention is All You Need. In NIPS, pages 5998-6008, 2017. 5, 16
[45] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful Are Graph Neural Networks? In ICLR, 2019. 5
[46] Weihua Hu, Matthias Fey, Marinka Zitiik Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for Machine Learning Graphs. In NeurIPS, pages 2020. 5, 17
[47] Kevin Yang, Kyle Swanson, Wengong Jin, Connor W. Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi S. Jaakkola, Klavs F. Jensen, and Regina Barzilay. Analyzing Learned Molecular Representations for Property Prediction. J. Chem. Inf. Model., 59(8):3370 3388, 2019. 5
[48] Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, Dominique Beaini, Recipe for a General, Powerful, Scalable Graph Transformer. In NeurlPS, pages 14501-14515. 2022. 5
[49] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling, E(n) Equivariant Graph Neural Networks. In ICML, pages 9323-9333 2021.
[50] Thomas G. Dictterich, Richard H. Lathrop, and Tomás Lozano-Pérez Solving the Multiple Instance Problem with Axis-Parallel Rectangles. Artif. Intell., (9(1-))31-7,, 1997. 6
[51] Oded Maron and Tomás Lozeno-Pérez. A Framework for Multiple-Instance Learning In NIPS, pages 570-576, 1997. 6
[52] Maximilian Ilse, Jakub M. Tomezak, and Max Welling. Attention-based Deep Multiple Instance Learning. In ICML, pages 2241, 2018.
[53] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnaáás Póczos, Ruslan R. Salakhutdinov, and Alexander Smola. Deep Sets. In NIPS, pages 3391-3401, 2017.
[54] Demitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR, 2015. 6
[55] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015. 7, 17
[56] Carlo Adamo and Vincenzo Barone. Toward Reliable Density Functional Methods Without Adjustable Parameters: The PBEO Model. J. Chem. Phys., 110(13):6188 6170, 1999. 14
[57] A. Verloop, W. Hoogenstraaten, and J. Tipker. Development and Application of New Steric Substituent Parameters in Drug Design. In F. Ariëns, editor, Drug Design, volume 11 of Medicinal Chemistry: A Series of Monographs, pages 165-207. Academic Press, Amsterdam, 1976. 15
12

---

Published as a conference paper at ICLR 2024
[58] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapcau, Matthicu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-leam: Machine Learning in Python. J. Mach. Learn. Res., 12:2825 2830, 2011. 16
[59]
Philip Gage. New Algorithm for Data Compression. Users J. 12(2):23-3,, 1994. 16
[60]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veto Martin Raison. Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurlPS, pages 8024-8035, 2019. 17
[61] Matthias Fey and Jan Eric Lenssen. Fast Graph Representation Learning with PyTorch Geometrie, In RLGM@ 2019. 17
13

---

Published as a conference paper at ICLR 2024
Supplementary Material for MARCEL
Dataset Description
14
A.I Drugs-75K
14
A.2 Kraken
15
A.3 EE
15
A.4 BDE
16
Implementation Details
16
B.1 Implementation of ID Models
16
B.2 Featurizations of Molecules for 2D Models
17
B.3
Hyperparameter Specifications and Experimental Environments
17
Additional Experiments on Evaluation Schemes of the Conformer Sampling Strategy
17
Raw Data
18
A
DATASET DESCRIPTION
MARCEL include four datasets that cover a diverse range of chemical space, which focuses on four relevant tasks for both molecules and reactions, with emphasis on Boltzmann- Theory (DFT) level. Detailed
properties of conformer ensembles computed at the Density-Function information regarding dataset access, data formatting, and loading procedures be found at GitHub repository https: / //gitttubb com wwtp///////////ELLLL.. Any subsequent updates will also be posted on this repository.
A.1 DRUGS-75K
Drugs-75K is subset of the GEOM-Drugs [27] dataset, which includes 75,099 drug-like molecules with at least rotatable bonds. The original GEOM-Drugs dataset was constructed using semi- empirical DFT methods, which is less accurate than full DFT. To curate the Drugs-75K subset, Auto3D [28] is used to generate and optimize the conformer ensembles for each molecule and AIMNet-NSE [29] used calculate three important DFT-based reactivity descriptors: ionization potential, electron affinity, and electronegativity [30].
Auto3D [28] efficiently generates high-quality conformers, with mean RMSD at around 0.2 when compared with DFT conformers. It has been used in other large conformer dataset generation [31]. Regarding the neural network surrogate AIMNET-NSE [29]. it mimics the PBEO/ma-def2-SV method of DFT, which is widely used in the chemistry community. Investigating their accuracy out of the scope of this paper, but are readily accessible from multiple sources [29, 56).
Objectives. The tasks are to predict the Boltzmann-averagen value of each property across the conformer ensemble (y) kn = Σc,ee PC,UC,, where yc, is conformer-ppecffic property. We are given each C, and the goal is to predict (y) from the molecular graph G, single conformer C, E C, or the set
Dataset preparation. In preparing the 75K version of GHOM-Drugs, we begin with the original SMILES strings of the molecules. We first exclude molecules that have less than rotatable bonds. To enable the utilization of AIMNet-NSE for descriptor computation, we retain only those molecules containing atoms H, C, N. O, F, Si, P, S. and CL. Further, we generate DFT-level conformers and compute their energies with Auto3D. Based these conformers, compute three chemical bond energy descriptors using AIMNet-NSE. We exclude conformers that Auto3D fails to converge and charged molecules that unable to be processed by AIMNet-NSE, which results in 75,099
14

---

Published as a conference paper at ICLR 2024
10
107
107
107
102
10'
10
10
10
10
10F
10F
107
107
0.0
7.5
(0) Sterimel Bg
Starimol
(c) Buriel Sterimol Bg
(4) Buried Sterinol I.
Figure S1: Histogram of the ratio of the variance of each conformer property to the variance of each Bolzzmann-averaged property in the Kraken dataset.
molecules. Subsequently, we compute molecular-level Boltzmann- averaged descriptors based on conformer-level descriptors. Finally, we undertake deduplication process as outlined in Section with RMSD threshold of 2.0, which yields total of 558,002 distinct conformers.
nolizman--averaged
Data availability and license. The original GEOM-Drugs dataset publicly available at ht hpsps: //githul digtthu..o. comm/Learningmatteemmiitt// geon but no license is specified. Our Drugs-75K can be accessed at https:/ / /github. cww Stt///XXDZZ//// Bttp//SXDZ///ARCCEE MARMMM/ tetie//treemmatii aeeaainddatattstDDeraaaa conformer ensembles and descriptors that generated, they are licensed under the Apache License.
As for the
A.2 KRAKEN
Kraken [33] is a dataset of 1,552 monodentate organophosphorus (III) ligands along with their DFT-computed conformer ensembles. In this study, we consider four 3D catalytic ligand descriptors exhibiting significant variance among conformers: Sterimol B5, Sterimol L, buried Sterimol Bs, and buried Sterimol L. These descriptors quantify the steric size of a substituent in Á, and are commonly employed for Quantitative Structure-Activity Relationship (QSAR) modeling. The buried Sterimol variants describe the steric effects within the first coordination sphere of metal [57].
Objectives. As in the Drugs-75K tasks, the goal is predict the Boltzmann-average value of each property across the conformer ensemble from the molecular graph G, single conformer C; E C, or the set C.
Dataset preparation. In this study, we utilize the original 3D geometry structures of molecules and their corresponding gallzmann-averagee properties provided in the Kraken dataset. Among the 78 physical-organic properties listed in the original dataset, we select four properties that demonstrate high variance across conformer ensembles, as illustrated in Figure $1.
Data availability and license. The Kraken dataset is publicly accessible at https: //kraknn cs. toronto. edu. Its copyright is retained by the original authors. Under the permission of the original authors, the Kraken dataset with the conformer ensembles and the four conformer-level descriptors used in this study can be accessed at https: //git gitthu..... wwt.ccmmm////.e /Stp////K////ELLL////// ttree//tee////eeeeeeeeee wwwree/manndaaaie datase ts/Krake /rraken.
A.3 EE
EE [34] is a dataset of 872 catalyst-substrate pairs involving 253 Rhodium (Rh)-bound atropisomeric catalysts derived from chiral bisphosphine, with 10 enamides substrates. The dataset includes conformations of catalyst-substric transition state complexes in two separate pro-S and pro-R configurations. The task to predict the Enantiomeric Excess (EE) of the chemical reaction involving the substrate, defined as the absolute ratio between the concentration of each enantiomer in the product distribution.
Objectives. EE depends on the conformer ensembles of each pro-R and pro-S complex. The goal is
to predict EE from the graphs of the catalyst and substrate (Geat: Gaub), conformer C₁. E ((R) and CiS) ((S) for each complex, or the ensembles (R) and (())
Dataset preparation. The conformer ensembles are generated with Q2MM, which automatically generates Transition State Force Fields (TSFFs) in order simulate the conformer ensembles of each
15

---

Published as a conference paper at ICLR 2024
prochiral transition state complex. Then, the EE values computed from the conformer ensembles by Boltzmann -averaggnng the activation energies for the competing transition states [34, 35J. Finally, conduct the same conformer deduplication process as described in Section with RMSD threshold of 1.0.
Data availability and license. As of now, the EE dataset is proprietary, given that the publication addressing the conformer ensembles is still under preparation. Therefore, access to the EE dataset is restricted to review purposes only. We anticipate making the EE dataset publicly accessible following the acceptance of the corresponding paper.
A.4 BDE
BDE [36] is dataset containing 5,915 organometallic catalysts ML, consisting of a metal center (M = Pd. Pt, Au, Ag, Cu, Ni) coordinated to two flexible organic ligands (L, and L2), each selected from 91 91-membered ligand library. The data includes conformations of each unbound catalyst, as well as conformations of the catalyst when bound to ethylene and bromide after oxidative addition with vinyl bromide. Each catalyst has an electronic binding energy. computed as the difference in the minimum energies of the bound-catalys complex and unbound catalyst, following the DFT- optimization of their respective conformer ensembles.
Although the binding energies are computed via DFT, the conformers provided for modeling are initially generated with Open Babel [37], followed by further geometric optimization steps, which ensures that the generated 3D structures are likely to be the global minimum energy conformers at the force field level [36, Supplementary Information]. We also note that obtaining DFT-optimized conformers for BDE is not feasible given the time-consuming nature of the process single geometric search using DFT can take 2 to days. Therefore, this realistically represents the setting in which precise conformer ensembles
unknown at inference.
Objectives. The task is to predict the binding energy from the graphs of the unbound and bound cata- lyst, sampled conformers C (uubbound) E ((ubbound) and C (hbuudd)) E C ((bound), or the ensembles Cambounn)
and (houndd)
Dataset preparation. We employ Open Babel [37] to produce conformers for each unbound catalyst and each bound complex. In order to avoid redundancy, we follow a deduplication process as outlined in Section For the unbound catalysts, RMSD threshold value of 0.5 is applied, whereas for the bound complexes, a threshold of 1.0 is used.
Data availability and license. The binding energy descriptors can be accessed at https: e. wamerriialstt.. alscloud. org/ wwgg/reord//2// 2018. 0014/v1 under the Creative Commons Attribution 4.0 International license. The conformers are publicly available at https: https://gitii //tp:/gggtttt.tt// wwt.hhu.....e...e www. SXXDZ//AR CEL/tree/man Beaaii/daatstts/Bee under the Apache license.
B IMPLEMENTATION DETAILS
B.1 IMPLEMENTATION OF 1D MODELS
For the random forest model that operates on fingerprints, we employ three molecular finger- print schemes: the Molecular ACCess System (MACCS) [42], Setendedd-Conecttvee Fingerprints (ECFP) [41]. and RDKit topological fingerprints [38]. Then, we concatenate their outputs into single vector, which might lead to some feature redundancy, given the possible overlaps in these three fingerprint representations of the molecular structure. To tackle this issue, we remove any features that exhibit a high correlation exceeding 90% with the other features. For implementation, employ Scikit-Learn [58] and compute fingerprints with RDKit [38].
For both LSTM and Transformer models that operate on SMILES strings, use a Byte-Pair Encoding (BPE)-based tokenizer [59] that is pretrained on PubChem 10M, which strikes a balance among character- and word-level representations and allows to handle large vocabularies in molecular corpora. For the Transformer model, we further follow the positional embedding scheme [44] to capture the positional relationship among tokens in the SMILES string.
16

---

Published as a conference paper at ICLR 2024
Table S1: A summary of node and edge features used in 2D GNN models.
Feature
Explanation
Atomi c&
Atomic number, representing the type of atom.
ChiralTag
Indicator of chirality, property of asymmetry.
TotalDegree
Sum of implicit and explicit bonds of atom.
FormalCharge
Charge of atom assuming equal sharing of bonding electrons.
Node
TotalNuumls
Total number of hydrogen atoms bonded to the atom.
NumRadi calErectrons
Count of unpaired electrons in atom.
Hybridization
Type of atomic orbital hybridization in the atom.
IsAromatic
Boolean indicating if the atom is part of aromatic ring.
IsInRing
Boolean indicating if the atom is part of any ring structure.
BondType
Type of the bond (c-g., single, double, triple, aromatic).
Edge
Stereo
Stereochemistry of the bond (e.g.. "none". "any". "Z", "E" for double bonds).
laConjugated
Boolean indicating the bond is part conjugated system.
B.2 FEATURIZATIONS OF MOLECULES FOR 2D MODELS
Following OGB [46], we employ a rich of features for atoms (nodes) and bonds (edges) for 2D GNN models. A complete list of node and features can be found in Table S1.
B.3 HYPERPARAMETER SPECIFICATIONS
EXPERIMENTAL ENVIRONMENTS
Each model is trained over 2,000 epochs using the Adam optimizer [55] with early stopping triggered there is no improvement in the training loss over 200 epochs. ensure a fair comparison, the hidden dimension size is uniformly set 128 for all models. Other hyperparameters mostly follow the original configurations as described in the respective papers. The complete hyperparameter set of each model be found in https: /tte::///ttttttt giggthu........ wtw/SXKDZ////ccccccccccc//ccc BAADDZ//MACEL//reee CeeeeCEL//teee////ii www.erreeemmmiiiiiee heaain/benchmarks/// arams.
We utilize Py PyTorch [60] and of the experiments are conducted on servers equipped with Nvidia A100 GPUs, each with 40GB of memory. For memory-intensive models such as GemNet and LEFTNet, H100 GPUs, each with 80GB memory. The cumulative computation time across all experiments amounts approximately 6,000 single GPU hours.
[61] implement all deep learning models. Most
servers with Nvidia
C ADDITIONAL EXPERIMENTS ON EVALUATION SCHEMES OF THE CONFORMER SAMPLING STRATEGY
In this section, conduct additional experiment the conformer ensemble learning strategies. We assess all 3D models on five tasks: Ionization Potential (IP) from the Drugs-75K dataset, B, and BurB, from the Kraken dataset, and tasks from the EE and BDE datasets.
In our previous setup, we evaluate the conformer sampling strategy using the lowest-energy conformer of each molecule evaluation time, provide direct comparison the single-conformer 3D models that are trained and tested with the lowest energy conformation. In these experiments, we continue to sample a random conformer uniformly from the conformer ensemble during training time, but consider two additional evaluation schemes: (1) evaluating model performance when encoding randomly sampled conformer, and (2) evaluating model -performance when averaging the per-conformer predictions across the entire conformer ensemble.
The results of these experiments are summarized Table S2. In the table, we refer to the original evaluation scheme as "fixed", and the additional schemes "random" and "all", respectively. We find that across all three schemes, using the lowest-energy conformer for evaluation consistently yields the best performance. This is expected, as the lowest-energy conformer contributes the most to ensemble- descriptors. The random conformer evaluation scheme generally yields the worst performance, which is likely due the introduction of noise from less relevant conformers at test.
17

---

Published as a conference paper at ICLR 2024
Table S2: Performance comparison of three conformer sampling variants with different evaluation strategies. All models are trained with randomly sampled conformer from the ensemble. The last column summarizes the average rank across all datasets for each base model.
Evaluation Strategy
Drugs-75K
Kraken
Average Rank
Model
EE
BDE
IP
B;
BurB;
Fixed
0,4452
0.3235
0.2086
20.3595
1.9737
SchNet
Random
0.4498
0.3682
0.2454
22.0380
2.4416
3
0.4428
0,3856
0.2407
18.0296
2.0106
2
Fixed 0.4395
0.3323
0.2237
15.0596
1.4741
2
DimeNet++
Random
0,4555
0.3549
0.2222
13.5681
1.4688
2
All
0.4479
0.3282
0.2001
12.3562
1.6270
Fixed
0.4066
0.2694
0.1796
12.0541
1.6059
GemNet Random
0.4250
0,4034
0.2534
16.1709
1.7894
3
All
0.4320
0.4523
0.2481
14.3952
1.6660
2
Fixed
0.4466
0.3441
0.2476
19.1521
1.9262
1
PaiNN Random
0,4770
0.3756
0,2478
21.3553
1,9411
3
All
0.4478
0.3458
0.2342
19.1955
1.8696
2
Fixed
0.4430
0.4524
0.2442
31.3733
2.5126
1
ClofNet Random
0.4530
0.4689
0.2736
31.3675
2.6310
2
All 0.4363
0.4749
0,2855
34.3203
2.0271
= 2
Fixed 0.4149
0.2834
0.2120
20.3358
1.5276
1
LEFTNet
Random 0,4518
0.3177
0.2344
20.3740
1.5842
3
All
0.4274
0.3152
0.2170
18.8945
1.8663
2
time. Interestingly, we observe occasional performance improvement when averaging the predictions
across all conformers in the ensemble, indicating that explicitly using ensemble-level information during evaluation can be beneficial.
D RAW DATA
The raw performance data with standard deviation of Table and Figure 3 is summarized in Table S3.
18

---

Published as a conference paper at ICLR 2024
Table S3: Raw performance data (mean + standard deviation) of representative 1D, 2D, 3D, and conformer ensemble MRL models in terms of absolute test error.
Drugs 75K
Kraket
Model
X
Hg
Barli
0,4957am2
02732aom
0.152
LSTM
0.4738
0,4648aams
0,2505
0,487%xcmmm
0,5142am
0.282aames
0,1924
64.0088m2.
2.827
Transormer
0.5850am
0,407%
0,961
0,8389
0.4929
0,278 Lower
08 66 nn
10,077
GIN
0435
0.4169mmm
0,2260lomn
0.3128
4000
0.1719uamts
0.120lomm
62 300
2.6 2.6308ozz
GIN-VV
0,2267
0,3567
381 28
ChenProp
0.595
0.4417axmm
0,244leoma
1.5452amm
0.3002ama
0,1948
61 61.636mm
2.661
GupllIPS
0331
0.4085
0.2312
0.3490
0.336aamm
0.266aam
0.1500
61 61 mux
2.4527
44%44mm
0.4207
0.224%
0.309%.m
0.1861
m.
DineNere4
0.44
0u4233usmm
0.2436aaar
0.4144aree
0.2077aam
0.152
14.641
1.4503
GemNer
0.008aase
0,3922axus
0.197keww
0,2789uass
0.3754u0ss
0.1782aume
0.1635
180338c
L 10300mm
PaiNN
0.505am
0.4495mmm
0.2324urw
0.344)um
0.477aa
0.167
2. Lowe
ClufNer
0.503amm
0.4251
0,2376evxx
0.2844amm
0,2529umm
LRPTNe
04144sme
0,2083
0.9073eors
0.4493-aroe
0:217mmmm
O 14866
19.7974.
SchNet
0 04452amm
0.4232oom
0,2245
0.3235xon
0,598
0.206aamm
0.1739xx
20.3595 soe
1.975
0.4217wmme
0.2432ew
0.3
0:2237
0.156l.amu
15005mmm
GemNet
04066uee
0,3910umm
0.2027
0.348
0.1796usms
0,1184uauu
12.054aamm
1,059
+Supllin
PaiNN
0.4066amc
0,4393ume
0,2351
0,546
0.358
0.2676ammo
0.154
19 19.152 Lazee
1,922
ClofNet
0 0.430auem
0.4237axess
0,2335
0.4344
0.5902u
0.2442axx0
0,1756uxx
31 31.3733 wie
2.5
Mean
0.583unm
0.4410mmm
0,237lacm
0,3075easi
0.2282xxao
0.1619
20.1392
SchNet
Deeplet
45377
0.4306wae
0,2385-rw
0.3105-00m
0.4322
2244
0.1535
0495... mas
2 mss
Atlenioon
0.456mmmm
0.4382axx
0.2380umm
6.2704amm
0.517amm
0.1443
2238ason
0.4340ummm
0.2425
0.2650
0.3828
0.1268omee
DineNett+
DeepSet
0.412amm
0,3944aum
0,2267axmn
0.2889wcoe
0.3458uac
0.1783uxr
0.1339
L.733
Attention
0.136
0.6030usm
0,2325
0.3718axmm
0.3628acc
0.1185
13,3433u.a Aws
04500
0,2289emen
0,1987
1914
Gem
DorpSet
04187
0.4002mmn
0,2169Gowx
0.2313
0.3386ars
189
0.0977mmm
139275u
Fasemiiic
Attantios
0.4212
0,2260kee 0.422
mm
0.3670a
0.3554are
0. 17770aa
0.1346
12.024
2.6m0mmmm
0.4425
0,2360eosa
0.2877aas
0.2450axz20
0.1817asoo
0.1472
16.429aama
PaNN
DeepSet
0.477a
0.4269
0.2294u
0,2225
0.36 (9ucee
16993amm
0.1324mmm
13,5570.sse
2.207
Attention
0.4641amms
0.4567axmm
0,2471acee
0,3496ucr
0,400u
0.2123aams
0.1506uams
19.1556
2.2335
Mran
4516..
0.430
0.2365uom
4.448
0.002.oon
199710..
CaNNe
DeqpSc
0.423t.amm
0.4033ummm
0.2199emam
0.3225
0,4742
0.2233amm
139647
2.357
Attention
0.430aamm
0.4107axee
0,2220mxm
0,3734
0.4933acii
0.2178ouss
0.1090aamm
26.7133 nus
2.
Mean
0 04402amm
0.2185 0.4267
0,2969ammi
0.3643ans
20888aas
0.6866
189245..x M
2. 20440
DeepSet
0,2069aowa
0,2664acxe
18.4199u. 0,1441a0u
Attention
0 a.4279aeer
4067
0.2198mmm
0.316lleors
0.4324ars
10207
0.1 11508mmmm
18 18.988.
2.6%
19


Reviews:
[{"review_id": "MwfLePJQJR", "reviewer": "Reviewer_4QFi", "summary": "This paper discusses the use of graph neural networks for ensemble-based learning of molecular representations. \nSpecifically, the paper introduces a molecular conformer ensemble learning benchmark, with the aim of evaluating\nthe potential of learning on conformer ensembles. The idea behind casting this problem as an ensemble-based\nlearning problem is that this could help take into account the dynamic aspects of molecules. Generally, the paper\nis well-written and contains a thorough comparison to state of the art results in the field.", "questions": "None", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's main strength, in this reviewer's view, is that it thoroughly compares its approach to the state of the art. Its main value is\nlikely that it can serve as a benchmarking basis for various approaches in the field. The main original aspect is the use of an ensemble-based approach, which affords to incorporate the dynamical aspect of molecules. The paper is also well written and meticulous at comparing to the state of the art,", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EJbDngspcU", "reviewer": "Reviewer_pme5", "summary": "This paper introduces a new benchmark named MARCEL, which consists of four tasks: Drugs-75K, Kraken, EE, and BDE. The goal is to evaluate the learning with multiple conformers for each molecule. In traditional evaluations of molecular machine learning, the dynamic nature of molecules taking on various possible conformers has been somewhat overlooked. MARCEL addresses this by setting up a possible set of conformers for each molecule and preparing a task to predict the Boltzmann average of molecular properties over the set of conformers, i.e. conformer ensembles. Using this benchmark data, the paper also provides comprehensive empirical evaluations of widely-used 1D, 2D, and 3D GNNs, also examining two strategies of ensemble learning in situations where a set of conformations can be used.", "questions": "Q1. While Drugs-75K, a subset of the GEOM-Drugs [27], Kraken [33], EE [34], BDE [36] all have associated citations, indicating they are secondary data, it was unclear whether the presented datasets are a simple curated version of these primary data, or if any original information was generated in this study. If there is any original information, it would be very helpful to be clarified.\n\nQ2. Regarding 'Dataset preparation' in the Supplementary Material, why are different methods used to generate conformers depending on the data? (Auto3D for Drugs-75k, Q2MM for EE, Open Babel with DTF?) Is this point not problematic for benchmarking several methods?\n\n\nQ3. While I understand that the 'two conformer ensemble learning strategies' are useful for predicting the 'Boltzmann-averaged value of each property across the conformer ensemble,' can they be said to be generally useful for predicting molecular properties? Could you provide any supported evidence for this claim?\n\n\nQ4. Is the BDE task also about predicting the Boltzmann-averaged value?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "In molecular machine learning, considering the dynamic structural transitions of molecules is an extremely important point. While there are studies predicting molecular dynamics simulations through machine learning, and existing research examining the impact and significance of conformers on machine learning predictions, the data and tasks are extremely limited. Thus, objectively comparing multiple methods on the same foundation is challenging. In this context, the benchmark proposed in this paper is extremely intriguing. Moreover, even if one wishes to consider multiple conformers for each molecule in machine learning evaluations, preparing it can be difficult without specialized knowledge. Considering these points, establishing such a benchmark and sharing it within the molecular machine learning community has the potential to enable more constructive methodological research and analysis.\n\nIn this paper, not only is a dataset provided, but comprehensive baseline evaluations are also given that would be useful for researchers looking to enter this field of study. In particular, comprehensive evaluation is conducted using multiple popular GNN models in 1D, 2D, and 3D. These results offer insight into how machine learning methods at each representation level are affected by actual conformation changes, providing very valuable knowledge.", "weaknesses": "In the four tasks developed in this study, the objective is defined as predicting the Boltzmann average of various properties over multiple conformers. Under this goal setting, it seems intuitive that using information from multiple conformers would naturally improve prediction accuracy. Therefore, it has not been proven that 'considering multiple conformers contributes to machine learning predictions of real data (e.g., actual experimental measurements of molecules rather than computed values).' In this sense, the utility of this benchmark remains a bit artificial, and practical values would be unclear.\n\nIt is possible that machine learning predictions based solely on the ground-state structure, as traditionally done, are already practically useful. Regarding how considering multiple conformers contributes to molecular machine learning, the contribution of this study might be limited.\n\nAdditionally, since all four datasets prepared are secondary data from referenced primary data, it's unclear how challenging it would be for researchers to prepare them on their own. While the paper mentions quality control and the removal of redundancies, it's unclear whether there is any original information added to this study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BJXotfFYYu", "reviewer": "Reviewer_m99Y", "summary": "This work presents MARCEL, a novel dataset and benchmark for studying molecular conformer ensemble learning. MARCEL curates multiple datasets in which every molecule has many molecular conformers, and benchmark several baseline methods for predicting molecular properties from multiple molecular conformers.", "questions": "In Table 2, which molecular conformers are used as inputs to 3D graph neural network models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: This work curates novel datasets and benchmarks for an under-explored problem of molecular conformer ensemble learning.  \nQuality: Detailed information about dataset curation, baseline experiment settings and results are clearly elaborated.  \nClarify: The writing of this paper is excellent and well-organized.  \nSignificance: The presented MARCEL benchmark will be useful and impactful for researchers to develop novel molecule representation learning methods on multiple molecular conformers.", "weaknesses": "(1) For 3D models, it is recommended to add at least one 3D graph transformer models as baseline, such as Equiformer [1].  \n(2) It is recommended to add discussions about [2] as [2] proposes a molecular conformer ensemble learning module named ConfDSS. Also, it is recommended to add it as a baseline if it can be applied to the task in MARCEL.\n\n[1] Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs. ICLR 2023.  \n[2] Fast Quantum Property Prediction via Deeper 2D and 3D Graph Networks. Arxiv 2106.08551.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)