Deep Molecular Representation Learning via Fusing Physical and Chemical Information
Authors: Shuwen Yang
Ziyao Li
Guojie Song
Lingsheng Cai
Venue: NeurIPS 2021 Poster (2021)
Keywords: molecular representation
molecular fingerprints
neural molecular dynamics
graph neural networks

Abstract: Molecular representation learning is the first yet vital step in combining deep learning and molecular science. To push the boundaries of molecular representation learning, we present PhysChem, a novel neural architecture that learns molecular representations via fusing physical and chemical information of molecules. PhysChem is composed of a physicist network (PhysNet) and a chemist network (ChemNet). PhysNet is a neural physical engine that learns molecular conformations through simulating molecular dynamics with parameterized forces; ChemNet implements geometry-aware deep message-passing to learn chemical / biomedical properties of molecules. Two networks specialize in their own tasks and cooperate by providing expertise to each other. By fusing physical and chemical information, PhysChem achieved state-of-the-art performances on MoleculeNet, a standard molecular machine learning benchmark. The effectiveness of PhysChem was further corroborated on cutting-edge datasets of SARS-CoV-2.

Deep Molecular Representation Learning via Fusing Physical and Chemical Information
Shuwen Yang" Ziyao Li²; Guojie Song' Lingsheng Cai! Key Laboratory of Machine Perception and Intelligence (MOE), Peking University, Beijing, China
CCntter for Data Science, Peking University, Beijing, China {swyang, leeeezy, gjsong, cailingsheng hanpshhnnn@@uuu..... edu. cn
Abstract
Molecular representation learning the first yet vital step in combining deep learn- ing and molecular science. To push the boundaries of molecular representation learning, we present PhysChem, a novel neural architecture that learns molecu- lar representations via fusing physical and chemical information of molecules. PhysChem is composed of a physicist network (PhysNet) and a chemist network (ChemNet).. PhysNet is neural physical engine that learns molecular conforma- tions through simulating molecular dynamics with parameterized forces; ChemNet implements gearetry-aware deep message-passinn passing learn chemical biomedical properties of molecules. Two networks specialize in their own tasks and cooperate by providing expertise each other. By fusing physical and chemical information, PhysChem achieved state-of-the-art performances on MoleculeNet, a standard molecular machine learning benchmark. The effectiveness of PhysChem was further corroborated on cutting-edge datasets of SARS-CoV-2.
Introduction
The intersection between deep learning and molecular science has recently caught the eye of re- searchers in both areas. Remarkable progress made in applications including molecular property prediction 36 [6]. molecular graph generation [37] 12 28), and virtual screening for drug discovery [I. yet learning representations for molecules remains the first yet vital step. Molecular repre- sentation learning, or learning molecular fingerprints, aims to encode input notations of molecules into numerical vectors, which later serve features for downstream tasks. Earlier deep molecular representation methods generally used off-the-shelf network architectures including message-passingg neural networks [ graph attention networks (36) and Transformers [IT 24). These methods took either line notations (e.g. SMILES") or graph notations (i.c. structural formulas) of molecules as inputs, whereas physical and chemical essence of molecules was largely neglected. Notably, trend of integrating 3D conformations (i.e. the 3D Cartesian coordinates of atoms) into molecular representations recently emerged [ 15 29) while most of these methods assume the availability of labeled conformations of target molecules.
In order push the boundaries of molecular representation learning, revisited molecules from both physical and chemical perspectives. Modern physicists generally regard molecules as particle systems that continuously move following the laws of (quantum) mechanics. The dominant conformations of molecules reflect the equilibriums of these micro mechanical systems, and are thus of wide interest.
"qqual Contribution.
t *Corresponding Author.
*SMILES (Simplified Molecular Input Line Entry Specification 34
) is widely used protocol that specifies
(non-unique) line notations for molecules, CCO for ethanol, for example.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Chemists, on the other hand, focus more on chemical bonds and functional groups of molecules, which denote the interactions of electrons and determine chemical/ biomedical properties such as solubility and toxicity, etc. Nevertheless, physical and chemical information of molecules is not orthogonal. For example, torsions of bond lengths and angles greatly influence the dynamics of particle systems. Therefore, an ideal molecular representation is not only expected to capture both physical and chemical information, but also to appropriately fuse the two types of information.
Based on the observations above, we propose PhysChem, a novel neural architecture that captures and fuses physical and chemical information of molecules. PhysChem is composed of two specialist networks, namely physicist network (PhysNet) and a chemist network (ChemNet), who understand molecules physically and chemically PhysNet is a neural physical engine that learns dominant conformations of molecules via simulating molecular dynamics in generalized space. In PhysNet, implicit positions and momenta of atoms are initialized by encoding input features. Forces between pairs of atoms are learned with neural networks, according to which the system moves following laws of classic mechanics. Final positions of atoms are supervised with labeled conformations under spatial-invariant losses. ChemNet utilizes a cressage-passingg framework [9] to capture chemical characteristics of atoms and bonds. ChemNet generates messages from atom states and local geometries, and then updates the states of both atoms and bonds. Output molecular representations are merged from atomic states and supervised with labeled chemical / biomedical properties. Besides focusing their own specialties, two networks also cooperate by sharing expertise: PhysNet consults the hidden representations of chemical bonds in ChemNet to generate torsion forces, whereas ChemNet leverages the local geometries of the intermediate conformations in PhysNet.
Compared with existing methods, PhysChem adopts a more elaborated as well as interpretable architecture that incarnates physical and chemical understandings of molecules. Moreover, as PhysNet learns molecular conformations from scratch, PhysChem does not require labeled conformations of test molecules. This extends the applicability of PhysChem to situations where such labels are drug candidates. We evaluated PhysChem
unavailable, for example, with neural-network-geneeet on several datasets in the MoleculeNet (35) benchmark, where PhysChem displayed state-of-the-an art performances on both conformation learning and property prediction tasks. Results on cutting- edge datasets of SARS-CoV-2 further proved the effectiveness of PhysChem.
2 Related Work
Molecular Representation Learning Early molecular fingerprints commonly encoded line or graph notations of molecules with rule-based algorithms [19
22
25 Along with the spurt of deep learning, (6). More recently, researchers started
deep molecular representations gradually prevailed 7
to focus on incorporating 3D conformations of molecules into their representations []] 4 29 15).
Models that leveraged 3D geometries of molecules generally performed better than those that simply used graph notations, whereas most 3D models required labeled conformations of the target molecules. This limited the applicability of these models. Among previous studies, message- message-passingg neural networks (MPNNs) proposed a universal framework of encoding molecular graphs, which assumed that nodes in graphs (atoms in molecules) passed messages to their neighbors, and then aggregated received messages to update their states. A general message-passing layer calculated
M(h,, hy Big); h: + U m.), i iED V, (EN(M)
Σ
where i,j E V were graph nodes, hs and es were states of nodes and edges, and M ((),U(() were the message and update functions. For graph-level tasks, MPNNs further defined readout function feactitt which merged the states of all nodes into graph representations. In this work, ChemNet modifies the message-passing scheme of MPNNs: messages are extended to incorporate geometries of local structures, and states of both atoms and bonds are updated. See Section 3.4 for more details.
Neural Physical Engines Recent studies showed that neural networks
capable of learning annotated (or pseudo) potentials and forces in particle systems, which made fast molecular simulations 38 [7] and protein-folding tasks (6 possible. Notably, it was further shown in [16] 27) that neural networks alone can simulate molecular dynamics for conformation prediction. As an instance,
'PhysNet and ChemNet are novel architectures, not to be confused with previous works with similar or
identical names (33 32 20

---

postiion
Confomaation
Decoder
Abootte feuluunes
atcune momealaa
Initializzrr
PhysNet Block
conform -ation
expertiee
bond siolee
ateees
moleecules
features
neg boad
R.gnecunaiin
Readont
#locn unog. passing
bund mag.. ChemNet Block
angle
y
chemical properties
Figure I: A sketch of the architecture of PhysChem.
HamNet [16] proposed a neural physical engine that operated on a generalized space, where positions and momentums of atoms were defined high-dimensional vectors. In the engine, atoms moved following Hamiltonian Equations with parameterized kinetic, potential and dissipation functions. PhysNet our work similar engine. Nevertheless, instead of learning parameterized energies and calculating their negative derivatives as forces, we directly parameterize the forces between each pair of atoms. In addition, HamNet considered gravitations and repulsions of molecules based on implicit positions, while it ignored the effects of chemical interactions: for example, the types of chemical bonds were ignored in the energy functions. PhysChem fixes this issue via cooperation mechanism between two specialist networks. Specifically, PhysNet takes chemical expertise (the bond states) from ChemNet and introduces torsion forces, i.e. forces that origin from torsions in chemical bonds, into the dynamics. See Section 3.3 for more details.
Multi-Task Learning and Model Fusion The cooperation mechanism in PhysChem shares similar motivations with multi-task learning and model fusion. Multi-task learning [39 o is now almost universally used deep learning models. Representations are shared among a collection related tasks in order to learn the common ideas. Model fusion, on the other hand, merges different models on identical tasks to improve performances 23 13. Notably, these techniques have been previously applied to molecular tasks (30 13 31.. In PhysChem, conformation learning and property prediction tasks are jointly trained, with two specialist networks fused to achieve better performances. Nevertheless, the cooperation mechanism in PhysChem roots from observations in physics and chemistry, and enjoys better interpretability than straight-forward forward ensemble strategies. The advantages of the cooperation mechanism over multi-task strategies are also empirically shown in Section 4
3 Proposed Method: PhysChem
3.1 Preliminaries
Notations In the statements and equations below, we use italic letters for scalars and indices, bold lower-case letters for (column) vectors, bold upper-case letters for matrices, calligraphic letters for sets, and normal letters for annotations. Common neural network layers are directly referred to: FC (x) denotes a fully-connected layer with input æ: GCN (A, X) denotes (vanilla) graph convolutional network (44 with adjacency matrix A and feature matrix X; GRU (8, æ)) denotes a cell of the Gated Recurrent Units (GRU) [ with recurrent state and input signal æ;; LSTM (fxx)) denotes Long Short-Term Memory network [10] with input signals (æ, J: MaxPool ((æ;) denotes a max-pooliee max-pooling layer with inputs (.t). Exact formulas are available in the appendix. the operator of concatenations. || denotes the le norm of the input vector.
Problem Definition PhysChem considers molecular representation learning supervised learning task. It takes notations of molecules as inputs, conformations and chemical / biomedical properties supervisions. PhysChem assumes pre-conducted featurization process, after which a molecule can be denoted as an attributed graph M = (V,E, n, m, X XY). Here, V is the set of n atoms, (x). ... xxx) is the matrix of atomic E V x V is the set of m chemical bonds, X E gaxd,

---

features, and X E Rm = (zf L that of bond features. Based on above inputs, PhysNet outputs the dominant conformations of molecules, and ChemNet outputs the representations as well as the predicted chemical / biomedical properties of molecules.
Overview Figure Dis is sketch of PhysChem. An initializer first encodes the inputs into initial atom and bond states (olo) and e for ChemNet, along with the initial atomic positions and momenta (q()) and ( in PhysNet. Subsequently, L PhysNet blocks simulate neural molecular dynamics in the generalized space; I. ChemNet blocks conduct geometry-aware messagg-passing for atoms and bonds. Between each couple of blocks, implicit conformations (qs) and states of chemical bonds (es) are shared as expertise. At the top of PhysNet, a conformation decoder transforms implicit atomic positions into the 3D Euclidean space; of ChemNet, sequence of readout layers aggregate atom states into molecular representations, based on which molecular properties are predicted with task-specific layers. Notably, two specialist networks jointly optimized in PhysChem.
3.2 Initializer
Given a molecular graph M (V,E, m, X", X*), the initializer generates the initial values for both PhysNet and ChemNet variables. In the initializer, we first encode the input features into initial atom and bond states with fully connected layers, i.e.
(0) = FC (z)) i V;
c =C (21,)) (i.5) E
(2)
We then adopt the initialization method in 16 to generate initial positions (o) and momenta (p(") for atoms: a bond strength adjacency matrix A R"** estimated with layers on bond features, according to which GCN captures the chemical environments of atoms (as i): an LSTM then determines unique positions for atoms, especially for those with identical chemical environments (carbons in benzene, for example). Denoted in formula, the initialization follows
activated FC
A(i,))
0,
(i,5) E
GCN (A, v(
(3)
FC sigmoid (
(i,j) E
{(
q
+ P:
((T)) iEV
(4)
where (0))
(0)0
(0) and V -, )m)) are the atom states, The order of atoms
in the LSTM is specified by the canonical SMILES of the molecule.
3.3 PhysNet
Overall, PhysNet simulates the dynamics of atoms in a generalized, dy-dimensional space (dy 2 3). In molecule as a classic dynamical system, atoms move following Newton's Second Law:
dg/di p/m. dp/di. =
(5)
where q. p and m are the position, momentum and mass of atom, and f is the force that the atom experiences. With uniform temporal discretization, the above equations may be approximated with
4++1 9, + PsT/m, Po+1 P. fST,
0, 2,
where is the index of timestamps and 7 is the temporal interval. The calculations in PhysNet simulate such process. Parameters of PhysNet blocks learn to derive the forces f from intermediate molecular conformations and states of chemical bonds. Correspondingly, two types of forces are modeled in PhysNet, namely the positional forces and the torsion forces.
Positional Forces The positional forces in PhysNet model the gravitations and repulsions between pairs of atoms. In conventional molecular force fields, these forces are generally modeled with (negative) derivatives of Lennard-Jones potentials. We therefore propose similar form for the positional forces taking atomic distances as determinants:
f (r T m,,,dd...... Tz.1 ||CC (q: 91)l i L.eV, (
(7)
where dy 4: 9: -q is the unitary directional vector from atom j to i. Instead of using l2 distances in the generalized space, we parameterized interaction distances T estimate the forces, which

---

increase the capability of the network. Here, -- approximates the landscape of derivatives of Lennard-Jones potential with lower-degres polynomials. The approximation is made to avoid numerical issues of using high-degree polynomials, most typically, the gradient explosion.
Torsion Forces The torsion forces model the mechanical effects of torsions in local structures. The torsion forces are defined between pairs of atoms that are directly connected by chemical bonds. Estimating local torsions only with positions and momenta of atoms is somehow suboptimal, as the network lacks prior knowledge of chemical characteristies of the bonds (default bond lengths, for example). Alternatively. we leverage the bond states (et,,) in ChemNet as chemical expertise to derive the torsion forces. Specifically, the torsion forces fo are calculated as
fjM = wy Wy J, Wy, = FC (eg,), (1,5) E
FC
As the torsion forces model chemical interactions, we do not explicitly incorporate atomic mass into the calculation. Notably, atomic information is implicitly considered in the torsion forces, as the bond states integrate atom states at both ends in ChemNet (see the next subsection).
Dynamics After estimating the positional and torsion forces, PhysNet simulates the dynamics of atoms following Equation (6). In the /-th block of PhysNet, S steps of dynamics are simulated:
(1++1)) = 9; -g,"))
+ T (l.))
(l.+11)
(1.a) =p; + +TS(le) 8 = 1, , 1.
li
(9)
mi
Here, 9;
(1,0]
qi 9i
(1+1)
qi (1.))
(p similarly). Si is the resultant force on atom i.e.
f
Σ JEV
+
Σ
(10)
(KI))
Conformation Decoder and Loss We use simple linear transformation to decode implicit atomic positions (and momenta) from the generalized space into the real 3D space:
R = QWaec: dec:
Wdes E Rx
(11)
where Q = (q).... qn)) E Raxdy is the position matrix in the generalized space, and I (f1,.... ,Pr) E =x that of the predicted 3D conformation. We further propose Conn-k (k-hop connectivity loss), a spatially invariant loss that supervises PhysNet based on local distance errors: C(K) E {0, 1 is the k- k-hop connectivity matrix and D the distance matrix then the k- k-hop connectivity loss is defined
1 C(())) o (D- O
D (0-D))(D-1)
LConn-& (R, R) =
(12)
where O the element- wise product, |1-11 is the Frobenius norm, (D, (D, are distance matrices of the real and predicted conformations (R, R), and ((() is the normalized k-hop connectivity matrix & The total loss of PhysNet is defined as the weighted sum of Conn-3 losses on all timestamps:
Lphys
Σ 1.S (1S5+( (L.z)
11
Comm-33 1= 1,2, ... ,,, 8 == 1,2,.....5
(13)
7
1,8
where 7) < 1 is decay factor and Z 0.01 is empirical normalization factor.
3.4 ChemNet
ChemNet conducts saassage-pasinng for both atoms and chemical bonds. Besides generating messages with atom states, ChemNet also considers local geometries including bond lengths and angles to adequately characterize local chemical environments. Specifically, in each ChemNet block, tripler descriptors are established in the atomic neighborhoods. Atoms merge relevant triplet descriptors to
اy = or 112 br -اا دyا
de (12ar - 6br 6b) Sy
(c!(5 if and only if atoms and j are or less-hop connected on the molecular graph.
'Duj ||r: ryll. where E R² are 3D coordinates of atoms.
((()) L (() 11/2 where & is the diagonal degree matrix with Li,i E, c.O
5

---

generate messages, and then aggregate received messages to update their states, Chemical bonds also update their states by aggregating the states of atoms both ends.
Triplet Descriptor The triplet deseriptors lijjk are descriptive vectors defined on all atom triplets
(,,j,k) with (1, & (i, E E
(1) FC
U BO U @ 1. +
ie + )ik) j, (N),,
(14)
where M(2) ( (i.5) E E denotes the atomic neighborhood of i, us are atom states, and L. = FC (I.,) and a FC (cos (4),,,)) are representations of bond lengths and bond angles. The motivation of constructing triplet descriptors is that these features are geometrically deterministic and invariant: i) all bond lengths and angles in an atomic neighborhood together compose a minimum set of variables that uniquely determine the geometry of the neighborhood (deterministic); ii) these features all enjoy translational and rotational invariances (invariant). When PhysNet and ChemNet are jointly optimized, L and Z()
j.i.k: are calculated from the intermediate 3D conformation in the 1-th PhysNet block; when real conformations of target molecules are available, these values can be replaced by the ground-truthe (e.g. ChemNet (real conf.) Section 4
Message-passing After establishing the triplet descriptors, ChemNet generates atomic messages by merging relevant descriptors. The message from atom j i in the (-th block is calculated
MaxPool (f< I (KE)))) j NN),,,
[ = 1,2, ,,
(15)
Subsequently, ChemNet utilizes a similar architecture to 36) to conduct message passing. Centric atoms aggregate the received messages with attention scores determined by the bond states:
(1) m
Σ JEN(()
(OIIEEN()) = softmax (fFC (e(3 1 (sEN()).
(16)
Atom states are then updated with GRU cells that take previous states as recurrent states. similar process is then conducted for all chemical bonds, where messages are generated by the states of atoms at both ends. Denoted in formula,
(1+1) GRUcell
((+1) GRU (e FC (of (1+1) @ BU eij
(17)
The molecular representation is finally read out with T global attentive layers (36). where a virtual meta-atom continuously collects messages from all atoms in the molecule and updates its state. We initialize and update the state of the meta-atom (Umew) following:
Representation Readout
(1) meta
(L)
{a liE v}
softmax (fFC (vii @ of ( | |iEV))
SEV
(0) melaa
FC
Σ 1
((+1) Umeta
= GRUeell
(+) m
1= 1,,, (18)
n1
LEV
The final state of the meta-atom used the molecular representation. Labeled chemical and or biomedical properties are used to supervise the representations, and the loss of ChemNet, Lihees is determined task sesssppccffically. The total loss of PhysChem is the weighted sum of losses the two networks controlled by hyperparameter X, i.e.
T T Aephys + Techem-
(19)
4 Experiments
4.1 Experimental Setup
Datasets and Featurization We evaluated PhysChem on quantum mechanics (QM7, QM8 and QM9) and physical chemistry (LIPOP, FREESOLV and ESOL) datasets, as well drug effectiveness datasets of the notorious SARS-CoV-2. Statisties of the datasets are in Table [ QM7, QM8 and QM9 contain stable organic molecules with up to 7, 8 or 9 heavy atoms. 3D atomic coordinates as

---

Table 1: Statistics of the datasets used in this paper.
Datasets
QM7
QM8
QM9
LIPOP
FREESOLV
ESOL
COVID19
molecules
7.160
21,786 16
133,885 12
4,200
642
1,128 1
14,332
# targets
1
13
Task type
Regression
Regression
Classification Biomedicine No
Category
Quantum Mechanics Yes
Physical Chemistry No
With conformation?
well as electrical properties of molecules were calculated with ab initio Density Functional Theory (DFT). LIPOP provides experimental results on lipophilicity of 4,200 organic compounds: FREESOLV provides experimental and calculated hydration free energy of 642 small molecules in water; ESOL provides water solubility data for 1 1228 compounds. The above six datasets were collected in the ggeeertt MoleculeNet benchmark [35 9 COVID [ is a collection of dataset: [1] generated by screening a
panel of SARS-Co CARS--oV-2-rrlated assays against approved drugs. 13 assays of 14,332 drugs were used in our experiments. We split all datasets into 8:1:1 as training, validation and test sets. For datasets with less than 100, 000 molecules, we trained models for 5 replicas with randomly split data and reported the means and standard deviations of performances, for QM9, we used splits with the same random seed across models. We used identical fcaturization process in (36] to derive feature matrices for all models and on all datasets.
Baselines For conformation learning tasks, we compared PhysChem with i) Distance Geometry method tuned with the Universal Force Fields (UFF) 21, which was implemented in the RDKit package [T| and thus referred to as RDKit; ii) CVGAE and CYGAE+UFF [8] which learned to generate low-energy molecular conformations with deep generative graph neural networks (either with or without UFF tuning); iii) HamEng [6). which learned stable conformations via simulating Hamiltonian mechanies with neural physical engines. For property prediction tasks, we compared PhysChem with i) MoleculeNet, for which we reported the best performances achieved by methods collected in (35) (before 2017); ii) 3DGCN [4]. which augmented conventional GCNs with input bond geometries; iii) DimeNet [15 which conducted directional message- waatgge--ssinggggg by representing
pairs of atoms; iv) Attentive FP
36 which used local and global attentive layers to derive molecular
29 which used communicative kernels to conduct deep message- passing. We conducted experiments with official implementations of HamEng, CVGAE,Attentive FP and CMPNN; for other baselines, identical evaluation schemes adopted, referred to the reported performances in corresponding citations and left unreported entries blank.
representations; and v) CMPNN
PhysChem Variants We also compared PhysChem with several variants, including i) PhysNet (s.a.), a stand-alone PhysNet that ignored all chemical expertise by setting e,, = 0; ii) ChemNet (s.a.), stand-alone ChemNet (ChemNet (s.a.) that ignored all physical expertise by equally setting all bond lengths and angles (L,,) = L, 1,a k a); iii) ChemNet (real conf.) and ChemNet (rdkit conf.), two ChemNet variants that leveraged 4.j and aj,,k in real conformations (real conf.) or RDKit-generated conformations (rdkit conf.); and iv) ChemNet (m.f.), a multi-task ChemNet variant that used straight-forward multi-task strategy for conformation learning and property prediction tasks: we applied the conformation decoder and loss on atom states v and optimized ChemNet with the weighted sum of losses of both tasks (i.e. Lioua in Equation (I9)..
Implementation and Training Details Unless otherwise specified, we used L = 2 pairs of blocks for PhysChem. In the initializer, we used 2-layer GCN and a 2-layer LSTM. In each PhysNet block, we set dy- = 3, S = 4 and r = 0.25. In each ChemNet block, we set the dimensions of atom and bond states as 128 and 64, correspondingly. In the representation readout block, we used T 1 global attentive layers with 256-dimensional meta-atom states. For property prediction tasks with multiple targets, the targets were first standardized and then fitted simultaneously. We the meen-square-erna eeror (MSE) loss for all regression tasks and the cross-entropy loss for all classification tasks to train the models. Other implementation details, such as hyperparameters of baselines, provided in the appendix.
The datasets are publicly http://llw.lae//eeccceloeaaaa/aa
1he datasets are available at htttps://opendata.. http://pppaaaa..a.ee www.at...i..o........o. hiw.aassiii..ooo/iiii...... (CC BY 4.0 license) and are continuously extended. The data used in our experiments were downloaded on February 16th, 2021. We use the 2020.03.1.0 version of the RDKit package at hattp://wwwrrkki/
7

---

w/ conf. Table 2: Performances of conformation learning on QM datasets. Results in more metrics are in the Appendix.
Dataset Metric
QM7 Distance MSE
QM8 Distance MSE
QM9 Distance MSE
Random Guess
2.597+000006
2.631+0.003
2.799
RDKit 2
1.236-00.011
1.635+0.006
1.920
CVGAE 18
1.110+0.001
1.053+00.07
1.052
single-task
CVGAE+UFF 18
1,10910,003
1049:10.008
1,048
HamEng [6
0.636110.036
0.596110.012
0.418
PhysNet (s.u.)
0.504
0.207
0.197
ChemNet (m.z.)
1.034=00002 0.492.10.027
0.690
0.626
multi-task
PhysChem
0.259-10.000
0.255
Table 3: Performances of property prediction on QM datasets. W/ and w/o conf. specify whether conformations of test molecules were leveraged. Italic entries directly referred to from citations. Individual MAEs for separate tasks on QM9 are listed in the Appendix.
Dataset Metric
QM7 MAE
QM8 Multi-MAE
QM9
Multi-MAE
Random Guess
178.8+6644
0.0343+000025
9.392
MoleculeNet 35
94.7+2.7
0.0150+00.0000
2.350
w/o conf.
Attentive FP 36
66.2112.8
0.0130
1.292
ChemNet (s.a.)
59.611..0
0.0105110.000.
1.209
ChemNet (m..f)
60.3112.4
0.0112-10.005
1.639
PhysChem
59.6+2233
0.0003
1.096
DimeNet [T5]
1.920
HamNet [6
1.194
ChemNet (rdlit conf.)
60.1
0.010011.0000
1.140
ChemNet (real conf.)
60.2+1.9
0.0098=000003
1.040
4.2 Results
Quantum Mechanical Datasets As real conformations available in the QM datasets, evaluated PhysChem on both conformation learning and property prediction tasks. On conformation learning tasks, Distance MSE metric is reported, which sums the squared errors of all pair-wise distances in each molecule, normalizes it by the number of atoms, and then takes averages across molecules. Note that this metric is equivalent to the Conn-o: loss with k oo in Equation 12). On property prediction tasks, the mean-absollte-erroot (MAE) for regression targets are reported. When multiple targets are requested (on QM8 and QM9), we report the Multi-MAE metric in [16) which sums the MAEs of all targets (standardized for QM9). Table 2 and show the results. On conformation learning tasks, PhysNet (s.a.) displays significant advantages on learning conformations of small molecules. Specifically, the comparison between PhysNet (s.a.) and HamEng indicates that directly learning forces neural physical engines may be superior learning energies and their derivatives. With massive data samples (QM9), the specialist, PhysNet (s.a.), obtains better results than PhysChem; while on datasets with limited samples (QM7), chemical expertise demonstrates its effectiveness. On property prediction tasks, PhysChem obtains state-of-the-art performances on QM datasets. The comparison between PhysChem, ChemNet (s.a.) and ChemNet (m.I.) shows that the elaborated cooperation mechanism in PhysChem is necessary, the straight-forward multi- task strategy leads to severe negative transfer. addition, the results of ChemNet (real conf.) and ChemNet (rdkit conf.) show that leveraging real (or generated, geometrically correct) conformations of test molecules indeed helps on some datasets, while the elevations are somehow insignificant.
Physical Chemical & Biomedical Datasets On LIPOP, FREESOLV, ESOL and COVID19 with no labeled conformations, we used EDKKit-eneraat conformations to satisfy models that requested conformations of training and or test molecules. Although these generated conformations are less accurate, distance geometries local structures are generally correctly displayed. We report (RMSE) for regression tasks, and the multi-class ROC-AUC (Multi- AUC) metric on COVID 19. Table shows the results. PhysChem again displays state-of-the-arr performances. Notably, the numbers of samples in physical chemical datasets (LIPOP, FREESOLV
segeeddean--quared--eee

---

Table 4: Performances of property prediction on LIPOP, FREESOLV, ESOL, and COVID19.
Dataset Metric
Lipophilicity RMSE
FreeSolv RMSE
ESOL RMSE
SARS-CoVe
Multi-AUCT
MoleculeNet
35
0.655
1.150
0.580
Attentive FP 36
0.509-00.036
0.962++0.17
0.612-0.027
0.695+0.000
w/o conf.
CMPNN 29
0.80810.129
0.547-10.011
0.69210.0007
ChemNet (s.u.)
0.517:10.016
0.730:1:.227
0..01
0.751
PhysChem
0..08
0.692.10.112
0,499
0.79210.011
3DGCN 4
0.82440.014
0.580=00.069
w/ conf.
HamNet [6]
0.557:10.014
0.0014
L0.011
ChemNet (ndkit conf.)
0.532.10.011
0.663-10.029
0.504 0.003
0.090
Centarates Eeiring Propetty Prccition
0.36
1.2
Digaee MS5
0:28
0.
HamEing
PhysChem
PhysChem"
PhysChen?
(a) Effect of hyperparameter X.
(b) Visualization results on QM9,
Figure 2: Further analyses of PhysChem.
and ESOL) are sparse, the conformation learning task for PhysNet is comparatively tough. This leads to a larger gap performances between PhysChem and ChemNet (rdkit conf.), yet the former is still better than most baselines. The largest elevation of performances is observed on COVID19 19, which indicates that PhysChem is further competent on more complicated, structure-dependent tasks.
Further Analyses Figure showsws the effect of the loss hyperparameter on QM9. When À S 0.1, increasing benefits both tasks. This indicates that aiding the learning of conformations also helps to predict chemical properties. With larger X, the property prediction task is then compromised. Figure 2(b) visualizes the predicted conformations of baselines and PhysNet blocks. Local structures such as bond lengths, angles and planarity of aromatic groups are better preserved by PhysChem.
Conclusion and Future Work
In this paper, we propose a novel neural architecture, PhysChem, that learns molccular representations via fusing physical and chemical information. Conformation learning and property prediction tasks are jointly trained in this architecture. Beyond straight-forward multi-task strategies, PhysChem adopts elaborated cooperation mechanism between specialist networks. State-of-the-an art performances were achieved on MoleculeNet and SARS-CoV.3 datasets. Nevertheless, there is still much space for advancement of PhysChem. For future work, straight-forwad improvement is to enlarge the capability of the model by further simplifying as well as deepening the architecture of PhysChem. In addition, proposing strategies to train PhysChem with massive unlabeled molecules is yet another promising direction.
For the machine learning community, work proposes more interpretable architecture on molecular machine learning tasks and demonstrates its effectiveness. We hope that the specialist networks and domain-related cooperation mechanism in PhysChem will inspire researchers in a wider area of deep learning to develop novel architectures under the same motivation. For the drug discovery community, PhysChem leads to direct applications ligand-related tasks including conformation and property prediction, pooteinlligame binding affinity prediction, etc. With limited possibility, the abuse of PhysChem in drug discovery may violate some ethics of life science.
Broader Impact
Acknowledgments This work was supported by the National Natural Science Foundation of China (Grant No. 61876006).

---

References
[1] S. D. Axen, X. P. Huang, E. L. Cáceres, L. Gendelev, B. L. Roth, and M. J. Keiser. A simple representation of three-dimensional molecular structure. Journal of medicinal chemistry, 60(17):7393 2017.
[2] Jeffrey M. Blaney and J. Scott Dixon. Distance Geometry Molecular Modeling, pages 299-335. John Wiley & Sons, Ltd, 2007.
[3] Kyle R Brimacombe, Tongan Zhao, Richard T Eastman, Xin Hu. Ke Wang, Mark Backus, Bolormaa Baljinnyam, Catherine Z Chen, Lu Chen, Tara Eicher, Marc Ferrer, Ying Fu, Kirill Gorshkov, Hui Guo, Quinlin M Hanson, Zina Itkin, Stephen C Kales, Carleen Klumpp-Thowe Thomas, Emily M Lee, Sam Michael, Tim Mierzwa, Andrew Patt, Manisha Pradhan, Alex Renn, Paul Shinn, Jonathan H Shrimp, Amit Viraktamath, Kelli M Wilson, Miao Xu, Alexey V Zakharov, Wei Zhu, Wei Zheng, Anton Simeonov, Ewy A Mathé, Donald C Lo, Matthew D Hall, and Min Shen. An OpenData portal to share COVID-19 drug repurposing data in real time. bioRxiv, 2020 Jun 5(2020.06.0) 135046), 2020.
[4] H. Cho and I. Choi. Enhanced deep-learning prediction of molecular properties via augmen- tation of bond topology. ChemMedChem, 14(1 (6(17):1604-16000 2019.
[5] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), pages 1724-1734, 2014.
[6] Michael Crawshaw. Multi-task learning with deep neural networks: A survey. CoRR, abs/2009.09796. 2020.
[7] David Duvenaud, Dougal Maclaurin, Jorge Aguilera- Iparraguiree, Rafael Gomez-Bombarelll Timothy Hirzel, Alán Espuru-Guzike and Ryan P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015 (NeurIPS 2015), pages 2224 2232, 2015.
[8] Francesco Gentile, Vibudh Agrawal, Michael Hsing, Anh-Tien Ton, Fuqiang Ban, Ulf Norinder, Martin E. Gleave, and Artem Cherkasov. Deep docking: deep learning platform for augmen- tation of structure based drug discovery. ACS Central Science, 6(6):939 9099999999999999999199 2020.
[9] Justin Gilmer, Samuel Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning (ICMI 2017), pages 1263-1272, 2017.
[10] Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(3: 11 1997.
[11] Shion Honda, Shoi Shi, and Hiroki R. Ueda. SMILES transformer: Pre-trained molecular fingerprint for low data drug discovery. CoRR, (bs//991.00433/ 2019.
[12] Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In Proceedings of the 35th International Conference on Machine Learning, (ICML 2018), 8), pages 2328 2337, 2018
[13] Derek Jones, Hyojin Kim, Xiaohua Zhang, Adam Zemla, Garrett Stevenson, William D. Bennett, Daniel A. Kirshner, Sergio E. Wong, Felice C. Lightstone, and Jonathan E. Allen. Improved protein-ligand binding affinity prediction with structure-based deep fusion inference. Journal of chemical information and modeling, 61(4):1583-1592 2021.
[14] Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017), 2017.
10

---

[15] Johannes Klicpera, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), 2020.
[16] Ziyao Li, Shuwen Yang, Guojie Song, and Lingsheng Cai. Conformation-guicl guided molecular representation with hamiltonian neural networks. In Proceedings of the 9th International Conference on Learning Representations (ICLR 2021), 2021.
[17] Denghui Lu, Han Wang, Mohan Chen, Lin Lin, Roberto Car, Weinan E, Weile Jia, and Linfeng Zhang. 86 PFLOPS deep potential molecular dynamics simulation of 100 million atoms with ab initio accuracy. Comput. Phys. Commun., 259: 237:14,, 2021.
[18] Elman Mansimov, Omar Mahmood, Seokho Kang, and Kyunghyun Cho. Molecular geometry prediction using a deep generative graph neural network. Scientific Report, 9(1,, 2019.
[19] H. L. Morgan. The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service. Journal of chemical documentation, 5(2):63- 112, 1965.
[20] Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Güinter Klambauer. Fréchet ChemNet Distance: A metric for generative models for molecules in drug discovery. Journal of chemical information and modeling. 58(9): (119):736-1744 1746611711111 2018.
[21] A. Rappé, C. Casewit, K. Colwell, W. Goddard, and W. M. Skiff. Uff, a full periodic table force field for molecular mechanics and molecular dynamics simulations. Journal of the American Chemical Society, 11::10024-10035, 1992.
[22] D. Rogers and M. Hahn. Extended-conmecctvvty fingerprints. Journal of chemical information and modeling, 50(5):742-754, 2010.
[23] Alina Roitberg, Tim Pollert, Monica Haurilet, Manuel Martin, and Rainer Stiefelhagen. Analysis of deep fusion strategies for multi-modal gesture recognition. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops 2019), pages 198-206, 2019.
[24] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS 2020), 2020.
[25] Julian Schwartz, Mahendra Awale, and Jean-Louis Reymond. Smifp (smiles fingerprint) chemical space for virtual screening and visualization of large databases of organic molecules. Journal of Chemical Information and Modeling, 5378):197-1198,, 2013.
[26] Andrew W. Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Zidek, Alexander W. R. Nelson, Alex Bridgland, Hugo Penedones, Stig Petersen, Karen Simonyan, Steve Crossan, Pushmeet Kohli, David T. Jones, David Silver, Koray Kavukeuoglu, and Demis Hassabis. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):00-7110 2020.
[27] Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In Proceedings of the 38th International Conference on Machine Learning (ICML 2021), 2021.
[28] Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow-based autoregressive model for molecular graph generation. In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), 2020.
[29] Ying Song, Shuangjia Zheng, Zhangming Niu, Zhang- Hua Fu, Yutong Lu, and Yuedong Yang. Communicative representation learning on attributed molecular graphs. In Proceedings of the Ninth International Joint Conference on Artificial Intelligence, (ICAI 2020), pages 2831 1238,2238,222 2020.
[30] S. Sosnin, M. Vashurina, M. Withnall, Pavel Karpov, Maxim V. Fedorov, and Tetko. A survey of multi-task learning methods in chemoinformatics. Molecular Informatics, 38, 2019.
11

---

[31] Garrett A. Stevenson, Derek Jones, Hyojin Kim, W. F. Drew Bennett, Brian J. Bennion, Monica Borucki, Feliza Bourguet, Aidan Epstein, Magdalena Franco, Brooke Harmon, Stewart He, Max P. Katz, Daniel A. Kirshner, Victoria Lao, Edmond Y. Lau, Jacky Lo, Kevin Mcl ougghin, Richard Mosesso, Deepa K. Murugesh, Oscar A. Negrete, Edwin A. Saada, Brent Segelke, Maxwell Stefan, Marisa W. Torres, Dina Weilhammer, Sergio E. Wong, Yue Yang. Adam Zemla, Xiaohua Zhang, Fangqiang Zhu, Felice C. Lightstone, and Jonathan E. Allen. High-throughput virtual screening of small molecule inhibitors for sars-cov-2 protein targets with deep fusion models. CoRR, abs/2 144.044777 2021.
[32] Oliver T. Unke and Markus Meuwly. PhysNet: neural network for predicting energies, forces, dipole moments, and partial charges. Journal of chemical theory computation, 15(6):3678-3695 2019.
[33] Zhihua Wang, Stefano Rosa, Bo Yang, Sen Wang, Niki Trigoni, and Andrew Markham. 3D- PhysNet: Learning the intuitive physies non-rigid object deformations. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (ICAI 2018), pages 4958-4964, 2018.
[34] D. Weininger. Smiles 1. introduction and encoding rules. Journal of chemical information and computer sciences, 28(31), 1988.
[35] Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science. 9(2):133-538) 2017.
[36] Z. Xiong. D. Wang, X. Liu, F. Zhong, X. Wan, X. Li. Z. Li, X. Luo, K. Chen, H. Jiang, and M. Zheng. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 63(16))8749-8777 2020.
[37] Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay S. Pande, and Jure Leskovec. Graph convolu- tional policy network for goal-directed molecular graph generation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018 (NeurIPS 2018), pages 6412-6422, 2018.
[38] Linfeng Zhang, Jiequn Han, Han Wang, Roberto Car, and E. Weinan. Deep potential molecular dynamics: A scalable model with the accuracy of quantum mechanics. Physical review letters, 1 120(14), 2018.
[39] Yu Zhang and Qiang Yang. A survey multi-task learning. CoRR, abs/1 707.707.000 00710011152 11, 2017.
12


Reviews:
[{"review_id": "mdprm5c_bEs", "reviewer": "Reviewer_tYCt", "summary": "This paper reports the use of a multi-task/model fusion approach to molecular representation learning: one NN learns to generate conformations through Hamiltonian dynamics with learned forces and another NN does supervised message-passing in an established way\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PROs\nThe architecture fusion is novel and seems effective. By making 3D information a label rather than an input the models are more likely to be transferable, even when there are no existing geometries to train on, as is often the case in large chemical libraries or new chemical spaces\n\nLearning forces directly makes energy not conserved, but it's fine for this application since there is no real \"energy\" being learned, and it's just a pseudo-energy (and pseudo-forces) to reach the desired geometries. I expect learning straightup pseudo-forces will become a standard in the field\n\nThe partitioning of pairwise interactions into the modified LJ and torsions coming from the Chemical Net is new and intereseting but it is not clear whether it's useful or important. A many-body MPNN model like MPNN (Schnet) could learn those terms without crosstalks between the models. I'd be intereseted in learning more about the role of the torsional terms, whether the can be ablated or replaced by learnable manybody MPNN terms in the potential and at what cost in performance / cost. \n\nCONs\nThe architectures themselves are not very innovative.\n\nThe RDKit RMSD for QMX data has me very worried. Most papers report around 0.3 RMSD between RDKit and DFT (and it has be low since the QMX geometries typically come from relaxing initial guesses produced with RDKit) (See arXiv:2105.07246v1 or any of the work from Jian Tang, for example) The values in table 2 and 3 are way too high, and raise concerns about how RMSD is being calculated (atom indexing issues?)\n\nWhat is a \"random\" guess for a geometry? Just sprinkling atoms in space will sure give more error than 2 A. \n\nThe justification for the low-order polynomial \"Lennard Jones\" is not great, but at the of the day it's an empirical choice, so it's fine. \n\nThe baselines for property prediction are a bit old, so it is not clear how the models stack. I'd encourage the authors to refer to more recent architectures for property prediction, including modern equivariant ones. \n\n\nSmall remarks:\n\"gravitations\" for \"attractions\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 h"}, "extra_text": {}}, {"review_id": "aan5iAddsKj", "reviewer": "Reviewer_HEMY", "summary": "The authors are interested in the linked tasks of predicting (i) 3D conformations and (ii) chemical/biomedical properties of molecules. They show that by designing a model (PhysChem) that tackles both of these tasks simultaneously, they can often do better on each individual task than when these tasks are considered separately. This is empirically demonstrated on the QM{7,8,9} datasets, the MoleculeNet  Physical Chemistry Benchmark [32], as well as a SARS-CoV-2 dataset [3], where the authors also compare to a variety of older approaches.\n\n\n\n\n### Further Details on the Architecture\nTo be more specific about the architecture, PhysChem consists of two networks (which share information at each layer and both take in as input the graph representation of a molecule):\n1. PhysNet, is used to predict the conformation of a molecule and it does this by first predicting the initial positions of the atoms (using a graph convolutional network (GCN) and RNN) before then updating these positions using the equations of motion under the action of interatomic forces output by a fully connected NN.\n2. ChemNet†, is used to predict chemical/biomedical properties of a molecule, and is a new variant of a message passing neural network [9].\n\n† This is a new network, not to be confused with previous networks also called \"ChemNet\", e.g.\nPreuer, K., Renz, P., Unterthiner, T., Hochreiter, S. and Klambauer, G. (2018) ‘Fréchet ChemNet Distance: A Metric for Generative Models for Molecules in Drug Discovery’, Journal of chemical information and modeling, 58", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**SUMMARY** \nI think the authors tackle an interesting problem, that of producing better representations of molecules by simultaneously training on different (but linked) tasks, and appear to achieve impressive empirical results. PhysNet alone is shown to outperform previous methods for predicting conformations. However, the proposed networks consist of several moving parts, and the architectural decisions seem fairly arbitrary at times; it would have been nice here to have some intuition or empirical/theoretical justification of the choices made (see limitations box below). Therefore, currently I have gone with a borderline acceptance score but am happy to raise this score if this point gets addressed in the rebuttal.\n\n\n\n## Originality\ni. The authors produce a novel architecture for predicting both molecular conformations and properties at the same time. Overall, the approach bears some similarities to HamNet [16], for instance the use of a neural physical engine to predict conformations along with the idea of a supplementary MPNN-like architecture to produce a molecular fingerprint. However, there are also some key differences (described in lines 81-91) such as the idea of parameterizing the forces directly or the idea of receiving information back from the MPNN into the physical engine. This ultimately results in better empirical results (Tables 1-3).\n \n## Quality\n\nii. As far as I can tell the submission is technically sound. The claims are supported empirically and state of the art performance is obtained (compared to some relatively recent methods, e.g. [16] at ICLR 2021). However, this is evaluated in terms of MSE/MAE and it would be nice to also have a comparison in terms of number of parameters/compute costs. There is actually some work in this direction in Appendix H, which shows PhysChem is competitive in this regard, but it would be nice to have this extended to the other models and brought into the main paper.\n\n\n## Clarity\n\niii. The approach is quite involved but, considering this, I thought that the authors did a good job at explaining the various steps involved. In particular, Figure 1 provides a helpful overall view of the approach and algorithms 1-3 (in the appendix) provide detailed descriptions of the individual steps involved. The authors provide code which I have glanced through. This is helpful but perhaps could be tidied up a little (e.g. there are a lot of top-level files and I am usure which are important, and also installation instructions should be provided in the README.md)\n\niv.One aspect relating to clarity that could be improved is better motivation of the choices made. Some of the architectural decisions seem a little ad-hoc and it would be nice to have some intuition for them. For instance, I'm confused as to the motivation behind using Lennard-Jones potentials (line 153) and then the subsequent simplification of these with lower-degree polynomials (line 158). It would be nice to have a more detailed background of this choice for a ML audience in the appendix. (I provide other examples of architectural decisions for which I think further explanation could be provided in the limitations box below.)\n\nv. On a much more minor point, the grammar/choice of words could perhaps be improved in a few places. However, this should be easy to fix. Here are a few examples:  \n    a. Line 66: \"Models that leveraged 3D geometries of molecules generally performed better than those simply used graph notations\" => \"those _that_ simply used graph notations\"?  \n    b. Line 145-146: \"and f is the force that the atom is compelled\" => \"the force that the atom _experiences_.\"...?  \n    c. Line 35: \"For example, torsions of bond lengths and angles greatly influence the dynamics the of particle systems\" => I think the \"the\" is unnecessary...?  \n\n\n## Significance\nvi. The results improve upon state of the art and therefore I believe may be of interest to other researchers or practitioners. Ablation tests are performed to distinguish the performance improvements due to the PhysNet vs ChemNet but I still am a bit confused as to what components of these individual networks are important (see limitations box below).\n\n\n## Other Miscellaneous Questions to the Authors\n\nvii. For the molecular properties tasks (e.g. Table 3 and 4) it appears much of the gains come down to the architecture of the MPNN network used, rather than the use of conformational information inside this network. Is this fair? Why do you think this network works better than the previous ones compared to? \n\nviii. Could PhysNet be used as an addition to other networks in the MPNN framework?\n\nix. I was surprised that only 4 time steps were used in estimating the position/momenta of the atoms in PhysNet. This seems rather low (for instance, in HamNet [16] my understanding is that they use the equivalent of 20 such steps)? Do the positions and momenta converge after so few steps?\n\nx. In the main paper (line 263) you state you set $d_f=128$, yet in the appendix (line 50) this is said to actually be 3? Which value did you actually use? I'm also having trouble reconciling the results in Figure S2 (of the Appendix) with Table 2 of the main paper. For instance, (assuming $d_f=3$) Figure S2 states the results on conformation learning on QM7 are ~0.52; however, in Table 2 they are stated at ~0.49. I would be grateful if you could point out where I am going wrong?\n\nxi. In addition to reporting multi-MAE in the main paper it would be nice to have the individual MAEs for the separate tasks in QM9 in the appendix if possible (for consistency with previous work, e.g. [32, Table S9; 33, Table ST5]).\n\nxii. In figure 2, what do the superscripts for the different PhysChem models represent (e.g. 0,1,2)? (sorry if I missed this).\n\nxiii The LSTM (Eq 4) outputs specific atom coordinates, but I believe the ultimate loss is invariant to rotations/translations of these coordinates? Does this mean the LSTM obtains the same loss even if it arbitrarily translates all initial positions by the same amount? Why not fix the first atom's initial position always to the origin?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "## Limitations\nA. My understanding is that the approach proposes several modifications to the approach used in HamNet [16] but it wasn't clear to me which of these are important to obtain the improved results/why these changes have been made. Examples include:\n1. PhysNet predicts the forces directly rather than the energies  (as in HamNet [16]).\n2. PhysNet obtains back from ChemNet MPNN information (in the form of \"torsion forces\") -- in HamNet the information only flows from the physical engine to the MPNN.\n3. PhysNet uses a different conformation decoder loss (Eqn 13).\n4. ChemNet uses a bespoke message passing scheme, relying on messages that are computed from triplets of graph nodes (as opposed to the ordinary use of pairs -- Section 3.4).  \n\nIt would be good if empirical evidence/better motivation of these choices was provided.\n\nB. At a higher-level, it would be nice if the general idea of sharing information between conformational and chemical property prediction networks is shown to be useful in a way that is agnostic to the specific architectures used for the separate networks. At the moment this contribution seems somewhat muddled up with the contributions of providing new (and somewhat bespoke) network architectures for these tasks.\n\nC. It actually seems (Table 3) that sharing the information from ChemNet into PhysNet actually degrades performance on the larger datasets QM8/QM9. I would be interested to hear why PhysNet cannot ignore this information if it is not helpful? This also seems to be in conflict with the central claim of the paper (that sharing information between these different tasks is helpful)?\n\n\n## Societal impact\nD. The authors include a Broader Impact subsection in section 5. Here the authors state: \"With limited possibility, the abuse of PhysChem in drug discovery may violate some ethics of life science.\" However, I admit to not understanding the specific potential negative societal impacts that they envisage and would have found an explicit example helpful.\n\nE. In this section the authors also state that their approach provides a more interpretable architecture. However, I would have liked more details on what they meant/some evidence backing up such claims.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "WA2nWkN6AjS", "reviewer": "Reviewer_1R5X", "summary": "The authors proposed PhysChem network for molecule representation. PhysChem consists of four modules, the initializer, the PhysNet, ChemNet and readout. Specifically, in PhysNet, the position, momentum and mass of an atom are modeled using rules in mechanics. The authors conduct a series of experiments to verify the proposed algorithm.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tThere are many previous work on QM9. You can find the results by searching \n(1)\thttps://paperswithcode.com/sota/formation-energy-on-qm9. \n(2)\thttps://paperswithcode.com/sota/drug-discovery-on-qm9\nAnd QM9 has lots of (sub)tasks. I have two questions towards this point:\n(i)\tIn Table 2, 3, 4, do you use the same way to get the training/validation/test data?\n(ii)\tA comparison with [ref1] should be discussed.\n\n2.\tSome ablation/discussion are required:\na.\tWhat if you use Dihedral angle like [15]?\nb.\tIn PhysNet, do we need both Positional Force and Torsion Forces? What if we use only one of them?\n\n3.\tI’m a little confused by your results. In Table 2, why not provide the results of PhysChem? In Table 3, w/ conf, what if we use PhysChem? More surprisingly, on QM7, ChemNet (s.a.) achieves the best results even without conf information.\n\n4.\tThe RDKit generated conformations are not accurate but useful as illustrated in Table 3. What if we start from RDKit and then finetuning using your proposed method?\n\n[ref1] Molecular Mechanics-Driven Graph Neural Network with Multiplex Graph for Molecular Structures, https://arxiv.org/pdf/2011.07457v1.pdf\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "9-7UkomB7lG", "reviewer": "Reviewer_2R2G", "summary": "This paper presents a novel method called PhysChem for molecular representation learning by fusing physical and chemical information of molecules. PhysChem consists of two cooperating neural networks, one called PhysNet and another called ChemNet. PhysNet learns 3D conformations by simulating molecular dynamics with parametrized forces, while ChemNet is a geometry-aware message passing network. The experiment results for MoleculeNet benchmark and SARS-Cov-2 datasets showed superior performance over the other state-of-the-art baselines.\n\nGNNs or Geometric NNs are mostly separated to 3D-geometry-aware ones used for quantum-calculation surrogates and 2D-abstraction ones for QSARs predicting properties such as toxicity and solvability. Representation learning that integrates these two aspects is very interesting and promising.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very interesting paper that integrates two different aspects of molecules that have been often studied separately. The structure of molecules are quite compositional/combinatorial (e.g. bonds, functional groups, fragments, scaffolds, etc), which motivates us to directly input graphs into neural networks as we seen in GNNs; On the other hand, a molecules exhibits its function in 3D physical space, and act as particle systems that follows the laws of (quantum) mechanics, which is physically constrained rather than being handled as a pure combinatorial object.\n\nAs shown in recent studies such as [16](ICLR2021) and [26](ICML2021), neural networks alone can potentially simulate molecular dynamics for conformation prediction. It is thus quite natural to expect that these networks capture physical aspects of molecules in 3D. So, the paper's idea is to learn PhysNet and ChemNet simultaneously: PhysNet uses ChemNet features to have torsion forces in chemical bonds, while ChemNet uses bond-length and bond-angle features from PhysNet 3D conformation. Doing like this, PhysChem is applicable to the situations where the task would be 3D-conformation related, but such conformation information is not directly available. (like a recent KDDCup2021 \"PCQM4M-LSC\" task to predict DFT-calculated properties from 2D molecular graphs?)\n\nOverall I liked the idea, and but here are some comments.\n\n- For \"PhysChem (w/o conf.)\" in Table 3 and Table 4, the paper uses UFF-based conformation generation of RDKit for supervised training. Are these conformations unique for each molecule?   \nThe RDKit document says that \"Since the 2018.09 release of the RDKit, ETKDG is the default conformer generation method\", and it looks to be able to generate multiple conformations by running the distance geometry calculation multiple times from different random start points. These multiplicities don't affect the performance of PhysChem?\n\n- In Table 3, I'm not sure why ChemNet (standalone) can be better than or equal to ChemNet (multitask), PhysChem, and other baselines for QM7 and QM9. QM7 and QM9 is basically geometric-aware tasks, for example, energy is changed if the xyz coordinates are changed. In this sense, if we dropped 3d information as in ChemNet(standalone), it can be hard to predict these quantities. Do you have any rational explanations for this?\n\n- Similarly, what makes PhysNet(s.a) better than HamEng? Without torsion forces from ChemNet, apparently PhysNet would have similar-level features, or even simpler and rough approximation of Lennard-Jones potentials. What brings us this improvement over HamEng?\n\n- For ChemNet, the \"triplet descriptors\" are unique features, but the remaining would be within the scope of the standard customized MPNNs. So what makes this difference, for example, over MoleculeNet? The ChemNet architecture looks like a variant of gated graph neural networks (GGNNs), and how good if we just grab a standard GGNN here (for Table 3 and 4)?\n\n- Is there any risk of leakage by intensive hyper-parameter tuning for PhysNet, ChemNet, PhysChem? How these results are dependent on careful tuning of hyperparameter tuning?\n\n- The name \"PhysNet\" might be very confusing because there already exists a widely-cited model called \"PhysNet\" also applied to QM9 prediction.  \n  \nPhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments, and Partial Charges (2019)\nhttps://doi.org/10.1021/acs.jctc.9b00181  \nFor related work, check an ICML2021 paper https://arxiv.org/abs/2102.09844\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are partly described in the Conclusion section.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors proposed a new neural network model called PhysChem for molecule representation. In PhysChem, a PhysNet and a ChemNet interact with each other to facilitate the learning process. The authors conduct a series of experiments to verify the practical effectiveness proposed algorithm. Overall speaking, the reviewers are positive on this paper. They like the intention of the authors to integrate physical and chemical knowledge in the design of the neural network. But on the other hand, the reviewers also raise some concerns, including the experimental setting, the role of PhysNet, etc. The authors did a good job in addressing most of the concerns, and the consensus made among the reviewers is more leaning towards the positive side. Therefore my recommendation is ACCEPT as a poster.
Decision: Accept (Poster)