Deep Molecular Representation Learning via Fusing Physical and Chemical Information
Year: 2022

Abstract:
Molecular representation learning is the first yet vital step in combining deep learning and molecular science. To push the boundaries of molecular representation learning, we present PhysChem, a novel neural architecture that learns molecular representations via fusing physical and chemical information of molecules. PhysChem is composed of a physicist network (PhysNet) and a chemist network (ChemNet). PhysNet is a neural physical engine that learns molecular conformations through simulating molecular dynamics with parameterized forces; ChemNet implements geometry-aware deep message-passing to learn chemical / biomedical properties of molecules. Two networks specialize in their own tasks and cooperate by providing expertise to each other. By fusing physical and chemical information, PhysChem achieved state-of-the-art performances on MoleculeNet, a standard molecular machine learning benchmark. The effectiveness of PhysChem was further corroborated on cutting-edge datasets of SARS-CoV-2.

heading: 1 Introduction
text: The intersection between deep learning and molecular science has recently caught the eye of researchers in both areas. Remarkable progress was made in applications including molecular property prediction [36, 16], molecular graph generation [37, 12, 28], and virtual screening for drug discovery [8, 31], yet learning representations for molecules remains the first yet vital step. Molecular representation learning, or learning molecular fingerprints, aims to encode input notations of molecules into numerical vectors, which later serve as features for downstream tasks. Earlier deep molecular representation methods generally used off-the-shelf network architectures including message-passing neural networks [9], graph attention networks [36] and Transformers [11, 24]. These methods took either line notations (e.g. SMILES3) or graph notations (i.e. structural formulas) of molecules as inputs, whereas physical and chemical essence of molecules was largely neglected. Notably, a trend of integrating 3D conformations (i.e. the 3D Cartesian coordinates of atoms) into molecular representations recently emerged [4, 15, 29], while most of these methods assume the availability of labeled conformations of target molecules.
In order to push the boundaries of molecular representation learning, we revisited molecules from both physical and chemical perspectives. Modern physicists generally regard molecules as particle systems that continuously move following the laws of (quantum) mechanics. The dominant conformations of molecules reflect the equilibriums of these micro mechanical systems, and are thus of wide interest.
∗Equal Contribution. †Corresponding Author. 3SMILES (Simplified Molecular Input Line Entry Specification [34]) is a widely used protocol that specifies
(non-unique) line notations for molecules, CCO for ethanol, for example.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Chemists, on the other hand, focus more on chemical bonds and functional groups of molecules, which denote the interactions of electrons and determine chemical / biomedical properties such as solubility and toxicity, etc. Nevertheless, physical and chemical information of molecules is not orthogonal. For example, torsions of bond lengths and angles greatly influence the dynamics of particle systems. Therefore, an ideal molecular representation is not only expected to capture both physical and chemical information, but also to appropriately fuse the two types of information.
Based on the observations above, we propose PhysChem, a novel neural architecture that captures and fuses physical and chemical information of molecules. PhysChem is composed of two specialist networks, namely a physicist network (PhysNet) and a chemist network (ChemNet), who understand molecules physically and chemically.4 PhysNet is a neural physical engine that learns dominant conformations of molecules via simulating molecular dynamics in a generalized space. In PhysNet, implicit positions and momenta of atoms are initialized by encoding input features. Forces between pairs of atoms are learned with neural networks, according to which the system moves following laws of classic mechanics. Final positions of atoms are supervised with labeled conformations under spatial-invariant losses. ChemNet utilizes a message-passing framework [9] to capture chemical characteristics of atoms and bonds. ChemNet generates messages from atom states and local geometries, and then updates the states of both atoms and bonds. Output molecular representations are merged from atomic states and supervised with labeled chemical / biomedical properties. Besides focusing on their own specialties, two networks also cooperate by sharing expertise: PhysNet consults the hidden representations of chemical bonds in ChemNet to generate torsion forces, whereas ChemNet leverages the local geometries of the intermediate conformations in PhysNet.
Compared with existing methods, PhysChem adopts a more elaborated as well as interpretable architecture that incarnates physical and chemical understandings of molecules. Moreover, as PhysNet learns molecular conformations from scratch, PhysChem does not require labeled conformations of test molecules. This extends the applicability of PhysChem to situations where such labels are unavailable, for example, with neural-network-generated drug candidates. We evaluated PhysChem on several datasets in the MoleculeNet [35] benchmark, where PhysChem displayed state-of-the-art performances on both conformation learning and property prediction tasks. Results on cutting-edge datasets of SARS-CoV-2 further proved the effectiveness of PhysChem.

heading: 2 Related Work
text: Molecular Representation Learning Early molecular fingerprints commonly encoded line or graph notations of molecules with rule-based algorithms [19, 22, 25]. Along with the spurt of deep learning, deep molecular representations gradually prevailed [7, 9, 11, 36]. More recently, researchers started to focus on incorporating 3D conformations of molecules into their representations [1, 4, 29, 15]. Models that leveraged 3D geometries of molecules generally performed better than those that simply used graph notations, whereas most 3D models required labeled conformations of the target molecules. This limited the applicability of these models. Among previous studies, message-passing neural networks (MPNNs) proposed a universal framework of encoding molecular graphs, which assumed that nodes in graphs (atoms in molecules) passed messages to their neighbors, and then aggregated received messages to update their states. A general message-passing layer calculated
mi = ∑
j∈N (v)
M(hi,hj , ei,j), hi ← U(hi,mi), i ∈ V, (1)
where i, j ∈ V were graph nodes, hs and es were states of nodes and edges, and M(·), U(·) were the message and update functions. For graph-level tasks, MPNNs further defined a readout function which merged the states of all nodes into graph representations. In this work, ChemNet modifies the message-passing scheme of MPNNs: messages are extended to incorporate geometries of local structures, and states of both atoms and bonds are updated. See Section 3.4 for more details.
Neural Physical Engines Recent studies showed that neural networks are capable of learning annotated (or pseudo) potentials and forces in particle systems, which made fast molecular simulations [38, 17] and protein-folding tasks [26] possible. Notably, it was further shown in [16, 27] that neural networks alone can simulate molecular dynamics for conformation prediction. As an instance,
4PhysNet and ChemNet are novel architectures, not to be confused with previous works with similar or identical names [33, 32, 20].
HamNet [16] proposed a neural physical engine that operated on a generalized space, where positions and momentums of atoms were defined as high-dimensional vectors. In the engine, atoms moved following Hamiltonian Equations with parameterized kinetic, potential and dissipation functions. PhysNet in our work is a similar engine. Nevertheless, instead of learning parameterized energies and calculating their negative derivatives as forces, we directly parameterize the forces between each pair of atoms. In addition, HamNet considered gravitations and repulsions of molecules based on implicit positions, while it ignored the effects of chemical interactions: for example, the types of chemical bonds were ignored in the energy functions. PhysChem fixes this issue via the cooperation mechanism between two specialist networks. Specifically, PhysNet takes chemical expertise (the bond states) from ChemNet and introduces torsion forces, i.e. forces that origin from torsions in chemical bonds, into the dynamics. See Section 3.3 for more details.
Multi-Task Learning and Model Fusion The cooperation mechanism in PhysChem shares similar motivations with multi-task learning and model fusion. Multi-task learning [39, 6] is now almost universally used in deep learning models. Representations are shared among a collection of related tasks in order to learn the common ideas. Model fusion, on the other hand, merges different models on identical tasks to improve performances [23, 13]. Notably, these techniques have been previously applied to molecular tasks [30, 13, 31]. In PhysChem, conformation learning and property prediction tasks are jointly trained, with two specialist networks fused to achieve better performances. Nevertheless, the cooperation mechanism in PhysChem roots from observations in physics and chemistry, and enjoys better interpretability than straight-forward ensemble strategies. The advantages of the cooperation mechanism over multi-task strategies are also empirically shown in Section 4.

heading: 3 Proposed Method: PhysChem

heading: 3.1 Preliminaries
text: Notations In the statements and equations below, we use italic letters for scalars and indices, bold lower-case letters for (column) vectors, bold upper-case letters for matrices, calligraphic letters for sets, and normal letters for annotations. Common neural network layers are directly referred to: FC (x) denotes a fully-connected layer with input x; GCN (A,X) denotes a (vanilla) graph convolutional network (GCN)[14] with adjacency matrix A and feature matrix X; GRUcell (s,x) denotes a cell of the Gated Recurrent Units (GRU) [5] with recurrent state s and input signal x; LSTM ({xt}) denotes a Long Short-Term Memory network [10] with input signals {xt}; MaxPool ({xt}) denotes a max-pooling layer with inputs {xt}. Exact formulas are available in the appendix. ⊕ is the operator of concatenations. ∥·∥ denotes the l2 norm of the input vector. Problem Definition PhysChem considers molecular representation learning as a supervised learning task. It takes notations of molecules as inputs, conformations and chemical / biomedical properties as supervisions. PhysChem assumes a pre-conducted featurization process, after which a molecule can be denoted as an attributed graphM = (V, E , n,m,Xv,Xe). Here, V is the set of n atoms, E ⊂ V × V is the set of m chemical bonds, Xv ∈ Rn×dv = (xv1, · · · ,xvn) ⊤ is the matrix of atomic
features, and Xe ∈ Rm×de = (xe1, · · · ,xem) ⊤ that of bond features. Based on above inputs, PhysNet outputs the dominant conformations of molecules, and ChemNet outputs the representations as well as the predicted chemical / biomedical properties of molecules.
Overview Figure 1 is a sketch of PhysChem. An initializer first encodes the inputs into initial atom and bond states (v(0) and e(0)) for ChemNet, along with the initial atomic positions and momenta (q(0) and p(0)) in PhysNet. Subsequently, L PhysNet blocks simulate neural molecular dynamics in the generalized space; L ChemNet blocks conduct geometry-aware message-passing for atoms and bonds. Between each couple of blocks, implicit conformations (qs) and states of chemical bonds (es) are shared as expertise. At the top of PhysNet, a conformation decoder transforms implicit atomic positions into the 3D Euclidean space; of ChemNet, a sequence of readout layers aggregate atom states into molecular representations, based on which molecular properties are predicted with task-specific layers. Notably, two specialist networks are jointly optimized in PhysChem.

heading: 3.2 Initializer
text: Given a molecular graphM = (V, E , n,m,Xv,Xe), the initializer generates the initial values for both PhysNet and ChemNet variables. In the initializer, we first encode the input features into initial atom and bond states with fully connected layers, i.e.
v (0) i = FC (x v i ) , i ∈ V; e (0) i,j = FC ( xei,j ) , (i, j) ∈ E . (2)
We then adopt the initialization method in [16] to generate initial positions (q(0)) and momenta (p(0)) for atoms: a bond-strength adjacency matrix A ∈ Rn×n is estimated with sigmoid-activated FC layers on bond features, according to which a GCN captures the chemical environments of atoms (as ṽ); an LSTM then determines unique positions for atoms, especially for those with identical chemical environments (carbons in benzene, for example). Denoted in formula, the initialization follows
A(i, j) = { 0, (i, j) /∈ E FCsigmoid ( xei,j ) , (i, j) ∈ E , Ṽ = GCN ( A,V (0) ) , (3)
{( q (0) i ⊕ p (0) i )} = LSTM ({ṽi}) , i ∈ V (4)
where V (0) = ( v (0) 1 , · · · ,v (0) n )⊤ and Ṽ = (ṽ1, · · · , ṽn)⊤ are the atom states. The order of atoms
in the LSTM is specified by the canonical SMILES of the molecule.

heading: 3.3 PhysNet
text: Overall, PhysNet simulates the dynamics of atoms in a generalized, df -dimensional space (df ≥ 3). In a molecule as a classic dynamical system, atoms move following Newton’s Second Law:
dq/dt = p/m, dp/dt = f , (5)
where q, p and m are the position, momentum and mass of an atom, and f is the force that the atom experiences. With uniform temporal discretization, the above equations may be approximated with
qs+1 = qs + psτ/m, ps+1 = ps + fsτ, s = 0, 1, 2, · · · (6) where s is the index of timestamps and τ is the temporal interval. The calculations in PhysNet simulate such a process. Parameters of PhysNet blocks learn to derive the forces f from intermediate molecular conformations and states of chemical bonds. Correspondingly, two types of forces are modeled in PhysNet, namely the positional forces and the torsion forces.
Positional Forces The positional forces in PhysNet model the gravitations and repulsions between pairs of atoms. In conventional molecular force fields, these forces are generally modeled with (negative) derivatives of Lennard-Jones potentials. We therefore propose a similar form for the positional forces taking atomic distances as determinants:
f posj,i = (r −2 j,i − r −1 j,i )mjmidj,i, rj,i = ∥FC (qi − qj)∥ , i, j ∈ V, i ̸= j, (7)
where dj,i = qi−qj
∥qi−qj∥ is the unitary directional vector from atom j to i. Instead of using l2 distances in the generalized space, we use parameterized interaction distances rj,i to estimate the forces, which
increase the capability of the network. Here, r−2 − r−1 approximates the landscape of derivatives of Lennard-Jones potentials5 with lower-degree polynomials. The approximation is made to avoid numerical issues of using high-degree polynomials, most typically, the gradient explosion.
Torsion Forces The torsion forces model the mechanical effects of torsions in local structures. The torsion forces are defined between pairs of atoms that are directly connected by chemical bonds. Estimating local torsions only with positions and momenta of atoms is somehow suboptimal, as the network lacks prior knowledge of chemical characteristics of the bonds (default bond lengths, for example). Alternatively, we leverage the bond states (ei,j) in ChemNet as chemical expertise to derive the torsion forces. Specifically, the torsion forces f tor are calculated as
f torj,i = wj,idj,i, wj,i = FC (ej,i) , (i, j) ∈ E . (8)
As the torsion forces model chemical interactions, we do not explicitly incorporate atomic mass into the calculation. Notably, atomic information is implicitly considered in the torsion forces, as the bond states integrate atom states at both ends in ChemNet (see the next subsection).
Dynamics After estimating the positional and torsion forces, PhysNet simulates the dynamics of atoms following Equation (6). In the l-th block of PhysNet, S steps of dynamics are simulated:
q (l,s+1) i = q (l,s) i +
τ
mi p (l,s) i , p (l,s+1) i = p (l,s) i + τf (l,s) i , s = 0, 1, · · · , S − 1. (9)
Here, q(l,0)i = q (l) i , q (l+1) i = q (l,S) i (p similarly). fi is the resultant force on atom i, i.e. fi = ∑ j∈V f posj,i + ∑ (i,j)∈E f torj,i . (10)
Conformation Decoder and Loss We use a simple linear transformation to decode implicit atomic positions (and momenta) from the generalized space into the real 3D space:
R̂ = QWdec, Wdec ∈ Rdf×3, (11)
where Q = (q1, · · · , qn)⊤ ∈ Rn×df is the position matrix in the generalized space, and R̂ = (r̂1, · · · , r̂n)⊤ ∈ Rn×3 that of the predicted 3D conformation. We further propose Conn-k (k-hop connectivity loss), a spatially invariant loss that supervises PhysNet based on local distance errors: if C(k) ∈ {0, 1}n×n is the k-hop connectivity matrix6 and D is the distance matrix7, then the k-hop connectivity loss is defined as
LConn-k(R̂,R) = ∥∥∥∥ 1nĈ(k) ⊙ (D̂ −D)⊙ (D̂ −D) ∥∥∥∥ F , (12)
where ⊙ is the element-wise product, ∥·∥F is the Frobenius norm, (D, D̂) are distance matrices of the real and predicted conformations (R, R̂), and Ĉ(k) is the normalized k-hop connectivity matrix8. The total loss of PhysNet is defined as the weighted sum of Conn-3 losses on all timestamps:
Lphys = 1
Z ∑ l,s ηLS−(lS+s)L (l,s) Conn-3, l = 1, 2, · · · , L, s = 1, 2, · · · , S, (13)
where η < 1 is a decay factor and Z = 0.01 is an empirical normalization factor.

heading: 3.4 ChemNet
text: ChemNet conducts message-passing for both atoms and chemical bonds. Besides generating messages with atom states, ChemNet also considers local geometries including bond lengths and angles to adequately characterize local chemical environments. Specifically, in each ChemNet block, triplet descriptors are established in the atomic neighborhoods. Atoms merge relevant triplet descriptors to
5UL-J = ar−12 − br−6, fL-J = − ∂U L-J
∂x =
( 12ar−11 − 6br−5 ) ∂r ∂x
6C (k) i,j = 1 if and only if atoms i and j are k- or less-hop connected on the molecular graph. 7Di,j = ∥ri − rj∥, where r ∈ R3 are 3D coordinates of atoms. 8Ĉ(k) = L−1/2C(k)L−1/2, where L is the diagonal degree matrix with Li,i = ∑ j C (k) i,j .
generate messages, and then aggregate received messages to update their states. Chemical bonds also update their states by aggregating the states of atoms at both ends.
Triplet Descriptor The triplet descriptors ti,j,k are descriptive vectors defined on all atom triplets (i, j, k) with (i, j) ∈ E ∧ (i, k) ∈ E :
t (l) i,j,k = FC ( v (l) i ⊕ v (l) j ⊕ v (l) k ⊕ l (l) i,j ⊕ l (l) i,k ⊕ a (l) j,i,k ) , j, k ∈ N (i), (14)
where N (i) = {j | (i, j) ∈ E} denotes the atomic neighborhood of i, vs are atom states, and li,j = FC (li,j) and aj,i,k = FC (cos (∠j,i,k)) are representations of bond lengths and bond angles. The motivation of constructing triplet descriptors is that these features are geometrically deterministic and invariant: i) all bond lengths and angles in an atomic neighborhood together compose a minimum set of variables that uniquely determine the geometry of the neighborhood (deterministic); ii) these features all enjoy translational and rotational invariances (invariant). When PhysNet and ChemNet are jointly optimized, l(l)i,j and ∠ (l) j,i,k are calculated from the intermediate 3D conformation in the l-th PhysNet block; when real conformations of target molecules are available, these values can be replaced by the ground-truths (e.g. ChemNet (real conf.) in Section 4).
Message-passing After establishing the triplet descriptors, ChemNet generates atomic messages by merging relevant descriptors. The message from atom j to i in the l-th block is calculated as
m (l) i,j = MaxPool ({ t (l) i,j,k | k ∈ N (i) }) , j ∈ N (i), l = 1, 2, · · · , L. (15)
Subsequently, ChemNet utilizes a similar architecture to [36] to conduct message passing. Centric atoms aggregate the received messages with attention scores determined by the bond states:
m (l) i = ∑ j∈N (i) α (l) i,jm (l) i,j , { α (l) i,j | j ∈ N (i) } = softmax ({ FC ( e (l) i,j ) | j ∈ N (i) }) . (16)
Atom states are then updated with GRU cells that take previous states as recurrent states. A similar process is then conducted for all chemical bonds, where messages are generated by the states of atoms at both ends. Denoted in formula,
v (l+1) i = GRUcell ( v (l) i , m (l) i ) , e (l+1) i,j = GRUcell ( e (l) i,j , FC ( v (l+1) i ⊕ v (l+1) j )) . (17)
Representation Readout The molecular representation is finally read out with T global attentive layers [36], where a virtual meta-atom continuously collects messages from all atoms in the molecule and updates its state. We initialize and update the state of the meta-atom (vmeta) following:
m (t) meta = ∑ i∈V α (t) i v (L) i , { α (t) i | i ∈ V } = softmax ({ FC ( v (t) meta ⊕ v (L) i ) | i ∈ V }) ,
v (0) meta = FC
( 1
n ∑ i∈V v (L) i
) , v (t+1) meta = GRUcell ( v (t) meta,m (t) meta ) , t = 1, 2, · · · , T. (18)
The final state of the meta-atom is used as the molecular representation. Labeled chemical and / or biomedical properties are used to supervise the representations, and the loss of ChemNet, Lchem, is determined task-specifically. The total loss of PhysChem is the weighted sum of losses in the two networks controlled by a hyperparameter λ, i.e.
Ltotal = λLphys + Lchem. (19)

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets and Featurization We evaluated PhysChem on quantum mechanics (QM7, QM8 and QM9) and physical chemistry (LIPOP, FREESOLV and ESOL) datasets, as well as drug effectiveness datasets of the notorious SARS-CoV-2. Statistics of the datasets are in Table 1. QM7, QM8 and QM9 contain stable organic molecules with up to 7, 8 or 9 heavy atoms. 3D atomic coordinates as
well as electrical properties of molecules were calculated with ab initio Density Functional Theory (DFT). LIPOP provides experimental results on lipophilicity of 4,200 organic compounds; FREESOLV provides experimental and calculated hydration free energy of 642 small molecules in water; ESOL provides water solubility data for 1128 compounds. The above six datasets were collected in the MoleculeNet benchmark [35]9. COVID19 [3] is a collection of datasets10 generated by screening a panel of SARS-CoV-2-related assays against approved drugs. 13 assays of 14,332 drugs were used in our experiments. We split all datasets into 8:1:1 as training, validation and test sets. For datasets with less than 100, 000 molecules, we trained models for 5 replicas with randomly split data and reported the means and standard deviations of performances; for QM9, we used splits with the same random seed across models. We used identical featurization process in [36] to derive feature matrices (Xv,Xe) for all models and on all datasets.
Baselines For conformation learning tasks, we compared PhysChem with i) a Distance Geometry [2] method tuned with the Universal Force Fields (UFF) [21], which was implemented in the RDKit package11 and thus referred to as RDKit; ii) CVGAE and CVGAE+UFF [18], which learned to generate low-energy molecular conformations with deep generative graph neural networks (either with or without UFF tuning); iii) HamEng [16], which learned stable conformations via simulating Hamiltonian mechanics with neural physical engines. For property prediction tasks, we compared PhysChem with i) MoleculeNet, for which we reported the best performances achieved by methods collected in [35] (before 2017); ii) 3DGCN [4], which augmented conventional GCNs with input bond geometries; iii) DimeNet [15], which conducted directional message-passing by representing pairs of atoms; iv) Attentive FP [36], which used local and global attentive layers to derive molecular representations; and v) CMPNN [29], which used communicative kernels to conduct deep messagepassing. We conducted experiments with official implementations of HamEng, CVGAE,Attentive FP and CMPNN; for other baselines, as identical evaluation schemes were adopted, we referred to the reported performances in corresponding citations and left unreported entries blank.
PhysChem Variants We also compared PhysChem with several variants, including i) PhysNet (s.a.), a stand-alone PhysNet that ignored all chemical expertise by setting ej,i ≡ 0; ii) ChemNet (s.a.), a stand-alone ChemNet (ChemNet (s.a.)) that ignored all physical expertise by equally setting all bond lengths and angles (li,j ≡ l, aj,i,k ≡ a); iii) ChemNet (real conf.) and ChemNet (rdkit conf.), two ChemNet variants that leveraged li,j and aj,i,k in real conformations (real conf.) or RDKit-generated conformations (rdkit conf.); and iv) ChemNet (m.t.), a multi-task ChemNet variant that used a straight-forward multi-task strategy for conformation learning and property prediction tasks: we applied the conformation decoder and loss on atom states v and optimized ChemNet with the weighted sum of losses of both tasks (i.e. Ltotal in Equation (19)).
Implementation and Training Details Unless otherwise specified, we used L = 2 pairs of blocks for PhysChem. In the initializer, we used a 2-layer GCN and a 2-layer LSTM. In each PhysNet block, we set df = 3, S = 4 and τ = 0.25. In each ChemNet block, we set the dimensions of atom and bond states as 128 and 64, correspondingly. In the representation readout block, we used T = 1 global attentive layers with 256-dimensional meta-atom states. For property prediction tasks with multiple targets, the targets were first standardized and then fitted simultaneously. We use the mean-square-error (MSE) loss for all regression tasks and the cross-entropy loss for all classification tasks to train the models. Other implementation details, such as hyperparameters of baselines, are provided in the appendix.
9The datasets are publicly available at http://moleculenet.ai/datasets-1. 10The datasets are available at https://opendata.ncats.nih.gov/covid19/ (CC BY 4.0 license) and
are continuously extended. The data used in our experiments were downloaded on February 16th, 2021. 11We use the 2020.03.1.0 version of the RDKit package at http://www.rdkit.org/.

heading: 4.2 Results
text: Quantum Mechanical Datasets As real conformations are available in the QM datasets, we evaluated PhysChem on both conformation learning and property prediction tasks. On conformation learning tasks, a Distance MSE metric is reported, which sums the squared errors of all pair-wise distances in each molecule, normalizes it by the number of atoms, and then takes averages across molecules. Note that this metric is equivalent to the Conn-∞ loss with k = ∞ in Equation (12). On property prediction tasks, the mean-absolute-errors (MAE) for regression targets are reported. When multiple targets are requested (on QM8 and QM9), we report the Multi-MAE metric in [16] which sums the MAEs of all targets (standardized for QM9). Table 2 and 3 show the results. On conformation learning tasks, PhysNet (s.a.) displays significant advantages on learning conformations of small molecules. Specifically, the comparison between PhysNet (s.a.) and HamEng indicates that directly learning forces in neural physical engines may be superior to learning energies and their derivatives. With massive data samples (QM9), the specialist, PhysNet (s.a.), obtains better results than PhysChem; while on datasets with limited samples (QM7), chemical expertise demonstrates its effectiveness. On property prediction tasks, PhysChem obtains state-of-the-art performances on QM datasets. The comparison between PhysChem, ChemNet (s.a.) and ChemNet (m.t.) shows that the elaborated cooperation mechanism in PhysChem is necessary, as the straight-forward multitask strategy leads to severe negative transfer. In addition, the results of ChemNet (real conf.) and ChemNet (rdkit conf.) show that leveraging real (or generated, geometrically correct) conformations of test molecules indeed helps on some datasets, while the elevations are somehow insignificant.
Physical Chemical & Biomedical Datasets On LIPOP, FREESOLV, ESOL and COVID19 with no labeled conformations, we used RDKit-generated conformations to satisfy models that requested conformations of training and / or test molecules. Although these generated conformations are less accurate, distance geometries in local structures are generally correctly displayed. We report the rooted-mean-squared-errors (RMSE) for regression tasks, and the multi-class ROC-AUC (MultiAUC) metric on COVID19. Table 4 shows the results. PhysChem again displays state-of-the-art performances. Notably, as the numbers of samples in physical chemical datasets (LIPOP, FREESOLV
and ESOL) are sparse, the conformation learning task for PhysNet is comparatively tough. This leads to a larger gap of performances between PhysChem and ChemNet (rdkit conf.), yet the former is still better than most baselines. The largest elevation of performances is observed on COVID19, which indicates that PhysChem is further competent on more complicated, structure-dependent tasks.
Further Analyses Figure 2(a) shows the effect of the loss hyperparameter λ on QM9. When λ ≤ 0.1, increasing λ benefits both tasks. This indicates that aiding the learning of conformations also helps to predict chemical properties. With larger λ, the property prediction task is then compromised. Figure 2(b) visualizes the predicted conformations of baselines and PhysNet blocks. Local structures such as bond lengths, angles and planarity of aromatic groups are better preserved by PhysChem.

heading: 5 Conclusion and Future Work
text: In this paper, we propose a novel neural architecture, PhysChem, that learns molecular representations via fusing physical and chemical information. Conformation learning and property prediction tasks are jointly trained in this architecture. Beyond straight-forward multi-task strategies, PhysChem adopts an elaborated cooperation mechanism between two specialist networks. State-of-the-art performances were achieved on MoleculeNet and SARS-CoV-2 datasets. Nevertheless, there is still much space for advancement of PhysChem. For future work, a straight-forward improvement is to enlarge the capability of the model by further simplifying as well as deepening the architecture of PhysChem. In addition, proposing strategies to train PhysChem with massive unlabeled molecules is yet another promising direction.
Broader Impact For the machine learning community, our work proposes a more interpretable architecture on molecular machine learning tasks and demonstrates its effectiveness. We hope that the specialist networks and domain-related cooperation mechanism in PhysChem will inspire researchers in a wider area of deep learning to develop novel architectures under the same motivation. For the drug discovery community, PhysChem leads to direct applications on ligand-related tasks including conformation and property prediction, protein-ligand binding affinity prediction, etc. With limited possibility, the abuse of PhysChem in drug discovery may violate some ethics of life science.
Acknowledgments This work was supported by the National Natural Science Foundation of China (Grant No. 61876006).

Review summaries:
The authors are interested in the linked tasks of predicting (i) 3D conformations and (ii) chemical/biomedical properties of molecules. They show that by designing a model (PhysChem) that tackles both of these tasks simultaneously, they can often do better on each individual task than when these tasks are considered separately. This is empirically demonstrated on the QM{7,8,9} datasets, the MoleculeNet  Physical Chemistry Benchmark [32], as well as a SARS-CoV-2 dataset [3], where the authors also compare to a variety of older approaches.     ### Further Details on the Architecture To be more specific about the architecture, PhysChem consists of two networks (which share information at each layer and both take in as input the graph representation of a molecule): 1. PhysNet, is used to predict the conformation of a molecule and it does this by first predicting the initial positions of the atoms (using a graph convolutional network (GCN) and RNN) before then updating these positions using the equations of motion under the action of interatomic forces output by a fully connected NN. 2. ChemNet†, is used to predict chemical/biomedical properties of a molecule, and is a new variant of a message passing neural network [9].  † This is a new network, not to be confused with previous networks also called "ChemNet", e.g. Preuer, K., Renz, P., Unterthiner, T., Hochreiter, S. and Klambauer, G. (2018) ‘Fréchet ChemNet Distance: A Metric for Generative Models for Molecules in Drug Discovery’, Journal of chemical information and modeling, 58

The authors proposed PhysChem network for molecule representation. PhysChem consists of four modules, the initializer, the PhysNet, ChemNet and readout. Specifically, in PhysNet, the position, momentum and mass of an atom are modeled using rules in mechanics. The authors conduct a series of experiments to verify the proposed algorithm.

This paper presents a novel method called PhysChem for molecular representation learning by fusing physical and chemical information of molecules. PhysChem consists of two cooperating neural networks, one called PhysNet and another called ChemNet. PhysNet learns 3D conformations by simulating molecular dynamics with parametrized forces, while ChemNet is a geometry-aware message passing network. The experiment results for MoleculeNet benchmark and SARS-Cov-2 datasets showed superior performance over the other state-of-the-art baselines.  GNNs or Geometric NNs are mostly separated to 3D-geometry-aware ones used for quantum-calculation surrogates and 2D-abstraction ones for QSARs predicting properties such as toxicity and solvability. Representation learning that integrates these two aspects is very interesting and promising.