Year: 2021

Abstract:
Recent studies use geometric deep learning to represent molecules and predict properties. However, they are computationally expensive in capturing long-range dependencies and ignore the non-uniformity of interatomic distances. More importantly, few of them consider injecting the biochemical structure knowledge such as functional groups into model architectures. To overcome such issues, we introduce Molformer, a variant of the Transformer for molecular representations that exploits both semantic motifs and 3D spatial information. Specifically, Molformer extracts motifs based on functional groups and learns customized embeddings to store the semantic meanings of those informative substructures. In order to fully employ 3D geometry, we adopt a convolutional position encoding to achieve roto-translation invariance, a multi-scale self-attention mechanism to capture local fine-grained patterns with increasing contextual scales, and an attentive farthest point sampling algorithm to attain the molecular representation. We validate Molformer across several domains in quantum chemistry, physiology, and biophysics. Our experiments show better or competitive performance in those datasets. Our work provides a promising way to amalgamate 3D geometric information and make better usage of informative substructures in representing molecules.

heading: 1 INTRODUCTION
text: Spatial structures are among the most crucial factors to decide molecular properties and understand their principles of action in the physical world. For example, 3D structures of proteins provide valuable information for inferring biological interventions, such as structure-based drug development and targeted mutagenesis (Senior et al., 2020; Jumper et al., 2021; Baek et al., 2021). In chemistry, zeolites show obvious differences in separation properties caused by subtle changes in their 3D geometric compositions (Chai et al., 2020; Pfriem et al., 2021). Apart from that, in the pharmaceutical industry, the same compounds can have different 3D structures, resulting in different solubility (Zhang et al., 2017). To sum up, capturing 3D spatial structures is essential to accurately forecast molecular properties. Based on these facts, researchers have studied molecular representation learning techniques (Rao et al., 2019) to include 3D spatial information (Zhavoronkov et al., 2019).
The dominant 3D molecular models are Graph Neural Networks (GNNs) and 3D Convolutional Neural Networks (3DCNNs) (Derevyanko et al., 2018; Pagès et al., 2019; Townshend et al., 2019). GNNs create edges by using either chemical bonds or finding the neighbors of each node within a distance cutoff (Zhang et al., 2020b). They encode pairwise connectivity of atoms and require running multiple hops for an atom to reach to another. 3DCNNs encode translational and permutational symmetries, but need to stack deep layers to build direct connections between distant regions, incurring significant computational costs. In contrast, Transformers rely on the self-attention mechanism to capture long-term dependencies in parallel (Hernández & Amigó, 2021). Meanwhile, Equivariant Neural Networks (ENNs) (Thomas et al., 2018) have emerged as a new class of methods, where geometric transformations of their inputs lead to well-defined transformations of outputs. Some ENNs adopt Transformers as the backbone but fail to surmount the intrinsic drawbacks of this architecture, including its insensibility to local patterns among non-uniformly distancing atoms and its ineffi-
ciency to aggregate atom features. Some other Transformer-based methods have been proposed to fuse distance and graph neighbourhood information (Maziarka et al., 2020; 2021). However, they take no consideration of employing motifs, which are frequently-occurring substructures in molecules and can be leveraged to uncover global graph properties.
In this work, we present the Molformer on the basis of all preceding analysis. For the sake of injecting chemical domain knowledge, we construct a motif-template vocabulary based on functional groups and adopt trainable motif embeddings to maintain the semantic meanings of those essential substructures. Then with both motifs and atoms as input, Molformer operates on a fully-connected graph with direct connections between remote regions (Veličković et al., 2017; Joshi, 2020), which reduces computational burden of multi-hop GNNs and stacked 3DCNNs. However, this characteristic limits Molformer’s capacity in exploiting local structures and leads to poor generalization in unseen cases (Qi et al., 2017). Therefore, we propose a Multi-scale Self-Attention (MSA) module to recognize fine-grained patterns from neighborhoods. Moreover, we introduce a roto-translation invariant Convolutional Position Encoding (CPE) to depict position relationships among atoms and their adjacencies. After that, to retain a comprehensive representation of the entire molecule, we propose an Attentive Farthest Point Sampling (AFPS) module that selects important atoms with the assistance of the attention score map.
To summarize, our contributions are as follows:
• To the best of our knowledge, we are the foremost to incorporate motifs with knowledge of functional groups into a Transformer architecture for 3D molecular representation learning.
• We propose a novel MSA to extract local patterns, a roto-translation invariant CPE method to encode relative distance at a linear computational time cost, and a simple yet effective downsampling algorithm to gather molecular representations.
• We show significant improvements on several benchmarks in three domains. Code and all datasets are available at https://github.com/smiles724/Molformer.

heading: 2 PRELIMINARIES
text: Problem Definition. A molecule S = (E,P ) has N atoms and C atom classes, where E = {e1, ..., eN} ∈ RN×C contains the one-hot atom representations and P = {p1, ...,pN} ∈ RN×3 contains the 3D coordinates of each atom. Each one-hot ei can be converted to a dense vector xi = eiW
E , with xi ∈ Rdmodel and WE ∈ RC×dmodel being the embedding matrix. The 3D coordinates of the atom i is a three-dimensional vector pi = [pxi , p y i , p z i ]. A representation learning model f acts on S, obtaining its representation r = f(S). Then r is forwarded to a prediction model g and attain the prediction of a biochemical property ŷ = g(r).
Self-attention Mechanism. The Transformer (Vaswani et al., 2017) has become very successful due to its core component, self-attention. Given a set of input features {xi}i=1,...,N , the standard dot-product attention layer is as the following:
qi = fQ(xi), ki = fK(xi), vi = fV (xi), aij = qik T j / √ dk, zi = N∑ j=1 σ(aij)vj (1)
where {fQ, fK , fV } are embedding transformations, and {qi,ki,vi} are respectively the query, key, and value vectors with the same dimension dk. aij is the attention that the token i pays to the token j. σ denotes the Softmax function and zi is the output embedding of the token i. This formula conforms to a non-local network (Wang et al., 2018), indicating its inability to capture fine-grained patterns in a local context.
Position Encoding. Self-attention is invariant to permutation of the input (Dufter et al., 2021), and position encoding ensures that the Transformer will reveal positional information. Position encoding methods can be either based on absolute positions or relative distances. The former takes the raw position information as input and is sensitive to spatial transformations. The latter manipulates the attention score by incorporating relative distances (Guo et al., 2020a; Pan et al., 2021): aij = qik T j / √ dk + fPE(pi−pj), where fPE(·) is the position encoding function and is translation
invariant. The rotation invariance can be further accomplished by taking a L2-norm ||pi − pj ||2 (Chen et al., 2019b).

heading: 3 MOLFORMER
text: Molformer is based on the architecture of Transformer but adopts several significantly different and novel components (see Figure 1). First, a vocabulary of motif templates is constructed on the basis of functional groups and we extract all available motifs from each molecule. Then both atoms and motifs acquire their corresponding embeddings and are forwarded into L feature learning blocks. Each block consists of a convolutional position encoding, a multi-scale self-attention, and a feedforward network. After that, an attentive subsampling method is utilized to adaptively aggregate the molecular presentation, which is later fed into a predictor to forecast properties in a broad range of downstream tasks.

heading: 3.1 TRAINABLE MOTIF-BASED EMBEDDING
text: Motifs are frequently-occurring substructure patterns as well as the building blocks of complex molecular structures. They usually maintain semantic meanings and have great expressiveness of the biochemical characteristics of the whole molecule (Zhang et al., 2020a). In the chemical community, researchers have developed a set of standard criterion to recognize motifs with essential functionalities in molecules (Milo et al., 2002). Despite that, few of prior studies directly incorporate those informative motifs into their model architectures. To fill this gap, we define a series of momentous substructures using external domain knowledge, and introduce a trainable motif embeddings method to fully exploit them in our Molformer.
To begin with, all motifs are first extracted according to the motif vocabulary, which is built by functional groups. Practically, we rely on RDKit (Landrum, 2013) to draw them from the SMILES (Weininger, 1988) representation of each molecule. We assume M motifs {m1, ...,mM}
are detected in the molecule S, and each motif mi contains a certain number of at least two atoms. Then we regard each kind of motif as a new type of token and append them to the input. Therefore, the input for our Molformer becomes {x1, ...,xN ,xm1 , ...,xmM }, where xmi is obtained through an learnable embedding matrix WM ∈ RC′×dmodel and C ′ denotes the number of motif categories. As for the position of each motif, we adopt a weighted sum of the 3D coordinates of its component atoms as pmi = ∑ xi∈mi( wi∑ xi∈mi wi ) · pi, where wi are the atomic weights.
Our approach requires the model to automatically learn a customized embedding for each motif template through backpropagations, which follows a data-driven pattern. In some data-sufficient tasks, its greatest potential can be unlocked and those motif embeddings can be well trained. Nevertheless, in the case of few-shot learning or small datasets, each category of motif might only appear rare times. Those embeddings are not fully tuned and can be extremely biased and noisy, which will do little helps to the ultimate property prediction.

heading: 3.2 CONVOLUTIONAL POSITION ENCODING
text: To enable roto-translation invariance and take fully advantage of geometric information, instead of adding a term of fPE(pi − pj), we propose a CPE that applies a convolutional operation to the interatomic distance D ∈ RN×N :
Acov = Conv2d(D) A, (2)
where A = [ai,j ]i,j=1,···N ∈ RN×N is the attention matrix, Conv2d(·) denotes a 2D shallow convolutional network with a kernel size of 1× 1, and is the element-wise product. With multi-headed self-attention, Acov is expanded in the sense that Acov ∈ RH×N×N , and Conv2d(·) has H output channels. The CPE method induces O(N) convolution operations on each atom and can drastically reduce training time when the number of atoms is very large (Wu et al., 2021).

heading: 3.3 MULTI-SCALE SELF-ATTENTION
text: The self-attention mechanism in the Transformer is good at capturing global data patterns but ignores local context (Guo et al., 2020a). Exploiting local context has proven to be important for 3D spatial data such as 3D point clouds (Qi et al., 2017). Therefore, we impose a distance-based constraint in self-attention in order to extract multi-scaled patterns from both local and global contexts.
Guo et al. (2020b) propose to use integer-based distance to limit attention to local word neighbors, which cannot be used in molecules. This is because different types of molecules have different densities and molecules of the same type have different spatial regularity, which results in the nonuniformity of interatomic distances. Normally, small molecules have a mean interatomic distance of 1-2 Å (Angstrom, 10−10m), which is denser than large molecules like proteins with approximately 5 Å on average. To address that, we design a new multi-scale methodology to robustly capture details. Specifically, we mask atoms beyond a certain distance τs (a real number as opposed to an integer in Guo et al. (2020b)) at each scale s. We denote dij = ||pi−pj ||2 as the Euclidean distance between the i-th and j-th atom. The attention calculation is modified as:
aτsij = qik T j · 1{dij<τs}√
dk , zτsi = N∑ j=1 σ(aτsij )vj , (3)
where 1{dij<τs} is the indicator function. For small molecules, Equation 3 can be complementally combined with Equation 2. Then features extracted from S different scales {τs}s=1,...,S as well as the informative global feature are concatenated together to form a multi-scale representation, denoted by z′i = z τ1 i ⊕ ...⊕z τS i ⊕z global i ∈ R(S+1)dk . After that, z′i is forwarded into a multi-layer perceptron to be compressed as z′′i with the original dimension dk.

heading: 3.4 ATTENTIVE FARTHEST POINT SAMPLING
text: After having the atom embeddings {z′′i }i=1,...,N , we study how to obtain the molecular representation r. For GNNs, several readout functions such as set2set (Vinyals et al., 2015) and GGNN (Gilmer et al., 2017) are invented. For Transformer architectures, one way is via a virtual atom.
Algorithm 1 Pseudocode of Attentive Farthest Point Sampling Input: The attention score matrix A ∈ RN×N , a Euclidean distance matrix D ∈ RN×N . Output: K sampled points. 1: Ã← ∑ iAij ∈ RN . sum up the attention matrix along rows
2: D̃ ← D ∈ RN×N . normalize the distance matrix 3: P = {x#},M = {1, 2, ..., N} 4: while length(P) < k do 5: xnew = argmax
i∈M (min j∈P D̃ij + Ãi) . pick up the atom that maximize the objective
6: P .append(xnew),M.remove(xnew) 7: return P
Though as Ying et al. (2021) state, it significantly improves the performance of existing models in the leaderboard of Open Graph Benchmark (Hu et al., 2020), this way concentrates more on close adjacent atoms and less on distant ones, and can lead to inadvertent over-smoothing of information propagation (Ishiguro et al., 2019). Besides, it is difficult to locate a virtual node in 3D space and build connections to existing atoms. The other way selects a subset of atoms via a downsampling algorithm named Farthest Point Search (FPS), but it ignores atomic differences and has sensitivity to outlier points (Pan et al., 2021) as well as uncontrollable randomness. To address these issues, we propose a new algorithm named AFPS. It aims to sample atoms by not merely spatial distances, but also their significance in terms of attention scores.
Specifically, we choose the virtual atom x# as the starting point and initialize two lists P = {x#} andM = {1, ..., N} to store remaining candidate points. Then the process begins with the attention score matrix A ∈ RN×N and the interatomic distance matrix D ∈ RN×N . It can be easily proved that each row of A sums up to 1 after the Softmax operation along columns, i.e. ∑ jAij = 1 for ∀i ∈ [N ]. In order to obtain the importance of each atom in the self-attention computation, we accumulate A along rows and get Ã = ∑ iAij ∈ RN . Besides, we adopt the min-max normalization to rescale the distance matrix D into values between 0 and 1, and obtain D̃ = D−minDmaxD−minD .
After the above preprocess, we repeatedly move a point xnew from M to P , which ensures that xnew is as far from P as possible by maximizing D̃ij and also plays a crucial role in attention computation by maximizing Ãi. Mathematically, the AFPS aims to achieve the following objective:
max ∑ i∈M ( min j∈P\{i} D̃ij + Ãi) (4)
where is a hyperparameter to balance those two different goals. This process is repeated until P has reached K points. Algorithm 1 provides a greedy approximation solution to solve this AFPS optimization objective for sake of computational efficiency.
After that, sampled features {z′′i }i∈P are gathered by a Global Average Pooling layer (Lin et al., 2013) to attain the molecular representation r ∈ Rdk . Remarkably, our proposed AFPS has considerable difference and superiority over a body of previous hierarchical learning approaches (Eismann et al., 2020; 2021). Their subsampling operations are mainly designed for protein complexities, which have more uniform structures than small molecules. To be specific, they hierarchically use alpha carbons as the intermediate set of points and aggregate information at the level of those carbons for the entire complex. However, the structures of small molecules have no such a stable paradigm, and we provide a universal methodology to adaptively subsample atoms without any prior assumptions on the atom arrangement.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: We conduct extensive experiments on both small and large molecules (proteins) with various targets, including quantum chemistry, physiology, and biophysics. Table 1 summarises information of
benchmark datasets, such as the number of tasks and task types, the number of molecules and atom classes, the minimum and maximum number of atoms, and the density (mean interatomic distances) of all molecules.
Datasets. We test Molformer on a series of small molecule datasets, containing QM7 (Blum & Reymond, 2009), QM8 (Ramakrishnan et al., 2015), QM9 (Ramakrishnan et al., 2014), BBBP (Martins et al., 2012), ClinTox (Gayvert et al., 2016), and BACE (Subramanian et al., 2016) 2. QM7 is a subset of GDB-13 and composed of 7K molecules with up to 5 heavy atom types. QM8 and QM9 are subsets of GDB-17 with 22k molecules and 133K molecule respectively.
Additionally, we also inspect Molformer’s ability of learning mutual relations between proteins and molecules on the PDBbind dataset (Wang et al., 2005). We follow Townshend et al. (2020) and split protein-ligand complexes by protein sequence identity at 30%. As for the target, we predict pS = − log(S), where S is the binding affinity in Molar unit. In addition, we only use the pocket of each protein and put pocket-ligand pairs together as the input.
For QM9, we use the exact train/validation/test split as Townshend et al. (2020). For PDBbind, 90% of the data is used for training and the rest is divided equally between validation and test like Chen et al. (2019c). For others, we adopt the scaffold splitting method with a ratio of 8:1:1 for train/validation/test as Rong et al. (2020). More implementing details can be found in Appendix A.1
Baselines For small molecules, we compare our approach with a number of state-of-the-art baselines. TF_Robust (Ramsundar et al., 2015) takes molecular fingerprints as the input. GraphConv (Kipf & Welling, 2016), Weave (Kearnes et al., 2016), MPNN (Gilmer et al., 2017), Schnet (Schütt et al., 2018), MEGNet (Chen et al., 2019c), DMPNN (Yang et al., 2019), MGCN (Lu et al., 2019), AttentiveFP (Xiong et al., 2019), DimeNet++ (Klicpera et al., 2020), SphereNet (Liu et al., 2021), and SpinConv (Shuaibi et al., 2021) are all graph convolutional models. Graph Transformer (Chen et al., 2019a), MAT (Maziarka et al., 2020), R-MAT (Maziarka et al., 2021), SE(3)Transformer (Fuchs et al., 2020), and LieTransformer (Hutchinson et al., 2021) are Transformerbased models.
For PDBbind, we choose six baselines. DeepDTA (Öztürk et al., 2018) and DeepAffinity (Karimi et al., 2019) take in pairs of ligand and protein SMILES as input. Cormorant (Anderson et al., 2019) is an ENN that represents each atom by its absolute 3D coordinates. Schnet, 3DCNN and 3DGCN (Townshend et al., 2020) are 3D molecular representation methods.

heading: 4.2 RESULTS ON DOWNSTREAM TASKS
text: Molecules. Table 2 and Table 3 document the overall results of Molformer and baselines on small molecules datasets, where best performance is marked bold and the second best is underlined for clear comparison. It can be discovered that Molformer achieves the lowest MAE of 11.6 on QM7 and 0.009 on QM8, beating several strong baselines including DMPNN and Graph Transformer. While
1The total number of proteins in the full, unsplit PDBbind is 11K, but our experiment only uses 4K proteins at 30% sequence identity. Moreover, the number of atoms is the sum of both the pocket and molecules.
2For BBBP, ClinTox, and BACE, we use RDKit (Landrum, 2013) to procure 3D coordinates from SMILES.
not all state-of-the-art on QM9, Molformer offers competitive performance in 5 property regression tasks, which do not require thermochemical energy subtractions. Particularly, we outperforms all Transformer-based ENNs, including SE(3)-Transformer and LieTransformer. In classification problems, we surpass all non-pretrained methods and are only inferior to the pretrained GROVE. This accords to the fact that datasets with fewer samples can gain large improvements through the self-supervised pretraining (Rong et al., 2020).
Protein. Table 4 reports the Root-Mean-Squared Deviation (RMSD), the Pearson correlation (Rp), and the Spearman correlation (Rs) on PDBbind. Molformer achieves the lowest RMSD among all baselines and the best Pearson and Spearman correlations. As Wu et al. (2018) claim, appropriate featurizations which contains pertinent information is significant for PDBbind. However, an important observation in our work is that deep learning approaches with the full exploitation of 3D geometric information can perform better than conventional methods like DeepDTA and DeepAffinity, which use a set of physicochemical descriptors but ignore 3D structures.

heading: 5 ABLATION STUDY AND DISCUSSION

heading: 5.1 WHAT ARE THE EFFECTS OF EACH COMPONENT
text: We investigate the effectiveness of different modules of our Molformer in Table 5. It can be observed that CPE substantially boosts model’s performance compared with the naive method that immediately adds 3D coordinates as the atom input feature. In addition, AFPS is found to produce better
predictions than the control group, which utilizes the virtual node as the molecular representation. Moreover, MSA significantly reduces RMSD from 17.6 to 11.6 on QM7, but its improvements in QM8 are much smaller. This phenomenon indicates that MSA is an appropriate way to alleviate the problem of inadequate training in small datasets. It endows Molformer with capability to extract local features by regulating the scope of self-attention. However, as the data size gets larger and larger, Molformer does not require the assistance of MSA to abstract local patterns, since the parameters of CPE is properly trained. What’s more, the trainable motif-level embedding leads to a MAE decrease of 2.1 in QM7 and a RMSD drop of 0.011 in PDBbind, indicating its effectiveness in both small molecules and proteins.

heading: 5.2 HOW USEFUL IS THE TRAINABLE MOTIF-BASED EMBEDDINGS?
text: How to determine motifs are critical and crucial to our proposed trainable motif-based embeddings. In organic chemistry, a functional group is a substituent or moiety in a molecule that causes the molecule’s characteristic chemical reactions. The same functional group will undergo the same or similar chemical reactions regardless of the rest of the molecule’s composition (Smith, 2020). Therefore, we define motifs on the basis of functional groups and explore the contribution of four different categories. Specifically, we consider four common functional groups, including groups that contain only carbon and hydrogen (Hydrocarbons), groups that contain halogen (Haloalkanes), groups that contain oxygen, and groups that contain nitrogen (see the left part in Figure 2). The ablations (see the right part in Figure 2) demonstrate that Molformer can gain improvements from all sorts of motifs, where Hydrocarbons and Haloalkanes are the most and the least effective kinds, respectively. This is in line with the fact that Hydrocarbons occur most frequently in organic molecules. Moreover, our model achieves the best performance when all categories of the motifs are integrated, implying a promising direction to discover more effective motifs.

heading: 6 RELATED WORKS

heading: 6.1 3D MOLECULAR REPRESENTATION
text: Deep learning has been widely applied to predict molecular properties during past decades. Small molecules are usually represented as lower-dimensional representations such as 1D linear sequence,
including amino acid sequences and SMILES (Weininger, 1988), or 2D chemical bond graphs. In spite of that, more evidence indicates that 3D space structures lead to better modelling and superior performance. 3D models becomes a popular way to capture these complex geometries in a variety of bio-molecular applications using CNNs (Anand-Achim et al., 2021; Jiménez et al., 2018) and GNNs (Cho & Choi, 2018). Nonetheless, aforementioned methods have hardly been extended to the self-attention mechanism that is proven to be good at grabbing contextual feature (Tang et al., 2018) and long-range dependencies (Vaswani et al., 2017).
Attempts have been undertaken to address that issue throughout Transformers. Initially, molecules are in the form of SMILES to obtain corresponding representations (Honda et al., 2019; Pesciullesi et al., 2020; Morris et al., 2020; Rao et al., 2021) and conduct pretraining (Chithrananda et al., 2020). Some researchers combine the characteristics of GNN and Transformer to solve generative tasks (Ingraham et al., 2019) or fulfill equivariance (Fuchs et al., 2020).

heading: 6.2 MOTIF-BASED METHOD
text: Motifs have been proven to benefit many tasks from exploratory analysis to transfer learning (Henderson et al., 2012). Various algorithms have been proposed to exploit motifs for contrastive learning (Zhang et al., 2020a), self-supervised pretraining (Rong et al., 2020; Zhang et al., 2021), and generation (Jin et al., 2020). However, none of previous work tries to embody those informative motifs in their model architectures.

heading: 7 CONCLUSION
text: In this study, we present a universal neural architecture, Molformer, for 3D molecular representations. Our model extracts motifs with semantic meanings from each molecule based on functional groups and learn customized embeddings to facilitate property predictions. Moreover, it adopts a convolutional position encoding method to make a full use of spatial information and augments the self-attention mechanism with multiplicate scales to catch local features. Furthermore, a simple but efficient downsampling algorithm is introduced to better accumulate representations of an entire molecule. Our experiments show the superiority of our model on various scientific domains.

heading: A EXPERIMENTAL SETUP
text: A.1 EXPERIMENTAL DETAILS
Molformer Architecture. A standard Molformer has 6 multi-scale self-attention layers, and each layer has 3 scales and 8 heads. Normally, scales are set by τ = [ρ2 , ρ, 2ρ], where ρ is the density of each corresponding dataset. The number of selected atoms K and the weight ratio in AFPS is set as 4 and 0.1, respectively. We use ReLU as the activation function and a dropout rate of 0.1 for all layers. The input embedding size is 512 and the hidden size for FFN is 2048.
For BBBP and ClinTox, we use Molformer with 2 multi-scale self-attention layers with 4 heads. The scales are 0.8, 1.6, and 3.0 Å. The dropout rate is 0.2 and 0.6 for BBBP and ClinTox, respectively. For BACE, we use a standard Molformer but with a dropout rate of 0.2.
Training Details. We use Pytorch (Paszke et al., 2019) to implement Molformer and data parallelism in two GeForce RTX 3090. An Adam (Kingma & Ba, 2014) optimizer is used and a lambda scheduler is enforced to adjust it. We apply no weight decay there. Each model is trained with 300 epochs, except for PDBbind where we solely train the model for 30 epochs. For QM7 and QM8, we use a batch size of 64 and a learning rate of 10−4. For QM9, we use a batch size of 256 and a learning rate of 10−3. All hyper-parameters are tuned based on validation sets. For all molecular datasets, we impose no limitation on the input length and normalise the values of each regression task by mean and the standard deviation of the training set. We used grid search to tune the hyper-parameters of our model and baselines based on the validation dataset.
Motif Generation. We adopt RDKit (Landrum, 2013) to search motifs. However, QM8 and QM9 do not provide SMILES representations but only 3D coordinates, thus we cannot pull out motifs from these datasets. As for PDBbind, we only extract motifs of small molecules and leave out motifs in proteins.

heading: B ADDITIONAL EXPERIMENTAL RESULTS
text: B.1 CONFORMATION CLASSIFICATION
Task and Data. In order to explore the influence of multiple conformations, we introduce a new task, conformation classification, to evaluate model’s capacity to differentiate molecules with various low-energy conformations. We use the recent GEOM-QM9 (Axelrod & Gomez-Bombarelli, 2020) experiments. More specifically, GEOM-QM9 is an extension to QM9 dataset. It contains multiple conformations for most molecules, while the original QM9 only contains one.
We randomly draw 1000 different molecules from GEOM-QM9, each with 20 different conformations. Models are required to distinguish the molecular type given different conformations. We take a half of each molecular conformations as the training set and another half as the test split. Since it is a multi-class classification problem with 1000 classes, we compute the micro-average and macro-average ROC-AUC as well as the accuracy for numerical evaluations.
Results. Molformer achieves a micro-average and macro-average ROC-AUC of 1.0 and 1.0, and an accuracy of 0.999. This indicates strong robustness of our model against different spatial conformations of molecules.
B.2 AFPS VS. FPS.
To have a vivid understanding of the atom sampling algorithm, we conducted a case study on a random crystal (see Figure 3). Points selected by FPS are randomized and exclude vital atoms like the heavy metal Nickel (Ni). With the adoption of AFPS, sampled points include Ni and Nitrogen (N) besides that they keep remote distances from each other. Moreover, FPS integrates too many features of trivial atoms like Hydrogen (H) while misses out key atoms, which will significantly smooth the molecular representations and lead to poor predictions. This illustrative example firmly shows the effectiveness of our AFPS to offset disadvantages of the conventional FPS in 3D molecular representation.

Review summaries:
The paper presents an approach to represent 3D atomic structure in a machine learning context.   The paper distinguishes between representations for "small" and "large" molecules based on a metric that takes the spatial extension and number of atoms in the molecule into account. Structures of small molecules are represented by an interatomic distance map. Large molecules are represented by what the authors refer to as a "sinusoidal function-based absolute position encoding method". In both cases, a transformer architecture is used on top of this initial representation. The authors further introduce a subsampling procedure to select a subset of points/ atoms and aggregate information.   The authors demonstrate quantitative results on datasets relating to 1) small molecule property prediction (QM datasets), 2) protein-ligand binding (PDBbind dataset), and 3) a dataset from material science on metalorganic compounds (CoRE-MOF dataset)   

This paper proposes 3D-Transformers, which is a variant of Transformer that incorporates the 3D spatial information. The authors develop a multi-scale attention module, an adaptive position encoding module, and a farthest sampling scheme for the 3D-Transformer. The authors validate the performance of 3D-Transformer on three different domains.

In the paper, the authors proposed a new Transformer dedicated to working with molecular representation. It is a modification of classical Transformer using the information of 3D coordinates of atoms. 3D-Transformer operates on a fully-connected graph with direct connections between atoms.