Learning Substructure Invariance for Out-of-Distribution Molecular Representations
Authors: Nianzu Yang
Kaipeng Zeng
Qitian Wu
Xiaosong Jia
Junchi Yan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Invariant Learning
Out-of-Distribution
Molecule Representation Learning

Abstract: Molecule representation learning (MRL) has been extensively studied and current methods have shown promising power for various tasks, e.g., molecular property prediction and target  identification. However, a common hypothesis of existing methods is that either the model development or experimental evaluation is mostly based on i.i.d. data across training and testing. Such a hypothesis can be violated in real-world applications where testing molecules could come from new environments, bringing about serious performance degradation or unexpected prediction. We propose a new representation learning framework entitled MoleOOD to enhance the robustness of MRL models against such distribution shifts, motivated by an observation that the (bio)chemical properties of molecules are usually invariantly associated with certain privileged molecular substructures across different environments (e.g., scaffolds, sizes, etc.). Specifically, We introduce an environment inference model to identify the latent factors that impact data generation from different distributions in a fully data-driven manner. We also propose a new learning objective to guide the molecule encoder to leverage environment-invariant substructures that more stably relate with the labels across environments. Extensive experiments on ten real-world datasets demonstrate that our model has a stronger generalization ability than existing methods under various out-of-distribution (OOD) settings, despite the absence of manual specifications of environments. Particularly, our method achieves up to 5.9\% and 3.9\% improvement over the strongest baselines on OGB and DrugOOD benchmarks in terms of ROC-AUC, respectively. Our source code is publicly available at \url{https://github.com/yangnianzu0515/MoleOOD}.

Learning Substructure Invariance for Out-of-Distribution Molecular Representations
Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, Junchi Yan'
Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence
Shanghai Jiao Tong University
mennmiaazzu,zenkaiipeenn ech jia.i/oooiiiiioooooo.o.
yaw.assonggaannnncciiii yanjunchi j@s edu. cn
Abstract
Molecule representation learning (MRL) has been extensively studied and rent methods have shown promising power for various tasks, e.g., molecular property prediction and target identification. However, common hypothesis of existing methods is that either the model development or experimental evalua- tion is mostly based i.i.d. data across training and testing. Such hypothesis can be violated in real-world applications where testing molecules could come from new environments, bringing about serious performance degradation or unex- pected prediction. We propose new representation learning framework entitled MolcOOD enhance the robustness of MRL models against such distribution shifts, motivated by an observation that the (bio)chemical properties of molecules are usually invariantly associated with certain privileged molecular substructures across different environments (e.g., scaffolds, sizes, etc.). Specifically, We intro- duce an environment inference model to identify the latent factors that impact data generation from different distributions in fully data-driven manner. We also propose a new learning objective to guide the molecule encoder lever- age environment-invariant substructures that more stably relate with the labels across environments. Extensive experiments on ten real-world datasets demon- strate that our model has a stronger generalization ability than existing methods under various out-of ont-of-distribution (OOD) settings, despite the absence of manual specifications of environments. Particularly, our method achieves up 5.9% and 3.9% improvement over the strongest baselines on OGB and DrugOOD bench- marks in terms of ROC-AUC, respectively. Our source code is publicly available at https://gii tht.bb.ttttttt/t.//g/// com./raangiaaann01111/1110000000
Introduction
Predicting molecular properties plays an important role in many related applications like drug discovery [13] and material design (51). These professional tasks conventionally take great efforts by experts e.g. in chemistry and pharmacology. Recent years have witnessed inspiring breakthroughs on building effective machine learning models for scientific discovery, and solid progress has been made along the avenue of ML-based molecule representation learning (MRL). general, MRL aims at embedding molecule into a vector in latent space foundation model, top of which the learned representations could be used for a variety of downstream tasks, such as target identification [69], retrosynthetic analysis [65]. search of antibiotics [54), virtual screening 440 for drug discovery, etc.
The challenge, however, is that existing MRL methods are mostly based on an underlying hypothesis that training and testing molecules are independently sampled from an identical environment, yet
Junchi Yan is the correspondence author who is also with Shanghai Al Laboratory.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Environment: Scaffold
Environment: Size
Scaffold 3C Ring
Scaffold
6C-Ring
Smaller Size
Larger Size
HO
Ho
Cyclopropanol
1,4-Cyclohxaneediol
Formic Acid
Citric
Figure 1: Two examples. Left: the shared substructure hydroxy (-OH) invariantly contributes the water solubility of the two molecules which contain different scaffolds, i.e. sampled from different enviromments by definition. Right: the water solubility of the two molecules with different sizes can be attributed to the shared substructure carboxy (-COOH) invariantly, where different sizes are regarded as indicators define different environments.
real-world environments are often dynamic and uncertain, which requires the model to effectively handle distribution shifts. fact, the available experimental molecule data are rather limited while the candidate molecules to be tested are often diverse, coming from unknown environments. Taking the virtual screening [40] as an example (which is a common protocol in drug discovery and usually for target identification), the prediction model is typically trained some known target proteins. However, some unpredictable events like COVID-19 may occur, bringing new targets from unknown distributions. Similar scenarios where training and testing data are sampled from different distributions are common real world, posing an urgency for strengthening current MRL methods regarding antrof-diitribution (OOD) generalization 41
Existing methods devised for out-of-distribution generalization mostly focus on Euclidean data such as images, while few endeavors OOD generalization non-Euclidean data [61 39 In particular,
molecules, as a kind typical non-Euclidean data, i.e., graph-strucctured data, different from visual data in nature. The work [27] point out that existing OOD models [3] 55
18] 70, 50) fail
to exhibit significant improvement on MRL tasks against distribution shifts and even the simple Empirical Risk Minimization (ERM) (56) method outperforms these latest methods, which is also empirically verified by [16]. We aim to develop an OOD method tailored for molecules to solve the OOD generalization problem MRL in this paper.
We incorporate an effective prior in the molecule domain into our model design: the (bio)chemical properties of molecule are usually associated with a few privileged molecular substructures, which has been consistently shown by studies [31
72 29 across bio-informatics, pharmacy, and data mining. The common practice specifies environments as some prominent information of the molecules e.g. scaffold pattern [32] 23] and molecule size [27]. Fig. [ provides two illustrative examples. Let's first take look the left example, where two molecules Cyclopropanol (C₂ H6) and =4-Cyycloeexanediol H))))) contain different scaffold pattern the former is 3C-ring and the latter is 6C-ring. Thus, the data-gena mata-generating environments and the induced distributions which these two molecules are sampled from can be considered different [23]. Though sampled from different distributions, they are both readily soluble in water due the invariant substructure hydroxy (24) shared across different environments. As for the example on the right of Fig. the sizes of two molecules Formic Acid (CH202) and Citric Acid (O)H&O7) differ a lot. Consequently, they can also be considered as being sampled from different environments. Owing to the shared invarint substructure carboxy (-COOH), they are both readily soluable in water, too. Hence, promising paradigm would be to learn the causal data-generating invariance from the substructures across environments, regarding a certain property, for the OOD generalization purpose.
48
Another important observation for consideration is that existing specifications for environments are often handcrafted or rule-based and not structured, which could provide insufficient information for capturing the fundamental relations across domains from the casual data Sata-generating perspective.
*As a 2-D structural molecular framework [5], the scaffold reduces the chemical structure of molecule to its core components, which can be obtained by removing side chains and only reserving the rings and parts connecting rings [67]. The scaffold can be indicator to define specific environment [32 23).

---

Besides, some studies [27 [6] show that directly utilizing such environment labels as input when adapting existing OOD generalization methods MRL tasks can be problematic. Furthermore, manual specifications of environments may be unavailable in reality. Hence, we aim to develop label-free model that does not rely on the above ad-hoc environment labels. As shown later, our model can infer the environment labels in an unsupervised manner, namely for environment clustering.
To achieve robust molecule representation for OOD generalization and overcome potentially unreli- able environment labels, we devise a new MRL framework without explicitly using the environment label information. We first formulate OOD generalization for molecular property prediction by introducing a latent variable for environment essential cause behind the failure of existing MRL models and propose a new learning scheme based on the invariance principle 447 44 3, [1]. The training procedures contain two steps: 1) optimize an environment inference model from training data; 2) optimize a molecule encoder and a predictor. Our general framework can integrate existing GNN backbones and achieve improvements on four OGB molecular property prediction tasks [23], as shown in our experimental results. for a newly released benchmark for drug-oriented OOD learning [27], even without access to environment labels, our method can still outperform state-of-the-arr models that rely on environment labels for training in five out of six datasets. The contributions of this paper are:
that affect the data generation. Then we analyze the
We formulate the out-of-distribution (OOD) generalization problem for molecule representation learning (MRL), by particularly incorporating an important observation that the substructure of molecule convey invariant casual information across environments, regarding certain property prediction tasks. To our best knowledge, this is the first work that formulates the OOD problem in MRL background and proposes to leverage the invariance principle which opens new perspective for handling mnnstructure-wwar distribution shifts.
Under the new learning objective to learn robust representations. In particular, our model does not require environment labels which in fact can be noisy and unreliable, but instead achieve environment inference in an unsupervised manner. This design endows our model with practical applicability for molecular OOD learning where the manual specifications of the enviromments are often unavailable.
principle with specific substructure invariance priors, we propose
We conduct extensive experiments on ten public datasets. Results demonstrate that our model yields consistent and significant improvements over various existing MRL methods as backbones and also achieves competitive or even superior prediction compared to state-of-the-art models tailored to OOD learning with environment labels used as extra inputs in both training and testing. Particularly, our method achieves up to 5.9% higher ROC-AUC on public OGB molecular property prediction benchmarks than the counterpart model trained with traditional objective. Besides, for drug-oricnn -oriented benchmarks DrugOOD, when environment labels are not used, our model still outperforms several SOTA approaches tailored for general OOD learning (using environment labels as extra training information) by up to 3.9% w.r.t. ROC-AUC.
2 Backgrounds and Related Works
Out-of-Distribution Generalization. Deep neural networks are prone suffering significant per- formance degradation under distribution shifts, motivating a surge of works on OOD generalization. Recent studies 49 3 61 assume that there is a potential environment variable e accounting for the distribution shift between the training and testing data. In general cases the goal is to predict the target label y given the associated input x. Then, the OOD problem could formally formulated as:
min max ((x.))-p(x.y/===)) (() y)]e], eEE
(1)
where & denotes the support of environments, f(-) is the prediction model and I(:5-) represents loss function. Notice that Ee x/xxx))ee)))) [(f(()) y)]]] is called the risk function under a given environment e and denoted as Rx(x", ,y"
33
regarding invariant predictor learning, for solving the OOD generalization problem. These methods propose to find an invariant predictor that could uncover invariant relationships between inputs and targets across all environments (33]. The invariant predictor aims learn an invariant representation satisfying such a invariance principle: 1) sufficiency: shows sufficient predictive power for the target, 2) invariance:
Invariant Learning. There is an emerging line of research 49
3
10 12
3

---

contributes to equal (optimal) performance for the downstream tasks across all environments. Recent works adopt the invariance principle cornerstone for handling various distribution shifts on semi- structured data like graphs [6] 6 and sequences [66]. To our knowledge, our work is a pioneering attempt that leverages the invariance principle and incorporate useful domain knowledge for handling molecular graph classification tasks under distribution shifts.
Molecule Representation Learning. Existing molecule representation learning methods can be classified into two categories. The first is SMILES-based methods where SMILES refers to Sim- plified Molecular Input Line Entry System 22. They use language models process the textual representation (SMILES) of a molecule, for example, Transformer [57] or BERT [T]. SMILES is a linear encoding for molecules and highly depends on the traverse order of molecule graphs. There- fore its expressiveness is limited for problems like medication recommendation which we believe calls for ane-grained molecular structure extraction. Beyond the above linear encoding protocol, structure-based methods are also developed, which can be further classified into fingerprint-based and graph neural networks (GNN)-based methods. The molecular fingerprint techniques date back the Morgan fingerprints
43 However, those fingerprint-based methods are often handcrafted and not trained in an end-to-end fashion (26). Since molecules can be viewed as structured graphs, methods or graph transformers
tailored for graph data, such as graph neural networks be used to learn molecule representation.
28 19 24
62 can
Existing general OOD methods [ 55 18, 70 50 are not tailored to such non -uclliddd structured data, i.c. molecules. In several recent works on molecule property classification tasks [7] 59] the importance of molecular substructures has been emphasized and such inductive bias is incorporated into the design of those models. However, they are still based on the i.i.d. assumption and do not leverage those invariant substructure across different environments to achieve robust representations. In this paper, we propose a general framework orthogonal to these MRL studies to bridge OOD and MRL, which can adopt any existing MRL methods as the backbone to improve their robustness.
3 Methodology
3.1 Problem Formulation
We propose a OOD generalization framework tailored for molecule representation learning, entitled MoleOOD. All the random variables and the corresponding realizations are denoted bold and thin letters, respectively. We first formulate the OOD generalization problem for MRL.
OOD Generalization Problem on Molecule Representation Learning. A molecular graph can be represented as G = (V,E), E), where V is the graph's 's node set corresponding to atoms constituting the molecule and E' denotes the graph's 's edge sets corresponding to chemical bonds. The training and test- ing molecule graph datasets are denoted as grrain [(G,, } /
Nerain and great = {(G ys) ) Nen Notice that the test dataset is drawn outside the distribution of the training dataset. The goal of molecule representation learning task is to predict the target label y given the associated input molecule G. Based on Eq. [
we can formulate the OOD problem on MRL tasks as:
min max BiG.s y.)le.
EEE
The difficulty of this problem is that the training data only cover very limited environments in while the model is expected to perform well on all the environments.
We elaborate our approach in the context of molecule property classification tasks in this paper. Existing MRL methods do not differentiate invariant and spurious substructures. Hence, the spurious correlations between irrelevant substructures and the target label will be encoded to learned molecular representations. When tested on unseen environments, the downstream classifier will be easily misled by these spurious correlations 3. With the knowledge that (bio)chemical properties of a molecule are usually associated with few privileged substructures [31 48 72 29], we aim to suppress such spurious correlations and leverage environment-invarinn
substructures that more stably relate with the labels across environments to learn invariant molecular representations. Notice that the learned invariant molecular representations should satisfy the invariance principle mentioned in Sec. 2 We next introduce our method formally and then give the instantiation of our model.

---

3.2 Model Formulation
The framework contains two parts, the fronted molecule encoder for learning an "invariant representation" of the input molecule graph and the back-end predictor for final prediction. Solving the formulation in Eq. directly is intractable in practice since we cannot know all the environments, ie, obtain complete support set E. We resort to minimizing the expectation of risks from different environments known in the training data,
min E. 4
s.t. y e ((G),,
(3)
where f = wo 4 and 1I denotes probabilistic independence. All learnable parameters of the molecule encoder & and the predictor w are included in 0. Different from Eq. 2 add an extra invariance constraint y IL causality is challenging, we could rethink the problem the basis of information theory. Recall that we hope to let the molecule encoder leverage molecular representation ((G) given molecule G. Our goal is to maximize the predictive power of (GG) on y, which can be measured by mutual information between (G) and Mcanwhile probabilistic independence between y and e given & (G) can be achieved via minimizing their mutual information. For convenience, we denote Φ (G) as z and Eq. 3 can be approximately solved by:
(G),, which is used to suppress spurious correlations [10]. Since assessing
iewtonmmentinnvarinn substructures and learn
max 1 (z:y), s.t. min ely;) 42
w,6
Treating the outputs of and t as distribution go(2 and (o(y)z) respectively, Eq. 4 can be specified as:
max I(z;y), 8.t. de (y (z G)
min
elyze
(5)
as (x)
Now, we have arrived at a clearer but still intractable optimization objective. Before specifying the practical instantiation of Eq. 5 let's discuss on the environment variable e first.
In practice, due to the non-trivial efforts to label the molecular environments, manual specifications of the environments may be unavailable in many cases. We may directly label molecules different environments in terms of their scaffolds when the environment label is unavailable. But this is unreasonable in practice, because the final total environment number will be too large. Taking the dataset HIV for molecule property prediction tasks released by Open Graph Benchmark [23] as an example, OGB uses scaffold to split the molecules into different environments. Assuming that we regard each scaffold as an environment directly, 41, 127 molecules in HIV are partitioned into 19, 076 environments (see details Appendix D). This environment count is much larger than other OOD datasets from other domains, e.g. Camelyon! + 4. CivilComments [l g etc. Even though some datasets may provide manual specifications of environments, the environment counts are also too large, which is unfriendly to existing OOD models (27 [6]. Therefore, we propose to design an environment -nneeeenceeeeeeeeeeeee model & to partition the molecule into different environments with a relatively smaller environment count. We denote the environment count as hyper-parameter k.
Given prior p(e we need to maximize the log likelihood of pr (y G) and then obtain the posterior Pr (e y), which are parameterized by т. Since there is no analytical solutions to the true posterior, here we use variational inference (VI) to approximate it. Specifically, we introduce a variational distribution (e G. y) parameterized by to approximate (e)))
Proposition 1. The Evidence Lower BOund (ELBO) of the observed molecule graph and correspond- ing label tuple (G. y): L(T, k; (G. F.g, [log P- (y/ G,e)] - D (e. y) p- (elG)).
Our goal to minimize the Kullback-Leible (KL) divergence between g. G. y) and pr (el G, ie. Dx (e G. y) || p. (e(G.y) y)), which is equivalent to maximizing the ELBO in Proposition Then, the objective used to train this environmenntinference model is transformed to:
Leibo k;
[g logp- e)] Dxu p(e[
(6)
(G.vec
'Camelyon 17 is for tumor prediction, partitioning 455, 95- issue slides into 5 environments. 'CivilComments is for toxicity prediciton, partitioning 448, 000 online comments into 16 environments.
5

---

(a) Environment Inference 4x
Environment
(G,y)
Classifier
q. (. G.y)
eloo
Conditional
P+
GNN
e-
Complete Encoder
decompon
Linv
substructuress
query
OH
Predictor
Substructure
attentive
Encoder
pooling
(b) Molecular Representation Learning
Figure 2: Overview of our model. The whole training procedure is divided into two stages: 1)0p- timize the environment-inferenee model. Given an input molecule yG we first infer the latent environment variable This stage is trained under the guidance of Letbo- 2) Optimize the molecule encoder and the final predictor guided by Linv-
Let's look back to the objective given Eq. 5..55pnn and give an equivalent tractable objective in practical instantiation, which involves the environment-imferencc model defined above:
Line (0,
|lo (o((G) E.ce ca
SE,
log 2o G))
gg
(G.))
(G.y)
(7)
where 9 consists of the pairs of molecular graph G and corresponding label y under environment e.
Theorem 1. With ge(ylz) treated as variational distribution, minimizing term ! in Eq. 7 contributes mingo x). 1(y; elz). letting z show equal performance for the downstream tasks across all environments, i.e. p(y e) p(ylz).
Theorem 2. Regarding q (2) as a variational distribution, minimizing term 2 in Eq. 7 equuls (y)X))lyy2))qool G) y) letting show sufficient predictive power for downstream lasks.
Serving as theoretical justifications, Th. and Th. Drveal that optimizing the objective in Eq. 7 forces the learned representation to satisfy the invariance principle mentioned in Sec. ? thus ensuring valid solution for OOD problem defined in Eq. 2 Due to the limited space, the detailed proofs can be found in Appendix
3.3 Model Instantiations and Training
Environment-infere Module. For the approximate posterier model (e G, in principle should design module, entitled Environment Classifier, that takes (G,y) as the input and outputs the probabilistic distribution of e. We use a Graph Isomorphism Network (GIN) [63] to learn graph representation given G. Then, the concatenation of this graph representation and label vector fed to feed-forward network obtain probabilistic distribution with regard to We could set the prior p. (el G) to Uniform distribution or Gaussian distribution. As for p, (yl G, e), we also choose GNN model (e.g. GIN) followed by softmax activation function to model it. We call this module Conditional GNN because conditions on c. takes (G, c) as the input and outputs the probabilistic distribution of y.
The Molecule Encoder & The Final Predictor. Recall that we aim to learn an invariant substructure- aware molecular representation. Given molecule G, we can choose any molecule representation learning method learn a representation rG for the complete molecular graph. This part en- titled Complete Encoder. Mcanwhile, we decompose the input molecule into set of chemical

---

substructures using molecule segmentation method, e.g. breaking retrosynthetically interesting chemical substructures (BRICS) [44, which available an API in RDKit [34]. For each sub- structure, we consider using simple GNN to learn corresponding representation. We call this GNN Substructure Encoder. Then, considering rG as query with regard to substructures, we operate attentive pooling on these substructure representations to obtain new substructure-aware molecular representation. We then use this substructure-ave aware representation for downstream task. Guided by proposed learning objective, we can encode some invariant relationships between certain substructures and target properties into this representation. The Complete Encoder, the Substructure encoder and the attentive pooling operation constitute our Molecule Encoder o. As for the Predictor w, we implement it with a multi-layer perceptron, followed by a softmax function. The overview of our model is demonstrated in Fig.
Training. We adopt simple yet efficient two-stage training strategy to search for optimal parameters and the training procedure of our method is summarized in Algorithm !
optimizing the environment-inference model: k", T <- arg mäXx,7 Lelbo (T, Ki Gtrain).
2) optimizing the molecule encoder and the predictor: 6* +- arg ming Livr (0; Gerain T).
Algorithm 1:
The training procedure.
Input: Dataset Grrain {GG,, yt) Mi Number of training epochs for environment inference module E1: Number of training epochs for the molecule encoder and the predictor E2; Batch size B.
Output: Trained parameters 0.
1 Initialize parameters 0, F and k;
for i < to Ei do Sample data batches B = (91. 92. , from Grrain with batch size B; for jor]1 to do
9k}
Compute batch loss feebb I, k: 9j) according to Eq. 6 Backkpoppggte Letbo and optimize parameters т, Kr
7 Freeze the parameters Ki T:
for i <- 1 to Eg do
9
Sample data batches B = (91. (9b,...,,]]]] from grrain with batch size B; for j+rj+r1
G2....,
9&
10
do
11
Determine the environment of each sample (G, y) in G. by arg max. ou (e(G, y);); Compute batch loss Line (0; Gk, 7) according Eq. Backpropagate Line and optimize parameters 0;
12
13
14 Output the parameters 0;
4
Experiments
Experiments are performed on 10 benchmark datasets and repeated 5 times with mean and standard deviation reported, running on a machine with 10-20X CPU, RTX 3090 GPU and 128G RAM.
4.1 Datasets and Setups
Datasets and protocols. The four datasets BACE, BBBP, SIDER and HIV, are from by Open Graph Benchmark (OGB) (23). We use the default train/val/test split with ratio 8:1:1. Each split contains a set of scaffolds (almost) different to each other. Hence we believe that to certain degree, it provides an OOD test-bed as different Scaffold often suggest different data-generation environments. The other six datasets are generated by the dataset curator provided by DrugOOD [27]. DrugOOD provides more diverse splitting indicators than OGB, including assay, scaffold and size. To comprehensively evaluate the performance of our method under different environment definitions, we adopt these three different splitting schemes on categories IC50 and EC50 provided in DrugOOD, Then we obtain six datasets, EC50-s and IC50-s where the suffix * specifies the splitting scheme i.e. -astl//CCSO-assaa/aaaaoolldd
Notice that only the six datasets from DrugOOD provide manual specified environment labels. Refer to Appendix D for more details of datasets.

---

0.83
Eley
0.82
1.0
ROC A0A
0.82
ROC-M
0.8
Elos
0.80
0.6
0.78
0.79
0.4
0.76
15
0.77
0.5
0.0
20
40
60
80
100
Epoch
(a)
(b)
(c)
Figure 3: (a) Varying the specified environment number k. (b) Varying the trading-off parameter 3 in Eq- 1 (c) Risk curves of environments in the training process. All results from "GraphSAGE + ours.' on BACE dataset.
Table Performance comparison with baselines on out-of-distribution molecular property predic- tion datasets from Open Graph Benchmark (OGB) 23 in terms of ROC-AUC (%), namely, BACE, BBBP, SIDER and HIV. The best and the runner-up results are highlighted in bolded and underlined respectively. We emphasize the comparison against ** + virtual node', variant of the original method augmented by an additional node connecting to all nodes in the raw graphs [99 25 38).
Methods
BACE
BBBP
SIDER
HIV
GCN 30
80.01 + 3.49
67.92 + 1.07
58.90 + 1.30
76.35 + 2.01
GCN virtual node
77.51 3.07
68.19 1.86
60.71 1.34
75.76 2.21
GCN ours.
84.33 1.07
70.62 = 0.99
63.38 + 0.67
77.73 0.76
GIN
63
77.83 + 3.15
66.93 + 2.31
59.05 + 1.47
76.58 + 1.02
GIN virtual node
79.64 2.02
66.77 + 0.95
59.12 + 0.95
77.11 + 0.96
GIN ours.
81.09 2.03
69.84 1.84
61.63 1.08
78.31 0.24
GraphSAGE 21
77.41 1.19
70.58 0.58
58.00 0.95
76.98 1.13
GraphSAGE + virtual node
78.34 +: 2.08
69.29 + 0.99
59.48 + 1.37
77.28 = 1.53
GraphSAGE + ours.
82.95 0.85
71.02 0.75
61.09 0.28
79.39 0.51
Metric. As the concerned property prediction tasks all relate to classification, we report the ROC- AUC score which is also in line with previous MRL works 668 64 58).
Baselines. Ideally, any MRL method can be adapted into our method as backbone to improve their generalization ability against distribution shifts. We adapt three backbones: GCN 300, GIN and GraphSAGE (21) into our method. We compare the adapted version with the original method. We also compare against another augmented version "+ virtual node" [19 [55] 38) Furthermore, we compare our method with six OOD generalization methods on MRL tasks: ERM [56). IRM (3).
[63
DeepCoral (55). DANN [18,, MixUp 70) and GroupDro [50]. Due the fact that most of these methods require the manual specification of environments in dataset, we report this comparison on datasets from Dru only. Each of the method is configured using the same parameters reported in the original paper or selected by grid search. For the sake fairness, the embedding size of all
methods are set be equal in comparison. We specify the training details in the Appendix
4.2 Performance Comparison
Improvements existing MRL methods. As demonstrated in Table baselines obtain consistent improvements after adapted to methods across all the four datasets released by OGB in terms of ROC-AUC, Our method also beats the augmented version, "+ virtual node". of baselines on all datasets, i.c. adding virtual node. The results indicate that, orthogonal to prior studies MRL, our method is a general framework which can incorporate existing MRL methods and improve their generalization ability for OOD data. We attribute the superior performance of our method in molecular properties predictions under OOD setting to that, our proposed learning objective enforces the model learn environment-invario representations against distribution shifts.
Superiority to other OOD generalization methods. summarizes the results comparsion with six state-of-the thet methods tailored for OOD learning, where obtain the following observa-

---

Table 2: Evaluation with other OOD generalization methods on out-of-distribution datasets from DrugOOD (27] in terms of ROC-AÜC (%). The best and the runner-up in each columns are highlighted bolded and underlined respectively. Note the baselines except ERM and MixUp all require environment labels. All methods including ours use GIN 63 as backhones.
Datasct
IC50
EC50
Environment
Assay
Scaffold
Size
Assay
Scaffold
Size
ERM [56]
70.93 1 2.10
67.31 1 1.72
67.40 1 0.56
69.35 1 7.38
63.92 1 2.09
60.94 - 1.95
IRM
70.85 + 2.41
66.06 + 1.23
58.46 + 2.11
69.94 + 1.03
63.74 + 2.15
58.30
1.51
DeepCoral
55
69.82 1: 4.23
66.36 1: 2.57
59.21 +: 2.09
69.42 +: 3.35
63.66 +: 1.87
56.13 1.77
DANN 18
70.00 + 1.03
63.61 2.32
65.77 = 0.47
66.97 = 7.19
64.33 = 1.82
61.11 = 0.64
MixUp 70
70.22 1: 3.66
66.43 +: 1.08
67.77 +: 0.23
70.62 = 2.12
64,53 - 1.66
62.67 = 1.41
GroupDro
50
69.98 + 1.74
64.09 + 2.05
58.46 + 2.69
70,52 + 3.38
64.13 + 1.81
59.06
1,50
Ours.
71.38 + 0.68
68.02 + 0.55
66.51 = 0.55
73.25 = 1.24 66.69 = 0.34
65.09 = 0.90
Table 3: Ablation study on EC50-* by ROC-AUC (%). We show the results of MixUp that performs best among baselines on all EC50- datasets and the naive ERM, which minimizes the average empiri- cal loss on training data, for comparison. Notice that ERM and MixUp don't 't require manual specified environments labels. We also present the results of DANN, which requires manual specifications of environment and obtains competitive results with MixUp. All methods use GIN (63 backbone.
Method
Assay
Scaffold
Size
ERM (GIN+ + ERM loss)
69.35 + 7.38
63.92 + 2.09
60.94 + 1.95
MixUp
70.62 + 2.12
64.53 = 1.66
62.67 = 1.41
DANN
66.97 = 7.19
64.33 = 1.82
61.11 = 0.64
Our architecture + ERM loss GIN + new learning objective
71.44 + 2.02
65.99 0.42
64.23 = 0.71
72.07 + 1.14
66.33 - 1.38
64.43 + 1.10
DANN using our inferred environment label Our model using given environment label
68.83 + 2.44
64.95 + 1.07
62.56 + 1.54
71.94 + 2.77
66.29 0.85
63.38 1.20
Our full model
73.25 + 1.24 66.69 + 0.34
65.09 + 0.90
tions. Except on IC50-size, our method outperforms all baselines across all datasets due its ability to enforce the molecule encoder to leverage invironment-invaiian substructures that more stably relate with the labels across environments. Our method ranks the third on IC50-size after MixUp and ERM. Different from the other methods, Mi constructs more training exmaples and uses more data to train the model. That's 's why MixUp obtains best performance among all methods on IC50-size in our analysis. As for ERM, [27 16 have pointed out that simple ERM shows better performance compared to subsequent OOD methods when datasets have relatively large environment counts. Even though we have set the environment number l to a smaller value than the ground-trut truth number given by the dataset, we still need to prevent & from being too small (see discussion in Sec. 4.4). leading to our poorer performance than ERM on IC50-size.
4.3 Ablation Study of Components
We analyze the contributions of different model components to the final performance in this section. Table rreportss detailed ablation experimental results on EC50-assay, EC50-scaffold and EC50-size.
Attention-based architecture. We study the impact of the attention-based architecture introduced in Sec. 3.3 by assembling this architecture with ERM loss. We beat ERM and MixUp only with this architecture on three datasets. The results show that learning a representation for each substructure and then attentively aggregating these learned representations to obtain final substructure-awarc representation performs better than learning a representation for a complete molecular graph directly. This verifies assumption that the substructure perspective is of importance to boosting performance of existing MRL methods. With the aid of such a substructure -gaiiner learning architecture, the impact of our learning objective can be further strengthened.
New learning objective. To evaluate the impact of our proposed new learning objective, we equip GIN with this new objective. We can see compared to using the substructure-grainet learning archi- tecture only, only using the proposed new learning objective can bring more significant improvement. Thus, can attribute the main superiority of our full model to this new objective. Combined with
9

---

the architecture discussed above, the new objective is able to better guide the molecule encoder to learn cattromment-invariinn molecular representations against distribution shifts.
Environment inference. Now we turn to investigate the performance with respect to our proposed invironment-inferene module. One motivation for this module is that in reality manual specifications of environments may be unavailable due to the high price for labeling environments by experts. But when environment labels are available, how will be performance be like if directly utilizing the given environment partition? An ablation study is targeted on this, Taking the EC50-assay dataset as an example, it has given the environment partition and specifies 47 environments in total. We utilize the given environment partition directly and keep the remaining parts in line with our full model. The results show that utilizing the given environment label, our method still can beat ERM and MixUp. But compared to our full model where we set the environment number i to 20, it obtains inferior performance. Additionally, to further examine the effectiveness of our proposed environment inference method, we relabel the environment for each molecule for DANN according to our inferred environment partition. We can see that based on the new environment partition, DANN obtains better performance than using the initial given environment labels across three datasets. The reason why inferring environment instead can outperform directly using the given environment label is mainly and not structured. In contrast,
due to the existing given partitions are often handeraftedrule-baar letting the model learn a environment partition by itself may be more effective to some degree.
4.4 Hyper-parameler Sensitivity Study
We investigate the sensitivity of our method to these two hyper- parameters: the specified number of environments k. the trading-off parameter 3 in Eq. 7 Fig. 3(a) shows the performance regarding different environment number k. It shows that the pertormance of our methods degrades when k is too small (e.g. & = 5) or too large (e.g. k = 15). When h = i.e. we regard all training data as from only environment, the performance is the poorest. This justifies that partitioning the training samples into different environments is necessary. Fig. (3b) shb)s the results of our method by varying the trade-off parameter 3. Our method obtains the worst performance when 8 = This is mainly because Eq. 7 is reduced to the first term when 3 = 0. According to Theorem without the second term of Eq. 7 the sufficiency condition of invariance principle cannot be satisfied, resulting in the performance degradation.
4.5 Risk Dynamics
Additionally, to shed insights of the ability of our method to lower the risks of different environments, we visualize the risk dynamic curve of some environments in Fig. 33c) As is shown in Fig. B the difficulties of decreasing the risk on different environments are different. Though the risks some environments vibrate violently at the beginning of training process (e.g. Env I and Env 2), with time elapsing, risks on all environments can decrease stably.
5 Conclusion
We have proposed general framework which can incorporate any existing MRL method as backbone to improve their generalization ability against distribution shifts. Specifically, we devise new learning scheme with its equivalent practical instantiation. We also develop an environment inference model to identify each molecule's corresponding environment without need of manual specifications of environments. Extensive experimental results on ten datasets demonstrate that our model yields consistent and significant improvements over various existing MRI methods as backbones. Addi- tionally, our model achieves competitive or even superior performance compared to state-of-the-an models designed for OOD learning that require manual specified environment labels as extra inputs.
Acknowledgement
This work was partly supported by National Key Research and Development Program of China A2100A,0) National Natural Science Foundation of China (61972250, 72061127003) and Shanghai Municipal Science and Technology (Major) Project (22511105100, 2021SHZDZX0102),
10

---

References
[1] Al-Mawsaa,i, R. Dayam, L. Taheri, M. Witvrouw, Z. Debyser, and N. Neamati. Discovery of novel non-cytotoxic salicylhydrazide containing hiv-1 integrase inhibitors. Bioorganic & medicinal chemistry letters, 17(23):6472-6475 2007.
[2] E. Anderson, G. D. Veith, and D. Weininger. SMILES, line notation and computerized interpreter for chemical structures. US Environmental Protection Agency, Environmental Research Laboratory, 1987.
[3] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint wwX.vv:190..028980 2019.
[4] Bandi, Geessink, Q Manson, M. Van Dijk, M. Balkenhol, M. Hermsen, E. Bejnordi, B. Lee, K. Paeng. A. Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE transactions on medical imaging, 38(2):550-566 2018.
[5] G. W. Bemis and M. A. Murcko. The properties of known drugs. molecular frameworks. Journal of medicinal chemistry, 39815)22887-2893, 1996.
[6] B. Bevilacqua, Y. Zhou, and B. Ribeiro. Size-invariant graph representations for graph clas- sification extrapolations. In International Conference on Machine Learning, pages 837-851, 2021.
[7] G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Advances Neural Information Processing Systems (NeurIPS), pages 1178-2186, 2011.
[8] D. Borkan, Dixon, J. Sorensen, N. Thain, and L. Vasserman. Nuanced metries for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pages 491-500, 2019.
[9] Bühlmann. Invariance, causality and robustness. Statistical Science, 35(3):404442) 2020.
[10] S. Chang, Y. Zhang, M. Yu, and T. Jaakkola. Invariant rationalization. In International Conference on Machine Learning, pages 1458. PMLR, 2020.
[11] T. Cofala and O. Kramer. An evolutionary fragment-based approach to molecular fingerprint reconstruction. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1156-1163, 2022.
[12] E. Creager, J.-H. Jacobsen, and R. Zemel. Environment inference for invariant learning. In International Conference Machine Learning, pages 2189-2200. PMLR, 2021.
[13] L. David, A. Thakkar, R. Mercado, and O. Engkvist. Molecular representations ai-driven drug discovery: a review and practical guide. Journal of Cheminformatics, 12(1):1-22, 2020.
[14] Degen, C. Srggschidd-erlace A. Zaliani, and M. Rarey. On the of compiling and using drug-like' chemical fragment spaces. ChemMedChem, 2008.
[15] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 10...0/811000400 2018.
[16] M. Ding, K. Kong. J. Chen, J. Kirchenbauer, M. Goldblum, D. Wipf, F. Huang, and T. Goldstein. closer look at distribution shifts and out-of-distributioo generalization on graphs, 2022.
[17] V. P. Dwivedi and X. Bresson. A generalization of transformer networks to graphs. CoRR, abs/2012.09699, 2020.
[18] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and Lempitsky. aoverriadversariil training of neural networks. The journal of machine learning research, 17(1):2096-2034 2016.

---

[19] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272 PMLR, 2017.
[20] M. Glymour, J. Pearl, and N. P. Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
[21] W. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[22] M. D. Hoffman, of Machine Learning Research, 2013.
Blei, C. Wang, and Paisley. Stochastic variational inference. Journal
[23] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 18-2233, 2020.
[24] Ishida, T. Miyazaki, Y. Sugaya, and S. Omachi. Graph neural networks with multiple feature extraction paths for chemical property estimation. Molecules, 26(1 1):3125, 2021.
[25] K. Ishiguro, S.-i. Maeda, and M. Koyama. Graph warp module: auxiliary module for boosting the power of graph neural networks. arXiv preprint arXiv:1902.01000 2019.
[26] S. Jaeger, S. Fulle, and S. Turk. Mol2vec: unsupervised machine learning approach with chemical intuition. Journal of chemical information and modeling, 2018.
[27] Y. Ji, Zhang. J. Wu, B. Wu, L.-K. Huang, T. Xu, Y. Rong. L. Li, J. Ren, D. Xue, et al. Drugood: Out-o--distribution (ood) dataset curator and benchmark for ai-aided drug discovery-a focus on affinity prediction problems with noise annotations. arXiv preprint arXiv:2201.0966
2022.
[28] W. Jin, C. Coley, R. Barzilay, and T. Jaakkola. Predicting organic reaction outcomes with weisfeiler-lehman network. Advances in neural information processing systems, 30, 2017.
[29] Y.-T. Kao, S.-F. Wang, M.-H. Wu, S.-H. Her, Y.-H. Yang. C.-H. Lee, H.F. Lee, A.-R. R. Lee, L.-C. Chang, and L.-H. Pao. A substructure-bart screening approach to uncover n-nitrosamines in drug substances. Journal of Food Drug Analysis, 30(1), 2022.
[30] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks In 5th International Conference Learning Representations, ICLR 2017, 2017.
[31] Klekota and F. P. Roth. Chemical substructures that enrich for biological activity. Bioinfor- matics, 24(21):255 1:2558-25225 2008
[32] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[33] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville. -uitof-dissribution generalization via risk extrapolation (rex). In International Conference Machine Learning, pages 5815-5826. PMLR, 2021.
[34] G. Landrum et al. Rdkit: Open-source cheminformatics. 2006.
[35] J. Lee, Y. Lec, Kim, A. Kosiorek, S. Choi, and Y. W. Tch. Set transformer: A framework for attention-based permutation-innvaiann neural networks. In International conference machine learning, pages 3744-3753. PMLR, 2019.
[36] A. Leman and B. Weisfeiler. A reduction of a graph to a canonical form and an algebra arising during this reduction. Truuchno-Technicheskkyy Informatsiya, 2(9): 1968.
[37] X. Q Lewell, B. Judd, P. Watson, and M. Hann. Recap retrosynthetic combinatorial analysis procedure: powerful new technique for identifying privileged molecular fragments with useful applications combinatorial chemistry. Journal of chemical information and computer sciences, 38(3):511-522, 1998.
12

---

[38] Li, D. Cai, and X. He. Learning graph-level representation for drug discovery. arXiv preprint arXiv:1709.03741 2017.
[39] Z. Li, Q. Wu, F. Nie, and Yan. Graphde: generative framework for debiased learning and out-of-distribution detection graphs. In Advances in Neural Information Processing Systems, 2022.
[40] Lim, Ryu, K. Park, Y. J. Choe, Ham, and Kim. Predicting drug target interaction using a novel graph neural network with 3d structure-embeddeed graph representation. Journal of chemical information and modeling, 59(9):3981-33) 1-3988, 2019.
[41] Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In Conference on Learning Theory (COLT), 2009.
[42] D. Mendez, A. Gaulton, A. P. Bento, J. Chambers, M. De Veij, E. Félix. M. P. Magariños, J. F. Mosquera, P. Mutowo, M. Nowotka, et Chembl: towards direct deposition of bioassay data. 2019.
Nucleic acids research, 47(D1))
[43] H. L. Morgan. The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service. Journal of Chemical Documentation, 1965.
[44] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning (ICML), pages 10-18, 2013.
[45] N. Neamati, H. Hong, J. M. Owen, S. Sunder, H. E. Winslow, J. L. Christensen, H. Zhao, T. R. Burke, G. W. Milne, and Y. Pommier. silleyllyddrrinn-coottinnnng inhibitors of hiv-1 integrase: implication for a selective chelation in the integrase active site. Journal of medicinal chemistry, 41(17):32023322 1998.
[46] Pearl et Models, reasoning and inference. Cambridge, UK: CumbridgeUnivversiyyPress 19:2, 2000.
[47] J. Peters, P. Bühlmann, and N. Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series (Statistical Methodology), 78(5):947-1012, 2016.
[48] C. Phanus-umporn, W. Shoombuatong, V. Prachayasitikul, N. Anuwongcharoen, and C. Nan- tasenamat. Privileged substructures for anti-sickling activity via cheminformatic analysis. RSC advances, 2018.
[49] M. Rojas-Carulla, B. Schölkopf, R. Turner, and J. Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):139-1132, 2018.
[50] S. Sagawa, W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: 191111991737 0,0331, 2019.
[51] B. Sanchez-Lengeling, C. Outeiral, G. L. Guimaraes, and A. Aspuru-Guzik. Optimizing distributions over molecular space. an objective-reinforced generative adversarial network for inverse-design chemistry (organic). 2017.
[52] J. Shen, F. Cheng, Y. Xu, W. Li, and Y. Tang. Estimation of adme properties with substructure pattern recognition. Journal of chemical information and modeling, 1034):1034-104 2010.
[53] M. Singh, R. Divakaran, I.. S. K. Konda, and R. Kristam. A classification model for blood brain barrier penetration. Journal of Molecular Graphics and Modelling, 96:1076,6, 2020.
[54] M. Stokes, K. Yang, K. Swanson, W. Jin, A. Cubillos-Ruiz, N. M. Donghia, C. R. MacNair, S. French, L. A. Carfrae, Z. Bloom-Ackernann, et al. A deep learning approach to antibiotic discovery. Cell. 180(4):688-702 2020.
[55] B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 443-450. Springer, 2016.
[56] V. Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
13

---

[57] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention all you need. In NeulPS, 2017.
[58] H. Wang, W. Li, X. Jin, K. Cho, H. Ji, Han, and M. Burke. Chemiicl--eactiion-aaaar molecule representation learning. In International Conference on Learning Repressetaations, 2022.
[59] S. Wang, Z. Li, S. Zhang, M. Jiang, X. Wang, and Z. Wei. Molecular property prediction based on a multichannel substructure graph. IEEE Access, 8: 111861--8614, 2020.
[60] Y. Wang, R. Magar, C. Liang, and A. Barati Farimani. Improving molecular contrastive learning via faulty negative mitigation and decomposed fragment contrast. Journal of Chemical Information and Modeling, 2022.
[61] Q. Wu, H. Zhang, J. Yan, and D. Wipf. Handling distribution shifts on graphs: An invariance perspective. In International Conference on Learning Representations, 2022.
[62] Q Wu, W. Zhao, Z. Li, D. Wipf, and J. Yan. Nodeformer: scalable graph structure learning transformer for node classification. In Advances in Neural Information Processing Systems, 2022.
[63] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations, ICLR 2019, 2019.
[64] M. Xu, H. Wang, B. Ni, H. Guo, and Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning, pages 11548 1558. PMLR, 2021.
[65] C. Yan, Q. Ding, P. Zhao, S. Zheng, J. Yang, Y. Yu, and Huang. Retroxpert: Decompose retrosynthesis prediction like chemist. Advances in Neural Information Processing Systems, 33:11248-11258 2020.
[66] C. Yang, Q. Wu, Q Wen, Z. Zhou, L. Sun, and J. Yan. Towards out-of--istribuue sequential event prediction: A causal treatment. In Advances in neural information processing systems, 2022.
[67] A. B. Yongye, J. Waddell, and J. L. Medina-Franco. Molecular scaffold analysis of natural products databases in the public domain. Chemical biology & drug design, 80(5)7117-724, 2012.
[68] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen. Graph contrastive learning with augmentations. Advances Neural Information Processing Systems, 53:5812-5822 2020.
[69] X. Zeng, S. Zhu, W. Lu, Z. Liu, J. Huang, Y. Zhou, J. Fang, Y. Huang, H. Guo, L. Li, et al. Target identification among known drugs by deep learning from heterogeneous networks. Chemical Science 11(7::1775- 1797, 2020.
[70] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. arXiv preprint arXiv: 1710:17100004122 2017.
[71] X. Zhao, Zong, Z. Guan, K. Zhang, and W. Zhao. Substructure assembling network for graph classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[72] Zhu, Y. Liu, C. Wen, and Wu. Dgdfs: Dependence guided discriminative feature selection for predicting adverse drug-drug interaction. IEFE Transactions on Knowledge and Data Engineering. 2020.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix !
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix[]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A B
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See Appendix C
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Appendix
(c) Did you report error bars (e-g., with respect the random seed after running experi-
ments multiple times)? [Yes] See Section 4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section F
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? Appendix [
[Yes] See Section 4 and
(b) Did you mention the license of the assets? [Yes] See Section 4 and Appendix D
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you's 're
using/curating? [Yes] See Appendix D
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] See Appendix D
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "_4BFN_Q9d_v", "reviewer": "Reviewer_MgvC", "summary": "This work aims to address the problem of the distributional shift between training data and test data when solving the property prediction task.  The idea is to learn a representation that is invariant with respect to the environment as the environment change results in the distributional shift. The main assumption is that, regarding the property that we want to predict, there are specific structures that determine the property and other substructures that are not relevant (so no matter how that structure changes the property does not change ) and to avoid the distributional shift one needs to learn the representation that is invariant with respect to the change of the environment.", "questions": "1. The definition of distribution shift here is defined as having a different scaffold of the molecules from training to test set. In this case, is each molecule regarded as coming from a different environment as they all have different structures, it is not clear how we measure  there is a distribution shift (how different the structure of the molecule should be to be considered as there is a distributional shift? is the size of the molecules? the number of the atom, the type of the atom, to which degree) \n\n2. The main idea is that there is spurious structure and causal structure wrt the property.  First of all, this assumption holds for most of the properties (at least the ones that they use here). Can we really identify substructures that determine the property and are totally disentangled from the rest of the structure (spurious structure)? What if the property is related to the global structure of the molecule? \n\nIf it is the case that the assumption holds,  if we have enough data and enough representation power, isn't the predictive model should learn to predict the property from the casual structure and ignoring the rest?  which means the representation will be naturally pushed to learn such invariant representation.  Unless we are really in a data scarcity case,  and if we are in the data scarcity case then learning to infer the environmental factor (unsupervised) would be also hard. \n\n3. In equation 7, the first term and last term exactly the same except one is the sum over G^e and the other one is over G , G^e represents the graph G and y sampled from environment e, so if you take the expectation at the last term with respect to e (E_e) as you have there isn't this two-term exact the same?\n\n4. Intuitively, it seems like what the model does is given the graph and property, learn to infer the environment, and a predictive model that learns to predict the property y for the graph given environment e (objective 6). Then use this inference and predictive model to learn another predictive model that is not restricted to the environment. But my question is if the model learned from objective 6 can infer the environment and predict the property given that environment is the problem solved? so even if we have a distributional shift we can use it to predict the property for the graph from the new environment?", "limitations": "The assumptions and proposed model are not really convincing. \nThe experiment section is a bit confusing and does not provide very structured information. For example, in table 2, what are the properties that are being predicted, and what elements are considered as the environment (still scaffold? ). If scaffold, size, and assay are considered as the environment then what is the property where it is being predicted? Also if the environment label is accessible, does the model still do inference or use it directly? ", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strength: the problem they try to address is interesting and valuable to the community\nweakness: The assumptions and proposed model are not really convincing. The experiments section is not clearly described. Hard to follow for readers who are not familiar with the dataset, so at least in appendix explains for each data set what is the property they are predicting, and what is the environment they considering here, and how the dataset is split to train test sets to model the distributional shift. \n                  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4VPjzFH-936", "reviewer": "Reviewer_WE4q", "summary": "This paper is mainly designing a learning framework to learn causal-invariance molecular representations, improving the generalization of MRL in OOD settings. This paper observes that the molecule usually consists of two parts: one part is causal substructure that determines the behaviors of molecular properties, another part is spurious substructure that constitute the molecular structures but has no impact on molecular behaviors. Based on this observation, this paper builds a structural casual model to learn more robust molecular representation learning. Specifically, new model includes substructure-aware attention-based molecular encoders and VAE-based environment inference model. To enable two-stage optimization, this paper proposes a new learning objective function consisting of the ELBO objective and the ERM objective. ", "questions": "(1) Does different decomposing strategy affect the model performance?\n(2) What if the model is trained in a simple multi-task learning setting? In other words, setting environment prediction as an auxiliary task? Would this have equivalent performance to the proposed method? Why it is necessary to design environment prediction in a VAE learning way?", "limitations": "This reviewer thinks the proposed method in this paper has the major limitations that the scaffold and decomposing strategy may not be robust to OOD setting although adding this additional knowledge could improve robustness of MRL. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) This reviewer thinks that this paper has good novelty. It proposes an important but overlooked observation in molecular representation learning. In short, the problem addressed in this paper is significant, the proposed solution is sufficiently novelty and the motivation is reasonable; \n(2) The presentation of each section is quite clear and well-organized. It is not hard for this reviewer to understand the novel idea and elegant design of new MRL paradigm;\n(3) Sufficient experiments over different public benchmarks under OOD settings and ablation studies demonstrate the effectiveness of each component;\nWeaknesses:\n(1) Decomposing strategies (BRICS) and scaffold introduction as environment actually implicitly introduce additional knowledge to MRL. The decomposing step and scaffold introduction both follow hand-engineered rules by domain experts and may have non-trivial impacts on model performance. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1zPkExBJjSl", "reviewer": "Reviewer_pgdS", "summary": "## Summary\n\nThis paper introduces techniques to enhance the robustness of molecule representation learning against distribution shifts. The authors made the observation that bio-chemical properties of molecules are usually invariantly associated with certain substructures across different environments such as scaffolds, sizes, etc. There are three important pieces in their modeling process. One is architecture-wise they use graph-level embeddings to attend a list of substructures. Second they device a new learning objective from mutual information and invariant learning to help select causal substructures. Third they use a latent representation for environment to mitigate the current issue of dealing molecular environments (e.g., often man-made, not always available, too many if using scaffolds). Overall, it's a solid paper.", "questions": "## Questions\n\n- what's the exact definition of $\\mathcal{G}^e$, can you add text and math formula to explain?\n- it seems that the second term of eqn 7 could be simplified to $\\beta \\frac{1}{|\\mathcal{G}|} \\sum_{(G, y) \\in \\mathcal{G}} -log q_{\\theta}(y | G)$, can you confirm?\n- BBBP benchmark ROC seems much lower than typical methods would produce, is it some mistake? reference: https://arxiv.org/abs/2111.12951 appendix G table s1.\n- Often times bio-chemical properties are affected by interactions between substructures, would be interesting to see if adding self-attention in the molecule encoder would help", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- the paper addresses an important topic in molecule representation learning (distribution shift).\n- it introduces interesting innovative techniques to help mitigate issues of current methods\n\n## Weaknesses\n- some key notations are not clearly defined - care is needed for the writing", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept