Learning Invariant Molecular Representation in Latent Discrete Space
Authors: Xiang Zhuang
Qiang Zhang
Keyan Ding
Yatao Bian
Xiao Wang
Jingsong Lv
Hongyang Chen
Huajun Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: molecular representation learning
out-of-distribution

Abstract: Molecular representation learning lays the foundation for drug discovery. However, existing methods suffer from poor out-of-distribution (OOD) generalization, particularly when data for training and testing originate from different environments. To address this issue, we propose a new framework for learning molecular representations that exhibit invariance and robustness against distribution shifts. Specifically, we propose a strategy called  ``first-encoding-then-separation'' to identify invariant molecule features in the latent space, which deviates from conventional practices. Prior to the separation step, we introduce a residual vector quantization module that mitigates the over-fitting to training data distributions while preserving the expressivity of encoders. Furthermore, we design a task-agnostic self-supervised learning objective to encourage precise invariance identification, which enables our method widely applicable to a variety of tasks, such as regression and multi-label classification. Extensive experiments on 18 real-world molecular datasets demonstrate that our model achieves stronger generalization against state-of-the-art baselines in the presence of various distribution shifts.  Our code is available at https://github.com/HICAI-ZJU/iMoLD.

Learning Invariant Molecular Representation in Latent Discrete Space
Xiang Zhuang 12.3+ Qiang Zhangl
1.2.+ Keyan Ding", Yatao Bian", Xiao Wang", Jingsong Lv Hongyang Chen Huajun Chen 1,2.3f
'Colleee of Computer Science and Technology. Zhejiang University Haunhoog Global Scientific and Technological Innovation Center *Zhejiang University Ant Group Joint Laboratory of Knowledge Graph *Tencent AI Lab, School of Software, Beihang University, "Zeejiang Lab (zhuangxi ang, qiang. zhang cs dingkeyan, huajunsir]02ji edu. en yatao. bian@gmail. com, xiao wang@buaa, edu. cn {jingsongl, hongyang) janglab.
Abstract
Molecular representation learning lays the foundation for drug discovery. However, existing methods suffer from poor out-of-distribution (OOD) generalization, partic- ularly when data for training and testing originate from different environments. To address this issue, propose new framework for learning molecular representa- tions that exhibit invariance and robustness against distribution shifts. Specifically, to identify invariant
we propose strategy called "first-encodngg wwwncoddinnthee--sppaatttii molecule features in the latent space, which deviates from conventional practices. Prior to the separation step, introduce residual vector quantization module that mitigates the over-fitting to training data distributions while preserving the expressivity of encoders. Furthermore, we design lask-agnostic self-supervised learning objective to encourage precise invariance identification, which enables our method widely applicable a variety of tasks, such regression and multi- label classification. Extensive experiments on 18 real-world molecular datasets demonstrate that our model achieves stronger generalization against state-of of-t--oftthe-rr art baselines in the presence of various distribution shifts. Our code is available at /ttpts///itthu.... com/HICAI Z.JU/iMol..
Introduction
Computer-aided drug discovery has played important role in facilitating molecular design, aiming to reduce costs and alleviate the high risk experimental failure [ 2] In recent years, the
emergence of deep learning has led growing interest in molecular representation learning, which aims to encode molecules as low-dimensional and dense vectors B
6 g These
learned representations have demonstrated their availability in various tasks, including target structure prediction (9), binding affinity analysis [10], drug re-purposing IT and retrosynthesis 12.
Despite the significant progress in molecular representation methods, prevailing assumption in the traditional approaches is that data sources are independent and sampled from the same distribution. However, in practical drug development, molecules exhibit diverse characteristics and may originate from different distributions 13 14] For example, in virtual screening scenarios [15), the distribution shift occurs not only in the molecule itself, e.g., size [13 scaffold [16] changes, but also in the
Equual contribution. "ooresppondigg author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

target [14] e.g., the emergent COVID-19 19 leads to new target from unknown distributions. This out-o--disrriuution (OOD) problem poses a challenge to the generalization capability of molecular representation methods and results in the degradation of performance in downstream tasks [7 18]
Current studies mainly foeus on regular Euclidean data for OOD generalization. Most studies 18 19
22 21
[22] adopt the invariance principle [23] [24], which highlights the importance of focusing on the critical causal factors that remain invariant distribution shifts while overlooking the spurious
parts 23 19 20] Although the invariance principle has shown effectiveness on Euclidean data, its
application to -on-Euclid data necessitates further investigation and exploration. Molecules are often represented as graphs, typical non-Euclidean data, where atoms as nodes and bonds as edges, thereby preserving rich structural information [3. The complicated molecular graph structure makes
challenging to accurately distinguish the invariant causal parts from diverse spurious correlations [25]
Preliminary studies have made some attempts on molecular graphs [26] 25 27 (8, 29). They
Sper Sabgraph
Spurlens
explicitly divide graphs extract invariant sub- structures, at the granularity of nodes (28).
Sahyraph
edges 26 28 29], or motif's 255 These attempts can be summarized as the "first- separation ceeem-tteeeeeeee ahee-ennodingg paradigm (Figure (a)). which first divides the graph into invariant and spurious parts and then encodes each part separately. We argue that this practice is sub- optimal for extremely complex and entangled graphs, such as real-world molecules [30 311, since some intricate properties cannot be readily determined by analyzing a subset of the molec- ular structure 30 3. Besides, some meth- ods 25 27 require assumptions and inferences about the environmental distribution, which are often untenable for molecules due to the intri- cate environment. Additionally, the downstream tasks related to molecules are diverse, including regression and classification. However, some methods such CIGA (26) and DisC (32) can only be applied to single-label classification tasks, due to the constraints imposed by the invariant learning objective function.
Ceneriier
Invaniant Sularsph
Teveniat Repecsanation
Secaration
Bsooding
Spericen
GIX
Invariex
Regmorrzttoon
Encding
Segaatiion
(b)
Figure 1: (a) First-Separation-TTTe Tieiiion--hh--nneee Enco--nnn the input is separated using a subgraph generator, then each subgraph is encoded respectively. (b): First- Encoding- Then- the input is encoded, then the representation separated by a scorer.
To fill these gaps, we present a novel molecule invariant learning framework to effectively cap- ture the invariance of molecular graphs and achieve generalized representation against distribution shifts. In contrast to the conventional approaches, we propose a "first-nncoding-thee wwwroodinngtee--sepeeeee seenssepatttion strategy" (Figure (b)). Specifically, we first employ a Graph Neural Network (GNN) B3 34 35
to encode the molecule (i.e., encoding GNN), followed by a residual vector quantization module to alleviate the over-fitting to training data distributions while preserving the expressivity of the encoder. We then utilize another GNN to score the molecule representation (i.e., scoring GNN), which measures the contributions of each dimension to the target in latent space, resulting in a clear separation between invariant and spurious representations. Finally, we design a self-supervised learning objective [36] 37 38] that aims to encourage the identified invariant features to effectively preserve label-related information while discarding anvironment-relatee information. It is deserving of note that the objective is task-agnostic, which means that our method can be applied to various tasks, including regression and single- or multi-label classification.
Our main contributions be summarized as follows:
We propose a paradigm of "first-encoding- wwwpe-enooiin--te--eeeeeeeee sereanttttttion" using an encoding GNN and scoring GNN, which enables to effectivvly identify invariant features from highly complex graphs.
We introduce a residual vector quantization module that strikes a balance between model expressiv- ity and generalization. 'he quantization acts the bottleneck to enhance generalization, while the residual connection complements the model's 's expressivity.
We design a self-supervised invariant learning objective that facilitates the precise capture of invariant features. This objective is versatile, task-agnostic, and applicable to variety of tasks.

---

Scaffold
Size
Assay
aviironment
anviromment
caviroument
Train
Nates
Mssayd88811
environment
onvironment
cavironment
Test
Naves
PAssap 190374
Figure 2: An overview of distribution shifts in molecules. Distribution shifts occur when molecules originate from different scaffold, size or assay environments.
We conduct comprehensive experiments on a diverse set of real-world datasets with various distribution shifts. The experimental results demonstrate the superiority of our method compared to state-of-the-ar approaches.
2 Related Work
OOD Generalization and Invariance Principle. The susceptibility of deep neural networks to substantial performance degradation under distribution shifts has led to a proliferation of research focused on out-of-distribution (OOD) generalization (39]. Three lines of methods have been studied for OOD generalization on Euclidean data, including group distributionally robust optimization [40]
41 42
domain adaptation 43 44 45 and invariant learning 19 20 21). Group distributionally
robust optimization considers groups of distributions and optimize across all groups simultaneously.
Domain adaptation aims to align the data distributions but may fail to find an optimal predictor
without additional assumptions [19] 20, (66.. Invariant learning aims to learn invariant representation satisfying the invariant principle 220 24), which includes two assumptions: (1) sufficiency, meaning the representation has sufficient predictive abilities, (2) invariance, meaning representation is invariant to environmental changes. However, most methods require environmental labels, which are expensive to obtain for molecules (26], and direct application of these methods to complicated non-Euclidean molecular structure does not yield promising results B 22 26].
OOD Generalization on Graphs. Recently, there has been growing attention on the graph-level representations under distribution shifts from the perspective of invariant learning. Some methods 226
32
paradigm, and they make attempt to capture the invariant substructures by dividing nodes and edges in the explicit structural space. However, these methods suffer from the difficulty in dividing molecule graphs in raw space due to
27
28
(55
follows the sriss-separatio--the wwe-ttteeeeennnngge
the complex and entangled molecular structure 47 Moreover, MoleOOD (55 and GIL. (2 require inference of unavailable environmental labels. which entails prior assumptions on the environmental distribution, And the objective of invariant learning may hinder the application, e.g., CIGA (26] and DisC [32] can only apply to single-label classification rather than regression and multi-label tasks. Additionally, OOD-GNN [48) does not use the invariance principle. proposes to learn disentangled graph representations, but requires computing global weights for all data, leading to a high computational cost. OOD generalization on graphs can also be improved by another line of relevant works 129 49 47
GNN explainability [50] 51). which aims to provide rationale for prediction. However, they may fail in some distribution shift cases (66 And DIR (29) and GSAT [49 also divide graphs in the raw structural space. Although GREA 47 learns in the latent feature space, it only conducts separation on each node while neglecting the different significance of each representation dimensionality. In this work, we focus on the OOD generalization of molecular graphs, against multi-type of distribution shifts, e.g., scaffold, size, and assay, as shown in Figure 2
Vector Quantization. Vector Quantization (VQ) [52 53 acts as a bottleneck of representation learning. It discretizes continuous input data in the hidden space by assigning them to the nearest 56 have demonstrated its effectiveness to
vectors in a predefined codebook. Some studies 53 54 55 enhance model robustness against data corruptions. Other studies [57] 588 [59 find that taking VQ as an coter-component communication within neural networks can contribute to the model generalization ability. However, we posit that while VQ can improve generalization against distribution shifts, may also limit the model's expressivity and potentially lead to under-fitting. To address this concern, we propose to equip the conventional VQ with a residual connection to strike a balance between model generalization and expressivity.
3

---

Preliminaries
3.1 Problem Definition
We focus the OOD generalization of molecules. Let G be the molecule graph space and Y be the label space, the goal is to find a predictor f : G - y to map the input G G into the given set of datasets collected from multiple environments Eall =D= = (D"YeEE.. Each D contains pairs of an input molecule and its label: D = {(G, Yi) that are drawn from the joint distribution P(G, Y| = of environment However, the information environment e is always not available for molecules, thus we redefine the training joint distribution P Ptrain (G, Y) = P(G, E e), Ve Ptrain, and the testing joint distribution as Prest(C, G, Y) = P(G, Y|E = e), Ve E Fau Ferain. where Ptrain ( [, Y) # Prest G, Y). We denote joint distribution across all environment as Pat(Ge G, Y) P(G, Y e), Ve Eaul Formally, the goal is to learn an optimal predictor f based on training data and can generalize well across all distributions:
label Y E y. Generally,
S" arg min EiG, YY (G) Y)),
where -): is the empirical risk function. Moreover, since the joint distribution P(G, Y) can be written as P(Y P(Y G P G), the OOD problem can be refined into two cases, namely covariate and concept shift 60 61 62 63). In covariate shift, the distribution of input differs. Formally, P Ptrain G) * Prest G) and Prram GY = Pteat G)G). While concept shift occurs when the
conditional distribution changes as Ptrain (Y|))= # Ptest (Y))) and Ptrain (G) Ptest(()) We will consider and distinguish between both cases in our experiments.
3.2 Molecular Representation Learning
We denote a molecule graph by G = {V,E), where is the set nodes (c.g., atoms) and E V the set of edges (c.g.. chemical bonds). Generally, the predictor f can be denoted as pog, containing an encoder g G - R that extracts representation for each molecule and downstream classifier p: IRd }' that predicts the label with the molecular representation. In particular, the encoder operates two stages: firstly, by employing a graph neural network representations H according to the following equation:
3 34
35
to generate node
H [, b h hyl GNN(G) R
(2)
where h, E RRd is the representation of node v. Secondly, the encoder utilizes a readout operator to obtain the overall graph representation z:
z = (EADOUT E Rdd.
The readout operator can be implemented using simple, permutation invariant function such as average pooling.
Method
This section presents the details of our proposed method that learns invariant Molecular representation in Latent Discrete space, called iMoLD. Figure 3 shows the overview of iMoLD, which mainly consists of three steps: Using a GNN encoder and a residual vector quantization module obtain the molecule representation (Section 4.1 2) Separating the representation into invariant and spurious parts through GNN scorer (Section 4.2) 3) Optimizing the above process with task-agnostic
self-supervised learning objective (Section 4.3
4.1 Encoding with Residual Vector Quantization
The current mainstream methods 26
adopt the paradigm, which explicitly divides graphs into invariant and spurious substructures on the granularity of edge, node, or motif, and then encodes each substructure individually. In contrast, we use the opposite paradigm that first encodes the whole molecule followed by separation.
28
27 32

---

Batch of Inputs
First-Fan First-Eacoding- Then-Siparation
Task-agnostic Self-upprvised
Invariant Learning
Scoring GNN
8 E Rliis
GNN
Undebook
Spusious Representzzion
levariant Repeosntaton
CINN
RVQ
Siniarry
Enooding
H ER Mid
H R
XB
Figure An overview iMoLD. Firstly, given batch of inputs, we learn invariant and spurious rep- resentations (z and z 2P for each input in latent discrete space by first-encoding sercodinggthen---e neensseparation paradigm. An encoding GNN and an RVQ module are involved to obtain molecule representation, then the representation is separated through a scoring GNN. The invariant 2!"! is used to predict the label y. Then a task-agnostic self-supervised learning objective across the batch designed to facilitate the acquisition of reliable invariant !n
Specifically, given an input molecule G, we first use GNN to encode it, resulting in node representa- tions H:
H GNN E RIV xd
where GNN E represents the encoding GNN, |V] is the number of nodes in G, and d is the dimen- sionality of features. Inspired by the studies (57 59 that Vector Quantization (VQ) [2 53 is
helpful to improve the model generalization on computer vision tasks, we propose Residual Vector Quantization (RVQ) module to refine the obtained representations.
In the RVQ module, VQ is used to discretize continuous representations into discrete ones. Formally, it introduces shared learnable codebook as a discrete latent space: C {(e, e2,... e|c }. where each e E Rdd, For each node representation h, in H, VQ looks up and fetches the nearest neighbor in the codebook and outputs it the result. Mathematically,
Q(h,) ok.
where h = argmin ||, 0x (I....../)))
(5)
and Q(-) denotes the discretization operation which quantizes h, to e in the codebook.
The VQ operation acts as a bottleneck to enhance generalization and alleviate the -ver--iiigi issue caused by distribution shifts. However, also impairs the expressivity of the model by using limited discrete codebook to replace the original continuous input, suffering from a potential under- under-fitting issue. Accordingly, we propose to equip the conventional VQ with a residual connection to strike balance between model generalization and expressivity. In specific, incorporate both the continuous and discrete representations to update node representations H to H':
H' = [Q(hi) + h1, Q(h2) - he...., he......((hp))) + h
(6)
Similar to VQ-VAE
52 53 we employ the exponential moving average updates for the codebook:
(1)
(1 NO () + n (1
7)
m
(1)
m
(t-1)
h (1 - m)
N m
where ni is the number of node representations in the t-th mini-batch that are quantized to e₁, and n is a decay parameter between and 1.
(1)
4.2 Separation at Nodes and Features
After encoding, we separate the representation into invariant parts and spurious parts. It is worth noting that our separation not only performed at the node dimension but also takes into account the feature dimension in the latent space. The reasons are two-fold: 1) Distribution shifts on molecules
5

---

can occur at both the structure level and the attribute level [26], corresponding to the node dimension [V] and the feature dimension d in H', respectively. 2) The resulting representation may be highly entangled, thus it is advisable to perform separation on each dimension in the latent space.
Specifically, we another GNN
obtain the separating score S:
S (GNNs (G)) R
(8)
where GNN ş represents the scoring GNN, |V] is the number of nodes in G, and is the dimensionality of features. a(+) denotes the Sigmoid function to constrain each entry in S falls into the range of (0, 1). Then we can capture the invariant and complementary spurious features at both structure and attribute granularity in the latent representation space by applying the separating scores to node representations:
InV H'' C S, H
(1 S) S)
where Hn and Sp denote the invariant and spurious node representations respectively, and @ is the wiss product. Finally, the invariant and spurious representation (denoted as z ! and z Spu respectively) of can be generated by a readout operator:
z Inv (EADOUT(H Rdd Z (EADOUT(HH*)) E RR.
(10)
4.3 Learning Objective
Our OOD optimization objectives are composed of an invariant learning loss, task prediction loss, and two additional regularization losses.
Task-agnostic Self-supervised Invariant Learning. Invariant learning aims to optimize the en- coding GNN and the scoring GNNs to produce precise invariant and spurious representations. In particular, we need lnv to be invariant under environmental changes. Additionally, we expect the objective be independent of the downstream task, which allows the method to be not restricted to specific type of task. To achieve these, we design task-agnostic and self-supervised invariant learning objective. Specifically, we disturb z!!" via concatenating corresponding z, batch, resulting in augmented representation Z!!!:
Spu in shuffled
Spu
(11)
BJ
where @ denotes concatenation operator and B is batch size.
Inspired by simple self-supervised learning framework that takes different augmentation views as similar positive pairs and does not require negative samples (37), we treat !!n and Z!Te as positive pairs and push them to be similar, using an ML.P-basdd predictor (denoted as w) that transforms the output of one view and aligns to the other view. We minimize their negative cosine similarity:
Linv Σ sim(sg|z,"" [h) (z!""))
(12)
where sim(:,-) represents the formula of cosine similarity and sgl-] denotes stop- hoo-ggradient operation to prevent collapsing [37]. We employ Lim as our objective for invariant learning to ensure the invariance of z !nv against distribution shifts.
Task Prediction. The objective of task prediction is to provide an invariant presentation !! with sufficient predictive abilities. During training. the choice of prediction loss function depends on the type of task. For classification tasks, we employ the cross-entropy loss, while for regression tasks, we use the mean squared error loss. Take the binary classification task as example, the cross-entropy loss is computed between the predicted Di = p(z!"r) ) and the ground-truth label y::
B
Cpred Σ
(y log D1 + (1 yi) vi) log (1 11)
(13)

---

Table 1: Evaluation performance on GOOD benchmark. denotes abnormal results caused by under-fitting declared in the leaderboard, and / denotes that the method cannot be applied to this dataset. The best is marked with boldface and the second best is with underline.
GO0D-4IV 1
GOOD-ZINC
GOOD-ICBA
Metoo
scatrold
$30
seaffed
size
scattold
conaziate
2000eg1
conoziate
Conuriate
concege
covariate
coscegt
conazlate
concegt
covoriate
coocepe
FRM
59 001522.000 19)
72.48(1 (30)
59. 1902.39
61.9122.99)
(199)000134)
0.23 (.201900007))
17 11 54)
21 9960744
17-18/346)
(5,800.55)
BRM
70 1202 78)
71. 1.37)
59.06:1.59)
-30
022164300166
0.13990.0641)
(19540022099
0.53560.0055
16 168900.33 20)
22 (2.500435)
(7.880336)
(5.820.52)
VREX
06 (8.3413.54) 54)
72.3 12.21(1.42)
55.4% 2.222.26
61.21 61.31(2.00
0 0118153001544
011257000055)
1.131 1000007)
17 1200000
21 (1.050022)
17.ME0.35)
(5855847)
GnoppRO
58 1512 641
71 1.247 1.27)
57.05/2.86 561
59.771 59.771955
0 0.1878300125)
0.1323/0.0441
0.2577000477
0133300006
16 16.550..5 50)
21 (19100933
18 1874//550
15.210..6)
Cond
70.6912255 251
1.00
59.390
09.39.2-00 2:500
60, 60.50
0.17890000632
0.1308/0.0057)
(,299/000000
H0.00) 0122
17.0000.5 38)
22 22:000044)
17.6303311
16.8900.50)
DANN
50.4312-00 42)
71.70005501
62.39/2.60)
65.15(2.15)
0 0.1746000054)
0.1259/0.0042)
17.2000440 111340000000)
22.0010.22)
1777000200
13:7500400
Misup
106811.000 96)
71 11.8911177
59.
19
62.802255)
0 (20060000133)
0.119:000077)
D 1020110001900
1n.5700 53)
20 50000000
17:40/0.00
(37700.17)
DIR
58 68.4412551
71 11.40)1.48)
57.57)3775)
14,3N1455
0.36220000399
0.245:00445)
0 01578000412
225)
16.3300.99)
238210.89)
16.041114
16.89
GSAT
70.0711 (6)
72. (1.09))))
60.73:2.900 199
56,
1.77
1418:000777
0 0:21000005)
In 4500 17)
20 (4)
17.5%044
13.528.00
GREA
71 119812677
70
(m6)
60. (0.11(1.77
6.59
16 1622800334
2.223(1.3)
17.12/1011
13.62:1.6)
CAL
5M T21 10
19.3622:13 14)
56.164.75
15
18.0011300
1699/07711
13000065)
DuC
58.857.220 20)
64. (2.822677
48.35/3.54)
74.1166.99)
MevoXDD
5M (1.-69)
69 [8C 15)
18.521-30 7K1
15
0 (7550000000) II 199880016) n (1N00006))
12 12000000
12 12.001129)
12.600555)
10. 10.100.10)
CIGA
59 0011-977
71 (11661133)
61 618111166
71.6221-55)
IMoLD
22.9312229)
74.32/163)
62.8622.56)
77.451133)
0.141010.0044 (LI0IAW 1L186300139 (.009008007)
17.5200.11)
22.580966)
18.020073) 18,2011000
Scoring GNN Regularization. For the scoring GNNs, we apply a penalty to the separation of invariant features, preventing abundance or scarcity and ensuring reasonable selection. To achieve this, we introduce a regularization term Leg to constrain the size of the selected invariant features:
(J.S) [ x d
Geg
(14)
where J E Vixd denotes a matrix with all entries 1, (;; is the Frobenius dot product and 7 is a pre-defined threshold between O and 1.
encourage h, to remain close to the selected codebook embedding eμ in Equation (5) and prevent it from frequently fluctuating between different embeddings, we add commitment loss fem to foster that h, commits to eş and does not grow uncontrollably:
Codebook Regularization. In the RVQ module, following VQ- 52 53
Lemt EvEv ||g[ex]
h
(15)
Finally, the learning objective can be defined as the weighted sum of the above losses:
== Pred + λ Linv + A2hreg + Aafemt;
(16)
where Å₁, Ag and Ag are hyper-parameters to control the weights of Cinv (in Equation 12) ) Creg (in
Equation (14) and Lemi (in Equation (55), respectively.
5 Experiments
In this section, we conduct extensive experiments to answer the research questions. (RQ1) Can our method iMoLD achieve better OOD generalization performance against SOTA baselines? (RQ2) How does each of the component we propose contribute to the final performance? (RQ3) How can we understand the latent discrete space induced by the proposed RVQ?
5.1 Experimental Setup
Datasets. We employ two real-world benchmarks for OOD molecular representation learning. Details of datasets are in Appendix A
GOOD
which is a systematic benchmark tailored specifically for graph OOD problems. 63).
We utilize three molecular datasets for the graph prediction task: (1) GOOD-HIV [16), where the objective is binary classification to predict whether molecule can inhibit HIV; (2) GOOD-
ZINC 64 which a regression dataset aimed at predicting molecular solubility; and (3) GOOD- PCBA 16]. which includes 128 bioassays and forms 128 binary classification tasks. Each dataset comprises two environment-splittnn strategies (scaffold and size), and two shift types (covariate and concept) applied per splitting outcome, resulting in total of 12 distinct datasets.
7

---

Table 2: Evaluation performance DrugOOD benchmark. The best is marked with boldface and the second best is with underline.
Method
IC50
EC50
Assay
Scuffold
Size
Assay
Scaffold
Size
ERM
71.630.76)
68.79(0.47)
67.50(0.38)
67.39(2.90)
64.98(1.29
65.10(0.38)
IRM
71.15(0.57)
67.22(0.22)
67.77(2.71) 61.58(0.58)
63.86(1.36
59. 19.19().83)
Coral
71.28(0.91)
68.36(0.61)
64.53(0.32)
72.08(2.80)
64.83(1.64)
58.47(0.43)
Mp
71.49(1.08)
68.59(0.27)
67.79(0.39)
67.81(4.06)
65.77(1.83)
65.77(0.60)
DIR
69.84(1.41)
66.33/0.65)
62.92(1 1.89)
65.81(2.93)
63.76(3.22
61.56(4.23)
GSAT
70.59(0.43)
66.45(0.50)
66.70(0.37)
73.82(2.62)
64.25(0.63)
62.65(1.79)
GREA
70.23(1.17)
67.02(0.28)
66.59(0.56)
74.17(1.47)
64.50(0.78)
62.81(1.54)
CAL
(0)))
65.90(1.04)
66.42(0..0)
74.54(4.18)
65.19(0.87)
61.21((.76)
DisC
61.40(2.56)
62.70(2.11)
61.43(1.66
63.71(5.56)
60,57(2.27)
57.38(2.48)
MoleOOD
71.62(0.52)
68.58(1.14)
65.62(0.77)
72.69(1.46)
65.74(147)
65.51(1.24)
CIGA
71.86(1.37)
69.14(0.70)
66.92(0.54)
69. 69.15(5.79)
67.32(1.35)
65.65(0.82)
iMoLD
72.11(0.51)
68.84(0.58)
67.92(0.43)
77.48(1.70)
67.79(0.88)
67.09(0.91)
which is a OOD benchmark for Al-aided drug discovery. This benchmark provides strategies, including assay, scaffold and size, and applies these three splitting to two measurements (IC50 and EC50). As a result, we obtain 6 datasets, and each dataset contains binary classification task for drug target binding affinity prediction. A detailed description of each environment-splitiing strategy is also in Appendix
DrugOOD 13
three environment--plittenn
Baselines. We thoroughly compare our method against ERM 665 and two groups of OOD baselines: (1) general OOD algorithms used for Euclidean data, which include domain adaptation methods such as Coral 44 and DANN [43], group distributionally robust optimization method (Group- DRO [41), invariant learning methods such as IRM [19] and VREx 40 and data augmentation method (Mixup (66)). And (2) graph-specific algorithms, including Graph OOD algorithms such as CAL (28) DisC [32]. MoleOOD (25] and CIGA (26), as well as interpretable graph learning
methods such as DIR [29], GSAT [49] and GREA 47 Details of baselines and implementation are
in Appendix
Evaluation. We report the ROC-AUC score for GOOD-HIV and DrugOOD datasets as the task is binary classification. For GOOD-ZINC, we use the Mean Average Error (MAE) since the task is regression. While for GOOD-PCBA- we use Average Precision (AP) averaged over all tasks as the evaluation metric due extremely imbalanced classes. We experiments 10 times with different random seeds, select models based on the validation performance and report the mean and standard deviations on the test set.
5.2 Main Results (RQ1)
Table and Table present the empirical results GOOD and DrugOOD benchmarks, respectively. Our method iMoLD achieves the best performance on 16 of the 18 datasets and ranks second on the other two datasets. Among the compared baselines, the graph-specific OOD methods perform best on only datasets, and some general OOD methods outperform them on another 7 datasets. This suggests that although some advanced graph-specific OOD methods can achieve superior performance on some synthetic datasets (e.g., to predict whether a specific motif is present in a synthetic graph (29), they may not perform well on molecules due to the realistic and complex data structures and distribution shifts. In contrast, our method is able to achieve the best performance on most of the datasets, which indicates that the proposed identification of invariant features in the latent space is effective for applying the invariance principle to the molecular structure. We also observe that MoleOOD, a method designed specifically for molecules, does not perform well on GOOD-ZING and GOOD-PCBA, possibly due to its dependence on inferred environments. Inferring environments may become more challenging for larger-scale datasets, such as GOOD-ZINC and GOOD-PCBA which contain hundreds of thousands of data and more complex tasks (e.g.. PCBA has a total of 128 classification tasks). Our method does not require the inference of environment, and is shown to be effective datasets of diverse scales and tasks.
8

---

(a) FRM
(b) ERM (+VQ)
(c) FRM (+RVQ)
(d) iMoLD
Scorefraaaj-00000 Seoer/ul--78.5
Scowetnang---t Sore
Sconetmaagi- -22 AL Scorefaatt- D12-00/00,00, D01-1)m166
Scovriinatg-677.6 60.00 Searetial)-79 D(T-1)-0.19 DV7-1)=0.64
D12-00-8.45 2/T-1)-2.41
0/2-00 D12-1)ma.51
Figure 4: Visualization of the extracted features on training and validation set when the model achieves the highest score on the validation set. Score(train) and Score(val) vval) are ROC-AUC scores on the training and validation set, respectively. D(Y=0) and D(Y=/) are distances between the features on the training and validation sets of cach class.
5.3 Ablation Studies of Components (RQ2)
We conduct ablation studies to analyze how each proposed component contributes to the final performance. Three groups of variants are included: (1) The first group to investigate the RVQ module. It consists of ERM and ERM (+RVQ), to explore the performance of the simplest ERM and the role of applying the RVQ to it. It also consists of variants of our method, including w/o VQ, which uses H instead of H' in Equation ( to obtain i and HSpp, w/o R, which obtains H and Spu without a residual connection by using the output discretization Equation (5). (2) The second group is to investigate the impact of each individual training objective, including w/o Cinv. w/o Lrgg and w/o Lems, which remove Linvs Lreg and Lemt from the final objective function in Equation (16), respectively. (3) The third group consists of variants that replace our proposed self-supervised invariant learning objective with other methods, including w/ LCIGA and w/ CGREA. which represent the objective functions using CIGA [26] and GREA (7). respectively.
Table 3 reports the results on the scaffold- split of GOOD-HIV and the size-split of GOOD-PCBA We observe that the addi- tion of the RVQ module can improve per- formance without using any invariant learn- ing algorithms. This indicates that the designed RVQ is effective. Notably, on the GOOD -IV dataset, removing the VQ module results in sigmificant performance degradation. While the GOOD-PCBA dataset, removing the residual connection leads to significant performance drop. This observation may be attributed to the larger scale and more complex task of the GOOD-PCBA dataset. Although VQ im- proves the generalization, it reduces the expressivity capacity of the model. Thus, achieving balance between the benefits and weaknesses brought by VQ is necessary with the proposed RVQ module. By observing the second group, we can conclude that each module is instrumental to the final performance. Removing the invariant learning objective Line and commitment loss Cemt results in more pronounced performance degradation, suggesting that the invariant learning objective is effective and indispensable, as well as the need for a constraint term to justify the output of the encoder and the codebook in VQ. By comparing the third group of results, we find that our proposed learning objective outperforms the existing methods, which shows that our approach can not only be applied to a wider range of tasks but also achieve better performance. Moreover, to investigate the robustness of our model, we perform
Table 3: Ablation studies. denotes that the variant cannot be applied to this dataset.
GOOD-HIV-Scafioll
FOOD-PCBA-SSE Size
covariato
concept
covariate
concept
iMoLD
72.93(2299)
74.33(1.63)
18.02/0.73)
18.21(1.10)
ERM
69. 55(5559))
72.48.1.20 26)
17.75(0.46)
15.5000.55 55)
ERM (+RVQ)
70.18(1.07)
73.14(1.99)
17.84(0.17)
16.59(0.40
wdo VQ
70.18(1.60)
72.58/0.66)
17,88(1.57)
17.37(0.82)
wio
72.40(2.11)
73.0340.1 15)
17.37(0.62)
12.99(2009
w/o Cine
7L.8% 1.00)
17.55(0.34)
17.05(0.87)
wlo Cus
71.47(0.72)
72 39(0997)
17.33(0.68)
17.91(0.34)
70.20((007)
71.9801.34
17.24(1.02)
17.68/2.21)
w/ COIGA CaREA
71,23(1.31)
73.31(1.73)
/
72.33(1.23)
72.94/2.30)
17.93(0.55)
16.86(0.50)
sensitivity analysis in Appendix C.1
5.4 Analysis of Latent Discrete Space (RQ3)
To further explore why our method performs better and understand the superiority of the introduced discrete latent space, we visualize the projection of extracted z lov on both training and validation set

---

when the model achieves the best score on the validation set, using t-SNE [67 on the covariate-shift dataset of HIV-Scfffll in Figure (d). We also visualize the results of some baselines, including the vanilla ERM (Figure 4 (a)), the ERM equipped with VQ after encoder (ERM(+VQ)
Figure (b)), and with the RVQ module (ERM(+RVQ), Figure (c)). We additionally compute the 1-order Wasserstein distance (68] between the features on the training and validation sets of each class, to quantify the dissimilarity in the feature distribution across varying environments. We find that adding the VQ RVQ after the encoder results in a more uniform distribution of features and lower feature distances, due to the fact that VQ makes it possible to reuse previously encountered embeddings in new environments by discretizing them. Moreover, the feature distribution of our method is more uniform and the distance smaller. This suggests that our method is effective in identifying features that are invariant across different environments. Moreover, also observe that all ERM methods achieve lower validation scores with higher training scores, implying that these methods are prone overfitting Our method, the other hand, achieves not only higher validation score but also a higher corresponding training score, thereby demonstrating its ability to overcome the problem of easy overfitting and improve the generalization ability effectively.
6 Conclusion
In this work, we propose a new framework that learns invariant molecular representation against distribution shifts. We adopt a "first-nncoding- serniggttee--eeee "een--eprattion strategy, wherein a combination of encoding GNN and residual vector quantization is utilized to derive molecular representation in latent discrete space. Then we learn scoring GNN to identify invariant features from this representation. Moreover, we design a task-agnostic self-supervised learning objective to enable precise invariance identification and versatile applicability to various tasks. Extensive experiments on real-world datasets demonstrate the superiority of our method on the molecular OOD problem. Overall, our proposed framework presents a promising approach for learning invariant molecular representations and offers valuable insights for addressing distribution shifts in molecular data analysis.
Acknowledgments
This work was supported by the National Key Research and Development Program of China (2022YBB4500300), National Natural Science Foundation of China (NSFCU19B2027, NSFC91846204, NSFC62302433), joint project DH-2022ZY0012 from Donghai Lab, and sponsored by CCF-Tencent Open Fund (CCF-Tencent RAGR20230122) We want to express gratitude to the anonymous reviewers for their hard work and kind comments.
References
[1] Eugene N Muratov, Jürgen Bajorath, Robert P Sheridan, Igor V Tetko. Dmitry Filimonov, Vladimir Poroikov, Tudor I Oprea, Igor Baskin, Alexandre Varnek, Adrian Roitherg, et al. Qsar without borders. Chemical Society Reviews,
2020.
[2] Stephani Joy Y Macalino, Vijayakumar Gosu, Sunhye Hong, and Sun Choi. Role of computer- aided drug design in modern drug discovery. Archives of pharmacal research, 38: 38:1686-1701, 2015.
[3] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In /CML., volume 70 of Proceedings of Machine Learning Research, pages 1263-1272. PMLR, 2017.
[4] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et Analyzing learned molec- ular representations for property prediction. Journal of chemical information and modeling. 59(8):3370-33 3388, 2019.
[5] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer lcrge-scale molecular data. In NeurIPS, 2020.
10

---

[6] Yin Fang, Qiang Zhang, Haihong Yang, Xiang Zhuang, Shumin Deng, Wen Zhang, Ming Qin, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Molecular contrastive learning with chemical element knowledge graph. In AAAI, pages 3968-3976. AAAI Press, 2022.
[7] Yin Fang, Qiang Zhang, Ningyu Zhang, Zhuo Chen, Xiang Zhuang, Xin Shao, Xiaohui Fan, and Huajun Chen. Knowledge graph-enhanced molecular contrastive learning with functional prompt. Nature Machine Intelligence, pages 1-12, 2023.
[8] Xiang Zhuang, Qiang Zhang, Bin Wu, Keyan Ding. Yin Fang, and Huajun Chen. Graph sampling- baase meta-learning for molecular property prediction. In IJCAI, pages 4729-4733 ijeai.org. 2023.
[9] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(78733:5833 2021.
[10] Mostafa Karimi, Di Wu, Zhangyang Wang, and Yang Shen. Deepaffinity: interpretable deep learning of compound-protein affinity through unified recurrent and convolutional neural networks. Bioinformatics, 35(18) (3:38)33329-333 2019.
[11] Deisy Morselli Gysi, ftalo Do Valle, Marinka Zitnik, Asher Ameli, Xiao Gan, Onur Varol, Susan Dina Ghiassian, JJ Patten, Robert A Davey, Joseph Loscalzo, et al. Network medicine framework for identifying reuurospggg opportunities for covid-19. Proceedings of the National Academy of Sciences, 118(19):c202558111 2021.
[12] Connor W Coley, Luke Rogers, William Green, and Klavs F Jensen. Computer-assisted retrosynthesis based on molecular similarity. ACS central science, 3(12):1237-1245, 2017.
[13] Yuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bingzhe Wu, Long-Kai Kai Huang. Tingyang Xu, Yu Rong, Lanqing Li, Jie Ren, Ding Xue, Houtim Lai, Shaoyong Xu, Jing Feng, Wei Liu, Ping Luo, Shuigeng Zhou, Junzhou Huang, Peilin Zhao, and Yatao Bian. Drugood: Out-of-distributioo (OOD) dataset curator and benchmark for ai-aided drug discovery A focus on affinity prediction problems with noise annotations. CoRR, Abs/2201.096311 2022.
[14] David Mendez, Anna Gaulton, A Patrícia Bento, Jon Chambers, Marleen De Veij. Eloy Félix, María Paula Magariños, Juan F Mosquera, Prudence Mutowo, Michal Nowotka, et al. Chembl: towards direct deposition of bioassay data. Nucleic acids research, 470D1))D930-D940, 2019.
[15] Douglas B Kitchen, Hélène Decornez, John R Furr, and Jürgen Bajorath. Docking and scoring in virtual screening for drug discovery: methods and applications. Nature reviews Drug discovery, 3(1 1):955-949, 2004.
[16] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, (2):5133-330 2018.
[17] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In COLT, 2009.
[18] Krikamol Muandet, David Balduzzi. and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML (1), volume 28 of JMLR Workshop and Conference Proceedings, pages 10-18. JMLR.org. 2013.
[19] Martín Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization. CoRR,
2019.
[20] Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. In NewrIPS, pages 3438-3450, 2021.
[21] Elliot Creager, Jörn-Herik Jacobsen, and Richard S. Zemel. Environment inference for invariant learning. In ICMI., volume 139 of Proceedings of Machine Learning Research, pages 2189-2200. PMLR, 2021.
11

---

[22] Masanori Koyama and Shoichiro Yamaguchi. Out-of-distribution generalization with maximal invariant predictor. CoRR, 0ass/2008001883 2020.
[23] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society. Series B (Statistical Methodology), pages 947-1012, 2016.
[24] Mateo Rojas-Carulla, Bernhard Schölkopf, Richard E. Turner, and Jonas Peters. Invariant models for causal transfer learning. J. Mach. Learn. Res., 19:36:1-36:34, 2018.
[25] Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for -itto--distribution molecular representations. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[26] Yonggiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, Kaili Ma, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-of- distribution generalization on graphs. In Advances in Neural Information Processing Systems, 2022.
[27] Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for out-of-distribution generalization. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 11828-10 11841. Curran Associates, Inc., 2022.
[28] Yongduo Sui, Xiang Wang. Jiancan Wu, Min Lin, Xiangnan He, and Tat-Seng Chua. Causal attention for interpretable and generalizable graph classification. In KDD, pages 1696-1705. ACM, 2022.
[29] Yingxin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. In ICLR. OpenReview.net, 2022.
[30] Li-Juan Jiang, Milan Vašák, Bert L. Vallee, and Wolfgang Maret. Zinc transfer potentials of the a-and B-clusters of metallothionein are affected by domain interactions in the whole molecule. Proceedings of the National Academy of Sciences, 976))2553325508 2000.
[31] Marie Jose Huron and Pierre Claverie. Calculation of the interaction energy of one molecule with its whole surrounding. i. method and application to pure nonpolar compounds. The Journal of Physical Chemistry, 76(15):2123-211)
1972.
[32] Shaohua Fan, Xiao Wang, Yanhu Mo, Chuan Shi, and Jian Tang. Debiasing graph neural networks via learning disentangled causal substructure. CoRR, abs/2209.14107) 14107, 2022.
[33] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR (Poster). OpenReview.net, 2017.
[34] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pages 1024-1034, 2017.
[35] Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR. OpenReview.net, 2019.
[36] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[37] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
[38] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
12

---

[39] Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu. Han Yu, and Peng Cui. Towards out-t--distribution generalization: A survey. CoRR, abs/2108. 136244 2021.
[40] David Krueger, Ethan Caballero, Zhang. Rémi Le Priol, and Aaron C. Courville. nut-of-distributiin generalization via risk extrapolation (rex). In ICMI, volume 139 of Proceedings of Machine Learning Research, pages 5815-5826. PMLR, 2021.
Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai
[41] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In ICLR. OpenReview.net, 2020.
[42] Michael Zhang, Nimit Sharad Sohoni, Hongyang R. Zhang, Chelsea Finn, and Christopher Ré. Contect-n-contrast: a contrastive approach for improving robustness to spurious correlations. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 26484-26516. PMLR, 2022.
[43] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversaae training of neural networks. J. Mach. Learn. Res., 17:59:1-59:33 2016.
[44] Baochen Sun and Kate Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In ECCV Workshops (3), volume 9915 of Lecture Notes in Computer Science, pages 443-450 2016.
[45] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV (15), volume 11219 of Lecture Notes in Computer Science, pages 647-663. Springer, 2018.
[46] Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On learning invariant representations for domain adaptation. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 7523-7532. PMLR, 2019.
[47] Gang Liu, Tong Zhao, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Graph rationalization with environment-based augmentations. In KDD, pages 1069-1078. ACM, 2022.
[48] Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Ood-gnn: Out-of-distribution gen- eralized graph neural network. IEEE Transactions on Knowledge and Data Engineering, 2022.
[49] Siqi Miao, Mia Liu, and Pan Li. Interpretable and generalizable graph learning via stochastic attention mechanism. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 15524-15543. PMLR, 2022.
[50] Zhitao Ying. Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. In NeurIPS, pages 9240-9251, 2019.
[51] Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Explainability in graph neural networks:
A taxonomic survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[52] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NIPS, pages 6306-6306 6315, 2017.
[53] Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAF-2. In NewrlPS, pages 14837-14847, 2019.
[54] Woncheol Shin, Gyubok Lee, Jiyoung Lee, Joonseok Lee, and Edward Choi. Translation- equivariant image quantizer for bi-directional image-text generation. CoRR, abs/21 12.00384, 2021.
[55] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process., 30:495-507, 2022.
13

---

[56] Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z. Li. Mole-BERT: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations, 2023.
[57] Dianbo Liu, Alex Lamb, Kenji Kawaguchi, Anirudh Goyal, Chen Sun, Michael C. Mozer, and Yoshua Bengio. Discrete-valued neural communication. In NewrIPS, pages 1109-2121, 2021.
[58] Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. CoRR, abs/2207.1124)) 2022.
[59] Dianbo Liu, Alex Lamb, Xu Ji, Pascal Notsawo, Michael Mozer, Yoshua Bengio, and Kenji Kawaguchi. Adaptive discrete communication bottlenecks with dynamic vector quantization. CoRR, abs/2202.01334, 2022.
[60] Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern recognition, 45(1):521 2012.
[61] Joaquin Sitnonero-Candelle Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset shift in machine learning. Mit Press, 2008
[62] Gerhard Widmer and Miroslav Kubat. Learning in the presence of concept drift and hidden contexts. Machine learning, 23:69-101, 1996.
[63] Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. GOOD: A graph out-of-distribution benchmark. CoRR, Abss2200.008522 2022.
[64] Rafael GómezzBombarelli, Jennifer Wei, David Duvenaud, José Miguel Hernández-Looato
Timothy D Hirzel, Benjamín Sánchez-Lengeling Dennis Sheberla, Jorge chemical design using a data-driven Ryan Adams, and Alán Aspuru-Guzik. Automatic continuous representation of molecules. ACS central science, 4(2):268-276. 2018.
[65] Vladimir Cherkassky. The nature of statistical learning theory. IEEE Trans. Neural Networks, 8(6):1564 1997.
[66] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR (Poster). OpenReview.net, 2018.
[67] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911), 2008.
[68] Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
[69]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary Veto,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, pages 8024 8035, 2019.
[70] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometrie. CoRR, abs/ 103.0428,,, 2019.
14

---

A Details of Datasets
A.1 Dataset
In this paper, use 18 publicly benchmark datasets, 12 of which are from GOOD 63 benchmark. They are the combination of 3 datasets (GOOD-HIV, GOOD-ZINC and GOOD-PCBA). types of distribution shift (covariate and concept), and environment-splittnng strategies (scaffold and size). The rest datasets are from DrugOOD [3] benchmark, including IC50-Assay, IC50-Scaffold, IC50-Size, EC50-Assay, the suffix denotes the environment-splitinn strategies. This benchmark exclusively focuses on covariate shift. We use the latest data released on the official webpage based on the ChEMBL 30 [ We use the default dataset split proposed in each henchmark. For covariate shift, the training, validation and testing sets obtained based environments without interactions. For concept shift, a screening approach is leveraged to scan and select molecules in the dataset. Statistics of each dataset in Table
and EC50-Size. The prefix denotes the measurement and
Table 4: Dataset statistics.
Datasct
Task
Metric
#Train
#Test
#Tasks
scaffold
covariate
Binary Classification
ROC-AUC
24682
4133
4108
concept
Binary Classification
ROC-AUC
15209
9365
10037
covariate
Binary Classitiatiion
ROC-AUC
26169
4112
3961
size
concept
Binary Classification
ROC-AUC
14454
3096
10525
GOOD
scaffold
covariate
Regression
MAE
149674
24945
24946
ZINC
concept
Regression
MAE
101867
43539
60393
covariate
Regression
MAE
161893
24945
17042
size
concept
Regression
scaffold
covariate
Multi-task Binary Classification
262764
44019
43562
128
PCBA
concept
Multi-task Binary Classitication
AP
159158
90740
119821
128
covariate
Multi task Binary Classification
269990
43792
31925
128
size
concept
Multi-task Binary Classification
150121
32168
115208
128
assay
Binary Classification
ROC-AUC
34953
19475
19463
DrugOOD
IC50
scaffold
Binary Classitication
ROC-AUC
22025
19478
19480
size
Binary Classification
ROC-AUC
37497
17987
16761
assay scaffolld
Binary Classitication
ROC-AUC
4978
2761
2725
EC50
Binary Classification
ROC-AUC
2743
2723
2762
size
Binary Classification
ROC-AUC
5189
2495
2505
A.2 The Cause of Molecular Distribution Shift
The molecule data can be divided according to different environments, and distribution shifts occur when the source environments of data are different during training and testing, In this work, we investigate three types of environment-splitinng strategies, i.e., scaffold, size and assay. And the explanation of each environment are in Table
B Details of Implementation
B.1
Baselines
We adopt the following methods as baselines for comparison, one group of which are common approaches for non-Euclle Euclidean data:
ERM 65
minimizes the empirical loss on the training set.
IRM [19 seeks to find data representations across all enviromments by penalizing feature distribu- tions that have different optimal classifiers.
VREx 40 reduces the risk variances of training environments achieve both covariate robustness and invariant prediction.
htap:e:://rrugod..gii...
15

---

Table 5: Description of different environment splits leading to molecular distribution shifts.
Environment
Explanation
Scaffold
Molecular scaffold the fundamental structure of a molecule with desirable bioactive properties. Molecules with the same scaffold belong the same environment. Distribution shift arises when there is change the molecular scaffold.
Size
The size of molecule refers to the total number of atoms in the molecule. Molecular size is also an inherent structural characteristic of molecular graphs. The distribution shifi occurs when size changes.
Assay
Assay experimental method as examination determination for molecular characteristics. Due to variations in assay environments and targets, activity values measured by different assays often differ significantly. Samples tested within the assay belong single environment, while change in the assay leads to a distribution shift.
GroupDRO 41 minimizes the loss on the worst-performing group, subject to a constraint that ensures the loss each group remains close.
Coral encourages feature distributions consistent by penalizing differences in the means and covariances of feature distributions for each domain.
DANN 4 encourages features from different environments indistinguishable by adversarially training a regular classifier and domain classifier.
Mixup 66
augments data in training through data interpolation.
And the others are graph-specific methods:
DIR 29 discovers the subset of a graph as invariant rationale by conducting interventional data augmentation to create multiple distributions.
proposes to build an interpretable graph learning method through the attention mecha- nism and inject stochasticity into the attention to select label-relevant subgraphs.
GSAT 49
GREA identifies subgraph structures called rationales by environment replacement to create virtual data points to improve generalizability and interpretability.
CAI 28 proposes causal attention learning strategy for graph classification encourage GNNs to exploit causal features while ignoring the shortcut paths.
D 32 analyzes the generalization problem of GNNs in causal view and proposes disentan- gling Tramework for graphs to learn causal and bias substructure.
investigates the OOD problem on molecules and designs an environment infer- ence model and a substructure attention model to learn environment-ioe invironment-invariaan molecular substruc- tures.
MoleOOD m
CIGA 26 proposes information--thooneiicc objective extract the desired invariant subgraphs from the lens of causality.
B.2 Implementation
Experiments are conducted on one 24GB NVIDIA RTX 3090 GPU.
httpp://ggthhbb.ooo////xxxtttttt
16

---

ERM
75
ROC-ABC
74
ROC-AUC
73
72
71
[C]
Cure
Ours
Cura
FRM
ROC-A ROC-AUC
ROC-AUC
ROC-AUC
70
Aa
Figure 5: Hyper-parameter sensitivity analysis on the covariate-shift dataset of GOODHIV-Scaffol.
the results provided in the official leaderboard [2 For datasets in the DrugOOD benchmark, we use the official benchmark code 13 to get the results for ERM, IRM, Coral, and Mixup the latest version of datasets. The results for GroupDRO and DANN are not reported due to an error occurred while the code was running. For some basclines that do have reported results, implement them using public codes. All of the baselines are implemented using the GIN-Virtual [ 35 (on GOOD) or GIN (35) (on DrugOOD) as the GNN backbone that parameterized according the guidance of the respective benchmark. And we conduct a grid search to select hyper-panne parameters for all implemented baselines.
Baselines. For datasets in the GOOD benchmark,
Our method. We implement the proposed iMoLD in Pytorch [69] and PyG [70]. For all the datasets, select hyper-parameters by ranging the code book size C from {100, 500, 1000, 4000, 10000), threshold y from {0.1, 0.5, 0.7, 0.9}. A1 from (0.001, 0.01, 0.5). X2 from {0.01, 0.1, 0.5. 1), As from (0.01, 0.1, 0.3, 0.5, 1). and batch size from {32, 64, 128, 256, 512). For datasets in DrugOOD, we also select dropout rate from {0.1, 0.5). The maximum number of epochs is set to 200 and the learning rate is set 0.001. Please refer to Table detailed hyper-parameter configuration of various datasets. The hyper-parameter sensitivity analysis is in Appendix C.I
C Additional Experimental Results
C.1 Hyper-parameter Sensitivity Analysis
Take the dataset of covariate-shift split of GOODHIV-Scafold experiments to investigate the Hyper-paramett sensitivity, and the results are shown in Figure [5] We observe that the performance tends to improve first and then decrease slightly as the size of the codebook [C] increases. This is because small codebook limits the expressivity of the model, while large one cuts the advantage the discrete space. The effect of the threshold and there is no remarkable trend. As the \T X2 and A increase, the performance shows tendency increase first and then decrease, indicating that Cav. Lug and Cem are effective and can improve performance within reasonable range. We also observe that the standard deviation of performance increases A increases. which may be due the fact that too much weight on the supervisdd invariant learning objective may enhance or affect the performance. The standard deviation is the smallest when X2 = O, suggesting that neural networks may have more stable outcomes by learning adaptively when there are no constraints, but it is difficult to obtain higher performance. While the
example, conduct extensive
insignificant
12 httppp///oooorreaacccacsssee//es http:::://ttaal.ccc////cccaaaaaa
html
17

---

Table 6: Hyper-paramelct configuration.
CC]
batch-size
A1
A2
À
dropout
covariate
0.8
4000
128
0.01
0.5
0.1
scaffold
concept
0.7
4000
256
0.01
0.5
0.1
HIV
covariate
0.7
4000
256
0.01
size
concept
0.9
4000
1024
0.01
0.5
0.1
covariate
0.3
4000
32
0,01
0.5
0.1
scaffold
concept
0.5
4000
256
0.01
0.5
0.1
GOOD
ZINC
covariate
0.3
4000
256
0.01
0.5
0.1
size
concept
0.3
4000
64
0.0001
covariate
0.9
10000
32
0.0001
scaffold
concept
0.9
10000
32
0.0001
0.1
PCBA
covariate
0.9
10000
32
0.0001
0.1
size
concept
0.9
10000
32
0.0001
0.1
assay
0.7
1000
128
0.001
0.5
0.1
0.5
IC50
scaffold
0.9
1000
128
0.0001
0.5
0.1
0.5
size
0.7
1000
128
0.01
0.5
0.1
0.1
DrugOOD
assay
0.7
500
128
0.01
0.5
0.1
0.5
EC50
scaffold
0.3
500
128
0.001
size
0.7
500
128
0.001
0.5
0.1
0.1
standard deviation is the largest when A3 = O, indicating that the commitment loss in VQ can increase the performance stability.
18


Reviews:
[{"review_id": "UNLjJ0vOOG", "reviewer": "Reviewer_d9FY", "summary": "This paper presents a new approach to obtain robust molecular representation through a first-encoding-then-separation method. The proposed method utilizes a graph neural network (GNN) as a molecule encoder and applies a residual vector quantization module to modify the representation. Additionally, a scoring GNN is employed to separate the resulting representations into spurious and invariant categories. The learning process involves contrastive-based self-supervised learning (SSL) loss and task prediction losses. Experimental results on three molecule-related benchmarks demonstrate the superiority of the proposed method over traditional debiasing techniques and recent methods designed specifically for molecule debiasing. Ablation studies and visualization techniques are conducted to provide further insights and analysis.", "questions": "See above in the weakness part.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed first-encoding-then-separation approach is novel.\n2. Experiments on various datasets have shown that the proposed method has the ability to achieve better results.", "weaknesses": "1. The motivation is not clear. Why the first-encoding-then-separation approach is reasonable? \n2. The reason to combine different components is also not clear, making the technical contribution not strong. The current version is like a straight forward combination without sufficient insight or understanding on the problem. For example, why we need a RVQ module in the molecule representation?\n3. It is not clear why the proposed method has the ability to mitigate spurious biases to achieve better OOD results. Is there any theory to support that?\n3. The experiments are not sufficient. For example, only improved results have been demonstrated, without sufficient analysis. In the ablation study of different modules are not consistent on different data, making the technique very ad hoc. Though some visualization have been provided, they are not sufficient to support the claim that the proposed method obtain better invariant features. What does it mean by a uniform distribution? Is the uniform distribution equivalent to a good feature?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### (Q1) Motivation for \"first-encoding-then-separation\"\nIn contrast to previous works, we adopt a novel paradigm that encodes the whole molecule first, then do separation in the latent representation space. The motivation for this approach is detailed in the global response.\n\n### (Q2) Technical contribution\nWe explain the technical contributions in detail in the global response. Our analysis discerns the unsuitability of dividing in structural space. Instead, we advocate for the \"first-encoding-then-separation\" paradigm, bolstered by the integration of the RVQ module, which enhances the refinement of the encoded representation. Further, we recognize that the downstream tasks related to molecules are diverse, and some existing methods cannot be applied to all of them. Towards this end, we propose a \"task-agnostic self-supervised invariant learning objective\" that is applicable to all tasks. We believe our method is devised after we have analyzed the intrinsic differences between molecules and other graph data, with sufficient insight and understanding on the problem.\n### (Q3) Theory for mitigating spurious biases\nOur approach is theoretically supported. Notably, both the Invariance Principle and the VQ theory lend support to our model, contributing to its enhanced generalization performance.\n1. _Invariance Principle._ Our method follows the Invariance Principle, which guides us focusing on the causal factors that remain invariant to distribution shifts while overlooking the spurious parts. Formally, our objective can be represented as $\\min I({z}^{inv}, {z}^{spu}), \\max I({z}^{inv}, y)$，where ${z}^{inv}$ and ${z}^{spu}$ are defined in Equation 10 to represent invariant and spurious features, respectively. To practically achieve $\\min I({z}^{inv}, {z}^{spu})$, we leverage $\\max I({z}^{inv}, \\widetilde{{z}}^{inv})$ as an approximation, as outlined in Equation 12.\n2. _VQ._ The robustness of noise against distribution shifts gains theoretical validation in citations [56, 57, 58].\n   \n[56] Discrete-valued neural communication. NeurIPS'21.\n\n[57] Discrete key-value bottleneck. ICML'23.\n\n[58] Adaptive discrete communication bottlenecks with dynamic vector quantization. AAAI'23.\n\n### (Q4) Experiments\nWe conduct comprehensive experiments in Section 5 and Appendix, including performance comparison on 2 benchmark datasets (Table 1 and Table 2), ablation study (Table 3), visualization (Figure 4) and hyperparameter analysis (Figure 5). We will explain the ablation study and virilization here to address your questions.\n\n***Ablation study.***\nIn the ablation study, we explore 3 groups of model variants. The first two groups involve the sequential removal of individual modules. The third group focuses on the replacement of our proposed task-agnostic self-supervised invariant learning objective, $L_{inv}$ in Equation 16, with alternative counterparts. Specifically, \"w/ $L_{CIGA}$\" and \"$w/ L_{GREA}$\" denotes the substitution of $L_{inv}$ with the objective function of CIGA[1] and GREA[2], respectively. The \"/\" in Table 3 specifies that CIGA's objective function is incompatible with PCBA dataset due to its suitability solely for single-label classification tasks, while PCBA is a multi-label classification task. This is not a case of \"different modules are not consistent on different data\", but rather highlights the specificity of certain approaches to particular tasks. Our approach, in contrast, boasts a broader applicability that encompasses a range of tasks, setting it apart from task-specific alternatives.\n\n***Visualization.***\nIn visualization, we use two approaches to evaluate the goodness of features:\n- _Distance-based Evaluation._ We measure distances between features in class-specific training and validation sets to assess invariant feature quality, given that features of the same class should be nearer. Illustrated in Fig. 4 titles as \"D(Y=0)\" and \"D(Y=1)\", these distances represent data labeled 0 and 1 in training/validation sets. Notably, our method achieves the smallest distance, affirming robustness of our invariant features to environment shifts.\n- _Inference Score Analysis._ We compute inference scores for both training and validation sets, depicted as \"Score(train)\" and \"Score(val)\" in Fig. 4 titles. These scores represent performance on respective sets. We observe that other methods could not achieve the highest performance on both two sets, while our method is able to. This demonstrates the ability of our method to overcome the easy overfitting and achieve the best generalization ability. \n\nWe employ the above two approaches to assess the quality of features. The uniform distribution that emerges from our model is a noted phenomenon, yet it does not serve as a definitive criterion for evaluating the quality of the features. We use the term “uniform distribution” to refer that the features locate and expand uniformly in the latent space, with few isolated clusters. As depicted in Fig. 4, the visualization outcomes of the ERM, ERM(+VQ) and ERM(+RVQ) exhibit discernible clusters, unlike that obtained through our method.\n\n[1] Learning causally invariant representations for out-of-distribution generalization on graphs. NeurIPS'22.\n\n[2] Graph rationalization with environment-based augmentations. KDD'22.\n"}, {"review_id": "uAfZfp8ZIY", "reviewer": "Reviewer_czws", "summary": "The paper presents an invariant and robust representation learning approach for molecules to improve the out-of-distribution generalization performance of the predictive models. Specifically, they first map the molecule to the latent representation and then do a separation step where they separate the latent word into invariant and spurious representations.\nThey also propose using residual vector quantization on the latent representation to avoid over-fitting while preserving the expressiveness power of the encoder.", "questions": "1) Intuitively, what is the difference between applying the Frobenius norm directly to the S matrix versus the regularization defined in equation (14)?\n\n2) It is a bit confusing to use \"h\" in equation (2) and equation (12) to represent different meanings.\n\n3) It is not clear what effect the discrete term Q(h) has on the learned final note representation H', since H' is the sum of the discrete and continuous representations. Is it possible for the model to completely ignore the discretization step and focus only on the continuous representation?\n\n4) In line 191, it is mentioned that \"It is worth noting that our separation is not only performed at the node dimension but also takes into account the feature dimension in the latent space.\" I didn't quite understand this. Could you provide more explanation?\n\n5) Could you explain the intuition behind what S is learning in equation 8? Essentially, it seems  S is just reweighing every element in H'. Intuitively, for the parts of H' that are not very important/invariant/main motif, S should be low so that those elements mainly contribute to the spurious representation, and vice versa. But how does the model enforces this?\n\n6) I'm not sure if the learned high-level representation can be seen as the sum of the invariant and spurious representations. In other words, can we really break down the abstract learned representation of such a complex structure into invariant and spurious parts? Would each of these components eventually represent some substructures if decoded?\n\n7) The paper states that the model learns discrete latent representation, but according to equation 6, the continuous representation is added back to the discretized representation. Can we still claim that the final learned representation is discrete?\n\n8) It would be very helpful to provide a brief explanation of how the dataset is split, how the out-of-distribution is represented in the training/test/validation data, and what the terms \"covariates\" and \"concepts\" refer to in Table 1. This would provide context, especially for readers who are not familiar with the dataset.", "limitations": "The paper did not discuss the limitations of the work and there is no potential negative societal impact of the work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper tries to address an interesting problem.\n2. The proposed idea is novel. \n3. They included a detailed ablation study which helps identify the effectiveness of each component.", "weaknesses": "1)  The experimental results when compared to the baseline do not have noticeable improvement. \n2)  Some more details on the experiment section would be helpful, for example in Figure 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### (Q1) The difference between Frobenius norm and the regularization in Equation (14).\n\nThe $<\\mathbf{J},\\mathbf{S}>\\_F$ in Equation (14) is equal to the 1-order Frobenius norm of $S$. \nHowever, the Frobenius norm merely encapsulates the summation of the elements in the matrix, and the range of its value varies with the size of the matrix. To confine this variability to an appropriate scope, we therefore employ $\\frac{<\\mathbf{J},\\mathbf{S}>\\_F}{|\\mathcal{V}| \\times d} $ to constrain the norm between 0 and 1 despite the variance of matrix size. Intuitively, this is the ratio of the identified invariant features (as delineated in Equation (9) for the invariant and spurious features separation). And to prevent abundance or scarcity of the invariant features, we set a threshold $\\gamma$, tasked with optimizing the model to incline towards a selection of invariant attributes' proportion approximating the predefined $\\gamma$ value.\n\n### (Q2) Confusion of notation\nThank you for pointing this out, we will substitute the \"h\" in equation (12) with a different notation.\n\n### (Q3) The effect of the discretization\nVQ discretizes the continuous representation. Intuitively, it plays the role of a bottleneck, constraining the expressive ability of the neural network. This quality of VQ prevents overfitting  on the training distribution and improves the generalization as discussed in lines 119-124. However, VQ also potentially leads to under-fitting because of the limited expressivity. Thus we consider both continuous and discrete representations to strike a balance between generalization and expressivity. And the experimental results in Table 3 verifies the effectiveness of our approach. It can be observed that both discretization and the residual connection contribute to the performance, and removing residual connection leads to a significant performance degradation on PCBA while removing discretization has a more serious impact on HIV.\n\n### (Q4) Explanation of node dimension and feature dimension\nFor the representation matrix $\\mathbf{H}^\\prime \\in \\mathbb{R}^{|\\mathcal{V}|\\times d}$, the node dimension corresponds to the rows, spanning from 1 to $|\\mathcal{V}|$, while the feature dimension aligns with the columns, encompassing the range from 1 to $d$. Given that the score matrix $\\mathbf{S}$ has the same size with $\\mathbf{H}^\\prime$, and an element-wise product is conducted with $\\mathbf{S}$ and $\\mathbf{H}^\\prime$ to obtain invariant features. This separation process is employed at each element within the representation matrix, thus encompassing both the node and feature dimensions.\n\n### (Q5) What S is learning and how to enforce this\nThe element at $(i,j)$ position in $\\mathbf{S}$ denotes the contribution weight of the $(i,j)$ counterpart in $\\mathbf{H}^\\prime$ to the invariant representation. For not very important/invariant element in $\\mathbf{H}^\\prime$, the corresponding element in $\\mathbf{S}$ should be low, and vice versa. To enforce the model to produce precise scores as well as invariant representations, we design a task-agnostic self-supervised invariant learning objective as described in Section 4.3, and we use it with task prediction loss to jointly optimize the model.\n\n### (Q6) Separation into invariant and spurious parts\nIdentifying invariant parts is a viable strategy to solve the OOD issue, given that these invariant attributes maintain an exclusive affiliation with labels, remaining unaffected by environment shifts. The remaining parts subsequent to the extraction of these invariant attributes are termed as the \"spurious\" components. As shown in Figure 1, preliminary studies follow a \"first-separation-then-encoding\" paradigm, which first divides the graph into invariant and spurious substructures explicitly, and then encode each separately. We argue this practice is suboptimal for extremely complex and entangled molecule graphs. And the detailed motivation is illustrated in the global response. Thus we propose a \"first-encoding-then-separation\" paradigm, that encodes molecules first and then identify invariant features in the latent space. As mentioned by the reviewer, we cannot ensure complete separation of the invariant and spurious components in the abstract learned representation. This is primarily due to the complex and entangled characteristics of real-world molecule graphs. We can solely rely on analyzing the experimental results to show the enhanced separation achieved, as evidenced by its superior performance compared to other baseline methods. Decoding the invariant and spurious parts in the latent space into structural space is a worthy direction for further investigation, and it is believed that this can provide interpretability to the model. We leave this as future work.\n\n### (Q7) The claim of discrete representation\nSorry for the confusion. We did not mean the final representation obtained is discrete. In line 185 we indicate that the final representation is a combination of both continuous and discrete components. We use \"discrete space\" in the title to emphasize the Vector Quantization (VQ) operations to address molecule OOD problem, which is one of the contributions of this work. We will refine this claim in the final version.\n\n### (Q8) Explanation of dataset\nWe explain \"covariate\" and \"concept\" shift in lines 141-145. And we provide details of different distribution shifts in Figure 2 and Appendix A.2. We will provide an explanation of how the dataset is splited in Appendix. "}, {"review_id": "ZPpY0g6yT2", "reviewer": "Reviewer_GuEJ", "summary": "This paper proposes a molecular self-supervised learning method for out-of-distribution generalization. The authors introduces a \"first-encoding-then-separation\" framework to learn invariant features. For doing so, the authors design discrete latent space with VQ-VAE. The experimental results show that their method improves previous baselines in various out-of-distribution downstream tasks.", "questions": "- How are the hyperparameters chosen in Table 1, 2?\n\n- Is there specific intuition why discrete latent space is useful for out-of-distribution molecular representation learning?", "limitations": "Yes, the authors addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to understand.\n\n- The pre-training objective to separate invariant and spurious features seems to make sense to me.", "weaknesses": "- The complexity of the proposed method is high. The loss function contains several tunable parameters and the ablation study (in Figure 5) shows that the performance is quite dependent on the choice of hyperparameters. \n\n- It seems vague why discrete latent space is needed. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### (Q1) Hyperparameters\nThe results presented for the baseline models within the benchmark are derived from an exploration of their respective hyperparameters. In a congruent manner but limited by computational resources, we have searched a partial set of hyperparameters of our method and empirically set some hyperparameters to fixed values. The ranges of the hyperparameters are described in Appendix B.2. And the chosen hyperparameters are detailed in the uploaded PDF file in the global response.\n\n### (Q2) Intuitive understanding of why discrete latent space\nIn this work, we leverage VQ to discretize continuous representations into discrete ones. For every input representation, VQ looks up and fetches the nearest neighbor in a pre-defined codebook and takes it as output. Intuitively, VQ acts as a bottleneck. In instances where the input is subject to perturbation induced by distribution shifts, the act of discretization emerges as a potent mitigator of such noise, ensuring the output remains unaffected. Therefore, the discretization can enhance model generalization and alleviate the easy-over-fitting issue caused by distribution shifts.\n"}, {"review_id": "y9p2Fl1jXk", "reviewer": "Reviewer_zQUd", "summary": "While significant advances have been made in molecular representation approaches, conventional approaches typically assume that data sources are independent and sampled from the same distribution. However, molecules in real-world drug development often show different characteristics, which might be from a different distribution. This issue is called the out-of-distribution (OOD) problem. OOD challenges the generalization capability of molecular characterization methods and can degrade the performance of downstream tasks. Unlike previous studies' \"first-separation-then-encoding\" approach, this study proposes a \"first-encoding-then-separation\" molecular graph representation paradigm. Specifically, the authors first employ a GNN to encode the molecules and then employ a residual vector quantization module to alleviate the overfitting of the training data distribution while preserving the expressiveness of the encoder. Then, they score molecular representations using another GNN that measures the contribution of each dimension to the target in the latent space, thus clearly distinguishing between invariant and spurious representations. Finally, the authors propose a self-supervised learning objective that encourages the recognition of invariant features and effectively preserves label-relevant information while discarding environment-relevant information. The authors conducted experiments on real-world datasets. The experimental results show that the proposed method outperforms the SOTA methods.", "questions": "Could the authors show 3D visualization graphs representing the extracted features, as in Fig. 4?", "limitations": "None.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unlike the traditional \"first-separation-then-encoding\" approach, the authors propose a \"first-encoding-then-separation\" paradigm that uses an encoding GNN and a scoring GNN to identify invariant features from a graph. The authors use the residual vector quantization module to make a balance between the model's expressivity and generalization. The quantization is used as a bottleneck to strengthen the generalization, and the residual connection complements the model's expressivity. Moreover, the authors design a self-supervised invariant learning objective to facilitate the precise capture of invariant features. This objective is generic, task-independent, and applicable to a variety of tasks.\n\n- The model is cleared described.\n\n- The authors conducted comprehensive experiments on 18 real-world datasets. The experimental results show that the proposed model achieved stronger generalization against SOTA baselines.\n\n- Code has been released and will be valuable for future related research.", "weaknesses": "- There are some typos in the paper. For example, a lack of space before references in line 31.\n\n- OOD is repeatedly defined in lines 29 and 90. In addition, please use \"OOD\" for \"out-of-distribution\" that appears later in the text, such as lines 130 and 353.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### (Q1) Typos\nThanks for pointing out these typos, we will revise them accordingly in the updated version.\n\n### (Q2) 3D visualization\nWe provide the 2D and 3D visualization of extracted features together in the uploaded PDF file in the global response. The 3D visualization results show similar characteristics with their 2D counterparts. For instance, the distribution pattern of ERM representations displays increased discreteness, and both ERM(+VQ) and ERM(+RVQ) representations show partially extended flows. In contrast, iMolD yields a more uniform and consistent representation.\n"}, {"review_id": "Tzr3bwbVDZ", "reviewer": "Reviewer_gGdj", "summary": "This paper presents a new graph neural network architecture and objective function that encourages models to identify features that are invariant to distribution shifts in the data. The proposed method, iMoLD performs invariant feature extraction in the latent embedding space and leads to improved performance across an extensive set of molecular property prediction tasks.", "questions": "Q1) It is not clear to me why vector quantization (VQ) is the right “bottleneck” to use here. Other than restricting the model’s expressivity, which can be done in other ways such as weight regularization, why is VQ particularly suited for this setup?\n\nQ2) Why is the stop gradient applied in equation 12? Is this simply for computation efficiency / stability? If so, this should be made explicit in the text.\n\nQ3) For the GOOD-PCBA experiment, why is average precision (vs. average accuracy, recall, or ROC-AUC) used?\n\nQ4) I know that there is an extensive sensitivity analysis in the appendix, but what are the hyperparameter configurations for the reported results in the main text (Tables 1 and 2)? Are the “best” iMoLD models sensitive to hyperparameter choice or do you see a general trend as to which configurations perform best? \n", "limitations": "- The current methodology does seem quite intricate with many loss terms that are justified in a somewhat ad hoc manner.\n- There is no real discussion of limitations / potential pitfalls relative to previous work.\n", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The presented idea is novel and leads to improved performance across a variety of datasets and tasks.\n- Experimentation is extensive with good results.\n- The ablation analysis is Section 5.3 and sensitivity analysis from Appendix C are useful.", "weaknesses": "#### **Incorrect definitions in Section 3.1**\n- I believe there is some issue in the notation of Section 3.1. Specifically, the definitions of $P_{train}, P_{test}, P_{all}$ as collections of distributions, means that they are not themselves valid probability distributions. I believe some re-normalization would be required here.\n\n---\n\n#### **Use of term “Discrete Latent space” is unclear**\n- Why do the authors claim they have a **“discrete”** latent space? The residual connection between $\\mathbf{H}$ and the quantized representation means that embeddings are continuous. Additionally the element-wise gating to create $\\mathbf{H}^{\\mathrm{Inv}}$ and $\\mathbf{H}^{\\mathrm{Spu}}$ means the model does not have a discrete latent representation.\n\n---\n\n#### **Unclear elements about the learning objective**\n- The notation in Equation (11) is confusing. Specifically, what is the dimensionality of the $\\tilde{\\mathbf{z}}_i^{\\mathrm{Inv}}$? Are you concatenating multiple batch samples from $\\mathbf{z}^{\\mathrm{Spu}}$ to $\\mathbf{z}_i^{\\mathrm{Inv}}$ or just one random one? \n- There seems to be an inherent tension between the residual connection and the commitment loss $\\mathcal{L}_{\\mathrm{cmt}}$. That is, if this loss were perfectly minimized, then the residual connection would be negated.\n- The role of $\\gamma$ in the scoring regularization is not well described. \n\n---\n\n#### **Baseline presentation is confusing**\n- It seems that the authors are conflating baselines in terms of loss objectives and in terms of model/architecture designs. It would be good to clarify which baselines rely on the same architecture but have different objectives (e.g. ERM) and which constitute an entirely different modeling scheme (e.g., CIGA). For the baselines that simply differ in objective, it would be good to also make explicit (could go in Appendix) if any model / architecture adjustments were also applied.\n\n---\n\n#### **Other minor comments**\n- In line 147, the notation for edges $\\mathcal{E}$ is overloaded, since the same variable is used to denote environments in Section 3.1.\n- At the end of Section 5.4 (lines 341-345), the authors seem to be mixing the meaning of low/high in terms of whether low = “good” or low = ”bad”.\n- $D$ and $Score$ should be defined explicitly in Figure 4 caption.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### (Q1) Distribution definition\nThe $P_{train}$, $P_{test}$ and $P_{all}$ given in the manuscript are the form of a collection of distributions, but not a distribution, which needs to be renormalized to avoid ambiguity. We will clarify that in the manuscript.\n\n### (Q2) The claim of discrete space\nThe utilization of the term \"discrete space\" in the title highlights the incorporation of Vector Quantization (VQ) operation within the methodology. This emphasis is placed to underscore the utilization of VQ in tackling the OOD challenge related to molecules, which is a contribution within this study. We agree that the residual connection between the embedding H and its quantized representation is continuous. Indeed, we did not mean the final representation is discrete. In line 185 we indicate that the final representation is a combination of both continuous and discrete components. We will refine this claim in the final version.\n\n### (Q3) Some unclear elements\n- In equation (11), the dimensionality of $\\widetilde{\\mathbf{z}}^{\\mathrm{Inv}}_i$ is 2$d$. To derive $\\widetilde{\\mathbf{z}}^{\\mathrm{Inv}}_i$, we first shuffle a batch of $\\mathbf{z}^\\mathrm{Spu}$ randomly, then concatenate ${\\mathbf{z}}^{\\mathrm{Inv}}_i$ with the corresponding counterpart $\\mathbf{z}^\\mathrm{Spu}$ in the shuffled batch.\n\n- The effect of the commitment loss is to restrain the input $\\mathbf{h}$ , preventing them from excessively deviating from the embeddings $\\mathbf{e}$ within the codebook. It is crucial for $\\mathbf{e}$ to be in close proximity to $\\mathbf{h}$ to effectively serve as a quantized representation.  When the commitment loss is perfectly minimized, the distribution of $\\mathbf{e}$ and $\\mathbf{h}$ are overlapped. It is important to note that the residual connection cannot be fully eliminated. This residual gap persists due to inherent limitations imposed by the finite size of the codebook.\n\n- $\\gamma$ is used to constrain the size of the selected invariant features. As we employ $\\frac{<\\mathbf{J},\\mathbf{S}>\\_{F}}{|\\mathcal{V}| \\times d} $ to calculate the ratio of identified invariant features. To prevent abundance or scarcity of the invariant features, we set a threshold $\\gamma$. This threshold, in turn, functions to optimize the model towards the selection of an invariant feature ratio that closely approximates the predefined value of $\\gamma$.\n\n### (Q4) The baseline presentation\nThanks for this advice, we will add a table to illustrate the loss objective and model architecture design for each baseline in Appendix.\n\n### (Q5) Other minor comments\nThank you for pointing out these issues and we will revise the manuscript correspondingly.\n\n### (Q6) Why VQ is chosen\nWe choose VQ for the following main reasons:\n1. The citations [56, 57, 58] in lines 122-123 provide theoretical analyses that demonstrate how VQ can enhance noise robustness.\n2. Intuitively, VQ can act as a bottleneck. In instances where the input is subject to perturbation induced by distribution shifts, the act of discretization emerges as a potent mitigator of such noise, ensuring the output remains unaffected, thereby enhancing modal generalization and alleviating the issue of easy overfitting caused by distribution shifts.\n3. Certain weight regularization methods, such as IRM, do not perform well when confronted with the molecular OOD problem, as shown in Tables 1 and 2. Their performance is even worse than the ERM baseline.\n\n### (Q7) The stop-gradient in equation 12\nThe stop-gradient operation is used for preventing collapsing. This technique is presented in a self-supervised learning method (citation [36] in line 217). We will clarify this in the manuscript.\n\n### (Q8) Why average precision is used in PCBA\nDue to the extremely unbalanced classes (only 1.4% positive labels), we use the Average Precision (AP) as the evaluation metric.\n\n### (Q9) The hyperparameter configuration and the hyperparameter sensitivity\nThe chosen hyperparameters are detailed in the uploaded PDF file in the global response. Through parameter sensitivity analysis experiments, we observe that iMoLD outperforms ERM within a reasonable range of parameters. We also observe some general trends through hyperparameter analysis, for example, the performance tends to first improve and then slightly decrease as the codebook size increases, the standard deviation of performance increases as $\\lambda_1$ increases, and the standard deviation is the largest when $\\lambda_3$ is zero, indicating that the commitment loss in VQ can increase the performance stability.\n"}]

Author rebuttal:
We sincerely appreciate the reviewers for providing us with their valuable and affirmative comments regarding our submission. 
First of all, we want to clarify our technical contributions. The proposed method mainly consists of three technical contributions:
1. _**The first-encoding-then-separation paradigm.**_ Preliminary studies follow a "first-separation-then-encoding" paradigm, where the graph is first divided into invariant and spurious substructures, and then each part is encoded separately. We argue that this paradigm is suboptimal for extremely complex and entangled molecules, as we have illustrated in lines 48-53, since molecular intricate properties usually cannot be easily determined by analyzing a subset of molecular structures. Citations [29,30] in line 53, which discuss the necessity and importance of considering the molecule as a whole in the study of molecular physicochemical properties, substantiate the motivation. To go into more detail, molecules are composed of atoms, and atoms contact through a cloud of electrons to form covalent bonds. Typically, chemical properties are expressed through electrons, which can conduct across the backbone. The surface potential, polarity, and other chemical properties on some substructures are affected by the charges of atoms in other substructures. The explicit separation of the molecular structure entails a loss of information to inter-substructural interactions. Therefore, distinguishing between invariant and spurious parts in the structural space is suboptimal for molecules and we propose a "first-encoding-then-separation" paradigm to make the distinction in the latent representation space.
2. _**The RVQ module.**_ Since we do not divide the molecule in the structural space, but encode the whole molecule directly. When the environment changes, the obtained representation will be disturbed by the distribution shift noise. To alleviate this issue, we propose to use VQ to improve the generalization ability by discretizing the continuous representation. However, we observe that VQ would also limit the model's expressivity and potentially lead to under-fitting. To address this concern, we propose to equip VQ with a residual connection to strike a balance between generalization and expressivity.
3. _**The task-agnostic self-supervised invariant learning objective.**_ Following the Invariance Principle, we investigate to focus on the causal factors that remain invariant to distribution shifts while overlooking the spurious parts. To accurately make a separation between invariant features and spurious features, an invariant learning objective is needed. We have analyzed that downstream tasks related to molecules are diverse, including regression, single- and multi-label classification. However, the existing invariant learning objectives cannot be applied to certain molecular tasks/datasets (marked with "/" in Table 1). Therefore, we propose the task-agnostic self-supervised invariant learning objective, which is independent of the downstream task and makes our method applicable to various tasks.

In the following sections, we present a detailed point-by-point response to the questions raised.


Meta-review: This paper proposes a method to learn invariant molecule representation using a "first encoding and then separation" scheme. The proposed method employs two graph neural networks. The first one embeds the molecules with residual vector quantization, and the second one scores the nodes and features. The invariant representation is learned by a contrastive self-supervised learning loss, together with classification loss, and regularization. Experiments on benchmarks demonstrate the effectiveness of the proposed method as compared to debiasing methods.

The paper has been reviewed by 5 reviewers. All the reviews are positive after rebuttal and discussion.

Initially, the reviewers raised some concerns, such as the role of discrete latent space, hyper-parameter configuration, motivation, and technical contributions. These concerns were adequately addressed by the authors. One reviewer raised the rating to 5.

Overall, this paper makes a useful contribution to invariant molecule representation.
Decision: Accept (poster)