CFD: Learning Generalized Molecular Representation via Concept-Enhanced  Feedback Disentanglement
Authors: Aming WU
Cheng Deng
Venue: ICLR 2025 Poster (2025)
Keywords: Molecular Representation
Generalization
Feedback Disentanglement
Concepts
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: To accelerate biochemical research, e.g., drug and protein discovery, molecular representation learning (MRL) has attracted much attention. However, most existing methods follow the closed-set assumption that training and testing data share identical distribution, which limits their generalization abilities in out-of-distribution (OOD) cases. In this paper, we explore designing a new disentangled mechanism for learning generalized molecular representation that exhibits robustness against distribution shifts. And an approach of Concept-Enhanced Feedback Disentanglement (CFD) is proposed, whose goal is to exploit the feedback mechanism to learn distribution-agnostic representation. Specifically, we first propose two dedicated variational encoders to separately decompose distribution-agnostic and spurious features. Then, a set of molecule-aware concepts are tapped to focus on invariant substructure characteristics. By fusing these concepts into the disentangled distribution-agnostic features, the generalization ability of the learned molecular representation could be further enhanced. Next, we execute iteratively the disentangled operations based on a feedback received from the previous output. Finally, based on the outputs of multiple feedback iterations, we construct a self-supervised objective to promote the variational encoders to possess the disentangled capability. In the experiments, our method is verified on multiple real-world molecular datasets. The significant performance gains over state-of-the-art baselines demonstrate that our method can effectively disentangle generalized molecular representation in the presence of various distribution shifts. The source code will be released at https://github.com/AmingWu/MoleculeCFD.

Published as a conference paper at ICLR 2025
CFD: LEARNING GENERALIZED MOLECULAR REPRE- SENTATION VIA CONCEPT-ENHANCEN
FEEDBACK DISENTANGLEMENT
Aming Wu, Cheng Deng"
School of Electronic Engineering, Xidian University, Xi a'an, China amwu@xi di an. won.edu..com en, chdeng@mail ..llxxxiiie xidiann edu. on
ABSTRACT
To accelerate biochemical research, e.g., drug and protein discovery, molecular rep- resentation learning (MRL) has attracted much attention. However, most existing methods follow the closed-set assumption that training and testing data share iden- tical distribution, which limits their generalization abilities in oitor--distribution (OOD) cases. In this paper, we explore designing a new disentangled mechanism for learning generalized molecular representation that exhibits robustness against distribution shifts. And an approach
Enhanced Feedback Disentangle- ment (CFD) is proposed, whose goal is exploit the feedback mechanism to learn distribution-agnostiee representation. Specifically, we first propose two dedicated variational encoders separately decompose distribution-agnosiii and spurious features. Then, a set of molecule-aware concepts are tapped to focus on invari- ant substructure characteristies. By fusing these concepts into the disentangled distribution-annosicc features, the generalization ability of the learned molecular representation could be further enhanced. Next, we execute iteratively the disentan- gled operations based on feedback received from the previous output. Finally, based on the outputs of multiple feedback iterations, we construct a self-supervised objective to promote the variational encoders to possess the disentangled capability. In the experiments, our method is verified on multiple real-world molecular datasets. -frtc-rrtt basclines demonstrate that
The significant performance gains our method can effectively disentangle generalized molecular representation the presence of various distribution shifts. The source code will be released
INTRODUCTION
With the rejuvenation of deep neural networks, advance the development of biochemical research, e.g., drug discovery
et al.) (222) and protein design (Ingraham et al., (019). molecular representation learning (MRL. has attracted growing attention, which aims to transform molecules into low-dimensional and dense vectors (Gilmer et al.] 2017] Yang et al.) 2019 Rong ct al. 2020 Fang et al.) 2022, Yang et al., [2022] Zhuang et al. (023). By means of the learned molecular representation, many downstream tasks could be addressed effectively, including drug property prediction (Wang et al. 2024b), binding affinity analysis (Karimi et al. 2019). and search of
antibiotics (Stokes al. 2020). etc.
Though MRL has achieved significant progress, most existing methods often follow the closed-set assumption, i.e., the training and testing data share the same distribution. However, in practical biochemical research, molecules may exhibit many variants and originate from different distributions
( al. 2022 Mendez et al.) 2019). Taking the virtual screening (Lim al. 2019) the model may be affected by the distribution shift in the molecule itself, e.g., size (Ji et al. 2022 or scaffold (Wu al. 2018) changes. Furthermore, some unpredictable events like COVID-19 may bring new target molecules from unknown distributions. Obviously, these scenarios pose significant
example,
"Corresponding Author

---

Published as a conference paper at ICLR 2025
Spurious Subgraph
Spuzious Feature
GNN
Subgraph Generator
GNN
BNN
GNN
GNN
Feedbark
Feedback
Generalized Subgraph
Generalized
Concept
Concept
(a) First- -ww.raeeop-TTaeaeeeee Thw------------iinggggg
Spurious Fostume
Scomr
Variational Discntangle
Variational Disentangle
Varfational Disuntagge
GNN
Generalized Feature
Denatoo
0
Denation
Iterafion: T
(b) First-Encoding Then- Sopaa--pprttion
(c) Concupt- Enhancod Feedback Disentanglement
Figure 1: Three different mechanisms for disentangling generalized features. (a) and (b) commonly used disentangled methods (Zhuang et al. 2023). (a) first uses subgraph generator to separate the input, then each subgraph is encoded, respectively. (b) first encodes the input, then the representation is separated by scorer (Zhuang et al. (2023). (c) is our proposed mechanism for learning generalized representation, i.e., Concept-Enhanced Feedback Disentanglement, which exploits the iteration manner to perform variational disentangled operations based on the feedback received from the previous iteration's output. The combination of all decomposed generalized features is taken as the final generalized representation for downstream tasks.
challenges for closed-set assumption based MRL methods. To this end, improving the generalization of molecular representation is meaningful for reducing the impact of distribution shifts.
Existing methods (Ahuja al. 1021 Yang et al., 2024) mainly focus learning generalized representation for regular Euclidean data (c.g., images). Instead, to preserve rich structural information (Gilmer et al. 2017p. molecules often represented graphs, typical Euclidean data. where atoms and bonds are separately taken as nodes and edges. Thus, the methods (Ahuja et al.) 2021 Euclidean data could not be directly applied molecular graph data. Particularly, the work (Ji et al.) 2022) points out that existing OOD methods (Sagawa et al. 2019 Sun & Saenko 2016 Zhang] 2017) fail to significantly improve the performance of MRL tasks against distribution shifts. In this paper, we explore designing a disentangled mechanism for learning generalized molecular representation.
Yang et al. 2024) corresponding
Recently, some studies (Chen et al. 2022 Li ct al. 2022b Yang et al. 2022) have attempted to improve the generalization of molecular representation, which could classified into two types, i.e. -eaatt--ppaaiio--T1 -eeiiioon-hee----eeee Then-EEncding and Sirst-Encoding-Tiee "hen-Sppraation (Zhuang et al.) 2023). Specifically, as shown Fig. [ (a), the first type first divides the graph into generalized subgraphs and then encodes each part separately. However, this kind of method usually relies on the separation quality. For certain molecules involving complex structures, since there is no high-quality decomposition, the performance of these methods may be weakened severely. To this end, the work (Zhuang et al. 2203)) introduces a "First- -eeete-nnoodd---neee Thene--Teeeeeeeee "hhen-Separation mechanism. As shown in Fig. (b), first employs Graph Neural Network (GNN) (Kipf & Welling, 2017 Hammlton et al.) 2017) encode the molecule. Then, another GNN is utilized to calculate the score of molecule representation, which is taken the critical factor for separating generalized and spurious features. However, for extremely complex and entangled molecules, e.g., biomacromolecules, only leveraging the calculated score could not effectively obtain expected separation results. Meanwhile, this method (Zhuang et al. 2023) fails to focus on certain important substructures containing invariant characteristics, affecting its performance real-worle molecules.
spurious
In Fig. (c). we propose new disentangled mechanism, i.c., Concept-Enhanne Feedback Disentan- glement (CFD), which still follows the "First- -eensst-noddin--Tiin -hennng-Tee-eeeee -thnnnSeparation strategy and exploits the feedback mechanism (Zaiir et al. 2017) to learn generalized representation. Concretely, given an input molecule, we first exploit a GNN to encode its representation. Then, two dedicated variational encoders are designed to separately decompose distribution-agnosse ous features. Next, a series of concepts are learned to capture certain critical substructures involving invariant molecule characteristics. By fusing these concepts into the decomposed distribution aatribution-agnostic features, the generalization ability could be further strengthened. Later, we iteratively perform the above disentangled operations based feedback received from the previous iteration's 's output.
generalized features and spuri-
2

---

Published as a conference paper at ICLR 2025
Finally, based the outputs of multiple feedback iterations, self- sape-sueeeeieed objective is constructed to encourage the variational encoders to own the disentangled capability. Extensive experimental results multiple molecular datasets demonstrate the superiorities of our method.
The contributions are summarized as follows:
(I) We propose paradigm of feedback disentanglement that utilizes the iteration manner to decom- pose generalized molecular representation, improving the robustness against distribution shifts.
(2) We propose concept mining module focusing certain critical substructures involving invariant characteristics. By fusing these concepts into the disentangled representation, the generalization ability could be further enhanced.
(3) We conduct extensive experiments on a diverse set of real-world molecule datasets (Zhuagg et al. 2023) with various distribution shifts. The significant performance gains over multiple baselines demonstrate that our method can disentangle generalized molecular representation effectively, More- over, to further verify the ability of capturing important substructure, our method is evaluated new task, i.e. Molecule's ' Ground-State Conformation Prediction (Xu et al. (2024). We observe that plugging our method into Transformer architecture (Xu et al. (024) could still improve the prediction performance, indicating the superiorities of our method.
RELATED WORK
OOD Generalization. Recent research (Wang al., 2022a Robey al. 2021] Zhou et al. 2024)
has indicated that existing machine learning methods usually suffer from significant performance degradation under distribution shifts, motivating a surge of works on OOD generalization. Particularly, most existing methods (Evuru al., 2024 Ratner et al. 2017; Feder al. 2024) attempt to improve the generalization from the data and representation perspectives. Particularly, an effective strategy is to utilize data augmentation to improve the generalization. Zhou et al. (Zhou et al. 2020) utilize domain information for creating an additive noise to increase the diversity of training data distribution while preserving the semantic information of data. Xu et al. (Xu et al.. 2021a) exploit the classical Fourier transforee cansformationn to obtain the augmented images with diverse styles, which enhances the generalization ability effectively. Besides, by simply mixing up different styles of input data, the work (Zhou et al. 2021) could strengthen the generalization performance significantly.
Another effective strategy to learn invariant representations (Chang al. 2020 Creager et al., 202I). In general, the learned invariant representation should satisfy such a principle: 1) sufficiency: shows sufficient predictive power for the target, 2) invariance: contributes to optimal performance for the downstream tasks across all environments. Therefore, some works (Mahajan et al. 2021 Ly
al. 2022 Wang al. (202bb) explore learning invariant representation from the causal perspective. Furthermore, feature disentanglement (Li et al. 2022a Zhang et al.) 2022: Wu et al. 2021) is also a commonly used mechanism to decompose distributton-agnoottc representations. Though these methods have been demonstrated to be effective
many tasks, they often pay attention to regular Euclidean data (c.g., images). However, molecules are usually represented as graphs, which belong typical oon-uuclidean data. At this time, directly employing the above methods to process graph data does not achieve satisfactory performance. Thus, is meaningful to design a proper mechanism for learning generalized molecular representation.
OOD Generalization on Molecular Representation. Recently, in order to promote the application of biochemical research, learning generalized molecular representation has attracted growing attention. Some methods (Yang et al., 2022 Chen et al. 2022 Fan et al., 2022) follows the "first-eepraation then-encoding" paradigm. And they first design a dedicated module to disentangle the original input graph into two different parts. Then, two independent GNN are utilized to encode these two parts, respectively. However, this kind of methods are prone to relying on the separation results. For certain molecules with complex structures, these methods (Yang et al., 2022 Chen et al. 2022] Fan et al.) 2022) could not achieve the expected performance. To this end, Zhuang et al. (Zhuang et al. 2023) propose a new strategy, i.e., encode molecules. Then, disentangled module is designed to perform representation decomposition. Though this method has been shown be effective, for certain biomacromolecules, the proposed score module could not disentangle invariant representations successfully, limiting its application in real-world molecules. In this paper, explore new mechanism, i.e., Concept- Enhanced Feedback
Concretely, they first employ a GNN to
3

---

Published as a conference paper at ICLR 2025
Spm
H4 9pm
Sampling
Decoder
Spurious Encoder
spu
GNN
Molecule Concepts
GNN
Maximize KL(P ppenn
Feedback
Dew
Variational Discntangle
Sampling
Variational
Generalized
Decoder
Disentangle
Encoder
Gen
Enhanced Generalized Feature Disentanglement
Ieeaaion t
Iteration t+1
Figure 2: The details of method for disentangling generalized molecule representation. Our method mainly contains two components: (1) Feedback Disentanglement, and (2) Molecule Concept Extraction. Partieularly, based on the current extracted representation, two variational encoders designed to separately obtain spurious features and generalized features. By maximizing the distance between estimated distributions, the decomposition ability of two variational encoders could be enhanced. Next, series of concepts are learned to focus on certain important substructures involving invariant characteristics. By fusing these concepts into the disentangled features, the generalization ability could be further enhanced. Finally, by combining the generalized features from multiple iterations, the output could be taken as the final generalized representation for downstream tasks.
Disentanglement, which aims utilize the iteration manner for learning generalized molecular representation. Meanwhile, series of concepts are mined to focus on certain critical substructures, which further enhances the generalization ability. Extensive experimental results on multiple real- world molecule datasets demonstrate that our method could indeed disentangle distribution Distrbbution-annosiie invariant molecular representation.
CONCEPT-ENHANCED FEEDBACK DISENTANGLEMENT
In this paper, we focus on disentangling generalized molecular representation reduce the impact of distribution shifts. Here, we first define the OOD generalization problem for MRL.
In general, a molecular graph can be represented as (V, E), where V is the graph ' node set corresponding to atoms and E denotes the graph's 's edge sets corresponding to bonds. The goal of MRL task to predict the target label ) given the associated input molecule The OOD problem on MRL is defined as follows:
min max (eg. (Rg..:))--Pg..)))CC(((1.) eEf
where & represents the support of environments, f(.) (-) is the prediction model and C(.,)) indicates a loss function. (.G.)))) PC).- [C(f(G)). is called the risk function under given environment (Krueger et al. 2021). The reason of resulting in the OOD) problem on MRL lies in that the training data only cover very limited environments in & while the model is expected to perform well all environments. Meanwhile, the OOD problem can be categorized into two cases, i.e., covariate and concept shift (Krueger et 2021 Widmer & Kubat 1996). In covariate shift, the distribution of input differs. And the category space of input is the same. Instead, for concept shift, the distribution of input remains the same. But the category space between the training and testing data is inconsistent. In the experiments, we consider these two shift cases.
3.1 VARIATIONAL DISENTANGLEMEN FOR MOLECULAR REPRESENTATION
As shown in Fig. 2 our method follows "first-encoding 2023) for learning generalized molecule representation. Different from this work Zhuang al. 2023), ) we leverage the feedback idea (Zamir et al. 2017) to iteratively extract intributiionagnostte representation, which is more applicable to molecules with highly complex structures. Moreover,
wwwerodiinnnt-en-eeppeee then-separation" mechanism Zhuang et al.
4

---

Published as a conference paper at ICLR 2025
considering that this work (Zhuang et al. 2023) fails to focus on critical molecular substructures, we propose to mine a series of concepts that involve invariant characteristics of certain molecule substructures, which could be used to further improve the generalization of the learned representation. Next, we will introduce the details of our method.
Specifically, given input molecule 9, first utilize GNN to encode it. The output is F:
F == GNN(S) R xd
where [ is the number of atoms in G. and d is the dimensionality of the encoded features. Then, spurious encoder (i.e., w (.) and (-)) and generalized encoder (i.e., W-een() and (.)r"()), each consisting of three fully-connected layers, are designed to separately estimate the corresponding distribution. The processes are shown follows:
Spu W.S (Qt), Ti
Spm S (Qt), Pi Gen W. (Qr), of =W -WCen (Q:):
Pi
where Q: = MiZTer Z(er)). t = 0,....1 And Qo = F, Q1 = Zgee. 'I---I' indicates the concatenation operation. "ó(-)' represents fully-connected layer that transforms channels. 14 upu R <v and ugeen R v are the estimated means for spurious and generalized features at the current iteration l, of E RRV xd and ofen RlV represent the estimated variances for spurious and generalized features. Next, separately perform sampling operation obtain the disentangled features 7.5
ER xRIV and 7,Gen RR. RR/]
Spp DD (H; + exp(os ( Z D (yen(u + exp(o
where (.) and (.en(.) are two defined decoders, each consisting of three fully-connected layers. denotes Gaussian noise sampled from (O, I).
To promote 2,5Pz and 2,Gen to separately contain rich spurious and distribution-agnosti information, we first enlarge the distance between the two estimated distributions. Meanwhile, to encourage 2,Gen involve plentiful invariant characteristics related the input molecule, further define a contrast loss. Finally, the loss Lid for variational disentanglement is defined as follows:
exp(sim(Z," (((en FF)/T) (F)(=,
Lyd llog
X KL(P. Spu pGen), (5)
exp(sim ZGen F)/T) + exp(sim (Z... ,F)TT)
where sim sim[Z,een (Z,enn FF) denotes the average of the cosine similarity between all corresponding clements of 7 and F. and À are two hyper-parametery which are separately set to 1.0 and 0.01 in the experiments. p.Spu NHu of _Syo) and PGen ~N( gGen). Obviously, by minimizing the loss Li the gap between 2Gen and 2,5pu could be enlarged from both semantic and distribution perspectives, which enhances the disentangled ability of our method.
3.2 LEARNING MOLECULE CONCEPTS
In general, the critical substructures involving invariant characteristics usually play an important role in downstream tasks. To this end, we explore mining series of concepts that do not specialize to one particular type or class of molecule, which is instrumental in strengthening the generalization of the learned representation (Locatello et al., (202)).
Concretely, the concepts are defined as C in (Me,, o.) E kk where Pe and de are learnable parameters. indicates the number of concepts. Next, based on the disentangled representation 7GGnn IR Vlxd, we calculate descriptors that represent molecule-level information:
D exp(M;.)
M = KAN(Z,GE)) H.
(Z Ct),
(6)
exp(M,,s)
where = 1, 1,... K. Here, we employ one KAN layer (Liu et al. 2024 to transform 7Gen, which outputs M E R Different from MLPs having fixed activation functions on neurons, KANs have learnable activation functions (Liu et al. 2024), which is beneficial for improving the flexibility of the learned concepts. H TR Kx represents the output descriptors. 'ZGen C denotes the
5

---

Published as a conference paper at ICLR 2025
residual operation. By means of this operation, assign molecule substructure features to the corresponding concept C₁ E R Finally, the learned concepts are utilized to enhance the current representation 2,aem The corresponding processes are shown as follows:
H KAN(H,); 7, KAN(HH, ),(")))
where H, E Rlxkdis is the reshaped result of H. H. E Rixd Z E R is the output generalized representation the step f. By the concatenation operation, the learned concepts could fused into the current decomposed representation, which is conducive to enhancing the generalization of the output representation Z,Gen. the experiments, Z utilized as the representation for computing the corresponding task loss, e.g., molecule property prediction, which promotes the learned concepts to capture plentiful task-speeific invariant molecule characteristics.
3.3 FEEDBACK SEPARATING GENERALIZED MOLECULAR REPRESENTATION
Existing methods (Zhuang et al. 2023 Yang al. 2022) for learning generalized molecu- lar representation generally employ one-step disentangled strategy, i.e., directly separating the input into generalized and spurious parts. However, in practice, we may encounter some unknown biomacromolecules containing more atoms and highly complex structures. At this time, using the original one-step mechanism could not obtain satisfactory disentangled re- sults. To this end, exploit the feedback idea (Zamir et al. 2017) to iteratively separate gen- eralized molecular representation.
Snep
Feelback
Co
Q1
Spu
ZGen
pt Z
Gen
Z
Spr
Z
Q2
As shown in Fig. we assume that the number of feedback iterations is T. Given the current feature Q E Rlvlxd that is the concatenation of the previous leverage the above operations to disentangle Qi into Sppu and ZGen, where t == 0, ........ ., Then, Q++1 o( Z(Ger]) ) is taken as the input of the current step to repeat the above disentangled operations. In the computation graph of feedback disentanglement (as shown in Fig. 3}. we can observe that compared with One-Step disentanglement, the feedback strategy follows hierarchical disentangled mechanism, which is instrumental in improving the ability of decomposing the molecules with complex structures. Finally, through multiple feedback iterations,
Figure 3: Computation graph of One-Step vs Feedback Disentanglement For the molecules with complex struc- tures, using one-step disentanglement could obtain the expected separation results. Instead, the feedback mechanism with multiple itcration steps is beneficial for dccomposing complex structures.
iteration outputs, we first
obtain series of generalized outputs, i.e., (ZGen, Z .... ZGen), ) and spurious outputs, i.e., (ZSpu, Z constructed to close the distance between the output generalized features at each step and enlarge the distance berween the decomposed generalized and spurious features at same step, which strengthens the disentangled ability effectively:
ZSpa). A self-supervised objective is
Lia= -log( log(
ΣV Σ 1=0
sim AZnGen ZGen) 14s -
1 B T Σ sim sim(Z Z.Gen )SPH)),
(8)
x
where 3 is pareeerrr which is set to 0.5 in the experiments. Finally, we take the concatenation result, i.e., zGen = ([Z[en ZGen DGEn)) E [Rlvixd as the last generalized representation for downstream tasks, where (-)) indicates fully-connected layer that transforms the channels.
For specific downstream tasks, their goals are to provide the generalized representation ZGen with predictive capabilities. Since our method is to learn generalized molecule representations, it could be applied to different molecular tasks. During training, the choice of prediction loss function depends on the type of task. For classification loss, we employ the cross-entropy loss, while for regression tasks, utilize the mean squared error loss. Taking the binary classification task example, the cross-entropy loss is computed between the predicted probability )) Prob ZGen) and the

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation performance on GOOD benchmark. denotes abnormal results caused by under-fitting declared the Icaderboard, and denotes that the method cannot he applicd to this dataset. For different distribution shifts, our method obtains superior generalization performance.
GOOD-HIV
1
GOOD-ZING
GOOD-PCBA
Method
scaffold
Size
scaffeld
SIze
scaffold
Size
covariate
concepp
covariate
concepp
covariate
concept
covariate
concept
covariate
covariate
concept
17.11
17.75
IRM
70.17
71.78
59.91
-(-)
0.2161
0.1339
0.6984
0.1336
16.89
22.37
17.68
15.82
VREA
69.34
72.21
58.49
61.21
0.1815
0.1287
0.2270
0.1311
17.10
21.65
17.80
15.85
GroupDRO
68.15
71.48
57.75
59.77
0.1870
0.1323
0.2377
0.1335
16.55
21.91
16.74
15.21
Coral
17.83
DANN
69.43
71.70
62.38
65.15
0.1746
0.1269
0.2326
0.1348
17.20
22,03
17.71
15.78
Mixup
70.65
71.89
59.11
62.80
0.2066
0.1391
0.2531
0.1547
16.52
20.52
17.42
13.71
71 .10
0.2543
0.4578
0.3146
16.33
16.04
70.07
72.51
0.1418
16.45
17:57
GREA
71.98
70.76
60.11
60.96
0.1691
0,1157
0.2100
0.1273
16.28
20.23
17.12
13.82
CAL
69.12
72.49
59.34
56.16
15.87
18.62
16.92
13.01
DisC
58.85
64,82
74,11
MaleOOD
0.1996
12:90
12.64
CIGA
69.40
71.65
61.81
73.62
iMoLD
72.9/
74.32
62.86
77.43
0.1410
0.1014
0.1863
0.1029
17.32
22.58
18.02
18.21
Ours (CFD)
76.42
77.83
64.14
79.28
0.1187
0.0765
0.1421
0.0852
19.78
25.64
19.18
20.03
ground-truth label y:
Spred = Vlog + -2) log(1 10--)) -)),
(9)
where Prob(-) represents function that computes probability. Finally, the learning objective can be defined as the weighted sum of the above losses:
T
Crask Lpred + Lid +
Σ
(10)
t=
where and are hyper-parameters, which are separately set to 1 and 0.01 in the experiments.
4 EXPERIMENTS
In this section, we conduct extensive experiments to answer the research questions. (1) Can method CFD achieve better OOD generalization performance against baselines? (2) Does our method possess the ability to capture important substructures and improve the performance of molecular substructure prediction? (3) How does each component contribute to the final performance?
4.1 EXPERIMENTAL SETUP
Datasets. For OOD molecular representation learning, we follow the settings of the work (Zhuang et al. 2023) and employ two worldworld datasets, i.e., GOOD (Gui al. (2022) that is systematic benchmark tailored specifically for graph OOD problems, and DrugOOD (Ji et al., (2022) that is
OOD benchmark for Al-aided drug discovery. Besides, for molecule's 's ground-state prediction,
follow the settings of the work (Xu et al. (224)) and utilize Molecule3D (Xu et al. (021b) and QM9
(Ramakrishnan al. 2014) to evaluate the ability of our method for focusing on substructures. In Appendix, we will introduce more details about these datasets.
Implementation Details. Our method is used for learning specific molecule representation and could be plugged into existing methods (Zhuang et al. 2023) Xu et al., 2024). The number of the learned concepts is set 12. The number feedback iterations set It is worth noting that during feedback, we do not introduce new parameters.
For OOD molecular representation learning, we mainly compare our method with some common approaches for non non-Euclidean data, i.e., ERM (Vapnik] 2013). IRM (Arjovsky et al. 2019). VREx (Krueger et al. 2021), GroupDRO (Sagawa et al. 2019), Coral (Sun Saenko 2016). DANN (Ganin et al. 2016), Mixup (Zhang 2017). and iMoLD (Zhuang et al. 2023). Meanwhile, our method algorithms, e.g., graph OOD algorithms such as CAL (Sui
Baselines.
is compared with some al. 2022)_ DisC (Fan et al. 2022), MoleOOD (Yang al.) 2022), and CIGA (Chen et al. 2022). as
well as interpretable graph learning methods, e.g., DIR [Wu et al. 2022). GSAT (Miao et al. 2022).
and GREA (Liu et al. 2022).
7

---

Published as a conference paper at ICLR 2025
4.2 ANALYSIS OF OOD MOLECULAR REPRESENTATION LEARNING
To evaluate the generalization ability of our method, our method is evaluated on GOOD benchmark (Gui al. 2022). Table T shows the performance. We observe that some graph-specifie OOD methods could achieve effective performance on some synthetic datasets (e.g., to predict whether a specific motif is present in synthetic graph). However, they do not perform well on molecules with complex data structures and distributions, which shows that our method could apply to molecules with complex structures and improve the generalization performance. Besides, the performance of iMoLD (Zhuang et al. 2023) outperforms MoleOOD (Yang et al. (022), which shows the effectiveness of the strategy thinst-encodingtteen theensspprrttion. However, iMoLD (Zhuang et al. 2023) fails to capture the critical substructures that involve certain invariant characteristics, which may affect its performance. We can see that our method improves the performance significantly. This indicates that the feedback disentanglement could indeed decompose generalized features. Meanwhile, method could be applied to complex molecules. This also shows that the learned concepts are beneficial for capturing important substructures, which further strengthens the generalization ability.
Table 2 further evaluates our method on DrugOOD dataset (Ji et al.) 2022). Par- ticularly, the recent work MILI (Wang]
Table 2: Evaluation performance DrugOOD dataset.
IC50
EC50
Method
al. 2024a) follows -first-epparation- then-encoding" mechanism. And it first
Assay
Scaffold
Size
Assay
Scaffold
Size
ERM
71.63
68.79
67.50
67.39
64.98
65.10
identifies privileged substructure and de- velops dual-head GNN to achieve in- variant representations of substructures. When the molecular substructures are accurately identified, the generalization performance of OOD molecule repre- sentation could be improved effectively. We that method still outper- forms iMoLD (Zhuang et al. 2023) and MILI (Wang al.) 2024a). This demon- strates that using the feedback iteration idea beneficial for extracting general- ized molecular characteristics. By fusing the concepts capturing plentiful substructure content, the
IRM
71.15
67.22
61.58
67.77
63.86
59.19
Coral
71.28
68.36
64.53
72.08
64.83
58.47
MixUp
71.49
68.59
67.79
67.81
65.77
65.77
DIR
69.84
66.33
62.92
65.81
63.76
61.56
GSAT
70.59
66.45
66.70
73.82
64.25
62.65
GREA
70.23
67.02
66.59
74.17
64.50
62.81
CAL
70.09
65.90
66.42
74.54
65.19
61.21
DisC
61.40
62.70
61.43
63.71
60.57
57.38
MoleOOD
71.62
68.58
65.62
72.69
65.74
65.51
CIGA
71.86
69.14
66.92
69.15
67.32
65.65
iMoLD
72.11
68.84
67.92
77.48
67.79
67.09
MILI
72.67
69.58
68.40
77.11
68.07
65.97
Ours (CFD)
73.86
70.02
69.73
78.32
69.13
67.62
generalization ability could be further enhanced.
Table 3: Performance of molecule's ground-state prediction based on Molecule3D and QM9 datasets.
Validation
Test
D-MAEL
D-RMSEJ
C-RMSDJ
D-MAEL
D-RMSEL
C-RMSDJ
(a) Molecule3D Random Split
0,581
1.054
0.582
0.932
RDKit ETKDG
0.575
0.941
0.998
0.576
0.942
0.999
DeeperGCN- -AGNNN Xu al. 2021b
0.509
0.849
0.571
0.961
GINE (Hu ct al. 2019
0.590
1.014
1.116
0.592
1.018
1.116
GATv2 Brody
d Brody 2021
0.563
0.983
1.082
0.564
0.986
1.083
GPS (Rampase k ara
0.528
1.036
0.911
1.038
GTMGC [Xu ctal. 2024
0.432
0.719
0.712
0.433
0.721
0.713
GTMGC Ours
0.397
0.682
0.684
0.407
0.695
0.688
(b) QM9
0.358
0.722
0.615
RDKit ETKDG
0.355
0.621
0.691
0.355
0.621
0.689
GINE (u al. 2019
0.357
0.673
0.685
0.357
0.669
0.693
GATvž Brody Brody al. 2021
0.339
0.663
0.661
0,339
0.659
0.666
GPS Rampasck al 2022
0.326
0.644
0.662
0.326
0.640
0.666
GTMGC (uu cf al. 2024
0.262
0.468
0.362
0.264
0,470
0.367
GTMGC+
0.223
0.434
0.305
0.218
0.442
0.309
4.3 ANALYSIS OF MOLECULE'S GROUND-STATE PREDICTION
The molecular ground-state conformation refers to the lowest energy state of a molecule on its potential energy surface (Xu al. 2024). It belongs to substructures of original molecules, which
8

---

Published as a conference paper at ICLR 2025
H-
H
H
RH
Ground Truth
Prediction
Ground Truth
Prediction
Figure 4: The prediction results of the molecule's ground-state conformation. Compared with ground truth, method could localize the critical substructures, improving the prediction accuracy.
play crucial role determining the physical, chemical, and biological properties of molecules. By evaluating this task, our method is directly plugged into GTMGC (Xu et al. (024) that is transformer architecture. The training processes and optimization objectives are kept unchanged. In Table we can observe that plugging our method could improve GTMGC's prediction performance effectively. This indicates that our method able capture critical substructure characteristics. Meanwhile, our method could be plugged into Transformer architecture, showing our method's s flexibility.
directly observe CFD's ability for capturing critical substructures. Here,
In Fig. we provide two prediction examples from our method. We can observe that compared with ground truth results, our method could accu- rately capture those critical substruc- tures, further demonstrating the effec- tiveness of our method.
Table 4: Analysis of our CFD method. 'Feedback' and 'Concepts' separately indicate feedback disentanglement and molecule con- cept learning. 'Cov-Sold', 'Cet-Sold', 'Cov-Size' and 'Cet-Size' denote scaffold-covariitee, scaffold-concept, size-covariate, and size-concept shifts, respectively.
Feedback Concepts Cov-Sold Cet-Sold Cov-Size Cet-Size
0.1391
0.0923
0.1754
0,0983
0.1685
0.1172
0.2046
0.1268
0.1187
0.0765
0.1421
0.0852
4.4 ABLATION ANALYSIS
To evaluate the effectiveness of each component, we take GOOD-ZINC (Gui et al. 2022) as the dataset make an ablation analysis.
Analysis of our method. As shown in Fig.
Table 5: Analysis of feedback iteration number.
2] our method mainly consists of two com- ponents, i.e., feedback disentanglement and molecule concept mining. Meanwhile, these two modules could be taken as the independent module. In Table we make an ablation exper- iment. 'Concepts' represents that the concept learning module is plugged into ERM [Vapnik] 2013). We can that for different shifts, using feedback disentanglement and learning molecule concepts could improve the performance significantly. This shows that our method could effectively separate generalized features from spurious features. Meanwhile, the mined concepts could capture critical substructure characteristics, further enhancing the disentangled features' generalization.
Iterations
Cov-Sold Cet-Sold Cov-Size Cet-Size
1
0.1378
0.1012
0.1789
0.1021
4
0.1285
0.1008
0.1573
0.0963
8
0.1187
0.0765
0.1421
0.0852
12
0.1191
0.0799
0.1453
0.0844
16
0.1172
0.0805
0.1479
0.0867
Number of feedback iterations. To achieve the expected generalized representation, we employ multiple feedback iterations. In Table 5) we can see that the iteration number affects the generalization performance. For molecules with complex structures, using fewer iterations could not decompose the expected generalized representation effectively, which weakens the performance. Mcanwhile, we observe that the performance change is very small when the iteration number is larger than 12 for the given dataset. For method, the performance of using feedback iterations is the best.
Table Analysis of molecule concept number. Here, we only change the concept number. Other operations and settings are kept unchanged.
Number Cov-Sold Cet-Sold Cov-Size Cet-Size
8
0.1233
0.0892
0.1541
0.0927
12
0.1187
0.0765
0.1421
0.0852
16
0.1201
0.0743
0.1478
0.0874
Number of molecule concepts. As shown in Fig. concepts to focus on critical substructures. In Table we can observe that the concept number affects the performance. Utilizing fewer concepts could not capture sufficient substructure characteristics
explore extracting series of molecule

---

Published as a conference paper at ICLR 2025
effectively. which reduces the performance. For method, when the concept number is set to 12. the generalization performance is the best. Besides, during mining molecule concepts (equation 6 and equation 7 we employ KAN (Liu al. (2024) to improve the concept aceuracy. Here, find that replacing KAN with MIPP results in performance degradation. For example, for Cov-Sold case, the performance is decreased by around 0.6%, indicating KAN's effectiveness.
CONCLUSION
In this paper, we propose new method, i.c., Concept-Enhanced Feedback Disentanglement, for learning generalized molecular representation against distribution shifts. Particularly, by performing multiple feedback iterations, our method could progressively decompose expected features involving rich generalized information. Meanwhile, fusing the concepts that focus on substructures could further strengthen the generalization. Extensive experimental results on multiple datasets and tasks demonstrate that our method could indeed disentangle generalized molecular representations.
ACKNOWLEDGEMENT
This work is supported in part by the National Key R&D Program of China (No. 2023YFC3305660 Joint Fund Ministry of Education of China (8091 880918022149 8091 B02072204) National Natural Science Foundation of China (62472333, 62132016, 62171343), and Natural Science Basic Research Program of Shaanxi (2020IC-23).
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Sean-Christophr Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. NeurIPS, 34:3438-3450, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint 1rxiv:1907.02200 00.0999..299,,.... 2019.
Maria Boulougouri, Pierre Vandergheynst, and Daniel Probst. Molecular set representation learning. Nature Machine Intelligence, 6(7):754-763, 2024.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491, 2021.
Shiyu Chang, Yang Zhang. Mo Yu, and Tommi Jaakkola. Invariant rationalization. In ICML, pp. 1448-1458. PMI.R, 2020.
Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, MA Kaili, Binghui Xie, Tongliang Liu, Bo Han. and James Cheng. Learning causally invariant representations for dutto--distribution generalization on graphs. NewrlPS, 35:22131-22148 2022.
Yongqiang Chen, Yatao Bian, Kaiwen Zhou, Binghui Xie, Bo Han, and James Cheng. Does invariant graph learning via environment augmentation learn invariance? NeurlPS. 36, 2024.
Elliot Creager, Jömn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In /CML, pp. 2189-2200, PMLR, 2021.
Chandra Kiran Reddy Evuru, Sreyan Ghosh, Sonal Kumar, Utkarsh Tyagi, Dinesh Manocha, al. Coda: Constrained generation based data augmentation for low-resource nlp. arXiv preprint arXiv:2404.00411 00415, 2024.
Shaohua Fan, Xiao Wang, Yanhu Mo, Chuan Shi, and Jian Tang. Debiasing graph neural networks via learning disentangled causal substructure. NewrlPS, 25:24934-24944 2022.
Yin Fang, Qiang Zhang, Haihong Yang. Xiang Zhuang, Shumin Deng, Wen Zhang, Ming Qin, Zhuo Chen, Xiaohui Fan. and Huajun Chen. Molecular contrastive learning with chemical element 2022.
knowledge graph. In AAAI, volume 36, pp.
10

---

Published as a conference paper at ICLR 2025
Amir Feder, Yoav Wald, Claudia Shi, Suchi Saria, and David Blei. Causal-structure driven augmenta- tions for text ood generalization. NewrIPS, 36, 2024.
Yaroslav Ganin, Bvgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research, 17(59)11-35, 2016.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, pp. 1263 1272. PMLR, 2017.
Rafael Gómmez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lood Benjamín Mannchez-Lnggling.. Dennis Sheberla, Jorge sailllera--prraguireee Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik, Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268-277) 2018.
Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph intrrf-distribution benchmark. NeurIPS, 35:2059-2075, 2022.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. NeurlPS, 30, 2017.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprini arXiv: 021.19955........ 2019.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. NeurlPS, 32, 2019.
Yuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bingzhe Wu, Long-Kai Huang, Tingyang Xu, Yu Rong, Lanqing Li, Jie Ren, Ding Xue, et al. Drugood: Out-of-distribution (ood) dataset curator and benchmark for ai-aided drug discovery-a focus on affinity prediction problems with noise annotations. arXiv preprint arXiv:2201 Xiv:2201.006337, 2022.
Mostafa Karimi, Di Wu, Zhangyang Wang, and Yang Shen. Deepaffinity: interpretable deep learning of compound protein affinity through unified recurrent and convolutional neural networks. Bioinformatics, 35(18):3329-3338 2019.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks.
ICLR, 2017.
Andreï Nikolaevich Kolmogorov. On the representation of continuous fanctions of several variables by superpositions of continuuu functions of smaller number of variables. American Mathematical Society, 1961.
David Krueger, Ethan Caballero, Joern-Henrk Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. disttr-iisttibttionn generalization via risk extrapola- tion (rex). In ICML., pp. 5815-5826. PMLR, 2021.
Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Dongsheng Li, Kurt Keutzer, and Han Zhao. Invariant information bottleneck for domain generalization. In AAAI, volume 36, pp. 7399-7407, 2022a.
Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu, Learning invariant graph representations for out-of-distribution generalization. NeurIPS, 35:1 18:188-81181 2022b.
Jaechang Lim, Scongok Ryu, Kyubyong Park, Yo Joong Choe, Jiyeon Ham, and Woo Youn Kim.
Predicting darget interaction using novel graph neural network with 3d stracture-embedded graph representation. Journal of chemical information and modeling, (919)3391-39988 2019.
Gang Liu, Tong Zhao, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Graph rationalization with environment-basel augmentations. In KDD, pp. 1069-1078, 2022.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačié, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-arnol networks. arXiv preprint arXiv:2404. 19756, 2024.
11

---

Published as a conference paper at ICLR 2025
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. NeurlPS, 33:11525-11538 2020.
Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, Liu, Causality inspired representation learning for domain generalization. In CVPR, pp. 8046-8056, 2022.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In ICML, pp. 7313-7324, PMLR, 2021.
David Mendez, Anna Gaulton, Patricia Bento, Jon Chambers, Marleen De Veij, Eloy Félix, María Paula Magariñios, Juan F Mosquera, Prudence Mutowo, Michal Nowotka, et al. Chembl: towards direct deposition of bioassay data. Nucleic acids research, 47(D1):D930-DD 29400009900,.-- 2019.
Siqi Miao, Mia Liu, and Pan Li. Interpretable and generalizable graph learning via stochastic attention mechanism. In ICML, pp. 15524-15543. PMLR, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 180.0338,, 2018.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp. and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):-7,, 2014.
Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Do- minique Beaini. Recipe for a general, powerful, scalable graph transformer. NeurIPS, 35: 2022.
14501-14515, 2022.
Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learn- ing to compose domain-specific transformations for data augmentation. NeurIPS, 30, 2017.
Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. NeurlPS, 34:20210-20229, 2021.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. NeurlPS, 33:12559-12571, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: /9/ /,08731, 2019.
Ana
Elisabeth Rumetshofer, Sepp Hochreiter, and Günter Klambauer. Con- trastive learning of image-and structure-based representations in drug discovery. In ICLR2022 Machine Learning for Drug Discovery, 2022.
Jonathan M Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M
Donghia, Craig R MacNair, Shawn French, Lindsey Carfrae, Zohar Bloom-Ackermann, et al. A deep learning approach to antibiotic discovery. Cell, 180(4):688-702, 2020.
Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan Hc, and Tat-Seng Chua. Causal attention for interpretable and generalizable graph classification. In KDD, pp. 1696-1705, 2022.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16. 2016, Proceedings, Part III 14, pp. 443-450. Springer, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 30, 2017.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yigiang Chen, Wenjun Zeng, and S Yu Philip. Generalizing to unseen domains: A survey on domain generalization. IEEE fransactions on knowledge and data engineering, 805-)88052-802, 2022a.
12

---

Published as a conference paper at ICLR 2025
Ruijia Wang. Haoran Dai, Cheng Yang, Le Song, and Chuan Shi. Advancing molecule invariant representation via privileged substructure identification. In KDD, pp. 3188-3199, 2024a.
Xinyi Wang, Michael Saxon, Jiachen Li, Hongyang Zhang, Kun Zhang, and William Yang Wang. Causal balancing for domain generalization. arXiv preprint arXiv:2206.0522 2022b.
Yili Wang. Kaixiong Zhou, Ninghao Liu, Ying Wang, and Xin Wang. Efficient sharpness-aware minimization for molecular graph transformer models. ICLR, 2024b.
Gerhard Widmer and Miroslav Kubat. Learning in the presence of concept drift and hidden contexts. Machine learning, 23:69-101, 1996.
Aming Wu, Yahong Han, Linchao Zhu, and Yi Yang. Instance-invariant domain adaptive object detection via progressive disentanglement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4178-411 2021.
Ying-Xiu Xin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. ICLR, 2022.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9()):513-530, 2018.
Guikun Xu, Yongquan Jiang, PengChuan Lei, Yan Yang, and Jim Chen. Gimge: Using graph transformer to predict molecule's ground-state conformation. In ICLR, 2024.
Qinwei Xu, Ruipeng Zhang. Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based framework for domain generalization. In CVPR, pp. 14383-14392, 2021a.
Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3d: A benchmark for predicting 3d geometries from molecular graphs. arXiv preprint arXiv: 10.01717, 2021b.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. International Journal of Computer Vision, pp. 1-28, 2024.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59(8): 3370-3388, 2019.
Nianzu Yang, Kaipeng Zeng, Qitian WU, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for out-of-distributt molecular representations. NeurlPS, 35:12964-1299 2022.
Amir Zamir, Te-Lin Wu, Lin Sun, William Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese. Feedback networks. In CVPR, pp. 1308-1317, 2017.
Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P Xing. Towards principled disentanglement for domain generalization. CVPR, pp. 8024-8034, 2022.
Hongyi Zhang, mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 2017.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep adwain-adversarial image generation for domain generalisation. In AAAI, volume 34, pp. 13025-13032, 2020.
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint arXiv:2 (0f.::22044000000, 2021.
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Mixstyle neural networks for domain generalization and adaptation. International Journal of Computer Vision, 132(3):822-86,, 2024.
Xiang Zhuang, Qiang Zhang, Keyan Ding, Yatao Bian, Xiao Wang, Jingsong Lv, Hongyang Chen, and Huajun Chen. Learning invariant molecular representation in latent discrete space. NeurIPS, 36.78435-78452, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
To learn generalized molecular representation, this paper proposes a new method, i.c. Concept- Enhanced Feedback Disentanglement, aiming feedback decomposition and concepts involving critical substructures to improve the generalization of the learned representation. In the appendix, we provide more experimental details, additional analyses, and more prediction results.
B DATASETS AND METRICS
Datasets. For molecular representation learning, we employ two benchmarks, i.e. GOOD Gui jtt al. 2022) and DrugOOD (Ji et al., 2022). Particularly, we follow the work (Zhuang et al. 2023
to utilize three molecular datasets for the prediction task: (1) GOOD-HIV (Wu et al. 2018), where the purpose is binary classification to predict whether a molecule can inhibit HIV; (2) GOOD- et al. 2018), which is regression dataset aimed at predicting molecular solubility; and (3) GOOD-FCBA (Wu et al., 2018), which includes 128 bioassays and forms 128 binary classification tasks. Each dataset contains size), and two shift types (covariate and concept) are applied per splitting outcome, resulting in a total of 12 distinct datasets (as shown in Table [ Furthermore, DrugOOD (Ji et al., 2022) provides three env environment--pliting strategies, including assay, scaffold, and size, and applies these three splitting two measurements (IC50 and EC50). As result, we obtain 6 datasets (as shown Table 2). and each dataset contains a binary classification task for drug target binding affinity prediction
ZINC G-ómez-Bombarce
mnvironment-splittinn strategies (scaffold and
For molecule's 's ground-state prediction, we follow the work (Xu et al., 2024) utilize Molecule3D (Xu et al., 2021b) and QM9 (Wu et al. 2018) as benchmarks. Particularly, Molecule3D aims to use DNNs to predict the ground-state 3D geometries of molecules based solely on their 2D graph structure. And this large-scale dataset comprises approximately 4 million molecules, each with its own 2D molecular graph, ground-state 3D geometric structure, and four additional quantum properties. In this paper, we employ the random splitting according to the same distribution based on the molecule's core component. Besides, QM9 is a quantum chemistry dataset that provides geometry, energy, electronic, and thermodynamic properties for nearly 130,000 organic molecules with 9 heavy atoms, containing molecular most stable conformation.
Metrics. For molecular representation learning, report the ROC-AUC score for GOOD-HIV and DrugOOD datasets as the task is binary classification. For GOOD-ZINC, we use the Mean Average Error (MAB) since the task is regression. While for GOOD-PCBA, we use Average Precision (AP)
averaged over all tasks as the evaluation metric due to extremely imbalanced classes.
For molecule's ground-state prediction, follow the work (Xu et al. 2024) utilize three metrics, i.e,, D-MAF, D-RMSE, and C-RMSD. Concretely, given a dataset with N interatomic distances, the Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) between the prediction {d} J and the ground truth [d. J. used to evaluate the performance node-pair level:
N
D - MAE([d] (d
d d:l,
(11)
N
N (d: ddd-d))².
D - RMSE((
(12)
Additionally, C-RMSD between the ground truth G and the prediction G" is computed as follows:
1 9g -9,1
C - T-RMSD(G,C'
(13)
C THE CAUSE OF MOLECULAR DISTRIBUTION SHIFT
For molecule data, there exist many kinds distribution shifts. In this paper, we investigate three types of shift strategies, i.e., scaffold, size, and assay. Here, give more explanations.
14

---

Published as a conference paper at ICLR 2025
Ground Truth
Our Prediction
Ground Truth
Our Prediction
Figure 5: More prediction results of molecule's ' ground-state -taute conformation. We can observe that our method could accurately localize critical ground-state parts that determine the physical, chemical, and biological properties of the molecule in most cases, which demonstrates the effectiveness of our method for learning generalized molccule representation and capturing substructure characteristics.
(1) Scaffold Environment:
molecular scaffold the fundamental structure of molecule with desirable bioactive properties. Molecules with the same scaffold belong to the same environment. Distribution shift arises when there a change in the molecular scaffold.
(2) Size Environment: the size of molecule refers to the total number of atoms in the molecule. Molecular size is also an inherent structural characteristics of molecular graphs. The distribution shift occurs when size changes.
(3) Assay Environment:
assay is an experimental method as an examination or determination for molecular characteristics. Due variations in assay environments and targets, activity values measured by different assays often differ significantly. Samples tested within the same assay belong single environment, while change the assay leads distribution shift.
D MORE ABLATION EXPERIMENTS OF HYPER-PARAMETER
In equation 10 we utilize two hyper-panne ppper-parameters, i.e., 0n and az to balance the two loss terms, i.e., La and Led- Here, based on GOOD-ZING with Covariate-Scaffol shift, we make an ablation analysis. We only change the settings of hyper-parameters.
Analysis of hyper- parper-prrameter feedback disentangled process to decompose generalized representation effectively. Thus, the setting of hyper-parameter aı is important. Here, make experiment. When a is separately set to 0.5, 0.1, and 0.05, the corresponding performance is 0.1251, 0.1187, and 0.1299.
In equation 10 the goal of the loss term La is to encourage the
Analysis of hyper-pann parametrreeerrreeeee (22 In equation 10 the loss term Ct is to promote the decomposed features to involve plentiful invariant characteristies related to the input molecule while enlarging the estimated distribution gap between the generalized and spurious features. Thus, the parameer ag is still an important paree--parreter. Here, we make an experiment. When a2 is separately set to 0.05, 0.01, and 0.005, the performance is 1248, 0.1787, and 0. 1225.
15

---

Published as a conference paper at ICLR 2025
Algorithm 1 Training Process of Concept-Enhancee Feedback Disentanglement
Input:
Molecule Dataset {G, }. randomly initialized GNN, randomly initialized spurious encoder and generalized encoder, randomly initialized concept learning module, weight aj for the loss La.. weight 0g for the loss L Output: Trained parameters.
while train do
Sample molecules from the given dataset {G, Y). Utilize a GNN to extract the representation F. for 0 do
Qo =
Use equation to estimate corresponding means (i.e., Mo Spu and μu Gen) and variances (i.e., Spm and ofe")) for spurious and generalized features.
To
Perform sampling and encoding operations to obtain 2, and 2Gen based on equation 4 Learn molecule concepts C' and calculate the generalized representation 2Gen based on
equation 6 and equation] 7
end
for
f do
Q-= = -
Use equation to estimate corresponding means (i.e., Spu and μ (Gen) and variances (i.e., 01 Spp and " for spurious and generalized features.
Perform sampling and encoding operations to obtain ", and Z based on equation 4 Learn molecule concepts C' and calculate the generalized representation z based on equation 6 and equation] end
for
1 =2 do Q: ( 7[ea))-
Use equation 3 to estimate corresponding means (i.e., Spu and (Ge) and variances (i.e., -Spu and Gen) for spurious and generalized features.
Perform sampling and encoding operations to obtain Z,5pu and Z,Gen based on equation 4 Learn molecule concepts C' and calculate the generalized representation 7Gen based on equation and equation 7
end
Calculate the overall training objective Ltask using equation 5 equation equation
Update the trained parameters based on equation 10
end
Finally, Algorithm| [ shows the training processes our method. For step >= 2, the feedback from the outputs of the previous two steps. By means of feedback disentanglement, could decompose generalized molecular representation effectively, which improves the performance of downstream tasks. In Fig. 5 show more prediction results of molecule's ground-state conformation. We can see that our method could localize critical ground-state molecular substructures, which further demonstrates the superiorities of our method.
method
E MORE CHEMICAL INSPIRATION OF OUR METHOD
This method is mainly based on the following chemical observations:
(1) A chemical molecule is complex composed multiple atoms and bonds. The reason for the OOD problem is the structural change of molecules. For example, changes in molecular chirality can lead to deviations molecular properties. Thus, learning robust molecule representation is important for molecule- based tasks, e.g., drug discovery.
(2) Lines 082-095 have indicated that existing methods could classified into two types, i.e., First- Separation-
and First-Encoding" Encoding-Theen-Se Then-Separation (as shown in Fig. (a) and (b)). As indicated in Lines 272-283 and Fig. 3, these methods for learning generalized molecular repre- sentation generally employ one-step disentangled strategy, i.e., directly separating the input into
16

---

Published as a conference paper at ICLR 2025
generalized and spurious parts. However, in practice, may encounter unknown biomacro- molecules containing more atoms and highly complex structures, c.g., entanglement of multiple substructures. At this time, using the original one-step mechanism could not obtain satisfactory disentangled results. To this end, we exploit the feedback idea to iteratively and progressively separate generalized molecular representation.
(3) In general, the molecule's ground-state conformation belongs to the substructures of molecules and determines their properties. Therefore, it important to capture substructure-related characteristics. To this end, we define a series of concepts to build connections between molecule substructures and corresponding concepts, which further improves the performance.
F MORE EXPERIMENTS TO EVALUATE INVARIANCE
Table 7: To further demonstrate the ability of disentangling invariance, we further evaluate our method on Two-Piece Graph Datasets Chen (2024).
Datascts
f0.8, 0.9)
(0.7, 0.9}
Avg.
Fan
(022
45.06
39.42
42.24
CIGA
C hhnn ct al 2022
57.87
43.62
50.75
GALA hen et al 2024
76.42
72.50
74.46
Ours
78.26
74.32
76.29
We follow the work Chen et al. (2024) and further evaluate our method on Two-Piece Graph Datasets Chen et al. 2024). Particularly, each dataset is generated from variation of two-piece graph model, denoted as a, bj. where a refers to the invariant correlation strength and b refers to the spurious
correlation strength. The results are shown in Table 7
We can observe that our method indeed improves the performance. Particularly, when the invariant correlation strength is weaker than the spurious correlation strength, the performance of our method outperforms state-of oitte-of-th-arr methods, further demonstrating that our method is indeed beneficial for disentangling invariant characteristics.
G COMPARISON BETWEEN RVQ AND MOLECULE CONCEPT LEARNING
Our molecule concept learning module is significantly different from Residual Vector Quantization (RVQ) Zhuang et al. (2023). In essence, RVQ is to perform feature replacement. Our method is to build the connection capture critical molecule substructures:
(1) The motivation is different. RVQ aims to leverage the discretized continuous representation to improve the model generalization. As shown in Fig. 6 our method is learn series of concepts that could capture critical molecule structure information involving plentiful task-specific invariant molecule characteristics.
(2) The operations are different. RVQ first introduces a shared learnable codebook as discrete latent space. For each node representation in molecular graph, RVQ looks up and fetches the nearest neighbor in the codebook and outputs it as the result. Meanwhile, a sum operation is further used to strengthen the representation ability. During training, similar to VQ-VAE! Van Den Oord et al. (2017) RVQ employs the exponential moving average updates for the codebook.
Van
Differently, our concepts are first sampled from a learnable Gaussian distribution. And we employ KAN operation to promote the learning of concepts. Importantly, as shown in equation the weighted residual operation is to align critical molecule substructure features to the corresponding concept, building the connection between molecule substructures and corresponding concepts. During training, we only utilize the task loss, c.g., molecule property prediction, to promote the learned concepts to capture plentiful task-specific invariant molecule characteristics.
(3) Experimental results show the effectiveness our method. Taking GOOD-ZIN -INC as the example, for iMoLD Zhuang et al. (2023), first remove the RVQ operation and keep other operations unchanged. We observe that removing RVQ does not affect the performance. Instead, our concept mining module could be plugged into iMol.D further improve its performance. For the
17

---

Published as a conference paper at ICLR 2025
HO
NH2
CH3
COOH
HO
Molecule Concepts
CHs
C
CHa
HO
HaC
OH
CH3
HgN
Figure 6: Molecule concepts (colorful stars) are learned from all molecule structures, which are not specific to certain molecule structures. By means the residual operation (as shown solid arrows), we can build the connection between molecule substructures and corresponding concepts, which is beneficial for aligning critical molecule substructure features to the corresponding concept.
covariate case of scaffold, the performance is improved by 0.0142. Furthermore, the prediction results of the molecule's ground-state conformation in Fig. aaddd also show that our learned concepts beneficial for capturing critical substructures.
H FURTHER ANALYSIS OF THE LOSS FUNCTION
equation 10 is the optimization objective, consisting of three terms, i.e., Lpreds L (as shown in equation and L'd (as shown in equation 5
Proposition 1. Ctd mainly is a contrastive loss. Minimizing Ced could enlarge the gap between the disentangled 2,Gen and Z Spm Mcanwhile, 2,Gen is promoted to retain plentiful input-related content.
Analysis. The first term of C is related to the InfoNCE loss Oord al. (2018). Concretely, we set B1 = exp(sim(Z Z ,)) and B2 exp(sim( S F))):
Ellog Bi + B2
= E logB,| [B + BB)]
(14)
< 1[ B[log[3]]
Obviously, maximizing equation 14 narrows the semantic distance between 7GGen and the input F and enlarges the semantic gap between Z,5pz and F. Equally, this operation enlarges the gap between ZGen and SpSp which beneficial for strengthening the disentangled ability.
Proposition 2. Minimizing Lya is instrumental in achieving the invariant representation Zenn
Analysis. In equation and ΣT. sim Gen
Lja mainly consists of two terms, i.e., Σ D sim zgen )ist)
Z
Spa):
T
Σ
sim ZGGen Z
Dizj
Σ sim (ZGen 2 7.Spu
(15)
i= 0=
1=0
Maximizing equation 15 first enlarges the gap between Z,Gen and 205pu the i-th iteration, which beneficial for enhancing the disentangled ability. Secondly, Z,Gen and ZGen separately represent disentangled generalized outputs at different iterations. Here, we assume that when taking a represen- tation involving plentiful invariant information as the input, the disentangled output should contain less spurious information, i.c., the output is equal to the input. Therefore, narrowing the semantic distance is conducive to learning invariant representations across various environments.
18

---

Published as a conference paper at ICLR 2025
H
Figure Visualization examples of molecule concepts. Here, we directly project the captured concepts into the predicted molecules. The green regions represent the focused content.
I EXAMINING IMPORTANCE OF CONCEPTS ACROSS ITERATIONS
Table 5 analyzes the impact of iteration number. We observe that when the iteration number increases, inserting concepts could improve the performance effectively, which indicates the effec- tiveness of the learned concepts. To further analyze the impact of iteration number on concepts, also make a new ablation experiment, i.e., removing the concepts at the 4th, and 8th iteration. The results are shown as follows:
Table 8: Examining importance of concepts across iterations.
Iterations
Concepts
CovSold
CetSold
CovSize
CetSize
0.1285
0.1008
0.1573
0.0963
0.1402
0.1117
0.1738
0.1102
8
0.1187
0.0765
0.1421
0.0852
8
0.1189
0.0783
0.1449
0.0897
We observe that when the iteration number is small, inserting concepts could significantly affect the performance. Instead, when the iteration number large, the impact of concepts is somewhat weak. The reason may be that when the iteration number is small, leveraging concepts is beneficial for capturing critical substructures, improving the performance. As the iteration number increases, concepts progressively pay attention meaningful substructure characteristics, reducing the iteration impact on performance.
MORE ANALYSIS OF MOLECULE CONCEPTS
We perform an ablation experiment of the decoder that calculates task-related representation. We observe that for classification and regression tasks, using decoders with more layers leads to the performance degradation. The reason may be that this operation introduces more parameters, resulting the overfitting risk.
In Fig. we show some visualization examples of molecule concepts. Since these concepts are learned from all molecule data, they are not specific to certain kinds of molecules, which beneficial for improving the generalization of molecule concepts. For different predictions, can observe that the learned concepts could indeed focus on critical molecule substructures, which strengthens the performance of molecule prediction.
K ANALYSIS OF COMPUTATIONAL COSTS
We observe that introducing feedback iterations indeed increases computational costs. During training, based on the same batch size, the memory is increased by around 0.9GB, However, Fig. 8 we observe that the convergence speed is significantly faster than the baseline ZZuang al. (2033)
During inference, compared with one-step disentanglement, the inference time of using 8 iterations is increased by around 0.06s. Although the inference time is longer, the disentangled invariant representations are more accurate, which is instrumental in improving the generalization ability. Particularly, for molecules with multiple atoms and various structures, observe that employing
19

---

Published as a conference paper at ICLR 2025
Ours Base
1.6
1.4
1.2
1.0
0.8
0.6
0.4
25
50
75
100
125
150
175
200
Epoch
Figure 8: The training loss of the baseline Zhuang et al.
2023)
and our method.
our method could significantly improve the performance of downstream tasks, further demonstrating the effectiveness of our method.
L QUANTIFY CONTRIBUTION OF FEEDBACK MECHANISM
To quantify the contribution of feedback iterations, make ablation experiment of the iteration number. Here, we only change the iteration number. Other operations are kept unchanged. The results are shown in Table 3
Table Quantify contribution of feedback mechanism on highly complex molecules.
16
0.3014 0.1867
0.1341
0.1279
0.54
We can observe that compared with one-step disentanglement, using multiple feedback iterations indeed improves the performance, indicating that our method is beneficial for decomposing invariant molecule representations. Besides, also observe that when the iteration number is larger than 12, the performance improvement is limited. The reason that for the current dataset, using iterations sufficient for disentangling highly complex molecules.
Finally, we observe that complex molecules usually contain more atoms. Therefore, to further demonstrate the effectiveness of our method highly complex molecules, select those molecules with more than 12 atoms for additional evaluation. The results are shown in Table
We that for complex molecules, using one-step disentanglement could decompose invariant characteristics effectively. Using multiple feedback iterations could indeed decouple the representations involving plentiful invariant information, which strengthens the generalization ability.
M PROTEIN-LIGAND WONDING-AFFINII PREDICTION
For OOD molecular representation lcarning, in Table I and 2 show the corresponding results. Furthermore, in Table 3 our method is evaluated on an additional task, i.c., molecule's ground-state prediction. These results demonstrate that method could effectively disentangle invariant molecule characteristics, which strengthens the generalization ability.
Finally, further demonstrate the superiorities of our method, evaluate method protein- ligand aanding-affinity prediction. Binding affinity is an important metric in biology and medicinal chemistry that measures the strength of reversible association between biological macromolecules,
20

---

Published as a conference paper at ICLR 2025
such proteins or DNA, and molellle ligands, such drugs Boulougouri al. (2024). Here,
we follow the work Boulougouri et al. (2024) and employ the same datasets and metrics to verify our feedback disentanglement method. The results are shown in Table I0
Table 10: Analysis of protein-ligamt binding Lunding-affinity affinity prediction.
Method
RMSE
MAE
R
MAT
1.457
1.154
0.747
DimeNet
1.453
1.138
0.752
CMPNN
1,408
1.117
0.765
SR-BU BR-BIND Boulougouri et al. 2024)
1.383
1.122
0.780
CFD
1.356
1.073
0.794
Here, MAT, DimeNet, CMPNN are three GNN-based methods. RMSE, MAE, and R are three pre- defined metrics. We can observe that for the challenging protein-ligand binding-affinity prediction, employing our method still improves the performance for the given three metries. This further demonstrates that for biological macromolecules, using feedback disentanglement is meaningful, which improves the generalization of the learned molecule representations.
N MORE INTERPRETATIONS OF KAN
In general, molecules consist of multiple atoms and bonds. For certain biomacromolcculess, their structures are entanglement of various substructures, which poses the challenge for learning molecule concepts that focus on critical substructures.
Inspired by the Kolmogorov-Amole representation theorem Kolmogorov (1961). KAN allows learn- ing custom activations of the network. In this way, possible to analyze the contribution of individual components of the input, then providing a more transparent view of the network's 's decision- making process.
Thus, using KAN could sufficiently convert the analyzed components to the corresponding concepts that involve plentiful molecule-aware substructure information, which beneficial for strengthening the generalization of molecule representations.
21


Reviews:
[{"review_id": "vH546OOTFp", "reviewer": "Reviewer_VpCx", "summary": "The paper, titled \"CFD: Learning Generalized Molecular Representation via Concept-Enhanced Feedback Disentanglement,\" addresses the challenge of out-of-distribution (OOD) generalization in molecular representation learning. The proposed approach, called Concept-Enhanced Feedback Disentanglement (CFD), aims to enhance the robustness of molecular representations against distribution shifts. CFD incorporates a novel feedback mechanism and a concept mining module to disentangle molecular representations into distribution-agnostic generalized features and spurious features.\n\nThe method uses two dedicated variational encoders: one to extract invariant (distribution-agnostic) features and the other for spurious features. A set of molecule-aware concepts is introduced to capture critical substructures. The iterative feedback mechanism allows multiple stages of representation disentanglement, progressively refining the generalized features over iterations. The approach also utilizes a self-supervised objective to further enhance the disentangling process, making it effective in capturing important molecular substructures.\n\nExtensive experiments conducted on multiple real-world datasets, including GOOD and DrugOOD, demonstrate the proposed method’s superior OOD generalization performance compared to state-of-the-art baselines. CFD shows significant improvements, particularly in tasks involving distribution shifts related to scaffold and molecule size, as well as in predicting molecular ground-state conformations. The paper positions its contribution as a robust solution to learn generalized molecular representations that perform well under diverse distribution conditions.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed Concept-Enhanced Feedback Disentanglement (CFD) introduces an iterative feedback mechanism that allows for progressively refining the disentangled representations across multiple iterations. This is a novel application within the molecular representation domain and effectively addresses a significant challenge in learning generalized representations—namely, how to handle complex structures and high variability in molecular data. This feedback mechanism enables the model to overcome the limitations of traditional one-step disentanglement methods, which often fail for biomacromolecules or other complex molecular structures.\n\nThe integration of concept mining to identify invariant substructures is a key strength. By incorporating domain-relevant substructures into the generalized features, CFD not only enhances the model's generalization ability but also improves interpretability. This combination of disentanglement and concept-aware enhancement represents a thoughtful fusion of representation learning techniques, where each step complements the other to strengthen the robustness of the final representation.", "weaknesses": "There is insufficient analysis to quantify the specific contribution of this feedback mechanism compared to a one-step or non-feedback approach. Although there are qualitative claims about the benefits of the iterative process, including an ablation study that explicitly compares different iteration counts or contrasts the feedback approach against standard one-step disentanglement would be crucial to substantiate the claims regarding its effectiveness. Such analysis would provide a clearer understanding of when and how the feedback mechanism truly enhances generalization, particularly in the presence of highly complex molecular structures.\n\nThe paper does not explore how well the learned representations generalize to domains outside of those used for training. Given the emphasis on out-of-distribution generalization, it would be beneficial to evaluate CFD's effectiveness across other molecular tasks beyond those in the current dataset, such as predicting drug toxicity or protein-ligand binding affinity, which require different structural knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LEawHnQ7Dq", "reviewer": "Reviewer_x8vs", "summary": "This paper proposes a concept-enhanced disentanglement feedback mechanism for molecular representation learning. The authors innovatively introduce the feedback mechanism to learn distribution-agnostic representation and validate the method’s effectiveness through comprehensive experimental evaluations.", "questions": "1. How is the environment $e$ in Eq. 1 obtained? Does it require spurious labels?\n2. Does “concept” represent the environment  in Eq. 1? Are $f(·)$ in Eq. 1 and $\\varphi(·)$ in Line 305 the same?\n3. In Line 251-252: “we explore mining a series of concepts that do not specialize to one particular type or class of molecule.” What does that mean?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Extensive evaluation across multiple datasets demonstrates the method’s robustness and generalizability.\n2. The method achieves strong results on both classification and regression tasks, indicating its versatility.", "weaknesses": "1. Missing details: Some critical details are insufficiently explained. For instance, how are $\\mu_c$ and $\\sigma_c$ of the concept learned? What is the motivation for the residual term? How does the residual operation assigns molecular substructure features to the corresponding concept. Also, the invariance mentioned in Eq.1 lacks further explanation of the association with the methodology herein.\n2. Limited in-depth analysis: The effectiveness of KAN needs further comparison with other methods. It's essential to include ablation studies to explore the sensitivity of the proposed method to variations in this hyper-parameter.\n3. High computational complexity: In the feedback disentanglement iterations, each step treats disentangled features as independent feature. However, there may be overlaps among these features. And concatenating the disentangled features from each iteration step can lead to dimensional growth, introducing high redundancy.\n4. Unclear motivation: The rationale behind the disentanglement and subsequent concatenation approach is not fully substantiated. It remains unclear whether this method effectively demonstrates decoupling, as no concrete evidence is provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BCLtnifbVd", "reviewer": "Reviewer_6Fd3", "summary": "(1) The article presents an iterative model based on an encoding-then-separation scheme, designed to be robust against distribution shifts.\n\n(2) The model employs a concept-mining module that focuses on critical substructures with invariant characteristics, aiming for superior performance on out-of-distribution (OOD) data.\n\n(3)The authors conducted experiments on a diverse set of real-world molecular datasets, demonstrating significant performance gains over multiple baselines. Additionally, they evaluated the model on an unseen task, proving its effectiveness.\n\nAlthough there are some curiosities about the algorithm, overall it seems to be novel and effective for representation learning for molecules.", "questions": "(1) In the contrastive loss, the first term aims to maximize the similarity between the latent representations of the general encoder and the GNN encoder initially. However, given that there is no direct link between the GNN latent space and downstream tasks, it is unclear how the authors ensure that this high similarity will make the general embedding latent useful in the end.\n\n(2) Why is KAN used instead of conventional deep learning models? The article states:\n\n\"Unlike MLPs, which have fixed activation functions on neurons, KANs feature learnable activation functions (Liu et al., 2024), which enhances the flexibility of the learned concepts.\"\n\nBut what is the specific rationale behind this choice?\n\n(3) The molecular concept is a critical factor in the algorithm. Are there experiments, such as visualizations, that examine the importance of this concept across iterations?\n\n(4) Is there any analysis involving a decoder to determine if the molecular concept aligns with known scaffolds for the input molecules?\n\n(5) The algorithm appears rather complex. Is there any analysis of the computational costs for training and inference?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "(1) The method is novel and has demonstrated superiority over conventional models.\n\n(2) The method also performs well on out-of-distribution (OOD) data prediction tasks.", "weaknesses": "(1) The algorithm appears somewhat complicated. --> Question (2)\n\n(2) It is unclear how the contrastive loss contributes to the usefulness of the general latent representations, as it only enforces similarity between the GNN latent vector and the general latent vectors. --> Question (1), (3) and (4)\n\n(3) The model is complex and iterative, suggesting it may consume a significant amount of computational power. However, no analysis of this aspect has been conducted. --> Question (5)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4rfCpanzsh", "reviewer": "Reviewer_a9Jv", "summary": "This article introduces a \"First-Encoding-Then-Separation\" paradigm-based framework for out-of-distribution (OOD) molecular representation learning. The authors have designed an iterative disentanglement module that utilizes a feedback mechanism to gradually decompose molecular representations into spurious features and invariant features. The effectiveness of the proposed framework has been validated by the authors across multiple datasets and various tasks.", "questions": "1. Could the authors elaborate on the distinction between the Concept Mining presented in this work and the Residual Vector Quantization found in iMoLD[1]? The Concept Mining in this work appears to be simply a weighted sum of the vectors from the codebook of iMoLD.\n2. Could the authors supplement the manuscript with a proof analogous to the proofs of Theorem 1 and Theorem 2 in MoleOOD[2], demonstrating that minimizing the loss function presented in this article can help the model acquire invariant features capable of providing sufficient predictive power across various environments?\n3. Could the author clarify why the baselines CIGA, DisC, and CAL fail to adapt on the datasets GOOD-ZINC and GOOD-PCBA?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The writing is clear, and the paper is easy to follow.\n- The experimental results are very comprehensive, with ample ablation studies demonstrating the effectiveness of each module. There is also an exploration of the impact of various hyperparameters on the model.", "weaknesses": "- The article lacks chemical interpretability or inspiration. The methodology presented could be generalized to any graph-based out-of-distribution generalization problem, but the authors have limited the scope of the paper to molecular representation learning.\n- One of the core contributions of this article, concept mining, shows a high degree of similarity to the idea in another paper. See questions for details.\n- The article lacks theoretical support. See questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)