Improving Self-supervised Molecular Representation Learning using Persistent Homology
Authors: Yuankai Luo
Lei Shi
Veronika Thost
Venue: NeurIPS 2023 poster (2023)
Keywords: Graph Neural Networks
Molecular Representation Learning
Persistent Homology
Contrastive Learning
Self-supervised Learning
TL;DR: We propose novel approaches for self-supervised molecular representation learning based on persistent homology and thoroughly demonstrate their particular advantages.

Abstract: Self-supervised learning (SSL) has great potential for molecular representation learning given the complexity of molecular graphs, the large amounts of unlabelled data available, the considerable cost of obtaining labels experimentally, and the hence often only small training datasets. The importance of the topic is reflected in the variety of paradigms and architectures that have been investigated recently, most focus on designing views for contrastive learning.
In this paper, we study SSL based on persistent homology (PH), a mathematical tool for modeling topological features of data that persist across multiple scales. It has several unique features which particularly suit SSL, naturally offering: different views of the data, stability in terms of distance preservation, and the opportunity to flexibly incorporate domain knowledge.
We (1) investigate an autoencoder, which shows the general representational power of PH, and (2) propose a contrastive loss that complements existing approaches. 
We rigorously evaluate our approach for molecular property prediction and demonstrate its particular features in improving the embedding space:
after SSL, the representations are better and offer considerably more predictive power than the baselines over different probing tasks; our loss 
increases baseline performance, sometimes largely; and we often obtain substantial improvements over very small datasets, a common scenario in practice.

Improving Self-supervised Molecular Representation Learning using Persistent Homology
Yuankai Luo
Lei Shi
Beihang University luoyk@buaa edu. en
Beihang University leishi@buas edu, cn
Veronika Thost MIT-IBM Watson Al Lab, IBM Research veronika. thost@ibm.com
Abstract
Self-supervised learning (SSL) has great potential for molecular representation learning given the complexity of molecular graphs, the large amounts of unlabelled data available, the considerable cost of obtaining labels experimentally, and the hence often only small training datasets. The importance of the topie reflected in the variety of paradigms and architectures that have been investigated recently, most focus on designing views for contrastive learning. In this paper, we study SSL based persistent homology (PH), mathematical tool for modeling topo- logical features of data that persist across multiple scales. has several unique features which particularly suit SSL, naturally offering: different views of the data, stability in terms of distance preservation, and the opportunity to flexibly incorporate domain knowledge. We (1) investigate autoencoder, which shows the general representational power of PH. and (2) propose a contrastive loss that complements existing approaches. We rigorously evaluate approach for molec- ular property prediction and demonstrate its particular features in improving the embedding space: after SSL,, the representations are better and offer considerably more predictive power than the baselines over different probing tasks; our loss increases baseline performance, sometimes largely; and we often obtain substantial improvements over very small datasets, a common scenario in practice.
Introduction
Self-supervised learning (SSL) has great potential for molecular representation learning given the complexity of molecules, the large amounts of unlabelled data available, the considerable cost of obtaining labels experimentally, and the resulting often small datasets. The importance the topic is reflected in the variety of paradigms and architectures that are investigated [Xia et al. 2023a].
Most existing approaches use contrustive learning (CL) as proposed in (You et al. 2020 CL aims at learning embedding space by comparing training samples and encouraging representations from positive pairs of samples to be close in the embedding space while representations from negative pairs are pushed away from each other. However, usually, each given molecule is considered as its own class, that is, a positive sample pair consists of two different views the same molecule; and all other samples in batch used as the negative pairs during training. We observe that this represents a very coarse -aatined comparison basically ignoring all commonalities and differences between the given molecules. Therefore also current efforts in molecular CL are put in constructing views that capture the possible relations between molecules: GraphCL (You et al. 2020) proposes four
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Doath Sire
Topological Fingerprint
airthSir
Number
Figure 1: Topological fingerprints are constructed using a filter (e.g., atom number) and recording
resulting topological structures in a PD which is then vectorized.
simple augmentations (c.g., drop nodes), but fix two per dataset; JOAO You et al. 2021 extends
the former by automating the augmentation selection; GraphLoG [Xu et al. 2021 optimizes in local neighborhood the embeddings, based on similarity, and globally, using prototypes based on
semantics; and SimGRACE [Xia et al. 2022 creates views using second encoder, perturbed
version of the molecule's 's ones; also most recent works follow this paradigm [Wu et al. 2023
Quality of Embedding Spaces is Underinvestigated. Although differences in performance are often observed, they seem marginal and are not yet fully comprehended. With a few exceptions, the models are usually evaluated over the MoleculeNet benchmark [Wu et al. 2018a] only, a set of admittedly rather diverse downstream tasks. While those certainly provide insights into model performance, recent evaluation papers have pointed out that the picture may be very different when the models evaluated more detail [Sun et al.] [2022 Wang et al. 2022a Akhondzadeh et al.) 2023 Deng et al.)
2022]. In particular, [ARhondzadeh et al. power of the representations instead of a few, specific downstream tasks. This type of evaluation is also common in other areas of DL [Chen al.,
2023 propose to use linear probing analyse the actual
2020). (Deng al. 2022] consider other, smaller
datasets, and compare to established machine learning (ML) approaches.
We study molecular SSL based on persistent homology (PH). PH is mathematical tool for mod- eling topological features of data that persist across multiple scales. In a nutshell, the molecule graph can be constructed sequentially using custom filter, such that atom nodes and the corresponding bond edges only appear once they meet a given criterion (c.g., based on atomic mass); see Figure captures this sequence in persistence diagram (PD), and the area has developed various methods to translate these diagrams into topological fingerprints, which can be used for ML [Ali ct al. 2022]
[ PH then
Topological fingerprints have been used for supervised ML over molecules by chemists Krishnapriyan et al. 202T Demir et al., [2022] and recent work in AI has shown promising results compared to the commonly used ECFP embeddings [Rogers and Hahn 2010). Moreover, many of these fingerprints have been shown to be stable, in the sense that distances between the PDS reflected in
GraghLaGG
0.5
Gighla
RDC AUC
SuGRRCE
GraiMMV
0.8
the distances between the corresponding fingerprints. We point out that the unique features of topological finger- prints particularly suit molecular SSL, naturally offering: different views of the data (i.e.. . by switching the filter func- tion), stability in terms of distances, and the opportunity flexibly incorporate domain knowledge.
0.4
100200
400)
800
3,000
1 Training Sumples
Figure 2: TDL may yield great improve- ments in the low-data scenario; Coi.
Our Idea. We consider the topological fingerprints of the molecules in the pre-training data as views and exploit their stability to model the distances between the given molecules (i.e., instead of between a molecule and/or its views). In particular, we use them for fine-grained supervision SSL.
Contributions.
We propose a topological distance contrastive loss (TDL) which, as outlined above, shapes the embedding space by providing supervision for the relationships between molecules, especially the ones in the data. TDL is complementary and can be flexibly and efficiently applied to complement any CL approach. We extensively evaluate it combining TDL with four established CL models.
We also consider alternative paradigms for comparison: we study a straightforward topological fingerprints autoencoder (TAE).
Our evaluation particularly focuses on the potential improving the embedding space and we demonstrate the gain in representation power in detail empirically (see Section 4.1 TDL en-

---

ables the models to learn, in sense, calibrated distances; considerably improves linear probing performance; and the üineggranng supervision may mitigate deficiencies of individual models. Over downstream data (see Section 4.2) the performance increases depend more on the model and data, Notably, TDL is able to considerably improve GraphCI. and make competitive with SOTA. Moreover, TDL strongly improves various models in the low-data scenario (see Figure
Our implementation is available httte://gsttuuoco//////////////2
2 Background and Related Works
Graph Homology. Molecules are graphs G = (V,E) with nodes V. the atoms, and bond edges E. In algebraic topology. graph homology considers such graph G topological space. We focus on simplices: every node is a O-simplex, and every edge is a I-simplex. In the context of graphs, we obtain a 1-dimensional simplicial complex X V U E in easy way, by considering the simplices induced by G; the dimension is determined by the maximal dimension of the contained simplices.
Persistent Homology (PH). We introduce the most important concepts and, for more details, refer to [Dey and Wang 2022, Edelsbrunner and Harer, 2022]. Persistent homology is a mathematical tool for modeling topological features of data that persist across multiple scales, comparable to different resolutions. These features are captured in persistent diagrams which, in turn, can be vectorized in fingerprints. We outline the process simplified below and in Figure 1 see Appendix ( for details.
First, the goal is to construct nested sequence of subgraphs G1 S ... S GN G (1 S S N). As described above, these graphs can be considered as simplicial complexes, hence we have simplices which we can record in a persistence diagram. To this end, we consider one of the most common types of filtration methods: sublevel/superlevel filtrations. A sublevel filtration is a function f X - IR over all simplices. simple such function can be a vale-d function (e.g., map atom nodes mox(s(u) (v)). Denote
to their atomic number) that is expanded to the edges as f(u, by Xa the sublevel set of X, consisting of simplices whose filtration function values S M()), X. = {xr E (xx) S [a)]. As the threshold value increases from mingev f'() to f(v) let G. be the subgraph of G induced by Xa: i.c., G. = (Va, Ea) where Va {v E +Xa) and Eg {er= X.). This process yields nested sequence of subgraphs G1 C C S.... ... GGv G.
As Xa grows to X, new topological structures gradually appear (born) and disappear (die).
Second, persistence diagram (PD) is obtained as follows. For each topological structure a, PH records its first appearance and its first disappearance the filtration sequence. And this is represented by a unique pair (b,, d.). where 1 < b, I d I N. We call b, the birth time of a,, d the death time of a and d, - by the persistence of a. PH records all these birth and death times of the topological structures in persistence diagram (D) k-th homology group of G, and practice, i typically takes values or 1. This step is rather standard and corresponding software is available [Otter al. 2017].
{(br, dz)] a E H& (G:)). where H& (G;) denotes the
PH in ML. The vectorization of PDs for making them applicable in ML has been studied extensively Ali et al. 2022). and proposals range from simple statistical descriptions to more complex persistence images (Pls) [Adams et al., 2017]. In a nutshell, for PIs, the PD is considered as 2D surface, tranformed using Gaussian basis function, and finally discretized into a vector. Furthermore, the Euclidean distance between PIs is stable with respect to the 1- W-Wasserstein distance between PDs Adams et al. 2017). which essentially means that the former is bounded by a constant multiple of the latter. We focus on PIs based on the promising results reported supervised settings (Demir et al., 2022 Krishnapriyan et al. 2021]. Our study was inspired by the ToDD framework, applying pre- trained vision transformers to custom 2D topological fingerprints [Dmir et al., 2022). Interestingly, they use the distances between the transformer's 's molecule representations to construct suitable dataset for subsequent supervised training using triplet loss. Our focus is on SSL and we apply the ToDD fingerprints as more complex, expert fingerprints in comparison to the ones based on atomic mass only. Recently, various other approaches of integrating PH into ML are explored, but these are only coarsely related to our work (e.g., (Hom et al. 2022 Yan et al. 2022
Molecular SSL. Since the foundational work of Huu et al.) 2020 who have proposed several effective methods, such as the node context prediction task ContextPred, the area is advancing at great pace [Xia et al. 2023a Xie et al. 2022). Our proposal falls into the category of contrastive
learning as introduced in Section | Our study focuses on the potential of PH complement existing
3

---

models such as You et al. 2020] and follow-up approaches. There are other related CL approaches which we do not aim compare directly; e.g., works including 3D geometry [Liu al. 2022a Stärk et al. [022] or considerably scaling up the pre-training data (Ross et al. 2022] Zhou et al. 2023
In contrast, our focus is on improvement through exploiting unused facets of the data.
SSL based on Distances. We found only few works that explicitly incorporate distances into SSL. With regard graphs. [Kim et al. 2022] exploit the graph edit distance between a graph and views to similarly represent the actual distance in the embedding space inside the loss function (i.e., vs. invariance to the view transformation). This distance computation feasible since it can be easily obtained based on the transformation. Our work focuses on exploring in how far distances in terms of PH can be applied towards the same goal: moreover, this makes it feasible to model the distances between given samples. [Wang et al.] [2022b] consider loss very similar to our proposal based on the more coarse-graiee graiied HCFPs, instead of Pls, but they focus on chemistry aspects instead of SSL. more generally. Also [Zha et al.) 2022] model distances between given samples in the context of CL in similar way, but in the context of a supervised scenario, where the distances are the differences between given regression labels; note that they also list other coarser related works. [Taghanaki et al., 2021] apply custom distances inside a triplet loss but similarly exploit label information, in order to select the positive and negative samples. Beyond that, distances have been applied to obtain hard negatives (Zhang and Re Demir et al. 2022]. and are exploited in various other ways rather different from our method. For instance, several recent approaches aim to structure the embedding space by exploiting correlations between samples which are, in turn, obtained using nearest neighbors [Caron et al. 2020 Dwibedi al. |021 Ge et al. 2023). Also methods considering equivariance between view transformations implicitly model distances [Chuang et al. 2022 Devillers and Lefort 2023
Others. While our focus on CL and distances hints at close relationship to deep metric learning KAYA and BILGE 2019]. these works usually do not have explicit distances for supervision but, for instance, exploit labels. Observe that this indicates the unique nature of the distances PH provides us with. Lastly, SSL more generally (Rethmeier and Augenstein 2023] has naturally been inspiration for molecular SSL and is certainly one reason why the field was able to advance so fast. We use its insights by putting focus on linear probing and investigating dimensional collapse [Hua et al. 2021
3 Methodology
The main goal of SSL is to learn an embedding space that faithfully reflects the complexity of molecular graphs, captures the topological nature of the molecular representation space overall, and whose representations can be effectively adapted given labels during fine-tuning. We propose methods based on persistent homology and show that they naturally suit SSL. First, persistence images (or comparable topological fingerprints) offer great versatility in that they are able to flexibly represent knowledge about the graphs and allow for incorporating domain knowledge. Second, they capture this knowledge based on persistence diagrams, which is very different from and hence likely complementary to the common graph representation methods in deep learning. Third, and most importantly, their stability represents unique feature, which makes them ideal views for SSL.
We study two SSL approaches based on PH and evaluate them in detail in terms of both their impact on representation power (see Section 4.1 and on downstream performance (see Section 4.2)
order to study the impact of PH on SSL. in general, we consider a simple autoencoder architecture. Since we consider topological fingerprints to represent information that is complementary to that used in existing approaches (we also show that they are not ideal alone) and because their topological nature can be used to improve the latter in unique ways, we developed loss function based on contrastive learning that complements existing approaches.
In this paper, our focus is on obtaining initial insights about the potential PH offers for molecular SSL, hence we chose one basic solution and one providing unique impact. There are certainly other promising ways to be investigated in the future.
3.1 Topological Fingerprints AutoEncoder (TAE)
Autoencoders are designed to reconstruct certain inputs given context information for the input graph G. We consider topological fingerprints Ig as the reconstruction targets, specifically, Pls.

---

Readouu
E
Pepectinn
Graph Emoondiing
Graph Eimbedding:
Topological Space
Batch Molecules
Persiatent Homology
Topological Fingerprint
Topalogical Distance Contrastive Loes
Figure Overview Topological Distance Contrastive Learning.
The model itself employs typical graph encoder s(-) for computing em- beddings h, for the individual nodes v E V; for simplicity, we write by = e (G), where hy {hyle V) rep- resents all node representations. Next, we pass through projection head g(() and readout function R(-) (e.g.. mean pooling) to obtain the graph- level representation hag R(g(hv))
10'
103
101
10F
Coa Samiarityy
Coo Sim.artyy
Figure 3: Comparison of molecule similarity based PIs similarity between corresp. ECFPs (blue: 20% most similar).
Hence the context is the particular knowledge the graph encoder was de- signed to exploit; our evaluation, we consider the commonly used GIN et comparison with related works. As loss formula for this topological fingerprints fingleiicallnggerprintss autoencoder (TAE) we use the regular mean squared error: LIAE = Ee MSE (hg, IG).
et al. in order to ease
Observations. Given stable fingerprints such as PIs, one main usual criticism for autoencoders, the fact that they fail capture inter-molecule relationships, does not apply if the model is able to reliably learn the Pls, which we show TAE does; in particular, we observe strong correlation between PIs and their reconstructions (see Table 7 The successful application of topological fingerprints in supervised learning and the simplicity of the model justify the study of TAE for analysis, however, we note that the PIs employed may not necessarily capture all information which is critical for particular downstream task. For example, Figure 3 shows that PIs may offer generally fitting embedding space, in that randomly chosen molecule pairs with similar standard fingerprints based on substructures (blue, Tanimoto similarity of ECFPs) have rather similar PIs (x-axis, cosine similarity). However, there clear difference for the two depicted datasets, sometimes PIs as shown here, based on the ToDD filtration Demir et al. [2022), fail to fully capture structural similarity; and this directly reflected performance (see Table 4 While optimization on a case-by-case basis w.r.t. the choice of fingerprints is possible, this is not in the spirit of foundational SSL, requires expert knowledge or extensive tuning, and may still not be sufficient. For that reason, we suggest to apply them together with existing approaches and show that they offer unique benefits.
3.2 Topological Distance Contrastive Loss (TDL)
Contrastive learning aims learning an embedding space by comparing training samples and encouraging representations from positive pairs of examples to be close in the embedding space while representations from negative pairs are pushed away from each other. Current approaches usually consider each sample as its own class, that is, positive pair consists of two different views of it; and other samples in a batch are used as the negative pairs during training. We observe that this represents very mnars--griined comparison basically ignoring all commonalities and differences between the given molecules; this is also why current efforts graph CL. focus on constructing views that capture the possible relations between molecules.
5

---

Our Idea, Figure We exploit the stability of topoloicaa fingerprints such as PIs to model the distances between the given molecules (i.e., instead of just views of the same molecule) and to use them for fine-grained supervision in SSL (recall that stability means the distances between Pls reflect those between the topological persistence diagrams). This is very different from and complementary to related works in that it structures the embedding space in different way. The fingerprints are usually constructed in a way so that they capture information about the molecular graph structure; even if they do not capture the entire complexity of the molecules, they represent some, probably important aspects. Moreover, based on the topological nature, they may capture aspects not represented by the commonly used graph embeddings. In particular, they offer way to flexibly integrate expert knowledge into molecular SSL; observe that the usually employed GNNs have dimensions based on the available molecule information, hence including additional information, even if available only for some of the data, requires architecture changes.
We consider a batch of N graphs, {Gi) Similar as above, we first extract graph-level representation vectors hc. using a graph encoder s-)), followed by a readout function [((-). Here, we apply the projection head g()) later, to map the graph representations to another latent space and obtain the final graph embedding 24. Specifically, we apply two-layer MLP and hence a non-linear transformation, known to enhance performance [Chen et al. 2020]: 2 = 9 (R (E (G.)))
Let G, be the sample under consideration. Instead of constructing an artificial view, we consider all possible positive pairs of samples (Gn, Gm.) together with set of stable, topological fingerprints (T))e(1.N) for all graphs. Note that these can be computed rather efficiently, and they have to be computed only once for the given pre-training data. In nutshell, our loss adapts the regular NT-Xent [Sohn] 2016 Oord et al. 2018 Wu et al. 2018b) by considering only those negative pairs (G,,. G&) of samples where the Euclidean distance dis (In, 1x) between the corresponding fingerprints is greater than the one between I, and 4m- We compute the similarity score sim (Z =m)
Fn 2m/ [| ||=m]], and consider a temperature parameter and indicator function I[] E {0. 1) as usual. Our topological distance contrastive loss (TDL) is defined as follows, for the n-th sample:
LTDL,,
1
Σ M. mFn
log
N
TDL can be flexibly and efficiently applied to complement any graph CL fr.rrmework, e.g., in the form C, Lothers XLTDL,, where X determines its impact. In our evaluation, we used 1.
Further Intuition. Essentially, TDL provides a form of regularization. It encourages the model to push molecule representations less far away from the sample under consideration if they are similar to it in terms of the topological fingerprints. Given the stability of those, we hence obtain an embedding space with better calibrated distances (i.e., distances in terms of PH, between persistence diagrams of the molecule graphs). This can be partly observed theoretically, in the directions of the gradients; see Appendix for an initial analysis. We show this empirically by calculating the correlation between distances between the molecule representations after pre-training and the PIs (see Table D and by visualizing the distances in Figure 5 Our evaluation further shows that the fine-grained supervision may solve deficiencies of CL models in that it forces them to capture crucial features of the input. Further, the improved embedding space particularly suits low-data downstream scenarios.
On Views. While we consider the modeling of the sample relationships to be most unique and offer great potential to complement other models, we note that TDL can be similarly applied over views.
Evaluation
Do TAE and TDL. lead to, in sense, calibrated distances in the representation space?
Do we obtain improved representations more generally, based on established SSL metrics?
What impact do we see on downstream performance, and does it justify our proposal?
Our Models. We apply both TAE and TDL with different filtration functions, marked by subscript. First, we apply a most simple filtration atom based on atomic mass. This allows showing that, even by considering less information than the baseline GNNs, which additionally apply atom chirality and connectivity, topological modeling may exploit additional facets of the data. Since TAE is a standalone model, atomic mass is not enough to let it fully capture the molecular nature. For that

---

Table 1: Pearson correlation coefficients (%) between distances in embedding space and distances
between corresponding Pls, for samples from various MoleculeNet datasets. Highlighted are clear decreases and clear increases (i.e., considering standard deviation).
Tox21
ToxCast
Sider
Clin
MUV
HIV
BBBP
Bace
ContextPred
31.2 (0.4)
2.5 (0.0)
61.6 (0.2)
15.6 (0.5)
37.2 (0.1)
20.6 (0.1)
3.7 (0.3)
12.7 (0.2)
TAa
35.6 (0.3)
12.2 (0.6)
60.6 (0.3)
6.2 (1.6)
55.9 (0.1)
30.9 (0.1)
2.7 (0.7)
25.9 (0.2)
GraphCL
15.6 (0.4)
52.6 (0.8)
16.7 (0.8)
35.4
18.3 (0.2)
(1.8)
10.5 (0.2)
TDLutom
55.4 (0.2)
14.5 (0.4)
66.6 (0.3)
21.9 (0.7)
65.2 (0.3)
41.5 (0.1)
(1.4)
34.1 (0.3)
JOAO
25.1 (0.8)
3.2 (1.7)
62.4 (0.3)
20.2 (1.3)
48.0 (0.2)
29.5 (0.3)
2.7 (1.4)
24.8 (0.3)
TDLatom
49.3 (0.3)
(2.1)
65.2 (0.1)
26.2 (1.0)
60.2 (0.2)
40.6 (0.4)
6.7 (0.8)
36.0 (0.4)
SimGRACE
10.3 (2.9)
9.8 (0.6)
(1.9)
(1.6)
26.5 (0.3)
21.6 (0.2)
(2.1)
(0.1)
TDLgiom
48.8 (0.9)
12.8 (1.7)
61.5 (0.2)
20.2 (0.5)
71.9 (0.1)
49.1 (0.2)
(2.0)
38.9 (0.9
GraphLoG
16.8 (0.2)
2.9 (0.5)
34.2 (0.2)
3.6 (0.8)
(.2) (0.1)
3,3 (0.9)
12.6 (0.2)
TDD
44.2 (0.4)
11.8 (1.3)
50.5 (0.4)
(1.2)
63.9 (0.1)
46.2 (0.2)
1.9 (0.4)
40.3 (0.1)
reason, we consider three filtrations (atomic mass, heat kernel signature, node degree) and concatenate the corresponding Pls. denoted by ahd. Finally, to show the real potential of PH, we include the ToDD filtration Demir et al. [022)) which combines atomic mass with additional domain knowledge, partial charge and bond type, inside more complex
filtration. See Appendix [A]
Baselines Datasets. For a comprehensive evaluation, we apply TDL on variety of existing CL approaches: GraphCL, JOAO, GraphLoGi, and SimGRACE (see Section []. Note that this goes far beyond other CL extensions which are often evaluated with a single approach only (You et al.] 2021] Xia et al. 20236]. We also study TAE on top of the established ContextPred (Hu et al. 2020]. to get an idea of its complementary nature. Model configurations and experimental settings are described in Appendix A For pre-training, we considered the most common dataset following [Hu et al.
2020), million unlabeled molecules sampled from the ZINC15 database [Sterling and Irwin 2015]. For downstream evaluation, focus on the MoleculeNet benchmark [Wu et al. [2018]] here, the appendix contains experiments on several other datasets.
4.1 Analysis of Representations after Pre-training
Calibrated Distances in Embedding Space, I Figure
TAE successfully learns the PI space for the molecules in MoleculeNet in the sense that there is a strong corre- lation between the PIs and their reconstructions (Table appendix). For TDL, we observe for all baselines consid- erable increases in terms of correlation between distances in PI space and in representation space; we use the embed- dings from linear probing. There are two notable excep- tions and generally smaller increases on one dataset, BBBP. We also compared the ROGI score [Aldeghi et al. 2022) which, in nutshell, measures in how far (dis) (dis)similarity of molecules (i.e., we use Euclidean distance of embeddings) reflected in label similarity (Table 9 appendix). This gives us some idea in terms of downstream labels and the mixed results reflect the variety of the data. Finally, the alignment figure on the right clearly visualizes that the PI embedding space is much more fine- aine-grained in terms of distances than the GNN space, and that the distribution of GraphCL seems to get correctly adapted that distances between "positive" pairs generally shrink below the ones of "negative" pairs.
Bace
200
Figure 5: Normalized Fuclidean dis-
tances for pairs of embeddings after pre- training of same/different Bace class, dark/light: PITADD, green; GraphCL, blue; GraphCL+TDLnaD. red.
Mitigating Dimensional Collapse, Figure 6 One of the most interesting of our findings is the fact that GraphCL and Graphl LoG suffer from dimensional collapse (i.e., the embeddings span lower-dimensional subspace instead of the entire available embedding space, [Wang al. [002aa] also observe this for GraphCL) and that TDL successfully mitigates this. This can be observed in
7

---

SIMRRATT:
Grgaet
SINCRACT:
Grapstati
Stegitar Villes (log)
-10
-10
-10-
-21
11
-30
-30
-00
100
200
500
100
200
500
100
200
300
100
200
300
Singular Rank Indes
Sincula Valie Runk Indes
SSiagular Value Bask Indes
Siagglaa Value Rask Endex
Figure 6: Singular values of covariance matrices of the representations; Bace (left), Clintox (right).
Table 2: Lincar/MLP probing: molecular property prediction; binary classification, ROC-AUC (%).
Tox21
ToxCast
Sider
ClinTox
MUV
HIV
BBBP
Bace
Average
Plases
(0.3)
51,5
56.1 I (0.5)
45.5 (1.1)
(0.7)
68.4 (0.6)
47.1 (0.4)
(0,7
$4.08
PlaDD
65.8 (0.3)
50.3 (0.3)
58.1 (0.5)
(1.3)
(0.9)
70.8 (0.4)
57.1 (0.7)
67.8 (0.8)
60.31
ECFP
68.8 (0.3)
57.0
62.3 (0.3)
69.2 (0.8)
66.1 (0.4)
63.2 69.4
(0.6
66.20
ECFP || Plyan
69.6 (0.4)
56.3 (0.3)
60.9 (0.6)
76.7 (1.1)
64.0 (0.6)
(0.5)
(3.4
(1.0)
67.36
Places MLP
57.2 (0.5)
52.5 (0.4)
56.6 (0.7)
49.8 (1.4)
60.5 (1.6)
(1.0) (0.4)
(1.3
56.08
PlEoD. MLP
66.7 (0,3)
52.5 (0.4)
S8.6 (0.6)
61.8 (1.6)
60.1 (0.4)
(1.6(
$7.3 (0.9)
(1.3
62.03
ECFP, MLP
70.1 (0:4)
59.8 (0.4)
59.6 (0.6)
67.8 (0.9)
61.7 (0.8)
69.1
58.6
(1.7
64,85
ECFP || Plaan. MLP
71.1 (0.6)
57.8 (0.4)
59.2 (0.7)
80.7 (2.1)
64.9 (1.1)
72.8
63.1 (0.8)
76.7 (0.9)
68.28
TAEad TAEp00
67.7 (0.2)
61.2 (0.2)
55.8 (0.3)
58.1 (0.7)
(0.8)
72.5 (0.5)
61.1 (0.2)
74.3 (0.2)
65.11
(0.2)
60.8 (0.1)
61.1 (0.1)
68.4 (0.7)
(0.3)
73.9 (0.2)
61.6 (0.4)
67.6 (0.6)
67.01
ContextPred
68.4 (0.3)
59.1 (0.2)
59.4 (0.3)
(3.7(
(0.7)
68.9 (0.4)
59.1 (0.2)
64.4 (0.6)
61.69
TAE
69.7 (0.1)
59.2 (0.2)
59.5 (0.3)
56.1 (1.1)
76.5 (0.9)
68.9 (0.2)
61.1 (0.4)
65.6 (0.5)
64.58
+ TAE
(0.0)
59.8 (0.4)
60.0 (0.4)
53.3 (1.3)
70.8 (0.3)
(0.7)
60.9 (0.5)
62.7 (0.5)
63.31
GraphCL + TDL
(0.5)
59.4 (0.2)
54.6 (0.3)
59.8 (1.2)
(1.0)
63.7 (2.3)
62.4 (0.7)
71.1 (0.7)
63.20
(0.4) 61.1 (0.2)
59.7 (0.6)
65.3
76.1 (0.9)
68.2 (1.1)
65.4 (0.9)
76.4(1.1)
68.02
+ TDL:3DD
72.7 (0.5)
60.8 (0.4)
58.9 (0.8)
64.1 (1.7)
72.7 (14)
69.7 (1.2)
64.5 (0.8)
76.1 (1.3)
67.44
JOAO
70.6 (0.4)
60,5 (0.3)
$7.4 (0.6)
(2.6)
69.8 (1.9)
68.1 (0.9)
63.7 (0.3)
(1.2
64.42
+ 'TLLam
70.5 (0.3)
60.4 (0.2)
(1.5)
$4.6 (1.3)
(1.6)
68.2 (0.6)
65.2 (0.3)
72.7 (3.1)
65.41
TDLLED
71.7 (0.4)
61.3 (0.3)
(0.7)
$2.4
69.6 (1.7)
69.9 (0.6)
(0.5)
72.6 (0.9)
65.06
SanGRACE
64.6 (0.4)
59.1 (0.2)
54.9 (0.6)
63.4
(1.2)
(1.5)
65.4 (1.2)
67.8 (1.3)
63.61
+ TDD
68.6 (0.3)
61.1 (0.2)
$9.5 (0.4)
(1.7
69.7 (2.0)
(1.8)
60.6 (0.5)
72.1 (0.7)
65.41
TDLLED
70.1 (0.3)
60.3 (0.3)
59.1 (0.3)
65.1 (1.4)
(1.1
71.1 (0.7)
64.9
73.4
66.93
GraphL.oG
(0.2)
$7.9 (0.2)
$7.9 (0.3)
57.8 (0.9)
(1.I
(1.3)
$4.3 (0.7)
(0.9)
62.08
+ TDLLoo
72.1 (0.3)
62.0 (0.2)
(0.2)
56.6 (0.8)
(0.9)
70.4 (0.9)
61.2 (0.4)
76.8 (0.7)
66.59
+ TTD
70.7 (0.2)
60.7 (0.3)
61.5 (0.3)
59.5 (0.5)
72.9 (1.8
71.6 (0.8)
62.1 (0.3)
80.1 (0.4)
67.39
Table 3: Linear probing: given two molecules, predict distance between their Pls; MSE.
21x21
ToxCast
Sider
Clin'Tox
MUV
HIV
ВHBP
Bacc
ContexaPred
6.330 (0.007)
5.351 (0.084)
22,227 (0.102)
4.234 (0.039)
2.342 (0.001)
6.127 (0.009)
4,907 (0.072)
3.591 (0.038)
TARas
5,970 (0.002)
5.263 (0.003)
21.262 (0.070)
4,255 (0.023)
1.851 (0.003)
5.752 (0.023)
4.901 (0.024)
3,069 (0.012)
GraphiLL
6,873 (0.011)
(0.03))
24,019 (0.059)
1.537 (0.006)
2.411 (0.006)
5.996 (0.006)
5,666 (0.005)
3.858 (0.007)
+ TDD
5.371 (0631)
5.191 (0.023)
20.641 (0,022)
4,231 (0.025)
1.645 (0.004)
5.001 (0.022)
4,936 (0.002)
2,375 (0.008)
JOAD
6.465 (0.032)
5.277 (0.010)
21.299 (0.085)
4,277 (0.118)
2.061 (0.001)
(0.016)
4.995 (0.036)
2.782
+ IDM
5.642 (0.011)
5.187 (0.010)
20.515 (0.044))
4.185 (0.027)
1.749 (0.004)
5.011 (0.028)
4.979 (0.015)
2.377 (0.002)
SimGRACE
10.015 (0.050)
13.387 (4.770)
26.514 (0.049)
4,669 (0.017)
2.591 (0.005)
6.150 (0.023)
6,167 (0.035)
13.054 (0.938)
TDL,yer
5.359 (0.005)
5.263 (0.016)
20.630 (0.034)
4,071 (0.042)
1.545 (0.007)
(0.033)
(0.011)
2.296 (0.003)
GrphLLoG
6.877 (0.012)
5:279 (0.004)
24.073 (0.038)
4.482 (0.055)
2.857 (0.004)
6.591 (0.015)
5.029 (0.079)
3.918 (0.004)
+ 1IX
5,814 (0032)
5.212(0:018)
21.441 (0.044)
4,237 (0.015)
1.719 (0.005)
4.669 (0.010)
4.962 (0.015)
2,235 (0.005)
the singular values of the covariance matrix of the representations, where vanishing values hint at collapsed dimensions. The phenomenon has recently been observed in computer vision and there are only initial explanations to date [Hua et al., 202T]. However, here, we observe the collapse only over the downstream data. Our hypothesis is that TDL's fine-grained supervision forces the models to capture relevant features, which they might have neglected otherwise. We also depict the graphs for SinGRACE and JOAO, which look very different, reflecting the variety of the approaches. There is basically no difference for JOAO(+TDLL) while TDL. shrinks the values for SiGRRAEE; the latter less optimal and needs further investigation.
8

---

Table 4: Binary classification over MoleculeNet; ROC-AUC, % Pos. is min/med/mmx for multi-task.
Tox21
TuxCast
Sider
ChinTox
MUV
III
BBBP
Buce
Average
Molcaules
7,831
8,575
1,427
1,478
95,087
41,127
2,39
1,513
e Tasks
12
617
27
2
17
1
8 Posilives
2.4/4.6/12.0
0,2/1,3/005
1.566.3922.4
3,2,4
7.6/506/9366
0,0000300003
3,5
76.5
45.7
No pretrain (GIN)
(0.4)
61,7 (0.5)
(1.2(1.7)
(6.4)
(1.7(
75.5 (0.8)
65.7 (3.3)
(3.8)
67.15
AD-GCL Suresh al 2011
(0.8)
63.0 (0.7)
(0.7)
(3.5)
72.3 (1.6)
(0.9
72.67
IMeICLK Want cra 2010
75.1 (0:7)
63.5 (0.4)
(1.0)
$1.0
74.7 (1.9)
77.3 (1.2)
(1.2)
(1.0)
72:24
Mole-BEKT Kar & cica
76.8 (0.5)
64.3 (0.2)
62.8 (1.1)
(3.9
78.5 (L.8)
78.2 (0.8)
71.9 (L.6)
80.8 (1.4)
74.01
SEGA [Wa m. 2013
76.7 (0.4)
65.2 (0.9)
63.6 (0.3)
84.9 (0.9)
76.6 (2.4)
77.6 (1.3)
71.8 (1.0)
77.0 (0.4)
74.17
TAB
75.2
63.1 (0.3)
61.9.0089 (0.80
(1.9)
74.6 (L.8)
73.5 (2.1)
67.5
K2.5 (L.)
72.36
TAB
763 (0.9)
(0.5)
61.9 (0.8)
79.3 (3.6)
(3.2)
(1.1)
70.4 (0.8)
(1.4
73.22
ContexPPred
75.7
63.9 (0.6)
60.9 (0.6)
(3.9
75.8
(1.3
68.0 (2.0)
(1.2)
70:89
(0.5)
63.2 (0.4)
62.0
(1.6)
(1.2
(1.1)
72.53
TAEnnc
75.7 (0.4)
63.1 (0.3)
61.3 (0.5)
(1.3
(L.2(
77.6 (1.1)
(0.9)
(1.4)
72.09
GrzgkCI.
(0.7
62.4 (0.6)
60.5 (0.9)
(2.77
69.8 (2.7)
78.5 (1.2)
69.7 (0.7)
(1.4)
70 78
75.3 (0.4)
61.4 (0.3)
61.2 (0.6)
83.7 (2.7)
75.7 (0.8)
78.0 (0.9)
70.9 (0.6)
80.5 (0.8)
73.71
TDLua
(0.7)
64.2 (0.3)
61.5 (0.4)
85.2 (1.8)
(2.1
77.9 (0.8)
69.9 (0.9)
(1.9
73,88
JOA0
(0.5)
62.9 (0.5)
60.0 (0.8)
$1.3 (2.5)
(1.4(
76.7 (1.2)
(1.0)
77.3 (0.5)
71.89
TDD
(0.3)
63.8 (0.2)
60.6 (0.5)
76.
73.8 (1.9)
(8.3((.2)
(0.5
(0.6)
72.22
1DLman
7522 (0.3)
(3.6
61.5
(3.3)
74.6 (1.6)
71.4
(2.2
73.18
SimGRACE
(0.3)
62.6 (0.7)
(0.9)
75.5
75.4
(1.2( (0.6)
(2.0)
71.15
74.7 (0.5)
63.0 (0.3)
59.5 (0.4)
(1.5
(1.9
76.1 (1.3
69.5 (0.9)
(0.5)
71.59
TDLuau
(5.6
63.3 (0.5)
(0.8)
62.5
(2.0)
69.9 (0.8)
78.9 (1.6)
72.71
Gaught oG
(0.6)
63.4
59.3 (0.8)
20.1 (4.6)
(1.6)
76.1 (0.8)
69.6 (1.6)
82.1 (1.0)
71.41
TDLar
76.1 (0.7)
63.7 (0.4)
(1.0
(3.5)
75.7 (L.2)
76.2 (1.8)
69.6 (1.2)
K2.2 (1.5)
72.39
TDLavn
(0.9)
63.5 (0.7)
63.4 (0.3)
(1.9)
(L.1
(1.6
70.7 (0.9)
82,1
73.39
Linear Probing, Tables 3 10 We evaluated extensively using linear layer on the representations of the pre-trained graph encoders over the MoleculeNet data in terms of binary classification and a custom distance prediction task. The probing, also in terms of MLPs. demonstrates that PIs possess a complementary nature to ECFP. Furthermore, TAF, which we developed for comparison purposes only, is competitive with the baselines and improves ContextPred. In Appendix C we additionally show some results over the more challenging activity cliff data. Apart from JOAO+TDL, where the results are mixed, TDL yields overall impressive increases. Interestingly, the more complex ToDD-based version is not always better than the simpler one.
Discussion. The results demonstrate that both TAE and TDL successfully approach the PI space in terms of distances between molecules. Yet. it also has to be noted that the PIs we chose do not always capture the main characteristics of the sample space, as seen for BBBP. Similarly, the mixed ROGI scores, which incorporate downstream labels, hint at a varied downstream performance. Nevertheless, the specifics of particular downstream datasets should not be confused with the more general molecular representation space. In that respect, especially TDL. has been proven effective overall in improving representation power in a variety of probing tasks for all baselines. We have also shown that, on closer look, the effects can be very different with the considered baselines, depending on their individual deficiencies. For JOAO, we have seen improvement in distance representation but not as large improvements thereafter, suggesting that it captures similar features already. Recall, however, that JOAO "only" adapts GraphCL in that it chooses between views more flexibly. In this regard, TDL represents a more efficient adaptation of GraphCL, which turns out to be also more effective in terms of SSL generally, as we show next.
4.2 Evaluation on Downstream Tasks
Transfer Learning Benchmarks, Tables 4
First, note that the MoleculeNet benchmark contains various types of data. We have already observed the strong differences between Bace and BBBP in terms of our Pls. Here we additionally have some very large datsets and some with very many tasks; in both we expect the labels and their correlation to have increasing impact and reduce the one of the original embedding space. We report some most recent works that used the same pre- pre-training data to provide a context for interpreting our results. As expected, TAE alone only reaches decent average performance, yet surpasses SOTA on ClinTTx and Bace and also considerably improves ContextPred there. We see more variability than above in the effects TDL has on the individual baselines. We do not observe a general impact of dataset balance. To learn about the impact of labels the influence of multiple tasks, we ran some models also on individual tasks of the multi-task data and observe larger increases for TDL there (see figures in Appendix D). Altogether, the table shows mixed results for SimGRACE; some improvement for Graphl and, with a single exception, also for JOAO; and large improvements for GraphCL. Notably, when using the stronger
12, Figure 12

---

c- GogheT om
Gogal -G-MiMi Gaggtod
- GrgheT 4TM GraplC
Gopil aopaaa--mil.om oze GupiLoG
GagaCE
Smolt SmiRAACE
SITRRACE
XANO
0,5
0.8
NOC AUC
0.7
0.7
0.5
0.8
0.4
100220
400
100220
1002000
400
100200
1,000
Training Sumples
0 Training Samples
Triining Samples
3 Triinng Samples
Figure 7: Performance over smaller datasets, subsets of Bace (left) and ClinTox (right).
ToDD filtration, TDL demonstrates convincing improvements across all baselines and makes them competitive with SOTA.
The Low-Data Scenario, Figures 2 13 14 Inspired by the impressive increases see in the quality of representations after pre-training, we experimented with considerably smaller datasets. This is scenario which is relevant in practice since proprietary data from costly experiments is often only available in small numbers [Tom et al.] [023]. Here we see clearly that the improvement in representation space reflected in downstream performance. There is still some dependence on the dataset but, overall, TDL often yields remarkable improvements. The introduction figure shows that also other models show room for improvement; note that GraphMVP uses 3D information.
Further Analysis. Due to space constraints, we present only the most impactful findings in this section and the rest in the appendix, in particular: (1) Ablation Studies. Our approach offers great variability in terms of, for instance, the graph encoder, the construction of the Pis, or even the type of topological fingerprint more generally. (2) Unsupervised learning. We observe similar but less pronounced trends, likely due to the smaller size of the data and fewer features available. (3) PIs w/o SSL. In view of the promising results with PIs in supervised ML (Krishnapriyan
al. 1021] Demir et al. 2022], we consider the PIs with XGB and SVM, our approaches based on pre-training better. (4) Other datasets. We conducted experiments over several other datasets, including activity-cliff data, where TDL is shown beneficial more generally.
Discussion. In this section. we have shown results very different from the ones usually see. In the benchmark, we observe some improvements and, for GraphCL+TDL, numbers competitive with SOTA, but they do not reach the general, large impact we saw in Section 4.] However, we have such impact over smaller data. Altogether, TDL offers great benefits for all baselines we considered:
It leads to much better calibrated distances in the representation space in terms of PH. Since the magnitude of increase is generally remarkable but varies with the data, this shows potential for further improvement, especially, in terms of choosing the right Pls.
It considerably improves representations in terms of established SSL metrics.
Downstream performance reflects this gain of representation power specifically in the low- data scenario. Given the importance of this kind of data in the real world and the deficiencies of the models we checked, conclude that our proposal is not only novel and different in its distance- based modeling, but also that it advances molecular SSL in important aspects empirically.
5 Conclusions
In this paper, we focus on the quality of representations in molecular SSL., an important topic which has not been addressed greatly to date. We propose a novel approach of shaping the embedding space based on distances in terms of fingerprints from persistent homology. We have evaluated our poplloical-distacee--eee
contrastive loss extensively, and show that it solves deficiencies of existing models and considerably improves various baselines terms of representation power. In future work, we plan to analyze the relationship between specific PIs and downstream data in more detail, to further improve our approach, include external knowledge, and to put more focus on regression tasks.
10

---

Acknowledgments
This work was supported by National Key R&D Program of China (2021 (F03570B300)700, NSFC Grant 62172026, National Social Science Fund of China 22&ZD153, the Fundamental Research Funds for the Central Universities and SKLSDE; correspondence to Lei Shi.
References
Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z. Li. A systematic survey of chemical pre-trained models, 2023a.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances neural information processing systems, 33: 5812-5823, 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121-12132. PMLR, 2021.
Minghao Xu, Hang Wang. Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference Machine Learning, pages 1548-1 1558. PMIR, 2021.
Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. In Proceedings of the ACM Web Conference 2022, pages 1070-1079, 2022.
Junran Wu, Xueyuan Chen, Bowen Shi, Shangzhe Li, and Xu. Sega: Structural entropy guided 2023.
anchor view for graph contrastive learning, arXiv preprint
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9():5113-30, 2018a.
Ruoxi Sun, Hanjun Dai, and Adams Wei Yu. Does gnn pretraining help molecular representation? Advances in Neural Information Processing Systems, 35:12096: 15:1209-12109, 2022.
Hanchen Wang, Jean Kaddour, Shengchao Liu, Jian Tang, Matt Kusner, Joan Lasenby, and Qi Liu. Evaluating self-supervised learning for molecular graph embeddings. arXiv preprins arXiv:2206.08005 2022a.
Mohammad Sadegh Akhondzadeh, Vijay Lingam, and Aleksandar Bojchevski. Probing graph representations. In Francisco Ruiz, Jennifer Dy, and Jan- Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 11630-11649, PMLR, 25-27 Apr 2023. URL
mlr. prwss/ v26/ akwoooaadeodaaaaaaa.mmm html
Jianyuan Deng, Zhibo Yang, Hehe Wang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang. Taking respite from representation learning for molecular property prediction. arXiv preprint arXiv:2209.13449 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning. pages 1597-1607. PMLR, 2020.
Dashti Ali, Aras Asaad, Maria-Jose Jimenez, Vidit Nanda, Eduardo Taluzo-Hidalge and Manuel Soriano-Trigueros. A survey of vectorization methods in topological data analysis. arXiv preprint arXiv:2212..97033 2022.
Aditi S Krishnapriyan, Joseph Montoya, Maciej Haranezyk, Jens Hummelshoj, and Dmitriy Morozov. Machine learning with persistent homology and chemical word embeddings improves prediction accuracy and interpretability in metal-organic frameworks. Scientific reports, 11(1))8888, 2021.
11

---

Andac Demir, Baris Coskunuzer, Yulia Gel, Ignacio Segovia- Doooii--D Yuzhou Chen, and Bulent Kiziltan. Todd: Topological compound fingerprinting in computer-aice aided drug discovery. Advances in Neural Information Processing Systems, 35:27978-27997 2022.
David Rogers and Mathew Hahn. Extended-conmectivve fingerprints. Journal of chemical information and modeling, 50(5):742-754, 2010.
Tamal Krishna Dey and Yusu Wang. Computational topology for data analysis. Cambridge University Press, 2022.
Herbert Edelsbrunner and John L Harer. Computational topology: an introduction. Mathematical Society, 2022.
American
Nina Otter, Mason A Porter, Ulrike Tillmann, Peter Grindrod, and Heather A Harrington. A roadmap for the computation of persistent homology. EPJ Data Science, 6:1-38, 2017.
Henry Adams, Tegan Emerson, Michael Kirby. Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. Persistence images: A stable vector representation of persistent homology. Journal of Machine Learning Research, 18, 2017.
Max Horn, Edward De Brouwer, Michael Moor, Yves Moreau, Bastian Rieck, and Karsten Borgwardt. Topological graph neural networks, 2022.
Zuoyu Yan, Tengfei Ma, Liangcai Gao, Zhi Tang, Yusu Wang, and Chao Chen. Neural approximation of graph topological features. In Advances in Neural Information Processing Systems, 2022.
Weihua Hu*, Bowen Liu", Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2020. URL /tRL/tttts//opeereiiiei http:///pppaaeeeeeeeeeeerrrtrrda
Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. IEEE transactions on pattern analysis and machine intelligence, 45(2):2412-2444 2429, 2022.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. International Conference on Learning Representations, 2022a. URL httpp:://peenrevier wwt pttp://oroormm//d---uuulppppp
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pages 20479-20502. PMLR, 2022.
Jerret Ross, Brian Belgodere, Vijil Chenthamarakshann Inkit Padhi, Youssef Mroueh, and Payel Das. Molformer: Large scale chemical language representations capture molecular structure and properties. 2022.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023. URL hhtpss
Dongki Kim, Jinheon Baek, and Sung Ju Hwang. Graph self supervised learning with accurate discrepancy learning. arXiv preprint arXiv:2202.02988) 2022.
Yuyang Wang, Rishikesh Magar, Chen Liang, and Amir Barati Farimani. Improving molecular contrastive learning via faulty negative mitigation and decomposed fragment contrast. Journal of Chemical Information and Modeling, 62(1 ((11):2713-2725, 2022b.
Kaiwen Zha, Peng Cao, Yuzhe Yang, and Dina Katabi. Supervised contrastive regression. arXiv preprint arXiv:2210.01189, 2022.
Saeid A Taghanaki, Kristy Choi, Amir Hosein Khasahmadi, and Anirudh Goyal. Robust representa- tion learning via perceptual similarity metries. In International Conference on Machine Learning, pages 10043-10053. PMLR, 2021.
12

---

Michael Zhang and Christopher Re. Contrastive adapters for foundation model group robustness. In Advances in Neural Information Processing Systems.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 33:9912-9924, 2020.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588-9597, 2021.
Chongjian Ge, Jiangliu Wang. Zhan Tong, Shoufa Chen, Yibing Song, and Ping Luo. Soft neighbors are positive supporters in contrastive visual representation learning, 2023.
Yung-Sung Chuang, Rumen Dangovski. Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljačić, Shang-Wet Wen Li, Wen-tau Yih, Yoon Kim, and James Glass. Diffese: Difference-based contrastive learning for sentence embeddings. arXiv preprint arXiv:2204. 10298, 2022.
Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module improve visual instance discrimination. In The Eleventh International Conference on Learning Representations, 2023.
Mahmut KAYA and Hasan Sakir BiLGE. Deep metric learning: A survey, Symmetry, 11(9). 2019. ISSN 2073-8994. doi: 10.3390/syml 109 1091066. URL https:::// /wwww 9/1066
Nils Rethmeier and Isabelle Augenstein. A primer on contrastive pretraining in language processing: Methods, lessons learned, and perspectives. ACM Computing Surveys, 55(10):1-17, 2023.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decor- relation in self--upervised learning. In Proceedings of the IEÉE/CVF International Conference on Computer Vision, pages 9598-9608, 2021.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rX.0:1807.03348, 2018.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedingy of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018b.
Jun Xia. Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Li. Mole-bert: Rethinking pre- gre-training graph neural networks for molecules. 2023b.
Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling, 55(1 23311))2324-3337 2015.
Matteo Aldeghi, David E. Graff, Nathan Frey, Joseph A. Morrone, Edward O. Pyzer-Knapp. Kirk Jordan, and Connor W. Coley. Roughness of molecular property landscapes and its impact modellability, 2022.
Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. Advances in Neural Information Processing Systems, 34:15920-15933 2021.
Gary Tom, Riley J Hickman, Aniket Zinzuwadia. Afshan Mohajeri, Benjamin Sanchez-Leeneelnn and Alán Sspuru-Guzik, Calibration and generalizability probabilistic models on low-data chemical datasets with dionysus. Digital Discovery, 2023.
13

---

Bharath Ramsundar, Peter Eastman, Patrick Walters, and Vijay Pande. Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more. O' O'Reilly Media, 2019.
Fan- Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv: 01000, 2019.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong. Hongxia Yang. Chunjje Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 594-604, 2022.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee- Kong Lee. Motif- basif- graph self- supervised learning for molecular property prediction. Advances in Neural Information Processing Systems, 34:1 34:15870-15882, 2021.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre- traininin of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1857-1867, 2020.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33:12559- 2020.
David Cohen-Steiner, Herbert Edelsbrunner, and John Harer. Extending persistence using poincaré and lefschetz duality. Foundations of Computational Mathematics, 91):79-103, 2009.
Jian Sun, Maks Ovsjaníkov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graphics forum, volume 28, pages 1383-1392. Wiley Online Library, 2009.
Dávid Bajusz, Anita Rácz, and Károly Héberger. Why tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics, 7(1):1-13, 2015.
Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663, 2020.
Peter Bubenik et al. Statistical topological data analysis using persistence landscapes. J. Mach. Learn. Res., 16(1):77-102, 2015.
Frédéric Chazal, Brittany Terese Fasy, Fabrizio Lecci, Alessandro Rinaldo, and Larry Wasserman. Stochastic convergence of persistence landscapes and silhouettes. In Proceedings of the thirtieth annual symposium on Computational geometry, pages 474-483, 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference knowledge discovery and data mining, pages 785-794, 2016.
Zixuan Cang and Guo-Wei Wei. Topologynet: Topology based deep convolutional and multi-task neural networks for biomolecular property predictions. PLoS computational biology, 13(7): e1005690, 2017a
Zixuan Cang and Guo-Wei Wei. Analysis and prediction of protein folding energy changes upon mutation by element specific persistent homology. Bioinformatics, (3(22):3549-3557 2017b.
Due Duy Nguyen, Zixuan Cang, Kedi Wu, Menglun Wang, Yin Cao, and Guo-Wei Wei. Mathematical deep learning for pose and binding affinity prediction and ranking d3r grand challenges. Journal of computer-aider aided molecular design, 33:7,-82, 2019.
14

---

Kelin Xia and Guo- Wei Wei. Persistent homology analysis of protein structure. flexibility, and folding. International journal for numerical methods in biomedical engineering, 30(8):814-844, 2014.
Xiang Liu, Huitao Feng, Jie Wu, and Kelin Xia. Dowker complex based machine learning (deml) mod- els for protein-ligand binding affinity prediction. PLoS Computational Biology, 2022b.
Zixuan Cang, Lin Mu, and Guo-Wa Wei, Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening. PLoS computational biology, 14(1): el c1005929, 2018.
Xiang Liu, Xiangjun Wang, Jie Wu, and Kelin Xia. Hypergraph- based persistent cohomology (hpe) for molecular representations in drug design. Briefings in Bioinformatics, 22(5):bbaa41, 11, 2021.
15

---

A Datasets and Experimental Details
Pre-training Datasets. As usual, for GNN pre-training, we use a minimal of node and bond features that unambiguously describe the two-dimensional structure of molecules from the ZINCI5 database following previous works [Hu al. 2020
Node features:
Atom number: [1, 118]
Chirality tag: (unspecified, tetrahedral ew, tetrahedral cow, other)
Edge features:
Bond type: (single, double, triple, aromatic)
Bond direction: {-, endupright, enddownright)
We note that the atom number implicitly provides domain knowledge in that it captures important knowledge about the atom's 's charge.
Downstream Task Datasets. We focus on molecular property prediction, where we adopt the widely- used 8 binary classification datasets contained in MoleculeNet [Wu et al. 20T8a]. In addition, we experimented with 4 regression tasks from various low-data domains in line with the same setting as [Liu et al. 2022al. We provide detailed information of these datasets in Table 5 and more information can be found in Liu et al., 2022a]. To further evaluate the usefulness our models, we also use the opioids-related datasets from (Deng et al. 2022). The number (percentage) of activity cliff (AC) molecules are summarized Table 6 Notably, among MDR1, 1, MOR, DOR and KOR, nearly half molecules are equipped with activity cliff scaffolds. Finally, scaffold -ppit [Ramsundar al. [019] is used to splits graphs into train/val/test set as 80%/10%/10%% which mimics real-world use cases.
Table 5: Summary for the molecule datasets.
Dataset
Task
Tasks
# Molecules
BBBP
Classification
2,039
Tox21
Classification
12
7,831
ToxCast
Classification
617
8,576
Sider
Classification
27
1,427
ClinTox
Classification
1,478
MUV
Classification
17
93,087
HIV
Classification
41,127
Bace
Classification
1,513
ESOL
Regression
1,128
Lipo
Regression
4,200
Malaria
Regression
9,999
CEP
Regression
29,978
Table 6: Summary of activity cliffs in the opioids datasets.
Dataset
MDR1
CYP2D6
CYP3A4
MOR
DOR
KOR
# Mol. AC. Scaff. (%)
594 (41.3)
710 (31.0)
926 (25.2)
1,627 (46.1)
1.342 (41.6)
1,502 (45.2) Classifictton
Task
Classification
Classification
Classification
Classification
Classification
# Tasks
Computing Environment. The experiments are conducted with two RTX 3090 GPUs.
Model Configurations. Following Hu et al. 2020 we adopt 5-layer Graph Isomorphism Network (GIN) [Xu et al.) with 300-dimmaaa hidden units the backbone architecture. We use mean pooling as the readout function. During the pre-training stage, GNNs are pre-trained for 100 epochs with batch-size as 256 and the learning rate as 0.001. During the fine-tuning stage, we train for 100 epochs with batch-size 32, dropout rate as 0.5, and report the test performance using ROC-AUC the best validation epoch. evaluate the learned representations, we follow the linear probing (linear evaluation) [Akhondzadeh et al. 2023] where linear classifier (1 linear layer) is
16

---

trained on top of the frozen backbone. We probe two types of properties from the representation: the first is the molecular properties of downstream tasks, for which we use the same settings as fine-tuning and report ROC-AUC; the second is topological fingerprints (PIs) of downstream molecular graphs, for which we train for 50 epochs with batch-size as 256, dropout rate as 0.0, and report MSE and Pearson correlation coefficients between distances in embedding space and corresponding PI space. The experiments are run with 10 different random seeds, and we report mean and standard deviation.
Performance Comparison. We compare the proposed method with existing self-supervised graph representation learning algorithms (i.e. InfoGraph [Sun al., 2019]. EdgePred [Hamilton al.
2017], ContextPred (Hu et al. 2020], AttrMask (Hu et al., 20200.
GraphLoG [Xu et al.) [202]]
iMoICLR (Wang et [0022b)) AD-GCL [Suresh et [0.1). J. JOAO You et al. 2021]. SimGRACE
[iaa et al. 2022], GraphCL You et al. a 2020]. GraphMAE [Hou et al.
2022 MGSSL (Zhang et al.
2021]. J. GPT-GNN (Hu et al. 2020], G-Contextual [Rong et al. 2020]. G-Motif (Rong et al. 2020].
GVP [iu et al., 2022a D. We report the results from their own papers with two exceptions:
(1) for GraphMAE and GraphL.oG, they reported the test performance of the last epoch; and (2)
MGSSL and SimGRACE. they only used random seeds; and (3) iMoICLR used -10M molecules
for pre-training. We reproduce the results of these four models with the same protocol as (Hu* et al.
2020 utilizing the pre-trained models originally provided by the respective papers.
Topological Fingerprints. We set the highest dimension in the simplicial complex to 1. This doesn't t impact performance because molecular graphs, being planar, rarely possess loops of length 3; in fact, the majority of loops exhibit a length of 5 or more.
Firstly, we use the sublevel filtration method. In terms of filtration functions, we use atomic number. For every molecular graph, utilize an extended persistence module [Cohen-Steiner et al. 2009
Yan et al.) 2022] to compute persistent diagrams (as depicted in Figure I of (Yan et al.) 2022 and then vectorize them to get PIs as topological fingerprints. The extended module incorporates additional reverse process. Through this module, all loop features created will be killed in the end. As an example, in Figure [ I-dim extended persistence point that can be captured is (1,, 12). This is because the loop feature Is born at time t and then dies at time tz in the reverse process. Note that we only use atomic number as filtration functions for TDLatom- In addition to the atomic number, we also investigate the heat kernel signature (hks) with temperature values t = 0..1 (Sun et al., [2099] and node degree as filtration functions. Observe that this is more geometric information, compared the domain knowledge encoded in the atomic number (atom). Considering that TAE is straightforward model and there is too little information for PI Platom, we concatenated these 3 PIs as reconstruction targets for TA
We have also considered a specific filtration function from ToDD (Demir et al. 2022]. which incorporates atomic mass and, additionally, partial charges and bond types inside a so-called multi- VR filtration. In a nutshell, for each of those three types of information, we do not simply consider the given filtration, but construct a 2D filtration by filtering: in one dimension according to the above information and in the second dimension according to a VR filtration capturing the distances between atoms (Figure 5 in [Demir et al. 2022) illustrates this kind of filtration well.) Lastly, the 3 2D PIs are concatenated, and we compute distances based on the combination of Pls. In Appendix ablation results for various filtrations.
we report
In order explore the effectiveness of Pls, we calculate the widely-used Tanimoto coefficient (Bajusz let al. 2015) of the extended connectivity fingerprints (ECFPs) [Rogers and Hahn| 2010) between two molecules as their chemical similarity on 8 downstream tasks datasets. Then, we pick the molecule pairs with top 20% similarity as 'similar' ones (Blue) and the left 80% molecule pairs in the datasets are 'random' pairs (Yellow). In final, we calculate the cosine similarity of these molecule pairs based on Pls. Figure g shows molecules with similar ECFPs have rather similar Pls.
It is computationally efficient to apply topological fingerprints method. Extracting and computing Platom for 2M molecular graphs from ZINC15 dataset only spent 20 minutes in a single workstation with 8-core CPU, 64 GB of memory.
TAE+ Models. We hypothesize that TAE gives other pre-training approach good initial embedding space to start from. Therefore, this pre-training strategy is to first perform TAE pre-training and then other models pre-training. We study TAE+EdgePred, TAE+ AurMaak and TAE*ContextPred.
TDL. The temperature parameter 7 is set to 0.
17

---

TDL over Views. Like any other contrastive loss function, TDL can be utilized standalone objective by considering views. Here, we show preliminary experiments using formulation based on the loss of GraphCI. (You et al. 2020]. we also use its augmentations. A minibatch of N graphs is processed through contrastive learning, resulting 2.V augmented graphs. We denote the augmented views for the nth graph in the minibatch as Sn, and En,j. and modify TDL follows:
Σ MEILN]
log
exp (sim (Zn,is 2m.j))
/T)
N-1
ENE(L,N] It. (2diii amma)) exp (sim (Zn,ts 2k,)) /T)
1"
(a) Bace
(b) BBBP
(c) ClinTox
(d) Sider
(c) Tox21
(1) ToxCast
g HIV
(h) MUV
Figure 8: Similarity histograms of PIs on downstream datasets. Cosine similarity measures the similarity between the PIs while 'Random' (Yellow) and 'Similar' (Blue) are defined by the similarity between ECFPs.
B Analysis of Embeddings
Table 7: The Pearson correlation coefficient between real PIs and reconstructed PIs (i.e., TAE's embeddings) on the downstream data (the full dataset), after pre-training. We also report the overlap of the downstream data with the pre-training data, but do not observe considerable impact here.
Tox21
ToxCast
Sider
Clin
MUV
HIV
BBBP
Bace
Molecules
7,831
8,575
1,427
1,478
93,087
41,127
2,039
1,513
Molecules in ZINC15
628 (8%)
608 (7%)
(0%)
51 (4%)
7599 (8%)
925 (2%)
100 (5%)
0 (0%)
TAE
0.8572
0.7744
0.5939
0,8642
0.9044
0.7359
0.8660
0.8514
18

---

Table 8: 5-nearest neighbors classifier of CL methods for 8 downstream datasets, ROC-AUC (%).
Tox21
ToxCast
Sider
ClinTox
MUV
HIV
BBBP
Bace
Average
Platom Pl Plan
58.8
51.1
58.7
50.9
50.2
64.8
55.2
75.5
58.15
62.7
51.7
56.3
64.9
49.7
63.9
56.7
68.7
59.33
ЕCFP
63.8
54.6
59.1
50.7
54.0
68.1
59.3
77.0
60.82
GraphCL
65.3
55.8
60.6
56.4
52.0
70.4
57.9
71.4
61.23
GraphCL + TDLaium
67.2
56.7
62.6
57.5
54.4
71.1
62.0
70.7
62.78
GraphC1. + -DD
68.0
56.5
61.6
63.5
54.1
70.3
62.5
66.9
62.93
JOA0
66.3
55.7
59.9
57.2
53.2
72.0
58.9
70.2
61.67
JOAO TDL.som
66.9
55.4
58.9
62.5
52.7
69.1
60.1
70.4
62.00
JOAO + TDLLDD
67.1
56.8
60.5
61.2
51.5
67.7
59.5
72.7
62.13
SimGRACE
64.2
56.1
58.3
46.7
$5.1
68.2
58.4
70.4
59.67
SimGRACE + TDL
56.3
61.9
68.4
52.4
71.3
63.6
77.3
64.66
SimGRACE TDL
67.3
58.1
60.2
77.5
52.2
68.3
67.8
76.7
66.01
GraphLoG
63.0
55.9
59.8
68.9
52.2
68.2
56.8
79.4
63.02
GraphLoG + TDL
66.7
57.3
60.9
67.3
50.0
68.4
63.6
77.5
63.96
GraphLoG TDL1.D
56.9
61.7
58.2
53.1
63.8
78.2
63.92
Table 9: ROGI scores, lower score means smoother embedding space in terms of downstream labels.
ClinTox
Bace
BBBP
Sider
GraphCL
0.4496
0.7883
0.7033
0.6863
GraphCL + TDL
0.4549
0.7623
0.6973
0.7213
JOÀO
0.4675
0.7900
0.7344
0.7320
JOAO TDL
0,4668
0.7673
0.7284
0.7284
SimGRACE
0.4722
0.7919
0.7874
0.7308
SimGRACE + TDL gaomm
0.4594
0.7635
0.6989
0.7004
GraphLoG
0.4456
0.7480
0.6908
0.6815
GraphLoG TDL
0.4789
0,8161
0.7167
0.7320
Alignment Analysis. To evaluate alignment, construct positive and negative molecule pairs. Positive pairs in dataset are defined as those sharing identical molecular properties, while negative pairs exhibit differing properties. We randomly select 10k positive and negative pairs of molecular graphs from the Bace, BBBP, Clintox, and Sider datasets. Subsequently, we compute the normalized Euclidean distance of the embeddings and illustrate the results via histogram, as depicted in Figure and Figure 10 We apply TDL on top of GraphCL, SimGRACE, JOAO, and GraphLoG to represent the impact of TDL on the baselines.
The normalized curves demonstrate significant differences in the embedding space among different models in the distance dimension. It observable that TDL refines the distribution of distances to more closely align with that PIs. The distinction between TDL1>D and TDL explains why TDI -atom does enhance JOAO because JOAO's curve shows that it is in fact possible get such an atom PIs' distribution. Furthermore, JOAO reveals that additional domain knowledge beyond atom be effectively incorporated via TDLTaDD
C Additional Linear Probing
We continue with linear probing for molecular properties using the more challenging activity cliff dataset. Table 10 we show that TDL. often yields improvements.
D Additional Benchmark Results
Primary Results. The main results on 8 molecular property prediction tasks are listed in Table 11 Note that all SSL methods TTb pre-trained on the same dataset based on ZINC15. We observe that the average performance of GraphCL+TDL has surpassed these existing SSL methods.
19

---

GraphCl, Bace
SINGRACE, Bace
GraphLoG, Bace
JOAO, Bace
GraphCL, 686P
SimCRACE. 868P
GraphloG. 866P
JOA0. 863P
GrsphCL, Clintox
SimGRACE, Clintox
GraphLoG Clintxx
JOAD. Cintax
10
GraphCL, Slder
SIMGRACE, Sider
GraphloG, Slder
JoAD. Sider
Figure 9: Evaluating alignment in the embedding space influenced by TDI -atom- Dark/light shades denote positive/negative pairs, respectively. Blue illustrates the embedding of baselines, red depicts the embedding of Taaelines+TDDaaanee and green represents plato-
Table 10: Linear evaluation of molecular property prediction using opioids-related datasets in classification task, repeated 30 times with 30 different seeds, ROC-AUC (%).
MDRI
CYP2D6
CYP3A4
Dataset
ROC1 1
PRC
ROC f
PRC
ROC
PRC
GraphCI.
0.832 (0.)
0.674 (0.256)
0.512 (0.205)
0.019 (0.017)
0.763 (0.147)
0.327 (0.258)
GraphCL- + TDI
0.928 (0.081)
0.736 (0.241)
0.590 (0.251)
0.032 (0.043)
0.817 (3.)
0.296 (0.183)
MOR
DOR
KOR
Datasel
ROC 1
PRC 7
ROC
PRC1
ROC
PRC
GraphCI.
0.789 (0.060)
0.595 (0.108)
0.730 (0.065)
0.382 (0.091)
0.801 (0.048)
0.593 (0.099)
GraphCL + TDL
0,797 (0.050)
0.601 (0.109)
0.760 (0.065)
0.432 (0.117)
0.802 (0.063)
0,598 (0.105)
Additionally, also apply TDL on both AD-GCL and iMoICLR, with results in the table showing that TDL still provides good improvements.
Individual Tasks. In Table we observe that SimGRACE+TDLL performs poorly on Sider and ClinTox. To learn about the influence of multiple tasks, we run SimGRACE and GraphLoG on individual tasks of Sider and ClinTox. Figure 12 shows increases for TDLatom- We hence conclude that the task correlation exploited by the models considerably during fine-tuning in the multi-task setting, such that the improved SSL representation (which likely does not fit every downstream task under consideration) provided by TDL has less impact.
Activity Cliff Dataset. We also evaluate our methods using the opioids-related datasets. We fine-tune GraphCL and GraphCL+TDL and observe that GraphCL+TDL. outperformes GraphCL, as shown in the Table [22 This result is interesting in itself since it is often argued that SSL based on similarity in "just" the geometric space (i.e., ignoring domain knowledge about, e.g., matching pairs or activity cliffs) may yield problems in practice. However, here, observe that increased power of SSL
20

---

Graphei, Bace
SIMGRACE, Bace
GraphLoG, Bace
JOAO, Bace
GraphCL. 686P
SimORACE. 868P
GraphioG. 866P
JOA0, 8B3P
GrsphCL, Clintox
SimGRACE, Clintxx
GraphloG Ciintx
JOAD. Cintax
GraphCL, Sider
SIMGRACE, Sider
GraphloG, Sider
JOAD. Sider
Figure 10: Evaluating alignment in the embedding space influenced by TDLTED-- Dark/light shades denote positive/negative pairs, respectively. Blue illustrates the embedding of baselines, red depicts the embedding of Trainnss++DLLDDDD.. and green represents PSDD
(a) GraphCL
(b) GraphLoG
Figure 11: Singular values of covariance matrices of the representations; Pre-training datasets.
representation helps to increase the performance of existing methods, even on AC data likely also because they have issues that still need to be resolved, such the dimensional collapse observed.
Additional Pre-training Data. Additionally, we validate the effectiveness our methods on another pre-training dataset, GEOM (50K) [Liu et al. [022a]. For evaluation, study TAE and apply TDL on GraphCL and JOAO. follow all pre-training and fine-tuning settings [Liu et al. 2022a|, with the only difference being that we pre-train the results from their original paper. As shown Table 13, TAE alone reaches decent average performance, and surpasses SOTA ClinTox and HIV. And report the performance in regressive property prediction in Table 14 which shows that TDL enhances GraphCL and JOAO on many datasets. Please note that considered these experiments only obtain initial estimates; the GEOM data (which we used to be able to compare to related works) itself is too small for real pre-training the basleline models
models for 200 epochs instead. We compare
able exploit the 3D nature of the conformers.
21

---

Table 11: Results for 8 molecular property prediction tasks on ZINC15. For each downstream task, we report the mean (and standard deviation) ROC-AUC of 10 seeds.
Tox21
ToxCast
Clin'Tox
MUV
Average
Molecules
7,831
8.575
1.427
1,478
93,087
41.127
2,039
1,513
No pretrain
74.6 (0.4)
61.7 (0.5)
58.2 (1.7)
58.4 (6.4)
70.7 (1.8)
75.5 (0.8)
63.7
(2.4
67.15
InfoGraph
73.3 (0.6)
(0.4)
58.7 (0.6)
75.4 (4.3)
74.4 (1.8)
(0.9)
68.7 (0.6)
(2.3)
70.10
GraphLoG
(0.6)
(0.6)
(0.8)
70.1 (4.6)
(1.6
76.1 (0.8)
69.6
(1.0)
71.43
JOAQ
(0.3)
(0.5)
(0.8
81.3 (2.5)
(1.4)
76.7 (1.2)
70.2
(0.5)
71.89
SimGRACE
(0.3)
(0.7)
(0.9)
75.5 (2.0)
(1.3)
(0.6)
71.2 (1.1)
72.9
71.15
GraphMAE
(0.2
60.5 (0.3)
76.5 (3.0)
76.4 (2.0)
(6.6
(1.5
72.30
MGSSL
(0.6)
(0.5)
(1.0)
77.1 (4.5)
77.6 (0.4)
75.8 (0.4)
68.8 (0.6)
(8.9)
72.28
TDLato
(0.5
62.1 (0.5)
62.2 (0.9)
79.1 (3.8)
75.2
76.9
(1.8)
(1.1
72.02
TDL
(0.3)
(0.5)
61.5
80.7 (2.9)
(1.4)
(0.6
70.5 (0.9)
(0.6
73.02
EdgePred
(0.6)
64.1 (0.6)
60.4 (0.7)
64.1 (3.7)
74.1 (2.1)
(1.0)
67.3 (2.4)
79.9 (0.9)
70.27
TAE EdgePred
(0.4)
63.8 (0.5)
60.1 (0.7)
72.2 (2.4)
(1.7)
(1.0)
68.1 (0.8)
81.5 (1.4)
71.95
AttMaask
76.7 (0.4)
(0.5)
71.8 (4.1)
(1.4)
64.3 (1.1
TAE + AitMask
(0.2)
64.0 (0.3)
$8.2 (1.3)
(3.7)
77.2 (2.1)
68.7 (0.7)
(1.1
71.60
ContextProd
75.7 (0.7)
63.9 (0.6)
60.9 (0.6)
65.9 (3.8)
(1.7)
(1.0)
68.0 (2.0)
(9.6(1.2)
70.89
TAE ContextPred
(0.5)
63.2 (0.4)
62.0
74.6
77.7 (1.6)
68.9 (1.1)
80.7 (1.6)
72.53
GraphCL
(0.7
(0.6)
60.5 (0.9)
(2.7)
(2.7)
(1.2)
69.7 (0.7)
(1.4)
70.78
GraphCLL+ TDLamm
(0.4)
61.2 (0.3)
(2.7
75.7 (0.8)
(0.9)
70,9 (0.6)
80.5
73.71
AD-GCL
(0.8)
63.2 (0.7)
79.7 (3.5)
(1.6
(0.9)
(1.00
78.5 (0.8)
72.67
AD-GCL + TDDseo
(0.1
64.5 (0.6)
60.9 (1.3)
79.4 (2.8)
(4.77..7)
77.4 (0.9)
(1.1)
82.1 (1.4)
73.21
iMoICLR
(0.1
63.5 (0.4)
(1.9) (1.0)
(2.0
(1.2
(1.6
77.3 (1.0)
72.24
iMoCCLR TDI
(5.9
63.7 (0.3)
60.7 (0.8)
(1.9
(1.3
76.7 (0.7)
71.9
78.5 (1.3)
73.20
(a) Sider
(b) Sider
(c) ClinTox
(d) ClinTox
Figure 12: Performance on individual tasks of the multi-task data. We report the mean ROC-AUC of 10 seeds.
Table 12: Results for molecular property prediction using opioids-related datasets in classification tasks, repeated 30 times with 30 different seeds, ROC-AUC (%).
MDRI
CYP2D6
CYP3A4
Datasct
ROC 1
PRC
ROC f
PRC
ROC
PRC
GraphCI.
0.912 (0.152)
0.766 (0.21
0.558 (0.216)
0.079 (0.143)
0.803 (0.155)
0.310 (0.224) 0.418 (0.245)
GraphCL + TDLaom
0.934 (0.126)
0.770 (0.233)
0.565 (0.261)
0.082 (0.1
0.815 (0.159)
MOR
DOR
KOR
Datasct
ROC 1
PRC ?
ROC 1
PRC 1
ROC
PRC
GraphCL
0.861 (0.040)
0.738 (0. 105)
0.804 (0.038)
0.532 (0.076)
0.859 (0.044)
0.701 (0.085)
GraphCL + TDL
0.879 (0.036)
0.741 (0.093)
0.821 (0.049)
0,566 (0.092)
0.861 (0.041)
0.718 (0.058)
22

---

Table 13: Results for molecular property prediction tasks on GEOM. For each downstream task, report the mean (and standard deviation) ROC-AUC of seeds with scaffold splitting. Note that GraphMVP uses 3D geometry (conformers), while other models do not.
Tox21
ToxCast
Sider
ClinTox
MUV
HIV
BBBP
Bace
Average
# Molecules
7,831
8,575
1,427
1,478
93.087
41,127
2,039
1,513
No pretrain
74,6 (0.4)
61.7 (0.5)
58.2 (1.7)
58.4 (6.4)
70.7 (1.8)
75.5 (0.8)
65.7 (3.3)
72.4 (3.8)
67.15
InfoCiraph
73.0 (0.7)
62.0 (0.3)
59.2 (0.2)
75.1 (5.m)
(1.5)
(1.8)
69.2 (0.8)
73,9 (2.5)
70.10
EdgePred
74.5 (0.4)
60.8 (0.5)
56.7
(6.2)
(1.6)
64.5 (3.1)
65.64
ContextPred
73.3 (0.5)
62.8 (0.3)
(1.4)
73.7 (4.0)
(2.5)
71.2 (1.1)
78.6 (1.4)
70.89
AttrMask
74.2 (0.8)
62.5 (0.4)
60.4 (0.6)
(9.6)
73.9
(1.3)
70.2 (0.5)
77.2 (1.4)
70.16
GPT-GNN
75.3 (0.5)
62.2 (0.1)
(4.2)
(3.1)
76.1 (2.3)
77.5
68.27
GraphLoG
73.0
(2.3) (0.4)
(1.8
(1.7
73.4 (0.6)
(1.7
78.8 (0.7)
68.47
G-Contextual
(0.3)
(0.3)
58.4 (0.6)
(9.2)
(2.9
(0.9
(1.3
79.2 (0.3)
69.21
G-Motif
73.2
(0.5)
(0.6
(2.0)
(2.3
(1.4)
(3.4
73.4
70.14
GruphCL
(0.3)
62.8 (0.2)
(1.3
(4.2)
(1.0
(0.4)
68.7 (3.3)
70.64
JOAQ
(0.7)
62.7 (0.6)
66.3 (1.0
(2.0
(6.6
(6.6
72.9 (2.0)
69.57
GraphMVP
62.7 (0.4)
(1.3
(2.5)
74.8 (L.0
68.5
76.8 (1.I)
71.69
GraphMVP-C
(0.2)
63.1 (0.4)
(1.9
(4.2)
75.0
(1.2)
(2.((1))
81.2 (0.9)
73.07
TAF
74.9 (0.3)
62.3 (0.3)
60.2 (0.9)
(2.5)
(1.2(14)
(1.2)
70.9 (3.3) 78.4 (1.6)
72.18
Table 14: Results for 4 molecular property prediction tasks (regression) on GEOM. For each downstream task, we report the mean (and standard variance) RMSE of seeds with scaffold splitting.
ESOL
Lipo
Malaria
CEP
Average
No pretrain
1.178 (0.044)
0.744 (0.007)
1.127 (0.003)
1.254 (0.030)
1.07559
AttrMask
1.112 (0.048)
0.730 (0.004)
1.119 (0.014)
1,256 (0.000)
1.05199
ContextPred
1.196 (0.037)
0.702 (0.020)
1.101 (0.015)
1.243 (0.025)
1.06059
GraphMVP
1.091 (0.021)
0.718 (0.016)
1.114 (0.013)
1.236 (0.023)
1.03968
TAEa
1.129 (0.019)
0.723 (0.006)
1.105 (0.014)
1.265 (0.018)
1.05552
JOAO
1.120 (0.019)
0.708 (0.007)
1.145 (0.010)
1.293 (0.003)
1.06631
JOAO + TDL
1.086 (0.013)
0.713 (0.004)
1.120 (0.007)
1.265 (0.008)
1.04613
GraphCL
1.127 (0.023)
0.722 (0.004)
1.143 (0.015)
1.317 (0.010)
1.07725
GraphCL.+ + TDI catom
1.082 (0.008)
0.707 (0.009)
1.113 (0.007)
1.294 (0.011)
1 1.04943
23

---

GrghiaGGIDD...m Cripllai NITURRATTTTTI....
GrathaaG-ID....
Crppliac 1040
SIOCRACE
SIuCRACE
1040
05
0.5
0.5
0.7
0.6
0.0
0.5
0.4
0.4
100:00
-00
800
1.000
100300
1.000
100:000
-00
800
1.000
100030
1,000
Tuining Samples
Training Sumples
Training Samples
Traising Sumples
GryACCL-TDl.an
CegghLaGTTDl...
GrepCCL uoe
GrgttLeG
CoupaCL
Guzttte
KIAD-1I11 a..
SimtarAU3:
JOAD
XGAD
07
0.
0.55
0.50
0.00
0.5
0.5
01.5
100/20
60C
800
1,002
100200
4.0
500
1,000
100200
400
500
1,000
100250
400
500
1,000
Tuiining Samples
Traiting Sumples
Traising Sumples
Training Sumples
GIyACE.TDX....
Gruptt LGHTDL
PeppCC--TLL.. Coypect
GoghlaG-TDD
Goghes,
Goyhi aG
Gogalat
SuXGRACE ITDL
/OND/TBLL..
SiNGRRACE
JOAD TD
0.55
D.GG
6.7
0.6
01:0
0.6
0.56
0.55
0.5
0.5
7.000 2,00 8,00 4,000 5,200 5,000
1.00 2,000
4,000 4,000 5,000 5,000
1,000
2,000
4.000 00000
1,000
2,000 5,00 4000 5,000 6,000
TTtiiiing Samples
Triinig Samples
Traming Samples
Triining Somples
-- Le
Cipptlotis Grggh oft XAAD (DDD XAAD
Coppee
Goghl oG SDLOT XMM
Gogpect
SIGGRACE: IDL... StaGRACE
SINGRACE: /TT IDD... SinGRACE
0.5
0.7
0.7
0.7
0.6
0.6
0.65
0.5
0.306
0.305
3.99
4
7.0
7.41
TTaining Samples
TTaining Sumplix
-19f
TTnining Simples
T'Triining Sumplax
Figure 13: Performance over smaller datasets. Each datapoint is the mean ROC-AUC of 10 different splits while keeping scaffolds, with standard deviations shown as error bars. From the upper left to the lower right: Bace, Clin Sider, BBBP, Tox21, ToxCast, HIV, MUV.
24

---

0- GopicT-TDi
0
GepiCT-TTi
Gupple TDLaao
GrgAeL TDLsso
SintacALE51
0.5
0.7
0.0
0.G
0.5
800
1,002
100/00
-00
200
1.000
100200
1,00
Tuining Samples
Trinsing Samples
Training Samples
Traising Samples
Grapht + 1D
Ooppiaatio
Graght TTI
Goght aG-im
Cropect -1T1
Gropht Tfi-TT
SuGGRAEE
SiuGRACCE
XGND
SLUGRACE
JGMD TDlaco
0.11
0.6
0.50
0.4
*Tmiiiin Samples
'Inining Samples
Praining Samples
Training Somples
Figure 14: Performance over smaller datasets. Each datapoint is the mean ROC-AUC of 10 different splits while keeping scaffolds, with standard deviations shown as error bars. From the upper left to the lower right: Bace, BBBP, Clintox, Sider.
E
Low-Data Scenario
Here we perform a more detailed experiment to illustrate how model performances change with increasing numbers of training samples. We fine-tune multiple models (GraphCL,, JOAO, GraphLoG and SimGRACE) and TDL. versions on 8 molecular property prediction datasets with training sets of different size. The experiment of each low-data size is run with 10 different subsets of the original training sets while keeping scaffolds. Figure 13 and Figure [4 show that TDL often yields remarkable improvements. Specifically, we observe that TDL's impact seems to depend on two factors: (1) in how far it is able to resolve issues of the baseline (c.g., the dimensional collapse GraphCL), and (2) in how far the PIs used for TDL "suit" the dataset considered (see also Figure [8.. We observe that the latter yields a task which needs to be solved together with domain experts (i.e., finding the best knowledge use to create Pls).
25

---

F
Unsupervised Learning
Datasets. In the unsupervised context, we adopted various molecular benchmarks from TUDataset [Moris et al. 2020), including NCII, MUTAG, PROTEINS, and DD. Table 15 shows statistics for datasets.
Table 15: Summary for molecular datasets from the benchmark TUDataset.
Dataset
# Molecules
# Class
Avg. # Nodes
Avg. # Edges
NCI1
4,110
2
29.87
32.30
MUTAG
188
2
17.93
19.79
PROTEINS 1,113
2
39.06
72.82
DD
1,178
2
284.32
715.66
Configurations, For the unsupervised graph classification task, we first train a representation model contrastively using unlabeled data, then fix the representation model and train the classifier using labeled data. Following GraphCL (You et al. 2020]. we employ a 5-layer GIN with a hidden size of 128 as our representation model and utilize an SVM as our classifier. The GIN trained with a batch size of 128 and learning rate of 0.001. Regarding graph representation learning. models are trained for 20 epochs and tested every 10 epochs. We conduct a 10-fold cross-validation on every dataset. For each fold, we utilize 90% of the total data the unlabeled data for contrastive pre-training and the remaining 10% as the labeled testing data. Every experiment is repeated 5 times using different random seeds, with mean and standard deviation of accuracies (%) reported. We apply TDL on GraphCL. and SinGRACE. The temperature parameter t for TDL here is chosen from {0.1, 0.2, 0.5, 1.0, 10.0). Besides adding a TDL to the loss function in CL approaches, did not make any modifications to the models.
Results. Table 16 displays the comparison among different models for unsupervised learning, revealing that general performance improvement is achieved with the TDL. We also compute the normalized Euclidean distance of embeddings from these unsupervised datasets, and the histogram is illustrated Figure 15 The findings are consistent with our prior observations transfer learning; however, the impact may not be substantial due to the smaller number of training samples.
Table 16: Results of unsupervised learning. For each datasets, we report the mean (and standard deviation) accuracies of 5 seeds.
NCI1
MUTAG
DD
PROTEINS
GraphCL
77.87 (0.41)
86.80 (1.34)
78.62 (0.40)
74.39 (0.45)
GraphCL + TDL
80.06 (0.37)
89.12 (0.79)
79.88 (0.47)
75.59 (0.48)
SimGRACE
79.12 (0.44)
89.01 (1.31)
77.44 (1.11)
75.35 (0.09)
SimGRACE TDL
80.08 (0.31)
89.47 (1.09)
79.01 (0.84)
75.39 (0.57)
G Ablation Studies
Different Filtrations Geometry. Firstly, we consider detailed ablations of the filtration functions used in the graph filtration step of TAE That is, use individual PIs (i.e., each PI is/was constructed based on a separate filtration function) instead of concatenation as reconstruction targets for TAE Consequently, we obtain individual TAEs based on different PIs. We also conduct an experiment that involves integrating these 3 TAES during the fine-tuning of downstream tasks. More specifically, we individually fine-tune these models, concatenate their embeddings after the readout, and finally project through a projection head for downstream task prediction. As demonstrated in Table [7] concatenation during fine-tun mne-tuning (FT) does indeed combine the advantages of individual filtrations. These filtrations are very generic and the method is flexible in terms of what is useful downstream. That is, this approach would allow domain experts to flexibly use their own filtrations in combinations and together with existing methods, in dependence on the specific downstream data. This also suits the knowledge.
fact that specific downstream data may benefit from specific domain
26

---

GraphcL, PROTEINS
GraphCL, MUTAG
GraphCL, DO
GraphCL NC11
SiMGRACE, PROTEINS
SimGRACE, MUTAG
SimGRACE, DD
SimGRACE NCI1
Figure 15: Evaluating alignment in unsupervised datasets. Dark/light shades denote positive/negative pairs, respectively. Blue illustrates the embedding of baselines, red depicts the embedding of baselines+TDLuaum and green represents Platom-
For TDL, the filtration based on atom number provided overall best performance; we ran all exper- iments on PIs concatenated as above as well. We plan investigate more this direction in the future.
Different Filtrations Domain Knowledge. Furthermore, show results about incorporating different kinds of domain knowledge, here, atomic radius and electron affinity, as filtration functions. For TAE, we conduct experiments: (1) We pre-train TAE using individual PIs generated by atomic radius and electron affinity as reconstruction targets; (2) We concatenate these PIs (with the previous Pls) reconstruction targets. For TDL. we use these fingerprints and apply TDL on GraphCL. Table 17 shows that these two domain knowledge-basee filtration functions achieve commendable average performance. Yet, in summary, we do not observe considerable differences. Our hypothesis is that, similar to the atomic number, these kinds of domain knowledge capture the difference between the atoms, and hence not too different overall, Based on this investigation, we plan to experiment with more complex sources of domain knowledge (e.g.. the polarity of atoms, which important information for the possible relationships between atoms).
individual PIs topological
Different Filtrations Multiple Dimensions. Here we consider the multidimensional Vietoris-Rij the atomic number, heat kernel signature with I = 0.1 and node degree as filtration functions in the first dimension, respectively. Then we compute PDs and then vectorize them get 3 Pls. Finally, concatenate these PIs reconstruction targets to pre-train TAE.
(multi-VR) filtration (Demir [022]_ And
Please observe first that our study's (initial) research question has been the exploration of the usefulness PH and topological fingerprints for molecular SSL. Based on our investigation (see also Figure 8) there one very interesting outcome of our research and follow-up research question related to TDL: TDL is an instance a more general proposal "XDL", a novel loss function for pre-tnain which is exploits distances between training samples. We conducted some initial experiments only using the Tanimoto distance between standard ECFP fingerprints and obtain decent results; see Table 17 In future research, will investigate the generality and potential impact of XDL more (e.g.. combining ECFPs and topological fingerprints).
No Filtrations.
Different Vectorizations of PDs. Persistence Vectorizations transform the obtained Persistent Homology information (PDs) into either a ffunctional or a feature vector form. This representation more suitable for ML tools compared to PDs. In study, we employ Persistence Images for vectorization. Apart from this, we also explored other stable vectorizations for TAE, such as Persistence Landscapes [Bubenik al. 2015] and Silhouettes [Chazal al.] 2014]. As shown in Table 18 ablation experiments demonstrate that Persistence Images outperform the other two vectorization techniques.
Hyperparameters Analysis. We further perform a sensitivity analysis on the parameter of GraphCL + TDL1aDD across 8 downstream molecular tasks. The results presented in Table 19 It
27

---

is noteworthy that as À decreases, the weight of TDL also decreases, leading to noticeable decline performance. This observation underscores the effectiveness of TDL.
Table 17: Ablation study results for 8 molecular property prediction tasks on ZINCI5. For each downstream task, we report the mean (and standard deviation) ROC-AUC of 10 seeds.
Tox21
ToxCast
Sider
ClinTox
MUV
IIIV
DDDP
Bace
Average
No
61.7 (0.4)
(1.7)
58.4 (6.4)
70.7
75.5 (0.8)
66.7
72.4
67.15
TA
(0.8)
63.1 (0.3)
61.9 (0.8)
(1.6
(1.6
(2.1
(1.1
82.5(1.1)
72.36
74.6
61.8
62.5 (0.7)
(3.8
75.7 (2.1)
(0.8)
(1.0
(1.3
70.84
TAE
75.5 (0.3)
62.7 (0.3)
(0.6)
(1.4() (3.0)
(1.2
68.6 (0.5)
82.5 (0.8)
72.08
TAFagge
(0.6
62.8(0.5) (0:5)
(0.8)
(0.0
(2.0
73.4(1.)
68.1 (0.8)
81.9 (0.6)
71.34
75.7
(0.4)
(2.9
(1.1)
(0.8)
(1.1)
TAExmei
76.5 (0.6)
63.3 (0.7)
(1.4
76.2 (2.6)
(1.7
(1.3
(1.3)
(1.7)
71.71
TAEBeemo dfiec
64.0 (0.7)
(1.9)
68.9 (1.6
(1.3
72.55
of
73.9 (0.4)
63.8 (0.3)
62.5 (0.6)
(2.4)
(1.6
(4.9
66.6 (0.9)
(1.2)
72.86
TAPasaiv tluaice
75.8 (0.6)
63.4 (0.6)
(0.7)
76.8 (2.6)
75.1 (1.2)
76.4
(9.9
(1.3
72.64
TAEnan
76.8 (0.9)
64.0 (0.5)
61.9 (0.8)
79.3 (3.6)
75.8 (3.2)
(1.1
70.4 (0.8)
(1.6
73.22
GraphCL
(0.3)
62.8 (0.2)
60.1 (1.3)
(4.2)
77.1 (1.0)
(0.4)
68.7 (3.3)
70.64
GrapbCL TDLaam
75.3 (0.4)
64.4 (0.3)
(0.6)
83.7 (2.7)
75.7 (0.8)
(0.9)
70.9 (0.6)
80.5 (0.8)
73.71
GraphCL. TDD- TDL-Muik
74.3 (0.3)
63.2 (0.3)
61.3 (0.4)
85.5 (1.5)
77.6
77.33
70.7 (0.6)
(1.3)
73.02
GraphCL TDLaseme ammr
75.1
(0.4)
(3.0)
(1.4
(1.9)
(0.6)
GraphCL+ + TDL7.0
75.2 (0.7)
64.2 (0.3)
61.5 (0.4)
85.2 (1.8)
(2.9
77.9 (0.8)
69.9 (0.9)
81.2
73,88
GraphCL TDL
64.0 (0.4)
(0.6)
82.6
(1.1
(0.9)
70.1 (0.7) (1.7)
72.85
Table 18: Comparison of the different vectorizations.
Tox21
ToxCast
Sider
ClinTox
Average
TATla (Pensistence Images)
75.2 (0.8)
63.1 (0.3)
61.9 (0.8)
80.6 (1.9)
74.6 (1.8)
(2.1)
67.5 (1.1)
82.5 (1.1)
72.36
TAF (Persistence Landscapes)
75.6
63.9
(0.)
(1.4)
69.7 (1.6)
79.8
TAE (Silhoacties)
74.7 (0.6)
63.4 (0.6)
58.4 (0.7)
(2.4)
72.2
75.8
68.9 (0.9)
79.4
70.88
Table 19: Sensitivity analysis on the parameter X.
GraphCL. TDLma0
Tox21
ToxCast
Sider
ClinTox
MUV
HIV
BBBP
Bace
Average
0.1
74.2 (0.8)
63.1 (0.2)
60.8 (0.6)
76.4 (5.8)
73.3 (1.7)
77.2
70.2 (0.2)
71.6 (0.9)
70.85
0.5
74.0 (0.5)
63,2
60.0 (0.4)
(0.1)
(1.4)
77.2 (0.6)
69.9
(1.7)
71.59
1.0
75.2 (0.7)
64.2 (0.3)
61.5 (0.4)
(1.2
(7.9 (2.1)
69.9 (0.9)
81.2 (1.9)
73.88
A 74.8 (0.3)
(0.3)
61.7 (0.7)
(1.8
(0.8)
71.0 (1.0
(1.8
73.85
Different GNN Backbones. We apply TDL on GraphCI. As shown in Table 20 we verify the
generality of TDL by experimenting with three popular GNN models: GIN [Xu et al.). GCN (Kipf and Wellingl, and GraphSAGE [Hamilton et al. 2017). Pre-training with GIN achieves the best performance across datasets, hence our results confirm what has been observed in related works.
28

---

Table 20: Comparison of the pre-training gains, represented in terms of averaged ROC-AUC (%).
obtained by using different GNN architectures 8 datasets.
Model
GCN
GIN
GraphSAGE
No pretrain 68.77
67.15
68.32
GraphCL
69.32
70.78
70.08
GraphCL + TDLutom
72.53
73.71
71.16
H
PIs w/o SSL
Here provide initial results for comparison with 2 non pre-training ML models, i.c. support vector machine (SVM) and XGB [Chen and Guestrin 2016) to test the regular prediction on molecular property prediction tasks using Pls topological fingerprints. And also conduct smaller data experiments on SVM compared with TAE and TAE*Contex(Pred over 4 downstream datasets. Note that performed hyperparameter search produce these ML results. As can be observed in Table and Figure 16 the SSL. methods provide large benefits. We are certainly aware of the fact that there good hint of how SSL based on PH compares to supervised methods more generally.
complex methods use these Pls, yet this basic comparison gives
Table 21: Results of non pre-training method for molecular property prediction tasks on ZINC15, ROC-AUC (%).
Tox21
Sider
ClinTox
BBBP
Bace
SVM XGB
53.9
53.8
60.1
56.6
72.3
64.7
58.5
55.7
61.6
73.8
ContextPred
75.7
60.9
65.9
68.0
79.6
TA ContextPred
76.4
62.0
74.6
68.9
80.7
CoutettPed TAEar (CantesiPee
Cremestholl TAEai (ootesuPred
Conrstthod TAEa (CottesiPeed
Comoutted at CunzsdPal
0.8
0.5
0.35
0.6
0.3
0.5
Training Samples
Training Samples
Training Samples
*Tmmining Samples
Figure 16: Performance of SVM over smaller datas. Each datapoint is the mean ROC-AUC of 10 different splits while keeping scaffolds, with standard deviations shown as error bars. From the left to the right: Bace, BBBP, ClinTox, Sider.
I Theoretical Analysis of TDL
In this section, we delve into a theoretical analysis TDL, which was initially introduced in Section 3.2] We encourage readers refer to Section 3.2 for any specific notation, terminology, or formulation details.
Recall TDL is defined as follows:
exp (sim (Zns 2m //)
LTDL,
Σ
log
-
EkELL, k
exp (sim (2n, 2x) /T)
mE ME,NN misin
29

---

For the sake of convenience, we re-index the subscripts of samples, ensuring that the subscripts d; satisfy the sorted distances from the n-th sample, i.e., dis In) S Id, S dis(I,, 412) <
dis(T, (w., Idg) S ... S dis(In, In, Idy Let sim (Zn; Fm) = Zm to streamline the derivations. We
thus obtain
el (a
LDDL,,
Σ
log
N
ELELAN (3)
JEn,N-
We then analyze the gradients with respect to different samples.
Lemma 1, The gradient of TDL,,, with respect to the sample d, in latent space is formulated as:
1
ezn
Zn
N V ed, CTDL,
1j o
T
JELL
In particular, Rdj LTDL, x -nn and V xan= CTDL.. & mm
(snina,1-) Exerin (a
Proof. The results follow from direct computation. Let CTDLm.4, then we have:
log
el (30
LTDL N
Σ
log
Σ
LTDLm.4;
ExELiN- ol (be
N
JE[I,N-1]
JE(L,N-]]
Next, we discuss V zn. LIDIM.az
i<j
4 i=j -1
V xa, LIDLO
22 E1ElAN--11 24
i>j
Therefore, we conclude:
2n
V Vaa CTDL..
LTDLM.d;
Ekeli conitaall 1
1)
N
ad;
N-1
JE(I.N-I]
JE[L,)]
Then we have:
eF /T
zd, LTDL,, N
1)
2n
Σ EKElLNN-11 e BoyT
Fn /T
The quantity EnecIN- -d the derivative V Vz,, LIDL,, proportional - Similarly, we have:
< 0 is a strictly negative scalar, allowing us to conclude
sax=1 LIDL, N
Σ
Zn
ΣkELi @fni
JE(I,N-2 -2
Since Σ EeglAN 11 "n V Edg LTDL., points in the direction 2,.-
0 we conclude in this case that the derivative
30

---

The stability provided by certain topological fingerprints serves as mathematically grounded. well- studied, and efficient proxy for stability with respect to graphs. TDL as our objective based on the stability of these topological fingerprints. Consequently, for each sample, we can determine ranking of all other samples. Lemma I provides the gradient with respect to different samples. Specifically, samples that are distant from sample in the topological space point in the direction En (negative gradient direction), while those close to sample n point in the direction 2n- Consequently, molecules with more similar topological features are brought closer to each other in the embedding space.
J Potential of PH for Molecular ML
Persistent homology (PH) has emerged as a crucial method in topological data analysis, demonstrating significant potential in decoding complex patterns within molecular data, including protein prediction Cang and Wei. 2017a b Nguyen et al. [211 Xia and Wei [0144] Liu et al. [022b]] virtual screening CCang al. 2018 and drug design [Liu et al. [021]] The many works on PH in chemistry show
that the domain is convinced by the usefulness of topological fingerprints already, and we therefore
believe that this kind of knowledge should be investigated in the context of SSL, assuming that the
latter becomes more important with the advancement of foundation models.
K Limitations & Broader Impact
Broader Impact. Graph data has a pervasive impact in many diverse fields and the popularity certainly increases the danger of an application of the models with negative effects (even if just due to ignorance). As pointed out in this work w.r.. smaller data, we have to be aware of the assumptions we make in our research to inform readers how the results translate to the real world. Our goal to advance biomedicine, chemistry, and material science, where our research hopefully provides benefits one day, for both the current society and future generations.
Limitations. We have tried our best in addressing common issues, by providing very detailed analysis, by considering TDL on top of various baselines, by also reporting mixed results, and by providing all code. However, due time and resource constraints, and the broadness of our study, we were not able to experiment with a variety of topological fingerprints, analyze individual baselines (e.g., the dimensional collapse) more detail, or complete the theoretical investigations.
31


Reviews:
[{"review_id": "mNcpXxKDks", "reviewer": "Reviewer_bcsi", "summary": "This paper explores self supervised learning in the context of molecular representation, specifically based on persistent homology. The paper proposes an autoencoder to demonstrate the general representational power of PH and a contrastive-learning-based loss that can be applied to existing SSL approaches. The proposed approach is evaluated for molecular property predictions, showing improved representations and predictive power compared to baselines across different tasks. The claim is that the new loss function enhances baseline performance particularly with small datasets.", "questions": "- Since PH naturally offers multiple data views, I wonder whether the learned representations can be more explainable, that is, not just focusing on atom based attribution but also having explanations of more global features/subgroups, for instance, along the lines of \nBertolini et al. \"Beyond Atoms and Bonds: Contextual Explainability via Molecular Graphical Depictions\". ", "limitations": "The authors describe the limitations of their approach. ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and the idea is novel and interesting. ", "weaknesses": "- Given the technical nature of PH, and its origin in the domain of topological data analysis, a more mathematical foundation of the methods in the paper would be desired. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the interesting comments!**\n\nThe paper does not explicitly discuss the points mentioned in the review because these are challenging topics in themselves, but they nicely underline the future research potential and we are happy to discuss them further.\n\n**W1 More mathematical foundation would be desired.**\n\nWe agree on that and definitely plan to investigate the learnt embeddings from a theoretical viewpoint, but we consider that out of the scope of this current, initial work on PH in the context of molecular SSL.\n\nIt is not straightforward how to leverage the topological distances to obtain stability and generalization results, this deserves deeper study.\nA second topic which we address empirically is the increase in the rank of the embeddings. However, there we observe a strong dependence on the baselines, which will make the study more challenging. Moreover, this direction is only lately studied in SSL in general, with measurement methods presented as recently as at ICML 2023 [1].\n\nWe sincerely thank the reviewer for recognizing that this kind of studies is beyond the focus of our paper and for not letting it influence the score too negatively.\n\n\n**Q1 Explainability**\n\nWe see at least two ways in which topological features may offer particular advantages. First, the filtrations can be designed based on arbitrary domain knowledge, and this knowledge may indeed be re-discovered in the embeddings if both the filtration and the embedding are chosen carefully. For instance, there are certain kinds of porous crystals called MOFs whose pore diameters can be captured by using a combination of VR filtration and PIs [2]; in a nutshell, the filtration iteratively \"aggregates\" the pore-delimiting atoms, a novel topological structure appears once the process is finished, and the length of the aggregation is recorded in the persistence diagram (this example uses 2D topological voids and hence goes beyond the graph homology we introduce, but we think it is very illustrative). Note that for this kind of analysis the model has to be encouraged to learn the underlying topological fingerprint, e.g., in the way TAE learns PIs.\n\nOur CL-based approach uses the fingerprints in a different way, to retain the overall structure of the PH-based embedding space, in terms of distances. This offers a different angle of explainability, reflecting the relations/similarity between molecules. We believe that this is a unique feature of PH-based fingerprints and particularly interesting for SSL in that it directly supports the main goal of pre-training, learning a comprehensive embedding space. Also model calibration (important for interpreting the results) is critically relying on distance awareness [3].\n\n-----------------------------------------\n\n[1] Garrido, et al. \"Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank.\" ICML, 2023.\n\n[2] Krishnapriyan, et al. \"Machine learning with persistent homology and chemical word embeddings improves prediction accuracy and interpretability in metal‑organic frameworks\" Scientific reports 11.1 (2021): 8888.\n\n[3] Liu, et al. \"Simple and principled uncertainty estimation with deterministic deep learning via distance awareness.\" NeurIPS, 2020."}, {"review_id": "l7PsMbmuNM", "reviewer": "Reviewer_9rP5", "summary": "The paper proposes two approaches to leverage topological information (obtained from persistent homology) for molecular representation learning in a self-supervised setting. The first (TAE) uses an encoder-decoder architecture whose decoder aims to recover topological fingerprints. The second approach (TDL) consists of a contrastive loss based on the similarity between topological fingerprints. The latter is combined with existing contrastive learning methods. Experiments on linear probing and downstream prediction tasks show the efficacy of the proposals.\n", "questions": "1. Could the authors elaborate on why the fact that 'the Euclidean distance between PIs is stable with respect to the 1-Wasserstein distance between PDs' (line 111-112) is relevant here? Don't we want stability wrt to the input graphs? \n2. What does the paper mean by calibrated distances?\n3. Can the proposed methods be extended to employ learnable filtration functions?\n4. A significant part of the experiments is devoted to showing the alignment between the learned molecular representations and the topological fingerprints. Isn't it naturally expected from the proposed design (e.g., additional loss term)?\n5. Could the authors elaborate more on the fact that TAE can capture inter-molecule relationships if they learn the PIs? Can't GNN embeddings also learn important structural information and be stable?\n6. Have the authors considered applying only TDL without other loss terms from CL methods? If yes, how well does it work?\n7. The paragraph 'linear probing' (line 281) says 'we evaluated extensively using MLPs on the representations of the pre-trained graph encoders...'. Shouldn't this employ linear models instead of MLPs?\n8. Duplicated text, e.g.,\n    - Lines 1-5 --> 20-23\n    - Lines 24-31 --> 199-206\n    - Lines 5-6 --> 37-38\n9. Typos:\n    - GraphCL propose (line 32)\n    - JOAO extend (line 33)\n    - and to exploit (line 65)\n    - TDL provides as a form of regularization (line 233)", "limitations": "The authors mention limitations in the main paper (section 5).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Ablation studies: There is a substantial number of experiments and ablation studies.\n- I like the simplicity of the proposed approach. \n- Flexibility: TDL can be combined with most SSL approaches.", "weaknesses": "- Overall, I believe the paper provides limited insight to support the proposals. Also, it does not discuss which structural information the proposed approach captures but not existing methods. From a conceptual level, we know that 1-WL GNNs cannot capture information even from simple homology (e.g., number of independent cycles of a graph). Thus, TAE has inherent limits/failures. In other words, the topological information we loose after pushing a graph through a GNN (which would be captured by TDA) cannot be recovered from GNN embeddings.\n- Results on downstream tasks: Based on Table 4, the gains from TDL look marginal. The gain is less than one standard deviation from the base model for many datasets.\n- Incorporation of domain knowledge: The claim that the proposal allows for incorporating domain knowledge seems overstated. The basis for such a claim comes from the choice of the filtration function. However, it is unclear how different filtration functions affect the topological embeddings --- thus, domain experts cannot leverage their knowledge to choose the filtration functions.\n- TAE vs. TDL...which one should we use? The paper says that \"TAE, which we developed for comparison purposes only...\" (line 283). I am unsure whether TAE should be introduced as a main contribution or as a baseline (in the experiments) for assessing the feasibility of learning the topological fingerprints with a simple architecture. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the very detailed feedback!**\n\nWe reformulated the particular benefits of PH for molecular SSL below and also adapted the paper. We hope that this clarifies the initial confusion about our contribution, please let us know in case further details are needed!\n\nW2 and W3 are addressed in the global reply. And for W1 and W3, please also see the reply to recC, Q5.\n\n\n**W1 & W4 Clarification on TAE**\n\nObserve that TAE is encouraged to directly learn the PI via the objective function, hence it does not necessarily have to reproduce the algorithmic procedure. To verify its ability, we ran preliminary experiments applying the pretrained, not finetuned TAE for PI prediction on the evaluation data. The Pearson correlation coefficients between the predicted and real PIs show that it approximates the latter PIs fairly well:\n\n|      | Tox21  | ToxCast | Sider  | ClinTox | MUV    | HIV    | BBBP   | Bace   |\n| ---- | ------ | ------- | ------ | ------- | ------ | ------ | ------ | ------ |\n| TAE  | 0.8572 | 0.7744  | 0.5939 | 0.8642  | 0.9044 | 0.7359 | 0.8660 | 0.8514 |\n\nMoreover, TAE is intended as simple and straightforward baseline for PIs in SSL, in the way the area uses the baselines of Hu et al. as context for interpreting results. In particular, since TDL does not directly encourage the model to learn PIs but rather the relations between them, the comparison to TAE allows us to evaluate the effectiveness of this rather abstract goal. Nevertheless, note that our comparison is only coarse since it is based on numbers (i.e., instead of on the actual predictions). A similar, even closer comparison could be drawn by considering regular CL using the PIs as embeddings, as outlined by Reviewer XtrW.\n\n\n**Q1 & Q5 Relevance of Stability of Topological Fingerprints, and Relation to TAE**\n\nStability w.r.t. the input graphs is definitely the goal, but we need an appropriate metric for the data space. There is no unique such metric for graphs, and we are only aware of few such works (e.g., a recent paper proposes a custom tree-mover's distance [1]). Our paper's hypotheses are that \n\n- the stability certain topological fingerprints offer is a mathematically grounded, well-studied, and efficient proxy for the stability w.r.t. graphs, which we can use complementary to existing approaches;\n- stable representations support the learning of a well-structured embedding space, which is particularly important for SSL;\n- the fact that they are generic in the filtration function gives domain experts the opportunity to flexibly inject their knowledge, which particularly suits molecular representation learning (see also the references mentioned by Reviewer recC). \n\nAltogether, this motivates us to study persistent homology in the context of SSL over molecular graphs.\n\nAssuming TAE learns PI-based representations sufficiently well, then the distances (\"relations\") between its embeddings reflect the ones between the corresponding input graphs in terms of the 1-Wasserstein distance between their persistence diagrams.\n\nAs for example [1] show, stability can indeed be defined for GNN embeddings. Our paper is intended to complement this research by investigating how we can leverage the body of existing works in persistent homology in the context of graph representation learning and, in particular, in molecular SSL.\n\n\n**Q2 What does the paper mean by calibrated distances?**\n\nTDL intends to \"adjust\" the distances between the embeddings such that they better reflect the ones between the PIs. In this sense, they get calibrated. We are aware that the notion is not perfect but did not find a better one. We are definitely open for suggestions! For now, we added this explanation to the paper. \n\n**Q3 Can Filtration Functions be Learnt?**\n\nWhile this should be possible in general, it might be expensive in an SSL setting since the topological fingerprints then have to be reconstructed in each epoch. Furthermore, in the context of our TAE and TDL, the objective functions are not clear.\n\n**Q4 Why Experiments showing Architecture Alignment?**\n\nWhile we indeed designed our architecture as carefully as possible, we do not think that such design intentions or even theoretical architecture guarantees have to necessarily translate into practice (e.g., GIN has been shown to be very expressive, but there are datasets where the conventional GCN is superior [2]). Therefore, we consider these experiments showing alignment of the architecture as considerable contribution, in particular, also because the relation to the topological fingerprints offers various topics for future investigation (see also the response to Reviewer bcsi).\n\n**Q6 What about TDL w/o other Losses?**\n\nGiven that TDL does not incorporate the regular, proven views used in CL, we do not recommend this setting. *We did some such experiments, interestingly obtaining quite good performance, now* (see U3kd, Q1 & Q3 TAE + TDL?). Note that regular CL views could certainly be constructed based on PIs or other topological fingerprints. In our initial study, we chose a different, more novel focus, but we are investigating alternative options now.\n\n\n**Q7-Q9 Minor Comments**\n\nThank you for checking on this level of detail! The \"MLP\" in the context of linear probing is indeed a mistake, we used a simple linear layer. We also fixed the remaining items.\n\n----------------------------------------------------------------------------------------\n[1] Chuang, et al. \"Tree Mover's Distance: Bridging Graph Metrics and Stability of Graph Neural Networks.\" Advances in Neural Information Processing Systems 35 (2022): 2944-2957.\n\n[2] Dwivedi et al. \"Benchmarking graph neural networks.\" JMLR (2023)."}, {"review_id": "i0YAcPYbxW", "reviewer": "Reviewer_U3kd", "summary": "This paper proposes two molecular self-supervised learning methods, which consists of fingerprint autoencoder and topological distance contrastive learning. The insight behind this paper is to utilize topological fingerprint as a supervision in self-supervised learning. Thus, the authors reconstruct the topological fingerprint of a given molecule with autoencoder and filter out similar molecules in negative views in contrastive learning based on the similarity in topological distance space. The experimental results show that their method improves previous baselines in various downstream tasks.", "questions": "- In Table 4, why some baseline methods are combined with TAE while others are combined with TDL? Can't TAE and TDL combined jointly?\n\n- How is the performance of TAE (or TDL) jointly trained with SEGA? Does this improve SEGA itself or SEGA + other method (e.g., SEGA + ContextPred)?\n\n- I would be convinced with the experimental results if the authors compare \"TAE + TDL\" vs other methods (Please refer \"Flexibility of TDL\" in Weakness).", "limitations": "Yes. The authors addressed the limitations.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "- The paper is well written and easy to understand.\n\n- The experimental results are comprehensive; the authors considers several setups such as linear probing and fine-tuning.", "weaknesses": "- Lack of Novelty: Excluding similar molecules from negative sample set is already considered in [1]. Conceptually, the difference of TDL and [1] is that TDL utilizes PH and [1] utilizes ECFP fingerprint (I know that the loss of [1] is based on augmented molecules, but I think this does not make big difference). This limits the novelty of this paper.\n\n- Table 1 does not support the effectiveness of proposed method: Correlating the distance in embedding space with the distance in corresponding PIs are not the main purpose of molecular representation learning. If PIs are indeed very important, then why should we use learned representation of proposed method? Can't we just utilize PIs as the molecular representation? In other words, Table 1 and Table 17 seem to contradict.\n\n- Insufficient rationalization of the usage of PIs: In molecular domain, ECFP fingerprint is a widely applied molecular representation since it reflect the substructure-wise molecular information. Why should we use PIs in molecular representation learning?\n\n- Flexibility of TDL: The authors insisted that TDL can be flexibly and efficient applied with any graph contrastive learning framework. However, any other two existing methods can be composed with each other to improve the performance. For example, ContextPred + GraphCL is possible and the flexibility is not the unique feature of TDL. \n\n- Table 4 seems weak: TDL (or TAE) combined with existing method does improve the overall performance. However, Mole-BERT and SEGA shows better performance than the proposed method.\n\n----Sorry for confusion. I added the reference.\n\n[1] Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast, Wang et al., JCIM 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for pointing out this indeed very related paper!** We should not have missed such closely related work, and we hope that the below delimitation resolves some of the related issues pointed out in your review. In fact, the detailed comparison highlights the novelty of our work.\n\nW3 and Q2 are addressed in the global reply.\n\n\n**W1  Clarification on Novelty: Comparison to [1]**\n\n- **Our paper's focus.** We study the benefits of PH for molecular SSL and propose technology that is suitable for incorporating PH into SSL, rather than just a loss that excludes \"false\" negative examples.\n- **We do not consider views.** The fact that we focus on input examples represents a considerable difference. While the similarity between rather similar molecules, such as views, might be captured by ECFP, the latter turned out to be not as effective as PIs in TDL (Tab. 15). We hypothesize that this is due to the fact that input examples that are overall different but have structural similarities might have too similar ECFPs for a fine-grained distance regularization.\n- **We filter the graphs based on the distances.** We experimented with weighting as used in [1] in the beginning but our method showed better performance in our setting. The weighting might work with ECFP since these fingerprints are discrete and hence rather coarse grained, while it may give too confusing signals to the model with PIs.\n- **Pretraining Data.** [1] used ~10M molecules. We reran their model for a fair comparison. \n- **Results.** With TDL, we obtain similar improvement in fine-tuning and linear probing as with the other models we considered.\n\n    |           | Tox21      | ToxCast    | Sider      | MUV        | ClinTox    | HIV        | BBBP       | Bace       | Average |\n    | --------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ------- |\n    | [1]       | 75.1 (0.7) | 63.5 (0.4) | 59.4 (1.0) | 74.7 (1.9) | 81.0 (2.6) | 77.3 (1.2) | 69.6 (1.2) | 77.3 (1.0) | 72.24   |\n    | [1] + TDL | 75.9 (0.6) | 63.7 (0.3) | 60.7 (0.8) | 75.1 (1.3) | 83.8 (1.9) | 76.7 (0.7) | 71.2 (0.9) | 78.5 (1.3) | 73.20   |\n    | [1]       | 68.8 (0.4) | 60.4 (0.3) | 57.5 (1.1) | 59.3 (2.2) | 73.3 (2.1) | 67.5 (0.7) | 63.8 (0.8) | 7.9 (1.2)  | 65.43   |\n    | [1] + TDL | 69.8 (0.3) | 61.1 (0.4) | 59.0 (0.4) | 61.7 (1.7) | 72.8 (1.0) | 69.6 (0.8) | 64.4 (0.4) | 74.7 (0.9) | 66.64   |\n\n**W2 & W5 On the Demonstration of Effectiveness**\n\nAssuming the filtration functions are carefully chosen and possibly include external knowledge, PIs will likely offer useful features for molecular representation learning (see also the references suggested by Reviewer recC). Nevertheless, our paper does not intend to compete with the existing body of graph SSL research and outperform specific SOTA approaches, but rather *improve* it by exploiting the unique features of PH in a complementary way. This is also why our initial work on the topic focuses on TDL rather than a custom, standalone CL-based loss, where PH could be used for constructing views, which would likely be a better model than the simple TAE. *Table 1 shows that TDL is effective in slightly moving the molecule embeddings in the embedding space towards the structure of the PI space*. Learning both embeddings that are rich in information, to fit multiple possible downstream scenarios, and a well-structured embedding space are especially important in SSL (which has more specific requirements than regular molecular representation learning).  \n\nRe SOTA, please note our discussion of fine-tuning experiments in SSL in the global reply, our results with stronger filtrations, and the improvement we obtain for AD-GCL.\n\n\n**W4 \"Flexibility\" of TDL**\n\nIt is true that other approaches can be combined as well. What we mean is that TDL covers a dimension which is not addressed by the regular models. Hence, \"flexibly\" is intended to convey the fact that it clearly adds a novel form of regularization which is likely effective.\nWe did not want to cause confusion with that wording, which is indeed not crystal clear, we are definitely open to alternative suggestions. \n\n\n**Q1 & Q3 TAE + TDL?**\n\nSince TDL's objective is a direct consequence of TAE's objective we do not expect particular benefits of this combination. In the paper, we only ran TAE in combination with ContextPred to obtain an idea of its performance in such combinations, but this is more intended as a side experiment. In combination with other CL-based approach, TAE's objective would be rather strict in that it enforces direct embedding similarity, which might contradict the other model's objectives. TDL is more general (and in a certain sense flexible) in that it regularizes adaptively. \n\nAs noted above (W2 & W5), standalone TDL is not designed for SOTA comparison since it is clearly missing CL representation power by not considering views. We hope that our explanations provide clarification in this regard and that our additional experimental results help to justify our contribution. In case there is doubt left, please let us know.\n\nFor analysis purpose, we ran the experiments and the results largely match our expectations. Yet, standalone TDL is surprisingly effective. \n|                 | Tox21      | ToxCast    | Sider      | ClinTox    | MUV        | HIV        | BBBP       | Bace       | Average |\n| --------------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ------- |\n| TDL  | 75.8 (0.5) | 62.1 (0.5) | 62.2 (0.9) | 79.1 (3.8) | 75.2 (2.3) | 76.9 (0.9) | 66.5 (1.8) | 78.4 (1.1) | 72.02   |\n| TAE+TDL         | 76.2 (0.3) | 62.9 (0.4) | 60.6 (1.0) | 81.7 (1.8) | 74.2 (1.5) | 76.2 (1.1) | 67.4 (0.8) | 83.0 (1.5) | 72.78   |\n| TAE+GraphCL+TDL | 76.0 (0.3) | 63.7 (0.4) | 62.6 (0.6) | 82.8 (2.3) | 75.4 (2.3) | 77.4 (0.6) | 69.8 (0.6) | 81.8 (0.9) | 73.69   |"}, {"review_id": "hElqVYCC6m", "reviewer": "Reviewer_XtrW", "summary": "Paper uses self-supervised learning tools for graph representation learning by facilitating topological data analysis (TDA) methods. In particular, for molecular representation learning, the authors use persistent homology outputs to improve the embeddings obtained by GNNs. They evaluated their model in molecular property prediction problem, and consistently got performance improvements.", "questions": "Instead of using PH output  in the loss function to improve GNN embeddings, did you consider directly combining them, e.g., by simply concatenating PIs with GNN embeddings? I know this is a completely different approach, but this is a more direct method to combine both outputs.", "limitations": "There are various choices to be made in several places for the model. On one side, it gives flexibility to adapt the model to different settings, but on the other hand, it needs expertise in several domains for fine-tuning.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "GNNs and TDA are both very successful and completely different methods in graph representation learning. In the past years, there are several approaches to integrate these two methods effectively. With this aim, the paper proposes a new way to use TDA output to improve node embeddings in GNNs by using contrastive learning ideas. The idea is novel and has a lot of room for improvement.\n\nMolecular Representation Learning is a significant application area for graph representation learning. The authors applied their model in this domain, in particular, molecular property prediction. They obtain strong results on this important question.\n\nThe paper's experimental part and ML details are strong. The authors made an in-depth analysis of the model from various angles.", "weaknesses": "The experimental results (Table 4) do not show significant improvements in several cases. \n\nThe results only report the performance of internal models. It would be nice to see the comparison with the SOTA results on these datasets. \n\nPH construction seems weak as it does not use clique complexes, and only uses nodes and edges in the filtration, i.e., the top dimension is set to be 1. This filtration are not commonly used in graphs as it reduces PH to only node and edge counting by using a simple Euler Characteristics argument. However, fortunately, this does not affect their performance in this setting since molecular graphs are planar and do not have loops of length 3, as all loops have length $\\geq 5$. The authors should add a note for nonexperts that for molecular graphs, this trivial filtration setting is equivalent to the traditional clique complex setting for sublevel filtration because of the special structures of molecular graphs (no cycle of length 3). For TDL, PI is a good choice, but for TAE, it looks weird. To be used in such a loss function, there are better stable PH vectorizations, e.g., Silhouette, landscape.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the insightful comments!**\n\nW1 and W2 are addressed in the global reply. We hope that especially our explanation there and the additional results clarify our proposal. The suggestion of comparing to SOTA also in our extra experiments (W2) was very helpful and underlines our contribution.\nPlease let us know in case any questions are left!\n\n\n**W3 Clarification on PH Construction**\n\nThis is a good point, and we added that to the paper. We admittedly dropped many details since we were afraid that too much theory on persistent homology may prevent readers and ML researchers from seeing and getting convinced of its most important features. In fact, to the best of our knowledge, the theory is only more recently considered in the field; e.g., in the context of graph representation learning, graph homology has been studied in more detail in [1]. \n\nWe did not intend to propose TAE as an architecture but rather as a means to compare to and obtain and estimate for the usefulness of PIs in an SSL setting, since we apply them for TDL (see also the response to Reviewer 9rP5, W1). We will mention this more clearly.\n\n\n**Q1 What about Concatenating PIs with GNN embeddings?**\n\nThis is a valid architecture proposal, and we also think that using PH for constructing views deserves further study. \n\nSince we consider the distance-based approach we propose with TDL to offer unique advantages for existing models (i.e., by incorporating regularization in terms of the relations between the input graphs, which the models do not explicitly consider), the possibility to improve a number of those by applying our loss on top, and its potential for follow-up research (see also the response to Reviewer bcsi), we chose to focus on this one in this very initial paper. \n\n**Limitation: Variety of Design Choices**\n\nWhile this can be considered as a limitation, we think that it rather shows the potential of the research direction we propose. Further research is definitely needed before the possible architectures can be reliably used in practice. However, the many works on PH in chemistry (see also the references suggested by Reviewer recC) show that the domain is convinced by the usefulness of topological fingerprints already and we therefore believe that this kind of knowledge should be considered in SSL, assuming that the latter becomes more important with the advancement of foundation models.\n\nWe now added some text about this to the paper to clarify the intention of our work, to motivate further study, and to also avoid confusion as it appeared in other reviews (e.g., that we recommend to use a specific filtration function, what we don't).\n\n------------------------------------------------------------\n[1] Rieck, Bastian. \"On the Expressivity of Persistent Homology in Graph Learning.\" arXiv preprint arXiv:2302.09826 (2023)."}, {"review_id": "16jruowUNg", "reviewer": "Reviewer_recC", "summary": "In this manuscript, the authors have developed an interesting self-supervised learning model, by the incorporation of persistent homology into contrastive learning module. More specifically, a special topological distance based contrastive loss is proposed. The model is novel, and the results are very promising. However, I have some concerns about the persistent homology analysis part.\n", "questions": "1)\tAs a powerful tool in topological data analysis, persistent homology (PH) has demonstrated great power in molecular data analysis, PH-based molecular descriptors and fingerprints have already been extensively tested on various benchmark datesets and shown better performance than not only traditional molecular descriptors, such as ECFP, Morgan, daylight, etc, but also many deep learning models. Many related important references are not discussed in the paper, such as\n\nZ. X. Cang, Lin Mu and Guo-Wei Wei, Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening, PLOS Computational Biology, 14(1), e100592 (2018).\n\nZ. X. Cang and Guo-Wei Wei, TopologyNet: Topology based deep convolutional and multi-task neural networks for biomolecular property predictions, PLOS Computational Biology, 13(7), e1005690 (2017).\n\nZ. X. Cang and Guo-Wei Wei, Element specific persistent homology for the analysis and prediction of protein folding stability upon mutation, Bioinformatics,  33, 3549-3557 (2017).\n\nZ. X. Cang, Lin Mu and Guo-Wei Wei, Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening, PLOS Computational Biology,  14(1), e100592 (2018).\n\nDuc Duy Nguyen, Zixuan Cang, Kedi Wu, Menglun Wang, Yin Cao and Guo-Wei Wei, Mathematical deep learning for pose and binding affinity prediction and ranking in D3R Grand Challenges, Journal of Computer Aided Molecular Design, 33, 71-82  (2019).\n\nDuc Nguyen, Zixuan Cang, and Guo-Wei Wei, A review of mathematical representations of biomolecular data, Physical Chemistry Chemical Physics, 22, 4343-4367 (2020).\n\nXiang Liu, Xiangjun Wang, Jie Wu, and Kelin Xia, \"Hypergraph based persistent cohomology (HPC) for molecular representations in drug design.\" Briefings In Bioinformatics, 22 (5), bbaa411 (2021)\n\nXiang Liu, Huitao Feng, Jie Wu, and Kelin Xia, \"Dowker complex based machine learning (DCML) models for protein-ligand binding affinity prediction.\" PLOS Computational Biology, 18(4), e1009943 (2022)\n\nChi Seng Pun, Si Xian Lee, and Kelin Xia, \"Persistent-homology-based machine learning: a survey and a comparative study.\" Artificial Intelligence Review, (2022)\n\nD. Vijay Anand, Qiang Xu, Junjie Wee, Kelin Xia, and Tze Chien Sum, \"Topological feature engineering for machine learning based halide perovskite materials design\", npj Computational Materials, 8 (203) (2022)\n\n2) Mathematically, the filtration process from PH will only generate a nested sequence of simplicial complexes! In Figure 1, the plotted “chemical subgraphs” (during the filtration process) are not the general simplicial complexes (Vietoris-Rips complex or Alpha complex), as there are double bonds. Note that a double bond is illustrated as two edges which appear simultaneous among two vertices. This representation is not mathematically rigorous! The authors can use the common graph but add edge features to denote the double bond! Note that it is possible to use cellular complex to represent “double bonds”, but their persistent homology will be different!\n\n3) Page 3, line 100, their filtration ends with the original graph. In this way, their Betti_1 bars will never die, as there are NO 2-simplexes in their model? The authors are suggested to double check this setting. More discussion will be given below. \n\n4) Page 3, line 106, the authors mention that they use $H_k (G_i)$? They should specify what is the range for the integer $k$. Further, in their topological loss, do they consider both the Betti_0 and Betti_1, or just Betti_0/Betti_1?\n\n5) Page 5, line 209-212, “The fingerprints are usually constructed in a way so that they capture information about the molecular graph structure and sometimes additional domain knowledge; even if they do not capture the entire complexity of the molecules, they represent some, probably important aspects.” Even though PH models are important, they can only characterize homological information, such as individual components, circles, voids, cavities. In biomolecular, the Betti_0 is usually related to covalent bonds, Betti_1 is related to pentagon (sugar ring) and hexagon (benzene ring). These findings have already been widely known, many references can be found,\n\nKelin Xia, Xin Feng, Yiying Tong and Guo-Wei Wei, \"Persistent homology for the quantitative prediction of fullerene stability.\" Journal of Computational Chemistry, 36, 408-422(2015).\n\nKelin Xia and Guo-Wei Wei, \"Persistent homology analysis of protein structure, flexibility and folding.\" International Journal for Numerical Methods in Biomedical Engineering, 30(8), 814-844(2014)\n\nZhenyu Meng, D Vijay Anand, Yunpeng Lu, Jie Wu, and Kelin Xia, \"Weighted persistent homology for biomolecular data analysis.\" Scientific Report, 10 (1), 1-15 (2020)\n\nThe authors are suggested to add an example or some more discussions to explain what the information is captured in their PH models. Further, the general filtration is usually based on atomic distance or some special weighted distances, in this way, higher-dimensional simplicial complexes are generated!  And both Betti_0, Betti_1, and Betti_2 have their clear biological meanings! If the filtration process ends with the original graph (as illustrated in Page 3, line 100), there are no Betti_2 information, and Betti_1 bars will never die! The authors are suggested to compare with these existing approaches to show their advantages or add some more discussions about the difference. \n", "limitations": "Missing many important related literature. The advantage of their filtration process is not clear.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors have developed an interesting self-supervised learning model, by the incorporation of persistent homology into contrastive learning module. More specifically, a special topological distance based contrastive loss is proposed. The model is novel, and the results are very promising.", "weaknesses": "The PH model is not explained clearly. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for carefully checking our theory part!**\n\nWe are sorry for the unnecessary confusion caused by the missing details. \nWe hope that the additional explanations resolve the points mentioned, esp. also Q1 and Q5, which are part of the global reply. In particular, please note that we did not intend to propose a particular filtration function \nrather give an example. We consider this to better be left to the expertise of PH experts and dependent on the actual application.\n\n**Clarification on Theoretical Background**\n\n- Q2: The “double bonds” in Figure 1 serve as a graphical illustration of the edge features rather than part of the actual filtration. In our experiments, the presence of a double bond is also treated as an edge feature, which is clearly stated in Appendix A. We realize that this is confusing in the context where the filtration is introduced.\n- Q3: We intended to introduce a simplified scenario and give the more complex details about the version we actually used (an extended persistence module [1]) in the appendix, but later forgot to add a comment and the latter details. The extended persistence module [1] captures Betti_1 features, ensuring that they will also be killed in the end.\n- Q4: k is either 0 or 1. In TDL we use both in concatenation. \n\nWe updated all these accordingly now.\n\n**Q5 Topological Features in Embeddings**\n\nIn regular molecule representation learning, it is indeed possible to draw rather direct conclusions about what is or can be captured in the embeddings if we use very straightforward filtration methods (e.g., no multi-parameter filtrations, which have been shown to be very powerful recently). However, only the TAE baseline we consider actually is optimized for learning PH-based embeddings. TDL has a more abstract goal and uses the information captured by the topological fingerprints only indirectly, for regularization. We do not expect the learnt embeddings to explicitly capture topological features and therefore did not include such examples into the paper, since this might be misleading. We will add more discussion about the topic more generally (e.g., for TAE, and also topological views for CL could capture such features) to underline the potential PH may offer in SSL. This also fits well with the proposal of Reviewer XtrW to mention alternative, potentially useful architectures.\n\n---------------------------------------------------------\n[1] Cohen-Steiner et al. \"Extending persistence using Poincaré and Lefschetz duality.\" Foundations of Computational Mathematics 9.1 (2009): 79-103."}]

Author rebuttal:

**We thank all reviewers for the very fair, detailed, and constructive feedback!**

We address all comments below and are happy to provide additional information if needed.

---------------------------
**G1 Summary of Additional Experiments Suggested by Reviewers**

- **Technically similar approach** ([1] suggested by U3kd). Performance well below GraphCL+TDL when it is pre-trained over the same data (~2M molecules rather than the ~10M considered in [1]), and TDL yields improvement.
- **Standalone TDL** (9rP5 Q6). Though not recommended, achieves a competitive average ROC-AUC of 72.02, *without using views*.
- **Recent AD-GCL** (U3kd Q2, XtrW W2). TDL increases average performance from 72.67 to 73.21. Our baselines+TDL often outperform AD-GCL in our additional experiments by large margins (e.g., linear probing).
- **w/ Stronger Filtration, ToDD [6]** (9rP5 W1, XtrW W3, recC Q5). Convincing improvements, *across several CL baselines*: 
    * Linear probing: average performance raised to >= 67 (vs. <= 64 w/o TDL), which is comparable to fine-tuned GIN
    * Fine-tuning: improvement between 1.3 and 2.0 for all models
    * Low-data fine-tuning: performance improvements of up to 10%

**Please see the attached pdf.** We report some results and additional side experiments in the reviews.

---------------------------
**G2 Comparison to SOTA** (U3kd Q2, XtrW W2)

The SEGA paper appeared on arXiv on May 8 and the code is not yet available. We tried to run Mole-BERT before for quite some time but we did not succeed, and all issues on its repository have gotten deleted so far, without having been addressed. 
We now ran TDL on top of AD-GCL and see good improvement (see attachment). Moreover, the inclusion of AD-GCL into the other experiments nicely *highlights the complementary nature and potential of TDL: overall, it makes the baselines outperform AD-GCL, often by large margins*.

---------------------------
**G3 Power and Complementary Nature of PH-based Embeddings, compared to regular GNN embeddings or ECFP** (9rP5 W1&W3, U3kd W3, XtrW W3, recC Q1&Q5)

- **References.** In the submission we only mentioned the work about molecule representation learning using persistent homology which is closest to ours and omitted the majority of works in the area since the technical focus of our paper is on SSL, a specific setting with its own challenges. However, as suggested by Reviewer recC and as it is shown in the reviews, PH is less known in the ML community and *we can back our claim that it represents a proven and powerful method for representation learning over molecules according to domain scientists*. We will add more discussion about this to the appendix and hope that we thereby address 9rP5 W3 (partially), U3kd W3, and recC Q1.
- **Power of Filtration Function.** In the submission, we focused on a most simple filtration function based on atom symbols, which does not use more domain knowledge than related works. This allows us showing that TDL adds a novel dimension of knowledge (i.e., the explicit distance regularization) even in this scenario. We also introduced the most simple filtration in the theory part, to convey the overall idea. *We did not intend to recommend a particular filtration, TDL is generic there*. To provide more support for our proposal of using PH (9rP5 W1, XtrW W3, recC Q5), *we report hopefully convincing results in the attachment using a stronger filtration* function including more domain knowledge, the one proposed in [6].

---------------------------
**G4 Relevance of Fine-Tuning Experiments** (9rP5 W1&W2, XtrW W1)

Please note that "the experimental results" should not be equated with Table 4 alone. We do not consider the fine-tuning experiments to show our method's effectiveness best. TDL shines in other scenarios.

Furthermore:
- In drug discovery, downstream benchmarking results rarely translate into practice which is getting recognized more and more recently (e.g., see the ICML 2023 panel on the topic [1] or [2]). Hence, the *generally* minor model differences over Moleculenet should be interpreted with care.
- SOTA works in SSL in other domains sometimes do not consider fine-tuning experiments at all [3, 4], since the data contains various forms of additional confounding factors (e.g., label distribution, dataset balance) which make it hard to get an insight into the actual influence of the SSL embeddings. Generally, fine-tuning experiments are considered as one possible label-based evaluation protocol, often only considered after linear probing etc. [5]. 

The Moleculenet benchmark used with scaffold split represents a challenging setting, which gives good estimates about model performance in certain downstream scenarios. However, we think that graph SSL should extend the experiment setting introduced by Hu et al. in 2020, which solely relies on fine-tuning, and evaluate more broadly.

TDL *significantly* improves a range of well-known and more recent graph SSL approaches on Moleculenet
- in linear probing and
- in low-data experiments on subsets of the benchmark.

Our additional results using a stronger filtration method even improve upon the numbers reported in the submission and also show that PH may lead to considerable increases in fine-tuning.


---------------------------
[1] ICML 2023 Panel: Fostering the Development of Impactful AI Models in Drug Discovery.

[2] Tossou et al. "Real-World Molecular Out-Of-Distribution: Specification and Investigation." ChemRxiv.

[3] Devillers, et al. "EquiMod: An Equivariance Module to Improve Visual Instance Discrimination." ICLR 2023.

[4] Ermolov, Aleksandr, et al. "Whitening for self-supervised representation learning." ICML 2021.

[5] Balestriero, Randall, et al. "A cookbook of self-supervised learning." arXiv preprint arXiv:2304.12210 (2023).

[6] ToDD: Topological Compound Fingerprinting in Computer-Aided Drug Discovery, NeurIPS 2022.

Meta-review: This paper presents a novel way for self-supervised molecular representation learning using novel algorithms from computational topology. All reviewers agreed about the significance of the topic and the relevance of the proposed method. While one reviewer raised some concerns about the novelty (`U3kd`), I find no such faults to be part of the current write-up. In addition, the authors provided a strong and comprehensive rebuttal that served to clarify numerous relevant aspects. I am thus happy to endorse this paper for publication. The authors are **strongly encouraged** to make use of the suggestions by reviewers and integrate the additional experiments mentioned in the rebuttal. Moreover, a revision in terms of improving the clarity would be beneficial in order to make this work available to a broader audience. I trust the authors to address these points in their revision.
Decision: Accept (poster)