Title: Improving Self-supervised Molecular Representation Learning using Persistent Homology
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Networks

Molecular Representation Learning

Persistent Homology

Contrastive Learning

Self-supervised Learning
Abstract: Self-supervised learning (SSL) has great potential for molecular representation learning given the complexity of molecular graphs, the large amounts of unlabelled data available, the considerable cost of obtaining labels experimentally, and the hence often only small training datasets. The importance of the topic is reflected in the variety of paradigms and architectures that have been investigated recently, most focus on designing views for contrastive learning.
In this paper, we study SSL based on persistent homology (PH), a mathematical tool for modeling topological features of data that persist across multiple scales. It has several unique features which particularly suit SSL, naturally offering: different views of the data, stability in terms of distance preservation, and the opportunity to flexibly incorporate domain knowledge.
We (1) investigate an autoencoder, which shows the general representational power of PH, and (2) propose a contrastive loss that complements existing approaches. 
We rigorously evaluate our approach for molecular property prediction and demonstrate its particular features in improving the embedding space:
after SSL, the representations are better and offer considerably more predictive power than the baselines over different probing tasks; our loss 
increases baseline performance, sometimes largely; and we often obtain substantial improvements over very small datasets, a common scenario in practice.

Improving Self-supervised Molecular Representation Learning using Persistent Homology

Abstract

Self-supervised learning (SSL) has great potential for molecular representation learning given the complexity of molecular graphs, the large amounts of unlabelled data available, the considerable cost of obtaining labels experimentally, and the hence often only small training datasets. The importance of the topic is reflected in the variety of paradigms and architectures that have been investigated recently, most focus on designing views for contrastive learning. In this paper, we study SSL based on persistent homology (PH), a mathematical tool for modeling topological features of data that persist across multiple scales. It has several unique features which particularly suit SSL, naturally offering: different views of the data, stability in terms of distance preservation, and the opportunity to flexibly incorporate domain knowledge. We (1) investigate an autoencoder, which shows the general representational power of PH, and (2) propose a contrastive loss that complements existing approaches. We rigorously evaluate our approach for molecular property prediction and demonstrate its particular features in improving the embedding space: after SSL, the representations are better and offer considerably more predictive power than the baselines over different probing tasks; our loss increases baseline performance, sometimes largely; and we often obtain substantial improvements over very small datasets, a common scenario in practice.

1 Introduction

Self-supervised learning (SSL) has great potential for molecular representation learning given the complexity of molecules, the large amounts of unlabelled data available, the considerable cost of obtaining labels experimentally, and the resulting often small datasets. The importance of the topic is reflected in the variety of paradigms and architectures that are investigated [Xia et al., 2023a].

Most existing approaches use contrastive learning (CL) as proposed in [You et al., 2020]. CL aims at learning an embedding space by comparing training samples and encouraging representations from positive pairs of samples to be close in the embedding space while representations from negative pairs are pushed away from each other. However, usually, each given molecule is considered as its own class, that is, a positive sample pair consists of two different views of the same molecule; and all other samples in a batch are used as the negative pairs during training. We observe that this represents a very coarse-grained comparison basically ignoring all commonalities and differences between the given molecules. Therefore also current efforts in molecular CL are put in constructing views that capture the possible relations between molecules: GraphCL [You et al., 2020] proposes four

Deathtime

O

1

6

7

8

AtomNumber

Birthtime

Fingerprint

simple augmentations (e.g., drop nodes), but fix two per dataset; JOAO [You et al., 2021] extends the former by automating the augmentation selection; GraphLoG [Xu et al., 2021] optimizes in a local neighborhood of the embeddings, based on similarity, and globally, using prototypes based on semantics; and SimGRACE [Xia et al., 2022] creates views using a second encoder, a perturbed version of the molecule's ones; also most recent works follow this paradigm [Wu et al., 2023].

Quality of Embedding Spaces is Underinvestigated. Although differences in performance are often observed, they seem marginal and are not yet fully comprehended. With a few exceptions, the models are usually evaluated over the MoleculeNet benchmark [Wu et al., 2018a] only, a set of admittedly rather diverse downstream tasks. While those certainly provide insights into model performance, recent evaluation papers have pointed out that the picture may be very different when the models are evaluated in more detail [Sun et al., 2022, Wang et al., 2022a, Akhondzadeh et al., 2023, Deng et al., 2022]. In particular, [Akhondzadeh et al., 2023] propose to use linear probing to analyse the actual power of the representations instead of a few, specific downstream tasks. This type of evaluation is also common in other areas of DL [Chen et al., 2020]. [Deng et al., 2022] consider other, smaller datasets, and compare to established machine learning (ML) approaches.

We study molecular SSL based on persistent homology (PH). PH is a mathematical tool for modeling topological features of data that persist across multiple scales. In a nutshell, the molecule graph can be constructed sequentially using a custom filter, such that atom nodes and the corresponding bond edges only appear once they meet a given criterion (e.g., based on atomic mass); see Figure 1. PH then captures this sequence in a persistence diagram (PD), and the area has developed various methods to translate these diagrams into topological fingerprints , which can be used for ML [Ali et al., 2022].

Topological fingerprints have been used for supervised ML over molecules by chemists [Krishnapriyan et al., 2021, Demir et al., 2022] and recent work in AI has shown promising results compared to the commonly used ECFP embeddings [Rogers and Hahn, 2010]. Moreover, many of these fingerprints have been shown to be stable , in the sense that distances between the PDs are reflected in the distances between the corresponding fingerprints. We point out that the unique features of topological fingerprints particularly suit molecular SSL , naturally offering: different views of the data (i.e., by switching the filter function), stability in terms of distances, and the opportunity to flexibly incorporate domain knowledge.

100200

400

800

1

,

000

0

.

4

0

.

6

0

.

8

ROC-AUC

GraphLoG+TDLToDD

GraphLoG

SimGRACE+TDLToDD

SimGRACE

GraphMVP

ADGCL

Our Idea. We consider the topological fingerprints of the molecules in the pre-training data as views and exploit their stability to model the distances between the given molecules (i.e., instead of between a molecule and/or its views). In particular, we use them for fine-grained supervision in SSL.

Contributions.

We propose a topological distance contrastive loss (TDL) which, as outlined above, shapes the embedding space by providing supervision for the relationships between molecules, especially the ones in the data. TDL is complementary and can be flexibly and efficiently applied to complement any CL approach . We extensively evaluate it combining TDL with four established CL models.

We also consider alternative paradigms for comparison : we study a straightforward topological fingerprints autoencoder (TAE) .

Our evaluation particularly focuses on the potential in improving the embedding space and we demonstrate the gain in representation power in detail empirically (see Section 4.1): TDL en-

ables the models to learn, in a sense, calibrated distances; considerably improves linear probing performance; and the fine-grained supervision may mitigate deficiencies of individual models.

Over downstream data (see Section 4.2) the performance increases depend more on the model and data. Notably, TDL is able to considerably improve GraphCL and make it competitive with SOTA. Moreover, TDL strongly improves various models in the low-data scenario (see Figure 2).

Our implementation is available at https://github.com/LUOyk1999/Molecular-homology .

2 Background and Related Works

Graph Homology. Molecules are graphs G = ( V, E ) with nodes V , the atoms, and bond edges E . In algebraic topology, graph homology considers such a graph G as a topological space. We focus on simplices : every node is a 0-simplex, and every edge is a 1-simplex. In the context of graphs, we obtain a 1-dimensional simplicial complex X = V ∪ E in an easy way, by considering the simplices induced by G ; the dimension is determined by the maximal dimension of the contained simplices.

Persistent Homology (PH). We introduce the most important concepts and, for more details, refer to [Dey and Wang, 2022, Edelsbrunner and Harer, 2022]. Persistent homology is a mathematical tool for modeling topological features of data that persist across multiple scales, comparable to different resolutions. These features are captured in persistent diagrams which, in turn, can be vectorized in fingerprints. We outline the process simplified below and in Figure 1; see Appendix A for details.

First, the goal is to construct a nested sequence of subgraphs G 1 ⊆ ... ⊆ G N = G ( 1 ≤ i ≤ N ). As described above, these graphs can be considered as simplicial complexes, hence we have simplices which we can record in a persistence diagram. To this end, we consider one of the most common types of filtration methods: sublevel/superlevel filtrations. A sublevel filtration is a function f : X → R over all simplices. A simple such function f can be a node-valued function (e.g., map atom nodes to their atomic number) that is expanded to the edges as f ( u, v ) = max ( f ( u ) , f ( v )) . Denote by X a the sublevel set of X , consisting of simplices whose filtration function values ≤ f ( a ) , X a = { x ∈ X | f ( x ) ≤ f ( a ) } . As the threshold value a increases from min v ∈ V f ( v ) to max v ∈ V f ( v ) , let G a be the subgraph of G induced by X a ; i.e., G a = ( V a , E a ) where V a = { v ∈ X a } and E a = { e rs ∈ X a } . This process yields a nested sequence of subgraphs G 1 ⊆ G 2 ⊆ ... ⊆ G N = G . As X a grows to X , new topological structures gradually appear (born) and disappear (die).

Second, a persistence diagram (PD) is obtained as follows. For each topological structure σ , PH records its first appearance and its first disappearance in the filtration sequence. And this is represented by a unique pair ( b σ , d σ ) , where 1 ≤ b σ ≤ d σ ≤ N . We call b σ the birth time of σ , d σ the death time of σ and d σ -b σ the persistence of σ . PH records all these birth and death times of the topological structures in persistence diagram PD ( G ) = { ( b σ , d σ ) | σ ∈ H k ( G i ) } , where H k ( G i ) denotes the k -th homology group of G i , and in practice, k typically takes values 0 or 1. This step is rather standard and corresponding software is available [Otter et al., 2017].

PH in ML. The vectorization of PDs for making them applicable in ML has been studied extensively [Ali et al., 2022], and proposals range from simple statistical descriptions to more complex persistence images (PIs) [Adams et al., 2017]. In a nutshell, for PIs, the PD is considered as a 2D surface, tranformed using a Gaussian basis function, and finally discretized into a vector. Furthermore, the Euclidean distance between PIs is stable with respect to the 1-Wasserstein distance between PDs [Adams et al., 2017], which essentially means that the former is bounded by a constant multiple of the latter. We focus on PIs based on the promising results reported in supervised settings [Demir et al., 2022, Krishnapriyan et al., 2021]. Our study was inspired by the ToDD framework, applying pretrained vision transformers to custom 2D topological fingerprints [Demir et al., 2022]. Interestingly, they use the distances between the transformer's molecule representations to construct a suitable dataset for subsequent supervised training using triplet loss. Our focus is on SSL and we apply the ToDD fingerprints as more complex, expert fingerprints in comparison to the ones based on atomic mass only. Recently, various other approaches of integrating PH into ML are explored, but these are only coarsely related to our work (e.g., [Horn et al., 2022, Yan et al., 2022]).

Molecular SSL. Since the foundational work of [Hu* et al., 2020], who have proposed several effective methods, such as the node context prediction task ContextPred, the area is advancing at great pace [Xia et al., 2023a, Xie et al., 2022]. Our proposal falls into the category of contrastive learning as introduced in Section 1. Our study focuses on the potential of PH to complement existing

models such as [You et al., 2020] and follow-up approaches. There are other related CL approaches to which we do not aim to compare to directly; e.g., works including 3D geometry [Liu et al., 2022a, Stärk et al., 2022] or considerably scaling up the pre-training data [Ross et al., 2022, Zhou et al., 2023]. In contrast, our focus is on improvement through exploiting unused facets of the data.

SSL based on Distances. We found only few works that explicitly incorporate distances into SSL. With regard to graphs, [Kim et al., 2022] exploit the graph edit distance between a graph and views to similarly represent the actual distance in the embedding space inside the loss function (i.e., vs. invariance to the view transformation). This distance computation is feasible since it can be easily obtained based on the transformation. Our work focuses on exploring in how far distances in terms of PH can be applied towards the same goal; moreover, this makes it feasible to model the distances between given samples. [Wang et al., 2022b] consider a loss very similar to our proposal based on the more coarse-grained ECFPs, instead of PIs, but they focus on chemistry aspects instead of SSL more generally. Also [Zha et al., 2022] model distances between given samples in the context of CL in a similar way, but in the context of a supervised scenario, where the distances are the differences between given regression labels; note that they also list other coarser related works. [Taghanaki et al., 2021] apply custom distances inside a triplet loss but similarly exploit label information, in order to select the positive and negative samples. Beyond that, distances have been applied to obtain hard negatives [Zhang and Re, Demir et al., 2022], and are exploited in various other ways rather different from our method. For instance, several recent approaches aim to structure the embedding space by exploiting correlations between samples which are, in turn, obtained using nearest neighbors [Caron et al., 2020, Dwibedi et al., 2021, Ge et al., 2023]. Also methods considering equivariance between view transformations implicitly model distances [Chuang et al., 2022, Devillers and Lefort, 2023].

Others. While our focus on CL and distances hints at a close relationship to deep metric learning [KAYA and B ˙ ILGE, 2019], these works usually do not have explicit distances for supervision but, for instance, exploit labels. Observe that this indicates the unique nature of the distances PH provides us with. Lastly, SSL more generally [Rethmeier and Augenstein, 2023] has naturally been inspiration for molecular SSL and is certainly one reason why the field was able to advance so fast. We use its insights by putting focus on linear probing and investigating dimensional collapse [Hua et al., 2021].

3 Methodology

The main goal of SSL is to learn an embedding space that faithfully reflects the complexity of molecular graphs, captures the topological nature of the molecular representation space overall, and whose representations can be effectively adapted given labels during fine-tuning. We propose methods based on persistent homology and show that they naturally suit SSL. First, persistence images (or comparable topological fingerprints) offer great versatility in that they are able to flexibly represent knowledge about the graphs and allow for incorporating domain knowledge. Second, they capture this knowledge based on persistence diagrams, which is very different from - and hence likely complementary to - the common graph representation methods in deep learning. Third, and most importantly, their stability represents a unique feature, which makes them ideal views for SSL.

We study two SSL approaches based on PH and evaluate them in detail in terms of both their impact on representation power (see Section 4.1) and on downstream performance (see Section 4.2):

In order to study the impact of PH on SSL in general, we consider a simple autoencoder architecture.

Since we consider topological fingerprints to represent information that is complementary to that used in existing approaches (we also show that they are not ideal alone) and because their topological nature can be used to improve the latter in unique ways, we developed a loss function based on contrastive learning that complements existing approaches.

In this paper, our focus is on obtaining initial insights about the potential PH offers for molecular SSL, hence we chose one basic solution and one providing unique impact. There are certainly other promising ways to be investigated in the future.

3.1 Topological Fingerprints AutoEncoder (TAE)

Autoencoders are designed to reconstruct certain inputs given context information for the input graph G . We consider topological fingerprints I G as the reconstruction targets, specifically, PIs.

Readout

E

Projection

Head

Z

Graph

GNN Encoder

Embedding

Z

GraphEmbeddings

Topological

Space

BatchofMolecules

Persistent

Topological

Homology

Fingerprint

Topological Distance

m

Contrastive Loss

The model itself employs a typical graph encoder ε ( · ) for computing embeddings h v for the individual nodes v ∈ V ; for simplicity, we write h V = ε ( G ) , where h V = { h v | v ∈ V } represents all node representations. Next, we pass it through a projection head g ( · ) and readout function R ( · ) (e.g., mean pooling) to obtain the graphlevel representation h G = R ( g ( h V )) . Hence the context is the particular knowledge the graph encoder was de-

102

102

Density

Density

101

101

100

100

0.3

0.4

0.5

0.6

0.7

0.8

6'0

1.0

0.2

0.4

0.6

0.8

1.0

Cosine Similarity, Pls

Cosine Similarity, Pls

signed to exploit; in our evaluation, we consider the commonly used GIN [Xu et al.] in order to ease comparison with related works. As our loss formula for this topological fingerprints autoencoder (TAE) we use the regular mean squared error: L TAE = ∑ G MSE ( h G , I G ) .

Observations. Given stable fingerprints such as PIs, one main usual criticism for autoencoders, the fact that they fail to capture inter-molecule relationships, does not apply if the model is able to reliably learn the PIs, which we show TAE does; in particular, we observe a strong correlation between PIs and their reconstructions (see Table 7). The successful application of topological fingerprints in supervised learning and the simplicity of the model justify the study of TAE for analysis, however, we note that the PIs employed may not necessarily capture all information which is critical for a particular downstream task. For example, Figure 3 shows that PIs may offer a generally fitting embedding space, in that randomly chosen molecule pairs with similar standard fingerprints based on substructures (blue, Tanimoto similarity of ECFPs) have rather similar PIs (x-axis, cosine similarity). However, there is a clear difference for the two depicted datasets, sometimes PIs as shown here, based on the ToDD filtration Demir et al. [2022], fail to fully capture structural similarity; and this is directly reflected in performance (see Table 4). While optimization on a case-by-case basis w.r.t. the choice of fingerprints is possible, this is not in the spirit of foundational SSL, requires expert knowledge or extensive tuning, and may still not be sufficient. For that reason, we suggest to apply them together with existing approaches and show that they offer unique benefits.

3.2 Topological Distance Contrastive Loss (TDL)

Contrastive learning aims at learning an embedding space by comparing training samples and encouraging representations from positive pairs of examples to be close in the embedding space while representations from negative pairs are pushed away from each other. Current approaches usually consider each sample as its own class, that is, a positive pair consists of two different views of it; and all other samples in a batch are used as the negative pairs during training. We observe that this represents a very coarse-grained comparison basically ignoring all commonalities and differences between the given molecules; this is also why current efforts in graph CL focus on constructing views that capture the possible relations between molecules.

Our Idea, Figure 4. We exploit the stability of topological fingerprints such as PIs to model the distances between the given molecules (i.e., instead of just views of the same molecule) and to use them for fine-grained supervision in SSL (recall that stability means the distances between PIs reflect those between the topological persistence diagrams). This is very different from and complementary to related works in that it structures the embedding space in a different way . The fingerprints are usually constructed in a way so that they capture information about the molecular graph structure; even if they do not capture the entire complexity of the molecules, they represent some, probably important aspects. Moreover, based on the topological nature, they may capture aspects not represented by the commonly used graph embeddings . In particular, they offer a way to flexibly integrate expert knowledge into molecular SSL; observe that the usually employed GNNs have dimensions based on the available molecule information, hence including additional information, even if available only for some of the data, requires architecture changes.

We consider a batch of N graphs, { G i } i ∈ [1 ,N ] . Similar as above, we first extract graph-level representation vectors h G i using a graph encoder ε ( · ) , followed by a readout function R ( · ) . Here, we apply the projection head g ( · ) later, to map the graph representations to another latent space and obtain the final graph embedding z i . Specifically, we apply a two-layer MLP and hence a non-linear transformation, known to enhance performance [Chen et al., 2020]: z i = g ( R ( ε ( G i ))) Let G n be the sample under consideration. Instead of constructing an artificial view, we consider all possible positive pairs of samples ( G n , G m ) together with a set of stable, topological fingerprints { I i } i ∈ [1 ,N ] for all graphs. Note that these can be computed rather efficiently, and they have to be computed only once for the given pre-training data. In a nutshell, our loss adapts the regular NT-Xent [Sohn, 2016, Oord et al., 2018, Wu et al., 2018b] by considering only those negative pairs ( G n , G k ) of samples where the Euclidean distance dis ( I n , I k ) between the corresponding fingerprints is greater than the one between I n and I m . We compute the similarity score as sim ( z n , z m ) = z n ⊤ z m / ∥ z n ∥ ∥ z m ∥ , and consider a temperature parameter τ and indicator function I [ · ] ∈ { 0 , 1 } as usual. Our topological distance contrastive loss (TDL) is defined as follows, for the n-th sample:

̸

̸

TDL can be flexibly and efficiently applied to complement any graph CL framework , e.g., in the form L n = L others + λ L TDL n , where λ determines its impact. In our evaluation, we used λ = 1 .

Further Intuition. Essentially, TDL provides a form of regularization. It encourages the model to push molecule representations less far away from the sample under consideration if they are similar to it in terms of the topological fingerprints. Given the stability of those, we hence obtain an embedding space with better calibrated distances (i.e., distances in terms of PH, between persistence diagrams of the molecule graphs). This can be partly observed theoretically, in the directions of the gradients; see Appendix I for an initial analysis. We show this empirically by calculating the correlation between distances between the molecule representations after pre-training and the PIs (see Table 1), and by visualizing the distances in Figure 5. Our evaluation further shows that the fine-grained supervision may solve deficiencies of CL models in that it forces them to capture crucial features of the input. Further, the improved embedding space particularly suits low-data downstream scenarios.

On Views. While we consider the modeling of the sample relationships to be most unique and offer great potential to complement other models, we note that TDL can be similarly applied over views.

4 Evaluation

Do TAE and TDL lead to, in a sense, calibrated distances in the representation space ?

Do we obtain improved representations more generally, based on established SSL metrics?

What impact do we see on downstream performance , and does it justify our proposal?

Our Models. We apply both TAE and TDL with different filtration functions, marked by a subscript. First, we apply a most simple filtration atom based on atomic mass. This allows showing that, even by considering less information than the baseline GNNs, which additionally apply atom chirality and connectivity, topological modeling may exploit additional facets of the data. Since TAE is a standalone model, atomic mass is not enough to let it fully capture the molecular nature. For that

reason, we consider three filtrations (atomic mass, heat kernel signature, node degree) and concatenate the corresponding PIs, denoted by ahd . Finally, to show the real potential of PH, we include the ToDD filtration Demir et al. [2022], which combines atomic mass with additional domain knowledge, partial charge and bond type, inside a more complex multi-dimensional filtration. See Appendix A.

Baselines & Datasets. For a comprehensive evaluation, we apply TDL on a variety of existing CL approaches: GraphCL, JOAO, GraphLoG, and SimGRACE (see Section 1). Note that this goes far beyond other CL extensions which are often evaluated with a single approach only [You et al., 2021, Xia et al., 2023b]. We also study TAE on top of the established ContextPred [Hu* et al., 2020], to get an idea of its complementary nature. Model configurations and experimental settings are described in Appendix A. For pre-training, we considered the most common dataset following [Hu* et al., 2020], 2 million unlabeled molecules sampled from the ZINC15 database [Sterling and Irwin, 2015]. For downstream evaluation, we focus on the MoleculeNet benchmark [Wu et al., 2018a] here, the appendix contains experiments on several other datasets.

4.1 Analysis of Representations after Pre-training

Calibrated Distances in Embedding Space, Tables 1, 7, 9, Figure 5.

TAE successfully learns the PI space for the molecules in MoleculeNet in the sense that there is a strong correlation between the PIs and their reconstructions (Table 7, appendix). For TDL, we observe for all baselines considerable increases in terms of correlation between distances in PI space and in representation space; we use the embeddings from linear probing. There are two notable exceptions and generally smaller increases on one dataset, BBBP. We also compared the ROGI score [Aldeghi et al., 2022] which, in a nutshell, measures in how far (dis)similarity of molecules (i.e., we use Euclidean distance of embeddings) is reflected in label similarity (Table 9, appendix). This gives us some idea in terms of downstream labels and the mixed results reflect the variety of the data. Finally, the alignment figure on the right clearly visualizes that the PI embedding space is much more fine-grained in terms of distances than the GNN space, and that the distribution of GraphCL seems to get correctly adapted in that distances

between 'positive' pairs generally shrink below the ones of 'negative' pairs.

Mitigating Dimensional Collapse, Figure 6. One of the most interesting of our findings is the fact that GraphCL and GraphLoG suffer from dimensional collapse (i.e., the embeddings span a lower-dimensional subspace instead of the entire available embedding space, [Wang et al., 2022a] also observe this for GraphCL) and that TDL successfully mitigates this. This can be observed in

0

100

200

300

-

30

-

20

-

10

0

# Singular Value Rank Index

Singular Values (log)

GraphCL+TDLatom

GraphCL

GraphLoG+TDLatom

GraphLoG

0

100

200

300

-

20

-

10

0

# Singular Value Rank Index

SimGRACE+TDLatom

SimGRACE

JOAO+TDLatom

JOAO

0

100

200

300

-

30

-

20

-

10

0

# Singular Value Rank Index

GraphCL+TDLatom

GraphCL

GraphLoG+TDLatom

GraphLoG

0

100

200

300

-

20

-

15

-

10

-

5

0

# Singular Value Rank Index

SimGRACE+TDLatom

SimGRACE

JOAO+TDLatom

JOAO

the singular values of the covariance matrix of the representations, where vanishing values hint at collapsed dimensions. The phenomenon has recently been observed in computer vision and there are only initial explanations to date [Hua et al., 2021]. However, here, we observe the collapse only over the downstream data. Our hypothesis is that TDL's fine-grained supervision forces the models to capture relevant features, which they might have neglected otherwise. We also depict the graphs for SimGRACE and JOAO, which look very different, reflecting the variety of the approaches. There is basically no difference for JOAO(+TDL), while TDL shrinks the values for SimGRACE; the latter is less optimal and needs further investigation.

Linear Probing, Tables 2, 3, 10. We evaluated extensively using a linear layer on the representations of the pre-trained graph encoders over the MoleculeNet data in terms of binary classification and a custom distance prediction task. The probing, also in terms of MLPs, demonstrates that PIs possess a complementary nature to ECFP. Furthermore, TAE, which we developed for comparison purposes only, is competitive with the baselines and improves ContextPred. In Appendix C, we additionally show some results over the more challenging activity cliff data. Apart from JOAO+TDL, where the results are mixed, TDL yields overall impressive increases. Interestingly, the more complex ToDD-based version is not always better than the simpler one.

Discussion. The results demonstrate that both TAE and TDL successfully approach the PI space in terms of distances between molecules. Yet, it also has to be noted that the PIs we chose do not always capture the main characteristics of the sample space, as seen for BBBP. Similarly, the mixed ROGI scores, which incorporate downstream labels, hint at a varied downstream performance. Nevertheless, the specifics of particular downstream datasets should not be confused with the more general molecular representation space. In that respect, especially TDL has been proven effective overall in improving representation power in a variety of probing tasks for all baselines . We have also shown that, on closer look, the effects can be very different with the considered baselines, depending on their individual deficiencies. For JOAO, we have seen an improvement in distance representation but not as large improvements thereafter, suggesting that it captures similar features already. Recall, however, that JOAO 'only' adapts GraphCL in that it chooses between views more flexibly. In this regard, TDL represents a more efficient adaptation of GraphCL, which turns out to be also more effective in terms of SSL generally, as we show next.

4.2 Evaluation on Downstream Tasks

Transfer Learning Benchmarks, Tables 4, 12, Figure 12. First, note that the MoleculeNet benchmark contains various types of data. We have already observed the strong differences between Bace and BBBP in terms of our PIs. Here we additionally have some very large datsets and some with very many tasks; in both we expect the labels and their correlation to have increasing impact and reduce the one of the original embedding space. We report some most recent works that used the same pre-training data to provide a context for interpreting our results. As expected, TAE alone only reaches decent average performance, yet surpasses SOTA on ClinTox and Bace and also considerably improves ContextPred there. We see more variability than above in the effects TDL has on the individual baselines. We do not observe a general impact of dataset balance. To learn about the impact of labels the influence of multiple tasks, we ran some models also on individual tasks of the multi-task data and observe larger increases for TDL there (see figures in Appendix D). Altogether, the table shows mixed results for SimGRACE; some improvement for GraphLoG and, with a single exception, also for JOAO; and large improvements for GraphCL. Notably, when using the stronger

100200

400

800

1

,

000

0

.

5

0

.

6

0

.

7

0

.

8

ROC-AUC

GraphCL+TDLToDD

GraphCL

SimGRACE+TDLToDD

SimGRACE

100200

400

800

1

,

000

0

.

6

0

.

7

0

.

8

GraphLoG+TDLToDD

GraphLoG

JOAO+TDLToDD

JOAO

100200

400

800

1

,

000

0

.

4

0

.

6

0

.

8

GraphCL+TDLToDD

GraphCL

SimGRACE+TDLToDD

SimGRACE

100200

400

800

1

,

000

0

.

4

0

.

6

0

.

8

GraphLoG+TDLToDD

GraphLoG

JOAO+TDLToDD

JOAO

ToDD filtration, TDL demonstrates convincing improvements across all baselines and makes them competitive with SOTA.

The Low-Data Scenario, Figures 2, 7, 13, 14. Inspired by the impressive increases we see in the quality of representations after pre-training, we experimented with considerably smaller datasets. This is a scenario which is relevant in practice since proprietary data from costly experiments is often only available in small numbers [Tom et al., 2023]. Here we see clearly that the improvement in representation space is reflected in downstream performance. There is still some dependence on the dataset but, overall, TDL often yields remarkable improvements. The introduction figure shows that also other models show room for improvement; note that GraphMVP uses 3D information.

Further Analysis. Due to space constraints, we present only the most impactful findings in this section and the rest in the appendix, in particular: (1) Ablation Studies. Our approach offers great variability in terms of, for instance, the graph encoder, the construction of the PIs, or even the type of topological fingerprint more generally. (2) Unsupervised learning. We observe similar but less pronounced trends, likely due to the smaller size of the data and fewer features available. (3) PIs w/o SSL. In view of the promising results with PIs in supervised ML [Krishnapriyan et al., 2021, Demir et al., 2022], we consider the PIs with XGB and SVM, our approaches based on pre-training are better. (4) Other datasets. We conducted experiments over several other datasets, including activity-cliff data, where TDL is shown beneficial more generally.

Discussion. In this section, we have shown results very different from the ones we usually see. In the benchmark, we observe some improvements and, for GraphCL+TDL, numbers competitive with SOTA, but they do not reach the general, large impact we saw in Section 4.1. However, we have such impact over smaller data. Altogether, TDL offers great benefits for all baselines we considered:

It leads to much better calibrated distances in the representation space in terms of PH. Since the magnitude of increase is generally remarkable but varies with the data, this shows potential for further improvement, especially, in terms of choosing the right PIs.

It considerably improves representations in terms of established SSL metrics.

Downstream performance reflects this gain of representation power specifically in the lowdata scenario . Given the importance of this kind of data in the real world and the deficiencies of the models we checked, we conclude that our proposal is not only novel and different in its distance-based modeling, but also that it advances molecular SSL in important aspects empirically.

5 Conclusions

In this paper, we focus on the quality of representations in molecular SSL, an important topic which has not been addressed greatly to date. We propose a novel approach of shaping the embedding space based on distances in terms of fingerprints from persistent homology. We have evaluated our topological-distance-based contrastive loss extensively, and show that it solves deficiencies of existing models and considerably improves various baselines in terms of representation power. In future work, we plan to analyze the relationship between specific PIs and downstream data in more detail, to further improve our approach, include external knowledge, and to put more focus on regression tasks.

Acknowledgments

This work was supported by National Key R&D Program of China (2021YFB3500700), NSFC Grant 62172026, National Social Science Fund of China 22&ZD153, the Fundamental Research Funds for the Central Universities and SKLSDE; correspondence to Lei Shi.

References

Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z. Li. A systematic survey of chemical pre-trained models, 2023a.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems , 33: 5812-5823, 2020.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning , pages 12121-12132. PMLR, 2021.

Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning , pages 11548-11558. PMLR, 2021.

Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. In Proceedings of the ACM Web Conference 2022 , pages 1070-1079, 2022.

Junran Wu, Xueyuan Chen, Bowen Shi, Shangzhe Li, and Ke Xu. Sega: Structural entropy guided anchor view for graph contrastive learning. arXiv preprint arXiv:2305.04501 , 2023.

Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science , 9(2):513-530, 2018a.

Ruoxi Sun, Hanjun Dai, and Adams Wei Yu. Does gnn pretraining help molecular representation? Advances in Neural Information Processing Systems , 35:12096-12109, 2022.

Hanchen Wang, Jean Kaddour, Shengchao Liu, Jian Tang, Matt Kusner, Joan Lasenby, and Qi Liu. Evaluating self-supervised learning for molecular graph embeddings. arXiv preprint arXiv:2206.08005 , 2022a.

Mohammad Sadegh Akhondzadeh, Vijay Lingam, and Aleksandar Bojchevski. Probing graph representations. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 11630-11649. PMLR, 25-27 Apr 2023. URL https://proceedings.mlr.press/v206/akhondzadeh23a.html .

Jianyuan Deng, Zhibo Yang, Hehe Wang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang. Taking a respite from representation learning for molecular property prediction. arXiv preprint arXiv:2209.13492 , 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Dashti Ali, Aras Asaad, Maria-Jose Jimenez, Vidit Nanda, Eduardo Paluzo-Hidalgo, and Manuel Soriano-Trigueros. A survey of vectorization methods in topological data analysis. arXiv preprint arXiv:2212.09703 , 2022.

Aditi S Krishnapriyan, Joseph Montoya, Maciej Haranczyk, Jens Hummelshøj, and Dmitriy Morozov. Machine learning with persistent homology and chemical word embeddings improves prediction accuracy and interpretability in metal-organic frameworks. Scientific reports , 11(1):8888, 2021.

Andac Demir, Baris Coskunuzer, Yulia Gel, Ignacio Segovia-Dominguez, Yuzhou Chen, and Bulent Kiziltan. Todd: Topological compound fingerprinting in computer-aided drug discovery. Advances in Neural Information Processing Systems , 35:27978-27993, 2022.

David Rogers and Mathew Hahn. Extended-connectivity fingerprints. Journal of chemical information and modeling , 50(5):742-754, 2010.

Tamal Krishna Dey and Yusu Wang. Computational topology for data analysis . Cambridge University Press, 2022.

Herbert Edelsbrunner and John L Harer. Computational topology: an introduction . American Mathematical Society, 2022.

Nina Otter, Mason A Porter, Ulrike Tillmann, Peter Grindrod, and Heather A Harrington. A roadmap for the computation of persistent homology. EPJ Data Science , 6:1-38, 2017.

Henry Adams, Tegan Emerson, Michael Kirby, Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. Persistence images: A stable vector representation of persistent homology. Journal of Machine Learning Research , 18, 2017.

Max Horn, Edward De Brouwer, Michael Moor, Yves Moreau, Bastian Rieck, and Karsten Borgwardt. Topological graph neural networks, 2022.

Zuoyu Yan, Tengfei Ma, Liangcai Gao, Zhi Tang, Yusu Wang, and Chao Chen. Neural approximation of graph topological features. In Advances in Neural Information Processing Systems , 2022.

Weihua Hu*, Bowen Liu*, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=HJlWWJSFDH .

Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. IEEE transactions on pattern analysis and machine intelligence , 45(2):2412-2429, 2022.

Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pretraining molecular graph representation with 3d geometry. In International Conference on Learning Representations , 2022a. URL https://openreview.net/forum?id=xQUe1pOKPam .

Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning , pages 20479-20502. PMLR, 2022.

Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das. Molformer: Large scale chemical language representations capture molecular structure and properties. 2022.

Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations , 2023. URL https: //openreview.net/forum?id=6K2RM6wVqKu .

Dongki Kim, Jinheon Baek, and Sung Ju Hwang. Graph self-supervised learning with accurate discrepancy learning. arXiv preprint arXiv:2202.02989 , 2022.

Yuyang Wang, Rishikesh Magar, Chen Liang, and Amir Barati Farimani. Improving molecular contrastive learning via faulty negative mitigation and decomposed fragment contrast. Journal of Chemical Information and Modeling , 62(11):2713-2725, 2022b.

Kaiwen Zha, Peng Cao, Yuzhe Yang, and Dina Katabi. Supervised contrastive regression. arXiv preprint arXiv:2210.01189 , 2022.

Saeid A Taghanaki, Kristy Choi, Amir Hosein Khasahmadi, and Anirudh Goyal. Robust representation learning via perceptual similarity metrics. In International Conference on Machine Learning , pages 10043-10053. PMLR, 2021.

Michael Zhang and Christopher Re. Contrastive adapters for foundation model group robustness. In Advances in Neural Information Processing Systems .

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33:9912-9924, 2020.

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9588-9597, 2021.

Chongjian Ge, Jiangliu Wang, Zhan Tong, Shoufa Chen, Yibing Song, and Ping Luo. Soft neighbors are positive supporters in contrastive visual representation learning, 2023.

Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljaˇ ci´ c, Shang-Wen Li, Wen-tau Yih, Yoon Kim, and James Glass. Diffcse: Difference-based contrastive learning for sentence embeddings. arXiv preprint arXiv:2204.10298 , 2022.

Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module to improve visual instance discrimination. In The Eleventh International Conference on Learning Representations , 2023.

Mahmut KAYA and Hasan ¸ Sakir B ˙ ILGE. Deep metric learning: A survey. Symmetry , 11(9), 2019. ISSN 2073-8994. doi: 10.3390/sym11091066. URL https://www.mdpi.com/2073-8994/11/ 9/1066 .

Nils Rethmeier and Isabelle Augenstein. A primer on contrastive pretraining in language processing: Methods, lessons learned, and perspectives. ACM Computing Surveys , 55(10):1-17, 2023.

Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9598-9608, 2021.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations .

Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems , 29, 2016.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via nonparametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3733-3742, 2018b.

Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. 2023b.

Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling , 55(11):2324-2337, 2015.

Matteo Aldeghi, David E. Graff, Nathan Frey, Joseph A. Morrone, Edward O. Pyzer-Knapp, Kirk E. Jordan, and Connor W. Coley. Roughness of molecular property landscapes and its impact on modellability, 2022.

Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. Advances in Neural Information Processing Systems , 34:15920-15933, 2021.

Gary Tom, Riley J Hickman, Aniket Zinzuwadia, Afshan Mohajeri, Benjamin Sanchez-Lengeling, and Alán Aspuru-Guzik. Calibration and generalizability of probabilistic models on low-data chemical datasets with dionysus. Digital Discovery , 2023.

Bharath Ramsundar, Peter Eastman, Patrick Walters, and Vijay Pande. Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more . O'Reilly Media, 2019.

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semisupervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000 , 2019.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 594-604, 2022.

Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph selfsupervised learning for molecular property prediction. Advances in Neural Information Processing Systems , 34:15870-15882, 2021.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1857-1867, 2020.

Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems , 33:12559-12571, 2020.

David Cohen-Steiner, Herbert Edelsbrunner, and John Harer. Extending persistence using poincaré and lefschetz duality. Foundations of Computational Mathematics , 9(1):79-103, 2009.

Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graphics forum , volume 28, pages 1383-1392. Wiley Online Library, 2009.

Dávid Bajusz, Anita Rácz, and Károly Héberger. Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics , 7(1):1-13, 2015.

Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663 , 2020.

Peter Bubenik et al. Statistical topological data analysis using persistence landscapes. J. Mach. Learn. Res. , 16(1):77-102, 2015.

Frédéric Chazal, Brittany Terese Fasy, Fabrizio Lecci, Alessandro Rinaldo, and Larry Wasserman. Stochastic convergence of persistence landscapes and silhouettes. In Proceedings of the thirtieth annual symposium on Computational geometry , pages 474-483, 2014.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations .

Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining , pages 785-794, 2016.

Zixuan Cang and Guo-Wei Wei. Topologynet: Topology based deep convolutional and multi-task neural networks for biomolecular property predictions. PLoS computational biology , 13(7): e1005690, 2017a.

Zixuan Cang and Guo-Wei Wei. Analysis and prediction of protein folding energy changes upon mutation by element specific persistent homology. Bioinformatics , 33(22):3549-3557, 2017b.

Duc Duy Nguyen, Zixuan Cang, Kedi Wu, Menglun Wang, Yin Cao, and Guo-Wei Wei. Mathematical deep learning for pose and binding affinity prediction and ranking in d3r grand challenges. Journal of computer-aided molecular design , 33:71-82, 2019.

Kelin Xia and Guo-Wei Wei. Persistent homology analysis of protein structure, flexibility, and folding. International journal for numerical methods in biomedical engineering , 30(8):814-844, 2014.

Xiang Liu, Huitao Feng, Jie Wu, and Kelin Xia. Dowker complex based machine learning (dcml) models for protein-ligand binding affinity prediction. PLoS Computational Biology , 18(4):e1009943, 2022b.

Zixuan Cang, Lin Mu, and Guo-Wei Wei. Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening. PLoS computational biology , 14(1): e1005929, 2018.

Xiang Liu, Xiangjun Wang, Jie Wu, and Kelin Xia. Hypergraph-based persistent cohomology (hpc) for molecular representations in drug design. Briefings in Bioinformatics , 22(5):bbaa411, 2021.

A Datasets and Experimental Details

Pre-training Datasets. As usual, for GNN pre-training, we use a minimal set of node and bond features that unambiguously describe the two-dimensional structure of molecules from the ZINC15 database following previous works [Hu* et al., 2020].

Node features:

-Atom number: [1 , 118]

-Chirality tag: { unspecified, tetrahedral cw, tetrahedral ccw, other }

Edge features:

-Bond type: { single, double, triple, aromatic }

-Bond direction: { -, endupright, enddownright }

We note that the atom number implicitly provides domain knowledge in that it captures important knowledge about the atom's charge.

Downstream Task Datasets. We focus on molecular property prediction, where we adopt the widelyused 8 binary classification datasets contained in MoleculeNet [Wu et al., 2018a]. In addition, we experimented with 4 regression tasks from various low-data domains in line with the same setting as [Liu et al., 2022a]. We provide detailed information of these datasets in Table 5 and more information can be found in [Liu et al., 2022a]. To further evaluate the usefulness of our models, we also use the opioids-related datasets from [Deng et al., 2022]. The number (percentage) of activity cliff (AC) molecules are summarized in Table 6. Notably, among MDR1, MOR, DOR and KOR, nearly half molecules are equipped with activity cliff scaffolds. Finally, scaffold-split [Ramsundar et al., 2019] is used to splits graphs into train/val/test set as 80%/10%/10% which mimics real-world use cases.

Computing Environment. The experiments are conducted with two RTX 3090 GPUs.

Model Configurations. Following [Hu* et al., 2020], we adopt a 5-layer Graph Isomorphism Network (GIN) [Xu et al.] with 300-dimensional hidden units as the backbone architecture. We use mean pooling as the readout function. During the pre-training stage, GNNs are pre-trained for 100 epochs with batch-size as 256 and the learning rate as 0.001. During the fine-tuning stage, we train for 100 epochs with batch-size as 32, dropout rate as 0.5, and report the test performance using ROC-AUC at the best validation epoch. To evaluate the learned representations, we follow the linear probing (linear evaluation) [Akhondzadeh et al., 2023], where a linear classifier (1 linear layer) is

trained on top of the frozen backbone. We probe two types of properties from the representation: the first is the molecular properties of downstream tasks, for which we use the same settings as fine-tuning and report ROC-AUC; the second is topological fingerprints (PIs) of downstream molecular graphs, for which we train for 50 epochs with batch-size as 256, dropout rate as 0.0, and report MSE and Pearson correlation coefficients between distances in embedding space and corresponding PI space. The experiments are run with 10 different random seeds, and we report mean and standard deviation.

Performance Comparison. We compare the proposed method with existing self-supervised graph representation learning algorithms (i.e. InfoGraph [Sun et al., 2019], EdgePred [Hamilton et al., 2017], ContextPred [Hu* et al., 2020], AttrMask [Hu* et al., 2020], GraphLoG [Xu et al., 2021], iMolCLR [Wang et al., 2022b], AD-GCL [Suresh et al., 2021], JOAO [You et al., 2021], SimGRACE [Xia et al., 2022], GraphCL [You et al., 2020], GraphMAE [Hou et al., 2022], MGSSL [Zhang et al., 2021], GPT-GNN [Hu et al., 2020], G-Contextual [Rong et al., 2020], G-Motif [Rong et al., 2020], GraphMVP [Liu et al., 2022a]). We report the results from their own papers with two exceptions: (1) for GraphMAE and GraphLoG, they reported the test performance of the last epoch; and (2) MGSSL and SimGRACE, they only used 3 random seeds; and (3) iMolCLR used ~10M molecules for pre-training. We reproduce the results of these four models with the same protocol as [Hu* et al., 2020], utilizing the pre-trained models originally provided by the respective papers.

Topological Fingerprints. We set the highest dimension in the simplicial complex to 1. This doesn't impact performance because molecular graphs, being planar, rarely possess loops of length 3; in fact, the majority of loops exhibit a length of 5 or more.

Firstly, we use the sublevel filtration method. In terms of filtration functions, we use atomic number. For every molecular graph, we utilize an extended persistence module [Cohen-Steiner et al., 2009, Yan et al., 2022] to compute persistent diagrams (as depicted in Figure 1 of [Yan et al., 2022]) and then vectorize them to get PIs as topological fingerprints. The extended module incorporates a additional reverse process. Through this module, all loop features created will be killed in the end. As an example, in Figure 1, a 1-dim extended persistence point that can be captured is ( t 4 , t 2 ) . This is because the loop feature is born at time t 4 and then dies at time t 2 in the reverse process. Note that we only use atomic number as filtration functions for TDLatom. In addition to the atomic number, we also investigate the heat kernel signature (hks) with temperature values t = 0 . 1 [Sun et al., 2009] and node degree as filtration functions. Observe that this is more geometric information, compared to the domain knowledge encoded in the atomic number (atom). Considering that TAE is a straightforward model and there is too little information for PIatom, we concatenated these 3 PIs as reconstruction targets for TAEahd.

We have also considered a specific filtration function from ToDD [Demir et al., 2022], which incorporates atomic mass and, additionally, partial charges and bond types inside a so-called multiVR filtration. In a nutshell, for each of those three types of information, we do not simply consider the given filtration, but construct a 2D filtration by filtering: in one dimension according to the above information and in the second dimension according to a VR filtration capturing the distances between atoms (Figure 5 in [Demir et al., 2022] illustrates this kind of filtration well.) Lastly, the 3 2D PIs are concatenated, and we compute distances based on the combination of PIs. In Appendix G, we report ablation results for various filtrations.

In order to explore the effectiveness of PIs, we calculate the widely-used Tanimoto coefficient [Bajusz et al., 2015] of the extended connectivity fingerprints (ECFPs) [Rogers and Hahn, 2010] between two molecules as their chemical similarity on 8 downstream tasks datasets. Then, we pick the molecule pairs with top 20% similarity as 'similar' ones (Blue) and the left 80% molecule pairs in the datasets are 'random' pairs (Yellow). In final, we calculate the cosine similarity of these molecule pairs based on PIs. Figure 8 shows molecules with similar ECFPs have rather similar PIs.

It is computationally efficient to apply topological fingerprints method. Extracting and computing PIatom for 2M molecular graphs from ZINC15 dataset only spent 20 minutes in a single workstation with 8-core CPU, 64 GB of memory.

TAE+ Models. We hypothesize that TAE gives other pre-training approach a good initial embedding space to start from. Therefore, this pre-training strategy is to first perform TAE pre-training and then other models pre-training. We study TAE+EdgePred, TAE+AttrMask and TAE+ContextPred.

TDL. The temperature parameter τ is set to 0.1.

TDL over Views. Like any other contrastive loss function, TDL can be utilized as a standalone objective by considering views. Here, we show preliminary experiments using a formulation based on the loss of GraphCL [You et al., 2020], we also use its augmentations. A minibatch of N graphs is processed through contrastive learning, resulting in 2 N augmented graphs. We denote the augmented views for the n th graph in the minibatch as z n,i and z n,j , and modify TDL as follows:

(a) Bace

(b) BBBP

(c) ClinTox

(d) Sider

(e) Tox21

(f) ToxCast

(g) HIV

(h) MUV

10-

r0T

10

101

10

10+

10°

109

100

0.2

0.4

Cosine Similiarity

0.6

0.8

1.0

0.0

0.2

Cosine Similarity

0.4

0.6

0.8

1.0

0.0

0.2

Cosine Similarity

0.4

0.6

0.8

1.0

0.0

0.2

Cosine Similarity

0.4

0.6

0.8

1.0

101

10-

101

Density

10°

10#

109

10°

10-1

10-1

10~1

10~1

0.0

2'0

Cosine Similarity

0.4

0.6

0.8

1.0

0.0

0.2

Cosine Similarity

0.4

0.6

0.8

1.0

0.0

0.2

Cosine Similarity

0.4

0.6

0.8

1.0

0.0

0.2

Cosine Similarity

0.4

0.6

0.8

1.0

B Analysis of Embeddings

Alignment Analysis. To evaluate alignment, we construct positive and negative molecule pairs. Positive pairs in a dataset are defined as those sharing identical molecular properties, while negative pairs exhibit differing properties. We randomly select 10k positive and negative pairs of molecular graphs from the Bace, BBBP, Clintox, and Sider datasets. Subsequently, we compute the normalized Euclidean distance of the embeddings and illustrate the results via a histogram, as depicted in Figure 9 and Figure 10. We apply TDL on top of GraphCL, SimGRACE, JOAO, and GraphLoG to represent the impact of TDL on the baselines.

The normalized curves demonstrate significant differences in the embedding space among different models in the distance dimension. It is observable that TDL refines the distribution of distances to more closely align with that of PIs. The distinction between TDLToDD and TDLatom explains why TDLatom does not enhance JOAO because JOAO's curve shows that it is in fact possible to get such an atom PIs' distribution. Furthermore, JOAO reveals that additional domain knowledge beyond atom can be effectively incorporated via TDLToDD.

C Additional Linear Probing

We continue with linear probing for molecular properties using the more challenging activity cliff dataset. In Table 10, we show that TDL often yields improvements.

D Additional Benchmark Results

Primary Results. The main results on 8 molecular property prediction tasks are listed in Table 11. Note that all SSL methods in Table 11 are pre-trained on the same dataset based on ZINC15. We observe that the average performance of GraphCL+TDL has surpassed these existing SSL methods.

GraphCL,Bace

SimGRACE,Bace

GraphLoG,Bace

JOAO,Bace

1200

700

1000

1200

1400

600

800

500

400

300

400

002

L.5c

0.00

0.25

0S'0

0.75

1.25

1.50

1.75

GraphCL,BBBP

SimGRACE,BBBP

GraphLoG,BBBP

JOAO,BBBP

2000

1200

1750

800

1500

1250

600

1000

750

200

5

0.25

0.50

0.75

1.00

0.25

0.75

1.00

1.50

1.75

GraphCL,Clintox

SimGRACE,Clintox

GraphLoG,Clintox

JOAO,Clintox

1600

000

1400

1750

1200

1000

008

400

600

750

400

200

00'0

5Z0

L00

1.25

L.50

0.00

0.25

0.50

0.75

1.00

1.25

1.50

0.00

0.75

1.00

1.25

1.50

1.75

GraphCL,Sider

SimGRACE,Sider

GraphLoG,Sider

JOAO,Sider

1600

1400

1750

1200

1500

4000

1250

500

000

1000

40

300

400

200

002

00

0.00

0.25

0.50

0.75

1.00

1.25

0.00

0.75

Additionally, we also apply TDL on both AD-GCL and iMolCLR, with results in the table showing that TDL still provides good improvements.

Individual Tasks. In Table 4, we observe that SimGRACE+TDL performs poorly on Sider and ClinTox. To learn about the influence of multiple tasks, we run SimGRACE and GraphLoG on individual tasks of Sider and ClinTox. Figure 12 shows increases for TDLatom. We hence conclude that the task correlation is exploited by the models considerably during fine-tuning in the multi-task setting, such that the improved SSL representation (which likely does not fit every downstream task under consideration) provided by TDL has less impact.

Activity Cliff Dataset. We also evaluate our methods using the opioids-related datasets. We fine-tune GraphCL and GraphCL+TDL and observe that GraphCL+TDL outperformes GraphCL, as shown in the Table 12. This result is interesting in itself since it is often argued that SSL based on similarity in 'just' the geometric space (i.e., ignoring domain knowledge about, e.g., matching pairs or activity cliffs) may yield problems in practice. However, here, we observe that increased power of SSL

1A001

GraphCL,Bace

SimGRACE,Bace

GraphLoG,Bace

JOAO,Bace

800

1400

000

500

800

400

200

0.75

GraphCL,BBBP

SimGRACE,BBBP

GraphLoG，BBBP

JOAO,BBBP

2000

600

1200

1750

500

1000

800

1500

1250

400

1000

300

600

750

800

500

200

250

00

0.00

0.25

0.50

0.75

1.00

1.25

L.50

0.00

0.50

0.75

1.00

1.25

1.50

GraphCL,Clintox

SimGRACE,Clintox

GraphLoG,Clintox

JOAO,Clintox

1400

1200

1600

2000

1750

1500

500

00

1000

1250

400

800

1000

300

400

750

500

00

250

100

0.2

L

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.72

LV

Z'T

0.00

0.25

0.50

0.75

1.25

1.50

1.75

GraphCL,Sider

SimGRACE,Sider

GraphLoG,Sider

JOAO,Sider

1600-

2000

1400

1200

1750

1500

500

1000

1250

400

000

1000

600

750

400

300

200

250

Z0

000

0.25

0.50

0.75

1.00

1.25

L50

(a) GraphCL

(b) GraphLoG

IPqde.6

102

anjeA aeynfurs

201

100

10-1

50

Singular Value Rank Index

100

150

200

250

graphlog

101

Singular Value

200

10-1

10-3

10-2

0

50

Singular Value Rank Index

100

150

200

250

00E

representation helps to increase the performance of existing methods, even on AC data - likely also because they have issues that still need to be resolved, such as the dimensional collapse we observed.

Additional Pre-training Data. Additionally, we validate the effectiveness of our methods on another pre-training dataset, GEOM (50K) [Liu et al., 2022a]. For an evaluation, we study TAE and apply TDL on GraphCL and JOAO. We follow all pre-training and fine-tuning settings of [Liu et al., 2022a], with the only difference being that we pre-train our models for 200 epochs instead. We compare the results from their original paper. As shown in Table 13, TAE alone reaches decent average performance, and surpasses SOTA on ClinTox and HIV. And we report the performance in regressive property prediction in Table 14 which shows that TDL enhances GraphCL and JOAO on many datasets. Please note that we considered these experiments only to obtain initial estimates; the GEOM data (which we used to be able to compare to related works) itself is too small for real pre-training if the basleline models are not able to exploit the 3D nature of the conformers.

(a) Sider

(b) Sider

(c) ClinTox

(d) ClinTox

Comparison

Comparison

Comparison

Comparison

0.7

0.6

SimGRACE+TDL

SimGRACE

0.6

0.7

GraphLoG+TDL

GraphLoG

0.7

0.6

0.7

0.6

0.5

0.5

0.5

0.4

0.4

0.4

0.4

E0

0.3

0.3

0.3

0.2

0.2 -

0.2

0.2

0.1

0.0

Hak

101112

0.1

0.0

101112

0.1

0.0

SimGRACE+TDL

SimGRACE

Task

0.1

0.0

GraphLoG+TDL

GraphLoG

Task

100200

400

800

1

,

000

0

.

5

0

.

6

0

.

7

0

.

8

GraphLoG+TDLatom

GraphLoG

SimGRACE+TDLatom

SimGRACE

100200

400

800

1

,

000

0

.

6

0

.

7

0

.

8

GraphLoG+TDLatom

GraphLoG

JOAO+TDLatom

JOAO

100200

400

800

1

,

000

0

.

4

0

.

6

0

.

8

GraphLoG+TDLatom

GraphLoG

SimGRACE+TDLatom

SimGRACE

100200

400

800

1

,

000

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

GraphLoG+TDLatom

GraphLoG

JOAO+TDLatom

JOAO

100200

400

800

1

,

000

0

.

5

0

.

55

0

.

6

GraphCL+TDLatom

GraphCL

SimGRACE+TDLatom

SimGRACE

100200

400

800

1

,

000

0

.

5

0

.

55

0

.

6

GraphLoG+TDLatom

GraphLoG

JOAO+TDLatom

JOAO

100200

400

800

1

,

000

0

.

5

0

.

6

0

.

7

GraphCL+TDLatom

GraphCL

SimGRACE+TDLatom

SimGRACE

100200

400

800

1

,

000

0

.

5

0

.

6

0

.

7

GraphLoG+TDLatom

GraphLoG

JOAO+TDLatom

JOAO

1

,

000 2

,

000 3

,

000 4

,

000 5

,

000 6

,

000

0

.

6

0

.

7

GraphCL+TDLatom

GraphCL

SimGRACE+TDLatom

SimGRACE

1

,

000 2

,

000 3

,

000 4

,

000 5

,

000 6

,

000

0

.

6

0

.

7

GraphLoG+TDLatom

GraphLoG

JOAO+TDLatom

JOAO

1

,

000 2

,

000 3

,

000 4

,

000 5

,

000 6

,

000

0

.

5

0

.

55

0

.

6

0

.

65

GraphCL+TDLatom

GraphCL

SimGRACE+TDLatom

SimGRACE

1

,

000 2

,

000 3

,

000 4

,

000 5

,

000 6

,

000

0

.

5

0

.

55

0

.

6

0

.

65

GraphLoG+TDLatom

GraphLoG

JOAO+TDLatom

JOAO

0

.

30

.

6

1

2

3

.

29

·

10

4

0

.

5

0

.

6

0

.

7

0

.

8

GraphCL+TDLatom

GraphCL

SimGRACE+TDLatom

SimGRACE

0

.

30

.

6

1

2

3

.

29

·

10

4

0

.

5

0

.

6

0

.

7

0

.

8

GraphLoG+TDLatom

GraphLoG

JOAO+TDLatom

JOAO

1

2

3

4

7

.

44

·

10

4

0

.

6

0

.

7

0

.

8

GraphCL+TDLatom

GraphCL

SimGRACE+TDLatom

SimGRACE

1

2

3

4

7

.

44

·

10

4

0

.

6

0

.

7

GraphLoG+TDLatom

GraphLoG

JOAO+TDLatom

JOAO

100200

400

800

1

,

000

0

.

6

0

.

7

0

.

8

GraphCL+TDLatom

GraphCL+TDLToDD

SimGRACE+TDLatom

SimGRACE+TDLToDD

100200

400

800

1

,

000

0

.

6

0

.

7

0

.

8

GraphLoG+TDLatom

GraphLoG+TDLToDD

JOAO+TDLatom

JOAO+TDLToDD

100200

400

800

1

,

000

0

.

5

0

.

6

0

.

7

GraphCL+TDLatom

GraphCL+TDLToDD

SimGRACE+TDLatom

SimGRACE+TDLToDD

100200

400

800

1

,

000

0

.

5

0

.

6

0

.

7

GraphLoG+TDLatom

GraphLoG+TDLToDD

JOAO+TDLatom

JOAO+TDLToDD

100200

400

800

1

,

000

0

.

4

0

.

6

0

.

8

GraphCL+TDLatom

GraphCL+TDLToDD

SimGRACE+TDLatom

SimGRACE+TDLToDD

100200

400

800

1

,

000

0

.

6

0

.

8

GraphLoG+TDLatom

GraphLoG+TDLToDD

JOAO+TDLatom

JOAO+TDLToDD

100200

400

800

1

,

000

0

.

5

0

.

55

0

.

6

GraphCL+TDLatom

GraphCL+TDLToDD

SimGRACE+TDLatom

SimGRACE+TDLToDD

100200

400

800

1

,

000

0

.

5

0

.

55

0

.

6

GraphLoG+TDLatom

GraphLoG+TDLToDD

JOAO+TDLatom

JOAO+TDLToDD

E Low-Data Scenario

Here we perform a more detailed experiment to illustrate how model performances change with increasing numbers of training samples. We fine-tune multiple models (GraphCL, JOAO, GraphLoG and SimGRACE) and TDL versions on 8 molecular property prediction datasets with training sets of different size. The experiment of each low-data size is run with 10 different subsets of the original training sets while keeping scaffolds. Figure 13 and Figure 14 show that TDL often yields remarkable improvements. Specifically, we observe that TDL's impact seems to depend on two factors: (1) in how far it is able to resolve issues of the baseline (e.g., the dimensional collapse of GraphCL), and (2) in how far the PIs used for TDL 'suit' the dataset considered (see also Figure 8). We observe that the latter yields a task which needs to be solved together with domain experts (i.e., finding the best knowledge to use to create PIs).

F Unsupervised Learning

Datasets. In the unsupervised context, we adopted various molecular benchmarks from TUDataset [Morris et al., 2020], including NCI1, MUTAG, PROTEINS, and DD. Table 15 shows statistics for datasets.

Configurations. For the unsupervised graph classification task, we first train a representation model contrastively using unlabeled data, then fix the representation model and train the classifier using labeled data. Following GraphCL [You et al., 2020], we employ a 5-layer GIN with a hidden size of 128 as our representation model and utilize an SVM as our classifier. The GIN is trained with a batch size of 128 and a learning rate of 0.001. Regarding graph representation learning, models are trained for 20 epochs and tested every 10 epochs. We conduct a 10-fold cross-validation on every dataset. For each fold, we utilize 90% of the total data as the unlabeled data for contrastive pre-training and the remaining 10% as the labeled testing data. Every experiment is repeated 5 times using different random seeds, with mean and standard deviation of accuracies (%) reported. We apply TDL on GraphCL and SimGRACE. The temperature parameter τ for TDL here is chosen from { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 , 10 . 0 } . Besides adding a TDL to the loss function in CL approaches, we did not make any modifications to the models.

Results. Table 16 displays the comparison among different models for unsupervised learning, revealing that a general performance improvement is achieved with the TDL. We also compute the normalized Euclidean distance of embeddings from these unsupervised datasets, and the histogram is illustrated in Figure 15. The findings are consistent with our prior observations in transfer learning; however, the impact may not be as substantial due to the smaller number of training samples.

G Ablation Studies

Different Filtrations - Geometry. Firstly, we consider detailed ablations of the filtration functions used in the graph filtration step of TAEahd. That is, we use individual PIs (i.e., each PI is/was constructed based on a separate filtration function) instead of concatenation as reconstruction targets for TAEahd. Consequently, we obtain 3 individual TAEs based on different PIs. We also conduct an experiment that involves integrating these 3 TAEs during the fine-tuning of downstream tasks. More specifically, we individually fine-tune these models, concatenate their embeddings after the readout, and finally project through a projection head for downstream task prediction. As demonstrated in Table 17, concatenation during fine-tuning (FT) does indeed combine the advantages of individual filtrations. These filtrations are very generic and the method is flexible in terms of what is useful downstream. That is, this approach would allow domain experts to flexibly use their own filtrations in combinations and together with existing methods, in dependence on the specific downstream data. This also suits the well-known fact that specific downstream data may benefit from specific domain knowledge.

GraphCL,PROTEINS

GraphCL,MUTAG

GraphCL,DD

GraphCL,NCI1

00E

500

2500

2000

700

400

2000

600

1500

1500

500

300

400

1000

1000

300

500

500

心

100

100

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

0.0

0.2

0.4

0.6

0.8

1.0

1.2

L.4

L6

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

SimGRACE,PROTEINS

SimGRACE,MUTAG

SimGRACE,DD

SimGRACE, NCI1

0008

25000

2000

7000

4000

20000

0009

1500

5000

3000

15000

4000

10000

1000

3000

2000

000S

00S

2000

1000

1000

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

0.00

0.25

0.50

0.75

1.00

1.25

1.50

0.00

0.25

0.500.75

1.00

1.25

1.50

1.75

For TDL, the filtration based on atom number provided overall best performance; we ran all experiments on PIs concatenated as above as well. We plan to investigate more in this direction in the future.

Different Filtrations - Domain Knowledge. Furthermore, we show results about incorporating different kinds of domain knowledge, here, atomic radius and electron affinity, as filtration functions. For TAE, we conduct two experiments: (1) We pre-train TAE using individual PIs generated by atomic radius and electron affinity as reconstruction targets; (2) We concatenate these 5 PIs (with the previous 3 PIs) as reconstruction targets. For TDL, we use these two individual PIs as topological fingerprints and apply TDL on GraphCL. Table 17 shows that these two domain knowledge-based filtration functions achieve commendable average performance. Yet, in summary, we do not observe considerable differences. Our hypothesis is that, similar to the atomic number, these kinds of domain knowledge capture the difference between the atoms, and hence are not too different overall. Based on this investigation, we plan to experiment with more complex sources of domain knowledge (e.g., the polarity of atoms, which is important information for the possible relationships between atoms).

Different Filtrations - Multiple Dimensions . Here we consider the multidimensional Vietoris-Rips (multi-VR) filtration [Demir et al., 2022]. And we use the atomic number, heat kernel signature with t = 0 . 1 and node degree as filtration functions in the first dimension, respectively. Then we compute PDs and then vectorize them to get 3 PIs. Finally, we concatenate these 3 PIs as reconstruction targets to pre-train TAE.

No Filtrations. Please observe first that our study's (initial) research question has been the exploration of the usefulness of PH and topological fingerprints for molecular SSL. Based on our investigation (see also Figure 8), there is one very interesting outcome of our research and follow-up research question related to TDL: TDL is an instance of a more general proposal 'XDL', a novel loss function for pre-training which is exploits distances between training samples. We conducted some initial experiments only using the Tanimoto distance between standard ECFP fingerprints and obtain decent results; see Table 17. In future research, we will investigate the generality and potential impact of XDL more (e.g., combining ECFPs and topological fingerprints).

Different Vectorizations of PDs . Persistence Vectorizations transform the obtained Persistent Homology information (PDs) into either a functional or a feature vector form. This representation is more suitable for ML tools compared to PDs. In our study, we employ Persistence Images for vectorization. Apart from this, we also explored other stable vectorizations for TAE, such as Persistence Landscapes [Bubenik et al., 2015] and Silhouettes [Chazal et al., 2014]. As shown in Table 18, ablation experiments demonstrate that Persistence Images outperform the other two vectorization techniques.

Hyperparameters Analysis . We further perform a sensitivity analysis on the parameter λ of GraphCL + TDLToDD across 8 downstream molecular tasks. The results are presented in Table 19. It

is noteworthy that as λ decreases, the weight of TDL also decreases, leading to a noticeable decline in performance. This observation underscores the effectiveness of TDL.

Different GNN Backbones . We apply TDL on GraphCL. As shown in Table 20, we verify the generality of TDL by experimenting with three popular GNN models: GIN [Xu et al.], GCN [Kipf and Welling], and GraphSAGE [Hamilton et al., 2017]. Pre-training with GIN achieves the best performance across datasets, hence our results confirm what has been observed in related works.

H PIs w/o SSL

Here we provide initial results for a comparison with 2 non pre-training ML models, i.e. support vector machine (SVM) and XGB [Chen and Guestrin, 2016] to test the regular prediction on 5 molecular property prediction tasks using PIs as topological fingerprints. And we also conduct smaller data experiments on SVM compared with TAE and TAE+ContextPred over 4 downstream datasets. Note that we performed a hyperparameter search to produce these ML results. As can be observed in Table 21 and Figure 16, the SSL methods provide large benefits. We are certainly aware of the fact that there are more complex methods to use these PIs, yet this basic comparison gives a good hint of how SSL based on PH compares to supervised methods more generally.

100200

400

800

1

,

000

0

.

5

0

.

6

0

.

7

0

.

8

ContextPred

TAEahd+ContextPred

SVM

100200

400

800

1

,

000

0

.

5

0

.

6

0

.

7

ContextPred

TAEahd+ContextPred

SVM

100200

400

800

1

,

000

0

.

4

0

.

6

0

.

8

ContextPred

TAEahd+ContextPred

SVM

100200

400

800

1

,

000

0

.

5

0

.

55

0

.

6

ContextPred

TAEahd+ContextPred

SVM

I Theoretical Analysis of TDL

In this section, we delve into a theoretical analysis of TDL, which was initially introduced in Section 3.2. We encourage readers to refer to Section 3.2 for any specific notation, terminology, or formulation details.

Recall TDL is defined as follows:

̸

̸

For the sake of convenience, we re-index the subscripts of samples, ensuring that the subscripts d i satisfy the sorted distances from the n-th sample, i.e., dis ( I n , I n ) ≤ dis ( I n , I d 1 ) ≤ dis ( I n , I d 2 ) ≤ dis ( I n , I d 3 ) ≤ ... ≤ dis ( I n , I d N -1 ) . Let sim ( z n , z m ) = z n z m to streamline the derivations. We thus obtain

We then analyze the gradients with respect to different samples.

Lemma 1. The gradient of TDL n with respect to the sample d i in latent space is formulated as:

In particular, ∇ z d 1 L TDL n ∝ -z n and ∇ z d N -1 L TDL n ∝ z n .

Proof. The results follow from direct computation. Let L TDL n,d j = -log e ( zn · z d j /τ ) ∑ k ∈ [ j,N -1] e ( zn · z d k /τ ) , then we have:

Next, we discuss ∇ z d i L TDL n,d j :

Therefore, we conclude:

Then we have:

The quantity ( e zn · z d 1 /τ ∑ k ∈ [1 ,N -1] e zn · z d k /τ -1 ) < 0 is a strictly negative scalar, allowing us to conclude the derivative ∇ z d 1 L TDL n is proportional to -z n . Similarly, we have:

Since ( ∑ j ∈ [1 ,N -2] e zn · z d N -1 /τ ∑ k ∈ [ j,N -1] e zn · z d k /τ ) > 0 we conclude in this case that the derivative ∇ z d N -1 L TDL n points in the direction z n .

The stability provided by certain topological fingerprints serves as a mathematically grounded, wellstudied, and efficient proxy for stability with respect to graphs. TDL as our objective is based on the stability of these topological fingerprints. Consequently, for each sample, we can determine a ranking of all other samples. Lemma 1 provides the gradient with respect to different samples. Specifically, samples that are distant from sample n in the topological space point in the direction -z n (negative gradient direction), while those close to sample n point in the direction z n . Consequently, molecules with more similar topological features are brought closer to each other in the embedding space.

J Potential of PH for Molecular ML

Persistent homology (PH) has emerged as a crucial method in topological data analysis, demonstrating significant potential in decoding complex patterns within molecular data, including protein prediction [Cang and Wei, 2017a,b, Nguyen et al., 2019, Xia and Wei, 2014, Liu et al., 2022b], virtual screening [Cang et al., 2018], and drug design [Liu et al., 2021]. The many works on PH in chemistry show that the domain is convinced by the usefulness of topological fingerprints already, and we therefore believe that this kind of knowledge should be investigated in the context of SSL, assuming that the latter becomes more important with the advancement of foundation models.

K Limitations & Broader Impact

Broader Impact. Graph data has a pervasive impact in many diverse fields and the popularity certainly increases the danger of an application of the models with negative effects (even if just due to ignorance). As pointed out in this work w.r.t. smaller data, we have to be aware of the assumptions we make in our research to inform readers on how the results translate to the real world. Our goal is to advance biomedicine, chemistry, and material science, where our research hopefully provides benefits one day, for both the current society and future generations.

Limitations. We have tried our best in addressing common issues, by providing a very detailed analysis, by considering TDL on top of various baselines, by also reporting mixed results, and by providing all code. However, due to time and resource constraints, and the broadness of our study, we were not able to experiment with a variety of topological fingerprints, analyze individual baselines (e.g., the dimensional collapse) in more detail, or complete the theoretical investigations.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this manuscript, the authors have developed an interesting self-supervised learning model, by the incorporation of persistent homology into contrastive learning module. More specifically, a special topological distance based contrastive loss is proposed. The model is novel, and the results are very promising. However, I have some concerns about the persistent homology analysis part.\n\nSTRENGTHS:\nThe authors have developed an interesting self-supervised learning model, by the incorporation of persistent homology into contrastive learning module. More specifically, a special topological distance based contrastive loss is proposed. The model is novel, and the results are very promising.\n\nWEAKNESSES:\nThe PH model is not explained clearly.\n\nQUESTIONS:\n1)\tAs a powerful tool in topological data analysis, persistent homology (PH) has demonstrated great power in molecular data analysis, PH-based molecular descriptors and fingerprints have already been extensively tested on various benchmark datesets and shown better performance than not only traditional molecular descriptors, such as ECFP, Morgan, daylight, etc, but also many deep learning models. Many related important references are not discussed in the paper, such as\n\nZ. X. Cang, Lin Mu and Guo-Wei Wei, Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening, PLOS Computational Biology, 14(1), e100592 (2018).\n\nZ. X. Cang and Guo-Wei Wei, TopologyNet: Topology based deep convolutional and multi-task neural networks for biomolecular property predictions, PLOS Computational Biology, 13(7), e1005690 (2017).\n\nZ. X. Cang and Guo-Wei Wei, Element specific persistent homology for the analysis and prediction of protein folding stability upon mutation, Bioinformatics,  33, 3549-3557 (2017).\n\nZ. X. Cang, Lin Mu and Guo-Wei Wei, Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening, PLOS Computational Biology,  14(1), e100592 (2018).\n\nDuc Duy Nguyen, Zixuan Cang, Kedi Wu, Menglun Wang, Yin Cao and Guo-Wei Wei, Mathematical deep learning for pose and binding affinity prediction and ranking in D3R Grand Challenges, Journal of Computer Aided Molecular Design, 33, 71-82  (2019).\n\nDuc Nguyen, Zixuan Cang, and Guo-Wei Wei, A review of mathematical representations of biomolecular data, Physical Chemistry Chemical Physics, 22, 4343-4367 (2020).\n\nXiang Liu, Xiangjun Wang, Jie Wu, and Kelin Xia, \"Hypergraph based persistent cohomology (HPC) for molecular representations in drug design.\" Briefings In Bioinformatics, 22 (5), bbaa411 (2021)\n\nXiang Liu, Huitao Feng, Jie Wu, and Kelin Xia, \"Dowker complex based machine learning (DCML) models for protein-ligand binding affinity prediction.\" PLOS Computational Biology, 18(4), e1009943 (2022)\n\nChi Seng Pun, Si Xian Lee, and Kelin Xia, \"Persistent-homology-based machine learning: a survey and a comparative study.\" Artificial Intelligence Review, (2022)\n\nD. Vijay Anand, Qiang Xu, Junjie Wee, Kelin Xia, and Tze Chien Sum, \"Topological feature engineering for machine learning based halide perovskite materials design\", npj Computational Materials, 8 (203) (2022)\n\n2) Mathematically, the filtration process from PH will only generate a nested sequence of simplicial complexes! In Figure 1, the plotted “chemical subgraphs” (during the filtration process) are not the general simplicial complexes (Vietoris-Rips complex or Alpha complex), as there are double bonds. Note that a double bond is illustrated as two edges which appear simultaneous among two vertices. This representation is not mathematically rigorous! The authors can use the common graph but add edge features to denote the double bond! Note that it is possible to use cellular complex to represent “double bonds”, but their persistent homology will be different!\n\n3) Page 3, line 100, their filtration ends with the original graph. In this way, their Betti_1 bars will never die, as there are NO 2-simplexes in their model? The authors are suggested to double check this setting. More discussion will be given below. \n\n4) Page 3, line 106, the authors mention that they use $H_k (G_i)$? They should specify what is the range for the integer $k$. Further, in their topological loss, do they consider both the Betti_0 and Betti_1, or just Betti_0/Betti_1?\n\n5) Page 5, line 209-212, “The fingerprints are usually constructed in a way so that they capture information about the molecular graph structure and sometimes additional domain knowledge; even if they do not capture the entire complexity of the molecules, they represent some, probably important aspects.” Even though PH models are important, they can only characterize homological information, such as individual components, circles, voids, cavities. In biomolecular, the Betti_0 is usually related to covalent bonds, Betti_1 is related to pentagon (sugar ring) and hexagon (benzene ring). These findings have already been widely known, many references can be found,\n\nKelin Xia, Xin Feng, Yiying Tong and Guo-Wei Wei, \"Persistent homology for the quantitative prediction of fullerene stability.\" Journal of Computational Chemistry, 36, 408-422(2015).\n\nKelin Xia and Guo-Wei Wei, \"Persistent homology analysis of protein structure, flexibility and folding.\" International Journal for Numerical Methods in Biomedical Engineering, 30(8), 814-844(2014)\n\nZhenyu Meng, D Vijay Anand, Yunpeng Lu, Jie Wu, and Kelin Xia, \"Weighted persistent homology for biomolecular data analysis.\" Scientific Report, 10 (1), 1-15 (2020)\n\nThe authors are suggested to add an example or some more discussions to explain what the information is captured in their PH models. Further, the general filtration is usually based on atomic distance or some special weighted distances, in this way, higher-dimensional simplicial complexes are generated!  And both Betti_0, Betti_1, and Betti_2 have their clear biological meanings! If the filtration process ends with the original graph (as illustrated in Page 3, line 100), there are no Betti_2 information, and Betti_1 bars will never die! The authors are suggested to compare with these existing approaches to show their advantages or add some more discussions about the difference.\n\nLIMITATIONS:\nMissing many important related literature. The advantage of their filtration process is not clear.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper uses self-supervised learning tools for graph representation learning by facilitating topological data analysis (TDA) methods. In particular, for molecular representation learning, the authors use persistent homology outputs to improve the embeddings obtained by GNNs. They evaluated their model in molecular property prediction problem, and consistently got performance improvements.\n\nSTRENGTHS:\nGNNs and TDA are both very successful and completely different methods in graph representation learning. In the past years, there are several approaches to integrate these two methods effectively. With this aim, the paper proposes a new way to use TDA output to improve node embeddings in GNNs by using contrastive learning ideas. The idea is novel and has a lot of room for improvement.\n\nMolecular Representation Learning is a significant application area for graph representation learning. The authors applied their model in this domain, in particular, molecular property prediction. They obtain strong results on this important question.\n\nThe paper's experimental part and ML details are strong. The authors made an in-depth analysis of the model from various angles.\n\nWEAKNESSES:\nThe experimental results (Table 4) do not show significant improvements in several cases. \n\nThe results only report the performance of internal models. It would be nice to see the comparison with the SOTA results on these datasets. \n\nPH construction seems weak as it does not use clique complexes, and only uses nodes and edges in the filtration, i.e., the top dimension is set to be 1. This filtration are not commonly used in graphs as it reduces PH to only node and edge counting by using a simple Euler Characteristics argument. However, fortunately, this does not affect their performance in this setting since molecular graphs are planar and do not have loops of length 3, as all loops have length $\\geq 5$. The authors should add a note for nonexperts that for molecular graphs, this trivial filtration setting is equivalent to the traditional clique complex setting for sublevel filtration because of the special structures of molecular graphs (no cycle of length 3). For TDL, PI is a good choice, but for TAE, it looks weird. To be used in such a loss function, there are better stable PH vectorizations, e.g., Silhouette, landscape.\n\nQUESTIONS:\nInstead of using PH output  in the loss function to improve GNN embeddings, did you consider directly combining them, e.g., by simply concatenating PIs with GNN embeddings? I know this is a completely different approach, but this is a more direct method to combine both outputs.\n\nLIMITATIONS:\nThere are various choices to be made in several places for the model. On one side, it gives flexibility to adapt the model to different settings, but on the other hand, it needs expertise in several domains for fine-tuning.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two molecular self-supervised learning methods, which consists of fingerprint autoencoder and topological distance contrastive learning. The insight behind this paper is to utilize topological fingerprint as a supervision in self-supervised learning. Thus, the authors reconstruct the topological fingerprint of a given molecule with autoencoder and filter out similar molecules in negative views in contrastive learning based on the similarity in topological distance space. The experimental results show that their method improves previous baselines in various downstream tasks.\n\nSTRENGTHS:\n- The paper is well written and easy to understand.\n\n- The experimental results are comprehensive; the authors considers several setups such as linear probing and fine-tuning.\n\nWEAKNESSES:\n- Lack of Novelty: Excluding similar molecules from negative sample set is already considered in [1]. Conceptually, the difference of TDL and [1] is that TDL utilizes PH and [1] utilizes ECFP fingerprint (I know that the loss of [1] is based on augmented molecules, but I think this does not make big difference). This limits the novelty of this paper.\n\n- Table 1 does not support the effectiveness of proposed method: Correlating the distance in embedding space with the distance in corresponding PIs are not the main purpose of molecular representation learning. If PIs are indeed very important, then why should we use learned representation of proposed method? Can't we just utilize PIs as the molecular representation? In other words, Table 1 and Table 17 seem to contradict.\n\n- Insufficient rationalization of the usage of PIs: In molecular domain, ECFP fingerprint is a widely applied molecular representation since it reflect the substructure-wise molecular information. Why should we use PIs in molecular representation learning?\n\n- Flexibility of TDL: The authors insisted that TDL can be flexibly and efficient applied with any graph contrastive learning framework. However, any other two existing methods can be composed with each other to improve the performance. For example, ContextPred + GraphCL is possible and the flexibility is not the unique feature of TDL. \n\n- Table 4 seems weak: TDL (or TAE) combined with existing method does improve the overall performance. However, Mole-BERT and SEGA shows better performance than the proposed method.\n\n----Sorry for confusion. I added the reference.\n\n[1] Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast, Wang et al., JCIM 2022\n\nQUESTIONS:\n- In Table 4, why some baseline methods are combined with TAE while others are combined with TDL? Can't TAE and TDL combined jointly?\n\n- How is the performance of TAE (or TDL) jointly trained with SEGA? Does this improve SEGA itself or SEGA + other method (e.g., SEGA + ContextPred)?\n\n- I would be convinced with the experimental results if the authors compare \"TAE + TDL\" vs other methods (Please refer \"Flexibility of TDL\" in Weakness).\n\nLIMITATIONS:\nYes. The authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two approaches to leverage topological information (obtained from persistent homology) for molecular representation learning in a self-supervised setting. The first (TAE) uses an encoder-decoder architecture whose decoder aims to recover topological fingerprints. The second approach (TDL) consists of a contrastive loss based on the similarity between topological fingerprints. The latter is combined with existing contrastive learning methods. Experiments on linear probing and downstream prediction tasks show the efficacy of the proposals.\n\nSTRENGTHS:\n- Ablation studies: There is a substantial number of experiments and ablation studies.\n- I like the simplicity of the proposed approach. \n- Flexibility: TDL can be combined with most SSL approaches.\n\nWEAKNESSES:\n- Overall, I believe the paper provides limited insight to support the proposals. Also, it does not discuss which structural information the proposed approach captures but not existing methods. From a conceptual level, we know that 1-WL GNNs cannot capture information even from simple homology (e.g., number of independent cycles of a graph). Thus, TAE has inherent limits/failures. In other words, the topological information we loose after pushing a graph through a GNN (which would be captured by TDA) cannot be recovered from GNN embeddings.\n- Results on downstream tasks: Based on Table 4, the gains from TDL look marginal. The gain is less than one standard deviation from the base model for many datasets.\n- Incorporation of domain knowledge: The claim that the proposal allows for incorporating domain knowledge seems overstated. The basis for such a claim comes from the choice of the filtration function. However, it is unclear how different filtration functions affect the topological embeddings --- thus, domain experts cannot leverage their knowledge to choose the filtration functions.\n- TAE vs. TDL...which one should we use? The paper says that \"TAE, which we developed for comparison purposes only...\" (line 283). I am unsure whether TAE should be introduced as a main contribution or as a baseline (in the experiments) for assessing the feasibility of learning the topological fingerprints with a simple architecture.\n\nQUESTIONS:\n1. Could the authors elaborate on why the fact that 'the Euclidean distance between PIs is stable with respect to the 1-Wasserstein distance between PDs' (line 111-112) is relevant here? Don't we want stability wrt to the input graphs? \n2. What does the paper mean by calibrated distances?\n3. Can the proposed methods be extended to employ learnable filtration functions?\n4. A significant part of the experiments is devoted to showing the alignment between the learned molecular representations and the topological fingerprints. Isn't it naturally expected from the proposed design (e.g., additional loss term)?\n5. Could the authors elaborate more on the fact that TAE can capture inter-molecule relationships if they learn the PIs? Can't GNN embeddings also learn important structural information and be stable?\n6. Have the authors considered applying only TDL without other loss terms from CL methods? If yes, how well does it work?\n7. The paragraph 'linear probing' (line 281) says 'we evaluated extensively using MLPs on the representations of the pre-trained graph encoders...'. Shouldn't this employ linear models instead of MLPs?\n8. Duplicated text, e.g.,\n    - Lines 1-5 --> 20-23\n    - Lines 24-31 --> 199-206\n    - Lines 5-6 --> 37-38\n9. Typos:\n    - GraphCL propose (line 32)\n    - JOAO extend (line 33)\n    - and to exploit (line 65)\n    - TDL provides as a form of regularization (line 233)\n\nLIMITATIONS:\nThe authors mention limitations in the main paper (section 5).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper explores self supervised learning in the context of molecular representation, specifically based on persistent homology. The paper proposes an autoencoder to demonstrate the general representational power of PH and a contrastive-learning-based loss that can be applied to existing SSL approaches. The proposed approach is evaluated for molecular property predictions, showing improved representations and predictive power compared to baselines across different tasks. The claim is that the new loss function enhances baseline performance particularly with small datasets.

STRENGTHS:
The paper is well written and the idea is novel and interesting.

WEAKNESSES:
- Given the technical nature of PH, and its origin in the domain of topological data analysis, a more mathematical foundation of the methods in the paper would be desired.

QUESTIONS:
- Since PH naturally offers multiple data views, I wonder whether the learned representations can be more explainable, that is, not just focusing on atom based attribution but also having explanations of more global features/subgroups, for instance, along the lines of 
Bertolini et al. "Beyond Atoms and Bonds: Contextual Explainability via Molecular Graphical Depictions".

LIMITATIONS:
The authors describe the limitations of their approach.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11