SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Further Intuition. Essentially, TDL provides a form of regularization. It encourages the model to push molecule representations less far away from the sample under consideration if they are similar to it in terms of the topological fingerprints. Given the stability of those, we hence obtain an embedding space with better calibrated distances (i.e., distances in terms of PH, between persistence diagrams of the molecule graphs). This can be partly observed theoretically, in the directions of the gradients; see Appendix I for an initial analysis. We show this empirically by calculating the correlation between distances between the molecule representations after pre-training and the PIs (see Table 1), and by visualizing the distances in Figure 5. Our evaluation further shows that the fine-grained supervision may solve deficiencies of CL models in that it forces them to capture crucial features of the input. Further, the improved embedding space particularly suits low-data downstream scenarios.
On Views. While we consider the modeling of the sample relationships to be most unique and offer great potential to complement other models, we note that TDL can be similarly applied over views.
- Do TAE and TDL lead to, in a sense, calibrated distances in the representation space?
- Do we obtain improved representations more generally, based on established SSL metrics?
- What impact do we see on downstream performance, and does it justify our proposal?
Our Models. We apply both TAE and TDL with different filtration functions, marked by a subscript. First, we apply a most simple filtration atom based on atomic mass. This allows showing that, even by considering less information than the baseline GNNs, which additionally apply atom chirality and connectivity, topological modeling may exploit additional facets of the data. Since TAE is a standalone model, atomic mass is not enough to let it fully capture the molecular nature. For that
Table 1: Pearson correlation coefficients (\%) between distances in embedding space and distances between corresponding PIs, for samples from various MoleculeNet datasets. Highlighted are clear decreases and clear increases (i.e., considering standard deviation).
Tox21	ToxCast	Sider	ClinTox	MUV	HIV	BBBP	Bace
ContextPred	31.2 (0.4)	2.5 (0.0)	61.6 (0.2)	15.6 (0.5)	37.2 (0.1)	20.6 (0.1)	3.7 (0.3)	12.7 (0.2)
+ TAEahd	35.6 (0.3)	12.2 (0.6)	60.6 (0.3)	6.2 (1.6)	55.9 (0.1)	30.9 (0.1)	2.7 (0.7)	25.9 (0.2)
GraphCL	15.6 (0.4)	7.4 (0.8)	52.6 (0.2)	16.7 (0.8)	35.4 (0.2)	18.3 (0.2)	6.2 (1.8)	10.5 (0.2)
+ TDLatom	55.4 (0.2)	14.5 (0.4)	66.6 (0.3)	21.9 (0.7)	65.2 (0.3)	41.5 (0.1)	7.0 (1.4)	34.1 (0.3)
JOAO	25.1 (0.8)	3.2 (1.7)	62.4 (0.3)	20.2 (1.3)	48.0 (0.2)	29.5 (0.3)	2.7 (1.4)	24.8 (0.3)
+ TDLatom	49.3 (0.3)	14.6 (2.1)	65.2 (0.1)	26.2 (1.0)	60.2 (0.2)	40.6 (0.4)	6.7 (0.8)	36.0 (0.4)
SimGRACE	10.3 (2.9)	9.8 (0.6)	59.9 (1.1)	4.8 (1.6)	26.5 (0.3)	21.6 (0.2)	9.5 (2.1)	4.4 (0.1)
+ TDLatom	48.8 (0.9)	12.8 (1.7)	61.5 (0.2)	20.2 (0.5)	71.9 (0.1)	49.1 (0.2)	8.5 (2.0)	38.9 (0.1)
GraphLoG	16.8 (0.2)	2.9 (0.5)	34.2 (0.2)	3.6 (0.8)	20.4 (0.1)	9.4 (0.2)	3.3 (0.9)	12.6 (0.2)
+ TDLatom	44.2 (0.4)	11.8 (1.3)	50.5 (0.4)	22.4 (1.2)	63.9 (0.1)	46.2 (0.2)	1.9 (0.4)	40.3 (0.1)
reason, we consider three filtrations (atomic mass, heat kernel signature, node degree) and concatenate the corresponding PIs, denoted by ahd. Finally, to show the real potential of PH, we include the ToDD filtration Demir et al. [2022], which combines atomic mass with additional domain knowledge, partial charge and bond type, inside a more complex multi-dimensional filtration. See Appendix A.
Baselines & Datasets. For a comprehensive evaluation, we apply TDL on a variety of existing CL approaches: GraphCL, JOAO, GraphLoG, and SimGRACE (see Section 1). Note that this goes far beyond other CL extensions which are often evaluated with a single approach only [You et al., 2021, Xia et al., 2023b]. We also study TAE on top of the established ContextPred [Hu* et al., 2020], to get an idea of its complementary nature. Model configurations and experimental settings are described in Appendix A. For pre-training, we considered the most common dataset following [Hu* et al., 2020], 2 million unlabeled molecules sampled from the ZINC15 database [Sterling and Irwin, 2015]. For downstream evaluation, we focus on the MoleculeNet benchmark [Wu et al., 2018a] here, the appendix contains experiments on several other datasets.
4.1 Analysis of Representations after Pre-training
Calibrated Distances in Embedding Space, Tables 1, 7, 9, Figure 5.
TAE successfully learns the PI space for the molecules in MoleculeNet in the sense that there is a strong correlation between the PIs and their reconstructions (Table 7, appendix). For TDL, we observe for all baselines considerable increases in terms of correlation between distances in PI space and in representation space; we use the embeddings from linear probing. There are two notable exceptions and generally smaller increases on one dataset, BBBP. We also compared the ROGI score [Aldeghi et al., 2022] which, in a nutshell, measures in how far (dis)similarity of molecules (i.e., we use Euclidean distance of embeddings) is reflected in label similarity (Table 9, appendix). This gives us some idea in terms of downstream labels and the mixed results reflect the variety of the data. Finally, the alignment figure on the right clearly visualizes that the PI embedding space is much more fine-grained in terms of distances than the GNN space, and that the distribution of GraphCL seems to get correctly adapted in that distances
Figure 5: Normalized Euclidean distances for pairs of embeddings after pretraining of same/different Bace class, dark/light:  $\mathrm{PI}_{\mathrm{ToDD}}$ , green; GraphCL, blue; GraphCL+TDL $_{\mathrm{ToDD}}$ , red.
between "positive" pairs generally shrink below the ones of "negative" pairs.
Mitigating Dimensional Collapse, Figure 6. One of the most interesting of our findings is the fact that GraphCL and GraphLoG suffer from dimensional collapse (i.e., the embeddings span a lower-dimensional subspace instead of the entire available embedding space, [Wang et al., 2022a] also observe this for GraphCL) and that TDL successfully mitigates this. This can be observed in
Figure 6: Singular values of covariance matrices of the representations; Bace (left), Clintox (right).
Table 2: Linear/MLP probing: molecular property prediction; binary classification, ROC-AUC (%).
Tox21	ToxCast	Sider	ClinTox	MUV	HIV	BBBP	Bace	Average
PIatom	56.9 (0.3)	51.5 (0.4)	56.1 (0.5)	45.5 (1.1)	58.6 (0.7)	68.4 (0.6)	47.1 (0.4)	48.6 (0.7)	54.08
PIToDD	65.8 (0.3)	50.3 (0.3)	58.1 (0.5)	56.9 (1.3)	55.7 (0.9)	70.8 (0.4)	57.1 (0.7)	67.8 (0.8)	60.31
ECFP	68.8 (0.3)	57.0 (0.2)	62.3 (0.3)	69.2 (0.8)	66.1 (0.4)	69.4 (0.2)	63.2 (0.3)	73.6 (0.8)	66.20
ECFP || PIToDD	69.6 (0.4)	56.3 (0.3)	60.9 (0.6)	76.7 (1.1)	64.0 (0.6)	71.6 (0.5)	63.0 (0.4)	76.8 (1.0)	67.36
PIatom, MLP	57.2 (0.5)	52.5 (0.4)	56.6 (0.7)	49.8 (1.4)	60.5 (1.6)	69.9 (0.4)	48.8 (1.0)	53.3 (1.1)	56.08
PIToDD, MLP	66.7 (0.3)	52.5 (0.4)	58.6 (0.6)	61.8 (1.6)	60.1 (0.4)	71.6 (0.7)	57.3 (0.9)	68.2 (1.3)	62.03
ECFP, MLP	70.1 (0.4)	59.8 (0.4)	59.6 (0.6)	67.8 (0.9)	61.7 (0.8)	69.1 (1.0)	58.6 (1.3)	72.1 (1.7)	64.85
ECFP || PIToDD, MLP	71.1 (0.6)	57.8 (0.4)	59.2 (0.7)	80.7 (2.1)	64.9 (1.1)	72.8 (1.7)	63.1 (0.8)	76.7 (0.9)	68.28
TAEahd	67.7 (0.2)	61.2 (0.2)	55.8 (0.3)	58.1 (0.7)	70.2 (0.8)	72.5 (0.5)	61.1 (0.2)	74.3 (0.2)	65.11
TAEToDD	70.4 (0.2)	60.8 (0.1)	61.1 (0.1)	68.4 (0.7)	72.3 (0.3)	73.9 (0.2)	61.6 (0.4)	67.6 (0.6)	67.01
ContextPred	68.4 (0.3)	59.1 (0.2)	59.4 (0.3)	43.2 (1.7)	71.0 (0.7)	68.9 (0.4)	59.1 (0.2)	64.4 (0.6)	61.69
+ TAEahd	69.7 (0.1)	59.2 (0.2)	59.5 (0.3)	56.1 (1.1)	76.5 (0.9)	68.9 (0.2)	61.1 (0.4)	65.6 (0.5)	64.58
+ TAEToDD	69.0 (0.1)	59.8 (0.4)	60.0 (0.4)	53.3 (1.3)	70.8 (0.3)	70.0 (0.7)	60.9 (0.5)	62.7 (0.5)	63.31
GraphCL	64.4 (0.5)	59.4 (0.2)	54.6 (0.3)	59.8 (1.2)	70.2 (1.0)	63.7 (2.3)	62.4 (0.7)	71.1 (0.7)	63.20
+ TDLatom	72.0 (0.4)	61.1 (0.2)	59.7 (0.6)	65.3 (1.3)	76.1 (0.9)	68.2 (1.1)	65.4 (0.9)	76.4 (1.1)	68.02
+ TDLToDD	72.7 (0.5)	60.8 (0.4)	58.9 (0.8)	64.1 (1.7)	72.7 (1.4)	69.7 (1.2)	64.5 (0.8)	76.1 (1.3)	67.44
JOAO	70.6 (0.4)	60.5 (0.3)	57.4 (0.6)	54.1 (2.6)	69.8 (1.9)	68.1 (0.9)	63.7 (0.3)	71.2 (1.0)	64.42
+ TDLatom	70.5 (0.3)	60.4 (0.2)	57.8 (1.5)	54.6 (1.3)	74.2 (1.6)	68.2 (0.6)	65.2 (0.3)	72.7 (3.1)	65.41
+ TDLToDD	71.7 (0.4)	61.3 (0.3)	58.9 (0.7)	52.4 (1.7)	69.6 (1.7)	69.9 (0.6)	64.1 (0.5)	72.6 (0.9)	65.06
SimGRACE	64.6 (0.4)	59.1 (0.2)	54.9 (0.6)	63.4 (2.6)	67.4 (1.2)	66.3 (1.5)	65.4 (1.2)	67.8 (1.3)	63.61
+ TDLatom	68.6 (0.3)	61.1 (0.2)	59.5 (0.4)	62.2 (1.7)	69.7 (2.0)	69.5 (1.8)	60.6 (0.5)	72.1 (0.7)	65.41
+ TDLToDD	70.1 (0.3)	60.3 (0.3)	59.1 (0.3)	65.1 (1.4)	71.4 (1.1)	71.1 (0.7)	64.9 (0.6)	73.4 (0.8)	66.93
GraphLoG	67.2 (0.2)	57.9 (0.2)	57.9 (0.3)	57.8 (0.9)	64.2 (1.1)	65.0 (1.3)	54.3 (0.7)	72.3 (0.9)	62.08
+ TDLatom	72.1 (0.3)	62.0 (0.2)	60.7 (0.2)	56.6 (0.8)	73.0 (0.9)	70.4 (0.9)	61.2 (0.4)	76.8 (0.7)	66.59
+ TDLToDD	70.7 (0.2)	60.7 (0.3)	61.5 (0.3)	59.5 (0.5)	72.9 (1.8)	71.6 (0.8)	62.1 (0.3)	80.1 (0.4)	67.39
Table 3: Linear probing: given two molecules, predict distance between their PIs; MSE.
Tox21	ToxCast	Sider	ClinTox	MUV	HIV	BBBP	Bace
ContextPred	6.330 (0.007)	5.351 (0.084)	22.227 (0.102)	4.234 (0.039)	2.342 (0.001)	6.127 (0.009)	4.907 (0.072)	3.591 (0.038)
+ TAEahd	5.970 (0.002)	5.263 (0.003)	21.262 (0.070)	4.255 (0.023)	1.851 (0.003)	5.752 (0.023)	4.901 (0.024)	3.069 (0.012)
GraphCL	6.873 (0.011)	5.440 (0.031)	24.019 (0.059)	4.537 (0.006)	2.411 (0.006)	5.996 (0.006)	5.666 (0.003)	3.858 (0.007)
+ TDLatom	5.371 (0.031)	5.191 (0.023)	20.641 (0.022)	4.231 (0.025)	1.645 (0.004)	5.001 (0.022)	4.936 (0.002)	2.375 (0.008)
JOAO	6.465 (0.032)	5.277 (0.010)	21.299 (0.085)	4.277 (0.118)	2.061 (0.001)	5.418 (0.016)	4.995 (0.036)	2.782 (0.004)
+ TDLatom	5.642 (0.011)	5.187 (0.010)	20.515 (0.044)	4.185 (0.027)	1.749 (0.004)	5.011 (0.028)	4.979 (0.015)	2.377 (0.002)
SimGRACE	10.015 (0.050)	13.387 (4.770)	26.514 (0.049)	4.669 (0.017)	2.591 (0.005)	6.150 (0.023)	6.167 (0.035)	13.054 (0.938)
+ TDLatom	5.359 (0.005)	5.263 (0.016)	20.630 (0.034)	4.071 (0.042)	1.545 (0.007)	4.714 (0.033)	5.131 (0.011)	2.296 (0.003)
GraphLoG	6.877 (0.012)	5.279 (0.004)	24.073 (0.038)	4.482 (0.055)	2.857 (0.004)	6.591 (0.015)	5.029 (0.079)	3.918 (0.004)
+ TDLatom	5.814 (0.032)	5.212 (0.018)	21.441 (0.044)	4.237 (0.015)	1.719 (0.005)	4.669 (0.010)	4.962 (0.015)	2.235 (0.005)
the singular values of the covariance matrix of the representations, where vanishing values hint at collapsed dimensions. The phenomenon has recently been observed in computer vision and there are only initial explanations to date [Hua et al., 2021]. However, here, we observe the collapse only over the downstream data. Our hypothesis is that TDL's fine-grained supervision forces the models to capture relevant features, which they might have neglected otherwise. We also depict the graphs for SimGRACE and JOAO, which look very different, reflecting the variety of the approaches. There is basically no difference for JOAO(+TDL), while TDL shrinks the values for SimGRACE; the latter is less optimal and needs further investigation.
Table 4: Binary classification over MoleculeNet; ROC-AUC, % Pos. is min/med/max for multi-task.
Tox21	ToxCast	Sider	ClinTox	MUV	HIV	BBBP	Bace	Average
# Molecules	7,831	8,575	1,427	1,478	93,087	41,127	2,039	1,513	-
# Tasks	12	617	27	2	17	1	1	1
% Positives	2.4/4.6/12.0	0.2/1.3/20.5	1.5/66.3/92.4	7.6/50.6/93.6	0.03/0.03/0.03	3.5	76.5	45.7
No pretrain (GIN)	74.6 (0.4)	61.7 (0.5)	58.2 (1.7)	58.4 (6.4)	70.7 (1.8)	75.5 (0.8)	65.7 (3.3)	72.4 (3.8)	67.15
AD-GCL [Suresh et al., 2021]	76.5 (0.8)	63.0 (0.7)	63.2 (0.7)	79.7 (3.5)	72.3 (1.6)	78.2 (0.9)	70.0 (1.0)	78.5 (0.8)	72.67
iMolCLR [Wang et al., 2022b]	75.1 (0.7)	63.5 (0.4)	59.4 (1.0)	81.0 (2.6)	74.7 (1.9)	77.3 (1.2)	69.6 (1.2)	77.3 (1.0)	72.24
Mole-BERT [Xia et al., 2023b]	76.8 (0.5)	64.3 (0.2)	62.8 (1.1)	78.9 (3.0)	78.6 (1.8)	78.2 (0.8)	71.9 (1.6)	80.8 (1.4)	74.04
SEGA [Wu et al., 2023]	76.7 (0.4)	65.2 (0.9)	63.6 (0.3)	84.9 (0.9)	76.6 (2.4)	77.6 (1.3)	71.8 (1.0)	77.0 (0.4)	74.17
\( TAE_{ahd} \)	75.2 (0.8)	63.1 (0.3)	61.9 (0.8)	80.6 (1.9)	74.6 (1.8)	73.5 (2.1)	67.5 (1.1)	82.5 (1.1)	72.36
\( TAE_{ToDD} \)	76.8 (0.9)	64.0 (0.5)	61.9 (0.8)	79.3 (3.6)	75.8 (3.2)	75.9 (1.1)	70.4 (0.8)	81.6 (1.4)	73.22
ContextPred	75.7 (0.7)	63.9 (0.6)	60.9 (0.6)	65.9 (3.8)	75.8 (1.7)	77.3 (1.0)	68.0 (2.0)	79.6 (1.2)	70.89
+ \( TAE_{ahd} \)	76.4 (0.5)	63.2 (0.4)	62.0 (0.7)	74.6 (4.4)	76.7 (1.6)	77.7 (1.2)	68.9 (1.1)	80.7 (1.6)	72.53
+ \( TAE_{ToDD} \)	75.7 (0.4)	63.1 (0.3)	61.3 (0.5)	72.1 (1.3)	77.2 (1.8)	77.6 (1.1)	69.6 (0.9)	80.1 (1.4)	72.09
GraphCL	73.9 (0.7)	62.4 (0.6)	60.5 (0.9)	76.0 (2.7)	69.8 (2.7)	78.5 (1.2)	69.7 (0.7)	75.4 (1.4)	70.78
+ \( TDL_{atom} \)	75.3 (0.4)	64.4 (0.3)	61.2 (0.6)	83.7 (2.7)	75.7 (0.8)	78.0 (0.9)	70.9 (0.6)	80.5 (0.8)	73.71
+ \( TDL_{ToDD} \)	75.2 (0.7)	64.2 (0.3)	61.5 (0.4)	85.2 (1.8)	75.9 (2.1)	77.9 (0.8)	69.9 (0.9)	81.2 (1.9)	73.88
JOAO	75.0 (0.3)	62.9 (0.5)	60.0 (0.8)	81.3 (2.5)	71.7 (1.4)	76.7 (1.2)	70.2 (1.0)	77.3 (0.5)	71.89
+ \( TDL_{atom} \)	75.5 (0.3)	63.8 (0.2)	60.6 (0.5)	76.8 (1.5)	73.8 (1.9)	78.3 (1.2)	70.3 (0.5)	78.7 (0.6)	72.22
+ \( TDL_{ToDD} \)	75.2 (0.3)	63.6 (0.2)	61.6 (0.6)	80.7 (3.3)	74.6 (1.6)	77.4 (0.9)	71.3 (0.8)	81.0 (2.2)	73.18
SimGRACE	74.4 (0.3)	62.6 (0.7)	60.2 (0.9)	75.5 (2.0)	75.4 (1.3)	75.0 (0.6)	71.2 (1.1)	74.9 (2.0)	71.15
+ \( TDL_{atom} \)	74.7 (0.5)	63.0 (0.3)	59.5 (0.4)	73.7 (1.5)	75.9 (1.6)	77.3 (1.1)	69.5 (0.9)	79.1 (0.5)	71.59
+ \( TDL_{ToDD} \)	75.6 (0.4)	63.3 (0.5)	59.9 (0.8)	82.4 (2.5)	75.6 (2.0)	76.1 (1.3)	69.9 (0.8)	78.9 (1.6)	72.71
GraphLoG	75.0 (0.6)	63.4 (0.6)	59.3 (0.8)	70.1 (4.6)	75.5 (1.6)	76.1 (0.8)	69.6 (1.6)	82.1 (1.0)	71.43
+ \( TDL_{atom} \)	76.1 (0.7)	63.7 (0.4)	59.9 (1.0)	75.7 (3.5)	75.7 (1.2)	76.2 (1.8)	69.6 (1.2)	82.2 (1.5)	72.39
+ \( TDL_{ToDD} \)	75.9 (0.8)	63.5 (0.7)	63.4 (0.3)	79.8 (1.9)	75.6 (1.1)	76.2 (1.6)	70.7 (0.9)	82.1 (1.9)	73.39
Linear Probing, Tables 2, 3, 10. We evaluated extensively using a linear layer on the representations of the pre-trained graph encoders over the MoleculeNet data in terms of binary classification and a custom distance prediction task. The probing, also in terms of MLPs, demonstrates that PIs possess a complementary nature to ECFP. Furthermore, TAE, which we developed for comparison purposes only, is competitive with the baselines and improves ContextPred. In Appendix C, we additionally show some results over the more challenging activity cliff data. Apart from JOAO+TDL, where the results are mixed, TDL yields overall impressive increases. Interestingly, the more complex ToDD-based version is not always better than the simpler one.
Discussion. The results demonstrate that both TAE and TDL successfully approach the PI space in terms of distances between molecules. Yet, it also has to be noted that the PIs we chose do not always capture the main characteristics of the sample space, as seen for BBBP. Similarly, the mixed ROGI scores, which incorporate downstream labels, hint at a varied downstream performance. Nevertheless, the specifics of particular downstream datasets should not be confused with the more general molecular representation space. In that respect, especially TDL has been proven effective overall in improving representation power in a variety of probing tasks for all baselines. We have also shown that, on closer look, the effects can be very different with the considered baselines, depending on their individual deficiencies. For JOAO, we have seen an improvement in distance representation but not as large improvements thereafter, suggesting that it captures similar features already. Recall, however, that JOAO "only" adapts GraphCL in that it chooses between views more flexibly. In this regard, TDL represents a more efficient adaptation of GraphCL, which turns out to be also more effective in terms of SSL generally, as we show next.
4.2 Evaluation on Downstream Tasks
Transfer Learning Benchmarks, Tables 4, 12, Figure 12. First, note that the MoleculeNet benchmark contains various types of data. We have already observed the strong differences between Bace and BBBP in terms of our PIs. Here we additionally have some very large datasets and some with very many tasks; in both we expect the labels and their correlation to have increasing impact and reduce the one of the original embedding space. We report some most recent works that used the same pre-training data to provide a context for interpreting our results. As expected, TAE alone only reaches decent average performance, yet surpasses SOTA on ClinTox and Bace and also considerably improves ContextPred there. We see more variability than above in the effects TDL has on the individual baselines. We do not observe a general impact of dataset balance. To learn about the impact of labels the influence of multiple tasks, we ran some models also on individual tasks of the multi-task data and observe larger increases for TDL there (see figures in Appendix D). Altogether, the table shows mixed results for SimGRACE; some improvement for GraphLoG and, with a single exception, also for JOAO; and large improvements for GraphCL. Notably, when using the stronger
Figure 7: Performance over smaller datasets, subsets of Bace (left) and ClinTox (right).
ToDD filtration, TDL demonstrates convincing improvements across all baselines and makes them competitive with SOTA.
The Low-Data Scenario, Figures 2, 7, 13, 14. Inspired by the impressive increases we see in the quality of representations after pre-training, we experimented with considerably smaller datasets. This is a scenario which is relevant in practice since proprietary data from costly experiments is often only available in small numbers [Tom et al., 2023]. Here we see clearly that the improvement in representation space is reflected in downstream performance. There is still some dependence on the dataset but, overall, TDL often yields remarkable improvements. The introduction figure shows that also other models show room for improvement; note that GraphMVP uses 3D information.
Further Analysis. Due to space constraints, we present only the most impactful findings in this section and the rest in the appendix, in particular: (1) Ablation Studies. Our approach offers great variability in terms of, for instance, the graph encoder, the construction of the PIs, or even the type of topological fingerprint more generally. (2) Unsupervised learning. We observe similar but less pronounced trends, likely due to the smaller size of the data and fewer features available. (3) PIs w/o SSL. In view of the promising results with PIs in supervised ML [Krishnapriyan et al., 2021, Demir et al., 2022], we consider the PIs with XGB and SVM, our approaches based on pre-training are better. (4) Other datasets. We conducted experiments over several other datasets, including activity-cliff data, where TDL is shown beneficial more generally.
Discussion. In this section, we have shown results very different from the ones we usually see. In the benchmark, we observe some improvements and, for GraphCL+TDL, numbers competitive with SOTA, but they do not reach the general, large impact we saw in Section 4.1. However, we have such impact over smaller data. Altogether, TDL offers great benefits for all baselines we considered:
- It leads to much better calibrated distances in the representation space in terms of PH. Since the magnitude of increase is generally remarkable but varies with the data, this shows potential for further improvement, especially, in terms of choosing the right PIs.
- It considerably improves representations in terms of established SSL metrics.
- Downstream performance reflects this gain of representation power specifically in the low-data scenario. Given the importance of this kind of data in the real world and the deficiencies of the models we checked, we conclude that our proposal is not only novel and different in its distance-based modeling, but also that it advances molecular SSL in important aspects empirically.
In this paper, we focus on the quality of representations in molecular SSL, an important topic which has not been addressed greatly to date. We propose a novel approach of shaping the embedding space based on distances in terms of fingerprints from persistent homology. We have evaluated our topological-distance-based contrastive loss extensively, and show that it solves deficiencies of existing models and considerably improves various baselines in terms of representation power. In future work, we plan to analyze the relationship between specific PIs and downstream data in more detail, to further improve our approach, include external knowledge, and to put more focus on regression tasks.
This work was supported by National Key R&D Program of China (2021YFB3500700), NSFC Grant 62172026, National Social Science Fund of China 22&ZD153, the Fundamental Research Funds for the Central Universities and SKLSDE; correspondence to Lei Shi.
Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z. Li. A systematic survey of chemical pre-trained models, 2023a.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33: 5812-5823, 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121-12132. PMLR, 2021.
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning, pages 11548-11558. PMLR, 2021.
Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. In Proceedings of the ACM Web Conference 2022, pages 1070-1079, 2022.
Junran Wu, Xueyuan Chen, Bowen Shi, Shangzhe Li, and Ke Xu. Sega: Structural entropy guided anchor view for graph contrastive learning. arXiv preprint arXiv:2305.04501, 2023.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513-530, 2018a.
Ruoxi Sun, Hanjun Dai, and Adams Wei Yu. Does gnn pretraining help molecular representation? Advances in Neural Information Processing Systems, 35:12096-12109, 2022.
Hanchen Wang, Jean Kaddour, Shengchao Liu, Jian Tang, Matt Kusner, Joan Lasenby, and Qi Liu. Evaluating self-supervised learning for molecular graph embeddings. arXiv preprint arXiv:2206.08005, 2022a.
Mohammad Sadegh Akhondzadeh, Vijay Lingam, and Aleksandar Bojchevski. Probing graph representations. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 11630-11649. PMLR, 25-27 Apr 2023. URL https://proceedings.mlr.press/v206/akhondzadeh23a.html.
Jianyuan Deng, Zhibo Yang, Hehe Wang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang. Taking a respite from representation learning for molecular property prediction. arXiv preprint arXiv:2209.13492, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
Dashti Ali, Aras Asaad, Maria-Jose Jimenez, Vidit Nanda, Eduardo Paluzo-Hidalgo, and Manuel Soriano-Trigueros. A survey of vectorization methods in topological data analysis. arXiv preprint arXiv:2212.09703, 2022.
Aditi S Krishnapriyan, Joseph Montoya, Maciej Haranczyk, Jens Hummelshøj, and Dmitriy Morozov. Machine learning with persistent homology and chemical word embeddings improves prediction accuracy and interpretability in metal-organic frameworks. Scientific reports, 11(1):8888, 2021.