SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kelin Xia and Guo-Wei Wei. Persistent homology analysis of protein structure, flexibility, and folding. International journal for numerical methods in biomedical engineering, 30(8):814-844, 2014.
Xiang Liu, Huitao Feng, Jie Wu, and Kelin Xia. Dowker complex based machine learning (dcml) models for protein-ligand binding affinity prediction. PLoS Computational Biology, 18(4):e1009943, 2022b.
Zixuan Cang, Lin Mu, and Guo-Wei Wei. Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening. PLoS computational biology, 14(1): e1005929, 2018.
Xiang Liu, Xiangjun Wang, Jie Wu, and Kelin Xia. Hypergraph-based persistent cohomology (hpc) for molecular representations in drug design. Briefings in Bioinformatics, 22(5):bbaa411, 2021.
A Datasets and Experimental Details
Pre-training Datasets. As usual, for GNN pre-training, we use a minimal set of node and bond features that unambiguously describe the two-dimensional structure of molecules from the ZINC15 database following previous works [Hu* et al., 2020].
- Node features:
- Atom number: [1, 118]
- Chirality tag: {unspecified, tetrahedral cw, tetrahedral ccw, other}
- Edge features:
Bond type: {single, double, triple, aromatic}
Bond direction:  $\{-$ , endupright, enddownright\}
We note that the atom number implicitly provides domain knowledge in that it captures important knowledge about the atom's charge.
Downstream Task Datasets. We focus on molecular property prediction, where we adopt the widely-used 8 binary classification datasets contained in MoleculeNet [Wu et al., 2018a]. In addition, we experimented with 4 regression tasks from various low-data domains in line with the same setting as [Liu et al., 2022a]. We provide detailed information of these datasets in Table 5 and more information can be found in [Liu et al., 2022a]. To further evaluate the usefulness of our models, we also use the opioids-related datasets from [Deng et al., 2022]. The number (percentage) of activity cliff (AC) molecules are summarized in Table 6. Notably, among MDR1, MOR, DOR and KOR, nearly half molecules are equipped with activity cliff scaffolds. Finally, scaffold-split [Ramsundar et al., 2019] is used to splits graphs into train/val/test set as  $80\% /10\% /10\%$  which mimics real-world use cases.
Table 5: Summary for the molecule datasets.
Dataset	Task	# Tasks	# Molecules
BBBP	Classification	1	2,039
Tox21	Classification	12	7,831
ToxCast	Classification	617	8,576
Sider	Classification	27	1,427
ClinTox	Classification	2	1,478
MUV	Classification	17	93,087
HIV	Classification	1	41,127
Bace	Classification	1	1,513
ESOL	Regression	1	1,128
Lipo	Regression	1	4,200
Malaria	Regression	1	9,999
CEP	Regression	1	29,978
Table 6: Summary of activity cliffs in the opioids datasets.
Dataset	MDR1	CYP2D6	CYP3A4	MOR	DOR	KOR
# Mol. AC. Saff. (%)	594 (41.3)	710 (31.0)	926 (25.2)	1,627 (46.1)	1,342 (41.6)	1,502 (45.2)
Task	Classification	Classification	Classification	Classification	Classification	Classification
# Tasks	1	1	1	1	1	1
Computing Environment. The experiments are conducted with two RTX 3090 GPUs.
Model Configurations. Following [Hu* et al., 2020], we adopt a 5-layer Graph Isomorphism Network (GIN) [Xu et al.] with 300-dimensional hidden units as the backbone architecture. We use mean pooling as the readout function. During the pre-training stage, GNNs are pre-trained for 100 epochs with batch-size as 256 and the learning rate as 0.001. During the fine-tuning stage, we train for 100 epochs with batch-size as 32, dropout rate as 0.5, and report the test performance using ROC-AUC at the best validation epoch. To evaluate the learned representations, we follow the linear probing (linear evaluation) [Akhondzadeh et al., 2023], where a linear classifier (1 linear layer) is
trained on top of the frozen backbone. We probe two types of properties from the representation: the first is the molecular properties of downstream tasks, for which we use the same settings as fine-tuning and report ROC-AUC; the second is topological fingerprints (PIs) of downstream molecular graphs, for which we train for 50 epochs with batch-size as 256, dropout rate as 0.0, and report MSE and Pearson correlation coefficients between distances in embedding space and corresponding PI space. The experiments are run with 10 different random seeds, and we report mean and standard deviation.
Performance Comparison. We compare the proposed method with existing self-supervised graph representation learning algorithms (i.e. InfoGraph [Sun et al., 2019], EdgePred [Hamilton et al., 2017], ContextPred [Hu* et al., 2020], AttrMask [Hu* et al., 2020], GraphLoG [Xu et al., 2021], iMolCLR [Wang et al., 2022b], AD-GCL [Suresh et al., 2021], JOAO [You et al., 2021], SimGRACE [Xia et al., 2022], GraphCL [You et al., 2020], GraphMAE [Hou et al., 2022], MGSSL [Zhang et al., 2021], GPT-GNN [Hu et al., 2020], G-Contextual [Rong et al., 2020], G-Motif [Rong et al., 2020], GraphMVP [Liu et al., 2022a]). We report the results from their own papers with two exceptions: (1) for GraphMAE and GraphLoG, they reported the test performance of the last epoch; and (2) MGSSL and SimGRACE, they only used 3 random seeds; and (3) iMolCLR used  $\sim 10\mathrm{M}$  molecules for pre-training. We reproduce the results of these four models with the same protocol as [Hu* et al., 2020], utilizing the pre-trained models originally provided by the respective papers.
Topological Fingerprints. We set the highest dimension in the simplicial complex to 1. This doesn't impact performance because molecular graphs, being planar, rarely possess loops of length 3; in fact, the majority of loops exhibit a length of 5 or more.
Firstly, we use the sublevel filtration method. In terms of filtration functions, we use atomic number. For every molecular graph, we utilize an extended persistence module [Cohen-Steiner et al., 2009, Yan et al., 2022] to compute persistent diagrams (as depicted in Figure 1 of [Yan et al., 2022]) and then vectorize them to get PIs as topological fingerprints. The extended module incorporates a additional reverse process. Through this module, all loop features created will be killed in the end. As an example, in Figure 1, a 1-dim extended persistence point that can be captured is  $(t_4,t_2)$ . This is because the loop feature is born at time  $t_4$  and then dies at time  $t_2$  in the reverse process. Note that we only use atomic number as filtration functions for  $\mathrm{TDL}_{\mathrm{atom}}$ . In addition to the atomic number, we also investigate the heat kernel signature (hks) with temperature values  $t = 0.1$  [Sun et al., 2009] and node degree as filtration functions. Observe that this is more geometric information, compared to the domain knowledge encoded in the atomic number (atom). Considering that TAE is a straightforward model and there is too little information for  $\mathrm{PI}_{\mathrm{atom}}$ , we concatenated these 3 PIs as reconstruction targets for  $\mathrm{TAE}_{\mathrm{ahd}}$ .
We have also considered a specific filtration function from ToDD [Demir et al., 2022], which incorporates atomic mass and, additionally, partial charges and bond types inside a so-called multi-VR filtration. In a nutshell, for each of those three types of information, we do not simply consider the given filtration, but construct a 2D filtration by filtering: in one dimension according to the above information and in the second dimension according to a VR filtration capturing the distances between atoms (Figure 5 in [Demir et al., 2022] illustrates this kind of filtration well.) Lastly, the 3 2D PIs are concatenated, and we compute distances based on the combination of PIs. In Appendix G, we report ablation results for various filtrations.
In order to explore the effectiveness of PIs, we calculate the widely-used Tanimoto coefficient [Bajusz et al., 2015] of the extended connectivity fingerprints (ECFPs) [Rogers and Hahn, 2010] between two molecules as their chemical similarity on 8 downstream tasks datasets. Then, we pick the molecule pairs with top  $20\%$  similarity as 'similar' ones (Blue) and the left  $80\%$  molecule pairs in the datasets are 'random' pairs (Yellow). In final, we calculate the cosine similarity of these molecule pairs based on PIs. Figure 8 shows molecules with similar ECFPs have rather similar PIs.
It is computationally efficient to apply topological fingerprints method. Extracting and computing  $\mathrm{PI}_{\mathrm{atom}}$  for 2M molecular graphs from ZINC15 dataset only spent 20 minutes in a single workstation with 8-core CPU, 64 GB of memory.
TAE+ Models. We hypothesize that TAE gives other pre-training approach a good initial embedding space to start from. Therefore, this pre-training strategy is to first perform TAE pre-training and then other models pre-training. We study TAE+EdgePred, TAE+AttrMask and TAE+ContextPred.
TDL. The temperature parameter  $\tau$  is set to 0.1.
TDL over Views. Like any other contrastive loss function, TDL can be utilized as a standalone objective by considering views. Here, we show preliminary experiments using a formulation based on the loss of GraphCL [You et al., 2020], we also use its augmentations. A minibatch of  $N$  graphs is processed through contrastive learning, resulting in  $2N$  augmented graphs. We denote the augmented views for the  $n$ th graph in the minibatch as  $z_{n,i}$  and  $z_{n,j}$ , and modify TDL as follows:
$$
\mathcal {L} _ {\mathrm {T D L} _ {n}} ^ {v i e w s} = \frac {1}{N - 1} \sum_ {m \in [ 1, N ]} - \log \frac {\exp \left(s i m \left(z _ {n , i} , z _ {m , j}\right) / \tau\right)}{\sum_ {k \in [ 1 , N ]} \mathbb {I} _ {[ d i s (I _ {n , i} , I _ {k , j}) \geq d i s (I _ {n , i} , I _ {m , j}) ]} \cdot \exp \left(s i m \left(z _ {n , i} , z _ {k , j}\right) / \tau\right)}
$$
(a) Bace
(b) BBBP
(c) ClinTox
(d) Sider
(e) Tox21
(f) ToxCast
(g) HIV
(h) MUV
Figure 8: Similarity histograms of PIs on 8 downstream datasets. Cosine similarity measures the similarity between the PIs while 'Random' (Yellow) and 'Similar' (Blue) are defined by the similarity between ECFPs.
B Analysis of Embeddings
Table 7: The Pearson correlation coefficient between real PIs and reconstructed PIs (i.e., TAE's embeddings) on the downstream data (the full dataset), after pre-training. We also report the overlap of the downstream data with the pre-training data, but do not observe considerable impact here.
Tox21	ToxCast	Sider	ClinTox	MUV	HIV	BBBP	Bace
# Molecules	7,831	8,575	1,427	1,478	93,087	41,127	2,039	1,513
# Molecules in ZINC15	628 (8%)	608 (7%)	1 (0%)	51 (4%)	7599 (8%)	925 (2%)	100 (5%)	0 (0%)
TAEahd	0.8572	0.7744	0.5939	0.8642	0.9044	0.7359	0.8660	0.8514
Table 8: 5-nearest neighbors classifier of CL methods for 8 downstream datasets, ROC-AUC (%).
Tox21	ToxCast	Sider	ClinTox	MUV	HIV	BBBP	Bace	Average
PIatom	58.8	51.1	58.7	50.9	50.2	64.8	55.2	75.5	58.15
PIToDD	62.7	51.7	56.3	64.9	49.7	63.9	56.7	68.7	59.33
ECFP	63.8	54.6	59.1	50.7	54.0	68.1	59.3	77.0	60.82
GraphCL	65.3	55.8	60.6	56.4	52.0	70.4	57.9	71.4	61.23
GraphCL + TDLatom	67.2	56.7	62.6	57.5	54.4	71.1	62.0	70.7	62.78
GraphCL + TDLToDD	68.0	56.5	61.6	63.5	54.1	70.3	62.5	66.9	62.93
JOAO	66.3	55.7	59.9	57.2	53.2	72.0	58.9	70.2	61.67
JOAO + TDLatom	66.9	55.4	58.9	62.5	52.7	69.1	60.1	70.4	62.00
JOAO + TDLToDD	67.1	56.8	60.5	61.2	51.5	67.7	59.5	72.7	62.13
SimGRACE	64.2	56.1	58.3	46.7	55.1	68.2	58.4	70.4	59.67
SimGRACE + TDLatom	66.1	56.3	61.9	68.4	52.4	71.3	63.6	77.3	64.66
SimGRACE + TDLToDD	67.3	58.1	60.2	77.5	52.2	68.3	67.8	76.7	66.01
GraphLoG	63.0	55.9	59.8	68.9	52.2	68.2	56.8	79.4	63.02
GraphLoG + TDLatom	66.7	57.3	60.9	67.3	50.0	68.4	63.6	77.5	63.96
GraphLoG + TDLToDD	67.4	56.9	61.7	58.2	53.1	72.1	63.8	78.2	63.92
Table 9: ROGI scores, lower score means smoother embedding space in terms of downstream labels.
ClinTox	Bace	BBBP	Sider
GraphCL	0.4496	0.7883	0.7033	0.6863
GraphCL + TDLatom	0.4549	0.7623	0.6973	0.7213
JOAO	0.4675	0.7900	0.7344	0.7320
JOAO + TDLatom	0.4668	0.7673	0.7284	0.7284
SimGRACE	0.4722	0.7919	0.7874	0.7308
SimGRACE + TDLatom	0.4594	0.7635	0.6989	0.7004
GraphLoG	0.4456	0.7480	0.6908	0.6815
GraphLoG + TDLatom	0.4789	0.8161	0.7167	0.7320
Alignment Analysis. To evaluate alignment, we construct positive and negative molecule pairs. Positive pairs in a dataset are defined as those sharing identical molecular properties, while negative pairs exhibit differing properties. We randomly select 10k positive and negative pairs of molecular graphs from the Bace, BBBP, Clintox, and Sider datasets. Subsequently, we compute the normalized Euclidean distance of the embeddings and illustrate the results via a histogram, as depicted in Figure 9 and Figure 10. We apply TDL on top of GraphCL, SimGRACE, JOAO, and GraphLoG to represent the impact of TDL on the baselines.
The normalized curves demonstrate significant differences in the embedding space among different models in the distance dimension. It is observable that TDL refines the distribution of distances to more closely align with that of PIs. The distinction between  $\mathrm{TDL}_{\mathrm{ToDD}}$  and  $\mathrm{TDL}_{\mathrm{atom}}$  explains why  $\mathrm{TDL}_{\mathrm{atom}}$  does not enhance JOAO because JOAO's curve shows that it is in fact possible to get such an atom PIs' distribution. Furthermore, JOAO reveals that additional domain knowledge beyond atom can be effectively incorporated via  $\mathrm{TDL}_{\mathrm{ToDD}}$ .
C Additional Linear Probing
We continue with linear probing for molecular properties using the more challenging activity cliff dataset. In Table 10, we show that TDL often yields improvements.
D Additional Benchmark Results
Primary Results. The main results on 8 molecular property prediction tasks are listed in Table 11. Note that all SSL methods in Table 11 are pre-trained on the same dataset based on ZINC15. We observe that the average performance of GraphCL+TDL has surpassed these existing SSL methods.
Figure 9: Evaluating alignment in the embedding space influenced by  $\mathrm{TDL}_{\mathrm{atom}}$ . Dark/light shades denote positive/negative pairs, respectively. Blue illustrates the embedding of baselines, red depicts the embedding of baselines  $+\mathrm{TDL}_{\mathrm{atom}}$ , and green represents  $\mathrm{PI}_{\mathrm{atom}}$ .
Table 10: Linear evaluation of molecular property prediction using opioids-related datasets in classification task, repeated 30 times with 30 different seeds, ROC-AUC (\%).
Dataset	MDR1	CYP2D6	CYP3A4
ROC↑	PRC↑	ROC↑	PRC↑	ROC↑	PRC↑
GraphCL	0.832 (0.172)	0.674 (0.256)	0.512 (0.205)	0.019 (0.017)	0.763 (0.147)	0.327 (0.258)
GraphCL + TDLatom	0.928 (0.081)	0.736 (0.241)	0.590 (0.251)	0.032 (0.043)	0.817 (0.133)	0.296 (0.183)
Dataset	MOR	DOR	KOR
ROC↑	PRC↑	ROC↑	PRC↑	ROC↑	PRC↑
GraphCL	0.789 (0.060)	0.595 (0.108)	0.730 (0.065)	0.382 (0.091)	0.801 (0.048)	0.593 (0.099)
GraphCL + TDLatom	0.797 (0.050)	0.601 (0.109)	0.760 (0.065)	0.432 (0.117)	0.802 (0.063)	0.598 (0.105)
Additionally, we also apply TDL on both AD-GCL and iMolCLR, with results in the table showing that TDL still provides good improvements.
Individual Tasks. In Table 4, we observe that SimGRACE+TDL performs poorly on Sider and ClinTox. To learn about the influence of multiple tasks, we run SimGRACE and GraphLoG on individual tasks of Sider and ClinTox. Figure 12 shows increases for  $\mathrm{TDL}_{\mathrm{atom}}$ . We hence conclude that the task correlation is exploited by the models considerably during fine-tuning in the multi-task setting, such that the improved SSL representation (which likely does not fit every downstream task under consideration) provided by TDL has less impact.
Activity Cliff Dataset. We also evaluate our methods using the opioids-related datasets. We fine-tune GraphCL and GraphCL+TDL and observe that GraphCL+TDL outperforms GraphCL, as shown in the Table 12. This result is interesting in itself since it is often argued that SSL based on similarity in "just" the geometric space (i.e., ignoring domain knowledge about, e.g., matching pairs or activity cliffs) may yield problems in practice. However, here, we observe that increased power of SSL
Figure 10: Evaluating alignment in the embedding space influenced by  $\mathrm{TDL}_{\mathrm{ToDD}}$ . Dark/light shades denote positive/negative pairs, respectively. Blue illustrates the embedding of baselines, red depicts the embedding of baselines  $+\mathrm{TDL}_{\mathrm{ToDD}}$ , and green represents  $\mathrm{PI}_{\mathrm{ToDD}}$ .
(a) GraphCL
(b) GraphLoG
Figure 11: Singular values of covariance matrices of the representations; Pre-training datasets.
representation helps to increase the performance of existing methods, even on AC data - likely also because they have issues that still need to be resolved, such as the dimensional collapse we observed.
Additional Pre-training Data. Additionally, we validate the effectiveness of our methods on another pre-training dataset, GEOM (50K) [Liu et al., 2022a]. For an evaluation, we study TAE and apply TDL on GraphCL and JOAO. We follow all pre-training and fine-tuning settings of [Liu et al., 2022a], with the only difference being that we pre-train our models for 200 epochs instead. We compare the results from their original paper. As shown in Table 13, TAE alone reaches decent average performance, and surpasses SOTA on ClinTox and HIV. And we report the performance in regressive property prediction in Table 14 which shows that TDL enhances GraphCL and JOAO on many datasets. Please note that we considered these experiments only to obtain initial estimates; the GEOM data (which we used to be able to compare to related works) itself is too small for real pre-training if the baseline models are not able to exploit the 3D nature of the conformers.
Table 11: Results for 8 molecular property prediction tasks on ZINC15. For each downstream task, we report the mean (and standard deviation) ROC-AUC of 10 seeds.
Tox21	ToxCast	Sider	ClinTox	MUV	HIV	BBBP	Bace	Average
# Molecules	7,831	8,575	1,427	1,478	93,087	41,127	2,039	1,513	-
No pretrain	74.6 (0.4)	61.7 (0.5)	58.2 (1.7)	58.4 (6.4)	70.7 (1.8)	75.5 (0.8)	65.7 (3.3)	72.4 (3.8)	67.15
InfoGraph	73.3 (0.6)	61.8 (0.4)	58.7 (0.6)	75.4 (4.3)	74.4 (1.8)	74.2 (0.9)	68.7 (0.6)	74.3 (2.6)	70.10
GraphLoG	75.0 (0.6)	63.4 (0.6)	59.3 (0.8)	70.1 (4.6)	75.5 (1.6)	76.1 (0.8)	69.6 (1.6)	82.1 (1.0)	71.43
JOAO	75.0 (0.3)	62.9 (0.5)	60.0 (0.8)	81.3 (2.5)	71.7 (1.4)	76.7 (1.2)	70.2 (1.0)	77.3 (0.5)	71.89
SimGRACE	74.4 (0.3)	62.6 (0.7)	60.2 (0.9)	75.5 (2.0)	75.4 (1.3)	75.0 (0.6)	71.2 (1.1)	74.9 (2.0)	71.15
GraphMAE	75.2 (0.9)	63.6 (0.3)	60.5 (1.2)	76.5 (3.0)	76.4 (2.0)	76.8 (0.6)	71.2 (1.0)	78.2 (1.5)	72.30
MGSSL	75.2 (0.6)	63.3 (0.5)	61.6 (1.0)	77.1 (4.5)	77.6 (0.4)	75.8 (0.4)	68.8 (0.6)	78.8 (0.9)	72.28
\( \text{TDL}_{\text{atom}} \)	75.8 (0.5)	62.1 (0.5)	62.2 (0.9)	79.1 (3.8)	75.2 (2.3)	76.9 (0.9)	66.5 (1.8)	78.4 (1.1)	72.02
\( \text{TDL}_{\text{views}} \)	75.7 (0.3)	64.3 (0.5)	61.5 (0.5)	80.7 (2.9)	73.3 (1.4)	77.6 (0.8)	70.5 (0.9)	80.6 (1.1)	73.02
EdgePred	76.0 (0.6)	64.1 (0.6)	60.4 (0.7)	64.1 (3.7)	74.1 (2.1)	76.3 (1.0)	67.3 (2.4)	79.9 (0.9)	70.27
\( \text{TAE}_{\text{ahd}} + \text{EdgePred} \)	77.2 (0.4)	63.8 (0.5)	60.1 (0.7)	72.2 (2.4)	75.9 (1.7)	76.8 (1.0)	68.1 (0.8)	81.5 (1.4)	71.95
AttrMask	76.7 (0.4)	64.2 (0.5)	61.0 (0.7)	71.8 (4.1)	74.7 (1.4)	77.2 (1.1)	64.3 (2.8)	79.3 (1.6)	71.15
\( \text{TAE}_{\text{ahd}} + \text{AttrMask} \)	76.6 (0.2)	64.0 (0.3)	58.2 (1.3)	71.6 (3.7)	75.5 (2.1)	77.2 (0.6)	68.7 (0.7)	81.0 (1.1)	71.60
ContextPred	75.7 (0.7)	63.9 (0.6)	60.9 (0.6)	65.9 (3.8)	75.8 (1.7)	77.3 (1.0)	68.0 (2.0)	79.6 (1.2)	70.89
\( \text{TAE}_{\text{ahd}} + \text{ContextPred} \)	76.4 (0.5)	63.2 (0.4)	62.0 (0.7)	74.6 (4.4)	76.7 (1.6)	77.7 (1.2)	68.9 (1.1)	80.7 (1.6)	72.53
GraphCL	73.9 (0.7)	62.4 (0.6)	60.5 (0.9)	76.0 (2.7)	69.8 (2.7)	78.5 (1.2)	69.7 (0.7)	75.4 (1.4)	70.78
GraphCL + \( \text{TDL}_{\text{atom}} \)	75.3 (0.4)	64.4 (0.3)	61.2 (0.6)	83.7 (2.7)	75.7 (0.8)	78.0 (0.9)	70.9 (0.6)	80.5 (0.8)	73.71
AD-GCL	76.5 (0.8)	63.0 (0.7)	63.2 (0.7)	79.7 (3.5)	72.3 (1.6)	78.2 (0.9)	70.0 (1.0)	78.5 (0.8)	72.67
AD-GCL + \( \text{TDL}_{\text{atom}} \)	75.1 (0.5)	64.5 (0.6)	60.9 (1.3)	79.4 (2.8)	74.4 (1.7)	77.4 (0.9)	71.9 (1.1)	82.1 (1.4)	73.21
iMolCLR	75.1 (0.7)	63.5 (0.4)	59.4 (1.0)	74.7 (1.9)	81.0 (2.6)	77.3 (1.2)	69.6 (1.2)	77.3 (1.0)	72.24
iMolCLR + \( \text{TDL}_{\text{atom}} \)	75.9 (0.6)	63.7 (0.3)	60.7 (0.8)	83.8 (1.9)	75.1 (1.3)	76.7 (0.7)	71.2 (0.9)	78.5 (1.3)	73.20
(a) Sider
(b) Sider
Figure 12: Performance on individual tasks of the multi-task data. We report the mean ROC-AUC of 10 seeds.
(d) ClinTox
Table 12: Results for molecular property prediction using opioids-related datasets in classification tasks, repeated 30 times with 30 different seeds, ROC-AUC  $(\%)$ .
Dataset	MDR1	CYP2D6	CYP3A4
ROC↑	PRC↑	ROC↑	PRC↑	ROC↑	PRC↑
GraphCL	0.912 (0.152)	0.766 (0.211)	0.558 (0.216)	0.079 (0.143)	0.803 (0.155)	0.310 (0.224)
GraphCL + TDLatom	0.934 (0.126)	0.770 (0.233)	0.565 (0.261)	0.082 (0.116)	0.815 (0.159)	0.418 (0.245)
Dataset	MOR	DOR	KOR
ROC↑	PRC↑	ROC↑	PRC↑	ROC↑	PRC↑
GraphCL	0.861 (0.040)	0.738 (0.105)	0.804 (0.038)	0.532 (0.076)	0.859 (0.044)	0.701 (0.085)
GraphCL + TDLatom	0.879 (0.036)	0.741 (0.093)	0.821 (0.049)	0.566 (0.092)	0.861 (0.041)	0.718 (0.058)
Table 13: Results for 8 molecular property prediction tasks on GEOM. For each downstream task, we report the mean (and standard deviation) ROC-AUC of 3 seeds with scaffold splitting. Note that GraphMVP uses 3D geometry (conformers), while other models do not.
Tox21	ToxCast	Sider	ClinTox	MUV	HIV	BBBP	Bace	Average
# Molecules	7,831	8,575	1,427	1,478	93,087	41,127	2,039	1,513	-
No pretrain	74.6 (0.4)	61.7 (0.5)	58.2 (1.7)	58.4 (6.4)	70.7 (1.8)	75.5 (0.8)	65.7 (3.3)	72.4 (3.8)	67.15
InfoGraph	73.0 (0.7)	62.0 (0.3)	59.2 (0.2)	75.1 (5.0)	74.0 (1.5)	74.5 (1.8)	69.2 (0.8)	73.9 (2.5)	70.10
EdgePred	74.5 (0.4)	60.8 (0.5)	56.7 (0.1)	55.8 (6.2)	73.3 (1.6)	75.1 (0.8)	64.5 (3.1)	64.6 (4.7)	65.64
ContextPred	73.3 (0.5)	62.8 (0.3)	59.3 (1.4)	73.7 (4.0)	72.5 (2.2)	75.8 (1.1)	71.2 (0.9)	78.6 (1.4)	70.89
AttrMask	74.2 (0.8)	62.5 (0.4)	60.4 (0.6)	68.6 (9.6)	73.9 (1.3)	74.3 (1.3)	70.2 (0.5)	77.2 (1.4)	70.16
GPT-GNN	75.3 (0.5)	62.2 (0.1)	57.5 (4.2)	57.8 (3.1)	76.1 (2.3)	75.1 (0.2)	64.5 (1.1)	77.6 (0.5)	68.27
GraphLoG	73.0 (0.3)	62.2 (0.4)	57.4 (2.3)	62.0 (1.8)	73.1 (1.7)	73.4 (0.6)	67.8 (1.7)	78.8 (0.7)	68.47
G-Contextual	75.2 (0.3)	62.6 (0.3)	58.4 (0.6)	59.9 (8.2)	72.3 (0.9)	75.9 (0.9)	70.3 (1.6)	79.2 (0.3)	69.21
G-Motif	73.2 (0.8)	62.6 (0.5)	60.6 (1.1)	77.8 (2.0)	73.3 (2.0)	73.8 (1.4)	66.4 (3.4)	73.4 (4.0)	70.14
GraphCL	75.0 (0.3)	62.8 (0.2)	60.1 (1.3)	78.9 (4.2)	77.1 (1.0)	75.0 (0.4)	67.5 (3.3)	68.7 (7.8)	70.64
JOAO	74.4 (0.7)	62.7 (0.6)	60.7 (1.0)	66.3 (3.9)	77.0 (2.2)	76.6 (0.5)	66.0 (0.6)	72.9 (2.0)	69.57
GraphMVP	74.5 (0.4)	62.7 (0.1)	62.3 (1.6)	79.0 (2.5)	75.0 (1.4)	74.8 (1.4)	68.5 (0.2)	76.8 (1.1)	71.69
GraphMVP-C	74.4 (0.2)	63.1 (0.4)	63.9 (1.2)	77.5 (4.2)	75.0 (1.0)	77.0 (1.2)	72.4 (1.6)	81.2 (0.9)	73.07
TAEahd	74.9 (0.3)	62.3 (0.3)	60.2 (0.9)	81.9 (2.5)	71.2 (1.4)	77.6 (1.2)	70.9 (3.3)	78.4 (1.6)	72.18
Table 14: Results for 4 molecular property prediction tasks (regression) on GEOM. For each downstream task, we report the mean (and standard variance) RMSE of 3 seeds with scaffold splitting.
ESOL	Lipo	Malaria	CEP	Average
No pretrain	1.178 (0.044)	0.744 (0.007)	1.127 (0.003)	1.254 (0.030)	1.07559
AttrMask	1.112 (0.048)	0.730 (0.004)	1.119 (0.014)	1.256 (0.000)	1.05419
ContextPred	1.196 (0.037)	0.702 (0.020)	1.101 (0.015)	1.243 (0.025)	1.06059
GraphMVP	1.091 (0.021)	0.718 (0.016)	1.114 (0.013)	1.236 (0.023)	1.03968
TAEahd	1.129 (0.019)	0.723 (0.006)	1.105 (0.014)	1.265 (0.018)	1.05552
JOAO	1.120 (0.019)	0.708 (0.007)	1.145 (0.010)	1.293 (0.003)	1.06631
JOAO + TDLatom	1.086 (0.013)	0.713 (0.004)	1.120 (0.007)	1.265 (0.008)	1.04613
GraphCL	1.127 (0.023)	0.722 (0.004)	1.143 (0.015)	1.317 (0.010)	1.07725
GraphCL + TDLatom	1.082 (0.008)	0.707 (0.009)	1.113 (0.007)	1.294 (0.011)	1.04943
# Training Samples
Training Samples
Training Samples .104
Training Samples
Figure 13: Performance over smaller datasets. Each datapoint is the mean ROC-AUC of 10 different splits while keeping scaffolds, with standard deviations shown as error bars. From the upper left to the lower right: Bace, ClinTox, Sider, BBBP, Tox21, ToxCast, HIV, MUV.