SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4) RANDOM outperforms almost every GSSL method on node- and pair-level metrics, therefore incorporating the randomised features would bring substantial advantages for tasks that are based on local geometry [72]. We observe that among six node- and pair-level graph metrics, randomised embeddings perform the best in four. We further verify this via T-SNE embeddings in Fig. 4, where randomised embeddings can form more interpretable clusters w.r.t. "Atom Nodes". In Appendix C, we provide a numerical analysis to show that one of the reasons might be the
choice of the GNN layer's initialisation and summation message-passing function. We further empirically experiment with different initialisation strategies, finding that the discriminative power of randomised embeddings on these local metrics disappears.
Figure 4: RANDOM generates more interpretable latent space: T-SNE visualisation of node embeddings produced by RANDOM, ATTRMASK, and GRAPHCL on BBBP. Each dot represents an atom from a molecular graph, coloured by node degrees. We notice that RANDOM embeddings form more coherent clusters, consistent with their performance on node-level tasks. While this behaviour is not observed in other atom-level metrics such as node centrality (Fig. 8).
summation message-passing function. We further summarisation strategies, finding that the discriminative local metrics disappears.
5) RANDOM falls short of predicting graph-level metrics, which is in alignment with the fact that all GSSL methods outperform RANDOM on the graph-level MPP tasks. However, ranking top on recovering graph topological metrics does not guarantee the embeddings have the best generalisation on downstream MPP tasks.
6) Performance in topological metrics aligns with pre-training objectives. EDGEPred and ATTRMASK set the pre-training objectives to predict the adjacency matrix and masked nodes and edges, respectively; the embeddings extracted from these methods compose more information on the node- and pair-level topological metrics. Also, incorporating 3D geometry in the pre-training (i.e., GRAPHMVP) helps retain pair-level topological properties. It aligns with the fact that geometry helps improve target discovery [73-75], as atom pair interactions play a major role.
Table 3: Benchmarking topological properties. We report the mean square error or the cross entropy on eight datasets (i.e., smaller is better). For each topological metric, the best and second-best scores are marked **bold** and **bold**, respectively. We omit such annotations for the "Clustering coefficient (i.e., Cluster)" metric as all test errors are similar. We also report the average ranks of each GSSL method on these metrics ("R"), grouped by the three abstract levels of topology. We use the pre-training configurations that achieve the best MPP test performance with fixed embeddings, as reported in Tables 1 and 9.
Node	Pair	Graph
Degree	Cent.	Cluster	R	Link	Jaccord	Katz	R	Diameter	Conn.	Cycle	Assort.	R
RANDOM	0.001	0.008	0.003	1.5	0.078	0.012	0.017	2.8	177.924	0.087	2.933	0.029	8.6
EDGEPred	0.031	0.009	0.003	3	0.067	0.014	0.016	2.2	159.825	0.073	2.596	0.026	6.5
ATTRMASK	0.009	0.009	0.003	2.7	0.082	0.015	0.020	4.7	110.793	0.062	2.207	0.019	2
GPT-GNN	0.123	0.009	0.003	4.7	0.014	0.021	0.029	5.3	111.688	0.074	2.854	0.026	6.5
INFOGRAPH	0.054	0.009	0.004	5	0.088	0.019	0.021	6	84.339	0.066	2.100	0.029	6
CONT.PRED	0.164	0.010	0.004	8	0.014	0.021	0.045	5.7	138.304	0.067	2.150	0.027	5.3
GROVER	0.120	0.012	0.004	8.2	0.114	0.047	0.059	10	78.352	0.064	2.058	0.021	3.3
GRAPHCL	0.060	0.010	0.004	6.7	0.084	0.028	0.026	7.3	90.336	0.066	2.287	0.026	6.1
JOAO	0.067	0.010	0.004	7	0.089	0.041	0.025	8	95.335	0.063	2.352	0.024	5.3
GRAPHMVP	0.199	0.010	0.004	8.3	0.077	0.014	0.017	3	109.198	0.065	2.372	0.030	5.5
6.2 Substructure properties
We use Cramér's V statistics to identify the five substructures mostly associated with downstream biochemical properties, as reported in Table 5, detailed in Table 14. We probe the pre-trained embeddings to predict these substructures. We provide the complete results and plot the distributions of all substructures in Appendix D.
Findings. Table 4 shows the results, where we bold the best and underline the worst scores of each substructure. We report the Spearman rank correlation and p-values between the performance of recognising substructures and predicting molecular properties. We highlight the following findings.
7) Substructure detection performance correlates well with MPP performance. Pre-trained embeddings notably surpass the RANDOM in both substructure detection and Multiple Property Prediction (MPP)
Table 4: Benchmarking substructure properties. We report the mean square errors on the test split averaged over the eight downstream MPP datasets.
allylic	amide	benzene	ether	halogen
RANDOM	0.959	16.917	1.100	2.024	1.127
EDGE PRED	0.780	14.173	0.797	1.608	0.939
ATTRMASK	0.926	14.703	0.976	1.742	0.501
GPT-GNN	0.872	15.629	0.783	1.912	0.341
INFOGRAPH	0.740	6.747	0.583	1.128	0.706
CONT.PRED	1.040	16.636	0.980	1.787	1.075
GROVER	0.715	6.576	0.558	0.957	0.298
GRAPHCL	0.652	7.598	0.525	1.077	0.319
JOAO	0.654	7.926	0.531	1.071	0.310
GRAPHMVP	0.905	6.992	0.649	1.037	0.311
Corr.	0.830	0.770	0.915	0.879	0.806
p-value	3e-3	9e-3	2e-4	8e-4	5e-3
Table 5: Cramér's V between molecular substructure counts and binary properties, averaged over 678 property prediction tasks or eight downstream datasets.
allylic	benzene	amide	ether	halogen
Task	0.1144	0.1630	0.0881	0.1034	0.1721
Dataset	0.1024	0.1227	0.1336	0.1083	0.1086
Figure 5: Dimensional collapse analysis on BBBP. Left: graph; Right: node.
tasks, with the sole exception of continuous prediction in the "allyli" category. The superior performance of GSSLs in predicting molecular properties could be attributed to their capacity for substructure awareness. This observation is further corroborated by the high positive rank correlation and substantial statistical significance, as indicated by all p-values being less than  $1\%$ . This evidence suggests that integrating substructure awareness into GSSL methods could potentially enhance the accuracy of molecular property predictions.
8) Motif-based and Contrastive-based GSSL methods have better substructure awareness. GROVER, GRAPHCL, JOAO, and JOAOv2 perform consistently better than most other GSSL methods on substructure detection also in the MPP tasks. Note that the optimal pre-training configurations for GROVER is "Motif"-based loss<sup>1</sup>, as reported in Table 9.
6.3 Embedding space properties
Findings. Based on the above results, we summarise the following findings. For alignment, we randomly select 10k positive and negative pairs of molecular graphs from BBBP and Tox21 datasets, calculate the cosine similarity and plot the histogram in Fig. 6. We choose AT-
Table 6: Benchmarking embedding space properties. We report the rank correlations and p-values.
Node Embed	Graph Embed	Uniformity
Correlation	0.806	0.927	0.842
p-value	4e-3	6e-3	2e-3
TRMASK and GRAPHCL to represent generative and contrastive GSSL methods, respectively. Table 16 presents uniformity values as defined in [6]; Fig. 5 plots the magnitude of the singular values in the logarithm scale provided in Table 15.
9) Compared with the RANDOM initialised GNN, GSSL methods give rise to better alignments, promote more uniform features and lift the spectrum.
GSSL embeddings form distinguishable distributions for positive/negative pairs, while the RANDOM embeddings do not (a phenomenon often referred to as over-smoothing [76], see Fig. 6). All the GSSL methods have better uniformly distributed embeddings on all datasets (in Table 16). However, we found that a better alignment is not necessary to achieve better generalisation for the domain of the molecular graph (Table 16). The singular values of stacked GSSL embeddings (both node and graph) are larger than RANDOM's by multiple magnitudes; also, we observe that the magnitude of the spectrum positively correlates with the downstream MPP performance (in Tables 6 and 15).
In this work, we challenged common practices in evaluating graph self-supervised learned embeddings of molecular graphs. First, we extensively searched the optimal hyperparameters and evaluated GSSL methods on common molecular property prediction tasks in an unbiased and controlled manner. Next, we presented MOLGRAPHVAL, which is a diverse collection of probe tasks divided into three categories: (i) topological properties, (ii) substructure properties, and (iii) embedding space properties. Then we evaluated GSSL methods on MOLGRAPHVAL and found surprising insights not revealed by the evaluation of MPP tasks alone.
Here the concepts of "Motif" and "Substructure" are identical, under the context of molecular graphs.
The purpose of this work is to complement current evaluation practices with probe tasks and metrics that reveal novel insights, rather than arguing about which combinations of pre-training tasks yield the best downstream performance. Also, as our primary focus is the pre-trained GNN encoders, we leave the investigations of comparing probing and fine-tuning embeddings in the future. Our empirical findings suggest that there are many open questions on how to learn robust molecular graph embeddings without labels and a better understanding of these, along with a new methodology for solving some of the issues mentioned earlier (e.g., dimensional collapse), are yet to come. Nevertheless, we are optimistic that the tasks pro
Figure 6: Alignment in the embedding space.
posed in this paper will benefit the GSSL research community to tackle these challenges and applied scientists in fields like drug discovery to yield additional insights that can help their problem.
We thank Le Song, Anima Anandkumar, Matthew Welborn for valuable discussions.
A Overview 17
B Pre-Training 17
B.1 GEOM dataset 17
B.2 GSSL methods 17
B.3 Hyperparameters search 18
B.4 Probe models 19
B.5 Computation efficiency 19
B.6 Practical guides for future research 20
C Randomised embeddings 20
D Substructure 21
D.1 Discussions on substructure counting 21
D.2 Detailed description and performance 21
D.3 Cramer's V 25
D.4 Distribution 25
E More results on metrics 28
E.1 Topological metric 28
E.2 Spectrum 30
E.3 Uniformity 32
F Extending to more GSSL methods and datasets 32
G Potential ethical and social implications 32
A Overview
Automated MOLGRAPH EVAL pipeline. In the codebase of MOLGRAPH, we provide setup scripts (in "env/") for both the docker and conda virtual environment. The end-to-end benchmarking pipeline consists of three modules (in Fig. 7):
- Pre-training the GNN models (other GNN model classes such as MPNN, GCN, GAT are implemented in the codebase);
- Extracting the node/graph/pair-level embeddings from the pretrained or the randomly initialised GNN models;
- Probing the quality of embeddings with the proposed metrics.
We have meticulously packaged the components of MOL-GRAPHEVAL for ease of access and potential extension. The pre-trained methods are housed in "src/pretrainers", model libraries in "src/models", pre-training and downstream datasets in "src/datasets", and probing metrics in "src/validation". This modular design allows flexibility and extensibility to incorporate new model architectures and datasets. All configurations, specified in YAML files and processed by argument parsers in "src config", are managed by scripts (with templates provided in "script"). Furthermore, we have implemented loggers to chronicle training/validation/testing curves during both pre-training and probing. For added convenience, templates for visualising embeddings and analysing datasets are also available.
Figure 7: MOLGRAPHEVAL.
We open-sourced MOLGRAPHEval in https://github.com/hansen7/MolGraphEval.
B Pre-Training
We next describe the additional details of the pre-training used in the MOLGRAPH evaluation benchmark.
B.1 GEOM dataset
We avoid using the updated version of GEOM ('New drug-like molecules' and 'MoleculeNet') to remove the overlap between pre-training and downstream datasets. Compared with other molecular datasets that contain 3D conformation structures, GEOM [46] has the following advantages:
- Preciseness. Compared with toolkits like RDKIT or MMFF [77] (used in studies such as ChemRL-GEM [78]), DFT-based calculations (used in GEOM) will provide more precise computation results on the 3D molecular conformation structures [79-83]. Such errors in the molecular geometries have been proven harmful to property predictions (Appendix B of [84])
- Comprehensive. GEOM provides a more comprehensive collection in comparison with other quantum chemistry-based datasets (e.g., QM9 [85], Atom3D [86]) in terms of quantity and diversity. In comparison with ZINC15 and SAVI used in this concurrent study [49], GEOM provides accurate 3D conformation structures of molecules, which allows to compare more GSSL methods such as GraphMVP.
B.2 GSSL methods
Self-Supervised Learning (SSL) generally bifurcates into contrastive and generative methodologies, each characterised by its unique supervised signal as highlighted by [87]. Contrastive SSL operates by contrasting representations at the inter-data level, while generative SSL emphasises intra-data level reconstruction. Both strategies have been the subject of comprehensive research.
Contrastive GSSL creates multi-layered views of each graph, each capturing different granularity levels, from nodes and subgraphs to the complete graph. It aligns representations for views originating from the same data and distinguishes those from unrelated datasets, targeting a unified
embedding space. The effectiveness of various approaches largely hinges on their view design. For example, INFOGRAPH contrasts node and graph views, while GRAPHCL and JOAO delve into graph-level transformations. Further avenues to improve Contrastive GSSL include:
- CLEAR [88] captures graph structure at both global and local scopes, enhancing semantic information granularity and consistency between multiple views;
- PGCL [89] addresses sampling bias by clustering graphs into groups represented by prototype vectors, focusing on the dataset's global semantics;
- iGCL [90] employs a Siamese architecture to generate positive samples sans data augmentation. The proposed ID loss eschews negative sampling while promoting feature-wise discriminability.
Generative GSSL zeroes in on reconstructing the graph structure, striving to derive representations that capture the core characteristics of the data. Noteworthy examples in this category include EDGEPRED and ATTRMASK, which predict adjacency matrices and mask tokens, respectively. Meanwhile, GPT-GNN employs an auto-regressive approach tailored for holistic graph reconstruction. In line with this methodology, masked graph autoencoders, as cited in [91-93], have garnered considerable attention.
GROVER-Motif leverages domain-specific knowledge to extract motifs from molecules, assigning SSL the role of predicting motif presence. Diverging from the paradigms of contrastive and generative GSSL, recent explorations like [94] categorise this approach as predictive GSSL. In this framework, the supervisory signals are derived from self-generated labels.
There is a limited body of work dedicated to understanding GSSL methods. In a notable study, Akhondzadeh et al. [95] explored the use of probing tasks to measure and contrast the richness of graph representations derived from various models. A significant revelation from this study is that transformer-based GNNs capture chemically pertinent information more effectively than message-passing GNNs. Further integrating the data from 3D structures, Liu et al. [96] presented Geom3D — a comprehensive framework for benchmarking geometric representation learning techniques applicable to molecules, proteins, and materials. This framework encompasses 16 cutting-edge geometric models and evaluates their efficacy across 46 diverse scientific challenges, spanning small molecules, proteins, and crystalline substances. An innovative aspect of Geom3D is its approach to categorising geometric models into three groups: invariant, spherical frame equivariant, and vector frame equivariant.
Table 7: Hyperparameters search space of GSSL.
METHOD	HYPERPARAMETERS	# MODELS
EDGE PRED	LEARNING RATE	15
ATTRMASK	MASK RATE, LEARNING RATE	300
GPT-GNN	LEARNING RATE	15
INFOGRAPH	LEARNING RATE	15
GROVER	LEARNING RATE, “ CONTEXTURAL” OR “MOTIF”-BASED LOSS	30
CONT.PRED	LEARNING RATE, CONTEXT SIZE, # NEGATIVE SAMPLES	300
GRAPHCL	LEARNING RATE, AUG STRENGTH, AUG PROB	360
JOAO	LEARNING RATE, GAMMA, LOSS VERSION (V1 OR V2)	300
GRAPHMVP	LEARNING RATE, TEMPERATURE, ALPHA2, # CONFORMER	540
Total	1875
B.3 Hyperparameters search
We search the optimal hyperparameters of pre-training methods, details are summarised in Tables 7 to 9. We select the best hyperparameter of each GSSL method based on their averaged score on downstream datasets (in Table 1, linear models).
We provide details in calculating the 1875 GNN configurations in Table 7. The number of probe models is calculated as follows:  $1875*8$  (MPP datasets)  $*2$  (Fix, FT)  $*3$  (Seed) + 9 (GSSL, Optimal)  $*10$  (Topological Metrics)  $*3$  (Seed) + 9 (GSSL, Optimal)  $*24$  (Substructure)  $*3$  (Seed) = 90918. It takes over 4 terabytes to save these pre-trained models.
B.4 Probe models
Ideally, probe models should be neither too simple to capture the representation's information, nor too powerful to learn precise property prediction themselves. If overly powerful, the probe's performance might not accurately reflect the information embedded in the representations. In light of these complexities, we select a linear model as our probe, aligning with the choice prevalent in most probe studies.
Table 8: Range of Grid Search on Hyperparameters Space.
HYPERPARAMETERS	RANGE
LEARNING RATE, ALL BUT GRAPHMVP	[0.01, 0.005, 0.001, 0.0005, 0.0001]
LEARNING RATE, GRAPHMVP	[0.001, 0.0005, 0.0001]
MASK RATE	[0.05, 0.10, ..., 0.95]
CONTEXT SIZE	[2, 3, 4, 5]
# NEGATIVE SAMPLES	[1, 2, 3, 4, 5]
AUG STRENGTH	[0.2, 0.4, 0.6, 0.8]
AUG PROBABILITY	[0.1, 0.2, ..., 1.0]
GAMMA	[0.1, 0.2, ..., 1.0]
TEMPERATURE	[0.1, 0.2, 0.5, 1, 2]
ALPHA2	[0.1, 1, 10]
# CONFORMER	[1, 5, 10, 20]
Table 9: Optimal hyperparameters based on linear probing and finetuning scores on MPP tasks.
METHOD	OPTIMAL HYPERPARAMETERS (left: probing / right: fine-tuning)
EDGE PRED	LEARNING RATE=1e-2/1e-2
ATTRMASK	LEARNING RATE=1e-4/5e-4, MASK RATE=0.85/0.50
GPT-GNN	LEARNING RATE=1e-2/1e-4
INFOGRAPH	LEARNING RATE=1e-4/1e-4
GROVER	LEARNING RATE=1e-4/1e-3, “MOTIF”/“ CONTEXTUAL”-BASED LOSS
CONT.PRED	LEARNING RATE=1e-3/5e-3, CONTEXT SIZE=1/1, # NEGATIVE SAMPLES=5/1
GRAPHCL	LEARNING RATE=1e-3/1e-3, AUG STRENGTH=0.2/0.6, AUG PROB=0.8/0.5,
JOAO	LEARNING RATE=1e-3/1e-3, GAMMA=0.9/0.6, “V1”/“V1”-VERSION LOSS
GRAPHMVP	LEARNING RATE=5e-4/5e-4, ALPHA2=0.1/10.0, TEMPERATURE=0.1/0.2, # CONFORMER=5/5
B.5 Computation efficiency
We present the number of trainable parameters alongside the average training time per epoch (utilising a single A100 GPU) for each GSSL method in Table 10.
Table 10: Computational efficiency of GSSL methods
METHOD	NUMBER OF PARAMETERS (Million)	TRAINING TIME (Second)
EDGE PRED	7.462	101
ATTRIMASK	7.606	38
GPT-GNN	7.606	972
INFOGRAPH	7.823	40
GROVER	7.566	39
CONT.PRED	12.00	202
GRAPHCL	8.186	65
JOAO	8.186	382
GRAPHMVP	15.84	119
B.6 Practical guides for future research
In summary, for prospective advancements in Graph SSL, the following practical guidelines should be considered:
- Develop novel pretext tasks that emphasise beneficial invariances and geometry, as unveiled by MOLGRAPHEVAL probes. This includes tasks that enhance substructure modelling and preserve local topology.
- Relying solely on either probing or fine-tuning may not provide a comprehensive understanding. It's noteworthy that weak probing performance doesn't necessarily correlate with subpar fine-tuning outcomes (e.g., GRAPHMVP). The method of choice should be based on the specific downstream task, whether it's property prediction, generation, optimisation, or interaction modelling.
- Innovate superior data augmentation techniques specifically for molecular graphs to produce valuable views for contrastive learning. Probes can serve as an instrumental means to assess the quality of these augmentations.
- The prevailing notion of achieving a more uniform embedding space is helpful doesn't always hold true in the context of molecular graphs.
- Additionally, probes can be harnessed as a potent tool to scrutinise the impacts of varied negative sampling and augmentation strategies, exemplified by the comparison between GRAPHCL and JOAO.
C Randomised embeddings
How GNN models are initialised. We first analyse how weights in the GNNs are initialised (PyTorch and PyG).
- Edge Embedding Layers uses 'xavier uniform', essentially samples from uniform distribution
- GNN Layers in fact only have MLP weights (see PyG Doc), same initialisation as Linear layers.
- Linear Layers samples from uniform distribution for both weight and bias (PyTorch Doc)
Since all the weights (Edge embedding layers, GNN layers, and Linear layers) in the GINs are extracted from some uniform distribution of some positive ranges. As the GIN layer essentially consists of multiplications and additions, the expected statistics of the node embeddings from randomised GINs are proportional to the number of connected neighbours (i.e., node degrees). Therefore, the randomised embeddings form discriminative clusters in Fig. 4. As for other node-level metrics (in Fig. 8), we don't observe a good clustering formed from randomised embeddings.
Figure 8: T-SNE visualisation of node centrality produced by RANDOM (three different seeds).
D Substructure
D.1 Discussions on substructure counting
A recent study [97] asserts that message-passing neural networks (MPNNs), including Graph Isomorphism Networks (GIN), struggle with the exact counting of certain induced-subgraph structures. While this observation does not directly indicate that Graph Self-Supervised Learning (GSSL) aids in precise subgraph counting, we find that the knowledge acquired from GSSL pre-training can be construed as substructure awareness.
Conversely, another study [98] demonstrates that in the context of molecular graphs, the theoretical expressiveness limitations of MPNNs, as described by the Weisfeiler-Lehman Isomorphism test, do not impair generalisation performance in real-world datasets. As evidenced in Table .6, GIN models, especially those that are pre-trained, exhibit substantial proficiency in identifying substructures.
In conclusion, the question of whether the expressiveness of the backbone model is a limiting factor in molecular domains remains a topic of ongoing debate. The investigation of the backbone model operates independently of the GSSL pretraining analysis undertaken in our study. As such, we propose deferring this line of inquiry for future exploration.
D.2 Detailed description and performance
In Table 11, we provide the descriptions of the molecular substructures (mainly from documents on the rdkit.Chem.Fragments, textbooks [99] and Wikipedia). We also listed some molecular properties that are affected by these substructures. Table 12 and Table 13 report the detailed performance of substructure property prediction.
Table 12: Substructure detection, part I. We **bold** the best and **underline** the worst scores.
allylic	amide	amidine	azo	benzene	epoxide	ether	furan	guanido	halogen	imidazole	imide
RANDOM	0.959	16.917	0.054	0.020	1.100	0.024	2.024	0.036	0.126	1.127	0.080	0.062
EDGEPred	0.780	14.173	0.046	0.018	0.797	0.021	1.608	0.033	0.098	0.939	0.074	0.031
ATTRMASK	0.926	14.703	0.047	0.019	0.976	0.022	1.742	0.028	0.112	0.501	0.077	0.029
GPT-GNN	0.872	15.629	0.044	0.017	0.783	0.021	1.912	0.023	0.117	0.341	0.077	0.037
INFOGRAPH	0.740	6.747	0.050	0.019	0.583	0.022	1.128	0.021	0.086	0.706	0.062	0.038
CONT.PRED	1.040	16.636	0.053	0.020	0.980	0.023	1.787	0.034	0.126	1.075	0.078	0.033
GROVER	0.715	6.576	0.025	0.008	0.558	0.023	0.957	0.008	0.064	0.298	0.069	0.021
GRAPHCL	0.652	7.598	0.039	0.016	0.525	0.023	1.077	0.012	0.080	0.319	0.051	0.026
JOAO	0.654	7.926	0.043	0.015	0.531	0.023	1.071	0.013	0.085	0.310	0.048	0.026
GRAPHMVP	0.905	6.992	0.043	0.017	0.649	0.019	1.037	0.019	0.084	0.311	0.060	0.027
SSL Worse (#)	1	0	0	0	0	0	0	0	0	0	0	0