SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 13: Substructure detection, part II. We bold the best and underline the worst scores.
lactam morpholine	NO	oxazole	piperidine	piperazine	pyridine	tetrazole	thiazole	thiophene	urea
RANDOM	0.018	0.031	0.022	0.009	0.212	0.058	0.176	0.014	0.040	0.052	0.045
EDGEPred	0.016	0.020	0.022	0.008	0.182	0.048	0.158	0.014	0.039	0.046	0.041
ATTRMASK	0.016	0.028	0.022	0.008	0.192	0.053	0.174	0.014	0.038	0.044	0.044
GPT-GNN	0.012	0.021	0.022	0.008	0.177	0.049	0.128	0.014	0.039	0.041	0.037
INFOGRAPH	0.012	0.026	0.024	0.007	0.185	0.048	0.127	0.016	0.029	0.036	0.046
CONT.PRED	0.016	0.031	0.022	0.008	0.214	0.059	0.170	0.014	0.039	0.047	0.045
GROVER	0.014	0.022	0.028	0.004	0.158	0.043	0.096	0.005	0.018	0.018	0.014
GRAPHCL	0.014	0.019	0.021	0.006	0.169	0.031	0.084	0.009	0.023	0.018	0.023
JOAO	0.014	0.021	0.020	0.006	0.168	0.035	0.082	0.009	0.022	0.018	0.025
GRAPHMVP	0.011	0.021	0.022	0.006	0.155	0.038	0.129	0.009	0.027	0.025	0.028
SSL Worse (#)	0	0	2	0	1	1	0	1	0	0	1
D.3 Cramer's V
Cramér's V quantifies the strength of the association between the molecular substructure counts (i.e., chemical fragments) and their biochemical properties. It is defined as:
$$
V = \sqrt {\chi^ {2} / (n \cdot \min  (k - 1 , r - 1))} = \sqrt {\chi^ {2} / n} \quad (r \equiv 2) \tag {5}
$$
where  $n$  is the sample size,  $k$  and  $r$  are the total number of substructure counts and property categories (binary), respectively. The Chi-squared statistics  $\chi^2$  is then calculated as:
$$
\chi^ {2} = \sum_ {i, j} \left(n _ {(i, j)} - n _ {(i, \cdot)} \cdot n _ {(\cdot , j)} / n\right) ^ {2} / \left(n _ {(i, \cdot)} \cdot n _ {(\cdot , j)} / n\right) \tag {6}
$$
where  $n_{(i,j)}$  is the total occurrence for the pair of  $(i,j)$ . Here  $i$  is the specific count of a certain substructure, and  $j$  represents the certain outcome of a molecular biochemical property. Cramér's V value ranges from 0 to 1, representing the associated strength between two categorical variables.
Table 14: Cramér's V between molecular substructure counts and downstream properties.
Pre-training	BBBP	Tox21	ToxCast	Sider	ClinTox	MUV	HIV	Bace	Avg (Task)	Avg (Data)
allylic	0.1602	0.1345	0.1156	0.1276	0.0935	0.0413	0.0280	0.1186	0.1144	0.1024
amide	0.2692	0.0490	0.0858	0.1841	0.1326	0.0235	0.0689	0.2556	0.0881	0.1336
amidine	0.0360	0.0291	0.0412	0.0323	0.0158	0.0117	0.0396	0.1328	0.0399	0.0423
azo	0.0400	0.0399	0.0393	0.0277	0.0123	0.0007	0.2082	-	0.0384	0.0526
benzene	0.1476	0.1632	0.1691	0.1149	0.1112	0.0289	0.1374	0.1091	0.1630	0.1227
epoxide	0.0273	0.0481	0.0449	0.0300	0.0049	0.0005	0.0086	-	0.0437	0.0235
ether	0.2314	0.0694	0.1060	0.1069	0.1023	0.0185	0.0498	0.1821	0.1034	0.1083
furan	0.0635	0.0257	0.0387	0.0227	0.0061	0.0311	0.0148	0.0135	0.0375	0.0270
guanido	0.0765	0.0201	0.0509	0.0715	0.0286	0.0057	0.0094	0.1088	0.0499	0.0464
halogen	0.1488	0.0849	0.1827	0.0773	0.0908	0.0143	0.0347	0.2353	0.1721	0.1086
imidazole	0.0601	0.0427	0.0492	0.0460	0.1212	0.0102	0.0398	0.1280	0.0483	0.0622
imide	0.0951	0.0246	0.0401	0.0428	0.0518	0.0094	0.0188	-	0.0392	0.0404
lactam	0.4263	0.0184	0.0116	0.0646	0.0543	0.0006	0.0048	-	0.0182	0.0830
morpholine	0.0512	0.0126	0.0343	0.0268	0.0425	0.0068	0.0101	0.0668	0.0329	0.0314
N_O	0.0438	0.0195	0.0467	0.0391	0.0709	0.0195	0.0144	0.0537	0.0452	0.0385
oxazole	0.0126	0.0184	0.0321	0.0359	0.0123	0.0079	0.0080	0.0364	0.0312	0.0205
piperidine	0.1450	0.0305	0.0844	0.0575	0.0418	0.0079	0.0226	0.0935	0.0803	0.0604
piperzine	0.0509	0.0214	0.0421	0.0776	0.0648	0.0111	0.0192	0.0063	0.0424	0.0367
pyridine	0.0598	0.0402	0.0549	0.0338	0.0833	0.0129	0.0300	0.1747	0.0529	0.0612
tetrazole	0.1161	0.0158	0.0251	0.0300	0.0286	0.0083	0.0123	0.0334	0.0247	0.0337
thiazole	0.1389	0.0521	0.0345	0.0445	0.0183	0.0118	0.0173	0.0539	0.0348	0.0464
thiophene	0.0356	0.0467	0.0472	0.0315	0.0113	0.0166	0.0081	0.0438	0.0456	0.0301
urea	0.0790	0.0236	0.0506	0.0471	0.0268	0.0079	0.0329	0.0516	0.0489	0.0399
D.4 Distribution
We plot the distribution of 24 molecular substructures, the y-axis (i.e., counts) is log scale. Alongside each substructure name, we include its average counts (i.e., ground truth). While certain substructures, such as amidine and azo, are infrequently found in the molecules from the eight downstream datasets, we note that others like Amide and Benzene are prevalent. These substructures are recognised for their significant relevance to molecular properties, as detailed in Table 4.
Allylic Oxide.
Amide.
Amidine.
AZO.
Epoxide.
Furan.
Guanido.
Halogen.
Imidazole.
Imide.
Lactam.
Morpholine.
N_O.
Oxazole.
Piperdine.
Piperzine.
Pyridine.
Tetrazole.
Thiazole.
Thiophene.
Urea.
E More results on metrics
E.1 Topological metric
We have depicted the distribution of structural metrics and substructures with respect to the downstream datasets using histograms. Please note that the vertical axes may occasionally be represented on a logarithmic scale.
Node Degree.
Node Centrality.
Node Clustering Coefficient.
Link Prediction.
Jaccard Coefficient.
Katz Index.
Cycle Basis.
Graph Diameter.
Aassortativity Coefficient.
Average Clustering Coefficient.
E.2 Spectrum
Our observations suggest a positive correlation between the magnitudes of singular values in pretrained embedding spaces and their performance in the downstream MPP tasks.
Table 15: Benchmarking the magnitude of the singular values (node/graph) in the spectrum, BBBP.
Dimension	1	50	100	200	300
RANDOM	1.96e-01/4.81e-02	8.14e-06/3.35e-07	1.08e-06/3.87e-08	9.10e-08/2.20e-09	2.44e-09/4.08e-11
EDGE PRED	1.25e+00/2.12e-01	5.30e-03/5.96e-04	3.92e-04/3.56e-05	1.31e-05/7.10e-07	1.97e-07/6.79e-09
ATTRMASK	9.97e+01/7.57e+00	9.71e-03/1.16e-03	2.06e-03/2.02e-04	2.95e-04/1.96e-05	1.86e-05/9.00e-07
GPT-GNN	7.78e+01/2.46e+01	4.21e-03/6.04e-04	6.26e-04/7.11e-05	5.32e-05/4.40e-06	1.29e-06/6.50e-08
INFOGRAPH	1.29e+02/6.56e+01	9.52e-01/3.23e-01	2.68e-01/6.54e-02	2.43e-02/3.38e-03	2.80e-05/2.69e-06
CONT.PRED	1.81e+01/5.89e+00	3.40e-04/3.72e-05	4.36e-05/3.42e-06	1.65e-06/9.08e-08	1.14e-08/3.92e-10
GROVER	2.21e+02/9.86e+01	2.18e+00/5.11e-01	1.13e-01/2.04e-02	1.68e-02/1.97e-03	1.51e-03/1.36e-04
GRAPHCL	7.63e+01/3.40e+01	4.26e-01/7.27e-02	1.45e-02/1.76e-03	6.84e-04/5.11e-05	2.42e-05/1.37e-06
JOAO	8.12e+01/3.58e+01	4.16e-01/7.89e-02	2.18e-02/2.67e-03	8.63e-04/6.74e-05	2.64e-05/1.52e-06
GRAPHMVP	2.07e+01/1.17e+01	2.57e-01/8.22e-02	1.94e-02/4.00e-03	4.39e-05/5.75e-06	1.76e-06/1.71e-07
Correlation	0.661/0.781	0.806/0.927	0.879/0.903	0.697/0.770	0.794/0.794
We provide more visualisations of the spectrum of the embedding space from different datasets.
Singular Value Rank Index
Figure 9: Spectrum of the GSSL embedding space on Tox21 dataset, Left: Node; Right: Graph.
Figure 10: Spectrum of the GSSL embedding space on Toxcast dataset, Left: Node; Right: Graph.
Figure 11: Spectrum of the GSSL embedding space on Sider dataset, Left: Node; Right: Graph.
Figure 12: Spectrum of the GSSL embedding space on Clintox dataset, Left: Node; Right: Graph.
Figure 13: Spectrum of the GSSL embedding space on MUV dataset, Left: Node; Right: Graph.
Figure 14: Spectrum of the GSSL embedding space on HIV dataset, Left: Node; Right: Graph.
Figure 15: Spectrum of the GSSL embedding space on BACE dataset, Left: Node; Right: Graph.
E.3 Uniformity
We provide more comprehensive results on the uniformity metric of the embedding space in Table 16. The upper bound for uniformity is zero, while the lower bound, as derived from Corollary 1 in the Appendix of [6], is both data and hyperparameter-dependent.
Table 16: Evaluating GSSL methods on uniformity.
Dataset	BBBP	Tox21	ToxCast	Sider	ClinTox	MUV	HIV	Bace
RANDOM	-0.214	-0.315	-0.313	-0.282	-0.242	-0.128	-0.245	-0.079
EDGEPred	-2.319	-2.969	-2.909	-2.424	-2.384	-2.634	-2.804	-1.653
ATTRMASK	-8.332	-9.348	-9.560	-7.786	-8.399	-9.241	-9.979	-6.389
GPT-GNN	-5.714	-5.701	-5.773	-5.550	-5.708	-5.279	-5.068	-4.977
INFOGRAPH	-10.065	-10.925	-11.310	-7.623	-9.831	-15.337	-13.506	-9.620
CONT.PRED	-2.636	-3.074	-3.209	-2.748	-2.780	-2.390	-2.865	-1.708
GROVER	-10.208	-12.142	-12.356	-11.907	-9.975	-15.772	-14.244	-10.512
GRAPHCL	-10.010	-11.073	-11.458	-10.155	-9.845	-13.390	-12.530	-8.513
JOAO	-10.000	-10.955	-11.360	-10.012	-9.846	-13.450	-12.505	-8.523
GRAPHMVP	-8.864	-9.770	-9.863	-8.782	-8.825	-12.405	-11.301	-7.540
Correlation	0.842	-0.600	0.097	-0.210	0.309	0.169	0.821	0.285
p-value	0.002	0.067	0.789	0.559	0.384	0.641	0.004	0.425
F Extending to more GSSL methods and datasets
We demonstrate that the proposed MOLGRAPHEVAL benchmark can readily be extended to incorporate more pre-training methods, such as GRAPHMAE and GRCL, as well as larger datasets, like the 2M ZINC15. It's pertinent to note that the downstream datasets can also be employed for pre-training. Importantly, we observe that thorough optimisation of pre-training hyperparameters can serve as a crucial factor for performance improvements in pre-training, notwithstanding the advancements in the design of pre-training tasks.
Table 17: Evaluating GSSL methods on molecular property prediction tasks, on 2M ZINC15. For each downstream dataset, we report the mean and standard deviation of the ROC-AUC scores over three random scaffold splits. The performance scores are based on the fixed pre-trained embeddings with linear probe models, we also report the average ROC-AUC scores with fine-tuned pre-trained GNN on MPP tasks ("Avg (FT)").
BBBP	Tox21	ToxCast	Sider	ClinTox	MUV	HIV	Bace	Avg	Avg (FT)
# Molecules	2,039	7,831	8,575	1,427	1,478	93,087	41,127	1,513	-	-
# Tasks	1	12	617	27	2	17	1	1	-	-
RANDOM	50.7 ±2.5	64.9 ±0.5	53.2 ±0.3	53.2 ±1.1	63.1 ±2.3	62.1 ±1.3	66.1 ±0.7	63.4 ±1.8	59.60	66.16
ATTRMASK	49.8 ±0.6	66.7 ±0.3	52.9 ±0.4	53.8 ±1.7	62.2 ±2.9	52.8 ±1.7	69.0 ±1.4	66.6 ±4.9	59.22	69.49
GRAPHCL	64.9 ±0.5	70.5 ±0.6	56.1 ±0.2	58.0 ±1.4	63.4 ±3.1	61.2 ±1.6	75.6 ±0.9	70.9 ±3.8	65.07	70.09
GRAPHMAE	58.6 ±2.3	64.4 ±0.6	55.3 ±0.1	57.0 ±1.8	67.4 ±1.9	57.3 ±1.5	71.6 ±1.2	51.6 ±3.6	60.41	68.91
GRCL	61.8 ±2.1	67.5 ±0.4	54.3 ±0.4	55.8 ±1.3	62.0 ±2.4	61.9 ±1.4	68.8 ±0.9	68.6 ±1.5	62.58	67.64
G Potential ethical and social implications
There are a few potential ethical considerations that could arise from the molecular graph representation learning methods discussed in :
Bias and fairness: We recognise that the molecular datasets used for pre-training and benchmarking may inadvertently encode biases or lack diversity, which could lead to models that unfairly under
Table 18: Benchmarking topological properties, on 2M ZINC15. We report the mean square error or the cross entropy on eight datasets (i.e., smaller is better).
Node	Pair	Graph
Degree	Cent.	Cluster	Link	Jaccord	Katz	Diameter	Conn.	Cycle	Assort.
RANDOM	0.001	0.008	0.003	0.078	0.012	0.017	177.924	0.087	2.933	0.029
ATTRMASK	0.058	0.010	0.003	0.082	0.013	0.020	142.832	0.071	3.232	0.027
GRAPHCL	0.063	0.009	0.003	0.079	0.018	0.022	87.426	0.064	1.716	0.017
GRAPHMAE	0.032	0.011	0.004	6.885	0.215	0.038	123.677	0.066	3.094	0.025
RGCL	0.356	0.011	0.003	0.075	0.013	0.017	124.869	0.067	3.528	0.029
perform on certain data subsets. We have made efforts to use representative datasets, but further testing on diverse molecules and monitoring for fairness issues remain important future directions.
Relatedly, an inherent risk with data-driven methods is "stereotyping" along chemical lines, where models reinforce historical assumptions. We believe conscious testing on new classes of molecules can mitigate this. The goal is to build broad molecular understanding.
Dual use: As with many advances in chemistry ML, dual use concerns exist. While we aim to accelerate beneficial therapeutics, we strongly discourage misuse of these methods to design harmful substances. We support efforts toward responsible AI practices.
Environmental impact: Training large molecular graph models consumes significant computational resources and energy, efforts should be made to minimise the environmental impact, e.g., by using efficient methods and carbon-neutral compute.
Privacy: While molecular graph data is less sensitive than data about individuals, any efforts to link models back to original private datasets should be handled carefully.
Overall, we make efforts to align research practices with principles of sharing benefits, avoiding harm, mitigating biases, and practising transparency. We welcome feedback from the community as we continue working to address these ethical dimensions. Our goal is innovation that aligns with broad social good.