SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 8 and Table 9 show statistics regarding the benchmark datasets used to evaluate our models. In Table 8, we list the number of learning instances and the number of binary tasks in each chemical dataset. Out of 8 datasets, 6 are small-size and 2 are medium-size. Table 9 presents long-range
datasets and predictive tasks on peptides. The peptide datasets have similar learning instances but with different predictive tasks.
Table 8: Binary Chemical Benchmarks
Datasets	Descriptions	Number of Graphs	Number of Tasks
BBBP	Blood-brain barrier permeability	2039	1
Tox21	Toxicology on 12 biological targets	7831	12
ToxCast	Toxicology measurements via high-throughput screening	8575	617
SIDER	Adverse drug reactions of marketed medicines	1427	27
ClinTox	Drugs that failed clinical trials for toxicity reasons	1478	2
MUV	Validation of virtual screening techniques	93087	17
HIV	Ability to inhibit HIV replication	41127	1
BACE	Binding results for inhibitors of human β-secretase 1	1513	1
Table 9: Long-range Peptide Benchmarks
Datasets	Peptides-func	Peptides-struct
Number of Graphs	15535	15535
Number of Tasks	1	5
Number of Classes	10	N/A
Task types	Multi-label classification	Multi-label regression
Descriptions	Peptide functions: antibacterial, antiviral, etc	3D properties: length, sphericity, etc
To maintain comparability with prior works, we use different featurizations for molecules from the binary chemical benchmarks and the long-range benchmarks. For data from Table 8, we follow the featurization used by [15], in which there are 2 input atom features and 2 input bond features  ${}^{1}$  . For data from Table 9,we use the standard featurization provided in Open Graph Benchmark [14], in which there are 9 input atom features and 3 input bond features  ${}^{2}$  . The latter featurization scheme is more up-to-date and adopted by recent works on machine learning in the chemical space  $\left\lbrack  {8,{50},{51}}\right\rbrack$  .
C Vocabulary
C.1 Vocabulary Statistics
In Figure 4, we present statistics on fragments in the extracted vocabulary, molecular graphs, and fragment graphs. Subplot c shows the distribution of fragments based on their sizes. The smallest fragments consist of singleton atoms, while the largest one contains 21 atoms. The majority of fragments comprise 1 to 14 atoms. Although the extracted vocabulary is precise, its fragments are both large and highly prevalent, which is not the case for fragmentations used in previous works. In fact, the most prevalent fragments are size-independent. As shown in subplot d, common fragments are well distributed among all fragment sizes. This high prevalence throughout the vocabulary fosters pattern recognition, thereby contributing to the competitive results achieved by our models. The presence of large fragments also positively impacts the size of the fragment graphs. Subplot a and subplot b depict the size distributions of the fragment graphs and the molecular graphs, respectively. Relative to the molecular graphs, fragment graphs are more sparse and considerably smaller in size. This property, combined with the optimized fragments, enables them to capture the higher-order connectivity patterns of graph components effectively.
C.2 Vocabulary Size and Downstream Performance
The vocabulary size may influence the performance of pretrained models on downstream tasks. In Table 10, we report this effect when varying the vocabulary size from 200 to 3, 200, doubling the size with each step. The results suggest that in general, for each task, there is an optimal vocabulary size. For instance,  $\mathrm{GraphFP}_C$  performs the best on ClinTox and HIV when pretrained with a vocabulary of size 800. The optimal vocabulary size for Tox21 is 400 while the optimal size for BBBP falls
$^{1}$ Hu et al, ICLR 2020, Appendix C
$^{2}$ https://github.com/snap-stanford/ogb/blob/master/ogb/util/features.py
Figure 4: (a) Size distribution of fragment graphs (b) Size distribution of molecular graphs (c) Size distribution of fragments from the extracted vocabulary (d) Blue plot shows the occurrences of fragments in the vocabulary and purple plot shows the sizes of the fragments.
between 400 to 800. SIDER favors smaller vocabulary sizes while MUV favors larger vocabulary sizes. Varying vocabulary sizes seem to have little impact on the performance of ToxCast. BACE perceives the most interesting trend when the performance drops at vocabulary size 400 and rises again when the size increases.
Table 10: Downstream performances with  $\mathrm{GIN}_C$  pretrained on vocabulary of various sizes.
Vocab Size	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE
200	69.8 ± 1.5	75.6 ± 0.8	63.5 ± 0.8	61.3 ± 0.7	74.8 ± 4.3	74.9 ± 1.8	76.9 ± 1.0	79.4 ± 1.6
400	71.6 ± 1.4	75.8 ± 0.5	63.8 ± 0.4	61.3 ± 0.7	75.2 ± 6.6	75.7 ± 2.5	75.9 ± 1.0	77.5 ± 2.5
800	71.5 ± 1.6	75.5 ± 0.4	63.8 ± 0.6	61.4 ± 0.9	78.6 ± 2.7	77.2 ± 1.5	76.3 ± 1.0	78.2 ± 3.4
1600	71.1 ± 1.6	75.4 ± 0.8	63.9 ± 0.9	60.4 ± 0.5	76.4 ± 4.6	76.1 ± 2.1	75.3 ± 1.6	79.0 ± 4.3
3200	71.3 ± 0.8	75.4 ± 0.4	63.7 ± 0.6	60.6 ± 0.5	75.5 ± 5.9	77.1 ± 1.9	75.1 ± 1.3	79.4 ± 4.5
C.3 Fragment Graph Statistics
We report several statistics regarding the fragment graphs from the prepared pretraining dataset described in Section 4.1. Table 11 lists the number of graphs for each fragment graph size. The smallest fragment graphs are of size 1 (standalone fragment) while the largest fragment graphs are of size 30. The majority of fragment graphs are of smaller sizes. This indicates that the fragmentation algorithm was able to extract large and frequent fragments, resulting in small fragment graphs that can capture higher-order connectivity.
To illustrate the connectivity within fragment graphs, for each fragment graph size, we report the average number of edges in Table 12. To save space, we only consider fragment graphs with a maximum size of 10. Since our molecular graphs are connected, our fragment graphs are also connected (i.e., no disconnected island). From the above table, we can see that, given a fragment graph with size  $n$ , the average number of edges connecting the fragments varies from around  $n - 1$
Table 11: Number of Fragment Graphs by Size.
Size	1	2	3	4	5	6	7	8	9	10	11	...	30
Num Graphs	167	8433	76024	144940	112426	52607	20999	7742	3334	1570	788	...	30
to  $n$ . When the fragment graphs are small, the average number of edges is closer to  $n - 1$ , indicating that the fragment graphs are mostly tree-like. As the fragment graph size increases, more loops appear and thus the average number of edges deviates further from  $n - 1$ .
Table 12: Average number of edges by graph size.
Size	1	2	3	4	5	6	7	8	9	10
Average Num Edges	0.00	1.00	2.02	3.07	4.17	5.33	6.51	7.74	8.88	10.11
Finally, in Table 13, with varying vocabulary sizes, we report the average fragment graph size and the average number of edges for the whole pretraining dataset. As the vocabulary size increases, the size of fragment graphs decreases in general since larger vocabularies contain larger fragments.
Table 13: Average graph size and average number of edges with varying vocabulary size.
Vocabulary Size	200	400	800	1600	3200
Average Graph Size	5.92	5.17	4.61	4.15	3.73
Average Num Edges	5.23	4.40	3.78	3.28	2.84