SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Computational resource. We perform experiments on an NVIDIA DGX A100 server. Each individual experiment can be run on a single GPU without exceeding 30 GBs of GPU memory.
D.1 Compared Methods
Motif-based tokenizers. We now elaborate on the details of the two compared motif-based tokenizers:
- MGSSL [1] employs the BRICS [35] method for molecule fragmentation (Section 2.2). To obtain more fine-grained fragments, MGSSL employs two additional rules to break the BRICS's output fragments: 1) separate the single atoms attached to cycles; 2) if a connected subgraph comprising three or more atoms is not part of a cycle, break it down as a new fragment.
- RelMole [18] combines the fragmentation functions of Cycles and FGs for molecule fragmentation (Section 2.2). Further, it extracts the carbon-carbon single bonds that are not covered in the previous step as new fragments.
We use the motif vocabulary provided by their paper for molecule fragmentation. Given a molecule, we convert its fragmented motifs into one-hot encodings, which serve as the reconstruction targets.
Pretrained GNN-based tokenizers. We use the atomic numbers as the node features and exclude edge features in pretrained GNN-based tokenizers. We show in Appendix E that incorporating edge
Table 9: Experimental settings for pretraining on 2 million molecules from ZINC15 and fine-tuning on eight datasets in MoleculeNet: BBBP, Tox21, ToxCast, Sider, ClinTox, MUV, HIV, and Bace.
(a) Node and edge features.
Type	Range
Node features	atomic numbers	1~118
chirality tag	{unspecified, tetrahedral cw, tetrahedral ccw, other}
Edge features	bond type	{single, double, triple, aromatic}
bond direction	{-, endupright, enddownright}
(b) Hyperparameters.
Encoder	pretrain	fine-tuning
lr	batch size	epochs	lr	batch size	epochs
GINE	1e-3	1024	100	{1e-3, 1e-4}	32	100
GTS	1e-4	2048	100	{1e-4, 1e-5}	32	100
features in GNN tokenizers can decrease performance. Due to the removal of edge features, the tokenizer uses the architecture of GIN [37] instead of GINE [12]. We have reported performances of GNN-based tokenizers that are pretrained by GraphCL [4], GraphMAE [9], and VQ-VAE [10, 43]. The implementation of VQ-VAE follows [10] and groups the latent codes by atomic numbers. We strictly follow the procedure in the mentioned papers to pretrain GNNs, which are later used as tokenizers.
Simple GNN-based tokenizers (SGTs). An SGT uses the node feature of atomic number. It uses the graph encoder's linear embedding function of atomic numbers. We present the graph operators for our tested SGTs below:
$$
\text {G I N :} \quad \omega (\mathbf {A}) = \mathbf {A} + (1 + \epsilon) \mathbf {I}, \tag {17}
$$
$$
\text {G C N :} \quad \omega (\mathbf {A}) = \tilde {\mathbf {D}} ^ {- 1 / 2} \tilde {\mathbf {A}} \tilde {\mathbf {D}} ^ {- 1 / 2}, \tag {18}
$$
$$
\text {G r a p h S A G E :} \quad \omega (\mathbf {A}) = \tilde {\mathbf {D}} ^ {- 1} \tilde {\mathbf {A}}, \tag {19}
$$
where  $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$  and  $\tilde{\mathbf{D}}$  is the degree matrix of  $\tilde{\mathbf{A}}$ ;  $\epsilon$  is set to 0.5 empirically.
Baselines. We now describe the details of our reported baseline methods:
- Infomax [69] learns node representations by maximizing the mutual information between the local summaries of node patches and the patches' graph-level global summaries.
- ContextPred [12] uses the embeddings of subgraphs to predict their context graph structures.
- InfoGraph [70] conducts graph representation learning by maximizing the mutual information between graph-level representations and local substructures of various scales.
- GraphCL [4] performs graph-level contrastive learning with combinations of four graph augmentations, namely node dropping, edge perturbation, subgraph cropping, and feature masking.
- JOAO [71] proposes a framework to automatically search proper data augmentations for GCL.
- AD-GCL [72] applies adversarial learning for adaptive graph augmentation to remove the redundant information in graph samples.
- GraphLOG [73] leverages clustering to construct hierarchical prototypes of graph samples. They further contrast each local instance with its corresponding higher prototype for contrastive learning.
- RGCL [74] trains a rationale generator to identify the causal subgraph in graph augmentation. Each graph's causal subgraph and its complement are leveraged in contrastive learning.
- BGRL [62] trains an online encoder by learning to predict the output of a target encoder. The target encoder shares the same architecture as the online encoder and is updated through exponentially moving average. The inputs of the online encoder and the target encoder are two different graph augmentations.
Table 10: Experimental setting for pretraining on the 50 thousand molecules from the GEOM dataset and fine-tuning on the four molecular property prediction (regression) datasets and DTA datasets.
(a) Node and edge features.
Type	Range
Node features	atomic numbers	1~118
chirality tag	{unspecified, tetrahedralcw, tetrahedralccw, other}
node degree	0~10
formal_charge	-5~5
number of H	0~8
number of radical e	0~4
hybridization	{sp, sp2, sp3, sp3d, sp3d2}
is aromatic	{false, true}
is in ring	{false, true}
Edge features	bond type	{single, double, triple, aromatic}
bond stereo	{stereonone, stereoz, stereoe, stereocis, stereotrans, stereoany}
is conjugated	{false, true}
(b) Hyperparameters and their search spaces. We use the performance on the validation set for hyperparameter tuning. **Bold** indicates the final value used in experiments.
Encoder	pretrain	fine-tune (regression)	fine-tune (DTA)
lr	batch size	epochs	lr	batch size	epochs	lr	batch size	epochs
GINE	1e-3	1024	100	1e-3	{32, 128, 256}	100	{1e-4, 2e-4}	128	500
GTS	1e-4	1024	300	{1e-4, 2e-4, 3e-4}	32	100	{1e-4, 2e-4}	128	500
- GraphMAE [9] shows that a linear classifier is insufficient for decoding node types. It applies a GNN for decoding and proposes remask to decouple the functions of the encoder and decoder in the autoencoder.
- GraphMVP [45] uses a contrastive loss and a generative loss to connect the 2-dimensional view and 3-dimensional view of the same molecule, in order to inject the 3-dimensional knowledge into the 2-dimensional graph encoder.
- S2GAE [75] randomly masks a portion of edges of graphs and pretrain the graph encoder to predict the missing edges.
- GraphMAE2 [76] applies multi-view random re-mask decoding as a regularization for MGM pretraining.
- Mole-BERT [10] combines a contrastive learning objective and a masked atom modeling objective for MRL. Specifically, they observe that mask atom prediction is an overly easy pretraining task. Therefore, they employ a GNN tokenizer pretrained by VQ-VAE [43] to generate more complex reconstruction targets for masked atom modeling.
D.2 Experimenets in Section 4 and Table 5
Here we elaborate the experimental setting for pretraining on 2 million molecules from ZINC15 [42] and fine-tuning on the eight classification datasets in MoleculeNet [28]: BBBP, Tox21, ToxCast, Sider, ClinTox, MUV, HIV, and Bace. This setting covers the experiments in Section 4 and Table 5.
Molecule representations. For SimSGT and other compared methods, we follow previous works [12, 4] and use a minimal set of molecule features as the graph representations (Table 9a). These features unambiguously describe the two-dimensional structure of molecules.
Hyper-parameters. Table 9b summarizes the hyper-parameters. We use different hyper-parameters given different graph encoders. The architectures of the two graph encoders are borrowed from previous works: GINE [12] and GTS [27]. We use large batch sizes of 1024 and 2048 to speed up pretraining. We do not use dropout during pretraining. During fine-tuning, we  $50\%$  dropout in GINE layers and  $30\%$  dropout in transformer layers. The learning rate for the MUV dataset is 10 times smaller than other datasets. Following [4, 73], we report the last epoch's test performance. We
Table 11: Hyperparameters for fine-tuning on the QM datasets.
QM Dataset	batch size	lr
QM7	32	4e-4
QM8	256	1e-3
QM9	256	1e-3
report the mean performances and the standard deviations across 10 random seeds. Baselines are reproduced using the same setting.
Linear probing experiments. Here we elaborate on the settings of our linear probing experiments (Figure 5b and Figure 7). Specifically, we randomly split the 2 million molecules from ZINC15 into train set  $(90\%)$  and test set  $(10\%)$ . We train the MGM models on the train set and save the encoder's checkpoint every epoch. The linear classifiers are trained for 1000 epochs on the encoder's frozen hidden representations. We train linear classifiers using 25600 molecule samples from the training set and evaluate them on the whole test set.
- Probing masked atom types (Figure 5b). We let linear classifiers predict the masked atom types using the masked atoms' hidden representations. During linear probing, we disable remask-v2 to obtain the masked atoms' hidden representations. Molecules are randomly masked by 0.35 during probing. We use accuracy (\%) as the evaluation metric.
- Probing FGs (Figure 7). Following [2], we extract 85 types of FGs for each molecule using RDkit [30]. FGs are represented by 85-dimensional binary vectors, whose each dimension indicates the presence of a certain FG. Afterward, we train multi-label linear classifiers on the frozen encoder's mean pooling outputs for FG prediction. Molecules are not masked during probing. We use ROC-AUC (\%) as the evaluation metric.
D.3 Experiments in Table 6
We present the experimental setting for pretraining on the 50 thousand molecules from GEOM [46] and fine-tuning on the four molecule property prediction (regression) datasets and two DTA datasets. Our experimental setting follows that in [45]. This setting covers the experiments in Table 6.
Molecule representations. In the graph autoencoder, we use 9-dimensional node features and 3-dimensional edge features of molecules provided by the OGB [77] package, following GraphMVP [45]. The features are summarized in Table 10a. Note that, our tokenizer uses only the atomic numbers as node features and does not use edge features.
Hyper-parameters. The hyperparameters are summarized in Table 10b. We tune the hyperparameters in the fine-tuning stage using the validation performance. Following [45], we report the test performance at the epoch selected by the validation performance. We do not use dropout during pretraining. During fine-tuning, we  $50\%$  dropout in GINE layers and  $30\%$  dropout in transformer layers.
For a fair comparison, we reproduce Mole-BERT [10]'s performance by pretraining on the 50 thousand molecule samples from the GEOM dataset [46]. The original Mole-BERT is trained on a larger dataset of 2 million molecules from ZINC15 [42].
D.4 Experiments in Table 7
The hyperparameters for fine-tuning on the QM datasets are reported in Table 11.
In this section, we provide more experimental results. If not especially noted, these experiments employ an autoencoder of GTS encoder and GTS-Small decoder with remask-v2, and a tokenizer of a single-layer SGT of GIN. Other settings follow that in Appendix D.2.
Influence of edge features for pretrained GNN-based tokenizer. We ablate the impact of the "bond type" and "bond direction" edge features in pretrained GNN-based tokenizers. We use GINE and
Table 12: Average transfer learning ROC-AUC (%) scores on the eight classification datasets in MoleculeNet. Including edge features in tokenizers decreases the performance.
Tokenizer	Edge feature	Tokenizer GNN's depth
1	2	3	4	5
Pretrain, GraphCL	X	75.1	74.5	74.2	74.0	74.6
✓	74.2	74.7	73.7	73.8	73.2
Pretrain, GraphMAE	X	75.1	74.9	74.9	75.4	75.2
✓	74.6	74.6	74.3	74.6	75.0
Figure 8: SimSGT pretraining on the 2 millions molecules from ZINC15.
GIN for the tokenizer with and without edge features. Table 12 shows that including edge features in GNN-based tokenizers negatively influences the transfer learning performance. Therefore, we exclude edge features from pretrained GNN-based tokenizers in our experiments.
Influence of Batch Normalization layers in SGTs. The Batch Normalization [41] (BN) layers in SGTs are crucial to avoid loss vanishment. Figure 8a presents a comparison between SGT "with vs without BN". Without the BN layer, the MSE loss drops to lower than 0.01 within a few steps of pretraining. Such small loss values lead to significant model underfitting.
As shown by Figure 8b, the token values of SGT without BN follow a sharp distribution: the values are primarily distributed around zero, and their standard deviation (std) is smaller than 0.35. This minor std issue might be caused by the smoothing effect of GNNs [78]. An expressive neural network (i.e., a graph autoencoder) can quickly fit this sharp target distribution and minimize the loss to a negligible value, causing the problem of loss vanishment. However, if a BN layer is used, it forces each dimension of the tokenizer output to have an std of 1.00, so as to "spread out" the distribution of the SGT tokens. These new SGT tokens of a larger std are harder to fit. They keep the MSE loss at a reasonable range of  $0.10 \sim 0.15$  (Figure 8a).
Mask ratio. We apply random node masking throughout the experiments [12]. Figure 9 presents SimSGT's sensitivity with respect to the mask ratios. SimSGT is not sensitive to mask ratios such that a wide range of ratios  $(0.25\sim 0.45)$  can generate competitive performances. The ratio of 0.35 achieves the best performance. This ratio is much lower than that for images, where a ratio of 0.75 can generate promising performances [15].
Balancing the distribution of reconstruction targets. As shown in Figure 11, the popularly used ZINC15 dataset includes 12 types of atoms, and  $95\%$  of the atoms are distributed on the top three atom types. This skewed distribution renders the node-level token reconstruction an easy pretraining task [10]. Figure 10 shows that the accuracy of predicting node-level tokens converges quickly. Such an easy pretraining task can lead to suboptimal performance as suggested by existing SSL literature [79, 80]. In Figure 11, we show that the induced subgraphs of a single-layer SGT (i.e., one-hop rooted subtrees) follow a more balanced distribution than the distribution of nodes. SGT tokens also have a larger vocabulary size: ZINC15 includes 555 types of one-hop rooted subtrees. Consequently, the accuracy of predicting tokens of a single-layer SGT takes more epochs to converge (Figure 10).
Pooling Method for Subgraph Representations. In previous experiments, we use mean pooling to obtain the subgraph representations for motif-based tokenizers, following the method of obtaining graph representations in [4, 12]. Here we add results for MGSSL tokenizer using sum and max
Figure 9: Average ROC-AUC  $(\%)$  scores with respect to different node masking ratios. The performances are evaluated on the eight classification datasets in MoleculeNet.
Figure 10: Token prediction accuracies. SGT token prediction is conducted by calculating the Euclidean distance between the autoencoder's output and the vocabulary of all SGT tokens.
Figure 11: Distributions of graph fragments in MGM. Statistics come from molecules in ZINC15 [42]. For the subgraph types, a colon ‘:’ separates the center node and the neighbor nodes. For example, C:CN denotes a subgraph with a center of a Carbon and neighbors of a Carbon and a Nitrogen.
pooling in Table 13. The results show that mean pooling yields the highest performance, affirming the soundness of our previous experiments.
Full results of Section 4. We provide the full results of experiments in Section 4. Table 14 contains the full results of Table 3a and Table 4. Table 15 contains the full results of Table 3b and Figure 6.
Table 13: ROC-AUC (%) scores on eight MoleculeNet datasets. Compared methods use GTS encoder and GTS-Small decoder with remask-v2 decoding.
Dataset	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	Avg.	GAIN
Motif, MGSSL, Mean	72.5±0.9	77.5±0.4	65.2±0.6	60.7±0.9	85.0±3.5	79.9±1.5	78.0±1.5	83.0±1.0	75.2	5.3
Motif, MGSSL, Max	71.5±0.9	75.8±1.2	66.2±0.7	60.7±1.3	82.6±2.3	78.9±1.8	76.5±1.4	83.8±1.6	74.5	4.6
Motif, MGSSL, Sum	71.7±1.4	75.9±0.6	66.1±0.7	60.4±1.4	83.4±1.5	79.5±1.0	76.8±1.2	84.2±1.1	74.8	4.8
Table 14: Transfer learning ROC-AUC (%) scores on the eight classification datasets in MoleculeNet.
Encoder	Decoder	Remask	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	Avg.
GINE	Linear	-	72.5±0.8	76.0±0.4	63.7±0.5	60.1±0.7	81.2±2.6	74.2±1.6	78.0±0.8	79.6±1.4	73.2
GINE	GINE-Small	-	70.9±0.6	75.1±0.5	63.5±0.4	61.0±0.4	79.1±2.6	76.0±0.5	76.3±0.5	82.5±0.9	73.0
GINE	GINE-Small	v1	70.2±1.0	76.4±0.6	64.2±0.4	61.9±0.8	80.9±1.8	77.8±1.1	78.1±1.1	83.6±1.1	74.1
GTS	Linear	-	72.9±1.0	76.6±0.8	63.8±0.8	58.3±1.3	81.9±5.5	78.5±1.5	78.0±2.0	83.1±0.9	74.1
GTS	GTS-Small	-	72.0±0.6	74.7±0.4	63.7±0.4	58.9±0.6	86.0±2.0	78.9±1.2	77.3±0.7	81.0±0.7	74.1
GTS	GTS-Small	v1	71.3±1.0	77.0±1.2	66.2±0.6	60.6±1.4	84.5±3.4	81.5±1.5	77.0±1.6	83.5±1.2	75.2
GTS	GTS-Small	v2	72.2±0.9	76.8±0.9	65.9±0.8	61.7±0.8	85.7±1.8	81.4±1.4	78.0±1.9	84.3±0.6	75.8
GTS	GTS-Tiny	v2	71.9±1.2	77.2±1.1	65.6±0.5	61.7±1.4	82.9±2.4	79.6±1.4	76.8±1.3	82.1±1.5	74.7
GTS	GTS	v2	70.7±1.2	76.4±0.9	66.1±0.4	60.3±0.9	84.7±4.6	79.6±0.7	76.8±1.9	84.5±0.8	74.9
Table 15: Transfer learning ROC-AUC (%) scores on the eight classification datasets in MoleculeNet.
Tokenizer	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	Avg.
Node	70.3±0.9	76.4±1.0	65.7±0.7	61.7±0.9	81.9±3.5	79.8±0.7	77.4±1.8	84.6±1.1	74.7
Motif, MGSSL	72.5±0.9	77.5±0.4	65.2±0.6	60.7±0.9	85.0±3.5	79.9±1.5	78.0±1.5	83.0±1.0	75.2
Motif, RelMole	71.4±1.3	77.1±0.4	66.3±0.6	58.9±1.2	80.7±2.7	79.2±1.4	78.0±1.0	83.6±1.0	74.4
Pretrain, GraphCL, 1 layer GIN	72.2±1.4	76.8±0.4	66.0±0.8	60.8±1.2	81.4±2.5	81.1±1.5	78.4±1.3	83.8±0.9	75.1
Pretrain, GraphCL, 2 layer GIN	70.8±0.6	76.7±0.8	66.3±0.4	60.6±1.2	84.3±3.3	77.2±1.4	76.3±2.1	83.8±1.2	74.5
Pretrain, GraphCL, 3 layer GIN	70.6±0.8	77.1±0.6	65.4±0.5	59.3±1.4	81.2±3.8	79.4±2.3	76.4±1.9	84.2±1.1	74.2
Pretrain, GraphCL, 4 layer GIN	72.1±0.9	76.9±0.6	65.7±0.7	59.6±0.9	77.6±3.5	81.7±1.2	77.6±2.2	81.1±1.1	74.0
Pretrain, GraphCL, 5 layer GIN	71.4±0.7	76.9±0.7	66.5±0.7	60.0±1.2	80.8±2.3	81.0±1.1	77.7±1.2	82.5±1.5	74.6
Pretrain, VQ-VAE, 1 layer GIN	72.2±0.9	77.0±0.6	66.5±0.6	61.3±1.8	82.8±3.7	79.1±2.0	77.4±1.5	84.2±0.8	75.1
Pretrain, VQ-VAE, 2 layer GIN	71.5±0.8	76.6±0.6	65.9±0.7	60.3±0.7	82.1±2.0	81.5±2.4	77.2±1.9	84.3±1.1	74.9
Pretrain, VQ-VAE, 3 layer GIN	71.9±0.9	76.7±0.9	65.8±0.8	61.2±1.8	79.5±2.5	80.0±0.9	78.0±1.3	81.7±0.9	74.4
Pretrain, VQ-VAE, 4 layer GIN	72.5±1.0	77.0±0.6	66.3±0.3	61.7±1.5	86.7±2.2	80.3±1.6	77.6±1.3	82.7±0.9	75.6
Pretrain, VQ-VAE, 5 layer GIN	72.0±0.9	76.8±0.6	65.6±0.6	61.5±1.1	84.3±1.3	80.6±1.0	78.1±1.4	81.9±0.8	75.1
Pretrain, GraphMAE, 1 layer GIN	72.0±1.1	77.3±0.6	66.3±0.3	60.9±1.5	83.7±2.7	79.9±1.4	76.3±2.2	84.0±1.6	75.1
Pretrain, GraphMAE, 2 layer GIN	71.9±0.8	77.6±0.6	66.0±0.5	60.8±1.8	81.9±3.9	79.0±1.6	76.5±2.4	85.3±0.7	74.9
Pretrain, GraphMAE, 3 layer GIN	71.4±0.7	77.6±0.8	65.8±0.3	61.1±1.7	82.2±2.9	79.2±1.5	77.4±2.1	84.3±0.9	74.9
Pretrain, GraphMAE, 4 layer GIN	72.3±0.7	76.6±0.7	66.1±0.8	62.0±1.2	83.3±2.1	80.1±2.2	77.9±1.7	85.0±0.9	75.4
Pretrain, GraphMAE, 5 layer GIN	72.6±0.6	76.4±0.5	65.7±0.6	62.4±1.3	84.0±2.8	80.0±1.3	78.7±1.3	81.5±1.3	75.2
SGT, 1 layer GIN	72.2±0.9	76.8±0.9	65.9±0.8	61.7±0.8	85.7±1.8	81.4±1.4	78.0±1.9	84.3±0.6	75.8
SGT, 2 layer GIN	71.3±0.7	77.0±0.9	66.2±0.8	61.5±0.8	84.9±2.0	80.7±2.0	78.1±1.1	84.5±0.8	75.5
SGT, 3 layer GIN	71.7±0.8	77.6±0.8	66.2±0.6	61.2±2.4	85.8±2.6	80.4±1.1	77.7±2.1	84.1±1.4	75.6
SGT, 4 layer GIN	72.0±0.9	77.1±1.2	65.4±1.0	61.7±1.5	83.8±2.2	80.1±1.9	77.5±1.2	84.7±0.9	75.3
SGT, 5 layer GIN	70.7±0.7	77.0±0.7	65.9±0.8	61.1±1.6	83.1±1.6	79.9±1.5	77.6±1.6	83.9±1.4	74.9
SGT, 1 layer GCN	71.9±0.9	77.8±1.0	66.5±0.7	62.0±0.8	85.2±1.6	79.2±1.4	77.9±2.3	84.4±2.1	75.6
SGT, 3 layer GCN	71.1±1.0	77.4±0.8	66.3±0.4	61.6±1.1	84.4±2.5	78.7±1.1	77.7±2.4	85.1±1.3	75.3
SGT, 5 layer GCN	71.0±1.0	76.5±0.6	65.6±0.3	61.9±1.3	83.7±2.1	78.3±2.0	76.7±1.2	84.9±1.0	74.8
SGT, 1 layer GraphSAGE	72.5±0.7	77.1±0.8	66.7±0.5	61.3±1.0	86.2±1.8	80.7±1.5	76.6±1.7	83.5±1.0	75.6
SGT, 3 layer GraphSAGE	70.3±1.1	77.9±0.7	65.4±0.6	60.3±1.0	87.4±3.7	79.2±1.9	77.6±2.1	84.9±0.5	75.4
SGT, 5 layer GraphSAGE	71.6±1.0	76.5±0.7	66.4±0.7	60.8±1.4	85.6±2.4	78.6±1.3	77.0±1.6	84.4±0.9	75.1