SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We pretrain GNNs according to the process discussed in section 3. The pretrained models are evaluated on various chemical benchmarks.
Datasets We use a processed subset containing 456K molecules from the ChEMBL database [24] for pretraining. A fragment vocabulary of size 800 is extracted as described in Section 3.1.1. To ensure possible fragmentation of unseen molecules, we further complete the vocabulary with atoms not existing in the pretraining set, totaling 908 unique fragments. For downstream evaluation, we consider 8 binary graph classification tasks from MoleculeNet [36] with scaffold split [15]. Moreover, to assess the ability of the models in recognizing global arrangement, we consider two graph prediction tasks on large peptide molecules from the Long-range Graph Benchmark [7]. Long-range graph benchmarks are split using stratified random split. More information regarding datasets is provided in Appendix B.
Models For graph classification benchmarks, we model our molecular encoders  $\mathrm{GNN}_M$  with the 5-layer Graph Isomorphism Network (GIN) [41] as in previous works [15, 23], using the same featurization. Similarly, we model the fragment encoder  $\mathrm{GNN}_F$  with a shallower 2-layer GIN since fragment graphs are notably smaller. For long-range benchmarks, both  $\mathrm{GNN}_M$  and  $\mathrm{GNN}_F$  are GIN with 5 layers, with the featurization of molecular graphs based on the Open Graph Benchmark [14]. All encoders have hidden dimensions of size 300. For more details, please refer to Appendix A.
Pretraining and Finetuning All pretrainings are done in 100 epochs, with AdamW optimizer, batch size 256, and initial learning rate  $1 \times 10^{-3}$ . We reduce the learning rate by a factor of 0.1 every 5 epochs without improvement. We use the models at the last pretraining epoch for finetuning. On graph classification benchmarks, to ensure comparability, our finetuning setting is mostly similar to that of previous works [15, 23]: 100 epochs, Adam optimizer, batch size 256, initial learning rate  $1 \times 10^{-3}$ , and dropout rate chosen from  $\{0.0, 0.5\}$ . We reduce the learning rate by a factor of 0.3 every 30 epochs. On long-range benchmarks, the setting is similar except that we finetune for 200 epochs and factor the learning rate by 0.5 every 20 epochs without improvement. We find that using averaging as FRAGPOOL( $\cdot$ ) works well. All experiments are run on individual Tesla V100 GPUs.
Baselines We compare to several notable pretraining baselines, including predictive methods (AttrMask & ContextPred [15], G-Motif & G-Contextual (GROVER) [27]), generative method (GPT-GNN [16]), contrastive methods (GraphLoG [42], GraphCL [46], JOAO, JOAOvs [45]), contrastive method with privileged knowledge (GraphMVP [23]), and fragment-based method (MGSSL [49]). For long-range prediction, we compare our method with popular GNN architectures: GCN [19], GCNII [2], GIN [41], and GatedGCN [1] with and without Random Walk Spatial Encoding [6].
Our goal is to benchmark the quality of our proposed pretraining methods with existing work. The settings in our experiments are chosen to fulfill this objective. In general, our settings follow closely those from previous works to ensure comparability. Specifically, we use the same embedding model (5-layer GIN), featurization, and overall similar hyperparameters and amount of pretraining data as those in the baselines [15, 23, 42, 45, 46, 49]. We limit the amount of hyperparameter tuning for the same reason. In general, self-supervised learning and property prediction on molecular graphs are important research topics and a wide variety of methods have been proposed in terms of both the pretraining task and the learning model, resulting in impressive results [8, 50, 51]. Some of these performances stem from deeper neural network design, extensive featurization, and large-scale pretraining. Pretraining at the scale of [8, 51] requires hundreds of GBs or even a TB of memory.
Table 1: Test ROC-AUC on binary molecular property prediction benchmarks using different pretraining strategies in GraphFP. The top-3 performances on each dataset are shown in red color, with red being the best result, red being the second best result, and red being the third best result. The 2 rightmost column shows the average performance ranking (lower value means better ranking) and the average AUC. The last 5 rows show the performances of our methods, with  $C$ ,  $P$ , and  $F$  indicate contrastive pretraining, predictive pretraining, and inclusion of fragment encoders in downstream prediction, respectively.
Pretraining Strategies	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	Avg. Rank	Avg. AUC
AttrMasking [15]	64.3 ± 2.8	76.7 ± 0.4	64.2 ± 0.5	61.0 ± 0.7	71.8 ± 4.1	74.7 ± 1.4	77.2 ± 1.1	79.3 ± 1.6	7.88	71.15
ContextPred [15]	68.0 ± 2.0	75.7 ± 0.7	63.9 ± 0.6	60.9 ± 0.6	65.9 ± 3.8	75.8 ± 1.7	77.3 ± 1.0	79.6 ± 1.2	7.56	70.89
G-Motif [27]	66.9 ± 3.1	73.6 ± 0.7	62.3 ± 0.6	61.0 ± 1.5	77.7 ± 2.7	73.0 ± 1.8	73.8 ± 1.2	73.0 ± 3.3	14.25	70.16
G-Contextual [27]	69.9 ± 2.1	75.0 ± 0.6	62.8 ± 0.7	58.7 ± 1.0	60.6 ± 5.2	72.1 ± 0.7	76.3 ± 1.5	79.3 ± 1.1	11.88	69.34
GPT-GNN [16]	64.5 ± 1.4	74.9 ± 0.3	62.5 ± 0.4	58.1 ± 0.3	58.3 ± 5.2	75.9 ± 2.3	65.2 ± 2.1	77.9 ± 3.2	13.63	67.16
GraphLoG [42]	67.8 ± 1.9	75.1 ± 1.0	62.4 ± 0.2	59.5 ± 1.5	65.3 ± 3.2	73.6 ± 1.2	73.7 ± 0.9	80.2 ± 3.5	12.56	69.70
GraphCL [46]	69.7 ± 0.7	73.9 ± 0.7	62.4 ± 0.6	60.5 ± 0.9	76.0 ± 2.7	69.8 ± 2.7	78.5 ± 1.2	75.4 ± 1.4	12.13	70.78
JOAO [45]	70.2 ± 1.0	75.0 ± 0.3	62.9 ± 0.5	60.0 ± 0.8	81.3 ± 2.5	71.7 ± 1.4	76.7 ± 1.2	77.3 ± 0.5	9.56	71.89
JOAOv2 [45]	71.4 ± 0.9	74.3 ± 0.6	63.2 ± 0.5	60.5 ± 0.7	81.0 ± 1.6	73.7 ± 1.0	77.5 ± 1.2	75.5 ± 1.3	8.94	72.14
GraphMVP [23]	68.5 ± 0.2	74.5 ± 0.4	62.7 ± 0.1	62.3 ± 1.6	79.0 ± 2.5	75.0 ± 1.4	74.8 ± 1.4	76.8 ± 1.1	10.00	71.70
MGSSL [49]	68.9 ± 2.5	74.9 ± 0.6	63.3 ± 0.5	57.7 ± 0.7	67.5 ± 5.5	73.2 ± 1.9	75.7 ± 1.3	82.1 ± 2.7	10.94	70.41
GraphFP-JTc	71.5 ± 0.9	75.2 ± 0.5	63.6 ± 0.5	62.0 ± 1.0	77.7 ± 4.5	76.0 ± 2.2	75.6 ± 1.0	79.7 ± 1.3	6.13	72.66
GraphFP-JTcF	70.2 ± 1.7	72.7 ± 0.8	62.5 ± 0.9	59.3 ± 1.3	75.9 ± 5.6	73.9 ± 1.3	73.0 ± 1.9	74.2 ± 2.8	13.56	70.21
GraphFPc	71.5 ± 1.6	75.5 ± 0.4	63.8 ± 0.6	61.4 ± 0.9	78.6 ± 2.7	77.2 ± 1.5	76.3 ± 1.0	78.2 ± 3.4	5.50	72.81
GraphFPp	68.2 ± 1.2	76.0 ± 0.5	63.2 ± 0.7	59.3 ± 1.0	53.8 ± 3.8	74.5 ± 2.1	76.7 ± 1.0	80.7 ± 4.8	9.50	69.05
GraphFPCP	71.3 ± 1.7	75.5 ± 0.5	64.7 ± 0.2	61.3 ± 0.6	73.7 ± 3.9	76.6 ± 1.8	76.3 ± 1.0	81.3 ± 2.2	5.19	72.59
GraphFPcF	70.1 ± 1.8	74.3 ± 0.3	65.3 ± 0.8	64.7 ± 1.0	87.7 ± 5.8	74.5 ± 1.8	76.1 ± 2.0	77.1 ± 2.1	7.25	73.73
GraphFPcPF	72.0 ± 1.7	74.0 ± 0.7	63.9 ± 0.9	63.6 ± 1.2	84.7 ± 5.8	75.4 ± 1.9	78.0 ± 1.5	80.5 ± 1.8	4.56	74.01
Due to limited resources, we leave the evaluation of GraphFP and other baselines when pretrained on such larger datasets for future work.
4.2 Results on Graph Classification Benchmarks
Table 1 reports our results on chemical graph classification benchmarks. For each dataset, we report the mean and error from 10 independent runs with predefined seeds. Except for GraphLoG [42] and MGSSL [49], the results of other baselines are collected from the literature [15, 23, 37, 45, 46]. To ensure a comprehensive evaluation, we conduct experiments with all possible combinations of the proposed strategies, which include contrastive pretraining (denoted as  $C$ ), predictive pretraining (denoted as  $P$ ), and inclusion of fragment encoders in downstream prediction (denoted as  $F$ ). Because  $F$  requires  $C$ , all possible combinations of these components are  $\{C, P, CP, CF, CPF\}$ , with the corresponding pretrained models  $\mathrm{GraphFP}_C$ ,  $\mathrm{GraphFP}_P$ ,  $\mathrm{GraphFP}_{CP}$ ,  $\mathrm{GraphFP}_{CF}$ , and  $\mathrm{GraphFP}_{CPF}$ . For  $\mathrm{GraphFP}_{CP}$ , we choose  $\alpha = 0.3$  and for  $\mathrm{GraphFP}_{CPF}$ , we choose  $\alpha = 0.1$ . The choices of  $\alpha$  are reported in Appendix A. We also compare to  $\mathrm{GraphFP-JT}_C$  and  $\mathrm{GraphFP-JT}_{CF}$ , which are variations of our models pretrained with the fragmentation from [18]. The fragments used by [18] are generally smaller, resulting in larger fragment graphs. We found 5-layer GIN models encode these larger graphs better. Our models are competitive in all benchmarks, obtaining the best performance in 5 out of 8 downstream datasets. We report the average ranking and the average AUC of the models. As reported in Table 1, the GNNs with our proposed fragment-based pretraining and finetuning strategies achieve the best average rankings and average AUCs across baselines. Moreover, adding the strategies successively, i.e.  $C, CP$ , and  $CPF$ , improves the average downstream rankings, confirming the individual effectiveness of each strategy in supporting the learning on molecular graphs. We attribute the remarkable performances of our methods to the capability of the pretrained embeddings and the fragment graph encoder in capturing higher-order structural patterns. For instance, the BBBP benchmark requires predicting blood-brain barrier permeability, where the molecular shape, size, and interaction with the transporting proteins play significant roles in determining the outcome. Capturing such information necessitates a deep understanding of the global molecular structure, which is the goal of our pretraining strategies.
4.3 Results on Long-range Chemical Benchmarks
In Table 2, we compare fragment-based pretraining and finetuning of GraphFP with GNN baselines on two long-range benchmarks: PEPTIDE-FUNC containing 10 classification tasks regarding peptide functions and PEPTIDE-STRUCT containing five regression tasks regarding 3D structural information [7]. Because fragment graphs of peptides are extremely large for effective extraction of structural backbones, we exclude predictive pretraining in this experiment. On both benchmarks, the results
Table 2: Performances on PEPTIDE-FUNC (graph classification) and PEPTIDE-STRUCT (graph regression). These tasks require capturing long-range interactions within large peptide molecules. Best performances are colored red.
Methods	Peptide-func Test AP	Peptide-struct Test MAE
GCN	0.5930 ± 0.0023	0.3496 ± 0.0013
GCNII	0.5543 ± 0.0078	0.3471 ± 0.0010
GIN	0.5498 ± 0.0079	0.3547 ± 0.0045
GatedGCN	0.5864 ± 0.0077	0.3420 ± 0.0013
GatedGCN+RWSE	0.6069 ± 0.0035	0.3357 ± 0.0006
GraphFP CF	0.6267 ± 0.0073	0.3137 ± 0.0019
Table 3: Effects on varying the size of the vocabulary
Models	Fragmentation	ROC-AUC
Avg Frag Size	Avg Frag Graph Size	SIDER	ClinTox	HIV
GraphFP800	2.40	4.61	61.4 ± 0.9	78.6 ± 2.7	76.3 ± 1.0
GraphFP1600	2.47	4.15	60.4 ± 0.5	76.4 ± 4.6	75.3 ± 1.6
GraphFP3200	2.53	3.73	60.6 ± 0.5	75.5 ± 5.9	75.1 ± 1.3
show that the proposed method outperforms other GNN baselines, including GatedGCN+RWSE with positional encoding. In particular, our pretrained model  $\mathrm{GraphFP}_{CF}$  obtains about  $14\%$  improvement on PEPTIDE-FUNC and  $11.5\%$  improvement on PEPTIDE-STRUCT compared to vanilla GIN. As the tasks require recognizing long-range structural information, these improvements indicate the effectiveness of our strategies in capturing the global arrangement of molecular graph components.
Pretraining with Various GNN Architectures In Table 4, we report the performances of some other GNN architectures besides GIN. The pretraining and finetuning conditions are similar to those described in Section 4.1 and Table 6. All architectures share the same input features, hidden dimensions, and the number of layers. On average, GIN, being the most expressive GNN among the ones shown, performs the best. This observation agrees with the conclusions from previous works [15, 41].
Figure 3: (left) t-SNE plot showing the embeddings of a common fragment. Each color corresponds to a structural backbone. (right) t-SNE plot showing the embeddings of atoms in a molecule with colors distinguishing fragment membership. Round markers indicate atom embeddings while square markers indicate fragment-based pooling of atom embeddings.
Table 4: Downstream performances with models pretrained with various GNN architectures.
Architectures	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	Avg. Rank
GIN	71.3 ± 0.8	75.8 ± 0.3	63.9 ± 0.4	61.3 ± 0.4	63.2 ± 1.0	76.2 ± 2.0	76.1 ± 1.4	82.5 ± 1.9	1.88
GCN	70.5 ± 1.2	74.3 ± 0.3	63.3 ± 0.5	61.5 ± 0.8	78.1 ± 5.8	77.8 ± 1.8	74.6 ± 0.9	76.2 ± 2.1	2.13
GraphSage	72.3 ± 1.5	74.5 ± 0.6	62.8 ± 0.3	61.2 ± 0.8	69.6 ± 3.1	77.3 ± 1.3	77.1 ± 1.1	78.0 ± 3.6	2.00
Effects on Varying the Size of the Vocabulary The size of the fragment vocabulary likely influences the quality of pretraining and finetuning since it dictates the resolution of fragment graphs. To investigate such effects, we prepare two additional vocabularies with size 1, 600 and size 3, 200. We repeat the same contrastive pretraining as in Section 4.1 using the new vocabularies. As the vocabulary size increases, more unique and larger fragments are discovered, increasing the average fragment size and reducing the average fragment graph size. Table 10 shows that the performances worsen on some downstream benchmarks as the vocabulary size grows larger. We conjecture a few possible reasons. Firstly, a larger vocabulary means more parameters to optimize. Second, smaller fragment graphs represent an excessively loose view of the graph, resulting in a loss of structural information. In general, vocabulary size is an important hyperparameter that greatly affects the quality of self-supervised learning. In [20], the authors gave a discussion on selecting an optimal vocabulary size according to the entropy-sparsity trade-off.
Visualizing the Learned Embeddings As shown in Figure 3, we visualize the learned embeddings via t-SNE [33]. The left plot illustrates the embeddings produced by the pretrained fragment encoder  $\mathrm{GNN}_F$  of a common fragment. Each dot corresponds to a molecule in which the fragment appears. The color of a dot indicates the structural backbone of the molecule it represents. For visibility, we only show the most common backbones. The embeddings are reasonably separated according to the structural backbones, indicating that the fragment embeddings capture higher-order structural information. Notice that the fragment encoder  $\mathrm{GNN}_F$  is not directly trained to recognize structural backbones. The right plot shows the embeddings produced by the molecule encoder  $\mathrm{GNN}_M$  of atoms within a molecule. The embeddings of atoms within the same fragment are clustered together. Interestingly, some clusters, shown in green, purple, and black, are arranged similarly to how they appear in the original molecule. This observation confirms that the collective embedding of nodes can capture higher-order connectivity.
In this paper, we proposed contrastive and predictive learning strategies for pretraining GNNs based on graph fragmentation. Using an optimized fragment vocabulary, we pretrain two separate encoders for molecular graphs and fragment graphs, thus capturing structural information at different resolutions. When benchmarked on chemical and long-range peptide datasets, our method achieves competitive or better results compared to existing methods. Moving forward, we plan to further improve the pretraining via larger datasets, more extensive featurizations, better fragmentations, and more optimal representations. We also plan to extend the fragment-based techniques to other learning problems, including novel molecule generation and interpretability.
This project is supported by the BioPACIFIC Materials Innovation Platform of the National Science Foundation under Award # DMR-1933487 and high-performance computing infrastructure from the California NanoSystem Institute at the University of California Santa Barbara.
[1] Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiv:1711.07553, 2017.
[2] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[39] Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical Review Letters, 120(14):145301, 2018.
[40] Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2412-2429, 2023.
[41] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
[42] Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[43] Ling Xue and Jurgen Bajorath. Molecular descriptors in chemoinformatics, computational combinatorial chemistry, and virtual screening. Combinatorial Chemistry & High Throughput Screening, 3(5):363-372, 2000.
[44] Ronghui You, Shuwei Yao, Hiroshi Mamitsuka, and Shanfeng Zhu. Deepgraphgo: graph neural network for large-scale, multispecies protein function prediction. Bioinformatics, 37(Supplement 1):i262-i271, 2021.
[45] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[46] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Advances in Neural Information Processing Systems, 2020.
[47] Xiangxiang Zeng, Hongxin Xiang, Linhui Yu, Jianmin Wang, Kenli Li, Ruth Nussinov, and Feixiong Cheng. Accurate prediction of molecular properties and drug targets using a self-supervised image representation learning framework. Nature Machine Intelligence, 4(11):1004-1016, 2022.
[48] Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. arXiv preprint arXiv:2012.12533, 2020.
[49] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self-supervised learning for molecular property prediction. In Advances in Neural Information Processing Systems, 2021.
[50] Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. Unified 2d and 3d pre-training of molecular representations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2626-2636, 2022.
[51] Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Wengang Zhou, Tao Qin, Houqiang Li, and Tie-Yan Liu. Dual-view molecular pre-training. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3615-3627, 2023.
A Experimental Settings
A.1 Graph Isomorphism Network
Given a graph  $G = (V, E)$  with node attributes  $x_v$  for  $v \in V$  and edge attributes  $e_{uv}$  for  $(u, v) \in E$ , after the  $k$ -th layer of the GNN, the embedding of node  $v$  is updated as:
$$
h _ {v} ^ {(k)} = U _ {k} \left(h _ {v} ^ {(k - 1)}, M _ {k} \left(\left\{\left(h _ {v} ^ {(k - 1)}, h _ {u} ^ {(k - 1)}, e _ {u v}\right) \mid u \in N (v) \right\}\right)\right), \tag {7}
$$
where  $N(v)$  is the set of neighbors of  $v$  and  $h_v^{(0)} = x_v$ . The graph embedding  $h_G$  is obtained via:
$$
h _ {G} = R \left(\left\{h _ {v} ^ {(K)} \mid v \in V \right\}\right), \tag {8}
$$
where  $K$  is the number of layers and  $R$  is the readout function. The overall graph embedding  $h_G$  is used for graph-level prediction, as in Section 4.1. In our work as well as some previous works, Graph Isomorphism Network (GIN) with edge encoding [15, 23, 42] is often used as the underlying architectures of learning models. In particular:
$$
h _ {v} ^ {(k)} = \operatorname {R e L U} \left(\mathbf {M L P} ^ {(k)} \left(\sum_ {u \in N (v) \cup \{v \}} h _ {v} ^ {(k - 1)} + \sum_ {e \in \{(u, v) | v \in N (v) \cup \{v \} \}} h _ {e} ^ {(k - 1)}\right)\right), \tag {9}
$$
where  $h_e^{(k)}$  is the learnable edge embedding at the  $k$ -th layer. The mean readout is used to obtain the graph embedding:
$$
h _ {G} = \operatorname {M E A N} \left(\left\{h _ {v} ^ {(K)} \mid v \in V \right\}\right) \tag {10}
$$
A.2 Model and Training Configurations
The configurations of  $\mathrm{GNN}_M$  and  $\mathrm{GNN}_F$  are provided in Table 5. The pretraining and finetuning setups are provided in Table 6. In general, our parameters are comparable to those of previous works with which we compare [15, 23, 42]. Note that the number of node features and edge features is much fewer than that in [8, 50, 51]. On downstream tasks, we only finetune the dropout rate, selected from  $\{0.0, 0.5\}$ .
Table 5: Model Configuration
Methods	Parameters	Values
GNNM	Convolution type	GIN
Number of atom features	2
Number of atom features for long-range benchmarks	9
Number of edge features	2
Number of edge features for long-range benchmarks	3
Dimension of hidden embeddings	300
Aggregation	SUM
Number of layers	5
Readout	MEAN
GNNF	Convolution type	GIN
Number of fragment features	1
Dimension of hidden embeddings	300
Aggregation	SUM
Number of layers	2
Number of layers for long-range benchmarks	5
Readout	MEAN
A.3 Combination of Predictive and Contrastive Pretraining
We combine the contrastive pretraining and predictive pretraining proposed in Section 3 by optimizing the joint objective function:
$$
\mathcal {L} = \alpha \mathcal {L} _ {P} + (1 - \alpha) \mathcal {L} _ {C}, \tag {11}
$$
Table 6: Training Configuration
Phases	Hyperparameters	Values
Pretraining	Number of epochs	100
Batch size	256
Optimizer	AdamW
Weight decay	0.01
Learning rate	0.001
Learning rate decay	Reduce on plateau
Learning rate decay factor	0.1
Learning rate decay patience	5 epochs
Finetuning, chemical	Number of epochs	100
Batch size	256
Dropout	{0.0,0.5}
Optimizer	Adam
Weight decay	0.0
Learning rate	0.001
Learning rate decay	Step decay
Number of steps before decay	30
Learning rate decay	0.3
Finetuning, long-range	Number of epochs	100
Batch size	128
Dropout	{0.0,0.5}
Optimizer	Adam
Weight decay	0.0
Learning rate	0.001
Learning rate decay	Reduce on plateau
Learning rate decay factor	0.5
Learning rate decay patience	20 epochs
where  $\mathcal{L}_P$  is the predictive objective,  $\mathcal{L}_C$  is the contrastive objective, and  $\alpha$  is the weight hyperparameter. We select  $\alpha$  from  $\{0.1, 0.2, 0.3, 0.4, 0.5\}$ , using finetuning results as the performance indicator. Table 7 shows the performances using models trained with different values of  $\alpha$ .
Table 7: Downstream performances with models pretrained using different values of the weight hyperparameter  $\alpha$ . Subscripts  $C, P,$  and  $F$  indicate contrastive pretraining, predictive pretraining, and inclusion of fragment encoders in downstream prediction, respectively. Best results for either  $CP$  or  $CPF$  models are shown in red. The rightmost column shows the average performance ranking for either  $CP$  or  $CPF$  models (lower value means better ranking).
Models	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	Avg. Rank
CP, α = 0.1	71.3 ± 0.8	75.8 ± 0.3	63.9 ± 0.4	61.3 ± 0.4	63.2 ± 1.0	76.2 ± 2.0	76.1 ± 1.4	82.5 ± 1.9	2.88
CP, α = 0.2	70.8 ± 1.9	75.3 ± 0.3	64.1 ± 0.3	62.1 ± 0.5	64.7 ± 7.9	76.5 ± 1.2	76.0 ± 1.1	83.2 ± 1.4	2.63
CP, α = 0.3	71.3 ± 1.7	75.5 ± 0.5	64.7 ± 0.2	61.3 ± 0.6	73.7 ± 3.9	76.6 ± 1.8	76.3 ± 1.0	81.3 ± 2.2	2.13
CP, α = 0.4	70.1 ± 2.3	76.1 ± 0.4	64.0 ± 0.3	61.2 ± 0.6	60.1 ± 6.7	75.6 ± 2.0	75.7 ± 1.0	82.1 ± 2.2	3.81
CP, α = 0.5	70.6 ± 1.1	76.1 ± 0.3	64.6 ± 0.5	59.2 ± 0.9	65.7 ± 4.2	74.1 ± 2.4	75.6 ± 1.1	81.9 ± 1.3	3.56
CPF, α = 0.1	72.0 ± 1.7	74.0 ± 0.7	63.9 ± 0.9	63.6 ± 1.2	84.7 ± 8.8	75.4 ± 1.9	78.0 ± 1.5	80.5 ± 1.8	1.63
CPF, α = 0.2	69.4 ± 1.3	73.7 ± 0.8	64.5 ± 0.4	63.3 ± 0.9	76.9 ± 3.2	73.0 ± 4.2	77.6 ± 1.3	79.9 ± 1.0	4.06
CPF, α = 0.3	70.0 ± 1.6	73.9 ± 0.5	65.0 ± 0.5	62.6 ± 0.9	83.0 ± 5.6	74.2 ± 2.7	77.6 ± 1.2	78.2 ± 1.0	3.25
CPF, α = 0.4	71.7 ± 0.9	73.3 ± 0.8	64.6 ± 0.5	63.5 ± 0.8	83.0 ± 6.9	73.3 ± 3.0	77.7 ± 2.1	78.8 ± 2.8	3.06
CPF, α = 0.5	70.3 ± 0.9	74.2 ± 0.7	64.8 ± 0.5	62.7 ± 0.9	79.3 ± 3.1	73.4 ± 2.8	76.7 ± 1.3	80.1 ± 1.2	3.00