Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules
Authors: Zhiyuan Liu
Yaorui Shi
An Zhang
Enzhi Zhang
Kenji Kawaguchi
Xiang Wang
Tat-Seng Chua
Venue: NeurIPS 2023 poster (2023)
Keywords: Molecular Representation Learning
Masked Graph Modeling
Graph Tokenizer

Abstract: Masked graph modeling excels in the self-supervised representation learning of molecular graphs. Scrutinizing previous studies, we can reveal a common scheme consisting of three key components: (1) graph tokenizer, which breaks a molecular graph into smaller fragments (\ie subgraphs) and converts them into tokens; (2) graph masking, which corrupts the graph with masks; (3) graph autoencoder, which first applies an encoder on the masked graph to generate the representations, and then employs a decoder on the representations to recover the tokens of the original graph. However, the previous MGM studies focus extensively on graph masking and encoder, while there is limited understanding of tokenizer and decoder. To bridge the gap, we first summarize popular molecule tokenizers at the granularity of node, edge, motif, and Graph Neural Networks (GNNs), and then examine their roles as the MGM's reconstruction targets. Further, we explore the potential of adopting an expressive decoder in MGM. Our results show that a subgraph-level tokenizer and a sufficiently expressive decoder with remask decoding have a \yuan{large impact on the encoder's representation learning}. Finally, we propose a novel MGM method SimSGT, featuring a Simple GNN-based Tokenizer (SGT) and an effective decoding strategy. We empirically validate that our method outperforms the existing molecule self-supervised learning methods. Our codes and checkpoints are available at https://github.com/syr-cn/SimSGT.

Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules
Zhiyuan Liu
Yaorui Shi
An Zhang
Enzhi Zhang
Kenji Kawaguchit Xiang Wang!- Tat-Seng Chua
National University of Singapore, UUniversity of Science and Technology of China Hokkaido University
{acharkq, shiyaorui xiangwang12233 2gggggail1222m11.111 com, anzhang@u nus. edu enzhi. .zhang.aa....................... n6delms hokudai ac. .c. {kenji, chuats) nus edu
Abstract
Masked graph modeling excels in the self-supervised representation learning of molecular graphs. Scrutinizing previous studies, consisting of three key components: (1) graph tokenizer, which breaks a molecular graph into smaller fragments (i.e., subgraphs) and converts them into tokens; (2) graph masking, which corrupts the graph with masks; (3) graph autoencoder, which first applies encoder the masked graph to generate the representations, and then employs a decoder on the representations to recover the tokens of the original graph. However, the previous MGM studies focus extensively on graph masking and encoder, while there is limited understanding of tokenizer and decoder. To bridge the gap, we first summarize popular molecule tokenizers at the granularity of node, edge, motif, and Graph Neural Networks (GNNs), and then examine their roles as the MGM's reconstruction targets. Further, we explore the potential of adopting an expressive decoder in MGM. Our results show that a subgraph- tokenizer and a sufficiently expressive decoder with remask decoding have a large impact on the encoder's representation learning. Finally, we propose a novel MGM method SimSGT, featuring Simple GNN-based Tokenizer (SGT) and an effective decoding strategy. We empirically validate that our method outperforms the existing molecule self-supervised learning methods. Our codes and checkpoints are available at https: // gi thub. com/ str///yrroar cn/ //mmSST
reveal common scheme
Introduction
Molecular representation learning (MRL) [1, 2, 3] is a critical research area with numerous vital downstream applications, such as molecular property prediction [4], drug discovery [5, 6], and retrosynthesis [7, 8]. Given that molecules can be represented as graphs, graph self-supervised learning (SSL) natural fit for this problem. Among the various graph SSL techniques, Masked Graph Modeling (MGM) has recently garnered significant interest [9, 10. 11].
In this paper, we study MRL through MGM, aiming to pretrain a molecule encoder for subsequent fine- tuning in downstream applications. After looking at the masked modeling methods in graph [12, 9, 10], language [13, 14], and computer vision [15, 16], summarize that MGM relies three key components graph tokenizer, graph masking, and graph autoencoder, as Figure shows:
Graph tokenizer. Given graph g, the graph tokenizer employs graph fragmentation func- tion [l, 17, 18, 2] to break g into smaller subgraphs, such as nodes and motifs. Then, these
"Corresponding author. Xiang Wang is also affiliated with Institute of Artificial Intelligence, Institute of Dataspace, Hefei Comprehensive National Science Center.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Fragments
Tokens
Benzene
ring
H
Mayring
Carbonyi
0
group
Hydroxyl group
Graph masking
Tokemizer
N
Reconstruction targets
M
Inputs
H
Graph Encoder
Graph Decoder
M
Figure 2: Example of subgraph-
Autoencoder
level patterns in molecule. SMILES: CC(=O)ecl
Figure 1: The pipeline of Masked Graph Modeling.
fragments mapped into fixed-length tokens to serve the targets being reconstructed later. Clearly, the granularity of graph tokens determines the abstraction level of representations in masked modeling [19, 16]. This is especially relevant for molecules, whose properties are largely determined by patterns at the granularity of subgraphs [20]. For example, the molecule shown in Figure 2 contains benzene ring subgraph. Benzene ring confers the molecule aromaticity, making it more stable than saturated compounds that only have single bonds [21]. Therefore, applying graph tokenizers that generate subgraph-level tokens might improve the downstream performances.
Graph masking. Before feeding into the autoencoder, is corrupted by adding random noise, typically through randomly masking nodes or dropping edges [12, 11]. Graph masking crucial to prevent the autocncoder from merely copying the input, and guide the autoencoder to learn the relationships between co-occurring graph patterns.
Graph autoencoder. Graph autoencoder consists of graph encoder and a graph decoder [12, 9]. The graph encoder generates the corrupted graph's hidden representations, based on which the graph decoder attempts to recover the corrupted information. The encoder and the decoder are jointly optimized by minimizing the distance between the decoder's outputs and the reconstruction targets, i.e., the graph tokens induced by the graph tokenizer. Given the complex subgraph- tokens as targets, an effective reconstruction might demand a sufficiently expressive graph decoder.
Although all three components mentioned above are crucial, previous MGM studies primarily focus on graph masking [12, 11, 22, 23] and graph encoder [2, 24, 25, 26]. with less emphasis on the tokenizers and decoders. For example, while there exist extensive motif-bat base- fragmentation functions for MRL [1, 17, 18, 2], they have been overlooked as tokenizers for MGM. Moreover, many previous works [12, 10, 11, 22, 23, 2, 25] employ a linear or MLP decoder for graph reconstruction, leaving more expressive decoders largely unexplored.
In this work, we first summarize the various fragmentation functions as graph tokenizers, at the granularity of nodes, edges, motifs, and Graph Neural Networks (GNNs). Given this summary, we systematically evaluate their empirical performances for MGM. Our analysis shows that reconstruct- ing subgraph-level tokens in MGM can improve over the node tokens. Moreover, we find that sufficiently expressive decoder combined with remask decoding [9] could improve the encoder's rep- resentation quality. Notably, remask "decouples" the encoder and decoder, redirecting the encoder's focus away from molecule reconstruction and more towards MRL, leading to better downstream performances. In summary, we reveal that incorporating a subgraph- tokenizer and a sufficiently expressive decoder with remask decoding gives rise to improved MGM performance.
Based on the findings above, we propose novel pretraining framework Masked Graph Modeling with Simple GNN-based Tokenizer (SimSGT). SimSGT employs a Simple GNN-based Tokenizer (SGT) that removes the nonlinear update function in each GNN layer. Surprisingly, we show that single-layer SGT demonstrates competitive or better performances compared to other pretrained GNN-based and chemistry-inspiree tokenizers. SimSGT adopts the GraphTrans [27] architecture for its encoder and smaller GraphTrans for its decoder, in order to provide sufficient capacity for both tasks of MRL and molecule reconstruction. Furthermore, we propose remask-v2 to decouple the encoder and decoder of the GraphTrans architecture. Finally, SimSGT is validated on downstream molecular property prediction and drug-target affinity tasks [28, 29], surpassing the leading graph SSL methods (e.g., GraphMAE [9] and Molc-BERT [10]).

---

2 Preliminary
In this section, we begin with the introduction of MGM. Then, we provide a categorization of existing graph tokenizers. Finally, we discuss the architecture of graph autoencoders for MGM.
Notations. Let & denote the space of graphs. A molecule can be represented as graph g = (V,E) G. where V is the set of nodes and & is the set of edges. Each node E V is associated with node feature 2; A and each edge (i.j) E & is associated with an edge feature eij E R The graph g's structure can also represented by its adjacency matrix A E {0, 1)MMX)M) (VI, such that Ai = (i,j) E & and Aij = O otherwise.
2.1 Preliminary: Masked Graph Modeling
Here we illustrate MGM's three key steps: graph tokenizer, graph masking, and graph autoencoder.
Graph tokenizer. Given a graph we leverage graph tokenizer tok(g) = {y' = m(() E le e fg)] to generate its graph tokens as the reconstruction targets. The tokenizer tok() (.) is composed of a fragmentation function f that breaks into a set of subgraphs (g) = it = (V: & S g). and mapping function ml()) E IRd that transforms the subgraphs into fixed-length vectors. In this work, allow (g) to include overlapped subgraphs enlarge the scope of graph tokenizers.
Graph masking. Further, we add noises to g by random node masking. Here we do not use edge dropping because Hou et al. [9] empirically show that edge dropping easily leads to performance drop downstream tasks. Specifically, node masking samples a random subset of nodes Vm S V and replaces their features with special token my Rdo We denote the masked node feature by Ny:
mo:
Vi Vm
X;.
otherwise
Graph autoencoder. The corrupted graph ÿ is then fed into graph autoencoder for graph re- construction. We defer the details the graph autoencoder's architecture to Section 2.3. Let {z.li E / be the node-wise outputs of the graph autoencoder. We obtain subgraph l's prediction y, = MEAN([Z,lt E V.)) by mean pooling the representations its nodes, not especially noted. The graph autoencoder is trained by minimizing the distance between the predictions {ýelt f()
and the targets tg)(g) = {y.lt E f(g)}. The reconstruction loss is accumulated on tokens that include corrupted information {ttt fg)., V, n Vm *0}:
Lg
1
Σ
((y...).
(2)
[f(g]
where ((.)) the loss function, dependent on the type yt. We use mean square error [15] for E(..-) when is continuous vector, and use cross-entrops [16] for f(.,-) when yr is discrete value.
2.2 Revisiting Molecule Tokenizers
Scrutinizing the current MRL methods, we summarize the molecule tokenizers into four distinct categories, as Table shows. A detailed description of the first three categories systematically provided here, while the Simple GNN-based Tokenizer is introduced in Section 3.
Table 1: Summary of graph tokenizers.
Tokenizers
Subgraph types
Tokens
Potential limitations
Node, edge
Nodes and edges
Features of nodes and edges
Low-level feature
Motif
FGs, eycles, etc.
Motif types
Rely expert knowledge Extra pretraining for tokenizer
Pretrained GNN
Rooted subtrees
Frozen GNN representations
Simple GNN
Rooted subtrees
Frozen GNN representations
Node, edge tokenizer [12, 11j. A graph's 's nodes and edges can be used as graph tokens directly:
tok (g) = {y: = az/i E V},
tokedge = {y., = (i,)( E E}.
Figure 3a illustrates the use of atomic numbers of nodes and bond types of edges as graph tokens in molecule, These have been widely applied in previous research [12, 11, 25, 24], largely due their
3

---

vecabulary
Periodic
types
Functional groups,
Cycles BRICS,
Fragment
Gather
Remaining.
nodes
One-bot mup
One-hot mapp
Cyclo
(a) Node, edge lokenizer
(b) Morif-bnsed tokenizer
GNNs
Pretralning
Molecule Dataset
Pretrained GNNs,
Boot modic
140p seigthom
neighhors
Z-hop mesuug--poing
Mexagg-passing inducal anhtrees
(c) Pretrained GNN-based tokenizer
Figure 3: Examples for the first three types of graph tokenizers and their induced subgraphs. (b) A motif-based tokenizer that applies the fragmentation functions of cycles and the remaining nodes. (c) A two-layer GIN-based tokenizer that extracts 2-hop rooted subtrees for every node in the graph.
simplicity. However, atomic numbers and bond types are low-level features. Reconstructing them may be suboptimal for downstream tasks that require high-level understanding of graph semantics.
Motif-based tokenizer. Motifs are statistically significant subgraph patterns in graph structures. For molecules, functional groups (FGs) motifs that manually curated by experts based on the FGs' biochemical characteristics [30, 31]. For example, molecules that contain benzene rings exhibit the property of aromaticity. Considering that the manually curated FGs are limited and cannot fully cover all molecules, previous works [32, 1, 18] employ ceamistry-inspired fragmentation functions for motif discovery. Here summarize the commonly used fragmentation functions:
FGs [18, 2]. FG is molecule's subgraph that exhibits consistent chemical behaviors across various compounds. In chemistry toolkits [30, 31], the substructural patterns of FGs are described
|Sa be a of SMARTS patterns for FGs, and let by the SMARTS language [33]. Let S = {sa F=
(g) be the function returning the set of s FGs in FG-based fragmentation works as:
f So) UsES,Pa(g).
(4)
Cycles [32, 1, 18]. Cycles in molecules are often extracted as motifs due to their potential chemical significance. Figure 3b depicts the process of fragmenting five-node cycle moif. two cycles overlapped more than two atoms, they can be merged because they constitute a bridged compound [34]. Let Cn represent eycle of nodes. They can be written as:
Scycte(g)
[1] Cj.. Egg,
feycle-merge (g) (t, Utylt,,tt feycw j. j.lt 2),
(6)
BRICS [35, 1J. BRICS fragments molecule at the potential cleavage sites, where chemical bonds can be broken under certain environmental conditions or catalysts. The key step of BRICS is to identify molecule g's potential cleavage sites, denoted by ((g). This is achieved by finding bonds with both sides matching one of the pre-defined "FG-like" substructural patterns S in BRICS:
((g) = U(E (&, U UEEUUE-Ci 11,
E (a((g,S)),
where - = g(V \ Vij denotes deleting g's nodes in and the corresponding incident edges [36]; E\ (EE contains the bond that connects and g g-- Next, g fragmented into the maximum subgraphs that contain no bonds in the cleavage sites (g)):
fgRICS(9) (g) n n 0, (tR n [t].t S g
(8)

---

Note that, the original BRICS includes more rules, such as constraint on t and t's combinations. We present only the key step here for simplicity.
Remaining nodes and edges [32, 18]: Given another fragmentation function fo. the nodes and edges that are not included in any of fo 's outputs are treated as individual subgraphs. This improves fo is coverage on the original graph. Figure 3b shows example of fragmenting remaining nodes after feyele: nodes that are not in any cycles are treated as individual subgraphs.
To obtain more fine-graine greeggriiedd subgraphs, previous works [1. 32, 18] usually combine several fragmen- tation functions together by unions (e.g., (i) U (z)g)) or compositions (e.g.. (/(())( E (g)) Let /mxit be the final fragmentation function after combination. We break every molecule in the dataset by fmour and gather motif vocabulary M, which filters out infrequent motifs by a threshold.
Then, given a new molecule g', we can generate its tokens by one-hot encoding its motifs f (g')
tok (y: = one-hot(t, MM) E fmoar(g'))
(9)
Pretrained GNN-based tokenizer [10]. Pretrained GNNs can serve as graph tokenizers. Take a
k-layer Graph Isormophism Network (GIN) [37] as an example. Its node embedding summarizes the structural information of the node's k-hop rooted subtree, making it a subgraph-level graph token (Figure 3c). A GIN performs the fragmentation and the mapping simultaneously. It can be written as:
tokGIN(9) {y: SG (1) )j V}. h (k) COMBINE (k) (n. (h11) AGGREGATE (4) ((M)) (()))),
(10)
(11)
where AGGREGATE(() collects information from node i's neighbors, based on which COMBINE(.)
updates i's representation. SG(() denotes stop-gradient, which stops the gradient flow to the tokemizer during MGM pretraining. In addition to GINs, other GNNs can also serve as tokenizers. To obtain meaningful graph tokens, we pretrain a GNN before employing it as a tokenizer [10]. Once pretrained, this GNN is frozen and employed for the subsequent MGM pretraining. In Section 4, we evaluate the prevalent graph pretraining strategies for tokenizers. Given that GNN-based tokenizers provide node-wise tokens, we directly minimize the distance between the graph tokens and the autoencoder's
outputs {z, } 1=1 of the masked nodes Vma. i.e., Lo = P Σ EiEV.. E(ý: = Zi V.).
2.3 Revisiting Graph Autoencoders
Background. Graph autoencoder consists of a graph encoder and a graph decoder. We pretrain them with the objective of graph reconstruction. Once well pre- trained, the encoder saved for downstream tasks. MGM works [12, 2] usually adopt expressive graph encoders like GINEs [12] and Graph Transformers [2]. However, the ex- ploration on expressive decoders has been limited. Many previous works [12, 10, 11, 22, 2, 25] apply a linear or an MLP decoder, similar to BERT's design [13].
Table 2: The compared GNN architec- tures for encoders and decoders.
Model
GINE, dim 300 Transformer, dim 128
Linear
GINE
layer
GINE
layer
GTS
layer
layer
GTS-Small
layer
1 Jayer
GTS-Tiny
layer
1 layer
However, recent studies [15, 38] have revealed the disparity between representation learning and reconstruction tasks. They show that sufficiently expressive decoder could improve the encoder's representation quality. Delving into these studies, we identify two key elements to improve the representation quality: a sufficiently expressive decoder and remask decoding [9].
Sufficiently expressive decoder. Following [15, 38], we devise smaller versions of the encoder's architecture as decoders. We adopt the GINE [12] and GraphTrans [27] (denoted as GTS) as encoders. GTS stacks transformer layers top of the GINE layers to improve the ability of modeling global interactions. Table 2 summarizes their different versions that we compare in this work.
Remask decoding [9]. Remask controls the focus of the encoder and the decoder. Let (hali E V) be the encoder's 's node-wise hidden representations for the masked graph. Remask decoding masks the hidden representations of the masked nodes Vm again by special token m E Ed before feeding them into the decoder. Formally, the remasked hidden representation h; is as follows:
h, = (mi:
Vie V.
(12)
otherwise
5

---

Simple GNN- bused Tokenizer
Stop gradicat
53
y2
Random masking
Weight shuring
Minimize
distance
Trandoomeee
V2
GINES
y:
y:
5%
Encoder
Decoder
Figure 4: Overview of the SimSGT's framework.
Remask constrains the encoder's ability on predicting the corrupted information by removing the encoder's representation the masked part. The encoder is enforced to generate effective rep- resentations for the unmasked part, to provide signals for the decoder for graph reconstruction.
3 Methodology
In this section, we present our method Masked Graph Modeling with Simple GNN-based Tokenizer (SimSGT) (Figure 4), Specifically, applies the GTS [27] architecture for both its encoder and decoder. SimSGT features Simple GNN-based Tokenizer (SGT), and employs new remask strategy to decouple the encoder and decoder of the GTS architecture.
Simple GNN-based Tokenizer. SGT simplifies existing aggregation-bass based GNNs [37] by removing the nonlinear update functions in GNN layers. Is inspired by studies showing that carefully designed graph operators can generate effective node representations [39, 40). Formally, a k-layer SGT is:
toksor (g) {y, ( H ( V}.
(13)
H (0) Embedding(2s) x.,) Rd,
Vi E V,
(14)
H (1) = w(A) - e Plli
151sk
(15)
H (t) BatchNorm(Ei (H),,
(16)
where Embedding(-) is a linear layer that uses the weights of the encoder's node embedding function; ( is the i-th row of (();; Batchhorm(.)) is a standard Batch Normalization layer without the trainable scaling and shifting parameters [41]; and w( () is the graph operator that represents the original GNN's aggregation function. For example, GIN has w A + + e)I and GCN has
w(A) D-11/ 1/2 AD 1/2, where À A + and D is the degree matrix of A.
Note that, SGT does not have trainable weights, allowing its deployment without pretraining. Its tokenization ability relies the graph operator w (A) that summarizes each node's neighbor informa- tion. Additionally, we concatenate the outputs of every SGT layer to include multi-scale information. We show in experiments that SGT transforms the original GNN into effective tokenizer.
Graph autoencoder. SimSGT employs the GTS architecture as the encoder, and a smaller version of GTS (i.e., GTS-Small in Table 2) the decoder. This architecture follows the asymmetric encoder-decoder design in previous works [15, 38]. Further, we propose a new remask strategy, named remask-v2, to decouple SimSGT's encoder layers and decoder layers.
Remask-v2. Remask-v2 v2 constrains the encoder's ability on predicting the corrupted information by dropping the masked nodes' Vm representations before the Transformer layers (Figure 4). After the Transformer layers, we pad special mask tokens m, E I to make up the previously dropped nodes' hidden representations. Compared to the original remask, remask-v2 additionally prevents the Transformer layers from processing the masked nodes. It thereby avoids the gap of processing masked nodes in pretraining but not in fine-tuning [15].

---

Table Average ROC-AUC (%) scores on the eight classification datasets MoleculeNet.
(a) Ablating the decoder. The tok-
(b) Ablating the tokenizer. Darker blue indicates higher performance.
enizer is single-layer SGT of GIN.
The encoder is GTS and decoder GTS-Small, Remask-v2 is used.
Encoder Decoder
Remask Avg.
GNN tokenizer 's depth
GINE
Linear
73.2
Tokenizer
2
3
GINE
GINE-Small
73.0
Node
74.7
GINE
GINE-Small
vl
74.4
Motif, MGSSL
75.2
Motif, RelMole
74.4
GTS
Linear
74.1
Pretrain, GIN, GraphCL
75.1
74.5
74.2
74.0
74.6
GTS
GTS-Smal
74.1
Pretrain, GIN, VQ-VAE
75.1
74.9
74.4
75.6
75.1
GTS
GTS-Small
75.2
Pretrain, GIN, GraphMAE
75.1
74.9
74.9
75.4
75.2
GTS
GTS-Small
v2
75.8
SGT, GIN
75.8
75.5
75.6
75.3
74.9
0:16
Decoder: Linear
100
GTS-Smal
76.00
1-ayer 3-layer
5-layer
Decoder: GTS-Smell
Decoder: GTS-Smal with remask-v1
Avg. ROC-AUC/SS (9
75.75
0.14
Decoder: GTS-Smell with remeek-v1
Decoder: GT S-S-m with remask-v2
Decoder: GTS-Small with remaskev2
Baseline, random intiaized encoder
7544
MSE
0.12
Linear prosing
76.3
75.1
0.10
80
75.00
74.9
74.8
0.08
Pretrain epochs
80
40
epochs
60
74.75
GIN
GCN
GraphSAGE
(a) MSE loss for pretraining
(b) Probe encoder for masked atoms.
Figure 6: MGM with SGT of different GNNs. GTS encoder and GTS-Small decoder.
Figure 5: GTS encoder. Tokenizer is a single-layer SGT of GIN.
4
Rethinking Masked Graph Modeling for Molecules
Experimental setting. In this section, we perform experiments to assess the roles of tokenizer and decoder MGM for molecules. Our experiments follow the transfer learning setting in [12, 9J. We pretrain MGM models on 2 million molecules from ZINC15 [42], and evalute the pretrained models on eight classification datasets MoleculeNet [28]: BBBP, Tox21, ToxCast, Sider, ClinTox, MUV, HIV, and Bace. These downstream datasets divided into train/valid/tes sets by scaffold split to provide an -uito--distt evaluation setting. We report the mean performances and standard deviations the downstream datasets across ten random seeds. Throughout the experiments, random node masking ratio 0.35. More detailed experimental settings can be found in Appendix D.
4.1 Rethinking Decoder
We investigate the impact of an expressive decoder on MGM's performance. single-layer GIN- based SGT tokenizer is utilized in these experiments. Table 3a summarizes the results.
Finding 1. A sufficiently expressive decoder with remask decoding is crucial for MGM. Table 3a shows that using sufficiently expressive decoder with remask decoding significantly improve downstream performance. This can be attributed to the disparity between molecule reconstruction and MRL tasks: the last several layers in the autoencoder will be specialized molecule reconstruction, while losing some representation ability [15, 38J. When using a linear decoder, the last several layers in the encoder will be reconstruction-speciiee specialized, which yield suboptimal results in fine-tuning
sufficiently expressive decoder is essential to account for the reconstruction specialization. As shown by Figure 5a, employing an expressive decoder results in significantly lower reconstruction loss than a linear decoder. However, increasing the decoder's 's expressiveness without remask decoding cannot improve the downstream performance (Table 3a). This leads our exploration remask.
Finding 2. Remask constrains the encoder's efforts on graph reconstruction for effective MRL. Figure 5b shows that the linear probing accuracy of the masked atom types on an encoder pretrained with remask is significantly lower than the accuracy on encoder pretrained without remask. This shows that remask makes the encoder spend less effort on predicting the corrupted information. Moreover, remask only slightly sacrifices the autoencoder's reconstruction ability, when paired with the GTS- Small decoder (Figure 5a). Combining the
Table 4: Testing decoder's size. Remask-v2 is used.
Encoder Decoder Avg.
GTS
GTS-Tiny 74.7
GTS
GTS-Small 75.8
GTS
GTS
74.9
7

---

observation that remask improves downstream performances (Table 3a), it indicates that remask constrains the encoder's efforts on graph reconstruction, allowing focus on MRL.
In addition, remask-v2 outperforms remask-v1 with GTS architecture. This improvement can be attributed remask-v2's ability to prevent the Transformer layers from processing the masked nodes, avoiding the gap of using masked nodes in pretraining but not in fine-tuning, Finally, we test decoder's sizes in Table 4. When the encoder GTS, GTS-Smal decoder provides the best performance.
4.2 Rethinking Tokenizer
We investigate the impact of tokenizers on MGM's performance. the following experiments, the graph autoencoder employs the GTS encoder and the GTS-Small decoder with remask-v2, given their superior performances in the previous section. The results are summarized in Table 3b.
Compared tokenizers. We use the node tokenizer as the baseline. For motif-based tokenizers, we employ the leading fragmentation methods: MGSSL [1] and RelMole [18]. For GNN-based tokenizers, we compare the prevalent pretraining strategies GraphCL [4], GraphMAE [9], and VQ-VAE [10, 43] - to pretrain tokenizers on the 2 million molecules from ZINC15.
Finding 3. Reconstructing subgraph-level tokens can give rise to MRL. Table 3b shows that, given the appropriate setup, incorporating motif-hased tokenizers GNN-based tokenizers in MGM can provide better downstream performances than the node tokenizer. This observation underscores the importance of applying a subgraph-level tokenizer in MGM.
Finding 4. Single-layer SGT outperforms or matches other tokenizers. Table 3b shows that a single-layer SGT, applied to GIN, delivers comparable performance to four-layer GIN pretrained by VQ-VAE, and surpasses other tokenizers. Further, Figure 6 shows that SGT can transform GCN and GraphSAGE into competitive tokenizers. We attribute SGTs' good performances to the effectiveness of their graph operators in extracting structural information. It has been shown that linear graph operators can effectively summarize structural patterns for node classification [39, 40].
Finding 5. GNN-based tokenizers have achieved higher perfor- mances than motif- tokenizers. We hypothesize that this due to GNN's s ability of summarizing structural patterns. When using GNN's s representations as reconstruction targets, the distance between targets reflects the similarity between their underlying sub- graphs- a nuanced relationship that the one-hot encoding of motifs fails to capture. We leave the potential incorporation of GNNs into motif- tokenizers for future works.
ROC-AUC (96)
.90
85
Node tokenizer Single-layer lagee--laerrra SGT
25
75
100
Protrain apochs
Finally, we show that although GNN-based tokenizers are agnostic to chemistry knowledge, incorporating them in MGM can improve the recognition of FGs. In Figure 7, we use a linear classifier probe the encoder's mean pooling output to predict the FGs within the molecule. We use RDkit [30] to extract 85 types of FG. Details are in Appendix D. It can be seen that, incorporating a single-layer SGT in MGM improves the encoder's ability to identify FGs in comparison to the node tokenizer.
Figure 7: Linear probing en- coder's output for FGs.
5 Comparison to the State-of-the-Art Methods
In this section, we compare SimSGT to the leading molecule SSL methods for molecular property prediction and a broader range of downstream tasks. For fair comparison, we report the performances of SimSGT and its variant that uses the GINE architecture. This variant employs a GINE as encoder and GINE-Smal
decoder (Table 2), and implements remask-vl decoding.
Molecular property prediction. The molecular property prediction experiment follows the same transfer learning setting as Section 3. Table 5 presents the results. It can be observed that SimSGT outperforms all the baselines in average performances with both the GTS and GINE architectures. Notably, SimSGT with GTS establishes a new totte-of-the-arr of 75.8% ROC-AUC. It exceeds the second method by an absolute 1.8% points in average performance and achieves the best performance in five out of the eight molecular property prediction datasets. SimSGT with GINE outperforms the baselines by 0.4% points in average performance. These improvements demonstrate the effectiveness of our proposed tokenizer and decoder to improve MGM performance for molecules.
8

---

Table 5: Transfer learning ROC-AUC (%) scores on eight MoleculeNet datasets. Bold denotes the best performance. Baselines are reproduced using source codes. All methods use the GTS encoder.
Dataset
BBBP
Tox21
ToxCast
Sider
Clin'ox
MUV
HIV
Bace
Avg.
No Pre-train
68.721.3
75.5 64.5a08
58.0s1.
71.2+4.9 69.7.1. 74.2x22 77.2.1.
69.9
Infomax
69.2a0.7
74.620.5
61.8a08
60.1
74.8
74,8
75.0a1. 76.3a18
70.8
ContextPred
68.521.1
75.3a06
63.2a0s
61.3208
74.9
70.9
76.8.t, 80.221.8
71.4
GraphCL
70.4
73.8a10
63.1a04
60,4213
77.8
73.8220
75.6109 78.321.1
71.6
JOAO
70.8
75.2+1.5
63.8+nx
61.0sa9
78.7+3.
75.7+26
77.0m12 77.5+1.2
72.5
ADGCL
67.9+1.
73.0+t.
63.2404
60.2+1.0
78.8+16
75.6+25
75.5+13 73.4+16
71.0
BGRL
72.7=0.
75.8+1.
65.1
60.4+1.2
77.6x4.1
76.7+28
77.1+12 74.7+26
72.5
GraphLOG
67.6+16
76.0+08
63.6s06
59,8+2.1
79.
72.8+18
72.5+16 83.6+20
71.9
RGCL
71.4
75.7=0.4
63.9403
60.920.
80.0+1.
75,9+1.
77.8406 79.9.1.
73.2
S2GAE
67.6-20
69.6s1.
58.7+08
55,4
59.6+1.1
60.1
68.0s3.7
68.622.1
63.5
Mole-BERT
70.8 76.6
63.7a0s
59.2+1.1
77.2a1.1 7
76.5
82.8s14
73.0
GraphMAE
71.7=08
76.0
65.8
60.0s1.0
79.2x2.
76.3x19
75.9t18 81.7+16
73.3
GraphMAE2
71.6+16 75.9
65.6407
59.6806
78. 78.8 78.5
76. 142.2 81.0
73.4
SimSGT 72.2+0. 76.8s09 65.9.as 61.7sa8 85.7+1. 81.4+14 78.0s1.9 84.3+a.6
75.8
Table 6: Transfer learning performance for molecular property prediction (regression) and drug target affinity (regression). Bold indicates the best performance and underline indicates the second best. *
denotes reproduced result using released codes. Other baseline results are borrowed from [ICLR' 22].
Molecular Property Prediction (RMSE 4)
Drug -arget Affinity (MSE L)
ESOL
Lipo
Malaria
CEP
Avg.
Davis
KIBA
Avg.
No Pre-train
1.178
0.744
1.127
1.2542amm
1.076
0.286
0.206sne4
0.246
ContextPred
1. 1.196
0.702an.220
1.101
1.243aams
1.061
0.279a000:
0. 0.198a0004
0,238
Aurkask
1.112
0.730
1.119
1.256xame
1.054
0.291=0omm
0.203
0,248
JOAO
1.120sn019
0.708
1.145
1.293
1.066
0.28 20004
0. 19
0.239
GraphMVP
1.064.0.0s
0.691
1000s
1.106s001a
1.228
1.022
0.274anon:
0. 17
0.225
Mole-BERT
1.192aa.00s
0.706
1.1
1.078
1.024
0.277=0004
0.210s
0.243
SimSGT, GINE
1.039=0.022
1.090samm 0.670
1.060=amit
0.965
0.263
0. 04
0.204
SimSGT, GTS
0.917
0.695an.12
1.078
1.036
0.932
0.251
0. 2.153
0.202
Broader range of downstream tasks. We report the transfer learning performances regressive molccular property prediction and drug-target affinity (DTA) tasks. DTA aims to predict the affinity
substitute the molecule encoder in [44] with a SimSGT pretrained encoder to evaluate DTA performances. Following the experimental setting in [45],
scores between molecular drugs and target proteins [44, 29]. Specifically,
pretrain SimSGT the 50 thousand molecule samples from the GEOM dataset [46] and report the mean performances and standard deviations across three random seeds. We report the RMSE for the molecular property prediction datasets with scaffold splitting, and report the MSE for the DTT datasets with random splitting. The results are summarized in Table 6. can be observed that SimSGT achieves significant improvement over the baseline models.
Table 7: Mean Average Error (MAE) performance on the QM datasets. GTS encoder is used.
QM7
QM8 12
QM9
#Tasks
12
GraphCL
80.423.
0.0200a0x00
5.7620.37
GraphMAE
78.4
0.0190a0.0:
5,84
Mole-BERT
79.8u26
0.0
5.75
SimSGT
75.4
0.0183,0.000
5.53
Quantum chemistry property prediction. We report performances of predicting the quantum chemistry properties of molecules [47]. Following [48], we divide the downstream datasets by scaffold split. This experiment reuses the model checkpoints from Table 5, which are pretrained on million molecules from ZINC15. Specifically, we attach two-layer MI.P after the pretrained molecule encoders and fine-tune the models for property prediction. We report average performances

---

Table 8: Time spent for pretraining 100 epochs on ZINCI5 when using the GTS encoder.
Model
GraphMAE
Mole-BERT
S2GAE
GraphMAE2
SimSGT
Pretrain Time
527 min
2199 min
1763 min
1195 min
645 min
and standard deviations across 10 random seeds. The performances are reported in Table 7. We can observe that SimSGT consistently outperforms representative baselines of GraphCL, GraphMAE, and Mole-BERT.
Computational cost. We compare the wall-clock pretraining time for SimSGT and key baselines in Table 8. We can observe that: 1) SimSGT's pretraining time on par with GraphMAA [9]. This efficiency is largely attributed the minimal computational overhead of our SGT tokenizer; 2) In comparison to Mole-BERT [10], the prior benchmark in molecule SSL, SimSGT is approximately three times faster. The computational demands of Mole-BERT can be attributed to its combined approach of MGM training and contrastive learning.
6 Conclusion and Future Works
In this work, we extensively investigate the roles of tokenizer and decoder in MGM for molecules. We compile and evaluate comprehensive range of molecule fragmentation functions as molecule tokenizers. The results reveal that subgraph-level tokenizer gives rise to MRL performance. Further, we show by empirical analysis that sufficiently expressive decoder with remask decoding improves the molecule encoder's 's representation quality. In light of these findings, we introduce SimSGT, novel MGM approach with subgraph-level tokens. SimSGT features a Simple GNN-bases Tokenizer capable of transforming GNNs into effective graph tokenizers. It further adopts the GTS architecture for encoder and decoder, and incorporates a new remask strategy. SimSGT exhibits substantial improvements over existing molecule SSL methods. For future works, the potential application of molccule tokenizers to joint molecule-tex. modeling [3], remains an interesting direction.
Acknowledgement
This research supported by the National Natural Science Foundation of China (92270101 14) and the University Synergy Innovation Program of Anhui Province (GXXT-2022-040) This material is based upon work supported by the Google Cloud Research Credit program with the award (6NW8- CF7K-3AG4-IWHD) This research is supported by NExT Research Center.
References
[1] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self-supervised learning for molecular property prediction. NeurIPS, pages 15870-15882. 2021.
[2] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang. and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. In NeurIPS, 2020.
[3] Zhiyuan Liu, Sihang Li. Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In EMNLP, 2023.
[4] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS, 2020.
[5] Laurianne David, Amol Thakkar, Rocío Mercado, and Ola Engkvist. Molecular representations in ai-driven drug discovery: a review and practical guide. Journal of Cheminformatics, 12(1):1-
22, 2020.
[6] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals. and George E Dahl. Neural message passing for quantum chemistry. In ICMI., pages 1263-1272. PMLR, 2017.
[7] Chaochao Yan, Qianggang Ding, Peilin Zhao, Shuangjia Zheng, Jinyu Yang, Yang Yu, and Junzhou Huang. Retroxpert: Decompose retrosynthesis prediction like A chemist. In NewrIPS. 2020.
[8] Umit V Ucak, Islambek Ashyrmamatov, Junsu Ko, and Juyong Lee. Retrosynthetic reac- tion pathway prediction through neural machine translation of atomic environments. Nature communications, 13(1):1186 2022.
10

---

[9] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjje Wang. and Jie Tang. Graphmae: Self- sulf-vueer masked graph autoencoders. In KDD, pages 594-604 ACM, 2022.
[10] Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z. Li. Molc-BERT: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations, 2023.
[11] Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In ICML, volume 119 of Proceedings of Machine Learning Research, pages 10871-10880. PMLR, 2020.
[12] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitikk, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In ICLR, 2020.
[13] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACI-HLLI, pages 4171-4186. Association for Computational Linguistics, 2019.
[14] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 19979-15988. IEEE, 2022.
[16] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In ICLR. OpenReview.net, 2022.
[17] Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou Mocl: Data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. In KDD, pages 3585-3594. ACM, 2021.
[18] Zewei Ji, Runhan Shi, Jiarui Lu, Fang Li, and Yang Yang. Relmole: Molecular representation learning based on two-level graph similarities. Journal Chemical Information Modeling, 62(22):5361-533
2022.
[19] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In ICML., pages 8821-8831, 2021.
[20] Mukund Deshpande, Michihiro Kuramochi, Nikil Wale, and George Karypis. Frequent substructure- approaches for classifying chemical compounds. IEEE Transactions on Knowledge and Data Engineering, 17(8):1036- 2005.
[21] Jonathan Clayden, Nick Greeves, and Stuart Warren. Organic chemistry. Oxford university press, 2012.
[22] Jintang Li, Ruofan Wu, Wangbin Sun, Liang Chen, Sheng Tian, Liang Zhu, Changhua Meng, Zibin Zheng, and Weiqiang Wang. Maskgae: Masked graph modeling meets graph autoencoders. CoRR, abs/2205. 10053, 2022.
[23] Chun Wang, Shirui Pan, Guodong Long, Xingquan Zhu, and Jing Jiang, MGAE: marginalized graph autoencoder for graph clustering. In CIKM, pages 889-898. ACM, 2017.
[24] Sixiao Zhang, Hongxu Chen, Haoran Yang, Xiangguo Sun, Philip S. Yu, and Guandong Xu. Graph masked autoencoders with transformers. CORR, abs/2202.08391, 2022.
[25] Ganqu Cui, Jie Zhou, Cheng Yang, and Zhiyuan Liu. Adaptive graph encoder for attributed graph embedding. In KDD, pages 976-985. ACM, 2020.
[26] Xiao Liu, Shiyu Zhao, Kai Su, Yukuo Cen, Jiezhong Qiu, Mengdi Zhang, Wei Wu, Yuxiao Dong, and Jie Tang. Mask and reason: Pre-training knowledge graph transformers for complex logical queries. In KDD, pages 1 1120-112 1130. ACM, 2022.
[27] Zhanghao Wu, Paras Jain, Matthew A. Wright, Azalia Mirhoseini, Joseph E. Gonzalez, and lon Stoica. Representing long-range context for graph neural networks with global attention. In NeurlPS, pages 13266-13279, 2021.
[28] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 92):513-530, 2018.

---

[29] Tapio Pahikkala, Antti Airola, Sami Pietilä, Sushil Shakyawar, Agnieszka Szwajda, Jing Tang. and Tero Aittokallio. Toward more realistic drug-target interaction predictions. Briefings Bioinform. 16(2):325-337, 2015.
[30] Greg Landrum. Rdkit documentation. Release. 1(1-79):4, 2013.
[31] Elena S Salmina, Norbert Haider, and Igor V Tetko. Extended functional groups (efg): an efficient set for chemical characterization and structure-activity relationship studies of chemical compounds. Molecules, 21(1):1, 2015.
[32] Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In ICML, volume 80 of Proceedings of Machine Learning Research, pages 2328-2337. PMLR, 2018.
[33] Daylight Chemical Information Systems, Inc. http:///gg.....oo//a////////////
Daylight
Theory
Manual.
[34] Jonathan Clayden, Nick Greeves, and Stuart Warren. Organic chemistry. Oxford university press, 2012.
[35] Jörg Degen, Christof Brescheid--erlaccie Andrea Zaliani, and Matthias Rarey. On the art of compiling and using drug-like' chemical fragment spaces. ChemMedChem: Chemistry Enabling Drug Discovery, 3000)150331507, 2008
[36] Reinhard Diestel. Graph Theory, 4th Edition, volume 173 of Graduate texts in mathematics. Springer, 2012.
[37] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[38] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, NeurlPS, 2022.
[39] Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Wein- berger. Simplifying graph convolutional networks. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 6861-6871. PMLR, 2019.
[40] Lei Chen, Zhengdao Chen, and Joan Bruna. On graph neural networks versus auap-auugmetted mlps. In ICLR. OpenReview.net, 2021.
[41] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICMI., volume 37 of JMLR Workshop and Conference Proceedings, pages 448 456. JMLR.org, 2015.
[42] Teague Sterling and John J. Irwin. ZINC 15 ligand discovery for everyone. J. Chem. Inf. Model., 55(11):2322 2337, 2015.
[43] Ali Razavi, Aäron den Oord, and Oriol Vinyals. Generating diverse figh-iidlitty images with VAE-2 In NeurlPS, pages 14837-14847, 2019.
[44] Thin Nguyen, Hang Le, Thomas P. Quinn, Tri Nguyen. Thuc Duy Le, and Svetha Venkatesh. Graphdta: predicting drug-targed binding affinity with graph neural networks. Bioinform.. 111): 2021.
[45] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby. Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. In ICLR. OpenReview.net, 2022.
[46] Simon Axelrod and Rafael Gómez-Bombareil Bombarelli. GEOM: energy-annotated molecular conforma- tions for property prediction and molecular generation. CoRR, abs/2006. 055311, 2020.
[47] Quantum machine. http: / http:///uuuuuummmataaa
wrg/dataset////////ea/oa Accessed 2023-03.
[48] Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geeheery-eehanced molecular representation learning for property prediction. Nat. Mach. Intell., 4(2):127-134, 2022.
[49] Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael M. Bronstein, and Federico Monti. SIGN: scalable inception graph neural networks. CoRR, abs/2004.1 11188, 2020.
12

---

[50] Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In JCML, volume 119 of Proceedings of Machine Learning Research, pages 4839-4848, PMLR, 2020.
[51] Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. arXiv preprint arXiv:2012. 2020.
[52] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In
NeurIPS, 2020.
[53] Steven Bird. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69-72, 2006.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.
[55] Po- Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In NeurlPS, 2022.
[56] Alan Baade, Puyuan Peng, and David Harwath. MAE-AST: masked autoencoding audio spectrogram transformer. In INTERSPEECH, pages 2438-2442. ISCA, 2022.
[57] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
[58] Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In ACL (System Demonstrations), pages 55-60, 2014.
[59] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL (1). The Association for Computer Linguistics, 2016.
[60]
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Blena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent- new approach to self-supervised learning. In NeurlPS, 2020.
[61] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
[62] Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veličković, and Michal Valko. Large-scale representation learning on graphs via bootstrapping. In ICLR, 2022.
[63] Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M. Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. In ICLR. OpenReview.net, 2022.
[64] Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any GNN with local structure awareness. In ICLR. OpenReview.net, 2022.
[65] Ziqiao Zhang, Jihong Guan, and Shuigeng Zhou. Fragat: fragment-oriented multi-scale graph attention model for molccular property prediction. Bioinform.. (711))298112987) 2021.
[66] Fabrizio Frasca, Beatrice Bevilacqua. Michael M. Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. In NeurIPS, 2022.
[67] Chendi Qian. Gaurav Rattan, Floris Geerts, Mathias Niepert, and Christopher Morris. Ordered subgraph aggregation networks. In NewrIPS, 2022.
[68] Bohang Zhang, Guhao Feng, Yiheng Du, Di He, and Liwei Wang. A complete expres- siveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests. arXiv preprint 2302.0000,, 2023.
13

---

[69] Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR, 2019.
[70] Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In ICI.R 2020.
[71] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In ICML, Proceedings Machine Learning Research, pages 12121-12132, 2021.
[72] Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. In NeurlPS, pages 15920-15933, 2021.
[73] Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph- level representation learning with local and global structure. In ICMI., volume 139, pages 11548-1158,, 2021.
[74] Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan He, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In ICML, pages 13052-13065, 2022.
[75] Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. S2GAE: self-supervised graph autoencoders are generalizable learners with graph masking. In WSDM, pages 787-795, ACM, 2023.
[76] Zhenyu Hou, Yufei He, Yukuo Cen. Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. Graphmae2: A decoding-enhanced enhanced masked self-supervised graph learner. In WWW, volume abs/2304.04779 2023.
[77] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.0068) 2020.
[78] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. AAAI, pages 3538-3545. AAAI Press, 2018.
[79] Joshua David Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In ICLR. OpenReview.net, 2021.
[80] Yoav Levine, Barak Lenz, Opher Lieber, Omri Abend, Kevin Leyton-Brown, Moshe Tennen- holtz, and Yoav Shoham. Pmi-masking: Principled masking of correlated spans. In ICLR. OpenReview.net,222 2021.
14

---

A Limitations
Our results and analysis on the graph tokenizer and graph decoder are confined to the task of MGM pretraining. Different tokenizers and decoders might offer advantages for other generative modeling methods, such as autoregressive modeling [32].
SGTs [39, 40] are limited in expressive power for graph structures compared to standard GNNs, like GINs [37]. Theoretically, the separation of expressiveness power between SGTs and standard GNNs grows exponentially in the GNN's depth [40]. However, SGTs exhibit comparable, if not better, performances to pretrained GNN-based tokenizers, as demonstrated in Table 3b. We attribute this intriguing observation to two key factors. Firstly, SGT's (i.e., simple GNNs) are still powerful and can "distinguish almost all non-isomorphic graphs" [40]. They have shown decent results in practice [39,45 49]. Secondly, we conjecture that a better pretraining method for GNN-based tokenizers could exist, but current pretraining techniques do not fully harness the potential GNNs in their roles as effective tokenizers. Indeed, the significant difference in performance between GraphCL and VQ-VAE (Table 3b) emphasizes the impact of pretraining methods on the tokenizer's performance. We leave the investigation of how to effectively pretrain GNN-based tokenizers future works.
B Related Works
We have included the literature review of MGM in the main body of the paper. Here we elaborate on the literature review in the following areas.
Molecule SSL with motifs. Motifs are statistically significant subgraph patterns [32, 50], and have
been applied in existing molecule SSL methods. Autoregressive pretraining methods [32, 1, 50] generate motifs instead of nodes at each generation step, in order to improve the validity of the generated molecules. Motifs are also used in contrastive learning [17, 51, 18]. Sun et al. [17] substitute motifs within molecule with their chemically similar counterparts to create high-quality augmentations. [51, 18] construct molecules' views at the motif-level to supplement the original views at the atom-level. In predictive pretraining, Rong et al. [2] pretrain graph encoder to predict the FGs inside the molecule. These previous works have developed extensive molecule fragmentation methods for motif discovery. However, these fragmentation methods have been overlooked tokenizers in MGM pretraining. Our work addresses this gap by summarizing the common fragmentation rules and examining the performances of the selected fragmentation methods in MGM pretraining.
Data tokenization. Tokenization is a data pew-ppreeessseee processing technique that divides the original data into smaller elements and converts them to tokens. It is widely used in NLP to split sentences into word- level units [13, 52, 53]. Due to the surging interests Transformers [54], tokenization also applied on images [15, 16] and audios [55, 56]. Tokenization fragments these data into sequences of patches to fit the shapes of transformer's 's input and output. A tokenizer can be designed by heuristics [57], incorporating domain knowledge [58], and pretraining the target dataset [59, 19, 433. In this work, we study graph tokenizers, which are less explored in previous works.
Relations to contrastive learning. When using a GNN-based tokenizer, MGM involves minimizing the distances between the outputs from two network branches (i.e., the tokenizer branch and the autoencoder branch). At first glance, this design might seem similar to the contrastive learning methods of BYOL [60], SimSiam [61], and BGRL [62], which also minimize the output differences between two network branches. However, a closer inspection reveals several critical distinctions between MGM and these methods. Firstly, MGM feeds uncorrupted data to the tokenizer branch and feeds corrupted data to the autoencoder branch, encouraging the autoencoder to reconstruct the missing information. In contrast, BOL, SimSiam, and BGRL use corrupted data in both of their branches, constituting different training objectives. Secondly, while BYOL, SimSiam, and BGRL employ nearly identical architectures for their two branches, MGM can adopt distinctly different architectures for its autoencoder and tokenizer. In our best-performing experiment, the autoencoder has more than ten layers of GNNs and Transformers, while the tokenizer is a shallow single-layer network (Table 3). Finally, MGM employs remask decoding constrain the encoder's ability reconstruction, which is not used in contrastive learning methods [60, 61, 62].
Subgraph-enhanced Graph Neural Network. Subgraph-enchancedd GNN [63, 64, 65] refers to an emerging class of GNNs that fragments graph into subgraphs before encoding, in order to improve the GNNs' expressivenss [66, 67, 68]. The common graph fragmentation method is node-wise, such
15

---

that each fragmented subgraph is associated with unique node in the original graph. For example, ESAN [63] obtains subgraphs by sampling ego-networks or deleting one node from the original graph. Given the subgraphs, subgraph-enchanced GNNs generate node embeddings in every subgraph by applying series of equivariant peessage-passing layers [63, 64]. Finally, these embeddings are pooled to output the graph embedding. Our work is related to subgraph-enhance GNNs that we also study graph fragmentation. The major distinction is that we focus on using the tokens derived from these fragmented graphs as the reconstruction targets in MGM for molecules.
C Pseudo Code
We present the pseudocode SimSGT. This code uses a single-layer SGT of GIN as an example.
Algorithm Pytorch style pscudocode of SimSGT
phi: graph rho: graph decader
def SgT anbed):
SCT: single-layer GIR texenizer
x, enge_index -
nessage passing
-pooaagaee (x),ed(x), (x)b)()),)) edge_index) (Itepa) enbed (x)
normalization layer batchnorn(x)
for g in loader:
masking
g_hat, a_pos randon__asking((g # m_pos: mask positions
tokenization.
linear layer
sG embeed. detach: stop- gradient
autoencoder forvard
y_bat rhe (renask (phi (g.bat), m_pos))
mininize lass loss sestance_losee hy_hat [osp y [o=])
l0sa. backuard
D Experimental Setup
Computational resource. We perform experiments on an NVIDIA DGX A100 server. Each
individual experiment be single GPU without exceeding 30 GBs of GPU memory.
D.1 Compared Methods
Motif-based tokenizers. We now elaborate on the details the two compared motif-based tokeniz- ers:
MGSSL [1] employs the BRICS [35] method for molecule fragmentation (Section 2.2). To obtain fine-grained fragments, MGSSL employs additional rules to break the BRICS's output fragments: 1) separate the single atoms attached to cycles; 2) a connected subgraph comprising fragment.
three or atoms is not part of cycle, break down
RelMole [18] combines the fragmentation functions of Cycles and FGs for molecule fragmentation (Section 2.2). Further, extracts the carbon-carbon single bonds that are not covered in the previous step new fragments.
We use the motif vocabulary provided by their paper for molecule fragmentation. Given a molecule, convert its fragmented motifs into one-hot encodings, which serve the reconstruction targets.
Pretrained GNN-based tokenizers. We use the atomic numbers as the node features and exclude edge features in pretrained GNN-based tokenizers. We show in Appendix E that incorporating edge
16

---

Table Experimental settings for pretraining on million molecules from ZINCI5 and fine-tuning on eight datasets in MoleculeNet: BBBP, Tox21, ToxCast, Sider, ClinTox, MUV, HIV, and Bace.
(a) Node edge features.
Type
Range
atomic numbers chirality tag
1~118
Node features
(unspecified, tetrahedral ew, tetrahedral cew, other)
bond type
(single, double, triple, aromatic]
Edge features
bond direction
{-, endupright, enddownright)
(b) Hyperparameters.
pretrain
fine fine-tuning
Encoder
Ir
batch size
epochs
Ir
batch size
epochs
GINE
le-3
1024
100
{1e-3, le-4)
32
100
GTS
le-4
2048
100
(1e-4, le-5)
32
100
features in GNN tokenizers can decrease performance. Due to the removal of edge features, the tokenizer uses the architecture of GIN [37] instead of GINE [12]. We have reported performances of GNN-based tokenizers that are pretrained by GraphCL [4], GraphMAE [9], and VQ- [10, 43]. The implementation of VQ- VAE follows [10] and groups the latent codes by atomic numbers. We strictly follow the procedure the mentioned papers to pretrain GNNs, which are later used as tokenizers.
Simple GNN-based tokenizers (SGTs). An SGT uses the node feature atomic number. It uses the graph encoder's linear embedding function of atomic numbers. We present the graph operators for our tested SGT's below:
GIN:
(AA) A + +1++)L
(17)
GCN:
((A) --1/2AD 1/2 11/2 GraphSAGE: (AA) = D-
(18)
(19)
where Á = + and D is the degree matrix of A; e set to 0.5 empirically.
Baselines. We now describe the details of our reported baseline methods:
Infomax [69] learns node representations by maximizing the mutual information between the local summaries of node patches and the patches' graph-level global summaries.
ContextPred [12] uses the embeddings of subgraphs to predict their context graph structures.
InfoGraph [70] conducts graph representation learning by maximizing the mutual information
between graph-level representations and local substructures of various scales.
GraphCL [4] performs graph-level contrastive learning with combinations of four graph augmen-
tations, namely node dropping, edge perturbation, subgraph cropping. and feature masking.
JOAO [71] proposes a framework to automatically search proper data augmentations for GCL.
AD-GCL [72] applies adversarial learning for adaptive graph augmentation remove the redundant information in graph samples.
GraphLOG [73] leverages clustering to construct hierarchical prototypes of graph samples. They further contrast each local instance with its corresponding higher prototype for contrastive learning.
RGCL [74] trains rationale generator to identify the causal subgraph in graph augmentation.
Each graph's 's causal subgraph and its complement are leveraged in contrastive learning.
BGRL [62] trains an online encoder by learning to predict the output of a target encoder. The target encoder shares the same architecture as the online encoder and is updated through exponentially moving average. The inputs of the online encoder and the target encoder are two different graph augmentations.
17

---

Table 10: Experimental setting for pretraining on the 50 thousand molecules from the GEOM dataset and fine-tuning on the four molecular property prediction (regression) datasets and DTA datasets.
(a) Node edge features.
Type
Range
Node features
atomic numbers chirality tag node degree formal charge number of H number of radical hybridization is aromatic ring bond type bond stereo is conjugated
1 118
(unspecified, tetrahcdralew, tetrahedralccw, other)
0~~10
-5~5
0-~8
0--4
[sp, sp2, sp3, sp3d, sp3d2) (aale,, true]
(false, true)
Edge features
(single, double, triple, aromatic] (stereonone, stereoz, stereoe, stereocis, stereotrans, stereoany] [false, true]
(b) Hyperparameters and their search spaces. use the performance on validation set for hyperparameter tuning. Bold indicates the final value used in experiments.
pretrain
fine-tune tune (regression)
fine-tune (DTA)
Encoder
batch size epochs
batch size
epochs
Ir
batch size epochs
GINE
le-3
1024
100
lc-3
[32, 128, 256)
100
[1c-4, 2e-4)
128 128
500
GTS
le-4
1024
300
[le-4, 2e-4, 3e-4)
32
100
[le-4, 2e-4)
500
GraphMAE [9] shows that linear classifier is insufficient for decoding node types. It applies GNN for decoding and proposes remask decouple the functions of the encoder and decoder in the autoencoder.
GraphMVP [45] uses a contrastive loss and a generative loss to connect the 2-dimensional view and 3-dimensional view of the same molecule, in order to inject the 3-dimensional knowledge into the 2-dimensional graph encoder.
S2GAE [75] randomly masks portion of edges of graphs and pretrain the graph encoder to predict the missing edges.
GraphMAE2 [76] applies multi-view random re-mask decoding as regularization for MGM pretraining.
Mole-BERT [10] combines a contrastive learning objective and a masked atom modeling objective for MRL. Specifically, they observe that mask atom prediction is an overly easy pretraining task. Therefore, they employ GNN tokenizer pretrained by VQ- VQ-V [43] to generate more complex reconstruction targets for masked atom modeling.
D.2 Experimenets in Section 4 and Table 5
Here we elaborate the experimental setting for pretraining on million molecules from ZINC15 [42] and fine-tuning the eight classification datasets in MoleculeNet [28]: BBBP, Tox21, ToxCast, Sider, ClinToo, MUV, HIV, and Bace. This setting covers the experiments in Section 4 and Table 5.
Molecule representations. For SimSGT and other compared methods, follow previous works [12, 4] and use a minimal set molecule features as the graph representations (Table 9a). These features unambiguously describe the two-dimensional structure of molecules.
Hyper-parameters. Table 9b summarizes the paper-parameters. We use different given different graph encoders. The architectures of the two graph encoders are borrowed from previous works: GINE [12] and GTS [27]. We use large batch sizes of 1024 and 2048 to speed up pretraining. We do not use dropout during pretraining. During fine-tuning, we 50% dropout in GINE layers and 30% dropout transformer layers. The learning rate for the MUV dataset is 10 times smaller than other datasets. Following [4, 73], report the last epoch's 's test performance. We
18

---

Table 11: Hyperparameters for fine-tuning on the QM datasets.
QM Dataset
batch size
Ir
QM7
32
4e-4
QM8
256
le-3
QM9
256
le-3
report the mean performances and the standard deviations across 10 random seeds. Baselines reproduced using the same setting.
Linear probing experiments. Here we elaborate on the settings of our linear probing experiments (Figure 5b and Figure 7). Specifically, we randomly split the million molecules from ZINC15 into train set (90%) and test set (10%). We train the MGM models the train set and save the encoder's checkpoint every epoch. The linear classifiers are trained for 1000 epochs on the encoder's frozen hidden representations. We train linear classifiers using 25600 molecule samples from the training set and evaluate them on the whole test set.
Probing masked atom types (Figure 5b). We let linear classifiers predict the masked atom types using the masked atoms' hidden representations. During linear probing, we disable remask-v2 to obtain the masked atoms' hidden representations. Molecules randomly masked by 0.35 during probing. We use accuracy (%) as the evaluation metric.
Probing FGs (Figure 7). Following [2], we extract 85 types of FGs for each molecule using RDkit [30]. FGs are represented by 85-dimensional binary vectors, whose each dimension indicates the presence a certain FG. Afterward, we train multi-label linear classifiers on the frozen encoder's mean pooling outputs for FG prediction. Molecules are not masked during probing. We use ROC-AUC (%) as the evaluation metric.
D.3 Experiments in Table
We present the experimental setting for pretraining on the 50 thousand molecules from GEOM [46] and fine-tuning on the four molecule property prediction (regression) datasets and two DTA datasets. Our experimental setting follows that in [45]. This setting covers the experiments in Table 6.
Molecule representations. In the graph autoencoder, 3-dimensional edge features of molecules provided by the OGB [77] package, following Graph- MVP [45]. The features summarized in Table 10a. Note that, tokenizer uses only the atomic numbers as node features and does not use edge features.
9-dimensional node features and
Exper-parameters. The hyperparameters eters in the fine-tuning stage using the validation performance. Following [45], we report the test performance at the epoch selected by the validation performance. We do not use dropout during pretraining. During fine-tuning, we 50% dropout GINE layers and 30% dropout in transformer layers.
summarized in Table 10b. We tune the hyperparam-
For a fair comparison, we reproduce Mole-BERT [10]'s performance by pretraining on the 50 thousand molecule samples from the GEOM dataset [46]. The original Mole-BERT is trained on larger dataset of million molecules from ZINC15 [42].
D.4 Experiments in Table
The hyperparameters for fine-tuning on the QM datasets are reported in Table 11.
E More Experimental Results
In this section, provide more experimental results. If not especially noted, these experiments employ an autoencoder of GTS encoder and ETSl decoder with remask-v2, and a tokenizer of single-layer SGT of GIN. Other settings follow that in Appendix D.2.
Influence of edge features for pretrained GNN-based tokenizer. We ablate the impact of the "bond type" and "bond direction" edge features in pretrained GNN-based tokenizers. We use GINE and
19

---

Table 12: Average transfer learning ROC-AUC (%) scores on the eight classification datasets in
MoleculeNet. Including edge features in tokenizers decreases the performance.
Tokenizer GNN's depth
Tokenizer
Edge feature
1
2
3
4
5
x
75.1
74.5
74.2
74.0
74.6
Pretrain, GraphCL.
/
74.2
74.7
73.7
73.8
73.2
x
75.1
74.9
74.9
75.4
75.2
Pretrain, GraphMAE
/
74.6
74.6
74.3
74.6
75.0
1e6
with BN without BN
MSE loss
1.0
without
Count
0.5
0.5
0.1
0.0
50
100
150
200
Pretrain steps
Value
(a) MSE loss curve with respect pretrain steps. (b) Histograms for the values of the tokenizer's output.
Figure SimSGT pretraining on the millions molecules from ZINC15.
GIN for the tokenizer with and without edge features. Table 12 shows that including edge features in GNN-based tokenizers negatively influences the transfer learning performance. Therefore, we exclude edge features from pretrained GNN-based tokenizers in our experiments.
Influence of Batch Normalization layers in SGTSs. The Batch Normalization [41] (BN) layers in SGT' are crucial to avoid loss vanishment. Figure 8a presents a comparison between SGT "with us without BN Without the BN layer, the MSE loss drops to lower than 0.01 within few steps of pretraining. Such small loss values lead to significant model underfitting.
As shown by Figure 8b, the token values of SGT without BN follow sharp distribution: the values are primarily distributed around zero, and their standard deviation (std) is smaller than 0.35. This minor std issue might be caused by the smoothing effect of GNNs [78]. An expressive neural network (i.e., a graph autoencoder) can quickly this sharp target distribution and minimize the loss to a negligible value, causing the problem of loss vanishment. However, if BN layer is used, it forces each dimension of the tokenizer output to have an std of 1.00, so as to "spread out" the distribution of the SGT tokens. These new SGT tokens of larger std harder fit. They keep the MSE loss at reasonable range of 0.10 ~ 0.15 (Figure 8a).
Mask ratio. We apply random node masking throughout the experiments [12]. Figure 9 presents SimSGT's sensitivity with respect to the mask ratios. SimSGT is not sensitive to mask ratios such that wide range of ratios (0.25 ~ 0.45) can generate competitive performances. The ratio of 0.35 achieves the best performance. This ratio is much lower than that for images, where a ratio of 0.75 can generate promising performances [15].
Balancing the distribution of reconstruction targets. As shown in Figure 11, popularly used ZINC15 dataset includes 12 types of atoms, and 95% of the atoms are distributed the top three atom types. This skewed distribution renders the node-level token reconstruction an easy pretraining task [10]. Figure 10 shows that the accuracy of predicting node-level tokens converges quickly. Such an easy pretraining task can lead to suboptimal performance as suggested by existing SSL literature [79, 80]. In Figure 11, show that the induced subgraphs of single-layer SGT (i.e., one-hop rooted subtrees) follow a more balanced distribution than the distribution of nodes. SGT tokens also have larger vocabulary size: ZINC15 includes 555 types of one-hop rooted subtrees. Consequently, the accuracy of predicting tokens of a single-layer SGT takes more epochs to converge (Figure 10).
Pooling Method for Subgraph Representations. In previous experiments, we use mean pooling to obtain the subgraph representations for motif-based tokenizers, following the method of obtaining graph representations in [4, 12]. Here we add results for MGSSL tokenizer using sum and max
20

---

100
78
75.8
95
ROC AUC (1)
75.5
75.4
Aosuaey (9)
75.2
75.2
74,8
node-loval taken
subgraphiievel token from SGT
0.25
0.30
0.35
0.40
0.45
0.50
0.55
20
60
80
100
Mask ratio
Pretrain epochs
Figure 9: Average ROC-AUC (%) scores with respect to different node masking ratios. The performances are evaluated on the eight classification datasets in MoleculeNet.
Figure 10: Token prediction accuracies. SGT token prediction is conducted by calculating the Euclidean distance be- tween the autoencoder's 's output and the vocabulary of all SGT tokens.
14.0
Distribution
Distribution
19%
Laru
37%
110
T2
56
93
84
80
60
Figure 11: Distributions of graph fragments in MGM. Statistics come from molecules in ZINC15 [42]. For the subgraph types, colon separates the center node and the neighbor nodes. For example, C:CN denotes subgraph with center of Carbon and neighbors of Carbon and Nitrogen.
pooling in Table 13. The results show that mean pooling yields the highest performance, affirming the soundness of our previous experiments.
Full results of Section 4. We provide the full results of experiments in Section Table contains the full results of Table 3a and Table 4. Table 15 contains the full results of Table 3b and Figure 6.
21

---

Table 13: ROC-AUC (%) scores on eight MoleculeNet datasets. Compared methods use GTS encoder and GTS-Small decoder with remask-v2 decoding.
Dataset
BBBP
Tox21
ToxCast SIDER ClinTox MUV
HIV
BACE
Avg.
GAIN
Motif, MGSSL, Mcan 72.5.09 77.5
65.2 60. 7ea9 85.0at 79.9a15 78.0a1s 83.0ata
75.2
5.3
Motif, MGSSL, Max
71.5e09
75.8
66.2
60.7
82.6
78.9218
76.5 83.8
74.5
4.6
Motif, MGSSL, Sum
71.7a1A
75.9
66.1
60.4
83,4 79.5a1.
76.8a12
84.2at.
74.8
4.8
Table 14: Transfer learning ROC-AUC (%) scores on the eight classification datasets in MoleculeNet.
Encoder
Decoder
Remask BBBP Tox21 ToxCast SIDER ClinTox MUV
HIV
BACE
AAy..
GINE
Linear
72.5ams 76.0aa4 63.7a0.
60.1 81.2a26 74.2a16 78.0ans 79.6.14
73.2
GINE
GINE-Small
70.9 75.
63.5.o.
61.0an4
79.1a26
76.0so5
76.3
82.5an9
73.0
GINE
GINE-Small
v1
70.2a1.
76.4s06
64.2s0.4
61.9ans
80.9at.
77.8a1.1
18.1 83.6s11
74,1
GTS
Linear
72.9e1o 76.6
63.8ea8
58.3a13
81.9555
78.5s1.
78.0eza
74.1
GTS
GTS-Small
72.0-a6 74.7-04
63.7-04
58.9
86,0k20
78.9-12
77.3-07
81.0=a7
74.1
GTS
GTS-Small
v1
71.3+10 77.0.12
66.2-06
60.6
84,5
81.5+15 77.0
83.5
75.2
GTS
GTS-Small
v2
72.2at9 76.8209 65,920x
61.7aus
85.7at8
81.421A
78.0a19 84.3
75.8
GTS
GTS-Tiny
v2
71.9212 77.2s1.1 65.6
61.7a14
82.9a24
79.6s1.4
76.8 82.1
74.7
GTS
GTS
v2
70.7a12
76.4s09
16.
60.3a09
84.7246
79.6.07
76.8 84.5s03
74.9
Table 15: Transfer learning ROC-AUC (%) scores on the eight classification datasets in MoleculeNet.
Tokenizer
BBBP Tox21 ToxCast SIDER ClinTox MUV
HIV
BACE
Avg.
Node
70,3 76.4 65.7a07 61.7am9 81.9235 79.8 77.4x18 84.6.11
74.7
Motif, MGSSL
72.5.a9
77.5.04
65.2a0s 60.7am9 85.0a3.
79.9a15
78.0x15
83.0ain
75.2
Motif, RelMole
71.4 77.1
66.3s0s
58.9s12
80.7a27
79.2s14 78.0x10
83.6.10
74.4
Pretrain, GraphCI. 1 layer GIN
72.2+14 76.8s04
66.0ees
60.8
81,4-25
81.1 78.4
83.8+a9
75.1
Pretrain, GraphCI 2 layer GIN
70.8+a6 76.7.08
66.3-04
60.6+1.
84,3+33
77.2+14 76.3x21 83.8+12
74.5
Pretrain, GraphCI, 3 layer GIN
70.6 77.1
65.4.0,
59,3u14 81.2s3.8
79.4s23 76.4
84,2.1.1
74,2
Pretrain, GraphCL, 4 layer GIN
12. 76.9206
65.7.07
59.6s09
77.6:3.
81.7a12 77.6x22
81.1
74.0
Pretrain, GraphCL, 5 layer GIN
71.4 76.9
66.5
60.0s12 80.8,2.3
81.0st1
77.7
82,521.
74,6
Pretrain, VQ-VAF, layer GIN
72.2+09 77.0s0s
66.5-06
61.3t1
82.8+37
79.1 77.4k15
84,2+ax
75.1
Pretrain, VQ-VAF, 2 layer GIN
71.5.a8 76.6
65.9-07
60,3107
82,1-20
81.5.2a 77.2x19
84.3+1.
74.9
Pretrain, VQ-VAF, layer GIN
71.9.09
76.7s09
65.8.os
61.2a18 79.5.25
80.0.ps 78.0+13
81.7sa9
74.4
Pretrain, VQ-VAE, 4 layer GIN
72.5.1.0
77.0a0.
66.3:03
61.7a15 86,7,22
80.3,16
77.6 82.7
75.6
Pretrain, VQ-VAE, 5 layer GIN
72.0aa9
76,8206
65.62066
61.5H1.1 84.321.3
80.6
18.1 81,9208
75.1
Pretrain, GraphMAE, layer GIN
72.0at.1 77.3206
66.3
60.9
83,7a27
79.9214
76.3222 84,0216
75.1
Pretrain, GraphMAE, 2 layer GIN 71.9208 77.6206
66.020.
60.8
81.9239
79.0216 76.5224 85,
74.9
Pretrain, GraphMAE. 3 layer GIN 71.420. 77. 77.6.0x
65 65.3 61.1 82.2a19
79.2.15 77.4.21 84.3g0u
74.9
Pretrain, GraphMAE, 4 layer GIN 72.3 76.6
16. 62,0+12
83.3-21
80.
77.9.17
85,0ga)
75.4
Pretrain, GraphMAE, 5 layer GIN 72.6.00
76,4
65.7 62.4+13 84.0.2s
80.0A13
78.7
81,521.
75.2
SGT, layer GIN
72.2a0. 76.8.0.
65.920s
61.7203 85,7218
81.4214
78.0x19
84,3
75.8
SGT, layer GIN
71.3
77.0g09
66.220s
61.5w0s 84,9220
80.7420
18.1 84.52a8
75.5
SGT. layer GIN
71.7 77.62ox
66.2206
61.2224 85,8216
80.4211
77.7x21
84.
75.6
SGT, 4 layer GIN
72.0za. 17.1at2
65.4210
61.7a15 83,8222
80.1.19
77.5x12
84.7
75.3
SGT, layer GIN
70.7 77.0a07
65.9ass
61.1ai6
83.1
79.9.15
77.6x16
83,9a1.
74.9
SGT, layer GCN
71.9 77.8
66.5
62,0mns
85,2=1.6
79.2.14
77.9t23
84.4221
75.6
SGT, layer GCN
1 77.4
66.3-64
61.641.1
84.4-2
78.7.11 77.7+24
85.
75.3
SGT. layer GCN
71.0a1.0 76.520.
65.6203
61.9113
83.7a21
78.3220 76.7x12 84,9a1
74.8
SGT, layer GruphSAGE
72.5wa7
77.1
66.7a0s
61.3210
86,2a18
80.7a15 76.6x17 83,5a10
75.6
SGT, layer GraphSAGE
70.3
77.9
65.4
60.3r10
87,4a3
79.2.19 77.6x21
84,9aa
75.4
SGT, layer GraphSAGE
71.6.1.0
76.5
66.4-6.7
60.8
85.6
78.6.13
6 84.4
75.1
22


Reviews:
[{"review_id": "rTwUYQWFED", "reviewer": "Reviewer_9pKQ", "summary": "This paper proposes a masked graph modeling framework called Simple GNN-based Tokenizer (SGT) for molecular graph analysis. Extensive experiments show the performance of the proposed method.", "questions": "N/A", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The article is well-written and easy to understand.\n2. The proposed framework is attractive to this research community.\n3. The experimental results further corroborate the authors' view.", "weaknesses": "1. The limitations of the previous methods are not explained clearly. What is the limited understanding of tokenizer and decoder?\n2. Theoretical analysis is weak. The paper only provides experimental results to support its idea, but it remains unclear why removing the nonlinear update function in each GNN layer can train a better encoder.\n3.  In Table 5, the improvement of the proposed method over some baselines (e.g., RGCL) is incremental, which can not well support its effectiveness.\n4. Lack of comparison of computational time.\n5. Missing baselines [1,2].\n[1] S2GAE: Self-Supervised Graph Autoencoders are Generalizable Learners with Graph Masking.\n[2] GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  **Q1.** The limitations of the previous methods are not explained clearly.\n\n**Response:** Thanks for your comments. Although the limitations of prior methods have been elucidated (as outlined in Lines 51-54 and Table 1, and supported by findings in Sections 4.1 and 4.2), we will clarify more:\n\n1. Tokenizer: Most previous research does not consider the potential of existing motif-based fragmentation functions as tokenizers.\n\n2. Decoder: Previous studies mostly adpot a linear or an MLP decoder for graph reconstruction, but unexploring more expressive decoders.\n\nFurthermore, we summarize the limitations in the following table and add it in our revision.\n\n**Table1: Potential limitation of tokenizers in previous works.**\n\n| Tokenizers     | Potential Limitation            |\n| -------------- | ------------------------------- |\n| Node, edge     | Low-level feature               |\n| Motif          | Rely on expert knowledge        |\n| Pretrained GNN | Extra pretraining for tokenizer |\n| Ours, SimSGT   | -                               |\n\n**Table2: Potential limitation of decoders in previous works.**\n\n| Model        | Decoder          | Sufficiently Expressive | Remask | Avoid processing masked nodes? |\n| ------------ | ---------------- | ----------------------- | ------ | ------------------------------ |\n| Others       | Linear, MLP      | x                       | -      | x                              |\n| GraphMAE     | Single-layer GNN | x                       | v1     | x                              |\n| Ours, SimSGT | GTS-Small        | Y                       | v2     | Y                              |\n\n\n\n> **Q2.** In Table 5, the improvement of the proposed method over some baselines (e.g., RGCL) is incremental...\n\n**Response:** We appreciate your feedback. We would like to respectfully highlight the leading performance of our SimSGT model, especially when juxtaposed with the most competitive baselines, as evidenced in the outlined tables:\n\n**Table3**\n\n| Dataset   | BBBP     | Tox21    | ToxCast  | SIDER    | ClinTox  | MUV      | HIV      | BACE     | Avg. | Improvement |\n| --------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | ---- | ----------- |\n| Mole-BERT | 71.9±1.6 | 76.8±0.5 | 64.3±0.2 | 62.8±1.1 | 78.9±3.0 | 78.6±1.8 | 78.2±0.8 | 80.8±1.4 | 74.0 |             |\n| SimSGT    | 72.2±0.9 | 76.8±0.9 | 65.9±0.8 | 61.7±0.8 | 85.7±1.8 | 81.4±1.4 | 78.0±1.9 | 84.3±0.6 | 75.8 | 1.8         |\n\n**Table4**\n\n|          | Molecular Property  Prediction |             |             |             |           | Drug-Target Affinity |             |           |\n| -------- | ------------------------------ | ----------- | ----------- | ----------- | --------- | -------------------- | ----------- | --------- |\n|          | ESOL                           | Lipo        | Malaria     | CEP         | Avg.      | Davis                | KIBA        | Avg.      |\n| GraphMVP | 1.064±0.045                    | 0.691±0.013 | 1.106±0.013 | 1.228±0.001 | 1.022     | 0.274±0.002          | 0.175±0.001 | 0.225     |\n| SimSGT   | 1.039±0.012                    | 0.670±0.015 | 1.090±0.013 | 1.060±0.011 | **0.965** | 0.263±0.006          | 0.144±0.001 | **0.204** |\n\n\n\n1. In Table 3, SimSGT significantly surpasses the strongest baselines, demonstrating an improvements of 1.8% in average ROC-AUC.\n\n2. In Table 4, SimSGT consistently posts lower RMSE and MSE values than all the competing baselines across every datasets.\n\nMoreover, owing to the difficulity of molecular self-supervised learning, existing literature [4,5,6,7] indicates steady and gradual advancements, rather than groundbreaking leaps as expected. Given the intricacies and nuanced progress inherent to this field, we kindly urge the reviewer to re-assess our contributions and the results we presented.\n\n> **Q3.** Lack of comparison of computational time.\n\n**Response:** Thank you for your valuable suggestion. To address this, we have incorporated the wall-clock pretraining time for SimSGT and key baselines in Table 2 of our updated pdf. Our findings indicate that: \n\n1. SimSGT's pretraining time is on par with GraphMAE [5]. This efficiency is largely attributed to the minimal computational overhead of our SGT tokenizer.\n\n2. In comparison to Mole-BERT [4], the prior benchmark in molecule SSL, SimSGT is approximately three times faster. The computational demands of Mole-BERT can be attributed to its combined approach of MGM training and contrastive learning.\n\nThis insightful comparison will certainly be integrated into our revision. We're grateful for your astute feedback, which has undeniably enriched our presentation.\n\n>  **Q4.** Missing baselines: 1) S2GAE: Self-Supervised Graph Autoencoders are Generalizable Learners with Graph Masking; 2) GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner.\n\n**Response:** Thanks for the suggestion. We have included these the new baselines in our updated Table 1. We can observe that SimSGT maintains the better performances over baselines. Note that, the performance of S2GAE is worse than expected. This might be caused by the mismatch between its pretraining task (*i.e.*, link prediction) and MRL's objective (*i.e.*, graph classification). In Table 9, we similarly observe that using edge features in the reconstruction target leads to worse performances.\n\n**Reference:**\n\n[1] How Powerful are Graph Neural Networks? In ICLR 2019.\n\n[2] Simplifying Graph Convolutional Networks. In ICML 2019.\n\n[3] How Powerful are Spectral Graph Neural Networks. In ICML 2022.\n\n[4] MOLE-BERT: RETHINKING PRE-TRAINING GRAPH NEURAL NETWORKS FOR MOLECULES. In ICLR 2023.\n\n[5] GraphMAE: Self-Supervised Masked Graph Autoencoders. In KDD 2022.\n\n[6] Let Invariant Rationale Discovery Inspire Graph Contrastive Learning. In ICML 2022.\n\n[7] Self-supervised Graph-level Representation Learning with Local and Global Structure. In ICML 2021.\n"}, {"review_id": "AJtAvLQo3N", "reviewer": "Reviewer_PiLY", "summary": "The authors attempt to categorize existing approaches for pretraining neural networks on molecular graphs and assess their contributions to pretraining quality.  They then propose a new strategy for pretraining molecular graphs and compare it to existing results.  ", "questions": "This is primarily a question for future work and my own academic interest in the field.  In many practical applications for learning on molecular graphs SOTA approaches typically consist of regression on classical fingerprint techniques, combined using ensembling methods (note that this typically isn't well-represented on public leaderboards, often due to the fact that these datasets can become overfit and that there is substantial research interest in neural approaches).  It seems like modifying the methods proposed here to use fingerprint-based methods  instead of the GNN-based tokenizer, e.g. by enumerating the Morgan fingerprints an atom takes place and hashing, could be a fruitful research direction.  I'd be curious to hear the authors thoughts on this, or alternative deterministic featurizations to the GNN-inspired tokenizer.", "limitations": "Limitatinos are appropriately addressed.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "I was very impressed by this paper: the research was well-motivated, and the approach used was novel.  I also found the paper informative and easy to read: the authors clearly describe how they came to the conclusions they did and what motivated their explorations.  \n", "weaknesses": "Maybe I was looking in the wrong places, but I struggled to find something to criticize here.  The only thing that I could find is that the SimSGT framework is strongly reminiscent of the famous BYOL work (https://doi.org/10.48550/arXiv.2006.07733) and a short discussion of the similarities between the two would improve the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We're genuinely pleased to hear that you found the research well-motivated and the approach novel. Your positive feedback on the paper's clarity and informativeness means a lot to us. The constructive feedback provided will undoubtedly help us further refine and improve our work. We appreciate your positive remarks and thoughtful engagement with our work.\n\n> **Q1:** Maybe I was looking in the wrong places, but I struggled to find something to criticize here. The only thing that I could find is that the SimSGT framework is strongly reminiscent of the famous BYOL work and a short discussion of the similarities between the two would improve the paper.\n\n**Ressponse:** Thank you for insightful observation regarding the resemblance between our SimSGT framework and BYOL [1]. Here we present a brief discussion on the similarities and distinctions between our method and BYOL, along with other contrastive learning methods.\n\n\n\nSimSGT involves minimizing the distances between the outputs from two network branches (*i.e.*, the tokenizer branch and the autoencoder branch). This design is similar to the contrastive learning methods of BYOL [1], SimSiam [2], and BGRL [3], which also minimize the output differences between two network branches. However, a closer inspection reveals several critical distinctions between MGM and these methods. Firstly, MGM feeds uncorrupted data to the tokenizer branch and feeds corrupted data to the autoencoder branch, encouraging the autoencoder to reconstruct the missing information. In contrast, BYOL, SimSiam, and BGRL use corrupted data in both of their branches, constituting different training objectives. Secondly, while BYOL, SimSiam, and BGRL employ nearly identical architectures for their two branches, MGM can adopt distinctly different architectures for its autoencoder and tokenizer. In our best-performing experiment, the autoencoder has more than ten layers of GNNs and Transformers, while the tokenizer is a shallow single-layer network. Finally, MGM employs remask decoding to constrain the encoder's ability on reconstruction, which is not used in contrastive learning methods [1,2,3].\n\nYou can find a relevant discussion at Line 540-553 in the Related Works section (Appendix B).\n\n>  **Q2.** This is primarily a question for future work and my own academic interest in the field. In many practical applications for learning on molecular graphs SOTA approaches typically consist of regression on classical fingerprint techniques, combined using ensembling methods (note that this typically isn't well-represented on public leaderboards, often due to the fact that these datasets can become overfit and that there is substantial research interest in neural approaches). It seems like modifying the methods proposed here to use fingerprint-based methods instead of the GNN-based tokenizer, e.g. by enumerating the Morgan fingerprints an atom takes place and hashing, could be a fruitful research direction. I'd be curious to hear the authors thoughts on this, or alternative deterministic featurizations to the GNN-inspired tokenizer.\n\n**Response:** Thank you for the inspiring comment! Indeed, fingerprint-based methods like Morgan fingerprints and ECFP [4] can also be used as graph tokenizers to provide reconstruction targets for MGM. Actually, these fingerprint-based methods can be seen as special GNNs with hashing functions as the update layer for node representations. Therefore, given their demonstrated effectiveness in many tasks, it's certainly compelling to employ them as graph tokenizers.        \n\n\n\n**Reference:**\n\n[1] Bootstrap your own latent - A new approach to self-supervised learning. In NeurIPS 2020.\n\n[2] Exploring simple siamese representation learning. In CVPR 2021.\n\n[3] Large-scale representation learning on graphs via bootstrapping. In ICLR 2022.\n\n[4] Extended-connectivity fingerprint. In Journal of chemical information and modeling 2010.\n"}, {"review_id": "Npd5y6Ni18", "reviewer": "Reviewer_Lzf1", "summary": "This paper mainly revisits the graph tokenizers and the graph autoencoders in Masked Graph Modeling (MGM) frameworks. Authors examine the roles of different tokenizers as the MGM’s reconstructions targets and propose a simple GNN-based tokenizer method and a decoding strategy. The experiment results show the effectiveness of the designed tokenizer and decoding strategy, as well as some insightful findings.", "questions": "1. What are the real differences between the proposed tokenizer and the pretrained GNN-based tokenizer? Can the proposed method be considered as a special case of the previous GNN-based tokenizer? Authors are encouraged to investigate more about the insights behind the different tokenizers. \n2. In the experiments, the authors use ZINC15 as the pretrained dataset. What if more pretrained datasets are used? Will the proposed method still perform well in different size of pretrained dataset? Authors are encouraged to do more experiments or do some theoretical analysis about why the proposed method work well than others.\n\n\nThe response from authors addressed these questions and clarified the confused parts in the paper.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and mostly easy to understand.\n2. The revisiting part is clear and the summary of current tokenizers and autoencoders are systematic.  \n3. The code is provided by an anonymous link.", "weaknesses": "1. The motivation for the proposed method is somehow unclear and the novelty is limited. The revisiting part takes a lot of space to introduce the existing tokenizers and decoding strategies. There’s no explicit insight drawn from the revisiting results, which can be helpful for designing the proposed method. \n2. The technical contributions of this work are less and somewhat incremental. The proposed methods are all based on the current MGM framework and only change a little about the tokenizer and remask strategy. Concretely, the proposed simple GNN-based tokenizer is based on the previous pretrained GNN-based tokenizer concept and the remask-v2 has already been widely used in many scenarios to prevent transformer layers from processing masked items.\n3. From the experiments in Table3, the improvements of the proposed tokenizer and decoder are limited. In Table3 (b), when the depth of tokenizer is 4 or 5, the performance of proposed tokenizer is even worse than previous tokenizers. Additionally, there’s no standard error reported. \n4. The comparison of Table5 and Table6 is somehow unfair. The GNN encoders or decoders of baselines may be different from this work. So if the GNN encoders of this work are strong, the higher performance may be because of the encoders, rather than the proposed tokenizers and decoding strategy of this work.\n5. There’s no theoretical analysis of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  **Q1.** The motivation ... is somehow unclear and the novelty is limited ... There’s no explicit insight drawn from the revisiting results, which can be helpful for designing the proposed method.\n\n**Response:** We appreciate your insights, but wish to respectfully emphasize our contribution and novelty. Our primary objective is not just to introduce another model, but to critically analyze and rethink the design paradigms (especially tokenizer and decoder choices) in MGM for molecule SSL. Our motivation with novelty are summarized in the following points:\n\n1. Systematic Scrutiny: We believe we're pioneering in our effort to meticulously scrutinize the prevalent MGM design choices for molecule SSL. Our work sheds light on the inherent advantages and disadvantages of both tokenizers and decoders.\n\n2. Simple yet Effective Design: Our SimSGT represents a simple yet effective approach to address identified limitations in tokenization and decoding. While simplicity is at its core, its effectiveness in addressing complex challenges cannot be understated.\n\n3. Extensive evlauation: We've conducted a comprehensive set of experiments to underscore the superiority of SimSGT.\n\nWe will make these points clearly in the revision. We kindly urge the reviewer to re-assess our contributions to molecule SSL.                            \n\n> **Q2.** The technical contributions of this work are less and somewhat incremental. The proposed methods are all based on the current MGM framework and only change a little about the tokenizer and remask strategy ...\n\n**Response:** Thanks. Our focus is on **rethinking** MGM design choices for molecules. We acknowledge that most individual components are from previous works, although our implementation can be different. Moreover, the main contribution of this work is not any single component, but the evaluation and the unique composition of these components. Through comparisons in Table3, Table4, Figure5, and Figure6, we demonstrate the advantages of our particular design choices.\n\n> **Q3.** In Table3(b), when the depth of tokenizer is 4 or 5, the performance of proposed tokenizer is even worse than previous tokenizers. \n\n**Response:** Thank you for noting the performance at specific depths in Table3(b).\n\nThe tokenizer's depth is a hyperparameter, and SGT's effectiveness should be judged by its maximum or mean performances. In Table3(b), SGT shows the highest max and mean performances. Regarding the different behavior between pretrained GNN-based tokenizers and SGT concerning depth, we have a discussion in the response to Q4 of Reviewer Qgfd.\n\n> **Q4.** The comparison of Table5 and Table6 is somehow unfair. The GNN encoders or decoders of baselines may be different from this work ...\n\n**Response:** We agree that a fair comparison is important, and we have taken measures to ensure fairness:\n\n* **Encoder.** To ensure fair comparison, we have 1) categorized methods according to encoder types in Table5; 2) reported the performance of SimSGT, GINE, which uses the same encoder as the baselines, in Table6.\n* **Decoder.** Recall that, we have compared MGM baselines using our proposed decoder and re-masking strategy in Table3(b). In Tables5 and 6, we deliberately retain the original decoders of MGM baselines. Altering this would introduce multiple variables to control, thereby complicating the comparison. By keeping the original design, we maintain a clear comparison.\n\n> **Q5.** There’s no theoretical analysis ... Authors are encouraged to do more experiments or do some theoretical analysis about why the proposed method work well than others.\n\n**Response:** We agree that theories can bring insights to research. However, we respectfully argue that empirical studies offer equally valuable insights.\n\nIn this submission, we provide extensive experiments to validate our findings and designs in both decoders and tokenizers. For examples:\n\n* In Section 4.1, our experiments reveal that “a sufficiently expressive decoder with remask decoding is crucial for MGM”, which is not recognized by previous MGM works; \n* In Section 4.2, our experiments reveal a surprisingly simple but effective method for molecule tokenization: “Single-layer SGT outperforms or matches other tokenizers.”\n\nThrough sharing these findings, we hope to provide insights that may be valuable to other researchers. We welcome further discussion on this subject.\n\n>  **Q6.** In experiments, the authors use ZINC15 as the pretrained dataset ... Will the proposed method still perform well in different size of pretrained dataset? ...\n\n**Response:** Thanks for the advice about more datasets. We have indeed used different datasets for pretraining: ZINC15 (Table 5, 50 thousand molecules) and GEOM (Table 6, 2 million molecules). Note that, these datasets meet the proposed size requirement. On both datasets, we observe that SimSGT shows improvements when pretrained. \n\n**Reference:**\n\n[1] Masked Autoencoders Are Scalable Vision Learners. In CVPR 2022.\n\n[2] Masked autoencoders as spatiotemporal learners. In NeurIPS 2022.\n"}, {"review_id": "ciCDRjxzAP", "reviewer": "Reviewer_tR9n", "summary": "The paper examines the effectiveness of tokenizer and decoder in the self-supervised representation learning of molecular graph following masked auto-encoding framework. Specifically, the paper adopts GraphTrans architecture for its encoder and a smaller GraphTrans for its decoder. A simple GNN-based architecture serves as tokenizer in the framework. The proposed method is pre-trained on 2 millions molecules and is evaluated using 8 classification datasets. The ablation analysis is conducted for the proposed architecture.", "questions": "-  My main concern is about the reported performance which may cause unfair comparison.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The idea of improving tokenizer and decoder in self-supervised learning framework is well-motivated.", "weaknesses": "- The proposed method utilizes a simple GGN-based architecture to learn the feature embeddings. However, these feature embeddings only serve as target for masked auto-encoding. I think it is misleading to call it tokenizer.\n- The performance of GraphMAE [1] and Mole-BERT [2] in Table 5 is lower than one reported in the original paper.\n\n[1] Graphmae: Self-supervised masked graph autoencoders. Hou et al. In KDD 2022.\n[2]  Mole-BERT: Rethinking pre-training graph neural networks for molecules. Xia et al. In ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1.** The proposed method utilizes a simple GGN-based architecture to learn the feature embeddings. However, these feature embeddings only serve as target for masked auto-encoding. I think it is misleading to call it tokenizer.\n\n**Response:** Thank you for your insightful comment on the module names used in our work. We concur that the term \"tokenizer\" traditionally refers to the module defining both input and target in NLP tasks. However, prior research in computer vision and molecular studies [1,2,3] also utilizes the term \"tokenizer\" for modules that solely specify the reconstruction targets. Hence, we chose to align our terminology with these works for consistency.\n\n\n\n>  **Q2.** The performance of GraphMAE and Mole-BERT in Table 5 is lower than one reported in the original paper.\n\n**Response:** Thank you for bringing our attention to the performance metrics. Following your suggestions, we have updated the reported performance of GraphMAE and Mole-BERT in Table 1 of the newly uploaded pdf, based on the original scores in their respective papers. Despite these adjustments, SimSGT still outperforms all the baselines w.r.t. average performance. \n\nBesides, we would like to elucidate our decision to reproduce the baselines. Our primary objective was to ensure a fair and controlled comparison. Given the potential variances that can arise due to software differences, hardware variations, or even random seed discrepancies, direct reuse of reported results from other papers might introduce unintended biases. By reproducing the baselines in the same environment and under identical conditions as our proposed method, SimSGT, we aimed to minimize these external influences and offer a more genuine head-to-head comparison. Now, we keep both the reproduced and original performance.\n\nHere we also try to interpret the discrepancies between reproduced and original scores. The inconsistency between our reproduced results and the original scores can be partially attributed to numerical precision in computation. This issue is common when using GNNs because the scatter operation is non-deterministic in PyTorch. Some relevant discussions can be found by searching “reproducible scatter random” under the issue page of PyG’s github repo.\n\n**Reference:**\n\n[1] BEiT: BERT Pre-Training of Image Transformers. In ICLR 2022.\n\n[2] BEIT V2: Masked Image Modeling with Vector-Quantized Visual Tokenizers. In arxiv 2022.\n\n[3] An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling. In CVPR 2023\n\n[2] Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules. In ICLR 2023.\n"}, {"review_id": "SzJlYLYCCV", "reviewer": "Reviewer_Qgfd", "summary": "In this work, the authors investigate tokenization and decoder in masked graph modeling (MGM), and present Simple GNN-based Tokenizer (SGT) as well as stacking GINE and GraphTrans (GTS) as an efficient decoder. SGT uses non-trainable linear graph aggregation to get tokenization of graphs. It also introduces a variant of remask for the encoder and decoder of GTS. Experiments comparing to other SSL baselines show the proposed method achieves superior performance on multiple molecular property benchmarks. Comprehensive ablations studies also demonstrated the effectiveness of SGT as well as GTS in molecular representation learning compared to other design choices in MGM. ", "questions": "Major questions: \n1. SGT applies only linear aggregation operations to graphs without nonlinear update functions. What will happen if nonlinearity is included on SGT?\n2. In Finding 2, the authors write \"it indicates that remask constrains the encoder’s efforts on graph reconstruction, allowing it to focus on MRL\" which suggests an interesting mismatch between graph reconstruction and MRL. Can the authors discuss more about why the mismatch exists and how it provides insights into designing self-supervised learning framework for MRL. \n3. Table 3b suggests that pretrained GNN tokenizer benefits from depth, however, the proposed SGT achieves the best performance with one single layer. What could be the cause of such observation?\n4. The work benchmarks on multiple MoleculeNet and DTA tasks. There are still important quantum-mechanics-related benchmarks (e.g., QM9, MD17) that are not included. Though the proposed method built upon 2D graphs is unlikely to compete with SOTA equivariant GNNs that leverage 3D information. Benchmarking against other SSL baselines on QM benchmarks provides a more comprehensive evaluation. \n5. In Table 1, why are Mole-BERT and GraphMAE included as baselines in GINE encoder but not in GTS encoder setting?\n\nMinor questions:\n1. In section 2.1, the authors mention using mean pooling to obtain representation of the subgraph. Some works also integrate max or summation pooling. Will that affect the performance in molecular property predictions?\n2. From my perspective, the authors explain too many details regarding BRICS in section 2.2. Yet BRICS is not the focus of the work. The authors may consider truncating the paragraphs. ", "limitations": "The authors have included reasonable discussions regarding potential limitations in the supplementary material. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow overall. \n2. Systematic study of tokenization and decoder can be a valuable contribution to MGM for molecules. Since multiple works have applied masked modeling in molecular representation learning but few have extensively studied the design space of the MGM. \n3. The paper includes comprehensive investigations of MGM design choices in section 4, which validates the effectiveness of the proposed SGT and decoder. \n4. Technical details as well as complete results are reported in supplementary material, demonstrating the experimental soundness. ", "weaknesses": "From my perspective, the major weakness is the miss of quantum-mechanics-related benchmarks. QM-related properties play an important role in applications. Thus performance on QM benchmarks can further help evaluate the effectiveness of the proposed method. \n\nPlease find more details in the \"Questions\" below. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  **Q1.** The performance on the quantum mechanics benchmarks, e.g., QM9, MD17.\n\n**Response:** Thanks. We have included the results on the QM7, QM8, and QM9 datasets in our updated Table 3. This experiment reuses the model checkpoints and settings in Table 5. We observe that SimSGT consistently outperforms representative baselines of GraphCL, GraphMAE, and Mole-BERT. \n\nNote that, our results on the QM9 differ in scale from Uni-Mol[7]. This is because of a different setting: we use all the 12 tasks in QM9, while Uni-Mol uses 3 tasks. \n\n>  **Q2.** SGT applies only linear aggregation operations to graphs without nonlinear update functions. What will happen if nonlinearity is included on SGT? \n\n**Response:** Thanks for the question. If nonlinear update functions (both nonlinear activation function and weights) are included, two potential scenarios arise:\n\n* **GNN tokenizer is pretrained.** This is the same as the pretrained GNN-based tokenizer reported in Table 3 (b).\n\n* **GNN tokenizer is not pretrained.** This will lead to failed MGM pretraining. To show this, let’s further consider two cases:\n\n\t*  **Stop-gradient is used.** The un-pretrained GNN tokenizer, having random weights, will likely generate low-quality tokens,  leading to failed pretraining.\n\n  *  **Stop-gradient is not used.** This will lead to representation collapse, as discussed in [1,2]. The tokenizer branch and autoencoder branch will output constant values to minimize the loss, but constant values are not meaningful representations.\n\n>  **Q3.** More discussions on why the mismatch between graph reconstruction and MRL exists and how it provides insights into designing SSL framework for MRL.\n\n**Response:** The mismatch exists because the graph reconstruction objective demands the graph autoencoder’s last layer to output raw features (*e.g.*, nodes/edge features). However, these raw features are not optimal for graph prediction tasks, the primary goal of MRL.\n\nTo deal with the mismatch, previous works in CV [3,4] have noticed that the autoencoder’s encoder outputs are representations of high-level semantics, when 1) pairing with a sufficiently expressive decoder and 2) excluding the masked elements in encoder. These masked elements are only included in the decoder, to prevent the encoder from learning to reconstruct raw features.\n\nTo adapt these insights for MRL, we use a sufficiently expressive decoder in a similar manner. However, excluding masked nodes in encoder is nontrivial because it can easily corrupt the graph structure. This leads to significant performance drops in our preliminary experiments. To avoid corrupting structures, we decide to: 1) retain the masked nodes in the encoder’s GNN layers, which explicitly use graph structures; and 2) only exclude masked nodes in the encoder’s Transformer layers, which do not use graph structures. \n\n> **Q4.** Why pretrained GNN tokenizer benefits from depth, but SGT does not?\n\n**Response:** Thanks for the question regarding the differing behavior between the pretrained GNN tokenizer and SGT. We present our explanations below:\n\n**Pretrained GNN Tokenizers.** The performance of pretrained GNN-based tokenizers relies heavily on the pretraining process. As evidenced in Table 3(b), changing the pretraining methods, such as from GraphMAE to GraphCL, leads to noticeable difference in performance. Additionally, since the pretraining methods we examined were designed for deep-layer GNNs, these tokenizers naturally perform best with increased layers. The pretrained weights seem to alleviate over-smoothing issues that could otherwise arise with depth.\n\n**SGTs.** SGTs are not affected by pretraining. Their performance depends solely on the graph operator used. We conjecture that the decrease in performance seen with added depth in SGT is due to the over-smoothing effect that is often found in deep GNNs. Nevertheless, it's essential to emphasize that despite facing the over-smoothing issue, SGT demonstrates better or comparable performancse among all compared tokenizers. \n\nThanks for the insightful observation. We will include this discussion in the limitation section in our revised submission.\n\n>  **Q5.** Missing baselines of running Mole-BERT and GraphMAE with the GTS encdoer.\n\n**Response:** Thanks for the suggestion. We have now included Mole-BERT and GraphMAE results when using GTS encoder in our newly uploaded PDF file. As Table 1 shows, SimSGT maintains better performances on average. \n\nOur initial submission did not include these results due to time constraints for reproducing the baselines. \n\n> **Q6.** The authors mention using mean pooling to obtain subgraph representations. Will using max/sum pooling affect the performance?\n\n**Response:** Thank you for the suggestion. We use mean pooling to follow the method of obtaining graph representations in [5,6]. We agree that different pooling strategies may impact performance. Hence, we add results for MGSSL tokenizer using sum and max pooling in our updated Table 4. The results show that mean pooling yields the highest performance, affirming the soundness of our earlier experiments.\n\n>  **Q7.** The authors explain too many details regarding BRICS in section 2.2.\n\n**Response:** Thank you for your suggestion. We acknowledge that the explanation of BRICS may be excessive. We will move some details to Appendix, allowing us have a more focused narrative in the main paper.\n\n\n\n**Reference:**\n\n[1] Exploring Simple Siamese Representation Learning. In CVPR 2021.\n\n[2] Understanding Self-Supervised Learning Dynamics without Contrastive Pairs. In ICML 2021.\n\n[3] Masked autoencoders as spatiotemporal learners. In NeurIPS 2022.\n\n[4] Masked Autoencoders Are Scalable Vision Learners. In CVPR 2022.\n\n[5] Graph Contrastive Learning with Augmentations. In NeurIPS 2021.\n\n[6] Strategies for Pre-training Graph Neural Networks. In ICLR 2020.\n\n[7] UNI-MOL: A UNIVERSAL 3D MOLECULAR REPRESENTATION LEARNING FRAMEWORK. In ICLR 2023\n"}]

Author rebuttal:
We appreciate all the reviewers' efforts for reviewing this submission. Our submission has received diverse ratings, including one strong accept (8), one weak accept (6), one borderline accept (5), one borderline reject (4), and one reject (3). 

We would like to thank all the reviewers for providing insightful comments and valuable suggestions. In resposne to the reviewer's request, we have uploaded a PDF file with updated experimental results. The updated experimental results include:

* **[Reviewer Qgfd, tR9n]:** The performances of Mole-BERT and GraphMAE when using the GraphTrans encoder, and their performances reported in the original paper. Present in new Table 1.

* **[Reviewer 9pKQ]:** The performances of two new baselines: S2GAE[1] and GraphMAE2[2]. Present in new Table 1.

* **[Reviewer Qgfd]:** Performances on a new downstream dataset of quantum mechanics: QM9 [3]. Present in new Table 3.

* **[Reviewer 9pKQ]:** The comparison of computational time. Present in new Table 2.

* **[Reviewer Qgfd]:** Testing motif-based tokenizer's performances when using max and sum pooling. Present in new Table 4.



Here we also present the response to one common question regarding the difference between our proposed Simple GNN-based Tokenizer (SGT) and the pretrained GNN-based tokenizers.



> **Q1. The proposed tokenizer and the pretrained GNN-based tokenizer**
>
> * **[Reviewer Lzf1].** What are the real differences between the proposed tokenizer and the pretrained GNN-based tokenizer? Can the proposed method be considered as a special case of the previous GNN-based tokenizer? ...
>
> *  **[Reviewer** **9pKQ].** It remains unclear why removing the nonlinear update function in each GNN layer can train a better encoder.



**Response:** Thanks for your valuable insights. We've summarized the key technical distinctions between our SGTs and pretrained GNN tokenizers as follows:

1. **Nonlinearity Elimination.** Once nonlinear update functions are removed, SGT essentially is a linear combination of graph operators, while pretrained GNN tokenizers mostly adopt nonlinear graph convolution layers. Linear GNNs have been studied in previous works [5,6,7]. [5] theoretically show that simple GCN operators function as low-pass filters in the graph spectral domain. [6] proves that linear GNNs are universal approximators under some mild conditions. Additionally, [5,6] show that GNNs without nonlinearity can perform comparablely to conventional GNNs.

2. **Nonparametric & Non-trainable Nature.** SGT is a linear combination of graph operators without any trainable parameters, while the GNN tokenizers usually hold trainable parameters to optimize. These parameter- and training-tree characters make SGT more efficient and simpler to adopt, faciliating the subsequent pretraining of graph encoder.

3. **Nonlinearities in Conventional GNNs.** Conventional GNNs include nonlinear update functions to enhance the **potential** expressiveness [4]. However, the **real-world** expressiveness and performances of GNNs hinge heavily on the pretraining method and data. This is shown by results in Table3(b): changing the pretraining methods, such as from GraphMAE to GraphCL, leads to noticeable difference in performance. 



**Why pretrained GNN-based tokenizers do not outperform SGT?**

We confer that a discrepancy exists between existing molecule pretraining methods and the objectives of molecule tokenizers. This is backed by the observation that pretraining method can largely influence the performances of pretrained GNN-based tokenizers: in Table 3 (b), changing the pretraining methods, such as from GraphMAE to GraphCL, leads to noticeable difference in performance. Indeed, only Group VQ-VAE [8] in the existing pretraining methods are designed for molecule tokenization, and it might need more exploration to define a "good" molecule tokenizer. This work can be part of this exploration.

At the current stage, considering that pretrained GNN-based tokenizers do not outperform the SGT, we chooce SGT in our framework, which also eliminates the need for costly pretraining.

**Reference:**

[1] S2GAE: Self-Supervised Graph Autoencoders are Generalizable Learners with Graph Masking. In WSDM 2023.

[2] GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner. In WWW 2023.

[3] Quantum chemistry structures and properties of 134 kilo molecules. In Scientific Data 2014.

[4] How Powerful are Graph Neural Networks? In ICLR 2019.

[5] Simplifying Graph Convolutional Networks. In ICML 2019.

[6] How Powerful are Spectral Graph Neural Networks. In ICML 2022.

[7] On graph neural networks versus graph-augmented mlps. In ICLR 2021.

[8] MOLE-BERT: RETHINKING PRE-TRAINING GRAPH NEURAL NETWORKS FOR MOLECULES. In ICLR 2023.

Meta-review: Post rebuttal, all the reviewers agree on acceptance with their main concerns resolved. The AC checked all the materials and concurs that the paper has made a valuable contribution to the "masked-modeling" idea in molecular representation learning, calling for the community's attention to tokenizer and decoder beyond previous explorations. Therefore recommending acceptance. Please incorporate necessary changes in the final version.
Decision: Accept (poster)