SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiyuan Liu† Yaorui Shi‡ An Zhang† Enzhi Zhang§ Kenji Kawaguchi† Xiang Wang‡* Tat-Seng Chua†
†National University of Singapore, ‡University of Science and Technology of China §Hokkaido University
{acharkq,shiyaorui,xiangwang1223}@gmail.com,anzhang@u.nus.edu enzhi.zhang.n6@elms.hokudai.ac.jp,{kenji,chuats} @comp.nus.edu.sg
Masked graph modeling excels in the self-supervised representation learning of molecular graphs. Scrutinizing previous studies, we can reveal a common scheme consisting of three key components: (1) graph tokenizer, which breaks a molecular graph into smaller fragments (i.e., subgraphs) and converts them into tokens; (2) graph masking, which corrupts the graph with masks; (3) graph autoencoder, which first applies an encoder on the masked graph to generate the representations, and then employs a decoder on the representations to recover the tokens of the original graph. However, the previous MGM studies focus extensively on graph masking and encoder, while there is limited understanding of tokenizer and decoder. To bridge the gap, we first summarize popular molecule tokenizers at the granularity of node, edge, motif, and Graph Neural Networks (GNNs), and then examine their roles as the MGM's reconstruction targets. Further, we explore the potential of adopting an expressive decoder in MGM. Our results show that a subgraph-level tokenizer and a sufficiently expressive decoder with remask decoding have a large impact on the encoder's representation learning. Finally, we propose a novel MGM method SimSGT, featuring a Simple GNN-based Tokenizer (SGT) and an effective decoding strategy. We empirically validate that our method outperforms the existing molecule self-supervised learning methods. Our codes and checkpoints are available at https://github.com/syr-cn/SimSGT.
Molecular representation learning (MRL) [1, 2, 3] is a critical research area with numerous vital downstream applications, such as molecular property prediction [4], drug discovery [5, 6], and retrosynthesis [7, 8]. Given that molecules can be represented as graphs, graph self-supervised learning (SSL) is a natural fit for this problem. Among the various graph SSL techniques, Masked Graph Modeling (MGM) has recently garnered significant interest [9, 10, 11].
In this paper, we study MRL through MGM, aiming to pretrain a molecule encoder for subsequent fine-tuning in downstream applications. After looking at the masked modeling methods in graph [12, 9, 10], language [13, 14], and computer vision [15, 16], we summarize that MGM relies on three key components - graph tokenizer, graph masking, and graph autoencoder, as Figure 1 shows:
- Graph tokenizer. Given a graph  $g$ , the graph tokenizer employs a graph fragmentation function [1, 17, 18, 2] to break  $g$  into smaller subgraphs, such as nodes and motifs. Then, these
*Corresponding author. Xiang Wang is also affiliated with Institute of Artificial Intelligence, Institute of Dataspace, Hefei Comprehensive National Science Center.
Figure 1: The pipeline of Masked Graph Modeling.
Figure 2: Example of subgraphlevel patterns in a molecule. SMILES: CC(=O)Nc1cccc(O)c1.
fragments are mapped into fixed-length tokens to serve as the targets being reconstructed later. Clearly, the granularity of graph tokens determines the abstraction level of representations in masked modeling [19, 16]. This is especially relevant for molecules, whose properties are largely determined by patterns at the granularity of subgraphs [20]. For example, the molecule shown in Figure 2 contains a benzene ring subgraph. Benzene ring confers the molecule aromaticity, making it more stable than saturated compounds that only have single bonds [21]. Therefore, applying graph tokenizers that generate subgraph-level tokens might improve the downstream performances.
- Graph masking. Before feeding into the autoencoder,  $g$  is corrupted by adding random noise, typically through randomly masking nodes or dropping edges [12, 11]. Graph masking is crucial to prevent the autoencoder from merely copying the input, and guide the autoencoder to learn the relationships between co-occurring graph patterns.
- Graph autoencoder. Graph autoencoder consists of a graph encoder and a graph decoder [12, 9]. The graph encoder generates the corrupted graph's hidden representations, based on which the graph decoder attempts to recover the corrupted information. The encoder and the decoder are jointly optimized by minimizing the distance between the decoder's outputs and the reconstruction targets, i.e., the graph tokens induced by the graph tokenizer. Given the complex subgraph-level tokens as targets, an effective reconstruction might demand a sufficiently expressive graph decoder.
Although all three components mentioned above are crucial, previous MGM studies primarily focus on graph masking [12, 11, 22, 23] and graph encoder [2, 24, 25, 26], with less emphasis on the tokenizers and decoders. For example, while there exist extensive motif-based fragmentation functions for MRL [1, 17, 18, 2], they have been overlooked as tokenizers for MGM. Moreover, many previous works [12, 10, 11, 22, 23, 2, 25] employ a linear or MLP decoder for graph reconstruction, leaving more expressive decoders largely unexplored.
In this work, we first summarize the various fragmentation functions as graph tokenizers, at the granularity of nodes, edges, motifs, and Graph Neural Networks (GNNs). Given this summary, we systematically evaluate their empirical performances for MGM. Our analysis shows that reconstructing subgraph-level tokens in MGM can improve over the node tokens. Moreover, we find that a sufficiently expressive decoder combined with remask decoding [9] could improve the encoder's representation quality. Notably, remask "decouples" the encoder and decoder, redirecting the encoder's focus away from molecule reconstruction and more towards MRL, leading to better downstream performances. In summary, we reveal that incorporating a subgraph-level tokenizer and a sufficiently expressive decoder with remask decoding gives rise to improved MGM performance.
Based on the findings above, we propose a novel pretraining framework - Masked Graph Modeling with a Simple GNN-based Tokenizer (SimSGT). SimSGT employs a Simple GNN-based Tokenizer (SGT) that removes the nonlinear update function in each GNN layer. Surprisingly, we show that a single-layer SGT demonstrates competitive or better performances compared to other pretrained GNN-based and chemistry-inspired tokenizers. SimSGT adopts the GraphTrans [27] architecture for its encoder and a smaller GraphTrans for its decoder, in order to provide sufficient capacity for both the tasks of MRL and molecule reconstruction. Furthermore, we propose remask-v2 to decouple the encoder and decoder of the GraphTrans architecture. Finally, SimSGT is validated on downstream molecular property prediction and drug-target affinity tasks [28, 29], surpassing the leading graph SSL methods (e.g., GraphMAE [9] and Mole-BERT [10]).
In this section, we begin with the introduction of MGM. Then, we provide a categorization of existing graph tokenizers. Finally, we discuss the architecture of graph autoencoders for MGM.
Notations. Let  $\mathcal{G}$  denote the space of graphs. A molecule can be represented as a graph  $g = (\mathcal{V},\mathcal{E})\in$ $\mathcal{G}$ , where  $\mathcal{V}$  is the set of nodes and  $\mathcal{E}$  is the set of edges. Each node  $i\in \mathcal{V}$  is associated with a node feature  $\pmb {x}_i\in \mathbb{R}^{d_0}$  and each edge  $(i,j)\in \mathcal{E}$  is associated with an edge feature  $\pmb {e}_{ij}\in \mathbb{R}^{d_1}$ . The graph  $g$  's structure can also be represented by its adjacency matrix  $\mathbf{A}\in \{0,1\}^{|\mathcal{V}|\times |\mathcal{V}|}$ , such that  $\mathbf{A}_{ij} = 1$  if  $(i,j)\in \mathcal{E}$  and  $\mathbf{A}_{ij} = 0$  otherwise.
2.1 Preliminary:Masked Graph Modeling
Here we illustrate MGM's three key steps: graph tokenizer, graph masking, and graph autoencoder.
Graph tokenizer. Given a graph  $g$ , we leverage a graph tokenizer  $tok(g) = \{ \mathbf{y}_t = m(t) \in \mathbb{R}^d | t \in f(g) \}$  to generate its graph tokens as the reconstruction targets. The tokenizer  $tok(\cdot)$  is composed of a fragmentation function  $f$  that breaks  $g$  into a set of subgraphs  $f(g) = \{ t = (\mathcal{V}_t, \mathcal{E}_t) | t \subseteq g \}$ , and a mapping function  $m(t) \in \mathbb{R}^d$  that transforms the subgraphs into fixed-length vectors. In this work, we allow  $f(g)$  to include overlapped subgraphs to enlarge the scope of graph tokenizers.
Graph masking. Further, we add noises to  $g$  by random node masking. Here we do not use edge dropping because Hou et al. [9] empirically show that edge dropping easily leads to performance drop in downstream tasks. Specifically, node masking samples a random subset of nodes  $\mathcal{V}_m \subseteq \mathcal{V}$  and replaces their features with a special token  $\mathbf{m}_0 \in \mathbb{R}^{d_0}$ . We denote the masked node feature by  $\tilde{\mathbf{x}}_i$ :
$$
\tilde {\mathbf {x}} _ {i} = \left\{ \begin{array}{l l} \mathbf {m} _ {0}, & \forall i \in \mathcal {V} _ {m} \\ \mathbf {x} _ {i}, & \text {o t h e r w i s e} \end{array} . \right. \tag {1}
$$
Graph autoencoder. The corrupted graph  $\tilde{g}$  is then fed into a graph autoencoder for graph reconstruction. We defer the details of the graph autoencoder's architecture to Section 2.3. Let  $\{\mathbf{z}_i|i\in \mathcal{V}\}$  be the node-wise outputs of the graph autoencoder. We obtain subgraph  $t$  's prediction  $\hat{\mathbf{y}}_t = \mathrm{MEAN}(\{\mathbf{z}_i|i\in \mathcal{V}_t\})$  by mean pooling the representations of its nodes, if not especially noted. The graph autoencoder is trained by minimizing the distance between the predictions  $\{\hat{\mathbf{y}}_t|t\in f(g)\}$  and the targets  $tok(g) = \{\mathbf{y}_t|t\in f(g)\}$ . The reconstruction loss is accumulated on tokens that include corrupted information  $\{t|t\in f(g),\mathcal{V}_t\cap \mathcal{V}_m\neq \emptyset \}$ :
$$
L _ {0} = \frac {1}{| f (g) |} \sum_ {t \in f (g), \mathcal {V} _ {t} \cap \mathcal {V} _ {m} \neq \emptyset} \ell \left(\hat {\mathbf {y}} _ {t}, \mathbf {y} _ {t}\right), \tag {2}
$$
where  $\ell (\cdot ,\cdot)$  is the loss function, dependent on the type of  $\mathbf{y}_t$ . We use mean square error [15] for  $\ell (\cdot ,\cdot)$  when  $\mathbf{y}_t$  is a continuous vector, and use cross-entropy [16] for  $\ell (\cdot ,\cdot)$  when  $\mathbf{y}_t$  is a discrete value.
2.2 Revisiting Molecule Tokenizers
Scrutinizing the current MRL methods, we summarize the molecule tokenizers into four distinct categories, as Table 1 shows. A detailed description of the first three categories is systematically provided here, while the Simple GNN-based Tokenizer is introduced in Section 3.
Table 1: Summary of graph tokenizers.
Tokenizer	Subgraph types	Tokens	Potential limitations
Node, edge	Nodes and edges	Features of nodes and edges	Low-level feature
Motif	FGs, cycles, etc.	Motif types	Rely on expert knowledge
Pretrained GNN	Rooted subtrees	Frozen GNN representations	Extra pretraining for tokenizer
Simple GNN	Rooted subtrees	Frozen GNN representations	-
Node, edge tokenizer [12, 11]. A graph's nodes and edges can be used as graph tokens directly:
$$
\operatorname {t o k} _ {\text {n o d e}} (g) = \left\{\mathbf {y} _ {i} = \boldsymbol {x} _ {i} | i \in \mathcal {V} \right\}, \quad \operatorname {t o k} _ {\text {e d g e}} (g) = \left\{\mathbf {y} _ {i j} = \boldsymbol {e} _ {i j} | (i, j) \in \mathcal {E} \right\}. \tag {3}
$$
Figure 3a illustrates the use of atomic numbers of nodes and bond types of edges as graph tokens in a molecule. These have been widely applied in previous research [12, 11, 25, 24], largely due to their
(a) Node, edge tokenizer
(b) Motif-based tokenizer
(c) Pretrained GNN-based tokenizer
Figure 3: Examples for the first three types of graph tokenizers and their induced subgraphs. (b) A motif-based tokenizer that applies the fragmentation functions of cycles and the remaining nodes. (c) A two-layer GIN-based tokenizer that extracts 2-hop rooted subtrees for every node in the graph.
simplicity. However, atomic numbers and bond types are low-level features. Reconstructing them may be suboptimal for downstream tasks that require a high-level understanding of graph semantics.
Motif-based tokenizer. Motifs are statistically significant subgraph patterns in graph structures. For molecules, functional groups (FGs) are motifs that are manually curated by experts based on the FGs' biochemical characteristics [30, 31]. For example, molecules that contain benzene rings exhibit the property of aromaticity. Considering that the manually curated FGs are limited and cannot fully cover all molecules, previous works [32, 1, 18] employ chemistry-inspired fragmentation functions for motif discovery. Here we summarize the commonly used fragmentation functions:
- FGs [18, 2]. An FG is a molecule's subgraph that exhibits consistent chemical behaviors across various compounds. In chemistry toolkits [30, 31], the substructural patterns of FGs are described by the SMARTS language [33]. Let  $S_0 = \{s_i\}_{i=1}^{|S_0|}$  be a set of SMARTS patterns for FGs, and let  $p_s(g)$  be the function returning the set of  $s$  FGs in  $g$ . FG-based fragmentation works as:
$$
f _ {\mathrm {F G}} (g, \mathcal {S} _ {0}) = \cup_ {s \in \mathcal {S} _ {0}} p _ {s} (g), \tag {4}
$$
- Cycles [32, 1, 18]. Cycles in molecules are often extracted as motifs due to their potential chemical significance. Figure 3b depicts the process of fragmenting a five-node cycle as a moif. If two cycles overlapped more than two atoms, they can be merged because they constitute a bridged compound [34]. Let  $C_n$  represent a cycle of  $n$  nodes. They can be written as:
$$
f _ {\text {c y c l e}} (g) = \{t | t = C _ {| t |}, t \subseteq g \}, \tag {5}
$$
$$
f _ {\text {c y c l e - m e r g e}} (g) = \left\{t _ {i} \cup t _ {j} \mid t _ {i}, t _ {j} \in f _ {\text {c y c l e}} (g), i \neq j, \left| t _ {i} \cap t _ {j} \right| > 2 \right\}, \tag {6}
$$
- BRICS [35, 1]. BRICS fragments a molecule at the potential cleavage sites, where chemical bonds can be broken under certain environmental conditions or catalysts. The key step of BRICS is to identify a molecule  $g$ 's potential cleavage sites, denoted by  $\psi(g)$ . This is achieved by finding bonds with both sides matching one of the pre-defined "FG-like" substructural patterns  $S_{1}$  in BRICS:
$$
\psi (g) = \cup \left\{\mathcal {E} \backslash \left(\mathcal {E} _ {t} \cup \mathcal {E} _ {g - t}\right) | t, g - t \in f _ {\mathrm {F G}} (g, \mathcal {S} _ {1}) \right\}, \tag {7}
$$
where  $g - t = g[\mathcal{V} \setminus \mathcal{V}_t]$  denotes deleting  $g$ 's nodes in  $t$  and the corresponding incident edges [36];  $\mathcal{E} \setminus (\mathcal{E}_t \cup \mathcal{E}_{g - t})$  contains the bond that connects  $t$  and  $g - t$ . Next,  $g$  is fragmented into the maximum subgraphs that contain no bonds in the cleavage sites  $\psi(g)$ :
$$
f _ {\mathrm {B R I C S}} (g) = \{t | \psi (g) \cap \mathcal {E} _ {t} = \emptyset , f _ {\mathrm {B R I C S}} (t) \cap 2 ^ {t} = \{t \}, t \subseteq g \}. \tag {8}
$$
Note that, the original BRICS includes more rules, such as a constraint on  $t$  and  $g - t$ 's combinations. We present only the key step here for simplicity.
- Remaining nodes and edges [32, 18]: Given another fragmentation function  $f_{0}$ , the nodes and edges that are not included in any of  $f_{0}$ 's outputs are treated as individual subgraphs. This improves  $f_{0}$ 's coverage on the original graph. Figure 3b shows an example of fragmenting remaining nodes after  $f_{\text{cycle}}$ : nodes that are not in any cycles are treated as individual subgraphs.
To obtain more fine-grained subgraphs, previous works [1, 32, 18] usually combine several fragmentation functions together by unions (e.g.,  $f_{1}(g) \cup f_{2}(g)$ ) or compositions (e.g.,  $\{f_{2}(t) | t \in f_{1}(g)\}$ ). Let  $f_{\mathrm{motif}}$  be the final fragmentation function after combination. We break every molecule in the dataset by  $f_{\mathrm{motif}}$  and gather a motif vocabulary  $\mathcal{M}$ , which filters out infrequent motifs by a threshold. Then, given a new molecule  $g'$ , we can generate its tokens by one-hot encoding its motifs  $f_{\mathrm{motif}}(g')$ :
$$
\operatorname {t o k} _ {\text {m o t i f}} \left(g ^ {\prime}\right) = \left\{\mathbf {y} _ {t} = \text {o n e - h o t} (t, \mathcal {M}) \mid t \in f _ {\text {m o t i f}} \left(g ^ {\prime}\right) \right\}. \tag {9}
$$
Pretrained GNN-based tokenizer [10]. Pretrained GNNs can serve as graph tokenizers. Take a  $k$ -layer Graph Isormorphism Network (GIN) [37] as an example. Its node embedding summarizes the structural information of the node's  $k$ -hop rooted subtree, making it a subgraph-level graph token (Figure 3c). A GIN performs the fragmentation and the mapping simultaneously. It can be written as:
$$
t o k _ {\mathrm {G I N}} (g) = \left\{\mathbf {y} _ {i} = \operatorname {S G} \left(\boldsymbol {h} _ {i} ^ {(k)}\right) \mid i \in \mathcal {V} \right\}, \tag {10}
$$
$$
\boldsymbol {h} _ {i} ^ {(k)} = \operatorname {C O M B I N E} ^ {(k)} \left(\boldsymbol {h} _ {i} ^ {(k - 1)}, \operatorname {A G G R E G A T E} ^ {(k)} \left(\left\{\boldsymbol {h} _ {j} ^ {k - 1}, j \in \mathcal {N} (i) \right\}\right)\right), \tag {11}
$$
where AGGREGATE(\cdot) collects information from node  $i$ 's neighbors, based on which COMBINE(\cdot) updates  $i$ 's representation.  $\mathrm{SG}(\cdot)$  denotes stop-gradient, which stops the gradient flow to the tokenizer during MGM pretraining. In addition to GINs, other GNNs can also serve as tokenizers. To obtain meaningful graph tokens, we pretrain a GNN before employing it as a tokenizer [10]. Once pretrained, this GNN is frozen and employed for the subsequent MGM pretraining. In Section 4, we evaluate the prevalent graph pretraining strategies for tokenizers. Given that GNN-based tokenizers provide node-wise tokens, we directly minimize the distance between the graph tokens and the autoencoder's outputs  $\{\mathbf{z}_i\}_{i=1}^{|\mathcal{V}|}$  of the masked nodes  $\mathcal{V}_m$ , i.e.,  $L_0 = \frac{1}{|\mathcal{V}_m|} \sum_{i \in \mathcal{V}_m} \ell(\hat{\mathbf{y}}_i = \mathbf{z}_i, \mathbf{y}_i)$ .
2.3 Revisiting Graph Autoencoders
Background. Graph autoencoder consists of a graph encoder and a graph decoder. We pretrain them with the objective of graph reconstruction. Once well pretrained, the encoder is saved for downstream tasks. MGM works [12, 2] usually adopt expressive graph encoders like GINEs [12] and Graph Transformers [2]. However, the exploration on expressive decoders has been limited. Many previous works [12, 10, 11, 22, 2, 25] apply a linear or an MLP decoder, similar to BERT's design [13].
Table 2: The compared GNN architectures for encoders and decoders.
Model	GINE, dim 300	Transformer, dim 128
Linear	-	-
GINE	5 layer	-
GINE-Small	3 layer	-
GTS	5 layer	4 layer
GTS-Small	3 layer	1 layer
GTS-Tiny	1 layer	1 layer
However, recent studies [15, 38] have revealed the disparity between representation learning and reconstruction tasks. They show that a sufficiently expressive decoder could improve the encoder's representation quality. Delving into these studies, we identify two key elements to improve the representation quality: a sufficiently expressive decoder and remask decoding [9].
Sufficiently expressive decoder. Following [15, 38], we devise smaller versions of the encoder's architecture as decoders. We adopt the GINE [12] and GraphTrans [27] (denoted as GTS) as encoders. GTS stacks transformer layers on top of the GINE layers to improve the ability of modeling global interactions. Table 2 summarizes their different versions that we compare in this work.
Remark decoding [9]. Remask controls the focus of the encoder and the decoder. Let  $\{h_i | i \in \mathcal{V}\}$  be the encoder's node-wise hidden representations for the masked graph. Remask decoding masks the hidden representations of the masked nodes  $\mathcal{V}_m$  again by a special token  $\mathbf{m}_1 \in \mathbb{R}^d$  before feeding them into the decoder. Formally, the remasked hidden representation  $\tilde{\pmb{h}}_i$  is as follows:
$$
\tilde {\boldsymbol {h}} _ {i} = \left\{ \begin{array}{l l} \mathbf {m} _ {1}, & \forall i \in \mathcal {V} _ {m} \\ \boldsymbol {h} _ {i}, & \text {o t h e r w i s e} \end{array} . \right. \tag {12}
$$
Figure 4: Overview of the SimSGT's framework.
Remask constrains the encoder's ability on predicting the corrupted information by removing the encoder's representation on the masked part. The encoder is enforced to generate effective representations for the unmasked part, to provide signals for the decoder for graph reconstruction.
In this section, we present our method - Masked Graph Modeling with a Simple GNN-based Tokenizer (SimSGT) (Figure 4). Specifically, it applies the GTS [27] architecture for both its encoder and decoder. SimSGT features a Simple GNN-based Tokenizer (SGT), and employs a new remask strategy to decouple the encoder and decoder of the GTS architecture.
Simple GNN-based Tokenizer. SGT simplifies existing aggregation-based GNNs [37] by removing the nonlinear update functions in GNN layers. It is inspired by studies showing that carefully designed graph operators can generate effective node representations [39, 40]. Formally, a  $k$ -layer SGT is:
$$
\operatorname {t o k} _ {\mathrm {S G T}} (g) = \left\{\mathbf {y} _ {i} = \mathbf {S G} \left(\left[ \mathbf {H} _ {i} ^ {(1)}, \dots , \mathbf {H} _ {i} ^ {(k)} \right]\right) \mid i \in \mathcal {V} \right\}, \tag {13}
$$
$$
\mathbf {H} _ {i} ^ {(0)} = \operatorname {E m b e d d i n g} \left(\boldsymbol {x} _ {i}\right) \in \mathbb {R} ^ {d}, \quad \forall i \in \mathcal {V}, \tag {14}
$$
$$
\hat {\mathbf {H}} ^ {(l)} = \omega (\mathbf {A}) \cdot \mathbf {H} ^ {(l - 1)} \in \mathbb {R} ^ {| \mathcal {V} | \times d}, \quad 1 \leq l \leq k, \tag {15}
$$
$$
\mathbf {H} ^ {(l)} = \operatorname {B a t c h N o r m} \left(\hat {\mathbf {H}} ^ {(l)}\right), \tag {16}
$$
where Embedding  $(\cdot)$  is a linear layer that uses the weights of the encoder's node embedding function;  $\mathbf{H}_i^{(l)}$  is the  $i$ -th row of  $\mathbf{H}^{(l)}$ ; BatchNorm  $(\cdot)$  is a standard Batch Normalization layer without the trainable scaling and shifting parameters [41]; and  $\omega(\mathbf{A})$  is the graph operator that represents the original GNN's aggregation function. For example, GIN has  $\omega(\mathbf{A}) = \mathbf{A} + (1 + \epsilon)\mathbf{I}$  and GCN has  $\omega(\mathbf{A}) = \tilde{\mathbf{D}}^{-1/2}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-1/2}$ , where  $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$  and  $\tilde{\mathbf{D}}$  is the degree matrix of  $\tilde{\mathbf{A}}$ .
Note that, SGT does not have trainable weights, allowing its deployment without pretraining. Its tokenization ability relies on the graph operator  $\omega(\mathbf{A})$  that summarizes each node's neighbor information. Additionally, we concatenate the outputs of every SGT layer to include multi-scale information. We show in experiments that SGT transforms the original GNN into an effective tokenizer.
Graph autoencoder. SimSGT employs the GTS architecture as the encoder, and a smaller version of GTS (i.e., GTS-Small in Table 2) as the decoder. This architecture follows the asymmetric encoder-decoder design in previous works [15, 38]. Further, we propose a new remask strategy, named remark-v2, to decouple SimSGT's encoder layers and decoder layers,
Remark-v2. Remask-v2 constrains the encoder's ability on predicting the corrupted information by dropping the masked nodes'  $\mathcal{V}_m$  representations before the Transformer layers (Figure 4). After the Transformer layers, we pad special mask tokens  $\mathbf{m}_1 \in \mathbb{R}^d$  to make up the previously dropped nodes' hidden representations. Compared to the original remask, remask-v2 additionally prevents the Transformer layers from processing the masked nodes. It thereby avoids the gap of processing masked nodes in pretraining but not in fine-tuning [15].
(a) Ablating the decoder. The tokenizer is a single-layer SGT of GIN.