Title: Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules
Venue: NeurIPS 2023
Accepted: yes
Keywords: Molecular Representation Learning

Masked Graph Modeling

Graph Tokenizer
Abstract: Masked graph modeling excels in the self-supervised representation learning of molecular graphs. Scrutinizing previous studies, we can reveal a common scheme consisting of three key components: (1) graph tokenizer, which breaks a molecular graph into smaller fragments (\ie subgraphs) and converts them into tokens; (2) graph masking, which corrupts the graph with masks; (3) graph autoencoder, which first applies an encoder on the masked graph to generate the representations, and then employs a decoder on the representations to recover the tokens of the original graph. However, the previous MGM studies focus extensively on graph masking and encoder, while there is limited understanding of tokenizer and decoder. To bridge the gap, we first summarize popular molecule tokenizers at the granularity of node, edge, motif, and Graph Neural Networks (GNNs), and then examine their roles as the MGM's reconstruction targets. Further, we explore the potential of adopting an expressive decoder in MGM. Our results show that a subgraph-level tokenizer and a sufficiently expressive decoder with remask decoding have a \yuan{large impact on the encoder's representation learning}. Finally, we propose a novel MGM method SimSGT, featuring a Simple GNN-based Tokenizer (SGT) and an effective decoding strategy. We empirically validate that our method outperforms the existing molecule self-supervised learning methods. Our codes and checkpoints are available at https://github.com/syr-cn/SimSGT.

Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules

Abstract

Masked graph modeling excels in the self-supervised representation learning of molecular graphs. Scrutinizing previous studies, we can reveal a common scheme consisting of three key components: (1) graph tokenizer, which breaks a molecular graph into smaller fragments ( i.e., subgraphs) and converts them into tokens; (2) graph masking, which corrupts the graph with masks; (3) graph autoencoder, which first applies an encoder on the masked graph to generate the representations, and then employs a decoder on the representations to recover the tokens of the original graph. However, the previous MGM studies focus extensively on graph masking and encoder, while there is limited understanding of tokenizer and decoder. To bridge the gap, we first summarize popular molecule tokenizers at the granularity of node, edge, motif, and Graph Neural Networks (GNNs), and then examine their roles as the MGM's reconstruction targets. Further, we explore the potential of adopting an expressive decoder in MGM. Our results show that a subgraph-level tokenizer and a sufficiently expressive decoder with remask decoding have a large impact on the encoder's representation learning. Finally, we propose a novel MGMmethod SimSGT , featuring a S imple G NN-based T okenizer ( SGT ) and an effective decoding strategy. We empirically validate that our method outperforms the existing molecule self-supervised learning methods. Our codes and checkpoints are available at https://github.com/syr-cn/SimSGT .

1 Introduction

Molecular representation learning (MRL) [1, 2, 3] is a critical research area with numerous vital downstream applications, such as molecular property prediction [4], drug discovery [5, 6], and retrosynthesis [7, 8]. Given that molecules can be represented as graphs, graph self-supervised learning (SSL) is a natural fit for this problem. Among the various graph SSL techniques, Masked Graph Modeling (MGM) has recently garnered significant interest [9, 10, 11].

In this paper, we study MRL through MGM, aiming to pretrain a molecule encoder for subsequent finetuning in downstream applications. After looking at the masked modeling methods in graph [12, 9, 10], language [13, 14], and computer vision [15, 16], we summarize that MGM relies on three key components - graph tokenizer, graph masking, and graph autoencoder, as Figure 1 shows:

Graph tokenizer. Given a graph g , the graph tokenizer employs a graph fragmentation function [1, 17, 18, 2] to break g into smaller subgraphs, such as nodes and motifs. Then, these

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

<latexi sh

1_b

64="VwW7GA

S

FYQ

XP

9u

o

>

B8

c

DL

gM

T

+q

d

Iv

y

NU

E

KJ

20

m

C

f

/

n

H

z

r

jR

k

Zp

3

5

O

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

<latexi sh

1_b

64="VwW7GA

S

FYQ

XP

9u

o

>

B8

c

DL

gM

T

+q

d

Iv

y

NU

E

KJ

20

m

C

f

/

n

H

z

r

jR

k

Zp

3

5

O

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="VwW7GA

S

FYQ

XP

9u

o

>

B8

c

DL

gM

T

+q

d

Iv

y

NU

E

KJ

20

m

C

f

/

n

H

z

r

jR

k

Zp

3

5

O

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="VwW7GA

S

FYQ

XP

9u

o

>

B8

c

DL

gM

T

+q

d

Iv

y

NU

E

KJ

20

m

C

f

/

n

H

z

r

jR

k

Zp

3

5

O

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

Graph

Encoder

Reconstruction targets

Tokenizer

Graph masking

C

C

C

C

N

N

F

F

C

C

N

F

Graph

Decoder

Inputs

Fragments

N

N

C

F

F

C

C

C

Tokens

M

M

M

M

Autoencoder

Mapping

C

C

C

C

C

C

C

C

N

H

O

O

H

Benzene

ring

Hydroxyl

group

Carbonyl

group

fragments are mapped into fixed-length tokens to serve as the targets being reconstructed later. Clearly, the granularity of graph tokens determines the abstraction level of representations in masked modeling [19, 16]. This is especially relevant for molecules, whose properties are largely determined by patterns at the granularity of subgraphs [20]. For example, the molecule shown in Figure 2 contains a benzene ring subgraph. Benzene ring confers the molecule aromaticity, making it more stable than saturated compounds that only have single bonds [21]. Therefore, applying graph tokenizers that generate subgraph-level tokens might improve the downstream performances.

Graph masking. Before feeding into the autoencoder, g is corrupted by adding random noise, typically through randomly masking nodes or dropping edges [12, 11]. Graph masking is crucial to prevent the autoencoder from merely copying the input, and guide the autoencoder to learn the relationships between co-occurring graph patterns.

Graph autoencoder. Graph autoencoder consists of a graph encoder and a graph decoder [12, 9]. The graph encoder generates the corrupted graph's hidden representations, based on which the graph decoder attempts to recover the corrupted information. The encoder and the decoder are jointly optimized by minimizing the distance between the decoder's outputs and the reconstruction targets, i.e., the graph tokens induced by the graph tokenizer. Given the complex subgraph-level tokens as targets, an effective reconstruction might demand a sufficiently expressive graph decoder.

Although all three components mentioned above are crucial, previous MGM studies primarily focus on graph masking [12, 11, 22, 23] and graph encoder [2, 24, 25, 26], with less emphasis on the tokenizers and decoders. For example, while there exist extensive motif-based fragmentation functions for MRL [1, 17, 18, 2], they have been overlooked as tokenizers for MGM. Moreover, many previous works [12, 10, 11, 22, 23, 2, 25] employ a linear or MLP decoder for graph reconstruction, leaving more expressive decoders largely unexplored.

In this work, we first summarize the various fragmentation functions as graph tokenizers, at the granularity of nodes, edges, motifs, and Graph Neural Networks (GNNs). Given this summary, we systematically evaluate their empirical performances for MGM. Our analysis shows that reconstructing subgraph-level tokens in MGM can improve over the node tokens. Moreover, we find that a sufficiently expressive decoder combined with remask decoding [9] could improve the encoder's representation quality. Notably, remask 'decouples' the encoder and decoder, redirecting the encoder's focus away from molecule reconstruction and more towards MRL, leading to better downstream performances. In summary, we reveal that incorporating a subgraph-level tokenizer and a sufficiently expressive decoder with remask decoding gives rise to improved MGM performance.

Based on the findings above, we propose a novel pretraining framework - Masked Graph Modeling with a Simple GNN-based Tokenizer ( SimSGT ). SimSGT employs a S imple G NN-based T okenizer ( SGT ) that removes the nonlinear update function in each GNN layer. Surprisingly, we show that a single-layer SGT demonstrates competitive or better performances compared to other pretrained GNN-based and chemistry-inspired tokenizers. SimSGT adopts the GraphTrans [27] architecture for its encoder and a smaller GraphTrans for its decoder, in order to provide sufficient capacity for both the tasks of MRL and molecule reconstruction. Furthermore, we propose remask-v2 to decouple the encoder and decoder of the GraphTrans architecture. Finally, SimSGT is validated on downstream molecular property prediction and drug-target affinity tasks [28, 29], surpassing the leading graph SSL methods ( e.g., GraphMAE [9] and Mole-BERT [10]).

<latexi sh

1_b

64="W0nq3HkK7

G

MuDU

mJ2

yVw

>A

B

c

ZC

SgN

FI

P

o

+dz

Qr

Y

E

X

T

O

8

5

R

L

vf

j

/

9

p

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="Dv

B

dz

HnKP

p/FZ5f

L

yq

w

>A

73

c

V

SgN

EJ

G

Mr

Y

8

Q

T

C

U

k

RX

O

I

o

u

W

2

j

9

0

m

+

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="W0nq3HkK7

G

MuDU

mJ2

yVw

>A

B

c

ZC

SgN

FI

P

o

+dz

Qr

Y

E

X

T

O

8

5

R

L

vf

j

/

9

p

<latexi sh

1_b

64="Dv

B

dz

HnKP

p/FZ5f

L

yq

w

>A

73

c

V

SgN

EJ

G

Mr

Y

8

Q

T

C

U

k

RX

O

I

o

u

W

2

j

9

0

m

+

2 Preliminary

In this section, we begin with the introduction of MGM. Then, we provide a categorization of existing graph tokenizers. Finally, we discuss the architecture of graph autoencoders for MGM.

Notations. Let G denote the space of graphs. A molecule can be represented as a graph g = ( V , E ) ∈ G , where V is the set of nodes and E is the set of edges. Each node i ∈ V is associated with a node feature x i ∈ R d 0 and each edge ( i, j ) ∈ E is associated with an edge feature e ij ∈ R d 1 . The graph g 's structure can also be represented by its adjacency matrix A ∈ { 0 , 1 } |V|×|V| , such that A ij = 1 if ( i, j ) ∈ E and A ij = 0 otherwise.

2.1 Preliminary: Masked Graph Modeling

Here we illustrate MGM's three key steps: graph tokenizer, graph masking, and graph autoencoder.

Graph tokenizer. Given a graph g , we leverage a graph tokenizer tok ( g ) = { y t = m ( t ) ∈ R d | t ∈ f ( g ) } to generate its graph tokens as the reconstruction targets. The tokenizer tok ( · ) is composed of a fragmentation function f that breaks g into a set of subgraphs f ( g ) = { t = ( V t , E t ) | t ⊆ g } , and a mapping function m ( t ) ∈ R d that transforms the subgraphs into fixed-length vectors. In this work, we allow f ( g ) to include overlapped subgraphs to enlarge the scope of graph tokenizers.

Graph masking. Further, we add noises to g by random node masking. Here we do not use edge dropping because Hou et al. [9] empirically show that edge dropping easily leads to performance drop in downstream tasks. Specifically, node masking samples a random subset of nodes V m ⊆ V and replaces their features with a special token m 0 ∈ R d 0 . We denote the masked node feature by ˜ x i :

Graph autoencoder. The corrupted graph ˜ g is then fed into a graph autoencoder for graph reconstruction. We defer the details of the graph autoencoder's architecture to Section 2.3. Let { z i | i ∈ V} be the node-wise outputs of the graph autoencoder. We obtain subgraph t 's prediction ˆ y t = MEAN ( { z i | i ∈ V t } ) by mean pooling the representations of its nodes, if not especially noted. The graph autoencoder is trained by minimizing the distance between the predictions { ˆ y t | t ∈ f ( g ) } and the targets tok ( g ) = { y t | t ∈ f ( g ) } . The reconstruction loss is accumulated on tokens that include corrupted information { t | t ∈ f ( g ) , V t ∩ V m = ∅} :

̸

where ℓ ( · , · ) is the loss function, dependent on the type of y t . We use mean square error [15] for ℓ ( · , · ) when y t is a continuous vector, and use cross-entropy [16] for ℓ ( · , · ) when y t is a discrete value.

2.2 Revisiting Molecule Tokenizers

Scrutinizing the current MRL methods, we summarize the molecule tokenizers into four distinct categories, as Table 1 shows. A detailed description of the first three categories is systematically provided here, while the Simple GNN-based Tokenizer is introduced in Section 3.

Node, edge tokenizer [12, 11]. A graph's nodes and edges can be used as graph tokens directly:

Figure 3a illustrates the use of atomic numbers of nodes and bond types of edges as graph tokens in a molecule. These have been widely applied in previous research [12, 11, 25, 24], largely due to their

̸

<latexi sh

1_b

64="8KMrdpZ

Qg9C

RnX

YW

mDG

I

>A

B73

c

S

N

F

P

o

+

z

u

U

E

w

k

T

O

5

L

H

0vf

j

/

y

J

q

2

V

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="8KMrdpZ

Qg9C

RnX

YW

mDG

I

>A

B73

c

S

N

F

P

o

+

z

u

U

E

w

k

T

O

5

L

H

0vf

j

/

y

J

q

2

V

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="8KMrdpZ

Qg9C

RnX

YW

mDG

I

>A

B73

c

S

N

F

P

o

+

z

u

U

E

w

k

T

O

5

L

H

0vf

j

/

y

J

q

2

V

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="8KMrdpZ

Qg9C

RnX

YW

mDG

I

>A

B73

c

S

N

F

P

o

+

z

u

U

E

w

k

T

O

5

L

H

0vf

j

/

y

J

q

2

V

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="p

TqXNU

m

w

5+

AEdCfP

Vg

>

B73

c

Z

S

FI

o

r

D

Ku zYGM

LJ

H

O

8

Q

0

y/

9

n

W

v

2

j

R

k

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="

+oK

MY

AHkB8QPZ

z0

/dv

c

>

73

DJSgN

EI

rXG

F

CTm

L

Rj

p

n

w5

y

O

9

q

2

W

u

V

U

f

<latexi sh

1_b

64="

+oK

MY

AHkB8QPZ

z0

/dv

c

>

73

DJSgN

EI

rXG

F

CTm

L

Rj

p

n

w5

y

O

9

q

2

W

u

V

U

f

<latexi sh

1_b

64="

+oK

MY

AHkB8QPZ

z0

/dv

c

>

73

DJSgN

EI

rXG

F

CTm

L

Rj

p

n

w5

y

O

9

q

2

W

u

V

U

f

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="

+oK

MY

AHkB8QPZ

z0

/dv

c

>

73

DJSgN

EI

rXG

F

CTm

L

Rj

p

n

w5

y

O

9

q

2

W

u

V

U

f

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="p

TqXNU

m

w

5+

AEdCfP

Vg

>

B73

c

Z

S

FI

o

r

D

Ku zYGM

LJ

H

O

8

Q

0

y/

9

n

W

v

2

j

R

k

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

<latexi sh

1_b

64="p

TqXNU

m

w

5+

AEdCfP

Vg

>

B73

c

Z

S

FI

o

r

D

Ku zYGM

LJ

H

O

8

Q

0

y/

9

n

W

v

2

j

R

k

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="

+oK

MY

AHkB8QPZ

z0

/dv

c

>

73

DJSgN

EI

rXG

F

CTm

L

Rj

p

n

w5

y

O

9

q

2

W

u

V

U

f

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="

+oK

MY

AHkB8QPZ

z0

/dv

c

>

73

DJSgN

EI

rXG

F

CTm

L

Rj

p

n

w5

y

O

9

q

2

W

u

V

U

f

<latexi sh

1_b

64="

+oK

MY

AHkB8QPZ

z0

/dv

c

>

73

DJSgN

EI

rXG

F

CTm

L

Rj

p

n

w5

y

O

9

q

2

W

u

V

U

f

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="75o/j cDp

FTO9

dgL

E3r

M

>A

B

ZC

S

N

I

P

U

q

Ku zk

Yy

X

J

Ww

H

2n

8

G

R

f

V

+

Q

0

m

v

<latexi sh

1_b

64="

+oK

MY

AHkB8QPZ

z0

/dv

c

>

73

DJSgN

EI

rXG

F

CTm

L

Rj

p

n

w5

y

O

9

q

2

W

u

V

U

f

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="X

rz

Ao

DR

Z3PFy mcO

>

B7

V

LSgN

EJ

fUY9

Q

8

0

wT

CW

2

p

M

d

vH

Ku/

G

q

I

+

j

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="X

rz

Ao

DR

Z3PFy mcO

>

B7

V

LSgN

EJ

fUY9

Q

8

0

wT

CW

2

p

M

d

vH

Ku/

G

q

I

+

j

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="

ZU

AI

TFc5

yY

2

L

k3

>

B7

VD

SgN

EOz

G

Mr

H

8

dQw

j

C

nJ

RX

p/

oK

q

u

m

+

W9v

P

0

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="0S

VrQm9F

+

DEdwRy7

28

>A

B

3

c

L

gN

Oz

G

M

H

Y

J

Zp

P

TK

v

k

I

X

W/n

f

q

C

j

5

u

U

o

<latexi sh

1_b

64="+gW

L

yFP/zU

k

w

>A

B73

c

V

S

N

EJ

Mr

H

Y

8

dQ

j

C

n

RX

p

O

I

T

oK

q

u

2

Z

9

5

f

v0

m

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="jO3v+

V7

/Ygo

n

UqTr

f

I

>A

B

c

DL

J

E

zF

9

H

R

N

y

8

5Z

P

kw0

X

d

p

m

W

2

u

K

S

Q

G

C

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="wj5

2Z

Ly

J

oAS

C

/

EpR8

>

B73

c

VD

Tg

OzF

+I

9

H

Nr

M

Y

kd

v

P

0Uqn

X

W

m

Q

G

f

u

K

<latexi sh

1_b

64="qCV0fKD

vEA

w

n/j

o

+

TyM

>

B73

c

LSgN

J

r

UY9

OQ

X8

I

m

k

p

P

F

G

H

W

dz

R

u

2

Z

5

Root nodes

1-hop neighbors

2-hop neighbors

C

C

C

C

N

N

F

F

C

O

O

O

C

O

C

C

C

N

N

F

F

C

(a) Node, edge tokenizer

(b) Motif-based tokenizer

Cycle

Remaining

nodes

C

C

C

C

N

N

F

F

C

C

C

N

C

C

C

C

N

N

…

Message-passing induced rooted subtrees

2-hop message-passing

Fragment

Motif vocabulary

One-hot map

C

C

C

N

N

C

C

C

C

C

C

C

C

C

C

S

…

F

Gather

Chemistry knowledge:

Functional groups,

Cycles, BRICS, …

Molecule Dataset

G

NN

s

Pretrained G

NN

s

Pretraining

Periodic table

Bond types

One-hot map

C

C

…

C

C

F

F

F

F

F

F

F

F

(c) Pretrained GNN-based tokenizer

simplicity. However, atomic numbers and bond types are low-level features. Reconstructing them may be suboptimal for downstream tasks that require a high-level understanding of graph semantics.

Motif-based tokenizer. Motifs are statistically significant subgraph patterns in graph structures. For molecules, functional groups (FGs) are motifs that are manually curated by experts based on the FGs' biochemical characteristics [30, 31]. For example, molecules that contain benzene rings exhibit the property of aromaticity. Considering that the manually curated FGs are limited and cannot fully cover all molecules, previous works [32, 1, 18] employ chemistry-inspired fragmentation functions for motif discovery. Here we summarize the commonly used fragmentation functions:

FGs [18, 2]. An FG is a molecule's subgraph that exhibits consistent chemical behaviors across various compounds. In chemistry toolkits [30, 31], the substructural patterns of FGs are described by the SMARTS language [33]. Let S 0 = { s i } |S 0 | i =1 be a set of SMARTS patterns for FGs, and let p s ( g ) be the function returning the set of s FGs in g . FG-based fragmentation works as:

Cycles [32, 1, 18]. Cycles in molecules are often extracted as motifs due to their potential chemical significance. Figure 3b depicts the process of fragmenting a five-node cycle as a moif. If two cycles overlapped more than two atoms, they can be merged because they constitute a bridged compound [34]. Let C n represent a cycle of n nodes. They can be written as:

̸

BRICS [35, 1]. BRICS fragments a molecule at the potential cleavage sites, where chemical bonds can be broken under certain environmental conditions or catalysts. The key step of BRICS is to identify a molecule g 's potential cleavage sites, denoted by ψ ( g ) . This is achieved by finding bonds with both sides matching one of the pre-defined 'FG-like' substructural patterns S 1 in BRICS:

where g -t = g [ V \ V t ] denotes deleting g 's nodes in t and the corresponding incident edges [36]; E \ ( E t ∪E g -t ) contains the bond that connects t and g -t . Next, g is fragmented into the maximum subgraphs that contain no bonds in the cleavage sites ψ ( g ) :

Note that, the original BRICS includes more rules, such as a constraint on t and g -t 's combinations. We present only the key step here for simplicity.

Remaining nodes and edges [32, 18]: Given another fragmentation function f 0 , the nodes and edges that are not included in any of f 0 's outputs are treated as individual subgraphs. This improves f 0 's coverage on the original graph. Figure 3b shows an example of fragmenting remaining nodes after f cycle: nodes that are not in any cycles are treated as individual subgraphs.

To obtain more fine-grained subgraphs, previous works [1, 32, 18] usually combine several fragmentation functions together by unions ( e.g., f 1 ( g ) ∪ f 2 ( g ) ) or compositions ( e.g., { f 2 ( t ) | t ∈ f 1 ( g ) } ). Let f motif be the final fragmentation function after combination. We break every molecule in the dataset by f motif and gather a motif vocabulary M , which filters out infrequent motifs by a threshold. Then, given a new molecule g ′ , we can generate its tokens by one-hot encoding its motifs f motif ( g ′ ) :

Pretrained GNN-based tokenizer [10]. Pretrained GNNs can serve as graph tokenizers. Take a k -layer Graph Isormophism Network (GIN) [37] as an example. Its node embedding summarizes the structural information of the node's k -hop rooted subtree, making it a subgraph-level graph token (Figure 3c). A GIN performs the fragmentation and the mapping simultaneously. It can be written as:

where AGGREGATE ( · ) collects information from node i 's neighbors, based on which COMBINE ( · ) updates i 's representation. SG ( · ) denotes stop-gradient, which stops the gradient flow to the tokenizer during MGM pretraining. In addition to GINs, other GNNs can also serve as tokenizers. To obtain meaningful graph tokens, we pretrain a GNN before employing it as a tokenizer [10]. Once pretrained, this GNN is frozen and employed for the subsequent MGM pretraining. In Section 4, we evaluate the prevalent graph pretraining strategies for tokenizers. Given that GNN-based tokenizers provide node-wise tokens, we directly minimize the distance between the graph tokens and the autoencoder's outputs { z i } |V| i =1 of the masked nodes V m , i.e., L 0 = 1 |V m | ∑ i ∈V m ℓ (ˆ y i = z i , y i ) .

2.3 Revisiting Graph Autoencoders

Background. Graph autoencoder consists of a graph encoder and a graph decoder. We pretrain them with the objective of graph reconstruction. Once well pretrained, the encoder is saved for downstream tasks. MGM works [12, 2] usually adopt expressive graph encoders like GINEs [12] and Graph Transformers [2]. However, the exploration on expressive decoders has been limited. Many previous works [12, 10, 11, 22, 2, 25] apply a linear or an MLP decoder, similar to BERT's design [13].

However, recent studies [15, 38] have revealed the disparity between representation learning and reconstruction tasks. They show that a sufficiently expressive decoder could improve the encoder's representation quality. Delving into these studies, we identify two key elements to improve the representation quality: a sufficiently expressive decoder and remask decoding [9].

Sufficiently expressive decoder. Following [15, 38], we devise smaller versions of the encoder's architecture as decoders. We adopt the GINE [12] and G raph T ran s [27] (denoted as GTS ) as encoders. GTS stacks transformer layers on top of the GINE layers to improve the ability of modeling global interactions. Table 2 summarizes their different versions that we compare in this work.

Remask decoding [9]. Remask controls the focus of the encoder and the decoder. Let { h i | i ∈ V} be the encoder's node-wise hidden representations for the masked graph. Remask decoding masks the hidden representations of the masked nodes V m again by a special token m 1 ∈ R d before feeding them into the decoder. Formally, the remasked hidden representation ˜ h i is as follows:

<latexi sh

1_b

64="Jd wI

CU3PE

o

9LRN

Tf

>A

Dn

c

Z

S

F

Y

V

r

zW

X

Gq

0Mk

y

v

/jp

2g

8

HO

K

+

u

7

Q

B

5

m

<latexi sh

1_b

64="zoNv89ufV

5

OEI

dH

gr

Q

>A

B7n

c

S

J3U

q/

L

p

K

X

CF

0

Wy2m

Z

R

P

j

w

G

MD

T

+

kY

<latexi sh

1_b

64="2uHGK

q

Y

W

o

>A

B

X

c

DLSgN

EOy

r

U

9

I5

J

m

n

Q

d

p

M

/7C

w

Fv

3

j

Z

R

z

<latexi sh

1_b

64="2uHGK

q

Y

W

o

>A

B

X

c

DLSgN

EOy

r

U

9

I5

J

m

n

Q

d

p

M

/7C

w

Fv

3

j

Z

R

z

<latexi sh

1_b

64="2uHGK

q

Y

W

o

>A

B

X

c

DLSgN

EOy

r

U

9

I5

J

m

n

Q

d

p

M

/7C

w

Fv

3

j

Z

R

z

<latexi sh

1_b

64="

LBTz foc

+vF3

2

7muP0pG

A

>

VD

S

N

W

r

q

Y

K

n

gH9CEMJ

O

Z

k

y

U

w

5

X

d/

I

j

R

Q

8

<latexi sh

1_b

64="ok+

DcQ

YN59

fMuqFVw

UK

g

>A

B

3

LS

J

2vW

d

X

T

7

m

0

j

C

P

/

n

R

p

O

y

H

I

8

E

r

z

Z

G

<latexi sh

1_b

64="w

GB

HmXnQ/0rPvOSC

2Kuf

>A

+3

c

VDL

N

F

p

7d

I

U

8

E

J

M

y

5

W

z

9

j

q

Y

o

Rg

T

k

Z

<latexi sh

1_b

64="u8M

Cc

B+0

rLw

HA

FGp

7

>

3

VD

S

N

2

f

d

q5KI

y

oU

mk

R

v

J

/

T

g

P

O

nj

Z

9

W

Q

Yz

E

X

<latexi sh

1_b

64="V

RIA5G8NFMpQB

E

>

+3

c

DLS

J

2vW

dugkXoq

H

C

7

O

T

z

w

n

P

K

r

m

f

Y

j

9

y

<latexi sh

1_b

64="PSq0V

g82

O

3

UCpEDAw

>

c

L

N

F

r

f

7d

5KIW

y

o

mk

MH

R

v

G

Ju/

T

nj

Z

9

Q

Yz

X

<latexi sh

1_b

64="0

B

OJGk

MRI

r5H

U

Q

>A

83

c

VDLS

N

F

2p

f

Z

u

ov

E

C

7

T

g

y

/

Y

q

W

+

m

n

P

K

w

X

<latexi sh

1_b

64="

CDMTw9Gon

B3NPfkdU

XYm+F

>A

8

c

V

LS

2pr

Z

u

vQ

y

0

J

I

/

R

z

O

7

E

j

K

W

g

q

5

H

<latexi sh

1_b

64="c

Z3

QzYJ

+uN

EX

I

P

mk

>A

B8

VDLS

F

2pr fU

d

ov

q

CG5

7

O

T

0Mg

y

/

9W

H

n

j

R

K

w

<latexi sh

1_b

64="oSpj

I

A

OHXN0

u9/

k

Pf

>

B+3

c

VDL

FJ

U

2vW

d

Q

nEq

CG5

7

T

8m

Y

r

K

Z

y

M

z

<latexi sh

1_b

64="GFPn

3

5N/

Do9rqY8X

WIKc

>A

B

V

LSgM

T

7

+

d

vQ

ZkR

uC

m

0

w

J

f

2

U

p

z

H

E

y

O

j

<latexi sh

1_b

64="

r7/2Rk

L

I

HufqdN

GyAz

o

>

B83

c

VD

S

F

p

Z

vQ

UmK

X

j

9g

JP

0

Y

C

O

+

n

M

T

5

W

w

E

<latexi sh

1_b

64="Y7R

GUP

kAJ

Q

K

uf wMB

>

83

c

VDLS

N

F

2pr

Zd ov

X

Cm

n

5

O

W

H

z

9

+

j

q

/

E

T

I

0

g

y

<latexi sh

1_b

64="

G20HE

YgK7p3I

d

L

c

>A

B8

VD

S

N

F

r

fU

u

ovQ

n

q

C

5

O

T

M

kz

y

9W

+

m

P

j

J

R

w

X

<latexi sh

1_b

64="ok+

DcQ

YN59

fMuqFVw

UK

g

>A

B

3

LS

J

2vW

d

X

T

7

m

0

j

C

P

/

n

R

p

O

y

H

I

8

E

r

z

Z

G

<latexi sh

1_b

64="w

GB

HmXnQ/0rPvOSC

2Kuf

>A

+3

c

VDL

N

F

p

7d

I

U

8

E

J

M

y

5

W

z

9

j

q

Y

o

Rg

T

k

Z

<latexi sh

1_b

64="oSpj

I

A

OHXN0

u9/

k

Pf

>

B+3

c

VDL

FJ

U

2vW

d

Q

nEq

CG5

7

T

8m

Y

r

K

Z

y

M

z

<latexi sh

1_b

64="

r7/2Rk

L

I

HufqdN

GyAz

o

>

B83

c

VD

S

F

p

Z

vQ

UmK

X

j

9g

JP

0

Y

C

O

+

n

M

T

5

W

w

E

<latexi sh

1_b

64="Y7R

GUP

kAJ

Q

K

uf wMB

>

83

c

VDLS

N

F

2pr

Zd ov

X

Cm

n

5

O

W

H

z

9

+

j

q

/

E

T

I

0

g

y

<latexi sh

1_b

64="

G20HE

YgK7p3I

d

L

c

>A

B8

VD

S

N

F

r

fU

u

ovQ

n

q

C

5

O

T

M

kz

y

9W

+

m

P

j

J

R

w

X

Random masking

Stop gradient

Minimize

distance

Simple GNN-based Tokenizer

Embed

GINEs

Weight

sharing

Encoder

1

2

3

4

5

6

7

1

2

3

4

5

6

7

Concat

Embed

BN

k

×

ˆ

y

1

ˆ

y

2

ˆ

y

3

ˆ

y

4

ˆ

y

5

ˆ

y

6

ˆ

y

7

ˆ

y

2

ˆ

y

3

ˆ

y

5

y

1

y

2

y

3

y

4

y

5

y

6

y

7

y

2

y

3

y

5

Transformers

1

4

6

7

1

4

6

7

m

1

m

1

m

1

GINEs

Transformers

Decoder

ω

(

A

)

H

(

l

-

1)

1

2

3

4

5

6

7

1

4

6

7

Remask constrains the encoder's ability on predicting the corrupted information by removing the encoder's representation on the masked part. The encoder is enforced to generate effective representations for the unmasked part, to provide signals for the decoder for graph reconstruction.

3 Methodology

In this section, we present our method - Masked Graph Modeling with a Simple GNN-based Tokenizer ( SimSGT ) (Figure 4). Specifically, it applies the GTS [27] architecture for both its encoder and decoder. SimSGT features a S imple G NN-based T okenizer ( SGT ), and employs a new remask strategy to decouple the encoder and decoder of the GTS architecture.

Simple GNN-based Tokenizer. SGT simplifies existing aggregation-based GNNs [37] by removing the nonlinear update functions in GNN layers. It is inspired by studies showing that carefully designed graph operators can generate effective node representations [39, 40]. Formally, a k -layer SGT is:

where Embedding ( · ) is a linear layer that uses the weights of the encoder's node embedding function; H ( l ) i is the i -th row of H ( l ) ; BatchNorm ( · ) is a standard Batch Normalization layer without the trainable scaling and shifting parameters [41]; and ω ( A ) is the graph operator that represents the original GNN's aggregation function. For example, GIN has ω ( A ) = A +(1 + ϵ ) I and GCN has ω ( A ) = ˜ D -1 / 2 ˜ A ˜ D -1 / 2 , where ˜ A = A + I and ˜ D is the degree matrix of ˜ A .

Note that, SGT does not have trainable weights, allowing its deployment without pretraining. Its tokenization ability relies on the graph operator ω ( A ) that summarizes each node's neighbor information. Additionally, we concatenate the outputs of every SGT layer to include multi-scale information. We show in experiments that SGT transforms the original GNN into an effective tokenizer.

Graph autoencoder. SimSGT employs the GTS architecture as the encoder, and a smaller version of GTS ( i.e., GTS-Small in Table 2) as the decoder. This architecture follows the asymmetric encoder-decoder design in previous works [15, 38]. Further, we propose a new remask strategy, named remask-v2 , to decouple SimSGT's encoder layers and decoder layers,

Remask-v2. Remask-v2 constrains the encoder's ability on predicting the corrupted information by dropping the masked nodes' V m representations before the Transformer layers (Figure 4). After the Transformer layers, we pad special mask tokens m 1 ∈ R d to make up the previously dropped nodes' hidden representations. Compared to the original remask, remask-v2 additionally prevents the Transformer layers from processing the masked nodes. It thereby avoids the gap of processing masked nodes in pretraining but not in fine-tuning [15].

(a) Ablating the decoder. The tokenizer is a single-layer SGT of GIN. (b) Ablating the tokenizer. Darker blue indicates higher performance. The encoder is GTS and decoder is GTS-Small. Remask-v2 is used.

Encoder Decoder

GINE

Linear

-

73.2

GINE

GINE-Small

-

73.0

GINE

GINE-Small

v1

74.4

GTS

Linear

-

74.1

GTS

GTS-Small

-

74.1

GTS

GTS-Small

v1

75.2

GTS

GTS-Small

v2

75.8

GNN tokenizer 's depth

Tokenizer

-

1

2

3

4

5

Node

74.7

Motif, MGSSL

75.2

Motif, RelMole

74.4

Pretrain, GIN, GraphCL

75.1

74.5

74.2

74.0

74.6

Pretrain, GIN, VQ-VAE

75.1

74.9

74.4

75.6

75.1

Pretrain, GIN, GraphMAE

75.1

74.9

74.9

75.4

75.2

SGT, GIN

75.8

75.5

75.6

75.3

74.9

0

20

40

60

80

100

Pretrain epochs

0.08

0.10

0.12

0.14

0.16

MSE loss

Decoder: Linear

Decoder: GTS-Small

Decoder: GTS-Small with remask-v1

Decoder: GTS-Small with remask-v2

0

20

40

60

80

100

Pretrain epochs

75

80

85

90

95

100

Linear probing accuracy (%)

Decoder: GTS-Small

Decoder: GTS-Small with remask-v1

Decoder: GTS-Small with remask-v2

Baseline, random initialized encoder

GIN

GCN

GraphSAGE

74.75

75.00

75.25

75.50

75.75

76.00

Avg. ROC-AUC (%)

75.8

75.6

75.6

75.6

75.3

75.4

74.9

74.8

75.1

1-layer

3-layer

5-layer

(a) MSE loss for pretraining

(b) Probe encoder for masked atoms.

4 Rethinking Masked Graph Modeling for Molecules

Experimental setting. In this section, we perform experiments to assess the roles of tokenizer and decoder in MGM for molecules. Our experiments follow the transfer learning setting in [12, 9]. We pretrain MGM models on 2 million molecules from ZINC15 [42], and evalute the pretrained models on eight classification datasets in MoleculeNet [28]: BBBP, Tox21, ToxCast, Sider, ClinTox, MUV, HIV, and Bace. These downstream datasets are divided into train/valid/test sets by scaffold split to provide an out-of-distribution evaluation setting. We report the mean performances and standard deviations on the downstream datasets across ten random seeds. Throughout the experiments, we use random node masking of ratio 0 . 35 . More detailed experimental settings can be found in Appendix D.

4.1 Rethinking Decoder

We investigate the impact of an expressive decoder on MGM's performance. A single-layer GINbased SGT tokenizer is utilized in these experiments. Table 3a summarizes the results.

Finding 1. A sufficiently expressive decoder with remask decoding is crucial for MGM. Table 3a shows that using a sufficiently expressive decoder with remask decoding can significantly improve downstream performance. This can be attributed to the disparity between molecule reconstruction and MRL tasks: the last several layers in the autoencoder will be specialized on molecule reconstruction, while losing some representation ability [15, 38]. When using a linear decoder, the last several layers in the encoder will be reconstruction-specialized, which can yield suboptimal results in fine-tuning.

A sufficiently expressive decoder is essential to account for the reconstruction specialization. As shown by Figure 5a, employing an expressive decoder results in significantly lower reconstruction loss than a linear decoder. However, increasing the decoder's expressiveness without remask decoding cannot improve the downstream performance (Table 3a). This leads to our exploration on remask.

Finding 2. Remask constrains the encoder's efforts on graph reconstruction for effective MRL. Figure 5b shows that the linear probing accuracy of the masked atom types on an encoder pretrained with remask is significantly lower than the accuracy on an encoder pretrained without remask. This shows that remask makes the encoder spend less effort on predicting the corrupted information. Moreover, remask only slightly sacrifices the autoencoder's reconstruction ability, when paired with the GTS-Small decoder (Figure 5a). Combining the

observation that remask improves downstream performances (Table 3a), it indicates that remask constrains the encoder's efforts on graph reconstruction, allowing it to focus on MRL.

In addition, remask-v2 outperforms remask-v1 with GTS architecture. This improvement can be attributed to remask-v2's ability to prevent the Transformer layers from processing the masked nodes, avoiding the gap of using masked nodes in pretraining but not in fine-tuning. Finally, we test decoder's sizes in Table 4. When the encoder is GTS, GTS-Small decoder provides the best performance.

4.2 Rethinking Tokenizer

We investigate the impact of tokenizers on MGM's performance. In the following experiments, the graph autoencoder employs the GTS encoder and the GTS-Small decoder with remask-v2, given their superior performances in the previous section. The results are summarized in Table 3b.

Compared tokenizers. We use the node tokenizer as the baseline. For motif-based tokenizers, we employ the leading fragmentation methods: MGSSL [1] and RelMole [18]. For GNN-based tokenizers, we compare the prevalent pretraining strategies - GraphCL [4], GraphMAE [9], and VQ-VAE [10, 43] - to pretrain tokenizers on the 2 million molecules from ZINC15.

Finding 3. Reconstructing subgraph-level tokens can give rise to MRL. Table 3b shows that, given the appropriate setup, incorporating motif-based tokenizers or GNN-based tokenizers in MGM can provide better downstream performances than the node tokenizer. This observation underscores the importance of applying a subgraph-level tokenizer in MGM.

Finding 4. Single-layer SGT outperforms or matches other tokenizers. Table 3b shows that a single-layer SGT, applied to GIN, delivers comparable performance to a four-layer GIN pretrained by VQ-VAE, and surpasses other tokenizers. Further, Figure 6 shows that SGT can transform GCN and GraphSAGE into competitive tokenizers. We attribute SGTs' good performances to the effectiveness of their graph operators in extracting structural information. It has been shown that linear graph operators can effectively summarize structural patterns for node classification [39, 40].

Finding 5. GNN-based tokenizers have achieved higher performances than motif-based tokenizers. We hypothesize that this is due to GNN's ability of summarizing structural patterns. When using GNN's representations as reconstruction targets, the distance between targets reflects the similarity between their underlying subgraphs - a nuanced relationship that the one-hot encoding of motifs fails to capture. We leave the potential incorporation of GNNs into motif-based tokenizers for future works.

Finally, we show that although GNN-based tokenizers are agnostic to chemistry knowledge, incorporating them in MGM can improve the recognition of FGs. In Figure 7, we use a linear classifier to

0

25

50

75

100

Pretrain epochs

75

80

85

90

95

ROC-AUC (%)

Node tokenizer

Single-layer SGT

probe the encoder's mean pooling output to predict the FGs within the molecule. We use RDkit [30] to extract 85 types of FG. Details are in Appendix D. It can be seen that, incorporating a single-layer SGT in MGM improves the encoder's ability to identify FGs in comparison to the node tokenizer.

5 Comparison to the State-of-the-Art Methods

In this section, we compare SimSGT to the leading molecule SSL methods for molecular property prediction and a broader range of downstream tasks. For fair comparison, we report the performances of SimSGT and its variant that uses the GINE architecture. This variant employs a GINE as encoder and a GINE-Small as decoder (Table 2), and implements remask-v1 decoding.

Molecular property prediction. The molecular property prediction experiment follows the same transfer learning setting as Section 3. Table 5 presents the results. It can be observed that SimSGT outperforms all the baselines in average performances with both the GTS and GINE architectures. Notably, SimSGT with GTS establishes a new state-of-the-art of 75 . 8% ROC-AUC. It exceeds the second method by an absolute 1 . 8% points in average performance and achieves the best performance in five out of the eight molecular property prediction datasets. SimSGT with GINE outperforms the baselines by 0 . 4% points in average performance. These improvements demonstrate the effectiveness of our proposed tokenizer and decoder to improve MGM performance for molecules.

Broader range of downstream tasks. We report the transfer learning performances on regressive molecular property prediction and drug-target affinity (DTA) tasks. DTA aims to predict the affinity scores between molecular drugs and target proteins [44, 29]. Specifically, we substitute the molecule encoder in [44] with a SimSGT pretrained encoder to evaluate DTA performances. Following the experimental setting in [45], we pretrain SimSGT on the 50 thousand molecule samples from the GEOM dataset [46] and report the mean performances and standard deviations across three random seeds. We report the RMSE for the molecular property prediction datasets with scaffold splitting, and report the MSE for the DTA datasets with random splitting. The results are summarized in Table 6. It can be observed that SimSGT achieves significant improvement over the baseline models.

Quantum chemistry property prediction. We report performances of predicting the quantum chemistry properties of molecules [47]. Following [48], we divide the downstream datasets by scaffold split. This experiment reuses the model checkpoints from Table 5, which are pretrained on 2 million molecules from ZINC15. Specifically, we attach a two-layer MLP after the pretrained molecule encoders and fine-tune the models for property prediction. We report average performances

and standard deviations across 10 random seeds. The performances are reported in Table 7. We can observe that SimSGT consistently outperforms representative baselines of GraphCL, GraphMAE, and Mole-BERT.

Computational cost. We compare the wall-clock pretraining time for SimSGT and key baselines in Table 8. We can observe that: 1) SimSGT's pretraining time is on par with GraphMAE [9]. This efficiency is largely attributed to the minimal computational overhead of our SGT tokenizer; 2) In comparison to Mole-BERT [10], the prior benchmark in molecule SSL, SimSGT is approximately three times faster. The computational demands of Mole-BERT can be attributed to its combined approach of MGM training and contrastive learning.

6 Conclusion and Future Works

In this work, we extensively investigate the roles of tokenizer and decoder in MGM for molecules. We compile and evaluate a comprehensive range of molecule fragmentation functions as molecule tokenizers. The results reveal that a subgraph-level tokenizer gives rise to MRL performance. Further, we show by empirical analysis that a sufficiently expressive decoder with remask decoding improves the molecule encoder's representation quality. In light of these findings, we introduce SimSGT, a novel MGM approach with subgraph-level tokens. SimSGT features a Simple GNN-based Tokenizer capable of transforming GNNs into effective graph tokenizers. It further adopts the GTS architecture for its encoder and decoder, and incorporates a new remask strategy. SimSGT exhibits substantial improvements over existing molecule SSL methods. For future works, the potential application of molecule tokenizers to joint molecule-text modeling [3], remains an interesting direction.

References

Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self-supervised learning for molecular property prediction. In NeurIPS , pages 15870-15882, 2021.

Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. In NeurIPS , 2020.

Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In EMNLP , 2023.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS , 2020.

Laurianne David, Amol Thakkar, Rocío Mercado, and Ola Engkvist. Molecular representations in ai-driven drug discovery: a review and practical guide. Journal of Cheminformatics , 12(1):122, 2020.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML , pages 1263-1272. PMLR, 2017.

Chaochao Yan, Qianggang Ding, Peilin Zhao, Shuangjia Zheng, Jinyu Yang, Yang Yu, and Junzhou Huang. Retroxpert: Decompose retrosynthesis prediction like A chemist. In NeurIPS , 2020.

Umit V Ucak, Islambek Ashyrmamatov, Junsu Ko, and Juyong Lee. Retrosynthetic reaction pathway prediction through neural machine translation of atomic environments. Nature communications , 13(1):1186, 2022.

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In KDD , pages 594-604. ACM, 2022.

Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z. Li. Mole-BERT: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations , 2023.

Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In ICML , volume 119 of Proceedings of Machine Learning Research , pages 10871-10880. PMLR, 2020.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In ICLR , 2020.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , pages 4171-4186. Association for Computational Linguistics, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21:140:1-140:67, 2020.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR , pages 15979-15988. IEEE, 2022.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In ICLR . OpenReview.net, 2022.

Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou. Mocl: Data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. In KDD , pages 3585-3594. ACM, 2021.

Zewei Ji, Runhan Shi, Jiarui Lu, Fang Li, and Yang Yang. Relmole: Molecular representation learning based on two-level graph similarities. Journal Chemical Information Modeling , 62(22):5361-5372, 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML , pages 8821-8831, 2021.

Mukund Deshpande, Michihiro Kuramochi, Nikil Wale, and George Karypis. Frequent substructure-based approaches for classifying chemical compounds. IEEE Transactions on Knowledge and Data Engineering , 17(8):1036-1050, 2005.

Jonathan Clayden, Nick Greeves, and Stuart Warren. Organic chemistry . Oxford university press, 2012.

Jintang Li, Ruofan Wu, Wangbin Sun, Liang Chen, Sheng Tian, Liang Zhu, Changhua Meng, Zibin Zheng, and Weiqiang Wang. Maskgae: Masked graph modeling meets graph autoencoders. CoRR , abs/2205.10053, 2022.

Chun Wang, Shirui Pan, Guodong Long, Xingquan Zhu, and Jing Jiang. MGAE: marginalized graph autoencoder for graph clustering. In CIKM , pages 889-898. ACM, 2017.

Sixiao Zhang, Hongxu Chen, Haoran Yang, Xiangguo Sun, Philip S. Yu, and Guandong Xu. Graph masked autoencoders with transformers. CoRR , abs/2202.08391, 2022.

Ganqu Cui, Jie Zhou, Cheng Yang, and Zhiyuan Liu. Adaptive graph encoder for attributed graph embedding. In KDD , pages 976-985. ACM, 2020.

Xiao Liu, Shiyu Zhao, Kai Su, Yukuo Cen, Jiezhong Qiu, Mengdi Zhang, Wei Wu, Yuxiao Dong, and Jie Tang. Mask and reason: Pre-training knowledge graph transformers for complex logical queries. In KDD , pages 1120-1130. ACM, 2022.

Zhanghao Wu, Paras Jain, Matthew A. Wright, Azalia Mirhoseini, Joseph E. Gonzalez, and Ion Stoica. Representing long-range context for graph neural networks with global attention. In NeurIPS , pages 13266-13279, 2021.

Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science , 9(2):513-530, 2018.

Tapio Pahikkala, Antti Airola, Sami Pietilä, Sushil Shakyawar, Agnieszka Szwajda, Jing Tang, and Tero Aittokallio. Toward more realistic drug-target interaction predictions. Briefings Bioinform. , 16(2):325-337, 2015.

Greg Landrum. Rdkit documentation. Release , 1(1-79):4, 2013.

Elena S Salmina, Norbert Haider, and Igor V Tetko. Extended functional groups (efg): an efficient set for chemical characterization and structure-activity relationship studies of chemical compounds. Molecules , 21(1):1, 2015.

Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In ICML , volume 80 of Proceedings of Machine Learning Research , pages 2328-2337. PMLR, 2018.

Daylight Chemical Information Systems, Inc. Daylight Theory Manual . http://www.daylight.com/dayhtml/doc/theory/theory.smarts.html.

Jonathan Clayden, Nick Greeves, and Stuart Warren. Organic chemistry . Oxford university press, 2012.

Jörg Degen, Christof Wegscheid-Gerlach, Andrea Zaliani, and Matthias Rarey. On the art of compiling and using'drug-like'chemical fragment spaces. ChemMedChem: Chemistry Enabling Drug Discovery , 3(10):1503-1507, 2008.

Reinhard Diestel. Graph Theory, 4th Edition , volume 173 of Graduate texts in mathematics . Springer, 2012.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR , 2019.

Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, NeurIPS , 2022.

Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In ICML , volume 97 of Proceedings of Machine Learning Research , pages 6861-6871. PMLR, 2019.

Lei Chen, Zhengdao Chen, and Joan Bruna. On graph neural networks versus graph-augmented mlps. In ICLR . OpenReview.net, 2021.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML , volume 37 of JMLR Workshop and Conference Proceedings , pages 448-456. JMLR.org, 2015.

Teague Sterling and John J. Irwin. ZINC 15 - ligand discovery for everyone. J. Chem. Inf. Model. , 55(11):2324-2337, 2015.

Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In NeurIPS , pages 14837-14847, 2019.

Thin Nguyen, Hang Le, Thomas P. Quinn, Tri Nguyen, Thuc Duy Le, and Svetha Venkatesh. Graphdta: predicting drug-target binding affinity with graph neural networks. Bioinform. , 37(8):1140-1147, 2021.

Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. In ICLR . OpenReview.net, 2022.

Simon Axelrod and Rafael Gómez-Bombarelli. GEOM: energy-annotated molecular conformations for property prediction and molecular generation. CoRR , abs/2006.05531, 2020.

Quantum machine. http://quantum-machine.org/datasets/ . Accessed 2023-03.

Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geometry-enhanced molecular representation learning for property prediction. Nat. Mach. Intell. , 4(2):127-134, 2022.

Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael M. Bronstein, and Federico Monti. SIGN: scalable inception graph neural networks. CoRR , abs/2004.11198, 2020.

Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In ICML , volume 119 of Proceedings of Machine Learning Research , pages 4839-4848. PMLR, 2020.

Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. arXiv preprint arXiv:2012.12533 , 2020.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS , 2020.

Steven Bird. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions , pages 69-72, 2006.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , pages 5998-6008, 2017.

Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In NeurIPS , 2022.

Alan Baade, Puyuan Peng, and David Harwath. MAE-AST: masked autoencoding audio spectrogram transformer. In INTERSPEECH , pages 2438-2442. ISCA, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR . OpenReview.net, 2021.

Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In ACL (System Demonstrations) , pages 55-60, 2014.

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL (1) . The Association for Computer Linguistics, 2016.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent - A new approach to self-supervised learning. In NeurIPS , 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR , 2021.

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veliˇ ckovi´ c, and Michal Valko. Large-scale representation learning on graphs via bootstrapping. In ICLR , 2022.

Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M. Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. In ICLR . OpenReview.net, 2022.

Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any GNN with local structure awareness. In ICLR . OpenReview.net, 2022.

Ziqiao Zhang, Jihong Guan, and Shuigeng Zhou. Fragat: a fragment-oriented multi-scale graph attention model for molecular property prediction. Bioinform. , 37(18):2981-2987, 2021.

Fabrizio Frasca, Beatrice Bevilacqua, Michael M. Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. In NeurIPS , 2022.

Chendi Qian, Gaurav Rattan, Floris Geerts, Mathias Niepert, and Christopher Morris. Ordered subgraph aggregation networks. In NeurIPS , 2022.

Bohang Zhang, Guhao Feng, Yiheng Du, Di He, and Liwei Wang. A complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests. arXiv preprint arXiv:2302.07090 , 2023.

Petar Veliˇ ckovi´ c, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR , 2019.

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In ICLR , 2020.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In ICML , Proceedings of Machine Learning Research, pages 12121-12132, 2021.

Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. In NeurIPS , pages 15920-15933, 2021.

Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graphlevel representation learning with local and global structure. In ICML , volume 139, pages 11548-11558, 2021.

Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan He, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In ICML , pages 13052-13065, 2022.

Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. S2GAE: self-supervised graph autoencoders are generalizable learners with graph masking. In WSDM , pages 787-795. ACM, 2023.

Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. Graphmae2: A decoding-enhanced masked self-supervised graph learner. In WWW , volume abs/2304.04779, 2023.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 , 2020.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI , pages 3538-3545. AAAI Press, 2018.

Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In ICLR . OpenReview.net, 2021.

Yoav Levine, Barak Lenz, Opher Lieber, Omri Abend, Kevin Leyton-Brown, Moshe Tennenholtz, and Yoav Shoham. Pmi-masking: Principled masking of correlated spans. In ICLR . OpenReview.net, 2021.

A Limitations

Our results and analysis on the graph tokenizer and graph decoder are confined to the task of MGM pretraining. Different tokenizers and decoders might offer advantages for other generative modeling methods, such as autoregressive modeling [32].

SGTs [39, 40] are limited in expressive power for graph structures compared to standard GNNs, like GINs [37]. Theoretically, the separation of expressiveness power between SGTs and standard GNNs grows exponentially in the GNN's depth [40]. However, SGTs exhibit comparable, if not better, performances to pretrained GNN-based tokenizers, as demonstrated in Table 3b. We attribute this intriguing observation to two key factors. Firstly, SGTs ( i.e., simple GNNs) are still powerful and can 'distinguish almost all non-isomorphic graphs' [40]. They have shown decent results in practice [39, 49]. Secondly, we conjecture that a better pretraining method for GNN-based tokenizers could exist, but current pretraining techniques do not fully harness the potential of GNNs in their roles as effective tokenizers. Indeed, the significant difference in performance between GraphCL and VQ-VAE (Table 3b) emphasizes the impact of pretraining methods on the tokenizer's performance. We leave the investigation of how to effectively pretrain GNN-based tokenizers as future works.

B Related Works

We have included the literature review of MGM in the main body of the paper. Here we elaborate on the literature review in the following areas.

Molecule SSL with motifs. Motifs are statistically significant subgraph patterns [32, 50], and have been applied in existing molecule SSL methods. Autoregressive pretraining methods [32, 1, 50] generate motifs instead of nodes at each generation step, in order to improve the validity of the generated molecules. Motifs are also used in contrastive learning [17, 51, 18]. Sun et al. [17] substitute motifs within a molecule with their chemically similar counterparts to create high-quality augmentations. [51, 18] construct molecules' views at the motif-level to supplement the original views at the atom-level. In predictive pretraining, Rong et al. [2] pretrain a graph encoder to predict the FGs inside the molecule. These previous works have developed extensive molecule fragmentation methods for motif discovery. However, these fragmentation methods have been overlooked as tokenizers in MGMpretraining. Our work addresses this gap by summarizing the common fragmentation rules and examining the performances of the selected fragmentation methods in MGM pretraining.

Data tokenization. Tokenization is a data pre-processing technique that divides the original data into smaller elements and converts them to tokens. It is widely used in NLP to split sentences into wordlevel units [13, 52, 53]. Due to the surging interests in Transformers [54], tokenization is also applied on images [15, 16] and audios [55, 56]. Tokenization fragments these data into sequences of patches to fit the shapes of transformer's input and output. A tokenizer can be designed by heuristics [57], incorporating domain knowledge [58], and pretraining on the target dataset [59, 19, 43]. In this work, we study graph tokenizers, which are less explored in previous works.

Relations to contrastive learning. When using a GNN-based tokenizer, MGM involves minimizing the distances between the outputs from two network branches ( i.e., the tokenizer branch and the autoencoder branch). At first glance, this design might seem similar to the contrastive learning methods of BYOL [60], SimSiam [61], and BGRL [62], which also minimize the output differences between two network branches. However, a closer inspection reveals several critical distinctions between MGM and these methods. Firstly, MGM feeds uncorrupted data to the tokenizer branch and feeds corrupted data to the autoencoder branch, encouraging the autoencoder to reconstruct the missing information. In contrast, BYOL, SimSiam, and BGRL use corrupted data in both of their branches, constituting different training objectives. Secondly, while BYOL, SimSiam, and BGRL employ nearly identical architectures for their two branches, MGM can adopt distinctly different architectures for its autoencoder and tokenizer. In our best-performing experiment, the autoencoder has more than ten layers of GNNs and Transformers, while the tokenizer is a shallow single-layer network (Table 3). Finally, MGM employs remask decoding to constrain the encoder's ability on reconstruction, which is not used in contrastive learning methods [60, 61, 62].

Subgraph-enhanced Graph Neural Network. Subgraph-enchanced GNN [63, 64, 65] refers to an emerging class of GNNs that fragments a graph into subgraphs before encoding, in order to improve the GNNs' expressivenss [66, 67, 68]. The common graph fragmentation method is node-wise, such

that each fragmented subgraph is associated with a unique node in the original graph. For example, ESAN [63] obtains subgraphs by sampling ego-networks or deleting one node from the original graph. Given the subgraphs, subgraph-enchanced GNNs generate node embeddings in every subgraph by applying a series of equivariant message-passing layers [63, 64]. Finally, these embeddings are pooled to output the graph embedding. Our work is related to subgraph-enhanced GNNs that we also study graph fragmentation. The major distinction is that we focus on using the tokens derived from these fragmented graphs as the reconstruction targets in MGM for molecules.

C Pseudo Code

We present the pseudocode of SimSGT. This code uses a single-layer SGT of GIN as an example.

Algorithm 1 Pytorch style pseudocode of SimSGT

D Experimental Setup

Computational resource. We perform experiments on an NVIDIA DGX A100 server. Each individual experiment can be run on a single GPU without exceeding 30 GBs of GPU memory.

D.1 Compared Methods

Motif-based tokenizers. We now elaborate on the details of the two compared motif-based tokenizers:

MGSSL [1] employs the BRICS [35] method for molecule fragmentation (Section 2.2). To obtain more fine-grained fragments, MGSSL employs two additional rules to break the BRICS's output fragments: 1) separate the single atoms attached to cycles; 2) if a connected subgraph comprising three or more atoms is not part of a cycle, break it down as a new fragment.

RelMole [18] combines the fragmentation functions of Cycles and FGs for molecule fragmentation (Section 2.2). Further, it extracts the carbon-carbon single bonds that are not covered in the previous step as new fragments.

We use the motif vocabulary provided by their paper for molecule fragmentation. Given a molecule, we convert its fragmented motifs into one-hot encodings, which serve as the reconstruction targets.

Pretrained GNN-based tokenizers. We use the atomic numbers as the node features and exclude edge features in pretrained GNN-based tokenizers. We show in Appendix E that incorporating edge

(b) Hyperparameters.

features in GNN tokenizers can decrease performance. Due to the removal of edge features, the tokenizer uses the architecture of GIN [37] instead of GINE [12]. We have reported performances of GNN-based tokenizers that are pretrained by GraphCL [4], GraphMAE [9], and VQ-VAE [10, 43]. The implementation of VQ-VAE follows [10] and groups the latent codes by atomic numbers. We strictly follow the procedure in the mentioned papers to pretrain GNNs, which are later used as tokenizers.

Simple GNN-based tokenizers (SGTs). An SGT uses the node feature of atomic number. It uses the graph encoder's linear embedding function of atomic numbers. We present the graph operators for our tested SGTs below:

where ˜ A = A + I and ˜ D is the degree matrix of ˜ A ; ϵ is set to 0 . 5 empirically.

Baselines. We now describe the details of our reported baseline methods:

Infomax [69] learns node representations by maximizing the mutual information between the local summaries of node patches and the patches' graph-level global summaries.

ContextPred [12] uses the embeddings of subgraphs to predict their context graph structures.

InfoGraph [70] conducts graph representation learning by maximizing the mutual information between graph-level representations and local substructures of various scales.

GraphCL [4] performs graph-level contrastive learning with combinations of four graph augmentations, namely node dropping, edge perturbation, subgraph cropping, and feature masking.

JOAO [71] proposes a framework to automatically search proper data augmentations for GCL.

AD-GCL [72] applies adversarial learning for adaptive graph augmentation to remove the redundant information in graph samples.

GraphLOG [73] leverages clustering to construct hierarchical prototypes of graph samples. They further contrast each local instance with its corresponding higher prototype for contrastive learning.

RGCL [74] trains a rationale generator to identify the causal subgraph in graph augmentation. Each graph's causal subgraph and its complement are leveraged in contrastive learning.

BGRL [62] trains an online encoder by learning to predict the output of a target encoder. The target encoder shares the same architecture as the online encoder and is updated through exponentially moving average. The inputs of the online encoder and the target encoder are two different graph augmentations.

(a) Node and edge features.

(b) Hyperparameters and their search spaces. We use the performance on the validation set for hyperparameter tuning. Bold indicates the final value used in experiments.

GraphMAE [9] shows that a linear classifier is insufficient for decoding node types. It applies a GNN for decoding and proposes remask to decouple the functions of the encoder and decoder in the autoencoder.

GraphMVP [45] uses a contrastive loss and a generative loss to connect the 2-dimensional view and 3-dimensional view of the same molecule, in order to inject the 3-dimensional knowledge into the 2-dimensional graph encoder.

S2GAE [75] randomly masks a portion of edges of graphs and pretrain the graph encoder to predict the missing edges.

GraphMAE2 [76] applies multi-view random re-mask decoding as a regularization for MGM pretraining.

Mole-BERT [10] combines a contrastive learning objective and a masked atom modeling objective for MRL. Specifically, they observe that mask atom prediction is an overly easy pretraining task. Therefore, they employ a GNN tokenizer pretrained by VQ-VAE [43] to generate more complex reconstruction targets for masked atom modeling.

D.2 Experimenets in Section 4 and Table 5

Here we elaborate the experimental setting for pretraining on 2 million molecules from ZINC15 [42] and fine-tuning on the eight classification datasets in MoleculeNet [28]: BBBP, Tox21, ToxCast, Sider, ClinTox, MUV, HIV, and Bace. This setting covers the experiments in Section 4 and Table 5.

Molecule representations. For SimSGT and other compared methods, we follow previous works [12, 4] and use a minimal set of molecule features as the graph representations (Table 9a). These features unambiguously describe the two-dimensional structure of molecules.

Hyper-parameters. Table 9b summarizes the hyper-parameters. We use different hyper-parameters given different graph encoders. The architectures of the two graph encoders are borrowed from previous works: GINE [12] and GTS [27]. We use large batch sizes of 1024 and 2048 to speed up pretraining. We do not use dropout during pretraining. During fine-tuning, we 50% dropout in GINE layers and 30% dropout in transformer layers. The learning rate for the MUV dataset is 10 times smaller than other datasets. Following [4, 73], we report the last epoch's test performance. We

report the mean performances and the standard deviations across 10 random seeds. Baselines are reproduced using the same setting.

Linear probing experiments. Here we elaborate on the settings of our linear probing experiments (Figure 5b and Figure 7). Specifically, we randomly split the 2 million molecules from ZINC15 into train set (90%) and test set (10%). We train the MGM models on the train set and save the encoder's checkpoint every epoch. The linear classifiers are trained for 1000 epochs on the encoder's frozen hidden representations. We train linear classifiers using 25600 molecule samples from the training set and evaluate them on the whole test set.

Probing masked atom types (Figure 5b). We let linear classifiers predict the masked atom types using the masked atoms' hidden representations. During linear probing, we disable remask-v2 to obtain the masked atoms' hidden representations. Molecules are randomly masked by 0.35 during probing. We use accuracy (%) as the evaluation metric.

Probing FGs (Figure 7). Following [2], we extract 85 types of FGs for each molecule using RDkit [30]. FGs are represented by 85-dimensional binary vectors, whose each dimension indicates the presence of a certain FG. Afterward, we train multi-label linear classifiers on the frozen encoder's mean pooling outputs for FG prediction. Molecules are not masked during probing. We use ROC-AUC (%) as the evaluation metric.

D.3 Experiments in Table 6

We present the experimental setting for pretraining on the 50 thousand molecules from GEOM [46] and fine-tuning on the four molecule property prediction (regression) datasets and two DTA datasets. Our experimental setting follows that in [45]. This setting covers the experiments in Table 6.

Molecule representations. In the graph autoencoder, we use 9-dimensional node features and 3-dimensional edge features of molecules provided by the OGB [77] package, following GraphMVP [45]. The features are summarized in Table 10a. Note that, our tokenizer uses only the atomic numbers as node features and does not use edge features.

Hyper-parameters. The hyperparameters are summarized in Table 10b. We tune the hyperparameters in the fine-tuning stage using the validation performance. Following [45], we report the test performance at the epoch selected by the validation performance. We do not use dropout during pretraining. During fine-tuning, we 50% dropout in GINE layers and 30% dropout in transformer layers.

For a fair comparison, we reproduce Mole-BERT [10]'s performance by pretraining on the 50 thousand molecule samples from the GEOM dataset [46]. The original Mole-BERT is trained on a larger dataset of 2 million molecules from ZINC15 [42].

D.4 Experiments in Table 7

The hyperparameters for fine-tuning on the QM datasets are reported in Table 11.

E More Experimental Results

In this section, we provide more experimental results. If not especially noted, these experiments employ an autoencoder of GTS encoder and GTS-Small decoder with remask-v2, and a tokenizer of a single-layer SGT of GIN. Other settings follow that in Appendix D.2.

Influence of edge features for pretrained GNN-based tokenizer. We ablate the impact of the 'bond type' and 'bond direction' edge features in pretrained GNN-based tokenizers. We use GINE and

0

50

100

150

200

Pretrain steps

0.1

0.5

1.0

MSE loss

with BN

without BN

-4

-2

-1

0

1

2

4

Value

0.0

0.5

1.0

Count

1e6

with BN

without BN

(a) MSE loss curve with respect to pretrain steps. (b) Histograms for the values of the tokenizer's output.

GIN for the tokenizer with and without edge features. Table 12 shows that including edge features in GNN-based tokenizers negatively influences the transfer learning performance. Therefore, we exclude edge features from pretrained GNN-based tokenizers in our experiments.

Influence of Batch Normalization layers in SGTs. The Batch Normalization [41] (BN) layers in SGTs are crucial to avoid loss vanishment. Figure 8a presents a comparison between SGT 'with vs without BN'. Without the BN layer, the MSE loss drops to lower than 0 . 01 within a few steps of pretraining. Such small loss values lead to significant model underfitting.

As shown by Figure 8b, the token values of SGT without BN follow a sharp distribution: the values are primarily distributed around zero, and their standard deviation (std) is smaller than 0 . 35 . This minor std issue might be caused by the smoothing effect of GNNs [78]. An expressive neural network ( i.e., a graph autoencoder) can quickly fit this sharp target distribution and minimize the loss to a negligible value, causing the problem of loss vanishment. However, if a BN layer is used, it forces each dimension of the tokenizer output to have an std of 1 . 00 , so as to 'spread out' the distribution of the SGT tokens. These new SGT tokens of a larger std are harder to fit. They keep the MSE loss at a reasonable range of 0 . 10 ∼ 0 . 15 (Figure 8a).

Mask ratio. We apply random node masking throughout the experiments [12]. Figure 9 presents SimSGT's sensitivity with respect to the mask ratios. SimSGT is not sensitive to mask ratios such that a wide range of ratios (0.25 ∼ 0.45) can generate competitive performances. The ratio of 0.35 achieves the best performance. This ratio is much lower than that for images, where a ratio of 0.75 can generate promising performances [15].

Balancing the distribution of reconstruction targets. As shown in Figure 11, the popularly used ZINC15 dataset includes 12 types of atoms, and 95% of the atoms are distributed on the top three atom types. This skewed distribution renders the node-level token reconstruction an easy pretraining task [10]. Figure 10 shows that the accuracy of predicting node-level tokens converges quickly. Such an easy pretraining task can lead to suboptimal performance as suggested by existing SSL literature [79, 80]. In Figure 11, we show that the induced subgraphs of a single-layer SGT ( i.e., one-hop rooted subtrees) follow a more balanced distribution than the distribution of nodes. SGT tokens also have a larger vocabulary size: ZINC15 includes 555 types of one-hop rooted subtrees. Consequently, the accuracy of predicting tokens of a single-layer SGT takes more epochs to converge (Figure 10).

Pooling Method for Subgraph Representations. In previous experiments, we use mean pooling to obtain the subgraph representations for motif-based tokenizers, following the method of obtaining graph representations in [4, 12]. Here we add results for MGSSL tokenizer using sum and max

0.25

0.30

0.35

0.40

0.45

0.50

0.55

Mask ratio

74

75

76

ROC-AUC (%)

75.2

75.5

75.8

75.4

75.2

74.5

74.8

































C

N

O

S

F

Cl

Br

I

P

B

Si

Sn

Node type

0%

10%

20%

30%

40%

50%

60%

70%

Proportion

74.0

11.0

10.8

1.9

1.1

0.9

0.3

0.0

0.0

0.0

0.0

0.0

Distribution of nodes

C:C2

C:C3

C:CN

O:C

C:C2N

C:C

N:C2 C:CNOC:C2O O:C2

N:C3

C:CO

O:S

N:CN

F:C

C:O

others

Subgraph type

0%

5%

10%

15%

20%

25%

30%

35%

40%

Proportion

29.6

7.9

7.2

5.6

5.6

4.8

4.6

3.9

3.4

3.3

3.1

1.8

1.3

1.3

1.1

1.1

14.5

Distribution of one-hop rooted subtrees

pooling in Table 13. The results show that mean pooling yields the highest performance, affirming the soundness of our previous experiments.

Full results of Section 4. We provide the full results of experiments in Section 4. Table 14 contains the full results of Table 3a and Table 4. Table 15 contains the full results of Table 3b and Figure 6.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors investigate tokenization and decoder in masked graph modeling (MGM), and present Simple GNN-based Tokenizer (SGT) as well as stacking GINE and GraphTrans (GTS) as an efficient decoder. SGT uses non-trainable linear graph aggregation to get tokenization of graphs. It also introduces a variant of remask for the encoder and decoder of GTS. Experiments comparing to other SSL baselines show the proposed method achieves superior performance on multiple molecular property benchmarks. Comprehensive ablations studies also demonstrated the effectiveness of SGT as well as GTS in molecular representation learning compared to other design choices in MGM.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow overall. \n2. Systematic study of tokenization and decoder can be a valuable contribution to MGM for molecules. Since multiple works have applied masked modeling in molecular representation learning but few have extensively studied the design space of the MGM. \n3. The paper includes comprehensive investigations of MGM design choices in section 4, which validates the effectiveness of the proposed SGT and decoder. \n4. Technical details as well as complete results are reported in supplementary material, demonstrating the experimental soundness.\n\nWEAKNESSES:\nFrom my perspective, the major weakness is the miss of quantum-mechanics-related benchmarks. QM-related properties play an important role in applications. Thus performance on QM benchmarks can further help evaluate the effectiveness of the proposed method. \n\nPlease find more details in the \"Questions\" below.\n\nQUESTIONS:\nMajor questions: \n1. SGT applies only linear aggregation operations to graphs without nonlinear update functions. What will happen if nonlinearity is included on SGT?\n2. In Finding 2, the authors write \"it indicates that remask constrains the encoder’s efforts on graph reconstruction, allowing it to focus on MRL\" which suggests an interesting mismatch between graph reconstruction and MRL. Can the authors discuss more about why the mismatch exists and how it provides insights into designing self-supervised learning framework for MRL. \n3. Table 3b suggests that pretrained GNN tokenizer benefits from depth, however, the proposed SGT achieves the best performance with one single layer. What could be the cause of such observation?\n4. The work benchmarks on multiple MoleculeNet and DTA tasks. There are still important quantum-mechanics-related benchmarks (e.g., QM9, MD17) that are not included. Though the proposed method built upon 2D graphs is unlikely to compete with SOTA equivariant GNNs that leverage 3D information. Benchmarking against other SSL baselines on QM benchmarks provides a more comprehensive evaluation. \n5. In Table 1, why are Mole-BERT and GraphMAE included as baselines in GINE encoder but not in GTS encoder setting?\n\nMinor questions:\n1. In section 2.1, the authors mention using mean pooling to obtain representation of the subgraph. Some works also integrate max or summation pooling. Will that affect the performance in molecular property predictions?\n2. From my perspective, the authors explain too many details regarding BRICS in section 2.2. Yet BRICS is not the focus of the work. The authors may consider truncating the paragraphs.\n\nLIMITATIONS:\nThe authors have included reasonable discussions regarding potential limitations in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
The paper examines the effectiveness of tokenizer and decoder in the self-supervised representation learning of molecular graph following masked auto-encoding framework. Specifically, the paper adopts GraphTrans architecture for its encoder and a smaller GraphTrans for its decoder. A simple GNN-based architecture serves as tokenizer in the framework. The proposed method is pre-trained on 2 millions molecules and is evaluated using 8 classification datasets. The ablation analysis is conducted for the proposed architecture.

STRENGTHS:
- The paper is well-written and easy to follow.
- The idea of improving tokenizer and decoder in self-supervised learning framework is well-motivated.

WEAKNESSES:
- The proposed method utilizes a simple GGN-based architecture to learn the feature embeddings. However, these feature embeddings only serve as target for masked auto-encoding. I think it is misleading to call it tokenizer.
- The performance of GraphMAE [1] and Mole-BERT [2] in Table 5 is lower than one reported in the original paper.

[1] Graphmae: Self-supervised masked graph autoencoders. Hou et al. In KDD 2022.
[2]  Mole-BERT: Rethinking pre-training graph neural networks for molecules. Xia et al. In ICLR 2023.

QUESTIONS:
-  My main concern is about the reported performance which may cause unfair comparison.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly revisits the graph tokenizers and the graph autoencoders in Masked Graph Modeling (MGM) frameworks. Authors examine the roles of different tokenizers as the MGM’s reconstructions targets and propose a simple GNN-based tokenizer method and a decoding strategy. The experiment results show the effectiveness of the designed tokenizer and decoding strategy, as well as some insightful findings.\n\nSTRENGTHS:\n1. The paper is well written and mostly easy to understand.\n2. The revisiting part is clear and the summary of current tokenizers and autoencoders are systematic.  \n3. The code is provided by an anonymous link.\n\nWEAKNESSES:\n1. The motivation for the proposed method is somehow unclear and the novelty is limited. The revisiting part takes a lot of space to introduce the existing tokenizers and decoding strategies. There’s no explicit insight drawn from the revisiting results, which can be helpful for designing the proposed method. \n2. The technical contributions of this work are less and somewhat incremental. The proposed methods are all based on the current MGM framework and only change a little about the tokenizer and remask strategy. Concretely, the proposed simple GNN-based tokenizer is based on the previous pretrained GNN-based tokenizer concept and the remask-v2 has already been widely used in many scenarios to prevent transformer layers from processing masked items.\n3. From the experiments in Table3, the improvements of the proposed tokenizer and decoder are limited. In Table3 (b), when the depth of tokenizer is 4 or 5, the performance of proposed tokenizer is even worse than previous tokenizers. Additionally, there’s no standard error reported. \n4. The comparison of Table5 and Table6 is somehow unfair. The GNN encoders or decoders of baselines may be different from this work. So if the GNN encoders of this work are strong, the higher performance may be because of the encoders, rather than the proposed tokenizers and decoding strategy of this work.\n5. There’s no theoretical analysis of the proposed method.\n\nQUESTIONS:\n1. What are the real differences between the proposed tokenizer and the pretrained GNN-based tokenizer? Can the proposed method be considered as a special case of the previous GNN-based tokenizer? Authors are encouraged to investigate more about the insights behind the different tokenizers. \n2. In the experiments, the authors use ZINC15 as the pretrained dataset. What if more pretrained datasets are used? Will the proposed method still perform well in different size of pretrained dataset? Authors are encouraged to do more experiments or do some theoretical analysis about why the proposed method work well than others.\n\n\nThe response from authors addressed these questions and clarified the confused parts in the paper.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors attempt to categorize existing approaches for pretraining neural networks on molecular graphs and assess their contributions to pretraining quality.  They then propose a new strategy for pretraining molecular graphs and compare it to existing results.

STRENGTHS:
I was very impressed by this paper: the research was well-motivated, and the approach used was novel.  I also found the paper informative and easy to read: the authors clearly describe how they came to the conclusions they did and what motivated their explorations.

WEAKNESSES:
Maybe I was looking in the wrong places, but I struggled to find something to criticize here.  The only thing that I could find is that the SimSGT framework is strongly reminiscent of the famous BYOL work (https://doi.org/10.48550/arXiv.2006.07733) and a short discussion of the similarities between the two would improve the paper.

QUESTIONS:
This is primarily a question for future work and my own academic interest in the field.  In many practical applications for learning on molecular graphs SOTA approaches typically consist of regression on classical fingerprint techniques, combined using ensembling methods (note that this typically isn't well-represented on public leaderboards, often due to the fact that these datasets can become overfit and that there is substantial research interest in neural approaches).  It seems like modifying the methods proposed here to use fingerprint-based methods  instead of the GNN-based tokenizer, e.g. by enumerating the Morgan fingerprints an atom takes place and hashing, could be a fruitful research direction.  I'd be curious to hear the authors thoughts on this, or alternative deterministic featurizations to the GNN-inspired tokenizer.

LIMITATIONS:
Limitatinos are appropriately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper proposes a masked graph modeling framework called Simple GNN-based Tokenizer (SGT) for molecular graph analysis. Extensive experiments show the performance of the proposed method.

STRENGTHS:
1. The article is well-written and easy to understand.
2. The proposed framework is attractive to this research community.
3. The experimental results further corroborate the authors' view.

WEAKNESSES:
1. The limitations of the previous methods are not explained clearly. What is the limited understanding of tokenizer and decoder?
2. Theoretical analysis is weak. The paper only provides experimental results to support its idea, but it remains unclear why removing the nonlinear update function in each GNN layer can train a better encoder.
3.  In Table 5, the improvement of the proposed method over some baselines (e.g., RGCL) is incremental, which can not well support its effectiveness.
4. Lack of comparison of computational time.
5. Missing baselines [1,2].
[1] S2GAE: Self-Supervised Graph Autoencoders are Generalizable Learners with Graph Masking.
[2] GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner.

QUESTIONS:
N/A

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07