DiP-GNN: Discriminative Pre-Training of Graph Neural Networks
Authors: Simiao Zuo
Haoming Jiang
Qingyu Yin
Xianfeng Tang
Bing Yin
Tuo Zhao
Venue: Submitted to ICLR 2023 (2023)

Abstract: Graph neural network (GNN) pre-training methods have been proposed to enhance the power of GNNs. Specifically, a GNN is first pre-trained on a large-scale unlabeled graph and then fine-tuned on a separate small labeled graph for downstream applications, such as node classification. One popular pre-training method is to mask out a proportion of the edges, and a GNN is trained to recover them. However, such a generative method suffers from graph mismatch. That is, the masked graph input to the GNN deviates from the original graph. To alleviate this issue, we propose DiP-GNN (Discriminative Pre-training of Graph Neural Networks). Specifically, we train a generator to recover identities of the masked edges, and simultaneously, we train a discriminator to distinguish the generated edges from the original graph's edges. The discriminator is subsequently used for downstream fine-tuning. In our pre-training framework, the graph seen by the discriminator better matches the original graph because the generator can recover a proportion of the masked edges. Extensive experiments on large-scale homogeneous and heterogeneous graphs demonstrate the effectiveness of the proposed framework. Our code will be publicly available.

Under review as a conference paper at ICLR 2023
DIP-GNN: DISCRIMINATIVE PRE-TRAINING OF GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Graph neural network (GNN) pre-training methods have been proposed to en- hance the power of GNNs. Specifically. a GNN is first pre-trained on large-scale unlabeled graph and then fine-tuned on separate small labeled graph for down- stream applications, such as node classification. One popular pre-training method is mask out proportion of the edges, and a GNN trained to recover them. However, such generative method suffers from graph mismatch. That is, the masked graph input to the GNN deviates from the original graph. To alleviate this issue, propose DiP-GNN (Discriminative Pre-training of Graph Neural Networks). Specifically, we train a generator to recover identities of the masked edges, and simultaneously, we train discriminator to distinguish the generated edges from the original graph's 's edges. The discriminator is subsequently used for downstream fine-tuning. In our pre-training framework, the graph seen by the discriminator better matches the original graph because the generator can recover a proportion of the masked edges. Extensive experiments on large-scale homo- geneous and heterogeneous graphs demonstrate the effectiveness of the proposed framework. Our code will be publicly available.
INTRODUCTION
Graph neural networks (GNNs) have achieved superior performance in various applications, such as node classification (Kipf Welling, 2017), knowledge graph modeling (Schlichtkrull et al., 2018) and recommendation systems (Ying et al., 2018). To enhance the power of GNNs, generative pre- training methods are developed (Hu et al., 2020b). During the pre-training stage, a GNN incor- large-scale unlabeled graph in a self-supervised
porates topological information by training manner. Then, the pre-trained model is fine-tuned on a separate small labeled graph for downstream applications. Generative GNN pre-training is akin to masked language modeling in language model pre-training (Devlin et al., 2019). That is, for an input graph, we first randomly mask out a propor- tion of the edges, and then a GNN is trained to recover the original identity of the masked edges.
One major drawback with the abovementioned approach is graph mismatch. That is, the input graph to the GNN deviates from the original one since considerable amount of edges are dropped. This causes changes in topological information, c.g., node connectivity. Consequently, the learned node embeddings may not be desirable.
To mitigate the above issues, we propose DiP-GNN (Discriminative Pre-training of Graph Neural Networks). In DiP-GNN, we simultaneously train a generator and a discriminator. The generator is trained similar to existing generative pre-training approaches, where the model seeks to recover the masked edges and outputs a reconstructed graph. Subsequently, the reconstructed graph is fed to the discriminator, which predicts whether each edge resides in the original graph (i.e., true edge) or is wrongly constructed by the generator (i.e., a fake edge). After pre-training, we fine-tune the discriminator on downstream tasks. Figure illustrates is related to Generative Adversarial Nets (GAN, Goodfellow et al. 2014), and detailed discussions presented in Section 3.4. We remark that similar approaches have been used in natural language processing (Clark et al., 2020). However, we identify the graph mismatch problem (see Section 4.5), which is specific to graph-relater related applications and is not observed in natural language processing.
training framework. Note that our work
The proposed framework is more advantageous than generative pre-training. This is because the reconstructed graph fed to the discriminator better matches the original graph compared with the

---

Under review as a conference paper at ICLR 2023
XA
Illustration of DiP-GNN. From left to right: Original graph; Graph with two masked edges (dashed lines); Reconstructed graph created by the generator (generated edges are the dashed red lines); Discriminator labels each edge /G] (generated) or [O] (original), where there are two wrong labels (shown in red).
Figure 1:
masked graph fed to the generator. Consequently, the discriminator can learn better node embed- dings. Such a better alignment is because the generator recovers the masked edges during pre- training, i.e, we observe that nearly 40% of the missing edges can recovered. We remark that in framework, the graph fed to the generator has missing edges, while the graph fed to the discrim- inator contains wrong edges since the generator may make erroncous predictions. However, em- than wrong ones, making discriminative pre-training
pirically we find that missing edges hurt more desirable (see Section 4.5 in the experiments).
We demonstrate effectiveness of DiP-GNN large-scale homogeneous and heterogeneous graphs. Results show that the proposed method significantly outperforms existing generative pre-training and self-supervised learning approaches. For example, on the homogeneous Reddit dataset (Hamil- ton et al.. 2017) that contains 230k nodes, we obtain an improvement of 1.1 in terms of FI score; and on the heterogeneous OAG-CS graph (Tang al., 2008) that contains 1.1M nodes, obtain an improvement of 2.8 in terms of MRR score in the paper field prediction task.
BACKGROUND
Graph Neural Networks. Graph neural networks compute a node's representation by aggregating information from the node's neighbors. Concretely, for multi-layer GNN, the feature vector h of node v at the k-th layer is
(R) Aggregate (f
{(1!t [((t-1) Vu Neighbor( 1/ Combinc (a! (0(0,,h'-1 1))
where Neighbor(e) denotes all the neighbor nodes of v. Various implementations of Aggregate(.) and Combine(-) are proposed for both homogencous (Defferrard et al., 2016; Kipf & Welling, 2017; Velickovic et al., 2018; Xu et al., 2019) and heterogeneous graphs (Schlichtkrull et al., 2018; Wang al., 2019; Zhang al., 2019; Hu al., 2020c).
Graph Neural Network Pre- Fre-Trainin. Previous unsupervised learning methods leverage the graph's 's proximity (Tang al., 2015) information gathered by random walks (Perozzi al., 2014; Grover & Leskovec, 2016; Dong et al., 2017: Qiu et al., 2018). However, the learned embeddings nodes, limiting the methods' applicabilityy Other unsupervised learn- ing algorithms adopt contrastive lcarning (Hassani & Ahmadi, 2020; Qiu et al.. 2020; Zhu et al., 2020; 2021; You al., 2020; 2021). That is, we generate two views of the same graph, and then maximize agreement of node presentations in the two views. However, our experiments reveal that these methods do not scale well extremely large graphs with millions of nodes.
cannot be transferred
Many GNN pre-training methods focus on generative objectives. For example, GAE (Graph Auto- Encoder, Kipf Welling 2016) proposes to reconstruct the graph structure; GraphSAGE (Hamilton et al., 2017) optimizes an unsupervised loss derived from a (Deep Graph Infomax, Velickovic et al. 2019) maximizes the mutual information between node representations and graph summary representation.
metrie; and DGI
There are also pre-training methods that extract graph-level representations, i.c., models are trained on large amount of small graphs instead of single large graph. For example, Hu et al. 2020a propose pre- pre-training methods that operate on both graph and node level; and InfoGraph (Sun al., 2020) proposes maximize the mutual information between graph representations and representa- tions of the graphs' sub-structures. In this work, we focus on pre- pre-training GNNs on a single large graph instead of multiple small graphs.
2

---

The discriminator is trained to distinguish edges that are from the original graph (i.e., the true edges) and edges that are not (i.e., fake edges). Specifically, given the true edges Eune and the fake ones Eake we first compute (n)n)) [n, (%) for every node n E .V, where (- 0%) is the discrimina- Under review as a conference paper at ICLR 2023
3
METHOD
We formally introduce the proposed discriminative GNN pre-training framework DiP-GNN. The algorithm contains two ingredients that operate edges and features.
3.1 EDGE GENERATION AND DISCRIMINATION
Suppose have graph 9 (W,E), E), where M denotes all the nodes and & denotes all the edges. We randomly mask out proportion of the edges, such that & = Um. where & is the unmasked of edges and &m is the of edges that are masked out.
For a masked edge e = (M1, (2) E Em, where n and n2 are the two nodes connected by e, the generator's goal is to predict n1 given n2 and the unmasked edges E.. For each node n. we compute its representation fg (n) = (n, 0g) using the generator (- 0g), which is parameterized by Pe. We remark that the computation of ho (-) only relies on the unmasked edges E,,. We assume that the generation process of each edge independent. Then, we have the prediction probability
exp (d(hg((1)) (m1), (n())) p(n [ Ev) Enrec exp (r())g(()) hg (12)))
C= (m1) (W Neighbor(n2)). (eigbbbbr(())))))
(1)
Here, C is the candidate set for n1, which contains all the nodes that are not connected to n2 except nı itself. Moreover, the distance function d(:,,) is chosen trainable cosine similarity, i.e.,
(Weoss) u) TT [| ||a
d(u, v)
where Woos is a trainable weight. The training loss for the generator defined as
(9) (m, - -log I2,, Ew),
(3)
which is equivalent to maximizing the likelihood of correct predictions.
The goal of the generator is to recover the masked edges in Em. Therefore, after we train the generator, we use the trained model to generate & = { (n))EE, m2))E... where each ûn is the model's prediction as ú argmax,'ec p(n'Imz E Because the generator cannot correctly predict every edge, some edges in Eg are wrongly generated (i.e., not in Em). We refer such edges as fake edges, and the rest as true edges. Concretely, we denote the true edges Eiee = E, (E,, Nεg), i.c., the unmasked edges and the edges correctly generated by the generator. Correspondingly, denote the fake edges eake = \ Erve.
tor model parameterized by 02. We highlight that different from computing hg (.). the computation of ha(-) relies on all the edges, such that the discriminator can separate a fake edge from true one. Then, for each edge (Ri, (22) E Etruc U Etke. the discriminator outputs
Plake = p(e E &rake 1Eime: Etake) sigmoid (d(ha((n)) ha())))),
(4)
where d(.,-) is the distance function in Eq. 2. The training loss for the discriminator is the binary cross-entropy loss of predicting whether an edge is fake or not, defined as
(0g) EeES UE
le E {tae log(ptas) 1(e E Euve) log(1 Přake),
(5)
where 1(-) is the indicator function.
The edge loss is the weighted sum of the generator's and the discriminator's loss
( 0 = C,, + (0)),
(6)
where is hyper-parameter Note that structures of the generator S and the discriminator fā are flexible, e.g., they can be graph convolutional networks (GCN) graph attention networks (GAT).
3.2 FEATURE GENERATION AND DISCRIMINATION
In real-world applications, nodes are often associated with features. For example, in the Reddit dataset (Hamilton et al., 2017), a node's feature a vectorized representation the post corre- sponding the node. As another example, in citation networks (Tang al., 2008), paper's title
3

---

Under review as a conference paper at ICLR 2023
can be treated as node's feature. Previous work (Hu et al., 2020b) has demonstrated that generating features and edges simultaneously can improve the GNN's representation power.
Node features can be either texts (e.g.. in citation networks) vectors (c.g., in recommendation systems). In this section, we develop feature generation and discrimination procedures for texts. Vector features akin to encoded text features, and can use linear layers to generate and discriminate them. Details about vector features are deferred to Appendix B.
For text features, we parameterize both the feature generator and the feature discriminator using bi-directional Transformer models (Vaswani et al., 2017), similar to BERT (Devlin et al., 2019). Denote f (., 0g) trmg o emb,(-) the generator parameterized by 0g. where emby is the word embedding function and trmm, denotes subsequent Transformer layers. For an input text feature [_,,-_ ... 4z] where is the sequence length, we randomly select indices mask out, i.c. we randomly select an index set M C (1.... LJ. For masked position i M, the prediction probability given by
exp (xmb [g
p(x,1x)
Evocab exp (emb, (a') Bi (2'))) (x,) = tring (Wpro) (1x) emby (7)
Here Wproj is a trainable weight and hg (nx) is the representation of the node corresponding to computed by the edge generation GNN. Note that we concatenate the text embedding emby(1:) and the feature node's s embedding hg (mx), such that the feature generator can aggregate information from the graph structure. We train the generator by maximizing the probability of predicting the correct token, i.e., by minimizing the loss
(6 0 = Ex log p(a; x).
After we train the generator, we use the trained model to predict all the masked tokens, after which we obtain a new text feature x Here, we set -:50 = x; for & .M and N. = I; for i E M,, where T. = arginax/'evccal p(xx[x) is the generator's prediction.
The discriminator is trained to distinguish the fake tokens (i.e., wrongly generated tokens) from the true ones (i.e., the unmasked and correctly generated tokens) in xoon Similar to the generator, we denote f (..6)) = trma o emba(.) as the discriminator parameterized by 0/. For each position i, the discriminator's 's prediction probability is defined
pl= xx;)= sigmoid (w" (wTva((e vv(()"")), va(2;"") trina (wpro) [ha(nx)) emba(=;""))
Here w and Wproo are trainable weights and (1x) is the representation of the node corresponding to x computed by the edge discriminator GNN. The training loss for the discriminator is
() 0/) = Ex El= 11: x;) log(Pme) 1 (2 + x. log(1 - Piree)
(10)
where Prue p( r.)) and 1 is the indicator function.
The text feature loss is defined as
C( 0/. 0% 0%) c/ 0/) + XC)
(11)
where À is hyper-parameter.
3.3 MODEL TRAINING
We jointly minimize the edge loss and the feature loss, where the loss function is
C(B 05, 0% 8/) = (0", 83) + L (8 8/, 02, 05) = (C, + Co 05)) + ( 22103,0))
(12)
Here, À is the weight of the discriminator's loss. We remark that our framework is flexible be- cause the generator's is loss (L5 and C() is decoupled from the discriminator's (C; and C)). As such, existing generative pre-training methods can be applied to train the generator. In DiP-GNN, the discriminator has better quality than the generator because of the graph mismatch issue (see Sec- tion 4.5). Therefore, after pre-trat tre-training, we discard the generator and fine-tune the discriminator on downstream tasks. A detailed training pipeline is presented in Appendix A.
4

---

Under review as a conference paper at ICLR 2023
3.4 COMPARISON WITH GAN
We remark that our framework different from Generative Adversarial Nets (GAN, Goodfellow et al. 2014). In GAN, the generatorddisriminatto training framework is formulated as a min-max game, where the generator trained adversarially to fool the discriminator. The two models are updated using alternating gradient descent/ascent.
However, the min-max game formulation of GAN is not applicable our framework. This is be- cause in GNN pre-trianing, the generator generates discrete edges, unlike continuous pixel values in the image domain. Such property prohibits back poc-pproaggtton from the discriminator the gen- erator. Existing works (Wang et al., 2018) use reinforcement learning (specifically policy gradient) to circumvent the non-differentiabiiityy issue. However, reinforcement learning introduces extensive hyper-parametes tuning and suffers from scalability issues. For example, the largest graph used in Wang et al. 2018 only contains 18k nodes, whereas the smallest graph used in our experiments has about 233k nodes.
Additionally, the goal of GAN is to train good-quality generators, which is different from our focus. In our diseriminative pre-training framework, focus the discriminator because of better graph alignments. In practice, we find that accuracy of the generator is already high even without the discriminator, e.g., the accuracy is higher than 40% with 255 negative samples. And we observe that further improving the generator does not benefit downstream tasks.
EXPERIMENTS
We implement all the algorithms using PyTorch (Paszke al., 2019) and PyTorch Geometric (Fey & Lenssen, 2019). Experiments are conducted on NVIDIA A100 GPUs. By default, we use Hetero- geneous Graph Transformer (HGT, Hu al. 2020c) as the backbone GNN. We also discuss other choices in the experiments. Training and implementation details are deferred to Appendix C.
4.1 SETTINGS AND DATASETS
Settings. We consider node transfer setting in the experiments. In practice we often work with a single large-scale graph, on which labels are sparse. In this case, we can use the large amount of unlabeled data as the pre-training dataset, and the rest are treated as labeled fine-tuning nodes. Correspondingly, edges between pre-training nodes are added to the pre-training data, and edges between fine-tuning nodes are added to the fine-tuning data. In this way, the model cannot see the fine-tuning data during pre-training, and vice versa.
We remark that our setting is different from conventional self-supervised learning settings, namely we pre-train and fine-tune on two separate graphs. This meets the practical need transfer learning, e.g., trained GNN needs to transfer across locales and time spans in recommendation systems.
Homogeneous Graph. We use the Reddit dataset (Hamilton et al., 2017), which is a publicly available large-scale graph. In this graph, each node corresponds post, and is labeled with "subreddit". Each node has 603-dimensional feature vector constructed from the corresponding post. Two nodes (posts) are connected if the same user commented on both. The dataset contains posts from 50 subreddits sampled from posts initiated in September 2014. In total, there are 232.965 posts with an average node degree of 492. We use 70% of the data as the pre-training data, and the rest as the fine-tuning data, which are further split into training, validation, and test sets equally. We consider node classification as the downstream fine-tuning task.
Product Recommendation Graph. We collect in-house product recommendation data from an e-commerce website. We build a bi-partite graph with two node types: search queries and product ids. The dataset contains about 633k query nodes, 2.71M product nodes, and 228M edges. We sample 70% of the nodes (and corresponding edges) for pre-training, and the rest are evenly split for fine-tuning training, validation and testing. We consider link prediction as the downstream task, where for each validation and test query node, we randomly mask out 20% of its edges to recover. For each masked edge that corresponds to query node and a positive product node, we randomly sample 255 negative products. The task is to find the positive product out of the total 256 products.
5

---

Under review as a conference paper at ICLR 2023
Heterogeneous Graph. We use the OAG-CS dataset (Tang al., 2008; Sinha al., 2015), which is a publicly available heterogeneous graph containing computer science papers. The dataset contains over 1.1M nodes and 28. 48.4M edges. In this graph, there are five node types (institute, author, venue, paper and field) and ten edge types. The "field" nodes are further categorized into six levels from 1.0 to L,5, which are organized using hierarchical tree. Details are shown in Figure 2.
We use papers published before 2014 as the pre-training dataset (63%), papers published between 2014 (inclusive) and 2016 (inclusive) as the fine-tuning training set (20%), papers published in 2017 as the fine-tuning validation set (7%), and papers published after 2017 as the fine-tuning test set (10%). During fine-tuning, by default we only use 10% of the fine-tuning training data (i.e., 2% of the overall data) be- cause in practice labeled data are often scarce. We consider three tasks for fine-tuning: author name disambiguation (AD). paper field classifi- cation (PF) and paper venue classification (PV).
aftiliation
Institute
Author
confereneo!
firs:lastiothhr
in
joumaa/ preprint
cootain
Venue
Paper
Field
citation
Figure 2: Details of OAG-CS. There are 5 node types (in black) and 10 edge types (in red).
For paper field classification, we only consider 1.2 fields. the experiments, we use the pre- processed graph from Hu et al. 2020b.
4.2 IMPLEMENTATION DETAILS
Graph subsampling. In practice, graphs are often too large to fit in the hardware, e.g., the Reddit dataset (Hamilton et al., 2017) contains over 230k nodes. Therefore, we sample a dense subgraph from the large-scale graph in each training iteration. For homogeneous graphs, we apply the LADIES algorithm (Zou et al., 2019), which theoretically guarantees that the sampled nodes are highly inter-connected with each other and can maximally preserve the graph structure. For hetero- geneous graphs, we use the HGSampling algorithm (Hu et al., 2020b), which is a heterogeneous version of LADIES.
Node sampling for the edge generator. In the edge generator, for masked edge (s, t), we fix the node t and seek to identify the other node s. One approach to identify s from all the graph nodes, i.c., by setting C N in Eq. 1. However, this task is computationally intractable when the number of nodes is large, i.e., the model needs to find s out of hundreds of thousands of nodes. Therefore, sample some negative nodes { " such that (s) t) & & Then, the candidate set to generate the source node becomes {s, sy.... ... 2 Ruyg instead of all the graph nodes N. We remark that such a sampling approach is standard for GNN pre-training and link prediction (Hamilton et al., 2017: Sun al., 2020; Hu al., 2020b).
Edge sampling for the edge discriminator. In computing the loss for the discriminator, the number of edges in E,, is significantly larger than those in Eg. i.c., we only mask small proportion of the edges. To avoid the discriminator from outputting trivial predictions (i.e., all the edges belong to E.), we balance the two loss terms C2. Specifically, we sample Ed E, such that | a ElEgl where is a paree-pareeeter. Then, we compute LS on &g and E:. Note that the node representations ha are still computed using all the generated and unmasked edges E and E,,
4.3 BASELINES
We compare our method with several baselines in the experiments. For fair comparison, all the methods are trained for the same number of GPU hours.
GAE (Graph Auto-Encoder, Kipf& & Welling 2016) adopts an auto-encoder for unsupervised learn- ing on graphs. In GAE, node embeddings are learnt using a GNN, and we minimize the discrepancy between the original and the reconstructed adjacency matrix.
GraphSAGE (Hamilton et al., 2017) encourages embeddings of neighboring nodes to similar. For each node, the method learns a function that generates embeddings by sampling and aggregating features from the node's 's neighbors.

---

Under review as a conference paper at ICLR 2023
Table 1: Experimental results on homogeneous graphs. We report FI averaged over 10 runs for the Reddit data and MRR over 10 runs for the product recommendation data. The best results are shown bold.
Table Experimental results on OAG-CS (heterogeneous). Left to right: saper-field, paper-venue author-name name-dsaambiguation We report MRR over 10 runs. The best re- sults are shown bold.
Reddit
Recomm.
PF
PV
AD
w/o pre-train
87.3
46.3
w/o pre-train
32.7
19.6
60.0
GAB
88.5
56.7
GAE
40.3
24.5
62.5
GraphSAGE
88.0
53.0
GraphSAGE
37.8
22.1
62.9
DGI
87.7
53.3
DGI
38.1
22.5
63.0
GPT-GNN
89.6
58.6
GPT-GNN
41.6
25.6
63.1
GRACE
89.0
51.5
GRACE
38.0
21.5
62.0
GraphCL.
88.6
GraphCL.
38.0
22.0
61.5
JOAOv2
89.1
JOAOv2
38.6
23.5
62.8
DiP-GNN
90.7
60.1
DiP-GNN
44.1
27.7
65.6
38
DIP-ONN GPE-NN
68
35
Test MRR
66
Finc-tane
Test MRR
45
Test MRR
30
64
40
25
62
35
60
20
1.0
0.2
0.4
0.6
0.2
0.4
Percentage of labaled data
Percentage of lahaled data
Percentage of lahaled data
(a) Author disambiguation.
(b) Paper field classification.
Paper classification.
Figure 3: Model performance vs. amount of labeled data on OAG-CS.
DGI (Deep Graph Infomax, Velickovic et al. 2019) maximizes mutual information between node representations and corresponding high-level summaries of graphs. Thus, node's 's embedding sum- marizes a sub-graph centered around it.
GPT-GNN (Hu et al., 2020b) adopts a generative pre-training objective. The method generates edges by minimizing a link prediction objective, and incorporates node features in the framework.
GRACE (Graph Contrastive Representation, Zhu al. 2020) leverages a contrastive objective. The algorithm generates two views of the same graph through node and feature corruption, and then maximize agreement node representations in the two views.
GraphCL (You et al., 2020) is another graph contrastive learning approach that adopts node and edge augmentation techniques, such as node dropping and edge perturbation.
JOAO (Joint Augmentation Optimization, You et al. 2021) improves GraphCL by deigning a bi-level optimization objective to automatically and dynamically selects augmentation methods.
4.4
MAIN RESULTS
In Table and Table w/o pre-train means direct training on the fine-tuning dataset without pre- training. Results the Reddit dataset are Fl scores averaged over 10 runs, and results the product recommendation graph are MRR scores averaged over 10 runs. All the performance gain have passed hypothesis test with p-value < 0.05.
Table summarizes experimental results on the homogeneous graphs: Reddit and Recommenda- tion. We see that pre-training indeed benefits downstream tasks. For example, performance of GNN improves by at 2 0.4 Fl on Reddit (DGI) and 2 5.2 MRR on Recommendation (GRACE). Also, notice that among the baselines, generative approaches (GAE and GPT-GNN) yield promising per-

---

Under review as a conference paper at ICLR 2023
90.8
90.8
90.8
90.6
90.6
90.4
90.4
90.4
90.2
90.2
90.2
90.0
90.0
90.0
89.8
89.8
89.8
128
512
1.0
100
(a) Neg. nodes generation.
(b) Pos. edges for discrimination.
(c) Weight discriminator's loss.
Figure 4: Ablation experiments Reddit. By default, set the number of negative nodes to 256, the factor of positive edges to 1.0, and weight of the discriminator's loss to 20.
Table 3:
Test Fl score of
Table 4:
Test Fl of models with different backbone graph neural networks Reddit.
88.5
model variants on Reddit.
87.5
86.5
Model
F1
Model
HGT
GAT
85.5
Wrong Edges Missing Edges
Feature
90.7
84.5
Edges 90.4
w/o pretrain
87.3
86.4
0.0
0.2
0.5
0.8
Features
90.2
GPT-GNN
89.6
87.5
Figure 5: Fl proportion of manipulated edges on Reddit.
RandomEdges
89.8
DiP-GNN
90.7
88.5
formance. On the other hand, the contrastive method (GRACE, GraphCL and JOAO) does not scale well to large graphs, e.g., the OAG-CS graph which contains 1.1M nodes and 28.4M edges. By using the proposed discriminative pre-training framework, our method significantly outperforms the baseline approaches. For example, DiP-GNN outperforms GPT-GNN by 1.1 Reddit and 1.5 on Recommendation.
Experimental results the heterogeneous OAG-CS dataset are summarized in Table 2. Similar to the homogeneous graphs, notice that pre-training improves model performance by large margins. For example, pre-training improves MRR by least 5.1, 2.5 and 2.5 on the PF, PV and AD tasks, respectively. Moreover, by using the proposed training framework, models can learn better node embeddings and yield consistently better performance compared with all the baselines.
Recall that during fine-tuning on OAG-CS, we only use 10% of the labeled fine-tuning data (about 2% of the overall data). In Figure 3, we examine the effect of the amount of labeled data. We see that model performance improves when we increase the amount of labeled data. Also, notice that DiP-GNN consistently outperforms GPT-GNN in the three tasks under all the settings.
4.5 ANALYSIS
Comparison with semi-supervised learning. We com- pare DiP-GNN with semi-supervised learning method: C&S (CorrectSSmooth, Huang et al. 2020). Figure 6 summarizes the results. We that C&S yields a 0.5 improvement com- pared with the supervised learning method (i.e., w/o pre-train). However, performance of C&S is significantly lower than both DiP-GNN and other pre-training methods such as GPT-GNN.
90
88
86
Hyper-parameters. There are several hyper-parameters that we introduce in DiP-GNN: the number of negative nodes that are sampled for generating edges (Section 4.2); the num- ber of positive edges that are sampled for the discriminator's task (Section 4.2); and the weight the discriminator's loss (Eq. 12). Figure illustrate ablation experimental results the Reddit dataset. From the results, see that DiP-GNN is robust to these hyper-parameters. parameters. We remark that under all the settings, ours model behaves better than the best-performing baseline (89.6 for GPT-GNN).
w/o pre-train
C&S
DiP-GNN
Figure 6: Test Fl on Reddit.
8

---

Under review as a conference paper at ICLR 2023
60
DiP GNN
Table 5: Generator and discriminator performance vs. pro- portion of masked edges during pre-training. Coverage is the proportion of true edges input to the models.
58
GPE
Test MRR
544
52-
Masked%
Acc
Coverage Dis.
Gen.
Dis.
Gen.
Ratio
0.2
0.8
0.95
20
0.50
0.87
0.80
0.90
x1.13
Figure 7: proportion of masked edges on product recommendation.
Performance vs.
80
0.33
0.84
0.20
0.46
x2.30
95
0.20
0.80
0.05
0.24
x4.80
Model variants. We also examine variants of DiP-GNN. Recall that the generator and the dis- criminator operate on both edges and node features. We first check the contribution of these two factors. We also investigate the scenario where edges are randomly generated, and the discriminator still seeks to find the generated edges. Table 3 summarizes results on the Reddit dataset.
We see that by only using edges, model performance drops by 0.3; and by only using node features, performance drops by 0.5. This indicates that the graph structure plays a more important role in the proposed framework than the features. Also notice that performance of RandomEdges is unsatis- factory. This because implausible edges are generated when using a random generator, making the discriminator's 's task significantly easier. We remark that performance of all the model variants is better than the best persooeefnn baseline (89.6 for GPT-GNN).
Table 4 examines performance of our method and GPT-GNN using different backbone GNNs. Re- call that by default, we use HGT (Hu et al., 2020c) as the backbone. We see that when GAT (Velick- ovic et al., 2018) is used, performance of DiP-GNN is still significantly better than GPT-GNN.
Missing edges hurt more than wrong edges. In our pre-training framework, the generator is trained reconstruct the masked graph, after which the reconstructed graph fed to the discrimi- nator. During this procedure, the graph input to the generator has missing edges, and the graph input to the discriminator has wrong edges. From Figure 5, we see that wrong edges hurt less than missing ones. For example, model performance drops by 0.7% when 50% of wrong edges are added to the original graph, and performance decreases by 1.8% when 50% of original edges are missing. This indicates that performance relies on the amount of original edges seen by the models. Intuitively, wrong edges add noise to the graph, but they do not affect information flow. On the contrary, miss- ing edges cut information flow. Moreover, in practice work with graph attention models, and the attention mechanism can alleviate the wrong edges by assigning low attention scores to them.
Why is discriminative pre-training better? Figure 7 illustrates effect of the proportion of masked edges during pre-training. We see that when we increase the proportion from 0.2 to 0.8, performance of GPT-GNN drops by 6.1, whereas performance of DiP-GNN only drops by 3.3. This indicates that the generative pre- pre-training method is more sensitive to the masking proportion.
Table 5 summarizes pre-training quality. First, the generative task (i.e., the generator) is more diffi- cult than the discriminative task (i.e., the discriminator). For example, when we increase the propor- tion of masked edges from 20% to 80%, accuracy of the generator drops by 17% while accuracy of the discriminator only decreases by 3%. Second, the graph input to the discriminator better aligns with the original graph. For example, when 80% of the edges are masked, the discriminator sees 2.3 times more original edges than the generator. Therefore, the discriminative task is more advan- tageous because model quality relies on the number of observed original edges (Figure 5).
5 CONCLUSION AND DISCUSSIONS
We propose Discriminative Pre- Tre-Training of Graph Neural Networks (DiP-GNN), where we simul- taneously train a generator and a discriminator. During pre-training, we mask out some edges in the graph, and generator is trained to recover the masked edges. Subsequently, discriminator seeks to distinguish the generated edges from the original ones. Our framework is more advanta- geous than generative pre-training for two reasons: the graph inputted to the discriminator better matches the original graph: and the discriminative pre-training task better aligns with downstream node classification. We conduct extensive experiments to validate the effectiveness of DiP-GNN.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kevin Clark, Minh- Thang Luong, Quoc Le, and Christopher D. Manning. ELECTRA: pre- training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Erhiopia. April 26-30, 2020. OpenRe- view.net, 2020.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5- 10, 2016, Barcelona, Spain, pp. 3837-3845, 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume / (Long and Short Papers), pp. 4171-41886 Minneapolis, Minnesota, 2019. Association for Computational Linguisties. doi: 10.18653/v 18159/111N19-1423.
Yuxiao Dong, Nitesh V. Chawla, and Ananthram Swami. metapath2vee: Scalable representation learning for heterogeneous networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 17, 2017, pp. 135-144. ACM, 2017. doi: 10. 10.1145/3097983.39988)
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q Weinberger (eds.), Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Process- ing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 2672-26800 2014.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp. 2016. doi: 00.1145/9339772.293777
ACM,
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1024-1034, 2017.
Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-view view representation learning on graphs. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp- 4116-4126. PMLR, 2020.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020a.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. GPT-GNN: generative pre-training of graph neural networks. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (eds.), KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp. 1857-1867. ACM, 2020b.
Ziniu Hu, Yuxiao Dong Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Yennun Huang, Irwin King, Tie-Yan Liu, and Maarten van Steen (eds.), WWW '20: The Web
10

---

Under review as a conference paper at ICLR 2023
Conference 2020, Taipei, Taiwan, April 20-24, 2020, pp. 2704-22222111 2710. ACM/ IW3C2, 2020c. doi: 10.1145/33344333380027
Qian Huang. Horace He, Abhay Singh, Ser-Nam Lim, and Austin R Benson. Combining
label propagation and simple models out-performs graph neural networks. arXiv:2010. 13993, 2020.
arXiv preprint
Thomas N Kipf and Max Welling. arXiv: 1,000330, 2016.
Variational graph auto-encoders.
arXiv preprint
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett Advances Neural Information Process- ing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp.
2019.
Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk:
online learning of social represen- tations. In Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang. and Rayid Ghani (eds.), The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA August 24 27, 2014, pp. 701-710. ACM, 2014. doi: 10.1145/26233302222777
Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In Yi Chang, Chengxiang Zhai, Yan Liu, and Yoelle Maarek (eds.). Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9 2018, pp. 467.467. ACM, 2018, doi: 10. (00.1145/3159522.11539
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. GCC: graph contrastive coding for graph neural network pre-training. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (eds.), KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp. 1150-1160. ACM, 2020.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. European semantic web conference, pp. 697-607. Springer, 2018.
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, pp. 243-246, 2015.
Fan-Yo Yun-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. In 8th In- ternational Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. LINE: large-seale information network embedding. In Aldo Gangemi, Stefano Leonardi, and Alessandro Panconesi (eds.), Proceedings of the 24th International Conference on World Wide Web, WWW 2015, Flo- rence, Italy, May 18-22, 2015, pp. 1067-1077. ACM, 2015. doi: 10. 22.114//736777.274100
11

---

Under review as a conference paper at ICLR 2023
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Ametminer: extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 990-998, 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OvenReview.net, 2018.
Petar Velickovie, William Fedus, William L.. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Hongwei Wang, Jia Wang, Jialin Wang, Miao Zhao, Weinan Zhang, Fuzheng Zhang, Xing Xie, and Minyi Guo. Graphgan: Graph representation learning with generative adversarial nets. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the Sth AAAI Symposium on Educational Advances in Artificial In- telligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 2508-2515, AAAI Press, 2018
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S. Yu. Heterogeneous graph attention network. In Ling Liu, Ryen W. White, Amin Mantrach, Fabrizio Silvestri, Julian J. McAuley, Ricardo Baeza-Yates, and Leila Zia (eds.), The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pp. 2022-2032. ACM, 2019. doi: 10. 145// 3308558.3313562.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Rex Ying. Ruining He. Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Yike Guo and Faisal Farooq (eds.), Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018 pp- 974-983. ACM, 2018. doi: 10. 10.1145/221981993219999
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5823, 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning auto- mated. In International Conference on Machine Learning, pp. 12121-12132. PMLR, 2021.
Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V. Chawla. Heteroge- neous graph neural network. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Confer- ence Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pp. 793-803. ACM, 2019. doi: 10.1145/3222500333000
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint 200iv:2006. 04 13,, 2020.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pp. 2069-2080 2021.
12

---

Under review as a conference paper at ICLR 2023
Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. Layer-dependent importance sampling for training deep and large graph convolutional networks. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Ach-BBuc, Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11247-11256, 2019.
13

---

Under review as a conference paper at ICLR 2023
A DETAILED ALGORITHM
Algorithm is a detailed training pipeline of DiP-GNN. For graphs with vector features instead of text features, substitute the feature generation and discrimination modules with equations in Appendix B.
Algorithm 1: DiP-GNN: Discriminative Pre-training of Graph Neural Networks.
Input: Graph Gruli edge masking ratio; feature masking ratio; number of negative samples edge generator; proportion of positive samples for edge discriminator or; weight of the discriminator's loss number of training steps do // Graph subsampling.
for t= 0,
Sample subgraph = E) from Gtull;
// Edge generation.
Initialize the generated edge set & (} and the edge generation loss Lÿ 0; Construct the unmasked set of edges & and the masked set &m such that & = En. Emi Compute node embeddings using Ew:
for e (n1, 122) En do
Construct candidate set C for nı (ng is given during generation) via negative sampling; Generate (fi,,2)) where în E C:
Update the generated edge set Eg &g U {ē];
Update edge generation loss S.:
// Text Feature generation.
Initialize the feature generation loss C/ O:
for n E E do
For the node's 's text feature Xa, mask out some of its tokens;
Construct the generated text feature corr using the embedding of node n (computed during edge generation) and the feature generation Transformer model; Update the feature generation loss C/:
// Edge discrime nat ion. Initialize the edge discrimination loss L; 0; Compute node embeddings using &g Ex:
Sample Ed & such that |ad| = a Eyl:
for e = (n1, n22) E Eg U Ed do Determine if is generated using the embedding of nı and ne: Update the edge discrimination loss L3:
// Text feature discrimination ion. Initialize the feature discrimination loss C/ 0:
for n E N do
For the node's generated text feature _ determine whether each token is generated
using the embedding of node n (computed during edge discrimination) and the feature discrimination Transformer model;
Update the feature discrimination loss C/:
// Model updates.
Compute L (C; 4/) + X + </) and update the model;
Output: Trained discriminator ready for fine-tuning.
B GENERATION AND DISCRIMINATION OF VECTOR FEATURES
Node features be vectors instead of texts, e.g., the feature vector contain topological infor- mation such as connectivity information. this case both the generator and the discriminator are parameterized by linear layer.
14

---

Under review as a conference paper at ICLR 2023
To generate feature vectors, first randomly select some nodes Ng C N. For node E N, denote its feature vector Vn. then the feature generation loss
(W,) Σ l| vnll2 where V. W (n)).
nEN,
Here (n) is the representation of node and w is trainable weight. For node n N, we construct its corred feature your Vn, if n E Ng and va vn if E W N.
The discriminator's goal is to differentiate the generated features from the original ones. Specifically, the prediction probability is
p(n E Ngg) = sigmoid (Wāha(m))
where w/ a trainable weight. We remark that the node representation ha(2) is computed based on the corred feature yoorr Correspondingly, the discriminator's loss is
(Wa)) = Σ fn N logp(n E ENg) - 1(n E \W log(1 - p(n E (Ng)).
nEN
The vector feature loss L (0C, W. Hc W 24 W ( W computed similar to the text feature loss.
C IMPLEMENTATION AND TRAINING DETAILS
By default, we use Heterogeneous Graph Transformer (HGT, Hu al. 2020c) as the backbone GNN. In the experiments, the edge generator and discriminator have the same architecture, where we set the hidden dimension to 400, the number of layers to 3, and the number of attention heads 8. For the OAG dataset which contains text features, the feature generator and discriminator employs the same architecture: a layer bi-directional Transformer model, similar to BERT (Devlin et al., 2019), where neural network to 512.
the embedding dimension to 128 and the hidden dimension of the feed-forward
For pre- training, mask out 20% of the edges and 20% of the features (for text features mask out 20% of the tokens). We use AdamW (Loshchilov & Hutter, 2019) as the optimizer, where we to 0.001 and the weight decay 0.01. We adopt
10-8 the learning dropout ratio of 0.2 and gradient norm clipping of 0.5. For graph subsampling, we set the depth to and width 128. the same setting Hu al. 2020b.
3 (0.9, 0.999),
For fine-tuning, we use AdamW (Loshchilov & Hutter, 2019) as the optimizer, where we set 8 = (0.9, 0.999), e 10-6 and do not use weight decay. We use the same graph subsampling setting as pre-training The other paramm-rrssetess are detailed in Table 6.
Table 6: Hyper-parameten parameters for fine-tuning tasks.
Dataset
Task
Steps
Dropout
Learning rate
Gradient clipping
Reddit
2400
0.3
0.0015
0.5
Recomm.
1600
0.1
0.0010
0.5
PF
1600
0.2
0.0010
0.5
OAG-CS
PV
1600
0.2
0.0005
0.5
AD
1600
0.2
0.0005
0.5
15


Reviews:
[{"review_id": "urbdSBxvDr", "reviewer": "Reviewer_FMbA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a discriminative method for pre-training Graph Neural Networks. The main idea is to simultaneously train a generator to recover identities of the masked edges, and train a discriminator to distinguish the generated edges from the original graph’s edges. \n", "summary_of_the_review": "Overall, the proposed method makes sense. However, I feel the authors are selling the methodology wrong. Instead of saying that discriminative pre-training is better than generative training, the authors should present it as a method for applying generative and discriminative pre-training jointly. Otherwise, why don’t you just rely solely on discriminative pre-training?\n", "strength_and_weaknesses": "Strength\n- The proposed method makes sense.\n- The empirical results show the improved performance brought by the proposed methods on a number of  benchmarks.\n\nWeaknesses\n- There is no mentioning of n_1^hat or generated edges e_g in the loss of the generator. How should I understand that the generated edges are not captured in the generator loss?\n\n- Also, it is said that the discriminator is to distinguish edges that are from the original graph and edges that are generated. It would make sense to contrast the masked edges vs the generated edges, especially the ones that are different. However, the training loss for the discriminator seems to still focus on the unmasked edges vs generated edges. I don’t quite understand the intuition.\n\n- A masked edge has two end nodes n_1 and n_2. How do the authors decide which end node to predict?\n\n- Why is the same lambda shared between edge loss and feature loss, given these two losses are so different?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WFvGtAuyLOa", "reviewer": "Reviewer_S9Gb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, a pre-training method of GNN is proposed. The whole framework consists of generator and discriminator, and the discriminator is used as the pre-trained model. Therefore, the proposed method could be regarded as a variant of GAN on GNN, though there are significant differences between them. The authors also discuss the generation/discrimination of both edges and node features. ", "summary_of_the_review": "The paper is well-written and propose a method applied to the pre-training of GNN which is different from the existing GNN setings. \n\nHowever, I have some concerns about the experimental settings and the originality.", "strength_and_weaknesses": "### Strength\n- The paper is well-written and easy to follow. \n- The discussions of both edges and node features are sufficient. \n- The method is proposed to be applied to the pre-training of GNN which is different from the existing GNN setings\n\n### Weakness\n- Although the authors claim that \"*our setting is different from conventional self-supervised learning settings, namely we pre-train and fine-tune on two separate graphs*\", I still have some questions about the experiments: \n    - Could it show the superiority of DipGNN on transfer learning of GNN, since the authors claim \" This meets the practical need of transfer learning\"? \n    - The compared methods seems a little confusing. If the competitors are mainly the self-supervised/unsupervised methods, the methods proposed by Kaveh Hassani et al. (ICML 2020) and SGC (Felix Wu et al. ICML 2019) are missing. \n    - As DipGNN consists of pre-training and fine-tuning, do the all methods reported in Tables 1 and 2 have these two phases as well? \n    - As the pre-training technique is usually used in large-scale datasets, some experiments on OGB datasets may be convincing.\n\n- Although the details of generator and discriminator are different, the novelty seems a little common compared with the existing methods. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GPpUUssizu", "reviewer": "Reviewer_K6Uv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A popular pre-training GNN method is to mask out some edges and GNN is trained to recover them. However, there exists graph mismatch can mislead the performance. To handle this problem, the authors propose DiP-GNN which can better match the original graph. Specifically, the generator is trained to recover identities of the masked edges. Simultaneously, a discriminator is trained to distinguish the generated edges. Finally, the discriminator is chosen to accomplish the downstream tasks.", "summary_of_the_review": "To handle the graph mismatch in graph generative pre-training, this paper proposes an interesting DiP-GNN (Discriminative Pre-training of Graph Neural Network) pre-training framework. Specifically, the generator is trained to recover identities of the masked edges. Simultaneously, a discriminator is trained to distinguish the generated edges. \nThe major strength is that the proposed reconstructed graph fed to the discriminator will better match the original graph, compared with generative pre-training.\nThe major weakness is that there lacks some detail explanation about the statement and the generated graph. Apart from that, some equations lack explanation. Meanwhile, some typo errors need to be rectified such as adding the equation label.\nApart from that, I have an interesting question. In section 3.1, the authors mask some edges not nodes. So, if all adjacent edges of the node are masked, how to handle this node in training.\n", "strength_and_weaknesses": "### Strength:\nThe authors introduce generative mechanism into graph pre-training and propose a DiP-GNN (Discriminative Pre-training of Graph Neural Network) framework. Meanwhile, compared with generative pre-training, the proposed reconstructed graph fed to the discriminator will better match the original graph.\n\n### Weakness:\n1) Lack of analysis about the generated graph. As shown in Fig. 1, the different masked-edge impacts the performance. Hope the authors can clarify how to choose the masked edge and analysis the influence of the masked edge.\n2) Some statements are unclear and mislead the reader. For example, in section 1, the authors state that “we find that missing edges hurt more than wrong ones”, which is contrary to common sense but interesting. Although the authors utilize the experiment results to verify this point, I hope the author can explain and analyze it in more detail.\n3) Lack of some explanation of equations. For example, the loss function in section 3 lacks an explanation and formula label.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject