GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs
Authors: Yichuan Li
Kaize Ding
Kyumin Lee
Venue: EMNLP 2023 Findings (2023)
Keywords: textual attributed graph
text rich network
representation learning
graph neural network
language model
TL;DR: GRENADE combines language models & graph networks via self-supervised algorithms, excelling in text-attributed graph representation.

Abstract: Self-supervised representation learning on text-attributed graphs, which aims to create expressive and generalizable representations for various downstream tasks, has received increasing research attention lately. However, existing methods either struggle to capture the full extent of structural context information or rely on task-specific training labels, which largely hampers their effectiveness and generalizability in practice. To solve the problem of self-supervised representation learning on text-attributed graphs, we develop a novel Graph-Centric Language model -- GRENADE. 
Specifically, GRENADE harnesses the synergy of both pre-trained language model and graph neural network by optimizing with two specialized self-supervised learning algorithms: graph-centric contrastive learning and graph-centric knowledge alignment. The proposed graph-centric self-supervised learning algorithms effectively help GRENADE to capture informative textual semantics as well as structural context information on text-attributed graphs. Through extensive experiments, GRENADE shows its superiority over state-of-the-art methods.

GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs
Yichuan Li
Kaize Ding
Kyumin Lee
Worcester Polytechnic Institute
Northwestern University
Worcester Polytechnic Institute
yli29@wpi. edu
kaize. ding@northwestem. edu kmlee@wpi. edu
Abstract
Recent research has demonstrated that self- supervised learning (SSL) can substantially im- prove the effectiveness representation learning on text data (Reimers and Gurevych, 2019; Gao et al., 2021; Wu et al., 2020) without using human supervision. Those methods are commonly learned under the assumption that text documents are inde- pendently and identically distributed (i.i.d), which neglects the structural interdependencies among text nodes on TAG. However, the interdependen- cies between different text documents can provide valuable insights for understanding their semantic relationships. Take citation networks as an example, those academic papers (text nodes) that have cita- tion relationships often share similar topics. Hence, is necessary for SSL models to account for not only textual semantics but also structural context information.
Self-supervised representation learning on text- attributed graphs, which aims to create expres- sive and generalizable representations for vari- ous downstream tasks, has received increas- ing research attention lately. However, ex- isting methods either struggle to capture the full extent of structural context information rely task-specific training labels, which largely hampers their effectiveness and gener- alizability in practice. To solve problem of self-supere superviedd representation learning on text- attributed graphs, we develop a novel Graph- Centric Language model GRENADE. Specifi- cally, GRENADe exploits the synergistic effect of both pre-trained language model and graph neural network by optimizing with special- ized self supervisdd learning algorithms: graph- centric contrastive learning and graph-centric knowledge alignment. The proposed graph- centric self-supervised learning algorithms ef- fectively help GreNade to capture informative textual semantics well as structural context information on text-attributed graphs. Through extensive experiments, GreNADE shows its su- periority over state-of-the-arr methods. Im- plementation available at https://github.com bigheiniu/GRENNDE.
In fact, self-supervised representation learning TAG remains in its infancy: (i) Though recent research efforts (Zhao al., 2022; Chien al., 2021; Cohan al., 2020; Yasunaga et al., 2022) try to empower pre-trained language models (PLM) with structural context information, most of them still stay superficial by designing local structure- dependent SSL objectives. For example, both GIANT (Chien et al., 2021) and SPECTER (Cohan et al., 2020) train the language model by inferring the local neighborhood based on representations of text nodes. However, simply relying on those SSL objectives cannot help the PLM fully understand complex graph structures, especially compared to models like graph neural networks (GNN) (Kipf and Welling, 2017; Velickovic et al., 2018; Hamilton et al., 2017; Ding et al., 2022a); (ii) Meanwhile, another line of research (Mavromatis al., 2023; Zhao al., 2022) try to combine the advantages of both PLM and GNN by distilling the knowledge from one the other (Hinton al., 2015) and have shown promising results. Nonetheless, one major issue is that those methods are task-specific (e.g.,
1 Introduction
Text-Attributdd Graph (TAG) (Yang et al., 2021) (a.k.a,, Textual Graph) has been widely used for modeling a variety of real-world applications, such as information retrieval (Cohan et al., 2020; Yang et al., 2021), product recommendation (Zhu et al., 2021) and many more. In TAG, each node rep- resents text document, while the relationships among these text nodes are depicted by the edges, For instance, in citation networks, text nodes rep- resent academic papers, and edges are the citation relationship between different papers. To conduct different analytics tasks on TAG, the key is to learn expressive node representations for the text nodes.
Corrreppnnding Author.

---

semi-supervised node classification) and require numan-annotated labels enable knowledge distil- lation. Such an inherent limitation jeopardizes the versatility of their models for handling different and even unseen downstream tasks, which runs counter to the goal of SSL.
We conduct extensive experiments show that our model Grenade significantly and consistently outperforms state-of-the- methods on a wide spectrum of downstream tasks.
Problem Definition
To go beyond the existing learning paradigms and capture informative textual semantic and graph structure information, develop new model for self supervveed learning on TAG, namely GrENADE (Graph-Centrie Language Model). GRENADE built with PLM encoder along with an adju- vant GNN encoder that provides complementary knowledge for it. More importantly, GRENADE is learned through two new self-supervised learning algorithms: Graph-Centrr Contrastive Learning (GC-CL), structure-aware and augmentation-free contrastive learning algorithm that improves the representation expressiveness by leveraging the in- herent graph neighborhood information; and Graph- Centric Knowledge Alignment (GC-KA), which en- ables the PLM and GNN modules to reinforce each other by aligning their learned knowledge encoded in the text node representations.
We utilize bold lowercase letters such as d to represent vectors, bold capital letters like W to denote matrices and calligraphic capital letters like W represent sets. Let G (A, D) denote text-attributed graph with adjacency matrix A E {0, 1) DIx[DI and text set D. The A; when there is a connection between node and j. Each node represents text document which consists of a sequence of tokens D; [ww)
Notations.
Problem Given an inpul text-attributed graph (TAG) denoted as G=)A,D), our goal is to learn graph-centric language model PLM(-). that can generate expressive and generalizable representa- lion for an arbitray node on G: d,-PLM(D.). Note that the whole learning process is performed solely the input graph G without the utilization of human annotated labels.
Specifically, GC-CL enforees neighboring nodes share similar semantics in the latent space by considering them as positive pairs. Even without us- ing data augmentation, GC-CL performs node-wise contrastive learning to elicit the structural context information from TAG. In the meantime, GC-KA bridges the knowledge gap between PLM and GNN by performing dual-level knowledge alignment on the computed representations: at the node level, minimize the distance between the representations learned from two encoders that focus different modalities. the neighborhood level, we minimize the discrepancy between two neighborhood simi- larity distributions computed from PLM and GNN. By virtue of the two proposed graph-centric self- supervised learning algorithms, we are able to learn GrENade that can generate expressive and general- izable representations for various downstream tasks without using any human supervision. In summary, our work has the following contributions:
Proposed Approach: Graph-Centric Language Model (GRENADE)
To learn expressive representations from TAG in a self-supervised learning manner, we propose our Graph-Centrie Language Model GRENADE, which bridges the knowledge gap between Pre-trained Language Model (PLM) and Graph Neural Network (GNN). By optimizing two distinct encoders with a set of novel self-supervised learning algorithms, the PLM encoder and GNN encoder mutually reinforce each other, and we can finally derive our Graph- Centric Language Model (GReNadE). GNN. The overall framework is shown in Fig.
3.1 Model Architecture
Our proposed model Grenade is composed of Pre- trained Language Model (PLM) along with Graph Neural Network (GNN), which are optimized by a set of novel self-supervised learning algorithms. We first introduce the details about those two essential components as follows:
We develop GRENADE, which is graph-centric language model that addresses the undercxplored problem of self-supervised learning TAG. We propose two new self-supervised learning algorithms for TAG, which allow us to perform contrastive learning and knowledge alignment in graph-centric way.
The primary component PLM(-) is BERT (Devlin et al., 2018) based text encoder that projects sequence of tokens D, into vectorized text node representation d.:
PLM Encoder.
d,,P,
(1)

---

Graph-Centric CL
Graph-Centric Knowledge Alignment
lil..
NBH XA
5
6
all
TAG
GNN
PLM
Positive
Encoder
Encoder
Negative Instance
Figure I: Illustration of GRENADE. Given text-attributed graph (TAG), GRENADE is jointly optimized by two graph- centric self-supervised learning algorithms: graph-centric contrastive learning GC-CL and a dual-level graph- centric knowledge alignment, which comprises node-level alignment (MD-KA) and neighborhhod-level alignment (NBH-K- KA).
where d, is the hidden representation of the [CLS] token computed from the last layer of the PLM encoder.
for applying contrastive learning, while those aug- mentations may unexpectedly disturb the semantic meaning of the original instance (Lee et al., 2022).
GNN Encoder. As an adjuvant component, the GNN encoder GNN(.) is built with a stack of message- passing based GNN layers, which compute the node i's representation by iteratively aggregating and transforming the feature information from its neigh- borhood (Hamilton al., 2017). For each node i. its representation learned from a L-layer GNN encoder can be denoted as:
To counter the aforementioned issues, pro- graph-centric contrastive learning (GC-CL) algorithm, which structure-aware and augmentation-free. GC-CL exploits inherent graph knowledge from TAG and can applied both the PLM encoder and GNN encoder. As suggested by the Homophily principle (McPherson et al., 2001), ) neighboring nodes commonly share similar seman- tics, meaning that their representations should also be close to each other in the latent space. Based the PLM representation of node its K-hop neighboring nodes (t),, and the node excluded mini-batch instances B(i), the GC-CL objective for PLM can be defined as follows:
pose
e E GN", A)
(2)
where the input node feature matrix E0 is obtained from the hidden representations of [CLS] token from the last layer of a pre-trained BERT model.
3.2 Graph-Centris Contrastive Learning
In order to improve the learning capability of those two encoders without using any human-annotated labels, prevailing way is to conduct contrastive learning from either the text perspective (Gao et al., 2021) graph perspective (Ding et al., 2022c). However, most of the existing contrastive learn- ing methods have the following two limitations: (/) conventional instance-level contrastive learning methods merely encourage instance-wir wise discrimi- nation (Li et al., 2021b; Ding et al., 2023), which neglects the property of TAG, i.c., the relational information among text nodes. Hence, those in- stances that share similar semantics may be unde- sirably pushed away in the latent space; (2) existing methods commonly rely on arbitrary augmentation functions to generate different augmented views
sim(de.
Lec-CL
Σ log Σ Ejee()) si d d,)/ pEV(1)
[N(i]
(3)
where denotes the temperature and sim(:;) represents the cosine similarity function. Here C(i) = Ni)) U B()). Note that for node i, we con- sider its PLM representation di as the query instance. The positive instances are the representations of node i's K hhop neighboring nodes (dplp E (i))). Meanwhile, the negative instances the represen- tations of other text nodes excluding within the same mini- [j E B(i).
Similar to the PLM encoder, we also apply our GC-CL CL algorithm to the GNN encoder GNN(.). Specif-

---

ically, the objective function defined follows:
that is learned from the GNN encoder as the query. then construct the positive and negative instances based on the representations learned from the PLM encoder. Specifically, the positive instances include both the representation of node / as well as the representations of i's K -hoop neighboring nodes (i.e., {dplp (i))),, and the negative instances are the representations of other instances within the same mini-batch (d, J C B(i)). In the meantime, we also consider the di the query and construct its corresponding positive and negative instances in the same way. Here we omit the illustration for simplicity.
sp)(
Loc-cL2 [iii]
log Ejec() e esmm7s
(EN()
(4) where e; is the query instance. The positive in- stances are {eplp (N(1)) and the negative in- stances {eylj B(i)).
Apart from the conventional instance-level con- trastive learning counterparts, our graph-centric contrastive learning also enforces neighboring nodes to share similar representations. In a sense, this self-supervised learning algorithm is analogous to performing link prediction task based on the rep- resentations learned from the PLM encoder, which inherently elicits informative graph knowledge dur- ing the learning process.
By virtue of the proposed ND-KA algorithm, the representations of the same node learned from two separate encoders will be pulled together in the la- tent space. In the meantime, ND-KA also encourages neighboring nodes have similar representations across different modalities.
3.3 Graph-Centric Knowledge Alignment
In this work, our ultimate goal is to learn expres- sive and generalizable representations that encode informative textual semantics within each text node as well the relational information among nodes. However, individually conducting the graph-centric contrastive lcarning on either PLM or GNN is not enough due to the lack of knowledge exchange be- tween them. To better align and enhance the knowl- edge captured by the PLM and GNN encoders, we propose dual-level graph-centric knowledge align- ment algorithm TAG, which includes Node-Leve Knowledge Alignment (N) and Neighborhood- Level Knowledge Alignment (NBH-KA).
Neighborhood-Level Knowledge Alignment. To further facilitate knowledge alignment between PLM and GNN, we propose Neighborhood Level Knowledge Alignment (NBH-KA) to align the neigh- borhood similarity distributions learned from the two encoders. Specifically, NBH-KA first computes the neighborhood similarity distribution between the query node and its K-hop neighboring nodes (ii) well as the rest nodes within the same mini- batch B(i) for each encoder. Then we minimize the KL-divergence between the distributions to align the knowledge between two encoders. The corresponding learning objective is:
Node-Level Knowledge Alignment. Different from the previously introduced graph-centric con- trastive learning, which only focuses on single- modal contrasting, ND-KA tries to align the knowl- edge across the two encoders by performing graph- centric contrastive learning in a cross-modal form. For each node i, based on its representations learned from the PLM encoder and GNN encoder (i.e., d; and ei respectively), formulate the objective of ND-KA as follows:
LNEH-KA (KL(PPLL
KL (i) (i)) (2))
(6)
PPLx(t)==oftmaae sof((())=so djj/T),
Pam = softmaxxftxxxxio (sim(e; (sim(e,,e))/1) 0/)/T):
(sim
where PFlM(1) and Panx are the neighborhood similarity distributions for PLM encoder and GNN encoder respectively. From certain perspective, our NEH-KA algorithm can be considered self- supervised form of knowledge distillation. Specif- ically, NBH-KA KA leverages the neighborhood infor- mation as self-supervision guide the knowledge alignment process. Moreover, we conduct two-way knowledge alignment across two encoders, which is different from original knowledge distillation.
dp))
LND-XA
-1
Σ log (1)(
Miii
EjE((1) ds)m(e;,d;)/r
+ log Σ ΣjEcto si /2,
(5) where (1)) = (i) (1) and ((i) = N(1)
B(i). Note that for node i, we first consider e;

---

3.4 Model Learning
layer's hidden state of [CLS] token from frozen bert-base-uncased the text node representa- tion. As for OGB, we utilize the default features from benchmark datasets, such as averaged word embeddings and bag-of-words representations. (2) Text-only sulf-vupe representation learning models: +ERT+MLM and SimCSE (Gao et al., 2021); In BERT+MLM, we apply masked language model- ing BERT for the target TAG. SimCSE employs instance-wiss contrastive learning to learn text node representation. (3) Structure-augmennte language models: This calegory includes SPECTER (Co- han et al., 2020), GIANT (Chien et al., 2021) and GLEM (Zhao et al., 2022). SPECTER applies graph- centric contrastive learning on the language model, and GIANT employs the extreme multi-label classifi- cation train the language model for neighborhood prediction. It is noteworthy that GLEM utilizes task- specific labels to alternatively guide the pre-trane pre-trained language model PLM and graph neural networks GNN through self-knowledge distillation. Com- pared with GLEM, our proposed method GrenAde is fully self-supervised and does not rely on any numan-annotaten labels. The learned text node representations can be efficiently and effectively generalized to downstream tasks
In order to learn our graph-centric language model GRENADE on TAG without using human-annotated labels, we jointly optimize the proposed graph- centric contrastive learning and knowledge align- ment algorithms. For the sake of simplicity, define the overall training loss follows:
C-Coc-cL + Loc-cL2 + LND-KA KA + LNBH-KA- (7)
Once the training is finished, we can freeze the parameters of the PLM encoder and use to compute the representations of each text node with a forward pass. The computed representations can be further used for different downstream tasks.
4 Experiment
To evaluate the effectiveness of our approach GRENADE, we conduct comprehensive experiments on different datasets and various downstream tasks,
4.1 Experimental Setup
Evaluation Datasets. We evaluate the general- izability of the representations computed from different methods on three Open Graph Bench- mark (OGB) (Hu et al., 2020) datasets: ogbn-arxiv, ogbn-products, ogbl-citation2. These datasets are utilized to evaluate the performance of few-shot and full-shot node classification, node clustering, and link prediction tasks. It should be noted that ogbn- arxiv and ogbn-products datasets are not originally designed for link prediction evaluation. There- fore, we create two link prediction tasks based on these two datasets, respectively. Furthermore, we incorporate obgl-citation2 into our node classifica- tion experiment. The statistical information of the datasets is shown in Tab. 1. More comprehensive information regarding the dataset extension can be found in Appendix A.
Implementation Details. To ensure fair com- parison, we implemented all baseline methods and GRENAdE using the same language model, specif- ically pert-base-uncee method, GRENADE, we set the K-hop neighbor as 1, set the temperature parameter r to 0.05 in all the loss functions. The optimal hyperparameter (W(i) is discussed in $ 4.5. Please refer to Appendix B for additional implementation details.
For our proposed
4.2 Experimental Results
Few-shot Node Classification. To assess the gen- eralizability of learned representation to new tasks under low-data scenarios, we conduct experiments on few-shot node classification. Under this set- ting, the classification models are trained with varying numbers of labeled instances per class (k: - (2, 4, 8, 16)). We repeat the experiment 10 times and reported the average results along with the standard deviation. The classification models uti- lized in this evaluation are the multilayer perceptron (MLP) and GraphSAGE (Hamilton al., 2017). The hyperparameters for the classification models can be found in Appendix B. As the result showed in Tab. 2, several observations can be made: (7) In most cases,
Dataset
#Nodes
#Edges
#Classes
ogbu-arxiv
169,343
1,166,243
40
ogbn-products
2,449.029
61.859.140
47
ogbl-citations2
2,927,963
00.387.995
172
Table 1: Statistical information of the datasets.
Baseline Methods. The baseline methods in- cluded in our study encompass three categories: (1) Untuned representations: BERT (Devlin al., 2018) and OGB representations (Mikolov et al., 2013; Hu et al., 2020). For BERT, we extract the final

---

MLP
GraphSAGIE
Methods
=
==
==8
==
J ==
8 ==
==
ogbn-arxiv
OGB"
32.16+1.08
37.81+1.c6
42.33-1.07
45.84+0.47
49.91-3.48
35.52+1.42
19.30
62.21
BERT
34.06+2.78
40. 40.29.22.16
46.59-0.88
50.17+1.0
3..92-3.21
57.11+1.06
60.36+1.17
64.10 64.10=o.01
BERT+MLM
38.4 38.4144.11
46. 46.72
51.88 51.89_0.0
55.8 55.87 1::23
53.02
57 17.76
61.94
65. 65.222...m
28.8331.08
1.246
37-78811.0
43.225g..0
46.61=2.ur
33.866g1.20
57-75quu.9
62.39
SPECTER
50.15g2.21
54.46.00.0
58.74_0.83
61.63
53.85
39. 39.46g1.65
63. 13.001
06, 06.41.9.1
GIANT"
48.50+2.30
56. 1..00
5..80=1.0a
64.10
50.18-2.46
36.30=1.60
09. 09.24:11.3
63.48=0.17
2..11
45.52
83..7
55.39900.9
GREVADE
55.85
61.10 61.10
63.95
66.62..0.47
57.17 10.04
60.49 60.49...00 20.00
64.65
67.50.0.4
ogho-products
OGB"
9.47
13. 13.54.1.44
16.83_2.0
20.71
24.74ma.u
33. 33.14 13.11412.5
38.38
44. 44.1922.0
BERT
20.53
30.91
40.82_1.52
47.77
40.53
10.43
57. 67.29
59.03
BERT+MLM
38.64+a.35
47.15s8.41
55.95=0.50
39.57
54.73-4..6
60.80+2.65
64.63:11.4
07.75=1148
SimCSE
7.76
11.30
1..94_1.91
25.61
24.99
37.05 17.00511.73
44.74
50. 50.1322.04
SPECTER
27.86.4.14
2..70
51.51=1.64
27.63
4..74_4..8
51.91
57-72+2.52
60.77=1..6
GIANT
20.83
31.37
22.84_2.72
51.36
30.90=4.85
40.80+4..1
52.34422.00
58.38-2.30
GLEM
41.72 4.88
42 22.5043.45
43 43.0642.77
48.64-2201
GrENADE
24.51
49. 64 30
59.34
65.00 65.05.11.00
59.63
64.95 64.96
07.38
70.92
ogbl-citation2
OGB"
21.78
25.54 25.54
27.39_0.60
29.53
18.83_3.12
22.4 22.49
31.38
35.199...9
BERT
17.95
20.84
23.84_1.36
26.77
21.05_2.40
25.65, 15.65
28.0642.08
35.84_1.48
BERT+MIM
32.03
34.39a2a:
38.53_1.2
41.6620.05
26.28-2.55
28.1042.47
37.6941.39
42.63=1.21
SimCSE
13.27
16.68
20.01
23.57
20.01
28.1142.00
28.17+1.06
31.T1+1.28
SPECTER
30. 94.8142.94
35.31 35.31
39.74_1.92
42.31
31.57
35 35.6612.11
37 27.23.2.70
45.59
GIANT
38.9321.81
1.000
45.81=1.90
62.05go.77
35.75_2.7a
38 43.43
40.99.3.42
49.4421.08
GLEM
30.86
33.7811.s8
47.36
51.42_1.41
GRENADE
46.40=2.00
47.93=1.s4
50.61+0.71
53.75+0062
40.63+3.77
44.44=2.68
49.15+1.7s
52.41+104
Table 2: Experiment results of few-shot node classification." indicates that the text node representations are obtained from their official release. indicates no result for GLEM. This because the representation learning stage, GLEM will utilize the labeled dataset to train GNN.
oghn arxiv ARI
oghn products ARI
ogbl citation
Methods
ACC
NMI
ACC
NMI
ACC
ARI
NMI
OGB
39.57.c.30
12.76.1.22
17.43
38.53 1.43
16.46:4424
20.21
40.31 10.40
22.82.n.54
40.57 10.35
BERT
11.21
8..1
12.78
18.7
12.17
35. 0.82
40.57 16.31
22.58
33.3810.4
BERT+MI
40.07+0.00
15.06g0.17
19.24400:1
64.35gn.t
60.58=1.a1
54.84-o.3:
49. 49.13
31.91 20.40
44.44+0:44
SimCSF
33.42g0.71
5.80
9.6246.74
30.40.1.0
29.00g2.02
19.88_1.or
26.67
14.2610005
SPECTER
58. 50.00s
40.44:1.37
42.81+9.23
70. 70
69.82:1.a
58.90 0..
63.366.4..5
48.66.uu34
58.96
GIANT
58 D0120.0
39 39.6921.22
43.73g006
61. 61
17. 78.00:411
47.51=1.14
63, 133.06g0.63
18. 18.37
58 68..6800..0
GRENADE
61.961o.m
11.85
1..63
73.54__0.7
60.6411.04
64.141om
64.891o.m
50.22 po.42
59.68.n.21
Table 3: Experiment results of node clustering.
SSL based methods achieve better performance than non-SSL methods BBERT+MLM,SPECTES and GIANT GLEM), this indicates the significance of SSL in enhancing model transferability new tasks with limited labels. (2) Among satc--ofttheart TAG representation models, GrENAde achieves the best performance on these datasets. This indicates the superior generalization ability representations extracted by GRENADE. The designed knowledge alignment allows the GRENADE to integrate the pre- trained knowledge from PLM encoder and structure inductive bias learned by GNN encoder. These ex- pressive representations can be easily and efficiently generalized to few-shot learning tasks.
dataset under MLP, GraphSAGE (Hamilton et al., 2017) and RevGAT-KD (Li et al., 2021a). As the result shown in Tab. 4, we can observe that: (1) GrENaDe achieves the best performance across all the bascline methods. (2) The performance gap between GRENADE and some baseline methods like GIANT and GLEM becomes smaller more labeled data provided, but Grenade consistently better than these methods.
Node Clustering. In the node clustering task, we utilize the Icarned text node representations to train K K-means++ model for clustering instances. We apply the default hyperparameters of K-means++ as provided by scikit-learn (Pedregosa et al., 201 1). The number of clusters set to the number classes the dataset, and we assign the cluster label
Full Data Node Classification. We also conduct the node classification experiment with full training

---

Methods
arxiv
products
citation2
MLP
GraphSAGE
RovGAT-KD
MLP
GraphSAGE
MLP
GraphSAGE
OGB
2..50
71.49=0.27"
74.26+0.17"
61.06+0.0s"
75.81=0.40
48.98-1.74
62.10=0.22
BERT
62.91+0.60
70.97=0.a5
73.59
60.90 1.00
50.70=0.50
56.75=
60.38
67.71
T55665=0.14
70,39
70.86
80.90=0.11
66. 00.00
64.57=0.50
SinCSE
60.58
67.26g2o.2
73.6920.15
67.6920.05
80.41
53.91
=0.30
58.75.0.21
SPECTER
70.40
74.1920.18
20.67
75.38
79.42-29.3
54, 200.07
66.5820.25
GIANT
+3.08+0.06"
74.59=0.25
70.1220.16
79.8220.07
82.03-20.05
67. 67.24
70.32
GLEM
73.59
$2.02:0..r
68.25
GRENADE
73.1610.12
75.00.0.10
76.21
81.5810.1
83.11
68. 88..1..
70.89
Table 4: Supervised node classification performance comparison on benchmark datasets. Boldfaced numbers indicate the best performance downstream models. The represents the experiment results adopted from (Chien al., 2021), while denotes the experiment results adopted from (Zhao et al., 2022).
based on the most common label within each clus- ter. Following the evaluation protocol described in (Ding et al., 2022b), we report three clustering eval- uation metrics: accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI). We exclude the GLEM model from this evaluation since it requires the labels during representation learning. To ensure robustness, we perform 10 runs K-means++ with different random seeds and report the average results. As shown in Table 3. we observe that the structure augmented SSL methods outperform the text-only euffssuerrvied representation learning methods (GRENADE, GIANT, SPECTER BERT+MLM, SimCSE). This indicates structure-augmeented SSL methods can understand the context within graph structure that can lead to more accurate node representations, which in turn can lead to better clustering. Additionally, our proposed method GRENADE consistently out- performs all baseline methods. The improvement demonstrates that GRENAde can better preserve neighborhood information which will inform the clustering methods of how data points are intercon- nected or related each other.
uation metric used for this task is the mean recip- rocal rank (MRR), which measures the reciprocal rank of the positive instance among the negative instances for each query instance and takes the av- erage over all query instances. As shown in Fig. 2. observe that GrENAde significantly outperforms other approaches. In fact, GrENADE achieves least 4% performance improvement compared to methods that utilize structure-augmented self- supervised learning loss (SPECTER and GIANT) across all datasets. This demonstrates that GRENADE can better preserve the neighborhood information, which is consistent with the findings from 4.2.
4.3 Representation Visualization
To visually demonstrate the quality of the learned representations, we apply I-distributed stochastic neighbor embedding (NE) (Van der Maaten and Hinton, 2008) to for representation visualization. We compare GRENADE with two best-performing baseline methods, including SPECTER and GIANT on the arxiv dataset. In Fig. present the t t-SNE visualization of the embeddings for 10 randomly sampled classes comprising 5,000 subsampled in- stances. The colors in the visualization correspond to the labels of these subsampled instances. From Fig. 3, we observe GrEnAde exhibits denser clus- ters and more explicit boundaries among different classes compared to the baseline methods. This ob- servation confirms that GreNADE can learn compact intra-class and distinct inter-class representations.
OGa
MERT-MLM
SP SPECTER
GLEM
DERRT
GUANT
GEEADDE
MBB
YEA
06
24.4
ogen-arxiv
ogbi-titatio2
Figure 2: Experiment results of link prediction.
Link Prediction. Next, we evaluate the learned representation in predicting missing connections given existing connections from TAG. We aim to rank the positive candidates (1 or positive in- stances) higher than the negative candidates (1,000 negative instances) for each query node. The eval-
(a) SPECTER
(b) GIANT
(c) GRENADE
Figure 3: Representation visualization arxiv dataset.

---

4.4 Ablation Study
We contend that our methodology strikes harmo- nious balance between assimilating structural data and filtering out extraneous noise, thereby ensuring consistent performance our assessments.
To validate the effectiveness of graph-centric con- trastive learning and graph-centric knowledge align- ment, we conducted ablation study GRENADE. In this study, respectively remove GC-CL, ND-KA, and NBH-KA KA from the full model and report these model variants' performance in Tab. 5. In general, the full model GRENADE has the best performance in most cases, and we notice a performance de- cline when any of the components is removed or replaced, underscoring the significance of each component in GRENADE. Remarkably, we observe performance improvement in link prediction after removing graph-centric contrastive learning (w/o GC-CL> GRENADE in terms of MRR). Considering the task similarity between GC-CC CL and the link pre- diction, one possible explanation is that removing GC-CL CL could help the model mitigate overfitting and further improve performance for the link prediction task. Meanwhile, this observation, in turn shows that the dual-level graph-centric knowledge align- ment (ND-KA and NBH-KA) is effective for capturing structural context information from the TAG.
th.
12.5
23.0
12.3
12.7
Sace DiveBratton
Sice Crawiing
Lok Precition
Figure 4: Hyperparameter evaluation for K K-hop neigh- bors in the ogbn-arxiv dataset. Dashed lines indicate the peak performance of GrENAde with K 1.
1-Hop Neighbor Size. One crucial aspect of Grenade's SSL objectives is the hyperparameter [i)),, which controls the number of 1-hop neigh- bors considered for representation learning. To investigate the impact of subsampled neighbor size in GRENADE, we conduct a hyperparameter analysis on the full training dataset node classification, node clustering, and link prediction tasks. As shown in Fig. 5, we observe that GRENADE achieves its best performance with practical number of neighbors ((((i) for ogbn-arxiv and [N(i)) 1 for sehn-products). This finding particularly advan- tageous as it reduces the computational burden of the PLM encoder in graph-centre contrastive learn- ing and knowledge alignment between the PLM and GNN encoders.
Methods
MLP(ACC)
K-mcanst++ACCC
MRR
Avg. Rank,
GRENADE
73. 73.16
g1.96
52.73
1.33
12.00.
2.67
w/o KD-KA
68.0 10
60.69 20
29.15
3.33
w/o KB0I KA
73.0620.37
20050
52.56
2.67
Table 5: Ablation study of graph-centric contrastive learning (GC-CL) and gowlldge--lignmeee ogbn- arxiv datasets. "w/o" is the abbreviation of "without".
23.3
$1
4.5 Hyperparameter Analysis
22.4
15.1
K- K-hop Neighbors. We delved into understand- ing the impact of the K-hop neighbor selection on GRENADE'S efficiency. The choice of different K values directly affects the formulation of positive pairs in graph-centric contrastive learning (Eq. and Eq. 4), and the aligmment of knowledge between the graph neural network and the language model (Bq. 5 and Eq, 6). Based on the results presented in Fig. 4, it is evident that augmenting the hop distance adversely affects performance metrics in full data node classification (ACC of MLP), node clustering (ACC), and link prediction (MRR). This suggests that 1-hop neighbors optimally capture structural knowledge within our algorithm. However, when extending 2-hop 3-hop neighbors, there's heightened risk of integrating noisy data. This in- sight aligns with the conclusions drawn from research, specifically SPECTER (Cohan et al., 2020).
(a) ogbn-arxiv
76.7
12.3
12.4
19.3
Mode Classifiction
Node Clustringg
Lnn Pridition
(b) ogbn- products
Figure 5: Analysis of hyperparameters on both ogbn- arxiv and ogbn-products datasets. The dashed line represents GRENAdE'S 's optimal result when for ogbn-arxiv and [W(1] = for [N(i)
5 Related Work
Learning with TAG. This problem involves learn- ing text node representations that encode both tex- tual semantics and structural context information.

---

A prominent approach uses structure-augmeented SSL objectives to incorporate structure informa- tion into language models (Chien et al., 2021; Cohan et al., 2020; Yasunaga et al., 2022; Os- tendorff et al., 2022). For example, SPECTER and LinkBERT focus on node-pair relationships (Co- han et al., 2020; Yasunaga et al., 2022), while GIANT targets extreme multiclass neighborhood prediction (Chien al., 2021). A concurrent work PATTON (Jin et al., 2023) trains language model through network-c masked lan- guage modeling and link prediction. A parallel line of research tries to integrate pre-trained lan- guage models (PLMs) with graph neural networks (GNN) (Yang et al., 2021; Zhao et al., 2022; Mavro- matis et al., 2023; Shang et al., 2020; Zhang et al.. 2021). GraphFormers interweaves GNN with LMs' self-attention layers (Yang al., 2021). GLEM (Zhao al., 2022), GraDBERT (Mavromatis et al., 2023) and LRTN (Zhang et al., 2021) leverage labeled datasets for co-training between PLM and GNN. While they excel in node classification benchmarks, their reliance on labeled datasets limits the representa- tion's s generalization ability to new tasks.
Conclusion
In this paper, we introduce a self-supervised graph- centric language model: GRENADE, for learning expressive and generalized representation from tex- tual attributed graphs (TAG). GRENADE is learned through two self-supervised learning algorithms: (1) Graph-Centric Centric Contrastive Learning, which enables GRENADE to harness intrinsic graph knowl- edge through relational-aware and augmentation- free contrastive learning; and (2) Graph-Centric Knowledge Alignment, which facilitates the ex- change and strengthening of knowledge derived from the pre-trained language model encoder and the graph neural network encoder, thereby en- hancing the capture of relational information from TAG. We conduct experiments on four benchmark datasets under few-shot and full data node classifica- tion, node clustering, and link prediction tasks, and find that GrENAdE significantly and consistently outperforms baseline methods.
7 Limitations
In this section, we acknowledge the following constraints in our study: (1) Constraints on the Choice of Backbone Model. Our choice of the backbone model was restricted to the initialization of
Contrastive Learning. Contrastive learning is a self-supervised learning paradigm that aims to learn representations by distinguishing between positive and negative instances (Jaiswal et al., 2020). A key practice in contrastive learning is to use augmented versions of the same instance as positive instances and other instances as negative instances (Gao et al., 2021; He et al.; Radford et al., 2021). For ex- ample, SimCSE creates augmented views for each instance based on dropout (Srivastava et al., 2014). However, conventional instance-level contrastive learning only encourages instance-wise discrimi- nation (Li et al., 2021b) and commonly assumes different instances are i.i.d. which neglects the rela- tionships among different instances on TAG. Hence, conventional contrastive learning methods are inef- fective to learn expressive representation learning on TAG. To address those limitations, many recent methods seek to extend the design of positive pair construction by considering local neighborhood information (Cohan et al., 2020; Ostendorff et al., 2022). However, those methods cannot fully cap- ture complex graph structures. In contrast, our pro- posed method, GRENADE, leverages graph-centric contrastive learning and graph-centric knowledge alignment to fully exploit the structural context information from TAG.
in training GRENADE. This choice was necessitated by the limitations in computational resources available to us. Ex- ploration with alternative PLM backbones such GPT2(Radford et al., 2019) and RoBERTa(Liu al., 2019) has not been carried out and represents a promising direction for subsequent studics. (2) Comparison with Large Language Models. The nat- ural language processing domain has recently seen breakthroughs with state-of-the-art large language models like LLaMA(Touvron et al., 2023) and Chat- GPT (OpenAI, 2023), which have demonstrated exceptional performance in language understanding tasks. Our experiment confirms that GreNAde sur- passes existing representation learning techniques in TAG, but the performance of Grenade relative to these cutting-edge language models remains to be ascertained. (3) Breadth of Evaluation. In this work, we evaluate GrENAde primarily through node classification, node clustering, and link prediction tasks. However, there are other relevant evaluation dimensions. such as retrieval, reranking. co-view and others (Cohan et al., 2020). Future work will in- vestigate the applicability and capacity of GRENADE to broader tasks.

---

References
Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020.
Jinsong Chen, Kaiyuan Gao, Gaichao Li, and Kun He. 2023. Nagphormer: A tokenized graph transformer for node classification in large graphs.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. 2020. A survey on contrastive self-supervised learn- ing. Technologies. 9(1):2.
Eli Chien, Wei-Cheng Chang, Cho-Jui Hsich, Hsiang- Fu Yu, Jiong Zhang. Olgica Milenkovic, and In- derjit S Dhillon. 2021. Node feature extraction by self-supervised multi-scale neighborhood prediction. arXiv preprini arXiv:21 21002000410004
Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Xinyang Zhang, Qi Zhu, and Jiawei Han. 2023, Pat- ton: Language model pretraining on text-rich net- works. CoRR, abs/2305. 12268.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07880.
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In ICL.R
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and
Namkyeong Lee, Junseok Lee, and Chanyoung Park. 2022. Augmentation free self-supervised learning on graphs. In AAAI.
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. 202 Training graph neural net- works with 1000 layers. In ICML.
Kaize Ding, Jianling Wang, James Caverlee, and Huan Liu. 2022a. Meta propagation networks for graph few-shot semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. 202 lb. Prototypical contrastive learning unsupervised representations. In 9th International Conference Learning Representations, ICLR 2021.
Kaize Ding. Yancheng Wang, Yingzhen Yang, and Huan Liu. 2022b. Eliciting structural and semantic global knowledge in unsupervised graph contrastive learning. arXiv preprint 200.::2200000888..
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 190..1907..999
Kaize Ding. Yancheng Wang, Yingzhen Yang, and Huan Liu. 2023. Eliciting structural and semantic global knowledge in unsupervised graph contrastive learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 7378-7386.
Costas Mavromatis, Vassilis N Ioannidis, Shen Wang, Da Zheng, Soji Adeshina. Jun Ma, Han Zhao, Christos Faloutsos, and George Karypis. 2023. Train your own gnn teacher: Graph-aware distillation textual graphs arXiv preprint arXiv:2304. 106688
Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu. 2022c. Data augmentation for deep graph learning: A survey. ACM SIGKDD Explorations Newsletter.
Tianyu Gao, Xingeheng Yao, and Danqi Chen, 2021. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.0883
Miller McPherson, Lynn Smith-Lovin. and James M Cook. 2001. Birds of a feather: Homophily in social networks. Annual Review of Sociology, (1).
Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Advances in neural information processing systems. 30.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representa- tions in vector space. arXiv preprins arXiv: X
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsu- pervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Put- lern Recognition, CVPR 2020.
OpenAl. 2023, Gpt-4 technical report. arXiv preprint arXiv:2303.0877
Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp. and Georg Rehm. 2022. Neighborhood contrastive learning for scientific document represen- tations with citation embeddings. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503.02531.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Newral
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary

---

DeVito, Martin Raison, Alykhan Tejani, Sasank Chil- amkurthy, Benoit Steiner, Lu Pang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS.
Zhuofeng Wu, Sinong Wang. Jiatao Gu. Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv20012.15666
Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian. Sanjay Agrawal, Amit Amit Guangzhong Sun, and Xing Xie. 2021. Graphformers: Gon-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournaaap,au, M. Brucher, M. Perrot, and E. Duches- nay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. Linkbert: Pretraining language models with document links. arXiv preprint arXiv:2203. 15827.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021, Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th Interna- tional Conference on Machine Learning, ICML 2021, Proceedings of Machine Learning Research.
Xinyang Zhang, Chenwei Zhang, Xin Luna Dong, Jingho Shang, and Jiawei Han. 2021. Minimally Minimally-spperissed structure-rich text categorization via learning on text- rich networks. In The TheWebConf.
Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li. Xing Xie, and Jian Tang, 2022. Learning on large-scale text-attributed graphs via variational inference. arXiv preprint arXiv: 20. 14709.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models unsupervised multitask learners. OpenAl blog.
Jason Zhu, Yanling Cui. Yuming Liu, Hao Sun, Xue Li. Markus Pelger, Tianqi Yang. Liangjie Zhang, Ruofei Zhang. and Huasha Zhao. 2021. Texignn: Improving text encoder via graph neural network in sponsored search. In WWW.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprini arXiv: 10084.
Jingbo Shang. Xinyang Zhang, Liyuan Liu, Sha Li, and Jiawei Han. 2020. Nettaxo: Automated topic taxonomy construction from text-rich network. In Weenef
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting Journal of Machine Learning Re- search.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971.
Laurens Van der Maaten and Geoffrey Hinton. 2008 Visualizing data using I-sne, Journal of machine learning research, (11).
Petar Velickovie, Guillem Cucurull, Aranixa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Authony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, al. 2019. Huggingface's transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771.

---

A Dataset Details
Datascts
ogbn-arxiv
ogba-products
ogbl-citation2
hidden size
256
We extend ogbn-arxiv and egbn-productt for link prediction and we evaluate models on the test-split from these two node datasets. Specifically, for each source node i, we randomly choose its one neighbor the positive candidate and 1,000 negative candi- dates and would like the model to rank the positive candidate over the negative candidates. The nega- tive references are randomly-sampled from all the nodes from TAG that are not connected by i.
dropout
0.5
0.5
0.5
Ir
lc-3
Io-3
lo-3
epochs
500
50
50
batch_side size
-1
1024
1024
neighbors
-1--1
5-10
5-10
Table 7: Hyperparameter setting for GraphSAGE model in node classification.
Enhanced Node Classification Model. Besides MLP and GraphSage, incorporated the recent Graph Transformer Network, NAGphormer (Chen et al., 2023), into our node classification evalu- ation. As evidenced by the data in Tab. 8, our proposed approach consistently surpasses the base- line techniques (MERT+MLM, SPECTER, GIANT) in the few-shot and full-data node classification. This performance is consistent with the observations of using MLP and GraphSage.
For ogbl-citatio22, we also extend it for node classification. Here the task is to predict the subject areas of the subset of the nodes/papers that pub- lished in arxiv like ogbn-pan papers I 100M. We borrow the labels for oablooollcttteee Sgbllecttation2 from ogbn- paperss 100M. We align the nodes from ogbl-citation2 and ogbn- papers 100 through Microsoft academic graph pa- per ID. We split the data into vraining/validation/f by year that the papers published before 2017 the training dataset, between 2017-2018 is the valida- tion dataset and after 2018 is the test dataset.
Methods
à == k 4
8
= 16
All
B Implementation Details
BERT-NLM
42.47
19.29
56.78
60.52
66.37
SPECTER
59.66
63.26
69.
GIANT
40.57
44.81
54.27
58.93
65.82
Our proposed methodology was implemented using Py Torch version 1.13.1 (Paszke et al., 2019) and HuggingFace Transformers version 4.24.0 (Wolf al., 2019). The experiments were executed on A5000, A6000, and RTX 4090 GPUs. For GRENADE, the learning rate configured at 5e and AdamW optimizer is employed for training over the course epochs. The search space for the number of GNN layers, L, ranges from {1, 2, 4J. and further hyperparameter analysis is performed detailed in Fig. 6. The hyperparameters for few- shot node classification
70.85
Table 8: ogbn-arxiv node classification result on NAGphormer.
Additional Ablation Study. We have the ablation study on ogbn-arxiv and ogbn-products datasct un- der different model variations. In the second row of Tab. 9, the term ICL refers to instance- wiste cross- modality contrastive learning. It indicates that the positive pairs are formed between identically in- dexed document and node representations, while the negative pairs consist of other document and node representations within the minibatch. From Table 9, have consistent observation as 5 that each component of Grenade contributes the per- formance improvement.
shown in Tab. 6 and Tab. 7, respectively. It should be noticed that " 1" means utilize all the training data for batch size and all the neighbors for neighbor sampling.
Hyperparameters
Value
hidden size
256
In-Depth Hyperparameter Analysis. To evalu- ate the performance of the graph neural network encoder (GNN), we conduct an analysis of the hyper- parameter as depicted in Fig. 6. Our observations indicate that an optimal performance is achieved when L = 2.
dropout
0.5
Ir
le-4
epochs
300
butch size
-1
Table Hyperparameter setting for MLP model in node classification.
Inference Time Complexity Analysis
When pro- vided with an arbitrary node, Grenade is capable of generating the textual representation of the node without requiring graph information. This leads
C Additional Experimental Results
Full Data Node Classification.

---

Methods
MIL((AC))
K-manss<<ACC))
ogbu-arxav
MRR
Avg. Rank
MIP(ACC)
K-means+
ogbn-products
MRR
Rank.
GRENADE
73.16
61.96
52.73
1.67
81.58
73.54_1..0
72.39
2.00
GC-CL+ICL--hBB
72.20 20.00
-0.67
47.74
4.33
a0.00
72.48:1.42
65.72
3.67
GRENnne wio GC-
72.83
68.75.0.34
55.87
3.33
80.52_..5
71.251o.s
74.51
4.33
GRENADE
=2.63=u.20
60.69
70.37
GRINADE wio Nast KA
73.05
60.50
52.56
3.33
81.36 31.36g0-30
72.77g1144
72.08
3.33
GRENADE w/o XD- XA-KBII-XX
72.71=0.21
60.34.1.00
19.67
4.67
81. 11.17
73. 73.59
68.71
3.00
GEENAnE GC-
10:c0
68.00gn.77
80.35
71.10.
5.33
GRENADE wi GC- CL-4DDDDD XA
17.97 17.9720.18
22.49g0.11
0.56
8.00
66. 70go
44.83g0.00
21.08
8.00
Table Ablation study of graph-centric contrastive learning (CL) and knowledge-alignment between PLM encoder and GNN encoder (ND-KA and NBH-KA) ogbn-arxiv and ogbn-produnn datasets.
17.1
03.7
00.
51.6
72.5
(4) oghn-arxiv.
724
60.2
08.1
Rode flzsfooloo
Boce Clutoirgg
Lin Preecton
(b) ogbn-producte
Figure 6: Hyperparameter analysis on Dashed hori- zontal lines are the performance of chosen in this paper.
an efficiency that is on par with BERT (Devlin et al., 2018), while achieving approximately a 10% enhancement in performance for full data node classification.
Regarding the training efficiency, though our model requires longer training time compared to text-only PLM/SSL methods like BERT+MLM (Īh24m) and SimCSE (Gao al., 202I)(around 41m), are able achieve a great margin of per- formance improvement with reasonable additional training time (around 2h50m). Also, our model can achieve stable performance with 3-epoch training, which is more efficient than structure-basse con- trastive learning method such as SPECTER (Cohan et al., 2020) (around 3h19m for 3 epochs).


Reviews:
[{"review_id": "DdIiSsZOk4", "reviewer": "Reviewer_P3Mr", "summary": "", "questions": "The proposed approach is only adaptable to new tasks on the same graph. Given it demands sufficient compute, it is logical to analyze  how well the proposed framework is adaptable to new graphs of same family (e.g. citation) similar to domain adaptation of LLM. The authors are requested to comment on this aspect of the proposed framework.\n\nFor more questions, please refer from WEAKNESS section.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful comments. We have addressed the concerns below. If you have any further questions, please let us know.\n\n> **Q1**. Given that all the experimental datasets are homophilous, it would be interesting to explore the performance of GRENADE in non-homophilous graphs. How much the proposed SSL objectives stand up to in this scenario.\n\n**A1**. Thank you for bringing up the consideration of non-homophilous graphs. \nWe want to clarify that in this paper we mainly focus on the homophilous graphs, which are more common in the real-world. \nAs we mentioned in the paper, our proposed graph-centric contrastive learning and knowledge alignment algorithms also highly rely on the Homophily principle. Handling non-homophilous graphs is definitely an interesting future direction, but it is out of this paper’s scope. But our proposed work has the potential to adapt to non-homophilous graphs with careful design, such as revising the way of selecting positive samples and negative samples in our self-supervised learning objective functions. \n\n> **Q2**. There is no discussion on computational effort especially given that method is relying on PLM.\n\n**A2**. Regarding the training efficiency, though our model requires longer training time compared to text-only PLM/SSL methods like BERT+MLM (~ 1h24m) and SimCSE(~ 41m), we are able to achieve a great margin of performance improvement with reasonable additional training time (~ 2h50m). \nAlso, our model can achieve stable performance with 3-epoch training, which is more efficient than structure-based contrastive learning method such as SPECTER (~ 3h19m for 3 epochs). \n\nIn terms of the inference time, our method only needs to access the text document, thus it is computationally comaprable with text-only PLM/SSL methods like BERT (around 1h59m for ogbn-arxiv dataset) and more efficient than GNN-based methods. All the above reported time was evaluated on the ogbn-arxiv dataset with an Nvidia A6000 GPU. \n\n> **Q3**. There is no comparison with graph-only SSL approaches such as Auto-SSL [1]. Though it will not be an apple to add comparison since they lack Language encoders but will be interesting to examine the effectiveness of incorporating PLM.\n\n**A3**. Thank you for your insightful feedback. \nIt is worth noting that graph-only SSL techniques usually underperform TAG SSL methods (e.g., SPECTER[1] and GIANT[2]). \nThis is primarily because Graph-only SSL leverages graph-agnostic methods such as tf-idf, word2vec, BERT to extract numerical features from text and only relies on GNNs to learn the text node representations, which cannot capture the inherent correlations between text attributes and graph structure. \nTo comapre with graph-only SSL methods, we have incorporated two Graph-only SSL methods Auto-SSL[3] and VGAE[4] in the node classification evaluation (with full training data, the baseline results are adopted from the corresponding paper) on ogbn-arxiv. As observed from the results in the subsequent table, Text+Graph SSL outperforms Graph-only SSL in performance metrics and our model GRENADE achieves the best performance.\n\n\n|    SSL Types            | Method            | ogbn-arxiv |\n|----------------|-------------------|------------|\n| Graph-Only SSL | Word2Vec+Auto-SSL | 69.13      |\n| Graph-Only SSL | Word2Vec+VGAE     | 72.04      |\n| Graph-Only SSL | BERT+VGAE         | 72.44      |\n| TAG SSL        | GIANT             | 74.59      |\n| TAG SSL        | GRENADE           | **75.00**  |\n\n[1] Cohan, Arman, et al. \"SPECTER: Document-level Representation Learning using Citation-informed Transformers.\" ACL 2020.\n\n[2] Chien, Eli, et al. \"Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction.\" ICLR 2021.\n\n\n[3] Jin, Wei, et al. \"Automated Self-Supervised Learning for Graphs.\" ICLR 2021.\n\n\n[4] Kipf, Thomas N. et al. \"Variational Graph Auto-Encoders.\"  NIPS 2016.\n\n> **Q4**. In Table 2, it would be nice to include Graph Transformer Networks results in order to compare it with MLP and GraphSage.\n\n**A4**. Thank you for your suggestion. In addition to MLP and GraphSage, we have added a recent Graph Transfomer Network NAGphormer [1] as the classification model in the few-shot node classification evaluation. According to the results shown in the following table, our proposed method consistently outperforms the baseline methods (BERT+MLM, SPECTER, GIANT) on the few-shot node classification task, which is aligned with the observations of using MLP and GraphSage.\n\n**ogbn-arxiv node classification result on NAGphormer** \n\n|   Methods       | k=2       | k=4       | k=8       | k=16      | all       |\n|----------|-----------|-----------|-----------|-----------|-----------|\n| BERT+MLM | 42.47     | 49.29     | 56.78     | 60.52     | 66.37     |\n| SPECTER  | 49.89     | 54.79     | 59.66     | 63.26     | 69.11     |\n| GIANT    | 40.57     | 44.81     | 54.27     | 58.93     | 65.82     |\n| GRENADE  | **52.73** | **58.28** | **62.53** | **64.97** | **70.85** |\n\n[1] Chen, Jinsong, et al. \"NAGphormer: A tokenized graph transformer for node classification in large graphs.\" ICLR 2022.\n\n> **Q5**.  The proposed approach is only adaptable to new tasks on the same graph. Given it demands sufficient compute, it is logical to analyze how well the proposed framework is adaptable to new graphs of same family (e.g. citation) similar to domain adaptation of LLM. The authors are requested to comment on this aspect of the proposed framework.\n\n**A5**. Thank you for highlighting the importance of adaptability of our proposed approach to new graphs. Based on your suggestion, we conducted a new set of experiments to assess the adaptability of our model across graphs of same family as well as different families. Specifically, our model was trained on ogbn-arxiv and subsequently evaluated on ogbl-citation2 (same family) and ogbn-products (different families). As reported in the below table, our model achieves better adaptability compared to  the baseline methods in both cases. \n\n|          | arxiv->citation2 |                  |                  |  arxiv->products |                  |          |\n|----------|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:--------:|\n|          | k=16 ACC      | KMeans ACC       | MRR              | k=16 ACC      | KMeans ACC       | MRR      |\n| BERT+MLM |        30.9      |        25.78     |        9.04      |        44.75     | 45.74            | 27.22    |\n| SPECTER  |        33.01     |        36.71     |        23.11     |        47.52     |        58.85     | 45.39    |\n| GIANT    |        34.84     |        39.33     |        18.61     |        45.56     | 57.31            | 45.48    |\n| GRENADE  | **35.65** | **42.68** | **27.17** | **47.56** | **60.14** | **52.50** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "STRENGTHS:\n1. Although this work is not the first one to combine PLM and GNN for better representation learning, but it enables language model to infer global structural information resulting to a better Language encoder.\n2. Alignment is a much needed module in Multimodal space. The proposed objective of aligning Language and GNN encoder seems to be a very effective for SSL on graph specific tasks as validated by the ablation results.\n3. Overall the paper has a novel content and easy to follow.", "reasons_to_reject": "WEAKNESS:\n1. Given that all the experimental datasets are homophilous, it would be interesting to explore the performance of GRENADE in non-homophilous graphs. How much the proposed SSL objectives stand up to in this scenario.\n2. There is no discussion on computational effort especially given that method is relying on PLM. \n3. There is no comparison with graph-only SSL approaches such as Auto-SSL [1]. Though it will not be an apple to apple comparison since they lack Language encoders but will be interesting to examine the effectiveness of incorporating PLM.\n4. In Table 2, it would be nice to include Graph Transformer Networks results in order to compare it with MLP and GraphSage. \n\n[1] Jin, W., Liu, X., Zhao, X., Ma, Y., Shah, N., & Tang, J. (2021). Automated self-supervised learning for graphs. arXiv preprint arXiv:2106.05470.", "questions_to_authors": "The proposed approach is only adaptable to new tasks on the same graph. Given it demands sufficient compute, it is logical to analyze  how well the proposed framework is adaptable to new graphs of same family (e.g. citation) similar to domain adaptation of LLM. The authors are requested to comment on this aspect of the proposed framework.\n\nFor more questions, please refer from WEAKNESS section.", "paper_topic_and_main_contributions": "The paper is focused on self supervised learning (SSL) on Text augmented graphs by leveraging pre-trained language model (PLM) as well as Graph neural network (GNN).  The proposed method, GRENADE, proposed two new SSL algorithms, namely Graph centric contrastive learning and Graph centric knowledge alignment. Graph centric contrastive learning improves representation of both PLM and GNN encoding via neighborhood information. While Graph centric knowledge alignment enables alignment between the representation of PLM and GNN. The experiments on various graph-specific downstream tasks demonstrate the effectiveness of GRENADE over baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "No ethics review needed."}, "extra_text": {}}, {"review_id": "iseEC0zK1N", "reviewer": "Reviewer_esjL", "summary": "", "questions": "A. Why do the authors set the K-hop neighbor as 1? What about 2 or others?\nB. What about the effectiveness and generalizability of the proposed method?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for providing your feedback! Please find our detailed response addressing your concerns as follows:\n\n> **Q1**. This paper emphasizes that the proposed method can improve the representation expressiveness. However, it does not prove and analyze the expressiveness.\n\n**A1**. Thank you for raising the concern regarding the expressiveness of our proposed method. In our paper, we claim that by eliciting and aligning the knowledge between pretrained language model and graph neural network, our model can learn expressive node representations for each text document. To validate this point, we conducted multiple analyses in our experiments:\n- Quantitative Analysis:\n    1. Few shot node classification: In Table 2, our method outperformed GIANT, SPECTER et al baseline methods by 5% showing and proving the utility of our representations for tasks.\n    2. Node clustering: In Table 3, our representations led to more coherent clusters, evidenced by an improvement of at least 2% in the K-Means++.\n    3. Link prediction: In Figure 2 our method showed a 4% enhancement over the best baseline method SPECTER in ogbn-arxiv, demonstrating the expressiveness in relational patterns.\n- Qualitative Analysis:\n    4. Representation visualization: In Figure 3, we visualized the embeddings using t-SNE, and observed clear distinctions and patterns which further validated the rich expressiveness of our representations.\n\nThe consistent superior performance across varied tasks strongly indicates the enhanced expressiveness of the representations generated by our method. We will ensure that the paper emphasizes and details these results more clearly in the revised version by improving the readability.\n\n\n>  **Q2**. Why do the authors set the K-hop neighbor as 1? What about 2 or others? B. What about the effectiveness and generalizability of the proposed method?\n\n**A2**. Thank you for your question. Our choice of setting **K=1** is mainly based on the consideration of simplicity and effectiveness. \nIn practice, 1-hop neighbors are good enough for capturing the structure knowledge in our algorithm. \nWhen we consider 2-hop or 3-hop neighbors, there's an increased potential for introducing noisy information. \nThis observation is also consistent with the findings from the related work, SPECTER [1]. We believe our approach balances the trade-off between capturing structural information and mitigating noisy information, leading to stable performance in our evaluation. To further justify our choice, we conducted a hyperparameter analysis by varying the value of $K$ on the ogbn-arxiv datasets. The results, as illustrated in the following table, clearly show that increasing the hop distance degrades performance metrics across node classification (MLP ACC), node clustering (KMeans ACC), and link prediction (MRR).\n\n|     | MLP ACC   | Kmeans ACC | MRR       |\n|-----|-----------|------------|-----------|\n| K=3 | 71.97     | 61.06      | 47.48     |\n| K=2 | 72.21     | 60.60       | 49.59     |\n| K=1 | **73.16** | **61.96**  | **52.73** |\n\n\n[1] Cohan, Arman, et al. \"SPECTER: Document-level Representation Learning using Citation-informed Transformers.\" ACL 2020.\n\n> **Q3**. What about the effectiveness and generalizability of the proposed method?\n\n**A3**. In this paper, we propose a new method for self-supervised representation learning on text attributed graphs. \nBased on our proposed graph-centric contrastive learning and graph-centric knowledge alignment algorithms, we are able to learn a language model that can derive expressive representations for each text node. \nTo show the effectiveness and generalizability of our approach, we evaluate the quality of the learned text node representations on different down-stream task, including few-shot node classification, node clustering, and link prediction (more details can be found in our **A1**). \nThe superior performance across different downstream tasks clearly show the effectiveness and generalizability of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clear motivation.\n2. Extensive experiments show the Grenade is able to learn efficient representations for node classification and link prediction tasks.\n\n\n", "reasons_to_reject": "1. Poor presentation of the method section.\n2. This paper emphasizes that the proposed method can improve the representation expressiveness. However, it does not prove and analyze the expressiveness.\n\n", "questions_to_authors": "A. Why do the authors set the K-hop neighbor as 1? What about 2 or others?\nB. What about the effectiveness and generalizability of the proposed method?", "paper_topic_and_main_contributions": "The proposed Grenade is a self-supervised graph centric language mode for learning expressive and generalized representation from textual attributed graphs (TAG). The proposed Grenade capture informative textual semantic and graph structure information to addresses the underexplored problem of self-supervised learning on TAG.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RAtjc7k5F3", "reviewer": "Reviewer_6SrM", "summary": "", "questions": "None.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback about our work! \n\nRegarding your concern about reproducibility, we would like to mention that all the detailed hyper-parameter settings and their respective analyses are included in the appendix of our paper. This should provide clarity on the configurations used in our experiments. We have also uploaded our source code along with the training scripts to the supplementary materials. These resources are intended to offer other researchers an in-depth view of our implementation, further aiding in reproduction. After the acceptance of our paper, we plan to release the model checkpoints additionally in public.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The writing is clear and easy to follow.\n2. Two novel self-supervised learning algorithms are proposed to perform contrastive learning and knowledge alignment in a graph-centric way.\n3. Extensive experiments show that Grenade achieves significant and consistent improvements over state-of-the-art methods on a wide spectrum of downstream tasks.", "reasons_to_reject": "I do not see significant weaknesses or omissions in this paper.", "questions_to_authors": "None.", "paper_topic_and_main_contributions": "This paper focuses on self-supervised representation learning on text-attributed graphs. The authors argue that existing methods either struggle to capture the full extent of structural context information or rely on task-specific training labels. To solve such problem, this paper proposes a Graph-Centric Language model named Grenade. Grenade optimizes graph-centric contrastive learning and graph-centric knowledge alignment to exploit the synergistic effect of both pre-trained language model and graph neural network. The experimental results demonstrate the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces GRENADE, a self-supervised learning (SSL) approach for text-augmented graphs that leverages both pre-trained language models (PLMs) and graph neural networks (GNNs). GRENADE proposes two novel SSL algorithms, Graph-centric Contrastive Learning and Graph-centric Knowledge Alignment. The former improves the representation of PLM and GNN encoding by incorporating neighborhood information, while the latter aligns the representations of PLM and GNN. The experiments conducted on various graph-specific downstream tasks demonstrate the effectiveness of GRENADE over baselines.

The strengths of the paper lie in its novel content and clear presentation. GRENADE introduces innovative SSL objectives that effectively leverage both PLM and GNN representations. The alignment module addresses a crucial need in multimodal learning, and the ablation results validate its effectiveness. The paper is easy to follow, making the proposed method accessible to the reader. Additionally, the extensive experiments provide substantial evidence of GRENADE's performance superiority over existing methods.

While the paper presents several strengths, there are some areas that could be improved. It would be valuable to explore GRENADE's performance on non-homophilous graphs to assess the robustness of the proposed SSL objectives in diverse scenarios. The computational effort required by GRENADE is not discussed, which is an important consideration, particularly when relying on PLMs. Comparisons with graph-only SSL approaches like Auto-SSL could provide insights into the effectiveness of incorporating PLMs. Additionally, including results from Graph Transformer Networks in Table 2 would enable a comparison with other graph-based SSL methods like MLP and GraphSage.
Decision: Accept-Findings