WalkLM: A Uniform Language Model Fine-tuning Framework for Attributed Graph Embedding
Authors: Yanchao Tan
Zihao Zhou
Hang Lv
Weiming Liu
Carl Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: Arributed graph
unsupervised graph learning
language models
representation learning
TL;DR: Our framework, built on a natural integration of language models (LMs) and random walks (RWs), is straightforward, powerful and data-efficient.

Abstract: Graphs are widely used to model interconnected entities and improve downstream predictions in various real-world applications. However, real-world graphs nowadays are often associated with complex attributes on multiple types of nodes and even links that are hard to model uniformly, while the widely used graph neural networks (GNNs) often require sufficient training toward specific downstream predictions to achieve strong performance. In this work, we take a fundamentally different approach than GNNs, to simultaneously achieve deep joint modeling of complex attributes and flexible structures of real-world graphs and obtain unsupervised generic graph representations that are not limited to specific downstream predictions. Our framework, built on a natural integration of language models (LMs) and random walks (RWs), is straightforward, powerful and data-efficient. Specifically, we first perform attributed RWs on the graph and design an automated program to compose roughly meaningful textual sequences directly from the attributed RWs; then we fine-tune an LM using the RW-based textual sequences and extract embedding vectors from the LM, which encapsulates both attribute semantics and graph structures. In our experiments, we evaluate the learned node embeddings towards different downstream prediction tasks on multiple real-world attributed graph datasets and observe significant improvements over a comprehensive set of state-of-the-art unsupervised node embedding methods. We believe this work opens a door for more sophisticated technical designs and empirical evaluations toward the leverage of LMs for the modeling of real-world graphs.

WalkLM: A Uniform Language Model Fine-tuning Framework for Attributed Graph Embedding
Yanchao Tan
Zihao Zhou
Hang Lv College of Computer and
College of Computer and
College of Computer and
Data Science
Data Science
Data Science
Fuzhou University
Fuzhou University
Fuzhou University
Fuzhou, China
Fuzhou, China
Fuzhou, China
yctan&fzu. edu. cn
reviverkey@gmail.. com
1vhangkenn@gmail.cm com
Weiming Liu
Carl Yang"
College of Computer Science Zhejiang University Hangzhou, China 218310100z2 218310100zju.ee edu.
Department of Computer Science
Emory University
Atlanta, United States
j carlyynggemory.cee edu
Abstract
Graphs are widely used to model interconnected entities and improve downstream predictions various real-world applications. However, real-world graphs nowa- days are often associated with complex attributes multiple types of nodes and even links that are hard to model uniformly, while the widely used graph neural networks (GNNs) often require sufficient training toward specific downstream predictions to achieve strong performance. this work, we take a fundamentally different approach than GNNs, to simultaneously achieve deep joint modeling of complex attributes and flexible structures of real-world graphs and obtain unsu- pervised generic graph representations that not limited specific downstream predictions. Our framework, built on natural integration of language models (LMs) and random walks (RWs), is straightforward, powerful and data-efficient. Specifically, we first perform attributed RWs on the graph and design an automated program compose roughly meaningful textual sequences directly from the tributed RWs; then we fine-tune an LM using the RW-based textual sequences and extract embedding vectors from the LM, which encapsulates both attribute semantics and graph structures, In our experiments, we evaluate the learned node embeddings towards different downstream prediction tasks on multiple real-world attributed graph datasets and observe significant improvements over comprehen- sive set of satte-of-the-ase unsupervised node embedding methods. We believe this work opens a door for more sophisticated technical designs and empirical evaluations toward the leverage of LMs for the modeling roal-world graphs.
1 Introduction
Graphs are widely used to model interconnected entities, and they are critical in enhancing down- stream predictions in various real-world applications. Nowadays, real-world graphs are often associ- ated with complex attributes on multiple types nodes and even links [27] [8), and modeling such real-world graphs is non-trivial. For example, in the schema of clinical attributed graph constructed
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

has
inchuding
A 35-year-ole olddrr-oddddoo female PATIENT P246 has a VISIT V196 on May 3rd including a DISEASE E Epistadis given a VISIT V175 on May 13th for y6-year-old male PATIENT P359.
PATIENT
VISIT
DISEASE
for has
Pid
Vid Time
Age
Name
given
Sex
inchuding
given
(a) Schema of real-world attributed graph
(b) A Atributtdd random walk
(c) Composed text
Figure A toy example of the transtormation from a real-world attributed graph to the composed text. (a) is schema of real-world attributed graph on MIMIC-III that delineates how nodes (e.g, patients), edges (e.g, has between patients and visits), and the associated attributes (e.g., age) are organized and interconnected. (b) is attributed random walk for capturing structural information and can be composed to text in (c).
from MIMIC-II data (shown in Figure a)), there are multiple types of nodes such as patients, visits and diseases, each with their attributes such age, sex, time and name; there are also multiple types of links associated with different meanings. Such complex heterogeneous attributes of nodes and links attributes can hardly be modeled in a uniform space.
Although the widely used graph neural networks (GNNs) have shown remarkable successes in the modeling of attributed graphs for various downstream applications [5 53 53], the representation learning (a.k.a. embedding) of GNNs often requires sufficient training toward specific downstream predictions to achieve strong performance 76). While unsupervised training has also been explored for GNNs 23 65], the generic performances of unsupervised GNN embeddings across different downstream tasks are still unsatisfactory (as we will also show in experiments). Consequently, it important to devise a general purppos graph embedding method to simultaneously understand the complex node/link attributes and incorporate the flexible graph structures in an unsupervised manner.
However, two obstacles stand in the way of achieving this goal. First, the nature of the attributes can be intricate and diverse, thus understanding their semantics in uniform space is non-trivial. Second, the graph structures need to be accurately captured and incorporated into the embedding space, which is not straightforward due to the inherent flexibility and potential complexity of entity relations.
To address these issues, in this work, we take fundamentally different approach than GNNs, named WalkLM, which is a uniform framework to obtain unsupervised generic graph representations that are not limited specific downstream tasks. To this end, we first draw inspiration from the recent successes of language models (LMs), and propose to leverage LMs as uniform attribute models that can capture the intricate semantics of complex heterogeneous attributes of nodes and links. Secondly, we propose to leverage the classic tool of random walks (RWs) on graphs which have been shown effective capturing flexible graph topological structures by various studies [ 10
21 38
Specifically, we first generate attributed RWs on the graph (c.g.. has
5 in Figure (b).
and design an automated textualization program to compose roughly meaningful textual sequences directly from the attributed RWs. As shown in Figure [ (c). c). the composed text a mapping from the attributed RW in Figure [b), where a uniform automatic program firstly textualize different types of nodes (in different colors) by properly concatenating the nodes with the names and values of different attributes, and then textualize the whole RWs by concatenating the nodes and links. Furthermore, we fine-tune an LM using the RW-based textual sequences and extract embedding vectors from the L.M. The learned embeddings encapsulate both attribute semantics and graph structures, and can be flexibly utilized for arbitrary downstream tasks.
In our experiments, we take the node embeddings and evaluate them towards different downstream prediction tasks (e.g., node classification and link prediction) on multiple real-world attributed graph datasets and observe significant improvements over comprehensive set of state-of- of-the-art unsupervised node embedding methods (e.g.. WalkLM achieves an average 88.98% improvement over the state-of-theear baselines ranging from existing RW-based graph embedding methods to popular unsupervised GNN modes regarding both Macro-F1 and Micro-F1 metrics). We believe this
2

---

work paves the way for more sophisticated technical designs and empirical evaluations toward the leverage LMs for the modeling of real-world graphs.
Related Work
Graph Representation Learning, In recent years, a plethora of representation learning techniques proposed for graphs [3] [5
55 58 74]. In this work, we focus on the objective of learning embedding vectors for each node that characterizes its topological (and semantic) information the
graph. Among existing node embedding methods, many have analyzed and utilized the great promise of random walks (RWs) in capturing the topological structures of graphs [8] |1 38). However, the
above methods ignore the abundant attribute information surrounding the nodes and edges [32] In recent studies, Graph neural networks (GNNs) for learning node representations through aggregating information from neighboring nodes on graphs [14
However, most existing GNNs established in a supervised learning setting, which requires abundant task-specific labeled data that may not be available in real-world applications 4 76]. and the embeddings they learn are not generalizable across different downstream tasks 77 Although some studies tried to reduce the labeling effort by pre-training an expressive GNN model on unlabeled data with self-supervision methods (e.g., contrastive learning) [19 22 75 their performances in specific downstream tasks still rely much on the properly chosen superviinvv tasks and attribute encoders 47 73 that is, there
24 62
still lack a uniform framework for generic unsupervised attributed graph representation learning.
Language Models (LMs), With the massive corpora and powerful computation resources for pre- training, modern language models (LMs) derive various families
34 These LMs can be grouped
into: (1) auto-regressive LMs (e.g., GPT [40] and GPT--/33 1 (2) masked LMs (e.g.. BERT (7].
RoBERTa 31, and XLNet (64), and (3) encoder-decoder LMs (e.g.. BART 26 and T5
42 LMs
have been intensively studied by NLP researchers for various language-related tasks [6
43 In
our work, we innovatively utilize LMs as uniform attribute models for nodes and links in graphs for the first time. Note that, our work also readily generalizes to recent large language models (LL.Ms) (e.g..
InstructGPT 33 ChatGPT and GPT-4 (35) via appropriate parameter-efficien training approaches (e.g., LoRA 18 and prompt-tuning [25 30). However, those are orthogonal to the innovations in this work, for which we leave the exploration as immediate future works.
LMs with Knowledge Graph (KG). recent studies, combining LMs with KG has been widely applied in various real-world applications 37 Among existing methods, many have proposed to enhance LMs with KG for significantly improve the performance of L.Ms accessing domain-specific knowledge [46] 56 67 69), and the others proposed to harness the power of LMs for addressing KG-related tasks [65] 66). However, the above methods fail to effectively combine the rich semantic information of graphs with global topological information. Furthermore, as closest to us, [12] proposed to combine random walks and neural network language models to produce new word representations. However, it ignores the rich relational information between nodes and thus fails to learn richer topological information. Moreover, compared with modern LMs with extensive prior knowledge, the text-based encoder used in 22 5T fails to extract richer semantic information.
3 Preliminaries
3.1 Problem Formulation
Given an attributed graph G and multiple downstream tasks т, (c.g., node classification Tı and link prediction 72), the goal of WalkLM is to sufficiently model information in G and improve task performances on 7,. Specificallyy, we denote the graph as G = (V, E, Φ, V), where each node o; e V associated with attributes ((e)). and each edge e; E is associated with attributes (e:)).
To fully exploit both the semantic information & and V, and the structural information in V and E, we first design attributed random walks (RWs) based automated textualization program on G. where we can transform the attributed graph to the meaningful textual sequences w/ = (W.) 1 Then, we -WW(,))
fine-tune a graph-aware language model (LM) using the RW-based textual sequences W, find the optimized parameters of the L.,, and extract embedding vectors from the LM. Finally, we apply these embeddings Ti and 72 for performance evaluation.
3

---

Generall LM
Node classification
walk
Composed texts
Generic graph
Link prediction
embeddings
(a) Attributed RW-based textualization program
(b) Graph-aware LM fine-tuning
(c) Downstream tasks
Figure 2:
The overall framework of WalkLM: (a) An attributed random walk (RW) based tex-
tualization program obtains attributed RWs and composes roughly meaningful textual sequences directly from the attributed RWs. (b) A pre-trained general LM is fine-tuned in graph-aware fashion using the generated textual sequences to produce generic graph embeddings. (c) The learned graph embeddings can be applied to various downstream tasks.
3.2 Masked I Language Modeling
Masked Language Modeling (MLM) a widely-used and highly effective technique for training large-scale language models [7] In the MLM task, 15% of the language tokens are randomly chosen for prediction. If the i-th token in sequence
is chosen, it can replaced by (1) the [MASK] token 80% of the time, (2) random token 10% of the time, and (3) the unchanged i-th token 10% of the time, Then, token t will be used to predict the original token with following cross-entropy
loss 7 59 60
MMLM
Σ Σ E logp (L/TT)) Xex NeM
(1)
X
where X is set of training examples, M is the prediction set of the masked token, Ti (4,...., 4--1 li, (r.) is the surrounding token set of 4, and [T,al L-
4 Methodology
In this section, we present the proposed method WalkLM, which comprises two major components. The first component, the attributed random walk (RW) based textualization program, captures both topological structures and attribute information of the graph and composes corresponding text automatically. The second component, the graph-aware language model (L.M) fine-tuning, leverages pre-trained LM to encode the complex semantics along with the graph structure. The overall model architecture shown in Figure
4.1 Attributed RW-based Textualization Program
To model node/link attributes, traditional machine learning algorithms require standard process of vectorization, which transforms different types attributes into categorical or numerical features as model input. However, such vectorization process removes the actual semantic meanings of the attributes, and it cannot unify different types of attributes (e.g., ages, sexes, time, etc.) in the same space. Inspired by the recent successes of LMs, find it promising to leverage pre-trained LMs to understand the intricate semantics of complex heterogeneous attributes [29 34]. The key idea is to perform textualization instead of vectorization, that to transform different types of attributes into texts, which can then be modeled by the LMs in a uniform space. Therefore, we first design the following process for the textualization of individual entities in the graph (i.e., nodes and edges).
Entity-level Textualization. Inspired by a wide range of NLP tasks that leverage prompts to construct informative rule templates, we propose textualize attributed graph entities via rule-based program function P.)) that automatically concatenates attribute values with the corresponding attribute names, well attributes and the corresponding entity id. For example, shown in Figure for patient node U; with attributes (v;)) = {age: 35, ser : female, pid P246), the texualization program will convert it into P(v)) =<< A 35-ycar-old female patient P246 >. For edges, in this work, only consider simple relational edges such as has and including. which are already texts, but our framework is readily extensible to edges with more complex attributes.
For attributed graphs, after modeling the complex attributes of individual entities, the next challenge would be model the flexible graph topological structures. To this end, we propose to utilize the

---

powerful and efficient tool of RWs, as the foundation for the textualization of graphs. Specifically, we design the following process:
Walk-level Textualization. We first initiate an attributed RW W by randomly selecting a node vo and attaching the textualized node information P(v)) to W. Then, we extend W by ran- domly selecting an edge starting from vo as in a standard RW with a uniform probability as 1 divided by the number of out-edges of vo. with its terminating node vi, and appending the corresponding texts P(ei) and P(v)) to W. We can keep adding edges and nodes until the ran- dom walk terminates, such as based on a termination probability a. The final textual sequence W (P((e) P(ez), ). P(v))...... P....., P(vz-1),
P P(eL)) which corresponds to an actual at- tributed random walk with the length of 2L 1 on the graph, will be roughly meaningful sentence, such as the one shown in Figure (
After performing the RW for N times, we can obtain N attributed RWs as W- (W.) which constitutes a graph-aware corpus for training LMs without any downstream task supervision.
Discussion. We believe that the above proposed attributed RW-based textualization program is effective in capturing both complex attribute information and flexible topological structures of the graph in an unsupervised manner. Such ability arises from two critical properties: (1) Random walks are known to be capable of providing characteristic graph traits and reconstructing network proximity of nodes [ [33] To be specific, it has been proven that the distribution of random walks starting at a particular node, which can be estimated with sufficient numbers of random walks, can sufficiently preserve the subgraph structure around the node. This means a sufficient amount of attributed RWs from different nodes can well reflect the topological structures of graphs. (2) Our textualization program completely preserves the attributes of nodes and edges, as well as the whole RWs, and it presents such information as meaningful texts, which is natural for LMs to comprehend. Moreover, RWs are known to be efficient and highly parallelizable, where numerous threads can run simultancously to generate large amounts of RWs [9]. Note that, we only need to perform the rule-based textualization once for every node during the pre-processing stage, which is also efficient and highly amenable to parallelization.
4.2
Graph-Aware LM Fine- TunnTTiing
Despite the robust generalizability of LMs, fine-tuning remains necessary step [57 68 which allows the general LM to adapt its broad language capabilities to the specificities of the different attributed graphs.
As one of the mainstream language modeling techniques, masked language modeling (MLM) is
proven to sufficiently utilize textual semantic information for further fine-tuning LMs [39] 77).
To achieve the balance between effectiveness and efficiency, we propose a graph-aware LM fine-
tuning mechanism with knowledge distillation (2) 17 44). Specifically, we adopt a general LM
BiRtil (abbr. DROBERTA as our starting point for fine-tuning, where RoBERTa is a widely used successor of BERT 77. Note that, DRoBERTa can further reduce the size of the original RoBERTa model by 40% and achieve 60% faster training while retaining 97% capacity of RoBERTa's language understanding [4)) Then, we feed the attributed RW W W to the LM tokenizer and obtain the corresponding token list T = (41, /2,,.... ...,,,,,, < MASK > < MASK >2..... < MASK > > where 4; denotes the unmasked token, and MASK >; denotes the token chosen for prediction. In this way, we create training example X, (W, T) e X for fine-tuning, where X = X21.X2..... XN is
the whole training dataset. We adopt the entropy loss as the fine-tuning objective 7 44
which is formulated as follows:
LFT(
Σ X.ex
Σ M.M
log
(2)
Eny (Spm(m,))
where θ is the learnable parameters of our graph-aware LM, M the ground-truth set of the masked token, V is the token vocabulary, tx is the prediction token, ti is the ground-truth token, and Sim((.,4) 4)) is the similarity scoring function between I and ly.
After obtaining the fine-tuned LM based on MI.M, we can extract generic graph embeddings (e.g.. node embeddings based on node name). For example, we can access the representation of disease embedding in Figure () via extracting the embedding of Epistaxis.
5

---

Complexity Analysis. The time complexity of fine-tuning is O - l d), where Iter the number of iterations of training. is the number training examples, lang is the average length of input textual sequences for the LM and d is the dimension of embedding. We infuse global graph structure knowledge into the LM to distinguish similar positions instead of negative sampling. making it possible to fine-tune more efficiently 59
Model Extension. Our framework fundamental approach integrate LMs and RWs for generic attributed graph embedding, which can choose different LMs according to different tasks and domains (shown in our experiments Sec. [5.4) and generalize to recent large language models (LLMs) (e.g.. InstructGPT [36] ChatGPT and GPT-4 (35)) via appropriate parameter- training approaches such as LoRA [18]. full exploration of different LMs is orthogonal the main contributions in this work, which is left as future work.
Datasets Extension. Our method introduces the novel process of textualization, which converts gen- eral attributed graphs into text-like sequence data. This process allows us to leverage the capabilities of pre-trained language models for graph representation learning. Note that, our method only requires some meaningful attributes on the graphs, which are available in most real-world graphs such as biological networks, social networks, and knowledge graphs. Some preliminary experimental results of graph classification and KG-related datasets are shown in Appendix A.1 and Appendix A.2
4.3 Various Downstream Tasks
In this work, we focus on node embeddings since they are most commonly studied for graph representation learning, and it is straightforward to extract node embeddings from the fine-tuned LM based on node names (or node IDs the node has no meaningful name). However, WalkLM can also easily generate edge embeddings, by adding edge names (e.g.. relation names) or edge IDs to the textualization process, and even obtain subgraph/graph embeddings via appropriate embedding aggregation mechanisms. The extracted embeddings can be directly used fixed feature vectors to train downstream prediction models for tasks such as node classification or link prediction. Alternatively, these embeddings also serve initialization for more learnable embeddings in complex neural network models, which can be further updated according to the specific requirements of the downstream task.
Experiments
5.1 Experimental Setup
Datasets. We conduct extensive experiments on two real-world datasets, PubMed and MIMIC-II PubMed contains graph of genes, diseases, chemicals, and species. The nodes and edges are extracted according to [61] A relatively small fraction of diseases are grouped into eight categories. MIMIC-III contains graph of diseases, patients, and visits, where nodes and relations are extracted from clinical records. Diseases are classified into nineteen categories according to ICD-9-CM The detailed statistics are shown in Table 0
and
Table 1: Statistics of the datasets.
Dataset
#attribute type
#node type
#node
#link type
# link
#label
#lahel node
PubMed
8
4
63.109
10
244,986
8
454
MIMIC-II
10
32.267
559,290
19
4880
Competitors. We compare our proposed WalkLM with ten graph-oriented baselines that are designed for heterogeneous information networks (HINs) knowledge graphs (KG), which can handle different types of nodes and edges. We divided them into four groups as follows:
(1) RW- based methods: Meeapath2Vec (abbr. M2V) (8) proposes to use user-defined meta- meta-paths as guidance, so as learn node embeddings on HINs. HIN2Vec [00 carries out multiple prediction
'ntps://pubmed..cc www.i.i.iiiiii........o
hhtrp://physiionet.rr///// http:///ww....o..oo///////mmmmmm htnp:://www..dd..evv/eesidiiiioo

---

Table 2: Different downstream task results (%) with the corresponding std (=) on two datasets. The best performances are in bold and the second rumners are shaded in gray, where denotes a significant improvement according to the Wilcoxon signed rank significance test.
Task
Node Classification
Link Prediction
Dataset
PubMed
MIMIC-III
PubMed
MIMIC-III
Metric
Macro-F1
Micro-F1
Macro-F1
Micro-F1
AUC
MRR
AUC
MRR
M2V
15.35
20.27
19.69
29.24
74.53
89.58
75.05
88.32
(=1.27)
(+3.01)
(+0.62)
(+1.57)
(+3.79)
(+2.05)
(+0,41)
(+0.23)
HIN2Voe
11.57
18.92
19.12
28.05
74.21
90.56
73.46
88.10
(+1.23)
(+2.78)
(+1.32)
(+1.44)
(15.49)
(+1.06)
(+0.41)
(+0.14)
ConvE
16.06
19.16
24.44
32.89
76.48
92.27
69.56
84.88
(+3.69)
(+4.00)
(+1.28)
(10.86)
(14.31)
(+0.57)
(+0.36)
(+0.25)
ComplEx
13.93
18.27
9.82
21.39
79.81
91.79
63.86
81.40
(+2.59)
(+4.12)
(+0.56)
(#3.12)
(+0.97)
(=0.48)
(+0.42)
(=0.40)
SimKGC
21.97
30.83
51.62
58.50
79.62
91.43
67.73
84,86
(#3.51)
(#3.10)
(#1811)
(#1.52)
(+2.72)
(=0.48)
(+1.69)
(=0.54)
RGCN
12.50
18.50
7.19
14.55
72.08
88.20
57.31
73.91
(+2.36)
(+1.41)
(+0.77)
(+3.25)
(+1.13)
(+0.47)
(+0.71)
(+0.57)
HAN
15.29
16.95
6.98
14.73
70.57
87.89
(+2.87)
(+2.71)
(+0.58)
(+166)
(+1.58)
(+0.62)
HGT
11.98
20.12
8.03
17.79
77.24
89.63
64.01
81.54
(+2.23)
(43.89)
(10.87)
(10.83)
(13.50)
(40.84)
(10.36)
(10.56)
HcCo
10.32
18.01
10.78
15.26
65.04
83.29
53.13
71.81
(#1.12)
(+0.87)
(=0.41)
(+1.52)
(+1.26)
(=0.72)
(+0.47)
(=0.35)
SHGP
10.80
19.28
11.34
17.44
68.22
85,34
54.49
72.58
(+3.03)
(+0.91)
(+1.29)
(+1.49)
(+2.71)
(+0.48)
(40.33)
(+0.24)
LM
40.10
44.71
54.51
61.27
60.20
84.23
51.21
74,22
(XRoBERTa)
(#4.62)
(+3.68)
(=1.50)
(+1.22)
(+2.78)
(=1.71)
(+0.17)
(+0.26)
LM
59.43
61.53
70.26
72.67
$1.71
80.54
50.66
72.36
(GPT-2)
(+4.73)
(+3.43)
(+1.43)
(+0.90)
(+3.67)
(+2.49)
(+0.74)
(+0.86)
58.29
60.57
66.25
70.14
60.97
83.00
75.09
(DRoBERTa)
(+2.44)
(+2.11)
(+1.60)
(+1.52)
(+2,98)
(+0.40)
(+0.14)
(+0.29)
LM
13.83
22.70
14.32
24.59
72.35
88.86
58.62
78.78
+RGCN
(+0.73)
(13.25)
(+0.87)
(+1.17)
(14.34)
(+1.46
(40.50)
(20.10)
LM
12.81
21.79
10.49
20.57
82.97
89.98
65.01
82,28
+HGT
(+1.22)
(+3,54)
(+0.41)
(+0.97)
(+3.91)
(+0.88)
(+0.20)
(+0.30)
WalkIM
60.42*
62.33
75.16*
77.89*
85.65*
94.16*
82.15
92.78*
(+2.62)
(+3.13)
(10.93)
(10.70)
(+3.28)
(+0.37)
(10.67)
(+0.68)
training tasks jointly based target set of relations to learn node embeddings and meta-paths HINs.
(2) Relation learning- based methods: ConvE 6 proposes to use 2D convolution over embeddings
and multiple layers non-linear features to model KGs. ComplEx [49] handles large number of binary relations using complex-valued embeddings on KGs. SimKGC (52) proposes to elicit the implicitly stored knowledge from BERT and designs text-based contrastive learning mechanism for knowledge graph completion.
(3) Supervised heterogeneous graph neural networks (HGNNs): RGCN 451
proposes to apply GCN to model HINs or KGs. HAN [53] proposes to learn the importance between a node and its meta-path based neighbors on HINs. HGT 20 proposes to use each edge's 's type to parameterize the transformer-base self-attention architecture on HINs. For the above supervised HGNNs, we use link prediction loss introduced GraphSAGE [4 achieve unsupervised learning (i.e., without any
node labels), following existing studies on HINs [1] 61.
(4) Unsupervised HGNNs: HeCo [54 proposes co-contrastive learning mechanism for HGNNs.
SHGP 63 designs self-supervised pre-training method for HGNNs.
Settings. We mainly compare ten algorithms under the setting of unsupervised graph learning. The full code for this work is available All the models are optimized through the Adam optimizer and

---

Table 3: Node classification results (%) in the few-shot setting with Macro-FI (abbr. Ma-FI) and Micro-FI (abbr. Mi-FI) metrics.
Dataset
PuhMed
MIMIC-II
Setting
shot
3 shot
5 shot
shot
shot
5 shot
Metric
Mo-FI
Mi-FI
Ma-FI
Mi-FI
Ma-FI
Mi-FI
Ma-FI
Mi-FI
Ma-Fi
Mi-FI
Ma-FI
Mi-FI
Compliix
13.26
15.94
2,00
9.26
M2V
9.86
13:42
10:27
12.56
12.97
14.98
5.83
8.72
3.91
5.11
3.40
4.49
ConvE
13:25
13:45
8.84
10.93
11:25
13.53
5.85
6.75
5,61
7:24
6.31
7.69
RGCN
9.34
11,02
8.57
10,58
10.84
13.43
4.97
5.82
5.43
6.28
5.22
5.73
IIN22Ve
8.46
10.54
9.04
12.79
10.96
17.39
5.72
10.35
4.90
5.72
3.57
4.91
SHGP
11.73
4.12
5.36
NalkLM
28.09'
30.94
32.11
35.35
35.41
37.68"
23.33*
27.96*
34.19
40.49*
46.83
the learning rate is searched in [1e-4, 1e-2]. The hyper-parameters of baselines are chosen carefully based on either grid search or their official source codes. For all the methods, we use five-fold cross-validation for a more reliable evaluation of the model's performance. All the experiments are performed with two NVIDIA GTX 3090 Ti GPUs. For link prediction, we deploy the Large Margin Nearest Neighbor (LMNN) technique based on the embeddings generated by our WalkLM, Then we construct feature vectors for edges.
5.2 Node Classification
Main Results. For node classification, train separate one-layer MLP classifier based the learned embeddings on 80% of the labeled nodes and predict the remaining 20%. All the methods are trained in an unsupervised manner without classification labels. We evaluate WalkLM with Macro-FI (across all labels) and Micro-F1 (across all nodes).
As shown in Table our proposed WalkLK has superior performance, indicating the importance of leveraging both semantic and structural information in attributed graphs. WalkLM achieves 138.59% performance gains PubMed over the second- best performance average while achieving 39.37% average performance gains on MIMIC-III Specifically, SimKGC achieves second-best performance by effectively employing text-based contrastive learning, which leverages BERT to capture rich set of semantic information. Compared with SimKGC, WalkLM can effectively combine the complex semantic and graph structure information of attributed graphs, as accurately model the complex attributes of nodes. Although the HGNNs can naturally model attributes, their unsupervised training mechanisms likely do align well with the downstream prediction task of node classification.
Results in the Few-shot Setting. Since one key challenge of node classification lies in the generaliz- ability and adaptability of models [63]. we design few-shot setting to evaluate models in extending knowledge to unseen scenarios and adapting to new tasks with limited training data.
As shown in Table our framework can stay strong with small size of training data, where we win 275.45% performance gain over the second-best performance on average. Through the novel textualization process that converts general attributed graphs into text-like sequence data, our proposed WalkLM can leverage the capabilities of modern language models for graph representation learning. With the extensive pre-training of LMs on broad text corpora, WalkLM can easily understand meaningful node attributes given a new graph, while the random walk strategy further allows in the
to capture graph structures. Consequently, WalkLM maintains superior performance few-shot setting. Similar to the main results in Table 2 the ranking of baselines is fluctuating across datasets, where ConvE and M2V continue to exhibit promising performance. Note that ComplEx and HIN2Vec exhibit notable improvements in this setting, likely because HIN2Vec can also learn node representations based random walks and ComplEx can capture fine interactions through complex-valued vectors, thus being more capable of capturing comprehensive node information before supervision.
5.3 Link Prediction
For link prediction, we train all models with the randomly selected 80% links and evaluate towards the 20% held-out links. We use the Hadamard function to construct feature vectors for node pairs and train a two-layer MLP classifier on the 80% training links. We evaluate WalkLM with AUC (area under the ROC curve) and MRR (mean reciprocal rank). Note that, HAN cannot predict links on MIMIC-III for its restriction to embed only one type of node at a time, and thus it cannot predict links between different types of nodes on MIMIC-III [53].
8

---

29.. 10c2 2254 90,5
Loggt of bottaiin GpR C- PN
-5
1555
23,00 95.7
116
4204
12/6
(90
190
12,00 80,65 /0/77
12.8
Leel ofbo Kodio Ang Mineel
witphiiiiaahiiittttio. ACC
Louglre Loggeedlotae cotaaaaatin
Macool
ABC
Locad oot wotealiiiie
-0
100
200
X-6
108
200
500
Nomber immplet
127)
Namber of uxmple
12")
Vemmiiation poobehizzy
loranaben probiccklt
(a) N on node classification
(b) Non link prediction
(c) a on node classification
(d) a on link prediction
Figure 3: Analysis of the number of sampled walks N and the termination probability a.
As shown in Table 2 our fine-tuned WalkLM demonstrates outstanding performance in uncovering latent associations among nodes in attributed graphs. In general, WalkLM outperforms all ten baselines with an average of 5.97% performance gain over the second-best performance, showing that our pro- posed framework can learn accurate edge representation for link prediction. As a RW-based method, M2V can effectively employ meta-path- path-guided random walks to capture topological information and trace meta-path understand relations between nodes. As relation-leamingg methods, ConvE and ComplEx design different deep neural models to evaluate triplets. ConvE can achieve good performance by using convolutions over embeddings to mine relations between entities. ComplEx can capture relations via complex-valued embeddings, so to better represent the complex inherent relations among entities. Compare to M2V. ConvE, and ComplEX, WalkLM can effectively capture the complex relations by providing characteristic graph traits and reconstructing network proximity of nodes that inherit from RWs. On the other hand, the unsupervised HGNNs, especially HeCo and SHGP, perform rather poorly, again because their training mechanisms are not aligned with the link prediction task. Such observation is consistent with the results in the recent work (72), showing the heterogeneous approaches that only preserve certain-type entities fail to capture accurate representations for all kinds nodes.
5.4 Ablation Studies
To better understand our proposed techniques, we closely study our framework by selecting different LMs and varying the graph-aware IMM fine-tuning mechanism.
Compared with the graph-based baselines, the LM-based models (c.g., LM (XRoBERTa)) R. LM DG and LM (DRoBERT: To ) are able to learn accurate and rich node attributes. However, it difficult for them to mine the relations between nodes in the attributed graph, where all of them perform worse than the RW-based and relation learning-basee methods in the link prediction task. Considering the overall performance of the above three LMs on two different tasks and the goal of leariing graph embedding, we choose LM (DRoBERTa) as our starting point for fine- tuning.
Furthermore, we show that LM can further effectively integrate with existing heterogeneous graph al- gorithms, such as LM RGCN and LM + HGT, resulting in a notable performance enhancement over their individual methods. Compared with LM RGCN and LM- HGT, our proposed graph-aware LM fine-tuning can achieve the largest improvement gains based on the chosen LM (DRoBERTa) in both node classification and link prediction tasks, showing the effectiveness of capturing topological information together with semantics in modeling attributed graphs. The detailed analysis of the ablation studies shown in Appendix A.3
5.5 Hyper- paramep Studies
In this subsection, we investigate the model sensitivity the number of sampled walks N and the termination probability a, which are the major hyper-parameters WalKLM, For the space limitation, we show results on PubMed in Figure 3 and the results MIMIC-III in Appendix A.4 Overall, WalkLM is not sensitive to the two apper-parameters. where its performance increases slowly with N and a. Note that, too small N or large a cause the textualization data |C| to lose
Saugrrgihhhhoolllleeeeeeeeeerree Phutp://githhbbbu "htnps//githuh.coo//oopono http::/ghhhaaoooooommm//eeeee//o

---

(a) M2V PubMed
(b) ConvE PubMed
(c) HGT PubMed
(d) WalkLM PubMed
Vii Drew
incl
(e) M2V on MIMIC-III
(f) ConvE on MIMIC-III
(g) HGT on MIMIC-III
(h) WalkLM on MIMIC-III
Figure Visualization of different types of node embeddings on PubMed and MIMIC-III
sufficient information, while too large N or small a lead to extensive [C] and increase computational costs for fine-tuning. Setting N around 3 10 and around 0.05 seems appropriate to generate sufficient textual sequences, which can achieve good balance of performance and efficiency, We additionally investigate the model's sensitivity the quantity of language model masking samples, aiming to elucidate the parameter's influence on the performance of downstream tasks. The detailed experimental results are shown in Appendix A.4
5.6 Visualization
For an intuitive comparison, we visualize the embedding space of different types of nodes which are learned by M2V, ConvE, HGT, and our WalkLM, respectively. Specifically, we select gene/disease nodes on PubMed and patient/vist//diease nodes on MIMIC-III. The embeddings are further trans- formed into the 2-dimensional Euclidean space via the 1-SNE algorithm [50]. The nodes and links are both colored according to their types.
As shown in Figure M2V, ConvE, and HGT have blurred boundaries and even overlaps between different types of nodes, which are hard to distinguish. Our WalkLM shows the clearest boundaries between different types of nodes and the best within-type compactness, which indicate it can auto- matically organize heterogeneous nodes in a uniform space. Moreover, by connecting different types of nodes according the relations the data, we find that WalkLM can provide more discriminate distributions for different types of relations than others. The visualizations clearly demonstrate the advantages of WalkLM in capturing both attribute semantics and topological structures on graphs.
6 Conclusion
In this paper, we propose novel uniform language model fine-tuning framework for attributed graph embedding. The proposed WalkLM consists of two key modules, which encapsulate both attribute semantics and graph structures and obtain unsupervised generic graph representations. We conduct extensive experiments demonstrate the superior effectiveness of WalkLM against ofte-oof-hhe-rr baselines. For future work, it is intriguing to further design more sophisticated techniques and empirical evaluations toward the leverage of LMs and generalize our work to modern LLMs.
7 Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (No. 62300712660 Fujian Provincial Youth Education and Scientific Research Project under Grant JAT220811. I. Carl Yang was not supported by any fund from China.
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1 33:1877-1901, 2020.
[2] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541, 2006.
[3] Hongyun Cai, Vincent W. Zheng, and Kevin Chen-Chuan Chang. A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE Trans. Knowl. Data Eng., 169: 2018. doi: 10.1100/KKDE.208888874445 URL https:///oi..rrg// 1109/TKDE. 2018. 2807452
[4] Thilini Cooray and Ngai-Mar Cheung. Graph-wise common latent factor extraction for unsu- pervised graph representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6420 6428, 2022.
[5] Ganqu Cui, Jie Zhou, Cheng Yang, and Zhiyuan Liu, Adaptive graph encoder for attributed graph embedding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery data mining, pages 976-985. 2020.
[6] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp. and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[7] Jacob Devlin, Ming-Was Wei Chang, Kenton I Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810004805, 2018.
[8] Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. metapath2vee: Scalable representation learning for heterogeneous networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 135-144, 2017.
[9] Chantat Eksombatchai, Pranav Jindal, Jerry Zitao Liu, Yuchen Liu, Rahul Sharma, Charles Sugnet, Mark Ulrich, and Jure Leskovec. Pixie: A system for recommending 3+ billion items to 200+ million users real-time. Proceedings of the 2018 world wide web conference, pages 1775-1784, 2018.
[10] Tao-yang Fu, Wang-Chier Lee, and Zhen Lei. Hin2vec: Explore meta-paths heterogeneous in- formation networks for representation learning. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 1797-1806, 2017.
[1I] Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. Proceedings of The Web Conference 2020, pages 2331-2341. 2020.
[12] Josu Goikoetxea, Aitor Soroa, and Encko Agirre. Random walks and neural network language models on knowledge bases. In Proceedings of the 2015 conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies, pages 1434-1439, 2015.
[13] Aditya Grover and Jure Leskovec. node2vee: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864, 2016.
[14] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Advances in neural information processing systems, 30, 2017.
[15] William I.. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. IEEE Data Eng. Bull., 40(3):52-74, 2017. URL httt:://tt::tee computer org//ebbul/A17sppt/

---

[16] Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. A survey on recent approaches for natural language processing low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computaationa Linguistics: Human Language Technologies, pages 2545-255 -568,, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18666 18653/1120221 hww.:/l22nn.aaalllaaamaaaa m0in.201. URL
httpp::/accaantooooog/ org/2021 naacl maain. 201
[17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531/ 2015.
[18) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[19] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857-1867, 2020.
[20] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of the web conference 2020, pages 2704-2710, 2020.
[21] Sergey Ivanov and Evgeny Burnaev. Anonymous walk embeddings. In International conference on machine learning, pages 2186-21861109 1195. PMLR, 2018
[22] Xunqiang Jiang, Tianrui Jia, Yuan Fang, Chuan Shi, Zhe Lin, and Hui Wang. Pre-training on large-scale heterogeneous graph. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 756-766, 2021.
[23] Mingxuan Ju, Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye, and Chuxu Zhang. Multi-task self supervise graph neural networks enable stronger task generalization. In International Conference on Learning Representations (ICLR), 2023.
[24] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017.
[25] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.
[26] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-seqqueece pre- training for natural language generation, translation, and comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871 7871-7880, 2020.
[27] Zhiping Lin and Zhao Kang. Graph filter-based multi-view attributed graph clustering. IJCAI, pages 2723-2729, 2021.
[28] Changshu Liu, Liangjian Wen, Zhao Kang, Guangchun Luo, and Ling Tian. Self-supervised consensus representation learning for attributed graph. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2654-2662, 2021.
[29] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[30] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2 / 2021.
[31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint
2019.
12

---

[32] Qiheng Mao, Zemin Liu, Chenghao Liu, and Jianling Sun. Hinormer: Representation learning on heterogeneous information networks with graph transformer. In Proceedings of the ACM Web Conference 2023, pages 599-610, 2023.
[33] Silvio Micali and Zeyuan Allen Zhu. Reconstructing markov processes from independent and anonymous experiments. Discrete Applied Mathematics, 1000:08-12:: 2016.
[34] Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, llana Heinz, and Dan Roth. Recent advances natural language processing via large pre- pre-trained language models: A survey. arXiv preprint arXiv:2111.01244; 2021.
[35] OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[36] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2777 27744, 2022.
[37] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. arXiv preprint urXiv:2306.08302 2023.
[38] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701-710, 2014.
[39] Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey, Science China Technological Sciences, 63 (10):1872-1897, 2020.
[40] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
[41] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18)99, 2019.
[42] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(())5485-5551, 2020.
[43] Nils Rethmeier and Isabelle Augenstein. A primer on contrastive pretraining in language processing: Methods, lessons learned, and perspectives. ACM Computing Surveys, 55(10):1-17, 2023.
[44] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[45] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15, pages 593-607. Springer, 2018
[46] Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen. Ex- ploiting structured knowledge in text via graph-guide representation learning. arXiv preprint arXiv:2004.1422 2020.
[47] Mingchen Sun, Kaixiong Zhou, Xin He, Ying Wang, and Xin Wang. Gppt: Graph pre- trainin and prompt tuning to generalize graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1717-1727 2022.
[48] Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. S2gae: Self supervised graph autoencoders are generalizable learners with graph masking. In Proceed- ings of the Sixteenth ACM International Conference on Web Search and Data Mining, pages 787-795, 2023.
13

---

[49] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International conference on machine learning, pages 2071-2080. PMLR, 2016.
[50] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[51] Haoyu Wang. Vivek Kulkarni, and William Yang Wang. Dolores: deep contextualized knowl- edge graph embeddings. arXiv preprint arXiv:1811.0014 2018.
[52] Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. Simkge: Simple contrastive knowledge graph completion with pre-trained language models. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4281-4294, 2022.
[53] Xiao Wang. Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Het- erogeneous graph attention network. In The world wide web conference, pages 2022-2032, 2019.
[54] Xiao Wang, Nian Liu, Hui Han, and Chuan Shi. Self-supervised heterogeneous graph neural network with co-contrastive learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 1726-1736, 2021.
[55] Xiao Wang, Deyu Bo, Chuan Shi, Shaohua Fan, Yanfang Ye, and Philip S. Yu. A survey on heterogeneous graph embedding: Methods, techniques, applications and sources. IEEE Trans. Big Data, 9(2):415 2023. doi: 10.1 (0.1099TBDAAAA2222217555 URL https
[56] Xiaozhi Wang. Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176-194, 2021.
[57] Nicolas Webersinke, Mathias Kraus, Julia Bingler, and Markus Leippold. Climatebert: A pretrained language model for climate-related text. Available at SSRN, 2022.
[58] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. IEEE Trans. Neural Networks Learn Syst. 32(1): 24, 2021. doi: 10. 10.1109/TNNLS, 200.0299..000029788886666 URL https::/doi. org/ 10. 1109/ TNNLS 2020 2978386]
[59] Hongyuan Xu, Yunong Chen, Zichen Liu, Yanlong Wen, and Xiaojie Yuan. Taxoprompt: A prompt-based based generation method with taxonomic context for self-supervised taxonomy expan- sion. Proceedings of the Thirty-Firs International Joint Conference on Artificial Intelligence, IJCAI-22, pages 4432 4438, 2022.
[60] Bin Yan and Mingtao Pei. Clinical-bert: Vision-language pre-training for radiograph diagnosis and reports generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2982-2990 2022.
[61] Carl Yang, Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. Heterogeneous network representation learning: A unified framework with survey and benchmark. IEEE Transactions on Knowledge and Data Engineering. 34(10):4855 487835444777,,, 2020.
[62] Carl Yang. Jieyu Zhang, Haonan Wang. Sha Li. Myungwan Kim, Matt Walker, Yiou Xiao, and Jiawei Han. Relation learning on social networks with multi-modal graph edge variational autoencoders. In Proceedings of the 13th International Conference Web Search and Data Mining, pages 699-707, 2020.
[63] Yaming Yang, Ziyu Guan, Zhe Wang, Wei Zhao, Cai Xu, Weigang Lu, and Jianbin Huang. Self-supervised heterogeneous graph pre-training based on structural clustering. In Conference on Neural Information Processing Systems, 2022.
14

---

[64] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[65] Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion. arXiv preprint 190.:::9990903393 2019.
[66] Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. Exploring large language models for knowledge graph completion. arXiv preprint arXiv:2308 13916, 2023.
[67] Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning. Percy S Liang, and Jure Leskovec. Deep bidirectional language-knowledge graph pretraining. Advances in Neural Information Processing Systems, 35:37309-37323, 2022.
[68] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In International Conference on Learning Representations, 2022.
[69] Donghan Yu. Chenguang Zhu, Yiming Yang, and Michael Zeng. Jaket: Joint pre-training of knowledge graph and language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11630-1160 1638, 2022.
[70] Muhan Zhang and Yixin Chen. Link prediction based graph neural networks. Advances in neural information processing systems, 31, 2018.
[71] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[72] Rui Zhang, Arthur Zimek, and Peter Schneider-Kamp. A simple meta-path-free framework for heterogeneous network embedding. Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 2600-2609 2022.
[73] Wen Zhang, Yushan Zhu, Mingyang Chen, Yuxia Geng, Yufeng Huang. Yajing Xu, Wenting Song, and Huajun Chen. Structure pretraining and prompt tuning for knowledge graph transfer. In Proceedings of the ACM Web Conference 2023, pages 2581-2590, 2023.
[74] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al open, 1:57-81, 2020.
[75] Qi Zhu, Carl Yang, Yidan Xu, Haonan Wang, Chao Zhang, and Jiawei Han. Transfer learning of graph neural networks with ego-graph information maximization. Advances in Neural Information Processing Systems, 34:1766-1779, 2021.
[76] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pages 2069-2080, 2021.
[77] Zhaorui Zhu, Hongyi Yu, Caiyao Shen, Jianping Du, Zhixiang Shen, and Zhenyu Wang. Causal language model aided sequential decoding with natural redundancy. IEEE Transactions on Communications, 2023.
15

---

A Appendix
A.1 Experiments based on KG datasets
Datasets. We conduct extensive experiments two real-world KG datasets, i.e., Freebase and FB15K-237 12 Freebase contains graph of books, films, sports, and locations. The nodes and edges are extracted according to [61] A large portion of books are labeled into eight genres of literature. Each labeled book has only one label. FB 1B1K-5277 is a standard dataset in the knowledge graph community, which contains 310,1 16 triples with 14,541 entities and 237 relation types. Since we did not manually label the nodes, we only predicted whether a triple is correct or not on this dataset. We matched the entities with Wikidata entities and obtained metadata from Wikidata, and constructed rough attribute graph dataset by using the names and descriptions of the nodes as textualized features of the nodes, and directly applying their original relationship text as the edge textualized attributes.
Node Classification. As shown in Table our proposed WalkLM has superior performance, indicating the importance of leveraging both semantic and structural information in attributed graphs. WalkLM achieves 40.24% performance gains on Freebase over the second-best performance on average. Specifically, text-based Knowledge graph completion method, SimKGC can effectively employ text-based contrastive learning to capture a rich set of semantic information. Compared with SimKGC, WalkLM can effectively combine the complex semantic and graph structure information of attributed graphs, so as to accurately model the complex attributes of nodes.
We evaluate WalkLM with AUC and MRR. As shown in Table our fine-tuned WalkLM demonstrates outstanding performance in uncovering latent associations among nodes in attributed graphs. In general, WalkLM outperforms all ten baselines with an average of 2.05% performance gain over the second-best performance, showing that our proposed framework can learn accurate edge representation for link prediction. ComplEx and ConvE consistently demonstrate promising performance by effectively capturing generic node representations. However, as a text- based Knowledge graph completion method, SimKGC can sometimes outperform others in terms of the MRR metric, where SimKGC can enhance semantic similarity between nodes through contrastive learning based on bi-encoder architecture and three types of negatives. Compared with ConvE, ComplEX, and SimKGC, WalkLM can effectively capture the complex relations by providing text- based semantic traits of characteristic graph and reconstructing network proximity of nodes that inherit from RWs.
Link Prediction.
A.2 Graph-level Classification
Compared with node or edge classification, aggregating node embeddings for graph-level classifi- cation needs more context information. Furthermore, graph-level classification presents its own set of challenges, which require holistic capturing of graph structures and often do not rely much on attributes. Therefore, it is difficult to find universal representation learning approach that solves all different levels of graph mining tasks. Technically, adapting our method to graph-level classification necessitates some subtle decisions to make (such as whether to include graph ID as a virtual node). We've conducted preliminary analysis on aggregating our learned node embeddings for graph- tasks. Specifically, we adopt widely-used MUTAG dataset and use mean accuracy as the met- ric 48 71]. The results on the popular MUTAG dataset are listed in Table [5] Although the findings are encouraging and show the potential of WalkLM, further studies are still needed to establish a clear advantage of our approach over SOTA graph classification baselines.
A.3 Detailed Ablation Studies
From T we have the following observations: (1) Compared with the graph-based baselines, the LM-based models (e.g.. LM (XRoBERTa). LM (GPT-2), and LM (DRoBERTa)) are able to learn accurate and rich node attributes, leading to superior performance in node classification. For the PubMed dataset distributed on classes, LM (XRoBERTa), LM (GPT-2), and LM (DRoBERTa) achieve 63.78%, 135.04%, and 130.89% performance gains over the second-best performance on
11 "ntt://www.freebbss.coml
12 "htnps//papprssiihcccee.. httpa///w..ooeaaac////hh//aaa// https://ss wwt.://ww..dtoouuoddddda///rrrie
16

---

Table 4: Different downstream task results (%) with the corresponding std (+) on two KG datasets. The best performances are in bold and the second runners are shaded in gray, where * denotes significant improvement according to the Wilcoxon signed-rank significance test.
Task
Node Classification
Link Prediction
Dataset
Freebase
Freebase
FB15K-237
Metric
Macro-F1
Micro-F1
AUC
MRR
AUC
MRR
M2V
25.74+1.12
50,25=2.57
80.68=1811
88.97+0.93
90.35=0.50
96.78=0.19
Vec2
15.56:1.07
43.67=2.12
80.04+3.01
90.90 =0.87
79.68=0.83
92.85=0.40
ConvE
2.81
49.31+3.45
1..14+1.03
93.57+0.42
92.88=0.42
97.57=0.15
ComplEx
20.25+1.62
49.43+3.57
84.01+1.43
91.46+0.56
95.03=0.35
97.88=0.22
SimKGC
35.88+0.87
56.12=0.44
8.53
94.21+0.34
93.80=0.31
97.62=0.30
RGCN
1..37+1.54
45.86+1.03
82.75+0.89
91.52+0.64
85.88=0.35
89.84=0.19
HAN
1.77
39.30+2.18
80.73+1.37
91.111..4
82.06-0.53
89.31
HGT
19.97
47.9922.56
8..94+1844
89.6510.43
87.41=0.69
94.6210.34
HeCo
23.95
48.62.1.13
79.3210.86
87.4010.32
78.82-0.37
90,41
SHGP
13.83+1.27
39.07=1.39
78.37+1.77
85.52+0.69
78.56=0.33
89.84 89.84=0.21
XRoBERTa
18.1042.01
67.95
7..94&1.62
88.1710.91
75.62:1..22
91.10=0.71
GPT-2
49.24:2.12
6..78
60.45+2.43
83.29+1.87
68.87=1.21
8.72
DRoBERTa
11.76+1.24
69.51
1.82
91.21+1.17
84.15=0.63
93.39=0.39
LM+RGCN
28.38+0.63
53.37=2.27
83,63+1.81
96.38+0.67
87.72=0.50
94.47=0.46
LM+HGT
20.79:0.67
48.73=3.13
83.09=1.23
89.7920.35
88.18=0.61
94.85=0.27
WalkLM
55.01
71.36=1.53*
92.11=2.24*
9..54
95.65=0.18
98.35
Table 5: Accuracy results (%) of graph-level classification on MUTAG.
Dataset
MUTAG
Model
HeN22eec
ConvE
ComplEx
LM (DRoBERTa)
WalkLM w/o, graph-ID
WalkLM
Accuracy
78.72
77.64
78.69
79.23
79.77
81.39
average, respectively. For the MIMIC-III dataset on the total 19 classes, LM (XRoBERTa), LM (GPT-2), and LM (DRoBERTa) achieve 5.17%, 30.17%, and 20.12% average performance gains compared to the second-best performance, respectively.
(2) The choice of LMs affect the performance of fine-tuning, Due different pre-training corpora, LM (XRoBERTa) performs worse than LM (DRoBERTa) in most cases. Moreover, LM (GPT-2) achieves average of 3.30% improvement over LM (DRoBERTa) in node classification, while LM (DRoBERTa) achieves an average of 7.08% improvement over LM (GPT-2) in link prediction. different tasks and the goal of learning graph embedding, we choose LM (DRoBERTa) as our starting point for fine- tuning.
Considering the overall performance of the above three LMs
(3) Furthermore, LM can further effectively integrate with existing heterogeneous graph algorithms, resulting a notable performance enhancement over their individual methods. Specifically, compared with RGCN, LM + RGCN achieves an average of 50.38% improvement in node classification, and achieves up to 6.5 6.59% improvements in link prediction. Compared with HGT, LM + HGT achieves up to 30.64% improvements in node classification and 7.42% improvements in link prediction.
(4) Compared with LM RGCN and LM HGT, our proposed graph-aware LM fine-tuning can achieve the largest improvement gains based on the chosen LM (DRoBERTa) in both node classification and link prediction tasks, showing the effectiveness of capturing topological information together with semantics in modeling attributed graphs. Specifically, our WalkLM outperforms the chosen LM (DRoBERTa) by up to 13.45% in node classification. Particular, our WalkLM achieves up to 59. 9.0 improvements in link prediction, which demonstrates our WalkLM can better learn accurate edge representation for link prediction by the graph-aware LM fine-tuning.
A.4 Detailed Hyper-parameter Studies
We show the results of the model sensitivity on the number of sampled walks N and the termination probability a MIMIC-III in Figure Overall, WalkLM is not sensitive to the two hyper-parameters. where its performance increases slowly with N and a. Note that, setting N around 3 10 and around 0.05 seems appropriate to generate sufficient textual sequences and limit computational costs
17

---

Table 6: The detailed ablation results (%) with the corresponding std (=) on two datasets. The best performances are in bold and the second runners are shaded in gray, where * denotes significant improvement according to the Wilcoxon signed-rank significance test.
Task
Node Classification
Link Prediction
Dataset
PubMed
MIMIC-III
PubMed
MIMIC-III
Metric
Macro-F1
Micro-FI
Macro-F1
Micro-F1
AUC
MRR
AUC
M2V
15.35
20,27
19.69
29.24
74.53
89.58
75.05
88.32
(+1.27)
(+3.01)
(40.62)
(+1.57)
(+3.79)
(+2.05)
(a0.41)
(10.23)
HIN2Vec
11.57
18.92
19.12
28.05
74.21
90.56
73.46
88.
(+1.23)
(+2.78)
(#1.32)
(=1.44)
(#5.49)
(+1.06)
(+0.41)
(+0.14)
ConvE:
16.06
19.16
24.44
32.89
76.48
92.27
69.56
84.88
(#3.69)
(#4.00)
(#1.28)
(=0.86)
(+4.31)
(+0.57)
(+0.36)
(+0.25)
ComplEx
13.93
18.27
9.82
21.39
79.81
91.79
63.86
81.40
(+2.59)
(+4.12)
(+0.56)
(+3.12)
(+0.97)
(+0.48)
(+0.42)
(+0.40)
SimKGC
21.97
30.83
51.62
58,50
79.62
91.43
67.73
84.86
(+3.51)
(+3.10)
(+181)
(+1.52)
(+2.72)
(+0.48)
(+1.6))
(+0,54)
RGCN
12.50
18.50
7.19
14.55
72.08
88.20
57.31
73,91
(+2.36)
(+1.41)
(+0.77)
(+3.25)
(+1.13)
(+0.47)
(20.71)
(10.57)
HAN
15.29
16.95
14.73
70.57
(+2.87)
(+2.71)
(#0.58)
(=1.69)
(+1.58)
(+0.62)
HGT
11.98
20.12
8.03
17.79
77.24
89.63
64.01
81.54
(+2.23)
(#3.89)
(#0.87)
(00.83)
(+3.50)
(+0.84)
(+0.36)
(+0,56)
HcCo
10.32
18.01
10.78
15.26
65.04
83.29
53.13
71.81
(+1.12)
(+0.87)
(+0.41)
(+1,52)
(+1.26)
(+0.72)
(+0.47)
(+0.35)
SHGP
10.80
19.28
11.34
17.44
68.22
85.34
54.49
(+3.03)
(+0,91)
(#1.29)
(=1.49)
(+2.71)
(+0.48)
(+0.33)
(+0.24)
LM
40.10
44.71
54.51
61.27
60.20
84.23
51.21
74.22
(XRoBERTa)
(+4.62)
(+3.68)
(+1.50)
(+1.22)
(+2.78)
(+1.71)
(+0.17)
(+0.26)
LM
59.43
61.53
70.26
72.67
$1.71
80.54
50.66
72.36
(GPT-2)
(+4.73)
(#3.43)
(+1.43)
(=0.90)
(+3.67)
(+2.49)
(a0.74)
(10,86)
LM
58.29
60,57
66.25
70.14
60.97
83.00
51.44
75.09
(DRoBERTh)
(+2,44)
(+2.11)
(+1.60)
(+1.52)
(+2,98)
(+0.40)
(+0.14)
(10.29)
LM
13.83
22.70
14.32
24.59
72.35
88.86
58.62
78.78
+RGCN
(+0.73)
(+3.25)
(+0.87)
(+1.17)
(+4.34)
(+1.4))
(+0.50)
(+0.10)
LM
12.81
21.79
10.49
20.57
82.97
89.98
65.01
82.28
+HGT
(+1.22)
(+3.54)
(+0.41)
(s0.97)
(+3,91)
(+0.88)
(a0.20)
(+0,30)
WalKLM
60.42*
62.33*
75.16*
77.89*
85.65
94.16*
82.15
92.78*
(#2.62)
(#3.13)
(#0.93)
(=0.70)
(#3.28)
(+0.37)
(#0.67)
(+0.68)
Table 7: Different downstream task results (%) with ratio of masked samples m on PubMed.
Task
Node Classification
Link Prediction
Metric
Macro-Fl
Micro-F1
AUC
MRR
0.05
52.97
56.33
83.16
93.47
m 0.15
60.42*
62.33*
85.65*
94.16*
m = 0.25
53.80
56.09
82.92
93.75
m 0.35
52.22
55.61
82.38
92.72
for fine-tur fine-tuning, which can achieve good balance of performance and efficiency. Furthermore, for the ratio of masked samples m, the specific results are listed in Table Overall, WalkLM sensitive m, where the optimal value across different tasks is 0.1.5, which is consistent with the empirical
selection in our paper and the previous work 7 31
60
2541
1011 92.29 8033
1601 75.15 10/34
24
00.4
2220
Teigh www.raaturiiaaa Bra PR
Linga woooaattitaorr Maaee:
Langa anooaabbatsnn dea ACC
37.2
Lenghaf Justem: tneappooo
of www.aaalmmmmmmmmmmmmmmmm dan
Maune"1)
I00
202 7.0
DU0
200
160
250
Nazher of imppled walks
Nambor ocmppp
Tomaaaaiin probakklly
Vonsinatin pwwbacilay
(4) N on node classification Figure 5: Analysis of the number of sampled walks N and the termination probability
(b) on link prediction
(c) d on node classification
(d) a on link prediction
18


Reviews:
[{"review_id": "KASJRinmMP", "reviewer": "Reviewer_vuBn", "summary": "This paper proposes WalkLM, an unsupervised graph representation learning leveraging the power of the language model. WalkLM first samples a set of sequences of entities from attributed graphs by random walk and fine-tunes the language model on textualized walks. The learned embedding by these procedures is employed to node-level and edge-level downstream tasks. The authors demonstrate that WalkLM significantly outperforms baselines in node classification and link prediction tasks on two real-world datasets.\n", "questions": "See the weaknesses section.\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "WalkLM is a neat idea that combines graph representation learning and language modeling. This paper shows how the classic random walk approach can enhance graph representations using the expressiveness of the language model. The authors conduct various experiments including ablation studies, hyperparameter analysis, qualitative analysis, and efficiency analysis. There is no particular reason to reject the paper.\n", "weaknesses": "One weakness is the number of datasets used for evaluation. Enough experiments have been done, but only two datasets are covered in this paper. This paper proposed a new data format and corresponding training method beyond proposing a just new model architecture. So, it is necessary to clarify whether there are plenty of scenarios that this method can be easily used in the real world. What are the conditions of the tasks and datasets to which this method can be applied, and what datasets can be employed for WalkLM besides the two datasets? If possible, further experiments should be done on those datasets.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the overall positive evaluations and detailed suggestions. In the following, we focus on the main issues to provide our feedback:\n\n> **1:** The conditions of the tasks and datasets to apply, and the number of datasets used for evaluation.\n\nThe target of this work is general attributed graph representation learning, and the proposed WalkLM can learn node representations that can be used for various downstream tasks (e.g., disease prediction and topic modeling) and datasets (e.g., PubMed and MIMIC-III). In response to the reviewer's suggestions on adding more datasets for experiments, we have extended our experiments to include a dataset of FreeBase (as used in [1]). The results further confirm the generalizability of our proposed method:\n\n**Table1: Different downstream task results (%) on FreeBase.**\n\n| Dataset      |                     | FreeBase  |                 |           |\n| ------------ | ------------------- | --------- | --------------- | --------- |\n| Task         | Node Classification |           | Link Prediction |           |\n| Metric       | Macro-F1            | Micro-F1  | AUC             | MRR       |\n| M2V          | 25.74               | 50.25     | 80.68           | 88.97     |\n| HIN2Vec      | 15.56               | 43.67     | 80.04           | 90.90      |\n| ConvE        | 25.13               | 49.31     | 88.14           | 93.57     |\n| ComplEx      | 20.25               | 49.43     | 84.01           | 91.46     |\n| RGCN         | 15.37               | 45.86     | 82.75           | 91.52     |\n| HAN          | 14.25               | 39.30      | 80.73           | 91.61     |\n| HGT          | 19.97               | 47.99     | 81.94           | 89.65      |\n| HeCo         | 23.95               | 48.62     | 79.32           | 87.40      |\n| SHGP         | 13.83               | 39.07     | 78.37           | 85.52     |\n| LM(DRoBERTa) | 51.76               | 69.51     | 79.22           | 91.21     |\n| WalkLM       | **55.01**           | **71.36** | **92.11**       | **96.54** |\n\n**References**  \n[1] Heterogeneous network representation learning: A unified framework with survey and benchmark, TKDE, 2020.\n\n> **2:** The new data format and the real-world applicability. \n\nOur method does not introduce a new data format, but rather introduces the novel process of *textualization*, which converts general attributed graphs into text-like sequence data. This process allows us to leverage the capabilities of pre-trained language models for graph representation learning. Importantly, our method only requires some meaningful attributes on the graphs, which is available in most real-world graphs such as biological networks, social networks, and knowledge graphs."}, {"review_id": "YycPJ0hmDM", "reviewer": "Reviewer_4W5x", "summary": "The paper proposed a method for knowledge-graph-embedding (KBE) task using integration of language model and random walks. Specifically, authors first verbalized the path via random walks in KB, then, fine-tuning language model for the verbalized path, finally, using the embedding layer of language model as the embedding of KB. Also, the numerical studies compared 9 different baseline graph embedding methods on downstream tasks node classification and link prediction. The proposed method WalkLM shows a good performance compared to listed baselines.", "questions": "It would be great to include more recent KB + language works as baseline to show the good performance of presented method.", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well organized and easy to follow.\n2. The proposed method WalkLM shows a good performance compared to baselines.", "weaknesses": "1. My top concern is the novelty. The proposed method simply fine-tuned a language model based on path from random walk. However, similar ideas have been widely explored by previous papers like [1]-[4]. \n2. In section 2 related work, a lot of related papers about KB + language model besides [1]-[4] are not included.\n\n\n[1] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation\n[2] Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning\n[3] Deep Bidirectional Language-Knowledge Graph Pretraining\n[4] Jaket: Joint pre-training of knowledge graph and language understanding", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. Our responses are listed as follows:\n\n> **W1:** My top concern is the novelty. The proposed method simply fine-tuned a language model based on path from random walk. However, similar ideas have been widely explored by previous papers like [1]-[4].\n\nThe setting and technical design of our method are new, where we leverage the capability of modern LMs for general attributed graph representation learning via a novel graph textualization process. The approach is concretely backed up by graph theory and properly leverages the advantages of random walks, which have been shown effective in capturing flexible graph topological structures. Note that, we do not intend to claim much novelty in the LM fine-tuning process. However, we believe that fine-tuning LMs for vertical or non-textual domain data and tasks can indeed lead to important discoveries, and many impactful contributions in the field have hinged on endeavors as such [1]-[5].\n\n**Reference**  \n[1] Training language models to follow instructions with human feedback, NeurIPS, 2022.  \n[2] Fine-tuning language models to find agreement among humans with diverse preferences, NeurIPS, 2022.  \n[3] Generating training data with language models: Towards zero-shot language understanding, NeurIPS, 2022.  \n[4]  Solving Math Word Problems via Cooperative Reasoning induced Language Models, ACL, 2023.  \n[5] Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions, ACL, 2023.\n\n\n> **W2:** In section 2 related work, a lot of related papers about KB + language model besides [1]-[4] are not included.\n\nWe thank the reviewer for highlighting the related work on KB+LM. We will add the discussion such as regarding [1]-[4] in our related work. However, we want to emphasize that, the goal of this work is to leverage the capability of modern LMs for general attributed graph representation learning, instead of (1) leveraging KB for enhancing LMs [1]-[5] or (2) leveraging LM for KB completion [6]-[9]. Therefore, we believe this work is novel and significantly different from the existing work on KB+LM, and we will further clarify this in our revision.\n\n**Reference**  \n[1] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation, TACL, 2021.  \n[2] Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning, EMNLP, 2020.  \n[3] Deep Bidirectional Language-Knowledge Graph Pretraining, NeurIPS, 2021.  \n[4] Jaket: Joint pre-training of knowledge graph and language understanding, AAAI, 2022.  \n[5] Knowledge Enhanced Contextual Word Representations, EMNLP-IJCNLP, 2019.  \n[6] Fusing topology contexts and logical rules in language models for knowledge graph completion, Information Fusion, 2023.  \n[7] Multi-task learning for knowledge graph completion with pre-trained language models, COLING, 2020.  \n[8] From discrimination to generation: Knowledge graph completion with generative transformer, WWW, 2022.  \n[9] Multilingual Knowledge Graph Completion from Pretrained Language Models with Knowledge Constraints, ACL, 2023.  \n\n> **Q1:** It would be great to include more recent KB + language works as baseline to show the good performance of presented method.\n\nAs we discussed in the response to W2, our focus in this work is to leverage LMs for general attributed graph representation learning, which cannot be adequately achieved by existing KB+LM works. Although some studies such as [1] can perform KB representation learning, they do not apply to general attributed graphs (KBs usually do not have attributes). Furthermore, as suggested by Reviewer vuBn, we have added one KB dataset (i.e., Freebase [2]) to enhance the experimental results, which shows the generalizability of our method to actual KBs. \n\n**Table1: Different downstream task results (%) on FreeBase.**\n\n| Dataset      |                     | FreeBase  |                 |           |\n| ------------ | ------------------- | --------- | --------------- | --------- |\n| Task         | Node Classification |           | Link Prediction |           |\n| Metric       | Macro-F1            | Micro-F1  | AUC             | MRR       |\n| M2V          | 25.74               | 50.25     | 80.68           | 88.97     |\n| HIN2Vec      | 15.56               | 43.67     | 80.04           | 90.90      |\n| ConvE        | 25.13               | 49.31     | 88.14           | 93.57     |\n| ComplEx      | 20.25               | 49.43     | 84.01           | 91.46     |\n| RGCN         | 15.37               | 45.86     | 82.75           | 91.52     |\n| HAN          | 14.25               | 39.30      | 80.73           | 91.61     |\n| HGT          | 19.97               | 47.99     | 81.94           | 89.65      |\n| HeCo         | 23.95               | 48.62     | 79.32           | 87.40      |\n| SHGP         | 13.83               | 39.07     | 78.37           | 85.52     |\n| LM(DRoBERTa) | 51.76               | 69.51     | 79.22           | 91.21     |\n| WalkLM       | **55.01**           | **71.36** | **92.11**       | **96.54** |\n\n**Reference**   \n[1] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation, TACL, 2021.  \n[2] Heterogeneous network representation learning: A unified framework with survey and benchmark, TKDE, 2020.\n"}, {"review_id": "ev2kzuPqF7", "reviewer": "Reviewer_x99Y", "summary": "GNNs for training require sufficient training data for downstream tasks to achieve strong performance. Self supervised learning approaches are inefficient due to the presence of a variety of node attributes and complicated relations between nodes. Inspired from the success in LLMs, they convert the graphs into natural language to be consumed by pre -trained language models. Random walk sequences are extracted from the graph and are converted into text through entity level and walklevel textualization. Roberta model is then fine tuned to predict the masked node / edges as textual sequences. The approach is tried on various node classification and link prediction tasks with significant improvements. ", "questions": "1. Do the authors have an opinion on - applying the same technique for nontextual networks with nontextual Ids and numeric attributes? I think the bigger question is how well can transformer-like models embed graph structures without the necessity of using GNN-like graph embeddings. \n", "limitations": "1. The process of converting graphs to text as presented in the paper consists of entity level textualization which can be taxing as the number and types of nodes increase to millions.\n2. Extracting random walks is an expensive operation and this method will have scalability issues when the amount of nodes/edges increases to millions or billions\n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Since the main crux of the approach is how we textualize the graphs. The method is highly flexible to extract node, edge, sub-graph, path-specific or even graph embeddings. \n2. I like how self-sufficient the paper is. All essential material to understand and analyze has been fitted into the main paper. \n3. The technique has been compared to a lot of standard GNN approaches and in the ablation LLM + GNN have been combined to further enhance performance and use the best of both worlds. \n4. Overall the approach is elegant and easy to understand. It brings huge improvements to downstream tasks. I think the approach is a significant contribution to the domain of LLMs and graphs. \n", "weaknesses": "1. The authors do not present results on the graph level classification task. Only node classification and link prediction tasks have been presented. It would be interesting to see if aggregating node embeddings for graph-level tasks also performs well as this task needs more context as compared to node or edge classification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the overall positive summary and accurate description of our major contributions. As for the several weaknesses you mentioned, our responses are listed as follows:\n\n> **W1:** The authors do not present results on the graph level classification task. Only node classification and link prediction tasks have been presented. It would be interesting to see if aggregating node embeddings for graph-level tasks also performs well as this task needs more context as compared to node or edge classification.\n\nWe appreciate your constructive suggestion. Graph-level classification presents its own set of challenges, which require holistic capturing of graph structures and often do not rely much on attributes. Therefore, it is difficult to find a universal representation learning approach that solves all different levels of graph mining tasks. Technically, adapting our method to graph-level classification necessitates some subtle decisions to make (such as whether to include graph ID as a virtual node). Following the reviewer's suggestion, we've conducted a preliminary analysis on aggregating our learned node embeddings for graph-level tasks. Specifically, we adopt a widely-used MUTAG dataset and use mean accuracy as the metric [1][2]. The results on the popular MUTAG dataset are listed in Table 1. Although the findings are encouraging and show the potential of WalkLM, further studies are still needed to establish a clear advantage of our approach over SOTA graph classification baselines.\n\n**Table 1: Accuracy results (%) of graph-level classification on MUTAG.**\n| Dataset|      |  | |  MUTAG     |  |       |\n| :---: | :---: | :-----: | :----: | :----------: | :------------------: | :-------: |\n| Model | ConvE | ComplEx | HinVec | LM(DRoBERTa) | WalkLM w/o. graph-ID |  WalkLM   |\n|Accuracy | 77.64 |  78.69  | 78.72  |    79.23     |        79.77         | **81.39** |\n\n**Reference**  \n[1] S2GAE: Self-Supervised Graph Autoencoders are Generalizable Learners with Graph Masking, WSDM, 2023.  \n[2] An end-to-end deep learning architecture for graph classification, AAAI, 2018.\n\n> **Q1:** Do the authors have an opinion on - applying the same technique for nontextual networks with nontextual Ids and numeric attributes? I think the bigger question is how well can transformer-like models embed graph structures without the necessity of using GNN-like graph embeddings.\n\nThanks for the question.  In this work, we apply RW to help LM in capturing graph structures. In our experiments, we have found that nontextual IDs can be effectively embedded as new tokens that appear in different random-based sequences. However, we have not yet found an effective way to model fully nontextual networks and numeric attributes. It is promising to establish our framework for more general graph representation along with the active research on more powerful and transparent LMs.\n\n> **L1:** The process of converting graphs to text as presented in the paper consists of entity level textualization which can be taxing as the number and types of nodes increase to millions.\n\nWe only need to perform the rule-based textualization once for every node during the pre-processing stage, which is not only pretty fast but also highly amenable to parallelization.\n\n> **L2:** Extracting random walks is an expensive operation and this method will have scalability issues when the amount of nodes/edges increases to millions or billions.\n\nInstead of being a limitation, using random walks for capturing graph structures is in fact very efficient and scalable, which is an advantage of our method. Specifically, in industry, random walks on large graphs such as those with millions to billions of nodes can be done on CPUs with terabytes of memory with numerous threads in parallel [1]. \n\n**Reference**  \n[1] Pixie: A system for recommending 3+ billion items to 200+ million users in real-time, WWW, 2018.\n"}, {"review_id": "trdivrfxu9", "reviewer": "Reviewer_DtxK", "summary": "This paper studies the universal language model for generic graph representation learning. To model the complex attributes on multiple types of nodes and links with the consideration of graph structure, the authors proposed WalkLM. Specifically, to compose meaningful textual sequences, the attributed RWs and an automated program are exploited. Besides, to transfer the capability of language models, a fine-tuning strategy is designed to extract embedding vectors from the LM. Extensive experiments on real-world datasets show the superiority of proposed method over the state-of-the-art methods.", "questions": "NA", "limitations": "See the Weaknesses part.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1. This paper is well motivated. Considering the graph representations are generally limited to specific downstream predictions, and the performance may be unsatisfactory with unsupervised GNN ways, the authors integrate language models (LMs) and random walks (RWs) to obtain unsupervised generic graph representations\n\nS2. The paper is well written and organized, making it easy to follow for readers and the work archives promising different downstream task results. \n\nS3. The problem setting is quite realistic, and the experimental results are also encouraging. In my opinion, this work has potential to promote the application of general-purpose graph methods to more real-world problems.\n\nS4. The adoption of random walk is a quite highlight part of this paper, which can easily capture flexible graph topological structures without supervision for various independent downstream tasks. \n", "weaknesses": "W1. It is suggested to include the deeper analysis on why the proposed framework can stay strong with a small size of training data for results in the few-shot setting.\n\nW2. The analysis of the some hyperparameters need to be considered, where the hyperparameters may have an impact on the performance of the downstream tasks. For example, the latent dimension d and the number of masked samples.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their positive and detailed comments, which affirm the importance of the problem we studied. In response to the weaknesses, our answers are as follows:\n\n> **W1:** It is suggested to include the deeper analysis on why the proposed framework can stay strong with a small size of training data for results in the few-shot setting.\n\nThrough the novel textualization process that converts general attributed graphs into text-like sequence data, our proposed WalkLM can leverage the capabilities of modern language models for graph representation learning. With the extensive pre-training of LMs on broad text corpora, the model can easily understand meaningful node attributes given a new graph, while the random walk strategy further allows it to capture graph structures.  This is how WalkLM can consistently exhibit superior performance in the few-shot setting. \n\n> **W2:** The analysis of the some hyperparameters need to be considered, where the hyperparameters may have an impact on the performance of the downstream tasks. For example, the latent dimension d and the number of masked samples.\n\nWe have analyzed specific hyperparameters in Section 5.5 of our paper (i.e., the number of sampled walks $N$ and the termination probability $\\alpha$). Furthermore, we have added experiments on the suggested hyperparameters. Regarding the latent dimension $d$, it's a common practice to utilize the output dimension of the original LMs (e.g., the dimension in GPT is 768 [1]). For the ratio of masked samples $m$, the specific results are listed as follows, where the optimal value across different tasks is 0.15, which is consistent with the empirical selection in our paper and the previous work [2]-[4]:\n\n**Table 1: Different downstream task results (%) with varing $m$ on PubMed.**\n| Dataset      |                     | PubMed |                 |       |\n| ------------ | ------------------- | -------- | --------------- | ----- |\n|   Task   | Node Classification |           | Link Prediction |           |\n|  Metric  |      Macro-F1       | Micro-F1  |       AUC       |    MRR    |\n| $m=0.05$ |        52.97        |   56.33   |      83.16      |   93.47   |\n| $m=0.15$ |      **60.42**      | **62.34** |    **85.65**    | **94.16** |\n| $m=0.25$ |        53.80        |   56.09   |      82.92      |   93.75   |\n| $m=0.35$ |        52.22        |   55.61   |      82.38      |   92.72   |\n\n**Reference**  \n[1] Improving language understanding by generative pre-training, 2018.  \n[2] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Proceedings of naacL-HLT, 2019.  \n[3] RoBERTa: A Robustly Optimized BERT Pretraining Approach, arXiv e-prints, 2019.  \n[4] Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation, AAAI, 2022.\n\n"}]

Author rebuttal:

Meta-review: If the authors have access to the review and discussion text of reviewer DtxK, that will greatly help them revise the paper to highlight its novelty and technical strength *way* better than they have done in the submission itself.

The authors should do three more things to justify the high scores, otherwise the paper may have limited appeal.

First, they should discuss all these prior papers below (in rough order of priority) and justify the novelty of their ideas against the backdrop of these papers.

 * Random Walks and Neural Network Language Models on Knowledge Bases. Josu Goikoetxea, Aitor Soroa and Eneko Agirre. NAACL 2015. "Each time the random walk reaches a vertex, a word is emitted at random using the probabilities in the inverse dictionary. When the random walk terminates, the sequence of emitted words forms the pseudo sentence which is fed to the NNLM architecture."
 * [DOLORES: Deep Contextualized Knowledge Graph Embeddings](https://arxiv.org/pdf/1811.00147.pdf). Haoyu Wang, Vivek Kulkarni, William Yang Wang. AKBC 2020.  (See "path generator" and "learner".)
 * [Exploring Large Language Models for Knowledge Graph Completion](https://www.semanticscholar.org/paper/Exploring-Large-Language-Models-for-Knowledge-Graph-Yao-Peng/9acc39255945b895faf0354b51716b227da87ea7) Liang Yao, Jiazhen Peng, Chengsheng Mao, Yuan Luo.
 * [KGBERT] KG-BERT: BERT for Knowledge Graph Completion. Liang Yao, Chengsheng Mao, Yuan Luo. 2019. "Our method takes entity and relation descriptions of a triple as input and computes scoring function of the triple with the KG-BERT language model."
 * Plus, include the discussion of the papers pointed out by reviewer 4W5x.


Second, they should add experiments involving the following data sets that are standard in the knowledge graph community, and compare against state-of-the-art KG inference systems.  For an accepted paper, this might be a tall ask, but it is my duty.
 * FB15K, FB15K237
 * WN18, WN18RR
 * YAGO3-10

Reviewer DtxK is of the opinion that this is not needed, because "attributed graphs" ("with both node and edge attributes") are allegedly different from KGs.  I cannot understand this claim, because certainly a KG like Wikidata has diverse attributes, both textual (in many languages!) and often structured, associated with nodes and edges.  It is certainly possible and desirable to know how much pretrained corpus information can boost purely structural completion algorithms on the above KGs, also because highly reliable standard numbers are available for them, unlike the two data sets used by the authors.


Third, there needs to be more diagnostics and explanations for why WalkLM is performing better than the closest algorithms it is compared against.
Decision: Accept (poster)