Pretraining Language Models with Text-Attributed Heterogeneous Graphs
Authors: Tao Zou
Le Yu
Yifei Huang
Leilei Sun
Bowen Du
Venue: EMNLP 2023 Findings (2023)
Keywords: language model pretraining
text-attributed heterogeneous graphs
graph neural networks
TL;DR: we present a new pretraining framework for language models that explicitly considers the topological and heterogeneous information in text-attributed heterogeneous graphs.

Abstract: In many real-world scenarios (e.g., academic networks, social platforms), different types of entities are not only associated with texts but also connected by various relationships, which can be abstracted as Text-Attributed Heterogeneous Graphs (TAHGs). Current pretraining tasks for Language Models (LMs) primarily focus on separately learning the textual information of each entity and overlook the crucial aspect of capturing topological connections among entities in TAHGs. In this paper, we present a new pretraining framework for LMs that explicitly considers the topological and heterogeneous information in TAHGs. Firstly, we define a context graph as neighborhoods of a target node within specific orders and propose a topology-aware pretraining task to predict nodes involved in the context graph by jointly optimizing an LM and an auxiliary heterogeneous graph neural network. Secondly, based on the observation that some nodes are text-rich while others have little text, we devise a text augmentation strategy to enrich textless nodes with their neighbors' texts for handling the imbalance issue. We conduct link prediction and node classification tasks on three datasets from various domains. Experimental results demonstrate the superiority of our approach over existing methods and the rationality of each design. Our code is available at https://github.com/Hope-Rita/THLM.

Pretraining Language Models with Text-Attributed Heterogeneous Graphs
Tao Zou'; ; Le Yu'; Yifei Huang', LeiLei Sun If and Bowen Du 1,233 Du1.23 SKLSDE Lab, Beihang University, Beijing, China 2 Zhongguancun Laboratory, Beijing, China
3 School of Transportation Science and Engineering, Beihang University, Beijing, China fzoutao, yule, yifeihuang, leileisun, dubowen } @ buaa.edu.cn
Abstract
masked language modeling (Devlin al., 2019), next-token prediction (Radford al., 2018), autore- gressive blank infilling (Du et al., 2022)), PLMs can learn general contextual representations from texts in the large-scale unlabelled corpus.
In many real-world scenarios (e.g., academic networks, social platforms), different types of entities are not only associated with texts but also connected by various relationships, which can be abstracted as Text-Attributed Heteroge- neous Graphs (TAHGs). Current pretraining tasks for Language Models (LMs) primarily focus separately learning the textual infor- mation of each entity and overlook the cru- cial aspect capturing topological connections among entities in TAHGs. In this paper, present new pretraining framework for LMs that explicitly considers the topological and heterogeneous information in TAHGs. Firstly, we define a context graph as neighborhoods of a target node within specific orders and pro- pose topology-aware pretraining task to pre- dict nodes involved in the context graph by jointly optimizing LM and auxiliary het- erogeneous graph neural network. Secondly, based on the observation that some nodes are text-rich while others have little text, de- vise text augmentation strategy to enrich text- less nodes with their neighbors' texts for han- dling the imbalance issue. We conduct link pre- diction and node classification tasks on three datasets from various domains. Experimen- tal results demonstrate the superiority of our ratio-
(ndonch): TIUL Abstract
Grugh Trendhmme Droe
first-ortler neightors
Koyword (setine):
Gmpt petroris
higl-ordss
Author (aciam)) Nans
maightors
Jurt Ledenve
Figure 1: As an instance of TAHG, academic net- work contains three types of nodes (papers, authors, and keywords) with textual descriptions as well as their multi-relational connections.
In fact, texts not only carry semantic information but also are correlated with each other, which could be well represented by Text-Attributed Heteroge- neous Graphs (TAHGs) that include multi-typed nodes with textual descriptions as well as relations. See Figure 1 for example. Generally, TAHGs usually exhibit the following two challenges that are struggled to be handled by existing PLMs. Abundant Topological Information (CI). Both first- and higher-order connections exist in TAHGs and can reflect rich relationships. For instance, a apa- per can be linked to its references via first-order ci- tations and can also be correlated with other papers through high-order co-authorships. However, the commonly used pretraining tasks (Radford et al., 2018; 8; Devlin al., 2019; Du et al., 2022) just learn from texts independently and thus ignore the con- nections among different texts. Although some recent works have attempted to make PLMs aware of graph topology (Yasunaga et al., 2022: Chien et al., 2022), they only consider first-order relation- ships and fail to handle higher-order signals. Imbalanced Textual Descriptions of Nodes (C2). In TA TAHGs, nodes are heterogeneous and their car- ried texts are often in different magnitudes. For example, papers are described by both titles and
approach over existing methods nality of each design. Our code available https:// / tht.. cww/Hope-etttteeeeoooooe Rita/THLM.
1 Introduction
Pretrained Language Models (PLMs) (Devlin et al., 2019; Yang et al., 2019; Brown al., 2020; Lan al., 2020) that built upon the Transformer archi- tecture (Vaswani et al., 2017) have been success- fully applied in various downstream tasks such as automatic knowledge base construction (Bosselut et al., 2019) and machine translation (Herzig et al., 2020). Due to the design of pretraining tasks (e.g..
"Equal Contribution Corresponding Author

---

abstracts (rich-text nodes), while authors and key- words only have names or brief terms (textless nodes). Currently, how to pretrain LMs to com- prehensively capture the above characteristics of TAHGs still remains an open question.
2 Preliminaries
A Pretrained Language Model (PLM) can map an input sequence X (2,,,2, .1,) of L to- kens into their contextual representations H (h1, h2, h2) with the design of pretraining tasks like masked language modeling (Devlin et al., 2019), next-token prediction (Radford et al., 201 8), autoregressive blank infilling (Du al., 2022). this work, we mainly focus on the encoder-only PLMs (c.g.. BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019)) and leave the explorations of PLMs based on encoder-decoder or decoder-only architecture in the future.
In this paper, we propose new pretraining framework to integrate both Topological and Heterogeneous information in TAHGs into LMs, namely THLM. To address CI, we define a con- text graph as the neighborhoods of the central node within K orders and design a topology-awand pre- training task (context graph prediction) to predict neighbors in the context graph. To be specific, we first obtain the contextual representation of the cen- tral node by feeding its texts into an LM and com- pute the structural representation of nodes in the given TAHG by an auxiliary heterogeneous graph neural network. Then, we predict which nodes are involved in the context graph based on the repre- sentations, aiming to inject the multi-order topol- ogy learning ability of graph neural networks into LMs. To tackle C2, we devise text augmentation strategy, which enriches the semantics of textless nodes with their neighbors' texts and encodes the augmented texts by LMs. We conduct extensive ex- periments on three TAHGs from various domains to evaluate the model performance. Experimental results show that our approach could consistently outperform the state-of-the-ar on both link predic- tion and node classification tasks. We also provide an in-depth analysis of the context graph predic- tion pretraining task and text augmentation strategy. Our key contributions include:
A Text-Attributed Heterogeneous Graph (TAHG) (Shi et al., 2019) usually consists of multi- typed nodes as well as different kinds of relations that connect the nodes. Each node also associ- ated with textual descriptions of varying lengths. Mathematically, a TAHG can be represented by R, X),, where V. E, U and R de-
G = note the set of nodes, edges, node types, and edge types, respectively. Each node v E V belongs to type Φ(v) E U and each edge Cu,v has type R. X' is the set of textual descrip- tions of nodes. Note that TAHG should satisfy [ + |R| > 2.
(b(w,,)
Existing PLMs mainly focus on textual descrip- tions of each node separately, and thus fail to capture the correlations among different nodes in TAHGs (as explained in Section 1). To address this issue, propose new framework for pre- training LMs with TAHGs, aiming to obtain PLMs that are aware of the graph topology well the heterogeneous information.
We investigate the problem of pretraining LMs on more complicated data structure, i.c., TAHGs. Unlike most PLMs that can only learn from the textual description of each node, we present a new pretraining framework to enable LMs to capture the topological con- nections among different nodes.
Methodology
Figure 2 shows the overall framework of our pro- posed approach, which mainly consists of two com- ponents: topology-aware pretraining task and text augmentation strategy. Given TAHG, the first module extracts the context graph for a target node and predicts which nodes are involved in the con- text graph by jointly optimizing an LM and an auxiliary heterogeneous graph neural network. It aims to enable PL.Ss to capture both first-order and high-order topological information in TAHGs. Since some nodes may have little textual descrip- tions in TAHGs, the second component is further introduced tackle the imbalanced textual descrip- tions of nodes, which enriches the semantics of textless nodes by neighbors' texts. It is worth notic-
We introduce a topology-awar pretraining task to predict nodes in the context graph of target node. This task jointly optimizes LM and an auxiliary heterogeneous graph neural network, enabling the LMs to leverage both first- and high-order signals.
We devise a text augmentation strategy to en- rich the semantics of textless nodes to mitigate the text-imbalanced problem.

---

ing that after the pretraining stage, we discard the auxiliary heterogencous graph neural network and only apply the PLM for various downstream tasks.
TAHG 9 and obtain the representations of all the nodes in V as follows,
H fHGNNN (9) PER. R
(1)
3.1
Topology-aware Pretraining Task
where (HGNN(() can be implemented by any ex- isting heterogeneous graph neural networks. d is the hidden dimension. Then, we encode the textual description of target node u by an LM and derive its semantic representation by
To tackle the drawback that most existing PLMs cannot capture the connections between nodes with textual descriptions, some recent works have been proposed (Yasunaga et al., 2022; Chien et al., 2022). Although insightful, these methods solely focus on the modeling of first-order connections between nodes while ignoring high-order signals, which are proved to be essential in fields like network analysis (Grover and Leskovec, 2016; Cui et al., 2019), graph learning (Kipf and Welling, 2017; Hamilton et al., 2017) and recommender system (Wang al., 2019; He et al., 2020). To this end, we propose a topology-aware pretraining task (namely, context graph prediction) for helping LMs capture multi-order connections among different nodes.
14 MEAN (Xx)) E R
(2)
where fLM(() can be realized by the existing LMs. Besides, to capture the heterogeneity of node u, we introduce projection header in the last layer of the PLM. X, denotes the textual descriptions of node u. Next, we predict the probability that node is involved in the context graph 9 of via a binary classification task
Úv, sigmoid (NIM T W Wo() H.!
(3)
Context Graph Extraction. We first illus- trate the definition of the context graph of target node. Let N₂ be the set of first-order neighbors of node u in a given TAHG G U,, R. X). The context graph 9 of node is composed of neighbors that u can reach within K orders (including node itself) as well as their connections, which is represented by G (OK EKK. vk {v'le' N, A a vk 1] U Dk-1 is the node set of g and EK {(w', v') Elvl E yK A E VK denotes the edge set of G.K is obvious that D = {u} and V = U [u]. Based on the definition, we can extract the context graph of node u based on the given TAHG G. Note that when K 2 2, the con- text graph G will contain multi-order correlations between nodes, which provides opportunity to capture such information by learning from GK
where Wo(r) E IRdxd is a trainable transform ma- trix for node type Φ(v) E R. The ground truth Yu,e = if g contains U, and 0 otherwise.
Pretraining Process. In this work, we use BERT (Devlin et al., 2019) and R-HGNN (Yu et al., 2022) to implement fLar(-) )-) and JHGNN() (-), respectively. Since it is intractable to predict the appearing prob- abilities of all the nodes adopt negative sampling (Mikolov al., 2013) jointly optimize fLM (-) and fHGNNN.).. To gener- atc positive samples, we uniformly sample neigh- bors from a specific relation during each hop. The negative ones are sampled from the remaining node set V DK with a negative sampling ratio of 5 (i.e., five negative samples per positive sample). In ad- dition the CGP task, we incorporate the widely used Masked Language Modeling (MLM) task help LMs better handle texts. The final objective function for each node u E V
V in Equation (3), we
Context Graph Prediction. TAHGs not only contain multiple types of nodes and relations but also involve textual descriptions of nodes. Instead of pretraining single texts like most PLMs do, we present the Context Graph Prediction (CGP) for pretraining LMs on TAHGs to capture the rich information. Since LMs have been shown to be powerful in modeling texts (Devlin et al., 2019; Brown et al., 2020), the objective of CGP is to inject the graph learning ability of graph neural networks (Bing et al., 2022) into LMs.
Cu M.M CCGP log P( P(X,(|Nuxx.)>
Σ log Du.e BEVK
E E.j.P. log (I-Bure)
(4)
where Xx, is the corrupted version of node u's original textual descriptions X,, with a 40% mask- ing rate following (Wettig et al., 2023). P. de- notes the normal noise distribution. Additionally, the input feature of each node for the auxiliary het- erogeneous graph neural network is initialized by
Specifically, we first utilize an auxiliary hetero- geneous graph neural network to encode the input

---

structural embeddings
g
PraPa (V (negative sample)
Hetenggneous Graph Neural Network
V
text-aitriuteed
(positive sample)
heterogensous graph.
loctlec greph extrection
lextless nodes
X
X Rp
backpropagnda gradients to (0 QRcKN)
Lu
CGP
Text A ugmentation Strategy
Rich tuxt nodes
Mi
Language Model
context graph 5%
LM input.
textual representations..i..
Figure 2: Framework of the proposed approach.
its semantic representation based on Equation (2) which is shown to be better than a randomly- initialized trainable feature in the experiments.
h with more semantics. We empirically find that text augmentation strategy can bring nontrivial improvements without significant increment of the model's complexity.
3.2 Text Augmentation Strategy
As discussed in Section 1, the textual descriptions of different types of nodes in TAHGs are varying with different lengths, resulting in rich-text nodes and textless nodes. The exhaustive descriptions of rich-text nodes can well reveal their characteristics, while the brief descriptions of textless nodes insufficient to reflect their semanties and solely en- coding such descriptions would lead to suboptimal performance. Therefore, we devise a text augmen- tation strategy to tackle the imbalance issue, which first enriches the semantics of textless nodes by combining the textual descriptions of their neigh- bors according to the connections in TAHGs and then computes the augmented texts by LMs.
3.3 Fine-tuning in Downstream Tasks
After the pretraining process, discard the auxiliary heterogeneous graph neural network SHGNN()) and solely apply the pretrained L.M fLM() (-) to generate the semantic representations of nodes based on Equation (2). We select two graph- related downstream tasks for evaluation including link prediction and node classification. We em- ploy various headers at the top of SLM(()) to make exhaustive comparisons, including MultiLayer Per- ceptron (MLP), RGCN (Schlichtkrull et al., 2018), HeISANN (Hong et al., 2020), and R- R-HGNN (Yu et al., 2022). For downstream tasks, fLM() (-) is frozen for efficiency and only the headers can be fine-tuned. Please refer to the Appendix A.2 for detailed descriptions of the headers.
To be specific, for rich-text node u, use its texts with special tokens (Devlin al., 2019) as the input My, which denoted as [CLS] X,, [SEP]. For textless node u, we concatenate its texts and & sampled neighbors' texts as the input Ma. i.e., [CLS] X [SEP] XN3 [SEP] ... [SEP] XN; [SEP], where N represents the i-th sam- pled neighbor of Furthermore, in the case of nodes lacking text information, we employ the con- catenation of text sequences from neighbors. This approach enables the generation of significant se- mantic representations for such nodes, effectively addressing the issue of text imbalance. After the augmentation of texts, change the input of Equa- tion (2) from X, to Mn. obtain representation
Experiments
4.1 Datasets and Baselines
Datasets. We conduct experiments on three real- world datasets from different domains, including the academic network (OAG-Venue (Hu al., 2020b)), book publication (GoodReads (Wan and McAuley, 2018: Wan et al., 2019)), and patent ap- plication (Patents²). All the datasets have raw texts on all types of nodes, whose detailed descriptions and statistics are shown in the Appendix A.1.
Compared Methods. We compare THLM with several baselines to generate the representations of nodes and feed them into the headers for down- stream tasks. In particular, select six methods
Note that the initialization executed only once by using the official checkpoint BERT (Devlin al., 2019).
'Among the neighbors in Nr. we select rich-text nodes in priority. Moreover, if the size of Nr. is smaller or equal to lt; we will choose all the neighbors.
Phttps: http::://ww..uppt..e. gwwto.goo/

---

compute the node representations: BERT (Devlin al., 2019) and RoBERTa (Liu et al., 2019) are widely used PLMs; MetaPath (Dong et al., 2017) is a representative method for heterogeneous net- work embedding; MetaPath-BBRRT combines the textual and structural information as the represen- tations, LinkBERT (Yasunaga et al., 2022) and GI- ANT (Chien et al., 2022) are first-order topology- aware PLMs. Besides, we apply OAG-BERT (Liu al., 2022) to compare the performance of OAG- Venue. Detailed information about baselines is shown in the Appendix A.1. It is worth noticing that LinkBERT and GIANT are designed for ho- mogenneus graphs instead of TAHGs. Hence, we maintain the 2-order connections among rich-text nodes and remove the textless nodes to build ho- mogeneous graphs for these two methods for eval- uation. See Appendix A.6 for more details.
4.3 Evaluation Tasks
Link Prediction. On OAG-Venue, GoodReads, and Patents, the predictions are between paper- author, book-publisher, and patent-company, re- spectively. We use RMSE and MAE as evalua- tion metrics, whose descriptions are shown in Ap- pendix A.3. Considering the large number edges on the datasets, we use sampling strategy for link prediction. Specifically, the ratio of the edges used for training, validation, and testing is 30% 10%, and 10% in all datasets. Each edge is sociated with five/one/one negative edge(s) in the trainingyvalidaaioo/lestgg stage.
Node Classification. We classify the cate- gory of papers, books, and patents in Venue, GoodReads, and Patents. We use Micro-Precision, Micro-Recall,
Macro-Recall and NDCG to evaluate the performance of different models. Descriptions of the five metries are shown in Appendix A.3. Each paper in OAG-Venue only belongs to one venue, which could be formalized as a multi-class classification problem. Each patent or each book is categorized into one or more labels, resulting in multi-label classification problems.
4.2 Experimental Settings
Following the official configuration of BERT buse (110M params, (Devlin et al., 2019)), limit the input length of the text to 512 tokens. For the context graph prediction task, the number of or- ders K in extracting context graphs searched in [1, 2,3, 3, 1). For the text augmentation strategy, we search the number of neighbors for concatena- tion in 3.... We load the weights in BERT base checkpoint released from Transformers tools' for initialization. For R-HGNN, we set the hidden di- mension of node representations and relation repre- sentations to 786 and 64, respectively. The number of attention heads is We use the two-layered R-HGNN in the experiments. To optimize THLM, we use AdamW (Loshchilov and Hutter, 2019) as the optimizer with (21,32)= (0.9, 0.999), weight decay 0.01. For BERT we warm up the learn- ing rate for the first 8,000 steps up to 6e-5, then linear decay it. For R-HGNN, the learning rate is set to le-4. We set the dropout rate (Srivastava et al., 2014) of BERT and R-HGNN to 0.1. We train for 80,000 steps, and batch sizes of 32, 48 and 64 sequences with 512 tokens for Venue, GoodReads, and Patents, and with maximize uti- lization while meeting the device constraints. The pretraining process took about three days on four GeForce RTX 3090 GPUs (24GB memory). For downstream tasks, please Appendix A.4 for detailed settings of various headers.
4.4 Performance Comparison
Due to space limitations, we present the perfor- mance on RMSE and MAE for link prediction, as well as Micro-Precision and Micro-Recall for node classification, in Table For the performance on Macro- Precisson,, Macro-Recall and NDCG on three datasets in the node classification task, please refer to Appendix A.5. From Table 1 and Appendix A.5, we have the following conclusions.
Firstly, except for MetaPath, BERT and RoBERTa exhibit relatively poorer performance in link prediction across three datasets compared to other baselines. This suggests that incorpo- rating the struetural information from the graph can greatly enhance the performance of down- stream link prediction tasks. Moreover, RoBERTa achieves notable performance in node classification when compared to other baselines. This implies that leveraging better linguistic representations can further improve the overall performance.
Secondly, we observe that MctaPath, which solely captures the network embeddings, performs the worst performance among the evaluated meth- ods. However, when MetaPath is combined with se- mantic information, it achieves comparable or even superior performance compared to RoBERTa. This

---

Table 1: Performance of different methods on three datasets in two downstream tasks. The best and second-bess performances are boldfaced and underlined. *: THLM significantly outperforms the best baseline with p-value
0.05
Link Prodiction
Noda Classificatin
Modell
RMSE
MAF
Micre-Prcissont
Minn-Reeal((e)
HESANN
RGUN
K-HONN
HRSAKN
ROUN
K-HONN
MEP
HRKANN
ROCN
HONN
MLF
BRISANN
KGLK
K-HONK
BERT
B.1987
0.2149
0,1802
0.0648
0,0886
0.9447
0.2257
0.314
0.3136
0.3473
012255
0.3146
0.3136
113473
RoBERTa
0.1911
0.2152
0.1689
0.8814
0,0400
0.2527
0.3193
0.3341
035115
0.2527
0.3193
0.3311
0.3516
MataPath
0.2199
0.2415
0.1946
0.8422
0.0972
0.0544
0.1132
0.2693
0.2851
0.3011
0.1132
0,2693
0.2851
0.5011
OAG-Neume
MetaPaib-BERT
0:2213
02149
0,1651
0.881
0.074
0.0377
0.2307
0.3311
0,3317
0.3122
0.2307
0.331
0.3317
0.342
LinkBERT
0.1867
0.2299
0,1729
0.0288
0.0892
0,0435
02278
0.100
03508
0.2278
03105
0.3155
GIANT
0.2045
0.0022
0.1200
0.730
0.0761
0,0408
0,2280
0.3116
0.8074
0.3274
0.2280
0.3116
0.3074
0.3274
DAG-BERT
0.1918
0.2030
0,1772
0.634
0.0744
00086
02577
0.3214
0,3152
0.3125
0.2577
033211
0.3152
0.3425
THIM
0.1857
0.1893
0.15V1
0.0614
0.0722'
0J03S2
0.263
0.340
0.33MM
0.3575
0.2637
0,340
1L33WW
0.3575
BERT
0.3424
0.1735
0.1108
0.0408
0.0556
0,0150
0.7274
0.8238
0.8240
0.33%
0.0984
0.7909
0,7911
0,8061
RobERTa
B.1349
0.1268
0.1044
0,0189
0.7069
0.941
0,79K2
0,809
MotaPath
0.1882
0,1760
0,1520
0.639
0.0639
0,9470
0,1192
0.6ESS
0.6179
0,683
0.1432
0.6190
0,6200
0,5008
GoodReads
MenaPabbBBRR
0.1314
0.1195
0,1403
0.0825
0,0280
0,0300
0.7240
08855
0.8320
0.339
0
0.7928
0.7983
0
LiokBERT
0.1362
0.1135
0,0212
0
0.793
0
GIANT
0.1323
0.1129
0,1089
0.3755
0.0271
0.0191
0,7580
0,8200
0,830
0.839]
0.7277
0.7921
0.7969
0,8057
THIM
0,1286
0.115V
0.1000'
0
0/6271'
0.0162"
0.7769
113399
IJ5437
0.849
0,7459
0,8022'
0U1144
0,8157'
BERT
0.3274
0.3135
0.2744
0.1935
0.182y
0.1334
0,6248
0.6910
0.6445
0.3791
0.4006
0,192
0,3912
RoBERTa
0.3149
0.2926
0.2585
0.1836
0.1545
0.1119
DA30I
0U6735
0,7022
0
0.3671
0.4087
0.4261
114238
MotaPuth
0.4816
0.1842
0.4842
0.3372
0.3352
0.3353
0.996
0.4385
0,4548
0.4651
0.1211
02660
0.799
0.2824
Potents
MetsPuth- BERT
0.2922
0.2840
0.2571
0.1483
0.1440
0.0%44
0,243
0,6583
0.6877
0.3788
0.3994
0.4175
0,4173
LinkBERT
0,5080
0.3033
0.2601
0.1805
0,1758
0.1142
0.604
06719
0.7018
0.7075
0. 0.5946
0.1095
0.1277
0.4293
GIANT
0:2734
0.4444
0,2276
0.1537
0.1288
0,0576
06508
0.6709
0.6939
0.3940
0.4071
0.4242
0,4210
THLM
0,2522
0,2513
0.31V0'
0.1333
0
0.0848
8.7066
0.715
0.7324
0.733
0,4287
0,4344
0.444''
0.4467'
highlights the importance of incorporating both structural information and textual representations for each node to enhance overall performance.
Finally, THLM significantly outperforms the ex- isting models due to: 1) integrating multi-order graph topology proximity into language models, which enables the model to capture a more com- prehensive understanding of the graph topology; 2) enhancing the semantic representations for text- less nodes via aggregating the neighbors' textual descriptions, that generates more informative rep- resentations for textless nodes.
Third, we note that LinkBERT and GIANT achieve superior results in the majority of met- rics for link prediction. This highlights the ad- vantage of learning textual representations that consider the graph structure. However, both GI- ANT and LinkBERT may not yield satisfactory re- sults in node classification on the DAG-Venue and GoodReads. This could be attributed to two rea- sons: 1) these models primarily focus on first-order graph topology while overlooking the importance of high-order structures, which are crucial in these scenarios; 2) these models are designed specifically for homogeneous graphs and do not consider the presence of multiple types of relations within the graph. Consequently, their effectiveness is limited in TAHGs and may impede their performance.
4.5 Analysis of Context Graph Prediction
To explore the impact of incorporating multi-order graph topology into language models, we conduct several experiments. These experiments aim to in- vestigate the effects of both first- and high-order topology information, as well as the model's ability to capture structural information using R-HGNN. For the remaining experiments on the analysis of different components like CGP and the text aug- mentation strategy, we intentionally removed the MLM task to isolate its effects in THLM, namely 'HLM* in Figure 3 and Table 2.
Moreover, OAG-BERT demonstrates competi- tive results in link prediction and strong perfor- mance in node classification, thanks to its ability to capture heterogeneity and topology during pretrain- ing. This can be attributed to its capability to learn the heterogeneity and topology of graphs. How- ever, it should be noted that OAG-BERT primar- ily captures correlations between papers and their metadata, such as authors and institutions, over- looking high-order structural information. These findings highlight the importance of considering both graph structure and high-order relationships when developing models for graph-based tasks.
Evaluation on Multi-order Topology Infor- mation. To assess the significance of multi-order neighbors' topology, we vary the number of or- ders in extracting the context graph from I to 4. The corresponding results are illustrated in Figure 3. Besides, to examine the impact of high-order neighbors, we solely predict the 2-order neighbors in the context graph prediction task, as indicated by w/ 2-order CGP in Table 2. From the results, it is evident that THLM

---

achieves superior performance when predicting multi-order neighbors compared solely predict- ing I-order or 2-order neighbors. This suggests that modeling both first- and high-order structures enables LMs to acquire more comprehensive graph topology. Additionally, we observe that THLM exhibits better results when K is 2 in context graph prediction. However, its performance gradually declines we predict neighbors in higher layers, potentially due to the reduced importance of topo- logical information in those higher-order layers.
From the obtained results, we deduce that both the initial features and effective HGNNs contribute significantly to capturing graph topology and em- bedding informative node representations effec- tively. Firstly, unlike MLP, which fails to capture the contextualized graph structure in the context graph prediction task, RGCN allows for the em- bedding of fine-grained graph structural informa- tion, which facilitates better learning of the graph topology. Furthermore, the utilization of effective HGNNs such as R-HGNN enables the embedding of expressive structural representations for nodes, Secondly, R R-HGNN demonstrates its superior abil- ity to learn more comprehensive graph structures from nodes compared to using randomly initialized features. These findings underscore the importance of integrating both semantic and structural infor- mation to learn informative node representations.
HeSANN
AGUN -B-HUNN
158
677-
8-5-
5.4-
0.7-
Periernezce on GoodReads
(b) Perforaazee
4.6 Analysis of Text Augmentation Strategy
Figure 3: Effects of learning multi-order topology infor-
mation in TAHGs on node classification.
Table 3: Results on the node classification task in evalu- ating the effectiveness of text augmentation strategy.
Table 2: Evaluation the ability to learn informative representations via R-HGNN node classification
Dataset
Methods
MLP
HctSANN
RGCN R-HGNN
Deighbors-only
0.2597
0.3274
0.3165
0,3495
textless-only
0.2625
0.3290
0.3044
0.3516
Datasets
GCP
HeaSANN
R-JIGNN
OAG-Venue
TeeiI-Neighhor)
0,2611
0.3349
0.3201
0.3507
w/ ML P
0,2591
0.3195
0.3043
0.3379
TAS(2-Ncighbor)
0,2627
0.3380
0.3217
0,3549
w/ RGCN
0.2728
0.3323
0.3220
0.3547
TAi(h-Ne)ghbor)
0,2629
0,3383
0.3228
0,3854
MMG-Nenue
w/ ardrer CGP
0,2609
0.3357
0.3121
0,3488
neighbors-only 0.4855
0.7278
0.7132
0,7624
w/ random feats
0,2602
0.3271
0.3133
0.3487
textless-only
0,7453
0.8351
0.8397
0.8436
THLM"
0,3383
0,3554
GoodReads
TAS(1 Neighhhr)
0.7480
0.8353
0.8421
0.8469
w/ MLP
0.7528
0.8352
0.8376
0.8445
TAS(2-Neighbor)
0.7547
0.8381
0.8426
0.8475
w/ RGCN
0,7608
0.8380
0.8411
0,8512
GoodReads
w/ 2-order CGP
0,7512
0,8319
0.5355
0.8451
Neigh-Neig
0.7549
0.8382
0.8425
0.8485
w/ random feats
0,7523
0.8384
0.8406
0.8483
neighbors-only 0.6971
0.7040
0.7228
0.7224
THLM"
0,7549
0.8382
0.8425
0,8485
textless only 0.6856
0.6923
0.7139
0.7164
w/ MLP
0.6903
0.6963
0.7201
0.7208
Patents
TAS( (eeig--eigh 0.6959
0.7004
0.7211
0.7221
RGCN
0.6911
0.6986
0.7218
TAS(2 IAS(2-Neighborr 0.6960
0.7050
0.7219
0.7233
Patents
w/ 2-order CGP
0.6827
0.6876
0.7057
0,2068
(AS((3-Neighhor) 0.6948
0.7050
0,7275
0.7281
w/ random Teats
0.6908
0.7001
0.7107
0.7198
THLM"
0.6948
0,7050
0.7275
0,7280
To explore the potential of enhancing semantic information for textless nodes through our text aug- mentation strategy, we design three experimental variants. Firstly, we remove the text sequences of textless nodes and solely rely on the texts of their neighbors as inputs, denoted as "neighbors-only" We set the number of neighbors k: as 3 for con- catenation. Secondly, we only use the original text descriptions of textless nodes to derive textual em- beddings, namely "textless-only". Additionally, we employ the text augmentation strategy by varying the number of neighbors for concatenation from 1 3, denoted as "TAS(I-Neighbon", "TAS(2- Neighbor)", and "TAS Neighbo-", respectively. For all variants, we focus exclusively on the con- text graph prediction task to isolate the effects of
Evaluation on Learning Informative Node Features of R-HGNN. In this work, we adopt one of the state-of-the-ar HGNNs, i.e., R-HGNN with pre-initialized semantic features on nodes to obtain node representations. To examine the importance of learning informative node representations and complex graph structure in R-HGNN, we conduct experiments using two variants. Firstly, we replace R-HGNN with an MLP encoder or an alternative HGNN framework, i.c., RGCN (Schlichtkrull et al., 2018) in this experiment, denoted w/ MLP and w/ RGCN respectively. Secondly, we substitute the semantic node features with randomly initialized trainable features, referred to as w/ random feats. The performance results are presented in Table 2.

---

other factors. Due to space limitations, we present the Patero-Precision(( @ 1) metric for node classifi- cation in the experiments. Similar trends could be observed across other metrics.
achieves the best performance when it employs both two pretraining tasks for training, Removing either of these tasks leads to a decrease the re- sults. In particular, the context graph prediction task significantly contributes to the overall perfor- mance, demonstrating the substantial benefits of incorporating graph topology into our LM. Addi- tionally, the masked language modeling task helps capture the semantics within texts better and fur- ther enhances the model performance. Besides, we find that THLM w/o MLM performs better than the original BERT on two datasets, which contributes to our text augmentation strategy for textless nodes. This enhancement allows for better connectivity between the brief terms of textless nodes and their neighboring text sequences, resulting in improved contextual understanding and representation in pre- training PLMs.
From Table 3, we observe that both neighbors and textless nodes themselves are capable of learn- ing the semantic information for textless nodes. However, relying solely on either of them may lead insufficient textual representations for nodes. Furthermore, it found that using texts from more neighbors can enhance the semantic quality of text- less nodes. Nevertheless, considering the limita- tions on the input sequence length of language mod- els, we observe that THLM achieves similar perfor- mance when the number of l: is increased beyond 2. Therefore, strike balance between performance and computational efficiency while accommodat- ing sequence length limitations, we choose for concatenation in the text augmentation strategy. To ensure the reliability of our findings, we conduct the task five times using different seeds ranging from O to 4. Remarkably, all obtained p-values are below 0.05, indicating statistical significance and confirming the accuracy improvement achieved by our text augmentation strategy.
5
Related work
5.1 Pretrained Language Models
The objective of PLMs is leam general represen- tations of texts from large and unlabeled corpora via pretraining tasks, which could be applied to a variety of downstrcam tasks. Pretraining tasks that most PLMs widely used include 1) masked lan- guage modeling in BERT (Devlin et al.. 2019) and RoBERTa (Liu et al., 2019); 2) next token predic- tion in GPT models (Radford et al., 2018; Brown et al., 2020); and 3) autoregressive blank infilling in GLM (Du et al., 2022). However, these tasks sep- arately focus on the modeling within single texts and ignore the correlation among multiple texts. Recently, several works have been proposed to capture the connections between different texts Levine et al. (2022); Chien et al. (2022); Yasunaga et al. (2022). For example, Chien et al. (2022) inte- grated the graph topology into LMs by predicting the connected neighbors of cach node. Yasunaga et al. (2022) designed the document relation pre- diction task to pretrain LMs, which aims to clas- sify the type of relation (contiguous, random, and linked) existing between two input text segments. Although insightful, these methods just consider the first-order connections between texts and can- not leverage high-order signals, which may lead to suboptimal performance. In this paper, we aim to present new pretraining framework for LMs to help them comprehensively capture multi-order relationships as well as heterogeneous information in a more complicated data structure, i.e., TAHGs.
4.7 Effects of Two Pretraining Tasks
To study the importance of two pretraining tasks for downstream tasks, we use two variants of THIM to conduct the experiments, and the performance is shown in Figure 4. Specifically, THLM w/o CGP removes the context graph prediction task, which does not predict the context neighbors for the input node. THLM w/o MLM reduces the masked language modeling task, which ignores the textual dependencies in the sentences and only predicts the multi-order graph topology in the pretraining process, i.e., by predicting the neighbors involved in the context graphs for input nodes.
TITL NAR OGP
THEM MEN
024
5013-
2833
615-
Uob-
031
0:4
NLP
BASAAN ROCK
B-EL2S
NLP
HOSANN
REON
R HIGNN
(3) Peeferrrnee PNleE
Figure 4: Importance of two pretraining tasks on the node classification task.
From Figure 4, we can conclude that THLM

---

5.2 Heterogeneeus Graph Learning
which jointly optimizes an auxiliary graph neural network and an LM to predict which nodes are involved the context graph. To handle imbal- anced textual descriptions different nodes, text augmentation strategy was introduced, which en- riches the semantics of textless nodes by combin- ing their neighbors' texts. Experimental results on three datasets showed that our approach could significantly and consistently outperform existing methods across two downstream tasks.
Graph Neural Networks (GNNs) (Kipf and Welling, 2017: Hamilton et al., 2017) have gained much progress in graph learning, which are extensively upplied in modeling graph-structure data. Recently, many researchers have attempted to extend GNNs heterogencous graphs (Zhang al., 2019; Fu et al., 2020; Hong et al., 2020; Yu et al., 2020; Hu et al., 2020b; Lv et al., 2021), I), which are powerful in handling different types of nodes and relations as well as the graph topological information. In this work, we aim to inject the graph learning ability of heterogeneous graph neural networks into PLMs via topology-aware pretraining task.
Limitations
In this work, we pretrained language models on TAHGs and evaluated the model performance on link prediction and node classification tasks. Al- though our approach yielded substantial improve- ments over baselines, there are still several promis- ing directions for further investigation. Firstly, we just focused on pretraining encoder-only onnly LMs, and is necessary to validate whether encoder-decoder or decoder-only LMs can also benefit from the proposed pretraining task. Secondly, more down- stream tasks that are related to texts (e.g.. retrieval and reranking) can be compared in the experiments. Thirdly, it is interesting to explore the pretraining of LMs larger scales on TAHGs.
5.3 Text-rich Network Mining
Many real-worid scenarios (academic networks, patent graphs) can be represented by text-rich net- works, where nodes are associated with rich text descriptions. Existing methods for text-rich net- work mining can be divided into two categories. The first branch designs the cascade architecture to learn the textual information by Transformer (Vaswani et al., 2017) and network topology by graph neural networks separately (Zhu et al., 2021; Li et al., 2021; Pang et al., 2022). Another group nests GNNs into LMs to collaboratively explore the textual and topological information (Yang et al., 2021; Jin et al., 2022, 2023a,b). However, these works either mainly focus on the homogeneous graph or modify the architecture of LMS by incor- porating extra components. For example, Heter- formers (Jin et al., 2023b) is developed for text- rich heterogeneous networks, which aims to embed nodes with rich text and their one-hop neighbors by leveraging the power of both LMs and GNNs during pretraining and downstream tasks. Different from these works, we learn about the more compli- cated TAHGs and employ auxiliary heterogeneous graph neural networks to assist LMs capturing the rich information in TAHGs. After the pretrain- ing, we discard the auxiliary networks and only apply the pretrained LMs for downstream tasks without changing their original architectures.
Acknowledgements
This work was supported by the National Nat- ural Science Foundation of China (51991395, 62272023), and the Fundamental Research Funds for the Central Universities (No. YWF-23-L-717, No. YWF-23-L-1203).
References
Rui Bing, Guan Yuan, Mu Zhu, Fanrong Meng. Huifang Ma. and Shaojie Qiao. 2022. Heterogeneous graph neural networks analysis: a survey of techniques. evaluations and applications. Artificial Intelligence Review, pages 1-40.
Antoine Bosselut, Hannah Rashkin. Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for auto- matic knowledge graph construction. In ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4762-4779, Association for Com- putational Linguistics.
6 Conclusion
In this paper, we pretrained language models on more complicated text-atributed heterogeneous graphs, instead of plain texts. We proposed the context graph prediction task to inject the graph learning ability of graph neural networks into LMs,
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voo Voss, Gretchen Krueger, Tom Henighan, Rewon Child,

---

Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurlPS 2020, December 6-12, 2020, virtual.
Huiting Hong, Hantao Guo, Yucheng Lin, Xiaoqing Yang. Zang Li, and Jieping Ye. 2020. An attention- based graph neural network for heterogeneous struc- tural learning. In AAAI, pages 4132-4139, AAAI Press.
Weihua Hu, Matthias Fey, Marinka Zimik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020a. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual.
Eli Chien, Wei-Cheng Chang, Cho-Jui Hsich, Hsiang- Fu Yu. Jiong Zhang, Olgica Milenkovic, and Inder- jit S. Dhillon. 2022. Node feature extraction by self- supervised multi-scale neighborhood prediction. In ICLR 2022, Viriual Event, April 25-29, 2022.
Peng Cui, Xiao Wang, Jian Pei, and Wenwu Zhu. 2019. A survey on network embedding. IEEE Trans. Knowl. Data Eng., (1(3)8833-3)) 852.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020b. Heterogeneous graph transformer. In WWW '20, Taipei. Taiwan, April 20-24, 2020, pages 2704-2710. ACM/ IW3C2.
Jacob Devlin, Ming- Hiig-Weii&e Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019. Volume (Long and Short Papers), pages 4171-4186. Association for Compu- tational Linguisties.
Bowen Jin, Yu Zhang, Yu Meng, and Jiawei Han. 2023a. Edgeformers: Graph-empowered transformers for representation learning on textual-edge networks. CoRR, abs/2302. 10050.
Bowen Jin, Yu Zhang, Qi Zhu, and Jiawei Han. 2022. Heterformer: A transformer architecture for node representation learning on heterogeneous text-rich networks. CoRR, abs/2205. 10282..
Yuxiao Dong, Nitesh V. Chawla, and Ananthram Swami. 2017. metapath2vec: Scalable representation learn- ing for heterogeneous networks. In KDD, Halifax, NS, Canada, August 13 17, 2017, pages 135 144. ACM.
Bowen Jin, Yu Zhang, Qi Zhu, and Jiawei Han. 2023b. Heterformer: Transformer- basedd deep node represen- tation learning on heterogenoous text-rich networks. In KDD 2023, Long Beach. CA, USA, August 6-10, 2023, pages 1020-1031. ACM.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: general language model pretraining with autoregres- sive blank infilling. In ACI. 2022, Dublin, Ireland, May 22-27, 2022. pages 320 335. Association for Computational Linguistics.
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. ICLR 2017, Toulon, France, April 24- 26, 2017, Conference Track Proceedings. OpenRe- view.net.
Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. 2020. MAGNN: metapath aggregated graph neural network for heterogeneous graph embedding. In WWW '20 The Web Conference, pages 2331-2341. ACM/ II3C2.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sorieut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Aditya Grover and Jure Leskovee. 2016. node2vec: Scalable feature learning for networks. In KDD, San Francisco, CA, USA, August 13-17, 2016, pages 855- 864. ACM.
Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, and Amnon Shashua. 2022. The inductive bias of in-context learning: Rethinking pretraining example design. In ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning large graphs. In NIPS 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1024-1034.
Chaozhuo Li, Bochen Pang, Yuming Liu, Hao Sun, Zheng Liu, Xing Xie, Tianqi Yang. Yanling Cui, Liangjie Zhang, and Qi Zhang. 2021. Adsgnn: Behavior-graph augmented relevance modeling sponsored search. In SIGIR '21, Virtual Event, Canada, July 11-15, 2021, pages 223 232. ACM.
Xiangnan He, Kuan Deng, Xiang Wang. Yan Li. Yong- Dong Zhang, and Meng Wang. 2020. Lightgen: Sim- plifying and powering graph convolution network for recommendation. In SIGIR, pages
ACM.
Xiao Liu, Da Yin, Jingnan Zheng, Xingjian Zhang, Peng Zhang, Hongxia Yang. Yuxiao Dong, and Jie Tang. 2022. OAG-BERT: towards unified backhone lan- guage model for academic knowledge services. In KDD '22 Washington. DC. USA, August 14 18, 2022, pages 3418-3428. ACM.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisen- schlos. 2020. Tapas: Weakly supervised table pars- ing via pre-training. ACL 2020, pages 3320-4333, Association for Computational Linguistics.

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907..1692.
Mengting Wan, Rishabh Misra, Ndapa Nakashole, and Julian J. McAuley. 2019. Fine-grie line-grained spoiler detec- tion from large-scale review corpora. In ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 2605-2610. Association for Com- putational Linguistics.
Nlya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural graph collaborative filtering. In SIGIR, pages 165-174. ACM
Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng. Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, and Jie Tang. 2021. Are really making much progress?: Revisiting. bench- marking and refining heterogeneous graph neural net- works. In KDD 21, Virtual Event, Singapore, August 14-18, 2021, pages 1 1150-1160, ACM.
A Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023. Should you mask 15% in masked language modeling? In EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 2977-2992, Associa- tion for Computational Linguistics.
Junhan Yang. Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. 2021. Graphform- ers: Gnn-nested transformers for representation leamn ing on textual graph. NeurlPS 2021, December 6-14, 2021, virtual, pages 28798-28810.
Tomás Mikolov, Ila Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed repre- sentations of words and phrases and their composi- tionality. In NIPS 2013. Lake Tahoe, Nevada, United States, pages 31 191199
Bochen Pang. Chaozhuo Li, Yuming Liu, Jianxun Lian, Jianan Zhao, Hao Sun. Weiwei Deng, Xing Xie, and Qi Zhang. 2022. Improving relevance modeling via heterogeneous behavior graph learning in bing ads. In KDD '22 Washington, DC, USA, August 14- 18. 2022, pages 3713-3721. ACM.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime G. Car- bonell, Ruslan Salakhutdinov. and Quoe V. Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding, In NeurlPS 2019,, pages 5754- 5764.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. Linkbert: Pretraining language models with document links. In ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8003-8016 Association for Com- putational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training,
Michael Sejr Schlichtkrull, Thomas N. Kipf. Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In ESWC 2018, volume 10843 of Lecture Notes in Computer Science, pages 593-607. Springer.
Le Yu, Leilei Sun, Bowen Du, Chuanren Liu, Weifeng Lv. and Hui Xiong. 2020. Hybrid micro/macro level convolution for heterogeneous graph learning. CoRR, abs/2012.14722.
Yu Shi, Jiaming Shen, Yuchen Li, Naijing Zhang, Xin- wei He, Zhengzhi Lou, Qi Zhu, Matthew Walker, Myunghwan Kim, and Jiawei Han. 2019. Discover- ing hypernymy in text-rich heterogeneous informa- tion network by exploiting context granularity. In CIKM 2019, Beijing, China, November 3-7, 2019, pages 599-608. ACM.
Le Yu, Leilei Sun, Bowen Du, Chuanren Liu, Weifeng Lv, and Hui Xiong. 2022. Heterogeneous graph rep- resentation learning with relation awareness. IEEE Transactions on Knowledge and Data Engineering.
Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V. Chawla. 2019. Heterogeneous graph neural network. In KDD '19, pages 793-803 ACM.
Nitish Srivastava, Geoffrey F. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929- 1958.
Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Tianqi Yang. Liangjie Zhang, Ruofei Zhang, and Huasha Zhao. 2021. Textgnn: Improving text encoder via graph neural network in sponsored search. In WWW '21 Ljubljana, Slovenia, April 19- 23, 2021, pages 2848 2857, ACM/ IW3C2.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Mengting Wan and Julian J. McAuley, 2018. Item rec- ommendation on monotonic hehavior chains. In Proceedings of the 12th ACM Conference Rec- ommender Systems, RecSys 2018, Vancouver. BC, Canada, October 2-7, 2018, pages 86-94. ACM.

---

A Appendix
predicting the relation between two segments on Wikipedia and BookCorpus. GIANT (Chien al., 2022) extracts graph-aware node embeddings from raw text data via neighborhood prediction in the graph. OAG-BERT (Liu et al., 2022) is a pre- trained language model specialized in academic knowledge services, allowing for the incorporation of heterogeneous entities such as authors, institu- tions, and keywords into paper embeddings.
A.1 Datasets and Baselines
Datasets. Specific statistics of datasets are shown in Table 4 and detailed descriptions of datasets are shown as follows.
OAG- OAG-Venue' a heteroge- ncous graph followed by Hu et al. (2020b), which includes papers (P), authors (A), fields (F) and institutions (I). Each paper is pub- lished in single venue. We treat papers as rich-text nodes and extract the title and ab- stract parts as their text descriptions. Authors, fields, and institutions are regarded textless nodes, whose text descriptions are composed of their definitions or names.
A.2 Headers in Downstream Tasks
We apply four methods on downstream tasks, which could be shown as follows,
MLP relies exclusively on node features as input and uses the multilayer perceptron for prediction, which does not consider the graph information.
Following (Wan and McAuley, 2018; Wan al., 2019), we receive subset of GoodReads which contains books (B), authors (A) and publishers (P). Each book is categorized into one or more genres. We treat books as rich-text nodes and extract brief in- troductions as their text descriptions. Authors and publishers are regarded textless nodes, whose text descriptions their names.
ReadRe
RGCN incorporates the different relation- ships among nodes by using transformation matrices respectively the knowledge graphs (Schlichtkrull et al., 2018).
HetSANN aggregates different types of re- lations information from neighbors with a type-aware attention mechanism (Hong et al., 2020).
Patents: Patents is a heterogeneous graph collected from the USPTO?, which contains patent documents (P), applicants (A) and applied companies (C). Each patent is as- signed several International Patent Classifica- tion (IPC) codes. We treat patents as rich-text nodes and extract the title and abstract parts as their text descriptions. Applicants and com- panies use their names as text descriptions, regarded as textless nodes.
R-HGNN learns the relation-aware node rep- resentation by integrating tinengraeedd repre- sentation on each set of nodes within separate relations, and semantic representations across different relations (Yu et al., 2022).
A.3 Evaluation Metries
Baselines. We compare our model with the fol- lowing baselines: BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) are popular encoder- only pretraining language models. MetaPath (Dong et al., 2017) leverages meta-path-based random walks in the heterogeneous graph to generate node embeddings. MeRRPath+BERT combines the tex- tual embeddings embedded from BERT base and structural representations learned from MetaPath as node features. LinkBERT (Yasunaga et al., 2022) captures the dependencies across documents by
Seven metrics are adopted to comprchensively eval- uate the performance of different models in link prediction and node classification. In link predic- tion, use Root Mean Square Error (RMSE) and Mean Absolute Error (MAB) metrics. In node clas- sification, we use Precisore,, Prero-Precision, Macro- Recall, and Normalized Discounted Cumulative Gain (NDCG) metrics for evaluation. Details of the metries are shown below. RMSE evaluates the predicted ability for truth values, which calculates the error between predic- tion results and truth values. Given the prediction for all examples 9 = {Ū1, /2, .- Ím]. and the truth data y = {2r, y2, ). we calculate the
hnttps://githbb http://ggghuub.cmo//LLA-A-MM COm/UULLADNMpDHGTT
ucsdbookgraph/hon con/eng. eddu
"https://sites wttp//li..e...oglee...ee. cone.coo/eeeeeeeeeeeeeeeeeeeeeee
phttps://ww/ uww.w..ooooooooo

---

Table 4: Statistics the datascts.
Average Text Length
Classification Split Sets
Link Prediction Split Sets
Datasets
Nodes
Edges
Category
# Paper (P): 167,014
# P-F: 1,709,601
P: 243.497
Train: 106,724
Train: 144,030
# Author (A): 511,122
# P-P: 864,019
A: 5.667
OAG-Venue
242
Validation: 24,433
Validation: 48,010
# Field (F); 45,775
# A-I: 614,161
F: 3.690
Test: 35,847
Test: 48,010
# Institution (I): 9,090
P-A: 480,104
I: 5.882
# Book (B): 364,115
B: 163.577
Train: 254,880
Train: 139,988
B-A: 572,654
GoodReads
# Author (A): 154,418
A: 4,100
8
Validation: 54,617
Validation: 46,662
B-P: 466,626
Publisher (P): 40,135
P: 5.120
Test: 54,618
Test: 46,662
# Patent (P): 363,528
P: 139.436
Train: 254,469
Train: 110,277
P-C:: 367,598
Patents
Applicant (A): 182,561
A: 6,418
565
Validation: 54,529
Validation: 36,759
# P-A: 334,906
# Company (C): 1,000
C: 8,436
Test: 54,530
Test: 36,759
total RMSE as follows,
where TP(c)), FN(C.) is the number of true posi- tives, and false negatives for class ej respectively. Macro-averaged precision reflects the average ability to recognize the relevant elements rather than irrelevant ones in each class. We select the top-K probability predicted labels as predictions. Hence, Macro-Precision @ K is calculated by aver- aging all the precision values of all classes,
m
RMSE(
m ( y:)
MAE measures the absolute errors between pre- dictions and truth values. Given the prediction for all examples أ = {ūr, D2, ... 3m) ). and the truth data = (11, 32, 3m we calculate the total MAE as follows,
Macro-Precision @K Ecjec P(Š,S.a)
|C|
m 3//-
MAE(Ú. 9)
1
where S, S represents the predicted values and truth labels in the datasets, P(Š, S, c;) is the preci- sion value of class G₁.
m7
1=1
Micro-averagee precision measures the ability that recognizes more relevant elements than irrel- evant ones in all classes. We select the top-K pre- dicted labels as predictions for each sample. Hence, Peero-Preccsion@ee is the proportion positive predictions that are correct over all classes, which is calculated by.
Macro-averaged recall evaluates the average abil- ity to select all the relevant elements in each class. We select the top-K probability predicted labels as predictions. Hence, Macro-Recall @ K is calculated by averaging all the recall values of all classes,
Macro-Recall EacoR(Š, R(S,
Egec TP(cs) EGec ( FP(c))
|C|
Micro-Precision@ @K-
where S, represents the predicted values and truth labels in the datasets, R(S, c;) is the recall value of class c..
where TP(c)), FP(c)) is the number of true posi- tives, and false positives for class e; respectively. Micro-averaged recall evaluates the model's abil- ity in selecting all the relevant elements in all classes. We select the top-K probability predicted labels as predictions for each sample. Hence, Micro-Recall@1 is the proportion of positive la- bels that are correctly predicted over all classes. which is calculated by,
NDCG measures the ranking quality by consid- ering the orders of all labels. For each sample pi. NDCG is calculated by
S0 S.)) ΣK
[ogg(K
NDCG@K(p.)
Emin
minn(K
(I+T)
where S/ denotes the k-th predicted label of ex-
ample Pj- 8 (v, is when element is set S, otherwise O. We calculate the average NDCG all examples as a metric.
Egec TP(c)
Micro-Recall@K
Enccc TP(c:) FN(c.)

---

A.4 Detailed Settings in Downstream Tasks
Table 5: Performance on node classification in LinkBERT and GIANT.
In downstream tasks, we search the hidden di- mension of node representation for headers in [32, 64, 128, 256, 512]. For methods that atten- tion mechanisms, (i.c., HeISANN and R-HGNN), the number of attention heads is searched in [1,2, 8, 16). The training process is following R-HGNN (Yu et al., 2022).
Miero-Precisione I)
Datasets
Model
MLP
HeiSANN RGCN R-HGNN
LinkBERT1 order
0.6790
0.8100
0.8044
0.8302
GoodReads
-onder)
0.6967
0,8247
0.8284
0.8398
THLM
0.7769
0.8399
0.8437
0.84%
LinkBERT(I
0.5972
0.6421
0.6773
0,6734
Patents
GIANT(I-order
0.4793
0.6234
0.6323
0.6391
THLM
0.7066
0.7159
0,7324
0.7363
A.5 Detailed Experimental Results
textless nodes always lack sufficient textual content, leading to scarce semantic information. Hence, pre- dicting relationships between textless nodes and their neighbors becomes challenging for language models. 2) Apart from first-order neighbors, high- order neighbors provide more complex structure in- formation within the graph. By considering the lationships beyond the immediate neighbors, LMs could capture the graph topology across nodes more effectively and comprehensively. These find- ings highlight the importance of considering both first-order and high-order structure information in TAHGs and addressing the challenges of limited semantics on textless nodes. By tackling both prob- lems, our model can learn better in TAHGs.
We show the Macro-Prccision(C @ 1) and Macro- Recall( @1) 1) in the node classification task on three datasets in Table 7. Since the values of NDCG @ I) are the same Micro-Precision @ 1), we do not show duplicate results. Besides, since node classi- fication tasks on GoodReads and Patents belong to multi-label node classification, we show the perfor- mance on five metries when K is and 5 in Table 8 and Table 9 respectively.
A.6 LinkBERT & GIANT
In our baselines, LinkBERT and GIANT are specif- ically designed for homogeneous text-attributed graphs, which cannot be directly applied in TAHGs. To address this, convert the TAHGs into homo- geneous graphs that contain the set of rich-text nodes and their connections to ensure that all nodes contain rich semantic information in the graphs.
A.7 Effect of Distinguishing Treasured Structural Information
MI
For Patents and GoodReads, we extract the 2-order relationships in the graph and discard the textless nodes along with their relative edges to construct the homogeneous graphs. In the case of the OAG- Venue dataset, due to the high density of the second- order graph. we choose to construct a homogencous graph using a subset of crucial meta-path informa- tion to save the graph topology as much as possible, Inspired by Yu et al. (2022). we utilize the meta- path "P-F-P" (Paper-FielddPappe)) and the direct relation "P-P" (Paper-Paper) to build the homoge- neous graph for conducting experiments.
Pootaic
67-
0.11
16.
0.-
372
04-
Periomenee Goodlives
(b Petfenazce on Pateess
Figure 5: Precision node classification with different numbers in sampling negative candidates in the pretrain- ing process.
We investigate the effect of treasured structural information in the TAHGs. Specifically, we solely change the number of negative candidates for each positive entity in the context graph prediction task in [1, 3, 5, 7] in the pretraining stage. We present the performance of GoodReads and Patents on the @ 1) metric in the node classifica-
In addition previous experiments, we con- ducted another experiment to capture the first-order information the TAHGs while preserving the graph topology as much as possible. Specifically, we discard the hetcrogeneity of nodes and relation- ships in the graph to build a homogeneous graph, and the results are shown in Table 5.
tion task.
From Table 5, it is evident that pretraining LinkBERT and GIANT on TAHGs solely for order prediction may not yield optimal results. There are two key reasons for this observation: I)
From Figure 5, we could observe that the per- formance with smaller number or larger number in sampling negative candidates would be worse. This observation can be explained by two factors.

---

Firstly, the model may receive limited structural in- formation when selecting smaller number of nega- tive candidates, which hampers the model's ability to understand the underlying topology structure effectively. Secondly, sampling a larger number of negative candidates may bring noise topologi- cal information and make it difficult to distinguish meaningful patterns and relationships. Hence, the optimal performance is achieved when the num- ber of sampled negative candidates falls within proper range. By striking a balance between learn- ing sufficient topological information and avoiding excessive noise, the model can effectively capture the graph structure and achieve better performance in downstream tasks.
A.8 Performance on Large-scale Datasets
In our evaluation, we further test THLM on large- scale datasets (i.e., obgn-mag dataset (Hu et al., 2020a)) for the node classification task. The per- formance is shown in Table 6. We observe that THLM demonstrates scalability to larger datasets, outperforming baselines such as LinkBERT and GIANT. This outcome highlights the effectiveness of THLM, particularly its superior performance on the obgn-mag dataset.
Table 6: The accuracy results for node classification on the obgn-mag dataset.
Model MLP
HetSANN
RGCN
BERT 0.3754
0.5298
0.5484
RoBERTa
0.3770
0.5300
0.5490
LinkBERT*
0.3775
0.5230
0.5491
GIANT" 0.3903
0.5184
0.5256
THLM
0.3933
0.5353
0.5517

---

Table 7: Performance of different methods on three datasets in node classification. The best and second-best
performances boldfaced and underlined.
(oesooPreccisoo(( @ I)
Macro-Recall(CG 1)
Datasets
Model
MLP
HctSANN RGCN R-HGNN
MLP
HetSANN
RGCN R-HGNN
BERT
0.2110
0.3104
0.3119
0.3359
0.1992
0.3118
0.3060
0.3415
RoBERTa
0.2429
0.3264
0.3258
0.3598
0.2387
0.3187
0.3169
0.3412
MetaPath
0.0959
0.2593
0.2830
0.3005
0.0717
0.2731
0.2663
0.3019
OAG- Ge-uu
MetaPath+BERT
0.2094
0.3202
0.3248
0,3363
0.1991
0.3150
0.3180
0.3368
LinkBERT
0.2054
0.2921
0.3014
0.3479
0.2060
0.3057
0,2902
0.3233
GIANT*
0.2026
0.3078
0.3080
0.3381
0.2005
0.3097
0.2858
0.3188
THLM
0.2506
0.3375
0.3408
0.3562
0.2464
0.3330
0.3331
0.3537
BERT
0.7352
0.8273
0.8253
0,8421
0.7040
0.7960
0.7969
0,8112
RoBERTa
0.7420
0.8290
0.8328
0,8428
0.7134
0.7994
0.8039
0.8120
MetaPath
0.1786
0.6599
0.6560
0.6966
0.1371
0.6204
0.6225
0.6624
GoodReads
BeRT
0.7285
0.8286
0.8356
0.8425
0.7015
0.7978
0.8026
0.8104
LinkBERT
0.7178
0.8239
0.8276
0.8389
0.6917
0.7932
0.7987
0.8091
GIANT
0.7622
0.8273
0.8329
0.8418
0.7331
0.7970
0.8018
0.8109
THLM
0.7798
0.8472
0.8493
0.8515
0.7516
0.8148
0.8184
0.8209
BERT 0.3526
0.3876
0.4073
0,2994
0.1587
0.1864
0.1795
0.1335
RoBERTa
0,3262
0.3918
0.4185
0.4227
0.1506
0.1941
0.1801
0.1846
MetaPath
0.0854
0.1894
0.1862
0.2059
0.0153
0.0941
0.0930
0.0946
Patents
MetaPath+BERT
0.3330
0.3827
0.4072
0.4263
0.1577
0.1866
0.1842
091929
LinkBERT
0.3458
0.3838
0.4182
0.4515
0.1649
0.1858
0. 1884
0. 1920
GIANT*
0.3506
0.3904
0.4194
0.4327
0.1764
0.1995
0.1928
0.1944
THLM
0.4374
0.4364
0.4466
0.4974
0.2090
0.2128
0.2115
0.2281

---

Table 8: Performance of different methods GoodReads in node classification. performances are boldfaced and underlined.
The best and second-best
K=3
K-5
Metric
Model
MLP
HetSANN
RGCN
R-HGNN
MLP
HcIANNN
RGCN
R-HGNN
BERT
0.3402
0.3645
0.3520
0.3637
0.2146
0.2193
0.2095
0.2136
RoBERTa
0.3418
0.3602
0.3552
0.3707
0.2145
0,2174
0.2104
0.2166
MetaPath
0.1463
0.3374
0.3283
0.3417
0.1415
0.2187
0.2107
0.2142
Macro-Precision MetaPath+BBET
0.3377
0.3676
0.3605
0.3749
0.2138
0.2202
0.2137
0.2182
LinkBERT*
0.3350
0.3688
0.3543
0.3647
0.2118
0.2209
0.2114
0.2133
GIANT
0.3526
0.3671
0.3609
0.3702
0.2186
0.2187
0.2146
0.2189
THLM
0.3458
0.3753
0.3647
0.3717
0.2139
0,2232
0.2125
0.2136
BERT
0.9368
0.9766
0.9755
0.9804
0.9814
0.9941
0.9935
0.9947
RoBERTa
0.9431
0.9791
0.9792
0.9819
0.9851
0.9948
0.9945
0.9952
MetaPath
0,3950
0,8608
0.8585
0.8863
0.6461
0.9445
0,9395
0.9554
Macro-Recall
MetaPath+BBRT
0.9357
0.9762
0.9788
0,9803
0.9806
0,9939
0.9949
0.9950
LinkBERT*
0.9283
0.9756
0.9760
0.9785
0.9768
0.9938
0.9935
0.9942
GIANT
0.9502
0.9766
0.9775
0.9803
0.9862
0.9947
0.9945
0.9951
THIM
0.9615
0.9829
0.9846
0.9836
0.9899
0.9961
0.9959
0.9957
BERT
0.3252
0.3391
0.3386
0.3403
0.2046
0.2071
0.2070
0.2072
RoBERTa
0.3274
0.3399
0.3399
0.3409
0.2053
0,2072
0.2072
0.2073
MetaPath
0.1438
0.2999
0.2991
0.3086
0.1410
0.1976
0.1964
0.1995
Micro-Precision
MetaPath+ BERT
0,3249
0.3389
0.3399
0.3404
0.2044
0.2071
0.2073
0.2073
LinkBERT
0.3222
0.3388
0.3388
0.3397
0.2037
0.2071
0.2070
0.2071
GIANT
0.3299
0.3390
0.3393
0.3404
0.2056
0.2072
0.2072
0.2073
THLM
0,3338
0.3413
0.3418
0.3414
0.2063
0.2075
0.2075
0.2074
BERT
0.9368
0.9767
0.9753
0,9802
0.9821
0,9942
0.9936
0.9947
RoBERTa
0.9430
0.9790
0.9791
0.9820
0.9854
0.9948
0.9945
0.9954
MctaPath
0,4142
0.8638
0.8614
0.8888
0.6767
0.9484
0.9427
0.9578
Micro-Recall
MetaPath+BERT
0.9357
0.9762
0.9791
0.9805
0.9813
0.9941
0.9952
0.9952
LinkBERT
0,9281
0,9758
0.9758
0.9785
0.9777
0.9941
0.9936
0.9943
GIANT
0.9502
0.9764
0.9774
0.9804
0.9868
0.9948
0.9946
0.9953
THLM
0.9613
0.9831
0.9846
0.9835
0.9902
0.9962
0.9960
0.9957
BERT
0.8526
0.9164
0.9158
0.9252
0.8713
0.9236
0.9233
0.9312
RoBERTa
0.8600
0.9192
0.9209
0.9266
0.8776
0.9257
0.9273
0.9321
MctaPath
0.3008
0.7740
0.7735
0.8070
0.4089
0.8088
0.8072
0.8354
NDCG
MctaPath+BERT
0.8507
0.9170
0.9214
0.9253
0.8695
0.9244
0.9280
0.9314
LinkBERT*
0.8413
0,9149
0.9168
0.9231
0.8617
0.9224
0.9241
0.9295
GIANT
0.8732
0.9168
0.9195
0.9252
0.8883
0.9243
0.9266
0.9313
THLM
0,8879
0.9288
0.9310
0.9314
0.8998
0.9342
0.9357
0.9364

---

Table 9: Performance of different methods Patents node classification. The best and second-best performances are boldfaced and underlined.
K=3
K=5
Metric
Model
MLP
HetSANN
RGCN
R-HGNN
MLP
HctAANN
RGCN
R-HGNN
BERT
0.2012
0.2502
0.2634
0.2365
0.1425
0.1800
0.1883
0.1866
RoBERTa
0.2010
0.2421
0.2648
0.2886
0.1414
0.1725
0.1836
0.2136
MetaPath
0.0655
0.1321
0.1389
0.1579
0.0523
0.1062
0.1
0.1234
Macro-Precision
MetaPath+BBRT
0.2041
0.2541
0.2626
0.2914
0.1418
0.1818
0.1860
0.2098
LinkBERT*
0.2144
0.2443
0.2715
0.2933
0490
0.1758
0.1877
0.2194
GIANT
0.2181
0.2459
0.2692
0.2854
0.1518
0.1799
0.1882
0.2137
THLM
0.2541
0.2671
0,2827
0.3133
0.1761
0.1864
0.1950
0,2300
BERT
0.3036
0.3553
0.3592
0.2765
0.3824
0.4326
0.4493
0.3619
RoBERTa
0.3017
0.3598
0.3603
0.3618
0.3827
0.4430
0.4560
0.4526
MetaPath
0,0335
0,1889
0.1949
0.1884
0.0484
0.2446
0.2543
0.2465
Macro-Recall
MeIRPath+BERT
0.3027
0,3603
0.3610
0.3682
0.3810
0.4383
0.4522
0.4527
LinkBERT*
0.3186
0.3597
0.3714
0.3699
0.4038
0.4483
0.4631
0.4568
GIANT
0.3344
0,3591
0.3713
0.3654
0.4131
0.4324
0.4616
0.4511
THIM
0.3933
0.4023
0.4067
0.4111
0.4890
0.4886
0.5038
0.4976
BERT
0.3502
0.3636
0.3785
0.3599
0.2426
0,2502
0.2590
0.2488
RoBERTa
0.3566
0.3694
0.3845
0.3826
0.2472
0,2541
0.2626
0.2615
MetaPath
0.1286
0.2580
0.2695
0.2729
0.0971
0.1874
0.1941
0.1962
Micro-Precision
MetaPath+ BERT
0,3498
0.3646
0.3773
0.3775
0.2428
0.2507
0.2582
0.2584
LinkBERT
0.3609
0.3699
0.3841
0.3851
0.2494
0.2542
0.2625
0.2627
GIANT
0.3596
0.3656
0.3804
0.3775
0.2484
0.2505
0.2600
0.2583
THLM
0,3843
0.3863
0.3951
0.3959
0.2626
0.2627
0.2684
0.2686
BERT
0.6375
0.6618
0.6890
0.6552
0.7360
0.7591
0.7856
0.7548
RoBERTa
0.6491
0.6724
0.6998
0.6964
0.7499
0.7709
0.7968
0.7933
MctaPath
0,2341
0.4697
0.4905
0.4967
0.2945
0.5684
0.5888
0.5951
Micro-Recall
MetaPath+BBRT
0.6367
0.6636
0.6868
0.6871
0.7367
0.7606
0.7834
0.7838
LinkBERT
0,6570
0.6734
0.6992
0.7010
0.7567
0.7711
0.7963
0.7970
GIANT
0.6546
0.6655
0.6923
0.6871
0.7537
0.7598
0.7888
0.7835
THLM
0.6996
0.7032
0.7192
0.7207
0.7967
0.7970
0.8144
0.8148
BERT
0.6725
0.7025
0.7297
0.6921
0.7066
0.7353
0.7610
0.7262
RoBERTa
0.6854
0.7140
0.7417
0.7387
0.7200
0,7470
0.7726
0.7697
MctaPath
0.2467
0.4902
0.5103
0.5188
0.2734
0.5296
0.5487
0.5573
NDCG
MctaPath+BERT
0.6717
0.7024
0.7272
0.7280
0.7066
0.7351
0.7586
0.7594
L
0.6953
0.7154
0.7413
0.7444
0.7291
0.7478
0.7725
0.7748
GIANT
0.6936
0.7084
0.7354
0.7305
0.7275
0.7397
0.7665
0.7617
THLM
0.7442
0.7488
0.7652
0.7675
0.7752
0.7785
0.7950
0.7963


Reviews:
[{"review_id": "so8QlXAIgQ", "reviewer": "Reviewer_m9df", "summary": "", "questions": "* What's the main idea to deal with heterogeneous node types effectively?\n* What is A in line 144?\n* Is there something that makes it challenging to apply THLM  to encoder-decoder and decoder only models?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed comments. We have conducted experiments to illustrate the effectiveness and efficiency of the text augmentation strategy. We have also clarified the design on learning graph structure and heterogeneous node types and revised the typos in our paper. We have discussed the challenges of encoder-decoder and decoder-only PLMs in our work. We hope our answers can sufficiently address your concerns.\n\n**W1(part 1/2): Effectiveness of the text augmentation strategy.**\n\nWe have conducted significant experiments and presented the p-values to support our claims. Specifically, we performed the node classification task multiple times (5 times) using different seeds ranging from 0 to 4 for these two pre-trained LMs whether using text augmentation strategy in our THLM. \n\nBy comparing the results, we observed that all p-values were less than 0.05, indicating statistical significance and validating the accuracy gain achieved by our text augmentation strategy.\n\n\n| p-values  | MLP       | HetSANN   | RGCN      | RHGNN     |\n| --------- | --------- | --------- | --------- | --------- |\n| GoodReads | 1.432E-04 | 2.026E-04 | 1.440E-04 | 2.529E-04 |\n| Patents   | 1.940E-04 | 7.397E-04 | 3.678E-04 | 7.111E-04 |\n\n**W1(part 2/2): Efficiency of the text augmentation strategy.**\n\nWe further report the training time and performance (Micro-Precision on node classification in the following table) of different methods on GoodReads and Patents. The running time of THLM on GoodReads (Patents) with aggregating neighbors ranging from 1 to 3 is 12,240s, 13,800s, 14,220s (12,611s, 14,472s, 15,078s). However, we also observe that THLM obtains the best performance with acceptable increments in computational complexity. Therefore, we conclude that THLM can achieve a good trade-off between efficiency and effectiveness.\n\n| Dataset   | Model           | MLP    | HetSANN | RGCN   | R-HGNN |\n| --------- | --------------- | ------ | ------- | ------ | ------ |\n| GoodReads | TAS(1-Neighbor) | 0.7480 | 0.8353  | 0.8421 | 0.8469 |\n|           | TAS(2-Neighbor) | 0.7547 | 0.8381  | **0.8426** | 0.8475 |\n|           | TAS(3-Neighbor) |**0.7549** | **0.8382**  | 0.8425 | **0.8485** |\n| Patents   | TAS(1-Neighbor) | 0.6959 | 0.7004  | 0.7211 | 0.7221 |\n|           | TAS(2-Neighbor) | **0.6960** | *0.7050*  | 0.7219 | 0.7233 |\n|           | TAS(3-Neighbor) | 0.6948 | **0.7050**  | **0.7275** | **0.7281** |\n\nConsidering the statistical significance of the accuracy gain achieved through our text augmentation strategy and the careful consideration given to computational efficiency, we believe our approach offers a valuable trade-off between performance improvement and practical feasibility. \n\n\n**W2: Effectiveness of learning graph structure.**\n\nIn our study, we focus on pretraining a language model that effectively incorporates both the heterogeneity and topology of graphs. To achieve this, we introduce a topology-aware pretraining task where the language model predicts nodes in the context graph surrounding a target node. This task optimizes both the language model and an auxiliary heterogeneous graph neural network (HGNN), allowing the model to utilize both first-order and high-order signals from the graph. To assess the ability of the HGNN to capture graph topology in our framework, we conduct an experiment where we replace the HGNN with a simple Multi-Layer Perceptron (MLP) mechanism and two state-of-the-art HGNNs (RGCN, R-HGNN) for pretraining and testing on downstream tasks.\n\nCompared to approaches such as LinkBERT and GIANT, which rely on self-supervised tasks to learn the graph structure, our method takes advantage of an auxiliary HGNN to incorporate the topology structure. By doing so, we are able to effectively capture the intricate topology inherent in graphs and enhance the pretraining process.\n\nThe results presented in the following table, specifically the Micro-Precision on node classification, clearly demonstrate the significance of capturing graph topology. When our framework utilizes the HGNN to learn the inner structure, it achieves improved performance. On the other hand, using the MLP mechanism that neglects graph topology leads to a significant performance deterioration. Furthermore, by incorporating the auxiliary HGNN, our model surpasses the performance of both LinkBERT and GIANT.\n\n| Dataset   | Model     | MLP    | HetSANN | RGCN   | R-HGNN |\n| --------- | --------- | ------ | ------- | ------ | ------ |\n| GoodReads | w/ MLP    | 0.7528 | 0.8352  | 0.8376 | 0.8445 |\n|           | w/ RGCN   | 0.7608 | 0.8380  | 0.8411 | 0.8512 |\n|           | w/ R-HGNN | 0.7549 | 0.8382  | 0.8425 | 0.8485 |\n|           | LinkBERT  | 0.7244 | 0.8209  | 0.8259 | 0.8369 |\n|           | GIANT     | 0.6887 | 0.8098  | 0.8150 | 0.8258 |\n|           | THLM      | **0.7769** | **0.8439**  | **0.8437** | **0.8496** |\n| Patents   | w/ MLP    | 0.6903 | 0.6963  | 0.7201 | 0.7208 |\n|           | w/ RGCN   | 0.6911 | 0.6986  | 0.7184 | 0.7218 |\n|           | w/ R-HGNN | 0.6948 | 0.7050  | 0.7275 | 0.7281 |\n|           | LinkBERT  | 0.6333 | 0.6670  | 0.6968 | 0.6957 |\n|           | GIANT     | 0.6508 | 0.6701  | 0.6956 | 0.6999 |\n|           | THLM      | **0.7066** | **0.7159**  | **0.7324** | **0.7363** |\n\n**W3&Q1: Design for dealing with heterogeneous node types.**\n\nIn our study, we address the heterogeneity of the graph, including different node types, by employing an auxiliary HGNN (Heterogeneous Graph Neural Network). Unlike existing approaches such as LinkBERT and GIANT, which focus on homogeneous graphs, we utilize an HGNN to effectively encode the heterogeneous graph. We have the flexibility to choose among various HGNNs, each with its own characteristics. For instance, RGCN (Relational Graph Convolutional Network) aggregates information across different types of nodes and edges, while R-HGNN (Relation-aware Heterogeneous Graph Neural Network) learns relation-aware node representations by integrating fine-grained representations within each relation and semantic information across different relations. The results obtained using \"w/ RGCN\" and \"w/ R-HGNN\" indicate that our model achieves better performance on downstream tasks when utilizing stronger HGNNs, as they provide more informative node representations.\n\n**Q2: Explaination of $\\mathcal{A}$.**\n\nThanks for pointing this out. $\\mathcal{A}$ should be $\\mathcal{U}$, which means that the number of node types and edge types should be more than 1 if the graph belongs to a heterogeneous graph.\n\n**Q3: Challenges for encoder-decoder and decoder-only PLMs.**\n\nWe propose a new pretraining framework for LMs that incorporates topological and heterogeneous information from TAHGs to encode contextual representations. We focus on encoder-only models due to their proficiency in capturing context. \n\nEncoder-decoder PLMs, like T5, encode the input sentence and generate the output sequence using separate encoder and decoder components. However, in THLM, we only pretrain the encoder, which introduces inconsistency between the encoder and decoder, potentially affecting downstream tasks like machine translation. Decoder-only PLMs, such as GPT, generate the output sequence autoregressively during pretraining, but they may overlook essential information from the right context, leading to incomplete or less accurate semantic representations lacking global context consideration. Moreover, both PLM types prioritize sequence generation tasks, potentially introducing biases favoring fluent generation over accurate semantic representations.\n\nTo address these challenges, techniques like incorporating bidirectional context, modifying training processes, or introducing additional objectives may be explored to improve the quality and accuracy of semantic representations generated by encoder-decoder and decoder-only models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The framework is able to incorporate both graph structural information and textual information into learning node representations even when the node types are heterogeneous.\n* The framework seems to be easily applicable to various foundation GNN/LM models.\n* The experiments were conducted extensively. They cover various aspects of the proposed framework.", "reasons_to_reject": "* It is not quite convincing how much effective and efficient the text augmentation strategy is. Compared to the case where only textless node's own texts are used, the accuracy gain is not significant. Also, the increase in the amount of texts by taking neighbors' texts would make the overall computation efficiency worse.\n* Not sure if the graph topology is really well learned. Given a node, the framework draws samples without any consideration about how the selected neighbors are connected to the node. It just considers whether the sampled neighbor is part of K-order neighbors or not.\n* The paper claims the proposed framework is for heterogeneous graphs but it does not explain well how the node types are handled differently from other methods. The objectives used to optimize the model do not seem to care about the node types.", "questions_to_authors": "* What's the main idea to deal with heterogeneous node types effectively?\n* What is A in line 144?\n* Is there something that makes it challenging to apply THLM  to encoder-decoder and decoder only models?", "paper_topic_and_main_contributions": "This paper proposes a pre-training framework for heterogeneous graphs where each node is associated with texts. The existing LMs can learn the textual information well but they ignore the graph topology which could be important to perform well the representation learning for nodes and to eventually achieve high performance on downstream tasks. The proposed framework, THLM, basically attempts to address two problems: 1) how to learn graph structure with high-order topological information and 2) how to tackle imbalanced textual information of nodes. For the first one, it extracts a context graphs for each node and the a GNN model is trained so as to predict its neighbor nodes involved in the context graph. To enrich the textual information for a node with lack of text, the framework simply concatenates its neighbor nodes' texts and use them as input to LM model. The authors conducted extensive experiments, including three different datasets and six baseline methods, to demonstrate the effectiveness of the framework. For both node classification and link prediction which are used as downstream tasks, THLM achieves higher accuracy overall compared to the baselines. Also, the paper shows the impact of the text augmentation strategy and the impact of different orders in context graph extraction.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aCARDZTuNC", "reviewer": "Reviewer_xFg8", "summary": "", "questions": "- Could you repeat your experiments (the fine-tuning stage) multiple times, conduct a t-test, and report the p-values by comparing THLM with the strongest baseline in each column?\n\n- Could you compare THLM with OAG-BERT (on OAG-Venue only) and GraphFormers?\n\n- Can THLM deal with nodes without any text information?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We have strengthened the novelty of our work, and have conducted extensive experiments comparing THLM with baselines. Additionally, we have run OAG-BERT and done a comparison with THLM. At last, we elaborated on the process of dealing with nodes with no text in our model. These efforts will undoubtedly contribute to the overall strength and significance of my work.\n\n**W1: Novelty of our work.**\n\nIn this paper, we present a novel pretraining framework for Language Models (LMs) that explicitly incorporates the topological and heterogeneous information found in Text-Attributed Heterogeneous Graphs (TAHGs). We compare our model with existing works from two perspectives:\n\n* Unlike traditional Heterogeneous Graph Neural Networks (HGNNs) that focus on learning node representations, our objective is to integrate the graph learning capabilities of HGNNs into PLMs through a topology-aware pretraining task. To achieve this, we employ representative HGNNs to capture the graph topology, enabling LMs to leverage both first- and high-order graph structures. \n\n* Existing approaches for modeling text-rich networks typically concentrate on homogenous networks or modify the architecture of LMs by incorporating external components. In our work, we employ an auxiliary HGNN during LM pretraining, but subsequently discard the auxiliary networks and solely utilize the pretrained LMs for downstream tasks without altering their original architectures.\n\nIt should be noted that Heterformers are specifically designed for text-attributed networks, with a focus on embedding rich-text nodes and their surrounding neighbors. However, they explore the graph structure by combining LMs and GNNs in both pretraining and downstream tasks. Additionally, Heterformers may fail to capture the graph topology around textless nodes, limiting their ability to effectively learn the graph structure. To address this challenge and mitigate the text-imbalanced problem, we employ a text augmentation strategy that enriches the semantics of textless nodes. This augmentation strategy facilitates the learning of the graph structure around these nodes, enhancing the overall performance of our approach.\n\n\n\n**W2&Q1: Significance tests on THLM and baselines.**\n\nWe conducted significance tests to compare the performance of THLM with the strongest baseline in both link prediction and node classification tasks. In order to present the results within the given space limitations, we specifically showcase the p-values for the MAE and Micro-Precision metrics associated with these two tasks. Notably, all the calculated p-values are found to be less than 0.05, indicating significant superiority of our THLM model over the baselines. These results validate the effectiveness and improved performance of our approach in both link prediction and node classification tasks.\n\n| p-values  | HetSANN   | RGCN      | RHGNN     |\n| --------- | --------- | --------- | --------- |\n| Books     | 3.161E-05 | 2.907E-04 | 1.629E-05 |\n| Patents   | 2.059E-06 | 1.862E-05 | 4.131E-05 |\n| OAG-Venue | 4.512E-04 | 3.024E-04 | 1.783E-05 |\n\n| p-values  | MLP       | HetSANN   | RGCN      | RHGNN     |\n| --------- | --------- | --------- | --------- | --------- |\n| Books     | 5.985E-05 | 1.461E-06 | 1.107E-04 | 2.669E-07 |\n| Patents   | 5.220E-07 | 1.460E-06 | 1.070E-06 | 3.270E-05 |\n| OAG-Venue | 1.476E-05 | 2.212E-05 | 2.124E-03 | 5.983E-05 |\n\nIt is important to note that the performance evaluation of THLM in Table 1 and Table 2-3 is subtly nuanced. This is because we report the performance of THLM without MLM (Masked Language Modeling) in all remaining experiments, except for the results presented in Table 1 to isolate its effects.  Consequently, the performance of THLM in Table 2 and Table 3 actually refers to THLM without MLM. We acknowledge the need for clarification on this matter, and we will provide a more detailed explanation in the revised version of the paper.\n\n\n**W3&Q2: Comparison with OAG-BERT and GraphFormer.** \n\nThank you for providing insights into the related works. OAG-BERT is a pre-trained language model specialized in academic knowledge services, allowing for the incorporation of heterogeneous entities such as authors, institutions, and keywords into paper embeddings. On the other hand, GraphFormers introduces a GNN-nest Transformer framework to simultaneously capture the graph structure surrounding different papers and the semantics of their texts.\n\nWe show the performance of OAG-BERT and other models in the following table. However, it is worth noting that in our work, we encountered challenges when attempting to use GraphFormers. The requirement of pretraining the PLM with text information from the center node and its surrounding neighbors demands significant computational resources. As a result, we were unable to successfully run GraphFormers within the given time constraints. We plan to conduct experiments using GraphFormers in the future and include the results in our revised paper.\n\n|     Model     | HetSANN |  RGCN  | R-HGNN |\n| :-----------: | :-----: | :----: | :----: |\n|     BERT      | 0.1987  | 0.2149 | 0.1802 |\n|    Roberta    | 0.1931  | 0.2152 | 0.1689 |\n|   MetaPath    | 0.2199  | 0.2415 | 0.1946 |\n| MetaPath+BERT | 0.2213  | 0.2149 | 0.1651 |\n|   LinkBERT    | 0.1867  | 0.2229 | 0.1739 |\n|     GIANT     | 0.2045  | 0.2022 | 0.1709 |\n|   OAG-BERT    | 0.1918  | 0.2030 | 0.1772 |\n|     THLM      | **0.1857**  | **0.1893** | **0.1591** |\n\n| Model         | MLP    | HetSANN | RGCN   | R-HGNN |\n| ------------- | ------ | ------- | ------ | ------ |\n| BERT          | 0.2242 | 0.3146  | 0.3136 | 0.3473 |\n| Roberta       | 0.2527 | 0.3193  | 0.3341 | 0.3516 |\n| MetaPath      | 0.1132 | 0.2693  | 0.2851 | 0.3011 |\n| MetaPath+BERT | 0.2307 | 0.3311  | 0.3317 | 0.3472 |\n| LinkBERT      | 0.2278 | 0.3108  | 0.3110 | 0.3508 |\n| GIANT         | 0.2280 | 0.3116  | 0.3074 | 0.3274 |\n| OAG-BERT      | 0.2577 | 0.3214  | 0.3152 | 0.3425 |\n| THLM          | **0.2637** | **0.3409**  | **0.3398** | **0.3575** |\n\nUpon analyzing the results, it is evident that OAG-BERT achieves competitive results in link prediction when compared to LinkBERT and GIANT and demonstrates strong performance compared to the baselines in node classification. This can be attributed to its ability to learn the heterogeneity and topology of graphs during the pretraining phase. However, it is important to note that OAG-BERT primarily captures correlations between papers and their meta-data information, such as authors, venues, and institutions, while disregarding the high-order structural information. As a result, OAG-BERT performs relatively worse than our model, which takes into account the high-order structure and achieves improved performance in the task at hand.\n\n**Q3: Dealing with nodes without text sequences.**\n\nIn our experimental setup, we encountered nodes that did not have any text information associated with them. To address this issue, we implemented the text augmentation strategy by concatenating the text sequences from neighboring nodes. This approach allows us to generate meaningful semantic representations for these nodes, effectively mitigating the text-imbalanced problem. By incorporating the text information from neighboring nodes, THLM can handle nodes without any text descriptions in a consistent manner.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The idea of exploiting graph information in heterogeneous networks has practical value due to the prevalence of structural information in real-world texts. \n\n+ Different from previous studies (e.g., LinkBERT and GraphFormers) that considers first-order and homogeneous links only, this paper identifies the challenges of tackling higher-order and heterogeneous structures. The proposed framework of combining pre-trained language models and heterogeneous graph neural networks is intuitive and well-motivated.\n\n+ The authors conduct comprehensive ablation analyses and hyperparameter studies to validate their design choices and configurations.", "reasons_to_reject": "- The novelty of this work is not significant. Modeling higher-order network information has been extensively explored in (heterogeneous) GNN studies. Considering imbalanced text information in heterogeneous networks has been studied in Heterformer. There are also many related works combining language models and graph structures together using different architectures. Therefore, this work is more like integrating many existing techniques together.\n\n- Significance tests are missing. It is unclear whether the improvement of THLM over baselines and ablation versions is statistically significant or not. In fact, some gaps between THLM and baselines/ablation versions are quite subtle in Tables 1-3, therefore p-values should be reported.\n\n- Some relevant baselines, such as GraphFormers and OAG-BERT [1], are not compared in the experiments.\n\n[1] OAG-BERT: Towards a Unified Backbone Language Model for Academic Knowledge Services. KDD 2022.", "questions_to_authors": "- Could you repeat your experiments (the fine-tuning stage) multiple times, conduct a t-test, and report the p-values by comparing THLM with the strongest baseline in each column?\n\n- Could you compare THLM with OAG-BERT (on OAG-Venue only) and GraphFormers?\n\n- Can THLM deal with nodes without any text information?", "paper_topic_and_main_contributions": "This paper proposes a language model pre-training framework for text-attributed heterogeneous graphs (TAHGs). The key idea is to leverage not only text semantics but also topological connections between texts during pre-training. The authors propose to tackle two challenges, higher-order connections and imbalanced text information, which are less concerned in previous studies. To be specific, they train an auxiliary heterogeneous graph neural network and devise a text augmentation strategy to solve the two challenges, respectively. Experiments on three datasets and two tasks demonstrate the effectiveness of the proposed THLM model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "OAG-BERT [1] mentioned above"}, "extra_text": {}}, {"review_id": "5frB29Y4DI", "reviewer": "Reviewer_v8Uk", "summary": "", "questions": "A. In Section 2.1, the process of sampling k neighbors is not detailed. Could you clarify how this sampling is conducted?\nB. In Table 1, the authors reconstruct homogeneous graphs for GIANT and LinkBert. In terms of link prediction tasks, how is a fair comparison maintained with the original heterogeneous graphs?\nC. There are inconsistencies in the experimental section that need to be clarified by the authors. For instance, the results of THLM in Tables 2 and 3 differ from those in Table 1. THLM without CGP approximates BERT, but there's a significant discrepancy between the blue column for the Patents data in Figure 4 and the results for BERT in Table 1. Could the authors please explain this?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback. We appreciate your attention to the performance of our proposed approach on the OAG-Venue and MAG datasets and we show the performance in the following tables. Additionally, we have highlighted the novelty of our work and provided a thorough explanation of our design. Furthermore, we further clarified the performance results in the experimental section. These additions aim to enhance the overall clarity and comprehensibility of our paper.\n\n**W1: Experiments on OAG-Venue.**\n\nWe appreciate your feedback on the performance of our approach on the OAG-Venue dataset. We acknowledge that due to space limitations, we did not provide the performance on OAG-Venue in the remaining experiments, except for Table 1. However, we would like to inform you that we have included the performance (Micro-Precision on node classification) about the evaluation of Context Graph Prediction (Section 4.5) and ablation study (Section 4.7) in the following table, and similar observations can be made for other metrics as well. These results and the performance of the rest of the experiments align with the original paper's findings. We will ensure that all of the results on the OAG-Venue dataset are included in future revisions or updates of the paper.\n\n\n| Methods         | MLP        | HetSANN    | RGCN       | RHGNN      |\n| --------------- | ---------- | ---------- | ---------- | ---------- |\n| w/ MLP          | 0.2591     | 0.3195     | 0.3043     | 0.3379     |\n| w/ RGCN         | 0.2728     | 0.3323     | 0.322      | 0.3547     |\n| w/ 2-order CGP  | 0.2609     | 0.3357     | 0.3121     | 0.3488     |\n| w/ random feats |    0.2602        |   0.3271         |    0.3133        |     0.3487       |\n| THLM w/o MLM   | **0.2629** | **0.3383** | **0.3228** | **0.3554** |\n\n\n| Methods      | MLP    | HetSANN | RGCN   | RHGNN  |\n| ------------ | ------ | ------- | ------ | ------ |\n| THLM         | **0.2637** | **0.3409**  | **0.3398** | **0.3575** |\n| THLM w/o CGP | 0.2260 | 0.3275  | 0.3152 | 0.3485 |\n| THLM w/o MLM | 0.2629 | 0.3383  | 0.3228 | 0.3554 |\n\n**W2: Explanation of the novelty of THLM.**\n\nWe propose a novel pretraining framework for Language Models (LMs) that incorporates topological and heterogeneous information from Text-Attributed Heterogeneous Graphs (TAHGs). Unlike existing approaches such as LinkBERT and GIANT, which focus on capturing first-order connections between texts, our method emphasizes learning both first-hop and high-order topology in TAHGs. We also introduce a text augmentation strategy to enhance the representations of textless nodes by aggregating neighbors' text information.\n\nIn contrast to GLEM, where LMs and Graph Neural Networks (GNNs) are jointly used for latent representation inference, our approach utilizes Heterogeneous Graph Neural Networks (HGNNs) as auxiliary modules to incorporate graph topology into PLMs. Our goal is to integrate the graph learning capability of HGNNs into PLMs through a topology-aware pretraining task. After the pretraining phase, we discard the auxiliary networks and solely rely on the pretrained LMs, without modifying their original architectures. This allows us to effectively utilize the learned representations in various downstream tasks.\n\nOverall, we aim to present a new pretraining framework for LMs to help them comprehensively capture multi-order relationships as well as heterogeneous information in a more complicated data structure, i.e., TAHGs\n\n**W3: Design for the heterogeneity of node u in Equation 3.**\n\nIn our approach, to capture the heterogeneity of node $u$, we introduce a projection header in the last layer of the PLM, which is utilized in Equation 2. The process involves multiplying $h^u_{LM}$ with $W_{\\phi(u)}$, similar to how we handle the heterogeneity of node $v$. We acknowledge that the embedding process in Equation 3 could be clarified further, and we will ensure to provide a more detailed explanation in future versions of the paper. \n\n**W4: Performance on ogbn-mag datasets.**\n\nIn our evaluation, we tested THLM on the obgn-mag dataset for the node classification task. Considering the limited time available, we utilized hyperparameters from the OAG-Venue dataset, and we acknowledge that further hyperparameter searches are desirable for future experiments. The results of THLM on the test set, specifically the Micro-Precision on node classification, are presented in the following table.\n\nWe observed that THLM demonstrates scalability to larger datasets, outperforming baselines such as LinkBERT and GIANT. This outcome highlights the effectiveness of THLM, particularly its superior performance on the obgn-mag dataset.\n\n\n| Model    | MLP    | HetSANN | RGCN   |\n| -------- | ------ | ------- | ------ |\n| BERT     | 0.3754 | 0.5298  | 0.5484 |\n| RoBERTa  | 0.3770 | 0.5300  | 0.5490 |\n| LinkBERT | 0.3775 | 0.5230  | 0.5491 |\n| GIANT    | 0.3903 | 0.5184  | 0.5256 |\n| THLM     | **0.3933** | **0.5353**  | **0.5517** |\n\n**Q1: Explanation of the sampling process.**\n\nAfter carefully reviewing our work, we find that there is no explicit mention of the sampling process in Section 2.1. To provide a clear understanding of the sampling process, we would like to provide detailed explanations of both sampling processes employed in our work.\n\nFor our text augmentation strategy, we prioritize the utilization of text sampled from rich-text neighbors. We do this by uniformly selecting text from these neighbors, thereby enriching the textual information associated with textless nodes. However, in cases where there are no rich-text neighbors available around textless nodes, we resort to concatenating the text from neighboring textless nodes as an alternative strategy.\n\nRegarding the context graph prediction task, we adopt a uniform sampling strategy. Specifically, during each hop, we uniformly sample $k$ neighbors from a specific relation. These selected neighbors from all kinds of relations are considered positive candidates during the training process. In order to generate negative candidates, we employ a negative sampling strategy.\n\n**Q2: Comparison between THLM and LinkBERT, GIANT.**\n\nRegarding the LinkBERT and GIANT models, we followed a specific procedure for their pretraining and downstream tasks. We pretrained these models on reconstructed homogeneous graphs, generating contextual representations for all nodes based on their respective text sequences. However, during downstream tasks, we utilized the original heterogeneous graphs for prediction.\n\nDue to the nature of LinkBERT and GIANT, which are designed for homogeneous networks with rich-text nodes, we encountered a challenge on datasets such as Patents and GoodReads. In these datasets, there are no direct connections between rich-text nodes. To address this, we constructed a homogeneous graph by selecting connections up to 2 hops among rich-text nodes. This approach aimed to preserve the original topology as much as possible. These models are denoted as LinkBERT(\\*) and GIANT(\\*). However, it's important to note that the reconstructed graph overlooks the presence of textless nodes and may result in some loss of graph topology. To further investigate the impact of retaining the original topology, we conducted another experiment in which we discarded the heterogeneity of nodes and relationships in the original heterogeneous graph to build a homogeneous graph for pre-training, namely LinkBERT(1-hop) and GIANT(1-hop). The performance on link prediction and node classification is shown respectively as follows (we use RMSE and Micro-Precision metric on these two tasks, other metrics have similar observations). \n\n\n| Dataset   | Model           | HetSANN | RGCN   | R-HGNN |\n| --------- | --------------- | ------- | ------ | ------ |\n| GoodReads | LinkBERT(1-hop) | 0.1507  | 0.1423 | 0.1189 |\n|           | LinkBERT(*)     | 0.1471  | 0.1362 | 0.1135 |\n|           | GIANT(1-hop)    | 0.1365  | 0.1259 | 0.1066 |\n|           | GIANT(*)        | 0.1323  | 0.1179 | 0.1089 |\n|           | THLM            | **0.1206**  | **0.1159** | **0.1000** |\n| Patents   | LinkBERT(1-hop) | 0.3122  | 0.3079 | 0.2641 |\n|           | LinkBert(*)     | 0.3080  | 0.3033 | 0.2601 |\n|           | GIANT(1-hop)    | 0.2781  | 0.2492 | 0.2309 |\n|           | GIANT(*)        | 0.2734  | **0.2454** | 0.2276 |\n|           | THLM            | **0.2522**  | 0.2513 | **0.2190** |\n\n| Dataset   | Model           | MLP    | HetSANN | RGCN   | R-HGNN |\n| --------- | --------------- | ------ | ------- | ------ | ------ |\n| GoodReads | LinkBERT(1-hop) | 0.6790 | 0.8100  | 0.8044 | 0.8302 |\n|           | LinkBERT(*)     | 0.7244 | 0.8209  | 0.8259 | 0.8369 |\n|           | GIANT(1-hop)    | 0.6967 | 0.8247  | 0.8284 | 0.8398 |\n|           | GIANT(*)        | 0.6887 | 0.8098  | 0.8150 | 0.8258 |\n|           | THLM            | **0.7769** | **0.8439**  | **0.8437** | **0.8496** |\n| Patents   | LinkBERT(1-hop) | 0.5972 | 0.6421  | 0.6773 | 0.6734 |\n|           | LinkBERT(*)     | 0.6333 | 0.6670  | 0.6968 | 0.6957 |\n|           | GIANT(1-hop)    | 0.4793 | 0.6234  | 0.6323 | 0.6391 |\n|           | GIANT(*)        | 0.6508 | 0.6701  | 0.6956 | 0.6999 |\n|           | THLM            | **0.7066** | **0.7159**  | **0.7324** | **0.7363** |\n\nFrom the results, it is evident that pretraining LinkBERT and GIANT on TAHGs solely for 1-order prediction may not yield optimal results. There are two key reasons for this observation: 1) The homogeneous graph used in LinkBERT and GIANT is built on rich-text nodes. They focus on capturing the relationships among rich-text nodes, which are the second-hop relationships in our graph. Hence, they fail to learn the graph topology well with the original heterogeneous graph. 2) Textless nodes always lack sufficient textual content, leading to scarce semantic information. Hence, predicting relationships between textless nodes and their neighbors becomes challenging for language models.  These findings highlight the importance of considering both first-order and high-order structure information in TAHGs and addressing the challenges of limited semantics on textless nodes. By tackling both problems, our model can learn better in TAHGs. (See Appendix A.6 for detailed settings of GIANT and LinkBERT experiments)\n\n\n**Q3 (Part 1/2): Clarification on the performance in Table 1, Table 2, and Table 3.**\n\nIn Table 1 presented in our original paper, alongside the CGP task, we introduced the MLM task to improve the LM's ability to capture semantic information on text sequences. However, for the remaining experiments, involving the analysis of different components like CGP and the text augmentation strategy, we intentionally removed the MLM task to isolate its effects. Therefore, the performance results of THLM in the subsequent experiments, such as in Table 2 and Table 3 were obtained without the MLM task, as mentioned in Lines 451-452 of the paper. We will provide further elaboration in the revised version of the paper.\n\n\n**Q3 (Part 2/2): Clarification on the performance between THLM and BERT.**\n\nThe distinction between THLM without MLM and the original BERT stems from our text augmentation strategy for textless nodes. In the original BERT, we simply masked the brief terms of textless nodes. However, in THLM without MLM, we take a different approach by enhancing the semantics of textless nodes. This enhancement allows for better connectivity between the brief terms of textless nodes and their neighboring text sequences, resulting in improved contextual understanding and representation in pre-training PLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Text-Attributed Graph Learning is a significant area of study. Both the textual modality and the graph structure have unique characteristics, and pre-training a language model (LM) using a Heterogeneous Graph Neural Network (HGNN) is a challenging task. \n2. For predicting the nodes in the context graph of a target node, a topology-aware pre-training task is designed. \n3. The method leverages higher-order information from heterogeneous graphs, a feature that is relatively overlooked in homogeneous graphs. This point is corroborated in the experiments through comparison with homogeneous graph methods, such as GIANT and LinkBERT.\n4. The presentation is well-structured and easy to follow.\n", "reasons_to_reject": "1. Beyond Table 1, results on the OAG-Venue dataset are noticeably absent from the experimental analysis in the main text of the paper.\n2. The technical novelty of the paper is somewhat restricted, as both Masked Language Modeling (MLM) and Heterogeneous graph neural network (HGNN) are widely studied and extensively documented in existing literature.\n3. In Equation 3, the authors introduced a learnable matrix W_(ϕ(v)) to handle the heterogeneity of node v. However, it is unclear how the heterogeneity of node u is accounted for.\n4. The authors have only evaluated mid-sized datasets in this paper. It would be beneficial for the authors to consider including results from experiments on the ogbn-mag dataset to bolster the persuasiveness of their findings.\n", "questions_to_authors": "A. In Section 2.1, the process of sampling k neighbors is not detailed. Could you clarify how this sampling is conducted?\nB. In Table 1, the authors reconstruct homogeneous graphs for GIANT and LinkBert. In terms of link prediction tasks, how is a fair comparison maintained with the original heterogeneous graphs?\nC. There are inconsistencies in the experimental section that need to be clarified by the authors. For instance, the results of THLM in Tables 2 and 3 differ from those in Table 1. THLM without CGP approximates BERT, but there's a significant discrepancy between the blue column for the Patents data in Figure 4 and the results for BERT in Table 1. Could the authors please explain this?\n", "paper_topic_and_main_contributions": "This paper proposes a new pre-training framework for language models for Text-Attributed Heterogeneous Graphs (TAHGs). This method considers the topological relationships between texts in the heterogeneous graph during the pre-training of the language model, and introduces a text augmentation strategy to enhance the semantics of textless nodes.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Zhao J, Qu M, Li C, et al. Learning on large-scale text-attributed graphs via variational inference[J]. arXiv preprint arXiv:2210.14709, 2022."}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Strengths**:

1. The idea of pretraining using both text + graph info is interesting. Also, using graph info to handle text-richness imbalance is a good idea. It considers higher-order and heterogeneous structures.

2. The experiments were conducted extensively. New expts added as part of rebuttal process are relevant additions. Authors were also able to show that their method provides stat sig better results compared to methods already described in the paper and other recent methods like OAG-BERT.


**Weaknesses**:


1. Accuracy gains are not significantly large, considering the extra computations involved. But they are statistically significant.

2. Novelty is a little bit on lower side. As a reviewer indicates, earlier studies have tried combining language models and graph structures, and have handled imbalanced text in heterogeneous graph settings also.

3. Comparison with GraphFormers is missing.

**Suggestions**:


1. Please include a summary of several results presented in rebuttal, into the main paper.

2. Also, discussion on difference with Heterformer is nice. Please include it in related work.

3. Many clarifications during rebuttal are important for the reader. E.g., Clarification on the performance in Table 1, Table 2, and Table 3. Clarification on the performance between THLM and BERT. Please include them briefly in your main paper.
Decision: Accept-Findings