Unleashing the Power of Language Models in Text-Attributed Graph
Authors: Haoyu Kuang
Jiarong Xu
Haozhe Zhang
Zuyu Zhao
Qi Zhang
Xuanjing Huang
zhongyu wei
Venue: EMNLP 2023 Findings (2023)
Keywords: Hierarchical Text-attributed Graph
Pre-training
Self-supervised Tasks

Abstract: Representation learning on graph has been demonstrated to be a powerful tool for solving real-world problems. Text-attributed graph carries both semantic and structural information among different types of graphs. Existing works have paved the way for knowledge extraction of this type of data by leveraging language models or graph neural networks or combination of them. However, these works suffer from issues like underutilization of relationships between nodes or words or unaffordable memory cost. In this paper, we propose a Node Representation Update Pre-training Architecture based on Co-modeling Text and Graph (NRUP). In NRUP, we construct a hierarchical text-attributed graph that incorporates both original nodes and word nodes. Meanwhile, we apply four self-supervised tasks for different level of constructed graph. We further design the pre-training framework to update the features of nodes during training epochs. We conduct the experiment on the benchmark dataset ogbn-arxiv. Our method achieves outperformance compared to baselines, fully demonstrating its validity and generalization.

Unleashing the Power of Language Models in Text-Attributed Graph
Haoyu Kuang', Jiarong Xu²; Haozhe Zhang', Zuyu Zhao',
Qi Zhang', Xuanjing Huang', Zhongyu Wei!.
School of Data Science, Fudan University, China
School of Management, Fudan University, China
School of Computer Science, Fudan University, China
'Research Institute of Intelligent Complex Systems, Fudan University, China
'Huawei Technologies Co.,Lid, China
edu.cn, hykuang23em. fudan. edu. cn, {jiarongxu, qz, xjhuang, zywei jefudan. cn (zhanghaozhe7, zhaozuyu1 jehuawei. j com
Abstract
Text-attrbuuted Graph
Hierarchical Text-attributed Graph
Pre-training Tasks
Representation learning on graph has been demonstrated to be a powerful tool for solv- ing real-world problems. Text-attributed graph carries both semanntc and structural informa- tion among different types of graphs. Existing works have paved the way for knowledge ex- traction of this type data by leveraging lan- guage models graph neural networks com- bination of them. However, these works suffer from issues like underutilization of relation- ships between nodes words unaffordable memory cost. In this paper, we propose Node Representation Update Pre-training Architec- ture based on Co-modeling Text and Graph (NRUP). In NRUP, we construct a hierarchical lext-attributed graph that incorporates both ini- tial nodes and word nodes. Meanwhile, we ap- ply four self-supervised tasks for different level of constructed graph. We further design the pre-training framework to update the features of nodes during training epochs. We conduct the experiment on the benchmark dataset ogbn- arxiv. Our method achieves outperformance compared to baselines, fully demonstrating its validity and generalization.
Base Struchow
Node Attribute Raconstruction
Node-Word adges
Important Word Identification
Important
Words
Figure 1: An illustration of hierarchical text-attributed graph and corresponding self-supervised tasks for dif- ferent level.
Existing research on learning from text- attributed graph mainly falls into three lines: La))ang models(LMs) only, in these works, LMs(Kim, 2014; Vaswani et al., 2017) ap- plied leverage the local textual information of individual nodes and generate representation for them(Howard and Ruder, 2018). How- ever, structural relationship between nodes are ignored in this way. To leverage the re- lationships between nodes, an self-supervised learning framework GIANT(Chien et al., 2021) propose to guide the training of LM with graph structure. Nevertheless, LM-based meth- ods ignore the message passing among nodes; (2)Graph neural networks(GNNs) only(Kipf and Welling, 2017: Zhang and Chen, 2018), which are used to capture the structural properties of text-attributed graphs. Raw texts contained by each node are transformed to numerical features as node attributes(Hu et al., 2020a; Liu al., 2020; Hu al., 2020c) by graph-irrelevant meth- ods, such as bag-of-words, pre-trained bert in most previous studies. Clearly, the information contained in raw text is compressed and under- utilized. Additionally, relying solely on a fixed
1 Introduction
Text-attributed graphs, characterized by the associa- tion of nodes with text attributes (Yang et al., 2021), are prevalent in diverse real-world contexts. For instance, in paper citation networks, each paper is accompanied by textual content, while in social net- works, each user can be described through text de- scription. The investigation of learning techniques text-altributed graphs has garnered considerable attention in domains such graph learning, infor- mation retrieval, and natural language processing, reflecting the growing importance of understand- ing and analyzing textual information within the context of graph-bauc based structures.
"Corresponding author

---

representation of text may not be appropriate for certain scenarios. For instance, the term "Trans- former' can refer to a device used for adjusting the voltage of an electric power supply, while in an academic context, it signifies a specific model; (3) commiinttio of LMs and GNNs(Bi et al., 2021; Zhu et al.. 2021a), which boost the text embedding with graph structure. However, it suffers from se- vere scalability issues when facing with large-scale graph and huge parameters of LMs. To address this, GLEM (Zhao et al., 2023) leverages varia- tional EM framework to iteratively update both the LM and GNN modules, enabling scalability to real- world graphs. Nevertheless, GLEM relies on task- specific labels, resulting in node representations that are constrained to the specific task at hand. Generally, prior researches encounter issues such overlooking the relationships between nodes or words, scalability limitations, and a lack of gener- alizability.
paper-occurrence correlation, respectively. Third, to mutually boost representations of nodes and words, we employ a relational graph neural network (R-GNN) as the foundational model for acquiring knowledge from the hierarchical text- attributed graph. Furthermore, within our frame- work, we introduce two aggregators that iteratively refine the features of both nodes and words, lever- aging progressively optimized embeddings of pa- pers/words after designated number of training epochs.
Method
In this section, we present the entire training frame- work to learn paper and word representations simul- taneously without supervision based on R-GNNs, including the modeling method of text-attributed graph, self-supervised tasks and pre- pre-training archi- tecture, see overall framework in figure 2.
2.1 Hierarchical Text-attributed Graph
In this paper, we introduce a general text- attributed graph pre-training framework that could fully utilize the relationship between graph-based structure and textual information. The main contri- butions of our proposed research are follows.
To better establish the relationship between the raw text and the graph, we propose to construct a hierarchical text-attributed graph encompassing initial nodes and word nodes, inspired by Yao et al. (2019).
First, to enhance the modeling of textual infor- mation within nodes of text-attributed graphs, we construct a hierarchical text-attributed graph that incorporates both initial nodes and word nodes. More specifically, we further decouple the word nodes from the corpus consisting of the textual in- formation from all nodes. Then construct edges among nodes based on word occurrence in nodes (node-word edges) and word co-occurrence in the whole corpus (word-word edges), as shown in fig- ure 1. This enables us to capture the finer nuances of the text at a more granular level.
First, tokenize the raw text contained in the nodes, thereby acquiring all the individual words; Second, we construct hierarchical text- attributed graph that incorporates both initial nodes and word nodes.
Third, we construct edges among nodes for the hierarchical text-attributed graph. The relationship between initial nodes constitutes the edges between them; and we build paper-word edges based on word occurrence in papers; for edges between word nodes, we employ point-wise mutual informa- tion (PMJ) to measure the co-occurrence frequency between words to determine whether to build a edge. The PMI value is computed as follows:
Second, approaching the capability of generating effective representations adapted to various scenar- ios, we introduce multi-task graph pre-training framework. This framework encompasses various self-supervised tasks, such as link reconstruction, node attribute reconstruction, important word re- construction, and important word identification. The objective of link reconstruction is to capture the underlying structural patterns in general sense, while node attribute reconstruction aims to uncover the semantic relationships among nodes. Further- more, the tasks of important word reconstruction and important word identification are specifically designed for access to distinctive semantics and
PMT(i,) llog p(i.)) p()P(
p(i,j) = W(1))
W
p(1)
W(E
where i, represents two word nodes; W(i) is the number of sliding windows in the nodes that contain word W(i.)) i) is the number of sliding windows in the nodes that contain both word and j: and W is the total number of sliding windows in the corpus from all nodes. We construct edges

---

Co-modeling Text and Gruph
3. Node Representation Update Pre- Prainrain Architecture
Hierarchical Text-attributed Graph
Text1
Text-attributed Graph
Text2
Text3
Initial feature
Decouple Text
SSL Tasks
RGAT
Word Nodes
2. Self-supervised Tasks
Aggregator
Link Reconstruction
Node Attribute Reconstruction
Updated feature
Graph
AUTO ENCODER
Normalization
negative sampling
SSL
Important Words Identification
Important Words Reconstruction
BBB
Projection Head
Graph
Downstream Task
AUTO ENCODER
edge representation
Figure An illustration of of the entire process of NRUP. Co-modeling the text and graph based heterogeneous graph. 2). Self-supervised tasks as training objectives, 3). Pre-training architecture updates the features with aggregators.
for words with positive PMI value which suggests high semantic correlation of words.
attributed graph with initial nodes and words, we build bridge for information interaction. Conse- incorporate individual words into the training process, the initial nodes are endowed with knowledge from both interconnected nodes and their respective textual components.
Thus far, we have constructed the structure of hicrarchical text-attributed graph. Next, we need to assign corresponding node attributes to different types of nodes. Initially, we use pre-trained Bert model to generate word embedding for each word node as the attributes; Subsequently, we simply av- erage the embeddings of all word nodes in one node obtain the feature of initial nodes. Finally, obtain this hierarchical text-attributed graph with semantic and structural information coexisting.
quently,
2.2 Self-supervised Tasks
Appropriate tasks drive R-GNNs mine poten- tial structure and semantic information in hetero- geneous graph continuously. In our pre-training architecture, we apply tasks to fully mine the information of different level in hierarchical text- attributed graph, as shown figure
Let G = W, sp, o, X' Z) denotes the hierarchical text-attributed graph we built, where V and & represent the sets of nodes and edges respec- tively; J and IC represent sets of node types and edge types respectively; ip
2.2.1 Link Reconstruction
Briefly speaking, link reconstruction is predict existing edges between node pairs. Task Process: In the design of link reconstruction task, we regard it as binary classification problem, and train the model by negative sampling.
J the node type mapping function, while Φ + is the edge type mapping function; nax and g rep- resent the feature matrix of paper and word nodes respectively; and m denote the number of initial nodes and words respectively, and d denotes the feature dimension. The number of nodes | is the summation of the number of initial nodes and individual words.
First, we treat the edges existing in the graph positive samples, and sample some non-existent edges in the graph as negative samples.
Second, for each node pair (u, v) in the graph, we calculate their score: Eu,p = s(hu, hw) based on the representation h, and h,, where io is a dot
Intuitively, by constructing hierarchical text-

---

product, and can be any other function that com- putes similarity.
title of paper contains key information thus we move the stop words after tokenizing the title of the paper, and take the remaining words as important words. We mask the features of these important words and reconstruct them. The reconstruction loss is denoted as:
Third, labeling the positive sample as and the negative sample as 0, we can optimize R-GNN with the following loss function:
LL logo (eu,))
log(1- (og(1_o(euwx)) Pa(P)(
Σ
LIWR
Σ NEVIM
Wi
VIAS
|
10
where a denotes a activation function, and v; Pu(()) denotes the negative sampling distribution. We perform link reconstruction for edges among initial nodes.
2.2.4 Important Word Identification
Each node contain distinctive important words. Therefore, we design important word identifica- tion task with the objective of judging important words.
2.2.2 Node Attribute Reconstruction
In our work, use node attribute reconstruction task maximize the semantic information of hier- archical text-attributed graph, also known feature reconstruction in homogeneous graphs(Hou et al., 2022).
Task Process:
First, we label the paper-word edge differently according to whether it is a important word to this node, for important and 0 for unim- portant.
First, we sample a subset Venitial C Vinitiat. and mask their features with a mask token [MASK], i.c., a learnable vector [[M] E d and the feature matrix of word nodes remains unchanged. Thus the node feature ã can be defined as:
Task Process:
Second, we splice the representation ha and hm of each node-word pair (u, m) as the edge repre- sentation ha.m-
Third, we input the edge representation into a projection head to predict the edge label, then opti- mize with the following loss function:
z; - M 04 E V papee Fz Ui & V Ppaper
hum hu + hm y a.m projection head (hu.m))
24, U/ C V. woord
LIWT Σ - log o(1 um)) log am a,
Second, we input the feature matrix X, 2 and graph G into a graph encoder fe to obtain the la- tent code H. Then we replace H on masked node indices again with another mask token [DMASK].
where y' am denotes the probability of being predicted as an important word edge.
Third, input the re-masked code matrix 11 into decoder fa to obtain the reconstructed feature matrix W. Then optimize with the scaled cosine error:
2.3 Pre-training Framework
The representation of initial nodes and word nodes can be optimized simultancously based on hicrar- chical text-attributed graph. Motivated by this, we propose pre-traeeiine pre-traininig framework: Node Represen- tation Update Pre-training ArchitectureNNRUP), y7
CNAR
Σ
D Ppaper
||
PcCPpaper
In our framework, we use the constructed graph as the input; and we select a R-GNN model as our base model, then the combination of self- supervised tasks is served as the objectives of the pre-training stage. Furthermore, aggregators to update features of both initial nodes and words, which are denoted follows:
which is averaged loss over all masked initial nodes. The scaling factor ry is a hyper-parameter.
2.2.3 Important Word Reconstruction
design two
Intuitively, we believe that there are important words in the raw text of initial nodes which can reflect the semantic information of the node large extent. Motivated by this, design impor- tant word reconstruction task to reconstruct seman- tic information of important words.
Aggregalor.nstia
+- MEAN AGG(o, neigh Anitina tex M EAN (negghtt
Task Process: We need to define what are impor- tant words to the initial nodes. For instance, the
Aggregatorworc
MEAN AGG(n, neigh worrd +d MEAN AGGGGeighh Enitsol) Enitsol

---

c nnttaa word d = neigh word nneighword trat word
initial / nneigh insesal Smeriad
Insigh
sword meighwordd
paper and each directed edge indicates that one pa- per cites another one. In addition, all papers are also associated with the year that the corresponding paper was published.
where MEAN - AGG denotes mean- aggregator, which average the embeddings of the aggregated nodes; neight denotes 1-hop paper-neighbors of initial node o; while neight denotes 1hhop neighbess of word node n; c and d are adaptive parameters based on the av- erage neighbor count of papers and words respec- tively, jneighaate word denotes the number of word neighbors of the initial node, and weigh de- notes the number of initial node neighbors of the word node.
Downstream Tasks: We evaluate our model on two types of tasks, namely subject prediction and important words identification.
This task is regarding pre- diction of subject areas of arXiv CS papers, which are manually labeled by the paper's 's au- thors and arXiv moderators. Formally, the task can be formulated 40-class classification problem.
Subject Prediction:
After aggregation, we normalize the aggregated initial node and word embeddings separately to obtain the updated embedding matrix Unitiaa and Umxd Then we replace the initial node features with the updated matrix, and continue to train the same R-GNN with self-supervised tasks until con- vergence.
Important Words Identification: This task is designed to identify important words based on paper-word correspondence. Formally, it can be regarded binary classification problem.
3.2 Pre-Training and Fine-Tuning Setup
Basically, pre-training methods are designed to ob- tain the transferable knowledge from unlabeled datasets, thus pre-training models bring better rep- resentations for the downstream tasks. Therefore. to evaluate the effectiveness of our method, we propose two different setups.
2.4 Multi- Task Pre-training
Link reconstruction task tends to restore structural features, while node attribute reconstruction task focuses on semantic information. IWI and IWR tasks focus on deep mining of important words. Therefore, we combine these tasks to guide the representation learning of the R-GNN model. Loss function L can be denoted as:
The first setting is called Transductive Learn- ing, we pre-train and fine-tur the-tune on the same graph in this setting, which means all nodes are visible during both pre-traing and fine-tuning stage. The second one is called Inductive Learning. pre- train on one grpah and fine-tune on another graph in this setting. Generating representation for un- seen nodes in fine-tuning stage makes this setting more challengeable.
C = LNAR + A1Cin + deCiwr + AaCiwn
where \1, A2, A3 are hyper-pasameters.
3 Experiment Setup
In this section, we apply our entire pre-training method to a real-world citation network oghn- arxiv(Hu et al., 2020a) and report performance on the downstream tasks. To demonstrate the general- ization of our method, we considered two settings on the same dataset: Transductive Learning and Inductive Learning. Meanwhile, we select sev- eral baseline models prove the validity of our method.
Dataset
Nodes
Edges
Avg.degree
Split Ratio
Ogbn-arxiv
169343
1166243
13.7
41/13/17/29
Table 1: Data Split of Ogbn-arxiv
We propose to split the dataset into four parts as shown in Table based on the publication dates of the papers to adapt to the pre-training settings, where 41% % of papers published before 2017: 13% of papers published in 2017: 17% of papers pub- lished in 2018; 29% of papers published since 2019. Specifically, the descriptions of the two settings are as follows:
3.1 Dataset
The ogbn-arxiv dataset is bench- mark node classification dataset, representing the citation network between all Computer Science (CS) arxiv papers indexed by MAG. Each node with its raw text of title and abstract is an arXiv
Data Profiling:
Transductive Learning: Under this setting, we select all papers to involve in the pre- pre-training stage; In the fine-tuning stage, we propose to

---

Setting
Graph
Papers
Words
Paper-Paper edge
Paper-Wone edge
Word-Word edge
Transductive Leurning
Pre-Train/FineTTue
169343
17634
2484941
21175983
7286638
Pre-Train
694999
17634
534337
8359451
3634502
Inductive Learning
Fine-Tune
99844
17634
927020
12816390
4081080
Table 2: Constructed Hierarchical Text-attributed Graph
Subject Prediction
Important Words Identification
Method
Accuracy(%)
ROC-AUC
Transductive
Inductive
Transductive
Inductive
Feat
59.17 + 0.06
60.05 0.01
0.7463 0.0012
0.7715 + 0.0008
Attribute Masking
65.32 = 1.98
61.96 2.48
0.7405 0.0055
0.7752 + 0.0052
Edge Prediction
64.94 = 2.04
62.46 = 2.08
0.7394 + 0.0072
0.7736 + 0.0049
DGI
70.34 = 0.16
63.66 0.04
0.7473 + 0.0062
0.7707 + 0.0014
GPT-GNN
68.45 + 2.54
66.04 2.09
0.7403 0.0078
0.7689 + 0.0111
GraphMAF
71.75 + 0.17
67.42 + 0.35
0.7475 + 0.0032
0.7812 + 0.0009
GIANT
72.46 + 0.07
68.89 0.06
0.7433 0.0008
0.7725 1 0.0012
NRUP
72.33 = 0.14
69.67 0.12
0.7515 0.0011
0.7847 0.0023
Table 3: Main result of Subject Prediction and Important Words Identification; we report Accuracy in task subject prediction and ROC-AUC task important word indentification(bllded number is the best in that column).
train papers published until 2017, validate those published in 2018, and test those published since 2019.
cess, we train the model to minimize the loss L using AdamW Optimizer and cosine learning rate decay without warmup. We provide an explanation of the hyper-parameter settings for different losses in Appendix D. More details and pypeme-prrmmters can found in Appendix A.
Inductive Learning: Under this setting, we propose to pre-train pre-train on papers published until 2016, train on those published in 2017, vali- date on those published in 2018, and test on those published since 2019.
3.4 Baseline Models
To verify the effectiveness of our method, select several baseline models for comparison.
For the evaluation protocol, we conduct the same experimental process under two settings. First, we train a R-GNN encoder by the proposed NRUP based pre-train graph. Then we freeze the pa- rameters of the encoder and generate all the nodes' embeddings for fine-tune graph. For evaluation, we train linear classifier and report the mean and standard deviation of performance on the test nodes through 10 random initializations.
Feat: Fixed representation of paper generated by the BERT model.
Attribute Masking: Mask the attributes of some nodes in the graph and reconstruct masked attributes by projection head.
Edge Generation: Mask some edges and generate them based on nodes and remaining edges, and optimize the by contrastive loss.
3.3 Implementation Details
DGI al., 2019): Maximizing mutual information between patch representa- tions and corresponding high-level summaries of graphs.
Construction of Hierarchical Text-attributed Graph: We construct pre-train and fine-tune hier- archical text-attributed graphs based on ogbn-arxiv according to two settings as shown in Table 2. The process time for construction can be found in the Appendix E.
GPT-GNN(Hu et al., 2020c): A genera- tive pre-training framework, which trains the model by reconstructing node attributes and graph structure through joint optimization of attribute generation and edge generation.
Basic Settings: In NRUP, R-GAT is selected the base model. We update the features of both papers and words with aggregator after training for 2000 epochs, then we input normalized updated features into the same R- GAT. Throughout the entire pro-
EraphMAF(HHu al., 2022): A generative supe-sud learning framework, which re- constructs initial node features by masked

---

graph autoencoder, and the model is opti- mized by reconstruction loss.
archical text-attributed graph. We conduct experi- ment under Inductive Learning setting using two basic self-supervised tasks to verify that embed- dings of words have indeed been optimized.
GIhNT(Chieen et al., 2021): An SSL frame- work that generates numerical node features with graph-structured self-supervision by XR- Transformer.
Self-Supervised Task
Accuracy
Bert Embedding
60.05 + 0.01
4
Experiment Results
Word Embedding of NAR
66.18 + 0.01
4.1 Main Results
Paper Embedding of NAR
67.32 + 0.04
Tabel presents the performance of applying differ- ent pre-trian methods the same pre-train dataset and fine-tune test set.
Word Embedding of LR
67.92 = 0.04
Paper Embedding of LR
66.87 = 0.10
In task Subject Prediction we predict subject paper based on paper representation, and in task Important Words Identification we identify im- portant words based on concatenation representa- tion of paper and word. More details can be found in Appendix B.
Table 5: Optimized Word Embedding (Word embedding means the average representation of word nodes, Paper embedding means the node representation output by RGAT)
We average the optimized word representation obtained by training for 2000 epochs through a certain task as the representations of downstream papers, then we train a linear classifier on the down- stream data to predict the field of the paper. The experimental results in Tabel 5 show that the word node embeddings are optimized well as the pa- per nodes, which is the reason why update framework works.
In both Transductive Learning and Inductive Learning setting, our NRUP achieves better or com- petitive performance compared to the selected base- line models, demonstrating the effectiveness and transferability of our method.
4.2 Effect of Different Tasks
The experimental results demonstrate that using multi-task loss for optimization can help the model capture both semantic and structural information in heterogeneous graph simultaneously.
4.4 Effect of Important Word Reconstruction
In task important word reconstruction, it is truc that a word may be "important" to some papers but not to others. However, even though the word decoupled from the title of particular paper may not have significant impact on another one, it is still part of the overall content and meaning of that paper.
We further investigated the performance on the test dataset using different tasks under Induc- live Learning setting without embedding update. Tabel shows the results that scenario-specific tasks can bring benefits to basic tasks, and our NRUP with multi-loss achieve the best perfor- mance.
We focus on reconstructing the important words in this task which contribute to the overall seman- tic understanding of a particular paper. other words, the less informative words which are not "important" to any paper can be disregarded.
Self-Supervised Task
Accuracy
Node Attribute Reconstruction(NAAR)
66.65 0.14
Link Reconstruccton((RR)
65.98 0.11
NAR+LR
67.31 0.07
NAR+IWI
66.77 0.12
Method
Accuracy
NAR+IWR
66.85 = 0.04
NAR+ AWR
65.49 + 0.07
LR+IWI
66. 10 0.11
NAR
66.65 + 0.14
LR+IWR
66.29 0.09
NAR+ IWR
66.85 + 0.04
NRUP
67.56 + 0.03
Table 4: Effect of Different Tasks
Table 6: Bfffect of Important Word Reconstruction
Further, we have conducted experiments regard- ing to all words reconstruction(AAWR instead of important words(IWR) under Inductive learning setting, and the findings of these experiments
4.3 Optimized Word Embedding
The optimization of word's embedding is a charac- teristic of our co-modeling method based on hier-

---

Table 6 indicate that it is more effective not to re- construct the word nodes unless we have specific preferences or criteria for word selection.
Rasiis Find
C=0.0
4.5 Ablation Studies
To verify the effects of the main components in NRUP, we further conduct several ablation studics. We choose explore under Inductive Learning set- ting.
(a)
(b) d
Figure 3: Effect of Adaptive Parameter Wheen exploring the hyperparameter set d the adaptive parameter and vice versa)
Effect of Update Architecture: We explore the influence of update architecture, and table shows the results that pre-train with the update architec- ture or not. Without the update component, use certain self-supervised tasks for end-to-end pre- training, and keep the optimal pre-training model to generate the embeddings for downstream dataset. And in our architecture, we update the features halfway through the pre-training and retain the op- timal model in the later stage. The performance on downstream dataset indicates that our framework is effective.
Related work
Representation Learning Text-attributed Graphs: rext-attributed graphs al., 2021) are rich in semantic and structural information, pre- vious studics on text-attributed graphs can be di- vided into three aspects: LMs only, GNNs only, combination of LMs and GNNs.
Early works leverage language models(Kim, 2014; Vaswani et al., 2017) to learn word repre- sentation based sentence sequence. However, neglect of the relationship between nodes leads to underutilization of structural information. In order to leverage the interrelation of nodes more effectively, the graph's structural configuration is employed as a complementary resource alongside textual data, with the aim of augmenting the train- ing process of language models(Yang et al., 2020; Mou et al., 2023). Besides, GIANT(Chien et al., 2021) propose train LM with graph structure, but message passing among nodes ignored in LM--baed methods.
Method
Accuracy
NRUP
69.67 0.12
w/o Update Architecture
67.56 0.03
Overall Normalization
68.16 0.18
w/o Paper Normalization
69.12 + 0.03
w/o Word Normalization
68.54 + 0.07
w/o Normalization
68.42 + 0.11
Table 7: Effect of Update Architecture
The normalization plays an crucial role in the update pre-training frame- work which brings the updated feature matrix back to normal distribution, eliminated the effect of dis- tribution transfer. Table 7 shows the results that update feature without normalization and with dif- ferent normalization ways. We found that the effect of normalizing the feature matrix is significantly better than not performing it. Meanwhile, the effect normalizing the feature matrices of the papers and words separately better than the effect of overall normalizing. In brief, normalization brings improvements.
Effect of Normalization:
The development of GNNs(Kipf and Welling, 2017: 7; Zhang and Chen, 2018) brings new ideas for studying this data format. In these works(Hu et al.. 2020a), the raw text of nodes are transformed to numerical features node attributes using graph- irrelevant methods(Mikklon) et al., 2013; Devlin et al., 2019). Nevertheless. representations for text are fixed in this situation, resulting in undermining of text information.
Co-training approaches(Bii et al., 2021; Zhu et al., 2021a) with combination of LMs and GNNs have advantages of both models. However, it suffer from issues of scalability due to the size of graph and parameters of LMs. Recently, variational EM framework(Zhao al., 2023) propose to alterna- tively update the LM and GNN, but it relies on task-specific labels thus the learned representation cannot be applied to other scenario.
Effect of Adaptive Parameter: Aggregators in NRUP in charge of updating node features. We further explore the way and degree of aggregation by using fixed hyper-parameters instead of adaptive pareee-prrmeterr for dataset. Figure shows the results that when the value of parammtessssserss is around adaptive parameters, the effect better.
Heterogeneous Graph Pre-training: There are

---

studies on related pre-training methods in the field of heterogencous graph, which leads to more gen- eral representation generated by R-GNN encoder. Jiang et al. (2021 l.b) proposed two heterogeneous graph pre-training frameworks: PT-HGNN and CPT-HG, in which PT-HGNN proposed two pre- training tasks at the node level and pattern level, while CPT-HG proposed two pre-training tasks at the relation level and subgraph level, both of which achieved good results. These pre-training methods helps the model acquire the representation with generalization and effcotiveness.
References
Nesreen K. Ahmed, Ryan A. Rossi, Rong Zhou, John Boaz Lee, Xiangnan Kong, Theodore L. Willke. and Hoda Eldardiry. 2017. Inductive representation learning in large attributed graphs.
Shuxian Bi, Chaozhuo Li, Xiao Han, Zheng Liu, Xing Xie, Haizhen Huang, and Zengxuan Wen. 2021. Leveraging bidding graphs for advertiser-aware rele- vance modeling in sponsored search. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 2215-2224, Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Y Hammerla. 2019, Relational graph attention networks. arXiv preprint arXiv:
6 Conclusion
In this work, we propose to learn representations of papers and words simultancously via co-modeling the raw text and graph based on hicrarchical text- attributed graph. We design pre-training frame- work and corresponding self-supereised tasks for this scenario. Sufficient experiments conducted on the benchmark dataset ogbn-arxit arxiv demonstrate the effectiveness and generality of our method.
Eli Chien, Wei-Cheng Chang, Cho-Jui ui Hsieh, Hsiang- Fu Yu, Jiong Zhang, Olgica Milenkovic, and In- derjit S Dhillon. 2021. Node feature extraction by supervupe multi-scale neighborhood prediction. arXiv preprint urXiv:2111.0006
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Zhenyu Hou, Xiao Liu, Yukuo Cen. Yuxiao Dong. Hongxia Yang. Chunjic Wang. and Jie Tang. 2022. Graphmae: Self-supervised masked graph autoen- coders.
Limitations
In our work, we propose to construct a hierarchi- cal text-attributed graph to realize connections be- tween nodes and words. However, the size of con- structed heterogeneous graph is proportional to the number of initial nodes, and the memory complex- ity is proportional to size of graph structure. There- fore, with the increase of paper-word edges, the memory cost of NRUP may become unaffordable, which limits the scalability of our method. Mean- while, saper-parameters of loss function may vary in different datasets. We will manage to address these issues in future work.
Jeremy Howard and Sebastian Ruder. 2018, Universal language model fine-tuning for text classification.
Weihua Hu, Matthias Fey, Marinka Zitikk Yuxiao Dong, Hongyu Ren. Bowen Liu, Michele Catasta, and Jure Leskovec. 2020a. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118 22133.
Weihua Hu, Bowen Liu, Joseph Gomes. Marinka Zitmik, Percy Liang. Vijay Pande, and Jure Leskovec, 2020b. Strategies for pre-training graph neural networks.
Ziniu Hu, Yuxiao Dong, Kuansan Wang. Kai- Chang. and Yizhou Sun. 2020c. Gpt-gnn: Gener- ative pre-training of graph neural networks. In Pro- ceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery Data Mining, pages 1857-1867
Ethics Statement
Our work strictly adheres to the ACL Ethics Policy, and the data in this paper comes from Open Graph Benchmark.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020d. Heterogeneous graph transformer.
Shuai Hua, Xinxin Li, Yunpeng Jing, and Qunfeng Liu. 2022. A semantic hierarchical graph neural network for text classification.
Acknowledgements
This work is supported by National Natural Sci- ence Foundation of China (No. 6217020551) and Science and Technology Commission of Shanghai Municipality Grant (N0.21QA1400600).
Xunqiang Jiang, Tianrui Jia, Yuan Fang, Chuan Shi. Zhe Lin, and Hui Wang. 2021a. Pre-training on large- scale heterogeneous graph. In Proceedings of the 27th ACM SIGKDD conference on knowledge discov- ery & data mining, pages 756-766.

---

Xunqiang Jiang, Yuanfu Lu, Yuan Fang, and Chuan Shi. 2021b. Contrastive pre-training of gnns on hetero- geneous graphs. In Proceedings of the 30th ACM International Conference on Information & Knowl- edge Management, pages 803-812.
Junhan Yang. Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. 2021. Graphform- ers: Gnn-nested transformers for representation learn ing on textual graph.
Yoon Kim. 2014, Convolutional neural networks for sentence classification.
Yuqiao Yang. Xiaoqiang Lin, Geng Lin, Zengfeng Huang, Changjian Jiang. and Zhongyu Wei. 2020. Joint representation learning of legislator and leg- islation for roll call prediction. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 1424-1430. International Joint Conferences on Artificial Intelli- gence Organization. Main track.
Thomas N. Kipf and Max Welling, 2017. Semi- supervised classification with graph convolutional networks.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7342-733 1351,, On- line. Association for Computational Linguistics.
Liang Yao. Chengsheng Mao, and Yuan Luo. 2019.
Graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370-7377.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- rado, and Jeffrey Dean. 2013. Distributed representa- tions of words and phrases and their compositional- ity.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. Linkbert: Pretraining language models with document links.
Yuning You, Tianlong Chen, Yang Shen, and and Zhangyang Wang. 2021. Graph contrastive learning automated. In International Conference on Machine Learning. pages 12121-12132. PMLR.
Xinyi Mou, Zhongyu Wei, Qi Zhang, and Xuanjing Huang. 2023. UPPAM: A unified pre-traing pre-training ar- chitecture for political actor modeling based on lan- guage. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Pupers), pages 1 1199-12012, Toronto, Canada. Association for Computational Linguisties.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812- 5823.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang. Ming Ding, Kuansan Wang, and Jie Tang. 2020. Gec: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1150- 1160.
Seongjun Yun, Minhyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo Kim. 2020. Graph trans- former networks.
Muhan Zhang and Yixin Chen. 2018. Link prediction based on graph neural networks.
Meng Qu, Yoshua Bengio, and Jian Tang. 2020. Gmnn: Graph markov neural networks.
Wentao Zhang, Ziqi Yin, Zeang Sheng. Yang Li, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang. and Bin Cui. 2022. Graph attention multi-layer percep- tron. In Proceedings of the 28th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining. ACM.
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2017. Modeling relational data with graph convolu- tional networks.
Chuxiong Sun, Hongming Gu, and Jie Hu.2021 2021. Seal- able and adaptive graph neural networks with self- label-enhanced training.
Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. 2023. Learning on large-scale text-attributed graphs via variational inference.
Shantanu Thakoor, Corentin Tallec, Mohammad Ghesh- laghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veličković, and Michal Valko. 2021. Large- scale representation learning graphs via bootstrap- ping. arXiv preprint 1r20v:2102.06514.
Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Tianqi Yang, Liangjie Zhang, Ruofei Zhang, and Huasha Zhao. 2021a. TexIGNN: Improv- ing text encoder via graph neural network in spon- sored search. In Proceedings of the Web Conference 2021. ACM.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun. Xue Li, Markus Pelger, Tianqi Yang, Liangiie Zhang, Ruofei Zhang, and Huasha Zhao. 202 TexIGNN: Improv- ing text encoder via graph neural network in spon- sored search. In Proceedings of the Web Conference 2021. ACM.
Petar Velickovie, William Fedus, William L Hamil- ton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2019. Deep graph infomax. ICLR (Poster), 2(3)4.

---

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu. and Liang Wang. 2020. Deep graph contrastive repre- sentation learning, arXiv preprint arX1::2006.04331,
A Pre-training Details
This section illustrates some details during the pre- training stage.
Update Architecture Setting: We design the ag- gregators update the paper and word features using optimized embedding. Specificly, update the features after 2000 epochs training, and train R- GAT with the updated representations for another 2000 epochs. In Table 8, we outline the hyper- parameters our update architecture.
Hyper-parameters
Value
epochs
4000
optimizer
Adam W
hidden layer dimension
768
layers of R-GAT heads of R-GAT
2
4
drop-out of R-GAT activation norm weight decay Ir
0.5
prelu
layerormm
le-4
0.001
Table 8: Hyper-parameters of Update Architecture
Self-supervised Tasks Setting: the frame- work, we select 2 basic self-supervised tasks: Feature Reconstruction and Link Prediction, and 2 scenario-specific tasks: Identify Important Words and Reconstruct Important Words. And we opti- mize the R-GAT model with the multi-loss of these tasks. Table 9 outline the hyper-parameters of self- supervised tasks.
In task node attribute reconstruction, we follow GraphMAE (Hou et al., 2022) and seleet one- layer R-GNN as the decoder, because the R-GNN decoder can recover the input features of one node based on set of paper nodes and word nodes in- stead of only the node itself, and it consequently helps the encoder learn high-level latent code. And the scaling cosine error can get rid of the impact of dimensionality and vector norms thus improves the training stability of representation learning.
B
Fine-tuning Details
This section illustrates some details during the fine- tuning stage.
Subject Prediction Setting: In this downstream task, we leverage the embeddings output by R- GAT for prediction. We train a linear classifier on the fixed representations of downstream dataset.

---

Decoder in FR layers of decoder in FR heads of decoder in FR Academic Graph (OAG) dataset, which contains more than 178 million paper nodes and 2.236 bil- lion edges. Each paper is labeled with a set of research topics/fields (e.g., Physics and Medicine) and the publication date ranges from 1900 to 2019.
Hyper- Hlype--prrametrs
Value
negative to positive ratio LP edge type of negative LP
Paper-Paper R-GAT
mask rate of FR mask edge rate of FR Loss function FR
0.5
Method
Paper-Field
Paper-Venue
0.5
GPT-GNN
12.22 1.02
16.72 + 0.99
sce
GraphMAE
43.77 0.12
47.81 + 0.18
? FR dimension PH in IIW layers of PH in IIW mask rate of RIW
GIANT
44.59 0.07
48.98 + 0.09
768
NRUP
45.01 + 0.13
49.92 + 0.12
2
0.2
Table 12:
Hyper-parameters Update Architecture
Table 9: Hyper- parameters of Self-supervised Tasks
Due our limited computing resources, we only sampled 0.5% of the nodes and corresponding edges for the hierarchical text-attributed graph con- struction($50000 nodes) and pre-trainin pre-training. For the fine-tuning step, we consider the edge prediction of Paper-Field, Paper- Venue as two downstream tasks, Table 12 shows our results.
Tabel 10 outline the hyper-parameten of this task.
sapper-parameters
Value
epochs of linear
2000
optimizer of linear Ir of linear dimension of linear
adamW W
0.01
D
The hyper-parameters for the coefficients of different losses
768
classification number
40
During the pre-training process of ogbn-arxiv, we consider node attribute reconstruction as the funda- mental self-supervised task. This is because seman- tic information plays a crucial role in text-attributed graphs.
Table Hyper-parameters of Subject Prediction
Important Words Identification Setting: In this downstream task, we leverage concatenation paper and word embedding output by R-GAT for prediction. For baseline models, we use the concatenation of paper embedding output by GNN and initial bert embedding for prediction. We train a linear classifier the concatenation of represen- tations of downstream dataset. Tabel 11 outline the hyper-parameters of this task.
Also, in experiments, we observed that set- ting the values of λ1 A2; Å3 (which are hyperpa- rameters controlling the relative importance dif- ferent loss terms) to be around 0.1 to 0.2 yielded good results.
Empirical results indicate that sufficient mining of semantic information and appropriate learning of structural information can enable graph neural networks to acquire more useful transferable knowl- edge that beneficial for downstream tasks.
Hyper-parameters
Value
epochs of linear optimizer of linear
1000
adamW W
Ir of linear dimension linear
0.01
E
Cost of Graph Construction
1536
classification number 2
We run the construction process of hierarchical text- attributed graph to record the time complexity. The running time of construction process under two different settings as follows:
Table 11: fication
of Important Words Identi-
Transductive Learning: 7.5hr
C Other Experiments
Inductive Learning: 3.5hr
addition to the ogbn-arse arxiv dataset in this paper, we have conducted extra experiment on the Open
The experimental results indicate that the construc- tion step is affordable terms of time complexity.

---

Furthermore, it is worth highlighting that the construction step of the hicrarchical text-attributed graph only needs to be performed once and con- structed graphs are large-scale with 10-30 million edges. Once the graph is constructed, it can be utilized for both the pre- pre-training step and multiple downstream tasks.


Reviews:
[{"review_id": "qq3csZL94M", "reviewer": "Reviewer_y7tX", "summary": "", "questions": "* Are there experiment results on other dataset or other downstream tasks?\n* In Important Word Reconstruction, a word may be important to some papers but not to others. Wasn't it an issue when pre-training?\n* The hyper-parameters for the coefficients of different losses need to be clarified.\n* No edge weights are used. Any reasons?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer y7tX for their positive comments and valuable time. We address all concerns as follows.\n\n### Q1: Other experiments\nThanks for the suggestion! We have conducted extra experiment on the Open Academic Graph (OAG) dataset, which contains more than 178 million paper nodes and 2.236 billion edges. Each paper is labeled with a set of research topics/fields (e.g., Physics and Medicine) and the publication date ranges from 1900 to 2019.   \nDue to the limited time and our limited computing resources, we only sampled 0.5% of the nodes and corresponding edges for the hierarchical text-attributed graph construction(850000 nodes) and pre-training. For the fine-tuning step, we consider the edge prediction of Paper-Field, Paper-Venue as two downstream tasks. Our results are as follows:\n\n|Method|Paper-Field|Paper-Venue|\n|:------:|:------:|:------:|\n|GPT-GNN|$42.22 \\pm 1.02$|$46.72 \\pm 0.99$|\n|GraphMAE|$43.77 \\pm 0.12$|$47.81 \\pm 0.18$|\n|GIANT|$44.59 \\pm 0.07$|$49.28 \\pm 0.09$|\n|**NRUP**|$\\bf{45.01 \\pm 0.13}$|$\\bf{49.92 \\pm 0.12}$|\n\nThe results show that our NRUP achieves better performance compared to the competitive  baselines, demonstrating its effectiveness on a different dataset. Also, these downstream tasks are evaluated on **link-level(graph structure)**. We will update the final result in our revision in the future.\n\n### Q2: Baseline of LM and GNN\nFirst, we respectfully disagree with the comment that \"None of the baselines use LM\". In fact, baseline GIANT uses LM for pre-training. And our approach NRUP achieves competitive or even better performance compared to that.  \nSecond, the reason why we didn’t use the method that uses the combination of LM and GNN is that such an approach is effective when graphs are small. However, it becomes problematic once the graphs become very large.   \nThis is because the size of the computational graph is proportional to the size of the graph structure between nodes as well as the language model capacity, and the memory complexity is proportional to the size of the observed graph and the number of parameters in the LM. So this kind of method is not suitable under limited computing resources.\n\n### Q3: About “The downstream tasks are sort of replaying some of the objectives used in NRUP.”\nIt is worth emphasizing that we avoid performing our self-supervised tasks on the test part of downstream tasks in the pre-training step, ensuring that testdata objectives have not appeared during the pre-training step. In transductive learning, NRUP does not perform the task “Important Word Identification” on the test portion of downstream task; and in inductive learning, our approach involves pre-training and fine-tuning on two distinct graphs.   \nIt is common that downstream tasks share the same objectives with pre-training step in real world applications. For example, downstream link prediction task in biology$^{[1]}$ or chemistry$^{[2]}$ often involves link reconstruction during pre-training$^{[3]}$. Massive unlabeled data in pre-training provide inherent structure information in graphs.\n\n> [1] Stanfield, Z., Coşkun, M. & Koyutürk, M. Drug Response Prediction as a Link Prediction Problem. Sci Rep 7, 40321 (2017). https://doi.org/10.1038/srep40321   \n[2] Lu, Y., Guo, Y. & Korhonen, A. Link prediction in drug-target interactions network using similarity indices. BMC Bioinformatics 18, 39 (2017). https://doi.org/10.1186/s12859-017-1460-z  \n[3] Hu, Weihua, et al. \"Strategies for pre-training graph neural networks.\" arXiv preprint arXiv:1905.12265 (2019).\n\n\n### Q4: Issue of Important Word Reconstruction\nThis is a very good question! In this task, it is true that a word may be \"important\" to some papers but not to others. However, even though the word decoupled from the title of a particular paper may not have a significant impact on another one, it is still a part of the overall content and meaning of that paper.  \nWe focus on reconstructing the important words in this task which contribute to the overall semantic understanding of a particular paper. In other words, the less informative words which are not \"important\" to any paper can be disregarded.  \nFurther, we have conducted experiments regarding to all words reconstruction instead of important words under the same setting in Section 4.2, and the findings of these experiments indicate that it is more effective to not reconstruct the word nodes unless we have specific preferences or criteria for word selection.\n|Method|Accuracy(inductive learning)|\n|:------:|:------:|\n|NAR+ All Word Reconstruction|$65.49 \\pm 0.07$|\n|NAR|$66.65 \\pm 0.14$|\n|NAR+ Important Word Reconstruction|$66.85 \\pm 0.04$|\n\n### Q5: The hyper-parameters for the coefficients of different losses.\nThanks for the question! In our method, we consider node attribute reconstruction as the fundamental self-supervised task. This is because semantic information plays a crucial role in text-attributed graphs.   \nAlso, in our experiments, we observed that setting the values of $\\lambda_{1},\\lambda_{2},\\lambda_{3}$ (which are hyperparameters controlling the relative importance of different loss terms) to be around 0.1 to 0.2 yielded good results.  \nWe will include an explanation of hyper-parameters coefficients in the revision.\n\n### Q6: Edge weights\nThis is a very good question! We agree that edge weights often provide important information about the strength or significance of relationships between nodes. However, in the case of the ogbn-arxiv dataset, there is no available information on edge weights. More specifically, the dataset does not provide any specific measurements or values indicating the strength or intensity of the \"cite\" relationship between papers.  \nSince this information is not provided, it becomes difficult for us to accurately assign meaningful weights to the edges.  \nAs a result, we refrained from using edge weights in our method. We will further explore datasets with edge weights information and verify the effectiveness of our method on them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Overall, the paper is easy to read (although there are some confusing terms like \"original/initial nodes\").\n* The proposed framework can be generally applied to various LM and GNN models.", "reasons_to_reject": "* It is unclear whether the proposed framework is actually effective to capture the co-dynamics of graph structure and text attributes. The pre-training objectives seem to heavily rely on the textual attributes, and the second-order relationships of nodes are not considered at all, which has been known as very important to be able to well understand graph structure. Also, the two downstream tasks used for evaluation are related to only node attributes. There needs another tasks that evaluate on graph structure as well.\n* Only one dataset is used in the experiment, which makes the proposed idea sound less convincing.\n* The experiments do not seem fair. None of the baselines use LM. At least, one method that uses LM and GNN should have been compared together. Also, The downstream tasks are sort of replaying some of the objectives used in NRUP.", "questions_to_authors": "* Are there experiment results on other dataset or other downstream tasks?\n* In Important Word Reconstruction, a word may be important to some papers but not to others. Wasn't it an issue when pre-training?\n* The hyper-parameters for the coefficients of different losses need to be clarified.\n* No edge weights are used. Any reasons?", "paper_topic_and_main_contributions": "This papers proposes a pre-training framework for learning representations of nodes and words simultaneously in a text-attributed graph. The proposed framework, called NRUP, first constructs a hierarchical test-attributed graph where a set of nodes is composed of text elements (e.g., papers in citation networks) and words and they are connected to each other via citations or co-occurrences. The node representations are learned via four different self-supervised tasks. During pre-training, they are further optimized by aggregation and normalization. After pre-training, they are fine-tuned to perform downstream tasks. To demonstrate its effectiveness, the ogbn-arxiv dataset is used and two different downstream tasks are used for evaluation. The experimental results show the proposed method can achieve competitive performances compared to the baseline methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Figure 2, the figure numbers do not match their corresponding descriptions. (e.g., \"2. Self-supervised Tasks\" should be \"3. Self-supervised Tasks\")"}, "extra_text": {}}, {"review_id": "6K9XpBFVVO", "reviewer": "Reviewer_52xC", "summary": "", "questions": "A.In the experimental part, the authors only selected one dataset (Ogbn-arxiv) to test the performance of the model in the downstream tasks. Can this prove the universality of the model? Why do the authors use more datasets?\nB. The authors point out in the abstract that the existing work has a large memory cost problem, so how do you solve the memory cost problem? Do you need to compare the space complexity?\nC. What does the column \"words\" mean? Why are they the same in three rows?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 52xC for their positive comments and valuable time. We address all concerns as follows.\n### Q1: Other experiments  \nThanks for the suggestion! We have conducted extra experiment on the Open Academic Graph (OAG) dataset, which contains more than 178 million paper nodes and 2.236 billion edges. Each paper is labeled with a set of research topics/fields (e.g., Physics and Medicine) and the publication date ranges from 1900 to 2019.   \nDue to the limited time and our limited computing resources, we only sampled 0.5% of the nodes and corresponding edges for the hierarchical text-attributed graph construction(850000 nodes) and pre-training. For the fine-tuning step, we consider the the prediction of Paper-Field, Paper-Venue as two downstream tasks. Our results are as follows:\n\n|Method|Paper-Field|Paper-Venue|\n|:------:|:------:|:------:|\n|GPT-GNN|$42.22 \\pm 1.02$|$46.72 \\pm 0.99$|\n|GraphMAE|$43.77 \\pm 0.12$|$47.81 \\pm 0.18$|\n|GIANT|$44.59 \\pm 0.07$|$48.98 \\pm 0.09$|\n|**NRUP**|$\\bf{45.01 \\pm 0.13}$|$\\bf{49.92 \\pm 0.12}$|\n\nThe results show that NRUP achieves better performance compared to the competitive  baselines, demonstrating its effectiveness on other dataset. We will update the final result in our revision in the future.\n### Q2: Memory cost problem  \nThanks for the suggestion! Existing work suffers from the memory cost problem due to training language model(LM) or co-training LM and GNN in the pre-training step. However, in our NRUP, we use a pre-trained LM to generate the initial embedding for the word nodes and original nodes, and we exclusively rely on graph neural networks (GNN) during pre-training, which significantly reduces the number of parameters compared to LM-based methods.   \nThe parameters of different methods are as follows:\n\n|Method|Params|\n|:------:|:------:|\n|GPT-GNN|456192|\n|GraphMAE|300288|\n|GIANT|>7000000|\n|**NRUP**|748032|\n\nNote that the parameters of our method NRUP is slightly larger than GNN-based methods due to the additional word nodes, but significantly less than LM-based methods GIANT. It shows that our method is affordable in terms of space complexity. \n### Q3: The meaning of \"words\"  \nIn table 2, the column \"words\" indicates the number of word nodes in hierarchical text-attributed graph. To ensure the generality and effectiveness of our method, we use the **same tokenizer** to decouple words from the original text, which means words decoupled from the corpus are **shared**. Therefore, there is an equal number of word nodes in all three graphs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The paper introduces a hierarchical text-attributed graph construction and a multi-task graph pretraining framework, which address the underutilization of relationships between nodes and words.  \n2.Experimental results show the proposed method is superior to baseline methods both in terms of evaluation metrics and efficiency.\n\n", "reasons_to_reject": "1. Only one dataset is selected for the experiment.\n", "questions_to_authors": "A.In the experimental part, the authors only selected one dataset (Ogbn-arxiv) to test the performance of the model in the downstream tasks. Can this prove the universality of the model? Why do the authors use more datasets?\nB. The authors point out in the abstract that the existing work has a large memory cost problem, so how do you solve the memory cost problem? Do you need to compare the space complexity?\nC. What does the column \"words\" mean? Why are they the same in three rows?", "paper_topic_and_main_contributions": "The main contributions of this research are as follows:\n1. Constructed a hierarchical text-attributed graph that incorporates both original nodes and word nodes, which can capture the finer nuances of the text at a more granular level.\n2. Introduced a multi-task graph pre-training framework that encompasses various self-supervised tasks to adapted to various scenarios.\n3. Employ a relational graph neural network (R-GNN) as the foundational model for acquiring knowledge from the hierarchical text-attributed graph to mutually boost representations of nodes and words.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RnhaJNeuVT", "reviewer": "Reviewer_TszY", "summary": "", "questions": "None.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer TszY for their positive comments and valuable time. We address all concerns as follows.\n\n### Q1: Cost of graph construction\nThis is a very good question! We re-run the construction process of hierarchical text-attributed graph to record the time complexity. The running time of construction process under two different settings are as follows:  \n* Transductive Learning: 7.5hr\n* Inductive Learning: 3.5hr  \n\nThe experimental results indicate that the construction step is affordable in terms of time complexity. Furthermore, it is worth highlighting that the construction step of the hierarchical text-attributed graph only needs to be **performed once** and constructed graphs are large-scale with 10~30 million edges. Once the graph is constructed, it can be utilized for both the pre-training step and multiple downstream tasks.\n\n### Q2: Typo and Mismatched indices\nThanks for pointing out the typo and the mismatched indices. We will correct them in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The writing is clear and easy to follow.\n2. The construction of hierarchical text-attributed graph is reasonable and novel.\n3. The proposed pre-training framework encompasses various self-supervised tasks to ensure the generalization ability.", "reasons_to_reject": "Minor mistakes:\n1. The construction of hierarchical text-attributed graph seems to be costly.\n2. Typo in Abstract: L.18 \"diffferent\" -> \"different\".\n3. The numerical indices in the caption of Figure 2 are inconsistent with the display.", "questions_to_authors": "None.", "paper_topic_and_main_contributions": "This paper focuses on text-attributed graph representation learning. The authors argue that existing works suffer from issues like underutilization of relationship between nodes or words or unaffordable memory cost. In this paper, a Node Representation Update Pre-training Architecture (NRUP) is proposed based on Co-modeling Text and Graph. In NRUP, a hierarchical text-attributed graph is constructed to incorporates both original nodes and word nodes, and four self-supervised tasks is applied for pre-training. The experimental results on ogbn-arxiv dataset demenstrate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "Please refer to Reasons To Reject."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a pretraining framework for learning representations of nodes in text-attributed graphs, applicable to GNNs and other models of structured data. The authors demonstrate that their proposed approach is competitive with or marginally better than previous approaches, using node classification in scientific graphs as the test case. 

The primary flaw identified by reviewers was the decision by the authors to evaluate on only one of the standard benchmarks for the task. This weakened the empirical arguments in the paper. Since the initial submission, the authors have in their rebuttal provided another set of results, significantly strengthening the paper.

As the framework is quite general and appears to yield performance improvements (albeit only to a small degree), the method presented here may find wider applicability.
Decision: Accept-Findings