Learning on Large-scale Text-attributed Graphs via Variational Inference
Authors: Jianan Zhao
Meng Qu
Chaozhuo Li
Hao Yan
Qian Liu
Rui Li
Xing Xie
Jian Tang
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Language Model
Graph Neural Network
Node Classification

Abstract: This paper studies learning on text-attributed graphs (TAGs), where each node is associated with a text description. An ideal solution for such a problem would be integrating both the text and graph structure information with large language models and graph neural networks (GNNs). However, the problem becomes very challenging when graphs are large due to the high computational complexity brought by training large language models and  GNNs together. In this paper, we propose an efficient and effective solution to learning on large text-attributed graphs by fusing graph structure and language learning with a variational Expectation-Maximization (EM) framework, called GLEM. Instead of simultaneously training large language models and GNNs on big graphs, GLEM proposes to alternatively update the two modules in the E-step and M-step. Such a procedure allows training the two modules separately while simultaneously allowing the two modules to interact and mutually enhance each other. Extensive experiments on multiple data sets demonstrate the efficiency and effectiveness of the proposed approach.

Published as a conference paper at ICLR 2023
LEARNING ON LARGE-SCALE TEXT-ATTRIBUTED GRAPHS VIA VARIATIONAL INFERENCE
Jianan Zhao 1. Meng Qu Chaozhuo Li Hao Yan', Qian Liu', Rui Li Xing , Jian Tang".7.8 Mila- - Québec Al Institute, Microsoft Research Asia, Université de Montréal *Central South University, SSea Al Lab, Dalian University of Technology HEE Montréal, Canaddian Institute for Advanced Research (CIFAR)
ABSTRACT
This paper studies learning text-atrributed graphs (TAGs), where each node is associated with text description. An ideal solution for such a problem would be integrating both the text and graph structure information with large language mod- els and graph neural networks (GNNs). However, the problem becomes very chal- lenging when graphs are large due to the high computational complexity brought by training large language models and GNNs together. In this paper, we pro- pose efficient and effective solution to learning large text-attributed graphs by fusing graph structure and language learning with a variational Expectation- Maximization (EM) framework, called GLEM. Instead of simultaneously training large language models and GNNs on big graphs, GLEM proposes to alternatively update the two modules in the E-step and M-step. Such procedure allows train- ing the two modules separately while simultaneously allowing the two modules to interact and mutually enhance each other. Extensive experiments on multiple data sets demonstrate the efficiency and effectiveness of the proposed approach TT
INTRODUCTION
Graphs are ubiquitous in the real world. In many graphs, nodes often associated with text at- tributes, resulting in text-attributed graphs (TAGs) (Yang et al. 2021). For example, in social graphs, each user might have text description; in paper citation graphs, each paper is associated with its textual content. Learning on TAG has become an important research topic in multiple areas includ- ing graph learning, information retrieval, and natural language processing.
In this paper, we focus on a fundamental problem, learning effective node representations, which could be used for variety of applications such node classification and link prediction. Intuitively, a TAG is rich in textual and structural information, both of which could be beneficial for learning good node representations. The textual information presents rich semantics characterize the prop- erty of each node, and one could use a praitrained language model (LM) (e.g., BERT (Devlin et al.) 2019)) as text encoder. Meanwhile, the structural information preserves the proximity between nodes, and connected nodes are more likely to have similar representations. Such structural rela- tionships could be effectively modeled by graph neural network (GNN) via the message-passing mechanism. In summary, LMs leverage the local textual information of individual nodes, while GNNs use the global structural relationship among nodes.
An ideal approach for learning effective node representations is therefore to combine both the tex- tual information and graph structure. One straightforward solution is to cascade an LM-based text encoder and GNN-based message-pasee method suffers from severe scalability issues. This is because the memory complexity is propor- tional to the graph size neighborhood texts also encoded. Therefore, real-world TAGs where nodes are densely connected, the memory cost of this method would become unaffordable.
module and train both modules together. However, this
To address such problem, multiple solutions have been proposed. These methods reduce either the capacity of LMS or the size of graph structures for GNNs. More specifically, some studies choose to
"The first two authors contributed equally. 'Corresponding authors.
Codes available at https: / //i whtb.ccom/andyy///a/////

---

Published as a conference paper at ICLR 2023
fix the parameters of LMs without fine-tuning them (Liu et al. 2020b). Some other studies reduce graph structures via edge sampling and perform message gaasaggg only on the sampled edges (Zhu et all, 2021 Li et al. 2021a Yang et al.) 202T). Despite the improved scalability, reducing the LM capacity or graph size sacrifices the model effectiveness, leading to degraded performance of learning effective node representation. Therefore, we are wondering whether there exists scalable and effective approach to integrating large LMs and GNNs on large text-attributed graphs.
In this paper, we propose such an approach, named Graph and Language Learning by Expectation Maximization GLEM. In the GLEM, instead of simultaneously training both the LMs and GNNs. we leverage a variational EM framework (Neal & Hinton, 1998) to alternatively update the two modules. Take the node classification task as an example. The LM uses local textual information of each node to learn a good representation for label prediction, which thus models label distributions conditioned on text attributes. By contrast, for a node, the GNN leverages the labels and textual encodings of surrounding nodes for label prediction, and it essentially defines a global conditional label distribution. The two components are optimized to maximize variational lower bound of the log-likelihood function, which can be achieved by alternating between an E-step and an M-step, where at each step we fix one component update the other one. This separate training framework significantly improves the efficiency of GLEM, allowing it to scale up to real-world TAGs. In each step, one component presents pseudo-labels of nodes for the other component to mimic. By doing this, GLEM can effectively distill the local textual information and global structural information into both components, and thus GLEM enjoys better effectiveness in node classification. We con- duct extensive experiments on three benchmark datasets to demonstrate the superior performance of GLEM. Notably, by leveraging the merits of both graph learning and language learning, GLEM- LM achieves on par or even better performance than existing GNN models, GLEM-GNI achieves new sfate-of of oh-thhe-rt art results on ogbn-arxiv, ogbn-produee and ogbn-papers100m.
RELATED WORK
Representation learning text-attributed graphs (TAGs) (Yang et al. 2021) has been attracting growing attention in graph machine learning, and one of the most important problems is node clas- sification. The problem can be directly formalized as a text representation learning task, where the goal is to use the text feature of each node for learning. Early works resort to convolutional neural networks (Kim 2014 Shen et al.. 2014) or recurrent neural networks Tai et al. (2015). Recently, with the superior performance of transformers ( (Vaswani et al. (017) and pre-trained language mod- els (Devlin et al., 2019 Yang et al. 2019). LMs have become the go-to model for encoding con- textual semantics in sentences for text representation learning. At the same time, the problem can also be viewed as graph learning task that has been vastly developed by graph neural networks (GNNs) (Kipf & Welling, 2017: Velickovic et al.) 2018 Xu et al. [009 /Zhang & Chen 2018 Teru al. 2020). GNN takes numerical features as input and learns node representations by transform- ing representations and aggregating them according to the graph structure. With the capability of considering both node attributes and graph structures, GNNs have shown great performance in var- ious applications, including node classification and link prediction. Nevertheless, LMs and GNNs only focus on parts of observed information (i.c., textual or structural) for representation learning, and the results remain to be improved.
There are some recent efforts focusing on the combination of GNNs and LMs, which allows one to enjoy the merits of both models. One widely adopted way is to encode the texts of nodes with fixed LM, and further treat the LM embeddings as features to train a GNN for message passing. Recently, few methods propose utilize domain-adaptive pretraining GGururaggan al. (020) on TAGs and predict the graph structure using LMs (Chien et al. 2022 Yasunaga et al. 2022) to provide better LM embeddings for GNN. Despite better results, these LM embeddings still remain unlearnable in the GNN training phase. Such separated training paradigm ensures the model scalability as the number of trainable parameters in GNNs relatively small. However, its performance is hindered semantic modeling process. To overcome these limitations, endeavors have been made (Zhu et al. 2021 Li et al., 202 Yang et al. 2021 Bi et al. 2021: Pang et al. 2022) to co-train GNNs and LM under joint learning framework. However, such co-training approaches suffer from severe scalability issues as all the neighbors need to be encoded by language models from scratch, incurring huge extra computation costs. In practice, these models restrict the besssag--assiigg to very few, e.g. 3 (Li et al. 2021a Zhu et al. 202I), sampled first-hop neighbors, resulting in severe information loss.
by the task and topology-ireelvane
2

---

Published as a conference paper at ICLR 2023
To summarize, existing methods of fusing LMs and GNNs suffer from either unsatisfactory results or poor scalability. In contrast to these methods, GLEM uses a seeudd-iikelhoodd variational frame- work to integrate LM and GNN, which allows the two components to be trained separately, leading to good scalability. Also, GLEM encourages the collaboration of both components, so that it is able enjoys better effectiveness.
both textual semantics and structural semantics for representation learning, and thus
Besides, there are also some efforts using GNNs for text classification. Different from our approach, which uses GNNs to model the observed structural relationship between nodes, these methods as- sume graph structures are unobserved. They apply GNNs on synthetic graphs generated by words in text (Huang et al., 2019; ( Hu et al. [2019] ZZhang et al.) (020) or co-occurrence patterns between texts and words (Huang et al. 2019] Liu et al. (020a). As structures are observed the problem of node classification in text-attributed graphs, these methods cannot well address the studied problem.
Lastly, our work is related to GMNN Qu et al. (2019), which also uses a pseudo- likelihood varia- tional framework for node representation learning. However, GMNN aims to combine two GNNs for general graphs and it does not consider modeling textual features. Different from GMNN, GLEM focuses TAGs, which are more challenging to deal with. Also, GLEM fuses a GNN and an LM, which can better leverage both structural features and textual features in a TAG, and thus achieves state-of-tee of-thh--rtttttttt results on few benchmarks.
BACKGROUND
In this paper, focus learning representations for nodes in TAGs, where take node classifi- cation as an example for illustration. Before diving into the details of our proposed GLEMM, we start with presenting a few basic concepts, including the definition of TAGs and how LMs and GNNs can be used for node classification in TAGs.
3.1 TEXT-ATTRIBUTED GRAPH
Formally, a TAG Gs = A, sy) ) is composed of nodes V and their adjacency matrix A E RIV VV1, where each node E V is associated with sequential text feature (sentence) Sn. In this paper, we study the problem of node classification on TAGs. Given a few labeled nodes yz of LCV V, the goal is to predict the labels yo for the remaining unlabeled objects U VAL. L.
Intuitively, node labels can be predicted by using either the textual information or the structural information, and representative methods are language models (LMs) and graph neural networks (GNNs) respectively. Next, we introduce the high-level ideas of both methods.
3.2 LANGUAGE MODELS FOR NODE CLASSIFICATION
Language models aim to use the sentence S, of each node n for label prediction, resulting text classification task (Socher et al., 2013 Williams et al. 2018). The workflow of LMs can be characterized as below:
py(y.(s.)) Cat(y, softmax (h,))); h = SeqEnne, (Sa),
where SeqEnc,, is a text encoder such as a transformer-basee model (Vaswani et al. 2017 Yang al., 2019), which projects the sentence s, into vector representation h,, Afterwards, the node label distribution yn can be simply predicted by applying an MLP', with softmax function to h,
Leveraging deep architectures and pre-training on large-scale corpora, LMs achieve impressive re- sults on many text classification tasks (Devlin et al.) [019 Liu et al,, 2019). Nevertheless, the memory cost often high due to large model sizes. Also, for each node, LMs solely use its own sentence for classification, and the interactions between nodes are ignored, leading to sub-optimal results especially on nodes with insufficient text features.
3.3 GRAPH NEURAL NETWORKS FOR NODE CLASSIFICATION
Graph neural networks approach node classification by using the structural interactions between nodes. Specifically, GNNs leverage pesssag--psssing mechanism, which can be described
Ps (y (A) Cat(yn softmax(h(")));
h (MSG, (t-1)
A)),
(2)
3

---

Published as a conference paper at ICLR 2023
where & denotes the parameters of GNN, is activation function, MSGa(.) and AGG stand for the message and aggregation functions respectively, NB(n) denotes the neighbor nodes of n.
Given the initial text encodings (0) e.g. pre-trained LM embeddings. a GNN iteratively updates them by applying the message function and the aggregation function, so that the learned node rep- resentations and predictions well capture the structural interactions between nodes.
With the message-passing mechanism, GNNs are able to effectively leverage the structural informa- tion for node classification. Despite the good performance many graphs, GNNs not able to well utilize the textual information, and thus GNNs often suffer on nodes with few neighbors.
output
imput
Gruphh Structure
Text Embedding
M-Step
target
GNN training
Pseudo-labet by LM
Pseudo-label by GNN
E-Step
LM training
Text Artribute
Figure 1: The proposed GLEM framework trains GNN and LM separately in variational EM framework: In E-step, an LM is trained towards predicting both the gold labels and ENN-predicted pseudo-labels; In M-step, GNN is trained by predicting both gold labels and LM-inferred pseudo- labels using the embeddings and pseudo-labels predicted by LMM.
METHODOLOGY
In this section, we introduce our proposed approach which combines GNN and LM for node rep- resentation learning in TAGs. Existing methods either suffer from scalability issues or have poor results in downstream applications such as node classification. Therefore, we are looking for an approach that enjoys both good scalability and capacity.
Toward this goal, we take node classification as an example and propose GLEM. GLEM leverages a variational EM framework, where the LM uses the text information of each sole node to predict its label, which essentially models the label distribution conditioned on local text attribute; whereas the GNN leverages the text and label information of surrounding nodes for label prediction, which characterizes the global conditional label distribution. The two modules are optimized by alternating between E-step and M-step. In the E-step, fix the GNN and let the LM mimic the labels inferred by the GNN, allowing the global knowledge learned by the GNN to be distilled into the LM. In the M-step, we fix the LM, and the GNN is optimized by using the node representations learned by the LM as features and the node labels inferred by the LM as targets. By doing this, the GNN can effectively capture the global correlations of nodes for precise label prediction. With such a framework, the LM and GNN can be trained separately, leading to better scalability. Meanwhile, the LM and GNN are encouraged benefit each other, without sacrificing model performance.
4.1 THE PSEUDO-LIKELIHOOD VARIATIONAL FRAMEWORK
Our approach is based pseudo-likelihood variational framework, which offers principled and flexible formalization for model design. To be more specific, the framework tries to maximize the log-likeidooo function of the observed node labels, i.e., p(yz/sv. A)) Directly optimizing the func- tion often hard due to the unobserved node labels yu, and thus the framework instead optimizes the evidence lower bound below:
log plyplsyy lsv,,, A) 2 Bytyer (logsss/llooypp// Yo (sv. A) - log g [s:)]
where 4(vr (st) is a variational distribution and the above inequality holds for any q. The ELBO can be optimized by alternating between optimizing the distribution q (i.e., E-step) and the distribution p
4

---

Published as a conference paper at ICLR 2023
(i.e., M-step). In the E-step, aim at updating to minimize the KL divergence between y (sy) and : so that the above lower bound can be tightened. In the M-step, we then update
p towards maximizing the following ggeudo-lkkelihon (Besag 1975) function:
R (og yulsv, A)] Ro (y(uuls)) logp(yn/sv: A,yv)))
(4)
NEV
The pseudo-likelihodd variational framework yields formalization with two distributions max- imize data likelihood. The two distributions are trained via a separate E-step and M-step, and thus longer need the end-to-end training paradigm, leading to better scalability which naturally fits our scenario. Next, we introduce how we apply the framework to node classification in TAGs by instantiating the p and q distributions with GNNs and LMs respectively.
4.2 PARAMETERIZATION
The distribution g aims use the text information Stz define node label distribution. In GLEM, we use a mean-field form, assuming the labels of different nodes are independent and the label of each node only depends on its own text information, yielding the following form of factorization:
qe gy
II nEU
qe (yn ss)).
As introduced in Section 3.2, each term qo (yyy|ss)) can be modeled by a transformer- -based LM qo parameterized by 0, which effectively models the fine-grained token interactions by the attention mechanism (Vaswani et al. 2017).
On the other hand, the distribution defines conditional distribution P& (yn A, yv aiming to leverage the node features 8v, graph structure A, and other node labels yv to characterize the label distribution of each node Such formalization the message- passsnng mechanism. Thus, we model A. yv..) as a GNN ps parameterized by effectively model the structural interactions between nodes. Note that the GNN pa takes the node texts Sy as input to output the node label distribution. However, the node texts are discrete variables, which cannot be directly used by the GNN. Thus, in practice we first encode the node texts with the LM qe. and then use the obtained embeddings as surrogate of node texts for the GNN Pd-
be naturally captured by GNN through
In the following sections, we further explain how we optimize the LM qo and the GNN to let them collaborate with each other.
4.3 E-STEP: LM OPTIMIZATION
In the E-step, we fix the GNN and aim update the LM maximize the evidence lower bound. By doing this, the global semantic correlations between different nodes can be distilled into the LM.
Formally, maximizing the evidence lower bound with respect the LM is equivalent to mini- mizing the KL divergence between the posterior distribution and the variational distribution, i.c., (y)))s)ss)ll
A. yL)). However, directly optimizing the KL divergence is nontrivial, as the KL divergence relies on the entropy of qo (yulsu)) ). which is hard to deal with. To overcome the challenge, we follow the wake-sleep algorithm (Hinton et al. 1995) to minimize the reverse KL. divergence, yielding the following objective function to maximize with respect to the LM qo:
- A.y = E Ep (yo (ooooolss (log qe (so))s))) + const
= Σ E le..yv.lee.A.y [log go (yn ss.)) + const,
(6)
neu
which is tractable as longer need consider the entropy of (o (sy). Now, the sole difficulty lies computing the distribution Po (yn|sv, y Remember that in the original GNN which defines the distribution pyyy//sv A, (z).,, we aim to predict the label distribution of node based on the surrounding node labels yv However, in the above distribution P& (y /sy, A, y.), we only condition on the observed node labels yı. and the labels of other nodes are unspecified, so cannot compute the distribution directly with the GNN. In order to solve the problem, we propose
5

---

Published as a conference paper at ICLR 2023
to annotate all the unlabeled nodes in the graph with the pseudo-labels predicted by the LM, that we can approximate the distribution as follows:
|sv A,) A,
where S = (8nc En with each Yn ~ go (s.)
Besides, the labeled nodes also be used for training the L.M. Combining it with the above objec- tive function, we obtain the final objective function for training the LM:
( = a Σ
+ (1 a) Σ nEL
log g(yn
(8)
REU
where a is hyperparameter. Intuitively, the second term EnE,, log ((y../a)) ) is supervised objec- tive which uses the given labeled nodes for training. Meanwhile, the first term could be viewed as knowledge distilling process which teaches the LM by forcing it to predict the label distribution based on neighborhood lext-information.
4.4 M-STEP: GNN OPTIMIZATION
During the GNN phase, we aim at fixing the language model go and optimizing the graph neural network Ps to maximize the pseudo-likelihore as introduced in equation
To be more specific, we use the language model to generate node representations hy for all nodes and feed them into the graph neural network as text features for message passing. Besides, note that equation relies on expectation with respect to q0. which can be approximated by drawing a sample Yo from (yu (s In other words, we use the language model qe to predict pseudo-label Yn for each unlabeled node n E U, and combine all the labels IeE into fv. With both the node representations and pseudo-labels from the LM qo. the pseudo-likelihood can be rewritten as follows:
(Ø) = 3 Σ logpo( A.y.:,..ls,,,,........ Surr) + (1 - B) Σ log A, VLy. nEL
REU
where 3 is hyperparameter which is added to balance the weight of the two terms. Again, the first term can be viewed knowledge distillation process which injects the knowledge captured by the LM into the GNN via all the pseudo-labels. The second term is simply a supervised loss, where we use observed node labels for model training.
Finally, the workflow of the EM algorithm is summarized in Fig. | The optimization process iteratively does the E-step and the M-step. the E-step, the pseudo-labels predicted by the GNN together with the observed labels are utilized for LM training. In the M-step, the LM provides both text embeddings and pseudo-labels for the GNN, which are treated as input and target respectively for label prediction. Once trained, both the LM in E-step (denoted GLEM-LM) and the GNN (denoted as GLEM-GNN) in M-step can used for node label prediction.
EXPERIMENTS
In this section, conduct experiments to evaluate the proposed GLEM framework, where two set- tings are considered. The first setting transductive node classification, where given a few labeled nodes in TAG, we aim classify the of the nodes. Besides that, also consider structure- free inductive setting, and the goal to transfer models trained on labeled nodes to unseen nodes, for which we only observe the text attributes without knowing their connected neighbors.
5.1 EXPERIMENTAL SETUP
Datasets. Three TAG node classification benchmarks used in experiment, including ogbn- arxiv, ogon-pro, and ogbn-papers 100M (Hu et al. 2020). The statistics of these datasets are shown in Table [|
Compared Methods. We compare GLEM-LM and GLEM-GNN against LMs, GNNs, and meth- ods combining both of worlds. For language models, apply DeBERTa He al. (021 to

---

Published as a conference paper at ICLR 2023
Table 1: Statistics of the OGB datasets (Hu et al. 2020).
#Nodes
#Edges
Avg. Node Degree
Train Val / Test (%)
ogbn-arxiv (Arxiv)
169,343 2,449,029
1,166,243
13.7
/ 8 / 2 90
ogbn-products (Products)
61,859,140
50.5
ogbn- poon-paperslee 100M (Papers)
111,059,956
1,615,685,872
29.1
78 / 8 14
Node classification accuracy for the Arxiv and Products datasets. (mean + std%, the best results are bolded and the runner-ups are underlined). G t denotes the improvements of GLEM- GNN over the same GNN trained on XOOB: L 1 denotes the improvements of GLEM-LM over LM-Ft. "+" denotes additional tricks implemented in the original GNN models.
Table 2:
Datasets
Methods
GNN
Xora
XCIANV
XxLM
GLEM-GNN
G
LM-Ft
GLEM-LM
Lt
val
73.00
0.17
74.89 a: 0.17
47.56 = 1.91
76.86 + 0.19
3.86
75.27 = 0.09
0.44
0.90
GCN
71.74 0.29
73.29
48.19 1.47
0.113 0.416
4.19
74.13 0.04
75.21
1.58
72.77
0.16
0.11
76.45 0.05
3.68
75.27 = 0.09
75.32
SAGE
71.49
0.227
56.39 0.13
0.25
4.01
74.13
7..13
1.44
Arxiv
GAMLP
Text
val
56.53
0.11
0.00
0.14
75.01
7..15
0.11
75.62
76.95 =
14.75
19.09
74.13+0.00
15.27=0.00
7.364
74.48
0.44
2.04
RevGAT
Test
val
74.02
0.10
0.18
77.01
0.
0.09
07.49 0.23
70.21
0.30
76.97 + 0.19
2.48
2.95
74.13
7
0.04
0.075
0.
1.32
val
91.99 = 0.07
0.14
86.74 0.31
93.84 = 0.12
1.85
91.82 = 0.11
92.71
0.71
SAGE
79.21 0.15
82.33
71.09 0.65
83,16 0.19
3.95
79.63 = 0.12
0.
1.61
Products
GAMPP
Test
val
93.12 = 0.03
83.54 = 0.09
83.16
93,99 + 0.04
91.65 0.17
80.49
85,09 + 0.21
94.19 0.01
1.07
1.55
91.82 = 0.11
79.63 = 0.12
90.56
0.223
-1.26
2.60
93.02
93.64
92.78
0.03
91.82 0.11
SAGN+
Test
84.35 a: 0.09
86.67 +: 0.09
84.20 = 0.39
87.36 0.07
3.01
79.63 a 0.12
84.83
5.17
71.17
72.70 0.07
69.78 0.07
71.71 + 0.09
0.54
68.05 = 0.03
69.94
1.89
GAMLP
67.71 0.20
69.33 0.06
65.94
0,10
68. + 0.14
0.54
63.52 0.06
64.80 0.06
1.78
Papers
71.59 0.05
73.05 0.04
69.87 0.06
73.54 0.01
1.95
68.05
0.03
71.16
3.11
GAMLP+
Text
68.25 + 0.11
69.67 + 0.05
66.36 = 0.09
70.36 # 0.02
2.11
63.52 + 0.06
66.71 + 0.25
3.19
setting by fine-tuning it on labeled nodes, and we denote it as LM-Ft. For GNNs, few well- known GNNs are selected, i.e., GCN (Kipf & Welling, 2017) and GraphSAGE (Hamilton et al,, included, i.e. RevGAT (Li et al. 20211). GAMLP (Zhang et al. 2022). SAGN (Sun & Wu 202T). For each GNN, we try different kinds of node features, including (1) the feature of OGB, denoted XoGn: (2) the LM embedding inferenced by pre-trained I.M, i.e. the DeBERTa-bass checkpoint denoted as XPLM: (3) the GI-
2017). Three top-ranked baselines
leaderboards
ANT (Chien al. 2022) feature, denoted XGIANT-
Implementation Details. We adopt the DeBERTa (He et al. 2021) as the LM model and fine-tune for node classification provide initial checkpoints for LMs and further infer text embeddings and predictions for the first GNN M-step. To provide predictions for the first-LM E-step, we use pre-trained GNN predictions, e.g. the original GNN predictions, for the initial target labels. The best EM-iteration is chosen based on the validation accuracy of GLEM-GNN, During optimization, GLEEM can start with either the E-step or the M-step. For better performance, we let the better module generates pscudo-labels and train the other module first. For example, if pre-trained GNN outperforms pre-trained LM, we start with the E-step (LM training). For fair comparison against other feature learning methods such as GIANT, the hypee-paramettrs of GNNs are set to the best settings described in the paper or in the official repository, other parameters are tuned by grid search.
5.2 TRANSDUCTIVE NODE CLASSIFICATION
Main Results. Next, we evaluate GLEM in the transductive setting. The results of three OGB datasets are presented in Table 2 For LMs, we see that fine-tuned LMs (LM-Ftt have competitive results, showing the importance of text attributes in TAG. By further leveraging the structural information for message passing, our approach (GLEM-LM) achieves significant improvement over LMs, which demonstrates its advantage over LMs.
For GNN-based methods, we see that for each GNN model, using OGB and GIANT node embed- dings as GNN inputs (XOGB and XGIANT) yields strong results. However, these embeddings remain unchanged during training. By dynamically updating the LM to generate more useful node embed-
httts::/huggingge:eeet// www.c/ccc.omoommo co/mi www.cct

---

Published as a conference paper at ICLR 2023
Table 3: Experiments Arxiv (RevGAT GNN backhone) with different scale of L.M.
Methods
Val. accuracy
Test accuracy
# Parameters
GNN-XOGB
75.01 = 0.10
74.02
2,098,256
GNN-XGIANT
77.01 + 0.09
75.90 0.19
1,304,912
GLEM-GNN--bas
17.49
1,835,600
GLEM-GNN-large
77.92
77.62
2,228,816
LM-base-Ft
75.27
74.13 0.04
138,632,488
LM-large-Ft
75.08 + 0.06
73.81
405,204,000
GLEM-LM-bard
75.75
75.45
138,632,488
GLEM-LM-largg
77.16
76.80 0.05
405,204.008
Table 4: Structure free inductive experiments on ogbn-arxiv and ogbn-products. The validation and test accuracies, denoted w/ struct and wo struct, and their relative differences (the lower the better) are reported. Boldfaced numbers indicate the best performances in each group.
Type
Methods
Arxiv
Products wo struct
w/ struct
wo struct
diff
w/ struct
diff
XoGa
57.65 + 0.12
55.50 = 0.23
-2.15
75.54 + 0.14
61.06 + 0.08
-14.48
X
74.56 0.01
72.98 = 0.06
-1.58
91.79 # 0.01
79.93 + 0.22
-11.86
MLP
XiLtM-light
75.20 0.03
73.32 = 0.31
-1.88
91.96 + 0.01
79.38 0.14
-12.58
Xeep
75.57
73.90 0.08
-1.67
91.85 + 0.02
80.04 0.15
-11.11
Xoon-light
70.73
48.59
-22.14
90.54 0.04
51.23 0.17
-39.31
X Xight
0.02
0.09
-2.79
92.95 0.03
78.75
-14.20
GNN
Xoca-
72.67
0..92
-21.55
9
32.71 + 2.23
-59.14
XCLEM- -depp
7.079
7.119
-2.50
93.22 0.03
7.001
-1341
Fine-tune
75.27 # 0.09
7.01
-1.14
91.82 0.11
7
-12.19
LM
GLEM-llght
75,49 0.11
-0.99
91.90 0.06
79.53
-12.37
GLEMM-depp
75.59 a 0.08
74.60 0.05
-0.99
91.81 + 0.04
79.69 + 0.51
-12.12
dings and pseudo-labels for the GNN, GLEM-GNN significantly outperforms all the other methods with fixed node embeddings in most cases. Notably, GLEM-GNN achieves new state-of-the-art results on all three TAG datasets on the OGB benchmark.
Scalability. One key challenge of fusing LMs and GNNs lies in scalability. When using large LMs with numerous parameters, the combined method will suffer from severe scalability problems. GLEM cases this challenge through the EM-based optimization paradigm, allowing it be adapted to large LMs. To verify this claim, we train GLEM with DeBERTa-large (He et al., 2021) on ogbn-arxiv. The results reported in Table 3 We observe that GLEM is able to generalize to DeBERTa-large with about 0.4B parameters, showing the appealing scalability. Besides, for every LM, applying GLEM yields consistent improvement, which proves the effectiveness of GLEM.
5.3 STRUCTURE-FREB INDUCTIVE NODE CLASSIFICATION
Besides the transductive setting, inductive settings also important, where we aim to train models on nodes of a training graph, and then generalize models to unobserved nodes. In many real cases, these new nodes often low-degree nodes use the structural information for node classification. Therefore, we consider a challenging setting named structure-free inductive setting, where we assume for each test node, we only observe its text attributes without any connected neighbors. For this setting, we consider different types of meth- ods for label prediction, including GNN models (GNN), neural networks without using structural information (MLP), and GLEM. The results are shown in Table
isolated nodes, meaning that can hardly
We can see that the structure-fres inductive setting more challenging task, especially for GNNs where a sheer performance drop is observed. Meanwhile, by effectively fusing with graph learning, GLEMM-LM is able to consider local semantics as well as neighboring structural information, leading to more accurate structure-frre inductive predictions. Besides, the generated embeddings are able to boost other models (e.g., see XGLEM-deep in the MLP and LM sections), enabling both MLP and GNNs with better fructure-free inference ability.
8

---

Published as a conference paper at ICLR 2023
Table 5: Comparison of different training paradigms of fusing LM and GNNss The maximum batch size (max bsz.) and time/epoch are tested on a single NVIDIA Tesla V100 32GB GPU.
LM-FFt
Static SAGE-Xoca
Joint
GLEM
Datasets
Metric
DeBERTa-bod base
Joint- BERT-dny
GraphFomeers
GLEM-GNN
GLEM-LM
val, acc.
75.27 2: 0.09
72.77 + 0.16
71.58 0.18
73.33 + 0.06
76.45 + 0.05
75.32 a 0.04
test acc.
74.13 0.04
71.49 + 0.27
70,87 # 0.12
72.81 # 0.20
75,50 0.24
74,53 # 0.12
parameters
138,632,488
218,664
110,694,592
110,694,592
545,320
138,632,488
Anxiv
max bsz.
30
all nodes
200
180
all nodes 0.13s
30
time/epoch
2760s
0.09s
1827s
4824s
3801s
val, acc. acc..
91.82 0.11
91 11.99-m + 0.07
90.85 # 0.12
91.77 0.09
93.84 + 0.12
92.71 # 0.15
79.21
0.11
0.16
81,25 UL15
parameters
138.637,871
206.895 all nodes 8.1s
10.6999.775
110,699,975
548,911
138,637,871
Products
max bsz.
30 5460s
100
100
80000
30
timo/epoch
8456s
12574s
153s
7740s
GLEM GNN -O- GLEM-LM
78.0
72.0
88.0
Set
77.0
g
08.0
70.0
Tex Set.
84.0
86.0
75.0
66.0
82.0
74.0
64.0
80.0
73.0
62.0
78.0
0
2
0
2
0
2 4
ogbn-arxiv
ogbn-pand papers 100M
ogbn-prood bobn-proddccs
Figure 2: The convergence curves of GLEM on OGB datasets.
5.4 COMPARISON DIFFERENT TRAINING PARADIGMS
As discussed before, besides directly fine-tuning LM (denoted LM-Ft), few training paradigms have been proposed for fusing GNNs and LMs. One paradigm is to use fixed/static LMs to generate node embeddings for GNN to do label prediction (denoted Static). Besides that, another paradigm to restrict message passing to a few sampled neighbors, so that the memory cost can be reduced (denoted as Joint), In this section, we compare our proposed paradigm (GLEM) against the others. For each paradigm, we choose two models trained with it. The results are presented in Table 5 we that although static training has the best efficiency, its classification accuracy is not very high due to the restricted model capacity caused by the fixed LM. On the other hand, the joint training paradigm has the worst efficiency and effectiveness due the reduced graph structure. Finally, our proposed paradigm achieves the optimal classification results, thanks to its ability to encourage the collaboration of LMs and GNNs. Meanwhile, our paradigm remains close static training in terms of efficiency (time/epoch). To summarize, proposed approach achieves much better results than other paradigms without sacrificing efficiency.
5.5 CONVERGENCE ANALYSIS
In GLEM, we utilize the variational EM algorithm for optimization, which consists of an E-step training GLEM-LM and M-step training GLEM-GNN in each iteration. Here, we analyze the convergence of GLEM by looking into the training curves of validation accuracy on ogbn-arxiv and OGB-Products. From the results in Fig. We can clearly see that with each E-step and M-step, both the performance of GLEM-GNN and the GLEM-LM consistently increase to maximum point and converge in a few iterations. Notably, GLEM takes only one iteration converge on the ogbn-arxiv dataset, which is very efficient.
6 CONCLUSION
This paper studies how to fuse LMs and GNNs together for node representation learning in TAGs. We propose approach GLEM based on Secudo-likelihood variational framework. GLEM alter- natively updates the LM and GNN via an E-step and an M-step, allowing for better scalability. In each step, both GNN and LM are mutually enhanced by learning from pseudo-labels predicted by the other module, fusing graph and language learning together. Extensive experiments on multiple datasets in two settings demonstrate the effectiveness and efficiency of GLEM.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This project is supported by Twitter, Intel, the Microsoft Research Asia internship program, the Nat- ural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bird Gift Fund, a NRC Collab- orative R&D Project (AI4D- AI4D-CORE-06) as well as the IVADO Fundamental Research Project grant 11--00993353139722
REFERENCES
Julian Besag. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society: Series D (The Statistician), 24(3):179-195, 1975.
Shuxian Bi, Chaozhuo Li, Xiao Han, Zheng Liu, Xing Xie, Haizhen Huang, and Zengxuan Wen. Leveraging bidding graphs for advertiser-aware relevance modeling in sponsored search. In EMNLP, pp. 2215-2224, 2021.
Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovic, and Inderjit S. Dhillon. Node feature extraction by self-supervised multi-scale neighborhood predic- tion. In ICLR, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACI, 2019.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In ACL, 2020.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. In NeurlPS, 2017.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: decoding-enhanced bert with disentangled attention. In ICLR, 2021.
Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal. The" wake-sleep" algorithm for unsupervised neural networks. Science, 268(5214): 1161, 1995.
Linmei Hu, Tianchi Yang, Chuan Shi, Houye Ji, and Xiaoli Li. Heterogeneous graph attention networks for semi-super short text classification. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4821-4830, 2019.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning graphs. In NeurIPS, 2020.
Lianzhe Huang, Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. Text level graph neural network for text classification. arXiv preprint arXiv:1910.02356. 2019.
Yoon Kim, Convolutional neural networks for sentence classification. In EMNLP, 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Chaozhuo Li, Bochen Pang, Yuming Liu, Hao Sun, Zheng Liu, Xing Xie, Tianqi Yang, Yanling Cui, Liangjie Zhang, and Qi Zhang. Adsgnn: Behavior-gras graph augmented relevance modeling in sponsored search. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR, 2021a.
Guohao Li. Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In ICML., 2021b.
10

---

Published as a conference paper at ICLR 2023
Xien Liu, Xinxin You, Xiao Zhang, Ji Wu, and Ping Lv. Tensor graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8409-8416 2020a.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. 2019.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. Fine-grained fact verification with kernel graph attention network. ACI, 2020b.
Radford M. Neal and Geoffrey E. Hinton. A view of the em algorithm that justifies incremental, sparse, and other variants. Learning Graphical Models, pp. 355-368. Springer Netherlands, 1998.
Bochen Pang. Chaozhuo Li, Yuming Liu, Jianxun Lian, Jianan Zhao, Hao Sun, Weiwei Deng, Xing Xie, and Qi Zhang, Improving relevance modeling via heterogeneous behavior graph learning in bing ads. In KDD, pp. 3713-3721, 2022.
Meng Qu, Yoshua Bengio, and Jian Tang. GMNN: graph markov neural networks. In ICML, 2019.
Yelong Shen, Xiaodong He. Jianfeng Gao, Li Deng, and Grégoire Mesnil. Learning semantic rep- resentations using convolutional neural networks for web search. In wwW, 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Chuxiong Sun and Guoshi Wu. Scalable and adaptive graph neural networks with self-label- enhanced training. CoRR, abs/2104.09376, 2021.
Kai Sheng Tai, Richard Socher, and Christopher Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00005 2015.
Komal Teru, Etienne G. Denis, and William L. Hamilton, Inductive relation prediction by sub- graph reasoning. In ICML, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In NAACL, June 2018.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanic Jegelka. How powerful graph neural networks? ICLR, 2019.
Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning textual graph. In NewrIPS, 2021.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NewrIPS, 2019.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. Linkbert: Pretraining language models with document links. In ACI, 2022.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In NeurIPS, 2018.
Wentao Zhang, Ziqi Yin, Zeang Sheng, Yang Li, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, and Bin Cui. Graph attention multi-layer perceptron. In KDD, 2022.
11

---

Published as a conference paper at ICLR 2023
Yufeng Zhang, Xueli Yu, Zeyu Cui, Shu Wu, Zhongzhen Wen, and Liang Wang. Every docu- ment owns its structure: Inductive text classification via graph neural networks. arXiv preprint arXiv:2004.13826, 2020.
Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Tianqi Yang, Liangjie Zhang, Ruofei Zhang, and Huasha Zhao. Textgnn: Improving text encoder via graph neural network in sponsored search. In WWW, 2021.
12

---

Published as a conference paper at ICLR 2023
77.0
76.0
Tost Ace (%)
75.0
Test Acc (%)
73.0
75.0
71.0
0
69.0
0 0.133 2
4..45555 6 7
9
0 1 2
6 7 .8
(a) LM-PL--eighh
(b) GNN-PL PL. MANN-PL-Weight B
Figure 3: The effect of the and for GLEM-GCN.
GLEM-GNN 0 GLEM-IM
77.0
77.0
76.0
Test Ace
76.0
Test Acc
75.0
75.0
71.0
74.0
0.1
1 3
5 6
.8
.9
0
.05
.25
.5
.75 1
(a) SMM-II-Weeihht
(b) GNN- BONN-PL--VEE L--------WWe-- PL-Weight /
Figure 4: The effect of the a and / for -LEM-R.vGAT.
A SENSITIVITY ANALYSIS
Recall that GLEM fuses LMs and GNNs by training them separately with an E-step and an M-step, where at each step both observed labels and pseudo-labels are used for model training as shown in 8 and Eq For both objectives, we introduce coefficient, i.e., and (3) denoted as LM- PL.-weight and GNN-PL--weigh respectlively, to control the relative weight of pseudo-labels. Next, systematically analyze these two hyperparameters by investigating how the performance varies when changing the hyperparamcters. We treat GCN (Kipf & Welling 2017) and Rev RevGAT (Li al. 2021b) the backbone GNNs. The results on the OGBN-Arxiv dataset shown in Figure and Figure 4 respectivel..
clearly that the LM-PL--eight is an important parameter, GLEM-LM provides both node feature and pseudo-labels for GLEM-GNN. We also observe that guiding LM with pseudo labels of GNN consistently boosts the performance of both LM and GNN compared with optimizing LM with gold label only, i.c. knowledge of GNN into LM. On the other hand, the GNN-PL--eight that balance the important of LM pseudo-labels in training GLEM-GNN is not very sensitive. Lastly, we also observe that for different GNN and LM, the optimal and | varies, indicating these parameters should be carefully selected.
On one hand,
0, demonstrating the effectiveness of fusing the
B REPRODUCIBILITY STATEMENT
We provide our code in a public repository along with detailed instructions conducting experi- ments. Our experimental settings and implementation details are stated in Section 5.1 the important hyper-parameters are discussed in the Appendix.
13


Reviews:
[{"review_id": "OKHIN4CkY6", "reviewer": "Reviewer_qotH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a variational expectation maximization framework to jointly train the language model and the graph neural network for representation learning for text-attributed graphs. The approach is clearly motivated, mathematically principled, and empirically effective and efficient. I believe this paper makes a clear contribution to graph representation learning and opens new possibilities for scaling graph representation learning to even larger graphs and language models. ", "summary_of_the_review": "This paper proposes a variational expectation maximization framework to jointly train the language model and the graph neural network for representation learning for text-attributed graphs. The method novel and smart. Experiments demonstrate its effectiveness, efficiency and scalability. I believe this paper would make a valid contribution to the community. ", "strength_and_weaknesses": "**Strength**\n\n**Clear motivation**: this paper is clearly motivated to target on the problem of joint learning LM and GNN efficiently. \n\n**Mathematically principled**: the variational EM framework is elegant and very smart. Although variational EM is very well studied and already used for learning GNNs, I think the application of variational EM in this paper is still novel. \n\n**Solid experiment**: the experiment is conducted well, with solid comparison with GNNs and LMs and joint methods.The improvements are also clearly marked in Table 2. \n\n**Efficiency and scalability**: The efficiency (quick enough training time) and scalability (setting larger batch size) is also clear in table 5. \n\n**Weakness**\n\n**Multiple approximation happens in training:** the authors have clearly discussed all the places where approximation happens, which on itself is a good thing. Yet all these approximations (mean-field, weak-sleep, pseudo label, balancing parameters .etc) intuitively makes the final objective deviate very much from the original variational bound (or not sure if the final objective is a bound anymore). Although being orthodoxical to the original variational objective may not be practically good, it would still be mathematically more principled if the authors could establish more rigid analysis of the final objective (e.g., its relationship to the likelihood).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FCdAWBfxQAL", "reviewer": "Reviewer_5qjT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a methodology to perform classification on text-attributed graphs. This is an important problem that arises in many practical applications where the nodes of the graph are associated with some textual information. The paper studies a methodology that combines pre-trained language models (LM) and GNNs. The proposed framework, called GLEM, is trained using a variational EM algorithm. Specifically, fixing the GNN model which characterizes the global label distribution, the parameters of the LM are updated in the E-step. Then, in the M-step, the parameters of the GNN are optimized. The paper puts emphasis on the choice of the likelihood function to deal with scalability constraints. The proposed methodology is evaluated on several node classification tasks.", "summary_of_the_review": "Overall, I believe it’s an interesting formulation of the node classification task on text-attributed nodes. Nevertheless, I still have some concerns about the absence of baseline models. This point needs further clarification from the authors.", "strength_and_weaknesses": "**Strengths:**\n* Interesting formulation of GLEM. Jointly optimizing the LM and the GNN within a variational EM algorithm constitutes an elegant formulation of the problem.\n\n* The paper has performed experiments on large-scale datasets.\n\n**Weaknesses:**\n* The paper has mostly examined datasets that contain textual features. However, similar settings arise while dealing with the task of text classification using graph-based models. This problem, although it is highly relevant to the formulation studied here, it is not discussed at all in the paper. In the past, several approaches have been introduced for graph-based text classification —  none of them is mentioned in the paper. For instance, one of the very first papers for this task “Graph Convolutional Networks for Text Classification” at AAAI ’19, follows a similar formulation where a graph composed of document nodes is constructed. Is there any specific reason why such methodologies have not been used?\n\n* Despite targeting scalability, the paper does not discuss the convergence of the proposed model. The training step is quite complex, therefore studying the convergence of the model is important.\n\n* The settings of structure-free inductive learning are not very clearly presented. Since GLEM assumes that there is a graph structure that captures the relationships among text nodes, it is not straightforward to extend it to an inductive setting. How is this problem tackled here?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F3i_2BUrzfc", "reviewer": "Reviewer_oKS1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new method for node classification where nodes are associated with text attributes. The proposed framework consists of a language model (LM) and a graph neural network (GNN) and uses the variational EM algorithm that learns LM and GNN in the E and M steps respectively. The proposed method is evaluated on several large-scale graphs with text attributes in comparison with different LMs and GNNs used separately. ", "summary_of_the_review": "The variational EM formulation is interesting but the quality of the paper needs to be improved.\n\nThe author provided detailed information to my concerns of clarity. I've increased my rating.", "strength_and_weaknesses": "Strength:\n\n1. The general idea of this paper is performing joint training for LM and GNN and letting them help each other during training. This might be done with heuristic approaches. But in this paper, it is interesting to wrap up the alternative training of LM and GNN into an EM framework, which adds credit to the technical depth of the paper.\n\n2. The datasets and the baselines are well selected and the experiments are comprehensive in general including different settings and comprehensive ablation studies.\n\nWeaknesses:\n\n\n1. Some of the performance improvement of the proposed method is a bit marginal. For example, in Table 2, GLEM-GNN (the proposed method) is marginally better than the second best (X_{GIANT}) on arxiv and products (e.g., 75.90 ± 0.19 VS 76.74 ± 0.08 with RevGAT), not mentioning that GLEM-GNN did not outperform X_{GIANT} on papers.\n\n2. Several clarity issues (please see comments in the next section).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8yy7LlAUmu", "reviewer": "Reviewer_5Tau", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose the node representation method GLEM for Text-Attributed Graphs (TAG) based on a pseudo-likelihood variational framework.\n\nSpecifically, they alternatively update GNN (M-step) and LM (E-step) with pseudo-labels from each other.\n\nIn experiments, they validate their method achieves SOTA on large-scale TAG while maintaining a high level of efficiency.", "summary_of_the_review": "To my knowledge, this paper has sufficient merits to be accepted.\n\nFor reproducibility, there should be more information provided about implementation and experimental settings.\n\nPlease be aware that since I am unfamiliar with the works on representation learning of TAGs, I may have missed significant prior works.", "strength_and_weaknesses": "### Strengths\n\n- **Simple but powerful idea;** the authors prove that their method outperforms previous baselines on three benchmark datasets. This work has enough impact on the field of representation learning on TAG.\n\n### Weaknesses\n\n- **Lack of ablation studies;** In E-step and M-step, there are hyperparameters that balance the weight of two terms. However, I fail to find any design choice or ablation study on this. In addition, more details (optimizer, learning rate, batch size, …) should be described in the paper (at least appendix) to fully reproduce the methods and experiments in this work.\n\n### Suggestions\n\n- Questions\n    \n    - What is the difference between GLEM-LM and GLEM-GNN? Do they differ based on the starting point of optimization (E-step or M-step first)? \n    \n- Typos & suggestions\n    - Table 3: Dataset name (ogbg-papers) is missing in Table 3.\n    - Table 4: In Arxiv-MLP experiments, the boldface on diff is wrong. (on -1.67 instead of -1.58", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%