Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning
Authors: Xiaoxin He
Xavier Bresson
Thomas Laurent
Adam Perold
Yann LeCun
Bryan Hooi
Venue: ICLR 2024 poster (2024)
Keywords: large language models (LLM)
feature learning
text attributed graphs (TAG)
graph neural networks (GNN)
Primary area: learning on graphs and other geometries & topologies
TL;DR: We propose the first framework that leverages LLMs to enhance representation learning on text-attributed graphs, achieving SOTA results on four benchmark datasets.

Abstract: Representation learning on text-attributed graphs (TAGs) has become a critical research problem in recent years. A typical example of a TAG is a paper citation graph, where the text of each paper serves as node attributes. Initial graph neural network (GNN) pipelines handled these text attributes by transforming them into shallow or hand-crafted features, such as skip-gram or bag-of-words features. Recent efforts have focused on enhancing these pipelines with language models (LMs), which typically demand intricate designs and substantial computational resources. With the advent of powerful large language models (LLMs) such as GPT or Llama2, which demonstrate an ability to reason and to utilize general knowledge, there is a growing need for techniques which combine the textual modelling abilities of LLMs with the structural learning capabilities of GNNs. Hence, in this work, we focus on leveraging LLMs to capture textual information as features, which can be used to boost GNN performance on downstream tasks. A key innovation is our use of \emph{explanations as features}: we prompt an LLM to perform zero-shot classification, request textual explanations for its decision-making process, and design an \emph{LLM-to-LM interpreter} to translate these explanations into informative features for downstream GNNs. Our experiments demonstrate that our method achieves state-of-the-art results on well-established TAG datasets, including \texttt{Cora}, \texttt{PubMed}, \texttt{ogbn-arxiv}, as well as our newly introduced dataset, \texttt{tape-arxiv23}. Furthermore, our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on \texttt{ogbn-arxiv}. Lastly, we believe the versatility of the proposed method extends beyond TAGs and holds the potential to enhance other tasks involving graph-text data~\footnote{Our codes and datasets are available at: \url{https://github.com/XiaoxinHe/TAPE}}.

Published as a conference paper at ICLR 2024
HARNESSING EXPLANATIONS: LLM-TO-LM INTER- PRETER FOR ENHANCED TEXT-ATTRIBUTED GRAPH REPRESENTATION LEARNING
Xiaoxin He', Xavier Bresson', Thomas Laurent', Adam Perold Yann LeCun 1.5, Bryan Hooi National University of Singapore, *Loyola Marymount University EElement, Inc., "New York University, MMea AI (xiaoxin, xaviercs, bhooi] ( @omp. nus edu..dusgg tlaurent emwee edu ap@elementresearch. com, yannêcs .yu.edu
ABSTRACT
Representation learning on text-attributed graphs (TAGs) has become critical re- search problem in recent years. A typical example of TAG is a paper citation graph, where the text of each paper serves as node attributes. Initial graph neu- ral network (GNN) pipelines handled these text attributes by transforming them into shallow hand-crafted features, such skip-gram or bag-of-words features. Recent efforts have focused on enhancing these pipelines with language models (LMs), which typically demand intricate designs and substantial computational re- sources. With the advent of powerful large language models (LLMs) such GPT or Llama2, which demonstrate an ability to reason and to utilize general knowl- edge, there is growing need for techniques which combine the textual modelling abilities of LLMs with the structural learning capabilities of GNNs. Hence, in this work, focus on leveraging LLMs to capture textual information features, which can be used to boost GNN performance on downstream tasks. key in- novation is use of explanations features: prompt LLM to perform zero-shot classification, request textual explanations for its decision makinn pro- cess, and design LLM-IO-LM interpreter to translate these explanations into informative features for downstream GNNs. Our experiments demonstrate that our method achieves state-of-the-art results on well-established TAG datasets, in- cluding Cora, PubMed, ogbn- lebon-rrs arxiv, as well as our newly introduced dataset, axpevar Furthermore, our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn rxiiv Lastly, we believe the versatility of the proposed method extends beyond TAGs and holds the potential to enhance other tasks involving graph-text data T
INTRODUCTION
Many real-world graphs possess textual information, and are often referred to text-attributed graphs (Yang et al. 2021). In TAGs, nodes typically represent text entities, such documents or sentences, while edges signify relationships between these entities. For example, the ogbn- ceobn-ad aaraiaxiii dataset (Hu al. 2020a) represents citation network in TAG form, where each node corresponds to paper, with its title and abstract serving as node attributes. More generally, the combination of textual attributes with graph topology provides rich source of information, significantly enhane- ing representation learning for important applications, such as text classification (aag et al. 2015)
Wang et al. 2016 Yasunaga et al., 2017: Chien al., [021 Zhao et al.) 2022). recommendation systems (Zhu et al. 2021), social networks, and fake news detection (Liu et al. 2019
Representation learning on TAGs. Prior research has explored various approaches for represen- tation learning on TAGs. The standard GNN pipeline (illustrated in [ light yellow), first encodes the textual attributes of each node using shallow or hand-crafted features such as skip-
gram (Mikolov et al. 2013) bag-of-words (BoW) (Harris 1985) (refer to Table 9 The result-
Our codes and datasets are available https: https://github. http://tttttttttte cwm/ aox nHe//APM

---

Published as a conference paper at ICLR 2024
Overview TAPE: Title, Abstract, Prediction Explanation
Step Node Feature Extraction
Step 2: Downstream Tasks TAGs
Prodictionn cs. EsRR CsCL, C5.33
Abstract: curve odiestation, despite being
The poper abost new ditasee scone text driection and rocogaitonn whec is togic related to computer vision (caCV), The paper also menties the ue of deep leaming techniques such as DeconvNet, which fill under the sib categocy of artifccii inclllyeeee (cx.Al). The ditnect smuneaal and inubee less reoegnmtin which could also till under the sub ub-cciteeeos of information retriccal (es (II) and natural langnge procesenn (1,) Finally, the paper dicusses the effectivnnes of diffeent solutioss. which ondld be cvalittal under sub-catogery of machine leaming (Gs
Explasation:
urmmon uniettations roal world envintmanl..
Title: Tooal lest Compeeeeneee Donnee For Scene Text
Ditectio And Recegamon
Question: Which adXi sub calcaory dees this paper
belong so" Give likely atKi CS stborrggrios
commo reparited list ondered from most least likely, the sonn "os.X"", and provide your roasoning. Auswer:
machine lorning techniguss fallnng
Graph Stracture
Prompt
Unery
LLM GPT33 (17EB)
Response
Preduction:
LM Delerto (7280))
Auge
GNN RevtiA7 et
Abstract:
Esplaration:
Prozos
Fino-lame
Trainable
wiloou
Shallow Embeddinn Techniques
skjp-gawe/ hags waoch
Shallow Embelding Pipcline (eg.. OGGB
LM-Based Pipeline (eg., GLANT)
LLM--aaed Pipellnn (Ours)
Figure 1:
Our framework leverages large language models (LLMs) enhance representation learn- ing on TAGs. First, textual attributes of each node, i.e., title and abstract, are wrapped in custom prompt (green box) query the LLM, here GPT-3.5 (Brown al. 2020), which generates a ranked prediction list and explanation (yellow box). Next, the original text, predictions, and explanation are used fine-tune a language model (LM), here DeBERTa (He et al., [202]), and then transformed into vectorial node features. Finally, these enriched node features, i.e., horig Respl and /pred. are used in any downstream GNN, e.g., RevGAT (Li al. 202T) to predict unknown node classes.
ing node features then used input for GNN. For instance, the Open Graph Benchmark (OGB) (Hu et al.] 2020a) generated BoW and skip-gram (Mikolov et al. 2013) features for the ogbn--roddctt and ogbn- arxiv datasets respectively. These processed features are readily available within popular graph libraries, such as PyTorch Geometric (PyG) (Fey & Lenssen 2019) and Deep Graph Library (DGL) (Wang et al.) 2019). and have been widely used by the graph com- munity. However, these shallow text embeddings are limited in the complexity of the semantic features they can capture, especially when compared to approaches based on multi-layer I.Ms.
LM-based pipeline for TAGs. Recent works have therefore focused on designing L.M-baseé tech- niques to better capture the context and nuances of text within TAGs (Chien al.) 2021: Zhao al.) 2022 Dinh et al. 2022). In this approach, pre-trained LMs are fine-tuned and used to generate node embeddings that are tailored to the specific TAG tasks (depicted in Figure in light gray). For LM using a neighborhood prediction task, while Zhao ctt al. (202)) fine-tuned an LM to predict the label distribution from a GNN's outputs. LM-based models have achieved state-of-the art (SOTA) results in node classification on ogbn-a aaaariii i and ogbn-products (Zhao et al.) 2022). However, these works typically entail intricate designs and demand substantial computational resources. Furthermore, for scalability reasons, existing works mostly rely on relatively small LMs, such as BERT (Devlin et al. 2018) and DeBERTa (He et al.) 2021. and thus lack the complex reasoning abilities associated with larger language models.
example, Chien et al. (222))
Large Language Models. The advent of large pre-trained models, exemplified by GPT (Brown et al. 2020). has revolutionized the field of language modeling. LLMs have notably enhanced perfor- mance across various natural language processing (NLP) tasks, and enabled sophisticated language processing capabilities such as complex and zero-shot reasoning. Furthermore, scaling laws (Kaplan) et al. (020) have revealed predictable rules for performance improvements with model and training data size. Additionally, LLMs have exhibited "emergent abilities" that were not explicitly trained for, such arithmetic, multi-step reasoning and instruction following (Wei et al.] 2022). While LLMs have found new success in domains like computer vision (Tsimpoukelli et al.] 2021), their potential benefits when applied TAG tasks remain largely uncharted. This presents exciting and promising avenue for future research, and it is precisely this untapped potential that we aim to explore in this work.
2

---

Published as a conference paper at ICLR 2024
LMs vs. LLMs. In this paper, we make clear distinction between "LMs" and "LLMs". We use LMs to refer to relatively small language models that can be trained and fine-tumed within the con- straints of an academic lab budger. We refer to LLMs as very large language models that are capable of learning significantly more complex linguistic patterns than LMs, such as GPT-3/4. These mod- els typically have tens hundreds of billions of parameters and require substantial computational resources to train and use, e.g., GPT-3 was trained on supercomputer with 10,000 GPUs. The size and complexity of recent LLMs have raised concerns about their scalability, as they can be too large even to run inference on the machines typically available within academic research labs. To address this issue, LLMs are often made accessible through language modeling as a service (LMaaS) (Sun) et al. 2022). This approach enables developers to harness the power of LLMs without necessitating extensive computational resources or specialized expertise.
the context of this paper, one of our primary objectives is to extract information from an LLM in I.MaS-compatinn result, we do not require fine-tuning the LLM or extracting its logits; rather, we focus solely on obtaining its output in textual form. In contrast, existing LM-based techniques (Chien et al.) [02T]] Zhao et al. 2022 Dinh et al. 2022) are not directly compatible with LLMs, as they require fine- tuning of LMs, well as accessing their latent embeddings or logits, which GPT-3/4 do not provide. Consequently, to the best of our knowledge, the use of LLMs in TAG tasks remains unexplored.
manner. As
Preliminary study. To assess the potential of LLMs in enhancing representation learning for TAGs, we conducted an initial investigation into leveraging GPT-3355 for zero-shot classification on the oginnarxxa dataset. Using task-specific prompts consisting of paper titles. abstracts, and ques- tions, GPT-3.5 achieved a promising accuracy of 73.5%, along with high-quality text explanations, surpassing several fully trained GNN baselines like RevGAT (Li et al. (02T) with OGB features (70.8% accuracy), but falling short of the SOTA accuracy of 76.6% (Zhao et al. 2022).
The present work: LLM augmentation using explanations. We introduce novel framework that leverages LLMs to improve representation learning on TAGs. A key innovation is the concept of explanations as features. By prompting a powerful L.LM to explain its predictions, we extract its relevant prior knowledge and reasoning steps, making this information digestible for smaller models, akin to how human experts use explanations to convey insights. To illustrate this concept further, observe in Figure I that the explanations (in the yellow box) highlight and expand upon key crucial information from the text, such as "deep learning techniques such as DeconvNet," and the relationship between text recognition and information retrieval. These explanations draw from the LLM's general knowledge and serve as valuable features for enhancing subsequent TAG pipeline phases. In practice, we design a tailored prompt to query an LLM such as GPT or Llama2 generate both a ranked prediction list and a textual explanation for its predictions. These predictions and explanations are then transformed into informative node features through fine-tuning a smaller LM such as DeBERTa (He et al. (021) for the target task. providing tailored features for any downstream GNNs. This smaller model acts as an interpreter, facilitating seamless communication between the LLM (handling text) and the GNN (managing vectorial representation).
Our contributions are summarized as follows:
Novel LMaaS--ompatie approach. We propose the first LMaaS compatile approach, to the best of our knowledge, for leveraging LLMs to enhance representation learning on TAGs. Our innovations involve extracting explanations from an LLM, here GPT-3.5 and Llama2, and subsequently employing an LLM-to-LM interpreter to translate textual expla- nations into enriched node vector representations for downstream GNNs. Our approach improves modularity and efficiency compared to prior LM+GNN models.
SOTA performance. Extensive experiments demonstrate that our method significantly boost the performance of various GNN models across diverse datasets. Notably, we achieve top-1 performance on ogbn- sigon-axxii faster than GLEM, and also excel in the TAG versions of PubMed and Cora datasets.
arxiv v with significantly lower computation time, i.e., 2.88x
Data contribution. We provide open-source access to our codes, pre-trained networks and enriched features. Additionally, recognizing the absence of raw text data for Cora and PubMed in common repositories (e.g.. PyG, DGL), we have collected and released these datasets in TAG format. Furthermore, we introduce the new sape-arxiv2 citation graph dataset, extending beyond GPT-3's ' knowledge cutoff, i.e.. Sept. 2021. These datasets can serve as valuable resources for the NL.P and GNN research community.
3

---

Published as a conference paper at ICLR 2024
78
77
TА (Ours)
2.88x lower computation time
76
192min, 77.50%
GLEM
Accuracy (%)
75
55 Imin, 76.57%
74
73
LM 104min, 73.61%
LM
72
GNN w/ shallow embedding
71
GNN
L. M--bused GLEM
4min, 70.83%
LLM-based TAPE (Ours)
70
100
200
300
400
500
600
Total Time (min)
The performance trade-off between node classification accuracy and total training time on ogbn-arxix (Hu et al. 2020a) for various training approaches that combine language models (LMs) and graph neural networks (GNNs). The experiment employs DeBERTa-bar (He et al. 2021) as the LM backbone and RevGAT (Li et al. 2021) the GNN backbone, with the size of the marker indicating the number of parameters.
Figure 2:
2 RELATED WORK
Shallow embedding pipeline for TAGs. In the context of learning representations TAGs, common approach involves combining graph-based learning with language modeling techniques. One prevalent strategy is to transform text attributes into shallow hand-crafted features, such as skip-gram (Mikolov et al., 2013) or BoW (Harris) 1985) features. Detailed information is available in Table These engineered features then be fed as inputs to graph-basee learning algorithm, such as a graph convolutional network (GCN) (Kipf' & Welling, 2016). which learns embeddings capturing the graph structure while incorporating the extracted text features. Shallow embedding methods are widely used in the graph community due to their simplicity and computational ef- ficiency, such as for designing GNN architectures (Veličković al., 2017; Chiang al. 2019 Velickovic et al. 2019: Zhang et al. (02T]) or benchmarking graph learning (Yang et 2016 Hu
al. 2020a). However, they may have limitations in capturing complex semantic relationships and fully leveraging the richness of text attributes, particularly in scenarios involving intricate semantic relationships and contextual information.
LM-based pipeline for TAGs. To overcome the limitations of shallow embedding approaches, researchers have explored deep embedding techniques by fine-tuning pre-trained LMs, such as BERT (Devlin et al. 2018), to generate node embeddings that are specifically adapted to the domain and context of the TAGs. These deep embeddings effectively capture the semantic richness text various TAG-related tasks. Integrating LM-based embeddings and graph- based learning can be done through different approaches. One approach is to cascaded architecture, where the node features are first encoded independently by the LMs, and then fed into GNN models. This representation paradigm has been widely adopted in subsequent works, such as TexIGNN (Zhu al., 2021), GIANT (Chien et al.] 2021). GPT-GNN (Hu et al.) 2020b), SimTeg (Duan et al. (023), as well as in studies related to knowledge graphs (Yasunaga al. 202T. Zhang al. 2022) and fact verification (Liu al., 2019; Zhou et al. 2019) that beyond the scope of this work. An alternative approach involves fusing text encoding and graph aggregation into iterative workflow. enabling the model to refine both the text representations and
attributes, leading to improved performance
the node embeddings simultaneously. such as Graphormer YYang et al. 2021) DRAGON (Yasunaga)
et al. 2022). and GLEM (Zhao al.) 2022), name a few.
LLM-based pipeline for TAGs. Incorporating LLMs into TAG tasks presents a promising frontier. LLMs such as ChatGPT [Brown et al. 2020) by OpenAI, PaLM [Chowdhery et al. 2022) by Google, and LLaMA (Touvron et al. 2023) by Meta, have demonstrated their effectiveness across spectrum of NLP tasks. However, their potential benefits for TAG tasks have yet to be fully explored. While some recent research efforts have sought to evaluate the capacity of LLMs in understanding graph- structured data and enhance their graph processing capabilities (Wang et al. 2023 Zhang 2023 Guo et al. 2023), these endeavors, while valuable, may not be directly aligned with specific focus
4

---

Published as a conference paper at ICLR 2024
on TAGs. By exploring LLM- methods designed specifically for TAGs, we unlock new possibilities for improving TAG prediction performance and advancing our understanding of text attributes within graph-based data. Notably, our initial attempt has already inspired further research endeavors in this direction.
3 FORMALIZATION
In this section, we introduce notation and formalize some concepts related to language models, large language models, and graph neural networks for node classification on TAGs.
Text-attributed graphs. Formally, TAG can be represented as g = (V, A, ( Jnev), where V is sequential text associated
of N nodes, A E NR is the adjacency matrix, and 8n. =D4 with node n E V, with D as the words or tokens dictionary. and L,, as the sequence length. In this paper, we investigate node classification on TAGs. Specifically, given some labeled nodes C C V. the goal is to predict the labels of the remaining unlabeled nodes 21 = V V.L.
Language models for text classification. In the context of TAGs, LMs can be employed encode the text attributes associated with each node and learn representation that captures the semantic meaning of the text. Let 8, E D' denote the text attributes of node n, and LM be pre-trained network, such BERT (Devlin et al. 2018) DeBERTa (He et al., (021). Then, the text attributes of node n can encoded by applying the LM to 8,, as follows:
hn = (M(ss) E R
where lú, the output of the L.M, and is the dimension of the output vector.
To perform node classification, the output is employed as input to classifier, such as a logistic regression or a neural network. The goal is to learn a function that maps the encoded text attributes to the corresponding node labels.
Large language models and prompting. LLMs have introduced a new paradigm for task- adaptation known "pre-train, prompt, and predict", replacing the traditional "pre-train, fine-tune" procedure. In this paradigm, the LLM is first pre-trained on a large corpus of text data to learn general language representations. Then, rather than fine-tuning the model task-specific labeled data, the model is prompted with a natural language prompt that specifies the task and context, and the model generates the output directly based on the prompt and the input (Liu al., 2023).
The prompt can take various forms, such as a single sentence or longer passage, and can include additional information constraints guide the model's behavior: Let M be an LLM that takes as input sequence of tokens x = (I1, TE........... xgy) (91, .2..... ). The model M is typically trained to optimize conditional probability distribution P(y(z), which assigns a probability to each possible output sequence given æ. To include prompt p with the input sequence æ, we can concatenate them into a new sequence (p. æ1 f1, ......... x.y)-
and produces as output a sequence of tokens y =
We then use å to compute the conditional probability distribution p(y (.). Formally, the probability of the output sequence y given is:
p(y II plsllve,,
(2)
where Yei represents the prefix of sequence y up position - 1, and p(y (())))),, represents the probability of generating token given yez and æ.
Graph neural networks for node classification. In node classification, the task to label each node in a graph based on its attributes and connections with other nodes. GNNs operate by aggre- gating information from node's neighbors, then updating the node's 's representation based on the aggregated information. Formally, the k-th layer of a GNN is designed as:
h = (h AGG(EE)-1 e NH)) Idd.
(3)
where hf E IRd is the representation of node i at layer l: and N₁ C CVVVVV is the set of neighbors of node Function f² is a differentiable function that updates the representation of a node based on its previous-layer representation and the aggregated information from its neighbors. This function
5

---

Published as a conference paper at ICLR 2024
is typically implemented neural network layer (e.g., multi-layer perceptron, attention mechanism). AGG is also a differentiable function (e.g. sum, mean, etc.) that aggregates the representations of node's neighbors to produce summary vector. The final representation is fed into a fully connected layer and a softmax function for class prediction.
4 PROPOSED METHOD
In this section, we describe our LLM-bbsed pipeline designed for node classification on TAGs. As illustrated in Figure [ the key idea is to leverage the LLM's explanations informative features for a downstream GNN. To achieve this goal, our method involves three main steps: 1) LLM-based prediction and explanation generation, 2) fine-tuning LM interpreter, and 3) training a GNN.
4.1 GENERATING PREDICTIONS AND EXPLANATIONS WITH LLMS
As outlined in the introduction, our approach is designed to be LMaaS-compatibl given the scale of LLMs. This means that we aim to operate solely through API access an LLM,, using text-based input and output, without requiring fine-tuning the LLM or accessing its embeddings or logits.
In lieu of these requirements, our approach focuses on querying the LLM in an "open-ended" man- ner, ie., instructing the LLM to make multiple predictions and provide explanations for its deci- sions. By doing so, we aim to effectively extract its reasoning abilitics and general knowledge in text format. These text-based outputs then processed using LLM-to-LM interpreter create informative node features for downstream GNNs. With this objective, for each paper node E V, generate prompt that includes the title and abstract of the paper, along with open-ended question about the paper's topic. The specific phrasing of the question part of the prompt is tailored the task and dataset, as shown in Table 5 The general structure of the prompt is follows:
Abstract: [paper abstract] Title: [paper title]
Question: [ask the model to predict one or more class labels the paper, ordered from most to least likely, and provide explanations for its predictions] Answer:
Querying the LLM results in ranked prediction list and textual explanation for each paper:
(Ranked Predictions) [a ranked prediction list] (Explanations) (model-generated explanation for the predictions]
These predictions and explanations serve supplementary text attributes for the downstream LMs and GNN models, as detailed in the subsequent section.
4.2 FINE-TUNING LM INTERPRETER AND NODE FEATURE EXTRACTION
Original text and explanation features. Our initial step involves converting both the original text, i.e,, title and abstract, and the LLM's explanations into fixed-length node features suitable for downstream GNN applications. Our approach to fine-tune smaller LM, which acts as an "interpreter" for the LLM's text explanations. The rationale behind this step is that both the LLM and LM possess distinct advantages: the LLM has greater power and more knowledge but less flexible, while the LM has less skills but is compact enough to be fine-tuned to specific task. Thus, the LM serves to interpret the LLM's output for the GNN, with the text explanation acting as an effective intermediate medium for communication. Then, fine-tuning the LM enables it to extract the most valuable and task-relevant features from the explanations.
Concretely, first fine-tune treitrained LMs follows: let LMorig and LMappi be pre-trained LMs that take input the original sorig and the explanation expl text sequences, respectively. We obtain text embeddings for each source as follows:
hodg LMongg (iR) E E Nxd
hexpl = LMexpi (s (p E R Nxd.
(4)

---

Published as a conference paper at ICLR 2024
We further apply Multi-Layer Perceptron (MLP) to the output of the LMs obtain N C- dimensional prediction matrix representing the LM's predictions for each node (in logits):
Yorig orig E IR RRN Beapl MLP expt (hesp)) E RR
(5)
We fine-tune these LMs and MLPs using cross-entropy loss. Finally, the text embeddings from both sources, Rorig and hespt. are used as enriched features for training downstream GNNs.
Ranked prediction features. In addition to the explanations. the LLM also provides top-k ranked prediction list for each node, which adds valuable information. To incorporate this knowledge, the top-A predictions for node are first one-hot encoded as vectors Pi.1..... Pi, E ER. These vectors are subsequently concatenated into a kC-dimensional vector, followed by a linear transformation to produce a fixed-sized vector of length dp. This process produces prediction feature matrix as hpred E R xdp across all nodes.
In summary, we denote features as /TTAPR {Morig /eeply /preed where "TAPE" stands for Title, Abstract, Prediction and Explanation for each node. Importantly, our framework requires these features to remain frozen during downstream GNN training, ensuring that the LM and LLM do not participate in the GNN training process. This characteristic significantly enhances case-of-use, modularity, and efficiency compared to approaches like GLEM, which involve an expensive iterative LM-GNN training process. As result, we achieve substantial speedup over GLEM, egg, 2.88 speedup on ogbn-arxiv even when utilizing the same backbone LM and GNN.
4.3 GNN TRAINING ON ENRICHED FEATURES
Our final step is to train GNN using the ПTPPE features. We aim to achieve this without increasing the memory requirements of the GNN or making any changes to architecture. To accomplish this,
we use an ensemble approach, as a simple and effective way of combining the features. Specifically, we independently train GNN models forig. feepl. and fpred on the features hong. hespl and hpred respectively, predict the ground truth node labels:
/xrig/expl//pe foni//epp//ped (Norig/ hetp://pl///eet preds A) E IR NxC.
(6)
We then fuse these predictions by taking their average:
ý mean Nexpt Upea) E IR. NXC.
Each of the three models performs well individually shown in Table which validates the effec- tiveness of simple averaging. This strategy enables us to capture complementary information from diverse input sources, ultimately enhancing the overall model's performance.
4.4 THEORETICAL ANALYSIS
In this section, we aim to demonstrate that explanations generated by an LLM can be valuable features for smaller LM. Specifically, the explanations E are helpful if they possess fidelity in describing the LLM's reasoning; and the LLM is non-redundant,, utilizing information not used by the smaller LM. Let E be the textual explanations generated by an LLM; Zz and Z are embeddings from the LLM and smaller LM respectively, y is the target and H (-I.) is the conditional entropy. The detailed proof is Appendix(A]
Theorem. Given the following conditions 1) Fidelity: E is a good proxy for Zt. such that (Zd EE) 6, with 6 > 2) Non-redundancy: Zr contains information not present in Z, ex- pressed as H (y) Z, (ylZ) é', with e' > Then it follows that H (y) Z, E) < H (y) Z).
5 EXPERIMENTS
We evaluate TAPE on five TAG datasets: Cora McCCallum et al. 2000). PubMed (Sen et al. 2008), ogbn-arxiv, ogbn-productss (Hu et al., 2020a), and cape-arxite axoeeaxxi22.. For Cora and PubMed, raw text data the articles unavailable in common graph libraries such as PyG and DGL. Hence, we collected and formatted the missing text data for these datasets in TAG format. Additionally, given the popularity of these datasets, their TAG version will be released publicly for reproducibility and new research projects. For ogpnn-roouccs,, given its substantial scale of million nodes and 61 million edges and considering our academic resource budget, we conducted experiments subgraph sample. Details can be found in Appendix G
7

---

Published as a conference paper at ICLR 2024
5.1 MAIN RESULTS
Node classification accuracy for the Cora, PubMed, ogbn-arxii ogon-produce and tape-axxiv2 123iv2333 datasets. G ? denotes the improvements of our approach over the same GNN trained on shallow features ftshallow: L 1 denotes the improvements of our approach over LManeuune The results are averaged over four runs with different seeds, and the best results are in bold.
Table I:
Dataset
Method
GNN
LM
Ours
ROBANT
T
LLM
1.M
L-1
FIIAPE
MLP
0.0213
0.0000
0.6769
1 0,0278
13.35%
0.0685
Cora
GCN
0.8911 & 0.0015
0.8423 0.0053
2.33%
0.6769
0,7606 1 0.0378
16.59%
0.9119
0.0158
SAGE
0.8824 s 0.0009
0,8455
0,0028
5.28%
0.6769
0,7606
à 0.0378
18.13%
0.9290
0.0307
RevGAT
0 0.0000
0,8353 0.0038
4.14%
0.6769
0.7606 1 0.0378
18.04%
0,9280
0.0275
MLP
8.8635 + 0.0032
0.8175 + 010059
10.77%
0.9342
0,9494 + 0.0046
0,754
0.9565
0.0060
GCN
0.8031 0.0425
0.8419 0.0050
0.0046
0666%
0.0043
SAGE
B.888
0.0002
0.832 0.0082
8.305
0.9342
0.9494 0.0046
1.31%
0 0.0053
RevGAT
0,8850 à: 0.0005
0,8502 0.0048
8.52%
0.9342
0.9494 1 0.0046
1.15%
0.9604
0.0047
MLP
0.5356 2 0.0038
0,7308
0.0006
42.191
0.7250
0,7361 2 0.0004
3.07%
0.7587 0,0015
oobn-arxiv
GCN
0,7182 a 0.0027
0,7329
0.0010
4.21%
0.7350
0.7361 1 0.0004
2.16
0.7520
O. 0.0003
SAGE
0,7171
0.0017
0.7455 0.0014
0.0004
4.22%
D.DNKT
RevGAL
0.70K3 = 0.0017
0,7590
0.0019
9.42%
0.7250
0.7361 1 0.0004
5,28%6
0,7750
0.0012
MIPP
0.5385 0.0017
0.6125 0.0078
46.3%
0.7440
0.7297 + 0.0023
7.96%
0.7878 0.D0RE
agba-prodacts
GCN
SAGE
0,7052 2 0.0051
0.6913 = 0.0026
0.0119 0.6869
0.6977
0.0042
17.71%
13.39%
0.7440
0,7440
0,7297 + 0.0023
0,7297 1 0.0023
11.51%
9.58%
0.8137 0.0043
0.7996 0.004)
RevGAT
0.6964 + 0.0617
0,7189 + 0.0030
18.24%
0.7440
0,7297 + 0.0023
12,84%
(US234 0.0H36
MLP
B.6202 = 0.0064
0,5574
0.0092
35.20%%
0.7356
0,7358 = 0.0006
12.25%
(18385 0.0246
tape-arxiv23
GUN
SAGE
0.6:54 0.0062
a: 0.0037
0,5665
0,5672
0.0061
0.0032
20.45%
0.7356
0.7358 0,0006
a 0.0006
12.28%
8,90%
0.3080 0.0215
0,8388 0.0264
RevGAT
0.6563 = 0.0062
0,5834 0.0038
28.34%8
0.7356
0,755% # 0.0006
12.64%
(L8423 0.0256
We conduct comprehensive evaluation of our proposed TAPE method by comparing with existing GNN- and LM-based methods, with the results summarized in Table For GNN comparisons, we consider three widely utilized architectures: GCN (Kipf & Welling 2016). GraphSAGE (Sun al., 2021]. and RevGAT (Li et al., (021)) along with a basic MLP baseline that operates independently off graph-related information. We explore three types of node features: 1) shallow features (detailed in Table denoted as Ithhaloo. 2) GIANT features (Chien et al. (222)) /IGIANT, and 3) our proposed features ПTAPE, comprising horig. hexpl- and hgredd For LM-based methods, we investigate two ap- proaches: 1) fine-tuning DeBERTa on labeled nodes, denoted as LMinetune- and 2) using zero-shot ChatGPT (gpt-3.5-turbbl with the same prompts as our approach, denoted as LLM.
Our approach consistently outperforms other methods on all datasets and across all models, demon- strating its effectiveness in enhancing TAG representation learning. Among GNN-based methods, shallow features (i.e., hehallow) ) yields subpar performance, while LM- based features (i.e., hChAnT) im- proves results. In the case of LMs, fine-tuned LMs (i.e., [M also perform well. Our proposed novel features, leveraging the power of the LLM,, further enhance the results.
Additionally, we expanded our experimentation to include the open-source Llama2 (Touvron et al. 2023), demonstrating the feasibility of a cost-effective (free) alternative, shown in Table 4 Fur- thermore, to address the potential label leakage concern in LLM, we took the initiative to construct a novel dataset, namely tape-azxivas v23, comprising papers published in 2023 or later well beyond the knowledge cutoff for GPT-3.5. The results clearly illustrate strong generalization capabilities: while the L.M achieves 73.56% accuracy, our approach outperforms it with 84.23%.
5.2 SCALABILITY
Our proposed method surpasses not only pure LMs and shallow embedding pipelines but also the LM--ased pipelines on the ogbn-arxt dataset, achieving superior balance between accuracy and training time, as illustrated in Figure 2 Specifically, our method achieved significantly higher accuracy than the SOTA GLEM (Zhao et al. (2022) method while utilizing the same LM and GNN models. Furthermore, our approach requires only 2.88x less computation time. These efficiency improvements are attributed to our decoupled training approach for LMs and GNNs, avoiding the iterative (i.e., multi- stage) approach used in GLEM. Moreover, unlike the iterative approach, our model allows for parallelizing the training of LMaiig and LM further reducing overall training time when performed simultaneously.
8

---

Published as a conference paper at ICLR 2024
Table 2: Experiments on ogin-arxiv dataset with DeBERTa-base (He 2021 as LM backbone and RevGAT (Li et al. 2021) GNN backbone for comparison of different training paradigms of fusing LMs and GNNs, including our proposed method and the state-of-the-an GLEM method (Zhao et al. 2022). The validation and test accuracy, number of parameters, maximum batch size (Max bsz.), and total training time on 4 NVIDIA RTX A5000 24GB GPUs are reported.
Method
Val acc.
Test acc.
Params.
Max bsz.
Total time
LMorig
0.7503 0.0008
0.7361 0.0004
139,223,080
36
1.73h
GNN- GNN-/baalooo
0.7144 0.0021
0.7083 0.0017
427,728
all nodes
1.80min
GLEM-G-Step
0.7761 + 0.0005
0.7655 + 0.0029
1,837,136
all nodes
9.18h
GLEM-L-Step
0.7548 # 0.0039
0.7495 0.0037
138,632,488
36
TArg-LMorig
0.7503 + 0.0008
0.7361 + 0.0004
139,223,088
36
1.73h
TAPE-LMant
0.7506 0.0008
0.7432 0.0012
139,223,080
36
1.40h
TAPE-GNN-ImAve
0.7785 0.0016
0.7750 0.0012
1,837,136
all nodes
3.76min
Table 3: Ablation study on the ogbn-arxi v dataset, showing the effects of different node features on the performance. Node features include the original text attributes (horgg), the explanations (hespl and predicted pped generated by LLM, and the proposed method (/TAPE). Results are averaged over runs with different seeds. The best results are in bold.
Method
Hodg
hespi
Ppdd
hYAPE
val 0.7624 + 0.0007
0.7577 = 0.0008
0.7531 + 0.0006
0.7642 0.0003
GCN
test 0.7498 + 0.0018
0.7460 = 0.0013
0.7400 + 0.0007
0.7520 + 0.0003
val 0.7594 # 0.0012
0.7631 # 0.0016
0.7612 # 0.0010
0.7768 0.0016
SAGE
test 0.7420 = 0.0018
0.7535 = 0.002
0.7524 0.0015
0.7672 = 0.0007
val 0.7588 # 0.0021
0.7568 # 0.0027
0.7550 0.0015
0.7785 0.0016
RevGAT
test
0.7504 + 0.0020
0.7529 = 0.0052
0.7519 + 0.0031
0.7750 = 0.0012
5.3 ABLATION STUDY
We perform an ablation study on the ogbn-arxixn dataset (Hu et al. 2020a) to evaluate the rele- vance of each module within our framework. The results are summarized in Table 3 and Figure 4 Across all methods and for both the validation and test sets, our proposed method consistently out- performs the other settings. This underscores the value of incorporating explanations and predictions into node embeddings.
We provide time analysis and cost estimation in Appendix B detail tape-arxiv23 dataset col- lection in Appendix [ use open-sourced llama as the LLM in Appendix include a case study in Appendix [E] discuss prompt design in Appendix [F] examine LM finetuming effects in Appendix!! [I] explore the impact of various LMs in Appendix J and analyze memory usage in Appendix
CONCLUSION
Given the increasing importance of integrating text and relationships, coupled with the emergence of LLMs, we foresee that TAG tasks will attract even more attention in the coming years. The conver- gence of L.MSs and GNNs presents new opportunities for both research and industrial applications. As a pioneering work in this field, we believe that our contribution will serve as a strong baseline for future studies in this domain.
Limitation and future work. An inherent limitation of our approach lies in the requirement for customized prompts for each dataset. Currently, we rely manually crafted prompts, which may not be optimal for the node classification task for every dataset. The efficacy of these prompts may fluctuate depending on the specific characteristics of the dataset and the specific task at hand. Future work can focus on automating the prompt generation process, exploring alternative prompt designs, and addressing the challenges of dynamic and evolving TAGs.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
Bryan Hooi is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2023) (Grant A-8001996-00-00) and Xavier Bresson supported by NUS Grant ID R-252-000-BB9-1133 The authors would like to express their gratitude to the reviewers for their feedback, which has improved the clarity and contribution of the paper.
REPRODUCIBILITY STATEMENT
In this statement, we provide references to the relevant sections and materials that will assist readers and researchers in replicating our results.
Theorem. For a comprehensive understanding of the theorem presented in Section 4.4 please refer to Appendix(A for detailed proof.
Dataset deseription. We summarize datasets used in our study in Appendix G providing infor- mation on their sources and any necessary preprocessing steps. Additionally, for the newly intro- dataset, we offer a detailed description the data collection and processing
duced steps in Appendix C
Open access to codes, datasets, trained models, and enriched features. Our source code can be accessed at the following url: htttss://ggtti httpt//ggtttt.t/// com/ Xi aox nHe TAPE Within this reposi- tory, we provide script with step-by-step instructions on how to replicate the main results presented in paper. Additionally, we offer download links for the Cora and P PubMed datasets in TAG form, along with the new dataset tape-arx: v23. These datasets can serve as valuable resources for the NLP and GNN research community. Furthermore, this repository includes the checkpoints for all trained models (.ckpt) and the TAPE features (.emb) used in our project, making it easy for researchers focusing on downstream GNN tasks access enriched features.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsich. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 257 257,266,220 2019.
Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovie, and Inderjit S Dhillon. Node feature extraction by self-supervised multi-scale neighborhood predic- tion. arXiv preprint arXiv:2111.0006 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018.
Tu Anh Dinh, Jeroen den Boef, Joran Cornelisse, and Paul Groth. E2cg: End-to-end node classi- fication using graph topology and text-based node attributes. arXiv preprint arXiv:2208. 0.X0022008.04609, 2022.
Keyu Duan. Qian Liu, Tat-Seng Chua, Shuicheng Yan, Wei Tsang Ooi, Qizhe Xie, and Junxian He. Simteg: A frustratingly simple approach improves textual graph learning. arXiv preprint arXiv:2308. 02565, 2023.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv: 02428, 2019.
10

---

Published as a conference paper at ICLR 2024
Jiayan Guo, Lun Du, and Hengyu Liu. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint Stt::2305110666 2023.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs.
Advances in neural information processing systems, 30, 2017.
Zellig Harris. Distributional structure. The philosophy of linguisties, 1985.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced ennanned bert with disentangled attention. In International Conference on Learning Representations, 2021.
URL httpp:://pperreviee net. fort://frrriiddeee fto.:rrr..dxx22Ittttttttt
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 222333,, 2020a.
Ziniu Hu, Yuxiao Dong, Kuansan Wang. Kai-Wei Chang, and Yizhou Sun. Gpt-gun: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1857-1867, 2020b.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 1609.02977 2016.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In International conference on machine learning, pp. 6437-6449, PMLR, 2021.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human 2021.
falsehoods. arXiv preprint
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Computing Surveys, 55(9):1-35, 2023.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. Fine-grained fact verification with kerne graph attention network. arXiv preprint arXiv: 2019.
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 3:127-163, 2000.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionaliiy. Advances in neural information processing systems, 26, 2013.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embed- ding benchmark. arXiv preprint arXiv:2210.073116. 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint 19Xiv:1908.1000 2019.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. Al magazine, 29(3):93-93, 2008.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga- Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint (XXiv22006006155. 2022.
Chuxiong Sun, Hongming Gu, and Jie Hu. Scalable and adaptive graph neural networks with self- label-enhanced training. arXiv preprint arXiv: 104.09776 2021.

---

Published as a conference paper at ICLR 2024
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model ass--as-ate sesieelas-a-seervee a-service. In International Conference on Machine Learning. pp. 20841- 20855, PMLR, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 139771 2023.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, 34:200-213 212, 2021.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: (.1.:003, 2017.
Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? arXiv preprint arXiv:2305. 2023.
Kuansan Wang. Zhihong Shen, Chiyuan Huang, Chieh-Han Wu, Yuxiao Dong, and Anshul Kanakia. Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 1(1):396- 413, 2020.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03553. 2022.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, al. Deep graph library: graph-centrie, highly-performant package for graph neural networks. arXiv preprint 190.01390, 2019.
Suhang Wang, Jiliang Tang, Charu Aggarwal, and Huan Liu. Linked document embedding for classification. In Proceedings of the 25th ACM international on conference on information and knowledge management, pp. 115-124, 2016.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilitics of large language 2022.
models. arXiv preprint
Cheng Yang. Zhiyuan Liu, Deli Zhao, Maosong Sun, and Edward Y Chang. Network representation learning with rich text information. In IJCAI, volume 2015, pp. 2111-2117, 2015.
Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. Advances in Neural Information Processing 3 34:28798-288 2021.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning. pp. 40-48. PMLR, 2016.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. Graph-based neural multi-document summarization. arXiv preprint arXiv:/706.00m 2017.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. arXiv preprint 0rX.::2004.06378 2021.
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren. Xikun Zhang, Christopher D Manning, knowledge graph pretraining.
Percy S Liang and Jure Leskovec. Deep bidirectional Advances in Neural Information Processing Systems, 35:37309-37333 2022.
12

---

Published as a conference paper at ICLR 2024
Jiawei Zhang. Graph-toolfooree To empower Ilms with graph reasoning ability via prompt aug- mented by chatgpt. arXiv preprint arXi::230..11116, 2023.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah, Graph-less neural networks: Teaching old mlps new tricks via distillation. arXiv preprint arXiv:21 2021.
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning, and Jure Leskovec. Greaselm: Graph reasoning enhanced language models. In Inter- national conference on learning representattinns 2022.
Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. Learning large-scale text-attributed graphs via variational inference. arXiv preprint (.ii:22/0.147009 2022.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pp- 12697-12706. PMLR, 2021.
Jie Zhou, Xu Han, Cheng Yang. Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Gear: Graph-based evidence aggregating and reasoning for fact verification. arXiv preprint 10810:1908.01843, 2019.
Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Tianqi Yang. Liangjie Zhang, Ruofei Zhang, and Huasha Zhao Textgnn: Improving text encoder via graph neural network in sponsored search. In Proceedings of the Web Conference 2021, pp. 2848-28577 2021.
13

---

Published as a conference paper at ICLR 2024
A THEORETICAL ANALYSIS
In this section, we aim to demonstrate that explanations generated by an LLM can provide valuable features for another model (such smaller LM). This is true under key conditions:
1. Fidelity: The explanations effectively represent LLM's reasoning over the raw text, con- taining most of the information from the LLM's hidden state.
2. Non-redundancy: The LLM possesses unique knowledge not captured by another model.
We formulate our theorem as follows:
Theorem 1. Given the following conditions:
1) Fidelity: E is good proxy for Z such that
H() =,, E <>0
(8)
2) Non-redundancy: Z1, contains information not present in Z, expressed as
H( Z1) H C, <>c
(9)
Then, follows that:
Hy H(y
(10)
where E is textual explanations generated by an LLM, Zr. is the vectorial representation of the raw text modeled by the LLM, Z is the vectorial representation of the raw text modeled by the other model, y is the target and H----) is the conditional entropy.
Proof. We aim to demonstrate that the conditional entropy of given both Z and E, denoted as Hyy Z, E), less than the conditional entropy of y given only Z, denoted H (y) Z))
Starting with:
E)
(11)
We apply the properties of entropy to decompose this expression into two components:
Hly|Z,E) E) H(y) ZL. E) + I(y: Zz|Z, E) Now, we utilize the following upper bound of conditional mutual information:
(12)
T(y; Z2, E) HZz,lZ, H(Zrly,
(13)
(Zr E)
(14)
where the first line follows from the definition of mutual information, and the second line follows from the nonnegativity of conditional entropy.
Substituting equation 14 into equation
12
rewrite the conditional entropy as:
(yy Z, E) < H Hy) Zr, E) + H (Zr,
(15)
Since conditional entropy increases when conditioning on fewer variables, we further have:
H Zr, H HHvl ((z((E)
(16)
Applying the "Fidelity" and "Non-redundancy"" conditions:
B) H(ylZ) +c
(17)
Finally, <'> e, have:
H(y e' H(v)z)
(18)
14

---

Published as a conference paper at ICLR 2024
Consequently, have proven that:
< <
(19)
This completes the proof.
B TIME ANALYSIS AND MONEY ESTIMATION
Our primary dataset, ogbn-arxiv with 169,343 nodes and 1,166,243 edges, serves as a represen- tative case for our approach. On average, our input sequences consist of approximately 285 tokens, while the output sequences comprise around 164 tokens. For the ChatGPT-3.5 Turbo API, priced at $0.0015 per 1,000 input tokens and $0.002 per 1,000 output tokens, with token per minute rate limit of 90,000, the monetary estimation for ogbn a axii v is as follows:
Cost ((285 x 0.0015) 1000 + (164 x 0.002)/1000) x 169, 343 s 128 USD
(20)
Considering the token rate limit, we estimate the deployment time as follows:
Time 343, (90, 000/285) 536min 9h
(21)
Cost-Effective Alternatives. Additionally, we have explored cost-effective alternatives, such as leveraging open-source LLMs like llama2. The use of llama2 is entirely free,, and the querying process to 1amma2-13b-cha takes approximately 16 hours when utilizing 4 A5000 GPUs.
Efficiency through Single Query and Reuse. Our method requires only one query to the LLM, with predictions and explanations stored for subsequent use. This not only enhances efficiency but also minimizes the number of API calls, contributing to cost-effeettvveness. We also release the gpt responses for public use.
C ADDRESSING LABEL LEAKAGE CONCERNS WITH A NEW DATASET
GPT-3.5's training data might include certain arXiv papers, given its comprehensive ingestion of textual content from the internet. However, the precise composition of these arXiv papers within GPT-3.5's training remains undiselosed, rendering it infeasible to definitively identify their inclu- sion. It is essential to emphasize that the challenge of label leakage is widespread and affects var- ious language model benchmarks, such the prominent BIG-bench (Srivastava et al. 2022 and TruthfulQA (Lin et al. (202T).
To address this concern, we created novel dataset tape-arxi v2. for experiments. We made sure that this dataset only included papers published in 2023 or later, which is well beyond the knowledge cutoff for GPT-3.5, as was launched in November 2022. The creation of this new dataset was meticulously executed. We collected all cs.ArXiv papers published from January 2023 to September 2023 from the arXiv daily repository F We then utilized the Semantic Scholar API 3 to retrieve citation relationships. This process yielded comprehensive graph containing 46, 198 papers and 78,548 connections. Our codes to collect and build the dataset available at:
https:
htt.a/b.cooo/iiaaxiiee//t//eepp ctte__rxiv____ arsip_22rxivv_2233 2023
D LLAMA AS A COST-EFFICIENT ALTERNATIVE
We extend out experiment to the open-source LLM "Ilama-2- 1-ama-22-13bchat" (Ilama for short), which demonstrates the feasibility of cost-effective (free) alternative, see Table
13b-chat"
is worth noting that although llama exhibits a lower performance compared to GPT-3.5 in terms of both zero-shot accuracy and explanation quality, our pipeline still maintains its robust performance. As an illustration, we achieved an accuracy of 76.19% on the ogbn-azzn arxiv dataset using Ilama, slightly below the 77. 77.50% achieved with GPT-3.5. We attribute this impressive level of generaliza- tion to the complementary nature the explanations themselves, which serve as rich source of semantic information supplementing the original text such title and abstract.
httoo://arxiv.ordg escholachcoo wwc.olar. wwcholar..orrg/proottt///
https: http::::/wwwwsmmattt..
15

---

Published as a conference paper at ICLR 2024
Table 4: Node classification accuracy for the Cora, PubMed and
datasets.
13-oh-t3b-chat
GPT3,5
Dataset
Method
I.M
L.
I.I.M
M.Sutttuee
ATAPE
GCN
0.5746
0.6845
0.0194
0.9045 a 0.0231
0.6769
0.7606 0.0378
0,9119 0.0158
Cora
SAGE
0.5746
0.6845 0.0194
0.9170 + 0.0337
0.6769
0.7606 = 0.0378
0,9290 + 0.0307
RevGAT
0.5746
0.6845 0.0194
0,9113 = 0.0237
0.6769
0.7606 0.0578
0,9280 0.0275
GCN
0.3958
0,9121
0.9362 0.0050
0.9342
0.0046
0,9431 0.0043
PubMed
SAGE
0,3958
0,9121 0.0026
0.9581 = 0.0073
0.9342
0.9494 = 0.0046
0,9618 + 0.0053
RevGAT
0.3958
0,9121 0.0026
0.9561 = 0.0068
0.9342
0.9494 0.0046
0.9604 0.0047
GCN
0.4423
0.6941
0.0020
0.7418 =: 0.0031
0.7350
0.7361 = 0.00004
0.7520 a 0.0003
ogbn-arxiv
SAGE
0,4423
0.6941
0.2536
0.0004
0.7672
RevGAT
0.4423
0.6941 0.0020
0.7619 + 0.0027
0,7350
0.7361
0.0004
0,7750 0.0012
GCN
0.4452
0.7677 0.0042
0.8045 0.0264
0.7356
0.7832 0.0052
0 0.0215
tape-arxiv23
SAGE
0.4452
0.7677 0.0042
0.8378 + 0.0302
0.7356
0.7832 # 0.0052
0.8388 0.0264
RevGAT
0,4452
0.7677 0.0042
0.8407 0.0308
0.7356
0.7832 = 0.0052
0.8423 0.0256
E CASE STUDY
(a) Original text attributes features.
(b) Explanations features.
Figure 3: Case study comparing features for node classification on the PubMed dataset: (a) Original text attributes and (b) Explanations generated by LLMs. The GNN model trained with (b) accurately predicts the label for node 12390 (type 2 diabetes), while the model trained with (a) predicts the incorrect label (experimentally induced diabetes). This improvement can be attributed to the concise and focused nature of LLM-generated explanations, as well as their reasoning ability and utilization of external knowledge.
To investigate the impact of using explanations as features in improving node classification on TAGs, conduct an analysis on predicted samples from the PubMed dataset. Figure 3 presents a case where the GNN model trained with original text attributes as features incorrectly predicts the label for node 12390 (as experimentally induced diabetes), while the model trained with explanations generated by LLMs as features correctly predicts the label (as type 2 diabetes).
This improvement can be attributed to two main factors. Firstly, compared to the original text at- tributes, which consist of the title and abstract text, the explanations generated by the LLM are more concise and focused. This aids the subsequent LM in generating node embeddings that capture the essential semantics without the need to compress an excessive amount of information into fixed- length representation. Secondly, LLMs possess reasoning capabilities and the ability to leverage general knowledge, which prove crucial in achieving accurate predictions. For instance, the expla- nations generated by LLMs explicitly link type diabetes to MKR mice and db/db mice (which are common animal models of type 2 diabetes), as well the insulinopenic mice / streptozotocin to experimentally induced diabetes. This knowledge is either absent or only implicitly specified in the original text attributes.
16

---

Published as a conference paper at ICLR 2024
F PROMPT DESIGN
TableB the prompts used for various datasets. Each prompt includes the abstract and title of the paper, followed by task-specific question. The question is formulated to query the model about a particular aspect of the paper and request an explanation for the prediction. The answer section left blank for the model to fill in. Generally, our analysis finds that the current instructions allow the LLM to produce output that conforms well to the expected format without significant deviations, allowing the answers be straightforwardly extracted from the text output of the LLM.
Table 5: Prompts used in this work to query the LLM.
Datasct
Prompt
Cora
<abstract text> in Title: <title text> \n Question: Which of the following sub-categories of Al does this paper belong to: Case Based, Genetic Algorithms, Neural Networks, Probabilistic Methods, Reinforcement Learning. Rule Learning, Theory? If multiple options apply. provide a comma- separated list ordered from most least related, then for each choice you gave, explain how is present in the text. In In Answer:
Abstract:
Pubmed
Abstract: <abstract text> in Title: <title text> In Question: Does the paper involve any cases of Type diabetes, Type 2 diabetes, or Experimentally induced diabetes? Please give one more answers of either Type diabetes, Type 2 diabetes, or Ex- perimentally induced diabetes; if multiple options apply, provide comma -eeaaated ordered from most least related, then for each choice you gave, give a detailed explanation with quotes from the text explaining why it is related to the chosen op- tion. In In Answer:
ogbn-arxis
Abstract: <abstract text> In Title: <title text> In Question: Which arXiv CS sub- category does this paper belong to? Give 5 likely arXiv CS sub-categories comma-separated list ordered from most to least likely, in the form "cs.XX", and provide your reasoning.
Answer:
ogbn-preducts
<product description> Question: Which of the following cat- egory does this product belong to: 1) Home Kitchen, 2) Health & Personal Care, 3) Beauty, 4) Sports & Outdoors. 5) Books, 6) Patio, Lawn & Garden, 7) Toys & Games, 8) CDs & Vinyl. 9) Cell Phones & Accessories. 10) Grocery & Gourmet Food. 11) Arts, Crafts & Sewing, 12) Clothing, Shoes & Jewelry, 13) Electronics, 14) Movies & TV, 15) Software, 16) Video Games, 17) Automotive, 18) Pet Supplies, 19) Office Products, 20) Industrial & Scientific, 21) Musical Instruments, 22) Tools & Home Improvement, 23) Magazine Subscriptions, 24) Baby Products, 25) NAN, 26) Ap- pliances, 27) Kitchen & Dining, 28) Collectibles & Fine Art, 29) All Beauty, 30) Luxury Beauty, 31) Amazon Fashion, 32) Computers, 33) All Electronics, 34) Pur- chase Circles. 35) MP3 Players & Accessories, 36) Gift Cards, 37) Office School Supplies. 38) Home Improvement, 39) Camera & Photo, 40) GPS & Navigation, 41)
Product description:
Digital Music, 42) Car Electronics, 43) Baby, 44) Kindle Store, 45) Kindle Apps, 46) Furniture & Decor? Give 5 likely categories as comma-separated list ordered from most least likely, and provide your reasoning.
Answer:
tape-arxiv23
<title text> \n Question: Which arXiv CS sub- category does this paper belong to? Give likely arXiv CS sub-categories as a comma-separated list ordered from most to least likely, in the form "cs.XX", and provide your reasoning. in in Answer:
Abstract: <abstract text> \n Title:
Exploring Prompt Variations. We have extensively explored the influence of various prompts on the ogbn-arxiv dataset, as outlined in Table and Table 7
Table/]indiccaee that, generally, most prompts yield similar performance. However, minor perfor- mance improvement is observed when the title is positioned after the abstract. This finding aligns with the principle suggested by Zhao et al. (202T) that placing more critical information later in the prompt can be beneficial.
Further analysis presented in Table demonstrates a positive correlation between the LLM's zero- shot accuracy and the overall accuracy of our method, implying that higher zero-shot prediction scores lead to enhanced TAPE accuracy. Despite the variation in prompt designs, methodology
17

---

Published as a conference paper at ICLR 2024
consistently achieves similar accuracy levels, ranging from 0. 0.7660 0.7750 with the RevGAT as the GNN backbone. This consistency underscores the robustness of our proposed TAPE to different prompt configurations.
Table 6: Prompts used for our experiments studying the effect different prompts. Most prompts have similar performance.
Description
Prompt
Accuracy
Default prompt
Abstract: <abstract text> Title: <title text> Question: Which arXiv CS sub-category does this paper belong to? Give 5 likely arXiv CS sub-categories as comma-sspaaated list ordered from most least likely, in the form "cs.XX", and provide your reason- ing. Answer:
0.720
Title first
Titl: <title text> in Abstract: <abstract text> \n Question: Which arXiv CS sub-category does this paper belong to? Give likely arXiv CS sub-categories as a comma-separated list ordered from most to least likely, in the form "cs.XX", and provide your reason- ing. In In Answer:
0.695
Focus text content
Title <title text> in Abstract: <abstract text> \n Question: Which arXiv CS sub-category does this paper belong to? Give 5 likely arXiv CS sub-categories
0.695
comma-separated list ordered from most least likely, in the form "cs.XX". Focus only on content the actual text and avoid making false associations. Then provide reasoning.
Chain of thought prompt
Title: <title text> In Abstract: <abstract text> \n Question: Which arXiv CS sub-category does this paper belong to? Give 5 likely arXiv CS sub-categories most to least likely, the form "cs.XX". Please think about the categorization step by step manner and avoid making false associations. Then provide your reasoning.
0.705
comma-separated list ordered from
Table 7: Study of the robustness of prompt on ogbn agbn-arxiv dataset.
LLM (zero-shot)
TAРE (GCN)
TAPE (SAGE)
TAPE (RevGAT)
Default prompt
0.720
0.7520 + 0.0003
0.7672 # 0.0007
0.7750 - 0.0012
Focus on text content Chain of thought prompt
0.695
0.7425 + 0.0021
0.7598 = 0.0006
0.7660 = 0.0017
0.705
0.7424 0.0019
0.7597 # 0.0034
0.7667 = 0.0028
G DATASET
We conduct experiments on five TAGs Cora (McCallum et al.) 2000), PubMed (Sen et al., 2008), ogbn-arxiv ogbn-products (Hu et al.) 2020a), and t tape-arxiv23. For Cora and PubMed, we collected the raw text data since they are not available in common repositories like PyG and DGL. For ogbn-products, given its substantial scale of million nodes and 61 mil- lion edges, we have employed a node sampling strategy to obtain subgraph containing 54k nodes and 74k edges. Additionally, introduced the tape-arxive arxiv23 citation graph dataset, extending beyond the knowledge cutoff GPT-3. This dataset serves as a valuable resource for the research community. 8 provides summary of the dataset statistics.
G.1 DATASET DESCRIPTION
Cora McCallum et al. 2000). The Cora dataset comprises 2,708 scientific publications classified into one of seven classes case based, genetic algorithms, neural networks, probabilistic methods, reinforcement learning, rule learning, and theory, with citation network consisting of 5,429 links. The papers were selected in a way such that in the final corpus every paper cites or cited by at least other paper.
18

---

Published as a conference paper at ICLR 2024
Table 8: Statistics of the TAG datasets
Dataset
#Nodes
#Edges
Task
Metrie
Augmentation
Cora
2,708
5,429
7-class classif.
Aecuracy
Pubmed
19,717
44.338
3-class classif.
Accuracy
ogbn-arxiv
169,343
1,166,243
40-class classif,
Accuracy
ogbn-products (subset)
54,025
74,420
47-class classif.
Accuracy
tape-arxiv23
46,198
78,548
D0-class-classtee
Accuracy
PubMed (Sen et al., 2008). The Pubmed dataset consists of 19,717 scientific publications from PubMed database pertaining to diabetes classified into one of three classes Experimental induced diabetes, Type diabetes, and Type 2 diabetes. The citation network consists of 44,338 links.
ogbn-arxiv (Hu et al., 2020a). The ogbn-arxiv arxiv dataset a directed graph that represents the citation network between all computer science arXiv papers indexed by MAG (Wang et al.) 2020). Each node is an arXiv paper, and each directed edge indicates that one paper cites another one. The task is to predict the 40 subject areas of arXiv CS papers, e.g... es. cs.LG, and cs.OS, which are manually determined (i.e., labeled) by the paper's authors and arXiv moderators.
ogbn-products (Hu et al., 2020a). The ogbn-products dataset represents Amazon product co- co-purchasing network, with product descriptions as raw text. Nodes represent products sold in Amazon, and edges between
products indicate that the products purchased together. The task is to predict the category of a product in multi-class classification setup, where the 47 top- level categories are used for target labels.
tape-arxiv23. The tape-arx v2. dataset is a directed graph that represents the citation network between all computer science arXiv papers published in 2023 or later. Similar to ogon-arrnn arxiv, each node is an arXiv paper, and each directed edge indicates that one paper cites another one. The task to predict the 40 subject areas of arXiv CS papers, e.g.,, cs.Al, es.LG, and cs.OS, which are manually determined (i.e., labeled) by the paper's authors and arXiv moderators.
G.2 DATASET SPLITS AND RANDOM SEEDS
In our experiments, we adhered to specific dataset splits and employed random seeds for re- producibility. For the ogbn-arxiv and ogbn-products dataset, adopted the standard train/validationntest split provided by OGB (Hu et al. (020a). As for the Cora, PubMed datasets, and tape-arxi v23, data was allocated for training, 20% for validation, and 20% for testing. Additionally, we utilized random seeds to ensure the reproducibility of our experiments, enabling the consistent evaluation of our proposed method on the respective datasets, which can be found in our linked code repository.
performed the train/validationntest splits ourselves, where 60% of the
G.3 SHALLOW EMBEDDING METHODS FOR NODE FEATURE EXTRACTION
Table 9 provides an overview of the text preprocessing and feature extraction methods commonly used in graph libraries such as PyG and DGL,, which are widely adopted in GNN research.
These text preprocessing and feature extraction methods facilitate the extraction node features from the text attributes of TAG datasets, enabling the utilization of GNN models for node classifi- cation tasks. While these methods are easy to apply and computationally efficient, it is important to note that they rely traditional language modeling techniques that may not capture the full seman- tie meaning in the text. This limitation can impact the expressiveness of the extracted node features and potentially affect the development of techniques for downstream tasks.
H EXPERIMENT DETAILS
H.1 COMPUTING ENVIRONMENT AND RESOURCES
The implementation the proposed method utilized the PyG and DGL modules, which are licensed under the MIT License. The experiments conducted in computing environment with the
19

---

Published as a conference paper at ICLR 2024
Table 9: Details of text preprocessing and feature extraction methods used for TAG datasets.
Dataset
Methods
Features
Description
Cora
BoW
1,433
After stemming and removing stopwords there is vocabu- lary of size 1,433 unique words. All words with document frequency than 10 were removed.
PubMed
TF-IDF
500
Each publication in the dataset is described TF/IDF weighted word vector from a dictionary which consists 500 unique words.
ogbn-arxiv
skip-gram
128
The embeddings individual words are computed by run- ning the skip-gram model (Mikolov et al. 2013) over the MAG (Wang et al. 2020) corpus.
ogbn-products
BoW
100
Node features are generated by extracting BoW features from the product descriptions followed by Principal Com- ponent Analysis reduce the dimension 100.
tape-arxiv23
word2vec
300
The embeddings of individual words are computed by run- ning the word2vec model.
following specifications: LM-based experiments were performed on four NVIDIA RTX A5000 GPUs, each with 24GB VRAM. On the other hand, the ENNN-bsse experiments were conducted on single GPU.
H.2 HYPERPARAMETERS
Table 10 provides overview of the hyperparameters used for the GCN Kiipf & Welling 2016),
SAGE (Hamilton et al. 2017). and RevGAT (Li ct al. 202T) models. These hyperparameters were selected based the official OGB repository and the RevGAT and language model hyperparam- eters follow those used in the GLEM repository
important to note that these hyperparame- ters were tuned on per-dataset basis, but instead were used consistently across all three TAG datasets based on those from prior work, and also set consistently across both our proposed method and the baselines. This demonstrates the generality and ease of of method, well as its compatibility with existing GNN baselines.
Table 10: Hyperparameters for the GCN, SAGE, and RevGAT models.
Hyperparameters
GCN
SAGE
RevGAT
layers hidden dim
3
3
256
256
256
learning rate
0.01
0.01
0.002
dropout
0.5
0.5
0.75
epoch
1000
1000
1000
warmup epochs
0
0
50
early stop
50
50
50
H.3 DETAILED ABLATION STUDY
We conducted a detailed ablation study on the ogbn-arxiv dataset to assess the impact of different sources of node features. The study focused on three types of node features: original text features (horig), explanation as features (hespl), and predictions as features (hpred). We systematically removed one of these features at time while keeping the other components unchanged in our model.
The results of the ablation study are illustrated in Figure 4 The figure presents the performance of the model when each type of node feature is removed. It is observed that using the full set of features
20

---

Published as a conference paper at ICLR 2024
Ablation
GCN
SAGE
RevGAT
Full
0.7520 = 0.0003
0.7672
0.0007
0.7750 0.0012
horig
0.7471
0.0007
0.7433 0.0005
0.7656 0.0038
hexpl
0.7506 0.0011
0.7528 # 0.0024
0.7693 0.0033
hpred
0.7519 0.0019
0.7605 0.0008
0.7686 0.0051
Figure 4: Effect node features. We study the effects of different sources of node features on the ogbn- eegbn-axxt arxiv dataset, i.e., original text features (horig), explanation features (Aeep)) and predic- tions as features (Apea), by removing one of them turn from our model while keeping the other components unchanged.
yields the best performance, while leaving out any of the features leads to drop in performance. However, the extent of the performance drop may vary depending on the specific GNN model used.
This ablation study provides additional insights to complement the findings presented in section 553 While Table 3 compared the performance of using the full set of features versus using just one of them, this ablation study specifically focuses on comparing the performance of using the full set of features versus leaving of them out. Although the experimental design differs, the overall message conveyed remains consistent, emphasizing the significance considering all the various sources of node features for achieving optimal performance in node classification tasks.
1 EFFECT OF LM FINETUNING
We conduct ablation study ogbn-arxis to explore the impact of language model (LM) fine-tuning. Specifically, we aim to address the following research questions (RQs):
RQQ: Is fine-tuning the LM necessary?
* RQ2: Is it necessary to use different LMs for encoding the original text and explanations?
To address these questions, we examine three settings: 1) Without Fine-Tuning: Utilizing pre- trained IM to encode the original text and the explanations without any fine-tuning. 2) Fine-Tuuning (Same LM): Fine-tuning single LM for both the original text and the explanations. 3) Fine- Fine-Tuning (Different LMs): Fine-tuning two separate LMs, one for the original text and another for the expla- nations.
Table 11: Effect of LM finetuning on ogbn arxiv
LM
MLP
GCN
SAGE
RevGAT
Without Fine-Tuning
0.5797 0.0217
0.4178 # 0.1148
0.4507 = 0.0529
0.7507 + 0.0189
Fine-Tunin Tine-Tuning (Same LM)
0.7566 = 0.0015
0.7442 + 0.0012
0.7676 = 0.0032
0.7728 & 0.0014
Fine-Tuning (Different LMs)
0.7587 = 0.0015
0.7520 # 0.0003
0.7672 = 0.0007
0.7750 # 0.0012
Our observations include:
For RQ1: Table [ (nderscores the importance of fine-tuning the L.M. It reveals a marked decline in performance without fine-tuning, compared with the settings where the LM fine-tuned.
For RQ2: Fine-tuning, whether with the same LM or with different LMs, yields similar outcomes, with a slight advantage for using two distinct LMs. However, the marginal difference suggests that our approach could be simplified and expedited by utilizing single L.M.
J EFFECT OF DIFFERENT LMS
To access the influence of different LMs, expand investigation beyond deberta-bass. Specif- ically, following the approach taken in SimTAG (Duan et al. 2023). we include two additional
widely-used L.Ms from the MTEB (Muennighoff et al. 2022 leaderboard, The selection is based on
21

---

Published as a conference paper at ICLR 2024
their model size and performance in classification and retrieval tasks: All-roberta-lrge-- (Reimers & Gurevych
2019) and e5 e5-large (Wang et al. 2022). The outcomes of our study are detailed in
Table 12: Effect of different LMs on ogbn-arxiv
LM
MLP
GCN
SAGE
RevGAT
deberta-base
0.7587 # 0.0015
0.7520 0.0003
0.7672 # 0.0007
0.7750 0.0012
all-roberta-large-v
0.7587 = 0.0003
0.7412 + 0.0015
0.7695 0.0008
0.7737 0.0004
e5-large
0.7595 = 0.0015
0.7443 # 0.0021
0.7688 # 0.0010
0.7730 + 0.0006
Notably, model exhibits insensitivity to the choice of specific LM, underscoring its robustness to variations in LM selection.
Table 12
K
MEMORY UTILIZATIION
Table 13 presents the memory utilization for experiments conducted on the ogbn-arxiv dataset.
Table 13: Memory Usage on ogbn-arxiv dataset with DeBERTa-base ad LM backbone and RevGAT GNN backbone for comparision of different training paradigms of fusing LMss and GNNs, including our proposed method and the state-of-the-art GL.EM method. All experiments are performed on NVIDIA RTX A5000 24GB GPUs with a batch size of 36.
Memory
Model
Accuracy
LM
GNN
Pure LM
8,834 MB
0.7361 0.0004
GNN w/ shallow fcature
4,430 MB
0.7083 0.0017
LM-based GLEM
11,064 MB
8,112 MB
0.7657 0.0029
LLM-based TAPE (Ours)
8,834
4,430 MB
0.7750 0.0012
There is trade-off between memory consumption and accuracy. Our model appears to be the most efficient in terms of memory-to-accuracy ratio. It does not require more memory than the pure LM or GNN with shallow feature models, yet it delivers the best accuracy.
L GLEM
Zhao et al. (2022) evaluated GLEM on the ogbn-arxiv dataset. We extended our evaluation of GLEM with the Cora and PubMed datasets for a more comprehensive comparison with our method. Results are reported in Table 14
Table 14: GLEM Zhao et al.
2022
Dataset
GCN
SAGE
RevGAT
Cora
0.8732 + 0.0066
0.8801 = 0.0054
0.8856 0.006
PubMed
0.9469 + 0.0010
0.9459 + 0.0018
0,9471 0.002
ogbn-arxis
0.7593 + 0.0019
0.7550 # 0.0024
0.7697 + 0.0019
22


Reviews:
[{"review_id": "2iJJkcKwnO", "reviewer": "Reviewer_NoEV", "summary": "This paper proposes a method to solve TAG problems. The method uses an LLM to generate category prediction and explanation of the target node, then the explanation is used as an enhancing part of the node feature. An LM is used to encode the raw text feature and the additional explanation into hidden space. A GNN predictor receives the hidden features, the prediction of LLM, and the shallow embeddings to give the final classification results. Experiments on several TAGs show the effectiveness of the method.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow.\n2. Bagging LLM prediction, LM features and shallow features is reasonable for the node classification task.\n3. The experiments show the method can achieve SOTA performance on several benchmark datasets.", "weaknesses": "1. The main concern is that the method has little to do with graph. It seems like an application attempt of LLMs on natural language tasks, and TAG is just a scenario. Thus the contribution of the method is limited.\n2. Since the Debera need fully fine-tuning, training the method cost much more memory than pure GNN methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bqSDLBvRGL", "reviewer": "Reviewer_HEW1", "summary": "This paper proposes a framework for representation learning on text-attributed networks. The authors first prompt the LLM to obtain explanations and predictions for each node based on its textual information. Then they finetune language models on different textual attributes and obtain feature embeddings. Finally, the graph neural network adopts the learned embeddings from the second step to conduct the final prediction.", "questions": "Please refer to the comments raised in the weakness section.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The author proposes a method to augment LM with LLM-generated features.\n2. The paper is easy to follow.", "weaknesses": "1. Lack of baselines. Many important baselines that conduct learning with language models on TAGs including GraphFormers [1] and Patton [2] are missing in the experimental section.\n2. The theorem is not specific to this problem. The theorem in 4.4 is not particular for LLM and LM for TAGs, and needs strong assumptions.\n3. Lack of evaluation tasks. The paper claims to do representation learning on TAGs but only evaluates the node classification task. It would be better to add experiments on other tasks such as link prediction to evaluate the quality of the representations.\n4. Lack of ablation studies. 1) If we need to finetune the LM in step 2 or not?; 2) Why do we need to have different LM for original text encoding and explanation encoding?\n5. Limit novelty. While I appreciate the introduction of the new arxiv-2023 dataset, the technique contribution of this work is very limited, which basically introduces LLM to conduct data augmentation for node classification on TAGs.\n\n[1] J. Yang, et al. GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph. NeurIPs 2021. \n\n[2] B. Jin, et al. Patton: Language Model Pretraining on Text-Rich Networks. ACL 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cHYscOybi9", "reviewer": "Reviewer_Dm45", "summary": "This paper addresses the challenge of leveraging Large Language Models (LLMs) such as GPT to enhance the performance of Graph Neural Networks (GNNs) on Text-Attributed Graphs (TAGs). The authors propose an innovative approach called GRAPHTEXT, which combines LLMs' textual modeling abilities with GNNs' structural learning capabilities. The key innovation lies in using explanations generated by LLMs as features to boost GNN performance on downstream tasks.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method presents a novel and effective way to integrate the power of LLMs, like GPT, with GNNs to handle text-attributed graphs. By using LLMs to generate explanations and converting them into informative features for GNNs, it bridges the gap between textual information and graph structure, enabling more sophisticated graph reasoning.\n\n2. The experimental results demonstrate that the proposed method achieves state-of-the-art performance on well-established TAG datasets, including Cora, PubMed, ogbn-arxiv, and a newly introduced dataset, arXiv-2023. This underscores the effectiveness of the proposed approach in improving the accuracy of downstream tasks on TAGs.\n\n3.The proposed method not only enhances performance but also significantly speeds up training. It achieves a 2.88 times improvement over the closest baseline on the ogbn-arxiv dataset. This computational efficiency is crucial for practical applications and scalability.", "weaknesses": "1. The time analysis and money estimation are lacking. The paper utilizes the chatgpt API, nonetheless, there is a large restraint on the word limitation per day, and the cost of one dataset should also be taken into consideration. \n\n2. The robustness of the prompt is lacking. The paper proposed a single prompt for the node classification. Nonetheless, is another similar prompt can achieve a similar performance. \n\n3. There may lack of ablation study on the LM. Current LM focuses on BERT, while other LM are ignored. Recent paper demonstrates that they can achieve satisfying performance with only SentenceBert Embedding\n\n4. There lack some experimental results in table 1, especially Giant. \n\n[1] Duan, Keyu, et al. \"Simteg: A frustratingly simple approach improves textual graph learning.\" arXiv preprint arXiv:2308.02565 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)