LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Authors: Hang Gao
Huang Wenxuan
Fengge Wu
Zhao Junsuo
Changwen Zheng
Huaping Liu
Venue: ICML 2025 poster (2025)
Keywords: Large language models; Graph Neural Netwroks; Causal
Primary area: deep_learning->graph_neural_networks
TL;DR: This study analyzes the under-explored properties of LLM-enhanced GNNs using interchange interventions and proposes an optimization module to improve information transfer.

Abstract: The use of large language models (LLMs) as feature enhancers to optimize node representations, which are then used as inputs for graph neural networks (GNNs), has shown significant potential in graph representation learning. However, the fundamental properties of this approach remain underexplored. To address this issue, we propose conducting a more in-depth analysis of this issue based on the interchange intervention method. First, we construct a synthetic graph dataset with controllable causal relationships, enabling precise manipulation of semantic relationships and causal modeling to provide data for analysis. Using this dataset, we conduct interchange interventions to examine the deeper properties of LLM enhancers and GNNs, uncovering their underlying logic and internal mechanisms. Building on the analytical results, we design a plug-and-play optimization module to improve the information transfer between LLM enhancers and GNNs. Experiments across multiple datasets and models validate the proposed module.

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Hang Gao 12
Wenxuan Huang 123
Fengge Wu 123
Junsuo Zhao 123 Changwen Zheng 123 Huaping Liu"
Abstract
2024). One category of methods employs LLMs feature enhancers to optimize node representations (Chen al., 2023; Huang et al., 2024a), which are then used as inputs for GNNs to build a unified model. Such methods leverage the pre-trained knowledge of LLMs to generate richer and more semantically coherent features, addressing the lim- itations of traditional GNNs while incorporating domain knowledge from LLMs into the features (Yu al., 2023). They also demonstrate strong generalization capabilities in heterogeneous graph representation learning scenarios (Li et al., 2025). Numerous recent methods (Chen et al., 2023; Huang et al., 2023b; Liu et al., 2024) have adopted this LLM-enhance--plus-GG GNNN framework, achieving excellent results across various graph representation learning tasks.
The use of large language models (LLMs) as fea- ture enhancers to optimize node representations, which are then used as inputs for graph neural networks (GNNs), has shown significant poten- tial in graph representation learning. However, the fundamental properties of this approach re- main underexplored. To address this issue, we propose conducting more in-depth analysis of this issue based on the interchange intervention method. First, we construct a synthetic graph dataset with controllable causal relationships, en- abling precise manipulation of semantic relation- ships and causal modeling to provide data for analysis. Using this dataset, conduct inter- change interventions examine the deeper prop- erties of LLM enhancers and GNNs, uncovering their underlying logic and internal mechanisms. Building on the analytical results, we design a plug-and-play optimization module to improve the information transfer between LLM enhancers and GNNs. Experiments across multiple datasets and models validate the proposed module. Codes can be found in https: / git.ub. com/ wode htt///LLEECCSSS Mechanism..
However, despite their broad applications, there is notice- able lack of dedicated research examining the fundamental framework of the L.MM-nhhacer-pphe paeee-pluus--ie paradigm. The deeper properties and mechanisms underlying this frame- work remain largely unexplored. This paper aims to address this gap. Meanwhile, achieving this is far from straightfor- ward. Since both the LLM enhancer and GNN are composed of neural networks, each is inherently challenging model formally (Lu et al., 2024). When these two complex net- works are combined, conducting unified analysis becomes even more difficult.
To address this issue, we introduce the interchange intervention approach (Wu et al., 2023a; Huang et al., 2023a) from causality the- ory (Pearl, 2022) to per- form the analysis. Specifi- cally, we construct a novel synthetic graph dataset with controllable causal re- lationships. Here, causal relationships refer to the ground-trute cause-and- connections, which can be utilized to evaluate the model's ability to accurately capture and represent key underlying information. While synthetic graph datasets are often used to analyze the properties of graph representation learning algorithms (Wu et al., 2022). our dataset is further enhanced enable precise manipula-
1. Introduction
Neural Network
Hlggh- Level Causal Model
With the rapid development of LLMs (Brown et al., 2020: Devlin et al., 2019; Dubey et al., 2024), their semantic un- derstanding and feature generation capabilities have demon- strated significant potential across various fields (Tian et al., 2024; Mustapha, 2025; Liu et al., 2025). In the domain of graph representation learning, recent studies have integrated LLMs with GNNs to enhance performance (Mao et al.,
Variable
Match
Variable
Figure 1. A simple illustration of variable alignment achieved by interchange intervention.
'Equal contribution 'Institute of Software, Chinese Academy of Sciences "National Key Laboratory of Space Integrated In- formation System University of Chinese Academy of Sci- ences "Tsinghua University. Correspondence to: Fenge Wu <fengge@ Bainggo@iscas.accom as..s.ac.cn..
Proceedings of the 4 International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
tion of complex semantic relationships. This feature makes the dataset more suitable for evaluating the performance and characteristics of the LLM-enhhnncerpplu--CCK els. Based on this dataset, we generate training and testing data. Using the predefined data synthesis process, we model the intrinsic causal relationships in the data as a high-order causal model, which accurately captures the complex rela- tional structures within the dataset. Subsequently, we train and test the GNN model enhanced by the LLM enhancer using the synthetic data. Through the interchange interven- tion method, we systematically analyze the correspondence between the LLM-cchancer-pluss pluserrN--NNNNNNN model and the high- order causal model, aiming to uncover the internal logical structure of the black-box neural network. Figure provides simplified illustration of the objectives we aim to achieve. Furthermore, we theoretically validate the reliability of this analytical approach and derive series of important findings.
2022; Elazar al., 2022; Wu al., 2023b). A significant
body of research has also focused specifically causal modeling in GNNs or LLMs (Chen et al., 2022; Ko et al., 2023; Gao et al., 2023; 2024; Zhao et al., 2025), We, on the other hand, utilize the causal mechanism identification approach for the specific analysis of the LLM-enhancer- plus-GNN paradigm.
mod-
Enhancing GNNs with LLM. Using LLM for initial node feature processing followed by GNN to explore inter-node relationships has become a popular research direction (Mao et al., 2024; Fatemi et al., 2023; Chen et al., 2023; Huang et al., 2023b; Liu et al., 2024; Huang et al., 2024a; Tang et al., 2024). Among the methods within this field, some combine prompt learning for graph data enhancement (Liu et al., 2024; He et al., 2024; Tang et al., 2024), others em- phasize the use of LLMs for class-level information and application usages (Yu et al., 2023; Ren et al., 2024; Lyu et al., 2023). Our research aims to conduct a thorough eval- uation of the overall framework of these methods. Further related works can be found in Appendix A.
Additionally, based on our findings, we identify areas for improvement in the information transfer between the LLM enhancer and GNN. To address this, we developed a plug- and-play optimization module designed to better assist the LLM enhancer in optimizing GNN performance. The effec- tiveness of this module has been validated across multiple datasets and various models. Our contributions are summa- rized as follows:
3. Analysis
3.1. CCSG Dataset
Our goal is to investigate the LLM-enhancer-plu--GNN paradigm, studying its capabilities in data relationship mod- eling. To accurately assess such capabilities, it is essential to first identify the relationships present in the dataset. Given the challenges of determining causal relationships in general graph datasets, we have developed a semantically rich graph dataset with controllable internal causal relationships, re- Graph (CCSG)
We construct a synthetic graph dataset with control- lable causal relationships, capable of simulating com- plex semantic associations within graphs.
We design an analysis framework based on interchange intervention and demonstrate its effectiveness both the- oretically and experimentally. Additionally, we also conduct comprehensive study on the LLM-enhancer- Plus-GNN paradigm, yielding a series of findings.
ferred to as the Controlled dataset.
The CCSG dataset is constructed based on Wikipedia en- tries, incorporating various types of causal relationships that we have defined. Table presents a comparison between the CCSG dataset and other similar datasets. This includes attributed graph datasets (Citeseer, Wiki-CS), which con- tain semantic node attributes, as well as synthetic graph datasets (Synthetic Graph, Spurious-Mori are manually generated datasets that are used for analyzing the properties of GNNs. We provide a detailed introduction below.
We propose a novel optimization module to enhance the information transfer between the LLM enhancer and the GNN, validating its effectiveness through ex- periments on multiple datasets and models.
2. Related Works
CRCG), which
Causal Mechanism Identification within Neural Net- works.
Research on explaining and understanding deep learning models has been ongoing. For this purpose, some studies attempt to uncover interpretable causal mechanisms within neural networks (Geiger et al., 2020; 2021) and train- ing methods for inducing such interpretable mechanisms (Geiger et al., 2022b; Huang et al., 2023a). These methods can be classified into iterative nullspace projection (Rav- fogel et al., 2020; Elazar et al., 2021; Lovering & Pavlick, 2022), causal mediation analysis (Meng et al., 2022; Vig et al., 2020), and causal effect estimation (Abraham et al.,
3.1.1. NODE ATTRIBUTES
The node attributes of the CCSG dataset include manually generated features and 5,660 Wikipedia entries, organized into three main categories and fifteen subcategories, ensur- ing diverse information. This dataset forms the foundation for graph construction of CCSG dataset, with detailed de- scriptions provided in Appendix C.
2

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Table 1. Comparative analysis of our dataset with other similar datasets. The term "Total Combinations" refers the maximum possible number of combinations attainable when all available graphical elements are employed and juxtaposed in pairs.
Adjustable Node
Somantic Node
Adjustable Edges
Multi-order Relationship
Controllable Causal Relationship
Semantic Aware Relationship Manipulation
Total
Dalaset
Attributes
Attributes
Adjustment
Combinations (1)
Citeseer
(Kipf& & Welling, 2016)
Fixed
Fixed
Wiki
(Witmman & Fey, 2020)
X
Fixed
Fixed
Synthetic Graph
Adjustable
(Ying et al., 2019)
X
X
First-order Relationship
X
Spurious-Motf (Wu et al., 2022)
Adjustable
First onder Relationship
36
CRCG
Adjustable
(Gao et al., 2024)
V
X
First-order Relationship
X 3750
Adjustable
CCSG
Multi-order Relationship
226400
3.1.2. GRAPH CONSTRUCTION
and Gdlff We first input G into the model h(-). Then, we select a variable zh within l(()) and replace its value with the value it would take if Gdlff were the input. This produces new output for h(-) where the input remains G but the variable Zd reflects the state it would have under G We denote the resulting output as INTINT(h, Gorie, Gdift ) where INTINT(-) denotes the conducted interchange inter- vention operation.
For graph construction, the CCSG dataset ensures control- lable generation of node features, connections, and topolog- ical structures. Data generation is controlled in four aspects: 1) Node features, the semantic information of node features can be actively controlled based on the collected data. 2) Node correlation, node feature correlations are adjusted based on the collected categories, subclasses, and reference relationships. 3) Topological structure, diverse topolo- gies are introduced and linked to label data to analyze the impact of graph structure on outcomes. Such controllable graph construction enable the injection of predefined and will formulated causal relationships to be added to the data.
Subsequently, we perform interchange intervention on f(-) using both Gorig and Gdiff which is denoted as INTINT(J, Gong, Gullt Z!). Here, 2 refers to an in- ternal variable within f(-). Intuitively, if 2f is the hidden layer variable corresponding to Zh in the neu- ral network model, then INTINT( G Gdlit zd) and INTINT(J, Gorie, Gdlfft Z should be equal. We compute interchange intervention loss Ln to measure the discrepancy between them:
3.2. Interchange Intervention Based Evaluation
We analyze the JLM--nhanccrrplu--CN GNN paradigm by eval- uating its ability to model predefined causal relationships. To do this, we apply the interchange intervention method (Geiger et al., 2022a) from causal inference. This method treats the neural network as a low-level model and the dataset's causal relationships as a high-level causal model. By modifying the neural network's 's internal feature repre- sentations and comparing them to changes in the high-level causal model, it evaluates the correspondence between the two. This process enables us to explore the underlying mechanisms of the model in greater depth.
Σ Σ
G D DINNINNV Gaii Ccliff,
Lu
G2
GOEEg GaNEg
INTINV (f. Gonie Gdlfft C .Z1))
where D(-) represents the metric used to measure the difference between INTINV G Gilift 2d) and INTINV (umig Gthl 2²), e.g., for classification tasks, D(.) can be the cross-entropy loss. The set G denotes the dataset being utilized. We search for the optimal Z' that minimizes Lu, which would ensure Z in f(-) best aligns with 2h in h(-). See Section 3.2.3 for justifications. In this way, we can establish the correspondence between the low-level neural network and the high-level causal model.
3.2.1. EVALUATION METHOD OUTLINE
Based on the CCSG dataset, we first construct the data and high-level causal model, denoted h(-). The model h(-) represents the underlying causal relationships between the data and the corresponding labels, and it out- puts the ground truth label based on the input G. Our ob- jective is to employ an interchange intervention to iden- tify which hidden variables in the low-level neural net- work model specificall, an LLM enhancer-plus- GNN model-correspond to the variables in h(-))
3.2.2. RUNNING EXAMPLE
We present simplified example to illustrate the application of the interchange intervention-basee analysis method. As shown in Figure 2, the process is divided into four steps. Below, we provide detailed step-by-step explanation of the process.
Specifically, we select two different graph samples: Gurig
Step 1: Data Generation, this step, as shown in Figure 2,
3

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Step 1: Taata generation
Step 2: Obtaining the output of h()
Step 3: Ohtaining the output of RC)
Graph sample
Galli
bCC) Classes of the
b(O Classes of the neighborrs
[()
neighbors
L.M layer
L.M layer
Class 2
Jyyelayer
LLMelayer
Change Variable
Change
Graph sample
Class of the target node
Class of the target node
GNN layer
1..
Variable
Hidden
layer
GNN layer
Class I
Class
Outpul
Ourpet
calculation and
L1
Figure The framework of the running example.
we construct only two attributed graph samples, G1 and G2- In practice, the number of samples would be significantly larger. The task is to classify node depicted in the figure. During data generation, we also define the high-level causal model h(-) for this example. Specifically, we manually establish that the class of node v is determined by first identifying the classes of all its neighboring nodes and then selecting the most frequent class among them as the class of v. This logic can be formally expressed as C(v) m((C(()
within f() to facilitate analysis.
3.2.3. VALIDATION
To conduct validation of our analysis method, we introduce the concept of total effect from causal theory (Pearl, 2009).
Definition 3.1. Total effect (Pearl, 2009) represents the over- all causal impact of one variable Z on another variable Y. It is denoted by TF (Y), where z and z' are two specific values that can assume. The total effect xE (Y) quan- tifies the expected difference in the outcome Y when Z is set compared to when it is set to z'.
[()))), where C(.) represents the classes of nodes and (v)) denotes the set of neighbors of v, m(-) selects the most frequent class. Consequently, we have:
GG) = C(v) = m((((u) (())))
Therefore, we can formalize our objective more precisely: to find variables 2º in model h(-) and Z' in model f()) such that 2h and Z exhibit consistent total effects on the predicted outputs. As demonstrated in Equation 1, we aim to find the aforementioned variable Z by minimizing a loss function Lu We use the following theorem to prove the validity of this approach.
We then select C(us) within h(-) as Zh for performing the interchange intervention.
Step 2: Calculating INTINV Gariz, Grdif Grdit 2*). We se- lect G1 as Gorig and input it into h(-)) to accurately compute the values of various variables and outcomes within the model. We then select G2 Gdiff and input it into h(-), performing another round of calculations for the variables and outcomes. As shown in Figure 2, Step 2, when the input is Gorie, we replace the value of 2h, i.e., C(us), with its value obtained from Cdiff as input and compute the output. This final output represents INTINV(h, Gorig, Gdill ).
Theorem 3.2. Given a high-level causal model h(()) and low-level neural network model f()), both of which accu- rately map input graphs G E & to outputs Y E Y, such that Y = f(G) = h(G). Assume there exists a subset Z of the intermediate variables in f(-) and a bijective mapping
rj : 2 - Zh, where Zh represents certain variables in h(()) If there exists Z that minimizes the loss Lut we can conclude that the total effect TE gl wr' (Y5) of f()) is equal to the total effect TEg"." (Y") of hi(.) in all cases. Here, zh and zt represent the values of 2 and 7 respec- tively, given the same impul graph G. Similarly, zt and z!' represent their values for a different input graph G'.
Step 3: Calculating INTINV(J, Gorle, Gorie,, , Gdllt ).). We lect G1 as Gorig and input it into f()). Similarly, we select G2 as Gdift and input it into f-)). Since the optimal selec- tion of 2 is not yet known, we randomly choose certain hidden layer variables as Z We then take the values of z obtained when Giill is used as input and replace the cor- responding values in the model ʃ(() when Gorig used as input. The model then outputs (N Gunig Guhlff ).
The demonstration can be found in Appendix B.1. Theorem 3.2 establishes that, under the condition where the model f(-) performs sufficiently well and bijection exists be- tween the internal variables of f()) and the internal variable z of f(-),, minimizing Lu yields a Z whose total effects on the model outputs are exactly the same as those of the cor- responding ²h Equal total effects indicate that the neurons modeling the variable 2h have been precisely identified,
Step 4: Loss Calculation and Minimization. Repeat Step 2 and 3 to acquire all INTINV Gorie, Gdifff 7n) and INTINV(f, Gunig Gdilf ZF) values, then utilize Equation 2 to calculate the loss Ln- We then repeat the process to locate optimal z that minimizes Cu, finding the best alignment of 7h within f(),, so as to reveal part of the logical structure

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
thereby providing theoretical validation for the effectiveness of the proposed method. However, if the aforementioned bijection does not exist, how will the results change? The following corollary provides further analysis.
2º W
0.006
0.008
0.075
0,014
0.124
2
22
0.003
0.013
0,901
0,019
1,792
2t
0.011
0.013
0,112
3,277
Layer
Layer
Layer 2
Layer 3
Layer
(a) The results of node-level experiments where 2th is set the output variables V 1, 22, and V/ of hooee. (-).
Corollary 3.3. Even if the condition that there exists a subset Zt of the intermediate variables of f() satisfied n Z + Zh where di is bijective does not hold, if INTINV Gonls Gdtff 75) INTINV(h, G Gdill 7h) holds, the conclusion given in Theorem 3.2 remains valid.
4P
1.844
0.790
4-
1.520
0.089
Layer
Layer
Layer
Layer
Layer
(b) The results of node-level experiments where 2d is set to the output variables Φ1 and 4b of h m (-).
The demonstration can be found in Appendix B.2. Corol- lary 3.3 shows that even if the aforementioned bijection does not exist, the condition INTINV G Guhlt z²) INTINV (h, Gorig Gdiff 7*) ensures the existence of Z that fully corresponds to Zh, practical analysis, we can use the distance of L from its possible minimum value to assess the degree of matching between Z' and Z*.
2
0.128
0.060
0.005
0016
0,021
2h = Tr
1.029
0.369
0.137
0,052
0.228
2
3,086
4.400
1.113
1.999
2,260
Layer 0
Layer
Layer
Layer
Layor
(c) The results of graph-level experiments where z is set to the output variables I1, 22,, and 33 of ho (-).
Proposition 3.4. For high-level causal model h(-)) and a low-level neural network f()), both mapping imput graphs G E 9 to outputs Y E y such that Y = f(G) = h(G), if 2 within f() and 2 within h(-) minimize Lu to its optimal value Lip then Z aligns best with 2h and has an identical total effect on the output prediction as Zn. The minimal Lū is given by:
2º -0=
0.117
0.036
0.042
0.0.9
0.029
-00
0.235
0.719
0.217
0.1
Layer 0
Layer 1
Layer
Layer
Layer
(d) The results of graph-level experiments where 2hh is set to the output variables Qt and 22 of (-mph-2-))
Figure 3. Analytical experiment results. In the figure, each small square represents the optimal value of Lu for specific GNN layer. Green squares indicate the minimum value of Lu within the corresponding row. Details concerning the variables be found in Appendix C.
C
Σ GR GORES
1
Σ D (INTINV(H, Garii Ch ,
INTINV(h, Goris Galll (2).
(3)
satisfying the the following condition:
INTINV(J, Goris C 22) INTINV Garis, , )*)). (4)
Please refer to Appendix B.3 for the justification. In our experiments, we use cross-entropy loss as D(.), and the label probabilities predicted by f(-) are either 0 or 1. Therefore, in our experiments, Lt = O. Consequently, Lu to represent the distance between Ln and Lj.
where Goniz and G differ at or above the seale of indi- vidual nodes. Then, the internal variables within the GNN model are always sufficient to constitute Z
directly use
The proof can be found in Appendix B.4. According to the theorem, and taking into account the difficulty of fixed L.MSs in modeling causal relationships, the subsequent anal- ysis focuses on using the GNN model to examine causal relationship modeling within the framework.
3.3. Analytical Experiments
We analyze the L.Menhancer--lls-- GNN framework using the CCSG dataset and the approach from Section 3.2, look- ing for deeper insights and possible optimization of the framework. Following prior studies on causality in graph representation learning (Wu et al., 2022; Gao et al., 2024), we treat nodes as the smallest variable units, focusing on node relationships without modeling internal node inter- actions. Most approaches use fixed-panne parrmeerrettrr LLM hancers to reduce training costs (Liu al., 2024: Huang et al., 2023b), our experiments also follow this setting.
3. NODE-LEVEL ANALYSIS
We begin our analysis with node-level tasks by first con- structing node classification dataset that leverages the rich information in the CCSG and designing m-de h node (.) outputs the class of target node v. Fach graph sample is used to classify only one target node U. This ensures the elimination of potential mutual influences that may arise when multiple nodes are classified simultancously. which could prevent (.)e(()) from being accurately established.
Theorem 3.5. Given a high-level causal model h(-) and a variable h within h()), suppose there exists a variable 7 within f(),, where f(-) is a GNN with LLM enhancers,
5

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
200
ment for h (-) is positioned slightly ahead of the optimal alignment for 1:0de.2 (.).
175
130
GNN with
--
3.3.2. GRAPH-LEVEL ANALYSIS
4Laren
125
-
For graph-level task analysis, we manually set the utilized topological structures, node features, and their interrelations, which collectively form the graph sample and determine the label. The corresponding high-level causal model hemaph can be formulated as follows:
6N
4agers
L
675
GNN
Laaer
-2
2--9:-
GNN wah
-0
Layers
-0
h (G) = h htrpph,3 o hgraph,2 o hgraph, (G),
(6)
Layer Index
(a) The results from experiments where 2h is set to the output variables of h (-).. (-), with different number of GNN layers.
where (.)aab.1(.) processes the features of substructures, (-)) denotes the function locating the topological structures, and (-) denotes the final processing func- tion that outputs the graph label. The details can found in Appendix C.
GAN with Liddee Diamnise io
135
with Hibken
The results are presented in Figure 3(c) and Figure 3(d). Similar to the phenomenon observed in the node-level ex- periments, the optimal alignment for heraph. (-) is positioned ahead of the optimal alignment for / h/raph.2 (.).
125
100
Dinmiia of 25%
030
CSN with Hidden Mimeddan of
035
Based on the experimental results at both the node-level and graph-level, it can be found that when 2d is more strongly associated with node-level information and is closer the raw input data in the logical structure of the model h(.), its corresponding latent variable Z is more likely to appear in the shallow layers of the GNN structure. This indicates that the role of the LLM enhancer is to process node-level and raw data-level information. Such a finding also partially validates Theorem 3.5. summary, this conclusion can be further generalized into the following empirical finding.
Layer Index
(b) The results from experiments where 2 is set to the output variables of heugg. (-), with dif- ferent hidden dimension.
Figure 4, Experimental results under different GNN scales.
The formal definition of hode is as follows:
node (G) = node3, nonde.2 a A.ode.1(G). (G).
(5)
Here, the symbol o denotes function composition. The function (-) is responsible for processing single node feature, while h (-) captures and analyzes the relation- ships and properties among these nodes. Finally, 3. (.) serves as the ultimate processing function, outputting the features of the target node v. The details can be found in Appendix C. To ensure generalizability, we construct our model based commonly used GNNs and LLMs Specifically, the GNN module is im- plemented using GCN (Kipf & Welling, 2016) and LLM enhancer is built using Llama 3 (Dubey et al., 2024). To ensure precise analysis, we adjust the number of training epochs and modify the task difficulty so that all models achieve an accuracy of at least 90%.
For Exedd-aarmmeter LIM en- hancers, the features output by the LLM serve the function of representing information at the node level and the raw data level.
Empirical Finding 1.
3.3.3. DEEPER INSIGHT
For further analysis, we repeated the experiments under dif- ferent model scales by increasing the number of layers and the hidden dimensions of the GNN. The results are shown in Figure 4. The experiments reveal an intriguing phenomenon: for both node-level and graph-level variables, the alignment results of 2h within the GNN exhibit a certain regularity across different model scales. For example, as shown in Figure 4, despite variations in model depth, the alignment performance is consistently worse for vven-numbeeed layers than for odd- hud-breeeredd layers, with the worst performance observed in the final layer. Furthermore, for 2* = 43. the performance remains suboptimal across all settings. Figure 4(b) illustrates that regardless of the size of the hidden di- mensions, the optimal alignment results for zh consistently
L.MM-enhancer-plu--Gin
The results are presented in Figures 3(a) and 3(b). It can be observed that the best alignment of variables (marked in green) across different node levels occurs in layer 0 and layer 1. These positions can be regarded as the locations of the internal representations z within the GNN model. Additionally, it can also be observed that the optimal align-
6

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Optimal Ca
006
1.
Optimal Eg
ACC
0.8
0.8
0.02
0.5
0.6
GNN Layers
Layers
Number of Layers
Layers
(a) Node-level task alignment
(b) Node-level task model ac- curacy with different number of GNN Layers.
(c) Graph-level task align-
(d) Graph-level task model accuracy with different num- her of GNN Layers.
results with different number
ment results with different number of GNN Layers.
of GNN Layers.
Optimal Ca
0.0
0.9
Optimal
0.998
0.8
699
0.7
1133
Liassa2
Prumpt
Liuen2
Prompt
Liamd
Prompt
0.6
4/0
Procept
Libra.
Pronett
I.bro?
1larsi
Phorer
11M
Prinpt Pronne Prinpt
Elams? L.bred
6.98
Pronett Friepe Proeept
Lerad
Libral
LLM
Types
LLM Types
LLM Types
(e) Node-Ievel alignment results with different LLMs.
(I) Node-level task model ac- curacy with different LLMs.
(g) Graph-level task align- ment results with different
(h) Graph-level task model accuracy LLMs.
with
different
LLMs.
Figure 5. Experimental results under different model scales and the use of LLMs.
accuracy no longer forced to be above 90%. Within the results in Figure 5, it be observed that there is certain correlation between the optimal alignment, represented by Lu. and the model's performance.
Lu
0.6
0.5
Feature Location
Feature Location
(a) Node-level task align- ment results with different feature positions.
(b) Node-level task model accuracy with different fea- ture positions.
Empirical Finding 3. The analysis based on Lu can par- tially reflect the capability of the model. Specifically; lower optimal Lu value generally indicates stronger model capability and vice versa.
Optinal Ki
ACC
Extra experiments within Appendix E indicate the same. Combining with Empirical Finding 2, we hypothesize that the scaling up of the GNN only enlarges its structure, with- out enhancing its capacity for causal relation modeling. Meanwhile, Figures 5(e), 5(f), 5()). and 5()) demonstrate that more powerful LLMs enhance model performance. Yet, improving the L.LM backbone remains challenging due to the high resource cost. Empirical Finding shows that the LLM enhancer in the architecture primarily provides information both the node level and the raw data level. Thus, instead shift focus to the optimization of the connection between the LLM enhancer and GNN.
10
Feature Location
Fenture Location
(c) Graph-level task align- ment results with different feature positions.
(d) Graph-level task model accuracy with different fea- ture positions.
Figure Experimental results under different feature positions. The numbers represent the relative positions of the tokens corre- sponding to adopted features within entire output, ranging from minimum of to a maximum of 10.
occur in the middle layers. In Appendix E, we present nu- merous additional experiments, all of which corroborate the existence of this phenomenon. This phenomenon leads to the following conclusion:
We modified the features transmitted from the LLM en- hancer to the GNN and conducted experiments to assess their impact. Specifically, while conventional methods (Chen et al., 2023; Huang al., 2023b; Liu et al., 2024) typically select last-layer features corresponding specific token positions in the LLM enhancer output input for the subsequent GNN, we varied these positions to alter the transmitted information. Experimental results, as shown in Figures 6, demonstrate that changes in the selection of these positions significantly affect model performance, with greater impact than the previously discussed factors. As a
Empirical Finding 2.
After receiving imput from the LLM enhancer, the neural structure within the GNN exhibits relatively consistent logical pattern, maintaining certain degree of invariance despite changes in the model's s scale.
For further exploration, experiments were performed to compare alignment with accuracy. To facilitate analysis,

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Pronpe1
Frature: Caroopndding Difierret Tokens
Altenian Seare
pprangs
Prompe
LLM
Tranduener Enceder
GNN
Enhancer
Prempt
Pronpe Selectias
Figure Framework of the proposed AT module.
Table Performance comparison across different backbone 'matches LLMs and GNNs, Diff' denotes performance change achieved with AT module.
Dutaset
Method
GrighS
AT
w/o AT
Diff
w/o AT
Dimr
Lhmm22
83.71 d: L05
$4.67 0.79
+0.96
83.67 d: 1.34
$4.17 0.43
+0.50
$4.45 1: 0.98
85.12 L,44
+0.67
Qwen2
84.33
64.55 0.77
84.11
85.34
Cora
Lluma3
$4.98 +: L.00
36.74 0.32
+1.6
$4.87
0.85
86.29 0.21
+1.42
85.13 4: 0.26
56.43 0.22
+1.30
Llena2
= 0:46
0.84
83.86
042
42.00
1.51
54.77
+2.95
Pubmed
Qwen2
81.09 = 0.34
84.04 + 1.61
#2.95
81.45 4 0.29
83.11
0.14
+1.66
81
1:29
54.35 1.64
+2.47
Ulama3
82.06
1,35
84.43 1,42
+2.37
82.48
0:88
84.86 1.60
+2.38
82.23
0.79
85. 1,49
+3.09
Liama2
62.81 4: 0,98
61.13 0.82
+1.32
63.12 1: 0.75
61.82
0.42
+1.70
64.35 1: 1.73
66.78 0.43
+2.23
Instugram
Qwen2
63.12 1.45
64.45 0.98
+1.33
62.81 +: 1.44
64.76
1,18
+1.95
64.48
1: 0.87
66.23 0.77
+1.75
Llama3
64.86
0.23
65.51
68.40
67.17
result, we chose to optimize the model based on the selec- tion of these positions.
separately to generate attention scores. Specifically, for S we acquire attention matrix A':
A' = Q'K")
4. Method
where Q' and K denote the output Query and Key matrices of the final layer of the transformer encoder. Then, compute the average attention scores for vector a E IRT the j-th element of a' can be represented as:
Based on the previous experimental analysis, propose optimizing the data transmission between the LLM en- hancer and the GNN by introducing the -tenttionbbsee Transmission (AT) module. The design of the AT module simple and straightforward, primarily leveraging an at- tention mechanism select the most optimal parts of the information output by the LLM enhancer for downstream transmission, thereby optimizing the model. Figure 7 illus- trates the framework of the module.
m
A
for j = 11,,.... m.
(8)
m
we acquire q vectors, a², a a², Next, we applied the softmax function to normalize all the elements within all vectors together and acquired normalized results a', a², â Then, calculate the final output vector of the LLM enhancer:
Specifically, to conduct a more comprehensive feature search, we first use the LLM to generate different in- put prompts for the LLM enhancer, where y a hyperpa- rameter. As a result, we obtain q different feature sets, X!,X2...., X². Each X contains the features corre- sponding to the output tokens from the LLM enhancer using the i-th prompt. Specifically, X' = {=]]);=1- where at denotes the feature of the j-th token from the final layer of the LLM enhancer when the i-th prompt is provided as input.
(9)
qm
where will be utilized as the node feature and input into the GNN. We utilize ó epochs for training the selection of the prompts, after ó epochs, we fix the utilized prompt and remove the rest.
We validated our module on Cora (Kipf & Welling, 2016), Pubmed (Hamilton et al., 2017), and Instagram (Huang et al., 2024b) datasets. We utilize Llama2 (Touvron et al., 2023).
Since the number n of output tokens is not fixed and varies with different inputs, select a total of m feature vec- tors, where is hyperparameter. The indices of the selected vectors are determined based on n and m, and are computed as follows: [[min[ x m: 1)J, [2 x n ... min(m x m 1)]3. We then select the cor- m
Qwen2 (Yang et al., 2024a), and Llama3 (Dubey et al., 2024) the LLM backbones, and GCN (Kipf & Welling, 2016), GAT (Velickovic et al., 2018), 8), and GraphS (Hamilton et al., 2017) as the GNN backbones. The results are shown in Table 2. It can be observed that the AT module effective across various LLM frameworks. All experimental settings and details are provided in Appendix D.
responding features and denote them as {
Next, we input s?2 .s....,, ST into a transformer encoder
8

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
5. Conclusion
Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
This paper analyzes the LLM-enhannerrplls-CCCM paradigm using the CCSG dataset and interchange intervention. The proposed method is validated through theory and experi- ments. Additionally, novel AT module is proposed to optimize the paradigm further.
Chalupka, K., Eberhardt, F., and Perona, Causal feature learning: an overview. Behaviormetrika, 44:137-164, 2017.
Chen, Y., Zhang. Y., Bian, Y., Yang. Ma, K., Xie, B., Liu, T., Han, B., and Cheng, J. Learning causally invariant representations for out-of-distribution generalization on graphs. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: /papers. nips cc ttcppp/ppocccccc wwwp:/aappeeaee files /ttppppee_eeeeeepppp//222222222/ hhahs/hh/22hhhh/////// 8b 3 -bstrrt Abstract-Contee Confere html.
Impact Statement
This paper presents work whose goal to advance the field of Graph Representation Learning and LLM. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Acknowledgments
We would like to express our sincere gratitude to the re- viewers of this paper, as well as the Program Committee and Area Chairs, for their valuable comments and sugges- tions. This work is supported by the CAS Project for Young Scientists in Basic Research, Grant No. YSBR-040 and Application Innovation Plan, Project No. 825DBB3C
Chen, Z., Mao, H., Li, H., Jin, W., Wen, H., Wei, X., Wang, S.. Yin, D., Fan, W., Liu, H., and Tang, J. Exploring the potential of large language models (Ilmsiin learning on graphs. SIGKDD Explor, 25(2):42-61, 2023. doi: 10. 1145/3655 133/6655103.36551100 URL https: / /ttp:./oo//////// 10. 15.1145/33555
References
3655110.
Abraham, E. D., D'Oosterlinck, K., Feder, A., Gat, Y. O., Geiger, A., Potts, C., Reichart, R., and Wu, Z. Cebab: Es- timating the causal effects of real-world concepts on NLP model behavior. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171-41866 Association for Computational Linguistics, 2019. doi: 10.18653/V 1423. URL https://doi.crr 10.
Beckers, S., Eberhardt, F., and Halpern, J. Y. Approxi- mate causal abstractions. In Globerson, A. and Silva, R. (eds.), Proceedings of the Thirty-Fitth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviy, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pp. 606-615, AUAI Press, 2019. URL http: / http//prcceee....eeee mlr. press/ #115/beckers200 html.
9-
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A.,
Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A.. Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Allonsius, D.,, Song. D., Pintz, D. Livshits. D.,, Esiobu, D., Choud- hary, D., Mahajan, D., Garcia D., Perino, D., Hup- kes, D., Lakomkin, E., AlBadawy, E.,, Lobanova, E., Di- nan, E., Smith, F. M., Radenovic, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Tou- vron, H., Zarov, I., Ibarra, I. A., Kloumann, I. M., Misra, I., Eytimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J.,
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert- Voss. A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radfordd A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H.,, Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Bil- lock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang. J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasani,
A., Belinkov, Y., Chrupala, G., Hupkes, D., Pinter, Y., and Sajjad, H. (eds.), Proceedings of the Third Black- boxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP EMNLP 2020, On- line, November 2020, pp. 163-173. Association for Com- putational Linguistics, 2020. doi: 10.18653/V//2000 BLACKBOXNLP.L.16. URL 10. 10.18653/v1/2000 http://///.22....aaaoo.o.oo.//o/ www.cboxo.pcp....1.....
K., Plawiak, K., Li, K., Heafield, K., Stone, K., and et al. The llama 3 herd of models. CoRR, abs/2407.21783. URL https: 2024. doi: 10.48550/ARXXV.240...11 21783.
///doi.og///// 10. http://...4550///8
Elazar, Y. Ravfogel, S., Jacovi, A., and Goldberg, Y. Am- nesic probing: Behavioral explanation with amnesic counterfactuals, Trans. Assoc. Comput. Linguistics, 9: 160-175, 2021. doi: 10.1 (0.1162/TACL) https: doi..oo...//o.oo org/ /0/ http//l/taaaaaaaaaeeeaeaalaaaaaa 00035_000__000099
Geiger, A., Lu, H., Ieard, T., and Potts, C. Causal abstrac- tions of neural networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, vir- tual, pp. 9574-9586, 2021.
URL
Elazar, Y., Kassner, N., Ravfogel, S., Feder, A., Ravichan- der, A., Mosbach, M., Belinkov, Y., Schüitze, H., and Goldberg. Y. Mcasuring causal effects of data statisties on language model's 'factual' predictions. CoRR, abs/2207. 14251, 2022. doi: 10.48550/ARXIV. 2207.14251. URL https::/p//......... //toi//ooo///ooo/oo www.//r....... 48550/ arXiv.2200 14251.
Geiger, A., Wu, Z., Lu, H., Rozner, J., Kreiss, E., Icard, T., Goodman, N., and Potts, C. Inducing causal structure for interpretable neural networks. In International Con- ference on Machine Learning, pp. 7324-7338. PMLR, 2022a.
Fatemi, B., Halcrow, J., and Perozzi, B. Talk like graph: Encoding graphs for large language models. CoRR, abs/2310.04566 2023. doi: 10.48550/ARXIV. 2310.04560. URL https: /ttp:::///o...// htt::/do..orrr ort../....1..... 48550/ arxiv 2310. 04560.
Geiger, A., Wu, Z., Lu, H.,, Rozner, J., Kreiss, E., Icard, T., Goodman, N. D., and Potts, C. Inducing causal structure for interpretable neural networks. In Chaud- huri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceed- ings of Machine Learning Research, pp. 7324-7338. PMLR, 2022b. URL https: //pp//eeeeeengssseeeeeee/ce...se mllr. press/vl 62 /tte//geeee22eee html.
Gao, H., J., J., Qiang, Si, L., Xu, B., Zheng, C., and Sun, F. Robust causal graph representation learning against confounding effects. In Williams, B., Chen, Y., and Neville, J. (eds.), Thirty-Seventi AAAI Conference on Ar- tificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirleenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp. 7624-7632. AAAI Press, URL https:
Geiger, A., Wu, Z., Potts, C., Icard, T., and Goodman, N. D. Finding alignments between interpretable causal variables and distributed neural representations. In Locatello, F. and Didelez, V. (eds.), Causal Learning and Reasoning, 1-3 April 2024, Los Angeles, California, USA, volume 236 of Proceedings of Machine Learning Research, pp. 160-187. PMLR, 2024. URL https: /tLttttss//prooeeeeee http://pooceeeennns//ee// .mlr. press vvl//v23552// hetr://eigee2442 .html.
2023, doi: (0.1609/AAALL333165.5555 //doi org. /10 1609 aaai. v3716.
Gao, H., Yao, C., Li, J., Si, L., Jin, Y., Wu, F., Zheng, C., and Liu, H. Rethinking causal relationships learn- ing in graph neural networks. In Wooldridge, M. J., Dy, J. G., and Natarajan, S. (eds.), Thirty-Eighth AAAI Con- ference on Artificial Intelligence, AAAI 2024, Thirty-Sixil Conference on Innovative Applications of Artificial In- telligence, IAAI 2024, Fourteenth Symposium on Edu- cational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 12145- 12154. AAAI Press, 2024. doi: 10.1609//AALL 29103. URL htttps::/doi.. http://oiioooo htt.iioorgg11..0 wtt..//.0/.....//a...... aaai. 008111.29103
Hamilton, W. I.., Ying, Z., and Leskovec, J. Inductive representation learning on large graphs. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1024-1034, 2017. URL https: /tltpp:://rree pttp//eeecccee..e/ heurips.cc//e wourips..c/ppaee//2000/ / / cadsabbeee33ddseefffbbaaaaaaaa Aat.aa//77aeebbeeaa-b-bbrteeee html.
Geiger, A., Richardson, K., and Potts, C. Neural nat- ural language inference models partially embed theo- ries of lexical entailment and negation. In Alishahi,
He, X., Bresson, X., Laurent, T., Perold, A., LeCun, Y., and Hooi, B. Harnessing explanations: Lim-to-Im inter-
10

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
preter for enhanced text-attributed graph representation learning. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openrevice ew, net forum dd=RFF/cynVeeee
(eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 4300-4316. Associ- ation for Computational Linguistics, 2023. doi: 10.18653/ V 11/0233EMNN.-MMIIN22.. URI. ht https::/t///.. /doi. o 1tt/1...1663/v1122000 emnlp-mnnlp..... mimin... 2 261.
Huang, C., Ren, X., Tang, J., Yin, D., and Chawla, N. V. Large language models for graphs: Progresses and di- rections. In Chua, T., Ngo, C., Lee, R. K., Kumar, R., and Lauw, H. W. (eds.), Companion Proceedings the ACM on Web Conference 2024, WWW 2024, Singa- pore, Singapore, May 13-17, 2024, pp. 1284-1287. ACM, 2024a. doi: 10.1 0001144//55833353641111 ///ooi.org / 10. . 11 00444555553333 36422.
Li, G., Hu, D., Liu, Z., Zhang, X., and Lyu, H. Seman- tic reshuffling with LLM and heterogeneous graph auto- encoder for enhanced rumor detection. In Rambow, O., Wanner, L.,, Apidianaki, M., AI-Khalifa, H., Eugenio, B. D., and Schockaert, S. (eds.), Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pp. 8557-8572. Association for Computational Linguis- tics, 2025. URL https: /aa//aaaaaaaa http//aaaannooooo...ooooooooooea org/ c2r.5coolin-maiin 572/.
URL https:
Huang. J., Wu, Z., Mahowald, K., and Potts, C. Induc- ing character-level structure in subword-based language models with type-level interchange intervention train- ing. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Findings of the Association for Computa- tional Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 12163-12180, Association for Compu- tational Linguistics, 2023a. doi: 118865/31120233 FINDINGS-ACLL770 URL https::/t/////...tt /ttp///d.ooooo http///i..orgg//////// 1855//11/003.rriininssccca 77C.
Liu, A., Pan, L.,, Lu, Y., Li, J., Hu, X., Zhang, X., Wen, L., King, I., Xiong, H., and Yu, P. S. A sur- vey of text watermarking in the era of large language models. ACM Comput. Surv. 57(2):4::1 2025. doi: 10.1145/369122 URL htt.ttss::/// /ttp::://dooooooo /rr////0...o.g/./o//o/// 1145/ 6266666.
Liu, H., Feng, J., Kong, L., Liang, N., Tao. D., Chen, Y., and Zhang, M. One for all: Towards training one graph model for all classification tasks. In The Twelfth International Conference on Learning Representations, 2024.
Huang, Q.. Ren, H., Chen, P., Krzmanc, G., Zeng, D., Liang, P., and Leskovec, J. PRODIGY: enabling in-context learn- ing over graphs. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b.
Lovering, C. and Pavlick, E. Unit testing for concepts in neural networks. Trans. Assoc. Comput. Linguistics, 10: 1193-1208, 2022. doi: 10. 10.1162/TACL https://doi. http://ddoooorg////////// 0 11 /2/taccllccc _ataaacll__0000 62/taacla_001111 a_0111
URL
Huang, X., Han, K., Yang, Y., Bao, D., Tao, Q., Chai, Z., and
Lu, S., Mills, K. G., He, J., Liu, B., and Niu, D. Goat: Explaining graph neural networks via graph output at- tribution. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: / //openreieww Lew. net/ forum? wwt.mmiii=iianeeeee
Zhu, Q. Can GNN good adapter for Ilms? In Chua, T.,
Ngo, C., Kumar, R.. Lauw, H. W., and Lee. R. K. (eds.), Proceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024, pp. 893-904. ACM, 2024b. doi: 10.1 10.1145/3589334436660 36456777.. URL https: 311.///5833333333333
Lyu, H., Jiang, S., Zeng, H., Xia, Y., and Luo, J. Llm- rec: Personalized recommendation via prompting large language models. CoRR, abs/2307.15780 2023. doi: 10.48550/ARXXV22277.555.0.. URL https: //doi. org/10. 4BG.G/.555//aax...../
Iwasaki, Y. and Simon, H. A. Causality and model ab- straction. Artif. Intell., 67(1):143-194, 1994. doi: 10.1016/0000-3770299999900444 URL https:://oi.. org/10.1016/00044 3702 (94) 90014-0.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. In Advances in Neural Information Processing Systems (NewrIPS), pp. 1- 10, 2016. URL https: http///axivv../ ortps/..rggbbb///..0./ 02907.
Mao, Q., Liu, Z., Liu, C., Li, Z., and Sun, J. Advancing graph representation learning with large language mod- els: A comprehensive survey of techniques, 2024. URL hatp:://arxivv.rgg abs 2402 05952.
Massidda, R., Geiger, A., Icard, T., and Bacciu, D. Causal abstraction with soft interventions. In van der Schaar, M., Zhang, C., and Janzing, D. (eds.), Conference on Causal Learning and Reasoning, CLeaR 2023, 11-14
Ko, D., Lee, S., Kang, W., Roh, B., and Kim, H. Large lan- guage models are temporal and causal reasoners for video question answering. In Bouamor, H., Pino, J., and Bali, K.

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
April 2023, Amazon Development Center, Tübingen, Germany, April 11-14, 2023, volume 213 of Proceed- ings of Machine Learning Research, pp. 68-87. PMLR, 2023. URL https: httpt::::/ppccceednnge mmlr .tl.spprss// v213 3/ sa i dda html.
11-15, 2017, AUAI Press, 2017. URL http:://aaa/i org/vai http://aa22../pporceeeeee.//a//
pdf.
Tang, J., Yang, Y., Wei, W., Shi, L., Su, L., Cheng, S., Yin, D., and Huang, C. Graphgpt: Graph instruction tuning for large language models. In Yang, G. H., Wang, H., Han, S., Hauff, C., Zuccon, G., and Zhang, Y. (eds.), Proceedings of the 47th International ACM SIGIR Conference on Research and Development in In- formation Retrieval, SIGIR 2024, Washington DC, USA, July 14- 14-18, 2024, pp. 491-500. ACM, 2024. doi: 10. 1145/3626772.3657777 URL https: /tt///oo.ooooo org/ 10. 19.1145/3380
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in GPT. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), ), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022.
Mustapha, K. B. A survey of emerging applications of large language models for problems in mechan- ics, product design, and manufacturing. Adv. Eng- Informatics, 64:103066, 2025. doi: 10.1016/1.AEI 103066. URL http::////io..g//1.... j.aei. 2024. 13366.
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, x., Yang. Y., Chen, Q., Kim, W., Comeau, D. C., Isla- maj, R., Kapoor, A., Gao, X., and Lu, Z. Opportunities and challenges for chatgpt and large language models in biomedicine and health. Briefings in Bioinformat- ics, 25()):bb,d493, 01 2024. ISSN 1477-4054. doi: 10. 10.1093/bib/bbxd4493 URL pettps://dii.rgg// org//.0o......ooo 1093/bill /baal//bbb
Pearl, J. Causality: Models, Reasoning, and Inference. Cambridge University Press, second edition, 2009.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J.. Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Sub- ramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open founda- tion and fine-tuned chat models. CoRR, abs/2307. 09288, 2023. doi: 10.48550/ARXXV.22007... /10. 18/88555///aaa 2307 092888.
Pearl, J. Causality 1988-2001 introduction. In Geffner, H., Dechter, R., and Halpern, J. Y. (cds.), Probabilis- tic and Causal Inference: The Works of Judea Pearl, volume 36 of ACM Books, pp. 215-220. ACM, 2022. doi: 10. 10.1145/3001714.3507777 URL sttttss://ok: org/10.
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., and Goldberg, Y. Null it out: Guarding protected at- tributes by iterative nullspace projection. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, R. (eds.), Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, ACL 2020, On- line, July 5-10, 2020, pp. 7237-7256. Association for Computational Linguistics, 2020. doi: 10.18653/V1/ 2020.ACL- htts:: http://oii.oo /ttp:////ooo......./// org/10. 18653/v1 acl.00.ccll-o -ma.mmmmmmii..... 647.
URL https:
Ren, X., Wei, W., Xia, L., Su, L., Cheng, S., Wang, J., Yin, D., and Huang, C. Representation learning with large language models for recommendation. In Chua, T., Ngo, C., Kumar, R., Lauw, H. W., and Lee, R. K. (eds.), Proceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024, pp. 1464-3475, ACM, 2024. doi: 10. 0001144/33889344366655 36445558..4 URL https: //doi. org/ /tt//...11150
Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018 Vancouver. BC, Canada, April 30 May 3. 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https:/ http:::///pppeeeeeee wptp///ennweeeeee/e//p/ee net forum? id=rJXMpikC2.
3645458.
Rubenstein, P. K., Weichwald, S., Bongers, S., Mooij. J. M., Janzing, D., Groose-WWnnrrug M., and Schölkopf, B. Causal consistency of structural equation models. In Elidan, G., Kersting, K., and Ihler, A. (eds.), Proceedings of the Thirty- Third Conference on Uncertainty in Arti- ficial Intelligence, UAI 2017, Sydney, Australia, August
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. M. Investigating gender bias in language models using causal mediation analysis. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.). Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Informa-
12

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
tion Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Yang, Y., Huang, P., Cao, J., Ma, F., Zhang, J., and Li, J. Quantifying robustness adversarial word substitu- tions. In Joint European Conference on Machine Learn- ing and Knowledge Discovery in Databases, pp. 95 112. Springer, 2023.
Wang, F., Zhu, G., Yuan, C., and Huang, Y. LIm-enhanced cascaded multi-level learning temporal heterogeneous graphs. In Yang, G. H., Wang, H., Han, S., Hauff, C., Zuccon, G., and Zhang, Y. (eds.), Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Wash- ington DC, USA, July 14-18, 2024, pp- 512-521, ACM, 2024. doi: 10.1 00.11445/6626772.35573333 URL https: 10. 114 /tt.:///.....111//666667....3330
Yang. Y., Huang, P., Cao, J., Li, J., Lin, Y., and Ma, F. A prompt-based approach to adversarial example genera- tion and robustness enhancement. Frontiers of Computer Science, 18(4):184318; 2024b.
Ying, Z., Bourgeois, D., You, J., Zitnik, M., and Leskovec, J. Gnnexplainer: Generating explanations for graph neural networks. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché-Buu, F., Fox, E. B., and Gareett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9240- 9251, 2019. URL https://proceedeee
Wittmann, J. and Fey, M.
Wiki-cs: A wikipedia- based benchmark for graph neural networks. CoRR, abs/2007.02901. 2020. URL
Wu, Y., Wang, X., Zhang, A., He, X., and Chua, T. Discov- ering invariant rationales for graph neural networks. In The Tenth International Conference on Learning Repre- sentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. Btthtts::/ baattps:://peeeeetetee cpenreview. net/ fet//ooummiiddie jfrfiiw.............
neur http:ippsccppppee/2011////// #B007749877319901156644222223889 Abbstrract................ html.
Wu, Z., D' O'Oosterlickk, K., Geiger, A., Zur, A., and Potts, C. Causal proxy models for concept-based model explana- tions. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Con- ference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceed- ings of Machine Learning Research, pp. 37313-37334. PMLR, 2023a. URL https: https:::/tttceeettttttett http://pccceee....e//////o/o..sp mlr. press/v202 ww23b. htmll
Yu, J., Ren, Y., Gong, C., Tan, J., Li, X., and Zhang, X. Empower text-attributed graphs learning with large lan- guage models (Ims).. CoRR, abs/2310.09877/ 2023, doi: 10. 10:48550/AXXW.2230009822 URL https: //oi.. org/10. 48540/5550/00 (01550//aXivv. arxiv. 2310 09872.
Zhao, J., Fang. M., Zhang, K., and Pechenizkiy, M. Un- masking style sensitivity: A causal analysis of bias eval- uation instability in large language models. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2025, Vienna, Austria, July 27 August 1, 2025, pp. 16314-1638.... Association for Computational Linguistics, 2025. URL https: //aaaaaaaaacllgg/oo org/ 2025.. acl-long. 796/.
Wu, Z., Geiger, A., Icard, T., Potts, C., and Goodman, N. D. Interpretability at scale: Identifying causal mechanisms in alpaca. In Oh, A., Naumann, T. Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neu- ral Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin,
H. Tang, J., Wang, J., Yang, J., Tu, J., Zhang. J.. Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R. Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W. Deng, X., Zhou, X., Ren, X., Zhang, X., Wei,
X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang, Y., Wan,
Y. Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo. Z., and Fan, Z. Qwen2 technical report. CORR, abbs/2407.10677 2024a. doi: 10.48500/ARIIV2440.000// ht 48550/ //doi. wooo..org/110 11.11..4855555
13

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
A. Extended Related Works
A.1. Causal Abstraction within Neural Networks
Causal abstraction, originating from research in causal theory, involves refining and summarizing the causal relationships present in a low-level model into a high-level causal model (Iwasaki & Simon, 1994; Chalupka et al., 2017: Rubenstein et al., 2017). This process ensures that both models exhibit the same causal effects under soft or hard interventions (Massidda al., 2023). Achieving fully precise causal abstraction is challenging; therefore, the more commonly used and observed approach is the approximate causal abstraction. Recent research on causal abstraction have been widely applied within neural networks, which can be broadly classified into three key approaches: iterative nullspace projection, causal mediation analysis, and causal effect estimation (Beckers et al., 2019).
Iterative nullspace projection (Ravfogel et al., 2020; Elazar et al., 2021; Lovering & Pavlick, 2022) leverages linear transformations on a model's 's representation space to project certain attributes into the nullspace, effectively removing their influence. This technique has been employed identify the role of specific features in model decisions. For example, in (Ravfogel et al., 2020), the authors applied nullspace projection to remove gender-related information from word embeddings to assess its impact on downstream tasks. Other works (Elazar et al., 2021) extended this to different fairness-related contexts, assessing bias removal and its impact on neural network performance.
Causal mediation analysis (Vig et al., 2020; Meng et al., 2022) examines how an intermediary variable transmits the effect from one variable to another. Notably, (Vig et al., 2020) applied this framework to interpret attention mechanisms in Transformer models, showing how specific tokens influence model predictions through attention heads.
Causal effect estimation (Abraham al., 2022; Elazar al., 2022: Wu al., 2023b) focuses on quantifying the causal impact of changing specific input variables on the output. For example, (Abraham et al., 2022) explored how modifying specific neurons in a neural network can alter model predictions, allowing the identification of neurons with strong causal influence over particular tasks. Due to the complexity of large language models (LLMs), the relationships they model are easily influenced by various factors (Yang et al., 2023; 2024b), making their mechanisms more difficult to understand. Therefore, exploring their internal causal mechanisms represents a feasible approach to clarifying their underlying workings. (Wu et al., 2023b) demonstrated how interventions could uncover causal dependencies in integrated architectures like GNNs combined with LLMs, offering insights into their interaction and prciisinnmaking mechanisms. On the other hand, Interchange Intervention Training (IIT) (Geiger et al., 2022b) based methods achieve causal abstraction by aligning the causal effects between high-level and low-level models through the use of interchange interventions (Wu et al., 2023a; Huang et al., 2023a). These methods have demonstrated excellent results when applied to neural networks (Geiger et al., 2024).
These methods offer valuable tools for understanding the internal causal structure of complex deep learning models, providing interpretable insights into their operation. We build on this foundation to explore the causal mechanisms in GNNs integrated with I.MMs by applying these methodologies to elucidate the interactions between their components and uncover their causal dependencies.
A.2. LLM GNN Enhancers
With the emergence of LLMs (Brown et al., 2020; Devlin et al., 2019; Dubey et al., 2024), a new research direction has gained popularity in the field of graph representation learning: using LLMs for initial node feature processing, followed by GNNs for in-depth exploration and utilization of inter-node relationships. This combined approach is expected to address more complex graph data analysis problems effectively (Mao et al., 2024).
In this domain, related methods focus on optimizing the prompts used with LLMs and integrating the more flexible data processing capabilities that LLMs offer. This integration results in graph data that is more readily analyzable by subsequent GNN processes, allowing for better summarization and analysis of data features (Huang al., 2024a; Chen et al., 2023). Among these methods, most approaches focus the utilization of LLMs enhance the interpretation and alignment of graph data. For instance, TAPE (He et al., 2024) prompts LLM to perform zero-shot classification, requests textual explanations for its decision-making process, and designs an LLM-to-LMy interpreter to translate these explanations into informative features for downstream GNNs. Similarly, OFA (Liu et al., 2024) represents both graph data and tasks as nodes, aligning different textual contents into feature vectors of the same dimension. This alignment enables a consistent representation of data, which is then processed using GNN. By ensuring that all data, regardless of its initial format, can be uniformly processed, OFA improves the overall effectiveness of graph data analysis. CasMLN (Wang et al., 2024) also
14

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
follows this trend by introducing LLM-based external knowledge to effectively capture the implicit nature of graphs and node types, thereby enhancing type- and graph-level representations.
Some others utilize LLM to enhance class-level information. e.g., ENG (Yu et al., 2023) leverages LLMs to enhance class-level information and seamlessly introduces labeled nodes and edges without altering the raw dataset, facilitating node classification tasks in few-shot scenarios. Additionally, some works (Ren et al., 2024; Lyu et al., 2023) optimize node attribute information using LLMs in specific recommendation tasks, followed by subsequent graph learning.
All these methods follow the general framework of using LLMs for initial data processing, making the data more amenable to analysis, and then utilizing GNNs for deeper investigation into the relationships and structures within the data. Our research focuses on analyzing and expanding upon this overall framework, aiming to improve the integration and effectiveness of LLM and GNN combinations in graph representation learning.
B. Proofs
B.1. Proof of Theorem 3.2.
Theorem 3.2. Given a high-level causal model h(()) and a low-level neural network model f(.), both of which accurately map input graphs G G to outputs Y V, such that Y f(G)- h(G). Assume there exists subset Zt of the intermediate variables in f()) and bijective mapping nj Z + Zh, where 7h represents certain variables in h(-)- If there exists Z that minimizes the loss Lu, can conclude that the total effect TE TE (Y)) of J()) is equal to the total effect T2 (Y") of h(()) in all cases. Here, zd and 2 represent the values of Z and Z3, respectively, for the same input graph G. Similarty, z² and z represent their values for a different input graph G'.
To support the proof of Theorem 3.2, we first present the following lemma and provide the corresponding proof. Lemma B.1. Given the conditions within Theorem 3.2, if Lu reaches minimal, then for all Goris 9 and Gdiff E G, the following equation holds:
INTINV(J, (r G ) INTINV(h, Gonie,G Zn
(10)
Proof. According to Equation 1, Lu can be represented as:
Cu Σ Σ
DINTINV (h, G G G.dii 2z)), INNTIVV (f, Grnie G.dliff 2Z)).
(11)
GrEes Ga
as D-) denotes difference between INTINV (J; (J,Gong Gdilt 21) and INTINV (h, G Zh),. we have:
f Σ
Σ
D( DINNTINY (h, Gonig G ) INTINV (J, G G 22)
GrEG GHIEEg
Σ
Σ
D (INTINVV GOi Gdiff ( INTINV G Crfff 1²h) 2²)).
(12)
G G
Eomng Eomeg D [NTINN(() (h, Gorie Gdif (ZllI IN.IIIIVVVV (h, Godee Grlff ,rllffzzz)) is the minimal possible value for Lu. where Equation 10 holds. Next, demonstrate step by step that there exists a certain set of Z that allows Lu Eomeo Eounro D (INTINV(h, Gorig Gali , INTINV (h, G drifff ) Given the conditions within Theorem 3.2, there exists subset f of the intermediate variables of f()) that satisfied (z')) Zh As is bijective,
find z that satisfied (Zh) = zJ, furthermore, r7(2)) n11 (Za) = 2h. Then, by setting z - (Z*), we have:
INTINV (f. Gorig, dddiftt z²) = plateer ( ((Fre() \ , (Cori))) Un (h2" (Gditt)))
(13)
Where (.re(-) outputs all the intermediate variables of the hidden layers corresponding to Zf, and fz (Gurie) denotes the value of z given the input Gorig The function flatter.) represents the output of the model f(-) given the intermediate
15

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
variables output by fr"=(),, (-), and h h² (Gulft) denotes the value of 2h given Gdiff as input. Furthermore, given the conditions
within Theorem 3.2, have:
flatier (Gonle))))) = h (Gons))
(14)
therefore:
planer (Co \ ,2 (Coli) U 5² (GOTE) latter (n (Crote) \ 1122 (Gote) u (Gati))
(15)
Where (.) outputs the intermediate variables within h(-) up and including Zh, and ha denotes the subsequent computation of hi))) that produces the final output given the output of (.)) Furthermore, we have:
glatter (Ge \ lz (GunE) Un - (h²" (GuiE)) = h lateer heee (Goii) 112 (G uh (GaRH)).
(16)
Then, have:
INTINV (f, Gorig Gt z!) flatter (fre $2' (Gori)) Un 2h22
latee (nme (Gauie)) \ h²² (GmTi) Uh 2h (Adiitt
INTINV G Guilt zd)
(17)
Therefore,
conclude that the minimal possible Lomeg Eoseeg D [NTIIVV(] (h, Gorig Gallit 2")). INTINI (h, Gurig, Gulif z*) INTINV Gunig Gdill z3) INTINV Crorig Culill, 2²),, the lemma proved.
we
can
value of Lu which
is where
be reached,
Next, we demonstrate Theorem 3.2 based on Lemma B.1. According (Pearl, 2009), the total effect TE T (Y') be represented as:
TE ,- (Y DE x/ (Y + 1B21,20
= =E(Y' (x!"
(Pa(Y') (Pa'( z (x()) (x())
E (Pa(Y)) (="")) - -x())
(18)
where (Ez, and IE2,z:(Y)) denotes the natural direct effect and natural indirect effect respectively (Pearl, 2009).
Pa(-) denotes the ancestor variables. (=²) denotes the value of yJ given Z/ zf and Y (>(", , (Pa(Y/) 2 (=
denotes the value of y given z = xz while parents of y except for Z are set as given Z = =z/_
Then, according to the proof of Lemma B.1, we have:
E(Y*(2')
platter (pre(Grnnt) \ zz (Grii) Uz
(19)
G2
GREEg
and, we can also conclude that:
(z" (Pa(Y') (=)))) platter (fme \ nz (Guriz) U (z/ Pa(Y U (zf nPa(Ys))):
(20)
where Pa(Y'f) is certain feature variables within model f(-), Gorig represent the input sample here. Likewise, the following equation holds:
('', (Pa(Y 2z)((²)) platter (ore(( \ (G (z PP((Y')) nPa(Y!))).
(21)
16

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Based on equation 19, 20 and 21, we have:
- TE T (Y7) - E Y ( / -2))())))(((( (x)))))))
+ E ( (1 (z (Pa(X 22 (x/)) - E((Y((//)
E Slutter (G 1zd (GmT U (z/ \ Pa(Y))) (Pa(Y))))
platter (Goltt \ 1 (GmR))
g² GUEE
E platter (Goni)) gz (GaME) U (zf (Pa(Y (z) )pa(Y!)))
glaner
\ fzd (GmHE)
CUISE
E (INTINN( (C (GrtT)= 23 n Pa())))
planor GGOiE) \ 8z (Goni) U zz ) P))
g2 CUIEE
(F (Gronieae (Gaitty''' .2 \ ,Z' planee (foC \ /zt (GanE) U x4 )
(22)
GUME
where (Quniez²² and (Z" denotes the graph samples that satisfied fz ((Crotiez()) = xf and fzl ((Gurizzz)) =xx x5.
Based on the above derivation and Equation 17, we have:
TE |- (X = [NTINN(h, (Groieyz (Gddffiz=" x . (P(X")))
Mamer (G 112 (G u h 2h ((God))zz))
GuEEg
+ E IN (Godi))z (Guitys', \ )a((")))
Σ
h latter (GoniE) \ 2 (Gori)) U 3* ((Grreez())
g
GREES
((("(1??' ((CrriiEe" (Pa(Yr)) \ 2²)((²²' ((Cmme)z()
((Cori))())) -[(TE((2((CCOM)E())))
-E(
((Cmmiia" + +E(" ((" ( zz)(1²²)) ((Cmiiezz")) ))
((Gmieja',
- -((Y"(12* ((Cod)))()))
TE ((oono)st) j.a22 ((Cmmo))))) (rW).
(23)
where 1) (z can be represented as 2 ((Goni))z)) ), the theorem is proved.
B.2. Proof of Corollary 3.3.
Corollary 3.3. Given the condition that there exist subset 7. of the intermediate variables of f() satisfied 1y: 21 + 7hh where 1) is bijective does not hold, then if INTINV(f, Goris, Gdlfff Z2) = INTINV h, G Gillf, 2²h) holds, the conclusion given in Theorem 3.2 remains valid.
17

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
As Z'= (2*) no longer holds, we then locate set 2 that satisfied:
2' = nZ UU),
(24)
where U is the minimum set of extra variables that are required to determine the value of 21, We will next analyze the two possible scenarios, namely U IL Y and U y1 Y/. For case that U TT Y we could fix U as a constant set, and then have the following equation hold:
(25)
where (-) adopt set of constant to replace U. In such case, we could follow the same demonstration given in Theorem 3.2 and prove that:
TE (X*) = TE 122 ((Comezzl) , ((Cuu))e") (x*)).
(26)
For case that U ,1 Y', if INTINV (f, G Gailii 2 INTINV (h, Q G -zh) holds, we could follow Equation 22 and Equation 23 proof that Equation 26 holds. The corollary is proven.
B.3. Justificcaiin of Proposition 3.4
Proposition 3.4. For a high-level causal model h(-) and a low-level neural network f(),, both mapping input graphs GE G to outputs Y y such that Y [(G) (G)), 7' within f() and 7h within h(.)) minimize Lu its optimal value Cip then 2 aligns best with zh and has identical total effect on the output prediction as Zn. The minimal Ly is given by:
S = g
Σ
E
DIINTINV(H, G , INTINV G Gdllf Z²)).
(27)
Gg GWEE
The loss Cn defined in Equation can be used to measure the correspondence between the neural network model f()) and the higher-order causal model (t()).
Specifically, given a high-level causal model k(-) and a low-level neural network model f()), both of which accurately map input graphs G E 9 to outputs E V, such that Y f(G) I()),, the following holds:
1. For y within f() and Zn within h(-) that let Lu reach the minimal possible value Lit then Zy is the variable that best align with Zn and holds identical total effect for the output prediction with Zn. Li can be calculated with:
Ci = Σ Σ
D (INTINNV(h, (h, Gorig Gadlt 2d)), INTINV Gorig Gdill 2²)).
(28)
G G
2. For different variables Z and Z within f(-) and the corresponding loss Li and Ch. if Li > Ch we have total effect of f upon the output Y is strictly similar to 2h than z under D(). formally: formally:
TE v (Y - TE # (Y') TE 2h g (Y")).
(29)
From the proof of Theorem 3.2, we have that the minimal possible value of Lu is that:
Cu = Σ Σ D(NNINNV GooiieeGro ;dd INTINV (h, Croii, Golfff (dliftz)))
(30)
GrEgg G
under which Equation 10 holds. According to Corollary 3.3, if Equation 10 holds, and Y f(G) = h(G), we have the total effect TEz (Y') of f(-) is equal to the total effect TEg*,"" (Yh) of h(-). The conclusion justified.
B.4. Proof of Theorem 3.5
Theorem 3.5. Given high-level causal model h(.), variable nhh within h(-), if 2 let INTINV (f,, Gorig Co Z') INTINV (h, Gorig , )²,, and Gorig and Gori different in node level. variables within the GNN model are sufficient to align z1.
18

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
LLM Model
GNN Model
X1
21
2
7
2
2
2
X2
22
23
7
2
23
2
yr
XMI
2
2 11
21
ZM
2
IM
holds right-arrows point ZjA and ( (i)} holds right-arm arrows point to 2/. Some of the above-mentione arrows are omitted due to complexity. The red arrows represent relationships that may not necessarily exist.
Figure& 8. The graphical representation of the proposed SCM. Elements in set (Z)
Hg ش
To prove Theorem 3.5, need to conduct more detailed and accurate analysis of the variables within the model. Therefore, we introduce the concept of SCM (Structural Causal Model) (Pearl, 2009), which is framework that uses equations and directed graphs to represent and analyze causal relationships between variables.
Figure 8 shows the constructed SCM. In the figure, some variables have been omitted, and edges connecting or crossing the ellipses have also been removed from the demonstration, e.g., edge 2 + 2i is removed as it across the ellipses. In Figure 8, we use X, to denote the node attribute of i-th node, Z) represents the output variable of the 7-th hidden layer of the LLM according to the j-th node feature. / represents the node representation of the i-th node of layer in the GNN.
Within the SCM, for 2/. we have set Pa(2] - E A(() {i}), where (ii denotes the neighboring nodes of node Pa 2/)= - E N(i) U {+}} implies that Pa( Pa(Z)) consists of the node representations of the last layer that the corresponding node is adjacent or the same node i. We support the validity of the proposed SCM through the following lemma and its corresponding proof.
Lemma B.2. The SCM that is demonstrated within figure can represent the general causal relationships among variables in the LLM-enhancer-plus
model.
Proof. We demonstrate Lemma B.2 by showing that such SCM is consistent with the results generated using the IC algorithm (Pearl, 2009) for SCM construction. The IC algorithm is commonly used method for constructing SCMs, please refer to section 3.2 of (Pearl, 2009) for details. We follow the triple-step method of the IC algorithm for the proof.
Step 1. We connect X; with Z/ as there exist none set difff that satisfied X; TL | Furthermore, connect 2 with 2, due to the neural network architecture used by the LLM. We no longer connect any other variables in the LLM. Then, 27 is connected to 23. as such elements form the node representation of the GNN. Next, { E NN)) (i) and 2 is connected, as the node representation relies on the node representations of last layer that the corresponding node [V] adjacent or the same as node i. Yf and ( J k== are connected Y calculated through global pooling. For node J==1 correspondingly. And the others are not
classification, the predicted label shall connect with the elements within {26 [V]
connected for the same reason as the LLM part.
Step 2. Based on the feed-forward mechanism of neural networks, all edges should be oriented in the feed-forward direction.
Step 3. Since all edges are already directed, step three is unnecessary.
The constructed SCM consistent with that in Figure 8, thus the lemma proven.
With the proposed SCM within Figure 8 and Lemma B.2, we carry on demonstrating the proposed theorem. As the SCM
19

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
demonstrates, we have 23 block all causal routes from X3 to Y Therefore, we have:
INTINV G G .) INTINV (h, Gone, G 2)), (2)
(31)
IS 21,5 S ( s siss }. then the conclusion given the theorem naturally holds. Otherwise, for case that 38 E 2 E (Z) | و - [ 1 s } we have:
INTINV (1, G G )² INTINV (f, GOdiE , ( ,SUS)
INTINV (J, (J, (ConiiS ,Gdint S)
(32)
where S _ ( {2) si f.). (Goniess denotes the input that ensure
values changes to that of Gd as input. Based on equation 31, we have:
INTINV (f, (Gorees ,,ddiiff 8) = INTINV (f, (Croreys ,Crdfft ,.))
8' C () | V1, M
(33)
Therefore, we have the following holds for certain S:
INTINV (J, G
, Gullft 2r) = INTINV (J, G Gullff S)
(34)
INTINV (h, Godig Gddlt 2²",
(35)
8 2c (3) 155s 11,1 M
(36)
The theorem proved.
C. Details of The CCSG Dataset
C.1. Node Features
Below, we provide detailed introduction to the node features in CCSG. All data related CCSG, well as the methods used to construct the data, will be made open-source. We have also provided the corresponding data and code in the supplementary materials.
C.I.1. SELF-CONSTTRCCEE FEATURES
We independently constructed portion of the node features with the primary objective of thoroughly analyzing how graph representation learning models effectively handle graph data when the node features are relatively simple. This design aims ensure that the information within the dataset is predominantly reflected in the graph's topological structure, minimizing the interference of complex node features with model performance. By doing so, can better investigate the model's reliance on and ability to process structural information.
Specifically, we adopted two feature construction methods: First, we generated noise features based on random distributions. This approach introduces randomness to ensure feature neutrality while eliminating the influence of specific patterns, thereby enabling us to observe the model's s ability to capture topological information under random conditions. Second, we directly assign feature values to nodes based on their categories. such as assigning unique feature identifiers nodes of each class. This provides the model with the most basic classification information and allows test its performance under minimal feature conditions.
By employing these two feature construction methods, we ensure that the node feature design meets the analytical requirements while functioning as a controlled variable. This not only enhances the precision and interpretability of our experiments but also establishes a solid foundation for exploring the central role graph structural information in model learning.
20

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
C.1.2. NODE FEATURES SOURCE FROM WIKIPEDIA
We extracted a large number of entries from Wikipedia to construct semantically rich node features with known interrelations. As illustrated in Table the entries collected are categorized into three main classes: spaceflight, computer, and software. These domains are further subdivided into 15 specific subcategories, providing fine- geieggraiedd classification structure. Additionally, we recorded interlinking information among the entries in the dataset to facilitate the construction of training data with explicit causal relationships. Table 4 presents three representative examples of the data we collected, demonstrating the diversity and structure of the dataset. The contents of the "Background Categories," "Related Terms," and "Similar Entries" fields will be used identify similar nodes for constructing coherent graph structure. This approach ensures comprehensive and well-documented framework for downstream applications and analysis.
Table 3. Classes of the entries collected from Wikipedia.
Class
Count
Spacellight
1665
Spaceflight
401
Satellite
401
Rocket
260
Outer Space
302
Space Science
301
Computer
2205
Computer engincering
401
Automation
4U1
Computer security
601
Computer science
401
Computer hardware
401
Software
2005
Computer programming
4U1
Software testing
401
Application software
401
Software development
401
Software architecture
EO
Figure 9. Example Graph.
C.2. Node Correlations
With a diverse set rich, multi-category node features in place, the next step in constructing graph dataset is to build its topological structure. One approach to this to base the construction on the interrelationships between nodes, connecting them through associations and simple categorical relationships. For instance, this method is partially employed in the analytical experiments presented in the main text. In this work, leverage the interrelationships extracted from Wikipedia entries, combined with manually crafted node features, define these connections. This ensures that the causal relationships between nodes are explicitly established, resulting in a well-defined and interpretable graph topology.
C.3. Topological Structures
We constructed total of 10 distinct topological structures, with their types and specific configurations detailed in Table 5, In addition to commonly used graph structures in causal representation learning analyses (Wu et al., 2022; Gao et al., 2024), we incorporated a variety of graph topologies from graph theory, such as complete graphs and bipartite graphs. This was done to ensure comprehensive coverage of potential graph structures that could have causal effects on classification tasks. By diversifying the graph topologies, our approach aims to provide a robust framework for evaluating the influence of graph structure on causal representation learning and downstream analytical performance.
21

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Table 4. Samples of the collected data
Name
Content
Class
SubClass
Background Categories
Related Terms
Similar En-
tries
Computational social choice
Computational social choice is the stady of problems arise og- gregating the prefereaces of group agents using computer science and social choice theoey. focuses on efficiently computing voling out- the complexity manipola- tion, and representing preferences in combinstorial contexts.
Computer
Computer engineer- ing
Social choloe theory, Vor- ing theory. Computer science
Voting theory. Social choioe theory, Egal- itarian rule, Utilitarian rule, Agrocable subset, Anonymity, Arrow's impossibil- ily theorem, Bayesian regret, Budgei- proposal uggregation, Computational cial choice, Dictatorship mechanism, Dis- cursive dilemma. Electoral list. Elec- toral system, Extended sympathy. Frac- tional approval voting. Fractional social choice, Gibbard-Sattertheee Satterthwaise theorem. Gibbard's y theorem, Implicii utilitariun voting, Independennce ternatives, Intensity of preference, Lib- paradex, Christian List, May's the- orem, McKelvey Kkelvey-Scholielle Scholield chaos theo- graph, Median voting rule, Nakamura num- her, Neutrality. Optimal upportionment. Proportional-Fe fair rale, Quasirransiivee lation. Ranked voting. Rated voting. Sequential elimination method. Social Choice and Individual Values, Social wel- function, Unrestricted
Algocracy, Algorithmic
game theory.
Algorithmic
mechanism
design, Cake-
cutting. Fair division,
Hedonic
imelevant
rem, Mechanism design,
CEN/XFS
CEN/XPS (extensions for financial services) is client-server urchitee- ture for financial upplications on the Microsoft Windows platform, espe- cially for devices like EFTPOS tor- minals and tional standard promoted by the Fu- ropcan Committoe Standardiza- tien and based tensions for Financial Services de- veloped Microsoft, XFS allows financial institutions to choose I
Computer
Computer engineer- ing
Windows communi- cation and services, De- vice drivers, Embedded systems
Remoting, Administrative share, Bonjour Sleep Proxy. CEN/XFS, Chan- nel Definition Format, Discuvery and lunch, Distributed Component Object Model, Dynamic Data Exchange, Eter- Indexing Service, Internet nection Sharing. Internet Locator Server, Ipconfig. Layered Service Provider, Layer Topology Discovery. Link-Local Multicust Name Resolation, List prod that support SMB, LMHOSTS, Lo- cal Inter-Process Communication, Mi- cresoft Message Passing Interface, Mi- crosoft Message Queuing. Poison mes- sage, Microsoft Messenger service, Mi- crosoft Transaction Server
Xpeak, Auto- mated teller machine. Taller assist unit
WOSA Ex-
iDempiere
iDempiere is open ERP soflware fully navigable on various devices. includes CRM and SCM lunctions, and is community ered, unlike proprictary salu- tions.
Software
Software Develop- ment
Free ERP software,
Adaxa Suite. Adempiere, Apache OHBiz, Compiere, Dolibar,, ERPS, ERPNext, Iee liumV, V. IDempiere, InoERR IntarS, JFre, Kuali Foundation, LedgerSMB. Melas- Odoo,
OSGL, Java. Compiere. List of ERP software packages
Free business soltware
Tryton
Kallithea (software)
Kallithea is free and cross- platform souree code management system that focuses providing repository hosting services for col- laboration. offers features lorking, pull requests, code view,
Software
Software Develop- ment
Open source hosted devel- opment tools. Project man- agement soft- ware. Free programmed
Cloud9 IDE, Git Gitorious, Kallithea (software), Travis
Comparison project management software. List of tools for code review, parison of source code bosting facil- itics, Apache
tracking. was created as fork of Rhode- Code duc to changes the license terms. While earlier versions of RhodeCode were licensed under the GNU GPL version
Python, Free project management
sottware
troduced exceptions
Lunar
Re-
The Lunar Recemnassance Orbiter (LRO) NASA spacecraft cur- rently orbiting the Moon, collect- ing essential daia for future human and robotic missions. Launched 2009 as part of the Lunar Procur- sor Robotic Program, LRO ated detaiked 3-D mup of the lunar surface 100 meter resolution, cluding high- nigh-resolution images of Apollo landing sites
Spacellight
Space Sci-
Lunar Re- connaissance Orbiter, Misssions to the Moon, NASA space probes, Space probes launched in Satellites orbiting the Moon
Lunar Reconmaissance Orbiter. Diviner,
Exploration ICROSS.
connaissance Orhiter #Pay-
load
List of mis- sions the Moon, Lunar Atmosphere and Dust Explorer, Lanar out- (NASA),
2009,
Lopar water
22

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Table 5. Details concerning the utilized topological structures.
Structure Type
Grid
Circle
Chain
Tree
Star
Illustration
Nodes connected in a grid pattem. and number of nodes is adjustable.
Nodes form closed loop, and the number modes is adjustable.
Nodes are connected in a chaio without brunches, useful for over squashing. Bipartite Graph
Multiple nodes connect to central node, and number of nodes is adjustable.
Nodes form a tree, branching and height are adjustable,
Description
Structure Type
Complete Graph
Chordal Cycle
Bipartite Graph
Fixed Structure
Illustration
Every pair of modes is connected, and number of nodes is adjustable.
Graph with chords in the cycle, with adjustable nodes and chords.
Vertices divided into two disjoint sets, adjustable nodes.
Outer nodes connected in a ring with central node, adjustable nodes.
random graph structure with fixed topology.
Description
Table Accuracy corresponding different q values, using different numbers of prompts. Llama3 was used the enhancer. The standard deviation of results over 10 trials.
Datasct
q = 2
q= 3
q=
Cora
84.94 83.84.0.77
86.45
85.79 83.98.1..99
81:31:023
Puhmed
84.32 1.06
12.58.10.44
C.4. Node-level Analytical Experiments Dataset Details
For node-level classification datasets, as described in Section 3, we fix the nodes to be classified and ensure that their associated causal relationships are controllable and analyzable. A specific example is provided in Figure 9. We further utilize the subcategories and interrelationship information included in CCSG to construct edges between the related nodes, At the same time, we partition the nodes into sets based on the shortest path lengths from each node to the target node and the associations between nodes. The attributes of each node set are processed by modde. The outputs of p,nde..1 serve the inputs to (-). The function determines the output based on the categories of the node itself and its neighboring nodes, These outputs will serve as the input for /:mode.s(.), (-), which produces the final classification result for the node. 1,mode.3() (-) takes as input the outputs of (.)) corresponding to the multi-order neighbors of the target node v and generates the final result.
Specifically corresponding to the experimental results in Figures and 2, v 22, and V represent the categories of nodes distances of 1, 2, and 3 from node v. respectively. These categories are determined by the predominant category of these nodes. Subsequently, Φ and Φ are features constructed based the categories of nodes and the categories of their neighboring nodes. The difference lies in that Φ considers multi-order neighboring nodes.
For node-level analytical experiments, we construct up to 3,000 graph samples, each containing a single target node for classification, 3 classes. On average, each graph consists of 28.9 nodes. We select 500 samples for testing and all samples for training. The test set is utilized for interchange intervention based analysis. To reduce computational complexity, we also exclude sample pairs that remain completely unchanged before and after the interchange intervention based on h(.).
C.5. Graph-level Analytical Experiments Dataset Details
For graph-level classification datasets, we utilize the topological structures and node features together. Specifically, we use certain substructures and their connections within the topological structures as the output of herahh, (.). Subsequently, the output of (.) represents the category of the entire topological structure, while h hrrpph,3. (+) produces the final result based on the category of the topological structure and its connection patterns.
In the results shown in Figures 1 and 2, T1, P2, and I3 are defined as substructures of the topological structures, differing in the number of nodes they contain: Ij consists of structures with nodes, I'2 consists of structures with 6 nodes, and I's consists of structures with 9 nodes. 22 and Q2 specifically refer to the types of graph structures commonly used in causal representation learning analyses, as well as the graph topologies derived from graph theory. Both types of structures are present in each sample.
23

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
200
GNN
2
W2
Laayn
1.5
6 3
0.90
0.632
1365
0.90
0.893
0003
0014
0.224
1.5
GNN with
4 Laayn
2 4/
090
= 9
0.111
125
00901
0.99
01244
0051
0.911
DAKO
GNN with Lovers
9.21
0.294
0484
0057
0.13
D002
1.00
2 -
0.34
0.13
0.42
D044
0.00
6.44
0214
0.75
GNN
M--
0239
0.122
0.544
6A47
013:
0403
0121
606.
1804
0.21
1235
0.15
0409
1:42
0.50
2 +;
0063
GNN with 10 Layes
2 4-
0,42
0.35
0.500
0030
0:105
0425
0001
0.25
0166
0.39
0.94
0.25
4
0.40
0.25
6196
6431
0.344
000
0120
0014
0100
0.39
0.00
Layer Index
(a) Node-Ievel experiment results.
2,00
2
GNN with
2Tavr
2" T2
0.5%
1.75
13
1158
1776
2 T:
1.50
GNN wim Laayers
2 T22
2 Ta
2.00
1.133
1:279
1.25
22 T;
0111
600:
0015
8813
8314
0003
0M2?
8671
0.295
1.00
2
2
8277
6677
0016
0019
6.001
0015
0022
8631
6:03
0.75
GNN wim
Layers
2 Ta
1.275
6712
0.36
01441
0.155
0.18
0:333
2 Ta
0.50
2" T
BB82
0014
0215
0013
8214
0022
0.1%
GNN with
10 Lagun
2" T2
8.03
0.3M
0.134
0351
R007
0073
2374
8:38
0229
0.947
2n 133
1:440
1506
1.011
1.76
1350
1440
Layer
(b) Graph-level experiment results.
Figure Aligament results graph-level node-level experiments varying GNN layers.
For graph-level analytical experiments, we construct up to 1500 graph samples, use total of 10 topological structures, each of which consists of 20.5 node. We select 300 samples for testing and all samples for training. We utilize these samples in Node-level Analytical Experiments.
D. Experimental Details
D.1. Hyperparameters
We conducted all our experiments computer equipped with single A100 GPU, running the Ubuntu 20.04 operating system. The learning rate for the model was set to 0,001, and the total number of training epochs was 200. For the LLM components, we utilized pre-trained parameters from publicly available open-source models, keeping them fixed during our experiments. Unless explicitly stated otherwise. Llamaž and a GCN with layers and 256 hidden dimensions (if not specifically declared) were employed as the backbone networks in all analytical experiments. For experiments related to the module, the hyperparameter m was set to 2, 8 is set 10, and was set 10.
24

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
D.2. Implementation Details of the AT module
The AT module utilizes Transformer with a 4-layer network and hidden dimension of 512. It employs an LLM automatically generate prompts, with the content of the prompt as follows:
Prompt Generation
In order to summarize the content of <data information>, please provide <C> different prompts. The prompts should be in the format [Prompt Content] + [Content to be Summarized]. Only output the [Prompt Content].
<ata information> refers to the details regarding the dataset, whereas <q> noting that the AT module also supports the use of manually designed prompts, which can help in selecting more effective prompts.
represents the hyperparameter q. is worth
E. Extra Experiments.
In Figures 10 and 11, we provide a more comprehensive presentation of the experimental results, including the alignment outcomes from several additional experiments. is evident from all the experimental results that the phenomenon described in Empirical Finding 2 consistently emerges-indicalee logic structures that remain unaffected by these variations. Furthermore, the results illustrated in Figure 12 also reveal the correlation between the optimal Lu and the model's s accuracy. Figure 13 presents further experiments conducted with different datasets. Finally, Table summarizes the hyperparameter experiment for q.
that, the size of the model changes, retains certain specific
25

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
2,00
2.00
2
T
0461
0235
8.181
GNN with
1 F1
0.12
5000
0.094
4.01
0,211
Dimension
Hiddea
2 2
1:59
0367
0:56
1,75
Dimension
Hidden
2" 12-
1.29
0309
0.55
0137
0,228
16
2
1424
1570
2 Ta
L614
2310
20
GNN
T1
Hidden
Hidden
Dimensson
20
125
Dimension
122
0.353
1.25
64
2 Ta
of
Fa
1:324
1.40
L101
1.00
1,00
GNN with Dimension of 256
20a 11
0:077
8.00
0.077
GNN with
2 F1
0015
0.005
0.007
0.000
2a 22
0.204
0.091
6.129
0.75
Dimension of 256
Hidden
2 T2-
0.184
D007
B082
0.44
00014
1175
2 Ta
1.188
3.999
2 Ta
1816
1.12
1411
1.100
0.50
GNN with
8813
8.000
C111
GNN with
2 T1
50005
6.005
0.077
00015
Dinsension
Hilden
2
0163
Dimension
Hildden
2 T2
RU44
0.102
0.008
2 Ta
0.00
(a) Results of GNN with 2 layers.
(h) Results of GNN with 4 layers.
GNN with Hidden Dimension of 16
t
2
0671
0.530
0.211
0.34
ILWV9
0.115
0.295
1.75
2
1,50
GNN with Dimension th 2
2 r
0.28
0012
0.005
0.005
0
0.009
0015
1.25
2
.800
1.70
1.600
1.505
1.444
1.004
1957
1,00
GNN with Hidden of 256
2
0
0.33
2
0.344
0.115
0,301
0,100
0
0.058
0.182
0.75
2
2009
1.738
1.227
1.704
0.50
GNN with Hidden Dimension of 2048
2h T
0.08
0.006
0,004
0.802
0,005
0.011
0.15
2 T2
0.664
0.151
0.229
1,12
1.995
1.236
1.467
1.502
2.171
0,00
(c) Results of GNN with layers.
2.00
GNN
2d ri
0277
0.077
0015
0.019
0.088
0,015
0.002
0.34
0.033
Hidden Dimension of 16
2¹
1.75
4¹ 1's
2.07
3,575
2200
1,00
1924
1.951
1.145
1340
2.125
GNN with Hidden Dimension
2 1
0.086
8.19
0.25
0.133
0.025
0.34
0.037
2¹ T2
0.522
8.895
0.159
2r l8
1.492
2,044
1.403
1.955
2.143
1.378
1,475
1.912
2.72
GNN with Hidden
Dimension
0485
3.073
0.139
0.149
01067
0,990
0.076
0.197
0.503
0.75
256
2
2.044
0.50
GNN with Hidden Dimension of
2
0.077
0.443
8,022
0.015
0.021
0.066
0.039
0,208
0.125
0.187
0.137
00085
0.116
0.201
0.219
0.306
0.25
2
1.665
0.00
2
(d) Results of GNN with 8 layers.
Figure Alignment results of graph-level experiments with varying GNN layers and hidden dimensions.
26

---

LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
I3
Optinal Gi
0.9
Optimal Cg
00.3
ACC
0.8
0.7
16
256
20488
255
2045
Hidden Dimension
Dimension
Hidden Dimension
Dimension
Graph-level task align- (b) Graph-level task model Graph-level task align- (d) Graph-level task model ment results with number accuracy with number of ment results with number accuracy with number of of GNN layers set to and varying hidden dimensions.
GNN layers set to and vary- of GNN layers set to and
GNN layers set to 4 and vary- ing hidden dimensions.
ing hidden dimensions.
varying hidden dimensions.
T3
Optimal Cn
0.8
Optimal Eu
09
0.7
0.
0.8
0.5
1.5
64
2.56
2048
16
64
255
2045
2.66
2048
Hiddea Dimeasion
Hidden Dimension
Hidden Dimension
Hidhen Dimemiso
(c) Graph-level task align- (f) Graph-level task model ac- (g) Graph-level task align- (h) Graph-level task model
ment results with the number curacy with the number of ment results with the number accuracy with the number of of GNN layers varying hidden dimensions. ing hidden dimensions.
and GNN layers
and vary- of GNN layers
and GNN layers
and vary-
varying hidden dimensions. ing hidden dimensions.
Figure 12. Experimental results under different number of GNN layers and hidden dimensions.
0.123
0.223
0.145
0.134
0.136
2.
V2
2 4
1.002
1.202
1494
1:625
8/
0.612
2th
0.283
0.216
Layer
Tayer
T
Layer
Layer
Layer
Layer
Layer
Layer
Layer 4
The results of node-level experiments, where 2 corresponds
The results of node-level experiments, where 2h cor- responds to output variables & and 42 of 1 mode.2 (.). using random data as node features, are presented. Bold values indicate the minimum Ln.
output variables V,, V2 and W, of 1 pode. 1 (-), using OGBN-Arxiv data as node features, are presented. Bold values indicate the minimum Lu.
Figure 13, Extra experiments using OGBN-Arxiv and random data.
27


Reviews:
[{"review_id": "M1wd04UzVv", "reviewer": "Reviewer_2HXM", "summary": "This paper proposes a new analysis tool for LLM encoders for GNNs, based on the causal theory. The proposed method is evaluated in one synthetic dataset generated by the authors.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your analysis and suggestions for revising our paper. We greatly appreciate your feedback. Below, we will address each of the raised concerns.\n\n**Methods And Evaluation Criteria:**\n\nRegarding the AT module, we conducted experimental analysis using several public datasets. To demonstrate the generalizability of the proposed analytical method, we first note that it can be applied to any graph dataset capable of representing higher-order causal models. Currently, validating this requires synthetic graph datasets with controllable causal relationships, where node features are rich in semantic information to effectively showcase the power of large language models. However, due to the limited availability of such datasets, we constructed our own CCSG dataset. To further establish the robustness of our approach, we expanded our evaluation by incorporating node data from the ogbn-arxiv dataset and re-executing the node-level experiments. Additionally, we conducted experiments using random data. The results are as follows:\n\n**Table 1: The results of node-level experiments, where $ Z^{h} $ corresponds to the output variables $ \\Psi_1 $,$ \\Psi_2 $, and$ \\Psi_3 $ of $h^{\\text{node,1}}(\\cdot) $, using OGBN-Arxiv data and random data as node features, are presented. These results are consistent with those in the paper. Bold values indicate the minimum $ \\mathcal{L}_{\\text{II}} $.**\n\n|   | Layer 0             | Layer 1             | Layer 2             | Layer 3             | Layer 4             |\n|--------|---------------------|---------------------|---------------------|---------------------|---------------------|\n|$\\Psi_1$     | **0.123**      | 0.223      | 0.143  | 0.134      | 0.136      |\n|$\\Psi_2$    | **0.366**      | 1.337      | 0.956     | 0.998      | 1.332      |\n|$\\Psi_3$     | **0.612**      | 2.747      | 2.131      | 2.324      | 2.653      |\n\n**Table 2: The results of node-level experiments, where $ Z^{h} $ corresponds to the output variables $ \\Phi_1 $ and $ \\Phi_2 $ of $h^{\\text{node,2}}(\\cdot) $, using random data as node features, are presented. These results are consistent with those in the paper. Bold values indicate the minimum $ \\mathcal{L}_{\\text{II}} $.**\n\n|   | Layer 0             | Layer 1             | Layer 2             | Layer 3             | Layer 4             |\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|$\\Psi_1$     | 1.002    | **1.202** | 1.494    | 1.625    | 2.139     |\n|$\\Psi_1$     | 0.203    | **0.211**| 0.214    | 0.216    | 0.393    |\n\nWe will add these results to the paper as well.", "claims_and_evidence": "N.A.", "methods_and_evaluation_criteria": "No. The proposed method is only evaluated in one synthetic dataset, which I think is not enough to show its generality and effectiveness.", "theoretical_claims": "No. I do not have background in related theories.", "experimental_designs_or_analyses": "Refer to \"Methods And Evaluation Criteria\"", "relation_to_broader_scientific_literature": "The analysis of the interpretability of LLMs encoder for GNNs is beneficial for this research area.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "N.A.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hVyTpMcVqM", "reviewer": "Reviewer_iycs", "summary": "This paper presents a valuable analysis of the LLM-enhancer-plus-GNN framework, exploring its underlying mechanisms and identifying potential areas for improvement. The use of the CCSG dataset and the interchange intervention method provides a novel approach to understanding the relationship between LLMs and GNNs. The proposed AT module offers a practical solution to enhance the information transfer between these components, leading to improved performance. The paper is well-written and the results are presented clearly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your analysis and suggestions for revising our paper, and your support for our paper. We greatly appreciate your feedback. Below, we will address each of the raised concerns.\n\n**Methods And Evaluation Criteria:** \n1. Thank you for your suggestions. We have incorporated data from the ogbn-arxiv dataset and conducted additional experiments. The results are as follows:\n\n**Table 1: The results of node-level experiments, where $ Z^{h} $ corresponds to the output variables $ \\Psi_1 $,$ \\Psi_2 $, and$ \\Psi_3 $ of $h^{\\text{node,1}}(\\cdot) $, using OGBN-Arxiv data and random data as node features, are presented. These results are consistent with those in the paper. Bold values indicate the minimum $ \\mathcal{L}_{\\text{II}} $.**\n\n|   | Layer 0             | Layer 1             | Layer 2             | Layer 3             | Layer 4             |\n|--------|---------------------|---------------------|---------------------|---------------------|---------------------|\n|$\\Psi_1$     | **0.123**      | 0.223      | 0.143  | 0.134      | 0.136      |\n|$\\Psi_2$    | **0.366**      | 1.337      | 0.956     | 0.998      | 1.332      |\n|$\\Psi_3$     | **0.612**      | 2.747      | 2.131      | 2.324      | 2.653      |\n\nWe also conducted experiments using random data, and the results are as follows:\n\n**Table 2: The results of node-level experiments, where $ Z^{h} $ corresponds to the output variables $ \\Phi_1 $ and $ \\Phi_2 $ of $h^{\\text{node,2}}(\\cdot) $, using random data as node features, are presented. These results are consistent with those in the paper. Bold values indicate the minimum $ \\mathcal{L}_{\\text{II}} $.**\n\n|   | Layer 0             | Layer 1             | Layer 2             | Layer 3             | Layer 4             |\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|$\\Psi_1$     | 1.002    | **1.202** | 1.494    | 1.625    | 2.139     |\n|$\\Psi_1$     | 0.203    | **0.211**| 0.214    | 0.216    | 0.393    |\n\nWe will include these results in our paper.\n\n2. We compared our results with TAPE, and the outcomes are as follows:\n\n**Table 1: The results of node-level experiments, where $ Z^{h} $ corresponds to the output variables $ \\Psi_1 $,$ \\Psi_2 $, and$ \\Psi_3 $ of $h^{\\text{node,1}}(\\cdot) $, using TAPE as baseline, are presented. The results show that, due to its complexity, the TAPE method aligns with later layers of the neural network compared to those reported in the paper. We will include the relevant analysis results in our paper. Bold values indicate the minimum $ \\mathcal{L}_{\\text{II}} $.**\n\n|               | Layer0           | Layer1           | Layer2           | Layer3           | Layer4           |\n|---------------|------------------|------------------|------------------|------------------|------------------|\n| $\\Psi_1$     | 1.620 | **1.454**  | 1.468  | 1.544  | 2.911  |\n| $\\Psi_2$     | **1.276**  | 1.387 | 1.389 | 1.312  | 1.511  |\n| $\\Psi_3$     | **1.380**  | 1.614  | 1.621  | 1.399 | 1.946  |\n\n**Experimental Designs Or Analyses:** \n\nWe have included the experimental results for this hyperparameter and will incorporate these results into our paper: \n\n**Table 1: Accuracy corresponding to different q values, using different numbers of prompts. Llama3 was used as the enhancer. The standard deviation of results over 10 trials.**\n\n| Dataset | q=1            | q=2            | q=3            | q=4            |\n|---------|----------------|----------------|----------------|----------------|\n| Cora    | 84.94±0.81     | **86.45±0.49**     | 85.79±0.51     | 83.33±0.23     |\n| Pubmed  | 83.84±0.77     | **84.32±1.06**     | 83.98±0.39     | 82.58±0.44     |\n\nWe will also provide a detailed analysis of the hyperparameters. Thank you for your suggestion.", "claims_and_evidence": "The claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Overall, the methods and/or evaluation criteria proposed in this paper are reasonable. Here are two additional suggestions.\n\nSuggestions:\n1. Extend the Analysis to Other Benchmark: Evaluate the performance of the method on more datasets, such as ognb-arxiv, etc.\n2. Compare with Other Methods: Conduct a more comprehensive comparison with other LLM-enhancer-plus-GNN methods.", "theoretical_claims": "The theoretical Claims proposed by the authors of this paper are relatively reasonable.\nIn the node-level and graph-level analysis, the authors used the CCSG dataset and causal modeling methods to analyze the llm-augmentor+gnn framework, revealing that for a fixed-parameter LLM enhancer, the features output by the LLM have the function of representing information at both the node level and the raw data level.", "experimental_designs_or_analyses": "Lack the Analysis of hyperparameter q :In Chapter 4, the impact of setting the hyperparameter q on the results lacks sufficient discussion.", "relation_to_broader_scientific_literature": "The authors evaluated the impact of modifying the features transmitted from the LLM enhancer to the GNN on model performance, and discovered that token position selection in the LLM enhancer output has a significant effect on model performance. This finding is novel and can be used to improve the performance of the LLM enhancer + GNN framework.", "essential_references_not_discussed": "In my understanding, there are none.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xk2Of4tAiU", "reviewer": "Reviewer_XFjz", "summary": "This paper explores the use of large language models (LLMs) as feature enhancers for graph neural networks (GNNs) in graph representation learning, addressing the fundamental properties of this approach using the interchange intervention method from causality theory. To facilitate analysis, the authors construct a synthetic graph dataset with controllable causal relationships, enabling precise manipulation of semantic structures. Through systematic interchange interventions, they investigate the correspondence between the LLM-enhancer-plus-GNN model and a high-order causal model, uncovering the internal logical structure of the black-box neural network. Based on these insights, they propose a plug-and-play optimization module to improve information transfer between the LLM enhancer and the GNN. Experimental validation across multiple datasets and models demonstrates the effectiveness of this module, offering a deeper understanding and improved performance of LLM-enhanced GNNs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your analysis and suggestions for revising our paper,  and your support for our paper. We greatly appreciate your feedback. Below, we will address each of the raised concerns.\n\n**Methods And Evaluation Criteria**: \nIn the appendix, we provide a detailed description of the composition of our dataset. In fact, the semantic information in the Controlled Causal-Semantic Graph dataset is primarily reflected in the semantic richness of its node features. The relationships and labels in the graph are determined based on the entry categories and associations provided by Wikipedia, following a deterministic approach to ensure causality.\n\n**Experimental Designs Or Analyses**：\nOur proposed method can be applied to any graph dataset capable of abstracting higher-order causal models, a point we will clearly mention in the paper. Achieving this currently requires synthetic graph datasets with controllable causal relationships, where node features must contain rich semantic information to demonstrate the effectiveness of LLMs. Due to the limited availability of such datasets, we built our own CCSG dataset. To validate the generalizability of our method, we incorporated node data from the ogbn-arxiv dataset and reconducted the node-level experiments, we also conduct the experiments using random data, with the results as follows:\n\n**Table 1: The results of node-level experiments, where $ Z^{h} $ corresponds to the output variables $ \\Psi_1 $,$ \\Psi_2 $, and$ \\Psi_3 $ of $h^{\\text{node,1}}(\\cdot) $, using OGBN-Arxiv data and random data as node features, are presented. These results are consistent with those in the paper. Bold values indicate the minimum $ \\mathcal{L}_{\\text{II}} $.**\n\n|   | Layer 0             | Layer 1             | Layer 2             | Layer 3             | Layer 4             |\n|--------|---------------------|---------------------|---------------------|---------------------|---------------------|\n|$\\Psi_1$     | **0.123**      | 0.223      | 0.143  | 0.134      | 0.136      |\n|$\\Psi_2$    | **0.366**      | 1.337      | 0.956     | 0.998      | 1.332      |\n|$\\Psi_3$     | **0.612**      | 2.747      | 2.131      | 2.324      | 2.653      |\n\n**Table 2: The results of node-level experiments, where $ Z^{h} $ corresponds to the output variables $ \\Phi_1 $ and $ \\Phi_2 $ of $h^{\\text{node,2}}(\\cdot) $, using random data as node features, are presented. These results are consistent with those in the paper. Bold values indicate the minimum $ \\mathcal{L}_{\\text{II}} $.**\n\n|   | Layer 0             | Layer 1             | Layer 2             | Layer 3             | Layer 4             |\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|$\\Psi_1$     | 1.002    | **1.202** | 1.494    | 1.625    | 2.139     |\n|$\\Psi_1$     | 0.203    | **0.211**| 0.214    | 0.216    | 0.393    |\n\nWe will add these results to the paper as well.", "claims_and_evidence": "The paper provides clear and structured evidence for its claims through a combination of theoretical analysis, synthetic dataset experiments, and empirical validation across multiple models and datasets. The use of a synthetic graph dataset with controllable causal relationships strengthens the credibility of the analysis by allowing precise manipulation of semantic structures and causal dependencies.", "methods_and_evaluation_criteria": "The entire analysis is conducted on a specific dataset, namely the Controlled Causal-Semantic Graph. My concern is the difference between a Causal-Semantic Graph and a causal graph. Based on my understanding of this dataset, the causality is still represented in a semantic way rather than reflecting the actual mechanism.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "The proposed analysis framework should be more generalizable, and the Controlled Causal-Semantic Graph dataset should be one of its implementations. Or, in my opinion, the Controlled Causal-Semantic Graph dataset should be used to evaluate the proposed analysis framework.", "relation_to_broader_scientific_literature": "The key contributions of this paper build upon and extend prior research in understanding how LLM enhancers contribute to GNNs through a causal lens.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper presents a novel perspective by applying causal analysis through interchange interventions to investigate the role of LLM-enhanced GNNs. The originality lies in its effort to bridge causality and graph representation learning, particularly in quantifying how LLM-derived features align with causal structures in GNNs. The introduction of the Controlled Causal-Semantic Graph (CCSG) dataset is another notable contribution, as it enables a structured evaluation of causal relationships in graph-based learning.\n\nHowever, one limitation is the generality of the findings, as the analysis is heavily dependent on the CCSG dataset. While synthetic datasets allow for precise control over causal structures, it remains unclear whether the insights generalize to structured causal DAGs rather than a semantic causal graph.", "other_comments_or_suggestions": "Line 251: LM-enhancer-plus-GNN -> LLM-enhancer-plus-GNN", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Post-rebuttal, most reviewers (iycs, XFjz) have reached a consensus in favor of acceptance, with their main concerns (e.g., the need for broader experimental validation with additional baselines and datasets) addressed. After carefully reviewing the submission and the rebuttal, I agree that the paper presents a contribution to understanding the causality-inspired interplay between LLMs and GNNs. I recommend acceptance. Please ensure that the suggested revisions are incorporated into the final version.
Decision: Accept (poster)