UniGTE: Unified Graph–Text Encoding for Zero-Shot Generalization across Graph Tasks and Domains
Authors: Duo Wang
Yuan Zuo
Guangyue Lu
Junjie Wu
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Graph Neural Networks
Zero Shot Learning
Primary area: deep_learning

Abstract: Generalizing to unseen graph tasks without task-specific supervision is challenging: conventional graph neural networks are typically tied to a fixed label space, while large language models (LLMs) struggle to capture graph structure. We introduce UniGTE, an instruction-tuned encoder–decoder framework that unifies structural and semantic reasoning. The encoder augments a pretrained autoregressive LLM with learnable alignment tokens and a structure-aware graph–text attention mechanism, enabling it to attend jointly to a tokenized graph and a natural-language task prompt while remaining permutation-invariant to node order. This yields compact, task-aware graph representations. Conditioned solely on these representations, a frozen LLM decoder predicts and reconstructs: it outputs the task answer and simultaneously paraphrases the input graph in natural language. The reconstruction objective regularizes the encoder to preserve structural cues. UniGTE is instruction-tuned on five datasets spanning node-, edge-, and graph-level tasks across diverse domains, yet requires no fine-tuning at inference. It achieves new state-of-the-art zero-shot results on node classification, link prediction, graph classification and graph regression under cross-task and cross-domain settings, demonstrating that tight integration of graph structure with LLM semantics enables robust, transferable graph reasoning.

UniGTE: Unified Graph-Text Encoding for Zero-Shot Generalization across Graph Tasks and Domains
Duo Wang
Yuan Zuo" Guangyue Lu Junjie Wu
MIIT Key Laboratory of Data Intelligence and Management, Beihang University wwangduo58, zuoyuan, lugybuaa, wujj euuaa.edu.cn
Abstract
Generalizing unseen graph tasks without task-specific supervision is challenging: conventional graph neural networks are typically tied to a fixed label space, while large language models (LLMs) struggle capture graph structure. We introduce UniGTE, an instruction-tuned encoder-decoder framework that unifies structural and semantic reasoning. The encoder augments a pretrained autoregressive LLM with learnable aligument tokens and a structure-aware graph-text attention mecha- nism, enabling attend jointly a tokenized graph and a natural language task prompt while remaining intiuutttionnnnvaiiin to node order. This yields compact, lask-aware graph representations. Conditioned solely on these representations, frozen LLM decoder predicts and reconstructs: it outputs the task answer and simultaneously paraphrases the input graph natural language. The reconstruction objective regularizes the encoder to preserve structural cues. UniGTE is instruction- tuned on five datasets spanning node-, edge- and graph-level tasks across diverse domains, yet requires fine-tuning at inference. It achieves new state-of-the-ard zero-shot results on node classification, link prediction, graph classification and graph regression under cross-task and cross-dae dross-domain settings, demonstrating that tight integration of graph structure with LLM semantics enables robust, transferable graph reasoning.
1 Introduction
Zero-shot learning in graph machine learning seeks generalize to unseen tasks and domains without task-specific supervision. Although graph neural networks (GNNs) excel in fully supervised settings, they transfer poorly to new label spaces or data distributions without costly fine-tuning M Inspired by recent progress in natural language processing (NLP), prompt-based extensions have been proposed to enhance GNN generalization 3. However, the rigid architecture of conventional GNNs-especiall their task-specific output heads-still hampers adaptability in zero-shot scenarios.
The advent of large language models (LLMs) opens new avenues for zero-shot reasoning on graphs. 6 7). While simple, g Recent work therefore
A direct approach serializes graph data into text and feeds it to an LLM 44 this often underperforms because LLMs lack structural inductive bias
5
explores combining GNNs and LLMs, which can be grouped as follows.
LLMs enhancers. These methods keep GNN the primary predictor and employ the LLM only to inject auxiliary semantic signals-for for example, generating synthetic labels or textual node descriptions TO 12) Although such signals improve performance, the approaches inherit the architectural rigidity of GNNs and typically require retraining for new tasks. Replacing task-specific output heads with textual label embeddings enables limited zero-shot classification [13] [4] but does
not naturally extend to regression or other objectives, and semantic mismatch between graph and text remains an issue.
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Instruction tuning
Zero-shot
Attention Mechanism of Encnderé
Grapo Descrption
Task Details
NodeCCheitiiaaoonn Can lines domaing
Scauturt awart griph mtntion
Decuder
Tank Detzils
Graph Regacsiion
Large Language Model
Liven represetation of leute with the fnllowiing
Telloware graphtnod atamtonn
infomnaioo::
Modal Aligment Nintton
Queetion: Whhat the (eggh this melecale"
Plans provice a angle mumeiall valee rounded to tro
Task Details Node Casiitialion
places
Encoder
Large Language Model
LottA
Frown the reporsmtaaiio of paper with Solleweing informutan::
Craph Regrestin
Esol
Uneetion: Which ankie aur-- eategeey doee this paper beiong 5o? Pleac decels give soni likey
LIM
Molecalar Graphs
Greph Clasifiction Predistion Nede Clasiiication
Deeoder
Determine boady otegory within the retwock.
Cood Tolew
Toue
Tuned
Large Language Model
Ledeadige
Alignnl
Froezn
Answer Natural Language
Figure Framework of UniGTE
LLMs as predictors. Here the predictive role is assigned to the LLM, while GNN supplies structural information aligned to the LLM's semantic space-usually via self-supervised pretraining and cross-modal projection [15 16 17 Because the two components are trained separately, is hard to inject task-specific signals in a task-aware manner, limiting generalization. Deeper integrations such as GOFA [8] inject GNN features into LLM tokens inference time, boosting zero-shot performance but at a high computational cost and with persistent cross-task and cross-domain gaps.
Our proposal. We introduce UniGTE, unified encoder-decoder framework that is instruction-turee on a diverse suite of graph datasets (node classification, link prediction, graph classification; domains include citation networks, e-commerce graphs, molecular structures). The encoder augments pretrained autoregressive LLM to jointly consume a tokenized graph, natural-language task prompt, and fixed set of learnable alignment tokens. During self-attention, these alignment tokens aggregate structural and prompt signals into a compact task-aware graph representation. The decoder is a frozen autoregressive LLM that conditions on this representation to (i) generate the task prediction and (ii) reconstruct the graph prompt, with the latter providing auxiliary supervision via a prompt-level loss. This design yields single model that is permutation -inn--iiiinntt to node order, conditioned on task instructions, and capable of zero-shot generalization across modalities, tasks, and domains. Our key contributions
follows:
We present UniGTE-the the first unified encoder-decoder architecture that achieves zero-shot gener- alization across diverse graph tasks and domains without any task-specific fine-tuning.
UniGTE conditions graph representation learning on task prompts and embeds both graph structure and textual semantics in common space, enabling flexible adaptation across modalities and objectives.
Extensive experiments demonstrate state-of-the-art zero-shot results on node classification, link prediction, and graph regression across multiple domains.
2 Methodology
We present UniGTE, unified encoder-decodee framework for learning transferable graph represen- tations across heterogeneous tasks and domains. UniGTE instruction-tunee on diverse collection of graph datasets that cover multiple task families-noke node classification, link prediction, and graph classification and domains such as citation networks, e-commerce graphs, and molecular structures.
The architecture comprises an encoder and decoder. The encoder extends pretrained autoregressive large language model (LLM) to jointly process graph- graph-structured inputs and natural-language task prompts. Its input sequence comprises tokenized graph (e.g., node representations from a subgraph), a task-specific prompt, and a fixed set of learnable alignment tokens. These alignment tokens act as cross-modal anchors during self-attention, aggregating information from graph tokens guided by the

---

task prompt and distilling it into a compact, task-aware latent representaiion-tterenn -trreittiontteeeeedeee the task-awan
graph representation.
The decoder is frozen autoregressive LLM that conditions exclusively on the encoder outputs of the alignment tokens-that is, the task-aware graph representation. It autoregressively generates two outputs: (i) the task prediction and (ii) a reconstruction of the graph prompt. The latter serves as an auxiliary supervision signal, implemented via prompt-level loss that encourages the encoder to preserve the semantic content of the input graph. An overview of the framework is shown in Fig.
2.1 Task definition and notation
graph-learning task r E T is formally defined as 7 == {(9, of) , U (T"), where M is the number of graph instances. Each instance pairs graph G3 with target output yf, expressed sequence of text tokens that typically encodes a class label or a numerical value.
A graph is denoted 9 = EV,E, A, X, E), where V = {vi, 02,...,, UN is the node set and & {er, (a.ee,..., } the edge set. The adjacency matrix A E RNxN is defined by Aij 1 iff (vi, vy) E E. Node features are stored in E NNNx and edge features in E R REEIFE, with FNv and PE denoting their respective feature dimensions.
Each task is accompanied by natural-language instruction T = [TTese: Tieual containing two (d)
briefly states the objective-for for example, "Determine this node's 's most likely category within the network's 's classification schema." " The task- specific input Taasl = =[]] ..... w(''1 provides the contextualised information needed to solve the task, such "Given a representation of a paper with the following information: Title: (title), Abstract: (abstract). Question: Which arXiv CS sub-category does this paper belong to?"
components. The task description T [w]] ..... ......t (d)
2.2 Unified graph-text encoder: learning task-aware graph representations via LLM
In UniGTE, the encoder receives graph instance G., the task description T and fixed set of alignment tokens A (defined below), and embeds them into a shared latent space. The decoder, conditioned the encoder outputs corresponding to the alignment tokens-i.e. the task-aware graph representation-and and on the task-specific input Tremt then autoregressively produces the target sequence y.-
We first describe the unified graph-tex text encoder. Prior work shows that graph tasks depend on structural and attribute information to different extents [19 (0). To satisfy these varied requirements, the encoder must inject task cues into the aggregation process while retaining the ability to generalise across tasks and domains for zero-shot transfer. Inspired by the strong generalisation capacity of large language models (LLMs) in graph contexts, we build the encoder on a pretrained LLM that jointly encodes graph structure and task instructions, thereby learning task-aware representations.
2.2.1 Unified input formatting for graph tasks
To enable joint training across heterogeneous tasks and domains, we cast every task into a unified graph-level input. For node- and edge-level tasks, we extract n-hop subgraph centred on the target node or edge and treat all nodes in this subgraph as input tokens Tg:
Tg [w (g)
where each token (9) is obtained by encoding the node's s attribute text with a pretrained language model (PLM), and n |9] is the number of nodes in the subgraph. This abstraction lifts instance-level tasks to the graph level, promoting parameter sharing across task types (3].
To inject task semantics, we employ unified description template Tasse that specifies both the task type and a concise summary the input graph. Presenting this textual prompt alongside the graph tokens guides the encoder to produce latent representation that is simultaneously structure- and aask-
To bridge graph and text inputs in shared semantic space, we append fixed set of learnable alignment tokens:
A= [aj 22 ..... am
< +
3

---

where m is much smaller than the combined number of graph tokens n and text tokens E. These tokens distil and integrate information from both modalities and act as the sole interface to the decoder. This design unifies the representational spaces of graph and text while shrinking the overall token budget, allowing the model accommodate longer inputs within its sequence-length limit.
Given Tg, Tasse and A. the encoder input is
Xenc
[Tg
Tdesc
A]
R (R(+t(m)xdd,
where d, is the hidden dimension of the underlying language model.
2.2.2 Structure-awary graph-text attention
Beyond simple input formatting, we propose structure-awar graph-text attention mechanism. Existing methods often linearize graph nodes into token sequences and input them into large language models (LLMs). However, their performance heavily relies on fixed node ordering and degrades significantly under permutation [21]. This fragility stems from the absolute positional encoding in standard self-attention, which contradicts the permutation invariance inherent to graph data. To address this, we design a unified attention mechanism that jointly processes graph and text tokens while preserving structural invariance.
Let the projection matrices be Wo. WK. Wv xd,, and Wo R4 xd,, The scaled dot- product attention with relative position encoding, as applied in LLMs, is defined as
A (x) R(i (x, vdx
A = softmax(A). Attn = WXXW))Wo. (1)
where R(-) denotes the rotary positional encoding (RoPE) transformation
Cross-modal RoPE We extend RoPE accommodate graph, text, and alignment tokens. For text tokens, i and j represent absolute positions, and R(i j) operates as in standard RoPE. For graph tokens, assign shared, learnable position index <GraphPos>, such that i - 0 for all graph-node pairs. This causes R(0) to reduce to the identity matrix, removing order sensitivity. For cross-modal interactions, we compute the offset between text token positions and <GraphPos>, enabling the model to learn consistent alignments across modalities.
While this construction restores rede-permutation invariance, it discards explicit structural cues. We re-inject them through additive biases:
A; = (x R(WVORRR-AIIO (x + b(i.)),
dd
with b(i,j) = T(eskn) (bPE + b Edge)
BM. The three bias terms
Distance-basse structural bias distance between graph nodes as:
Following graph transformers 23 we embed the shortest-path
PE e (dist
where e(-) is learnable lookup table, and dist. (i.j) denotes the taott distance between nodes and j.
Edge-aware bias Graphs in practice feature heterogeneous edge types. For each edge es the shortest path (P, we compute
ben MLP (PMi""essiiiiii of eu.))
and aggregate by bt) natural-language description the model generalize to unseen edge semantics, benefiting zero-shot transfer.
Edge
(PJ) EBESP(E)) beg. Unlike discrete type embeddings 24 this

---

Masking bias
Finally, biy enforces directional constraints:
oc, A or
M
(5)
otherwise.
Graph tokens attend bidirectionally to capture connectivity; in addition, they may look forward into the text downstream instructions can refine their representations. Conversely, text tokens never attend to graph tokens, preventing information leakage. Alignment tokens obey the standard causal mask across both modalities.
Together, these biases endow our attention with (i) permutation invariance for graph inputs, (ii) explicit structural awareness, and (iii) flexible cross-modal interaction, enabling robust reasoning across diverse graph-text tasks.
2.2.3 Instruction tuning the unified graph-text encoder
After the encoder processes the input graph and prompt, we take the hidden states HA at the special alignment tokens A as task-aware graph representations. H.4 fuse structural cues from the graph with semantic hints from the task description and constitute the sole conditioning signal for the decoder.
wetectioottunning objective. We train concurrently on node-, edge- and graph-level tasks drawn from multiple domains. For task 5 with target sequence y" (yī ......., VT.)) instruction de the negative log- rigliike loss is
.........T...
and instance-specific
(Genc)
log P(ui H., Themin Vesi Oene). Σ t=1
(6)
Here Tiemil differs from the terse task description Tiee in that contains the concrete, instance-level input needed to solve т.
During instruction tuning we keep the decoder frozen and update only small subset of encoder parameters: LoRA adapters LORA alignment-token embeddings 0A,, the MLP weights ONLP that compute edge-aware bias, and the table 6g for ollative-position bias. We denote their union by m
Auxiliary prompt reconstruction. We additionally ask the decoder to reconstruct the graph description dg embedded in the prompt. This supervision encourages the alignment tokens to better encode structural information, while eliminating the need for a separate autoencoding stage. With (dg)) the auxiliary loss is
target tokens (de) (w)
(ds)
Lu )
La log P(wo
Cprompe (Penc)
Σ
(do))) HAs (dd)) Oenc).
Overall objective.
The total loss for task is the sum of the two terms:
Liotal (Denc) LTr (Denc)
Cprrompe (Denc).
(8)
In training minimise Σ, Linaa (Oene) with respect to Oenc-
2.3 Training and evaluation strategy
To assess the scalability and generalization ability of our model, curate diverse of graph datasets spanning multiple levels (node, edge, and graph) and domains. The benchmark includes both classification and regression tasks drawn from application areas such citation networks, e-commerce platforms, social media, and molecular graphs. Specifically, it comprises 17 datasets from five distinct domains, covering node classification, link prediction, graph classification, and graph regression tasks. Full details of the datasets are provided in the Appendix A We use subset of these datasets for instruction tuning, and directly evaluate the model's zero-shot performance on the remaining datasets without any further fine-tuning.

---

3
Experimental results
In this section, we conduct comprehensive experiments to validate the effectiveness of UniGTE. Our evaluation designed to address the following research questions:
RQI:
How well does UniGTE generalize to unseen datasets within the same domain (in-domain zero-shot)?
RQ2: Can UniGTE handle more challenging generalization settings, such transferring across domains or tasks unseen during training (cross-domain and cross-task zero-shot)?
RQ3: What are the respective contributions of task-aware graph encoding and alignment tokens to the zero-shot performance of UniGTE?
3.1 Experimental setup
Datasets We jointly train UniGTE five datasets: Arxiv 25. Children 26). Computer [26). FB15K237 [33. and ChEMBL (27). spanning node classification, link prediction. and graph clas- sification tasks. For Arxiv, Children, and Computer, construct both node classification and link prediction tasks to increase task diversity. After training, we evaluate UniGTE in zero-shot setting set of unseen datasets. For in-domain evaluation, (e.g., additional citation or e-commerce graphs). To assess cross-domain generalization, we evaluate datasets from different domains such as web graphs and social networks. Finally, evaluate cross-task generalization, we include a previously unseen graph regression task. Detailed descriptions of all training and evaluation datasets are provided in the Appendix A
use datasets from the same domains
Baselines We compare UniGTE against several recent state-of-the-ard models with demonstrated transfer and zero-shot capabilities. OFA [3] combines a GNN-based predictor with large language model (LLM) via prompt-based input augmentation. GraphGPT [15]. LLaGA (28). and TEA-
GLM 17 adopt an LLM as the primary predictor and align graph-text representations through either multi-layer perceptron (MLP) linear projection layer. GOFA [18] also employs an LLM the predictor but incorporates structural information through inter-layer graph aggregation within the LLM architecture. Due to the high computational cost of training and evaluating GraphGPT, report its results as provided in the original publication. For all other baselines, we re-ran the official implementations and conducted evaluations under our experimental setup. Detailed settings of the experimental environment can be found in Appendix
3.2 In-domain zero-shot generalization (RQ1)
To address RQ1, we evaluate each model's zero-shot performance on datasets from the same domains as those seen during training. These include citation networks (Pubmed (29) and Cora [30], with the latter being a more challenging variant featuring 70 classes), e-commerce datasets (Photo and Sports), and molecular graphs [HI, BACE, and PCBA). We report accuracy for node classification, and AUC for link prediction and graph classification, reflecting the standard metrics used for each task type.
As shown in Table UniGTE achieves the best overall performance across tasks and datasets, outperforming all baselines on the majority of benchmarks. Models relying on GNN-based predictors, such OFA, struggle generalize and exhibit weak transfer performance. Surprisingly, in most tasks, LLM-based methods like LLaGA and GraphGPT fail to outperform their base model, Vicuna- 7B, suggesting that their lack of permutation invariance hinders generalization-ccaagee in node ordering significantly impact their predictions.
Among the basclines. TEA-GLM applies pooling mechanism to produce fixed number of graph tokens, which preserves permutation invariance and contributes to better generalization node classification and link prediction tasks. However, its inability to incorporate task-specific signals leads to inconsistent performance across tasks and even negative transfer in graph classification. GOFA, trained using instruction tuning pipeline the official pre-trained checkpoint, achieves limited gains. Despite extensive pretraining, it underperforms in most tasks and fails to match even an untuned L.M in many cases.
In contrast, UniGTE demonstrates consistent positive transfer across all datasets and task types. This can be attributed to its use of task-specific signals during graph encoding, which enable the

---

Table 1: In-domain zero results. Bold and underline indicate the best and second-best results, respectively. N.S. denotes unsupported tasks.
Pubmed
Cora
Photo
Sports
BACE
HTV
PCBA
Pubmed
Photo
Model
Node Classification
Graph
Classification
Link Prediction
Vicuna-7B
0.721
0.155
0.384
0.371
0.492
0.467
0.497
0.502
0.576
OFA
0.237
0.189
0.317
0.047
0.483
0.404
0.424
0.499
0.499
GraphGPT-std
0.701
0.126
0.501
LLaGA
0.726
0.156
0.249
0.351
N.S.
N.S.
N.S.
0.740
0.659
TEA-GLM
0.781
0.202
0.418
0.357
0,467
0.498
0.434
0.663
0.675
GOFA
0.614
0.039
0.447
0.133
0.500
0.481
0.500
0.507
0,504
UniGTE
0.870
0.215
0.565
0.403
0.534
0.501
0.541
0.722
0.732
model to distinguish between task objectives, and its alignment tokens, which unify structural and semantic information. Together, these components contribute to UniGTE's superior generalization in in-domain, zero-shot scenarios.
3.3 Cross-domain and cross-task generalization (RQ2)
To evaluate the generalization ability of each model in more challenging settings, conduct zero- shot testing under both cross-domate and cross-task conditions. Specifically, we use datasets from domains not seen during Kaining-WikiCS. [32] (web links), Reddit, and Instagram (social networks) ass well as an entirely new task: graph regression, evaluated on Esol (34] Lipo, and We report accuracy for node classification and mean absolute
Freesoly K5
(MAF) for
regression.
As shown in Table 2 UniGTE outperforms all baselines across both domains and task types. Most baseline models show some degree of positive transfer, but results remain inconsistent. LLaGA and OFA do not support graph-level tasks due to limitations in model design. GOFA achieves stronger regression performance than TEA-GLM on some datasets, yet both models show clear trade-offs: they perform well on specific domains or tasks but fail generalize broadly.
In contrast, UniGTE consistently delivers strong performance across datasets and settings. The challenging nature of cross-domain and cross-task transfer underscores the importance of robust generalization. UniGTE's results demonstrate its ability to generalize effectively beyond the training distribution, highlighting the benefits of its unified graph-text representation and task-aware alignment mechanism.
3.4 Ablation study (RQ3)
We conduct ablation studies assess the contributions of two key components in UniGTE: alignment tokens and task-aware graph encoding. To evaluate the role alignment tokens, we remove them entirely and allow the decoder to generate outputs without their guidance. For task-aware graph encoding, we replace the task-specific description Taese with a fixed, generic prompt that is not tailored any specific task.
To provide a comprehensive analysis, we evaluate performance from both domain-level and task-level perspectives, averaging metrics across datasets within each category. To ensure comparability across different task arres-particclarty between classification and regression-we adopt a normalized MAE score defined as:
MAE MAE
MAE
MAE MAE
where MAEmin and MAE denote the minimum and maximum MAE values observed across all models and datasets. This normalization yields scores between 0 and I, where higher values indicate better performance, thus making them directly comparable to metrics such as accuracy and AUC.
As shown in Figure 2 both components contribute substantially to the model's generalization. "w/o AT" corresponds to the setting without alignment tokens, and "w/o TA" " reflects the ablation task- aware graph encoding. Across both perspectives, removing either component results in a consistent performance drop. The absence of alignment tokens significantly degrades performance, highlighting their role in capturing structured and semantic information essential for zero-shot inference. Similarly,
7

---

Table 2: Zero-shot performance on cross-domain node classification and cross-task graph regression.
Bold and underline indicate the best and second-best results, respectively. N.S. indicates tasks not supported by the model. Lower is better for regression (MAE).
WikiCS
Reddit
Instagram
Esol
Lipo
Freesolv
Model
Node Classification (Accuracy)
Graph Regression (MAE)
Vicuna-7B
0.290
0.309
0.391
6,58
11.22
64.11
OFA
0.361
0.498
0.580
N.S.
N.S.
N.S.
LLaGA
0.499
0.397
N.S.
N.S.
N.S.
TEA-GLM
0.449
0.491
0.479
14,90
9.76
13.35
GOFA
0.613
0.493
0.367
4.93
1.36
14.98
UniGTE
0.680
0.510
0.601
2.54
1.03
9.18
07
w//oAA
Nate Cb
Look
Gappi Ca
Gapp Rey
Glatis
Evantoment
Hel
Moleale
(a) Domain-level perspective
(b) Task-level perspective
Figure 2: Ablation study of alignment tokens and task-aware graph encoding, reported from both domain-level and task-level perspectives. "w/o AT indicates the removal of alignment tokens, while "w/o TA" refers replacing task-specific descriptions with generic prompt.
removing task-specific descriptions reduces performance across the board, confirming the importance of task-aware encoding in providing -ha-gga task signals.
Overall, these results underscore the necessity of both components: alignment tokens enhance generalization by bridging modalities, while task-aware encoding improves adaptability to diverse task objectives.
4 Related work
In this section, we review recent advances in zero-shot and transfer learning for graph machine learning. Existing approaches can be broadly categorized into two groups: (1) methods based solely on GNNs, and (2) methods that incorporate large language models (LLMMs). The LLM methods can be further divided into two subtypes: LLM an enhancer, where the LLM provides auxiliary semantic information assist graph models, and LLM as predictor, where the LLM itself performs prediction after being aligned with structural features.
4.1 Zero-shot and transfer learning with GNNs
A wide range of self-supervised learning techniques have been proposed to reduce the reliance of graph neural networks (GNNs) on labeled data and improve their generalization. For example, Deep Graph Infomax (DGI) [36] maximizes mutual information between local and global embeddings. Contrastive methods such as GraphCL [37], GCA B8), GCC [39], and JOAO [40] construct positive and negative views of graphs to learn invariant node representations. Generative approaches like
GraphMAE
adopt masked reconstruction objectives to jointly encode semantic and structural
Although effective, these models typically require task-specific fine-tuning on downstream datasets. To improve transferability, prompt-based GNNs have recently gained attention. GraphPrompt 2 proposes unified prompt template shared across pretraining and fine-tuning, improving knowledge transfer. ProG (3] reformulates node-level, link-level, and graph-level tasks into unified prompting

---

framework and leverages meta-learning for multi- task adaptation. However, these approaches still depend heavily on GNN-specific architectures, making it difficult to generalize across tasks datasets with varying output spaces without further retraining.
4.2 LLMs for graph zero-shot and transfer learning
4.2.1 LLMs as enhancers
One line of work leverages LLMs as semantic enhancers- geerrssiingngggggggge task descriptions, pseudo-
labels, or contextual cues guide GNNs during training [10 13 [4]. These methods benefit from the rich prior knowledge encoded in LLMs; however, the final prediction is still performed by GNNs, limiting their flexibility and generalization in unseen tasks. Some studies [33 14 further propose novel architectural designs that enable GNNs to support zero-shot transfer across datasets. Despite these improvements, such approaches still exhibit limited generalization ability and are inherently unsuitable for regression tasks.
4.2.2 LLMs as predictors
A second line of work treats LLMs as predictors. Some studies attempt to serialize graph data into textual sequences [5 4 which are then directly fed into LLMs for inference. While this approach enables zero-shot reasoning by leveraging pretrained language models, it often compromises structural fidelity, as LLMs primarily capture token-level co-occurrence patterns rather than
inductive biases [8]
Some methods like LLaGA [28 attempt to convert nodes into token sequences and input them directly into I.LMs. However, this approach loses permutation invariance-a key property of graph latta-making the output highly sensitive to node ordering (1]). To better preserve structural information, other works incorporate GNNs to extract graph features, which are then aligned with
LLM inputs. For example, GraphGPT [15] aligns a graph transformer encoder with LLM through a two-stage training process. UniGraph [16] employs masked word prediction to pretrain GNN and aligns with LLM embeddings via MLP-based projection. TEA-GLM 17 further proposes a feature-wise contrastive pretraining strategy, followed by lightweight projection for aligmment. Despite these advances, most existing methods treat GNNs and LLMs as loosely coupled modules and perform aligmment in a post hoc manner, limiting their adaptability to task-specific signals. As a result, such models often struggle in multi-task cross-domain scenarios.
GOFA 18 addresses this limitation by introducing a tighter integration strategy, inserting GNN layers between LLM transformer blocks enable token-level structural aggregation. However, this architecture incurs high computational costs and still faces performance challenges in zero-shot settings across diverse tasks and domains.
5
Limitations
While UniGTE achieves strong zero-shot generalization across a wide range of graph tasks and domains, its performance gains on link prediction tasks are less pronounced compared to node classification and graph regression. This may be due to the pairwise nature of link prediction, which poses unique challenges for prompt formulation and representation alignment. Exploring more effective strategies for encoding edge-level interactions and adapting task prompts for link prediction remains an important direction for future work.
6 Conclusion
We presented UniGTE, a unified encoder-decoder framework for zero-shot graph learning. UniGTE fuses graph structure and natural-language task instructions in a shared representation space via a commutation-invariann encoder with learnable alignment tokens, while frozen LLM decoder produces both task predictions and prompt reconstructions. This design supports flexible transfer across node-, edge-, and graph-level tasks and across disparate domains. Extensive experiments verify UniGTE's robustness, setting new zero-shot state-of-the-ar results under demanding cross-task and cross cross-domain conditions. Our study highlights the benefit of tightly integrating structural and semantic cues for broadly transferable graph reasoning.

---

7 Acknowledgement
This work was supported by the National Key R&D Program of China (2023YFC33004700) The work of Yuan Zuo was partially supported by the National Natural Science Foundation of China (72571019) 7253 1002) and the Shenzhen Science and Technology Program (CJGIZD2223372240220000 Dr. Junjie Wu's work was partially supported by the National Natural Science Foundation of China (72031001, 72242101) and Outstanding Young Scientist Program of Beijing Universities (JWZQ202402010002.
References
[1] Mingxuan Ju, Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye, and Chuxu Zhang. Multi-task supervid graph neural networks enable stronger task generalization. In The Eleventh International Conference on Learning Representations, 2023.
[2] Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, page 417-428, 2023.
[3] Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. All in one: Multi-task prompting for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 2120-2131, 2023.
[4] Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, and Jiliang Tang. Exploring the potential of large language models (LLMs) in learning on graph. NeurIPS 2023 Workshop: New Frontiers in Graph Learning, 2023.
[5] Jiayan Guo, Lun Du, and Hengyu Liu. Gpt4graph: Can large language models understand graph structured data ? an empirical evaluation and benchmarking. ArXiv, abs/2305.1500 2023.
[6] Heng Wang, Shangbin Feng. Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? In Thirtty-eventh Conference on Neural Information Processing Systems, 2023.
[7] Chang Liu and Bo Wu. Evaluating large language models on graphs: Performance insights and comparative analysis. arXiv preprint arXiv:2308, /224, 2023.
[8] Jin Huang, Xingjian Zhang, Qiaozhu Mei, and Jiaqi Ma. Can Ilms effectively leverage graph structural information: when and why. arXiv preprint arXiv:2309.16599, 2023.
[9] Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134 2023.
[10] Jianxiang Yu, Yuxiang Ren, Chenghua Gong, Jiaqi Tan, Xiang Li, and Xuecang Zhang. Em- power text-attributed graphs learning with large language models (Ilms). arXiv preprint arXiv:2310.09887. 2023.
[11] Lianghao Xia, Ben Kao, and Chao Huang. Opengraph: Towards open graph foundation models. arXiv preprini arXiv:2403.011 121, 2024.
[12] Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, Hui Liu, and Jiliang Tang. Label-free node classification on graphs with large language models (LLMs). In The Twelfth International Conference on Learning Representations, 2024.
[13] Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang. Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. In The Twelfth International Conference on Learning Representations, 2024.
[14) Yuhan Li, Peisong Wang, Zhixun Li, Jeffrey Xu Yu, and Jia Li. Zerog: Investigating cross- dataset zero-shot transfcrability in graphs. arXiv preprint 40X. 11235, 2024.
10

---

[15] Jiabin Tang, Yuhao Yang. Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. arXiv preprint arXiv:2310.1302, 2023.
[16] Yufei He and Bryan Hooi. Unigraph: Learning a cross-domain graph foundation model from natural language, arXiv preprint arXiv:2402. 13630, 2024.
[17] Duo Wang, Yuan Zuo, Fengzhi Li, and Junjie Wu. Llms as zero-shot graph learners: Align- ment of gnn representations with Ilm token embeddings. In Advances in Neural Information Processing Systems, 2024.
[18] Lecheng Kong, Jiarui Feng, Hao Liu, Chengsong Huang, Jiaxin Huang, Yixin Chen, and Muhan Zhang. GOFA: A generative one-for-all model for joint graph language modeling. In The Thirteenth International Conference on Learning Representations, 2025.
[19] Duong Chi Thang, Hoang Thanh Dat, Nguyen Thanh Tam, Jun Jo, Nguyen Quoe Viet Hung, and Karl Aberer. Nature vs. nurture: Feature structure for graph neural networks. Pattern Recognition Letters, 2022.
[20] Diana Gomes, Frederik Ruelens, Kyriakos Efthymiadis,, Ann Nowe, and Peter Vrancx. When are graph neural networks better than structure structre-agnostic methods? In I Can't 't Believe It's Nor Better Workshop: Understanding Deep Learning Through Empirical Falsification, 2022.
[21] Xixi Wu, Yifei Shen, Caihua Shan, Kaitao Song, Siwei Wang. Bohang Zhang, Jiarui Feng, Hong Cheng, Wei Chen, Yun Xiong, and Dongsheng Li. Can graph learning improve planning in LLM-based agents? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[22] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568:127063, 2024.
[23] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. In Advances in Neural Information Processing Systems, 2019.
[24] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems, pages 28877-28888, 2021.
[25] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems, pages 22 18-22133,, 2020.
[26] Hao Yan, Chaozhuo Li, Ruosong Long, Chao Yan, Jianan Zhao, Wenwen Zhuang, Jun Yin, Peiyan Zhang, Weihao Han, Hao Sun, Weiwei Deng, Qi Zhang, Lichao Sun, Xing Xie, and Senzhang Wang. A comprehensive study on text-attributed graphs: Benchmarking and rethink- ing. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[27] Anna Gaulton, Louisa J. Bellis, A. Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvnnn Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Over- ington. Chembl: a large-scale bioactivity database for drug discovery. Nucleic Acids Research, 2011.
[28] Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. Llaga: Large language and graph assistant. ICML, 2024.
[29] Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. Harnessing explanations: LLM-to-LMM interpreter for enhanced text-attributed graph repre- sentation learning. The Twelfth International Conference on Learning Representations, 2024.
[30] Zhihao Wen and Yuan Fang. Augmenting low-resource text classification with graph-grounded pre-training and prompting. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 506-516, 2023.
11

---

[31] Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg. Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: A benchmark for molecular machine learning, 2018.
[32] Péter Mernyei and Cătälina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural networks, 2022.
[33] Yuhan Li, Peisong Wang, Xiao Zhu, Aochuan Chen, Haiyun Jiang, Deng Cai, Victor Wai Kin Chan, and Jia Li. Glbench: A comprehensive benchmark for graph with large language models, 2024.
[34] Michael Withnall, Hongming Chen, and Igor V Tetko. Matched molecular pair analysis on large melting point datasets: big data perspective. ChemMedChem, 2018.
[35] Rodrigo Casasnovas, Joaquin Ortega- Castro, Juan Frau, Josefa Donoso, and Francisco Munoz. Theoretical pka calculations with continuum model solvents, alternative protocols to thermody- namic cycles. International Journal of Quantum Chemistry, 2014.
[36] Petar Veličković, William Fedus, William L. Hamilton, Pictro Lió, Yoshua Bengio, and Devon Hjelm. Deep graph infomax. In International Conference Learning Representations, 2019.
[37] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems, pages 28877-28888, 2021.
[38] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, page 2069-2080, 2021.
[39] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gec: Graph contrastive coding for graph neural network pre-training, In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, page 115-1160, 2020.
[40] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In ICML, 2021.
[41] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang. and Jie Tang. Graphmae: Sele-supe masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 594-604, 2022.
[42] Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. Graphmae2: A decoding-enhanced masked self-supervised graph learner. In Proceedings of the ACM Web Conference 2023, page 737-746, 2023.
[43] Mengmei Zhang, Mingwei Sun, Peng Wang. Shen Fan, Yanhu Mo, Xiaoxiao Xu, Hong Liu, Cheng Yang, and Chuan Shi. Graphtranslator: Aligning graph model to large language model for open-ended tasks. In Proceedings of the ACM Web Conference 2023, 2024.
12

---

A Dataset descriptions
Table 3; Dataset statistics
Domain
Dataset
Avg. #Nodes
Avg. #Edges
#Classes/Taske
#G
Arxiv
169,343
1,166,243
40
Citation
Pubmed
19,717
44,338
Cora
25,120
91,140
70
Computer
87,229
721,081
10
Photo
48,362
500,928
12
E-commerve
Children
76,875
1,554,578
24
Sports
173,055
1,773,500
13
Web link
WikiCS
11,701
216,123
10
Knowledge graph
FB15K237
14,541
310,116
237
Reddit
33,434
198,438
Social network
Instagram
11,339
144.010
ChEMBL
25.87
55.92
1048
365,065
HIV
25.51
54.95
41,127
BACE
34.09
73.72
1.513
Molecular
PCBA
25.97
$6.20
128
437,929
Esol
13.29
27.35
1,128
Lipo
27.04
59.00
4,200
Freesolv
8.72
16.76
642
a large-scale citation graph derived from arXiv Computer Science papers. Each node corresponds to paper and edges represent citation links between papers. The task is to classify each paper into one of 40 arXiv subcategories, such as "cs.LG" "cs.Al". This dataset serves as representative benchmark for large-scale node classification.
Arxiv Arxiv 25
Pubmed Pubmed [29] is citation network of biomedical research papers from the PubMed database. Each node is paper and edges correspond to citation links. The classification task involves assigning each paper one of three disease- related categories.
Cora The Cora 30 dataset is citation graph where each node corresponds to research paper, and each edge represents a citation link between papers. The dataset focuses on papers within the machine learning domain and includes 70 fine-grained categories, making the classification task particularly difficult.
Computer The Computer [26] dataset is co-purchased or co-viewed product graph, where each node represents product in the computer category, and edges indicate that two products were frequently co-purchased or co-viewed by users. The textual content associated with each node consists of user-generated reviews for the corresponding product.
Photo The Photo 26 dataset is e-commerce product graph where nodes represent photographic products, and edges indicate that two items were either co-purchased or co-viewed by users. The textual content of each node consists of user reviews associated with the corresponding product.
The Children [26] dataset co-purchased or co-viewed product graph focused on children's 's books. Nodes correspond to individual books, and edges connect books that were frequently browsed bought together. Each node associated with textual information including the book's title and descriptive metadata.
Children
The Sports (26 dataset is co-purchased co-viewed product graph in the sports domain. Nodes represent sports-related products, and edges indicate that two items were often purchased or viewed together. The associated text for each node consists of the product's title.
Sports
is a web link network constructed from English Wikipedia articles related to computer science. Nodes are individual articles, and directed edges represent hyperlinks between them. The node text is the full content of each article.
WikiCS WikiCS 32
13

---

Table 4: Task-specific descriptions used as Tesse
Task Type
Task Description (Téese)
Node Classification
Determine this node's most likety category within the network's s classification schema.
Link Prediction
Determine whether there is a specific relationship between these two nodes.
Graph Classification
Determine whether the molecule possesses specific physicochemical bioactivity properties.
Graph Regression
Predict the continuous numerical value of physicochemical or bioactivity property of the molecule.
a large-scale knowledge graph where each node represents an entity (e.g., person, location, or object) and each edge corresponds to a relational triple connecting two entities. Textual content for nodes is constructed from entity names and relation descriptions.
FB15K237 FB FB15K237
Reddit The Reddit 3 dataset is social interaction graph where nodes correspond to users and edges represent reply interactions in threads from specifie time period. Node-associated text consists of user posts from Reddit threads.
Instagram Instagram 33 is a social graph in which each node represents a user, and edges denote social connections such following relationships. The textual content associated with each node is extracted from users' ielf-introductions or profile descriptions.
CHEMBL CHEMBL 27 is molecular graph dataset where each graph corresponds to chemical compound. Nodes represent atoms, and edges denote chemical bonds. The textual information for each molecule is given by its SMILES (Simplified Molecular Input Line Entry System) representation.
The HIV 31 dataset consists of molecular graphs representing candidate compounds for HIV treatment. Nodes denote atoms and edges are chemical bonds. Each molecule described by its SMILES string.
HIV
BACE BACE 31 is molecular dataset used in bioactivity classification. Each graph molecule, with atoms as nodes and chemical bonds as edges. SMILES strings provide the molecular structure information in text format.
PCBA PCBA [31 is large-scale molecular dataset for virtual screening. Each graph a molecule, modeled by atoms and bonds, with SMILES strings representing the underlying chemical structure.
The Esol 34 dataset contains water-solubility data for chemical compounds. Each molecule modeled as a graph, with node and edge structures corresponding to atoms and bonds. SMILES strings serve as the textual representation.
Esol
Lipo
Lipo is molecular dataset focused lipophilicity prediction. Each molecule is represented as a graph with atoms as nodes and bonds as edges. The SMILES string encodes each molecule's structure in text form.
Freesoly Freesolv B5 consists of molecular graphs used for estimating hydration free energy. Each molecule is modeled by a graph of atoms and bonds. The SMILES representation is used as the text-based molecular description.
B Details of prompt descriptions
For each specific task т, we define high-level textual description Ti. that serves the general instruction for the task. This description is task-type dependent and is designed to guide the model's
14

---

Table 5: Task-specific instruction templates used as detail-
Dataset
Instruction Template (Firatt)
Antiv, Pubmed. Cora
Given representation of paper with the following information: Titl: [title]. Abstract: [ubstract]. J. Question: Which arviv CS sub-category does this paper belong to? Please
directly give the most likely answer from the following sub categories: (candidate lahels).
Children
Given representation of book with the following information: Name: [title], Content: [abstract], J. Question: Which calegory does this book belong to? Please directly give the most likely answer from the following categories: {candidate _labels).
Computer, Photo
Given representation of book with the following information: Name: [title), Content: [abstract). J. Question: Which category does this book belong to? Please directly give the most likely answer from the following categories: (candidate_labble
Sports
Given a representation of electronic product with the following information: Comment: [comment]. Question: Which category does this electronic product belong to? Please directly give the most likely answer from the following categories: [candidate_
WikiCS
Given a sepresentation of Wikipedia page with the following information: Name: [name]. Content: [content). Question: Which category does this Wikipedia page belong to? Pleuse directly give the most likely answer from the following categories: [candidate_labee
Reddit
Given representation of user with the following information: Previous posts: [posts). ). Question: Which cutegory does this user belong to? Pleuse directly give the most likely answer from the following categories: (candidate_labees
Instagram
Given representation of a user with the following information: Personal introduction: (introduction). Question: Which category does this user helong to? Please directly give most likely answer from the following categoriis:: [candidate_lahes labells)
FB15K237
Given the representation of two entities: First entity: Name: [name], Description: (de- scription). J. Second entity: Name: [name], J. Description: [description). Question: Which category should the relation between these two entities be classified as? Please directly give the most likely answer from the following categories: (candidate_ labels).
Arxiv-link, Pubmed-ling link
Given the representation of two papers: Title: First Paper: [title]. Second Paper: [title). Question: Do these two papers have citation relationships? Please choose the most likely unswer from: "Yes, they have citation relationships' or "No, they do not have citation relationships..
Children-link
Given the representation of two books: Title: First Book: [title), Second Book: (title). Question: Do these two books have co-purchased co-viewed relationships? Choose from: "Yes, they have co sorpucchased or vo-weeed relationships" or "No, they do not have co-purchased or co-viewed relationships".
Computer- link, Photo-link
Given the representation of two electronic products: Title: First Product: [comment), Second Product: [comment). Question: Do these two products have co-purchased or co-viewed relationships? Choose from: "Yes, they have co- co-purchased co-viewed relationships" or 'No, they do not have co-purchased or co-viewed relationships.
ChEMBL, HIV, BACE, PCBA
Given a representation of a molecule with the following information: SMILES: (smiles). Question: [task] Pleuse answer: "Yes, this molecule is effective to this assay" 'No, this molecule is not effective to this assay".
Esol, Lipo. Freesolv
Given a representation of molccule with the following information: SMILES; (smiles). Question: [tusk) Pleuse provide single numericul value rounded to decimal places.
understanding and response generation. The full list of task descriptions used in our experiments is shown in T
Additionally, we provide task-specific content instructions Timmy which include a description of the graph content and the corresponding question. The details are shown in the table 5
C Details of experimental setup
For data splitting, we follow standard splits for node classification, graph classification, and graph regression tasks. For link prediction, we randomly split the data into training/validatiooneest sets with a ratio of 8:1:1. To ensure fair comparison, all baseline models are evaluated using the same splits. Due to the large scale of the training data, we sample a subset of instances from each training dataset. Specifically, we sample 45,470 instances from Arxiy, 21,888 from Children, 31,378 from Computer, 10,000 each from Arxiv-link, Children-link, and Computer-link, 29,440 from FB15K237, and 74,242 from CLEMBL.
Datasets
15

---

Table 6: In-domain zero-shot results.
Pubmed
Cora
Photo
Sports
BACE
HIV
PCBA
Pubmed
Photo
Model
Node Classification
Graph
Classification
Link Prediction
w/o AT
0.721
0.135
0.384
0.371
0,492
0,467
0.497
0.502
0.576
w/o
0.766
0.211
0.399
0.311
0.487
0.499
0.489
0.609
0.710
UniGTE
0.870
0.215
0.565
0.403
0.534
0.501
0.541
0.722
0.732
Table 7: Zero-shot performance on cross-domain node classification and cross-task graph regression.
WikiCS
Reddit
Instagram
Esol
Lipo
Freesolv
Setting
Node Classification (Accuracy)
Graph Regression (MAE)
w/o AT
0.290
0.309
0.391
6.58
11.22
64.11
w/o TA
0.645
0.502
0.590
8.35
2.44
12.37
UniGTE
0.680
0.510
0.601
2.54
1.03
9.18
Table 8: Legality rate (%) across models and datasets
Dataset
Pubmed
Cora
Photo
Sports
WiliCS
Reddit
Instagram
HTV
BACE
PCBA
Esol
Lipo
Freesoly
Model
Logality rale (%)
Vicuna-7B
64.7
LLaGA
100
76.5
83.8
99.2
99.0
99.0
99.5
82.1
100
75.1
66.3
82.2
$7.0
TEA-GLM
100
95.6
96.2
100
70.9
96.8
99.7
100
100
100
80.6
88.1
45,0
UniGTE
100
95.0
99.7
99.7
99.6
100
100
100
100
100
99.1
99.8
100
Baselines For LLaGA and TEA-GLM, we re-run their training pipelines under our experimental settings using Vicuna-7B as the predictor. As GOFA requires extensive pretraining, we directly use the officially released pretrained checkpoint and conduct instruction tuning under our experimental setup. For all baselines, we follow the hyperparameter configurations provided in their respective original papers.
Training details We use BERT as the pretrained language model (PLM), and Vicuna-7B as the large language model (LLM). In our implementation of UniGTE, we set the LoRA learning rate and the MIP learning rate to 2e-4, while the learning rate for the graph relative position embedding is 2e-3. We use a batch size of 2, apply gradient clipping with maximum norm of 10, and perform gradient accumulation every steps. The number of alignment tokens fixed to 64 across all experiments. All experiments are conducted on machine with two NVIDIA A100 GPUs, each equipped with 80GB of memory.
D More experimental results
D.1 Ablation study details
We provide detailed ablation results for each dataset. The in-domain results are presented in Table 6 while the cross-domain and cross-task results are shown in Table As the tables indicate, removing the alignment tokens significantly degrades the model's transferability. Additionally, replacing the task-specific description with fixed, generic prompt leads to performance drops across different datasets. These findings validate the effectiveness of the proposed alignment tokens in enhancing generalization, and further demonstrate the importance incorporating task-specific signals into the encoding process.
D.2 Legality rate
The training process may affect the instruction- foolloionn--looiingg.o ability of LLMs in zero-shot scenarios. Specifically, while the model can generate appropriate outputs on the training set, it often fails to produce legality answers unseen datasets. This essentially reflects poor generalization ability.
16

---

Following the approach proposed in 43 we use the legality rate to measure the proportion of valid responses generated by the model on unseen datasets.
As shown in Table [ existing baselines exhibit poor instraction-following performance on certain datasets. In contrast, model consistently maintains strong instruction compliance across all unseen datasets, generating valid answers. This further demonstrates the superior generalization capability of our approach.
D.3 Efficiency and computational complexity analysis
We have conducted a comparison between LLM and UniGTE under a zero-shot setting using equivalent graph information. The results summarized in the tables below, where the first table reports inference time and the second table presents task-specific performance metrics, showing that UniGTE achieves significantly faster inference speed while maintaining superior accuracy.
Table 9: Inference time compared with the LLM
Sec sample (s)
Pubmed
Cora
PCBA
Photo_link
WikiCS
Instagram
Lipo
Vicuna-7B
0.8
2.6
3,2
1.7
0.7
3.3
10.2
UniGTE
0.5
0.6
1.0
1.1
0.3
0.3
2.0
Improvement
37.5%
78.7%
70.3%
34.1%
50.8%
91.7%
80.5%
Table 10: Task performance compared with the LLM
Task Metrie
Pubmed
Cora
PCBA
Photo link
WikiCS
Instagram
Lipo
Vicuna-7B UniGTE
0.721
0.155
0.497
0.576
0.290
0.391
11.220
0.870
0.215
0.541
0.732
0.680
0.601
1.030
Improvement 20.7%
38.7%
8.9%
27.1%
134.5%
53.7%
90.8%
Here, we compare the time complexity of GOFA. We adopt the same notation used in the GOFA paper: suppose a graph contains V nodes, E edges, and each node is represented by & tokens on average. GOFA, each node is expanded into tokens (k: 1), and the self-attention restricted within each node's s token group. Thus, the per-layer time complexity is: O - V) ((V)2), which is consistent with complexity reported the original GOFA paper. In contrast, UniGTE represents each node using single token and performs cross-node attention directly. Therefore, the complexity becomes: (((V (1))))) = O(V2). This is considerably lower, especially when & = 128 or higher required in GOFA for satisfactory performance. Furthermore, method leverages k-hop subgraph sampling, which keeps the number of nodes V relatively small, making O(V) < O(k))
and thus ((V2) < OVV)2).
In addition the Transformer layers of the LLM itself, GOFA also introduces external GNN layers. This component is not accounted for in its original complexity analysis. Assuming the average node degree is d, the GNN layer introduces an additional cost O(V d. k). Combining all these considerations, our method provides a clear computational advantage over GOFA.
To further validate this, compare per-sample inference time and accuracy under the zero-shot setting between GOFA and UniGTE. As shown in the table below, UniGTE achieves faster inference and better performance, demonstrating the efficiency and effectiveness of our design.
Table 11: Inference time compared with GOFA
Sec sample (s)
Pubmed
Cora
PCBA
WikiCS
Instagram
Lipo
GOFA
1.4
5.1
1.5
1.8
2.0
1.2
2.1
UniGTE
0.5
0.6
1.0
1.1
0.3
2.0
Improvement
63.2%
89.3%
34.5%
38.5%
83.6%
77.1%
5.7%

---

Table 12: Task performance compared with GOFA
Task Metrics
Pubmed
Cora
PCBA
Photo_ Jink
WikiCS
Instagram
Lipo
GOFA UniGTE
0.614
0.039
0,500
0.504
0.613
0.367
1.360
0.870
0.215
0,541
0.732
0.680
0.601
1.030
Improvement
41.7%
451.3%
8.2%
45.2%
10.9%
63.8%
24.3%
Table 13: Supervised results under SFT setting
Model
Arxiv
Children
Computer
Arxiv link
Children link
Computer link
FB15K237
GraphGPT-std
0.625
LLsGA
0.711
0.579
0.901
0.837
0.793
0.870
TEA-GLM
0.661
0.549
0.767
0.738
0.910
GOFA
0.659
0.190
0.514
0.772
0.505
0.563
0.731
UniGTE
0.713
0.431
0.589
0.913
0.813
0.765
0.901
D.4
Supervised results
We have conducted additional experiments under the SFT setting and compared UniGTE with relevant baselines that use LLMs predictors. The results presented below.
As shown in Table 13 UniGTE achieves competitive performance and yields strong results on most datasets, even surpassing state-of-the-art baselines several of them. This highlights its promising capability under SFT settings. These results demonstrate that although our model originally designed for zero-shot scenarios, it also performs robustly under full supervision, without significant degradation compared to existing approaches.
D.5 Graph understanding tasks
We have conducted additional experiments graph understanding tasks to demonstrate model's graph understanding/reasoning capabilities, and compared the results with relevant baselines.
Table 14: Comparison of various models graph reasoning tasks
Model
Pubmed_Coom
Cora_CONN
Pubmed_SPPD
Cora_SPD
Pubmed_ CN
Coru_CN
BACE_CYCLE
HIVCEYCLE
PYCLE
Vicuna-8B
0.551
3.505
1.51
2.55
5.07
12:55
2.02
1.37
2.11
LlaGA
0.688
0.553
3.72
7.08
TEA-GLM
0.699
B,223
3.06
2.34
5:27
7.79
699
7.39
9.74
GOFA
0,68
0.694
143
166
1.13
2.28
6.96
5.58
6.24
UniGTE
0,707
0.616
1.11
1.42
120
2.24
188
1.10
1.92
We directly evaluated the model trained in our original submission zero-shot graph understanding tasks, without any fine-tuning on these tasks. All other baselines were evaluated under the same zero-shot setting, except for GOFA, which incorporates some graph understanding tasks during its pretraining phase. To ensure fair comparison, all methods were tested on the same test sets. Following the experimental setup of GOFA and the references you kindly suggested, we constructed the following tasks and generated the data using real-world datasets. We used AUC for evaluating the CONN task, and MAE the other tasks.
CONN: Determine whether two nodes are connected
SPD: Predict the shortest path distance between two nodes
CN: Predict the number of common neighbors between two nodes
CYCLE: Predict the number of cycles in the graph
The results are summarized the Table [4] where the best result bolded and the second-best italicized. Our method achieves th- performance on most datasets. Notably, compared to Vicuna, which serves as our base model, our method consistently outperforms it across all tasks and minnsttt-highlgghiing the strong generalization and zero-shot capabilities of our approach.
18

---

Compared to other baselines, our method is either the best or competitive. It is worth noting that GOFA, benefiting from the inclusion of graph understanding tasks during its pretraining phase, performs relatively well on a few specific datasets. In contrast, our method has never encountered any graph understanding tasks during training, making the evaluation significantly more challenging and rigorous zero-shot setting. Despite this, our approach still delivers consistently strong performance, further demonstrating its generalization capability.
19

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We verify the contributions of our proposed method through experiments, and the results in the Sec. effectivvey demonstrate the contributions we outlined in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the current shortcomings of our method and future research
directions in Sec
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all necessary information ensure the reproducibility of main experimental results. Section 2 details the proposed methodology, while Sections 2.3 and detailed setting Appendix C outline the training procedures and implementation settings, All reported results in this paper can be reliably reproduced based on the disclosed configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
21

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use publicly available datasets in all experiments. To ensure reproducibility. we will include the complete source code in the supplementary materials in an anonymized form.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
mublicc/guides/Cddeeublisss sos.ssoossssooooocoooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips https//ccccbllicgggddee//e/eeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details on data splits, hyperparameter settings, and baseline configurations Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
22

---

Justification: Due to the high computational cost associated with training large language models (LLMs), do not report error bars or variance across multiple runs. However, our evaluation spans a wide range of datasets and task types across multiple domains, and the consistent performance our method across these diverse settings provides strong empirical support for the robustness and generalization of the proposed approach.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the computer resources in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We confirm that our work complies with the NeurlPS Code of Ethics, including ethical standards for data collection, usage, and experimentation.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
23

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The paper does not discuss societal impacts because focuses on foundational research in graph learning. The method is generic and does not directly involve application scenarios with immediate societal implications.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release any data or models that pose high risk, and this work does not involve introduce such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
24

---

Answer: [Yes]
Justification: We properly cite the original papers corresponding to all models and datasets used in work. Wherever applicable, we also ensure that comply with the licenses and terms of use these assets. For publicly available datasets and pretrained models, we use only those released under permissible licenses (e.g.. MIT, CC-BY).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ ddtasset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
25

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: This paper employs large language model (Vicuna-7B)
component of the proposed encoder-decoder framework. Specifically, the LLM is used as the decoder to generate task-specific outputs conditioned alignment tokens. The LLM's generalization ability is integral to the design of our method and plays central role achieving zero-shot performance across diverse graph tasks.
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
26


Reviews:
[{"review_id": "rdRi1ap2TL", "reviewer": "Reviewer_U6Sz", "summary": "The paper titled “UniGTE: Unified Graph-Text Encoding for Cross-Modal Retrieval and Captioning” proposes a unified framework that encodes both graph and text modalities into a shared embedding space. It introduces graph-aware attention and a contrastive training strategy to bridge the semantic gap between graphs and text. Experiments on benchmark datasets demonstrate that UniGTE achieves strong performance on tasks such as cross-modal retrieval and graph captioning, outperforming existing baselines and validating its generalization ability across modalities.", "questions": "1.\tAlignment Strategy Clarification:\nHow does UniGTE ensure that the learned representations of graphs and texts are semantically aligned beyond contrastive loss? Have you considered other alignment signals such as mutual information or co-attention mechanisms?\n2.\tGraph Variability Handling:\nHow does UniGTE perform when the input graphs vary significantly in size, sparsity, or topology? Does the model require any normalization or preprocessing to handle such variations?\n3.\tModality Imbalance:\nIn training or inference, does the model exhibit bias toward one modality (e.g., favoring text features over graph structure)? How is this addressed, if at all?\n4.\tGeneralization to Unseen Modalities or Tasks:\nCould the UniGTE framework be extended to incorporate additional modalities such as images or tables? If so, what would need to be modified?", "limitations": "No.\nThe paper does not explicitly discuss the limitations or potential negative societal impacts of UniGTE. While the technical contributions are clear, a more responsible and comprehensive treatment of broader implications would strengthen the work. Here are a few constructive suggestions:\n1.\tBias and Misalignment Risk: The model could potentially learn and reinforce biases present in the training data, especially if the text or graph inputs reflect real-world inequalities. The authors should address how this risk is mitigated or detected.\n2.\tMisuse Potential: Cross-modal retrieval systems could be used for surveillance or misinformation if misapplied. A short section acknowledging such risks and encouraging ethical use would be valuable.\n3.\tLimitations in Generalization: The paper would benefit from more discussion of performance trade-offs when transferring to out-of-domain tasks or graphs with drastically different structure or semantics.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.\tUnified Framework: UniGTE presents a novel approach to jointly encoding graphs and text into a shared representation space, which is a significant step toward bridging cross-modal understanding.\n2.\tGraph-Aware Attention Mechanism: The model effectively integrates graph structures into the attention computation, allowing better alignment of textual and structural information.\n3.\tStrong Experimental Results: The method achieves state-of-the-art or competitive performance on multiple tasks, including cross-modal retrieval and graph captioning, demonstrating its effectiveness and generalizability.\n4.\tTask Versatility: UniGTE supports both retrieval and generation tasks, showing the model’s adaptability across modalities and applications.\n\nWeaknesses:\n1.\tLimited Modal Diversity: While the model unifies graphs and text, it does not extend to other modalities (e.g., images, audio), which may limit its applicability in broader multimodal contexts.\n2.\tScalability Concerns: The paper lacks detailed analysis of computational efficiency and scalability when dealing with large-scale graphs or long text inputs.\n3.\tAblation Study Scope: Although some ablation is provided, further breakdown on the contribution of each graph component (e.g., edge types, node features) would strengthen the empirical analysis.\n4.\tInterpretability: Like many Transformer-based models, UniGTE may suffer from limited interpretability, especially in understanding how graph-text interactions are formed.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate that the reviewer understood and recognized the contribution of the work. We have addressed the questions as follows:\n> Weakness 1:  \"Limited Modal Diversity ...\", and Question 4: \"Generalization to Unseen Modalities or Tasks ...\"\n\nOur research primarily focuses on leveraging large language models to address challenges in graph machine learning. This direction has recently attracted significant attention, with numerous related works exploring similar objectives [1, 2, 3, 4]. Our goal is to further enhance generalization across tasks and domains based on this emerging paradigm.\nWhile our current implementation targets graph and text modalities, we believe the proposed UniGTE framework is readily extensible to additional modalities such as images. For instance, it is feasible to adopt a multimodal LLM as the base model and incorporate our alignment and interaction mechanisms within that broader setting. This is indeed a promising and exciting research direction, and we plan to explore it further in our future work.\n\n> Weakness 2: \"Scalability Concerns ...\"\n\nWe apologize for the lack of detailed analysis on computational efficiency and scalability in the original submission. Our method mitigates scalability issues by adopting a k-hop subgraph sampling strategy to obtain graph context. This ensures that even when handling large-scale graphs, the computational overhead remains manageable. Compared to approaches that use LLMs alone for graph tasks, our design significantly improves efficiency. Specifically, when relying solely on LLMs, the model must represent all neighbors and subgraph structures as textual input, which leads to long and redundant prompts. In contrast, UniGTE encodes each node into a compact token representation via a pretrained language model (Sec. 2.2.1) and captures structural dependencies through our structure-aware graph-text attention mechanism (Sec. 2.2.2), without requiring explicit textual enumeration of neighbors or graph topology. This not only reduces the input length substantially but also overcomes the limitations of LLMs in understanding implicit graph structures from text alone [5], thereby making our method more computationally efficient.\n\nTo further address the reviewer’s concern, we have conducted a comparison between LLM and UniGTE under a zero-shot setting using equivalent graph information. The results are summarized in the tables below, where the first table reports inference time and the second table presents task-specific performance metrics, showing that UniGTE achieves significantly faster inference speed while maintaining superior accuracy. \n\n|Sec / sample (s)|Pubmed|Cora|PCBA|Photo_link|WikiCS|Instagram|Lipo|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|LLM|0.8|2.6|3.2|1.7|0.7|3.3|10.2|\n|UniGTE|0.5|0.6|1.0|1.1|0.3|0.3|2.0|\n|Improvement|37.5%|78.7%|70.3%|34.1%|50.8%|91.7%|80.5%|\n\n|Task Metric|Pubmed|Cora|PCBA|Photo_link|WikiCS|Instagram|Lipo|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|LLM|0.721|0.155|0.497|0.576|0.290|0.391|11.22|\n|UniGTE|0.870|0.215|0.541|0.732|0.680|0.601|1.03|\n|Improvement|20.7%|38.7%|8.9%|27.1%|134.5%|53.7%|90.8%|\n\n> Weakness 3: \"Ablation Study Scope ... \"\n\nWe thank the reviewer for the constructive suggestion. Following the same experimental setup as in the original submission, we have conducted additional ablation studies to further analyze the contributions of individual graph components.\n\nIn particular, w/o NF and w/o EF denote the removal of node features and edge features, respectively. As in the ablation study in the original submission, we report results aggregated at both the domain level and task level. The results are shown in the table below.\nAs can be observed, removing either node or edge features leads to a decline in zero-shot performance, highlighting the importance of both components. Moreover, the degradation is more pronounced when node features are removed, indicating that node-level semantic information plays a more critical role in our model.\n\n|Setting|Node Cls|Link Pred|Graph Cls|Graph Reg (MAE)|Citation|E-Com|Web|Social|Molecular|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|w/o EF|0.528|0.645|0.479|4.320|0.528|0.450|0.665|0.538|0.479|\n|w/o NF|0.431|0.464|0.477|7.570|0.460|0.177|0.644|0.550|0.477|\n|UniGTE|0.549|0.727|0.525|4.250|0.543|0.484|0.680|0.556|0.525|\n\n> Weakness 4: \"Interpretability...\"\n\nWe apologize for any confusion caused and thank the reviewer for raising this important point. Regarding how graph-text interactions are formed, our model incorporates a structure-aware graph-text attention mechanism that explicitly facilitates the interplay between graph and textual modalities. This design allows not only message passing among graph nodes, but also enables nodes to attend to task-relevant textual signals during the aggregation process. In other words, graph nodes can dynamically adjust their representation by incorporating context from the accompanying text, which improves the model’s ability to generalize across different domains and tasks. This interaction is not static but shaped by task-specific cues from the text, resulting in more informed and adaptive representation learning. The effectiveness of this mechanism is further validated through our ablation studies, which show significant performance drops when components responsible for this interaction are removed.\n\n> Question 1: \"Alignment Strategy Clarification ...\"\n\nWe apologize for any confusion caused, and thank the reviewer for this insightful question. UniGTE ensures semantic alignment between graph and text representations through the following mechanisms:\n1. Our encoder employs a structure-aware graph-text attention mechanism that enables direct interaction between graph and textual inputs. This joint attention mechanism allows both modalities to influence each other’s representations during encoding, facilitating meaningful semantic alignment.\n2. We introduce learnable alignment tokens that act as a shared latent space for graph and text inputs. These tokens serve as cross-modal aggregation points that encode the fused semantics of both modalities [6]. \n3. In the decoder, task-specific predictions are made based on the output representations of these alignment tokens. This further enforces semantic consistency and alignment across modalities.\n\nRegarding the reviewer’s suggestion on co-attention mechanisms — this is indeed an excellent point. In fact, our structure-aware graph-text attention module is a form of co-attention mechanism, designed specifically to incorporate structural biases from the graph while attending over textual inputs. This module achieves more fine-grained and task-relevant graph-text interaction.\n\nFinally, the strong zero-shot performance across multiple domains and tasks empirically demonstrates the effectiveness of our alignment strategy.\n\n> Question 2: \"Graph Variability Handling ...\"\n\nWe thank the reviewer for this valuable question. To handle variations in graph size, sparsity, and topology, UniGTE extracts an n-hop subgraph centered on the target node or edge. This is a common and well-established approach in prior work [1, 2, 3]. In our current implementation, the number of hops and the maximum number of neighbors sampled at each hop are fixed, following standard settings in existing literature. We acknowledge that dynamically adjusting these hyperparameters based on the characteristics of each input graph—such as its size, density, or structural complexity—could potentially lead to improved performance and robustness. Exploring such adaptive strategies for subgraph extraction is an interesting direction, and we plan to investigate it in future work.\n\n> Question 3: \"Modality Imbalance ...\"\n\nThis is a very insightful question—thank you for raising it. Prior studies [5] have shown that LLMs inherently struggle to capture structural information from graphs and tend to rely predominantly on textual features for inference. In such settings, explicitly incorporating graph structures often fails to provide additional benefits. However, our approach addresses this limitation by introducing structure-aware graph-text attention, which explicitly encodes graph structure alongside text. To evaluate whether our model exhibits modality imbalance, we conducted an ablation study in the original submission comparing UniGTE with a variant that removes graph information (denoted as w/o AT). This variant effectively removes graph structural information from the reasoning process. As shown in our results, incorporating graph structure yields substantial performance improvements across domains and tasks, suggesting that our model not only utilizes graph structure effectively but also benefits significantly from its integration. This demonstrates that UniGTE does not suffer from strong modality bias and can leverage both modalities in a complementary manner.\n\nThank you again for reviewing our manuscript. We sincerely thank the reviewer for all valuable suggestions, and we will include a detailed analysis and the corresponding experimental results in the revised submission. Please kindly let us know if you have any follow-up questions. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.\n\n[1] Tang, J., et al. GraphGPT: Graph Instruction Tuning for Large Language Models. SIGIR, 2024.\n\n[2] Chen, R., et al. LLAGA: Large Language and Graph Assistant. ICML, 2024.\n\n[3] Wang, D., et al. LLMs as Zero-Shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings. NeurIPS, 2024.\n\n[4] Kong, L., et al. GOFA: A Generative One-for-All Model for Joint Graph Language Modeling. ICLR, 2025.\n\n[5] Jin Huang., et al. Can llms effectively leverage graph structural information: when and why. arXiv preprint, 2023. \n\n[6] Ge, Tao, et al. In-context autoencoder for context compression in a large language model. ICLR, 2024."}, {"review_id": "1sqcgsUfQE", "reviewer": "Reviewer_rHZg", "summary": "This work fundamentally advances how graphs and language interact in foundation models. A Unified Encoder-Decoder framework UniGTTE is proposed in the paper. Unlike existing approaches that either (a) serialize graphs losing structure, or (b) separately train GNN+LLM components, UniGTE's end-to-end instruction tuning enables structure-preserving zero-shot adaptation. The auxiliary reconstruction objective further ensures semantic-structural consistency.", "questions": "1. The claimed \"efficiency\" lacks comparative metrics against baselines.\n2. The method has limited data coverage and lacks verification on dynamic graphs and industrial-grade ultra-large-scale graphs, which affects its practical application potential.\n3. The cross-modal fusion mechanism lacks interpretability.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nUniGTE innovatively achieves a tight coupling of graph structure and natural language cues through learnable alignment tokens and graph-text attention mechanisms, solving the multimodal split problem. The dynamic graph embedding generated by this framework can adapt to different task requirements and demonstrate excellent zero-shot generalization capabilities in cross-task and cross-domain scenarios, significantly outperforming existing methods. The frozen LLM decoder design reduces training parameters while providing self-supervision signals through graph cue reconstruction tasks, effectively improving representation quality. Experimental verification shows that this method achieves state-of-the-art performance on diverse tasks such as node classification, link prediction, and graph regression.\n\nWeaknesses:\nUniGTE has obvious bottlenecks in computational efficiency. The serialization of graph structures and the design of frozen decoders lead to latency issues when processing large-scale graph data. The method has limited data coverage and lacks verification of dynamic graphs and industrial-grade ultra-large-scale graphs, which affects its practical application potential. The model's interpretability is insufficient, the cross-modal mechanism of alignment tokens lacks theoretical support, and the quality of graph reconstruction lacks quantitative evaluation indicators. Although it innovatively combines graph-text alignment, some technical components (such as adapter tokens and reconstruction targets) have conceptual overlaps with existing methods, and their differentiated contributions need to be further clarified.", "was_revised": true, "final_justification": "Based on the authors' clarifications in the rebuttal, I will slightly increase my original rating.", "rebuttal": "Thank you for your detailed review and thoughtful comments! We have addressed your questions as follows:\n> The claimed \"efficiency\" lacks comparative metrics against baselines. \n\nWe thank the reviewer for pointing this out. In the original submission, we mentioned that GOFA [1] incurs high computational cost, but we did not provide concrete comparative metrics to support our claim. To address this concern, we conducted additional experiments to evaluate and compare the single-sample inference time and accuracy of UniGTE and GOFA.\n\nTo ensure a fair comparison, both models were evaluated with a batch size of 1. We measured the time taken for each sample from input encoding to output generation. The results are summarized in the table below, where the first table reports inference time and the second table presents task-specific performance metrics. UniGTE achieves significantly faster inference time per sample while also delivering higher accuracy compared to GOFA. These results further substantiate the efficiency advantage of our proposed method.\n\n|Sec/sample(s)|Pubmed|Cora|PCBA|Photo_link|WikiCS|Instagram|Lipo|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|GOFA|1.4|5.1|1.5|1.8|2.0|1.2|2.1|\n|UniGTE|0.5|0.6|1.0|1.1|0.3|0.3|2.0|\n|Improvement|63.2%|89.3%|34.5%|38.5%|83.6%|77.1%|5.7%|\n\n|Task Metric|Pubmed|Cora|PCBA|Photo_link|WikiCS|Instagram|Lipo|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|GOFA|0.614|0.039|0.500|0.504|0.613|0.367|1.36|\n|UniGTE|0.870|0.215|0.541|0.732|0.680|0.601|1.03|\n|Improvement|41.7%|451.3%|8.2%|45.2%|10.9%|63.8%|24.3%|\n\n>  UniGTE has obvious bottlenecks ... when processing large-scale graph data.\n\nWe apologize for any confusion regarding our method’s computational efficiency. In fact, compared to approaches that rely solely on LLMs for graph tasks, UniGTE achieves significantly better efficiency. LLM must serialize all neighbors and subgraph structures into text, resulting in long, verbose prompts that are computationally expensive to process. In contrast, UniGTE encodes each node into a compact token representation via a pretrained language model (Sec. 2.2.1) and models structural dependencies through our structure-aware graph-text attention (Sec. 2.2.2). This eliminates the need to explicitly describe graph topology in natural language and allows the model to better capture structural semantics—thereby improving both efficiency and effectiveness [5].\n\nTo further address the reviewer’s concern, we conducted an comparison between UniGTE and LLM using equivalent graph information, following the experimental setting of GOFA [1]. The results show that UniGTE achieves significantly faster inference speed while maintaining superior accuracy.\n\n|Sec/sample(s)|Arxiv|Pubmed|Cora|PCBA|Photo_link|WikiCS|Instagram|Lipo|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|LLM|1.3|0.8|2.6|3.2|1.7|0.7|3.3|10.2|\n|UniGTE|0.7|0.5|0.6|1.0|1.1|0.3|0.3|2.0|\n|Improvement|46.2%|37.5%|78.7%|70.3%|34.1%|50.8%|91.7%|80.5%|\n\n|Task Metric|Arxiv|Pubmed|Cora|PCBA|Photo_link|WikiCS|Instagram|Lipo|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|LLM|0.347|0.721|0.155|0.497|0.576|0.290|0.391|64.110|\n|UniGTE|0.713|0.870|0.215|0.541|0.732|0.680|0.601|9.180|\n|Improvement|105.5%|20.7%|38.7%|8.9%|27.1%|134.5%|53.7%|85.7%|\n\nRegarding scalability to large-scale graphs, we mitigate this issue by employing a k-hop subgraph sampling strategy to extract relevant local contexts. This design ensures that computational cost does not grow prohibitively with the size of the original graph. For example, as shown in our experiments on the Pubmed and Arxiv datasets, while the inference time of LLM increases by 62.5% when moving to larger graphs, UniGTE exhibits a much smaller increase of only 40%, demonstrating better scalability and efficiency.\n\n> The method has limited data coverage ... which affects its practical application potential.\n\nWe sincerely thank the reviewer for this thoughtful comment. However, we respectfully disagree with the assessment. Our work focuses on addressing zero-shot learning in graph machine learning using LLMs. To the best of our knowledge, all existing studies in this area have so far conducted evaluations exclusively on static graphs [1, 2, 3, 4, 7]. Building upon these foundations, our method is validated across a wider range of domains and task types, thereby demonstrating improved robustness and generalization. We believe this constitutes a meaningful contribution and advancement over prior work.\n\nThat said, we greatly appreciate the reviewer’s suggestion. Indeed, dynamic graphs and industrial-grade ultra-large-scale graphs are distinct and important subfields in graph machine learning. As we discussed in a previous response, our use of k-hop subgraph sampling already offers a degree of scalability to large graphs. For dynamic graphs, we believe it is possible to further extend UniGTE by incorporating techniques specifically designed for temporal or evolving structures—an exciting direction we plan to explore in future work to improve the generality and practical applicability of our framework.\n\n> The model's interpretability is insufficient ... lacks quantitative evaluation indicators. \n\nWe appreciate the reviewer’s detailed feedback and would like to clarify several points to avoid potential misunderstandings. First, regarding the comment on graph reconstruction: we apologize for the confusion. We do not perform graph reconstruction. Instead, during instruction tuning, we include a prompt loss as part of the training objective. This technique has been shown in prior studies to enhance the effectiveness of fine-tuning for LLMs [8]. Since our method does not aim to reconstruct the original graph, we do not include quantitative evaluation indicators for graph reconstruction quality. We will make this distinction clearer in the revised version to prevent misunderstanding.\n\nSecond, as for the mechanism of alignment tokens, they can be viewed as a form of joint compressed representation of the graph and text modalities [6]. These learnable tokens absorb information from both graph and text inputs during encoding and are then used in the decoder to guide inference. This design helps the model effectively integrate graph structure and textual semantics. While empirical evidence in our ablation study supports their utility.\n\n> Although it innovatively combines ... differentiated contributions need to be further clarified.\n\nWe sincerely thank the reviewer for acknowledging the novelty of our work and for raising this important question. Our method makes the following key technical contributions:\n1. Structure-Aware Graph-Text Attention: To jointly process graph and text inputs, we propose a unified attention mechanism that incorporates graph structure into the attention computation. This structure-aware design enables the model to capture both semantic and topological dependencies across modalities.\n2. Task-Specific Information Integration: We further introduce task-specific signals that dynamically influence the message-passing process within the graph. This helps the model adapt to a wide variety of domains and task types.\n3. Alignment Tokens for Cross-Modal Bridging: To enhance generalization and zero-shot capabilities, we design a set of learnable alignment tokens that serve as a semantic bridge between graph and text modalities. These tokens absorb information from both inputs during encoding, and their hidden states are subsequently used by the decoder to guide task-specific inference.\n\nRegarding the reviewer’s concern about overlaps with adapter tokens and reconstruction targets, there may be a misunderstanding: Our alignment tokens are not equivalent to adapter tokens. Instead of functioning as modular add-ons to inject task-specific parameters, our alignment tokens serve as shared latent representations that jointly encode graph and text information and guide the decoder accordingly by their hidden states. To our knowledge, this use of alignment tokens for graph-text integration is novel. As for reconstruction targets, we apologize for any confusion caused by our terminology. As clarified in a previous response, we do not perform graph reconstruction. The so-called \"prompt reconstruction\" refers to the addition of a prompt loss during instruction tuning. This technique is not used for aligning graph and text in prior work to our knowledge.\n\nIn summary, the key methodological components of our approach are, to the best of our knowledge, introduced for the first time in this context. The strong experimental results further demonstrate their effectiveness and contribution to advancing graph-text alignment.\n\nThank you again for reviewing our manuscript. We have tried our best to address your questions, and have revised our paper based on your questions. We believe that our revisions adequately address your concerns. **We would greatly appreciate it if you could consider raising your score further**. Please kindly let us know if you have any follow-up questions and we are prepared to provide any additional information that may be helpful.\n\n[1] Kong, L., et al. GOFA: A Generative One-for-All Model for Joint Graph Language Modeling. ICLR, 2025.\n\n[2] Chen, R., et al. LLAGA: Large Language and Graph Assistant. ICML, 2024.\n\n[3] Wang, D., et al. LLMs as Zero-Shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings. NeurIPS, 2024.\n\n[4]Tang, J., et al. GraphGPT: Graph Instruction Tuning for Large Language Models. SIGIR, 2024.\n\n[5] Jin Huang., et al. Can llms effectively leverage graph structural information: when and why. arXiv preprint, 2023. \n\n[6] Ge, Tao, et al. In-context autoencoder for context compression in a large language model. ICLR, 2024.\n\n[7] He Y, et al. UniGraph: Learning a Cross-Domain Graph Foundation Model From Natural Language. arXiv preprint. 2024.\n\n[8] Huerta-Enochian M, et al. Instruction Fine-Tuning: Does Prompt Loss Matter?. arXiv preprint. 2024."}, {"review_id": "Xz8Hgq9RXA", "reviewer": "Reviewer_Tj2S", "summary": "This paper proposes UniGTE, which is an encoder-decoder framework, simultaneously aware of graph and text modalities in a unified architecture, particular focus on transferable/ zero-shot graph reasoning scenarios.", "questions": "1. Please provide more results on SFT settings.\n\n2. It is well to directly show the models' basic graph understanding performance (as well as the transferable performance).", "limitations": "No. The architecture should also consider the fundamental adaptability to the SFT setting, not only for zero-shot, to ensure its better usability.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\n1. This paper is well-written and easy to follow.\n\n2. The statement of introduction and motivation makes sense. It is time to advance the graph-text unified architecture.\n\n3. Experiment results demonstrate that the proposed methods can promote the zero-shot and transferable graph reasoning ability.\n\nWeaknesses:\n\n1. One core limitation is that, as a fundamental architecture/framework work, this paper only focuses on zero-shot performance, which is not common for such types of work. A unified framework can do well on zero-shot cases, but it cannot only work for zero-shot. It is also necessary that also present more promising results in fundamental tasks under SFT settings.\n\n2. Though the work has presented many downstream tasks to show its effectiveness. It lacks intrinsic/fundamental graph understanding and reasoning abilities for analysis under basic graph tasks. e.g., zero-shot question-answering in some graph analysis/algorithm problem, like cycle and connectivity detection, shortest path, hamilton path, topological sort, like performed by other graph reasoning works like NLGraph, GITA, etc\n\n[1] NLGraph, Can language models solve graph problems in natural language?, NeurIPS 2023\n\n[2] GITA, GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning, NeurIPS 2024.", "was_revised": true, "final_justification": "My main concerns are the clarity of the boundary of this method on the basic graph capabilities. It is not included in the original manuscript; however, it is significant to such work for generalization. Because:  1) I wonder whether this paper is targeted at providing a general solution for graph learning with unified graph/text inputs. If so, understanding the graph structure and computing simple heuristics on that is a fundamental capability that should be contained within the intrinsic model itself for such work. If such basic abilities are absent, it is worth doubting how this method can generalize to more boarder structured real-world data. In my opinion, this work should be more ambitious about generalization, which should be exactly its most core value among the same type of works. 2) To put it more simply, the bare graph structure is the most widespread in the real world. Any downstream graph task with more complex requirements explicitly includes G = {V, E} at least, and therefore the abilities on these pure graph topology/algorithm/heuristics are atom-like small stone to support more complex requirements in graph learning, and have strong potential for achieving transfer and generalization to general graph tasks according to their highest abstract level. Considering the title includes a big term, \"Generalization across Graph Tasks and Domains\", these most fundamental graph tasks should certainly be taken into consideration (They are not hard and easy to test, even only need synthetic data).\n\nAccording to the author's additional results, they addressed part of the concerns except for the additional experiment's robustness. However, considering the rebuttal time limit, I raised the score.", "rebuttal": "Thank you for your detailed review and thoughtful comments. We have addressed your questions as follows:\n> One core limitation is that, as a fundamental architecture/framework work, this paper only focuses on zero-shot performance, which is not common for such types of work. A unified framework can do well on zero-shot cases, but it cannot only work for zero-shot. It is also necessary that also present more promising results in fundamental tasks under SFT settings. / Please provide more results on SFT settings.\n\nWe thank the reviewer for this suggestion. However, we believe there may be a misunderstanding regarding the scope and objective of our research. Our work is motivated by the challenge of zero-shot learning in graph machine learning, with a particular focus on improving generalization via large language models. Thus, the core contribution of this paper lies in enabling cross-task and cross-domain generalization through a unified graph-text framework, rather than pursuing state-of-the-art results on fully supervised benchmarks. This research objective is well-aligned with several recent studies [1, 2], which similarly prioritize generalization and zero-shot evaluation over task-specific fine-tuning performance.\n\nHowever, we fully agree with the reviewer that including SFT results is important to ensure that the model remains competitive when fine-tuned. To that end, we have conducted additional experiments under the SFT setting and compared UniGTE with relevant baselines that use LLMs as predictors. The results are presented below (**bold** indicates the best result, and *italic* indicates the second-best).\n\nAs shown, UniGTE achieves competitive performance and yields strong results on most datasets, even surpassing state-of-the-art baselines on several of them. This highlights its promising capability under SFT settings. These results demonstrate that although our model is originally designed for zero-shot scenarios, it also performs robustly under full supervision, without significant degradation compared to existing approaches.\n\nWe appreciate the reviewer’s input and have added these SFT results to the revised submission to provide a more complete evaluation of our framework.\n\n| |Arxiv|Children|Computer|Arxiv_link|Children_link|Computer_link|FB15K237|\n|:----:|:---:|:------:|:------:|:---------:|:-----------:|:------------:|:------:|\n|GraphGPT-std|0.625|-|-|-|-|-|-|\n|LLaGA|*0.711*|*0.498*|*0.579*|*0.901*|**0.837**|**0.793**|0.870|\n|TEA-GLM|0.661|*0.477*|0.549|0.891|0.767|0.738|**0.910**|\n|GOFA|0.659|0.190|0.514|0.772|0.505|0.563|0.731|\n|UniGTE|**0.713**|0.431|**0.589**|**0.913**|*0.813*|*0.765*|*0.901*|\n\n> Though the work has presented many downstream tasks to show its effectiveness. It lacks intrinsic/fundamental graph understanding and reasoning abilities for analysis under basic graph tasks. \n\nWe thank the reviewer for the constructive feedback. However, we respectfully believe there may be a misalignment between our research focus and the concern raised. Our work addresses core problems in graph machine learning, with a particular focus on predictive tasks over graphs. This focus is consistent with recent related works in this domain [1, 2, 3, 4, 5]. In contrast, the tasks mentioned by the reviewer fall under a different line of research, often studied in the context of graph reasoning benchmarks. Our goal is to enhance zero-shot learning on graph-based predictive tasks. To that end, we have validated UniGTE across a wider range of domains and task types than prior work, demonstrating its strong generalization capability. We believe this sufficiently supports the contribution of our framework within the graph machine learning field.\n\nThat said, we acknowledge that graph reasoning is a valuable and challenging research direction. As noted by the reviewer, incorporating such tasks could further broaden the scope of our framework. We plan to explore this in future work by including reasoning-oriented data during training to enhance the model’s capacity for fundamental graph reasoning.\n\nThank you again for reviewing our manuscript. We have tried our best to address your questions, and have revised our paper based on your questions. We believe that our revisions adequately address your concerns. **We would greatly appreciate it if you could consider raising your score further**. Please kindly let us know if you have any follow-up questions and we are prepared to provide any additional information that may be helpful.\n\n[1] Kong, L., et al. GOFA: A Generative One-for-All Model for Joint Graph Language Modeling. ICLR, 2025.\n\n[2] Wang, D., et al. LLMs as Zero-Shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings. NeurIPS, 2024.\n\n[3] Chen, R., et al. LLaGA: Large Language and Graph Assistant. ICML, 2024.\n\n[4]Tang, J., et al. GraphGPT: Graph Instruction Tuning for Large Language Models. ACM SIGIR, 2024.\n\n[5] He Y, et al. UniGraph: Learning a Cross-Domain Graph Foundation Model From Natural Language. arXiv preprint arXiv:2402.13630. 2024."}, {"review_id": "99fiBRHhSG", "reviewer": "Reviewer_X65Y", "summary": "This paper proposes an instruction-tuned encoder-decoder framework for zero-shot generalization across graphs from different domains without any task-specific fine-tuning. The  proposed method is equipped with structure-aware attention mechanism to preserve the permutation invariance and graph structural information.", "questions": "Q1: I am confused about adding the additive biases in the cross-attention mechanism in equation 2. The output of equation 2 only computes the cross attention weight for different tokens, while the original positional encoding is added to the token embedding before the self-attention mechanism. How does it encode any node-permutation invariance information and structural awareness information as attention weight only adjusts the importance of different token embeddings?\n\nQ2: In the introduction (line 42), the authors mention that one drawback of GOFA is its high computational cost. The proposed method also includes different types of embeddings as the input of the encoder. Then, what's the time complexity of the proposed method?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The presentation of this paper is good and the paper is well-written.\n2. The experimental results show that the proposed method outperforms most baseline methods across 9 datasets in three types of tasks.\n3. The motivation of this paper is clear.\n\nWeaknesses:\n1. The intuition of including the additive biases in the attention mechanism need further justification.\n2. In the introduction (line 42), the authors mention that one drawback of GOFA is its high computational cost. The proposed method also include different types of embeddings as the input of the encoder. However, this paper does not include the time complexity of the proposed method.\n3. In the ablation study, the authors do not validate the necessity of three additive biases, which the authors claim to preserve the permutation invariance and structural awareness.\n4. There is a misuse of symbol in the paper. In line 106, $n$ is denoted as the number of hop, while in line 109, $n$ is denoted as number of nodes, which is inconsistent. n-hop neighbors of a node are the nodes located at a shortest distance of n from that node, rather than the number of nodes in the subgraph.", "was_revised": true, "final_justification": "The presentation of this paper is good and the paper is well-written with clear motivation. The experimental results show that the proposed method outperforms most baseline methods across 9 datasets in three types of tasks. During the rebuttal, the authors have address all my concerns. Thus, I maintain my current score.", "rebuttal": "We sincerely appreciate that the reviewer understood and recognized the contribution of the work! We have addressed the questions as follows:\n> The intuition of including the additive biases in the attention mechanism need further justification. / In the ablation study, the authors do not validate the necessity of three additive biases, which the authors claim to preserve the permutation invariance and structural awareness.\n\nWe thank the reviewer for raising this important concern. The three additive biases in our attention mechanism are designed to introduce structural priors and enhance graph-text interaction. Specifically:\n1. $b_{ij}^{\\text{PE}}$ encodes the relative position between nodes via their shortest path distance, allowing the model to incorporate graph structural awareness into attention weights.\n2. $b_{ij}^{\\text{Edge}}$ accounts for the heterogeneous edge types commonly found in real-world graphs. It introduces edge-specific biases to better model diverse relationships between nodes.\n3. $b_{ij}^{\\text{M}}$ enables bidirectional attention among graph nodes and allows graph nodes to attend to textual tokens, thus making node-level message passing adaptive to task-specific textual cues.\n\nTo further address the reviewer’s concern, we conducted an ablation study to evaluate the necessity of each additive bias. In particular, w/o PE denote the remove of $b_{ij}^{\\text{PE}}$ , w/o Edge denote the remove of $b_{ij}^{\\text{Edge}}$, w/o Mask denote the remove of $b_{ij}^{\\text{M}}$ by disabling our custom attention mask and reverting to the LLM’s original causal mask, effectively disabling bidirectional attention and graph-text cross-attention.\n\nWe report results following the same evaluation protocol as in the original submission, aggregating performance at both the domain level and task level. The results clearly show that removing any of the three bias terms degrades performance. In particular, removing the mask bias ($b_{ij}^{\\text{M}}$) causes the most significant drop. This confirms its critical role in enabling effective message passing among graph nodes and facilitating task-specific alignment via textual signals.\n\n|Setting|Node Cls|Link Pred|Graph Cls|Graph Reg (MAE)|Citation|E-Com|Web|Social|Molecular|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|w/o PE|0.531|0.642|0.494|5.267|0.518|0.469|0.659|0.542|0.494|\n|w/o Edge|0.528|0.645|0.479|4.320|0.528|0.450|0.665|0.538|0.479|\n|w/o Mask|0.485|0.525|0.488|5.190|0.519|0.319|0.629|0.545|0.488|\n|UniGTE|0.549|0.727|0.525|4.250|0.543|0.484|0.680|0.556|0.525|\n\n> Weakness 2 and Question 2 about computational cost comparing with GOFA\n\nWe thank the reviewer for pointing this out. We agree that a comparative time complexity analysis is important. We adopt the same notation used in the GOFA paper: suppose a graph contains $V$ nodes, $E$ edges, and each node is represented by $k$ tokens on average. In GOFA, each node is expanded into $k$ tokens ($k \\gg 1$), and the self-attention is restricted within each node's token group. Thus, the per-layer time complexity is:  $\\mathcal{O}(k \\cdot k \\cdot V) = \\mathcal{O}(V k^2)$,  which is consistent with the complexity reported in the original GOFA paper. In contrast, UniGTE represents each node using a single token and performs cross-node attention directly. Therefore, the complexity becomes:  $\\mathcal{O}((V \\cdot 1)^2) = \\mathcal{O}(V^2)$. This is considerably lower, especially when $k = 128$ or higher as required in GOFA for satisfactory performance. Furthermore, our method leverages $k$-hop subgraph sampling, which keeps the number of nodes $V$ relatively small, making $O(V) \\ll O(k^2)$ and thus $O(V^2) \\ll O(Vk^2)$.\n\nIn addition to the Transformer layers of the LLM itself, GOFA also introduces external GNN layers. This component is not accounted for in its original complexity analysis. Assuming the average node degree is $d$, the GNN layer introduces an additional cost of $\\mathcal{O}(V \\cdot d \\cdot k)$. Combining all these considerations, our method provides a clear computational advantage over GOFA.\n\nTo further validate this, we compare per-sample inference time and accuracy under the zero-shot setting between GOFA and UniGTE. As shown in the table below, UniGTE achieves faster inference and better performance, demonstrating the efficiency and effectiveness of our design.\n\n|Sec/sample(s)|Pubmed|Cora|PCBA|Photo_link|WikiCS|Instagram|Lipo|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|GOFA|1.4|5.1|1.5|1.8|2.0|1.2|2.1|\n|UniGTE|0.5|0.6|1.0|1.1|0.3|0.3|2.0|\n|Improvement|63.2%|89.3%|34.5%|38.5%|83.6%|77.1%|5.7%|\n\n|Task Metric|Pubmed|Cora|PCBA|Photo_link|WikiCS|Instagram|Lipo|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|GOFA|0.614|0.039|0.500|0.504|0.613|0.367|1.36|\n|UniGTE|0.870|0.215|0.541|0.732|0.680|0.601|1.03|\n|Improvement|41.7%|451.3%|8.2%|45.2%|10.9%|63.8%|24.3%|\n\n> There is a misuse of symbol in the paper. In line 106, n is denoted as the number of hop, while in line 109, n is denoted as number of nodes, which is inconsistent. n-hop neighbors of a node are the nodes located at a shortest distance of n from that node, rather than the number of nodes in the subgraph.\n\nWe sincerely thank the reviewer for carefully pointing out this inconsistency. We will correct this notation and revise the corresponding description. Thank you again for highlighting this issue. We will make sure it is addressed in the revised version.\n\n> I am confused about adding the additive biases in the cross-attention mechanism in equation 2. The output of equation 2 only computes the cross attention weight for different tokens, while the original positional encoding is added to the token embedding before the self-attention mechanism. How does it encode any node-permutation invariance information and structural awareness information as attention weight only adjusts the importance of different token embeddings?\n\nWe sincerely thank the reviewer for raising this critical question. We achieve node-permutation invariance and structural awareness through the additive bias in Equation (2) and the \"Cross-modal RoPE\" design described in Section 2.2.2 (lines 134–137), particularly via the use of a shared position index.\n\nConcretely, in Equation (2), the terms $i$ and $j$ represent the position indices of tokens within the full input sequence. For all graph node tokens, we assign the same position index. In contrast, text tokens retain their natural positions within the sentence. As a result, for any two graph node tokens, the relative positional encoding term $R(i−j) = R(0) = 1$, meaning no positional bias is introduced based on the order of graph nodes. This ensures that the model’s output is invariant to the input order of graph nodes, addressing the limitation identified in prior work [1, 2].\n\nAs for structural awareness, this is introduced through the bias term $b_{ij}^{\\text{PE}}$ in Equation (2), which is computed based on the shortest path distance between nodes $i$ and $j$ in the graph. This term is part of the additive bias and only applies to node-node token pairs, allowing attention weights to be modulated based on graph structure.\n\nIn summary, while the base attention weights reflect token-level interactions, the additive bias encodes structure-dependent adjustments, and the shared position index design enforces permutation invariance—together enabling UniGTE to model graph topology effectively within the cross-modal attention mechanism.\n\nThank you again for reviewing our manuscript. We have tried our best to address your questions, and have revised our paper based on your questions. We believe that our revisions adequately address your concerns. **We would greatly appreciate it if you could consider raising your score further**. Please kindly let us know if you have any follow-up questions and we are prepared to provide any additional information that may be helpful.\n\n[1] Kong, L., et al. GOFA: A Generative One-for-All Model for Joint Graph Language Modeling. ICLR, 2025.\n\n[2] Xixi Wu., et al. Can graph learning improve planning in LLM-based agents? NeurIPS, 2024."}]

Author rebuttal:

Meta-review: This paper proposes a novel and timely contribution at the intersection of graph learning and large language models. An instruction-tuned encoder augments a pretrained LLM with learnable alignment tokens and a structure-aware graph–text attention mechanism. The encoder processes both a tokenized graph and a task prompt with cross-modal attention. A variant of RoPE ensures permutation invariance over node order. A frozen LLM decoder then generates both the task answer and a natural-language reconstruction of the graph, with the reconstruction serving as a regularization signal. The reviewers generally agree that the work is technically sound, well-written, and demonstrates strong empirical results. The instruction-tuned encoder–decoder design with alignment tokens and structure-aware cross-modal attention represents a clear methodological innovation.
Decision: Accept (poster)