TAGA: Text-Attributed Graph Self-Supervised Learning by Synergizing Graph and Text Mutual Transformations

## Abstract

Abstract Text-Attributed Graphs (TAGs) enhance graph structures with natural language descriptions, enabling detailed representation of data and their relationships across a broad spectrum of real-world scenarios. Despite the potential for deeper insights, existing TAG representation learning primarily relies on supervised methods, necessitating extensive labeled data and limiting applicability across diverse contexts. This paper introduces a new self-supervised learning framework, T ext- A nd- G raph Multi-View A lignment ( TAGA ), which overcomes these constraints by integrating TAGs’ structural and semantic dimensions. TAGA constructus two complementary views: Text-of-Graph view, which organizes node texts into structured documents based on graph topology, and the Graph-of-Text view, which converts textual nodes and connections into graph data. By aligning representations from both views, TAGA captures joint textual and structural information. In addition, a novel structure-preserving random walk algorithm is proposed for efficient training on large-sized TAGs. Our framework demonstrates strong performance in zero-shot and few-shot scenarios across eight real-world datasets.

## 1 Introduction

Text-Attributed Graphs (TAGs) are text documents that are connected in graph structures, allowing for deeper analysis and interpretation of complex relationships Zhang et al. (); Jin et al. (, ). TAGs are prevalently used in numerous real-world applications, such as social networks Paranyushkin (); Myers et al. (), citation networks Liu et al. (), and recommendation systems Wu et al. (); Huang et al. ().
TAGs encompass textual content in both nodes and edges that elucidate the meaning of individual documents and who they are semantically correlated with. For instance, scientific article network is a type of TAGs that store the texts of research papers and how they are citing, criticizing, and summarizing each other in paragraphs. As exemplified in Figure 1(a), to elicit knowledge like “the first law proposed in Paper A is a special case of Paper B’s Theorem 1 when it is under macro scale and low velocity” from scientific article network, it requires jointly considering semantics, topology, and their entanglement in the TAG.

Representation learning of TAGs is a promising, yet open research area that starts to attract fast-increasing attention Ye et al. (); Wang et al. (); Chen et al. (); Hu et al. (); Huang et al. (); Fatemi et al. (); Tang et al. (); Li et al. (); He et al. (). Existing works typically use Pre-trained Language Models (PLMs) to generate textual embeddings from node texts, which are then processed by Graph Neural Networks (GNNs) to produce embedding of the TAG. However, these methods predominantly rely on supervised learning paradigms, which require extensively labeled data that is often unavailable in real-world scenarios. Moreover, the reliance on supervised tasks means that models are usually optimized for specific tasks and domains reflected in the training dataset, which significantly constrains their applicability to new domains or broader tasks. This limitation undermines the unique advantage of TAGs to leverage their universal linguistic attributes effectively. Although there are some graph pre-training models Hou et al. (); Veličković et al. (); You et al. (); Li et al. () operate in an unsupervised manner, they often focus on either graph topology or node features independently, neglecting the crucial interplay between textual semantics and structural information inherent in TAGs.

Therefore, there is a pressing need for a method that comprehensively addresses the unique nature of TAGs, seamlessly integrating both their structural and semantic dimensions within a unified unsupervised framework. This presents a significant research challenge with several substantial hurdles to overcome. Primarily, developing a representation that can simultaneously leverage the textual semantic content, the graph structure, and their complex interplay presents significant difficulties. The scarcity of labeled training data further exacerbates this issue, making traditional supervised approaches impractical and necessitating innovative unsupervised strategies. Furthermore, the computational demands of such representation learning are substantial. The integration of large PLMs for textual corpus processing to be considered in TAGs creates a significant computational burden.

In order to address the aforementioned challenges, this paper proposes a new self-supervised learning framework named Text-And-Graph Multi-View Alignment (TAGA). TAGA jointly preserves rich semantic information, topology information, and their interplay by aligning representations of TAGs from two complementary views: the Text-of-Graph view and the Graph-of-Text view.

Figure: Figure 1: Illustration of the two distinct views of TAGs: (left) Graph-of-Text and (right) Text-of-Graph. Graph-of-Text view constructs a graph-structured data over the individual text corpora, while Text-of-Graph view organizes the text node and their connection description in a hierarchical layout document. These two views can be mutually transformed to each other.
Refer to caption: x1.png

As illustrated in Figure 1, these two views offer different representation formats of a TAG yet contain equivalent information. Specifically, the Text-of-Graph view organizes node texts into a structured textual document according to the TAG’s topology. As exemplified in Figure 1(b), structured textual documents are universal ways to represent the relations among different text pieces in large corpus, especially in books, long articles, web files, etc. Here we propose a novel $\mathrm{Graph2Text}$ encoding module to automatically transfer a TAG to a structured textual document, which is readily to be processed by language models. Conversely, the Graph-of-Text view transforms textual nodes and topology into graph-structured data, which is then processed by a graph representation learning module (e.g. graph neural network). By aligning the representations learned from these two views, we encourage the learned representation to capture both textual and structural information, resulting in a unified, comprehensive representation of the TAG. Furthermore, to accelerate the training process, we propose a novel structure-preserving random walk algorithm. Finally, we demonstrate the strength of our proposed representation learning framework through extensive experiments on eight real-world datasets in zero-shot and few-shot prediction scenarios.

## 2 Related Works

### 2.1 Unsupervised Graph Pre-Train Methods

Existing unsupervised graph pre-training methods can be categorized into several categories based on their objectives and architectures. Graph autoencoder methods, graph autoencoder methods Kipf and Welling (); Hou et al. () convert node and edge features into low-dimensional embeddings, which are then used to reconstruct the original graph data. Contrastive learning approaches, like DGI Veličković et al. (), GraphCL You et al. (), GRACE Zhu et al. (), and S^3-CL Ding et al. (), generate perturbed graph pairs by altering structural features, such as adding or removing nodes and edges or masking features, aiming to align the embeddings of these modified graphs closer in the embedding space. However, these methods often produce domain-specific embeddings with limited generalization ability across different domains, reducing their effectiveness in data-scarce or label-limited scenarios.

Recent developments have also seen efforts Wen and Fang (); Tang et al. (); Li et al. () in aligning graph representations with textual representations. For instance, G2P2 Wen and Fang () employs contrastive learning to align GNN representations with text encoder outputs by averaging individual node text embeddings across various neighborhood hops during its pre-training phase. However, these methods often simplify the treatment of textual encoder embeddings for neighborhoods by averaging the embeddings of individual nodes. Similarly, GRENADE Li et al. () implements a dual-level alignment strategy. This approach not only aligns GNN and text encoder embeddings but also encourages embeddings of connected node pairs to exhibit similarity. This approach overlooks the underlying interactions within neighborhoods, leading to a loss of information that could be crucial for the contrastive objectives of alignment models.

### 2.2 Graph2Text Encoding Methods

Recently, research include approaches Ye et al. (); Wang et al. (); Chen et al. (); Hu et al. (); Huang et al. (); Fatemi et al. () that first transform the text-attributed graph into text sequence and then directly utilize LLMs as the predictor given the transformed text and corresponding question as input prompt. These methods typically designs text templates to explicitly describe local graph structure by stating nodes and how they are connected in plain text. For example, “The first node is …. The second node is …. …. First node connects to third node. Second node connects to …”. However, these methods do not present the structure in a natural language-speaking manner, which fails to fully leverage the pretrained capabilities of language models. This is due to the distributional shift between the transformed text from the graph and the original pretrained corpus, resulting in lower quality embeddings and high variance of performance Fatemi et al. ().

### 2.3 Efficient and Scalable Methods for Large-Size Graph Neighborhoods

Efficiency and scalability are crucial for deep graph learning, particularly when dealing with large graphs or high-order interactions. Traditional graph sampling techniques, such as node sampling Chen et al. (), edge sampling Hamilton et al. (), or subgraph sampling Zeng et al. (), aim to reduce neighborhood size. However, these methods may not be suitable for TAGs, as they can result in the loss of important hierarchical interactive connection during the random sampling process. Meanwhile, in the NLP domain, some efforts Peng et al. (); Han et al. (); Chen et al. (); Jiang et al. (); Ding et al. () have been made to address the long context issue of PLMs. These approaches typically involve compressing input tokens into latent vectors Jiang et al. () or modifying the attention mask Chen et al. (); Han et al. (); Ding et al. () to reduce significant interactions. However, these methods often fail to preserve the original structure of the input corpus and might alter the hierarchical layout.

## 3 Preliminaries

In our study, a Text-Attributed Graph (TAG) can be represented as $\mathcal{G}=\left(\mathcal{V},\mathcal{E},\mathcal{C}\right)$, where $\mathcal{V}=\{v_{1},v_{2},...,v_{N}\}$ is a set of $N$ nodes and $\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$ is the set of $M$ edges. $e_{ij}\in\mathcal{E}$ is an edge connecting nodes $v_{i}$ and $v_{j}\in\mathcal{V}$. $\mathcal{C}=\{C_{1},C_{2},\dots,C_{N}\}$ is the set of node textual features where each $C_{i}$ is the textual corpus associated with node $v_{i}\in\mathcal{V}$.

The main goal of this paper is to learn the representation $f(\mathcal{G})$ of a TAG $\mathcal{G}=\left(\mathcal{V},\mathcal{E},\mathcal{C}\right)$, which is an open research problem with several subsantial and unique challenges to be resolved. First, how the representation can jointly preserve the rich semantic information, graph information, and their interplay in TAG? Moreover, note that it is prohibitive to prepare label data so the unavailability of the training labels further troubles the representation learning. Second, the efficiency and scalability present a big challenge in representation learning of TAG because of the synergization of computational overhead of LLMs and the large corpus to be considered in the subgraph of TAG.

## 4 Methodology

Figure: Figure 2: Illustration of the proposed self-supervised learning framework. (a) Generation of different orders of Graph-of-Text views; (b) The $\mathrm{Graph2Text}$ module that transforms a Graph-of-Text view into a Graph-of-Text view; (c) The alignment module via hierarchical self-supervised learning.
Refer to caption: x2.png

To effectively and efficiently address the substantial challenges of unsupervised representation learning on Text-Attributed Graphs (TAGs), we propose a novel self-supervised learning framework called Text-And-Graph Multi-View Alignment (TAGA). Specifically, to jointly preserve both rich semantic information, topology information, and their interplay, we propose to learn and align the representations of TAG in two complementary views, namely text view and graph view. In particular, the text view is a Text-of-Graph, where the TAG’s node texts are organized according to the TAG’s topology into a textual document format, which inherently has the power to encompass logic and relational information. The graph view is a Graph-of-Text, where the TAG’s nodes and topology are turned into a graph structured data. Then the text view can be transformed by pretrained language models, which are adept at preserving textual information, while the graph view can be transformed by graph neural network, which are designed to guarantee preserving graph information. Therefore, by aligning the representations learned from these two views, we encourage the graph view’s representation to also capture textual information and the text view’s representation to also capture graph information. The above new idea is shown in Figure 2, where Figure 2(a) illustrates Graph-of-Text view while Figure 2(b) illustrates Text-of-Graph view, and their respectively transformed embeddings are aligned by our new TAG-hierarchical self-supervised learning framework, as detailed in Section 4.1. The details of our Text-of-Graph view are elaborated in Section 4.2. Finally, the acceleration of our learning process is detailed in Section 4.3.

### 4.1 Text-and-Graph Multi-View Alignment via TAG Hierarchical Self-Supervised Learning

Existing methods for learning representations in TAGs use Pre-trained Language Models (PLMs) to generate textual embeddings from node texts, which are then processed by GNNs for an aggregated TAG embedding. These methods typically require supervised labels for training, which are hard to obtain in real-world scenarios. Moreover, the resulting embeddings often lack generalization capabilities beyond their training data’s specific domain and task.

To address the challenge of unsupervised representation learning in TAGs, TAGA first constructs two complementary views of a TAG: Text-of-Graph and Graph-of-Text, detailed in Section 4.1.1. These views contain equivalent information but in different formats, allowing them to mutually supervise each other. To leverage the strengths of both views, a hierarchical self-supervised learning module, described in Section 4.1.2, aligns the embeddings from both views, effectively capturing the rich semantic and structural information within TAGs.

#### 4.1.1 Text-and-Graph Multi-View Construction

Our proposed framework TAGA first leverages two views of a TAG: Text-of-Graph (TofG) and Graph-of-Text (GofT). Each view can be defined at different neighborhood orders, allowing for a multi-order hierarchical representation that captures both the structural and semantic information within the TAG.
Specifically, a $k$-order TofG view represents a node’s $k$-hop neighborhood as a single textual corpus that encompasses all nodes and their connections within that neighborhood. This corpus is then processed by a PLM to extract semantic embeddings that capture the combined content and structure within that $k$-hop neighborhood.
In contrast, the corresponding $k$-order GofT view is constructed as a graph structure, where nodes represent lower order TofGs within the $k$-hop neighborhood. A GNN model is then applied to aggregate information from these connected lower order TofGs, capturing the overall neighborhood context. This ensures that both TofG and GofT views at the same order encode equivalent information about the neighborhood.

To illustrate, consider a node with a 3-hop neighborhood, as shown in Figure 2(a). Its 3-order TofG is constructed by transforming the entire 3-hop neighborhood as a single text corpus. Three distinct 3-order GofT views can then be created using TofGs of orders 0, 1, and 2 as nodes in the graph structure. To maintain information consistency, the number of GNN aggregation layers decreases with increasing TofG order: 3 layers for 0-order TofGs, 2 for 1-order TofGs, and 1 for 2-order TofGs. This ensures that each 3-order GofT view captures the same 3-hop neighborhood information as the 3-order TofG view, facilitating information equivalent self-supervised learning.

#### 4.1.2 Multi-View Alignment via TAG Hierarchical Self-Supervised Learning

Upon construction of both views at different orders, a hierarchical self-supervised learning module is proposed to align the embeddings from both views.
Given a TAG $\mathcal{G}$ with at most $K$-hop neighborhood size, for each node $v_{i}\in\mathcal{V}$, its $k$-hop neighborhood can be denoted as $\mathcal{N}_{k}(v_{i})$ and its corresponding $k$-order TofG view embedding can be represented as:

$$ $\begin{split}\mathbf{h}_{k}(v_{i})&=\mathrm{PLM}\left(\text{{TofG}}(v_{i};k) \right),\\ \text{{TofG}}(v_{i};k)&=\mathrm{Graph2Text}\left(v_{i}\cup\mathcal{N}(v_{i},k) \right),\end{split}$ (1) $$

where $\mathrm{PLM}$ is a pre-trained language model (e.g. BERT Devlin et al. () or LlaMA Touvron et al. ()). $\mathrm{Graph2Text}$ is an encoding template function that can transform individual nodes and edges text into a textual corpus. Meanwhile, its corresponding $k$-order GofT views embeddings can be denoted as GNN aggregated representations of lower order TofGs:

$$ $\begin{split}{\mathbf{b}}^{l}_{k}(v_{i})&=f^{(k-l)}\left(\{\mathbf{h}_{l}(v_{b })|v_{b}\in v_{i}\cup\mathcal{N}(v_{i},k-l)\}\right),\end{split}$ (2) $$

where $l$ covers from 0 to $k-1$ and $f^{(k-l)}$ denotes the GNN model with $k-l$ layers.

By aggregating $k-l$ layers of information over the connected $l$-order TofGs, the obtained $k$-order GofT embeddings cover equivalent information with the $k$-order TofG view embedding. Therefore, given all the embeddings from level $1$ to $K$, the supervision objective function can be written as:

$$ $\begin{split}\mathcal{L}_{\mathrm{positive}}=-\frac{1}{K|\mathcal{B}|}\sum_{v_ {i}\in\mathcal{B}}\sum_{k\in[1,K]}\sum_{l\in[0,k-1]}\rho\left({\mathbf{b}}^{l} _{k}(v_{i}),\mathbf{h}_{k}(v_{i})\right),\end{split}$ (3) $$

where $\mathcal{B}$ represents the minibatch and $\rho$ denotes a similarity function, such as cosine similarity.
Additionally, we include the negative samples that chosen from other nodes within the minibatch:

$$ $\mathcal{L}_{\mathrm{negative}}=\frac{1}{K|\mathcal{B}|}\sum_{v_{i},v_{j}\in \mathcal{B},v_{1}\neq v_{2}}\sum_{k\in[1,K]}\sum_{l\in[0,k-1]}\rho\left({ \mathbf{b}}^{l}_{k}(v_{i}),\mathbf{h}_{k}(v_{j})\right),$ (4) $$

Thus, the overall objective function can be denoted as:

$$ $\mathcal{L}=\mathcal{L}_{\mathrm{positive}}+\mathcal{L}_{\mathrm{negative}}$ (5) $$

Time Complexity Analysis.
Consider a TAG with a maximum $K$-hop neighborhood size, where each node has an average degree $d$ and text attribute length $L$. Assume the feature dimensionality is $F$. In the case of transformer-based PLMs, the time complexity for processing the TofG view of a node would be $O((dL)^{2}K^{2})$, due to the quadratic complexity of self-attention mechanisms with respect to input sequence length. In contrast, our method employs a GNN to aggregate information from lower-order TofGs, each of length $dL$. Assuming a GNN with constant complexity per layer, the time complexity for aggregating information from all $K$ levels of the GofT view would be $O(L^{2}dK)$. Our method achieves significantly higher efficiency than directly using PLMs for TofG views, with details available in the Appendix B.

### 4.2 Represent Text Neighborhood Information via Hierarchical Document Layout

The key to our proposed self-supervised learning framework is ensuring that the two distinct graph views (TofG and GofT) contain equivalent information. This necessitates constructing a TofG view through the $\mathrm{Graph2Text}$ module in Equation 1 that preserves all connectivity information present in the original TAG.
Existing methods Fatemi et al. (); Huang et al. (); Wen and Fang (); Tang et al. () often struggle to effectively represent the structural information of graphs in a way that is both comprehensive and natural to language model understanding. Some methods Tang et al. (); Wen and Fang () omit crucial connectivity information between nodes, while others Fatemi et al. (); Huang et al. () explicitly list all connections in a manner that is unnatural and difficult for language models to process. This discrepancy between the transformed graph text and the original pre-training corpus leads to a distributional shift, hindering the language model’s ability to generate high-quality embeddings that accurately reflect both the semantic and structural aspects of the graph.

To address this issue, we introduce a novel $\mathrm{Graph2Text}$ approach that transforms a graph neighborhood into a hierarchical text document. This hierarchical structure mirrors the original graph’s topology, ensuring that the document’s latent structure is equivalent to the graph itself. Crucially, the resulting document resembles a natural document, aligning with the distribution of majority text data used to pre-train PLMs. This alignment mitigates the distributional shift issue, allowing PLMs to generate embeddings that accurately reflect both the semantic and structural aspects of the graph.

Specifically, the structure of a node and its k-hop neighborhood can be represented as an ego graph, with the node itself as the root. This ego graph can be decomposed into a hierarchical tree backbone and a set of cross-edges, as illustrated in Figure 2(b). The reading order is established for the TofG document through a pre-order traversal of this tree structure (first visit the root, then the left subtree, then the right subtree), capturing the hierarchical relationships between nodes. To fully represent the neighborhood’s structure, we then incorporate cross-edges into the document. These cross-edges indicate connections from later sections of the document back to earlier ones, effectively mirroring the original graph’s topology within the text format.

As shown in Algorithm 1, the $k$-hop neighborhood of a target node $v$ in graph $G$ is represented as an ego-graph ${\mathcal{G}}(v,k)$. A breadth-first search (BFS) tree $\hat{\mathcal{T}}(v,k)$, rooted at $v$, provides a hierarchical structure for the document, while cross-edges (edges outside the BFS tree) are identified. A pre-order traversal of $\hat{\mathcal{T}}(v,k)$ establishes the document’s hierarchical layout, assigning each node a section number. Cross-edges are then integrated by adding references at source nodes to the sections containing their respective destination nodes, if the destination node appears earlier in the traversal. This approach ensures that the document faithfully reflects the graph’s structure.

Figure: Algorithm 1 Hierarchical Document Layout (HDL) for Graph2Text

### 4.3 Accelerating Training on Large TAGs with Structure-Preserving Random Walk

While TAGA significantly improves efficiency during inference by transferring knowledge from the PLM to a GNN model, the pre-training stage still encounters computational bottlenecks due to the quadratic complexity of transformers with respect to context length when generating TofG view embeddings. Existing graph sampling methods (e.g., node or edge dropping) can partially alleviate this issue, but at the cost of sacrificing some valuable neighborhood structure information, which is crucial for capturing the intricate relationships within TAGs.

To address this issue while preserving the structure of corpus, we propose a novel approach inspired by human reading patterns. Our method segments the hierarchical corpus into multiple related sub-corpora, mirroring how humans naturally engage with complex documents: starting with a general overview (top of the hierarchy) and delving into specific sections (sub-corpora). By navigating the corpus multiple times, focusing on different sub-corpora each time, the combined insights gained can effectively approximate the understanding achieved from processing the entire corpus.

To facilitate this behavior, we introduce a neighborhood traversal algorithm based on a random walk. This algorithm simulates a reader starting at the root node and progressing towards leaf nodes in the BFS tree, transitioning from general to specific information. Additionally, at each step, there is a probability $p$ of jumping to another node via cross-edges, imitating the non-linear navigation often observed in human reading (e.g., jumping to related topics or backtracking).
By averaging multiple random walk traversals, the generated paths can approximate the complete corpus. As detailed in Algorithm 2, each traversal begins at the root node $v$ and iteratively samples child nodes to form a path down the hierarchy. At each step, a jump to another node via cross-edges is possible with probability $p$. This traversal continues until reaching a predefined length or a leaf.

## 5 Experiments

In this section, the experimental settings are introduced first in Section 5.1, then the zero-shot and few-shot performances are presented in Section 5.2. We further present the effectiveness under transfer learning settings in Section 5.3. In addition, we measure model efficiency in Section 5.5. We verify the effectiveness of framework components through ablation studies in Section 5.4. The parameter sensitivity experiments are present in Appendix A.2 due to space limit.

### 5.1 Experimental Settings

Datasets. We evaluate on eight real-world text-attributed graph datasets across different domains. Specifically, three citation networks (Cora Yang et al. (), Pubmed Yang et al. () and Arxiv Hu et al. ()), two book networks (Children Shchur et al. () and History Shchur et al. ()), and three E-commerce networks (Computers Shchur et al. (), Photo Shchur et al. (), and Sports Yan et al. ()) are chosen as our evaluation datasets. Datasets statistics can be found in Table 1.

Comparison Methods.
We choose the textual embedding of the text corpus as the baseline, which is denoted as "PLM" in our experimental results tables. Additionally, we compare our proposed framework with four state-of-the-art unsupervised graph training methods that across different pre-train strategies. Specifically, GraphMAE Kipf and Welling () — utilizes masked autoencoder technique to predict of graph structure and node features. GraphCL You et al. () and GRACE Zhu et al. () applies various graph augmentations to generate contrastive pairs. G2P2 Wen and Fang () aligns GNN embeddings and text encoder embeddings through contrastive learning across various neighborhood hops.

Implementation Details.
We choose two different pre-trained language models (OpenAI’s text-embedding-3-small and UAE-Large-V1 Li and Li ()) to generate text embeddings for robust results. Commonly used GNN models (GCN Kipf and Welling (), GIN Hamilton et al. (), GraphSAGE Xu et al. ()) are chosen as the backbone model as the backbone model for both our method and all comparison methods. For a fair comparison, all models are required to adhere to the same GNN architecture, including the number of convolution layers and hidden dimensions. More details about hyperparameters can be found in Appendix A.1. For space considerations, further technical details regarding the implementation of zero-shot and few-shot learning can be found in Appendix B.

Table: Table 1: Performance in zero-shot and few-shot node classification for each dataset and setting. The best-performing model is highlighted in bold, and the second-best performing model is underlined.

### 5.2 Effectiveness Results

In this section, we assess the effectiveness of our proposed unsupervised representation learning framework compared to other methods under conditions of label scarcity. Our representation learning models are initially pre-trained on each TAG dataset without any supervised labels. After the pre-training phase, we evaluate the quality of the obtained node embeddings under zero-shot conditions by measuring the similarity between these embeddings and the corresponding text label embeddings. To further gauge performance in scenarios with limited labeled data, we conduct evaluations using 1, 3, 5, 10, 20, 50, and 100-shot settings. Due to space limitation, the results with text encoder UAE-Large-V1 under zero-shot and 1, 3, 5, 10, and 100-shot settings is reported in Table 1. Our acceleration method with random walk is denoted as “TAGA-rw”. The results with text-embedding-3-small and other few-shot settings can be found in Appendix A.3.

Zero-shot performance.
Table 1 presents node classification accuracy under zero-shot conditions, where our method consistently outperforms all comparison methods in seven out of eight datasets. On average, our method surpasses other graph pre-training methods by 47.84% and exceeds the second-best model by 6.78%. These findings demonstrate the enhanced ability of our pre-trained model to effectively learn representations that enable zero-shot predictions. Furthermore, compared to direct textual embeddings from the PLM, our method improves zero-shot performance by an average of 20.76%. This demonstrates our method’s capacity in integrating structural and textual information from neighborhoods over directly using the PLM.
Interestingly, our method exhibits a stronger performance advantage when dealing with data rich in textual information. Specifically, for the two citation networks (Arxiv and Cora), which possess significantly longer text attributes compared to other datasets, our method surpasses the second-best performing graph pretrained model by an average of 10.33%. This proves our method can effectively leverage the rich textual information.

Few-shot performance.
For few-shot experiments, our method consistently outperforms all comparison methods, achieving a 15.55% average improvement and surpassing the second-best model by 6.28% on average. Notably, our method exhibits a more pronounced advantage in scenarios with limited labeled data (<=5 shots), where it outperforms all other methods by an average of 19.79% and exceeds the second-best model by 7.91% on average. This underscores the effectiveness of our method, particularly in settings where few-shot learning is essential due to data labels constraints.

Remarks.
It is worth noting that for some datasets, the zero-shot performance of our method can match or even exceed few-shot predictive results, particularly when the number of training samples for few-shot learning is limited. For example, on five datasets (Arxiv, Children, Computers, Cora, and Pubmed), the zero-shot performance surpasses 1-shot performance by an average of 23.54%. Remarkably, the zero-shot performance can even be comparable to that of 5-shot. This demonstrates the strong potential of our method in scenarios where labeled data is scarce or unreachable.

### 5.3 Transfer Ability Analysis

Table: Table 2: Transfer learning results. The best-performing model is highlighted in bold.

In real-world applications, scenarios may arise where not only are labels difficult to obtain, but the data itself is also scarce. This necessitates the generalization of a pre-trained model to a data domain distinct from the pre-training data. Here we evaluate the zero-shot and few-shot performance under transfer learning settings. Specifically, the model is unsupervisedly pre-trained on the source data domain and then transferred to the target data domain. No further fine-tuning is performed for zero-shot prediction, and is fine-tuned using the limited training samples for few-shot prediction.

In Table 2, we present the performance of zero-shot and five-shot predictions across eight pairs of source and target datasets. The results demonstrate a clear advantage for our method in the zero-shot setting, where it consistently outperforms all other methods across all dataset pairs. Notably, our method achieves an average improvement of 26.5% over the second-best performing method.
In the five-shot setting, our method continues outperforming the second-best performing method by 4.53% on average. Particularly when transferring from Cora to Arxiv and Pubmed, and Children to History, our method achieves significant performance gain by 6.30% on average, demonstrating its ability to effectively leverage limited labeled data in the target domain.

### 5.4 Ablation Study

Table: Table 3: Ablation studies results of zero- and five-shot settings. Here “Full” denotes our full model.

To investigate the effectiveness of our proposed model compared to simpler heuristics, we conducted a series of ablation analyses. We began by considering textual embeddings obtained directly by applying the PLM to the Text of Graph views’ corpus at different orders. This allowed us to assess the impact of our training procedure compared to a simpler approach that relies solely on Text-of-Graph view representations.

In addition, we compare our full model with a variant, Glo-GofT, which only aligns the GNN embeddings that aggregate individual node’s text embeddings but removes all higher-order Graph-of-Text embeddings. The results of these ablation studies are presented in Table 3, which reveals that removing components of our full model generally leads to a decrease in performance. In the zero-shot setting, the full model outperforms the variant models by 2.79% to 8.49% on average, and ranges from 1.74% to 9.71% in the five-shot setting. These results underscore the contribution of each component to TAGA’s overall effectiveness.

### 5.5 Efficiency Analysis

To validate the efficiency and scalability of our proposed full method and random walk algorithm

Figure: Figure 3: (top) Comparison of the full method and the random walk algorithm in terms of the number of words, and (middle) training time, and (bottom) inference time comparison between PLM and TAGA in terms of the number of hops.
Refer to caption: x3.png

during both training and inference phases, we conduct experiments on the Cora dataset. We vary the number of hops from 0 to 5 and record the number of words in the input corpus, training time, and inference time. The results are presented in Figure 3.
As depicted in top figure, the exponential growth in input size for the full method compared to the near-linear growth of the random walk method demonstrates the our’s superior scalability in managing larger graph neighborhoods. The middle figure further demonstrates the efficiency advantage of the random walk algorithm, as its training time increases linearly with the number of hops, whereas the full method experiences a much steeper increase, becoming infeasible beyond 3 hops due to out-of-memory (OOM) errors. Finally, the bottom figure highlights the speedup achieved by our proposed method during inference compared to directly using a PLM. The inference time for our method remains linear growth trend across different hops, while the PLM-based approach suffers from rapidly increasing inference time with the hops number.

## 6 Conclusions

In this paper, we introduce TAGA, a novel self-supervised learning framework designed to address the challenges of unsupervised representation learning on TAGs. TAGA integrates both textual and structural information within TAGs by aligning representations from two complementary views: Text-of-Graph and Graph-of-Text. To enhance the preservation of structural information in the Text-of-Graph view, we propose a natural hierarchical document layout that mirrors the graph’s topology. Additionally, we introduce a structure-preserving random walk algorithm to accelerate the training process on large TAGs. Extensive experiments on eight real-world datasets demonstrate TAGA’s superior performance in zero-shot and few-shot learning scenarios, showcasing its strong generalization capabilities across diverse domains.

## Appendix A Additional Experimental Results and Settings

In this section, we present additional experimental settings and results due to the space limitation of the main paper.

### A.1 Additional Implementation Settings

All experiments are conducted on a 64-bit machine with a 16GB NVIDIA GPU.
Each experiment involves running the models 20 times with different random seeds to minimize variance due to specific data splits. Accuracy is adopted as the evaluation metric for node classification tasks.
Specifically, for smaller datasets such as Cora and PubMed, we employ 3 convolution layers, while for larger datasets, we utilize 2 layers. Latent dimension is aligned with the PLM embedding dimension. During the pre-train stage, the model is trained with 40,000 steps on each dataset with minibatch size 8. The learning rate is initialized as $1e^{-3}$ and with decay rate 0.999 each 10 steps. For zero-shot predictions, we utilize the entire dataset as the test set. In the case of $k$-shot predictions, we randomly select $k$ samples from each class to form the training set, dividing the remaining data into validation and test sets at a ratio of 1:9. All models undergo finetune for 100 epochs, and testing is based on the best validation results.

### A.2 Sensitivity Analysis

Figure: Figure 4: Comparison of five-shot performance between (top) different GNN encoder choices, and (middle) varying jumping ratio, and (bottom) maximum walk length of random walks.
Refer to caption: x4.png

In this section, we investigate the sensitivity of the key hyperparameters and their impact on TAGA’s performance. Specifically, we first evaluate how different GNN backbones (GCN, GIN, and GraphSAGE) affect performance. Then we evaluate how jumping ratio ($p$) and maximum walk length ($L$) would affect random walk’s performance. The results are presented in Figure 4.
The sensitivity analysis conducted on TAGA’s performance demonstrates that the method is robust across a range of hyperparameters. Specifically, the variance in performance across different GNN backbones is 0.84%, indicating a stable behavior regardless of the backbone employed. Similarly, adjustments in the jumping ratio ($p$) and maximum walk length ($L$) exhibit 0.33% and 0.76% variance on average, which underscores that our method is not sensitive to the hyperparameters chosen.

### A.3 Additional Effectiveness Analysis

We present additional zero-shot and few-shot performance under two different text encoders UAE-Large-V1 and Text-embedding-3-small. The zero-shot results are present in Table 4. The few-shot results with text encoder UAE-Large-V1 is present in Table 5, and few-shot results with text encoderText-embedding-3-small is present in Table 6. From the table, we can observe that our method TAGA consistently achieve the best performance on two different choices of text encoder models. This demonstrates the effectiveness and robustness of our proposed method.

Table: Table 4: Zero-shot node classification performance.

Table: Table 5: Performance of all few-shot node classification for each dataset. The text encoder choice is UAE-Large-V1.

Table: Table 6: Performance of all few-shot node classification for each dataset. The text encoder choice is Text-embedding-3-small.

## Appendix B Additional Technical Details

Efficiency Comparison with Directly Using PLM Embeddings.
It is worth noting that the textual embeddings of TofG views $\mathbf{h}(v_{i})$ can directly represent the entire TAG. However, it may cause significant scalability and efficiency issue during the inference phase. Existing PLMs typically adopts transformer architecture and it has a quadratic complexity with the input number of text tokens, this is especially important to TAGs since the number of input size grows exponentially with the number of neighborhood hops. By aligning the knowledge from PLM with GNN model through our framework, we can simultaneously maintain generalization ability of TAG embeddings and high efficiency and scalability to large-sized graphs.

Enabling Zero-Shot and Few-Shot Predictions.
Our pretrained strategy ensures that the embeddings obtained from the GNN models at each layer remain aligned within the textual embedding space. This alignment enables direct zero-shot predictions using the self-supervised trained embeddings without requiring any additional fine-tuning.

Specifically, suppose there are $L$ prediction labels $\{l_{1},l_{2},\ldots,l_{L}\}$. Their textual embeddings are obtained through the pretrained language model (PLM) as follows:

$$ $h^{(l)}(l_{i})=\mathrm{PLM}(l_{i})\quad\text{for }i\in\{1,\ldots,L\}$ (6) $$

The probability that node $v_{i}$ belongs to class $l_{j}$ is computed in an unsupervised manner by measuring the cosine similarity (or another appropriate similarity measure) between the learned GNN embeddings $h^{(g)}(v_{i})$ and the label textual embeddings $h^{(l)}(l_{j})$:

$$ $p(v_{i}\rightarrow l_{j})=\frac{e^{\rho(h^{(g)}(v_{i}),h^{(l)}(l_{j}))}}{\sum_ {k=1}^{L}e^{\rho(h^{(g)}(v_{i}),h^{(l)}(l_{k}))}}$ (7) $$

The final predicted class of node $v_{i}$ is determined as follows:

$$ ${l}(v_{i})=\operatorname{argmax}_{j}p(v_{i}\rightarrow l_{j})$ (8) $$

where ${l}(v_{i})$ is the predicted class label for node $v_{i}$, determined by selecting the class $l$ that maximizes the similarity measure $\rho$ between the GNN embedding of the node $h^{(g)}(v_{i})$ and each of the label embeddings $h^{(l)}(l_{j})$.

Additionally, to further refine the learned embeddings, we introduce a learnable transformation function for few-shot learning adaptation:

$$ $h^{(g)}_{\text{adapted}}(v_{i})=g(h^{(g)}(v_{i}),\mathcal{D}_{\text{support}})$ (9) $$

where $g$ represents a transformation function with learnable parameters (e.g., a multi-layer perceptron), and $\mathcal{D}_{\text{support}}$ denotes a set of support examples for few-shot learning. This adapted embedding $h^{(g)}_{\text{adapted}}$ is then utilized to compute the updated predictive probabilities:

$$ $p(v_{i}\rightarrow l_{j})=\frac{e^{\rho(h^{(g)}_{\text{adapted}}(v_{i}),h^{(l) }(l_{j}))}}{\sum_{k=1}^{L}e^{\rho(h^{(g)}_{\text{adapted}}(v_{i}),h^{(l)}(l_{k }))}}$ (10) $$

## Appendix C Limitations

This work aims to pioneer unsupervised representation learning in the text-attributed graph research domain. Our approach demonstrates significant performance improvements over existing state-of-the-art methods in zero-shot and few-shot prediction tasks. However, we acknowledge certain limitations.
While our work pushes the boundaries of graph foundation models, the model’s transfer capabilities may be limited when training and inference domains are vastly different (e.g., from social networks to chemical networks). We consider the development of a universal graph foundation model, capable of generalizing across diverse domains, to be an important direction for future research.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to Weaknesses.
soundness: 2
strengths: 1. The paper presents a novel dual-view framework that represents TAGs as both hierarchical documents and graph structures, marking a significant departure from traditional single-view approaches in graph representation learning.
2. The extensive evaluation demonstrates consistent performance improvements over six state-of-the-art methods across eight diverse datasets, with comprehensive ablation studies and strong transfer learning results validating the method's robustness.
3. The work addresses the fundamental challenge of jointly modeling text and structure in graph representation learning, offering a practical solution that reduces dependency on labeled data and shows strong performance on real-world applications.
summary: This paper introduces TAGA, a self-supervised learning framework for Text-Attributed Graphs (TAGs) that aligns two complementary views: Text-of-Graph, which organizes node texts into a structured hierarchical document, and Graph-of-Text, which represents nodes and connections as graph data. The framework includes a Graph2Text method for structure-preserving document generation and a random walk algorithm for efficient training. Experiments on eight real-world datasets demonstrate TAGA's superior performance in zero-shot and few-shot learning tasks across different domains.
weaknesses: While the paper presents a novel and valuable contribution to TAG representation learning, its innovation trends towards increasing complexity rather than seeking elegant solutions. The insufficient analysis of its sophisticated components leaves readers with an understanding of "what" but limited insights into "why", making it difficult to fully appreciate the fundamental principles behind its effectiveness：
1. The paper misses critical discussion and comparison with recent TAG self-supervised methods, particularly "UniGraph: Learning a Unified Cross-Domain Foundation Model for Text-Attributed Graphs". This omission raises concerns about the claimed novelty and makes it difficult to assess the true advancement over existing TAG pre-training approaches.
2.  The dual-view alignment architecture is unnecessarily restricted to self-supervised learning, while this structural design could potentially benefit from other learning paradigms. The paper fails to explore the architecture's potential beyond self-supervised learning.
3. The preprocessing steps for generating dual views (hierarchical documents and graph structures) incur substantial computational and storage costs, but the paper neither analyzes these overheads nor compares total computation time (preprocessing + training) with baseline methods.
4. The paper arbitrarily adopts BFS tree with cross-edge references as the document structuring method without any theoretical justification or experimental comparison with alternative organization strategies (like DFS, topological sorting, or community-based structuring), leaving a critical gap in understanding why this specific choice is optimal or even suitable for preserving graph structure in document format.
5. Despite claiming efficiency through random walk sampling, the paper lacks comprehensive experiments on large-scale graphs, missing analysis of how the random walk strategy affects model performance, and providing no comparison of end-to-end processing time with existing methods.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: I am open to adjust my score based on the authors rebuttal.
soundness: 2
strengths: The idea of hierarchically aligning graph and text is interesting.
summary: The paper introduces TAGA, a self-supervised framework for Text-Attributed Graphs (TAGs) that hierarchically  integrates structural and semantic information. TAGA captures joint insights effectively, and a new random walk algorithm ensures efficient training on large TAGs, demonstrating strong performance in zero-shot and few-shot scenarios.
weaknesses: 1.**Unclear Contributions**: The contributions are not clearly articulated. While the idea of aligning hierarchical textual and graph knowledge is compelling, simply stating that the work *"constructs two complementary views"* does not convey its novelty. Although the authors introduce related works on integrating text and graphs in pre-training and adaptation paradigms [1,2,3,4] in Section 2, the differences and unique contributions of this work should be highlighted in Section 1, as these distinctions form the primary contributions of this work.\
2. **Missing Related Works and Baselines**: For downstream tasks with limited labeled data, even in the absence of textual data, graph prompt tuning methods [5,6,7,8] have demonstrated strong performance and should be discussed and used as baselines for comparison.\
3. **Lack of Theoretical Proof for TAGA**: The paper lacks a theoretical proof for the effectiveness of TAGA, which would strengthen the validity of the proposed approach.

[1] Wen et al. Augmenting low-resource text classification with graph-grounded pre-training and prompting. SIGIR 2023\
[2] Wen et al. "Prompt tuning on graph-augmented low-resource text classification." TKDE 2024\
[3] Liu et al. "One for all: Towards training one graph model for all classification tasks." ICLR 2024\
[4] Tang et al. Graphgpt: Graph instruction tuning for large language models. SIGIR 2024\
[5] Yu et al. "A Survey of Few-Shot Learning on Graphs: from Meta-Learning to Pre-Training and Prompt Learning." arXiv 2024\
[6] Liu et al. "Graphprompt: Unifying pre-training and downstream tasks for graph neural networks." WWW 2023\
[7] Yu et al. "Generalized graph prompt: Toward a unification of pre-training and downstream tasks on graphs." TKDE 2024\
[8] Fang et al. "Universal prompt tuning for graph neural networks."  NeurIPS 2024.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: The research problem addressed in this work is highly significant to the graph field, and the graph2Text concept is innovative. However, the experimental setup and choice of competitive methods could be enhanced by incorporating more advanced baselines and a standardized graph pre-training evaluation framework. For further details, please refer to the Weaknesses section.
soundness: 3
strengths: - The proposed graph2Text scheme is novel and offers a general pipeline for transforming TAGs into text documents.
- A structure-preserving random walk strategy is introduced to further improve training efficiency on large graphs.
- Extensive experiments across various evaluation scenarios demonstrate the effectiveness of TAGA compared to leading baseline methods.
summary: This paper focuses on self-supervised learning on text-attributed graphs (TAGs) and introduces TAGA. Unlike prior efforts, TAGA employs a graph2Text scheme to transform a TAG into text documentation, enabling the direct use of pre-trained language models (PLMs) for graph encoding. Furthermore, layer-wise contrastive learning is applied using a hierarchical text structure extracted by graph2Text. Extensive experiments are conducted to demonstrate the effectiveness of TAGA compared to baseline methods across various evaluation scenarios.
weaknesses: - The evaluation setup is incomplete. Graph self-supervised learning is a well-established research area in the graph field. The standard evaluation setup is linear probing under semi-supervised or supervised settings. While the proposed model performs well in few-shot learning scenarios, it would be beneficial to compare all self-supervised graph models under standard semi-supervised settings, especially for datasets like Cora and PubMed, which were originally designed for semi-supervised learning. Please follow the standard semi-supervised splits in GCN paper, and compare against relevant baselines using this setup.
- More key related works should be included for comparison or at least discussed. For instance, GIANT [1], UniGraph [2], and UniGLM [3] are leading self-supervised learning methods on TAGs and should be compared. Additionally, works like [4] and [5] should be discussed, as they also integrate textual information and graph topology in a unified manner and support zero-shot and few-shot scenarios. Without a thorough discussion of these recent advances, it is challenging to assess the true capabilities of the graph2Text approach. Therefore, a dedicated subsection in the Related Work section is required to discuss these recent advances in self-supervised learning on TAGs, and explain how their approach compares or differs conceptually.
- Similarly, several graph pre-training methods should be considered for comparison or at least discussed, as they represent distinct directions in graph pre-training. Examples include MaskGAE[6] and S2GAE[7], which serve as edge-level graph-masked autoencoders.
- The experimental setup for baseline methods needs further clarification. Since many graph pre-training methods (e.g., GraphMAE, GraphCL, and GRACE) were not originally designed for TAGs, data preprocessing is required to encode textual attributes into node features. However, as shown in [5], the choice of PLM encoder can significantly impact the performance of these methods. For a fair comparison, you can add a paragraph in the experimental setup section detailing how to preprocess the textual attributes for the baseline methods. Additionally, conducting more experiments using different PLM backbones for the baselines and report the results.
- Given TAGA’s capability to train on large graphs, it would be interesting to examine its performance on TAG datasets of varying scales. For instance, would training TAGA on multiple TAGs rather than a single TAG graph further enhance its performance?

[1] Node feature extraction by selfsupervised multi-scale neighborhood prediction.

[2] UniGraph: Learning a Unified Cross-Domain Foundation Model for Text-Attributed Graphs.

[3] UniGLM: Training One Unified Language Model for Text-Attributed Graphs.

[4] Zerog: Investigating cross-dataset zero-shot transferability in graphs

[5] GAugLLM: Improving Graph Contrastive Learning for Text-attributed Graphs with Large Language Models

[6] Maskgae: Masked graph modeling meets graph autoencoders.

[7] S2GAE: Self-Supervised Graph Autoencoders Are Generalizable Learners with Graph Masking.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. There is a typo in the caption of Figure 2 "The Graph2Text module that transforms a Graph-of-Text view into a Graph-of-Text
view".
2. It would be more appropriate to place the table's title at the top.
3. In Tables 1 and 2, it can be observed that the performance of TAGA-rw, which uses the random walk method, is often better than when this method is not used. This is counterintuitive; could the authors provide an explanation for this?
4. Regarding the loss function, it seems that the number of positive and negative samples is not balanced. Should a hyperparameter be introduced to balance the losses of L_{positive} and L_{negative} ?
soundness: 2
strengths: - The writing is fluent and easily comprehensible.
- The authors focus on the issue of the lack of training labels in real-world scenarios and propose a self-supervised learning framework. They also emphasize leveraging the semantic relationships between node texts.
- A random walk method inspired by human reading patterns is proposed to reduce the time consumption of the proposed TAGA.
summary: The paper presents TAGA, a novel self-supervised learning framework designed to enhance representation learning on Text-Attributed Graphs (TAGs) by integrating both textual and structural information. By aligning the representation of Text-of-Graph and Graph-of-Text, TAGA effectively captures joint textual and structural information. Additionally, a structure-preserving random walk algorithm is introduced to facilitate efficient training on large TAGs.
weaknesses: - I believe that how to construct the Text-of-Graph view for each node is a key issue in this work. I can understand its construction method from Algorithm 1. However, I think the paper still lacks details, as it needs to demonstrate the original topological structure through the text. I hope the authors can provide an example to clarify this point.
- I think the authors need to elaborate more on why it is necessary to align the Graph-of-Text views of different orders (as in Figure 2(a))

Meta-review: The paper proposes an interesting idea of aligning graph and text contents hierarchically, with extensive experiments. 

However, the paper may not be ready for publication due to missing critical related work, requiring additional clarifications on the experimental setup, and lack of empirical demonstration on large-scale graphs, as raised by reviewers.

As the authors did not provide any response, it is unclear if these issues can be addressed satisfactory.

Decision: Reject