Title: SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Self-supervised Learning

Text-Attributed Graphs
Abstract: Large-scale pre-trained models have revolutionized Natural Language Processing (NLP) and Computer Vision (CV), showcasing remarkable cross-domain generalization abilities. However, in graph learning, models are typically trained on individual graph datasets, limiting their capacity to transfer knowledge across different graphs and tasks. This approach also heavily relies on large volumes of annotated data, which presents a significant challenge in resource-constrained settings. Unlike NLP and CV, graph-structured data presents unique challenges due to its inherent heterogeneity, including domain-specific feature spaces and structural diversity across various applications. To address these challenges, we propose a novel structure-aware self-supervised learning method for Text-Attributed Graphs (SSTAG). By leveraging text as a unified representation medium for graph learning, SSTAG bridges the gap between the semantic reasoning of Large Language Models (LLMs) and the structural modeling capabilities of Graph Neural Networks (GNNs). Our approach introduces a dual knowledge distillation framework that co-distills both LLMs and GNNs into structure-aware multilayer perceptrons (MLPs), enhancing the scalability of large-scale TAGs. Additionally, we introduce an in-memory mechanism that stores typical graph representations, aligning them with memory anchors in an in-memory repository to integrate invariant knowledge, thereby improving the model’s generalization ability. Extensive experiments demonstrate that SSTAG outperforms state-of-the-art models on cross-domain transfer learning tasks, achieves exceptional scalability, and reduces inference costs while maintaining competitive performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel self-supervised learning framework called SSTAG for text-attributed graphs that leverages large language models and graph neural networks to create a unified representation. It aims to bridge the gap between semantic reasoning and structural modeling by using text as a common medium. The framework includes a dual knowledge distillation process that transfers knowledge from LLMs and GNNs into a lightweight, structure-aware MLP, enhancing scalability and reducing inference costs. Additionally, it incorporates an in-memory mechanism to store typical graph representations, improving generalization across different domains. Extensive experiments demonstrate its superior performance in cross-domain transfer learning and scalability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The authors propose a novel framework for the task of OOD detection on graph data, which is highly practical and likely to attract broad interest.\n2. The paper is well-structured and easy to follow, making it accessible for readers with limited background knowledge.\n3. The authors demonstrate superior performance over state-of-the-art methods across diverse datasets.\n4. The authors conduct thorough experiments to validate the proposed approach's effectiveness.\n\nWeaknesses\n1. The authors use the PPR algorithm, which involves matrix inversion. The sampled subgraph has quadratic complexity.\n2. What kind of interpretable advantages can the graph + LLM framework provide?\n3. The authors use a consistency loss instead of a contrastive loss. What is the motivation behind this choice, and are there experiments to verify its advantages?\n4. The authors have demonstrated the superiority of their framework on various tasks, but it seems that they haven't tried the most common graph clustering task.\n5. While LLM-based ideas are well motivated, the mechanism could be further explained in terms of semantic alignment between textual and graph domains.\n6. There are a few minor grammatical errors in the paper. Please check them carefully.\n\nQUESTIONS:\n1. The authors use the PPR algorithm, which involves matrix inversion. The sampled subgraph has quadratic complexity.\n2. What kind of interpretable advantages can the graph + LLM framework provide?\n3. The authors use a consistency loss instead of a contrastive loss. What is the motivation behind this choice, and are there experiments to verify its advantages?\n4. The authors have demonstrated the superiority of their framework on various tasks, but it seems that they haven't tried the most common graph clustering task.\n5. While LLM-based ideas are well motivated, the mechanism could be further explained in terms of semantic alignment between textual and graph domains.\n6. There are a few minor grammatical errors in the paper. Please check them carefully.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose SSTAG, a structure-aware self-supervised framework tailored for text-attributed graphs, aiming to bridge the gap between the structural reasoning strengths of GNNs and the semantic understanding capabilities of LLMs. By leveraging text as a unified medium, SSTAG tackles the challenge of knowledge transfer across heterogeneous graph domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The structure of the paper is clear and easy to follow.\n2. The proposed method seems reasonable and sound.\n\nWeaknesses:\n1. The paper lacks discussions with GIANT. It is recommended that the authors clarify the distinctions and conduct comparisons with GIANT (e.g., in terms of methodology, performance, or application scenarios).\n2. The authors are recommended to compare with language-model-only approaches (i.e., by removing Equation 7) to demonstrate the performance gains brought by incorporating graph structures.\n3. The authors adopted a cross-domain approach for all self-supervised methods by conducting pre-training on ogbn-paper100M. However, GraphMAE and GraphCL were not specifically designed for cross-domain applications. They can achieve better results when directly performing self-supervised training on target datasets. Is such the evaluation fair?\n\nReferences:\n\n[1] Chien E, Chang W C, Hsieh C J, et al. Node feature extraction by self-supervised multi-scale neighborhood prediction. ICLR 2022.\n\nQUESTIONS:\n1. Based on the dataset description appendix, the standard split should have been used for fine-tuning on PubMed. Is the relatively poor performance of GCN on PubMed attributable to the use of ST language model for obtaining node representations?\n\n2. For molecular graph datasets, did the authors employ SMILES strings for encoding?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SSTAG, a novel self-supervised learning framework designed for text-attributed graphs. SSTAG leverages text as a unified representation medium to bridge the gap between the semantic reasoning capabilities of large language models (LLMs) and the structural modeling abilities of graph neural networks (GNNs). The framework includes a unified graph task module, a knowledge extraction module from LLMs, and a knowledge distillation module. It employs a dual knowledge distillation approach to co-distill LLMs and GNNs into a structure-aware multilayer perceptron (MLP), enhancing the scalability of large-scale text-attributed graphs. Additionally, it introduces an in-memory mechanism to store typical graph representations and align them with memory anchors in an in-memory repository, thereby improving the model's generalization capabilities. Experimental results demonstrate that SSTAG outperforms state-of-the-art methods in cross-domain transfer learning tasks, exhibits strong scalability, and reduces inference costs while maintaining competitive performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nClear Research Objective: The paper focuses on the challenge of cross-domain knowledge transfer in text-attributed graphs, proposing a self-supervised learning framework that combines the strengths of LLMs and GNNs. This aligns with current research trends and addresses practical application needs.\n\nInnovative Framework Design: The dual knowledge distillation framework and in-memory mechanism are innovative. By co-distilling LLMs and GNNs into a structure-aware MLP, the framework effectively integrates semantic and structural information, offering a new approach for text-attributed graph representation learning.\n\nComprehensive Experimental Validation: The experimental section is rich and thorough. The proposed method is evaluated on multiple benchmark datasets across various tasks, including node classification, link prediction, graph classification, and graph regression. The results demonstrate the superiority of SSTAG over state-of-the-art methods, providing strong empirical support for its effectiveness.\n\nGood Clarity: The paper is well-structured, with each section logically organized and clear in expression. The methodology is detailed, enabling readers to easily understand the research content and technical details.\n\n**Weaknesses**\n\nLimited Novelty: While the proposed framework achieves good performance, its core ideas draw heavily on existing self-supervised learning and knowledge distillation techniques. The novelty of the underlying principles may be somewhat lacking.\n\nInsufficient Theoretical Analysis: The paper focuses more on experimental validation and lacks in-depth theoretical analysis of the model's performance and generalization capabilities. For instance, it does not provide a rigorous theoretical proof of why the dual knowledge distillation framework can enhance model performance.\n\nExperimental Results Limited to Specific Datasets: Although the experimental results are promising, they are based on specific datasets. The generalizability of the proposed method to other types of graph datasets remains to be further verified.\n\nQUESTIONS:\nFramework Component Contributions: The paper mentions that SSTAG includes a unified graph task module, a knowledge extraction module from LLMs, and a knowledge distillation module. However, it does not detail the specific contributions of each component to the overall performance of the framework. I would like to ask the authors to conduct ablation studies to analyze the impact of each component on the model's performance. For example, how does the performance change when using only the unified graph task module or only the knowledge extraction module? This would help clarify the role of each component and enhance the understanding of the framework's effectiveness.\n\nHyperparameter Selection: The performance of the proposed framework may be influenced by various hyperparameters, such as the trade-off coefficients between different losses and the parameters of the in-memory mechanism. The paper does not provide detailed information on hyperparameter tuning. I would like to ask the authors to supplement the description of hyperparameter selection methods and sensitivity analyses. How were these hyperparameters determined? Are there any recommended ranges or values for different types of graph datasets? This information would be helpful for readers to apply the framework to their own research and practice.\n\nApplicability to Other Graph Types: The experimental section of the paper focuses on text-attributed graphs. However, graph data comes in various forms, such as graphs without text attributes or with other types of attributes. I would like to ask the authors about the applicability of SSTAG to other types of graph data. Can the framework be extended to handle graphs with non-text attributes? What modifications or optimizations would be needed for such extensions?\n\nLIMITATIONS:\nThe authors have briefly mentioned some limitations of their work in the paper, such as the potential limitations of the proposed framework in handling extremely large-scale graphs and dynamic graphs, as well as the trade-off between model accuracy and efficiency. However, the discussion of these limitations is relatively brief and lacks in-depth analysis. For instance, the paper could further explore the challenges the framework may face in dynamic graph scenarios and propose potential solutions. Additionally, while the paper acknowledges the computational and memory costs of the framework, it does not provide detailed analyses of its resource requirements and optimization strategies. Future work could include a more comprehensive evaluation of the framework's performance on dynamic graphs and large-scale graphs, as well as further optimization of its computational efficiency and memory usage.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SSTAG, a self-supervised pre-training method for text-attributed graphs (TAGs). It mainly integrates a masked language model with a GNN to achieved the ability to process TAGs in a structure-aware manner. The method is evaluated on 4 different tasks (node & graph classification, graph regression, and link prediction).\n\nSTRENGTHS AND WEAKNESSES:\n# Weaknesses\n\n- Neither code, nor training/model hyperparameters are provided with the paper. This severely limits the reproducibility of the paper. This dictates the majority of my score.\n    \n- A concern regarding Eq 2: Inferring from Eq 3, $\\pi_v$ should be scalar, however the result of eq 2. is a vector. This seems to indicate a mistake in the eq. 2.  \n    Additionally, no citation is provided for the PPR algorithm. The 3 papers cited in the paragraph start on line 144 do not mention PPR at all, and so one would expect a citation specifically for that technique.\n    \n- The denoted accuracies of GCN (and other supervised methods) seems very low in comparison to existing literature. E.g. In (this paper), the reported accuracy of GCN on WikiCS is ~80%, and on ogbn-Arxiv is ~73%.\n    \n- The integration of all components in the method is lacking complete explanation. Specifically, it is unclear to me where the memory bank is located in the pipeline. Is it after the distilled model (the structure-aware MLP)? It would be very helpful to have a schematic to explain how different pieces fit together.\n    \n- There are no comparisons provided for the  accuracy hit incurred by using a distilled MLP model instead of the GNN during inference.\n    \n\n# Strengths\n\n- The paper incudes a wide range of evaluations (Node classification, link prediction, graph classification, and graph regression).\n\nQUESTIONS:\n- In table 5: the training/pretraining times would make a lot more sense if the computational hardware used was also provided. Can you please specify the hardware used in the paper?\n    \n- Are the language models trained from scratch during SSTAG pretrianing? Or is a checkpoint used? If a checkpoint is used, is it finetuned during the SSTAG training or is it kept frozen?\n\nLIMITATIONS:\nNo, limitations have not been discussed in the text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}