Title: GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environments
Venue: NeurIPS 2025
Accepted: yes
Keywords: Graph data synthesis

text-attributed graphs

multi-agent framework

large language models

retrieval-augmented generation

data-limited environments
Abstract: The era of foundation models has revolutionized AI research, yet Graph Foundation Models (GFMs) remain constrained by the scarcity of large-scale graph corpora. Traditional graph data synthesis techniques primarily focus on simplistic structural operations, lacking the capacity to generate semantically rich nodes with meaningful textual attributes—a critical limitation for real-world applications. While large language models (LLMs) demonstrate exceptional text generation capabilities, their direct application to graph synthesis is impeded by context window limitations, hallucination phenomena, and structural consistency challenges. To address these issues, we introduce \textbf{GraphMaster}—the first multi-agent framework specifically designed for graph data synthesis in data-limited environments. GraphMaster orchestrates four specialized LLM agents (Manager, Perception, Enhancement, and Evaluation) that collaboratively optimize the synthesis process through iterative refinement, ensuring both semantic coherence and structural integrity. To rigorously evaluate our approach, we create new data-limited “Sub” variants of six standard graph benchmarks, specifically designed to test synthesis capabilities under realistic constraints. Additionally, we develop a novel interpretability assessment framework that combines human evaluation with a principled Grassmannian manifold-based analysis, providing both qualitative and quantitative measures of semantic coherence. Experimental results demonstrate that GraphMaster significantly outperforms traditional synthesis methods across multiple datasets, establishing a strong foundation for advancing GFMs in data-scarce environments.


Peer reviews:
comments: SUMMARY:
GraphMaster introduces a novel multi-agent framework for synthesizing text-attributed graphs (TAGs) in data-limited environments. 
It addresses challenges faced by existing LLMs in graph synthesis, such as context window limitations and hallucinations, by orchestrating four specialized LLM agents (Manager, Perception, Enhancement, and Evaluation).

STRENGTHS AND WEAKNESSES:
### Strengths
1. First multi-agent LLM framework specifically for data-limited TAG synthesis.
2. Introduces a dual-perspective interpretability assessment combining human evaluation with Grassmannian manifold analysis.

### Weaknesses
1. Current knowledge selection and LLM context limitations pose challenges for very large or highly heterogeneous graphs.  
2. The multi-agent architecture incurs substantial inference overhead due to multiple LLM calls per round.

QUESTIONS:
1. Can the framework be extended to synthesize more complex graph elements (e.g., hyper-edges, temporal dynamics) or entirely novel structural patterns beyond just nodes and edges?
2. Current knowledge selection and LLM context limitations pose challenges for very large or highly heterogeneous graphs. In particular, the Perception Agent’s reliance on PPR to extract subgraphs may introduce sampling bias, favoring structurally central nodes while potentially missing peripheral but semantically important regions. Evaluating the approach on larger and more complex graphs would help clarify its scalability and generality.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge that Graph Foundation Models face due to the lack of large, diverse graph datasets. Instead of naive structural heuristics, it leverages a retrieval-augmented, multi-agent LLM framework, comprising a Manager (to plan synthesis strategies), a Perception agent (to extract context-sized subgraphs), an Enhancement agent (to generate semantically rich nodes and edges), and an Evaluation agent (to iteratively assess and refine outputs) to produce text-attributed graphs with both structural fidelity and meaningful attributes. To validate this under realistic data scarcity, the authors introduce “Sub” variants of six standard graph benchmarks and propose a dual interpretability scheme combining expert human judgments with a Grassmannian-manifold analysis of embeddings. Experiments demonstrate that GraphMaster markedly outperforms traditional synthesis baselines in both semantic coherence and structural metrics, establishing a new paradigm for generating high-quality TAGs in low-data settings\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nThe authors don’t limit their evaluation to standard structural measures like degree distributions and clustering coefficients. Instead, they also bring in human evaluators to judge whether the generated graphs actually make sense semantically, and then back up these human judgments with a Grassmannian-manifold embedding analysis. By using both human feedback and embedding metrics, they convincingly show that GraphMaster produces graphs that are structurally accurate and semantically meaningful.\n\nAnother strong point is the realistic scenario tested in the paper. Instead of using artificial, overly simple datasets, the authors created reduced (\"Sub\") versions of six common benchmarks. This approach genuinely reflects real-world situations where only limited graph data is available, highlighting GraphMaster's potential usefulness for practical applications.\n\nFinally, the authors combine human-based evaluations with quantitative embedding analysis in a novel way. Human raters offer practical, intuitive insights about semantic quality, while the Grassmannian-manifold method gives a clear, quantitative measure of embedding similarity. Together, these two evaluations provide a balanced assessment of GraphMaster's effectiveness.\n\n## Weaknesses\nOne concern is the method’s computational cost. GraphMaster uses several large language model (LLM) agents repeatedly within a retrieval-augmented generation loop. Since the authors don’t discuss ways to potentially reduce these costs, like caching, batching, or using simplified models -- it remains unclear whether the method can scale efficiently or affordably for practical users dealing with larger graphs.\n\nThe human evaluation also lacks critical details. The authors don’t explain clearly how the evaluators were chosen, how many graphs they looked at, or if they generally agreed with each other. Without these details, it’s difficult to tell if the reported semantic coherence scores represent broad agreement or simply reflect a small, possibly biased set of examples.\n\nFinally, all experiments are limited to citation and academic graphs with textual attributes. The authors haven’t tested whether GraphMaster would work equally well on other types of graphs--like social networks, protein interaction networks, or knowledge graphs with more complex structures. Because of this, it’s hard to know how broadly theapproach could be applied beyond the specific types of graphs studied in the paper.\n\nQUESTIONS:\nCould the authors clarify how GraphMaster could be scaled efficiently for larger graphs or more constrained compute environments? For example, do you consider strategies like caching previous LLM outputs, batching inputs, or employing smaller/distilled models?\n\nCould the authors clarify details about the human evaluation process--such as how raters were selected, how many raters participated, how many samples were rated, and inter-rater agreement statistics?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGraphMaster is a novel multi-agent framework for synthesizing text-attributed graphs (TAGs). The authors design a system of four LLM-based agents, Manager, Perception, Enhancement, and Evaluation, which collectively generate new graph nodes and edges in an iterative way. \nEvaluation is conducted across six benchmarks using multiple GNN architectures. The authors also introduce data-limited variants of standard datasets and a dual-perspective interpretability framework, incorporating human evaluation and Grassmann manifold-based semantic metrics. GraphMaster outperforms all baselines, including custom-built LLM synthesis methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strenghts\n- They introduced the first TAG synthesis framework to use a modular, multi-agent LLM design under a retrieval-augmented generation (RAG) paradigm. \n\n- This paper has a comprehensive evaluation. They tested on six datasets and four GNNs (GCN, JKNet, GAT, GraphSAGE). The metrics used to evaluate are also extensive e.g. accuracy, F1, graph statistics.\n\n- The paper has theoretical foundations that are tightly linked to the agent design (Appendix J). \n- They have discussed time and space complexity analysis.\n\nThis paper is groundbreaking in its framework design, theoretical depth, and evaluation rigor. It sets a new direction for text-attributed graph synthesis and provides tools (metrics, datasets, proofs) that the community can build on.\n\n## Weaknesses\n- While the framework claims generality, the synthesis quality is not validated in physically meaningful graphs where semantics require domain knowledge (e.g., valid chemistry or logic).\n- This paper is hard to read and is quite dense. The writing could be improved.\n- The quality of Figure 2 and 3 could be improved.\n\nQUESTIONS:\n- How would GraphMaster perform on non-textual graphs (e.g., protein–protein interactions or sensor networks)?\n\n- Could this work be extended to multi-modal graphs?\n\n- How scalable is the multi-agent loop for large-scale graphs (>100K nodes), especially when multiple agents call the LLM iteratively?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGraphMaster introduces a hierarchical, retrieval-augmented generation (RAG) multi-agent framework—Manager, Perception, Enhancement, and Evaluation agents—for synthesizing text-attributed graphs under severe data constraints . It also proposes six “Sub” variants of standard benchmarks to rigorously evaluate data-limited performance and a dual-perspective interpretability assessment combining expert human judgments with Grassmann manifold analysis . Extensive experiments show GraphMaster outperforming both classic and recent LLM-based baselines across multiple datasets .\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* **Novel multi-agent RAG paradigm:** Decomposes graph synthesis into specialized agents—Manager for strategy, Perception for subgraph retrieval, Enhancement for node/edge generation, and Evaluation for iterative quality control—addressing semantic coherence and structural integrity in a unified framework .\n* **Data-limited benchmark creation:** Introduces “Sub” variants of six widely used TAG datasets to test synthesis methods under realistic scarcity scenarios .\n* **Dual-perspective interpretability evaluation:** Combines quantitative Grassmann manifold metrics with qualitative human expert traceability scores (Tscore = 0.92) to validate semantic coherence and transparency .\n* **Strong theoretical foundations:** Provides convergence guarantees, information-theoretic bounds for the Perception Agent, and concentration results for the Enhancement and Evaluation agents .\n* **Comprehensive empirical validation:** Ablation studies, hyperparameter sweeps, and comparisons with both classic and newly crafted LLM-based baselines demonstrate state-of-the-art performance across six datasets and four GNN architectures .\n\nWeaknesses:\n\n\n1. **Scalability to very large graphs** (Section 3.6, lines 213–223): The time complexity analysis acknowledges batching needs for “extremely large graphs,” but no empirical evaluation is provided. Could the authors evaluate GraphMaster on larger-scale datasets or integrate scalable sampling/inference strategies [1,2] to demonstrate practical scalability?\n2. **Hyperparameter sensitivity** (Section 4.4, lines 316–323): Ablation shows optimal settings for N=30 and M=15%, but lacks guidance on robustness across other values. Could the authors include sensitivity analyses or contour plots to illustrate performance variation, perhaps leveraging [3,4].\n3. **Details of self-created LLM baselines** (Section 4.1, lines 243–249): The paper introduces Mixed-LLM and Synthesis-LLM but defers implementation specifics to the appendix, making direct comparison difficult. Could the authors summarize key architectural differences and prompt designs in the main text and compare against recent LLM-based graph generation methods [5,6].\n4. **Human evaluation methodology** (Section 4.3, lines 284–292): Expert traceability scores are reported, but the selection criteria and demographic breakdown of the 50 reviewers are not described. Could the authors detail reviewer expertise, sampling protocol, and bias mitigation measures?\n5. **Comparison errors and potential question** (Section 4.1, Table 1): The F1 score for LLM4NG in arXiv2023 dataset is 81.2, which is better than the proposed method GraphMaster (63.4). However, GraphMaster is still highlighted as the best method. In addition, how the baseline achieves a near 20 percent advantage over the GraphMaster is not properly justified.\n\n[1] Chen, Jie, Tengfei Ma, and Cao Xiao. \"FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling.\" International Conference on Learning Representations. 2018.\n\n[2] Zeng, Hanqing, et al. \"GraphSAINT: Graph Sampling Based Inductive Learning Method.\" International Conference on Learning Representations.\n\n[3] You, Jiaxuan, Zhitao Ying, and Jure Leskovec. \"Design space for graph neural networks.\" Advances in Neural Information Processing Systems 33 (2020): 17009-17021.\n\n[4] Yu, Tong, and Hong Zhu. \"Hyper-parameter optimization: A review of algorithms and applications.\" arXiv preprint arXiv:2003.05689 (2020).\n\n[5] Dadauto, Caio V., Nelson LS da Fonseca, and Ricardo da S. Torres. \"Data-driven intra-autonomous systems graph generator.\" IEEE Transactions on Network and Service Management (2024).\n\n[6] Liu, Chengyi, et al. \"Generative diffusion models on graphs: methods and applications.\" Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. 2023.\n\nQUESTIONS:\n1. How does GraphMaster perform on large-scale graphs exceeding typical LLM context windows, and can scalable strategies like FastGCN or GraphSAINT be integrated?\n2. Can you provide a detailed sensitivity analysis for critical hyperparameters (N, M%) to guide practitioners in tuning the framework?\n3. Could you summarize the architectural and prompting details of Mixed-LLM and Synthesis-LLM in the main text and compare against recent graph generation baselines?\n4. What were the criteria for selecting the 50 expert reviewers, and how did you ensure the reliability and unbiased nature of their traceability scores?\n\nLIMITATIONS:\n* The Personalized PageRank–based Perception Agent may undersample semantically important peripheral communities in highly skewed or heterogeneous graphs, leading to “blind spots.”\n* Quality evaluation depends heavily on the chosen LLM, posing challenges for deployment on smaller or resource-constrained models.\n* The multi-agent design incurs substantial inference overhead; practical deployment for very large graphs requires advanced batching and optimization techniques.\n* Privacy, fairness, and potential semantic biases in the synthesized data remain to be rigorously examined .", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}