Title: $\texttt{G1}$: Teaching LLMs to Reason on Graphs with Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Graph

Large Language Models

Reasoning

Reinforcement Learning
Abstract: Although Large Language Models (LLMs) have demonstrated remarkable progress, their proficiency in graph-related tasks remains notably limited, hindering the development of truly general-purpose models. Previous attempts, including pretraining graph foundation models or employing supervised fine-tuning, often face challenges such as the scarcity of large-scale, universally represented graph data. We introduce $\texttt{G1}$, a simple yet effective approach demonstrating that Reinforcement Learning (RL) on synthetic graph-theoretic tasks can significantly scale LLMs' graph reasoning abilities. To enable RL training, we curate \erdos, the largest graph reasoning dataset to date comprising 50 diverse graph-theoretic tasks of varying difficulty levels, 100k training data and 5k test data, all drived from real-world graphs. With RL on \erdos, $\texttt{G1}$ obtains substantial improvements in graph reasoning, where our finetuned 3B model even outperforms Qwen2.5-72B-Instruct (24x size). RL-trained models also show strong zero-shot generalization to unseen tasks, domains, and graph encoding schemes, including other graph-theoretic benchmarks as well as real-world node classification and link prediction tasks, without compromising general reasoning abilities. Our findings offer an efficient, scalable path for building strong graph reasoners by finetuning LLMs with RL on graph-theoretic tasks, which combines the strengths of pretrained LLM capabilities with abundant, automatically generated synthetic data, suggesting that LLMs possess graph understanding abilities that RL can elicit successfully. Our implementation is open-sourced at https://github.com/PKU-ML/G1, with models and datasets hosted on Hugging Face collections https://huggingface.co/collections/PKU-ML/g1-683d659e992794fc99618cf2 for broader accessibility.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a new LLM approach training with reinforcement learning to reason on graph tasks. The authors conduct extensive experiments on multiple datasets including synthetic and real-world datasets, comparing to other state-of-the-art comparable LLMs. The method achieves better performance on various graph tasks compared to baselines even though it uses less number of parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is very well written and structured. It is easy to follow and read.\n2. The paper contributes to the research community by promising to share the code and all tasks that are used to train.\n3. The paper conducts many experiments on multiple different graph tasks, and each tasks and experiments are explained really well.\n4. The paper provides multiple versions of their methods which are serving as ablation study. They compare each method in training analysis and show the differences between the methods.\n\nWeakness:\n1. The method is only compared with methods with small parameters (there are some unknown methods and they may achieve better performance in some cases.) \n2. The method uses only 7B parameters. How would the method scale if we want to build the larger model?\n3. The paper does not really discuss the future directions, which is needed for the research paper.\n4. All the experiments are run only once. The authors mentioned that randomness is negligible. LLMs are known to give different answers based on the questions. So ML practice suggests that multiple run is needed.\n\nQUESTIONS:\n1. How can this model scale with it is being trained with very large number of parameters? Can you provide the running times of the current model? Can you provide some insights on what would happen if the method has much more parameters (e.g., 100B)?\n2. What are the future directions of this particular research? Can you provide a section that discusses limitations and future directions?\n3. Can you provide the robustness/correctness of the method by running the same questions multiple times on some tasks?\n\nLIMITATIONS:\nLimitations and societal impacts are not clear. I suggest authors to make them clear. They mentioned that the limitations are in experiment section, which I don't see them. Societal impacts are supposed to be in appendix, again I don't see them. Can authors make them clearer, maybe in a separate section? They can be included with future directions that I already suggested to add.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents G1, using reinforcement learning to teach LLMs to solve graph theoretic problems. First, a new dataset collection is constructed called Erdos (I love this name by the way), which provides an amount of graph-related tasks serving various reasoning tasks. The RL procedure follows commonly used GRPO. From the results G1 is very strong and could even beat larger models by reinforcing a small model.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: Very recent R1 technique on graphs with exciting results. The presentation is clear. It is shown that G1 can even be generalized beyond graphs.\n\nW1. Line 32 \"Different from prior work, we believe LLMs pretrained on Internet-scale data already possess graph reasoning ability.\" Dangerous! This is a very strong claim without any supporting evidence in these two lines.\n\nW2. The ending of introduction is inspiring yet not very clear. I would suggest more discussions on how to leverage G1 to build graph foundation models. This paper presents a grand position on graph foundation model (that there could be a paradigm shift with RL) but lacks supporting evidence and discussions.\n\nW3. A new dataset is presented, but only reported some details and the scales. The presented dataset, however, is not evaluated. I just have no idea of what the constructed dataset looks like, and if different prompt choice would affect evaluating the LLMs. In other words, how good and robust is the constructed dataset?\n\nW4. The proposed reinforcement learning technique is a data-centric method from my perspective. The most important thing is an amount of available graph-theoretic data with correct labels/ground-truth. The training is a mimic of GRPO, which means G1 could suffer from draw backs of GRPO, such as sampling inefficiency and optimizing instability. I don't see (or maybe I missed) study on these in 5.3 training analysis or appendix.\n\nW5. I just cannot intuitively understand how conducting G1 on graph theoretic reasoning tasks could improve beyond graphs, such as GSM8K and MATH. Given the distribution shift in the training data and test data, I would assume the performance to degrade, which is not the case. More discusion/clarification here is needed (would be better with some discussion also on non-graph-theoretic task generalizations).\n\nW6. Why don't you compare with GPT-4o or other SOTA LLMs? It's ok if not outperforming those models as they might have a very large number of parameters, but it would provide more evidence of how good G1-7B is achieving. \n\nW7. Nothing on efficiency, such as training time, is reported.\n\nQUESTIONS:\nQ1. Do you think there is any unique graph-specific things that can be leverages to improve the design of GRPO training on graphs?\n\nQ2. I love the name of the new dataset, but let's be careful. Graph theory and graph-theoretic reasoning is different. Some clarifications could be made here (not affecting the quality of the paper).\n\nLIMITATIONS:\nN/A. No remarkable negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel dataset covering more than 50 graph-theoretic tasks with varying levels of difficulty. Leveraging this dataset, they present G1, the first large language model trained with reinforcement learning for graph computational problems. G1 demonstrates superior performance compared to existing graph-focused language models on graph-related tasks, and exhibits zero-shot generalization abilities to non-graph domains such as mathematics.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. This paper is the first to employ reinforcement learning (RL) to train large language models (LLMs) for solving graph computational problems, and demonstrates performance that surpasses previous graph-focused LLMs.\n2. The authors propose a dataset that covers a wide range of graph-theoretic tasks with clearly defined difficulty levels, offering a valuable resource for both training and evaluation.\n3. The presented G1 model exhibits impressive zero-shot generalization capabilities, not only across unseen graph tasks but also in non-graph domains such as mathematics.\n\n**Weaknesses:**\n\n1. The experiments are limited to graphs with a maximum of 35 nodes, which is relatively small. It would be more convincing to see results on graphs with at least 60 nodes to better assess scalability.\n2. The paper lacks any analysis or statistics regarding the number of output tokens generated by the model, which could be important for understanding efficiency and solution complexity.\n3. The technical novelty appears somewhat limited. The reward design focuses solely on outcome-based rewards, which is relatively naïve; more sophisticated reward structures could be explored.\n\nQUESTIONS:\n1. In the reward design, the authors mainly discuss outcome-based rewards. Have the authors considered designing format rewards or intermediate-step rewards, or assigning different weights to various types of rewards? Would such modifications lead to improved performance?\n2. Could the authors provide a plot showing the relationship between RL training steps and response length? This would offer insights into the learning dynamics and solution complexity.\n3. In the generalization experiments (e.g., mathematics), does G1 exhibit longer chain-of-thought (CoT) responses or changes in reasoning patterns, such as increased stepwise thinking? Some analysis or examples would be helpful to understand the model's behavior in out-of-domain tasks.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work first compiles a large-scale graph-theoretic dataset Erdős, which includes 50 graph-theoretic reasoning tasks from NetworkX, totaling 100k/2k training/evaluation question-answer pairs. Answers are automatically solved by NetworkX solvers. Graph data is downsampled from real-world datasets to 5-35 nodes, and graph structures are represented using a unified edge list format. The work then proposes applying a reinforcement learning (RL) framework to graph reasoning tasks for large language models (LLMs), designing rule-based rewards for different graph reasoning tasks and using Group Relative Policy Optimization (GRPO) for RL training. Finally, RL training on Erdős significantly improves the performance of baseline models on the Erdős benchmark.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1)\tHeavy workload. This work includes three main contributions: organizing the dataset, migrating the rule-based reward GRPO to the field of graph reasoning, and training the graph reasoning model.\n2)\tNovel theme. Improving the reasoning ability of LLMs on graphs is a relatively less explored field.\n\nWeakness:\n\n1)\tThe work mainly focuses on migrating GRPO to the graph reasoning field of large models, and the innovation of the work itself is relatively limited.\n2)\tThere is a lack of comparison with other methods, such as directly using graph neural networks to solve graph reasoning problems, such as https://arxiv.org/abs/2209.11142\n3)\tThe evaluation of the trained model is limited to Erdős, lacking assessment on other graph reasoning datasets.\n\nQUESTIONS:\n1)\tMy main concern is whether it is appropriate to use LLM to solve graph-theoretic problems without semantic information. Intuitively, the advantage of LLM lies in its ability to understand the textual attributes of nodes, so it is naturally suitable for scenarios such as text-attributed graphs. However, for pure graph structure problems addressed in the paper, isn't it more efficient to use pure GNN reasoning? On the other hand, tasks in Erdős all have clear optimization objectives, so wouldn't it be more efficient to solve them directly with algorithms? If the authors can provide sufficient reasons for using LLM, I am willing to raise my score.\n2)\tCorresponding to Question 1, it is hoped that the authors can supplement the experiments with comparisons of some other methods for solving graph-theoretic problems.\n3)\tPossibly due to limited space and rich content, Erdős is a newly proposed graph reasoning dataset by the authors, but it lacks analysis, such as whether it should be compared with existing other graph reasoning datasets.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}