Title: Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Code LLM

Repository-Level

Agentless

Open-source

Multimodal

Graph
Abstract: Recent advances in Large Language Models (LLMs) have shown promise in function-level code generation, yet repository-level software engineering tasks remain challenging. Current solutions predominantly rely on proprietary LLM agents, which introduce unpredictability and limit accessibility, raising concerns about data privacy and model customization. This paper investigates whether open-source LLMs can effectively address repository-level tasks without requiring agent-based approaches. We demonstrate this is possible by enabling LLMs to comprehend functions and files within codebases through their semantic information and structural dependencies. To this end, we introduce Code Graph Models (CGMs), which integrate repository code graph structures into the LLM's attention mechanism and map node attributes to the LLM's input space using a specialized adapter. When combined with an agentless graph RAG framework, our approach achieves a 43.00% resolution rate on the SWE-bench Lite benchmark using the open-source Qwen2.5-72B model. This performance ranks first among open weight models, second among methods with open-source systems, and eighth overall, surpassing the previous best open-source model-based method by 12.33%.


Peer reviews:
comments: SUMMARY:
This paper offers a code graph model (CGM) system for repo-level SWE automation. It addressed a critical gap of open source LLMs in handling repo level tasks.

STRENGTHS AND WEAKNESSES:
The paper addresses a critical need. The methodology section is particularly strong, especially the Code Graph Construction and Code Graph Model components.

However, I find that the Experiment and Results sections are inadequate and lack clearly defined research questions. Currently, the paper proposes a method and immediately presents Experiment and Results without sufficient context. I recommend authors to address following questions:

1. What is the baseline?
2. Which factors contributed most significantly to the overall success of the proposed approach?
3. What are the fundamental research questions?
4. I would like to see a clear assessment of the differential contribution of the graph-aware attention mask. Specifically, what would the performance be using a vanilla attention mask instead? The same applies to the graph-based retriever. These two artifacts are among the most intriguing elements of the paper.

That said, there is a wealth of valuable information in the Appendix, especially Figures 4 and 5. I recommend the authors carefully review their paper and present it following a clear scientific structure: Hypotheses → Experiments → Results → Conclusion.

QUESTIONS:
Is it possible for the authors to reorganize the paper? I believe the content is quite deep and possibly with immense value. But, the presentation and organization perhaps need improvement.

LIMITATIONS:
The paper is Appendix-heavy. I think the authors should reconsider the paper organization.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary\n\nThe paper proposes a code graph-based model to resolve repository-level issue resolution problems like SWE Bench tasks. \n\n1. Their technique is a fixed workflow-based approach and does not depend on the agent to take autonomous steps. \n2. They model the entire repository as a graph structure, including the code repository structure, inheritance, and call graph structure.\n3. They have multiple steps in their procedure - \n\nThey comprehend and extract relevant entities from the natural language issue.\n\nThey perform a search on the code graph structure and extract the relevant files that include the nodes in the graph.\n\nThey perform re-ranking to pick the top most k files from the retrieved files. \n\nThey provide the selected graph structure as input to the LLM and modify the attention matrix supplied to the LLM. They then pass the actual file contents to the LLM. And ask the LLM to generate the final patch file.\n\nSTRENGTHS AND WEAKNESSES:\nThe writing of the paper can be improved; it would have been nice to start with a motivating example and to explain the methodology using that example. As the proposed method is very involved and complicated, understanding the paper was challenging. \n\nStrengths\n\n1. The work is significant as it proposes an open-source trained LLM that achieves on-par performance with closed-source models.\n\n2. The paper introduces a novel technique to integrate code graph structures into an LLM; they explicitly train the LLM to understand the code graph structure and to reconstruct the file contents given the code graph structure. \n\n3. They show impressive performance on the SWE Bench Lite benchmark as well as the CrossCodeEval benchmark.\n\nWeakness\n\n1. There are a lot of moving components in the proposed approach. It is not exactly clear what components of the approach can be used for tasks other than issue resolution. Please also see my question on what component is used for retrieval-based generation.\n\nQUESTIONS:\nIn Table 4, for the purposes of retrieval, were all the components used except for the final Reader? As the CGM reader is trained to output the final answer? If the Reader was also used during retrieval, how is the Reader used for the purposes of retrieval?\n\nLIMITATIONS:\nYes, for limitations.\n\nNo, for potential societal impacts. The implications of open source models on the issue of automation should be considered in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Code Graph Models (CGMs), which is a graph-enhanced Large Language Model (LLM) architecture, for comprehensive repository-level code understanding. This architecture integrates repository code graph structures into the LLM’s attention mechanism and map node attributes to the LLM’s input space using a specialized adapter. When combined with an agentless graph RAG framework, this approach achieves a 43.00% resolution rate on the SWE-bench Lite benchmark using the open-source Qwen2.5-72B model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This work proposes an agentless approach to effectively address repository-level code tasks. \n2. When combined with an agentless graph RAG framework, this approach achieves a 43.00% resolution rate on the SWE-bench Lite benchmark using the open-source Qwen2.5-72B model, which ranks first among open weight models, second among methods with open-source systems, and eighth overall, surpassing the previous best open-source model-based method by 12.33%.\n3. The codebase is released for reproducibility. \n\nWeaknesses:\n1. Although there are engineering contributions in this work, the proposed approach seems too complicated to follow. Detailed ablation study and in-depth analysis are required in the main content to help readers gain more insight on the architecture design. \n2. This work mentions some related work such as RepoGraph[1] and GraphCoder[2] also leverage a graph-based approach to address repository-level code tasks, however, this work neither conducts a comprehensive performance comparison nor gives sufficient discussions on why the proposed approach is better.\n3. The experimental results are too limited to confirm the effectiveness of the proposed approach. We are not clear whether the effectiveness of CGM comes from the open-sourced code LLM Qwen2.5-72B or the architecture itself, since this paper only presents results from Qwen2.5-72B. More code LLMs such as DeepSeek-Coder and CodeLlama can be used for comparisons. \n4. Although this work claims the approach outperforms a couple of agentless baselines, there is an obvious gap between this approach and those agent-based baselines. The paper does not offer a compelling justification for why its proposed method should be adopted over agent-based approaches in applications, especially given that the latter demonstrate superior performance.\n\n[1] Ouyang et al., RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph, ICLR2025.\n[2] Liu et al., GraphCoder: Enhancing Repository-Level Code Completion via Coarse-to-fine Retrieval Based on Code Context Graph. ASE 2024.\n\nQUESTIONS:\n1. Can you add more in-depth analysis in the ablation study? Also, I would suggest adjusting the main content and adding the results of ablation study to the main content, as they are important.\n2. Can you use 2-3 sentences to summarize your strengths that are superior to previous works like RepoGraph and GraphCoder?\n\nLIMITATIONS:\n1. This work might provide more results using different open-sourced LLMs to demonstrate the effectiveness of the proposed approach.\n2. This paper might add a new section to discuss the broader impact of this approach to give more insights.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the Code Graph Model (CGM), a framework that integrates code graph structures with open - source Large Language Models (LLMs) to address repository - level software engineering tasks. Traditional approaches relying on proprietary LLM agents face issues like unpredictability and limited accessibility. CGM overcomes these by enabling LLMs to understand codebase functions, files, and their dependencies through semantic and structural integration. It constructs a code graph for each repository, encoding node attributes and incorporating graph structures into the LLM's attention mechanism. Coupled with an agentless Graph Retrieval - Augmented Generation (RAG) framework, CGM achieves a 43.00% resolution rate on the SWE - bench Lite benchmark using the open - source Qwen2.5 - 72B model. This performance ranks highly among open - source models and demonstrates the feasibility of open - source LLMs in handling complex repository - level tasks without agents.\n\nSTRENGTHS AND WEAKNESSES:\n## **Strengths：**\n\n1. **Innovative Architectural Design:** Introduces Code Graph Models (CGM), achieving deep integration of code repository graph structures with open-source LLMs through semantic encoders and structure-aware attention mechanisms, solving the structural information loss problem caused by code graph linearization in traditional methods.\n\n2. **Efficient Agent-free Framework:** The developed Graph RAG framework requires only 4 modules  significantly reducing interaction complexity while achieving a 43% problem-solving rate on SWE-bench Lite using open-source models, surpassing closed-source model approaches like Claude 3.5 Sonnet.\n\n3. **Multimodal Information Integration Advantages:** Extends model context length by 512 times through a \"chunk encoding-adapter projection\" mechanism that compresses long text nodes into LLM-processable node representations, enabling simultaneous processing of graph structures (global dependencies) and complete code text (local details).\n\n4. **Strong Empirical Support:** Validates effectiveness in Python/Java cross-language scenarios, exceeding similar open-source model methods by 12.33% on SWE-bench Lite, while improving ES metrics by 19.8% in code completion tasks, demonstrating method generalization capability.\n\n5. **Interpretability Improvements:** Maintains stable performance even with 10% noisy inputs through controlled noise fine-tuning and structured attention masking, addressing the error accumulation pain point of traditional agent-based methods.\n\n## Weaknesses\n\n1. **No Analysis of TTS Performance:** Many frameworks (such as Agentless) employ test-time scaling to explore the diversity of model outputs. In the experimental setup of CGM, only a single sample was performed, without multiple sampling to explore the upper bound of the model’s capabilities.\n2. **Severe Performance Degradation for Small Models:** When using the Qwen2.5-Coder-7B-Ins model, only a 4% resolution success rate was achieved. In comparison, models of similar size such as Seed-Coder-7B-Ins can achieve a score of about 20%.\n3. **Limited Language Paradigms:** Experiments and tool design are conducted only on Python and Java (object-oriented languages). There is no exploration of the feasibility for functional, scripting, or more complex paradigms, and cross-paradigm transfer remains unaddressed.\n4. **Long Execution Time:** Constructing a repository-level code graph takes about 3 minutes. Combined with model inference time, this is significantly longer compared to other workflow-based frameworks.\n5. **Lack of Bad Case Analysis:** The paper lacks in-depth analysis of failure cases. Understanding why CGM fails in certain situations, such as incorrect handling of complex dependencies or loss of structural information, is crucial for further improvement.\n\nQUESTIONS:\n1. Please submit the results or statistical curves of multiple TTS (e.g., different temperature/top-k) sampling runs, showing the potential improvement and stability of the model’s performance under best-of-n sampling. If CGM demonstrates significant improvement under TTS strategies, this will directly raise my evaluation of the model’s capability and the experimental thoroughness of the paper.\n2. If the authors can provide a detailed analysis of failure cases—including failure statistics, attribution of failure patterns, and concrete discussion of possible remedies—it will greatly increase my confidence in the research depth and the prospects for future improvement of this work.\n3. The resolution rate of CGM-Multi on Java repositories is much lower than on Python , but the reasons were not thoroughly analyzed. Is this due to insufficient support for Java-specific syntax (such as annotations or generics) in the code graph construction process, or is it because the Java sample proportion in training data is low? If possible, could you supplement experiments with other languages in the multi SWE bench[1]? If the current approach cannot be applied to other languages, please clearly state this limitation in the discussion, and provide a road map if available.\n4. Have the authors conducted any analysis of how the amount of training data affects repair performance? If so, please consider presenting this analysis to demonstrate that the capabilities of the CGM approach can continue to improve with increased data.\n\n[1]Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}