SWINGARENA: Adversarial Programming Arena for Long-context GitHub Issue Solving
Authors: Wendong XU
Jing Xiong
Chenyang Zhao
Qiujiang Chen
Haoran Wang
Hui Shen
Zhongwei Wan
Jianbo Dai
Taiqiang Wu
He Xiao
Chaofan Tao
Zhuoqing Mao
Ying Sheng
Zhijiang Guo
Hongxia Yang
Bei Yu
Lingpeng Kong
Quanquan Gu
Ngai Wong
Venue: ICLR 2026 Oral (2026)
Keywords: Arena
Real-World GitHub Issues
Adversarial Programming
Retrieval-Augmented Generation
Continuous Integration
Code Benchmark
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We present \textsc{SwingArena}, a adversarial evaluation framework for Large Language Models (LLMs) that closely mirrors real-world software development workflows. Unlike traditional static benchmarks, \textsc{SwingArena} models the collaborative process of software iteration by pairing LLMs as \textit{submitters}, who generate patches, and \textit{reviewers}, who create test cases and verify the patches through continuous integration (CI) pipelines. To support these interactive evaluations, we introduce a retrieval-augmented code generation (RACG) module that efficiently handles long-context challenges by providing syntactically and semantically relevant code snippets from large codebases, supporting multiple programming languages (C++, Python, Rust, and Go). This enables the framework to scale across diverse tasks and contexts while respecting token limitations. Our experiments, using over 400 high-quality real-world GitHub issues selected from a pool of 2,300 issues, show that models like GPT-4o excel at aggressive patch generation, whereas DeepSeek and Gemini prioritize correctness in CI validation. \textsc{SwingArena} presents a scalable and extensible methodology for evaluating LLMs in realistic, CI-driven software development settings.

Published as a conference paper at ICLR 2026
SWINGARENA: ADVERSARIAL PROGRAMMING ARENA FOR LONG-CONTEX: GITHUB ISSUE SOLVING
Wendong :u Jing Xiong" Chenyang Zhao Qiujiang Chen 20 .Horran Wang Hui Shen', Zhongwei Wan .Jianbo Dai Taigiang Wu He Xiao Chaofan Tao Z. Morley Mao Ying Sheng Zhijiang Guo Hongxia Yang*, ,Bei Yu?, Lingpeng Kong' Quaanquan Gu² Gua, Wong'
1 'he University of Hong Kong University of California, Los Angeles TTinghua University
University of Michigan 'hhe Ohio State University University of Edinburgh
Thhe Chinese University of Hong Kong Thhe Hong Kong Polytechnic University
l HHng Kong University of Science and Technology (Guangzhou) 10 LMSYS Org
Website
GitHub
Huggingface
ABSTRACT
We present SWINGARENA, an adversarial evaluation framework for Large Lan- guage Models (LLMs) that approximates real-world software development work- flows. Unlike traditional static benchmarks, SWINGARENA models the collabo- rative process of software iteration by pairing LLMs as submitters, who generate patches, and reviewers, who create test cases and verify the patches through con- tinuous integration (CI) pipelines. To support these interactive evaluations, we introduce a vettieval-augmennt code generation (RACG) module that handles long- context challenges by providing relevant code snippets from large codebases across multiple programming languages (C++, Python, Rust, and Go). Our adversarial evaluation can surface limitations that are often overlooked by traditional evaluation settings. Our experiments, using over 400 high-quality real-world GitHub issues selected from pool of 2,300 issues, indicate differing behavioral tendencies across models in patch generation versus validation. SWINGARENT offers scalable and extensible approach to evaluating LLMs in Cl-driven software development settings.
INTRODUCTION
Large Language Models (LLMs) have become potent accelerators of software development (Chen) et al. 2021 Li et al. 2023 Lozhkoy et al., 2024), capable of synthesizing high-quality code snippets, automatically ly detecting and repairing defects, and providing interactive guidance throughout the development cycle. To assess these three capabilities- ccreeeess-oooeeeeeeee generation
codde-gnneaatoonn fidelity, automated debugging, and conversational assistance- benehmark must evaluate each dimension individually and in concert. Existing suites including HumanEval (Chen et al. (021) and MBPP (Austin et al.) (02T) focus on the functional correctness of concise, self-contained snippets, offering a valuable first glance at model proficiency. Nevertheless, their narrow scope fails to capture the richer, iterative workflows that typify modern software engineering. Recent efforts including SWE-Bench (Jimenez et al. 2023) move toward greater realism by grounding tasks in genuine GitHub issues and repositories. Nevertheless, they typically rely on static or partially simulated contexts like single unit tests, omitting the critical role of the full Continuous Integration (CI) pipeline and its automated safeguards that define professional development. Moreover, these benchmarks tend to assume a one-shot coding paradigm, whereas industrial software engineering is inherently iterative: debugging, testing, and incremental refinement are the norm, not the exception.
These authors contributed equally.

---

Published as a conference paper at ICLR 2026
In real-world software development, coding tasks involve collaborative, iterative workflows with complex project requirements and automated systems. For example, a common scenario involves a contributor submitting pull request (PR) to large open-source project, where a CI pipeline-onl often implemented with GitHub autions-uttomaiicalyy builds up validation environments, runs unit tests, enforces style guides, executes linters] and validates compatibility with the existing codebase. Any failure triggers an iterative dialogue between contributor and reviewers, where comments are exchanged and resolved, patches are applied, and checks are rerun until all pass, allowing the PR be approved and merged into the upstream codebase. This rrevieerrrr loop epitomizes modern collaborative workflows, yet robust evaluation framework for such interactions in LLMs still lacking, leaving critical aspect of real-world practice largely unaddressed by current benchmarks.
To meet the demands of this sal-world process, conventional benchmarks ZZan et al. 2024 2025
Rashid et al. 2025; Aleithan et al.) (024) often fall short by focusing only on basic questions like "Does the code pass unit test?" threshold that is far beneath the expectations of professional software development. A meaningful evaluation must instead ask, "Can the model submit code that valid, compliant, and able to pass a full CI pipeline and peer review?"
Current evaluation practices suffer from three critical blind spots. First, static benchmarks use fixed. predictable challenges that fail to capture the dynamic, adversarial nature of real software development where patches face adaptive scrutiny. Second, they evaluate single-agent performance in isolation, missing collaborative interactions and role-switching dynamics essential to modern software engineering workflows. Third, they focus narrowly on functional correctness while ignoring comprehensive quality gates that determine real-world success.
LLMs must overcome one of the core challenges in software engineering: effectively analyzing and interpreting over long contexts in code repositories. Realistic CI tasks involve sprawling code- bases where essential information is scattered across thousands of lines and multiple files. To enable fair evaluation across diverse model architectures and context window sizes, we implement a Retrieval-Augiee Augmented Code Generation (RACG) system that provides standardized context access using established retrieval components. This helps ensure models receive comparable context- useful for isolating the effects of our adversarial evaluation protocol from retrieval artifacts.
Thus, we introduce SWINGARENA, an adversarial evaluation framework that operationalizes full CI workflows as dynamic testing arenas for LLMs. Our contributions are:
A standardized adversarial CI evaluation protocol that pairs submitter and reviewer, executed on repository-native workflows (PK-style dual-role evaluation with role switching and clear scoring).
A multi-languaae long-context retrieval pipeline (RACG) combining syntax-aware chunking, dense reranking, and token- seeen-budgt-awwand packing across C++, Python, Rust, and Go, positioned as strong baseline to support SwingArena rather than standalone algorithmic contribution.
A curated, Cl-grounded dataset of 2,300 real GitHub issues with solutions (4 languages), including 400 evaluation instances (100 per language) and a 100-sample ablation split, with scripts to reproduce retrieval and CI execution.
We evaluate SWINGARENA across multiple state-of-the-art proprietary and open-source models, including GPT-4o, Claude-3.5, Gemini-2.0, DeepSeek-Ve V33, and several open-source alternatives, demonstrating the framework's 's broad applicability and revealing distinet behavioral patterns across different model architectures. Our experiments reveal that while some models excel at aggressive patch generation, others prioritize correctness and CI stability, highlighting the nuanced trade-offs that emerge in realistic software engineering scenarios.
We adopt an adversarial CI security, coverage) and the interaction between patching and testing that fixed suites under-report. This design enables us to measure end-to-end success on real pipelines and to quantify stability, reviewer effects, and RACG's s contribution under context limits across models and languages.
protocol rather than static tests cover repository-nan
gates (style,
'A linter is tool that automatically analyzes code for grammar errors, style issues, and potential bugs to ensure consistency and quality.
2

---

Published as a conference paper at ICLR 2026
Step1. Repo Mining
Step2. CI Test Filtering
Popular Repo
Repo
ma
Hi-
mot
C++
Python
Action
M M42
Rust
Instance 3
Instance Set
Instance Instance Instance 3
n Instance
SwingArena
Delete Imdlewant Polance
darryy in
Step4. Expert Filtering
Step3. LLM Filtering
Figure 1:
Overview of SWINGARENA data construction pipeline, including repository collection, pull request extraction, task instance creation, quality filtering, and multiple Cl-based validation.
RELATED WORK
2.1
BENCHMARKS FOR EVALUATING REAL-WORLD SOFTWARE ENGINEERING
Real-world software engincering is complex, requiring nuanced reasoning over large, evolving
codebases. While LLM-based agents have made significant progress (Yang et al. 2024 Zhang
et al. 2024 Xia et al., 2024), evaluating the quality of generated code remains challenging (Wang et al.
2025a SWE- Bench (Jimenez et al.] (203) takes an important step toward realism by using
GitHub issues, pull requests, and unit tests grounded in real repositories. However, it is limited to Python, focuses only on unit test success, and includes noisy or weakly aligned test cases. Recent multi-language extensions (Zan et al. 2024 2025 RRahid et al.) 2025 Aleithan al., 2024) often
require manual Docker setup, hindering automation and preventing the use of stricter CI-based arenas evaluate LLMs under realistic development workflows. How integrate real-world workflows of multi-language code development, submission, and review into evaluation remains an open problem.
2.2 CODE EVALUATION
Evaluating the effectiveness of code-generation agents remains challenge (Wang al.
202.c). Prior
work explores diverse assessment strategies (Evtikhiev et al. 2023 Yetistiren et al. 2022 Siddiq et al. 2024). but most focus function-level correctness (Mündler et al. 2024] Zhuge al. [024 Lu et al. 2024b Dunsin 2025) and overlook repo-level effects. We push evaluation to the software- wide scale by embedding generated patches into a CI pipeline, allowing us observe their impact on build stability, regression risk, and interactions across the codebase. Current benchmarks (Jain et al. 2024 Wang et al. 2025b) still lack the ability to stress-test models this way, as they do not incorporate an adversarial agent that detects corner cases and synthesizes new unit tests.
2.3 RETRIEVAL-AUGEENTED GENERATION
The context window limitations of LLMs present a major bottleneck for handling large-scale code- bases. Retrieval-augmenter generation (RAG) techniques (Jimenez et al., 2023; Xie et al. 2025 Yang et al., 2024) have emerged as a practical solution. While advanced code retrieval methods explore structured representations like code graphs or abstract syntax trees for more semantic under- standing, many current approaches still rely on lexical methods like BM25 (Robertson et al. 2009)
for document and function name retrieval, without incorporating static code analysis or fine-grained code structure understanding. This often hampers performance on tasks that require identifying relevant functional code blocks from complex repositories (Feng et al. 2020 Parvez et al. 2021 Zhang et al., 2023 Xia et al.) 2024). For ARENG we opted for a robust and language-agnostic RAG pipeline to serve as strong, reproducible baseline, acknowledging that more sophisticated, language-speciff retrieval strategies represent promising direction for future work.
3 SWINGARENA
3

---

Published as a conference paper at ICLR 2026
3.1 DATA CONSTRUCTION
This section details the data construction pipeline for SWINGARENA. The pipeline comprises several stages: Repository Mining, CI Test Filtering, LLM Filtering, and Expert Filtering. An overview of this process is presented in Figure
Repository Mining We identify high-quality repositories via the GitHub API, prioritizing those with high popularity including star count as a proxy for code quality and community validation. We assume that patches and unit tests merged into such repositories have typically undergone extensive expert review, increasing the likelihood of correctness. For each selected repository. we collect metadata including license, forks, and activity, and clone those meeting our predefined criteria. From these, we extract real-world PRs linked to their corresponding issues, along with code diffs, patch content, and metadata. We then convert raw PRs data into structured benchmark instances by extracting problem statements from PR descriptions and issues, gathering associated patches and available test cases. Each task enriched with contextual metadata including repo II, base commit SHA, patch, timestamp, and CI lists, enabling realistic and fine-grained evaluation of L.M performance. We release a curated, Cl-grounded dataset of 2,300 (issue, PR) pairs and provide 400 evaluation instances (100 per language) plus 100-sample ablation split. We include license-aware distribution and scripts for reproducible retrieval and CI execution. We focus on four programming languages- Rust,, Go, C++, and Python-basee based on their prevalence in open-source repositories and the maturity of their associated CI ecosystems. These languages dominate most large-scale codebases and account for a substantial portion of real-world software development activity.
CI Test Filtering After mining repositories, we integrate corresponding CI configurations including GitHub Actions and Travis CP2 for each PR to fully replicate the real-world end-to-end software development process, including each repository's 's testing and build requirements, We retain only instances that pass all CI checks, ensuring they meet project-specific quality standards. Instances with verified test coverage are prioritized. This step ensures that the benchmark consists of code changes that correspond with realdworld, automated validation pipelines. By integrating CI, our pipeline enforces strict validation, creating original instances pool that mirrors the constraints and practices professional software development.
LLM Filtering To improve dataset quality and balance instances' difficulty, we employ LLM-as-a- Judge (Zheng et al. 2023) to assess the clarity of each pull request's problem statement and estimate its overall difficulty. The LLM (Grok-3-beta (xAl 2024)) is required provide reasons for its clarity and difficulty assessments, ensuring each evaluation is supported by a rationale for the sake of final expert verification. This process enables us categorize tasks by complexity, creating a structured and balanced benchmark.
Expert Filtering Following LLM -u-M-as-a-Jude,, human experts finally reviewed and calibrated LLM- generated assessments. Annotators examine the clarity and difficulty scores, along with the model's rationales, and either confirm or correct them. If the model's 's justification is unclear or misaligned with the problem, human experts intervene to ensure consistent and accurate labeling. This step mitigates LLM hallucinations and filters out low-quality problem statements, incoherent instances, or those with unreachable expired multi-modal attachment. After all the data construction process, the final data statistics of SWINGARENA can be found in Appendix B
3.2 ARENA
In this section, we introduce SWINGARENA, adversarial evaluation framework that facilitates direct competition against models on a set of programming tasks. Unlike traditional benchmarks focused on static code snippet completion single unit tests, SWINGARENA creates a dynamic, interactive environment that simulates real-world software development workflows. Specifically, the framework emulates the process of submitting and reviewing code changes: one model assumes the role of submitter, proposing a candidate solution, while another acts as reviewer, critiquing the submission and crrucially-geneete This interactive setup accurately mirrors human software collaboration and enables richer evaluation along multiple dimensions, including reasoning depth, code robustness, and collaborative capability.
additional test cases to expose potential flaws or edge cases.
TTravis CT is continuous integration service that automates testing and deployment of code changes in software projects.
4

---

Published as a conference paper at ICLR 2026
Codebase
Isse Chunk:
Participant (Submitter)
Invalid Patch
Snippet
B gollige
Fancece
sarvery
n Generated Patch -53 4t B
Crark Eronccrr
Velid Patch
Issue
-
Carny Drbicore
cohedig
Participant (Reviewer)
Reranker
sample
Patch Verifier
Next Round
Participant
X
Fatch Falled
Codebase
Test Verifier
Generated Test
(Submitter)
Winner
Valld Generated Patch
Runil suttrnne
Participant
Switch Roles
Fac: sill f+
Potch
Participant
Winner
(Reviewer)
Participantt
Final Veritler
X Invalid Teat
Figure 2: Illustration of the SWINGARENA adversarial evaluation framework. The framework
simulates an adversarial software engineering workflow between two agents- a submitter and a reviewer-who alternate roles and iteratively refine their solutions based on CI feedback.
Interactive Environment SWINGARENA employs modular, multi-agent interactive environment in which language models act as software engineers. Each model receives the same task description and access to a shared codebase but may take different approaches to generate patches and unit test cases. Our end-to-end environment includes components for retrieving relevant code, extracting and ranking code chunks, and generating code patches. Code modifications are validated using real-worke CI pipelines, ensuring that models produce solutions that compile, run, and pass automated tests. This design takes significant step toward aligning model evaluation with real-world software engineering scenarios.
We denote a single round of adversarial patch and test case generation, evaluation, and scoring between the submitter and reviewer as a battle. Given a problem statement with a buggy program and its description, the submitter generates a patch to fix the issue, while the reviewer creates a test case to challenge the patch's 's correctness, A CI pipeline evaluates both: the patch is checked for compilation and correctness, and the test case assessed for its ability to expose flaws. The submitter receives a score of +1 for patches passing all tests (including the reviewer's s), or 1 for any failure. The reviewer receives a score of +1 their test fails the submitter's patch, revealing a fault, and -1 it fails the golden human patch. Models alternate roles across multiple rounds with CI feedback for iterative refinement, simulating dynamic software development.
Battle Protocol
Verification SWINGARENA integrates real-world CI workflows into the evaluation process verification mechanism. For each repository, existing CI pipelines supelines-including GitHub Actions are executed locally within container environments, preserving the exact logic, dependencies, and toolchains used by human developers. Language-specific execution logic is supported when needed including Rust's ca rgr system ensuring accurate and faithful testing. All evaluations run inside isolated Docker containers to guarantee reproducibility and avoid cross-task contamina- tion. Given that differrntt repositories require different runtime environments, developers can miti- gate environment- inconsistencies by supplying configuration files including Docker f le, .yaml, or envirormen . yml to automatically construct test environments containers for consistent and automated verification.
Evaluation SWINGARENA evaluates models through adversarial interactions between two agents-the patch generator (the submitter) and the test case generator (the reviewer) within real-world CI workflows. Each task begins by validating the baseline and golden human patch via CI to establish correctness references. In each round, the submitter proposes patch, compared against the golden human fix; incorrect patches incur penalties. The reviewer then generates a unit test intended to expose faults. The patch and test jointly applied and verified through CI: passing tests reward the submitter, and failing ones reward the reviewer. Both models take turns acting as the submitter and reviewer across rounds, ensuring balanced assessment of their patch and test generation capabilities. Evaluation metrics include binary task success, win rates, and Best@k, k, with
Pcargo is Rust's 's build tool and package manager, used to compile code, manage dependencies, and run
5

---

Published as a conference paper at ICLR 2026
CI logs and model outputs aggregated into structured reports.
Reviewer Test Quality Gates To control evaluation variance and prevent exploitative behavior, reviewer-generated tests must: compile and pass when applied to the golden patch; refrain from modifying production code or existing tests; limit edits in any new test file to a bounded number of lines; avoid sources of nondeterminism; and conform to repository linting and style guidelines. Any violation results in automatic test rejection and forfeiture of the reviewer's reward. See Appendix [ for more details.
3.3 AUGMMETTDD CODE GENERATION
Real-world programming tasks often involve reasoning over large, multi-file codebases where relevant context is scattered across thousands of lines. While prior work (Xia et al. 2024 Xie et al., (202)) has tried address long-context challenges (mainly in Python) via AST-based parsing (Foundation 2023), these approaches typically (i) emphasize a single language. (ii) lack a context packing policy under strict token budgets, and (iii) do not integrate with adversarial CI evaluation. To address these limitations, SWINGARENA introduces multi-language Astrivaal-Augmente Code Generation (RACG) framework that combines static code analysis and dense retrieval to efficiently extract, rank, and pack relevant code snippets to the model.
File Retriever During the file retrieval stage, SWINGARENA employs a coarse-to coarse-to-fine-rainde fine-grained filtering step using lightweight F eeeeriiieeee retrieval method, BM25, to rank source files based on lexical similarity to the problem description, treating the problem as a query and each file document. This step prunes irrelevant files early, narrowing the search space and boosting efficiency for subsequent dense retrieval. Only the top-k most relevant files, as ranked by BM25, are forwarded to the CodeChunker for decomposition into semantically meaningful code chunks. This hierarchical retrieval pipeline- from file-level sparse matching to chunk-level dense reranking- enables SWINGARENA scale effectively to large codebases while preserving high precision in final context selection.
This component utilizes classical sparse
CodeChunker SWINGARENA uses a hierarchical, syntax-aware chunking strategy via CodeChunker to decompose codebases into semantically meaningful units including functions, classes, and blocks, preserving structural integrity and improving retrieval precision. It supports multiple programming danguages-including Rust, Python, C++ and Go torthho language-specific parsing rules. When parsing is unavailable or impractical, the system falls back to regex-based heuristics tailored each language, ensuring robustness across diverse codebases.
CodeReranker Given the large number of candidate chunks in real-world codebases, SWINGARENA employs CodeRerankes to prioritize content effectively within the model's 's context window. uses CodeBERT (Feng et al. 2020) to encode both the problem statement and code chunks into dense vectors, ranking them by cosine similarity to identify the most relevant segments. Beyond naive top-k selection, we incorporate (i) language-aware tie-breaking (favoring definitions over usages), (ii) proximity bias (neighboring chunks near an already selected target receive a small boost), and (iii) de- de-duplication across files to avoid redundant inclusions.
Token Budget-Aware Context Management To optimize token usage under strict token limits, SWINGARENA employs a dynamic token budgeting mechanism that incrementally selects and packs code chunks until the token threshold is reached. Crucially, it adapts chunk granularity based on available context window size- favoring coarser chunks when space permits and switching to finer- grained chunks as the budget tightens. Each included chunk is enriched with metadata (file path, symbol kind, function/class name, line range). This policy is deterministic given the retrieval scores and budget, improving reproducibility across runs.
Variance Control in the Adversarial Arena The adversarial setting introduces interaction-induced variance. We bound variance via: (i) fixed system and user prompts; (ii) a capped number of rounds and retries; (iii) temperature=0 decoding in all primary evaluations, using controlled higher- temperature sampling only in the scaling-law study; (iv) unified CI recipes executed via act with pinned images; and (v) fixed random seeds for retrieval and any sampling components. These choices make our outcomes reproducible while retaining the benefits of interactive stress.
Battle Protocol We denote a single round of adversarial patch and test case generation, evaluation, and scoring between the submitter and reviewer as a battle. Given a problem statement with a buggy program and its description, the submitter generates a patch to fix the issue. Concurrently, the reviewer creates a test case designed not merely to validate, but to strategically challenge the patch's 's correctness by probing for edge cases and potential weaknesses. To foster this adversarial

---

Published as a conference paper at ICLR 2026
nature, the reviewer is provided with contextual hints including which parts of the code were most changed by the patch, and is prompted to design tests that specifically target the logic of the fix. A CI pipeline evaluates both: the patch is checked for compilation and correctness, and the test case is assessed for its ability to expose flaws. The submitter receives a score of +1 for patches passing -1 for any failure. The reviewer receives score of +1 if their
all tests (including the reviewer's), test fails the submitter's s patch, revealing a fault, and -1 if it fails the golden human patch. Models alternate roles across multiple rounds with CI feedback for iterative refinement, simulating dynamic software development.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Baselines We evaluate our method against several strong models: GPT-4o (Achiam et al. 2023).
general-purpose LLM optimized for multimodal reasoning: Claude-3.5 (Anthropic 2024), ) an
instruction-tur tuned model for code reasoning; Gemini-2.0 (Google 2023). multimodal model with robust programming capabilities; and DeepSeek-Ve V3 (Liu al. 2024a), a code-augmented model trained on large-scale software repositories. Besides, we use Cooen..--Cderr-BBB-ne 2024) to do ablation studies.
(Hui et al.
Data Division We collect over 2,300 issues with corresponding solutions from GitHub, covering Rust, Python, Go, and C++. This dataset is divided into two parts: 400 high-quality samples filtered from the full set (100 per language) are used for evaluation, while the remaining data reserved for future community-die across programming languages while ensuring experimental efficiency, design two settings: (1) battle scenario using the 400 selected samples, and (2) an ablation experiment using 100 samples (25 high-quality random samples from each language).
training. To comprchensively evaluate both the submitter and the reviewer
Metries Let T denote the set of tasks and l the number of independent attempts per task (when applicable). We formalize the following metrics.
Best @ k: task is counted as solved if at least one of lc independent generations succeeds. Formally, Best@k T ΣteT 1( S success(t, i)).
Cl pass rate: we report submitter-side and reviewer-side CI check pass rates averaged over tasks. Submitter CI Pass Rate (SPR) averages, for each task, the fraction of submitter-side checks passed by the generated patch (excluding reviewer tests), then averages across tasks. Reviewer CI Pass Rate (RPR) analogously averages the fraction of reviewer-generated tests that pass against the golden patch.
Formally, let Csub (t) denote submitter-side checks for task and Crev (f) reviewer-side checks. Then
SPR
1
Σ LET
1
Σ (ECua(t)
(pass(t,,)),
RPR 1
Σ LE
1 (t)
Σ CEC...
(pass((,, c)}.
(t)l(t)
C.e
Win Rate: the fraction of battles whose final outcome is that the submitter's patch passes all CI checks (including reviewer tests) and agrees with the golden fix. Note that Win Rate is adversarial: higher values may also indicate weaker reviewer tests, so it should be interpreted together with SPR/RPR.
Implementation Details For both patch and test case generation in the arena evaluation, we employ carefully crafted system and user prompts. The system prompt instructs the model to behave as a senior software engineer or test automation expert, while the user prompt provides the issue description, relevant code snippets, and contextual metadata. All model outputs are required to follow structured JSON schema enable automated downstream evaluation.
We set the generation temperature to to ensure deterministic outputs. The maximum number of generated tokens is configured based on the model's context window and the specific requirements of each task. For RACG, we limit the number of retrieved files to and allow a maximum of 16 code chunks per query, where each chunk corresponds to syntactic code chunk. The retrieval agent supports up to 3 retry attempts to handle transient failures or timeouts.
Battle Protocol Configuration: We configure the evaluation rounds through system parameters. In our experiments, we set total of 10 rounds for each battle, where each agent executes 5 rounds in each role (submitter and reviewer). This configuration aims to provide balanced assessment of both
7

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation of Code Submission vs. Test Submission Capabiliitess Among Proprietary LLMs.
Matchup
Submitter
Reviewer
RPR
SPR
Win Rate
GPT-4o vs GPT-4o
GPT-4o
GPT-4o
0.71
0.68
0.97
GPT-4o vs Claude
GPT-4o
Claude
0.65
0.55
0.90
GPT-4o vs Gemini
GPT-4o
Gemini
0.61
0.55
0.94
GPT-4o vs DeepSeek
GPT-4o
DeepSeek
0.61
0,55
0.94
Claude vs GPT-4o
Claude
GPT-4o
0.66
0.55
0.89
Claude vs Claude
Claude
Claude
0.62
0.62
1.00
Claude vs Gemini
Claude
Gemini
0.59
0,55
0.96
Claude vs DeepSeek
Claude
DeepSeek
0.64
0.54
0.90
Gemini vs GPT-4o
Gemini
GPT-4o
0,61
0.55
0.94
Gemini Claude
Gemini
Claude
0.60
0.56
0.96
Gemini vs Gemini
Gemini
Gemini
0.72
0.63
0.91
Gemini vs DeepSeek
Gemini
DeepSeek
0.64
0.64
1.00
DeepSeek vs GPT-4o DeepSeek
GPT-4o
0.60
0.55
0.95
DeepSeek vs Claude DeepSeck vs Gemini DeepSeek vs DeepSeek
DeepSeek
Claude
0.60
0.55
0.95
DeepSeek
Gemini
0.68
0.64
0.96
DeepSeek
DeepSeek
0.70
0.66
0.96
agents" capabilities while maintaining experimental efficiency. The battle terminates after completing all rounds, and the final win rate is computed from cumulative outcomes across rounds.
Fairness and Harmonization For fairness, we harmonize the maximum prompt-plus caaipp--pluu-geeratiio token budget across proprietary models to a common value B and do not exceed even if model supports a larger context window. We log API versions and evaluation dates, apply the same rate limits, and use identical decoding parameters (temperature, top-p). We also record API failures and retries and confirm in Appendix that excluding failed calls does not change rankings. Open-source model results are reported in T
Reproducibility and Artifacts We provide anonymized artifacts (prompts, JSON schemas, scripts, pinned images). The exact evaluation workflow summarized by Algorithm I
4.2
MAIN RESULT
Adversarial Programming Battle Outcomes Table presents comparative analysis of RPR, SPR across proprietary LLMs evaluated using the SWINGARENA adversarial framework. Each model is tested in both self-play scenarios and cross-play scenarios (Claude vs Gemini), alternating roles sub- mitter and reviewer. Several trends emerge: Strong Self-Consistency: All models show high win rates when reviewing their own submissions Claude (1.00), GPT-4o (0.97), Gemini (0.91), DeepSeek (0.96) indicatin strong internal alignment between patch generation and test case generation.
GPT-4o's Aggressive Patching Ad- vantage: GPT-4o achieves win rates 20.90 as a submitter regardless of the reviewer, highlighting its dom- inance in producing adversarially- strong patches. However, its relatively lower RPR/SPR scores (0.65/0.55 vs Claude) suggest variability in overall correctness. DeepSeek and Gemini's Reliability: Although their win rates submitters are slightly lower, DeepSeek and Gemini yield the highest CI pass rates (up to 0.66 and 0.64 respectively). reflecting their strength generating reliably test-passing code. Asymmetry Matchups: Pairwise comparisons reveal minor asymmetries (GPT-4o vs Claude at 0.90 vs Claude vs GPT-4o at 0.89), indicating the reviewer model subtly affects the outcome, likely due to differing review strictness. Key Insight: Overall, GPT-4o excels in assertive patch generation, while DeepSeek and Gemini prioritize correctness and CI stability. The evaluation further underscores the critical yet nuanced role reviewers play in determining adversarial outcomes.
Table 2: Best@3 3 across Models and Languages.
Model
Average
C++
Go
Rust
Python
Gemini
0.57
0.64
0.58
0.51
0.57
DeepSeek
0.59
0.64
0.61
0.58
0.52
GPT-4o
0.57
0.63
0.53
0.56
0.54
Claude
0.55
0.63
0.55
0.52
0.50
Laneuage-Specific Evaluation
8

---

Published as a conference paper at ICLR 2026
In addition to the overall performance compar- ison, we further analyze how each model per- forms across different programming languages.
0.70
0.66
0,67
0.69
0.61
0.62
0.664
As shown in Table 2 DeepSeek achieves the highest average Best@ score (0.59), followed closely by Gemini and GPT-4o (both at 0.57), and Claude (0.55). When broken down by language, all models perform best on C++ and relatively worse on Rust and Python, sug- gesting variation in model proficiency across language-specific problem formulations. No- tably, DeepSeek shows generally strong results
Win Rate
0.60
0.57
0,59
0.55
0.55
0.50
0.52
0.45
0.40
0..43
2
10 Number of Retries (k)
16
Figure 3: Best@k win rate.
across languages, particularly in Rust (0.58) and Go (0.61), suggesting more robust generalization on our tasks. These results indicate that DeepSeek exhibits relatively balanced multi- multi-language code reasoning ability among the evaluated models.
Open-source matchups. Additional results on open-source models summarized in Table
Best@l Sampling for Win Rate We analyze the probability of success when the submitter (Qwen2.5- Moder-7B-Instruce and reviewer (Qwen2.5-Coder (ander-B--Instruct) independently generate up to attempts per role at temperature 0.25. A task counts as successful if at least one attempt yields a passing outcome. This Best@k curve characterizes test-time scaling behavior under our arena protocol.
4.3 ABLATION STUDY ON COMPONENTS OF SWINGARENA
Ablation Study on RACG Table 3 reports ablation results for the RACG module on submitter. The upper section shows results across four programming languages (C++, Python, Go, Rust), comparing model performance with and without RACG. The lower section includes retrieval-based baselines: BM25 and Top-k related retrievals (with incorporating RACG generally improves both Best @ 3 and Win Rate. For instance, in the C++ setting, RACG raises @est@ from 0.38 to 0.42 and Win Rate from 0.77 to 0.84; similar gains are observed in several cases for Python, Rust, and Go. Compared retrieval-only methods, RACG-enhanced approaches often outperform BM25. Top-20 retrieval achieves the strongest baseline result (Best@3 = 0..43, Win Rate 0.73), representing 0.11 improvement in win rate over using BM25 alone (0.62). We acknowledge that our RACG design, particularly the fixed Top-5 file retrieval limit, may act bottleneck for complex issues requiring broader context. As detailed in our failure analysis in Appendix C notable portion of failures can be attributed to retrieval limitations. This suggests that while our RACG serves as a strong baseline, exploring more dynamic retrieval strategies a key avenue for future improvement.
2, 10, 20) followed by reranking. Across languages,
Patch Localization Accuracy Table shows the fraction of queries whose golden-patch file retrieved within the Top-2, Top-10, or Top-20 results under four strategies: lexical file-level BM25 and chunk-level retrieval over Block, Function, and Class units (with hits mapped back to their parent files). Finer granular- ity boosts accuracy-swicching from BM25 to class-level retrieval more than doubles the Top- 10 hit rate (0007%)--4887%
Table 3: RACG Ablation Comparison.
Method
Best@3
Win Rate
C++ w/ RACG
0.42
0.84
C++ w/o RACG
0.38
0.77
Python w/ RACG
0.46
0.84
Python w/o RACG
0.44
0.71
Rust w/ RACG
0.58
0.75
Rust w/o RACG
0.49
0.72
Go w/ RACG
0.45
0.80
Most of the im- provement arises early in the ranking; curves flatten beyond Top- 10, implying that the cor- rect file is usually exposed near the top of the list. BM25 can lag as relies primarily on term overlap, lacking the structural and semantic cues
Go w/o RACG
0.37
0.71
BM25
0.38
0.62
Top-2 Related
0.42
0.69
Top-10 Related
0.43
0.72
Top-20 Related
0.43
0.73
exploited by chunk- based methods. While class-level retrieval generally effective due to its rich contextual information and noise suppression, often exceeds the LLM's context window, limiting its practical utility. To address this, we adopt a block-level reranker, which offers a finer granularity that fits within context limits while still guiding locating the correct patch position.

---

Published as a conference paper at ICLR 2026
4.4 DATA ANALYSIS AND FAILURE PATTERNS
We discuss the data analysis and failure patterns in Appendix C In brief, prompts are much shorter than repository context (code dominates the token budget); token usage across model pairs remains manageable; and finer-grained retrieval substantially improves Top-10 file hit rates over BM25. See Figures and Table for details.
5 CONCLUSION
This paper introduces SWINGARENA, unified framework for evaluating and enhancing LLM- based program repair and test generation under real-world constraints. By modeling interactions between submitter and reviewer agents across multiple languages, it offers holistic benchmarking aligned with practical software workflows. To handle large, diverse code contexts, we propose Retrieval-Augmentt Code Generation (RACG) module combining static analysis. dense retrieval, and token- aware context packing. Experiments across four languages show SWINGARENA provides nuanced insights into model capabilities, revealing trade-offss between patch assertiveness, correctness, and review strictness, thus bringing evaluation closer to real-world scenarios.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. Swe-bench+: Enhanced coding benchmark for Ilms. arXiv preprint arXi::2410.06992, 2024.
Anthropic. Introducing claude 3.5 sonnet, 2024. Utt.pttpss./ww..atttrppp........ 3-aude-3-5-saae
Accessed: 2025-05-12.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint (eXxx:2208.07700 2021.
Syteeance_Seed Seed-coder Let the code model curate data for itself. arXiv preprint arXiv:upcoming. 2025.
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint (10i::2007.03374, 2021.
Daniel Dunsin. Enhancing software quality and efficiency: The role of generative ai in automated code generation and testing. arxiv, 2025.
Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. Out of the bleu: how should we assess quality of the code generation models? Journal of Systems and Software, 203:
111741, 2023.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.
Python Software Foundation. ast - abstract syntax trees. library/ast.c htmm 2023.
Google. Gemini: Our new large language model, December 2023. URL https: /tttss://blog ob Accessed: 2025-05-12.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. Testgeneval: A real world unit test generation and test completion benchmark. arXiv preprint arXiv:2410.00752, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint vrXiv:2310.067700 2023.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arxiv preprint wrXiv22055.06661 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. No need to lift finger anymore? assessing the quality of code generation by chatgpt. IEEE Transactions on Software Engineering, 2024b.
11

---

Published as a conference paper at ICLR 2026
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel LLmy-PPorreet Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint 17X1::2402.19173. 2024.
Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. Code agents are state of the art software testers. In ICML 2024 Workshop LLMs and Cognition, 2024.
Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Kai-Wei Chang, Retrieval augmented code generation and summarization. arXiv preprint arXiv: 108. 1601, 2021.
Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buccholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al. Swe- polybench: A multi-language benchmark for repository level evaluation of coding agents. arXiv preprint wxXvv25544.08703 2025.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389 2009.
Mohammed Latif Siddiq, Simantika Dristi, Joy Saha, and Joanna CS Santos. The fault in our stars: Quality assessment of code generation benchmarks. In 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM), pp. 201-212. IEEE, 2024.
Xinchen Wang, Pengfei Gao, Chao Peng, Ruida Hu, and Cuiyun Gao. Codevisionary: An agent- based framework for evaluating large language models in code generation. arXiv preprint arXiv:2504.13477 2025a.
Yibo Wang, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S Yu, and Chen Xing. Projecttest: project-level unit test generation benchmark and impact of error fixing mechanisms. arXiv preprint arXiv:2502.0655:5 06556,,,,66,,, 2025b.
You Wang, Michael Pradel, and Zhongxin Liu. Are" solved issues" in swe-bench really solved correctly? an empirical study. arXiv preprint a 15223, 2025c.
xAI. Grok-3, https://xx... 2024. Accessed: 2025-05-15.
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.00 01489, 2024.
Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. Swe-fixer: Training open-source lims for effective and efficient github issue resolution. arXiv preprint arXiv:2501.05040, 2025.
Jing Xiong, Chengming Li, Min Yang, Xiping Hu, and Bin Hu. Expression syntax information bot- tleneck for math word problems. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2166-2171, 2022.
Jing Xiong, Zixuan Li, Chuanyang Zheng, Zhijiang Guo, Yichun Yin, Enze Xie, Zhicheng Yang, Qingxing Cao, Haiming Wang, Xiongwei Han, et al. Dq-lore: Dual queries with low rank approximation re-ranking for in-context learning. arXiv preprint arXiv:2310.0295.. 2023a.
Jing Xiong, Jianhao Shen, Ye Yuan, Haiming Wang, Yichun Yin, Zhengying Liu, Lin Li, Zhijiang Guo, Qingxing Cao, Yinya Huang, et al. Trigo: Benchmarking formal mathematical proof reduction for generative language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11594-11633 2023b.
Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Lingpeng Kong, al. Uncomp: Uncertainty-awe -aware compressor for efficient large language model inference. 2024.
aware long-context
Jing Xiong, Qiujiang Chen, Fanghua Ye, Zhongwei Wan, Chuanyang Zheng, Chenyang Zhao, Hui Shen, Alexander Hanbo Li, Chaofan Tao, Haochen Tan, et al. Atts: Asynchronous test-time scaling via conformal prediction. arXiv preprint arXiv:2509. /5/48, 2025a.
12

---

Published as a conference paper at ICLR 2026
Jing Xiong, Jianghan Shen, Chuanyang Zheng, Zhongwei Wan, Chenyang Zhao, Chiwun Yang, Fanghua Ye, Hongxia Yang, Lingpeng Kong, and Ngai Wong. Parallelcomp: Parallel long-context compressor for length extrapolation. arXiv preprint arXhv:2502.14317, 2025b.
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems. 37:50528-5000 506622, 2024.
Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. Assessing the quality of github copilot's code generation. In Proceedings of the 18th international conference predictive models and data analytics in software engineering, pp. 62-71, 2022.
Daoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, Yifan Shi, Wei Liu, Dong Chen, Zongshuai Qi, Hao Yu, Lei Yu, et al. Swe-bench-jaaa: A github issue resolving benchmark for java. arXiv preprint arXiv:2408. 14354, 2024.
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et Multi-swe-bench: A multilingual benchmark for issue resolving. arXiv preprint arXiv:2504.02600. 2025.
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint vrXh::2401.07339, 2024.
Xiangyu Zhang, Yu Zhou, Guang Yang, and Taolue Chen. Syntax-aware retrieval augmented code generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1291-1302, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph F. Gonzalez, and Ion Stoica. Judging alm-as-a-judge with mt-bench and chatbot arena, 2023. URL abs/2330 05685
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang. Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the harrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931 2024.
Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang. Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang. Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.. 10934, 2024.
13

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have taken substantial measures to ensure the reproducibility of our work, see Appendix
STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
We employed a large language model to enhance the manuscript's language. such as improving grammar and phrasing. All research ideas, methods, experiments, analyses, figures/tables, and conclusions were solely developed by the authors, who take full responsibility for the content.
A
MORE EXPERIMENT RESULTS IN SWINGARENA
evaluate more open sourced models wheih are good code generation: Qwen2.5 Coder (Hui et al. 2024) is code-specific large language model from the Qwen family. Seed Coder (ByteDance __eeed 2025) a powerful, transparent, and parameter-efficient family of open-source code models provided by ByteDance Seed. DeepSeek Coder V2 (Zhu al. 2024 is code language model based on the Mixture of Experts (MoE) architecture.
Based the size of the parameters, divided the experimental subjects into two groups: (a) and Seed-Coder-8e IB-Instruct, and (b) Beeen..--oder--ne DeepSeek-Coder-V2 (16B). Table shows the detailed results of RPR, SPR across LLMs using SWINGARENA grouped by the size of parameters.
Swen2.5-Cooder-Inttueee
14B and
In the table, for of reading, abbreviate
Coodee---nrt Qwen2.5, Seed-Coder-
Instruct as Seed, and DeepSeck Coder V2 as DeepSeek.
B DATA FEATURE DISTRIBUTION
(ojos dop sowd sonbor soonbog bog
Seseen Tokin Seneee
1860
7560
0302
Iov Oulia
2000
Grl Raderwer
2000
Cr In
2000
2500
Ravbeeor
Subpettur
Subaettar
Submettur
Subaettur
Figure 4: Token usage heatmap for SWINGARENA. The darker the blue color, the higher the number of request tokens. The taller the bars, the higher the number of response tokens.
B. CLARITY AND DIFFICULTY DISTRIBUTION
Figure 5
presents the distribution of scores across two evaluation ciimensions-clavey and diffi- culty for four programming languages: Go, Python, C++, and Rust. The data has been filtered to exclude incomplete invalid entries, and the results are shown pic charts. Clarity distribution:
Table 4: Evaluation of Code Submission vs. Test Submission Capabilities Among Open Source
LLMs.
Matchup
Submitter
Reviewer
RPR
SPR
Win Rate
Qwen2.5-7B vs Qwen2.5-7B
Qwen2.5-7B
Seed-8B
0.56
0.49
0.87
Qwen2.5-7B vs Seed-8B
Qwen2.5-7B
Secd-8B
0.55
0.48
0.87
Seed-8B vs Seed-8B
Seed-8BB
Qwen2.5-7B
0.61
0.52
0.90
Seed-8B vs Qwen2.5-7B
Seed-8B
Qwen2.5-7B
0.57
0.52
0.89
Qwen2.5-14p 14B Dwen2.5-14BB
Dwen2.5-14BB
DeepSeek
0.58
0.52
0.91
Qwen2.5-14B DeepSeck
Qwen2.5-14p
DeepSeek
0.62
0.54
0.95
DeepSeck vs DeepSeek
DeepScek
Qwen2.5-14BB
0.61
0.55
0.94
DeepSeck vs Qwen2.5-14BB
DeepSeck
Qwen2.5-14PP
0.58
0.95
14

---

Published as a conference paper at ICLR 2026
Go clarity distribution
Python clarity distribution
clarity distribution
Rust clarity distribution
94.7%
93.7%
97.1%
94.2%
Go difficulty distribution
Python difficulty distribution other
Cpp difriculty distribution
Rust difficulty distribution
0.05
other
0.75
0.25
0.55
0.65
other
0.3
0.25
16.
15
8.0%
0.55
16.2%
6.8%
0.3
24.45
0.75
19.3%
0.15
14.7%
14.5%
0.3
13.9%
11.67
0.35
13.7%
12.0%
0%
11.3%
0.35
15.5%
13 13.0KL
12,250
10.5%
0.25
16.8%
0.25
0.35
0.55
0.45
0.45
Figure 5: Clarity and Difficulty Distribution.
Clarity ratings are provided on a two-level scale: 2 (moderately clear) and 3 (very clear). The vast majority of samples across all four languages fall into level 2. For instance, 97.1% Python samples are rated at clarity level 2, followed by Go (93.7%). C++ (94.2%), and Rust (94.7%). ii) Difficulty distribution: Difficulty assessed on quasi-continuous scale, including scores 0.15, 0.25, 0.35, 0.45, 0.55, 0.65, and 0.75. The distributions differ across languages. Python shows peak at 0.65 (24.4%), suggesting a considerable portion of users perceived higher difficulty. Go's difficulty ratings more balanced, with notable frequencies at 0.25, 0.35, 0.45, and 0.65. C++ shows concentration at 0.25 (19.3%) and 0.35 (16.8%). Rust displays a more evenly spread distribution with significant percentages at 0.65 (16.1% () and 0.35 (14.7%). iii) Other values: The category labeled as "other" refers to scores that do not fall within the main set of defined intervals, possibly due to rounding or uncertainty in the rating process. These values account for approximately 6% to 13% depending on the language. iv) Data filtering: All statistics are derived from filtered data to ensure consistency and reliability by excluding anomalous or incomplete records.
B.2 DATA LENGTH DISTRIBUTION
Figure and Figure 6b present the distribution of problem statements and patch length in different languages. By categorizing problem statements and patches into various buckets, all distributions are skewed the left. The most frequent lengths of problem statements and patches are concentrated within the 0-200 length range.
Across all languages, problem statement lengths show a sharp decline in frequency as the length increases. Python and Rust exhibit particularly steep drops, indicating that most problem descriptions in these languages tend be concise. Notably, the Python problem statements show the highest frequency in the shortest bucket (0-100), surpassing 250 occurrences, suggesting that Python prob- lems are often defined with minimal text. On the other hand, C++ and Rust show slightly broader distribution with longer tails, implying that problem statements in these languages occasionally require more verbosity or complexity description.
For patch length distributions (Figure 6b). a similar trend is observed. The majority of patches are short, with the highest frequencies in the 0-100 bucket across all languages. Python again shows the highest peak, indicating high volume of short patches, which may reflect Python's expressive syntax and brevity in fixing issues. Rust, while still skewed left, displays a longer tail compared to the others, suggesting that Rust patches might involve more lines of code more complex fixes. C++ also presents a longer tail, consistent with the language's 's verbosity and potential complexity in code modification.
15

---

Published as a conference paper at ICLR 2026
Go problem statement length disributton Python probtem state ment length distrburtin problem ment lengtth distibutiion
Rust prob statemenn lenghh diar
300
250-
202
200
Count
200-
Count
350
63
150
appspin
(a) Problem statement length distribution.
Go longth distribution
Python distribution
Cpp patch langth distribution
Rust patch length distribution
100
3t 8 B
8
Count
60-
Count
Countt
S
Count
8
20
20
20
(b) Patch length distribution.
Figure 6: Length distributions in different languages.
In summary, these distributions indicate that both problem statements and code patches are predomi- nantly short in length across all languages. However, languages like Rust and C++ show a slightly more distributed range, potentially reflecting their syntactic or structural characteristies that lead to longer problem descriptions or patches.
C DATA ANALYSIS
Length and Token Distributions We detail problem-statement and patch length dis- tributions, and token usage across model pairs/languages. Prompts much shorter than repository context (code dominates token budgets). See Appendix and Figure 4
Figure 7 shows the average lengths of problem statements and patches across four programming languages: Go, Python, C++, and Rust. i) Python features the longest problem statements (2369.9 tokens) and relatively long patches (852.6 tokens), suggesting detailed task descrip- tions and substantial code changes. ii) Go exhibits concise problem statements (286.7 tokens) and patches (287.6 to- kens), reflecting a more minimalistic style. iii) C++ and Rust present short problem statements (205-216 tokens) but longer patches (631.2 for C++, 1059.9 for Rust), indi- cating that even brief prompts require complex code modifications. Although problem statements and patches vary in length, their average sizes still significantly smaller than the total input code context (often spanning tens of thousands of tokens), highlighting that the primary bottleneck in input context window length stems from the code itself. Given the high invocation costs of proprietary models, efficiently managing the token budget becomes key challenge. More detailed statistics about the data can
Aeeenge Length
1039
Python
Figure 7: Average problem statement and patch length of different languages. When retrieving 20 files, LLMs are ex- posed to input code contexts with aver- age lengths 8,232 tokens (Go), 29,258 (Python), 54,483 (C++), and 36,875 (Rust), which exceed typical visualiza- tion scales thus not shown directly in the figure.
be found in Appendix(B B
Token Usage We summarize token consumption across model pairs and languages in Appendix Figure
16

---

Published as a conference paper at ICLR 2026
C.I STABILITY, REVIEWER EFFECTS, AND FAILURE TYPOLOGY
Stability across retries We report Best@k (k=1,3,5) and its variance across three random seeds for the controlled sampling study (temperature 0.25). Variances are modest (median std 0.02) and decrease with larger k, indicating that our capped-retry protocol effectively stabilizes outcomes without masking difficulty.
Reviewer effects Cross-play reveals systematic reviewer strictness. For each submitter, we com- pute CI pass deltas relative to self-play. Gemini and DeepSeek exhibit tighter reviewer distri- butions (median ASPR in [-0.02, () than GPT-4o (wider tails), consistent with our qualita- tive observation that GPT-4o is more permissive on style/format but more aggressive on patching. These effects explain mild asymmetries in pair- wise win rates and motivate reporting both RPR and SPR.
Table 5: File hit rates of different retrieval methods. Each value gives the fraction of queries whose correct file appears within the Top-2, Top-10, or Top-20 retrieved results.
Method
Top-2
Top-10
Top-20
BM25
0.182
0.207
0.195
Block Chunks
0.207
0.341
0.329
Function Chunks
0.277
0.371
0.368
Class Chunks
0.363
0.487
0.461
Failure typology Manual inspection of 100 failed trials (random, stratified by language) yields four
dominant classes: (F1) incomplete patching across files (31%), (F2) test fragility or flakiness (19%), (F3) style/security gate violations (24%). (F4) mismatch between retrieved contexi and true locus (26%). Correlating with data features (Appendix B), (FI) and (F4) rise with dispersed logic (longer tails in patch length) and lower clarity=2 samples; (F3) concentrates in repos with strict linters.
Failure Pattern Analysis To understand the limitations of current LLMs in software engineering tasks, we conduct a systematic analysis of failure patterns across our adversarial evaluation frame- work. Our investigation reveals four dominant failure modes that expose fundamental challenges in automated code generation and testing.
Cross-File Consistency Challenges (F1, 3)) The most prevalent failure mode involves incom- plete fixes that span multiple files. Models demonstrate strong local reasoning but struggle with distributed codebases, often fixing the primary symptom while neglecting related components. This manifests as interface-ing innrrtaatiimlleenttttiinnn mismatches, where implementation changes are not reflected in corresponding header files or API definitions. Additionally, models frequently fail to propagate changes through dependency chains, leading to cascading failures in dependent modules. The corre- lation between this failure mode and patch complexity (Spearman p = 0.67) suggests that current models lack the architectural reasoning capabilities required for multi-file software modifications.
Don-Deterministic Test Behaviors (F2, 19%): A significant portion of failures stems from test instability rather than code correctness issues. Models generate tests that are sensitive to timing, resource availability, or environmental conditions. Race conditions in concurrent programming scenarios are particularly problematic, with Python and Go showing higher susceptibility (23% and 21 % respectively) compared to systems languages like C++ (15%) and Rust (17%). This pattern reflects the challenge of generating robust tests in dynamic execution environments, highlighting the need for models to understand concurrency models and resource management principles.
Von-Functional Requirement Violations (F3, 24%).: ): Models often produce functionally correct solutions that fail to meet quality standards. Style violations, security vulnerabilities, and performance anti-patterns are common, particularly in repositories with strict CI policies (up 35% failure rate in high-security projects). This suggests a disconnect between functional correctness and software engineering best practices in current model training. Interestingly, newer models show improved awareness of modern security practices, indicating that training data recency plays a crucial role in non-functional requirement compliance.
Context Retrieval Limitations (F4, 26%)) Failures attributable to RACG limitations reveal fun- damental challenges in semantic code understanding. BM25-based retrieval often finds lexically similar but semantically irrelevant code, leading inappropriate fix applications. Cross-language dependencies and legacy code patterns pose particular challenges, with repository age showing strong correlation with retrieval failure rates (Spearman م 0.58). This suggests that current retrieval methods struggle with codebases that deviate from modern programming paradigms or involve complex polyglot architectures.
17

---

Published as a conference paper at ICLR 2026
Our analysis reveals distinct failure signatures across evaluated models. GPT-4o's aggressive patching approach results in higher cross-file consistency failures (38%), while Claude-3.5's focus functional correctness leads to more style violations (28%). DeepSeek-Ve V3, despite showing the lowest overall failure rate, exhibits higher context retrieval failures (32%), reflecting its dependency high-quality input context. Gemini-2.0 demonstrates balanced failure distribution but shows particular vulnerability in concurrent programming scenarios (25% F2 failures).
Model-Specific Error Profiles:
These findings suggest that advancing LLM capabilities in software engineering requires improve- ments architectural reasoning, semantic code understanding, and integration of con-functional requirements into the generation process.
Our systematic investigation of 100 failed trials reveals four dominant failure modes that highlight important limitations in current LLM capabilities for software engineering: cross-file consistency challenges (3)%) non-deteriinistic test behaviors (19%), non-functional requirement violations (24%), and context retrieval limitations (26%). These patterns reveal model- specific error profiles: GPT-4o's aggressive patching leads to higher cross-file failures, while Claude- 3.5 prioritizes functional correctness over style compliance. The analysis provides actionable insights for advancing architectural reasoning and semantic code understanding in future model development. See Appendix C for comprehensive failure pattern analysis and model-specific error profiles.
Failure Pattern Analysis:
RACG ablation extensions Beyond T we compare RACG with: (B1) BM25-only file retrieval; (B2) Top-k related examples without syntax-aware chunking; (B3) dense-only reranking over raw lines. RACG improves patch localization Top-10 10 by 7-15 points across languages and reduces token footprint by 12-18% equal Best@3, 3, supporting the hypothesis that structure-aware packing, not just better retrieval scores, contributes to gains under context constraints.
Algorithm Eyaluation of Model Performance SwingArena
Task instances, Patch Agent, Test Agent, System 2: Output: Model performance scores for Patch Agent and Test Agent
Input:
3: for each task instance
4: Initialize patch and test agents
5: Load problem statement, bug report, and code context
6: Patch Agent generates candidate patch
7: Validate patch using CI pipeline
8:
if Patch fails CI checks then
9:
Patch Agent loses point
else
11:
Patch Agent passes CI check
end if
13:
Test Agent generates test case
Validate test case using CI pipeline
15:
if Test fails to expose meaningful flaw then
16:
Test Agent loses point
17:
else
18:
Test Agent passes validation
19:
end if
20:
Apply patch and test case together in CI pipeline Validate integration using CI checks
21:
22:
if Integration passes then
Patch Agent earns point
24: else
Test Agent earns point
26: end if
27: end for
28: Reverse roles and repeat process
29: Aggregate points and generate final scores
30: Output: Final scores for Patch Agent and Test Agent
18

---

Published as a conference paper at ICLR 2026
D LLM EVALUATION WORKFLOW IN SWINGARENA
Overview, The evaluation of large language models (LLMs) within the SwingArena framework is structured to simulate realistic software engineering tasks and workflows. It employs an adversarial, dual-agent setting where one L.LM acts as a patch agent (fixing code) and the other as a test ageni (generating test cases). See Algorithm [ for more details.
D.1 ADDITIONAL DATA ANALYSIS DETAILS
Figure 6 summarizes problem-stateee statement and patch lengths across Go, Python, C++, and Rust, In the main text, we noted: (i) Python features the longest problem statements (2369.9 tokens) and relatively long patches (852.6 tokens), suggesting detailed task descriptions and substantial code changes; (ii) Go exhibits concise problem statements (286.7 tokens) and patches (287.6 6. tokens), reflecting more minimalistic style; (iii) C++ and Rust present short problem statements (205-216 tokens) but longer patches (631.2 for C++, 1059.9 for Rust), indicating that even brief prompts can require complex code modifications. Although problem statements and patches vary in length, their average sizes still significantly smaller than the total input code context (often spanning tens of thousands of tokens), highlighting that the primary bottleneck in input context window length stems from the code itself.
E ANNOTATOR DEMOGRAPHICS
To ensure the accuracy and reliability of our benchmark annotations, we assemble a quali- fied annotation team composed of 12 individuals with diverse backgrounds in computer science and software engineering. Specifically, the team consists of eight Ph.D. students, two Master's students, one undergraduate student, and one as- sistant professor. All annotators hold academic backgrounds computer science, with research or industry experience in areas including soft- ware engineering, natural L. language process- ing.
Table 6: File hit rates of different retrieval methods. Each value gives the fraction of queries whose correct file appears within the Top-2, Top- 10, or Top-20 retrieved results.
Method
Top-2
Top-10
Top-20
BM25
0.182
0.207
0.195
Block Chunks
0.207
0.341
0.329
Function Chunks
0.277
0.371
0.368
Class Chunks
0.363
0.487
0.461
Among them, the eight Ph.D. students specialize in topics including systems, formal theorem proving. software engineering, computer networks, and mobile computing. The two Masters have completed rigorous coursework algorithms, compilers, and system, and actively contribute to open-source software projects. The undergraduate students focus research in program synthesis. Additionally, one annotator is an experienced senior software engineer with over ten years of practical experience in CI/CD workflows and large-scale software engineering.
This annotation team was responsible for validating the dataset annotations, including both quality checking and difficulty estimation of the data. Their combined expertise ensured the benchmark reflects realistic developer behaviors and professional software engineering standards.
F LIMITATION
Despite its advancements in creating more realistic evaluation setting, SWINGARENA has several limitations.
RACG Module Limitations. The effectiveness of the proposed Retrieval Augmmette Code Gen- eration (RACG) system, crucial for handling long contexts, is dependent on the performance of its constituent parts (BM25 retrieval, syntax-aware chunking, dense reranking). While designed for scalability and precision, its ability to retrieve the most relevant context may degrade with extremely large. poorly structured, or highly domain-specific codebases not adequately represented in the training data of the dense reranker.
19

---

Published as a conference paper at ICLR 2026
Context Window Constraints. We acknowledge that limiting the retrieval context to 5 files with 16 chunks represents a practical trade-off between model context window, inference cost, and information coverage. This fixed-size retrieval window may fail to capture all relevant context needed to solve problems in extremely large monorepos or projects with poorly structured code. To better understand this limitation's impact, conducted systematic analysis of RACG's failure cases on our benchmark. We found that in approximately 15% of failure cases, model errors could be attributed the retrieval phase failing find critical context. These cases typically exhibit two characteristics: (1) fixing the problem requires coordinated modifications across more than files; (2) related code logic is distributed loosely without direct textual associations, making difficult for the BM25 algorithm to discover. Despite these limitations, our experimental results (as shown in Table 3 demonstrate that even this relatively simple retrieval strategy provides significant performance improvements, indicating that in many real-world scenarios, critical information is relatively concentrated.
Computational Overhead. The computational overhead of simulating full CI pipelines iteratively via act, coupled with the RACG process, means that evaluations are significantly more resource-intensee and time-consuming than static benchmarks like HumanEval or MBPP, potentially limiting the scale and frequency of testing across a vast array of models or tasks.
Future Directions. We believe overcoming these limitations is key focus for future work. We propose several promising directions including iterative retrieval (where models can dynamically initiate new retrieval requests based existing context), hierarchical retrieval croarse-grained file- level retrieval followed by fine-grained code block-level retrieval within relevant files), and hybrid retrieval (combining RACG's sparse retrieval with structured retrieval based on code graphs, including RepoGraph-like methods), all of which we are actively exploring.
G BROADER IMPACT
Positive Social Impact
SWINGARENA could accelerate the development and evaluation of LLMs for complex software engineering tasks by improving code generation fidelity, automating debugging, and providing sophisticated conversational assistance. This could lead increased productivity for developers, lower the barrier to entry for aspiring programmers, and enable the creation of more complex and innovative software applications. The ability of LLMs to effectively navigate and modify large codebases, as tested by SwingArena's long-context reasoning challenges, could be particularly beneficial in maintaining and evolving legacy systems. Furthermore, the automated detection and repair of defects within a simulated CI pipeline could lead to more robust and secure software, ultimately benefiting end-users.
Negative Social Impact
Improved code generation and debugging capabilities could be exploited for malicious purposes in- cluding generating highly effective malware or developing sophisticated cyberattack tools, potentially facilitating disinformation campaigns. Fairness another concern, as biases present the training data could be reflected in the generated code or debugging suggestions, perpetuating or amplifying existing socictal biases in software applications. Privacy considcrations arise from the potential for LLMs trained vast code datasets to inadvertently expose sensitive or proprictary information, and from the risks associated with sharing private codebase details with external models or services during development. Finally, security risks are introduced, including the possibility of poisoned training data injecting vulnerabilities into generated code or the exploitation to compromise software security. While Swing helps identify some weaknesses in realistic settings, continuous effort is needed to address these risks in the development and deployment of LLMs for software engineering.
flaws within the LLMs themselves
20

---

Published as a conference paper at ICLR 2026
H PROMPTS USED IN SWINGARENA
H.1 PATCH GENERATION PROMPTS
System Prompt for Patch Generation
You are an AI Senior Full-Stack Engineer specialized GitHub issue triage and bug fixing. You should only generate the fixed code, without any other text or ma markdown formatting.
System Prompt for Test Generation
You are AI Test Automation Engineer specializing in generating unit tests. You should only generate the test code, without any other text or markdown formatting.
Test Generation Prompt
You are required to develop unit tests for the specified code and its fix.
The issue details: [issue]
The code snippet: [code snippet]
The fixed code: [patch]
The test case sample: [sample]
Please provide the complete code without any explanati ma markdown.
Test-Only Evaluation Prompt
You are an expert code reviewer. Your task is to evaluate if patch passes the provided test case. You will be given:
1. test
2. A patch that aims to pass the test
Carefully analyze if the patch implementation correctly addresses the requirements outlined in the test case.
Provide a detailed led reasoning for your conclusion.
[Response format
B. 1]
21

---

Published as a conference paper at ICLR 2026
Golden Patch Comparison Prompt
You are an expert code reviewer. Your task is to evaluate if patch correctly solves a given problem based on:
1. The problem statement
2. test case
3. reference "golden" patch known to correctly solve the problem
Compare the candidate patch with the golden patch to determine they are functionally equivalent in terms of solving the problem and passing the test case.
Provide detailed reasoning for your conclusion.
[Response format same as B.
22

---

Published as a conference paper at ICLR 2026
H.2 TEST AGENT PROMPTS
Problem and Test Evaluation Prompts
You are a senior software engineer with over 10 years of solid experience in rust, epp, python, and go, You possess a deep understanding of these languages and their standard libraries, along with a strong sense of problem difficulty.
Your task is to evaluate the difficulty and clarity of a ceding problem from GitHub repository, given its "Problem Statement" and "Code Changes". You need to consider the following factors:
1. Clarity and complexity of the problem description: Is the problem geal, input, output, and constraints clearly defined? Are there any ambiguities missing critical details? 1s the problem's legic inherently complex?
2. Scope and depth of code changes required to the whole codebase: Does the modification involve single file/function or multiple modules? Dees it require understanding interactions between different parts the codcbase? What the overall amount of code change? Docs it impact the system's architecture?
3. Number of technical concepts that need to be understood: What specific programming language features, libraries, algorithms, design patterns, or domain-specitic knowledge are zequired to solve this problem? How complex are these concepts?
4. Potential edge cases and error handling requi Does the problem statement mention any specific edge cases error conditions ions to consider? Does the code change require adding or modi fying error handling logic? How complex are these edge cases
Based on these factors, you will provide Clarity Score and Difficulty Score with detailed explanations.
Here is the problem statement and code changes:
Problem em Statement:
(problem statement
Code Changes:
[patch]
First, provide your judgment of the Clarity Scoring (0, of the problem, along with your explanation:
0 (Invalid): Statement is incomprchensible or code changes are unrelated.
fSignificant Ambiguities): valid but lacks critical details including no input/outpur format.
Valid, clear, but minor details missing including edge cases not specified.
(Mostly Clear):
3 (Comprchensive); Valid, clear, with detailed requirements and examples.
23

---

Published as a conference paper at ICLR 2026
I'hen, provide a difficulty score between 0.0 and 1.0, along with your explanation:
0.0-0.2: Very easy, requires only basic code modi fications including fixing typo or changing constant.
0.2-0.4: Easy, requires understanding some code logic and making simple function statement modification. including fixing a simple bug or adding basic feature.
0.4-0.6: Medium, requi res understanding multiple concepts and making complex modi iedifications across several files, potentially involving scre edge case handl ing including impl empntinn new module with moderate complexity.
0.6-0.8: Hard, requires deep understanding of the codebase architecture and complex modi fications with signi ficant impact, involving handling numerous edge cases and potential performance considerations ions including refactoring a core component or implementing complex algorithm.
0.8-1.0: Very hard, requires acvanced technical knowledge, extensive experience, and tackling highly challenging problems with intricate logic, potentially involving system-level considerations or complex domain-specific knowledge including implementing new distributed consensus protocol.
Please return your response in the fellowing structured format:
<clarity score>integer between 0 and 3</clarity score>
<clarity explanation>You explanation for the clarity sccre.</ clarity explanation
<difficulty>float between 0.00 and 1.00</difficulty
<difficulty explamation>>our explanation for the difficulty score .</difficulty explanation
All prompts in framework designed with several key principles in mind: i) Clarity of Purpose: Each prompt clearly defines its specific role and expected outputs. ii) Structured Output: JSON- based response formats ensure consistent and parseable outputs. iii) Confidence Levels: A five-level confidence scale VERY_LOW to VERY HIGH enables nuanced assessment of evaluation reliability, iv) Comprehensive Coverage: The combination of different prompt types ensures thorough evaluation from multiple perspectives
The confidence levels used throughout the evaluation process are carefully defined to ensure consistent VERY_HIGH:
and meaningful assessments: i) VERY HIGH: Complete certainty with no doubts. iī) HIGH: Strong confidence with only trivial uncertainties. iii) MEDIUM: Reasonable confidence with minor doubts. iv) LOW: Significant uncertainties present. VERY LOW: Insufficient information for definitive judgment.
I
STEP RESULTS
We attach more phased inputs and outputs to demonstrate our work. It is important to note that we are displaying processed, well-formatted data. The actual data processed by our system JSON format.
Submitter Input
24

---

Published as a conference paper at ICLR 2026
A real sample of submitter input to the agent.
"name": ceatcch_eeeeattrrr
" "description"; "Analyze and medify code to resolve issues while preserving functionality. You should use codc_editor to process the Intput field information, You should use "'json... un to wrap the code_editor output.",
"arramters":
"type": "object""
"properties":
"reascning_trase
"type": "string",
"description": "Step-by-step analysis of the issue, explanation the root cause, and justification for the proposed solution. Do not use any markdown formatting."
code prode_edits";
"type": "array"
"description"; "List of specific code modi fications required to resolve the issue", "items":
"type": "object", "properties";
"file":
"type": "string", "description": "Relative path to the file that contains code requiring modification"
b
"code_te bocdde_to_bemmdifiiee
"type"; "string",
"description": "Exact code segment that needs be changed (must natch portion of the original file)"
"codd_eedi @oode__dited":
"type"; "string",
"description": "Improved version of the code segment that fixes the issue while maintaining compatibility with surrounding code"
"required;; "file",
"code_to hacee_tto_____eeeeeeeeeeeeeeeeee be_modified", "code_edited"
"required";
seaascning_trace "code_edits"
input":
"issue": "client2: pulling nen-existent model prints duplicate \"hot found\" error message\al41 what is the issue7\n\nFrom emxyng \n\n "'nn ollama run nonexistent tnp manifest model \" nonexistent not found...",
"original_code"; "Key relevant code chunks:\mmnnwwer relevance chunk Lines: 1733-1 & (SBII &
1**: File: 1lana/llama.
750%mln'* defined
(STBI_ NO_BPP) & defined (STBI & defined 8THI_NO_TGA) as defined (STRI_NO_G1HH ss defined (STRI_NO_P10) defined L_THH__N_PPPPP \n// nothing\ntelse ...",
25

---

Published as a conference paper at ICLR 2026
"file File_ppth": oaarrr//oeennn////ooo///////m//a server/routes.ge",
path";
servor/sched.gg",,
1mm/se/7e8r.99""
#1
Submitter Output
A real sample of submitter output from agent.
" "reasoning_tras trace";; "The issue arises because the error message is being printed twice. This happens in multiple files where the same error nandling logic is used. To fix this, we need to identify the common error handling logic and ensure that the error message is printed only once.", cooeeedits";
"file": "server/sched.ge "code to be modified"; "err fmt. (rroff this model may be incempatible with your version of ollama. If you previously pulled this model, tzy updating it by running 'ollama pull $s \"
"file"; "eererr/iteemm//rrrrssrtrrrvvrrr
"code_ to be modified"; "s. "s.Logger. Logättrs (E. (r.Context (), level, \" attp\", \nltitlterraatz, // report first in line to make it easy to find/o/iitttttt/ TODO (bmizerany) Write test to that are logginginititkt// all of this correctly. ..."
Reviewer Input
A real sample of reviewer input to agent.
"name": frest_generatorr"
"description": "You are an Test Automation Engineer specislizing in generating comprehensive unit tests. Your task is to analyze the provided code and create effective test cases that verify the functionality and edge cases. In input field, it includes the issue description (issue), the code snippet (original_code), related file path (file_path), and the patch generated by the generator agent (generated_ patch). You should provide test cases for the patch. Yatch.You should use "Jjson... should only propose less than 10 test cases.", "parameters";
to wrap your JSON output. You
"type": "object",
"properties";
"reasoning_ttrae"" {
"type": "string",
26

---

Published as a conference paper at ICLR 2026
"description"; "Step-by-step analysis of the code, explanation what needs tested, and justification for the test cases. Do not use any markdown formatting."
ttest_cases";
"type": "array",
"description". "List test cases to verify the functionality the code. For each test case, you should provide unique file name.",
"items":
"type": "object", "properties";
"file":
"type": "string",
"description": "Relative path the test file where the test case should added. You should not use same file name with other test cases."
"test_name"
"type": "string", "description": "Descriptive name of the test case."
"tess_code""
"type": "string",
"description": "Complete test code including setup, execution, and assertions"
"test_desctt dest__descrpttion:
"type": "string",
"description": "Brief description what the test case verifies"
"required"; [ "file", "test_name" "test_code", "test_descciiption"
"required";
"reasoning_trace"" "test_cases"
"input":
"issue": "client2: pulling ncn-existent model prints duplicate "not found)" error messagelatt What is the issue7\n\nFron lmxyng \n\n ' ollama run nonexistent Anpulling mani fest feat model {" nonexistent not found\nBrror: model nonexiistent! not found), nexit status 1 /n error gets printed twice. AnAnThis the behavior without the flag ....,
"original_code"" "Key relevant code chunks:
relevance chunk 1*+:\n- File: ilama/llama.cpe cptp.oooo/////bob..oooo..m..ob.. image. hhn- hh'n- Lines: 1733-1 " defined (STBI_NO_JPPS) & (S
& (STBI_ STBI_NO_TGA) & (STBI_
&& defined & (S &
(STRI_
"file_path":
sserrerr/nteernal//lieeet
"server/routes.go"
"server/sched.go",
27

---

Published as a conference paper at ICLR 2026
" "1im/server.go",
serrerr/inieemaa/m//m//////ppppp
generateed_pachh"
"server/sched.go" "ditt --git s/server/sched.or b/server/sched.gg nindex f397879. 97e774f 100644 wtw.rver/accee..coee.... go b/ server/sched.go
-425, +425,8 8 ea func (s 4Scheduler) load req *LlmRequest, f *ggml Sggml.GMM,, spus discover.@pulnnfoll // show a generalized compatibility erroz until there is better way toln itit// check for model compatibilityle \tltif errors. Irrors.Is gww..rruhssspprteeerro... err. (tr.Error(), \"failed to load modell") ('a-\e\eNtorr fmt. Errorf(\"SV this model may be incompatible with your version of cllama. ..."
\t
|I www.ings.onnaai
Reviewer Output
real sample of reviewer output from agent.
trace": "The issue is that the error message is being printed twice. The patch removes the duplicate error message by modifying the error assignment in the 'load' function of 'sched.go To verify this, need to test that the error message is now printed only once.", "test_cases";
"file": "server/sched
"test_name"; "Iettiaddiaaxxiirrattdddeeeeds
"test_code"; "package server\nimpon (enierrrors) "\n\ "testingt "in) V nfnnc aettlaaaddoonnwistttteModllllll
*testing. ('n mockerr :- Nerors.Newllf (\"nodel not found)") Vnreg :- &Llnkequest (model: & 6Scheduler() nil A.ttEErrez
Model \"nonexistent>" ()
expectedMsg := \"this := s.load (req, nil, nil) if
(E"xxpected error, got An))r
model may be incompatible with your version of 01lama. If you previously pulled this model, try updating it by running cllama pull nonexistent "\niif Error != expectedMag (Ant. Errorf (\ "Expected error message Vres got V$sV\", expectedMsg, err.Error (n))
"test_descrippion" "Tests that the error message is printed only once when attempting to load non-exi model."
"file": "server/sched test. "test_name"" "TestloodExiistii wtwiinggnooeeeeeoeerroooo "test_code"; "package server\nimport ('n\"testing'" (An) Anfune
rrstocaas:ittttngoooeeeeeorr (t *testing.T) (An reg :=
LimRequest
Scheduler[] Anf := (Shorl(ShorNName
/mokeequuettmodi::: moodel:
(rexisting'"])nn :=
gpus := discover.
GpuInfoList() () Anerr := s.load f, gpus) Anis err != nil (Ant. Errorf "Dnexpected error: Sv)", err) An)",
"test_deseription" "Tests that no error is returned when attempting to load an existing model."
28

---

Published as a conference paper at ICLR 2026
CI Result
We summarized CI outputs from act. Noticed that these are well-formatted since SwingArena processes json format.
[test/test-]] [test/test-1] docker pull f flatform- username ForcePull=tre
Run Set up job
|test/test-1]
CC='ges'
[test/test-1]
Success Main Setup Go
[test/test-1]
::addpath::
[test/test-1]
Run Main check that 'go generate' is clean go: downloading httttu..coommpf1////////// vl. 7.0
[test/test-1]
[test/test-1]
Success Main check that 'go generate' is clean
[test/test-1]
Run Main go test
[test/test-1]
go: downloading github. githubb.cmm//avegaggg-ggeeee v1.1.1
[test/test-1]
ok github.
[no test files] 0.015s
[test/test-]]
[test/test-1]
0.004s
[test/test-1]
Success Main go test
[test/test-1]
Run Complete job
[test/test-1]
Cleaning up container for job test
[test/test-1]
Success Complete job
[test/test-1]
Job succeeded
29


Reviews:
[{"review_id": "NtIn6ccdi7", "reviewer": "Reviewer_cSPp", "summary": "This paper proposes SWINGARENA, a dynamic adversarial evaluation framework for real software engineering tasks that pairs LLMs as submitters and reviewers who generate and test patches, respectively, through a continuous integration (CI) pipeline. SWINGARENA also leverages retrieval augmentation to retrieve the most relevant context from code bases for a variety of languages (C++, Python, Rust, and Go), spanning 400 issues and surfacing new problems, and also showing behavioral differences in models as patch generators and validators.", "questions": "What does “PK-style dual-role evaluation” on line 78 mean?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Introduces iterative and adversarial evaluation that incorporates software engineering in CI development scenarios and goes beyond mere unit tests.\n2. Propose a multi-language long context retrieval (RACG) pipeline for fetching relevant code context that combines syntax-aware chunking, dense reranking, and token-budget–aware packing across C++, Python, Rust, and Go.\n3. A dataset of 2300 real GitHub issues with 400 high-quality issues (100 per language) selected for evaluation.\n4. Benchmarking of several state-of-the-art open and closed-source LLMs.", "weaknesses": "1. The win rates of all models are very close to each other (almost every model gets 0.9 or above), which makes me question the utility of this benchmark in terms of model selection.\n2. Best@k values for all the models are also very close to each other, which makes it hard to judge which model is better. \n3. Retriever doesn’t seem to boost performance much in Table 3, especially for Best@3 for Python and C++. The authors also acknowledge a weakness in how many relevant files can be included (only 5 files) and that context retrieval leads to the most failures (26% according to Appendix C1). This makes the benchmark less reliable since the LLMs cannot perform optimally under these limitations. \n4. Missing citations for some relevant work, like CodeRAGBench [1], CrossCodeEval [2], and RepoCoder [3].\n\n[1] Wang, Zora Zhiruo, et al. \"Coderag-bench: Can retrieval augment code generation?.\" arXiv preprint arXiv:2406.14497 (2024).  \n[2] Ding, Yangruibo, et al. \"Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion.\" Advances in Neural Information Processing Systems 36 (2023): 46701-46723.  \n[3] Zhang, Fengji, et al. \"Repocoder: Repository-level code completion through iterative retrieval and generation.\" arXiv preprint arXiv:2303.12570 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P32WEUQtCL", "reviewer": "Reviewer_pKUf", "summary": "The paper introduces SWINGARENA, a framework for evaluating LLMs on software development tasks. Unlike static benchmarks, SWINGARENA simulates a collaborative workflow by pairing LLMs into \"submitter\" (patch generator) and \"reviewer\" (test case generator) roles. The evaluation is grounded in real-world GitHub issues and utilizes repository-native CI pipelines for verification. To manage the long-context nature of large codebases, the framework includes an RACG module.", "questions": "1. The results for GPT-4o seem contradictory. The text claims it has \"relatively lower RPR/SPR scores\", but also a \"dominance in producing adversarially-strong patches\" based on high win rates ($\\ge0.90$). Why a model have a low SPR but a high Win Rate?\n\n2. Given that the \"Win Rate\" metric is confounded by the reviewer's strength, would it not be more sound to evaluate the submitter's patch directly against the golden patch and the full, human-written test suite?\n\n3. In the RACG ablation (Table 3), what does the \"Top-k Related\" retrieval baseline consist of?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "present a new dataset of 2,300 CI-filtered issues across C++, Python, Rust, and Go, and provide experimental results for several proprietary and open-source models.", "weaknesses": "1. The paper's main \"battle\" metric, the Win Rate, is severely confounded. The \"Win Rate\" is defined as the submitter's patch passing all CI checks, including the reviewer's generated test. As the authors correctly note, \"higher values may also indicate weaker reviewer tests\". This confounding variable makes it impossible to draw clear conclusions about a submitter's absolute capabilities.\n\n2. The paper introduces a complex, multi-stage RACG module but explicitly states it is a \"baseline rather than a standalone algorithmic contribution\". The ablation study in Table 3  fails to justify its necessity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SVD31lxNHq", "reviewer": "Reviewer_kJ84", "summary": "This paper introduces an adversarial evaluation framework for LLMs in software engineering, arguing that existing benchmarks like HumanEval are too simplistic by focusing only on single-function unit tests. SwingArena aims to model real-world workflows by using actual long-context GitHub issues, executing generated patches against the full CI pipeline, and implementing an adversarial battle protocol. In this protocol, one LLM acts as a Submitter generating a patch, while another acts as a Reviewer generating new tests specifically to break the submitted patch. The authors find that evaluating models in this dynamic, CI-driven arena reveals nuanced model \"personalities\" (e.g., \"aggressive patchers\" v.s. \"reliable coders\") and surfaces important failure modes (like cross-file consistency errors) that static benchmarks overlook.", "questions": "- Continuous Integration pipelines can be computationally expensive to run repeatedly. Given that the reviewer agent analyzes the submitter's patch and generates targeted tests, could this reviewer component potentially be leveraged to reduce the cost of CI runs needed in an evaluation? For instance, could the reviewer's analysis provide a strong signal for early rejection of clearly incorrect patches before running the full CI, or could it intelligently select a subset of critical tests to run instead of the entire suite?\n- You mentioned that the quality gates for the reviewer-generated tests are crucial for preventing exploitative behavior and ensuring test validity. Could you provide more detail on how strictly these were enforced during the evaluations? For example, what was the approximate rejection rate for reviewer-generated tests that failed these gates, and what were the common reasons for rejection (e.g., failing against the golden patch, modifying production code, style violations)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-motivated, addressing a clear gap in existing research. It convincingly argues that the field must move beyond simple unit test success and instead evaluate models on their ability to produce code that is \"valid, compliant, and able to pass a full CI pipeline and peer review.\"\n- A core contribution is the new dataset of over 2300 real-world GitHub issues across four languages (C++, Python, Rust, Go). Each problem is CI-grounded, meaning the original human solution was verified to pass the full CI pipeline, ensuring a high-quality, realistic testbed.\n- The adversarial framework successfully reveals behavioral tendencies that static tests can't. For example, the paper found that GPT-4o acts as an \"aggressive patcher\" (achieving high win rates), whereas DeepSeek and Gemini \"prioritize correctness and CI stability\" (scoring higher on CI pass rates).", "weaknesses": "- It's hard to interpret the primary \"Win Rate\" metric given its adversarial nature. A model's success depends on the relative weakness of its opponent (the reviewer model). This makes it difficult to assess the absolute quality and correctness of a solution based on this metric alone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "goqFdrsPbj", "reviewer": "Reviewer_hYHi", "summary": "The paper introduces SWINGARENA, a CI-faithful, adversarial code-evaluation arena where an LLM “Submitter” proposes patches and an LLM (or human) “Reviewer” writes tests to break them; the roles can switch across rounds. It ships (i) a rigorously curated multi-language dataset with runnable CI, (ii) a Retrieval-Augmented Code Generation (RACG) pipeline to find relevant files and synthesize fixes, and (iii) arena metrics (e.g., Win Rate, SPR/RPR) to quantify progress. Experiments on real repositories show the arena surfaces harder, more realistic failure modes than static benchmarks and that better retrieval/testing policies materially improve pass rates.", "questions": "1) **Causal attribution:** How do you isolate the gain from the *adversarial, role-switching* protocol itself (vs. model scale, retrieval strength, prompt length)? Can you run controlled A/Bs holding RACG constant while removing role-switching, and report mediation analysis?\n\n2) **Real-world parity:** How closely do local CI runs match upstream (e.g., GitHub Actions) on pass/fail and runtime? Please provide a paired evaluation with agreement statistics and discuss observed drift.\n\n3) **Gaming & robustness:** Are there “score-hacking” strategies (e.g., submitter creates trivially detectable but non-critical faults; reviewer overfits to diffs)? What constraints or equilibrium analyses prevent metric inflation?\n\n4) **Scaling laws:** As model size, number of rounds, and repo size (files, dependency depth) grow, do we observe consistent scaling behavior? Which has higher marginal return: **more rounds** or **stronger/adaptive retrieval**?\n\n5) **Granular failure taxonomy:** What are the dominant failure modes per language (retrieval miss, build failure, semantic error)? Please add an error decomposition table linking failure modes to specific RACG sub-modules.\n\n6) **Cross-ecosystem generalization:** What minimal changes are needed to support Java (Maven/Gradle) and JS/TS monorepos? Any zero-/few-shot transfer experiments showing the framework’s portability?\n\n7) **Contamination audits:** How did you detect/prevent training–evaluation overlap for both LLMs and the reranker? \n\n8) **Metric resolution:** Beyond overall win rate, can you report **defect-type × difficulty** strata (and CIs) to demonstrate the framework discriminates hard cases rather than being dominated by easy issues?\n\n9) **Adaptive retrieval policy:** Can RACG expand beyond fixed Top-k after early failures? Please report the success-latency trade-off and whether adaptive policies change conclusions.\n\n10) **Cost & reproducibility:** What is the typical wall-clock time and token usage per match under your default settings? Could you provide scripts and a “minimal slice” that reproduces headline results on modest compute?\n\n> If the above analyses (especially #1, #2, #4, #7) are addressed, I would likely raise my overall score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The adversarial submitter-reviewer paradigm with role-switching is creative and mirrors real-world collaboration better than static benchmarks.\n2. Rigorous data curation: Three-stage filtering with human expert validation is commendable.\n3. Well-oraganized writing, easy to understand.", "weaknesses": "1) **Limited novelty / “integration over invention.”**  \n   The work skillfully engineers an end-to-end pipeline for realistic SE evaluation, but core techniques (retrieval, ranking, CI emulation, multi-turn prompting) largely reuse known components. \n2) **Under-granular ablations for RACG.**  \n   Current ablations are mostly on/off toggles or coarse retrieval settings, leaving it unclear which sub-module drives gains.\n   - **Actionable:** Provide per-component ablations on (i) chunk size & overlap, (ii) reranker family (bi-encoder vs. cross-encoder), (iii) proximity/structure priors (same-dir, same-package), (iv) adaptive Top-k vs. fixed, (v) failure-triggered expansion rules; add mediation analysis to quantify each component’s indirect effect on final win rate.\n3) **Language & ecosystem coverage is narrow.**  \n   Results cover C++/Python/Rust/Go but omit **Java/JS/TS** and build systems (**Maven/Gradle**, **pnpm/yarn/monorepo**). This limits external validity for common enterprise stacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)