From Assistant to Independent Developer — Are GPTs Ready for Software Development?
Authors: Dezhi Ran
Yuan Cao
Mengzhou Wu
Simin Chen
Yuzhe Guo
Jun Ren
Zihe Song
Hao Yu
Jialei Wei
Linyi Li
Wei Yang
Baishakhi Ray
Tao Xie
Venue: ICLR 2026 Poster (2026)
Keywords: software dvelopment
app development
coding agent
LLM
code model
Primary area: datasets and benchmarks
TL;DR: We show that while LLMs can generate code snippets well, they fail dramatically at real-world Android app development due to inability to handle complex and architectural reasoning.

Abstract: Large language models (LLMs) have demonstrated remarkable capability in function-level code generation tasks. 
Unlike isolated functions, real-world applications demand reasoning over the entire software system: developers must orchestrate how different components interact, maintain consistency across states over time, and ensure the application behaves correctly within the lifecycle and framework constraints. Yet, no existing benchmark adequately evaluates whether LLMs can bridge this gap and construct entire software systems from scratch.

To address this gap, we propose \tool, a benchmark consisting of 101 software development problems drawn from real-world Android apps. Given a natural language specification detailing the app functionality, a language model is tasked with \textbf{implementing the functionality into an Android app from scratch}. Developing an Android app from scratch requires understanding and coordinating app states, lifecycle management, and asynchronous operations, calling for LLMs to generate context-aware, robust, and maintainable code. To construct \tool, we design a multi-agent system to automatically summarize the main functionalities from app documents and navigate the app to synthesize test cases validating the functional correctness of app implementation. Following rigorous manual verification by Android development experts, \tool incorporates the test cases within an automated evaluation framework that enables reproducible assessment without human intervention, making it easily adoptable for future research. Our evaluation on 12 flagship LLMs show that all evaluated models achieve low effectiveness, with the best-performing model (GPT-5) developing only 18.8\% functionally correct applications, highlighting fundamental limitations in current models' ability to handle complex, multi-component software engineering challenges.

Published as a conference paper at ICLR 2026
FROM ASSISTANT TO INDEPENDENT DEVELOPER ARE GPTs READY FOR SOFTWARE DEVELOPMENT?
Dezhi Ran 1.2 Yuan Cao Mengzhou Wu Simin Chen², Yuzhe Guo Jun Ren', Zihe Song". Hao Yu Jialei Wei2.3 Linyi Li", Wei Yang?..
Baishakhi Ray": Tao Xie
Correspendence to: tacxielpku. edu. en
PPeking University Beijing Tongming LAIC $Columbia University
FFudan University "ong Kong University of Science and Technology "Simon Fraser University
" University of Texas Dallas
ABSTRACT
Large language models (LLMs) have demonstrated remarkable capability in function-level code generation tasks. Unlike isolated functions, real-world ap- plications demand reasoning over the entire software system: developers must orchestrate how different components interact, maintain consistency across states over time, and ensure the application behaves correctly within the lifecycle and framework constraints. Yet, no existing benchmark adequately evaluates whether LLMs can bridge this gap and construct entire software systems from scratch.
To address this gap, we propose APPFORGE, a benchmark consisting of 101 software development problems drawn from real-world Android apps. Given a natural language specification detailing the app functionality, a language model is tasked with implementing the functionality into Android app from scratch. Developing an Android app from scratch requires understanding and coordinat- ing app states, lifecycle management, and asynchronous operations, calling for LLMs to generate context-awer -awee, robust, and maintainable code. To construct APPFORGE, we design a multi-agent system to automatically summarize the main functionalities from app documents and navigate the app to synthesize test cases validating the functional correctness of app implementation. Following rigorous manual verification by Android development experts, APPFORGE incorporates the test cases within an automated evaluation framework that enables reproducible assessment without human intervention, making it easily adoptable for future research. Our evaluation on 12 flagship LLMs show that all evaluated models achieve low effectiveness, with the best-performing model (GPT-5) developing only 18.8% functionally correct applications, highlighting fundamental limitations in current models' ability handle complex. component software engineer- ing challenges. Our benchmark, including all data and code, is publicly available ngLAIC/ AppPPrrge...
/tttps://gtt thub. www/Tonooeneeoooooooo Www.omiggAIIC.................IC
INTRODUCTION
Large language models (LLMs) are reshaping the horizon of software engineering, Frontier code LLMS (OpenAI, 2023) are deeply integrated into developer's toolchains like GitHub Copilot (GitHub, 2025), Amazon CodeWhisperer (Amazon Web Services, 2025), and Claude Code (Anthropic, 2025). They are advancing from coding assistants to fully autonomous software developers (Yang et al., 2024), which hold significant potential to shape the next generation of software engineering.
Although existing benchmarks have advanced the evaluation of code LLMs, they primarily focus on generating isolated snippets or functions, which differs fundamentally from the system-level reasoning and integration required to build complete application. As result, they cannot determine whether current models are capable of end-to-end software development in real-world scenarios. For instance, HumanEval focuses self-contained, toy-level, function-level code generation (Chen

---

Published as a conference paper at ICLR 2026
et al., 2021; Austin et al., 2021), while SWE-Bench targets program repair tasks within an existing codebase, requiring only minor modifications to a few lines of code in the target repository (Jimenez al., 2024). None of the existing benchmarks effectively assess the end-to-ene software development capabilities of LLMs in the role of an independent software developer (Liu et al., 2023; Jain et al., 2024; White al., 2025; Zhu al., 2024; Rajore et al., 2024). To address the limitations of current benchmarks and evaluate whether LLMS can truly function as software engineers in real-world development scenarios, we argue for the creation of new benchmark that goes beyond narrow tasks and instead captures the full spectrum of software development.
Building such a benchmark is necessary because (1) it provides a comprehensive and realistic evalua- tion of LLMs' ability to perform software development tasks end-to-end. (2) bridging the gap between isolated code generation and real-world engineering and (3) providing insight how to leverage L.MSs for the next generation software engineering. However, there are several challenges in building such a benchmark: O Reflecting the Real- world Software Development Process: The benchmark should be realistic and faithfully represent the complexities and workflows of actual software development. Ensuring Sufficient Challenge and Diversity: The benchmark should be sufficiently challenging to differentiate model capabilities, covering diverse tasks such as design, implementation, debugging, and maintenance. @ Measuring End-to- End-to-End Development Performance: The benchmark should capture not only code correctness but also factors like code quality, maintainabilityy and integration within larger systems.
To address these challenges, we propose Android application (app) development as our benchmark domain, motivated by three key factors. First, Android represents one of the most significant software ecosystems globally. with over 2.6 million apps available (Technource, 2022), making it highly representative of real-world software development. Android development naturally involves creating complete projects with specific functional requirements, effectively capturing authentic development workflows. Second, developing Android apps from scratch provides inherent complexity through backend logic implementation, state management, UT design, and external API integration, ensuring sufficient difficulty and diversity for comprehensive evaluation. Third, the mature ecosystem of An- droid development tools, including static analyzers, testing frameworks, and emulation environments, enables rigorous automated assessment of various development aspects (Developers, 2025a;b;c).
Building on this intuition, we propose APPFORGE, the first benchmark for evaluating code LLMs specifically in Android app development. As illustrated in Figure 2, LLMs are tasked with generating complete Android apps from scratch based on natural language specifications. Once the code files are generated, APPFORGE automatically handles compilation into APK files, deployment on Android emulators, and comprehensive functionality validation against automated test case execution and systematic fuzzing. To ease the of A APPFORGE, the evaluation of APPFORGE is fully automated and encapsulated with a standalone docker for out-of-the-box usage.
To construct APPFORGE with scalability and rigor, we first collect real-world Android apps from F-Droid 2025c, a well-curated repository of open-source Android apps that provides real-world and actively maintained projects. Next, we leverage LLMs automatically extract and summarize functionality specifications from each app's 's documentation and source code. Subsequently, we leverage a GUT agent (Ran et al., 2024) to interact with the app, capturing its runtime behavior to validate and enrich the specification description to avoid task ambiguity. Finally, we engage Android development experts to verify the correctness of both specifications and synthesized test cases. This combination of automated processing and expert validation ensures both scalability and reliability in benchmark construction.
We evaluate 12 flagship LLMs (OpenAI, 2023; Guo et al., 2024; Di et al., 2024; Jiang et al., 2024;
Li et al., 2022) including GPT-5 (OpenAI, 2025) and Claude-4-Opur (Anthropic, 2025) as well as popular coding agents including Claude Code (Anthropic, 2025) on APPFORGE, revealing three key findings. First, all models achieve remarkably low performance with less than 20% of apps being functionally correct, and among these correct apps, half still encounter least one crash (detailed in Table 1). This contrasts sharply with saturated existing benchmarks (Chen et al., 2021; Jain al., 2024; White al., 2025), indicating a significant gap between current LLM capabilities and real-world development tasks, and that APPFORGE represents the next frontier of software engineering challenges. Second, we uncover that some LLMs evade app development tasks by sacrificing functionality integrity for compilation success. When given opportunities to improve their previous generations with compilation errors, GPT-4.1 (OpenAI, 2025) and Kimi-K2 K2 (Kimi Team,
2

---

Published as a conference paper at ICLR 2026
2025) delete the implementation of ervor-inducig functions instead of fixing them as illustrated in Figure 8, indicating an avoidance strategy that sidesteps eтror handling instead of demonstrating true debugging competence. Specifically, GPT-4.1 evades development in 91.09% of tasks, while Kimi K2 does so in 65.36% of tasks. Finally, for simple tasks like calculator implementation (Figure 6), L.MSs demonstrate promising performance, producing robust apps that surpass typical human-writeer code quality as illustrated in Figure 7, suggesting significant potential when complexity is appropriately managed for future software development.
We summarize our main contribution as follows:
New real-world problem: : We introduce end-to-end Android app development from scratch as comprehensive evaluation task for LLMs' software engineering capabilities.
New Benchmark: We construct APPFORGE, a benchmark with 101 diverse Android devel- opment tasks and fully automated evaluation suites.
Evaluation Results: We evaluate 12 flagship LLMs and analyze their limited performance and failure patterns in real-workh software engineering scenarios.
BACKGROUND & RELATED WORK
Code large language models (LLMs) have been advancing rapidly, where frontier code LLMs such GPT5 (OpenAI, 2025). Claude-Opus (Anthropic, 2025). Gemini-Pro (Google, 2025), and Qwen3-Coder (Yang et al., 2025) have reshaped the paradigm of software development. As software ecosystems such as Cursor (Cursor, 2025) and GitHub Copilot (GitHub, 2025) continue to mature, the application scenarios of code LLMs expand beyond code generation and completion to encompass debugging, test generation, and even autonomous software development.
In contrast to the wide application scenarios of code LLMs, benchmarks that evaluate code LLMs still largely focus on (1) function-level code generation and completion,
Implementation Scope Full Development From Scratch
such as HumanEval (Chen et al., 2021), MBPP (Austin
et al., 2021), and BigCodeBench (Zhuo et al.. 2024);
Coding Complexity
BigCadeBench
APPFORGE
and (2) patch generation and feature implementation with repository-level context, such as SWE-Bench (Jimenez
LiveCodeBench
functionlevel
repostoory
al., 2024), Web-bench (Xu al., 2025) and Lo- CoBench (Qiu et al., 2025). Some efforts transform static benchmarks into dynamic ones to combat data contami- nation, such as SWE- Bench-nnhhh- (Zhang al., 2025) and LiveCodeBench (Jain et al., 2024). As shown in Figure 1, APPFORGE goes beyond function-level code generation and patch generation. Compared to existing benchmarks, APPFORGE evaluates code LLM's capability to perform automated software development from scratch at the repository level. It incorporates rigorous evaluation empowered by automated test cases and system- atic fuzzing.
HumanEval
SWE-Bench
MBPP
EvalPlus
SWA-
DevEval
CodeXQLUE
Wab-Bench
FEA-Banch
Partall Completion
Figure 1: Our Work Compared with Ex- isting Code Generation Benchmarks.
APPForGE is the first benchmark for assessing LLM capabilities in Android development to our knowledge. Android apps are typically built with Java or Kotlin following Material Design and Android architecture guidelines, comprise multiple interconnected components (2025a). Android apps represent one of the most significant software ecosystems globally with over 2.6 million applications available (Technource, 2022), so we believe that Android development an ideal code LLM evaluation scenario that largely reflects the real-world software development process. F- Droid (2025c) is the leading open open-source Android app repository, serving millions users worldwide. F-Droid apps span diverse categories and their code undergo rigorous review process. APPFORGE is constructed from a diverse set of hualtquality F-Droid apps, and can be dynamically expanded using latest projects from D-Drodd. We defer more in-depth discussion of related work in Appendix A.
3

---

Published as a conference paper at ICLR 2026
APK Complling
Designer Ageur
GLI Agrot
Generate
</>
Pass
Yes
DUME
Earh
Eatrant
Specification
Formal
LLM Agents
App Code
Produce APK
F-Deokl
Explert Rago
Functionalit Descrppiin
App
Interaction Trace
Top 200 Apps
Task Refisement
Stabality
No
Functional Testing
Collected
Testing
Yis
Crashing
Fuzzing
Testing Scores
Install Device
Validated Test Cases
Fomoal Specllcatiin
Validittd
Papert
Tesk Specilication
Test Cosc
Figure 2: Workflow of APPFORGE.
Figure 3: Construction of APPFORGE.
APPFORGE
APPFORGE is a benchmark designed to evaluate LLMs' capabilities across the full software develop- ment lifecycle for Android applications, using real-world apps such Amaze File Manager (F-Droid, 2025a), Arcticons (F-Droid, 2025b), and Vanilla Music (F-Droid, 2025c). Given the natural language description of Android application, the task is generate the corresponding code implementation that not only faithfully realizes the described functionality and passes the associated tests, but also executes securely within the Android operating system.
3.1 ANDROID APP DEVELOPMENT TASK FORMULATION
Each task in APPFORGE includes three main fields: model input, model output, and the evaluation suite. An example of task instance is provided in Appendix B.1.
Model Input: The model input is natural language description that consists of three components: (I) high-level overview of the app's 's functionality along with detailed descriptions of the features corresponding to each functionality, (2) natural language test cases that specify how these functionali- ties should be implemented and validated, and (3) implementation constraints such API version requirements and expected output format specifications. Within the detailed feature descriptions, we also provide the specific resource IDs required for implementation. This design streamlining the overall evaluation process.
Expected Model Output: When prompting the LLM for app generation, the model is required to produce output in JSON format, where each key represents a filename and each value contains the corresponding code. This design enables automated project assembly and evaluation.
APPFORGE includes an automated evaluation pipeline consisting of three com- ponents: (1) an automatic compiler suite that parses the generated outputs. assembles them into an Android project, and compiles the project into an Android Package (APK); (2) a testing module that installs the APK onto an Android emulator and executes predefined test cases to validate func- tional correctness; and (3) lightweight fuzzer that evaluates the robustness and -handlnnn capabilities of the application under various edge cases and unexpected inputs. The evaluation reports four metries: (1) compilation success rate, (2) test pass rate, (3) crash rate, and (4) an overall performance score (detailed in Appendix B.3) representing the model's 's effectiveness on the given Android development benchmark (More implementation details could be found in Appendix B.4).
Evaluation Suite:
3.2 CONSTRUCTION OF APPFORGE
We construct our benchmark from real-world Android apps collected from F-Droid. Although each app on F-Droid comes with detailed documentation and README files, these resources are too large and unstructured to directly used as prompts for benchmarking L.LMs. To address this limitation, we need to regenerate the task, Specifically, we follow the pipeline below: (1) Seed App selection: choose apps based on diversity, complexity, and popularity. (2) UI navigation and trace recording: We use UI navigator tool to explore the selected apps and record the navigation traces along with each UI element's ID. This step provides detailed interaction data, enabling automatic evaluation. Since our UI navigation is dynamic process, the app can produce different traces. This dynamic mechanism allows us to generate diverse tasks from the same app. reducing the risk of data contamination. (3) Trace summarization: We combine the app documentation and navigation
4

---

Published as a conference paper at ICLR 2026
traces, such the element ID, then LLM to summarize each trace into natural language descriptions. (4) Human validation: Finally, we perform human validation to ensure the generated tasks are accurate and meaningful.
App Selection and Scraping. We begin by ranking apps based on a combination of popularity, complexity, diversity, and update frequency; detailed criteria are in Appendix B.5. From this ranking, we select the top 200 highest-scoring apps across different categories as seed apps for subsequent task creation, ensuring balanced coverage of Android development domains. For each selected app, we analyze its code repository to extract metadata, including descriptions from README files and release notes. We then summarize the app's 's core functionalities in natural language using a JSON format. These functionality descriptions are intentionally high-level and may be ambiguous.
Automatic App Navigation. We use an existing tool, UlAutomator (Android Developers, 2025), to install each seed app in an Android emulator and systematically record interaction traces. For every high-level functionality description, UI navigator performs goal-based navigation (Ran et al., 2024) starting from the app's main screen. Guided by the functionality description, the agent identifies and interacts with relevant UI elements while maintaining a detailed log of the process. At each step, captures the full UI tree using UlAutomator, including element properties such as text, resource-id, class, and bounds. The agent also documents the sequence of UI actions (e.g., clicks, text inputs, swipes). the target elements, and the reasoning behind each action, along with the resulting screen transitions and state changes. Once the target functionality is accomplished (e.g., logging in or sending message), the agent records the complete interaction trace. This goal-directed approach produces precise traces that capture the most natural paths for implementing each functionality.
Task Generation For Trace History. We then utilize a LLM to synthesizes precise task descriptions and test suites based on the captured interaction traces. First, the task refinement agent transforms each interaction trace into test case. Each test case consists of a sequence of UI actions and an oracle specifying the expected outcome of executing the action sequence. Each UI action is associated with UI element containing clear text or resource-id labels. For UI elements in seed apps that lack meaningful labels, the refinement agent generates context- contextapppropriate Each oracle is an assertion determining whether UI element exists or does not exist. The test case implemented Python script using the UIAutomator framework, enabling automated evaluation. Based on the synthesized test suites, the task refinement agent generates a task description detailing the core functionalities and their implementation. For each test script, the agent produces natural language descriptions that specify the sequence of UI interactions (c.g.. "click the button with login resource-id", "enter text in the username field") and the expected app states after these operations. This approach eliminates ambiguity by providing precise, actionable specifications that ensure any LLM human developer interpreting the task description will implement functionally equivalent apps that satisfy the same behavioral requirements.
idsource-ids to avoid ambiguity.
Android Developer Validation. To ensure quality control, five expert Android developers with a combined 30 years of experience reviewed all tasks for technical accuracy, feasibility, and alignment with wall-orrld practices. The validation process included checking task clarity and completeness, verifying non-trivial and unambiguous requirements, ensuring coverage of essential concepts across difficulty levels, and confirming the soundness of examples and constraints. Experts also validated test cases by examining expected outputs and the accuracy of automated testing. Each task underwent multiple review rounds until consensus was reached, yielding high-quality benchmarks that reflect authentic Android development challenges.
3.3 BENCHMARK SUITE AND DATA STATISTICS
We collect 101 high- Android development tasks, each representing the development of a com- plete Android application. The task distribution reflects real-world Android development patterns and emphasizes comprehensive application diversity: UI/Layout focused apps comprise 40%, covering complex view hierarchies, custom components, and responsive design; Backend Integration apps account for 32% including API consumption, data persistence, and background services; User Inter- action apps represent 94%, focusing on gesture handling, input validation, and navigation flows; and System Integration apps make up 63%. encompassing permissions, hardware access, and inter-app communication. Task complexity spans three difficulty levels based on implementation requirements: Beginner (37%, focusing single-activity apps with basic Android concepts), Intermediate (48%,
5

---

Published as a conference paper at ICLR 2026
requiring multi- component integration and moderate architectural complexity), and Advanced (15%, involving sophisticated architectural patterns, performance optimization, and complex system interac- tions). The diversity in app categories and complexity levels ensures that APPFORGE captures the full spectrum of Android development scenarios real-world practice.
3.4 KEY FEATURES OF APPFORGE
Traditional code generation benchmarks often focus on toy function-level tasks or partial repository generation, where much of the context is pre-defined and evaluation limited to functionality- approach shown to be insufficiently rigorous in prior work. contrast, APPFORGE draws on real-world Android applications from F-Droid, offering au- thentic, end-to-end development tasks that more faithfully capture practical software engineering challenges. Here, describe some key features:
Money
System
Games
Securty
Connectivit
Real-world Software Development Tasks. Since each task in APPFORGE is sourced from F-Droid and represents a real- world Android application that may have been installed on millions of devices worldwide, solving APPFORGE requires LLMs to demonstrate sophisticated skills and knowledge in full-stack Android development, including UI design, API integration, state management, and security considerations cetiddrratoosscccpplpaiiieeee rarely evaluated in traditional code generation benchmarks.
Theming
Graphico
Scienco
Sports & Health
Figure 4: Distribution of Category.
Diverse Task Categories. As shown in Figure 4, APPFORGE includes diverse range of apps. Each instance of APPFoRGE belongs to a unique category, making significantly more diverse than existing benchmarks (e.g., SWE-Bench includes only 12 different repositories from Python (Jimenez al., 2024) and concurrent work LoCoBench covers only mobile app categories (Qiu et al., 2025)).
Software-level Code Generation. This task challenges LLMs to generate coherent, end-to-end Android application code while understanding the semantics of APIs across different versions the Android framework and third-party libraries. Unlike function-level tasks, software-level generation requires reasoning about how components interact, handling persion-specifce behavior, and integrating multiple modules correctly. By requiring models to adapt to evolving APIs and manage compatibility, this task evaluates a deeper level of software engineering capability, beyond simple functionality, ensuring that generated applications both correct and maintainable.
Rigorous Functionality & Reliability Evaluation. Considering the fact that every software may contain some bug or defect, our benchmark includes both functionality and reliability evaluations. Our experiments demonstrate that incorporating reliability is essential, as it can uncover hidden crashes that would be missed by functionality testing alone.
Wide Solution Space. The task of full-application code generation in APPFORGE provides a level playing field for evaluating approaches ranging from standard models autonomous agents capable of reasoning and acting across an entire Android project. APPFORGE also encourages creative solutions, allowing models to produce implementations that may diverge from reference apps while still meeting functional, and security requirements.
4 EVALUATIONS
4.1 EVALUATION SETUP
We conduct comprehensive experiments on APPFORGE with 12 state-of-the-art L.MMs, including 7 proprictary models (Claude-5-Opus, CCaude44-Smmee(Atttppii 2025), Gemini-2.5-Pro (Google, 2025), GPT4.1 (OpenAI, 2025), GPT-5-Low, GPT-S-Medium. and GPT-5-High (OpenAI, 2025)) and 5 source models (DeepSeek-N) R1, DeepSeek- V3 (Guo et al., 2024), GLM-4.5 (Zhuo et al., 2024), Kimi K2 (Kimi Team. 2025), and Qwen3-Coder (Yang al., 2025)), along with two coding agents
(Yang et al., 2024) and Claude Code (Anthropic, 2025)) to evaluate the cutting-edge progress in fully automated software engineering. Details provided in Appendix C.

---

Published as a conference paper at ICLR 2026
Table I: Performance of LLMs on APPFORGE.
Compilation Feedback
tflle
BLOC
Compile
Test Pass
Crash
Success
#flle
BLOC
Compile
Test Pass
Crach
Success
Proprictury Modely
Claude 4-pus
9.11
396.94
50.20%
186.63
Claudo--Soonet
9.61
432.17
40.59%
10.35%
58.540 54.54%
0.99%
9.78
437.66
77.23%
18.36%
26.92%
3.96%
Gemini-2..-Pro
10.74
380.31
53.47%
19.63%
62.96%
7.92%
10.52
361.94
68.32%
21.63%
75.36%
13.86%
Giig
354.59
45.54%
21.90%
340.77
82.18%
29.07%
31.33%
GPT-4.1
8.00
367.43
6.93%
2.44%
28.57%
0.99%
2.68
58.41
74.26%
1.85%
94.67%
0.99%
Open-snurex Modeds
DeepSeck-R1
7,00
214.33
14.85%
1.90%
73.33%
0.00%
7.33
233.78
44.55%
12.29%
62.22%
4.95%
DeepSeek-VV V3
5.17
164.67
5.94%
2.235%
83.33%
0.99%
5.33
250.19
26.73%
10.40%
48.15%
4.95%
GLMM-.5
7.64
256.16
24.75%
72.000%
8:51
44.55%
75.56%
Kimi K2
6.82
239.82
16.83%
4.95%
76.47%
1.98%
5.10
168.60
41.58%
7.76%
69.05%
1.98%
Qwen3-Coder
5,29
209.00
27.72%
4,42%
75.00%
1.98%
6.20
241.21
85.15%
21.45%
29.07%
8.91%
4.2
MAIN RESULTS
All models struggle on APFFORGE. As shown in Table 1, all models achieve low performance on APPFORGE, with the best- cest-performing flagship model GPT-5 with high reasoning mode achieving only 14.85% success rate (developing 14.85% of apps passing all cases). When given chances to repair compilation errors in their previous development, the improvement is still marginal, with GPT-5 achieving only 18.81% success rate. Open Open-source source models perform considerably worse, all achieving less than 10% functional success rate after repairing compilation error.. While the high compilation rates flagship models demonstrate that existing models can generate syntactically correct programs, the consistently low test pass rates across all models reveal the fundamental challenge of generating functionally correct Android apps. In addition, over 50% of functionally correct apps crashes during runtime, highlighting that when LLMs successfully implement the required functionality, the generated code often lacks reliability necessary for real-world deployment.
Iterative refinement with compilation feedback does not signif- icantly improve functional correctness. The compilation eπror feedback substantially improves compilation success across all models, with notable improvements for Claude-4-Sonnet (40.59% to 77.23%) and Qwen3-Coder (27.72% to 85. 15%). However, this improvement does not translate proportionally to functional correctness, as test pass rates show modest gains. As illustrated in Figure 5, iterative refinement significantly improves compilation success for both Owwn3-Coder (33.7% to 98%) and DeepSeck- V3 (7.9% to 63.4%) However, the functional success rate, measured by passing test cases, saturates quickly after 2-3 it- crations, peaking around 23% for Qwen3-Code and 14% for DeepSeek-V3,
100
Gmael Recers
Rate (%6
60
40
20
Fix laration Rounds
Figure 5: Performance evolution with compilation feedback.
LLMs can develop robust, functionally correct apps simple development tasks. Despite overall low success rates, successful cases demonstrate that L.MMs can generate surprisingly sophisticated An- droid applications. As visualized in Fig- ure 6, there is a clear inverse relationship between app complexity and success rates for lower complexity tasks with enough sample sizes below 800 LOCs. Notably, successful cases often showcase proactive exception handling and defensive programming beyond ba- sic functional requirements. Figure illustrates an actual implementation by GPT-5 the Autostarts app, where it gracefully manages potential exceptions and provides fallback solutions.
Figure 6: Correlation between Lines Code (LOC) and evaluation metrics (Compile, Test Pass, and Suc- cess). Rolling means with uncertainty bands show per- formance variability across code complexity.
Some LLMs evade development tasks rather than repair their compilation errors. Interestingly, GPT4.1 and Kimi K2 evade development tasks during iterative refinement, where they delete faulty
7

---

Published as a conference paper at ICLR 2026
private vold openágpTnfo packagaNarre
ley {
intent it new Intent (Se:tings.
st.setData (Uri.
packageName)
FIAG_ACTTITTYY startActivity (it); catch sestivityoooFouddRooooo try
it.addFlage
Iatent fallback new
(Settings ACTION APPLICATION
10
fallback.setuart (Uri. puri. ("peckage:" + get.ackagehaame (1)) caarActivityyfall (follbackk
12
catch (Exception ex)
15
roast. maketext (this, "Uncbla to open Application Info", Teast LENGTH_ (SORT) .woowti)sshow..) sshow();
Figure 7: Proactive defensive programming implemented by GPT-5 on the Autostarts app.
// Original Geneartion (Compile Berors Mainkctivity javaryy...aaaaa. 1ava+45: ecrozs cannot rind symbol Intent it Intent (Mainkctivity this, Newiodblistactivvayy (LndViewBBIddRR.d. id. ac_add) setOnO LekListener (v Intent startastivittyooreenuu (it, 101);
(Mainketivide this, dewtoobllisthctiiiyy...ees
});
// --- netinement lpassing comp1 lon without implementation!
FindViewById (R. ad. settnnClickissten (v
Figure 8: GPT-4.1 evades development when fixing the compilation error Todo List app.
implementations instead of repairing them. GPT-4.1 shows dramatic reduction in generated number of files (from 8.00 to 2.68) and LOCs (from 367.43 to 58.41) when provided with compilation feedback. As shown in Figure 8, GPT-4.1 replaces the buggy implementation of the function with an empty body. This strategy successfully achieves the highest compilation rate improvement (from 6.93% 74.26%), but does good for implementing the required app functionality. Similar patterns are observed in Kimi K2, indicating that some LLMs may strategically simplify their solutions when faced with compilation challenges rather than addressing the underlying issues.
APPFORGE differentiates model capa- bilities better than existing code gener- ation benchmarks. While many mod- els achieve high and similar performance on traditional code generation benchmarks like HumanEval and SWE-bench, APp- FORGE help reveal performance gaps of LLMs for real-world software engineer- ing tasks with success rates spanning from 0.99% 14.85%. In addition, We visu- alize the model performance differentia- tion Figure 9. the performance variance on APPFORGE is substantially larger than SWE-bench, providing more nuanced dif- ferentiation of model capabilities. This suggests that APPFORGE captures the real-world software engineering challenges that are not adequately captured by previous code generation benchmarks.
-58
GF741
07741
0775
GPT5
8 8o
C4dovel
C-4pps
C40za
gm12
goss grAA
Source
Source
Figure 9: Pairwise relative performance differences be- tween models SWE-bench-verifies (Left) and APp- FORGE (Right). Green and red cells represent relatively superior and inferior performance, respectively, with color intensity indicating the magnitude of differences.
4.3 PERFORMANCE OF ADVANCED CODING AGENTS AND REASONING MODELS
Table 3: Performance GPT-5 with differ- ent reasoning levels on APPFORGE.
Table 2: Performance of coding agents on APPFORGE.
Agent
L.M
#Flle
#LOCC
Complle
Test Pass
Success
SWE
Cloude-4-Ope
10.76
558.40
71.29%
24.61%
11.88%
Level
#File
#LOC
Compile
Test Pass
Success
Qwen3-Coder
8.42
430.94
88%
22.21%
6.93%
CC Qwen3 Coder
5.34
280.66
76.24%
14.64%
6.93%
High
Medium
7.76
7.61
321.96 354.59
27.72% 45.54%
21.90%
11.11%
14.85%
3.96%
8

---

Published as a conference paper at ICLR 2026
Coding agents provide marginal improvements at substantial computational cost. As shown in Table 2, coding agents (mini-SWE -aggeeett as SWE, Claude Code as CC) exhibit slight improvements simpler bascline approaches, yet their performance gains modest. Specifically, the best- performing combination (mini-SWF-agent using Claude-4-Opus) achieves only an 1 1.88% functional success rate. Although these agents demonstrate potential for iterative refinement and error correction, their modest overall performance indicates that current agent-based frameworks still fall short of effectively overcoming critical challenges inherent to real-world software engineering tasks, such as multi-file integration and framework-spor -specficc complexity typical in Android app development.
Enhancing reasoning capabilities remain insufficient for Android development. Table demon- strates that increasing the reasoning level of GPT-5 leads to improved performance across all metrics, with the highest reasoning setting achieving 14.85% functional success compared to 2.97% at the low level. However, even with maximum reasoning enhancement, the absolute performance remains far from satisfactory for practical Android development, highlighting that the fundamental challenges of multi-fil file coordination, Tramework-spectie knowledge, and complex dependency management require more than enhanced reasoning alone.
4.4 ANALYSIS OF DEVELOPMENT CHALLENGES
Compilation Error Analysis. Figure 10 presents the distribution of compilation errors. The most prevalent error stems from "Android Resource Link- ing Failed", accounting for 39.7% of compilation errors. This compilation error is typically caused by missing misreferenced resource files in the gener- ated apps, highlighting current models' inadequate capability in comprehensive software engineering tasks that require systematic coordination across mul- tiple project components. An interesting observation is that GPT series models and Kimi-K2 encounter the issue that apps fail to compile due to missing android:exported declarations (an attribute requirement introduced in Android 12 (Android Develop- ers, 2021)), highlighting gap between LLM training data and current Android requirements. Though can be resolved by refinement, this interesting issue reflects models' strategies when handling conflicts between training data patterns and task instructions.
Cappltw Tocia Duckater
Figure 10: Distribution of compilation errors across generated Android apps.
Crash Analysis. Table presents the crash analysis results from fuzzing LLM- generated apps (full version available in the Appendix C). First, the "evade development" strategies employed by GPT-4.1 ultimately backfire runtime. While achieving higher compilation rates, the app fundamentally fail to start when executed, indicating that evasive compilation error fixes often intro- duce fundamental flaws, such as incomplete resource initialization that prevent proper app bootstrap- ping. Second, notably all crashes are native crashes rather than Java-level exceptions, indicating that the generated Java code itself generally robust with proper exception handling. This suggests that existing L.Ms excel at defensive programming practices and maintain good exception handling patterns as illustrated in Figure However, crashes occur when calling third-party libraries or interacting with OS services due parameter validation failures and contract mismatches. While LLMs demonstrate solid understanding of Java code, they lack sufficient knowledge of underlying implementation details and resource constraints. Consequently, seemingly safe Java code can trigger native-level issues when interfacing with lower-level components, highlighting the gap between surface-level language proficiency and deep system understanding required for software engineering.
Table 4: Runtime crash analysis across LLMs.
Native Crash w/o Fix
Failed to Start
Model
w/ Fix
w/o Fix
w/ Fix
GPT-4.1
0.0
11.0
2.0
66.0
Claude-Opus
48.0
48.0
9.0
11.0
Gemini-Pro
25.0
37.0
14.0
21.0
GPT-5 --gh
21.0
0.0
5.0
25.0

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, we have introduced APPFORGE, comprehensive benchmark for evaluating LLMs on real-world Android application development from scratch, revealing significant gaps between current capabilities and practical software engineering requirements. Through systematic evaluation of 12 state-of-the-art LLMs across 101 diverse development scenarios, we found that even the best-performing models achieve only modest success rates, contrasting sharply with their high performance on existing code generation benchmarks, suggesting that fundamental innovations rather than incremental improvements may be necessary toward fully automated software engincering.
ACKNOWLEDGMENTS
We thank the Android development and testing experts at WeChat, Tencent, for their kind and voluntary help in reviewing the benchmark. Dezhi Ran and Tao Xie are partially supported by National Natural Science Foundation of China under Grant Nos. 623B2006, 92464301, and U25A6023, Any opinions, findings, conclusions, recommendations expressed in this material those the authors and do not necessarily reflect the views of the funding agencies.
ETHICS AND REPRODUCIBILITY STATEMENT
This work adheres to the ICLR Code of Ethics. No human subjects animal experimentation were involved. No personally identifiable information was used, and no experiments posed privacy or security risks. We are committed transparency and integrity throughout the research process. We believe APPFORGE can be used for various purposes, including evaluating the cutting-edge capabilities of code LLMs for software engineering, training better software engineering models and agents, and as a seed benchmark for building larger benchmarks for application development, We have strictly adhered the license of open-source apps since
the runtime behavior of these apps instead of using their source code for constructing APPFORGE. However, we are concerned about potential misuse of APPFORGE for training models reverse engineering existing Android applications, making the plagiarism of apps a practical concern.
As benchmark paper, the benchmark has been made publicly available https: / / thub. com/
AppF orge. providing detailed documentation, leaderboard, and docker- ized environment to ensure easy reproduction and customized use. We have detailed the selection criteria in our Appendix and use popular open-source apps in F-Droid to allow reproducibility of task collection. The experimental setup is described in detail.
LLM USAGE
LLMs were employed solely to assist in writing and polishing the manuscript, including refining language, improving readability, and enhancing clarity. The LLM was used for tasks such sentence rephrasing, grammar checking, and improving overall flow.
The L.M was not involved ideation, research methodology, or experimental design. All research concepts, analyses, and results were developed and conducted by the authors. The authors take full responsibility for the manuscript content, including any text generated or polished by the LLM, and confirm that all LLM-assisted text adheres to ethical guidelines and does not constitute plagiarism or scientific misconduct.
REFERENCES
Amazon Web Services. Amazon CodeWhisperer.
amazz/ass.aaaaaaoo..oa com/
codewhisperer// 2025. Accessed: 2025-09-24.
Android Developers. Behavior changes: Apps targeting Android 12 exported com-
ponents requirement. bttps://developps android. com/ abcut/ Accessed: 2025-09-24.
behavvior-changee. 12teresrtteeeeeeeeeeeeeee 2021.
10

---

Published as a conference paper at ICLR 2026
Android Developers. UI Automator.
https:// http://ddeeelllpee..ddadddd.dddd httaeaaa........oo.aaaaaaaaaaaaa 2025. Accessed: 2025-09-24.
Anthropic. Models overview Claude docs.
claude, wwwaaudee.oeeeeeeeeeeeeeeeeeeeee /
about-clauee soooottclladdmmmmdele/////oee
2025.
Accessed: 2025-09-23.
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang. Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang. Qing Sun, Mingyue Shang. Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, and Ramesh Nallapati. Multi-lingual evaluation of code generation models. In The Eleventh International Conference Learning Representations, 2023. URL https:://// /ttp//ppeeeeeeeeeeeeeeeeeeeeeeee net /ttp//pfeee iideeiiddeee did Be7eee666nn
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie J. Cai, Michael Terry. Quoc V. Lc, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https: /ttps///arxvvvvoo org/ abss/
2108.0777 07732.
Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D> C, Arun lyer, Suresh Parthasarathy, Sriram Rajamani, Balasubramanyan Ashok, and Shashank Shet. CodePlan: Repository-level coding using LLMs and planning. Proceedings of the ACM Software Engineering, 1(FSE); URL https: /ttp:::://di... wtt.ddi.....oooooooooeeoooo 0tt.:/1...11///666 3643557.
675-698, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- ers. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020. URL https: http::///ooeeeeeeneeeeee wewp:eeeeediiinnnnneeeeeeeeeeeee wwwrips. cc/paper..... http:/appppleee//////alaaaa wwwwssiappeeeeeeeeeeeeeeaaaeeeee file/ htll://447cccbfoof66/11122222222 Paaeeee4aaPPap.e....
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, 0bb/2107.003374 2021. URL https: //arxiv. org/ /agg/bb//// 217770377770....
Simin Chen, Soroush Bateni, Sampath Grandhi, Xiaodi Li, Cong Liu, and Wei Yang. Denas: Automated rule generation by knowledge extraction from neural networks. In Proceedings of the 28th ACM joint meeting European software engineering conference and symposium the foundations of software engineering, pp. 813-825, 2020. URL https://doi.org///
114 1145/333 1145/3368083 3409733.
Simin Chen, Xiaoning Feng. Xiaohong Han, Cong Liu, and Wei Yang. PPM: Automated generation of diverse programming problems for benchmarking code generation models. Proceedings of the ACM on Software Engineering. 1(FSE): 1994 11111194-11111 2024a. URL https:ts::///o
11 1145//3 /445/333388.
Simin Chen, Pranav Pusarla, and Baishakhi Ray. Dvao::Eval: Dynamic benchmarking of reasoning capabilities in code large language models under data contamination. In Proceedings of the 42nd International Conference on Machine Learning, volume 267, pp. 8890-8909, 2025. URL html.
https: http///cceeiideee ngs ml www.spressvvee. 67 chen25ba .rr.eerrsssss.....r.....s.s.

---

Published as a conference paper at ICLR 2026
Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. ThatUniTest: A framework for LLM-based test generation. In Companion Proceedings of the 32nd ACM International Conference the Foundations of Software Engineering, pp- 572-576, 2024b. URL httpss: / /oii .org/r C 48550 arxiv 2305. 04764.
Cursor. Cursor: The best way to code with AL. https://cursot.comm/ 2025. Accessed: 2025-09-23.
Android Developers. Application fundamentals, 2025a. URL htttpe:/ddeveloper..oo android. http//gude/ccomoo htweeettttttudddaaanaaaaaaa Accessed: 2025-08-15.
/
Android Developers, Fragments, 2025b. URL /tt.ttttss///eeeepp.....aaddddda
fragments. Accessed: 2025-08-15.
Android Developers. Services overview, 2025c. UBL.tttpss//deeeeooorr...ddddddd heweellop/bbcccgrooudwwwoott////
Accessed: 2025-08-15.
Peng Di, Jianguo Li. Hang Yu, Wei Jiang, Wenting Cai, Yang Cao, Chaoyu Chen, Dajun Chen, Hongwei Chen, Liang Chen, Gang Fan. Jie Gong, Zi Gong. Wen Hu, Tingting Guo, Zhichao Lei, Ting Li, Zheng Li, Ming Liang, Cong Liao, Bingchang Liu, Jiachen Liu, Zhiwei Liu, Shaojun Lu, Min Shen, Guangpei Wang, Huan Wang. Zhi Wang. Zhaogui Xu, Jiawei Yang. Qing Ye, Gehao Zhang, Yu Zhang. Zelin Zhao, Xunjin Zheng, Hailian Zhou, Lifu Zhu, and Xianying Zhu. CodeFuse-13B: pretrained multi-lingual code large language model. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Pructice, pp.
418-429, 2024. URL htt.://///..a.orr........1111113
F-Droid. Amaze File http://///aaadddooo////////////p .omnoo filemanager/, 2025a. Accessed: 2025-09-24.
..zee....
F-Droid. Arcticons. arcticons/, 2025b. 20255. Accessed: 2025-09-24.
F-Droid. Vanilla Music. nttps://f-drof http:/i...rggeep.cceeeeeeeeeee/e http://awtrr..oooooeeoocapaappap android. vanilla, 2025c. Accessed: 2025-09-24.
Open Technology Fund. F-Droid sustainability. weeeectss-e-ssupooo 2025. Accessed: 2025-08-15.
GeeksforGiekss. opment, 2025.
A complete guide to learn XML for Android app devel-
wwwp http:///wwwwwweefgrgeeeeeeeeee gwwkseorgeees org/androdd/ URL https:
a- d-complete-a geeee--uudeeoooo learn for -odroiddd app- developeenttt//// Ac- cessed: 2025-08-15.
GitHub. GitHub http:://gtthh..ccm/aaatteeeeeees 2025-09-23.
2025, Accessed:
Google. Gemini 2.5 Pro model card, 2025. URL https///::/ ctte::::///mooelacreeeee cards. wi tt www.whhhte. wwwoggll.c.ooooooooo.oo.ooooooo. com/
assets/document. wwwpttttseeemiiiiiieme 5-pro pdf. Accessed: 2025-09-23.
Batu Guan, Xiao Wu, Yuanyuan Yuan, and Shaohua Li. your benchmark (still) useful? Dynamic benchmarking for code language models. arXiv preprint arXiv:2503. 266.:0633330 2025. URL https:
/laaxivvorrg///..o..l...oo..o.oa abs 2503. 06643.
Daya Guo, Qihao Zhu Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li. Fuli Luo, Yingfei Xiong, and Wenfeng Liang. DeepSeek-Code -Coeer: When the large language model meets programming the rise of code intelligence. arXiv preprint mmXiv:2401.14196 2024. URL https:
org/ abs 2011 14196.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.0797. 2024. URL
petps://aaxxe www.iv.oog/vaaae www./ogg/......ooooooo wtwp//////3000022 079744
12

---

Published as a conference paper at ICLR 2026
Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang. Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self-planning code generation with large language models. ACM Transactions on Software Engi- neering and Methodology. 33(7):1-30, 2024. http::///ob..cc//11111////////0/
Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https: htttttt///ppeeeeeeeeeeee //ppnrevie...e... Sett/Eorum?ddvVIB
6yN
Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. InferFix: End-to-end program repair with LLMs. In Proceedings of the 31st ACM Joint European Softwree Engineering Conference and Symposium on the Foundationss of Software Engineering, pp. 1646-1656, 2023. URL 3613892.
Kimi Team. Kimi K2: Open agentic intelligence. arXiv preprint arXiv:2507.2000 2025. URL
https: //axivi org/ alos 20534..
Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz. Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with AlphaCode. CoRR, abs/2203.078 14, 2022. URL https: htop::://diin org//000. 48550/ wat.ir.viii.///.. 2203. 07814.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023.
URL https: /dl acm. orgg doi /10 5555 36661 .2. 3667065.
Junwei Liu, Yixuan Chen, Mingwei Liu, Xin Peng, and Yiling Lou. Stall+: Boosting LLM-based repository-level code completion with static analysis. arXiv preprint arXiv:2406.1000 2024.
URL httph:tts////
//apii:://xxxxvv /awxiv..iivvvvvv///////a.//aaii/ org. abs/2406 10018.
Alex Mathai, Chenxi Huang, Petros Maniatis, Aleksandr Nogikh, Franjo Ivančić, Junfeng Yang, and Baishakhi Ray. KGYM: platform and dataset benchmark large language models Linux kernel crash resolution. In Advances in Neural Information Processing Systems, volume 37, pp. 78053-78078, 2024. URL https: httpss//proceedins..euee... cc/ http:/rppppp.cpppp_eeeeeeee. files/
Sat__Bnnchaarks_Tracee
René Mayrhofer, Jeffrey Vander Stoep, Chad Brubaker, and Nick Kralevich. The Android platform security model. ACM Transactions Privacy and Security, 24(3), 2021. URL https: //doi.
org/10.1145/3444
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023. URL UtUhttps://bii.o//111 01:55//axxxv.233..002
OpenAI. GPT-4.1 model. 2025. Accessed: 2025-09-23.
OpenAl. GPT-5 system card. 2025.
URL https://ccn.opeeaal.co
carstem-ccrd.ppp card. pdf. Accessed: 2025-09-23,
Qiwei Peng. Yekun Chai, and Xuhong Li. HumanEyal-XI: A multilingual code generation benchmark for cross-lingual natural language generalization. In Proceedings of the 2024 Join! International Conference on Computational Linguistics. Language Resources and Evaluation, pp. 8394, 2024. URL https: http:///aaaaattttaaaeeee www///////2/ lrec-main. 735.
F-Droid Project F-Droid anti-features documentation. https: http://fffoooidddoooooo /
Ant Anti-Feature// 2025a. Accessed: 2025-08-15.
13

---

Published as a conference paper at ICLR 2026
F-Droid Project. F-Droid inclusion how-to. httpss//E--ahcidd.oo///////dd
Inclusion Inclusion_Hov-Io// 2025b. Accessed: 2025-08-15.
Stypox / F-Droid Project. F-Droid website overview. https://stypox: getpoox.giitaii
qit lab io/
2025c. Accessed: 2025-08-15.
Xingzhi Qian, Xinran Zheng, Yiling He, Shuo Yang, and Lorenzo Cavallaro. LAMD: Context-driven Android malware detection and classification with LLMs. In 2025 IEEE Security and Privacy Workshops (SPW). pp. 126-136, 2025. URL https: //oii ieeecompttersoccttyyy .rrg/ org/
10.1109 7851 2025 00017.
Jielin Qiu, Zuxin Liu, Zhiwei Liu, Rithesh Murthy, Jianguo Zhang. Haolin Chen, Shiyu Wang, Ming Zhu, Liangwei Yang, Juntao Tan, Zhepeng Cen, Cheng Qian, Shelby Heinecke, Weiran Yao, Silvio Savarese, Caiming Xiong, and Huan Wang. LoCoBench: A benchmark for long-context large language models in complex software engineering. arXiv preprint arXiv:2509.09614 2025, URL aww.vv..vvvvvvv////i//.a. wttp://////g/.///// 09911.
Tanmay Rajore, Nishanth Chandran, Sunayana Sitaram, Divya Gupta, Rahul Sharma, Kashish Mittal, and Manohar Swaminathan. TRUCE: Private benchmarking to prevent contamination and improve comparative evaluation of LLMs. arXiv preprint arXiv:2403.0039, 2024. URL https:/ //tpxvaxxvvvv////////////////.v org/abs 2.03.00000000 240..00333....... 00393.
Dezhi Ran, Hao Wang. Zihe Song. Mengzhou Wu, Yuan Cao, Ying Zhang, Wei Yang, and Tao Xie. Guardian: A runtime framework for LLM-based UI exploration. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 958-970, 2024. URL
12 36880334.
Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ra- manathan, and Baishakhi Ray. Code-aware prompting: A study of coverage-guided test generation in regression setting using LLM. Proceedings of the ACM on Software Engineering, I(FSE): 951-971, 2024. URL https: //t::///ooii. org 10. 45/ /
Justin Sahs and Latifur Khan, A machine learning approach to Android malware detection. In 2012 European intelligence and security informatics conference, pp. 141-147, 2012. URL https:
(eddii.org/1i 1101 /009//EIII...... 34.
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. JEEE Transactions on Software Engineering, 50(1))85-105, 2023. URL https: //wi. wwt.oo../o.oo....oooo 1109 /TSE. 0020..3334993
Technource. Google Play store statistics: Facts and numbers you must know. https:/ // www. www.nhurcccccecceeo 2025-08-15.
stat statistics,, 2022. Accessed:
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang. Zijian Wang. Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. ReCode: Robustness evaluation of code generation models. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume 11w.11...163111111 1: Long Papers), pp. 138 18188-3843, 2023. URL https: 11t.://........1111133//// acl mel--ong.rr 773.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. LiveBench: challenging, contamination-free LLM benchmark. arXiv preprint arXiv:2406.I 19314, 2025. URL https: wtwaiiiivv. www.aiwww.......e.eeeeeeeeooeeee abs 6... 1.44..911..... 19314.
Chunqiu Steven Xia and Lingming Zhang. Automated program repair via conversation: Fixing 162 out of 337 bugs for S0.42 each using ChatGPT. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 819-831, 2024. URL https:
//
14

---

Published as a conference paper at ICLR 2026
Kai Xu, YiWei Mao, XinYi Guan, and ZiLong Feng. Web-Bench: A LLM code benchmark based on Web standards and frameworks, arXiv preprint arXiv:2505.07473, 2025. URL https: //t////aaaaa //arxivv.... org/ http//bbb//55... 07473.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang. Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388. 2025. URL https: / aariivv..v org/ abss 2505. 09388.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Proceedings of the 38th International Conference on Neural Information Processing Systems, vol- URL https: /dl///d.//////// acm. org /ttp://oo/1...55335. 0tt.:////..55553777111111
ume 37, pp. 10528-506555 2024. 3739517.
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. CoderEval: A benchmark of pragmatic code generation with generative pre- trained models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1-12, 2024. URL https: /oiiioiiio.o/ org htt.://1.111111 / 3555533395533333 3627533333333311
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. SWE-bench goes live! arXiv preprint arXiv:2505.23419, 2025. URL https::///... arxiv.. org/ 2at.//555.241...
Qin Zhu, Qingyuan Cheng, Runyu Peng. Xiaonan Li, Tengxiao Liu, Ru Peng. Xipeng Qiu, and Xuanjing Huang. Inference-time decontamination: Reusing leaked benchmarks for large language model evaluation. In Findings of the Association for Computationnl Linguistics: EMNLP 2024, pp. 9113-9129, 2024. URL https: /ates://acllaann /aaaaaaaannntoooooee org/oo/g///// 2024. findings-emnni 532/-
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li. Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujum Cheng. Jiawei Liu. Qian Liu, Zijian Wang. Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 15877, 2024. URL https: / //axxiv. org/abs/2406. 15877.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A DETAILED DISCUSSION OF RELATED WORK
ML for Software Engineering. Machine learning, including large language models (LLMs), is increasingly used to address real-world software engineering tasks due to their advantages over traditional program analysis techniques. Typical use cases include automatic code generation (Chen et al., 2021; Austin et al., 2021; Liu et al., 2023; Chen et al., 2024a), malware detection (Qian et al., 2025; Sahs & Khan, 2012; Chen et al., 2020), test generation (Chen al., 2024b; Ryan et al., 2024; Schäfer et al., 2023), and program repair (Jimenez et al., 2024; Jin et al., 2023; Xia & Zhang, 2024; Yang et al., 2024).
Most relevant to our APPFORGE are works that apply LLMs to automated code generation or code completion (Chen et al., 2025; Wang et al., 2023; Athiwaratkun et al., 2023). However, existing code generation datasets are largely limited to the function level (Chen et al., 2021; Austin et al., 2021; Liu et al., 2023; Chen et al., 2025; Yu al., 2024), and repository-level work focuses mainly on code completion rather than generation from scratch (Liu et al., 2024; Bairi et al., 2024). Compared with existing datasets, APPFORGE introduces a more realistic and challenging setting for evaluating the capability of LLMS to perform software development from scratch. This setting better reflects real- world development scenarios where models must synthesize coherent, functional, and maintainable codebases instead of compleing the missing lines in existing codebases.
Code Generation Benchmarks. Many benchmarks have been proposed to evaluate the code generation capabilities of LLMs (Guan et al., 2025; Chen et al., 2024a; Yu et al., 2024; Jimenez et al., 2024; Mathai et al., 2024). HumanEval and MBPP introduced human-crafted datasets focused on synthesizing function-leve code from natural language descriptions and have become standard benchmarks (Chen et al., 2021; Austin et al., 2021). Building on this, HumanEval-
(Peng et al., 2024) extended HumanEval to support multilingual settings. Moreover, EvalPlus (Liu et al., 2023) highlighted limitations in HumanEval and MBPP, particularly their limited test case coverage, and proposed a more rigorous evaluation benchmark. BigCodeBench (Zhuo et al., 2024) introduced a larger-scale benchmark designed to further evaluate LLMs' code generation capabilities. Beyond these function-level code generation benchmarks, recent repository-level benchmarks have also been proposed. For example, SWE-Bench (Jimenez et al., 2024) focuses on evaluating LLMs' patch generation ability at the repository level. Although effective, most benchmarks are static and lag behind LLM advancements, prompting the emergence of dynamic benchmarks for up-to- date, contamination-free evaluation. LiveCodeBench (Jain et al., 2024) collects newly released programming completion problems from online coding platforms to minimize data contamination. PPM (Chen et al., 2024a) and DyCodeEval (Chen et al., 2025) propose an automated method to generate new benchmark data the evaluation stage, mitigating potential data contamination. SWE- Bench-Live (Zhang et al., 2025) follows the LiveCodeBench schema to collect new patches from GitHub repositories, providing continuous and realistic evaluation environment.
Compared to existing code generation benchmarks, APPFORGE operates at the repository level and includes rigorous evaluation. It can be dynamically constructed by collecting latest projects from F-Droid, preserving a much broader range of challenges rooted in real-world software develop- mein- beyond closed-form completion or patch generation.
Android Application Ecosystem. Android applications represent one of the most significant software ecosystems globally, with over 2.6 million applications available on Google Play Store and millions more distributed through alternative channels (Technource, 2022). In Android ecosystems, F- Droid (2025c) is the leading open- repository, serving millions of users worldwide. Its rigorous review process and anti-feature labeling ensure high-quality software that often exemplifies Android development best practices. The repository spans diverse categories- proiesutvittcttitt....... multimedia, utilities, games, and specialized professional tools-many maintained by experienced developers and organizations (2025a; 2025b; 2025).
Android apps. typically built with Java or Kotlin following Material Design and Android architecture guidelines, comprise multiple interconnected components such as Activities, Fragments, XML layouts, and manifest configurations (2025a). Developing robust apps demands expertise across UI design, background services, data persistence, networking, device optimization, and security (Mayrhofer et al., 2021; Developers, 2025b; GeeksforGeeks, 2025; Developers, 2025c). Given the prevalence
16

---

Published as a conference paper at ICLR 2026
and complexity of the Android ecosystem, developing Android applications automatically provides significant commercial and practical impact. APPFORGE, filling a critical gap. provides the first benchmark for assessing LLM capabilities in Android development.
B
DETAILS OF APPFORGE
B.
TASK INSTANCES
Model Input
Expected Model Output
wel Instruction
aapp/src// aapp/src/main/And diderieefees xmll:
You are sutonomeus programmer tesked with modiying an Android application
"app/anc/ main/ wwwp///aamp/////eeee.ooo sttor/Maancttiitiiiissss
packuge org.
activityy
sepert androsd Att.-pp-cctiitt
Overview
andradd. www.ent. Contexts
An app that functions as a haric caltulator, allowing users to perform various anthmeticc coeeations including adiitionn subtraction, power.
multiplication diviion, square
Evaluation Suite
Fu Functionalities
festure Perform basic anthmetto operat ons. sub-feetue feeture 1-1: Add two numbers. When the user clicks on the button
Prociefined Test Coscs
APK
**...*
dispayyed the
Compiling
nittieddd by the resource-id "resutt" sub-feeture 1-2
(lape: 'dle. (Tog"""
festure 2: Parform ackanced anthmetic openations. teature 2-1: Caloulate square root number the expone atation of number.
Mopit Vaible.
Targe: Dogee:Teessuroos (Tessurce id": result) "teet 1087
Produced
Functional Testing
Fuzz Tes tistinn
#drt Imple mentaion Constraints
APK
Your oode will be complled with Gradle7755 5.1 and
Functional Testing Log
Fuzz Testing Log
Nexus_1LAR 1_APLSS
FAILED cakulator testesse imsiiill
RuZZING NG COMPLETED
Meaaz return ison soring of flles to revised and revised code in
elamentide tide rided by (rrsource-id"
Total Duration 600 seconds
tollowing formar:
"rem "wid" 401
lotal Gydee 30
Figure 11: Example of Task Instance.
B.2 STATISTICS OF SEED APPS.
Table 5: Statistics of Seed Apps used for Constructing APPFORGE.
LoC
Acts
# Files
Range
194-53K
1-21
2-508
Avg. 6367.2
3.7
44.0
Median
4530
3
26
B.3 EVALUATION METRIC CALCULATION
We employ four primary metrics that directly correspond the three evaluation stages to comprehen- sively assess the quality and functionality of generated Android code.
The compilation rate (Eq. 1) measures the percentage of generated code that successfully compiles into valid APKs without syntax dependency enrors during the compilation stage, indicating the basic correctness and completeness of the generated code structure. Here, Ncompiled denotes the number of successfully compiled cases and Nivua denotes the total number of tasks.
The test pass rate (Eq. 2) evaluates, using macro averaging, the mean percentage predefined test cases that pass across all compiled applications during the testing stage. This metric reflects how well the generated code implements the specified functionalities and meets behavioral requirements. For each compiled application (8)
passed denotes the number of passed test cases, and (8) woaa denotes the
17

---

Published as a conference paper at ICLR 2026
total number of test cases for that application. The macro average is computed by taking the mean of per-application pass rates over all Ncompited compiled applications.
The crash rate (Eq. 3) quantifies the percentage of compiled applications that crash terminate unexpectedly during the fuzz testing stage, assessing the robustness and stability of the generated code under various stress scenarios. Here, Nerashed_ apps upps denotes the number of compiled applications that experienced crashes during fuzz testing, and Ncompiled denotes the total number of compiled applications.
Additionally, the functional success rate (Eq. 4) measures the percentage of tasks that achieve both successful compilation and complete test suite passage, representing the overall functional correctness of the generated applications regardless of their robustness under stress testing. Here, Noompiled and_aanddddde rests_passed denotes the number of tasks meeting both criteria.
The pipeline captures detailed logs and execution traces throughout all three stages, enabling precise computation of these metries and comprehensive result reporting for large-scale benchmarking experiments.
N Weopiiled
Compilation Rate
100%
Nivial
Nevasited
(6)
Test Pass Rate
Σ
ppassed (5)
x 100%
(2)
N compiledd
rotal
N Verashed_appe
Crash Rate
x 100%
(3)
N Ncompiled
N Weompited_amm and__andsss lests passed
Functional Success Rate-
x 100%
(4)
Noval
B.4 IMPLEMENTATION DETAILS OF APPFORGE
To ensure consistent and reproducible evaluation across different systems, we establish a standardized execution environment that supports the entire evaluation workflow described above. The environment consists of two main components that work together to enable seamless automated evaluation. First, we utilize the official Android Emulator with API level 31 (Android 12) running on $86_64 architecture for APK installation and testing. This emulator configuration is packaged into Docker container that guarantees seamless deployment on Ubuntu systems without requiring any additional manual configuration. The containerized approach eliminates environment-specifa, issues and ensures that all evaluations are conducted under identical conditions, enabling reliable APK execution and configure standardized Gradle build environment within the
test case validation. Second, same container to handle the automated compilation process. This includes pre-installed Android SDK components, build tools, and dependency management configurations that are commonly used in Android development. The Gradle setup is optimized for automated compilation of generated code files into APKs, with appropriate timeout settings and resource allocation to handle various code complexity levels. The build environment is configured to provide detailed compilation error messages when needed, supporting the iterative refinement workflow for models that can benefit from error feedback.
B.5 RANKING CRITERIA OF APPS IN F-DROID
The ranking of F-Droid apps follows a sequential process that begins with filtering the dataset to retain only actively maintained and popular applications, requiring at least 50 GitHub stars, 10 forks, non-archived status, update date no earlier than 2022, and primary implementation in Java or Kotlin. For each app that passes the filter, an LLM assigns integer scores from to 5 for four evaluation aspects: maintainability, which measures how easy the source code is to understand and modify; reproducibility, which reflects the ability to produce consistent results under stable conditions; generality, which indicates how broadly the app can operate across different devices and configurations; and evaluation efficiency, which assesses the speed and resource requirements of building and testing the app. The average of these four scores forms the quality rating. The difficulty
18

---

Published as a conference paper at ICLR 2026
ranking is then determined by combining the LLM-generated complexity activities in the app and the total code size in bytes. Based on these combined criteria, the ranking process selects the top 40 apps within each difficulty level, resulting in final list of 200 top-ranked apps that are representative, maintainable, reproducible, and diverse in technical challenge, ensuring comprehensive benchmark for assessing the Android development capabilities of LLMss.
with the number of
B.6 PROMPT EXAMPLES
You are an autonomous programmer and you are modi fying default Android app template with empty activity. Your code will be compiled with Gradle7.5.2 and run on Nexus_ Texus_A_API_311
The default Android app template "File Structure" is shown below.
You replace add some files in the templates to implement the app.
"Fi le Structure":
-- app
build. build.gradle
src
main
-- AndroidManifest. xml
10
Your app should implement every feature in "App Features", and we' 12
11 test each of the features. Note that you should pay attention to the resource-id, content-dese, texts and other attributes provide with corresponding widgets in "App Features" and exactly match the attributes when implementing the widgets.
13
"App Features":
14
description: Android app for quickly sharing your current location.
15
feature Share your location via
16
17
Please return json string and only json string of files to be revised and the revised code in following format:
vaap:/sre/math/nntcciaaaaaaee.e :mateesstt.l....................
20
22
C DETAILED EXPERIMENT RESULTS
C.I
SETUP
All LLMs use identical task prompts provided by APPFORGE as well as the same hyperparameter settings (with the temperature set to 0.2 using greedy decoding (Brown et al., 2020)).
C.2 DETAILED STATISTICS
Figure 6 and 7 present the detailed statistics developed by different LLMss.
of compilation errors and runtime errors of the apps
C.3 FULL EXAMPLE OF DEFENSIVE PROGRAMMING BY GPT-5.
Listing 1 presents the full example of the defensive programming achieved by GPT-5.
19

---

Published as a conference paper at ICLR 2026
Table Compilation results of LLMs.
Congelo
Padkago Nave
Antl NOMERO
Wong ISX
Chaar
Expared Fagg
Regures
Conglaa Tixcet
Rurdy
Caskod Seced
40.0
8.0
34.0
18
0.3
200
0.0
35
0.5
3.55
GPT/41
700
0.
60
7.3
128
3364
7281
0.0
46
25.0
00
12.0
18
68
72
05
35
148
1343
4.
3.8
30
691
Despled-15
210
280
10
35.0
100
10
80
0.
35
Compuloone Sove Fendiied
Caale Sexnct
75.0
1.0
50
044
225
0.0
26
Cladk Oyes
88
0.34
GPT-411
13.0
05
130
193
10
1.3
6.2
4.00
18
8.0
0.
0.56
3.5
200
GLMESS
10
0.30
9.8
2.0
65
Daepteak-V
27.0
10
36.
31.0
034
18
30
10
0.0
276
Table 7: Application Runtime Error Statistics
Pass@
with Compilation Error Feedhack
Model
ANR
Native Crash
Failed Start
ANR
Native Crash
Failed Start
Kimi K2
0.0
11.0
3.0
1.0
13.0
18.0
DeepSeek-V3
0.0
4.0
2.0
0.0
11.0
4.0
Qwen3-Coder
0.0
19.0
4.0
0.0
16.0
9.0
GL.M-4.5
0.0
15.0
5.0
0.0
27.0
15.0
DeepSeek-R1
0.0
8.0
4.0
1.0
25.0
6.0
GPT-4.
0.0
0.0
2.0
0.0
11.0
66.0
Claude-5-Opus
0.0
48.0
9.0
0.0
48.0
11.0
GPT-5-High
1.0
21.0
5.0
0.0
0.0
25.0
Gemini-2.5-PM
0.0
25.0
14.0
0.0
37.0
21.0
Claude-4-Sonnet
0.0
21.0
10.0
0.0
0.0
20.0
Listing 1:
Case study showing proactive defensive programming implementation by GPT-5 in the Autostarts app. including robust null-safety checks, multi-level fallback strategies for system navigation, and guarded UI state transitions beyond basic requirements.
// --- Null Bull-astety
Input favold NPE1
private void filtertist (String quary) { String Iqvery null) *" query.trim() -tooowerCaae (Tocale ROOT) // densivee for (AppEntzy e appEatries) { boolear q.isempty e.tisle. toLewertase (Locale. cot..:ooee..c...aaaaaaooo.. conteins wwww..etvicibilityy... (match ? VISNLEESIBLE View.GoNE);
//
guard: search safely
filters
11 private void closeSearchifogeet]
visiLLE) ietenttannersseviiiis. 1 (Vie).CONE); searchInput setText ("");
(scarchContaines .it...g.i..iiissbblllllll
defensive visibility check
15
// reset state
15
filterbist ("*),
// restore full 1ist
16
18
giverride
20
public word onRackPreased
21
if deearchContaines 1- null s& searchCont wineeeootttiiinttn.eeenn oet.eegeee
guard
22
closeSearchiFOpen
consume back to close search instead of leaving activity
25
return;
25
super. (ngescnnncckkressed)))
27
28
//
Best effort nevigation with graceful degredation
private void openAgpindd (String packagetimme)
30
try
Intent (Sestings. WAW.ON AATLOOLAAPPPLCCIIIOOOOOOOO SECTLIGS)) Intent (Uri-parsa paappaaaaaaaiiaaaaaaa ("package:" packageName)
32
it st. (Uri
it.dddF aga (Tatert. PTAG AIAG_ACTTTTTY_ NASK)
20

---

Published as a conference paper at ICLR 2026
startAstivityyEE)
35
catch (Actsittyyoooaandkkccttt el // devica/RCM mi amotch fal bolkk try
37
Iatent fallbsck Intent (Settings
CECACCCTTTOOOODTTIILLSSS OCTAIIS_SCTTINNG)
38
sellback.setDate (Uri. ("pckkage"" + getzackagshame (1)) // fallback to self
40
catch (oxception ex) I (this, "Uncbla open Application Info", Teast LENGTH show()
// laat resort: user-viatble error
42
43
46
45
// Defensivee null lcchecheok optioonll
47
private void www.lennoorGoggeeuus..
|
AlertDialog. But (thi(th
49
ssetTitle ("Goggl"")
50
se:IIees Inew CharSequence () ("Disable", "Application Info"I, (d, which)
51
if (which
52
if (statusicoglaterring 1= mull]
// view presence not guarenteed across
layouts
53
(catusooolebuudin..eelaaaaa (View. VISIBLE);
55
Toast.makeTee. (this, "Google autostart disabled for During Startup", Teast. SENGTILSHORTI .shww()
56
eise if (which == 1)
57
( le. android andrriiidddgooo www.gooogigoeerrrrieoooo
59
3)
()
61
21


Reviews:
[{"review_id": "sKfGNhAKYE", "reviewer": "Reviewer_Fhv9", "summary": "The paper presents a benchmark for tackling end-to-end app development. This provides a basis for assessing the abilities of models to be independent developers of complex applications and not just individual aspects of software engineering cycle. \n\nThe benchmark is a collection of 101 Android applications sourced from F-droid, the open source Andriod repository. An interesting question in building this is that of how does one obtain the specifications for the task. The paper makes use of application traces from the original apps and then uses LLMs to synthesize test cases from which LLMs generate app description.  The evaluations consist of test cases thus generated. The entire set of tasks were manually verified and validated by software developers. \n\nThe evaluations demonstrate that the tasks are challenging for current state-of-the-art models and coding agents.", "questions": "See questions in weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The benchmark targets end-to-end application development. Relative to existing benchmarks this presents a new investigation, one that is also likely impactful. \n\n- The benchmark creation process is sound. The apps are sourced from real applications and thus represent real needs and the levels of complexity expected in real apps. \n\n- The evaluations test a wide range of models and some coding agents. The substantially low performance numbers indicate the difficulty of this task and provide a useful empirical point on the capabilities of LLMs for end-to-end software engineering. The analysis are generally sound and provide detailed insights into the failure modes and in some cases posits reasons for why. These provide useful insights for future work on this dataset.", "weaknesses": "The generation process of the target task descriptions and the test cases are well motivated. I also see that the task descriptions and the test cases have been validated. However, I see some areas of improvement here.  \n- First,  it is not clear why this would correspond to how users would want to provide descriptions of apps. It would help to motivate how the authors envision such a system being used and how app development specifications are produced. \n\n- Second, the Android Developer Validation section can be improved in terms of details. Currently, it outlines the kinds of checks being done on the task descriptions and the test cases. However, it leaves out some details on what happened when developers found errors, inconsistencies, or completeness problems. Did they modify the descriptions manually? Did multiple developers iterate over each others validation process? Was there any specific validation that checked whether the test cases were clearly aligned with the task descriptions?\n\n- In practice, apps are rarely developed with full functionality in one go, and specifications themselves are often refined in the process of development. Is it reasonable to expect models to develop the entire functionality in one attempt? The process outlined allows for iterative development of functionality in the sense that failures are allowed to be corrected. I wonder if it will be useful to test, if feasible, incrementally add functionalities to the app. At the very least some acknowledgement of how software engineering for apps operates in practice and what differences is being posed in the current setting would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L0oBuhKMS0", "reviewer": "Reviewer_Y9mg", "summary": "This paper introduces APPFORGE, a new end-to-end benchmark for evaluating LLMs on real-world software development tasks. APPFORGE requires models to develop full Android applications from scratch, given only natural-language specifications, unlike existing code benchmarks, are focus on a single task. Under human expert verification, each task contains a detailed functional description, natural-language test cases, and implementation constraints for comprehensive information to resolve the problem. Finally, they evaluate 12 SOTA models and two code agents to show that currently, LLMs are still far from handling such an end-to-end development task.", "questions": "1. It might be good to cite some secure code generation work as a shared related interest for LLM coding.\n\n[1] Seccodeplt: A unified platform for evaluating the security of code genai\n\n[2] SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code\n\n[3] CodeLMSec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models\n\n2. It might be good to cite more code agent works as a shared related solution for LLM coding.\n\n[4] OpenHands: An Open Platform for AI Software Developers as Generalist Agents\n\n[5] PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification\n\n[6] Agentless: Demystifying llm-based software engineering agents\n\n3. For future work, it might be exciting to extend to other ecosystems, like iOS, web or desktop. Because different scenarios might involve different programming languages and unique challenges for LLM.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel problem formulation. APPFORGE proposes code evaluation by shifting from “code generation” to “full software engineering.” It evaluates end-to-end development — design, implementation, integration, and runtime — not just isolated snippets.\n\n2. Complete and reproducible evaluation pipeline. The benchmark provides a fully automated, Dockerized workflow: natural-language task → JSON output → Gradle build → emulator test → fuzz evaluation. Both contain static and dynamic evaluation in multiple dimensions.\n\n3. Realistic and validated data construction. Tasks are grounded in real F-Droid apps, with automatic conversion by LLM and human-in-loop verification to guarantee the quality of the tasks. Also, the tasks contain many aspects and kinds of sub-tasks that are highly related to realistic development problems.", "weaknesses": "1. The JSON format output is not flexible enough for evaluation. It is understandable that this work is mainly focused on evaluating LLM rather than agents, and such an evaluation method can avoid a lot of environmental problems. But for the SOTA models, agentic tool call and planning are important capabilities, which should be useful and need to be evaluated. Maybe can provide more evaluation for agentic workflows with different models.\n\n2. Currently, generating code based on JSON format may lead to the model needing to output a large amount of code at once when the task is complex. Long outputs may cause a decrease in the quality of the generated code. However, it is uncertain whether this potential problem can be solved by using an agent, COT, or other multiple polling methods. Does the author have any experiments or ideas on that? And it might be good to show the time-consuming and cost for agent evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vskdtJJvbb", "reviewer": "Reviewer_TWQk", "summary": "This paper presents APPFORGE, a novel benchmark designed to evaluate the end-to-end software development capabilities of large language models (LLMs) in the context of Android app development. Unlike existing benchmarks that primarily assess function-level or patch-level code generation, APPFORGE targets real-world software engineering tasks, including system-level reasoning, component integration, UI design, state management, and exception handling. The benchmark includes 101 tasks automatically constructed and validated from real-world open-source Android apps, and features a fully automated evaluation pipeline involving compilation, test execution, and fuzzing. The authors evaluate 12 state-of-the-art LLMs (including GPT-5, Claude-Opus, and others), revealing that current models perform poorly on this benchmark—with a maximum success rate of only 18.8%—thus highlighting the significant gap between current capabilities and the demands of real-world software development.", "questions": "1. In Table 1, are the results under “with Compilation Error Feedback” obtained from a single iteration or multiple rounds of refinement? For instance, Qwen3-Coder reports 85.15% compilation success, but this seems inconsistent with the values in Figure 5. In Figure 5, on the line for Qwen3-Coder Compile, there seem to be no node values corresponding to 85.15%. Please clarify.\n2. Could the authors provide the prompt used for iterative generation based on the previous compilation error output?\n3. For the observed “avoidance behavior”, could the authors provide concrete cases and detailed analysis? Specifically, is this phenomenon a result of training data bias, a form of lazy model strategy, or prompt design shortcomings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The first to comprehensively evaluate LLMs in full Android application development from scratch, addressing a crucial gap in current code evaluation benchmarks.\n2. Tasks are derived from real-world, actively maintained F-Droid apps, ensuring diversity and authenticity.\n3. The benchmark combines automatic task construction and evaluation with expert validation, ensuring high-quality and reproducible results.\n4. The paper provides a breakdown of LLM failure cases, behavioral patterns (e.g., avoidance strategies), and complexity trends.", "weaknesses": "While the paper identifies that some models (e.g., GPT-4.1) evade development by deleting faulty code instead of fixing it, it does not offer a detailed exploration of this behavior. There is no case analysis or inspection of prompts provided to the model during iterations. The paper does not clarify whether this is due to training data bias, a form of “lazy” behavior by the model, or a failure in prompt engineering. This limits the explanatory depth of an otherwise critical observation.\n\nThe title poses a broad question—“Are GPTs Ready for Software Development?”—but the empirical scope is restricted to Android app development. While Android development is non-trivial, its characteristics (e.g., strict lifecycle constraints, XML UI design, framework conventions) do not generalize to other domains such as backend systems, web applications, or embedded systems. The gap between the title’s generality and the study’s actual scope should be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jFB96oZKaJ", "reviewer": "Reviewer_WwYL", "summary": "The paper presents a benchmark called AppForge. The goal is to evaluate LLM's capability, beyond generating isolated snippets or functions. The argument is that existing benchmarks do not test system-level reasoning and integration required to build a complete application. To this end, AppForge builds on Android application development as the benchmark domain, and it is based on the top 200 highest-scoring apps across different categories. Evaluations compare how 12 LLMs perform on AppForge.", "questions": "Please refer to the \"Weaknesses\" section, for my questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Reading this paper brings back the fond memory of developing Android apps.\n\n1. I can see how app development can support the claim of going beyond generating isolated code snippets and functions. The idea of using Android apps as the benchmark domain seems reasonable.\n\n2. While I haven't personally used a lot of LLM benchmarks, the explanation on how AppForge is different from others is clear to me.\n\n3. I also like that AppForge includes a diverse range of apps. And, evaluations (over 12 LLMs) suggest that AppForge can differentiate model capabilities better than existing code generation benchmarks.", "weaknesses": "I have been thinking whether I would use AppForge myself, and the following aspects are not clear to me:\n\n1. What LLM capabilities is AppForge really testing? I understand that the end-goal is generating fully-working Andorid apps, but app development is simply a scenario, not a capability. From the introduction, the capability seems to be system-level reasoning, but what is system-level reasoning? As a system researcher, I find that system-level reasoning can be an overloaded term: reasoning about system behavior, properties, states, and so on?\n\n2. Is the scoring system just pass-or-fail? Developing a fully-working app is non-trivial, and it involves a lot of planning and implementation steps. Is it possible that minor mistakes along the way result in app failure?\n\n3. AppForge is based on top 200 highest-scoring apps across many categories. I am wondering whether solution leaking would be a problem. In other words, are there open-sourced version of these 200 apps, whose source codes are present in the LLM training data? After all, these are highest-scoring apps, and it is likely that people have released open-sourced version?\n\n4. One minor question. Software development is typically imposed with budget constraints. How does AppForge ensure that the better-performing LLMs are not using excessively more budgets than others?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)