The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution
Authors: Fei Gu
Zi Liang
Jiahao MA
Hongzong LI
Venue: ICLR 2026 Poster (2026)
Keywords: AI programming assistants
large language models
code generation
Matthew effect
software ecosystem evolution
programming languages and frameworks
multilingual benchmarking
agentic coding
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: AI-assisted programming is rapidly reshaping software development, with large language models (LLMs) enabling new paradigms such as vibe coding and agentic coding. While prior works have focused on prompt design and code generation quality, the broader impact of LLM-driven development on the iterative dynamics of software engineering remains underexplored. In this paper, we conduct large-scale experiments on thousands of algorithmic programming tasks and hundreds of framework selection tasks to systematically investigate how AI-assisted programming interacts with the software ecosystem. Our analysis quantifies a substantial performance asymmetry: mainstream languages and frameworks achieve significantly higher success rates than niche ones. This disparity suggests a feedback loop consistent with the Matthew Effect, where data-rich ecosystems gain superior AI support. While not the sole driver of adoption, current models introduce a non-negligible productivity friction for niche technologies, representing a hidden bias in software evolution.

Published as a conference paper at ICLR 2026
THE MATTHEW EFFECT OF AI PROGRAMMING ASSIS- TANTS: A HIDDEN BIAS IN SOFTWARE EVOLUTION
Fei Gul Zi Liang? Jiahao Ma'
Hongzong Li
City University of Hong Kong (Dongguan)
The Hong Kong Polytechnic University
The University of Hong Kong
*The Hong Kong University of Science and Technology
ABSTRACT
Al-assisted programming is rapidly reshaping software development, with large language models (LLMs) enabling new paradigms such as vibe coding and agen- tie coding. While prior works have focused on prompt design and code generation quality, the broader impact of LLM--riven development on the iterative dynam- ics of software engineering remains underexplored. In this paper, we conduct large-scale experiments thousands of algorithmic programming tasks and hun- dreds of framework selection tasks to systematically investigate how Al-assisted programming interacts with the software ecosystem. Our analysis quantifies a substantial performance asymmetry: mainstream languages and frameworks achieve significantly higher success rates than niche ones. This disparity suggests a feedback loop consistent with the Matthew Effect, where data-rich ecosystems gain superior AI support. While not the sole driver of adoption, cur- rent models introduce non-negligible productivity friction for niche technolo- gies, representing hidden bias software evolution.
INTRODUCTION
Large language models (LLMs) have quickly become ubiquitous in software engineering practice, with nearly all programmers (Daigle & Stafi (024) utilizing AI coding tools. Tools such GitHub Copilot, Cursor, and integrated LLM- coding assistants now support developers algorithmic problem solving (Yan et al.) 2023), debugging, and even full-stack system construction. These ad- vances introduce new coding paradigms: vibe coding, where developers iterate by prompting rather than typing every line, and agentic coding, where autonomous agents plan and execute end-to-end development tasks. Vibe coding democratizes software development (Gadde 2025 by lowering barriers to creation, translating conceptual intent into executable implementation. Agent-based code generation highlights the transformative potential of multi-agent systems in addressing the limita- tions of standalone LLMs. Agentic Coding effectively handles real-world coding challenges (Wul et al. 2024) by leveraging external tools for retrieval, achieving significant improvements (Huang al. 2023 in pass rates across diverse benchmarks (Zhang et al., 2024). Collectively, AI Coding could be the silver bullet for software engineering.
Prior to this empirical reality check, there was widespread optimism that LLMs would serve "Great Equalizer." " Recent studies show that LLMS help narrow the skill gap for junior developers (Noy & Zhang, 2023) [etabobb] (024). However, the belief that they also function as a language equalizer, making specific syntax irrelevant suggested by (Huang 2024), has not yet been tested empirical settings. We challenge this assumption and argue that instead of flattening the land- scape, AI support introduces critical new factor that may disproportionately disadvantage niche ecosystems.
Long-term ecosystem-level consequences of AI programming assistance remain underexplored. This research gap critical because biases in training data and model behavior may systemat-
"Co-First Authors.
'Corresoondng author: 1 hhhongongguust..n.

---

Published as a conference paper at ICLR 2026
Crunier
Doplink
Techend Euchediog
87
Figure 1: Two-tier experimental pipeline combining algorithmic tasks and framework tasks.
ically influence which languages, frameworks, and paradigms thrive decline. Several obser- vations underscore why this matters. First, LLM performance is uneven across languages: high- resource ecosystems such Python achieve disproportionately strong results, while lower-resource languages receive much weaker support. For instance, the StarCoder dataset shows Python alone ac- counts for nearly 40% of its training corpus, while many other languages appear only marginally (Li) iet al. 2023). Similarly, CodeGen explicitly notes that model quality varies substantially depending on training data availability, with mainstream ecosystems benefiting disproportionately (Nijkamp) jet al., [2022.. Second, the bias extends beyond languages to frameworks and usage patterns. Al coding assistants often over-rely on established libraries, such as NumPy, which appears in up to 48% of completions even when alternatives may be more suitable, and they also display persistent preference for Python, which is selected 58% of the time for performance-critica tasks where other languages like Rust may be objectively hetter (Twist et al. (025). together, these patterns raise a central question: do Al tools genuinely empower innovation by lowering entry barriers, or do they inadvertently reinforce existing dominance hierarchies?
The hypothesis we explore that Al programming assistants exhibit a Matthew effect: "the rich get richer." This effect is rooted in the operational mechanics of LLMs, which are trained mas- sive datasets of publicly available code. Such dynamics risk creating lock-in effects that suppress experimentation and reduce opportunities for paradigm-shifting innovations, which is consistent with prior observations on programming language adoption and diffusion (Meyerovich & Rabkin 2013). Programming learners may increasingly favor languages where Al support is strongest, fur- ther consolidating existing trends (Prather et al. (023). The empirical research on language adoption demonstrates that ecosystem factors (libraries, existing code, community size), rather than purely technical merit, strongly drive which languages gain traction, implying that model-mediated pro- ductivity gains could differentially amplify preexisting popularity patterns (Cao al., 2022). ). Most existing studies of Al-assisted code generation focus on short-term, licro-level evaluations that measure model performance on narrow benchmarks or single-language datasets, which do not cap- ture the multi-faceted complexity of real-world software engineering. If this impact is overlooked, the resulting cycle, where popular languages receive more LLM support due to their prevalence in training data, risks reducing programming ecosystem diversity. Thus, AI assistance could simulta- neously lower barriers to entry while stifling long-term innovation.
Contributions. This paper makes three main contributions:
We construct the first large-scale benchmark combining algorithmic programming tasks (Total 3011 x 9 x 5 135, 495) and complex full-stack development tasks to assess AI programming assistants across languages and frameworks.
We design controlled evaluation methodology that isolates the effect of language and framework popularity, revealing structural biases beyond aggregate accuracy metrics.
We quantify substantial asymmetry in AI support across languages and frameworks, demonstrating patterns consistent with Matthew-effect dynamics. While separating AI- specific amplification from pre-existing structural biases remains an open empirical ques- tion, our findings reveal measurable "AI productivity tax" that correlates with ecosystem popularity.
Figure I presents a high-level overview of the two-tier experimental pipeline, illustrating both language-level algorithmic tasks and framework tasks
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
2.1 AI PROGRAMMING ASSISTANTS
Research on LLM-based programming has largely focused on improving code generation quality, prompt engineering, and usability. Systems like Codex and Copilot have demonstrated high produc- tivity gains in everyday development. More recent models such as GPT-4, Gemini, and DeepSeek exhibit strong reasoning and multi-step planning abilities, further lowering the barrier to complex programming. Recent work has examined the capabilities of Al-assisted code generation tools across diverse benchmarks. Early evaluations using HumanEval (Chen et al.) 2021) revealed that while Copilot often produced syntactically valid code, correctness rates were low and highly cor- related with the prevalence of languages in the training data (Yetistiren et al.) (022). Although HumanEval became an early standard for evaluating LLM coding proficiency (Jiang et al.) 2024). its limited number of problems restricts its applicability across all research contexts. To address this limitation, subsequent studies employed LeetCode problems. Copilot's best accuracy in Java (Nguyen & Nadi 2022), subsequent analysis extended multiple tools, reporting that Copilot ex- celled in Java, ChatGPT maintained strong cross-language consistency (Batista et al.) (2024). and Gemini performed best in JavaScript. Larger-scale evaluations showed Copilot's accuracy decreased with problem difficulty and varied substantially across languages (Mo et al. 2025). The hyperscale (Khan et al., 2024), demonstrated persistent challenges for program synthesis and translation, especially in less common languages, though its automatically collected data lacks manual verification and may introduce noise. Existing benchmarks offer valu- able insights into Al code generation, but most focus on a few popular languages. By contrast, use a standardized CaeCoode-aseee dataset to examine how language popularity affects performance.
multilingual
2.2 PROGRAMMING ECOSYSTEM EVOLUTION
The evolution of programming languages and frameworks has long been shaped by ecosystem fac- tors such as community size, tooling, and industry adoption, often outweighing intrinsic technical merit (Meyerovich & Rabkin| 2013). In longitudinal study of web framework popularity, [Swachal & Kulpa (2023) demonstrated that adoption trajectories vary considerably across ecosystems. AI as a New Adoption Factor: The emergence of AI assistants introduces complex new variable to this evolutionary dynamic. Recent literature presents a nuanced picture of Al-assisted development: while studies report substantial speedups specific tasks (Peng et al.] 2023). concerns regarding correctness, over-reliance, and workflow disruption persist (Weisz et al., 2024). This suggests that AI is not simple linear accelerator but a nuanced influence on developer experience. However, recent surveys highlight a critical risk within this influence: LLM performance is uneven and heav- ily biased toward widely used languages (Zhang et al.) (023). Human-curated benchmarks such as CodeArena reveal persistent cross-language disparities, suggesting that LLMs may reinforce the dominance of mainstream ecosystems like Python and JavaSeript (Yang et al.) 2024). Beyond lan- guage adoption, ecosystem-level inequalities also emerge when LLMs act autonomous coding agents: AgentBench (Liu et al. 2023) shows that proprietary models such as GPT-4 exhibit superior reasoning and multi-turn decision-making these findings point to a risk that Al programming assistants could entrench existing hierarchies and amplify long-term ecosystem imbalances.
while open-source models lag significantly. Together,
ENVIRONMENT AND BENCHMARK CONSTRUCTION
To evaluate AI programming assistants, we designed a benchmark focused on two core software engineering components: languages and frameworks. Using controlled variable method, we first assess performance on algorithmic tasks across nine programming languages to see if popularity impacts success. We then evaluate the AI's ability to build real-world applications using various frameworks, testing both its proficiency on six mainstream full-stack combinations for common CRUD tasks and its architectural reasoning in specialized scenarios where niche technologies might be superior to popular ones. This dual approach allows us to measure not only core coding ability but also whether Al assistants exhibit a bias towards mainstream technologies, even when more suitable alternatives exist.
3

---

Published as a conference paper at ICLR 2026
3.1
ALGORITHMIC TASKS
3.1.1 LANGUAGE SELECTION
We select nine languages guided (Table 1 by the June 2025 TIOBE Index (TIOBE Software BV 2025:: Python, C++, C. Java, JavaScript, Go, Rust, Erlang, and Racket. These cover a spectrum from top-ranked mainstream languages to niche or emerging languages, enabling examination of popularity effects. This design allows us to investigate whether language popularity correlates with the performance of Al-assisted code generation and submission success.
Table 1: Programming Languages Selected for Comparative Experiments and Their Popularity
Language
TIOBE Rank
GitHuu Repos
Trend (5-yr)
(Jun 2025)
(> 100 stars)
Python
185,000
Strong Growth
C++
85,000
Stable
C
3
42,000
Gradual Decline
Java
4
125,000
Gradual Decline
JavaScript
6
172,000
Stable
45,000
Rapid Growth
Rust
13
38,000
Rapid Growth
Erlang
46
1,200
Declining
Racket
N/R
450
Niche/Stable
3.1.2 TEST CASES FROM LEETCODE
To evaluate performance across these languages, source algorithmic tasks from LeetCode (Leet- Code 2025). an online platform chosen for its extensive collection of problems, robust online judg- ing system, and broad multilingual support. LeetCode accommodates nearly 20 programming lan- guages, including both widely used ones (C++, Java, Python) and less common ones (Rust, Scala, Elixir), making it an ideal environment for our multilingual study.
To assemble our dataset, we develop a script to systematically retrieve problem information via paginated POST requests to LeetCode's 's GraphQL endpoint. This process collects metadata such as problem titles, difficulty ratings, and tags, which are then processed into structured JSON files for analysis. We focus exclusively on publicly available, non-paid problems ensure reproducibility.
Using this approach, we collect a total of 3,011 problems, comprising 765 easy, 1,526 medium, and 720 hard problems. This dataset forms the foundation for our large-scale benchmarking. To manage solution submission and validation, we employ a distributed submissiin system with 15 accounts, implementing exponential backoff and rate-limiting mechanisms ensure scalable and reliable data collection.
We position these algorithmic tasks the 'canary the coal mine' for linguistic competency. While acknowledging that LeetCode does not capture the full software ecosystem, the high compile error rates observed in niche languages expose a fundamental deficiency. an L.LM cannot generate syntactically correct code for basic logic, this failure inherently precludes its effective application in broader and more complex engineering contexts.
3.2 FRAMEWORK SELECTION
Our evaluation employs two-tiered benchmark designed to systematically assess LLM capabili- ties across different ecosystem contexts. The first tier, General-PPrpose CRUD Applications, tests core code generation proficiency using six mainstream full-stack combinations selected by industry adoption metrics (GitHub stars, Stack Overflow activity, job postings), establishing performance baseline in common development scenarios. The second tier, Tasks with Divergent Technology Pathways, examines model reasoning beyond popularity biases by presenting architectural trade-offs (e.g., performance more suitable niche frameworks versus mainstream options. This structure enables a comprehensive assessment of both routine coding ability and adaptive architectural discernment.
development speed), evaluating whether LLMs identify and implement
4

---

Published as a conference paper at ICLR 2026
Table 2: Full-stack Combinations Selected for Comparative Experiments and Their Popularity
Stack
Components
GitHub Stars
Stack Overflow Tags
Job Description
Frequency
Java Enterprise
Vue + Spring Boot Hibernate
Vue: 200k+
Vae: 100k+
High
Spring Boot: 78.4k Hibernate: 460
Spring Boot: 100k-
Hibernate: 100k+
Modern JS
React + Express.s+ js Prisma
React: 223k+
React: 200k+
High
Express. js: 104k+
Expressij: 100k+
Prisma: 43,8k
Prisma: Less Common
Python Full-stack
Django (REST) Django ORM
Django: 85k+ DRF: 29,5k+
Django: 100k+ DRF: 50k+
High
Lightweight Go
Preact + Gin + GORM
Preact: 38k
Preact: 10k+
Medium
Gin: 423
Gin: Ik+
GORM: 38.8k
GORM: 10k
Modern Python
FastAPI + SQLAlchemy
Svelle:
Svelte: 20k-
Mediumn High
FastAPI: 89.4k
FastAPI: 20ke
SQLAlchemy: 10.9k
SQLAlehemy: 50k+
Rust Emerging
SolidJS + Actix Weh SeaORM
SolidIS: 34.2k
Salid1S: 5k+
Low
Actix Web: 23.3k
Actix Web: 5k+
ScaORM: 8.7k
ScaORM: Less Common
To systematically evaluate LLM-assisted software development, we construct a benchmark consist- ing of five categories of tasks jointly designed by domain experts and industry practitioners. These tasks cover both generic development scenarios and cases with clear technology route divergences, allowing for comprchensive evaluation of Al-assisted coding performance.
Our benchmark begins with a foundational set of (1) Generic Tasks, which includes 17 rep- resentative application scenarios frequently encountered in practice, such movie ticket book- ing and library management systems. To ensure comparability across diverse ecosystems, each task is implemented across six mainstream full-stack frameworks, ranging from popular combi- nations like Vue with Spring Boot to emerging stacks like SolidIS with Actix, as detailed in Ta- ble 2 Building upon this baseline, the evaluation progresses more specialized domains. For (2) High-Concurrency Systems, we assess tasks like real-time chat platforms, contrasting the mainstream Node. Sodeejs/Sockkt.I0 approach with the performance-oriennted solutions offered by Go/- Gin and Rust/Actix. The framework then addresses (3) Data-Intensiv Applications, using exam- ples like log analytics to compare the dominant Python/Pandas ecosystem against enterprise- focusee Scala/Spark and the niche scientific computing paradigm of Julia. Subsequently, to gauge perfor- mance in lower-level development, the fourth category focuses on (4) Systems Infrastructure, task- ing the models with creating lightweight API gateways and distributed key-value stores using Go, Elixir/Phoenix, and Rust/Axum to cover popular, fault-tolerant, and emerging systems languages, respectively. Finally, the benchmark explores (5) Alternative Programming Paradigms by quiring declarative or functional solutions for services like chatbots, thereby comparing mainstream imperative languages with the distinct approaches of Haskell, Clojure, or F#.
The selected stacks span a wide spectrum: mainstream (Python, JavaScript, Java), emerging (Go, Rust, Kotlin), and niche (Elixir, Haskell, Clojure, Julia). This enables amalysis not only of func- tional correctness but also of how LLMs handle underrepresented yet domain-relevant stacks. These frameworks well-regarded in specific communities (e.g.. concurrency, functional programming. scientific computing) but have limited adoption and significantly fewer resources in open-source datasets. This contrast allows us measure not only whether the generated projects are executable but also whether LLMs disproportionately favor mainstream stacks, even when alternative stacks may be more suitable for the given task scenario.
3.3 EXPERIMENTAL INFRASTRUCTURE
For both types of tasks, the same core methodology applied. The variation in implementation arises merely from modifying the technology stack paradigm specified in the prompt, while the functional requirements remain consistent. For the algorithmic tasks, the proprictary LLM APIs used are summarized in Appendix A.4. For the framework selection tasks, all work is performed
5

---

Published as a conference paper at ICLR 2026
using three AI programming tools directly: Cursor Pro (using Claude-4-Sonnet), CodeBuddy (using Claude-4-Sonnan and Visual Studio Code with GitHub Copilot (using GPT-5).
4 PROGRAMMING LANGUAGE ANALYSIS
4.1 AI CODING
For each of the 3,011 problems crawled from LeetCode, we apply a standardized procedure wherein the problem statement and constraints are formatted into consistent prompt template (for each of the nine selected programming languages). In total, this process results in over 135,495 individual code generation requests (3,011 problems 9 languages x 5 models), by calling the APIs of these five models: GPT-4o-mini (Hurst et al.] 2024), DeepSeek- V3 (Liu et al. 2024). Gemini-2.0-Flaan (Google 2025). Gemini-2.5-FFl Flash (Comanici al. 2025). Qwen3-Turbe (Yang et al. 2025). This
prompt is then submitted to each commercial closed-source LLM 's API to generate solutions.
Although request that the AI generate pure code, its responses occasionally contained natural language text or other non-executable content. We specifically design the process to extract pure, executable code from mixed-text responses. This systematic approach ensures that the final output consists merely of functional code that can be directly submitted to LeetCode without any addi- tional modifications, addressing the common challenge of irrelevant natural language explanations and cross cross- language code snippets in Al Al-generated content. By implementing multi-stage cleaning process, the tool first identifies and extracts code blocks from potential Markdown formatting, then applies language-specific regular expression patterns to remove all forms of comments and non- code elements. The technical implementation employs targeted regular expression patterns tailored to each programming language's syntax characteristics, including '/.." and '/# 2a for C-style lan- guages, 'H \n?' for Python, "%.\n?' for Erlang, and "".,*nn²' for Racket. This language-awas aware approach effectively removes both single-line and multi-line comments while preserving code func- tionality. The refinement process additionally incorporates whitespace normalization and explana- tory text filtration, resulting in clean, production- ready code that maintains the algorithmic integrity of the original Al-generated solution while eliminating all non-essential elements that would prevent immediate platform execution.
4.2 SOLUTION JUDGING
Each Al-generated solution is submitted without modifications to LeetCode's online judging system, with results systematically recorded for subsequent analysis. The platform categorizes submission outcomes into six distinct status types: Accepted, Compile Error, Wrong Answer, Runtime Error, Time Limit Exceeded and Memory Limit Exceeded. The primary evaluation metric is the Pass@1 accuracy, defined as the fraction of solutions accepted on their first submission attempt.
To support this large-scale evaluation while respecting LeetCode's 's operational policies, we imple- ment distributed submission system utilizing multiple accounts with proper authentication mecha- nisms, including CSRF tokens and session cookies. The system incorporates an exponential backoff strategy with an initial 2-second delay and a maximum of 32 seconds for retries to gracefully man- age HTTP 429 and other transient errors. Additionally, request throttling is enforced at a rate of 10 submissions per minute per account to prevent detection, avoid service disruption, and ensure ethical use of LeetCode's 's platform resources.
4.3 RESULTS
Our large-scale evaluation reveals pronounced performance gap between popular and less popular programming languages, a disparity that is substantial and consistent across all five state-of-thh-arr models tested. As shown in Table 3 mainstream languages including Python, JavaScript, Java, C and C++ achieve Pass rates exceeding 60% in top-performing models. In stark contrast, less pop- ular languages such as Erlang and Racket struggle dramatically, with success rates often below 25% and sometimes approaching zero. For instance, the best performefn model (DeepSeek- V3) achieves 79.81% Pass 1 for Python but only 24.31% for Erlang and 20.82% for Racket. This pattern con- firms that language popularity is a stronger predictor of Al coding success than model capability alone. This phenomenon, pronounced Matthew effect in Al-assisted programming. becomes even

---

Published as a conference paper at ICLR 2026
Table Experimental Results across five LLMs and nine programming languages, Pass denotes first-attempt success rate; enror categories are reported as raw counts.
Model
Lang
Pissef1
Accepted
Wrong Ans.
Compile Err.
Runtime Err.
Other Err.
Easy
Medium
Hard
2.5 Flash
Pytion
67.92%
2045
217
726
23
1104
331
C++
68 65%
2067
164
744
13
23
617
1103
347
C
58.59%
1761
170
1007
52
18
572
925
267
Java
68.6556
2067
157
739
39
9
612
1106
349
IavaSeript
64 50%
1942
275
0
781
13
592
1053
297
Go
50. 10.1
1512
105
1377
7
10
544
216
51.81%
Erlang
Rocket
17.10%
515
2184
200
18
235
Pythom
62.94%
1895
268
787
61
594
1025
275
Co
64
249
718
34
75
1048
278
Java
1963
42
1077
Gemini
JavaSeript
Gio
55 90FF
64.40%
1683
1939
357
237
1088
618
17
06
97
527
607
1057
907
275
249
Rust
50.384
1517
267
1144
44
71
501
793
223
Erlang
2918
93
0
Racket
11.05%
333
281
1995
350
52
180
139
14
Python
41 9896
1265
444
1265
38
387
714
162
GPT 4o mini
C++
C
38.4346
41.8%%
1157
1255
460
416
1234
1169
191
72
34
34
474
428
683
518
144
165
Java
45.50%
1370
452
1055
99
35
476
746
148
InvaSeript
45.575
564
1031
44
495
139
Gu
19.2
24.055%
Edlang 1.16%
77
2701
195
27
Racket
1.99%
60
147
2661
131
12
Python
Qwenk-Tutto
C++
1758
141
Java
32.55%
980
118
1886
19
337
151
JavaSSrrip
38, 633
1163
618
1196
34
450
616
97
Go
33.1.5%
998
403
1566
26
18
368
S33
77
Rust
2. 2,19%
66
22
2014
7
29
33
Erlang
0/
2873
138
Rocket
3.25%
201
176
Python
79.81%
2403
0
162
418
28
683
1294
426
DeepSeskoa
C++
78.8155
2373
450
133
28
27
674
1279
430
C
67. 67.78%
2041
268
497
122
83
668
1071
302
Java
79.38%
2390
412
152
29
28
681
1288
421
JavaSeript
Go
Rust
71.24%
2145
199
20
1161
340
Erlang
24.319
732
1445
373
396
378
321
Racket
20.825
627
1805
287
197
95
326
268
35
more dramatic when stratified by problem difficulty. As illustrated in Figure the performance gap widens substantially as complexity increases. For Easy problems, the difference between popular and niche languages ranges from 45 to 82 percentage points. This gap expands significantly to 58 to 95 points on Hard problems, indicating that the advantage data-rich languages scales non-linearly with reasoning complexity. On these Hard tasks, top models achieve 50 to 63% success with popu- lar languages but only 6% with less popular ones, demonstrating that superior model capability cannot compensate for the disadvantage of language unpopularity.
Gemini-2 2.5shlash
GP CPT--o-mini
DeepSeelo.V3
Vodiem
Puss Raie
Rate:
Pass Mate
Jeva Jnadhurpe
Rost
Erieeg
Jasa
Aait
Erims
Rocker
Priiss
CH
Go
Host
trimg
Rocket
Programming Langunge
Programming Langunge
Programainng Language
Figure 2: Pass rates across difficulty levels for top LLMs nine programming languages.

---

Published as a conference paper at ICLR 2026
Beyond success rates, the distribution of failure types reveals the mechanistic basis of this effect. For popular languages, most unsuccessful submissions are Wrong Answer or Runtime Errors, sug- gesting models generate semantically plausible but incorrect solutions. By contrast, failures in low- resource languages are dominated by Compile Errors, indicating models struggle to produce even syntactically valid code. This points deeper structural limitation: insufficient training exposure hinders the ability of models to internalize basic coding idioms. To ensure these differences are not due random variation, Table 4 the differences were statistically significant across all models (p < 0.001), confirming that the observed performance gaps reflect systematic biases.
conducted paired I-tests comparing Pass @ rates. As summarized in
Table 4: Statistical significance of Pass@1 differences between popular languages (Python, C++, C, Java, JavaScript) and less popular languages (Go, Rust, Erlang, Racket). All tests used paired t-tests across the 3 problems.
Model
Mean Difference (%)
p-value
DeepSeek-VV V3
+44.8
< 0.001
Gemini-2. 2.5-Filh
+42.3
0.001
Gemini-2.0-Fiah GPT-4o-mini Qurnb-Turbo
+40.5
0.001
+33.1
< 0.001
+28.9
< 0.001
The observed Matthew effect has profound implications for programming language ecosystems. As Al-assisted programming becomes pervasive, the massive performance advantage for popular languages may accelerate their dominance while marginalizing niche languages, regardless of their technical merits. This could ultimately reduce linguistic diversity in software development. Our study uniquely leverages "data contamination" as direct signal of language popularity, defining the overlap between test tasks and training data measure of language's 's representation in the training corpus. This premise dictated our deliberate selection of newly released LeetCode tasks from 2025. This choice minimizes "rote recall" from hirely-circulated problems and aligns the "contamination gap" with contemporary popularity trends, allowing us to establish a clearer causal chain: language popularity dictates training data coverage, which in turn drives AI performance.
FRAMEWORK ANALYSIS
After the evaluation of programming languages, we extend our study to software frameworks, which represent higher-level abstractions shaping developer workflows. Unlike languages, frameworks bundle architectural choices and toolchains, making them a crucial layer where LLM biases may influence ecosystem trajectories. Our analysis therefore examines whether similar Matthew ef- fect appears at the framework level, and to what extent mainstream stacks enjoy disproportionate advantages over niche alternatives.
5.1
VIBE CODING
For each development task, the implementation process across varying technology stacks fol- lowed rigorously controlled VibeCoding protocol using the Cursor(Claude-4SSomee CodeBuddy (Claude-4-Sonnet), and Copilot (GPT-5) in both Agent Mode (for high-level planning and multi-file generation) and Auto Mode (for inline code completion and contextual suggestions). The process commenced with an initial prompt that specified the functional requirements of the task along with the designated technology stack, no other contextual or syntactic guidance was provided. Through- out the implementation, the experimenter abstained from any manual coding, architectural input, or corrective intervention. The interaction was strictly limited to forwarding raw, unedited error mes- sages, whether from dependency installation, compilation, runtime execution, or functional short- comings, back into the chat interface as successive prompts. Each error message initiated a new, automated debugging attempt by the agent, continuing in iterative loop without additional human elaboration. The procedure terminated only when all core functional requirements were satisfacto- rily met and the application operated intended, or when predetermined cap on iterative attempts was reached. This approach ensured that the observed outcomes were attributable solely to the Al's autonomous capacity to reason about and implement solutions within each technological context.
8

---

Published as a conference paper at ICLR 2026
The empirical results provide strong evidence of Matthew Effect in programming framework adop- tion under Al-assisted coding. Specifically, the success rate and efficiency code generation were strongly skewed toward few dominant frameworks. For instance, Vue+Spring, React+Express, and Django consistently solved the majority of the 17 benchmark tasks, often within 1-3 attempts. In contrast, less prevalent frameworks such Svelte+FastAPI and SolidIS+Acti exhibited far higher failure rates; many tasks required more than five attempts or could not be completed at all.
The heatmap analysis (Figure 3 highlights this disparity. Successful completions clustered around the established frameworks, while newer or niche stacks displayed darker regions (representing repeated failures). Importantly, this pattern emerged across all categories of tasks, from lightweight personal applications (c.g., a birthday reminder tool) to more complex management systems (e.g., library management or inventory control). This consistency suggests that the observed bias is not task-specific but structural.
5.2
RESULTS
Al Code Generation Attempts (Lower is Better)
Meeting Reservation
1
Pet information Registration
1
2
1
X
Library Management System-
Personal To-Do List Tool
3
2
1
X
X
5
Personal Accounting Applet
Commodiity ity inventory Management
3
2
2
4
5
X
System-
2
Movie Ticket Booking System.
2
2
X
x
Equipment Borrowing Registration
2
Birthday Reminder Tool
X
1
3
Book Management System-
4
X
X
Express Pickup Code Management
2
2
3
Course Schedule inquiry System
3
2
2
X
Parking Lot Payment Record
2
1
3
2
Volunteer Activity Registration
2
2
5
commeroe System-
5
Fitness Center Session System-
X
x
Vue Sprinnnring
React Expresss
Django
Preact+ Gin
Svalte Svelte+fastAPi FastAPi
Sollys Acdix
Figure performance of 17 program tasks under technical frameworks
Moreover, even in tasks where less popular stacks are technically well-suited (c.g.. high-hhghhhhueee cigh-connurrency systems where Go Rust frameworks should excel), the models still disproportionately favored Python- and JavaScript-based solutions. This demonstrates that the disparity arises not purely from technical merit, but from the underlying training data distribution. Frameworks with richer online presence and broader community adoption provide significantly more exposure during model pre- training, resulting in higher generation quality and stability.
Taken together, these findings illustrate self-reinforcing feedback loop with significant implications for software ecosystem diversity: popular frameworks are easier for LLMs to generate successfully, developers relying on AI assistants are nudged toward these frameworks, and increased adoption further amplifies their online presence, ensuring even more model exposure in future iterations. Such dynamics exemplify the Matthew Effect in software ecosystems, where established technolo- gies "get richer" in terms of visibility, usability, and adoption. While convenient for practitioners using mainstream stacks, this trend risks stifling ecosystem diversity by systematically disadvantag- ing technically promising but less popular frameworks. The findings further reveal that framework maturity and ecosystem support significantly impact Al code generation, with emerging frameworks lagging behind, suggesting that --saeddd assistance could exacerbate adoption gaps between es- tablished and new technologies.

---

Published as a conference paper at ICLR 2026
As shown in Figure + the experiments of divergent-technnlogyy wewptttteeeooooooooonnicoo pattwayy reinforce this conclusion. In scenarios explicitly designed pit mainstream, middle-ground, and niche stacks against each other, for example, API gateways, distributed key-value stores, or chat- bot systems, the number of required human interventions diverged sharply. Mainstream stacks (A) typically converged in 1-2 correc- tion rounds, middle-ground stacks (B) required slightly more effort (2-3 interventions), while niche or emerging stacks (C) often demanded 5-10 rounds of guidance before producing runnable system. These results confirm that even when controlling for task type, ecosystem popularity heavily conditions the reliability of Al-generated code.
Log Statisties
Recommendation
Real-time Counter
Chat Room
Gateway
Distributed KV
Chatbot
Task Management
(Mainstream)
(Niche)
Figure 4: Results of Divergent Technology Path- way Benchmarks
CONCLUSION
This study provides the first large-scale empirical evidence of the Matthew effect in Al program- ming assistants, demonstrating how LLMs systematically amplify existing popularity hierarchies among programming languages and frameworks. Our findings reveal that mainstream technologies consistently achieve higher success rates in code generation, while niche and emerging alternatives face disproportionate failure rates that could potentially stifle innovation. We emphasize that tech- nical decision-making making is multi-dimmensonal: Al compatibility is not a universal veto that overrides established factors like runtime performance. However, our results quantify a substantial 'AI Pro- ductivity Tax' for niche languages. This creates a hidden friction consistent with Matthew-effee dynamics, which may disproportionately influence new projects and long-term ecosystem diversity. Moving forward, we plan to expand our benchmarks into broader domains, investigate collaborative multi-agent development scenarios, and develop methods to counteract ecosystem homogenization through diversity-awarr training and inference strategies.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We ensure reproducibility by releasing the complete benchmark dataset, prompt templates, and eval- uation code in our public repository at bhttps://gittiie
Marthe--attt hew Specifically, the implementation for al-
-Effect- -f-offfll- gorithmic tasks (LeetCode retrieval/submissionn and result logging) is provided under leet code while the framework-taie -takk prompts and related materials are under task prompt/. The bench- mark composition detailed Appendix [A] and the prompt design as well as the stage code- extraction pipeline are described in Appendix B Our experimental infrastructure is documented in Section [33] and we summarize the proprietary L.M APIs in Table For artifacts that exceed repos- itory size limits (e.g.. raw logs or large generated projects), we distribute them via GitHub Releases associated with the same repository.
AI P-I--Prose Progr amming
REFERENCES
Samuel Silvestre Batista, Bruno Branco, Otávio Castro, and Guilherme Avelino. Code on demand: A comparative analysis of the efficiency understandability and self-correction capability of copilot chatgpt and gemini. In Proceedings of the XXIII Brazilian Symposium on Software Quality, pp- 351-361, 2024.
Liu Cao, Sumit Roy, and Hao Yin. Resource allocation in 5g platoon communication: Modeling, analysis and optimization. IEEE Transactions on Vehicular Technology, 72(4):5035-5048, 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03372 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Kyle Daigle and GitHub Staff. Survey: The ai wave continues to grow on software develop- ment teams. GitHlu Blog. Available online: https://githbb bioghhewssinngghh//reeee//ssss w--wave-grows## tey-suurvy-findings, 2024.
Akhilesh Gadde. Democratizing software engineering through generative ai and vibe coding: The evolution of no-code development. Journal of Computer Science and Technology Studies, 7(4): 556-572, 2025.
Google. Gemini 2.0 Flash. https::/al..oooglee
2025. Accessed: August 2025.
Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agent- coder: Multi-agent-bastt based code generation with iterative testing and optimisation. arXiv preprint 1IX1::2312.13010, 2023.
Jensen Huang. The programming language is human: Keynote at computex 2024. Computex 2024 Keynote, NVIDIA Corporation, 2024. httpps:/wwiit hwtp://ww..vii.... wttp://dw....ccmee.s/o/oooooooo/
3/sessione on/ http://ccocopoeeeeeeeeeeeeee
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint S0Vv:22006005/555 2024.
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. Xcodeeval: An execution-barte large scale multilingual multitask bench- mark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp- 6766-6805, 2024.
11

---

Published as a conference paper at ICLR 2026
LeetCode. Leetcode online judge, 2025. URL htlttt://eetttcoeee Accessed: May-June 2025.
Hongzong Li and Jun Wang. Collaborative annealing power k-means++ clustering. Knowledge- Based Systems, 255:109593, 2022.
Hongzong Li and Jun Wang. CAPKM++ 2.0: An upgraded version of the collaborative annealing power k-means++ clustering algorithm. Knowledge-Badd Based Systems, 262:1 10241, 2023.
Hongzong and Jun Wang. From soft clustering to hard clustering: collaborative annealing fuzzy c-means algorithm. IEEE Transactions on Fuzzy Systems, 32(3):1 181-1194, 2024a.
Hongzong Li and Jun Wang. Capacitated clustering via majorization-miniiizztooo and collaborative neurodynamic optimization. IEEE Transactions on Neural Networks and Learning Systems, 35 (5):6679-6692, 2024b.
Hongzong Li and Jun Wang. A collaborative neurodynamic algorithm for quadratic unconstrained binary optimization. IEEE Transactions Emerging Topics in Computational Intelligence, 9(1): 2025a.
Hongzong Li and Jun Wang. Machine-cell and part-family formation via neurodynamics-driven constrained binary matrix factorization. IEEE Transactions on Systems, Man, and Cyhernetics: Systems,
2025b.
Hongzong Li, Jun Wang, Nian Zhang, and Wei Zhang. Binary matrix factorization via collaborative neurodynamic optimization. Neural Networks, 176:106348 2024.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412.194377 2024.
Xiao Liu, Hao Yu. Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms agents. arXiv preprini PrXiv:2308.03668 2023.
Hongxu Ma, Han Zhou, Kai Tian, Xuefeng Zhang, Chunjie Chen, Han Li, Jihong Guan, and Shuigeng Zhou. Gor: unified and extensible generative framework for ordinal regression. In The Fourteenth International Conference on Learning Representattions, 2026. URL https:
// /oopeneeview.o net.
Metabob. The hidden pitfalls of using Iims in software http://wwwwwoooooooo/aaa/oaaa/ar
e elcpmentt---ee haggyyllonnuageemooome---eetett-
wtwp:eeetttceeeiieeeeeeee-le---- ht-think. html 2024. Accessed: 2025-11-26.
Leo Meyerovich and Ariel S Rabkin. Empirical analysis of programming language adoption. In Proceedings of the 2013 ACM SIGPLAN international conference Object oriented program- ming systems languages & applications, pp. 1-18, 2013.
Ran Mo, Dongyu Wang, Wenjing Zhan, Yingjie Jiang, Yepeng Wang, Yuqi Zhao, Zengyang Li, and Yutao Ma. Assessing and analyzing the correctness github copilot's 's code suggestions. ACM Transactions Software Engineering and Methodology, 34(7):1-32, 2025.
Nhan Nguyen and Sarah Nadi. An empirical evaluation of github copilot's code suggestions. Pro- ceedings of the 19th International Conference on Mining Software Repositories, 1-5, 2022.
Erik Nijkamp. Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 1 13474, 2022.
12

---

Published as a conference paper at ICLR 2026
Shakked Noy and Whitney Zhang. Experimental evidence the productivity effects of generative artificial intelligence. Science, 381(6654):187-192.. 2023.
Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.08 06590, 2023.
James Prather, Paul Denny, Juho Leinonen, Brett A Becker, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, et al. The robots are here: Navi- gating the generative ai revolution in computing education. In Proceedings of the 2023 working group reports innovation and technology in computer science education, pp. 108-159. 2023.
Jakub Swacha and Artur Kulpa. Evolution of popularity and multiaspectual comparison of widely used web development frameworks. Electronics, 12(17):3563, 2023.
TIOBE Software BV. Tiobe index for june 2025, 2025. URL https: www www.tiioocoooooo ti obe- indexi Accessed: June 2025.
Lukas Twist, Jie M Zhang, Mark Harman, Don Syme, Joost Noppen, and Detlef Nauck. Llms love python: A study of llms' bias for programming languages and libraries. arXiv preprint
arXiv:2503. /7181, 2025.
Justin D. Weisz, Michael He, Michael Muller, et al. Design principles for generative ai applications. In Proceedings of the CHI Conference Human Factors in Computing Systems (CHI 24). ACM, 2024.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang. Jiale Liu, et al. Autogen: Enabling next-gen I1m applications via multi- agent conversations. In First Conference on Language Modeling, 2024.
Dapeng Yan, Zhipeng Gao, and Zhiming Liu. A closer look at different difficulty levels code gen- eration abilities chatgpt. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1887-1898. IEEE, 2023.
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 1xXiv:2505.09388 2025.
Jian Yang, Jiaxi Yang, Ke Jin, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Binyuan Hui, and Junyang Lin. Evaluating and aligning codellms on human preference. arXiv preprint arXiv:2412.052100 2024.
Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. Assessing the quality of github copilot's code genera- tion. In Proceedings of the 18th international conference on predictive models and data analytics in software engineering, pp. 62-71, 2022.
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.073333 2024.
Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. Unifying the perspectives of nlp and software enginecring: A survey on language models for code. arXiv preprint arXiv:2311.07988 2023.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
LLM USAGE STATEMENT
This study used large language models (LLMs) as core tool within the experimental process to generate and evaluate code samples. Our experiments involved calling the APIs of multiple LLM models (including GPT-4o-mini, DeepSeek- V3, Gemini-2.0-Flaan
and Qwen3- Turbo), making a total of over 135,495 code generation requests. These models were used to gener- ate solutions for thousands of algorithmic programming tasks and hundreds of framework selection tasks, systematically investigating how Al-assisted programming impacts the software ecosystem. We followed strict VibeCoding protocol, where the LLMs acted as autonomous agents to produce runnable code through iterative feedback error messages.
Gemini-2.5-Flas
To be clear, while LLMs were an integral part of the experimental process in this study, they were not used to generate the research ideas, experimental design, data analysis for this paper. These aspects were performed independently by the authors.
ETHICS STATEMENT
This study relies exclusively on data from LeetCode.cn, a platform independently operated by Lingkou Network (Shanghai) Co., Ltd. and governed by its own Terms of Service httpss//eetcode.c//eemsss
The cellectuullpropeert attorney confirmed that retrieving publicly available problem metadata, without bypassing technical protection, while respecting the Robots protocol, and for non-commercial academic research, is compliant with both Chinese law and the LeetCode.en en Terms of Service. We additionally confirmed our access pattern and usage constraints with LeetCode.cn staff.
All accounts used for submitting model-generated solutions were manually created, used solely for distributing evaluation load, and operated strictly within normal rate limits. We did not attempt to obtain platform benefits, circumvent protections, or access any non-public content. We not process, collect, store personal data, user- seer-genettted content, paid materials from LeetCode.cn. We do not redistribute proprietary problem statements or other restricted content. Researchers can reproduce evaluation pipeline using authenticated LeetCode web endpoints (problem retrieval via GraphQL and solution submission via standard session-based requests), subject to platform rate limits and Terms of Service.
A BENCHMARK SPECIFICATIONS
A.1 LEETCODE BENCHMARK COMPOSITION
The LeetCode benchmark used in this study comprises 3.011 programming problems collected from the platform. The dataset contains 765 Easy (25.4%), 1,526 Medium (50.7%), and 720 Hard (23.9%) problems, representing standard ordinal structure (Ma et al., 2026) that allows for fine-grained performance analysis.
Figure presents the distribution of the top 15 algorithmic topics by difficulty level. The most preva- lent tags include Array (1,777 problems), String (737 problems), and Hash Table (638 problems). Notably, Dynamic Programming problems are predominantly Medium (270) and Hard (287) diffi- culty, reflecting the challenging nature of this topic. Conversely, Two Pointers and Math problems show stronger representation in the Fasy and Medium categories.
A.2 DATA AVAILABILITY AND REPRODUCIBILITY
We release the benchmark construction scripts, prompt templates, evaluation pipeline, and analy- The experiment is fully reproducible using the provided codebase, though we note two practical con- straints: (1) the evaluation requires 7-10 weeks due to rate limiting and scale (135,495 code gener- ations), and (2) reproduction incurs significant API costs ($1800-$2,000 USD)).
sis code at: http::/github..oo/Faana.ccc Comm/FaanlGGG/TTeeMMateeee of -r--rrogramming--ssiiii
14

---

Published as a conference paper at ICLR 2026
Distribution of Top Algorithmic Topics Across Difficulty Levels
(16.5%)
00.15
(284555
Teol: 198
Two Pointers
(440%)
(537%)
203
Troe
135
Breath Finst Search
341
94
Tenl:
Bit Macipulation
(3t,0%)
613 (4./%)
(34
80
Teal: 214
Manix
(16215)
(8615))
(17995)
Teal: 248
Bintry Search
(1288))
(51.1%)
(4.2%)
Teal: 212
Depth-F
115.11
(569%)
(25.5)
292
Groody
(3.8%)
(22.55)
Serting
240
10
Math
251
147
Teal: S54
Dynamic Programming
19995
Teab: 575
Hash Table
(57.4%)
(1k:55)
Teal: 63
String
(8639g)
147 (47.85))
(28.5%)
737
Aray
(22.2%)
.E3
86
Percentage Distribution by Difticulty (%)
Easy
Modium
Hard
Figure 5: Distribution of top algorithmic topics across difficulty levels, showing the number and percentage of problems for each difficulty category
A.3 DETAILED DESCRIPTION THE 17 FENERAL-PURPOST CRUD TASKS
The table provides detailed description of the 17 general- purepos- application scenarios used to evaluate framework selection in Section These tasks were designed to cover a wide range of common software development requirements while maintaining comparable level of complexity.
A.4 PROPRIETARY LLM APIS
Table summarizes the proprietary large language models used in algorithmic task experiments. We include details on version identifiers, release dates, and knowledge cut-off points to ensure re- producibility and clarify the temporal alignment between training data and evaluation tasks.
B PROMPT ENGINEERING & CODE EXTRACTION METHODOLOGGY
B. STANDARDIZED PROMPT DESIGN AND IMPLEMENTATION
B.1.1 PRIMARY PROMPT TEMPLATE STRUCTURE
def __.enerate_prompt (self, problem_dade Dict) >> str:
title problem larp_ddaaaggee.. get ('title', (*)
description groblem_data.gee deseripption', "
prompt 2 gaw
Provide (self. language_name) solution for LeetCode probles '[title)'.
15

---

Published as a conference paper at ICLR 2026
Task Name
Core Functional Requirements
E-commerce System
Product CRUD, recommendation logic, order creation/status workflow.
Library
Management
Search books, track loans, calculate overdue fines, category statistics.
System
Personal
To-Do
List
Add, mark complete/delete, and filter todo items by date.
Tool
Meeting Room Reserva-
Display rooms, reserve/check availability in time slots, view bookings.
tion System
Personal
Accounting
Record income/expense (amount, category, note), view monthly summary.
Applet
Commodity Inventory
Add products (name, specs), update stock levels, log inhound/outhound.
Management
Class Roll Call System
Manage student list, record daily attendance (present/absent), count absences. Register pets (name, breed, owner). log vaccination records.
Pet Information Regis-
tration
Movie Ticket Booking
Display movie schedules, select seats, generate and manage orders.
System
Equipment Borrowing
Manage equipment (name, model), record borrow/retum dates user.
Registration
Birthday Reminder Tool
Add contacts and birthdays, list upcoming birthdays for the month. Enter parcel info (tracking #, code, recipient), mark picked up.
Express Pickup Code
Management
Course Schedule
In-
Maintain course info (name, teacher, room), display weekly/daily schedule.
quiry System
Parking Lot Payment
Record vehicle entrylexit timestamps, calculate fee based on duration.
Record
Volunteer Activity Reg-
Publish activities (name, time, location), register users, count participants.
istration
Book Management Sys-
Register books (title, donor, ISBN), record borrowing history.
tem
Fitness Session Con- sumption
Manage user memhership cards, deduct sessions on check-in, check halance.
Table 5: Detailed description of the 17 General-Purpose CRUD applicatio tasks used for framework evaluation.
Table 6: Summary of proprietary LLM APIs used in Algorithmic tasks.
Model
Version / Endpoint
Release
Knowledge Cut-off
-iT--o-mini
111-4o-mini-2024-777
Jul 2024
Oct 2023
DeepSeek-V
PeepSeek-VV-0022
Mar 2025
Mar 2025
Gemini-2.0-Flaan Gemini-2.0-flash-000
Feb 2025
Jun 2024
Gemini-2.5-Flaas Gemini-2.5-flash
Jun 2025
Jan 2025
Qwen3-Turbo
Apr 2025
Apr 2025
IMPORTANT: Do not think through the problem step by step. Just provide the code directly.
Code requirements:
1. Must compile and
LeetCode
2. comments or explanations
3. Only the solution codo
Probiem Description:
(description)
Code:
return prompt
B.1.2 SYSTEM-LEVER INSTRUCTION CONFIGURATION
16

---

Published as a conference paper at ICLR 2026
"systeminstructionn: "parts":
("text": "You are a code generation assistant. Provide only code without any explanations or thinking process. Do not think through problems step by step.")
]
B.2 MULTI-STAGE CODE EXTRACTION PIPELINE
B.2.1 CODE BLOCK BOUNDARY IDENTIFICATION
Code block start pattern detection start_patterns Mwen", I" <""""n", I"r., +?nn", mr 2
Standard code block markers (""'us In)
Generic code blocks (*))n)
$ Python muitiline string markers
Python multiline string markers
Code block end pattern detection end patterns =
In 1 18*$",
Standard end markers
End markers with newlines
r" e"Aner \5*$",
Python end markers $ Python end markers
r' \**""
B.2.2 LANGUAGE-SPECING SYNTAX CLEANING
Python: Remove # comments
if self. self.languagy in ["python3"]:
sub "I.",,' ", solution, flags-re.KULTILE C-family languages: Remove // and / comments
solution
elif selg.language in ["cpp", "java", "javascript", "go", "rust"]: solution, flags=re. solution, Dlag3=re.DDTALL)
solution = re.sub (r' ///= solution = re. re.sub (r' //*.*/>*/' Erlang: Remove $ comments "erlang":
elif sanguae
solution = re. sub (r' "8,+\ " solution, bong=-reXXULTIIIN Racket: Remove comments elif felf.language -- "racket":
solution re. arb ;.tin??, " solution, flags-re.
B.2.3 DEBUGGING ARTIFACT REMOVAL
Remove debugging statements and test code
solution = re. (r' console). (leg warn warn/error/infor (.+2X) ; " solution)
solution = re.sub (r' function'stter \wels=) \s* [\s\8] (\)))) solution)
solution = re. sub (r' document)...ee ", solution) # Remove DOM
operations
B.2.4 CODE QUALITY OPTIMIZATION
Remove excessive blank lines and explanatory text
solution = re.sub (r' An's+An' Ann''' solution) # Compress blank lines
solution re.split (r' An (2:This_ccoee explanation Explanattonn note: |Note:), solution, flags-re, IGNORECASE) [0] explanations
Remove trailing
17

---

Published as a conference paper at ICLR 2026
B.3 ROBUST ERROR HANDLING FRAMEWORK
B.3.1 EXPONENTIAL BACKOFF RETRY STRATEGY
max_retrics =
base_delat
max_delay 30
Exponential backoff + randon jitter delay = min (base delay (2 ** retry_count), max delay)
jitter random uniform 1)
sleep_time delay jitter
B.3.2 ERROR CLASSIFICATION AND HANDLING
if sesponse.status..
200:
Success case processing return parts parts[0]] ['textt ,, None
elif
-- 429;
Rate limiting retry with backoff error mag
rressrrrreeoosttattteee code)"
elif response.status. code 400 and response.statiene code 500:
Client error do not retry error_msg = (espstttscrrrr...sspssssssss stt.:essssssttuuscodddd.
else:
Server error zetry with backoff  "rsareetccoorr.rrrppoooootttttst
This comprehensive methodology framework ensured consistent, high-quality code generation across all nine programming languages while maintaining robustness through systematic error han- dling and validation procedures. The multi-stage extraction pipeline guaranteed that generated so- lutions met LeetCode's strict requirements for exccutable, comment-liee code submissions.
B.4 EXAMPLE INITIAL PROMPTS FOR FULL-STACK TASKS
Example initial prompt for the Movie Booking System task using the Modern JS stack:
Build a complete movie ticket becking web application using React for the frontend, Express.js for the backend, and Prisma with SQLite for the database.
The application should allow users to browse movies, view showtimes, select
seats, and complete a booking. Provide the complete code.
B.5 FRAMEWORK TASK PROMPT STRUCTURE WITH EXAMPLE
This section details the prompt structure used for the 17 general-purpose full-stack development tasks evaluated in this study. All tasks followed the same prompt pattern: detailed specification of functional requirements and database schema, followed by instructions for the specific technology stack to be used.
To illustrate this structure, we provide the complete prompt for the Meeting Room Booking System representative example. The prompts for the other 16 tasks followed an identical format, with their respective requirement specifications substituted accordingly.
B.5.1 EXAMPLE: MEETING ROOM BOOKING SYSTEM
Database schema:
users: id, username, email, full_nane department, created meet i ro_mm id, name, location, capacity, amenities, is_active, created_at.
18

---

Published as a conference paper at ICLR 2026
bookings: id, room_dd id, id, title, description, start_time, end_time, status, created_at, Lp
Requirements:
1. User Management:
JWT-based authentication
User profile management
Department -baed organizaticn
2. Meeting Room Management: CRUI operations for meeting recms Availability checking
Filtering by various criteria
3. Booking Management:
Create, view, update, delete bookings Time conflict detection status management
B.5.2 TECHNOLOGY STACK VARIATIONS
For each of the six technology stacks evaluated, only the TECHNOLOGY STACK portion of the prompt was modified while keeping the requirements identical. The specific stack instructions were:
Vue Spring Boot Hibernate (Java Enterprise):
Create complete meeting room bock system using Vue 3 for frontend and Spring Boot with Hibernate for backend. Use PostgreS0L database with the following schema:
Technical Specifications:
Frontend: Vue 3 with Compositicn API, Vue Router for navigation, Pinia for state management
Backend: Spring Boot with Spring Security for JWT authentication, Hibernate for CRM
Database: PostgreSOL with connection string:
postgresql:/ belpostgresemetinnnaaa...
API: RESTiul design with proper HITP status codes
Generate complete, runnable code including:
Spring Boot application with controllers, services, and repositories
Vue 3 components for all features
Proper error handling and validation
Database configuration and entity classes Installation and setup instructions
React + Express.js Prisma (Modern JS):
Develop a meeting room booking system using React 18 for frontend and Express. js with Prisma for backend. Use FostgreSOL databaso with the follewing schema:
Technical $pecifications:
Frontend: React with functicnal components and hecks, React Router for navigation
Backena: Express.js with JWT authentication, Prisma as CRM
Database: PostgreSQL with connection string:
stttgreesl::/posttrrree :meet i
API: RESTful endpoints with proper error handling
Generate complete, runnable code including:
Express. js server with routes, middleware, and controllers
React components with modern hocks
Prisma schema and migrations
Authentication system
Setup and deployment instructicns
Django REST Framework Django ORM (Python Full-stack);
19

---

Published as a conference paper at ICLR 2026
Create meeting booking system using Django REST Framework for
backend and modern JavaScript franework for frontend. Use
FostgreSQL database with the following schema:
Technical Specifications:
Backend: Django with Django REST Framework, Django ORK
Frontend: Use modern JavaScript framework (specify which one)
Database: PestgreSQL with connection string:
postgresql:/ /ppestgg// wetaaaaseeeeeeeeeeeeeeeeeeoeeeee
booking
RESTiul design with token authentication Generate complete, runnable code including:
API:
Djjngo models, serializers, views, and URis Frontend components and API integration Authentication system
Database migrations
Setup and running instructions
Preact + Gin + GORM (Lightweight Go):
Build lightweight meeting becking system using Preact for
frentend and Gin with GORM backend. Use PostgreSOL database with
the following schema:
Technical Specifications
Frontend: Preact with lightweight state management
Backena: Gin framework with JWT authentication, GORM ORM
Database: PostgreSQL with connection string:
ieettiggasst Sa booking API: RESTful design with minimal everhead
Generate complete, runnable code including: server with routes, middleware, and handlers
Freact components with minimal dependencies
GORM models and database operations AAtteetication system
Build and run instructions
Svelte + FastAPI + SQLAlchemy (Modern Python):
Develop modern meeting becking system using Svelte for frontend and FastAP1 with SQI.Ai chemy for backend. PostgresQI. database with the following schema:
Technical Specifications:
Frontend: Svelte with SvelteKit for routing
Backend: FastAPI with SQLAlchemy as ORM, Pydantic for validation
Database: PostgreSOL with connection string:
API: RESTful design with ОpenAPI documentation
Generate complete, runnable code including:
FastAPI application with routes, models, and schemas Svelte components with reactive programming
(SLlAcchemy models and database operations
Authentication system with JWT
Setup and running instructions
SolidJS Actix Web + SeaORM (Rust Emerging):
Create meeting
booking system using the emerging Rust stack: SolidJS frontend and Actix with Sea0RM for backend. Use PostgreSQL database with the following schema:
Technical Specifications
Frontend: SolidJS with fine-grained reactivity
Backend: Actix Web with JWT authentication, Sea0RM CRK
Database: PostgreSQL with connection string:
postgresql://posteee
htweeeaaaaaaaoooooooooo222222222 bookin
API: RESTiul design with focus on performance
Generate complete, runnable code including:
Actix Web server with routes, handlers, and mi ddleeware
20

---

Published as a conference paper at ICLR 2026
SolicJS components with reactive patterns
SeaORM entities database operations
Authentication system with JWT
Build and run instructions for both frontend and backend
B.6 PROMPTS FOR EXPERIMENTS WITH DIVERGENT TECHNOLOGY ROUTES
The following prompts were used instruct the Al coding tool. Eight experiments were de- signed to emphasize typical technical route divergences, covering scenarios such as API gateways, chat servers, data pipelines, task queues, GraphQL services, event streaming, edge inference, and blockchain explorers. Each task specifies three alternative stacks (A, B, C) representing distinct trade-offs in performance, ecosystem maturity, and adoption trends.
Listing I: Experiment 1: High-Concureeng API Gateway
Task: Build a high-coneurrency API gateway that forwards requests to backend services and supports basic rate limiting.
Technology stack (must use exactly this):
[Option A] Rust + Axum Tokio
10ption BJ Go Gin
(Option CI Python FastAFI + Uvicern
Requirements:
Accept НTTP requests on /api. Forward requests to mock backend service. Implement simple rate limiting per client IP.
Listing 2: Experiment 2: Real-Time Chat Server
Task:
Build simple chat server where clients connect and send messages to each other.
Technology stack (must use exactly this) [Option A] Elixir Phoenix Channels [Option 8] Node.js Socket.
[Option Cl Gorilla WebSocket
Requirements:
Start server.
Support multiple clients connecting. others.
Broadcast messages from one client
Listing 3: Experiment 3: Data Analytics Pipeline
Task: Build data analytics pipeline that ingests CSV data, processes aggregates, and exposes results through an API.
Technology stack (must use exactly this): 10ption Al Python Pandas FastAPI (0ption BJ Java + Spring Boot Apache Spark
[Option C] Julia + Genie.jl
Requirements:
Load CSV data (columns: us, event_type, timestamp). Compute aggregate counts per event_type. Expose results at /stats endpoint.
Listing 4: Experiment 4: Scalable Task Queue System
Task: Implement a background task queue system that accepts jobs vía an API and processes then asynchronously.
Technology stack (must use exactly this):
21

---

Published as a conference paper at ICLR 2026
[Option Python FastAPI + Celery Redis
[Option B] Asynq [Option CJ Rust + Tokio + Redis-rs
Requirements:
Expose /submit endpoint enqueue jobs.
Workers pull jobs and simulate processing with sleep. Expose /status to query job states.
Listing 5: Experiment 5: GraphQL API Service
Task: Implement a GraphQl API for blogging platform supporting posts and comments.
Technology stack (must use exactly this): [Option AJ Node.js Apollo Server [Option B] Python Strawberry GraphCL
[Option C] Rust + asyno-graphql
Requircments:
Define schema: Post (id, title, content), Comment (id, postId, text). Support queries: fetcc posts with comments.
Support mutation; add post, add comment.
Listing 6:
Experiment 6: Event Streaming Platform
Task: Build simple event streaming system that publishes and consunes messages.
Technology stack (must use exactly this): [Option A] Java + Spring Boot Kafka
[Option B] Go NATS
[Option C] Python FastAPI + RabbitMo (sio-pika)
Requirements:
Fublisher service produces events with topic name
Consumer service subscribes to topic and logs events. Demonstrate end-to-end event delivery.
Listing 7: Experiment 7: Edge Computing Microservice
Task: Build lightweight edge micrcservice that performs real-time image classification.
Technology stack (must use exactly this): [Option A] Python FastAPI + PyTorch Mobile [Option B] Tensorflow Lite C API
[Option C] Rust + tract (ONNX inference)
Requirements:
Accept image uploads via POST.
Run model inéerence and return predicted label. Keep runtime lightweight to simulate edge deployment.
Listing 8: Experiment 8: Blockchain Transaction Explorer
Task: Build a blockchain transaction explorer for a toy chain.
Technology stack (must use exactly this): [Option A] JavaScript React Express.js MongoD3
[Option B] Python Django PostgreSCL
(Option CJ Rust + Actix Web + SQLite
Requi rementa:
Store mock blockchain transacti (tx_id, from, to, arountt)
22

---

Published as a conference paper at ICLR 2026
Provide API query transactions by address.
Provice web interface display transaction histery.
C EXPERIMENTAL CONFIGURATION & HYPERPARAMETERS
C.I API CALL PARAMETERS
All API calls for code generation were made with the following parameters: temperature=0.. used higher
max0utputtToknn==6555 top top_p=0.99 95. Framework
tasks
temperature=0. to encourage exploration.
C.2 DETAILED CONHGURATION OF THE DISTRIBUTED SUBMISSION SYSTEM
The system used 15 LeetCode accounts. Exponential backoff was configured with: initial.delay=2s max.delay=32s, retry.attempts=5. Throttling was set requests per minute per account.
C.3 LLM TOOL VERSIONS AND SETTINGS FOR FRAMEWORK TASK EVALUATION
Cursor Pro (v0.41.2), CodeBuddy (v1.5.0), VS Code Copilot extension (v1.20.0). All tools were configured use their respective default settings for agentic interactions.
D SUPPLEMENTARY ANALYSES
D.1 ADDITIONAL MODEL RESULTS FOR FIGURE
Due to space constraints in the main body, the success rate (Pass@ @ 1) curves for two models (Gemini- 2.0-Flash and Qwen3-Turbo) across all difficulty levels and nine programming languages are pre- sented here in Figure 6. The trend observed in the main text-where performance degrades sig- nificantly for less popular languages (Erlang, Racket) especially on harder problems- consistent across all five evaluated models.
Gemini-2.0-Flaan
Qwen3-Turbo
100
100
Easy
Medium
Hard
Pass Rate (%)
Pass Rate (%)
60
40
440
Python
C++
Jaw
JavaSeript
Go
Rust
Erlang
Racket
Python
C++
Jaya
JnvaSeript
Go
Rust
Erlang
Racket
Programming Language
Programming Language
Figure 6: Pass rates across difficulty levels for Gemini-2.0-Fiaan and Qwen3-Turbe on nine pro- gramming languages. This figure complements Figure 1 in the main text.
23

---

Published as a conference paper at ICLR 2026
D.2 EXPERIMENTAL SCREENSHOTS
This section provides key screenshots of the experimental process for reference and reproducibility. Due the substantial size of the projects generated by the Al assistants during the Vibecoding process with some individual task projects exceeding IGB, it not feasible to include all outputs their entirety. Therefore, we present curated set of visual examples that best illustrate the scope and outcomes of our experiments.
Figure showcases the running user interfaces of six representative full-stack applications, demon- strating functional completeness across diverse tasks and technology stacks. The examples include meeting room booking system (React + Express.js) a gym membership card system (Vue + Spring Boot), volunteer activity registration system (Django REST), course selection system (Svelte FastAPI), movie ticket booking system (Preact Gin), and a pickup code management system (SolidJS + Actix Web).
Complementing the UI examples, Figure g presents architecture diagrams and performance met- rics for solutions implementing divergent technology pathways. These include high-concurrencc systems such as chat system (Elixir + Phoenix PubSub) and real-time counters (Rust + Yjs + Ac- tix Web; Python + Django Channels), alongside niche/extreme route task management applications built with Ruby on Rails, Clojure Ring, and F# + Giraffe. These visuals effectively highlight the trade-offs between mainstream, emerging, and niche technology stacks in specialized scenarios.
D.3 POTENTIAL EXTENSION: LATENT FACTOR ANALYSIS TASK-STACK OUTCOMES
The main text (i.e., Section 3 summarizes framework-leve bias via task x stack outcome heatmap. constructive next step to decompose this binary outcome matrix so that latent factors can be identified and interpreted, e.g., whether failure is driven more by ecosystem maturity (documen- tation, community size) or by toolchain fragility (build systems, runtime errors) rrather than only visualized. Constrained binary matrix factorization approximates the success/failure matrix by low- rank binary factors, yielding interpretable clusters of tasks and stacks (Li et al. 2024 Li & Wang 20256). Clustering procedures discover soft hard clusters of stacks (or tasks) that behave simi- larly under Al assistance (Li & Wang 2022] 2023 2024a b) 2025a). For this paper, such an analysis would be directly useful: it could separate which dimensions of "popularity" (e.g.. training-date prevalence vs. documentation quality) actually drive the Matthew effect, inform where to invest in diversity-awart tooling or benchmarking, and suggest which task stack combinations prioritize when evaluating or mitigating Al programming bias.
E DATA STATEMENT & LICENSES
E. LEETCODE DATA USAGE STATEMENT
The LeetCode problem data used in this study is publicly available on the LeetCode website. Our usage complies with LeetCode's Terms of Service. The collected dataset is intended for academic research purposes.
E.2
LICENSE
The license is provided in our repository: http:///gttttcooo/aaaaaC///o/tth Seeramminn-AAsistaaae
24

---

Published as a conference paper at ICLR 2026
(a) Mceting Room Booking System (React + jxpress.js)
(b) Gym Membership Card System (Vuc Spring Boot)
(c) Volunteer Activity Registration System (Django REST)
(d) Course Selection System (Svelte + FastAPP)
(c) Movic Ticket Booking System (Preact + Gin)
(f) Pickup Code Management System (SolidJS + Actix Web)
Figure 7: Running user interfaces of six representative full-stack applications generated by Al assistants,
demonstrating functional completeness across diverse tasks and technology stacks.
25

---

Published as a conference paper at ICLR 2026
(a) Brigh-Conurrrency Chat System (Elixir + Phoenix PubSub)
Poat Meve Courter
(b) High-Coneurrency Real-time Counter (Rust Yjs + Actix Web)
Confure-Cnn Real-time Counter (Python Django Channels)
(d) Niche/Extreme Route Task Mana on Rails)
(e) Niche/Extreme Route Task Manage RRing)
(1 Niche/Extreme Route Task Management(F# Giraffe)
Figure 8: Architecture diagrams and performance metries of Al-generated solutions divergent technology pathway tasks, highlighting the trade-off's between mainstream, emerging, and niche stacks in specialized see- narios.
26


Reviews:
[{"review_id": "pXW6V5hLom", "reviewer": "Reviewer_EznC", "summary": "- This paper highlights the Matthew Effect in Coding LLMs, where models do better on programming languages that are predominant in use and therefore predominant in the training set\n- The paper studies it in two ways: (1) Leetcode problems and (2) building CRUD applications\n- Creating a dataset of 3011 Leetcode problems across 8 languages, the paper tests 5 models and shows that models perform poorly on less-used languages. Similar observations are made for CRUD applications", "questions": "See above", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Notwithstanding the issues highlighted below, the papers' focus on studying model performance on low-resource languages is important.\n- The paper also does substantial experiments to highlight model performances on varying languages", "weaknesses": "- LeetCode does not have a public API. The paper mentions in Line 174 that the GraphQL endpoint is used to retrieve problem information. Can the authors comment on what this endpoint is, and whether it is the official endpoint?\n- LeetCode additionally prohibits scraping problems, which raises Terms of Service violation concerns. For this reason, I have flagged the paper for an ethical review.\n- To evaluate model-generated solutions, the authors create 15 user accounts on LeetCode and then submit solutions using these accounts with rate-limiting and other mechanisms to avoid detection. This also raises concerns about the misuse of the LeetCode platform\n- Due to the above issues, this dataset cannot be used to benchmark future models that aim to fix this issue. This limits reproducibility.\n- Section 4.1 mentions the list of post-processing steps taken on model-generated output to strip them of comments and other steps. I am not sure why these are required, and it seemed to me that the post-processing steps were excessive. Comments do not alter the program behavior, so why remove them?\n- What is the difference between the three graphs in Figure 2? There is no indication of how they differ\n- How are the vibe coding tasks evaluated? Are there automated test cases to evaluate the various functionalities?\n- Additionally, why propose a new dataset and not use an existing one with automated evaluation, such as the SWE-Bench Multilingual dataset (https://www.swebench.com/multilingual.html) for this purpose?\n- The SWE-Bench Multilingual results do not show the Matthew effect. Resolution rate on C/C++ is the lowest, while Rust is the highest. How would the authors reconcile the findings in this work, and what SWE-Bench Multilingual shows?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JydTGLUsdZ", "reviewer": "Reviewer_j1XZ", "summary": "The paper examines the \"Matthew Effect\" in AI programming assistants. The authors demonstrate that popular programming languages and frameworks receive disproportionately better support from LLMs compared to less popular alternatives. Through large-scale experiments (120,440 code generations across 8 languages and 5 models), the authors demonstrate that language popularity strongly correlates with AI code generation success rates, with performance gaps widening as problem difficulty increases.", "questions": "1. How can you disentangle language difficulty from popularity? Have you considered matched comparisons of \"languages with similar complexity\" but different popularity? (e.g., Python is not only popular but also easy - so it's hard to know whether the model does Python well because of its popularity or complexity)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "To my knowledge, it's the first systematic demonstration of popularity bias in AI coding assistants at both language and framework levels, with clear quantitative characterization.", "weaknesses": "1. The authors tested GPT-4o-mini, DeepSeek-V3, Gemini-2.0-Flash, Gemini-2.5-Flash, and Qwen3-Turbo, but excluded reasoning-specialized models like o4 or DeepSeek-R1 (although Qwen3-Turbo could be reasoning model). Given that most complex coding problems today are solved using these reasoning models, I'm a bit concerned about the generalizability.\n\n2. I wonder about the effects of Chain-of-Thoughts as well. CoT reasoning is utilized in most coding cases - would the presence or absence of chain-of-thought reasoning alter the popularity bias patterns?\n\n3. The paper shows correlational evidence between language popularity and model performance, but the underlying causal mechanism driving this relationship is a bit unclear. For instance, many popular languages tend to be easier, more user-friendly, and simpler to use (e.g., Python). Perhaps it's not language popularity but rather \"language difficulty\" that determines model performance - models might simply perform better on easier languages.\n\n4. The assumed link between training data representation and performance is not empirically verified. This leads to the concern about the causality of the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iwsZCIHxHs", "reviewer": "Reviewer_aCdD", "summary": "This paper evaluates different LLMs on coding tasks taken from LeetCode.\nBot surprisingly the paper reveals that the more popular a programming language or framework, the higher the success rate of LLM-generated code. Besides LeetCode, the authors also evaluate \"vibe coding\" revealing that successful code generation is related to a few popular frameworks.\n\nIn terms of languages, the authors evaluate as follows: Python, C++, Java, JavaScript, Go, Rust, Erlang, and Racket for five models: GPT-4o-mini, DeepSeek-V3, Gemini-2.0-Flash, Gemini-2.5-Flash, Qwen3-Turbo.\n\nFor the framework selection tasks, three AI programming tools are used: Cursor Pro (using Claude-4-Sonnet), CodeBuddy (using Claude-4-Sonnet), and Visual Studio Code with GitHub Copilot (using GPT-5).", "questions": "It is not clear to me what metrics were used for evaluating vibe coding (section 5.2). Was the evaluation done manually?\n\nIs there a reason you do not evaluate C? Or is that subsumed by C++?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "With the widespread use of llms in coding tasks it is important to understand their strengths and limitations.\n\nThe paper contributes new benchmarks that would be useful for evaluating future models particularly on less popular languages and frameworks.", "weaknesses": "The results are not surprising so it is not clear what this paper brings in terms of novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qe8ckNAXrh", "reviewer": "Reviewer_7q5h", "summary": "The authors of this paper proved empirically the existence of training data gaps/bias for full-stack engineering and algorithmic code generation tasks across various programming languages, software frameworks using five commercial LLMs. They have contributed a comprehensive benchmark dataset to track progress/improvement of this bias over time.", "questions": "Supplementary material seems to be missing in the submission, Where can i find the links to the code base and dataset to replicate this experiments?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- Excellent job on highlighting the gaps in training data for full-stack engineering and algorithmic code generation tasks across various programming languages, frameworks in the leading commercial LLMs\n- Good call out on LLMs potential to amplify these training biases in the emerging coding paradigms such as vibe coding.\n- Good presentation of the topic overall.", "weaknesses": "- The paper does not include justification for why the leetcode & CRUD task datasets were chosen as valid proxies for the larger programming ecosystem. Empirical scope of the study based on datasets explanation in section 3 seems narrow for below reasons \n    - LeetCode algorithmic problems scope is narrow as leetcode problems are self-contained exercises that capture coding ability rather than the broader aspects of software ecosystem (or) programming language evolution which is much broader with non-algorithmic aspects like software configuration, security, performance, maintenance for example.\n    - Similarly, the full-stack benchmark comprising only CRUD-style web-application tasks does not fully represent the breadth of the software ecosystem such as data engineering, distributed systems, embedded systems, or system integrations.\n\n- The discussion in the paper does not mention the existing structural biases in the software ecosystems pre-dating the LLMs. The software industry has favored certain languages/frameworks ecosystems for structural reasons (standardizations, secure implementations, maturity, ecosystem support). Without presenting the pre-LLM baselines for “Matthew Effect”, it is difficult to establish/quantify AI-induced amplification from historical continuation of these biases.\n\n- The paper simplifies language and framework selection by treating popularity and by extension AI compatibility as a dominant determinant for language/framework adoption and suggests - (Lines 20-23) “The phenomenon suggests that AI systems may reinforce existing popularity hierarchies, accelerating convergence around dominant tools while hindering diversity and innovation.” However, technical decision-making in the software ecosystem is not driven by popularity alone. It's a multi-dimensional decision carefully evaluated by senior engineers on the projects based on various factors like language/framework features, runtime performance, maintainability, ecosystem support, and business needs. So this claim need to be proven by considering the aforementioned aspects before reaching to conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7NSTLNXCaQ", "reviewer": "Reviewer_xgRU", "summary": "The paper explores how AI has influenced or will influence software development by investigating AI's performance asymmetry across different programming languages and frameworks. To this end, they constructed a large-scale benchmark to evaluate models' coding capabilities across eight languages and five categories of software development tasks. They reveal a Matthew effect: the more popular a programming language or framework is, the higher the success rate of LLM-generated code.", "questions": "Please see the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- They construct a large-scale benchmark covering various programming languages. Considering that existing benchmarks mainly focus on a limited set of languages such as C++ or Python, their benchmark is valuable for assessing models' coding capabilities across diverse languages.\n- Their evaluation is systematic, including various metrics and statistical analyses.\n- They connect their evaluation findings to a broader societal perspective by discussing the Matthew effect in AI programming assistants.", "weaknesses": "I don't think the results themselves are very surprising. It's well known that models trained with more examples in their training data tend to show higher performance on those tasks. Even though the paper describes the situation as a \"striking\" Matthew effect, I personally don’t find it surprising. In fact, as models evolve over time, such an effect might diminish due to their generally higher capabilities. Moreover, even if the Matthew effect persists, one could create specialized models for each programming language through fine-tuning to mitigate the issue.\n\nAnother question is how significantly the Matthew effect will impact software development. In any case, whether or not there is a Matthew effect in AI assistants, software development itself has long been centralized (as shown in Table 1). It is unclear whether the Matthew effect from AI will have a substantial influence on software development, or what the negative consequences of this situation might be. I think the paper needs to discuss the meaning of the Matthew effect, its negative consequences, and whether the effect will persist.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)