Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Authors: Maria Ivanova
Pavel Zadorozhny
Rodion Levichev
Ivan Petrov
Adamenko Pavel
Ivan Lopatin
Alexey Kutalev
Dmitrii Babaev
Venue: ICLR 2026 Poster (2026)
Keywords: Code Benchmark; Code LLMs; Cross Language Evaluation; Contamination; Overfitting
Primary area: datasets and benchmarks
TL;DR: We introduce a contamination-aware benchmark that evaluates code LLMs across 12 programming languages.

Abstract: LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering.

We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python.
Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB’s contamination controls and evaluation protocol.
Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python.

We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB’s primary limitation and exposing critical gaps in current LLM capabilities.

Published as a conference paper at ICLR 2026
MULTI-LCB: EXTENDING LIVECODEBENCH TO MULTIPLE PROGRAMMING LANGUAGES
Maria Ivanova*
Pavel Zadorozhny"
Rodion Levichev
Ivan Petrov'
Pavel Adamenko
Ivan Lopatin
Alexey Kutaley
Dmitrii Babaev
GGiaaCoee
Yandex School of Data Analysis, Applied Institute
httpp:://itthbb.com//
ABSTRACT
LiveCodeBench (LCB) has recently become a widely adopted benchmark for eval- uating large language models (LLMs) code-generation tasks. By curating com- petitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aree aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering.
We introduce Multi-LCB, benchmark for evaluating LLMs across twelve pro- gramming languages, including Python. Multi-LCB transforms Python tasks from the I.BB dataset into equivalent tasks in other languages while preserving I.CB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation com- petence and requiring models to sustain performance well beyond Python.
We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specifff contamination. and substantial disparities in multilingual performance. Our results establish Multi-LCB as rig- orous benchmark for pooogramming-languee code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.
INTRODUCTION
Large language models (LLMs) have recently demonstrated impressive capabilities in code-related tasks (Ridnik al., 2024; Lozhkov et al., 2024; Roziere et al., 2023; Li et al., 2022; Nijkamp al., 2022), powering applications such Al-assisted programming, automated debugging. and code translation. To measure these abilitics, benchmarks such as HumanEval (Chen et al., 2021). MBPP (Austin et al., 2021), and APPS (Hendrycks et al., 2021) have been widely adopted. However, these datasets suffer from well-documented limitations, including contamination from training corpora, narrow task scope, and weak correlation with human judgment. BiveChdeBench (LCB) (Jain al., 2024) addresses these shortcomings by continuously curating competitive competiiiveprogrammming problems, filtering them by release date, and enabling contamination-avvarr, continuously updatable evalua- tion. As result, LCB has quickly become a standard benchmark for evaluating LLMs on code- generation tasks (Google DeepMind. 2025; DeepSeek. 2025).
Despite these strengths, LCB (Jain et al., 2024) evaluates only Python. While convenient, this limitation overlooks a central reality of software engincering: developers routinely work across diverse programming languages, each with its own syntax, semantics, and idiomatic practices. An LLM capable of solving problems exclusively in Python may perform poorly when C++ is required
"Equal contribution. Correspondence Dmitrii Babaev: Amitri.. babaevêgmai com

---

Published as a conference paper at ICLR 2026
for systems programming. Java for enterprise software, JavaScript for web development. Current evaluations therefore leave open a critical question: can LLMs generalize coding competence across they overfitted to Python?
multiple programming languages,
In this work, we introduce Multi-LCB, an extension of LCB (Jain et al., 2024) to twelve program- ming languages while preserving its contamination controls and evaluation protocol. Multi-LCB replicates every LCB task across all supported languages, enabling direct comparison of model per- formance on identical problems in different programming languages and updating automatically as LCB evolves. We evaluate 24 reasoning- and instraction-oriented LLMs on Multi-LCB and uncover key findings:
Python is not always a reliable proxy for individual non-Python languages. Our results re- veal substantial and practically meaningful performance gaps across languages. In several cases, models that are stronger on Python do not retain their advantage in other languages.
2. Python overfitting. Models that perform strongly in Python often degrade sharply in other languages.
3. Language-specifee contamination. Evidence of data leakage varies by programming lan- guage, reflecting uneven distribution in pretraining corpora.
4. Substantial pungriirogranmingg-- lannggaannnnn-agguagge disparities. Models show large performance gaps across languages, with weaker results in statically typed or less prevalent languages.
Our main contributions are:
1. We extend LCB (Jain et al., 2024) to 12 programming languages without task loss, enabling direct comparison of LLM abilities to solve identical problems across different languages.
2. We provide a comprehensive evaluation of 24 instruction- and reasoning-orienter LLMs across these languages, revealing systematic multi programming languages performance gaps and evidence of manguage-specingg contamination.
3. We publicly release all prompts, source code and experimental configurations to facilitate reproducibility and future research.
These results establish Multi-LCB as a rigorous benchmark for multi iooogramming-langeene code evaluation, directly addressing LCB's Python-only limitation and providing foundation for devel- oping more robust, programming language agnostic coding models.
RELATED WORK
Single-language code benchmarks. Early onde-generation benchmarks evaluate functional cor- rectness almost exclusively in Python. HumanEval (Chen et al., 2021) contains 164 hand-written problems, each defined by a natural language prompt, a fixed function signature, and hidden unit tests; tasks are short, single-function programs created specifically for evaluation rather than drawn from programming contests. MBPP (Austin al., 2021) likewise offers small Python exercises aimed at introductory programming and interview practice. Subsequent datasets expanded scale and difficulty: APPS (Hendrycks al., 2021) aggregates competition and interview style problems with hidden test suites, CodeContests (Li et al., 2022) compiles algorithmic contest tasks with official judge input/output data, and CodeXGL.EE (Lu et al., 2021) provides a broad suite of generation, translation, and retrieval tasks. Despite their influence, these resources are static snapshots, lack re- lease date filtering to prevent training set contamination and are therefore largely saturated, remain heavily Python centric, and do not enforce unified STDIN/STDOUT protocol.
benchmarks. Several datasets extend code generation evaluation beyond Python. MBXP (Athiwaratkun et al., 2022) translates functional-fomme Python problems (c.g., HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021)) by rewriting function signatures and regenerating unit tests for each language. Even simple Python assertion like:
Multi-programmingg-annnn
assert binomial.coeff (5, 2) = 10
must be expanded into multi-line Java test code. This translation must be repeated separately for ery language and is sensitive to syntax and runtime differences. Concurrent work MultiPL-E (Cas- sano et 2023) similarly performs translation of HyaanFyal and MBPP (including their unit tests)
2

---

Published as a conference paper at ICLR 2026
Prompes
E.M
Navral langugee desorptin
generation:
yocn: ccpa *xn
STDINSTDOUT fermat
Execution
TCB
Quesion
Problem sample:
Natural language
questioo description
Jest examples
Test
LeetCode Problems Test Convener
pythos
Figure I: Multi-LCB overview. Top: LCB natural-language problem descriptions are wrapped into prompts specifying the target programming language and passed to the LLM for STDIN STDOUT code generation. AtCoder and Codeforces problem tests are passed directly to the execution stage. LeetCode problem
are transformed through dedicated converter produce equivalent STDIN/S STDOUT inputs. The generated code is compiled or executed in the target pro- gramming language and evaluated using Pass@1.
Bottom:
into 19 programming languages. HumanEval- Eval-XL (Peng et al., 2024) similarly expands EvaanEEal additional languages and provides standardized execution harness while preserving the func- tional, unit-test format. Multi-LCC avoids this by keeping only the natural-language description and converting hidden tests into language-agnostic STDIN/STDOUT format, for example: Input:
2
Output:
10
Other projects broaden language coverage in different ways. Ag-LiveCodeBench-x (Boruch- Gruszecki et al., 2025) reuses a subset LiveCodeBench tasks already in STDIN/STDOUT format and adds rarer targets such as Lua, R, Julia, OCaml, and Fortran. xCodeEval (Khan et al., 2023) likewise provides a unified multilingual execution framework and resembles our approach, but draws exclusively from Codeforces problems and is not continuously updated. McEval (Chai al., 2024) and BigCode Bench (Zhuo et al., 2024) once offered broad language coverage, but both are static and evaluate different task sets per language, hindering direct cross language comparison.
Contamination-avaan aware evaluation. LiveCodeBench (LCB) (Jain et al., 2024) introduced release date filtering and continuous collection of Python problems from three major competitive program- ming platforms: LeetCode, AtCoder, and Codeforces (see Appendix D.I for task statistics). By harvesting new tasks and filtering them by post-training release dates, LCB enables live, contam- ination aware evaluation of LLMs and has become de-facto standard for robust single language (Python) code assessment (Comanici et al., 2025; Yang et al., 2025; Liu et al., 2024). related effort, EvoCodeBench (Li et al., 2024). followed similar evolving design but not actively maintained and remained limited to Python. Multi-LCB builds directly on this foundation. reuses the entire LCB (Jain al., 2024) task pool and inherits its contamination controls.
3 BENCHMARK DESIGN
This section describes the approach, used to construct the Multi-LCBB benehmark. Figure illustrates the full pipeline. Please note, that although Multi-LCB is built on LCB, the same approach can be applied to any dataset with a comparable structure.
Data Source. Earlier versions of LCB supported several evaluation scenarios beyond code gener- ation: self-repair, code execution, and test output prediction. But the latest releases (v5-v6) focus exclusively code generation, the most widely benchmarked capability of modern LLMs. In this setting, model receives a natural language problem statement with sample input/output pairs and must synthesize program that passes all hidden test cases.
3

---

Published as a conference paper at ICLR 2026
To construct Multi-LCB, we load the desired version of the LCB code generation dataset from Hug- ging Face, retrieving Python problems and their metadata. We convert every release of LCB code generation dataset without modification, preserving all tasks from three platforms: LeetCode, AtCoder, and Codeforces. Each task includes a natural language description, input/output examples, and contest release date for contamination-awan is applied only to LeetCode's 's functional format tasks to ensure unified STDIN/STDOUT evaluation. Details about platforms and temporal distribution appear in the Appendix D.1.
filtering. Test conversion
Conversion of functional format. LCB supports two native task formats: STDINISTDOUT (as in AtCoder and Codeforces), where a program reads from standard input and writes to standard output, and Functional (as in LeetCode), where a specific function is implemented and invoked by the evaluation system. Directly extending the functional format to a multi-programming language benchmark is challenging. Each I LeetCode task provides Python starter code tightly coupled to its own testing harness. Producing equivalent starter code and call signatures for many target languages would require custom templates for every language, leading to unsustainable and error proor-prone process. To overcome this limitation, we designed an automatic conversion pipeline that rewrites every Functional task into unified STDIN/STDOUT format. This pipeline consists of two compo- nents: (1) prompt adaptation that reformats problem statements and examples for model input, and (2) test conversion that transforms all test cases for automated evaluation.
The pipeline first parses examples from the problem statement and reformats them into STDIN/STD- OUT format for inclusion in model prompts. (see Appendix C. Separately, it converts all test cases (both public and hidden) from the original format to enable unified automated evaluation. This unification allows single evaluation harness handle both the original STDIN/STDOUT prob- lems and the adapted functional tasks across all supported languages, Since the original benchmark behavior could theoretically appear. However,
based on Python, tasks involving tasks on LeetCode, AtCoder, and Codeforces are authored by human experts and are intentionally designed to avoid language-specific ambiguities, as these platforms support many programming languages. Consequently, Multi-I.CB requires no language-specific rewriting, and the tasks remain inherently aunguage-agnostic. Moreover, in our manual inspection of approximately 500 tasks, we did not find any cases in which language-dependent features introduced inconsistencies. Note that tasks unsuitable for strict input/output grading, such as those admitting multiple valid answers or requiring explicit data structure construction, are already excluded in the official LCB dataset that we load, so Multi-LCB inherits this filtering without any additional intervention. The remaining tasks are grouped by 1/O structure: Scalar: inputs and outputs are single, sealar values (e.g. inte- gers, floats, booleans, or simple strings); One-Dimensional involve one-dimensional arrays (lists) as input or output: -wo-Dimensional:: include exactly one two-dimensional array (matrix or jagged array) in the 1/O. As result, all functional tasks, including their examples and hidden tests, are con- sistently converted STDIN/STDOUT format: lists are space-separated, and for 2D arrays the first line specifies the number of rows, followed by row-wise ppace-separated values. This conversion applies to both the examples shown to the models and all test cases used for evaluation.
Code generation. We adopt a zero-shot prompting strategy that follows the original LiveCodeBench protocol. For each task, the benchmark constructs a prompt with three components:
1. a system message instructs the model to act as an expert programmer the target language (e.g.. 'You are an expert Python programmer....
2. a user message provides the complete natural language problem statement with ex- plicit STDINISTDOUT specifications and input/output sample cases already provided in the original problem descriptions;
3. a code-block placeholder indicates where the solution must be written:
*** python
# YOUR CODE HERE
.*"
44#
Answer: (use the provided format with backticks)
The code-block header is set to the target language (e.g., opp, java, python) to ensure correct syntax highlighting and parsing.
4

---

Published as a conference paper at ICLR 2026
Models are required output only the complete program source that reads from the standard input and writes to the standard output. High-level zero-shot template prompts for both native AtCoder and CodeForces tasks and adapted LeetCode problems included in the Appendix C for reference.
Automatic Testing and Evaluation. Correctness is assessed against a hidden suite of official test cases provided by the original contests. A program is marked correct only if it passes all tests without runtime errors or timeouts. For quantitative comparison we report Pass@1 1, the fraction of tasks for which the model's 's first generated solution passes every public and hidden test.
Together, these stages create a fully automated pipeline: a model receives a problem prompt, emits candidate solution, the code is securely compiled and executed, and the output graded against hidden tests -all all without human intervention. This process preserves LCB' s rigorous contamination controls while enabling direct, language-agnostte evaluation of code generation across the diverse set of languages supported by Multi-LCCB. Note, that the same set of tasks is used across evaluations on different programming languages, hence task difference does not hinder the comparison of the multi-language model capabilities.
3.1 LANGUAGE SET AND MOTIVATION
This study evaluates multilingual code generation across major programming languages: C++, C#,, Python, Java, Rust, Go, TypeScript, JavaScript, Ruby, PHP, Kotlin and Scala. The selection balances three criteria: (1) popularity based Github, StackOverFlow, RedMonk and TIOBE rankings, (2) stable infrastructure support through package managers like Conda for reproducible execution, and (3) paradigmatic diversity across compilation strategies, type systems, and memory management models. For detailed programming language rankings across multiple sources, as well as the runtime characteristics information, please Appendix E.
4 EXPERIMENT SETUP
Here we describe the experimental configuration used to evaluate LLMs on the Multi-LCB bench- mark.
Models We evaluate diverse set of 24 publicly available large language models (LLMs) span- ning from 7B to 685B parameters and covering both general-purpon and code-specialized do- mains. The pool includes instruction-tuned and bassonnnn--ugmenttee variants from the Qwen3, DeepSeek, OlympicCoder, OpenReasoning, and OpenCoder families, among others. Representative examples include ppp-osss-1000 DeepSeek-RL RR---2111-22888 and www.neessoonnn-eeettoooo.
(Medi um), ( -hin----ninnnnn-25000, We intentionally selected models to capture a wide variety of training paradigms (pure code pretraining, mixed-domain train- ing, instruction tuning, reasoning-enhanced fine-tuning). Appendix F.1 lists all checkpoints with their estimated training cut-off dates.
Hardware and Environment. All experiments were run on cluster of 16 NVIDIA H100 80 GB GPUs with CUDA 12.3 and Python 3. 1 inside Conda environments. Each programming language is executed inside an isolated sandbox container that bundles its corresponding compiler or interpreter GCC 13 for C++, Rust 1.79, OpenJDK 21, ,NET 8, CPython 3. Node.js js 20). The sandbox enforces strict resource limits: 6 s wall-time per test case, 4 GB memory, and no external network access. This ensures deterministic, secure, and language-agnostic execution.
GCC
Inference Protocol. Following the original LiveCodeBench protocol, we adopt zero-shot prompt- ing strategy. For each problem, we generate model-specific number of tokens (set according to its configuration) using nucleus sampling with temperature 0.2 and top-p 0.95, applying triple- backtick stop sequence to capture the complete code block. Models are served with vLLM (Kwon al., 2023) or SGLang (Zheng et al., 2024) for efficient batched decoding.
Evaluation Metrie We report Pass @1 (%) averaged on 10 runs as the primary metric, which mea- sures the fraction of problems solved correctly by the first generated solution. A solution is marked correct only if it compiles/interpress successfully and passes all hidden official test cases without runtime emrrr or timeouts.
5

---

Published as a conference paper at ICLR 2026
Table 1: Performance results on Multi-LCB for the tasks from February 2025 till May 2025. Scores represent the @ass@1 1 (%) metric averaged on 10 runs. Higher is better, bold is best, italic is the second best. Temperature t=0.2 (* reasoning mode)
Model
Pythos
C+1
Jaya
Bust
Ruby
PHP
Botlis
Scals
Aax
Gje0s01200 (Matiun)
23:
14
244
78.5.1
70.3
18. 215
71.
541 gm0
DecySokk 1R1--00
26.
646
58
63442
63.3
631
Gpt-cx-2118 (Nlddann
635-21
65.2
127
299
6.9
61.
5244
619
61.7
68.
624
43.1
59.
Oveid 306 A38 250-2
54. -20
65.
034
46.1 a.P
5L9
45.-23
36.5 4M
51. 140
421 2m
506 m::
435 22
Gpt-o-1220 12:80
518
35.
...
A228*
Ovesl: 128"
515
35.3-M4
S6D
42.
49.
40.3
4011
400
521
50.0
46.
35.6
45.6
Qwar1-H8 A
13.0
169
459
11.2
Gft- (Law)
6.122%
425-211
412
43.0
417 21
Qwen2 (oom)-22SB-A221 2nit
41.
=222
1955
41.5
42.
57.5
Oveil SB
463
39.7 3M4
36.
25.
388
163
2055
241
38.5
340 130
37.0
339
Owerl Codo 308 Baar
365-121 221
31.1 2:5
25. 112
MALD
200
14T
34.3
347 222
202
31
Ovenl S-Cder 32B fostl
21.5
56.
755
18.5
27.5
250
Seel Coder-BB frst
13420
260:11
23
231
57.0
31.
10
330 11
Openllor-Nmm-1100
644 434
4422
43
11.5 412
163 mm
105
100
28.13
1
15
11. xm
Beezticck- Ri---asii Qwee-328*
119
162
116
10.3
101 14
-M
129
23,2
Sweat.5-Code lasis
232
213--
192
175
133
157 1m
22
27
25
209
OpenratiRoay Nim-1 1-37p
255
11
3142
131
10
(98
16
249331
10
167
115
1022
37214
1829
111
Alg
182
Deguokk
5 EXPERIMENTS AND RESULTS
We evaluate a suite of frontier large language models on Multi-LCB, spanning 12 programming
languages and reporting Pass@] I averaged on 10 runs as the primary metric (Table 1). This section presents a detailed analysis of model performance on latest Dataset v6 (Feb 2025 May 2025) (Section 5.1), compares findings with single-language LiveCodeBench (LCBB results (Section 5.2), and investigates contamination signals (Section 5.3). For additional performance results at various sampling temperatures. @ass@5 and Pass @ metrics and other dataset releases (July 2024- May 2025 and the full I, 1,055-task benchmark) see appendix F.
5.1 EXPERIMENTS RESULTS ON MULTI-LCB
We study performance variations in models released more recently. Particularly, evaluate 24 recent large language models on Multi-LCB, restricting tasks to those released after 2025-02-01 to ensure live, post-cutoff evaluation and minimize any risk of training-data leakage. Model approxi- mate cutoff dates are listed in Appendix F.1 Table 4.
Table summarizes Pass @ 1 averaged
10 runs with temperature 1 = 0.2 perfor- mance across twelve programming languages on Dataset vó (Feb 2025 May 2025), while Figure 2 highlights the results for the 10 best- performing models.
Pythas 8D
Scala
Java
Our results reveal substantial and practically meaningful performance gaps across languages. For example, outper-
30
opt-oss-120B- (Medium)
forms owen3-23553 11222-Th--2055 on Go, Javascript, Typescript, Rust, Ruby and Kotlin, and DeepSeek-R1-0525 outper- forms Owen3- 235B-A22B-Thk-250 on Rust, Ruby and Scala, despite 1reem33-25BB-22B-Th--22
TypeScript
PHP
JavaSeript
Ruby
Kottin
be- ing consistently stronger on Python. This precisely why strong Python ability not always reliable proxy for true cross-lingual code generation competence and evaluation must consider performance in the target languages rather than relying on Python alone.
Oven3 2218 2ww3--258--05--111 AISB- ii 25---337 fleg (4.0%)
Ower 2208-220 -28 48.9%)
Upenrca coe 1rvv 4a
Dpced 506 Tie
35.24
20t 1ite 45.
Figure 2: Top-10 models by Pass@1

---

Published as a conference paper at ICLR 2026
Figure 3 plots per-model @ averaged 10 runs with sampling temperature t = 0.2 scores Python against the cross program- ming language average on Dataset v6 (Feb 2025-May May 2025).
Almost every point lies above the I = w.w.wwa. diagonal, demonstrating toward Python.
consistent bias Models without ex- plicit multi programming languages train- ing, such as -NmenRsn-Nmt-322 OpenCodeRsn-Mnt-- 1--2BB", show the starkest gap, exceeding 60% on Python while remaining below 30% across other languages.
Oepion Mo
O dew11/68h
and
O Twecci
Even the largest caasonng--augmenter mod- els, including DeepSeek-R2 retain measurable positive bias toward Python, though the disparity less pronounced.
Fween-233B-TTE
and
10
Aoerage Fuang laggages
Figure 3: Scatter of Python vs. Average Pass @
These results confirm that strong Python ability not necessarily reliable proxy for true cross- lingual code generation competence.
The most strongest models, Gpt-oss-120F (Medium), and DeepSeek -R1-e022----- establish a strong yet far-from-saatrrated frontier, while the next tier of high-performing models, such as - of reasoning-augmented variants can exceed the 50% mark. Most of the evaluated models remain below 40,, underscoring the benchmark's challenge of achieving robust multi programming lan- guage code generation correctness.
illustrates that only handful
Figure plots @ass@1 distribution across 12 languages on with sampling tempera- ture 0.2 on Dataset vó (Feb 2025 May 2025). Boxes show the interquar- tile range with the horizontal line marking the median and the red diamond indicat- ing the mean. This reveals a clear diffi- culty gradient. Python achieves the high- est mean Pass@1 of 0.482, with Java and C++ close behind at about 0.44. C#. Ruby, РHP, Go, Rust, Kotlin and JavaScript/- TypeScript form middle tier with means near 0.33-0.39, while Scala consistently trail means below 0.29. These gaps per- sist across the pop-performin models, re- flecting structural challenges such as com- pilation complexity, ownership semantics, and smaller ecosystem resources.
Figure 4: Pass@1 I distribution across 12 languages
We observe that Python consistently outperforms other languages on Multi-L.B.. This suggests that current L.MS are substantially more trained on Python code, especially for reasoning-ma mode training. and that cross-language knowledge transfer remains only partial. We suppose that model performance could be improved by increasing training coverage non-Python programming lan- guages.
5.2 COMPARISON WITH LIVECODEBENCI
To verify that our multilingual extensions preserve the fidelity of the original LiveCodeBench (LCB), compare Pass @ scores the Python subset of Multi-L.CB against the official results reported

---

Published as a conference paper at ICLR 2026
for LCB versions v4-v6. Table 2 reports original leaderboard results (ORIG) and our reproduced scores (OUR), with representing the absolute difference.
Table 2:
Comparison of reasoning/code models on Python across benchmark versions (v4-v6).
Original leaderboard values (ORIG, %) are contrasted with our reproduced scores (OUR, %). Dif- ference is computed A OUR ORIG.
Model
Benchmark version
ORIG (%)
OUR (%)
A (%)
Source
(weea3-23BB-44 Thinking-
86/202--500
Hugging Face
DespScek RI 0528
v6 [25002 2505]
68.7
66.3
-2.4
LCB leaderboard
Qwen3-30B 30B 31BB-33 Thinhini--25500
v6 [2502-22555
66.0
64.0
-2.0
Ulugging Fage
OpenRessoning- Nemotron-322 32B
v6 (2502-2505)
65.6
64.4
-1.2
LCR lesderboond
OyenCColsecoominng Nemotton-
(2502-22505
leaderhean
2wees--88-AAB0
(2502-2508
Higging Face
Owes3-235B-A22
v6 (2500-25505
58.7
58.9
3.2
LCB leaderboard
Qwen3-215R- 219--473B-mmmmm---000
v6 (2502-2505)
$1.8
43.8
-50
Hugzing Face
Qwen3-320 328
v5 [2410-2502]
61.3
-144
Owenl Toch
[2110-2502]
report
Owes3-208-A3B**
[2410-2502]
Oweat report
Owen3-8BB 88
v5 (2420-2502)
57.5
49.1
-3.6
Owen! Tech report
Seed Coder SB Instract
v5 12210-22502
24.7
19.8
-1.9
[lugging Face
OgenCadee
Nemotron- 1--322
(2408-25022
Hugging
OlympicCakr--22
(2408-2502 2502
Hugging Fave
OlympicCocen-78
+4-v5 (2408-2502)
35.6
-51
Hugzing Faes
QwenZ S-ddde-e-------nnnee
v4-15 [3408-2502]
28.3
27.6
-0.7
Hagging Face
Overall, reproduction is strong: differences are typically within a few percentage points, with a mean absolute deviation of only about 3%. For example, Twen3-2338 A22B- 1.22B-Thinning 2507 achieves 74.0% Pass @ in our evaluation versus 74.1% on the original v6 leaderboard (A = -0.1),
while DeepSeek-RO -1-eekk-R1-055 R1-8528 records 66.3% compared 78.7% (A 2244). Even for mod- els with larger gaps. such as (A -8.6), the rank ordering across models remains consistent.
111B-A22B-Inn-250 (A = -8.0) or Qwen3-8B+
These close alignments confirm that Multi-LCB's multilingual transformations introduce no artifi- cial difficulty for Python tasks. Performance differences instead reflect natural leaderboard variance and underscore that the multilingual benchmark faithfully reproduces the single-language LCB set- ting, ensuring that any additional challenges arise from genuine cross-language generalization rather than implementation artifacts.
5.3 CONTAMINATION ON MULTI-LCB
core design goal of Multi-LCB is contamination-aware evaluation via release-date filtering. time-wise analysis reveals clear evidence of residual contamination on older (pre- cutoff) problems. Figure 5 shows monthly Pass@1 trends for the top-10 models aver- aged across all programming languages scores are systematically higher on earlier months and exhibit step-like drops when the evaluation window crosses model cutoffs, followed by sustained lower per- formance on post-cutoff problems. Our main comparisons in Section restrict evaluation to tasks released on or after 2025-02-01, ensuring live, post-cutoff measurement. Under this setting, perfor- mance drops to a level that better reflects true generalization, whereas inflated scores on older windows are explained by pretraining exposure rather than genuine zero-contaminatioon generalization.
Nevertheless,
Figure 5: Monthly @ass I trends averaged across all programming languages for top- 10 models.
8

---

Published as a conference paper at ICLR 2026
6 LIMITATIONS AND THREATS TO VALIDITY
Language Coverage and Selection. Multi-LCB covers 12 programming languages but does not include important languages such Swift, Haskell, R, and others. 'he language selection is based on popularity rankings in 2025, which may not reflect specialized domains or emerging languages. Additionally, some languages have various dialects and versions that are not accounted for in our evaluation framework.
Task Complexity and Domain. While the selected programming languages span different domains (systems programming, web development, data science). the tasks themselves remain rooted in com- petitive programming. Although algorithmic problem-solving has indirect relevance to industrial coding capabilities, Multi-I.CB does not directly assess real-world software engineering scenarios such as API integration, debugging legacy code, or collaborative development workflows.
Evaluation Protocol Constraints. The strict STDIN/STDOUT format may introduce performance degradation not only due to algorithmic reasoning limitations but also due to syntax unfamiliarity, difficulty parsing input formats, or failure to follow output specifications. Models may fail tasks due to format compliance issues rather than core wooblem-solving deficits, potentially confounding our assessment of true multilingual coding competence.
Model Selection Bias. Our evaluation focuses exclusively on publicly available models, excluding proprietary systems that may represent the current state-of-the-art. This limitation means results reflect only subset of available models and may not accurately represent the real-world leaderboard of multilingual code generation capabilities.
Construct Validity. The automatic conversion from functional format to STDIN/STDOUT may alter task complexity differently across programming languages. Some languages may be more naturally suited for certain problem types, potentially creating unequal evaluation conditions that affect cross-language comparisons.
Internal Validity. Despite date-based filtering, hidden forms of contamination may persist through similar problem patterns solution templates present in training data. Additionally, models may exhibit temporal bias based on varying exposure to different programming languages during their training periods.
7 FUTURE WORK
Multi-LCB's modular design enables straightforward language expansion. We plan add Swift, Haskell, R, and Julia by defining their compilation commands and runtime environments. We will evaluate proprietary models (GPT-4, Claude, Gemini) to establish comprehensive multilin- gual leaderboards reflecting current state-of-th of-the-rtt performance. The STDIN/STDOUT framework directly supports LCB-Pro (Zheng et al., 2025) and other benchmarks requiring format conver- sion, enabling broader contamination-awaree multilingual evaluation without additional infrastruc- ture changes.
8 CONCLUSIONS
We introduced Multi-LCB, a contamination aware benehmark for evaluating large language mod- els on multilingual code generation. Multi-LCB provides an extensible framework spanning twelve programming languages and continuously updates with newly released problems. The conversion methodology extends beyond LCB to other Python benchmarks (e. g. LCB Pro (Zheng al., 2025)), offering general approach for multilingual code evaluation. By inheriting LiveCodeBench's 's live evaluation protocol and unified STDIN/STDOUT execution, enables rigorous, cross programming language assessment and mitigates data contamination that affects static benchmarks. Our experi- ments expose programming language specific contamination, evidence of Python overfitting, and significant performance gaps across programming languages. We hope Multi-LCB will serve durable resource for advancing the evaluation of code-oriented LLMs and guiding future research in multilingual program synthesis.

---

Published as a conference paper at ICLR 2026
REFERENCES
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang. Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, al. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:22 10. /4868, 2022.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language 108.07732, 2021.
models. arXiv preprint
Aleksander Boruch-Gruszecte Gruszecki, Yangtian Zi, Zixuan Wu, Tejas Oberoi, Carolyn Jane Anderson, Joy- deep Biswas, and Arjun Guha Agnostics: Learning to code in any programming language via reinforcement with a universal learning environment. arXiv preprint arXiv:2508. 04865, 2025.
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl- c: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 49(7):3675-361,, 2023.
Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.074444 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint wrxiv:2107.03374 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
DeepSeek. Deepseek-r1-0523 release, May 2025. URL https: deepseek. com/ com/news news/news250555 DeepSeek News release.
Google DeepMind.
Gemini 2.5: Our most intelligent model, March URL attps://blon google/techoooogeeeeeee /gogl//gogleeeeoooooooo deepmind/ mecii-mooe-ttinnkke updates march- Google DeepMind Blogpost.
2025.
Dan Hendrycks. Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874, 2021.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.0 079774. 2024.
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do Weishi Wang. Md Rizwan Parvez, and Shafiq Joty. xcodeeval: large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv preprint arXiv:2303.03004 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium operating systems princi- ples, pp. 611-1111166 6-66, 2023.
Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. Evocodebench: An evolving code gen- eration benchmark aligned with real-world code repositories. arXiv preprint arXiv:2404.00530 005999 2024.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser. Rémi Leblond, Tom Eccles, James Keeling. Felix Gimeno, Agustin Dal Lago, et al. Competition- code generation with alphacode. Science, 37866624):109211010 2022.
10

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint (rXiv:2402.19172 2024.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprini urXiv:2 04664, 2021.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprini arXiv:2203. 1 13474, 2022.
Qiwei Peng, Yekun Chai, and Xuhong Li. Humaneval-xl: A multilingual code generation bench- mark for cross-lingual natural language generalization. arXiv preprint arXiv:2402. /66944 2024.
Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code laaa: Open foundation models for code. arXiv preprint arXiv:2308. 23x.:2308../950. 2023.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338. 2025.
Lianmin Zheng. Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37: 62557-62583, 2024.
Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li. Stanley Wei, Hangyi Hao. Jianzhu Yao, et al. Livecodebench pro: How do olympiad medalists judge ilms in competitive programming? arXiv preprint arXiv:2506.1199 2025.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint 158777 2024.

---

Published as a conference paper at ICLR 2026
Appendix
CONTENTS
Legal Compliance and License
13
B UI of Multi-LCB
13
C Prompt Examples
13
C.1 AtCoder/CodeForces Example (native STDIN/STDOUT
14
C.2 LeetCode Example (adapted into STDIN/STDOUT)
14
Tasks Distribution
15
D.I Task Distribution by Difficulty and Platform
15
D.2 Task Distribution by I/O Data Dimensionality (LeetCode Functional Format)
16
E Programming language rankings and runtime characteristics
17
Experiments
18
F.1
Models overview
18
F22 Performance on the Multi-LCB (Feb-May 2025 Subset) Across Sampling Tempera-
tures
18
F3
Performance on the Multi-LBB (Jul 2024-May 2025 Subset)
21
F:4
Performance the Complete Multi-LCB Benchmark
21
G Computation time
22
Languages and Compiler Versions
22
I Platform Analysis
23
J Difficulty Analysis
26
K Temporal Analysis
29
Languages errors type
31
12

---

Published as a conference paper at ICLR 2026
A LEGAL COMPLIANCE AND LICENSE
The Multi-LCB benchmark contains no personally identifiable information, offensive content, or proprietary code. It is derived entirely from the publicly released LiveCodeBench (LCB) dataset, which itself sources only publicly accessible contest problems, reference solutions, and test cases from LeetCode, AtCoder, and Codeforces. Our redistribution and multi programming language transformation of LCB fall under Fair Use ($107. U.S. Copyright Act): the benchmark is provided solely for nom-commercial academic research, reproduces only the material necessary for evaluation, and does not diminish the market value of the original platforms or LCB. Multi-LCB is strictly an evaluation resource, no models are trained on these tasks, and released under CC BY-NC 4.0 license to ensure non-commercial use.
B UI OF MULTI-LCB
Figure 6 presents the web interface of Multi-LCB, displaying a subset of tasks released between January 2024 and December 2024. A time-range scroller at the top allows newly released prob-
interactively
select different time windows to filter tasks and monitor model performance lems. This interactive design highlights the live and continuously updated nature the benchmark, enabling researchers to track progress as fresh contest tasks are incorporated.
MULTI-LCB
100
Multi Live Code Bench
CoCode
Dataset TBD
@ubmit
02/01/22024
27/112/202
07/05/2m23
04/29/2223
02200 /2024
01/05/2004
29/08/2224
22/12/2024
06/04/2005
Position
Model
papsgi/svgl
pass@11pythoni
pasag1
p [javescript
passgi 1ltpesoriipl
Thinking 2507_cot
Swen3-233B-ABe
74.70%
84.94%
80.11%
73.05%
Qwen3-2358-
66.60%
76.77%
66.17%
61.34%
Owen3-308-A3BB
Thinking-2507
65.96%
79.74%
61.15%
57.06%
Srun3-328_coo
64.75%
74.72%
64.13%
64.31%
Saen3-308-A3B.00
73.61%
61.15%
61.71%
Figure 6: Multi-LCB web interface showing tasks released between January 2024 Decem- ber 2024, with an interactive time-range scroller for filtering and visualization.
C PROMPT EXAMPLES
This appendix shows example prompts from Multi-LCB. We distinguish the original problem text it appears the source platform and the additional instructions that add in order unify everything into the STDIN/STDOUT format. Original parts are placed in blue boxes, while added parts placed in red boxes.
13

---

Published as a conference paper at ICLR 2026
C.1 ArCODER/CODEFORCES EXAMPLE (NATIVE STDIN/STDOUT)
Original
### Question: Find the number of positive integers not greater than N that have exactly positive divisors.
Input: N
Output: Print the answer.
Constraints: S 10
Sample Input 1:
200
Sample Output I:
Added
## Format:
Read the inputs from STDIN solve the problem and write the answer to STDOUT (do not directly test on the sample inputs). Enclose your code within delimiters follows. Ensure that when the python program runs, it reads the inputs, runs the algorithm and writes output STDOUT.
mun python YOUR CODE HERE nun
## Answer: (use the provided format with triple quotes)
C.2 LEETCODE EXAMPLE (ADAPTED INTO STDIN/STDOUT)
Original
You are given an integer array enemyEnergies and an integer curzentEnergy... (original description)
### Question:
Example 1:
Input: enemyEnergies [3,2,2] eurrentEnergy =
Output: Explanation:
Several operations lead to a maximum of points (see original problem description).
Example 2:
Input: enemyEnergies [2], currentEnergy= 10
Output: 5
Explanation:
Performing the first operation 5 times on enemy 0 yields the maximum number of points.
Constraints:
. 1 v enemyEnergies. Length < 105
1 f enemyEnergiesjj < 109
0 S currentEnergy < 10
14

---

Published as a conference paper at ICLR 2026
Added
## Format:
Read the inputs from STDIN solve the problem and write the answer to STDOUT (do not directly test on the sample inputs). Enclose your code within delimiters as follows. Ensure that when the python program runs, it reads the inputs, runs the algorithm and writes output to STDOUT.
For 2D arrays, the first line indicates the number of rows, followed by newline-separated rows.
Sample Input 1:
2 2
Sample Output 1:
python
YOUR CODE HERE
### Answer: (use the provided format with triple quotes)
For non-Python settings, only the header of the code block is replaced (e.g.. """ cpp, #B. java). The rest of the prompt structure remains identical.
D TASKS DISTRIBUTION
D.1 TASK DISTRIBUTION BY DIFFICULTY AND PLATFFORM
LiveCodeBench (LCB) continuously aggregates competitive programming problems in Python from three major platforms: LeetCode, AtCoder, and Codeforces. Figure 7 shows the monthly distri- bution of tasks by difficulty, and Figure 8 presents the monthly distribution by source platform. To- gether, these figures highlight the steady inflow of new problems and the live, contamination-aware nature of LCB, and, by extension Multi-LCB.
LivaCodeBench: Monthly Task Distribution
Past Pab 2025:
Nedium
Hard
54
PMH05
6H-43
1033
201400
2013-0
200
2014.00
2023
2014
200400
2014
2004
200409
200411
2214-12
2005
Monea
Figure 7: Monthly distribution of Tasks by Difficulty.
15

---

Published as a conference paper at ICLR 2026
LiveCodeBench: Monthly Platform Distribution
LactCooc
Toull Teala 1005
LS0 CO A322
Atcedde 932 D57
BH46
PH577
Number
APL
2023
202004
200
201300
10
20
20000
201400
204-0
40
20
Figure 8: Monthly distribution of LCB tasks by platform.
Each platform hosts frequent contests whose tasks provide a natural language description of a prob- lem, example input/ouiput pairs, and hidden tests, ensuring that solutions must be fully correct to receive credit. Because every contest attracts thousands of participants and receives official editorial review, the problems are inherently vetted for clarity and correctness. Across the full lifetime of the dataset, the platform composition is as follows:
Codeforces:
problems known for wide range of difficulty and algorith- mic focus, almost exclusively in STDIN/STDOUT format.
LeetCode: Interview oriented challenges emphasizing data structures and algorithms, originally in a Functional format.
AtCoder: Algorithmically rich competitive programming
problems, typically
using
STDIN/STDOUT input/output.
D.2 TASK DISTRIBUTION BY I/O DATA DIMENSIONALITY (LEETCODE FUNCTIONAL FORMAT)
LiveCodeBench: Monthly Task Type Distribution (LeetCode)
Part teb 2029
Dimonkonul
Scala: 136 (38-55)
Taa- Dimeneoral
Cemussoona (14.55)
m-26
BM23
THK22
6M22
pV-20
LG L 21
Number of
mx2
2013
201500
203000
201407
20/00
201300
20
2014-1
201407
204-50
2010-2
2044-00
2014.00
2014.99
2044-9
2014.0
20
ol
2000-02
20
Figure Monthly task distribution by 1/O data dimensionality (Leettoode Functional format).
Figure presents the temporal distribution of LeetCode tasks grouped by the I/O data dimensionality of their Functional format. The plot highlights how problems with different input/output structures,
16

---

Published as a conference paper at ICLR 2026
such scalar values, one-dimensional arrays, and two-dimensional arrays, have entered the bench- mark over time, illustrating the variety of functional tasks inherited from LeetCode within the LCB dataset.
E PROGRAMMING LANGUAGE RANKINGS AND RUNTIME CHARACTERISTICS
This study evaluates multilingual code generation across major programming languages selected for their 2025 popularity and broad industrial relevance. Programming language rankings across multiple sources presented in Table 3.
Table 3: Programming language rankings across multiple sources (dates in footnotes)
Language
TIOBE'
GitHuub
Stack Overflow
RedMonk
Python
(26.98%)
4 (57%)
2
C++
(9.80%
9 (23%)
Java
4 (8.76%)
2
7 (29%)
3
C#
(4.87%)
8 (27%)
5
JavaScript
6 (3.36%)
4
1 (66%)
1
TypeScript
37 (0.28%)
6 (43%)
6
Go
7 (2.04%)
3
13 (~2%%
12
Rust
18 (1.01%)
13
14 (~2%)
19
Ruby
23 (0.76%)
8
18 (~11.%%
9
PHP
14 (1.28%)
7
12 (~15%)
4
Kotlin
20 (0.90%)
15
(~3%)
14
Scala
34 (0.41%) %)
14
29 (~1%)
14
These languages span a wide range of paradigms and runtime characteristics, capturing the diversity of real-world software development:
Compilation model:
Compiled/JIT C++. Rust, Go, Java, C#, Scala, Kotlin
Interpreted - Python, Ruby, PHP
Transpiled TypeScript + JavaScript
Type system:
-Static - C++, Rust, Go, Java, C#. Scala, Kotlin, TypeScript
-Dynamic Python, JavaScript, Ruby, PHP
Memory management:
RAAl/manual - C++
Ownership/borrowingg- Rust
Garbage collection Java, C#, Go, Scala, Kotlin, PHP, Ruby, JavaScrip//TypeSerpp
Runtime platforms:
Native C++, Rust, Go
JVM - Java, Scala, Kotlin
.ETT CLR - C#
Interpreters/VMs- Python, Ruby, PHP
JavaScript engines- - JavaScript, TypeScript
Domain ecosystems:
Systems/performance C++. Rust, Go
Enterprise/JVM and .NET- - Java, C#,, Scala, Kotlin
Web/backend and scripting - JavaScript, TypeScript, PHP, Ruby
Data/Al glue Python
17

---

Published as a conference paper at ICLR 2026
F EXPERIMENTS
F. MODELS OVERVIEW
We provide details for all models included in our study in Table 4.
Table 4: Overview of Large Language Models (* denotes reasoning mode)
Model
Short Name
Approximate Cutoff Date
Link
spenai/gpt-oss-1200
Gpt-oss- 1208 (Mediun/Low)
08/05/2025
gpt-oss
openulgpttoss-200
Gpt-oss-20B (Medium/Low)
08/05/2025
gpt-oss
A22B Thinking 2507
2507
Qwen3-2388 235B-A22H TTk*
10/31/2024
qwen
deepsee--a/DDepSSeek
DeepSeek- 1-0528
11/29/2024
deepssek-ni
Qwen/Qwen:: 30B A3B- A38- Instruct-2507
Qwen3 306 A33 Thk-2557--
10/31/2024
qwen
Qvven/Qwen3-322
Qwen3-32B-
qwen
Qwen/ONwen3223B--AW
10/31/2024
Owee/(Qwen3-300-AAAB
Qwen3-30B-A33
10/31/2024
qwen
Qwenn/Qwen3-14BB
Qwen3-14B*
10/3//2024
qwen
ogcu=1])Olymiiccoddr:223
OlympicCodr:322B*
open-ri
Qweus-255b-42bb-mst--22
10/31/2024
qwen
Qwen/Qwen3-8B
Qwen3-8B*
10/1//024
qwen
QweniQwenn-Codde::300
qwen
0wen3-30B-A3B-Jmst-22555
10/1//0044
qwen
OlympieCodee::78
open-ri
Qwem25-CCoder-32BB
03/23/2024
qwen
widdaa/OpenCodaeeessonnng
Nemeotoooonn Nemotron
DpenR<anNmtt3311
Seed-Coder-SB-Inet
pytedance-seed
Qwen/Qwen2.55 Coder 14B Instruct
Qwen2.5. 1ren2.5-Coder-11 1ane--1BB-nmmt Instr
03/23/2024
qwen mistralai ovidia deepseck-ai
mistralai/Devstrrl-Smllll
Dovatral-Small-2555
11/22/2024
uv Weltia/peeReasooningg Nemottron-32B
OpenR'n OpenRsa-Nmtt3225 Nnit-38B
DeepSeck-Codes-333
08/30/2023
F.2 PERFORMANCE ON THE MULTI-LCB (FEB-MAY 2025 SUBSET) ACROSS SAMPLING TEMPERATURES
F.2.1 PASS@I AVERAGED OVER PERFORMANCE VARIOUS SAMPLING TEMPERATURES
Table 5 report Pass@1 scores averaged over 10 runs sampling temperature 0.6. Each score dicates the percentage of problems solved correctly on the first attempt, with higher values reflecting better performance.
Table 5: Performance results at temperature 0.6 Scores represent the Pass@1 (%) metric averaged on 10 runs. Higher is better, bold is best, italic is the second best. (* reasoning mode)
Pythen a11
Kottn
Gperee 2000 (Medion)
12.
100 210
21
78.2
511
Owea7 3358 A218-1111 250- 2507*
74.
15.3 cs
748 119
57.7
68.6
63.4
65.
45.9
67.
656
55.2
617
DecpSeck 0028
68.41
67.
541
124
649
02. 1:2
6255 L.9
123
613
054
61.
(30
1.3
Qwea3 30S.
2507
Gpt-ne 1250"
16.--ts
512
SAB
535
0sen1-288--A221 Atpp
582
588-33
3KL
48.6
299 -3
452
434
459
4022
34.
Qvart-122
58.
Que11: lay
15.
49.9
32.0
437
Gjenee-XPP Y01 (Low)
157
47:5
15.
417
4500
33.544
631
Qoos1 151---58..71 Aor-- 2007
1..
43:4-117
25.1
31 21
203
21.
1
1:3
132
30S-A38
Qwead Oodes 308-A38 leus
36.0 474
35.1
35.5
252
266
163
345
33.8 112
31.5
35,0
20.9
31.2
Deeplerl-R1-Dene 2iot Qpons 12p*
211
119
41.0
133
21.1
k1
281
Quent S-tkee--1
274 97*
218 211-217
20.211
Kiai
200
23
26-3
21.
212
22.
122
364
142
214
23
22
31
11.
21.
222
Owei2.5 Codes 3B-
22.5
21.5
346
14.
22.
1.
176
22.4
110
21.3
198
21.3
OpenColdRan Raa Mac 1-328
02.8
412 13
318 143
8618
99
54
25
12
333
211 230
13
212
13:0
Derplickk- Caale-Codeee Coder Iral:
18.
1k.
140
103
11.0
121
146
Table 6 reports Pass @1 1 scores averaged over 10 runs sampling temperature t = 1.0. Each score indicates the percentage of problems solved correctly on the first attempt, with higher values reflect- ing better performance.
18

---

Published as a conference paper at ICLR 2026
Table 6: Performance results at temperature 1.0. Scores represent the Pass@1 (%) metric averaged on 10 runs. Higher is better, bold is best, italic is the second best. reasoning mode)
Python
Rui
Rain
PHP
Kotiin
Scala
Gue (Medivn)
51.1
67.6
Owex 3358-A378 A378-TM Thk 2007
1XT a3
75.0 237
1N7 pm
570
680
67.
56. 135
49.2 a.*
487
686
58.5 23
Gje (Madian)
662 8:38
050:11
631 210
18
61 615
226
615
019
016
DeeyBock
1206"
Owea] 205 A3B T5k-2507e
62.9
65.9
63.
45.2
49.
45.0
57.6
51.2
424
51.3
51.6
40.5
525
Oves3 125"
59.0 9:2
36.5 c2s
553 V1e
427
51.8
$1.2
$1. 6ax
512
46.5
35.7
40.5 5:54
0om1--15fl.771
18.1 =m1
164
27
234
2.
Qwml-1am 1ay
50.
347
90.5
502
31.
47.3
13.6
64 61.
Gpe-ee20pp Law)
473
15:30311
12.9
4EA
42A
434
314-2
42.1
Owea3 1258 A228 Lost 2507
14.5 115
42.5
1.3
45.7 114
36.7 120
27.3 23
230
435 12
38.5 1.a
43.
42.9
31.2
38.5
Ovest 30S A38 Int:
15.6
Qvees Saannl--Dde--eee 345-433 Ieer
16.
123
35.0
23.
263
23.1
343
31.
34.2
19.2
30.6
Deeplierk-X MI-------a Doni 3215
23.
16.
142 mt
19A
139
227
44.3
133 =30
211
9.2
25 25.
U 201r
15.1.11
21.3
2413
1031
23.3
247
28.5
122" Sead Cide 40 fwar
212
318:235
315
112
20
202
31.70
191
31.8
20.7
Owen? 5 Cide--iff--rr 1if--wttr
12.1 1:3
12.1
21
17.521
114
185
163-217
11.
191
316
188
20.
OpeaCateRes Nered
635
117
10.1
19
177 21
137 217
57.
Des
23.3
11.2
13. -22
203
11
DespSeck
Despaeck Coder 338---33---1 Iost
16.9
17.9
189
10.7
5.7
16.
23
34.4
118
16.1
13.00
F.2.2 PASS @5 PERFORMANCE AT DIFFERENT SAMPLING TEMPERATURES
Table 7, Table and Table reports Pass 5 scores sampling temperatures / 0.2, 0.6 and t = 1.0 respectively. Fach score indicates the percentage of problems solved correctly on the 5th attempt, with higher values reflecting better performance.
Table 7: Performance results at temperature 0.2. Scores represent the @ass@5 (%) metric.
Higher is better, bold is best, italic is the second best. (* reasoning mode)
Rusi
Gpe-ns 1208* (Midiaas
83.7
23.7
A3.1
$3.8
65.3
$2.6
77.5
85.8
83.2
20.4
$5.0
26.7
83.0
Queet 211-2228--2222 2507-
62.6
86.2
83.8
75.8
50.0
50.4
$1.8
75.3
65.0
$1.1
54.5
79.1
80.1
Dhop
76-1
797
13.9
71.1
76.1
71.5
77.8
75.2
77.8
79.1
77.2
Ques3 30B 2505*
71.5
67.6
70.3
Gpe 220 (Lov)
69.1
00.4
70.3
65.9
70.3
70.5
62.6
65.2
63.3
65.6
69.1
63.6
67.5
Qvest 235--228 A228
69.2
70.4
70.1
56.4
69.0
69.5
$6.9
66.8
S6.1
16.5
65.5
55.1
66.2
Qwerl-12H
68.1
68.
51.9
61.
83.8
67.
Quee3 148
45.0
60.9
Gpe 208 (Low)
59.
61:1
56.3
55.8
56.2
54.2
$3.8
5466
55.4
$7.8
59.5
48.5
50.3
Qwerl-Si"
56.0
51.8
51.0
44.6
$1.0
$1.9
40.3
36.3
$1.2
46.5
18.1
35,0
42.6
Q2ea2-278-A220 2507
36.8
31.6
$2.7
SOLE
40.6
Openi
19:1
659
38.9
46.0
45.6
75.1
39.1
413
Qwez3 30B-A3B A38 lase 2507
45.0
44.0
46.9
31.0
32.4
21.5
41.0
42.2
43.6
42.9
45.5
34.4
40.6
Queea3-Co Caari---odee--ae 30B A3B last
43.0
29.5
42.1
34.0
36.9
34.3
45.2
35.9
42.1
38.3
45.4
29/0
39.8
Theppoods- D--lll-------- tape
11.4
31.6
1611
11.3
41-1
16.7
401
Quea2.5 Coder 32B Instr
31.3
305
DerpSedk RI Desill Qwen 1ag
51.4
30.3
420
21.0
25.0
29.6
44.1
109
47.2
11.1
25.8
1055
296
Sead Coden dileeiili-ie lnar
21.6
26.0
$1.2
16.6
29.3
21.5
11.3
25.6
26.6
28.5
16.1
26.2
21.1
Deverail-Smalll---n
34.0
110
301
110
24.8
2600
33.4
DeeySeck Coden
196
Table 8: Performance results at temperature = 0.6. Scores represent the Pass@5 (%) metric. Higher is better, bold is best, italic is the second best. reasoning mode)
Madel
Pytbos
U++
Java
Go
18
T8
CP
Rust
Ruby
PHP
Koitll
Scala
Arg
Gpe-me-1200r (Madamm))
$4.1
63.4
Gpeoss 206 (Medumn)
15.5
79A
17.9
74.4
DerpSock R1- 11-
76.6
79.1
78.6
71.4
71.5
76.0
76.1
76.1
26.3
75.0
19.5
77.6
71.5
Qves1-301B 30B-338-------- Aa3--Ta---e Tak-2900
75.3
29.3
75.5
67.6
75.2
73.0
7222
69.9
62-4
71.8
70L3
99.1
71.5
Gpe 1200 aw)
20.5
08.6
70
66.7
63.4
71.5
66.T
48.1
01.0
64.7
Quez3
66.5
Qweal-14ge
65.4
63.9
636
$7.0
53.6
62.8
S0L4
SQLE
60.4
59.7
61.3
47.6
598
Doer3-W0B-440 Na?
644
65.0
66.8
16.9
65.9
63.4
16.1
$1.7
19.1
45.4
56.7
Gpe (Low)
9:4
51
51-3
16.6
16.6
$4.6
$6.1
15.4
50.4
51.5
Quee3 2358 A22B 2507
$1.0
Qwexl-3tth- A Inan- 2501
56.3
42.1
60.0
16.0
18.8
36.5
46.4
44.6
42.8
45.5
47.0
35.0
440
Openike Nam--ttfe
77.8
71.6
67.4
25.3
15.3
36.3
SF.3
100
41.2
39.8
18.9
23.5
438
DeeySook Disill D280
$6.3
42%8
31.8
45.1
36.2
54.6
41.4
53.4
4n
43.6
24.4
438
5.6
40.5
Qwesal-Oddtn- 30B A36- Aat-
42.4
40.3
41.3
34.0
35.9
31.3
42.4
41.
41.2
45.0
43.5
30.1
39.2
Owes7 Cadere Til Instr
14.1
35.0
1655
11.5
11.
14.0
11.1
33.3
174
34.5
15.0
11.1
179
TheppSodi-tt Bi---eall--ooo---lll
111
194
15.0
46.0
11.
11.6
113
Seed
32.7
31.0
31.2
31.1
311
2505
28,4
315
Qwes2.5 Coder 14B Instr
25.0
27.9
289
369
25.9
27.8
31.6
25.8
31.4
25.5
30.
21.3
266
Hhepplode- Nisleer
353
25.6
2700
11.
306
22.4
16.
744
26.2
38.9
35.
213
239
19

---

Published as a conference paper at ICLR 2026
Table 9: Performance results at temperature 1.0. Scores represent the Pass@5 (%) metric.
Higher is better, bold is best, italic is the second best. (* reasoning mode)
Modkl
Pytkos
Java
Go
15
15
Busi
Ruby
Kotlin
Ark
Gpeoss 1200 (Medamn
R2.8
85.9
64.6
8599
76.0
835
68.0
85.3
Qwezz 2358-2228888 A228 156-2507
85.2
65.2
A5.9
71.4
51.5
$2.4
62.4
62.0
67.8
29.5
84.0
79.0
67.1
Gpe-ss 208 (Meduur"
794
50.5
81.8
77.5
792
9055
75.2
71.5
78.2
77.4
92.6
65.9
78.01
DeepSook R1-053
71.1
78.5
61.6
72.5
71.
11.3
70.8
00.0
72.5
66.1
71.3
71.5
(Lo))
69.5
68.4
Qwea3 328
1144
70.1
70.3
54.1
70.9
5R.4
56.2
61.8
65.3
56.5
67.5
53.9
07.0
Qwer1-2hHl--8
706
71-
70.0
06.6
66-1
70.5
56.0
06.6
61.7
66.4
en.3
54.5
06.5
Qwer--HHlle 1ile
66-
61.
06.1
57.9
50-5
9617
51.3
6044
01.9
(Low)
5699
Qwee1 SB
03.1
56.7
56.7
52.8
59.3
$7.5
56.3
42.6
56.3
50.9
44.5
43.1
53.5
(0ee))-258B-22200 2500
543
56.1
59.6
51.1
44.1
46.1
55.7
46A
$2.3
55.0
52.9
48.9
518
DeeppSock -eeiiill--OOem--1ee
595
45.0
53-2
44.9
66.1
Openl en Name-771 3709
29.3
15.2
21.2
$3.3
9.5
41.5
42.9
43.8
ChanppcCodar-18 78
47.6
47.1
46.7
36.7
43.1
42.2
43.3
25.2
42.3
39.1
37.5
29.2
40.
Queet Coder 30B- A3B- laaz
44.5
40.0
433
34.4
37.6
30.3
45.0
42.1
40.01
40.0
46.3
32.6
40.2
Open OpenCresiten
71:1
66
008
344
76-1
311
44.8
8.5
44.5
11.4
194
9.1
DecpSeck Disill 14g*
14.7
Seed Coder SB lesk
32.0
29.3
30.6
25.0
30.7
29.6
31.7
308
31.9
31.8
30.5
31.9
36.
@wer2 Cateri kill-mnsrr
20-7
4022
908
15.0
298
19.0
11.7
24.6
51.8
296
11.0
26.5
209
Deverai-SSmalil-50
18.1
118
10.6
10.8
23.3
266
Deepseek:
25.7
23.3
21.3
36.0
888
33.5
22.4
F.2.3 PASS @ 10 PERFORMANCE AT DIFFERENT SAMPLING TEMPERATURES
Table 10, Table 1l and Table 12 reports Pass 10 scores at sampling temperatures / = 0.2, L = 0.6 and 1.0 respectively. Each score indicates the percentage of problems solved correctly on the 10th attempt, with higher values reflecting better performance.
Table 10: Performance results at temperature t = 0.2. Scores represent the Pass@ 10 (% metric. Higher is better, bold is best, italic is the second best. (* Rsn mode)
Madel
Pythin
C++
Jasa
Go
15
1%
CW
Rust
Rahy
PHP
Katlis
Scals
Arg
Gutcs (Meduai
2558 A228 Thinking 2007-
87.0
Gpoos 208 (Medtar))
80.2
56.0
83.4
84.0
$1.3
94.
11.
805
91.7
79.4
68.2
76.3
51.4
DecpBock 81 2028
20.9
62.4
832
76.
60.3
832
800
80.
50.9
50.9
54.3
51.1
61.3
Ones3
Thinkig
68.7
63.6
INB (Law)
12.5
15.6
12.5
67.5
Qnea3 2208 A228
12.5
76.5
73.3
695
11.5
74%
12.5
61.1
72.5
T1.
61.8
70.5
Ones3
72.5
73.1
74.1
67.2
12.5
73.
69.5
67.5
67.9
66.7
73.1
65.4
Owes 1uB
71.0
64.9
Qp-0s-00" (Low)
66.1
66.9
63.
60.3
503
56.6
59.5
66.9
55.7
56.
06.1
74.1
39.7
45.5
40.
65.2
18.3
40.5
55.0
20.5
52.4
Queat sp"
3537
31.2
2501
CoyppeChde--p 78
53.4
53.4
51.2
46.3
46.9
41.
31.4
SBA
45.0
45.6
38.9
47.0
Opankollkka--nt 128+
78.6
67.9
65.7
31.1
17.
26.7
59A
55.4
2811
45.4
Quest
51.2
33.8
11A
DespSeck- Doalll-Qee
55.0
52.
Queaa Code 3BB look
45.0
42.0
43.5
366
159
366
458
41.3
435
363
45.
15.6
41.0
DecpBeckSi Si Dusill-Qwaa 148+
542
35.9
45
250
159
26
469
153
512
153
35.5
145
345
Queal.5
Iear
32.5
80.5
295
215
881
Desoul Smudl 2505
29.0
27.5
260
267
363
365
235
35.5
298
161
235
364
Onea2.5 Codor 148 luss
25.
23.5
293
215
27.
360
394
264
103
390
235
22.9
344
206
175
252
Table 11: Performance results at temperature t 0.6. Scores represent the Pass@ 10 (%) metric. Higher is better, bold is best, italic is the second best. (* Rsn mode)
Madel
Pythen
Jase
Raby
Kattis
Cgeeme h0pe (Mullimo
M6.7
87.6
547
87.8
87.8
67.
81.0
$6.3
96.3
R5.5
16.3
B5.5
K7.6
Q A hhnking--
A-5
07.0
86.3
87.8
12.4
14.7
Despliok -022
81.2
63.2
51.7
839
194
81.5
K0L9
K11
18.6
Quest NB ASS www---neee---- 2507
76.1
00.9
721
83.4
15.3
81.7
79A
77.1
75.
50.5
THL
BS.7
78.6
Ggome (La))
72.7
14.1
69.3
71.5
12.5
59.5
75.5
Quest 2218-A22p
71.
25.5
192
79.8
11.0
66.1
Y1.
65.0
12.
6337
687
66.1
66.2
85.4
6712
5807
56.9
56.
55.0
(pest
656
626
65.4
554
58.8
55.4
Queat agr
562
53.7
653
62.
519
44.1
48.9
Ques)
A222 228.-2222-tna-22330
545
58.0
35.3
353
54.2
64.1
334
350
29.5
51.2
14.1
47.3
46.5
Opanke-Nnm-tii
52.5
619
79.
18.1
51.9
OpanCodeiion Smat 11-128
011
78.5
59.5
69.5
11.4
66.7
550
51.2
31.3
26.3
Quea3 305- ASB Iatr 2507
45.4
52.1
31.7
31.2
4500
S42
41.8
51.9
483
43.8
48.9
41.2
45.0
Ocympee 75*
46.5
41.
542
450
450
DespSeck- RL. Duall Queas 148
37.5
59.5
$1.9
412
36A
42
363
16A
55.8
195
35.
Qnea2.5 Seedd Iear
36.
37.4
405
35.
M65
307
21A
369
31A
369
15.1
368
314
36.
311
318
Saall-2500- 2505
32.5
Quea2 S.Clar 148 last
30.9
3055
344
305
360
31.
385
12
113
363
365
DecpSeck Caygeck-Call--ie Cat.k--oo--------aa 31B-
25.7
295
290
275
152
200
244
360
361
362
99
353
2755
20

---

Published as a conference paper at ICLR 2026
Table 12: Performance results at temperature 1.0. Scores represent the Pass 10 (%) metric. Higher is better, bold is best, italic is the second best. (* Rsn mode)
Python
Jasu
Ruby
PHF
Kotlia
Scala
1808 (Medium:
30.3
87.0
87.8
96.3
86-3
84n
Qwent Qw005-115B- 1148-1133300 A278 Thk-2507 2007
61.
96.1
85.5
87.8
10.3
98.1
A55
87.0
62.4
74.
87.5
64.0
55.3
Ojtoss 201 (Midiam)
K2.6
51.4
87.7
86.7
9244
55.5
8477
86.3
80-
61.7
805
75.
$4.7
Despleck-81-
718
73.3
65.7
05.7
71.0
T18
75.1
08.5
Qwan5-12B7
719
16-1
71.
T6.
71.0
74.8
76.3
T1
710
70.
673
RAM
15.3
04mni-7198-7779
71.
16.1
71.0
06.1
67.2
66.1
Oweek 316-A38
62.6
54.2
Gyt-se 218 (Low)
01.
64.1
66.3
63.4
50.3
64.9
58.8
61.
611
61.1
64.1
25.7
59.5
Osans-4p
56.6
61:2
66.3
MA.3
59.3
64.9
64.1
512
56.5
M1.1
46.5
47.5
51.5
Uwani-JisS-Aee indrd2077
56.7
3933
56.5
518
49.
OpeiRsn Non-328
51.9
52.T
15.3
328
Owen3: A3B-A38 Ina 2507
01.7
55.7
51.2
54.3
43.5
54.3
443
51.
53.4
16.9
194
43.5
15.3
OpenCodeRan Nom-l
07.5
51.2
71.8
35.1
71.0
16.2
63.4
13.7
26.7
51.9
46.1
35.1
30B-ABB-aatt
DeopSesk-B --l-- Duntl Quees Eilt
15.0
31.5
413
65.0
314
706
Owan? SChdde--Tli-nn Inl-
10.2
16.3
96.7
16.2
15.9
41.
11A
16.2
114
10.
158
174
405
Soold ooke- Ieet
11.1
163
11.
11A
12.8
141
1511
111
Qwen2
33.6
Small 2505+
32.1
264
DespSeek Coda: 338--33--uu lasa
26.0
25.0
29.0
31.3
27.5
31.3
2500
26.7
23.7
20.5
130
20.6
15.2
F.3 PERFORMANCE ON THE MULTI-LCB (JUL 2024-MAY 2025 SUBSET)
Table 13 reports Pass@ scores sampling temperature 0.2 for all evaluated models on the Multi-LCB subset containing tasks from July 2024 to May 2025. Each score reflects the percentage of problems solved correctly on the first attempt, with higher values indicating better performance.
Table 13: Performance results on Multi-LCB tasks from July 2024 till May 2025. Scores represent @ (%) metric (higher is better). (* reasoning mode)
Medel
Prtbon
C++
Juno
Go
C#
Rust
Rabe
PHP
Kottis
Scala
Ang
16.1
71.1
69:1
Qwead The
2158 A228
$2.9
Quee3 328
63.4
63.6
$9.5
43.7
$1.5
54.7
53.5
41.4
53.9
32.5
42.3
35.1
$1.3
Onee3 308-488
60.5
36.5
56.3
37.0
489
33.1
30.5
43.1
50.5
30.7
39.2
49.2
OkraS-14pe
485
NL.
45.3
UympsCatee
$1.3
51.9
54.6
42.9
45.3
43.5
40.1
OueeS-pp
46.9
08.5
42.5
27.2
39.
41-2
2417
42.3
18:1
2A.9
35.6
Owce5 388-38 Baan 2507
40.6
36.4
22.5
37.4
33.4
35.5
36
26.8
19.7
31.0
OyngkCGoden TB
26.6
25.7
Nemonnn 358
69.8
09.5
41.5
13.9
133
10.5
33.0
1.8
20.7
30.3
16.3
748
369
Qwee2 Coder 225 leatz
27.4
27.4
30.2
24.2
60
24.0
27.8
25.4
24.1
22.0
26.5
22.7
248
Secl-Cootrr B8-leat
21.1
Deviral-iral Small-262-
25.4
20.9
23.7
22.9
22.7
21.1
19.5
1911
21.3
200
13.1
200
OokeCnadfon--aan Neaee--e---ll I. 1.1-2
62.5
38.0
306
33
8.9
5.6
27.0
1.4
23.5
24.7
12.5
3.6
21.0
Deoptree Caneee---we--111 Till--iil-
12.7
16.7
11.1
F.4 PERFORMANCE ON THE COMPLETE MULTI-LCB BENCHMARK
Table 14: Performance results on Multi-LCB (n=1055 per language). Scores represent the Pass (%) metric (higher is better). (* reasoning mode)
Model
Pythen
Jacs
Gn
18
Hast
Ioaty
Kettta
Scala
Ang
Ouant 30h 10: 1997+
$0A
K0.3
15.5
911
63.5
61.
S6.
10:
50.6
66.5
Qnaad-22 20004-238--2121 A228"
11.5
12.0
72:0
002
64.
63.3
61.5
56.2
532
$2.4
53.1
40.0
62.
Oncak
31.
64.5
14.
Qwaak 148
13.4
67.2
55.9
47.8
65.5
63.9
63.7
41.3
S4.
660
41.3
35.0
ST.4
Owerk 2358 A228-1nan 2500 2507
59
194
29.
45.0
34.3
25.0
$5.4
33.5
39.5
54.
463
35.4
4k.7
Osari-app
66A
112
51.
36.3
51.6
55.6
218
144
1677
Oweni 300 tean 2003
13.5
15.4
11.5
31.
33.6
1066
44.2
-5.3
40.4
IT.
318
3611
11.4
Ouees-Coder Mit-A
07.8
43.5
2631
366
-3.1
416
61:7
21.
16.1
Connps-Coke-18 Owerd Cyben 48 luse
4500
43.0
4677
30.9
32.8
31.9
35.
185
34.5
18.5
28.5
19.8
122
33.6
16.0
35.1
263
32.1
25,0
$3.9
SLS
26.6
16.3
20.6
16.5
263
17.4
Seel Colos d8 bar
26.3
25.1
25.
28.5
274
13.4
67:2
16.1
85
11A
150
311
305
18:3
153
7.4
26.1
Sovertt-Smal--suus
30.3
21.
264
26.4
2641
24.0
13.9
26.9
Table 14 reports Pass scores at sampling temperature = 0.2, for all 19 evaluated models on
the complete Multi-L.CB benchmark, which contains 1,055 tasks per programming language. Each
21

---

Published as a conference paper at ICLR 2026
score reflects the percentage of problems solved correctly the first attempt, with higher values indicating better performance. Models marked with an asterisk (*) are reasoning-enhann enhancedhhed variants.
G COMPUTATION TIME
Table 15 reports the average compilation and execution time required to evaluate one full Multi-LCB run (1,050 tasks per language) across 90 parallel CPUs. On average, each language requires about 8 min 50 (+= 530 s) per model, with total wall-clock time of roughly 106 hours when aggregated over all twelve languages.
Execution cost varies noticeably by language. Ruby shows the highest mean time at 17 min 37 followed by Go and Python, each exceeding 11 minutes on average. In contrast, Kotlin, PHP, and JavaSeript complete evaluation in under 4 minutes. These differences primarily reflect compilation overheads and runtime performance of each language's toolchain, and they guide resource planning for future large-scale model evaluations.
Table 15: Evaluation times (compilation + execution on tests + matching) across programming languages. Runs were executed in parallel 90 CPUs 1050 tasks (vl-v6). Averages and standard deviations are computed across the measured models.
I Language
Avg. Time (mmzss)
Std. Dev. (mm:ss)
Avg. Time (8)
Std. Dev. (8)
C#
9:10
3:08
$50. 15
188.83
10:25
2:54
625.72
174.17
Go
12:36
2:36
756. 13
156.71
Java
10:44
2:55
644.07
175.96
JavaScript
3:44
1:05
224.73
65.14
Kotlin
1:46
194.87
106.23
PHP
0:49
209.82
49.78
Python
11:38
2:56
698.71
176.07
Ruby
3:27
1057.42
207.45
Rust
7:41
3:50
461.04
230.24
Scala
7:25
1:16
445.24
76.41
TypeScript
0:55
497.59
55.37
Average
8:50
4:12
$30.46
252.51
Total (sum)
106:05
6365.48
H LANGUAGES AND COMPILER VERSIONS
All experiments were conducted in a controlled environment using the following language runtimes and compiler versions to ensure consistency and reproducibility across all tasks Multi-LCB:
C++: gec 14.3.0
Java: OpenJDK 8.0.412
Python: 3.12.11
Rust: 1.88.0
Go: 1.22.12
Ruby: 3.3.6
JavaScript (Node.js): 20.19.4
TypeScript (Deno): 2.3.4
C# (Mono): 6.12...199
Compilers (general): 1.11.0
PHP: 8.1.0
Kotlin: 2.2.0
Scala: 2.1
pip: 25.2
These versions were used consistently for compilation, execution, and evaluation to guarantee re- producibility of all Multi-L.CB results.
22

---

Published as a conference paper at ICLR 2026
I PLATFORM ANALYSIS
Figures 10, 11, and 12 show performance comparison between LeetCode and AtCoder platforms across different programming languages. Models demonstrate varying capabilities depending on the platform, with some excelling on LeetCode's interview-style problems while others perform better on AtCoder's competitive programming tasks.
Generation Metrics Heatmap Group (PYTHON, C++ JAVA, C#)
100
DeepSeek
80
A229
$4.2
51.2
Down1-1 AIP
50.4
44.9
53.7
49.6
51.2
46.6
47.6
45.8
44.9
46.3
Olymp c2d
46.8
42.9
48.8
50.4
44.9
53.7
42.0
38.8
43.9
39.7
38.8
40.2
Qwen3 2359 A228 Instr 2307
42.7
34.7
47.6
41.2
30.6
47.6
47.3
38.0
52.4
45.0
34.7
51.2
Pass@j (%)
Models
Owen 6g*
40.5
38.8
41.5
41.2
38.8
42.7
38.2
34.7
40.2
32.1
30.6
32.9
Gaenn (Codee-10B-A Wh-istt Wi-instt
35.9
30.6
30.0
30.5
20.4
36.6
34.4
32.7
35.4
32.1
26.5
35.4
Owend Neen3-30--A-B MiB---ratt-30----
$8.9
30.8
43.9
31.3
18.4
39.0
35.1
26.5
40.2
33.6
26.5
37.8
OlympicCoder 78*
32.8
28.6
35.4
34.4
18.4
43.9
26.7
18.4
31.7
30.5
30.6
30.5
Owen2 5 Coder 328-1228- Prst-isst----
26.0
18.4
30.5
25.2
20.4
28.0
27.5
22.4
30.5
28.2
24.5
30.5
Open 2- Neete--errrr---- -218
61.8
67.3
58.5
44.3
45.1
34.4
32.9
29.0
24.5
Coden Bis-mntt-
22.1
20.4
23.2
23.7
26.5
26.0
24.4
DevitralSSmallzaee
19.8
Seeppree--cce------eee
PYTMON Overall
PYTHON LeatCode
Ry RYTHON ArCoder
Overall
C++ LantCodl C++ ArCodar
JAVA JAVA Overall
JAVA LeutConte
ArCoder
Overall
Ce
LaetCodle
C#
C# ArCodor
Languages Platforms
Figure 10: Code generation performance heatmap by platform for Python, C++, Java, and C#. Shows overall performance and platform-specine results (LeetCode vs AtCoder) across different models. Values represent Pass @ scores (%).
23

---

Published as a conference paper at ICLR 2026
Code Generation Metrics Heatmap Group 2 (RUBY, PHP, KOTLIN, JAVASCRIPT)
Owen3-359-A728-1 133---22---h----- 2507"
48.1
59.2
41.5
68.7
77.6
63.4
63.4
61.2
64.6
68.7
77.6
63.4
DeapScok-RL 0528"
56.5
51.0
59.8
56.5
51.0
59.8
58.0
55.1
59.8
55.7
49.0
59.8
Owen3-330B--8 1-30--30-111 ME------------0000 2507*
38.2
53.1
29.3
59.5
57.3
48.1
40.8
52.4
69.6
53.1
47.6
Owen3 A38
Oown]-1-0P-
45.8
46.9
45.1
42.7
43.9
30.2
42.7
44.9
Olymp cCoder 328
42.7
42.9
42.7
44.3
42.9
45.1
32.8
26.5
36.6
36.6
42.9
32,9
Gomm
40.5
$4.7
43.9
44.3
32.7
$1.2
39.7
30.6
45.1
26.7
12.4
51.7
Models
Ower 88*
35.1
38.8
32.9
30.5
32.7
29.3
22:1
12.2
28.0
35.1
38.8
32.9
Passe1
Gwen Sodd----d--308-44
20.8
37.4
26.5
Owen2 5 Coder 328
21.4
22.9
26.8
Openits n--emuuttr--222
22.1
32.7
15.9
15.3
17.1
16.8
20.4
14.6
9.2
8.2
Seed Coden BB instr
20.6
16.3
23.2
20.6
16.3
23.2
23.7
16.3
28.0
22.1
18.4
24.4
Devitral-Smal 2505
18.3
12.2
22.0
20.6
18.4
22.0
19.1
12:2
23.2
21.4
14.3
25.6
OpenCodeiisn-eemeee Nemeeertoo----erroo.n... 1-3-220
25.2
24.5
25.6
24.4
22.4
25.6
11.5
12.2
11.0
9.9
8.2
11.0
despsaek-codee watpoolk-oodd-33----nee 33biinst
4DD-nstruct
RUBY
Overall
LestCode
RUBY
RUBY
ArCoder
PHP Overall
Last
PHP
ArCodar
PHP
Overall
KOTLIN
Atcoder
JAVASChipT Overall AVASCRIPT PotCode
AtCodor
Languages and Platforms
Figure/ 11: Code generation performance heatmap by platform for Ruby, PHP. Kotlin, and JavaScript. Shows overall performance and platform-specfficc results (LeetCode vs AtCoder) across different models. Values represent Pass@ scores (%).
24

---

Published as a conference paper at ICLR 2026
Code Generation Metrics Heatmap Group (TYPESCRIPT, RUST, SCALA)
Deen3-1150-728---17 2507-
58.0
63.3
54.9
57.3
57.1
57.3
42.0
38.0
43.9
51.9
55.1
50.0
DespSool-R1 0528"
55.7
55.1
56.1
47.3
40.8
51.2
57.3
51.0
61.0
56.5
53.1
58.5
Qwen3-3006 Ne-2
42.7
40.2
41.2
47.8
50.4
38.9
36.7
80
Qwwn2-235-A428--
Owen3 Dwen3-300-K30- 48.9
46.3
35.1
39.0
28.6
Owen3-14B
44.3
46.9
42.7
36.6
24.5
43.9
30.5
26.5
32.9
29.0
30.6
28.0
Dlymp cCodee-121
31.3
34.7
29.3
32.8
20.4
40.2
31.3
24.5
35.4
25.2
22.4
26.0
Qwen3 2358 A22B 2507
19.8
16.3
22.0
38.2
26.5
45.1
38.9
24.5
47.6
29.0
28.6
29.3
Paseg1 (%)
Models
Qwer
36.8
40.2
18.3
17.1
22.1
22.0
2507
40
ClymolcCaden 70*
21.4
25.3
8.5
Owen2 5 Coder 328 instr
25.2
20.4
28.0
23.7
22.4
24.4
25.2
22.4
26.8
19.8
14.3
23.2
13.0
18.4
9.8
15.3
8.2
19.5
0.8
0.0
1.2
6.9
6.1
7.3
Seed Cocer instr
22.1
22.4
22.0
19.8
14.3
23.2
20.6
14.3
24.4
22.1
12.2
28.0
DevstralSmall-330
OpenCoderlsn-
19.5
Owen? S-Cnder: 4i-instrurt
21.4
14.3
25.8
21.4
24.4
17.6
22.0
18.3
10.2
23.2
TYPESCRIPT
PYPESCRIPT
LearCode SYPESCRPPT
AiCoder
6o Overall
Laat
ArCoder
G0
RUST Overall
RUST LeetCade
RUST
SCALA
Overall
SCALA
SCALA
ccooer
Languages and Platforms
Figure 12: Code generation performance heatmap by platform for TypeScript, Go, Rust, and Scala. Shows overall performance and platform-specific results (LeetCode vs AtCoder) across different models. Values represent Pass @ scores (%).
25

---

Published as a conference paper at ICLR 2026
J DIFFICULTY ANALYSIS
Figures 13, 14, and 15 present performance breakdown by difficulty levels (Easy, Medium, Hard) across programming languages. The results reveal significant performance degradation problem complexity increases, with Hard problems showing the largest performance gaps between models.
Difficulty Metrica Heatmap- Group (PYTHON, JAVA, CR)
Omard-2208 2207
71.0
54.3
07.2
40.2
91.3
00.7
07.9
SuepSeno R1 0522-
61.4
45.
96.3
64.
47.5
64.1
92.3
00.2
55.7
94.3
Oward 200 Nww13-000-0001122220
64.2
42.0
76.3
30
96.3
00.2
37.3
90.5
Owwr1-328r
60.2
22.8
667
21.3
93.5
66.7
09.6
Owerb 2wwe-2235--222--
$5.0
28.5
21.4
40-7
51.1
Owech 333
64.2
04.0
47.4
Olympictode
Pusses es0
Modelts
Qpentodttoo
19.1
10.3
61.3
15.4
18.3
61.3
5 Ooden iutrrct
12.8
Primoo
Overnal
WFTNOW
Medium
PYTHON
C+a Overall
Kaayy
C+
C+
Overrall
JAvA
JaUe
Latia
Hard
Overaill
Medioum
Languages and Difficulty Levels
Figure 13: Code generation performance heatmap by difficulty level for Python, C++, Java, and C#. Shows overall performance and fffficulty-speciffe results (Fasy, Medium, Hard) across different models. Values represent Pass @ scores (%).
26

---

Published as a conference paper at ICLR 2026
Difficulty Metrics Heatmap Group
(RUBY, PHP, KOTLIN, JAVASCRIPT)
Oweeh 20000-298---201--
58.0
08.6
Owech 303-3-33------00
13.3
61.
19.7
07.1
41.0
45.8
77.4
45.8
13.1
51.
11.5
77.4
462
43.5
00.6
OlymeluDdde 328
42.7
44.3
53.5
48
18.0
87.1
33.3
36
76.2
owesk 201e wiON ha022501
00.8
90.4
06.2
15.
48.4
96.5
41.9
21.4
29.7
93.4
64.9
11.5
26.7
64.9
17.8
14.1
Models
Owen3-a3r
25.1
00.5
46.2
4.0
30.5
77.4
20.8
6.6
22.1
51.
20.2
3.3
25.4
71.
66.2
Owann Code- (38-A33--nul
35.2
20.3
35.5
6.2
20.0
64.5
33.3
9.5
37.4
87.1
41.0
3.8
26.7
64.5
25.5
Oweed 900 Add tote 1007
$4.4
77.4
35.5
5.6
55.1
74.2
48
8-2
29.3
55.1
41.9
9.2
15.1
54.5
15.4
ClmoocCader 20
26.7
67.7
25.1
76.2
17.8
3.8
26.7
71.0
21.1
19.1
$1.
15.4
Gaan 333-mat
21.4
3.6
17.0
67.7
20.5
16.1
Openfann Semodron-32
20
58.1
Overnall
RUBY
Easy
RUBY
Ovemall
PHI
Mediule
Herd
KOTLUM
KOTLIN
Moniimm
KOTUIN
Mevld
JAVASCRIYY
Overall
HAVASCHeT Baoe JAVAS AIVASCRITT
AVASSCat
Difficulty Levels
Figure 14: Code generation performance heatmap by difficulty level for Ruby, PHP, Kotlin, and JavaScript. Shows overall performance and difficulty-speciff results (Easy, Medium, Hard) across different models. Values represent Pass @ I scores (%)
27

---

Published as a conference paper at ICLR 2026
Difficulty Metrics Heatmap Group
(TYPESCRIPT, GO, RUST, SCALA)
Ovenr -wwo0-235--222---
40.6
41.0
23.3
41.0
313
30.6
Owerd 30--332--3---0
46.3
13.7
33.3
6.2
74
48.7
28.8
74.
44-3
15.
38.4
35.9
19-7
742
30.8
25.0
20.8
25.0
23.3
73.0
25.8
0.2
32.8
71.5
41.6
6.6
21.3
76.2
20.8
9.8
25.2
61.0
28.8
$2.8
35.5
23.3
9.0
38.2
03.5
38.5
34.8
38.5
77.4
43.0
38.0
20.0
74.2
26.2
6.6
Pesoeg (%)
Models
Owerd-997
30.8
77.4
48.2
5.8
18:1
$4.8
12.8
333
22-3
67.
15.4
3.5
25.7
87.7
26.5
5.5
27.5
24.0
25.6
8.2
40.6
25.9
11.5
16.1
34.8
15.4
19.0
17.0
3.3
74.2
30.8
23.7
28.1
ClymroCaaeern
35.5
Smakk
OpenCadséerr denstern 1-322
5.3
12.9
16
0.0
32.0
deepoeck cece- 520
3.0
12.8
0.0
12.9
26
13.7
48.4
Ovent Coder 148 Instrvtt
21.4
71.8
21:4
26,3
1.8
94.
15.4
0.0
$1.8
12
Saky PRSCRPT
Hard
Overaal
60
60
Medifom
60
Hard
RUST
Bony!
Mediom
Overatl
SCALA
SCALA
Difficulty Levels
Figure 15: Code generation performance heatmap by difficulty level for TypeScript, Go, Rust, and Scala. Shows overall performance and difficulty-speciff results (Easy, Medium, Hard) across dif- ferent models. Values represent Pass @ scores (%).
28

---

Published as a conference paper at ICLR 2026
K TEMPORAL ANALYSIS
Figures 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, and 27 illustrate monthly performance trends from 2023 2025 across different programming languages. A notable declining trend is observed across all models and languages, with top-performing models dropping from approximately 80% 60% @ scores over time. This consistent degradation pattern appears universally across programming languages, suggesting systematic factors rather than Ianguage-speciffe issues. The decline may be attributed two primary factors: (1) data contamination effects, where models perform better on older, potentially seen problems, and (2) increasing problem complexity over time as benchmark creators develop more challenging tasks maintain discriminative power.
Moethly pessot Thends Pythhn Acress Hodels
Monthly passo Thends Acress Nodils
Figure 16: Monthly Pass trends for Python.
Figure 17: Monthly Pass trends for C++.
joss
Figure 18: Monthly @ trends for C#.
Figure 19: Monthly Pass @ trends for Java.
Trende Acrous Modelss
Monthly peanc] Thends for Go Across Hodel's
Figure 21:
Monthly Pass @ I trends for
Figure 20: Monthly Pass@1 trends for Go.
JavaScript.
29

---

Published as a conference paper at ICLR 2026
Figure 22: Monthly Pass@) trends for Kotlin.
Figure 23: Monthly Pass @ trends for PHP.
Monthly passo3 Thends for Suby Acress Nedels
Heethly parsg Trands for Rest Across Models
Figure 24: Monthly Pass @ trends for Ruby.
Figure 25: Monthly Pass @ trends Rust.
Neethlly pansatt Trends State Across Models
Figure 27: Monthly Pass @ trends for Type- Script.
Figure 26: Monthly @ trends for Scala.
30

---

Published as a conference paper at ICLR 2026
L LANGUAGES ERRORS TYPE
Figures 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38,3, 39, 40, 41, 42 and 43 illustrate detailed error break- downs across different programming languages and models. Several consistent patterns emerge:
1. Wrong-answer (WA) errors dominate across almost all languages and models. For every model, WA the largest source of failure in both Python and non-Python languages, indi- cating that the primary bottleneck remains algorithmic correctness rather than compilation or parsing.
2. Compiled languages show substantially more compiler- and type-related errors. Languages such C++, Java, Rust, and Go exhibit significantly higher rates of compilation errors (c.g., missing imports, type mismatches, incorrect signatures) compared to Python. This pattern is consistent across all models and reflects the challenge of generating syntactically valid and type-correct code when strict compilation pipelines are enforced.
3. Runtime exceptions increase in languages that require explicit input parsing. languages like Java, C#,, and Go, runtime errors (e.g., NullPointerFixeepionn IndexError, ValueFrror) are far more frequent than Python. This supports the hypothesis that the STDIN/STD OUT format, while uniform across languages, exposes weaknesses in model robustness to input handling and data conversion.
4. Timeout and resource-related failures appear more often in slower languages and for reasoning-tunee models. Java, Rust, and Go show noticeably more TimeoutExpired cases, likely because models occasionally generate inefficient implementations. Reasoning-heate models (e.g.. R1-0528, Nemotron-32BI they attempt more complex multi-step logic.
more prone to long-running solutions when
5. Empty-code and trivial-syntax errors are rare but nonzero. These errors appear mostly in smaller models (e.g., 7B-14B) and are nearly absent for 30B+ models. This indicates that larger models rarely fail the initial code-structuring stage, with most errors occurring deeper in the exccution pipeline.
6. Cross-model consistency in error profiles. Despite architectural and training differences, the overall error distributions are remarkably stable across models, demonstrating that: Python remains the least error-prone language, Compiled languages introduce predictable error modes, and Languages with verbose input/output handling (Java, C#, Go) amplify runtime failures.
7. Error distributions reinforce the observed performance gaps. The breakdowns offer a mech- anistic explanation for the Pass@1 disparities reported in the main results. For example, models underperforming in Rust and C++ do so not because they fail to produce solutions, but because syntactic and type-level correctness is significantly harder to achieve those languages.
Figure 28: prppseek-coder--5 33bee-----nnntrrett
Figure 29: PeepSeek-R1-0S5
31

---

Published as a conference paper at ICLR 2026
Figure 30: Devstral-Small-2500
Figure 31: OlympicCoder-77
Figure 33: 1.1-32B
OpenCoddeReasoiing-eeeen Nemotron
Figure 32: OlympieCoder-321 =
Figure 34: OpenReasoning-Neeorre
Figure 35: Qwen2.5-Coder 14B-Instruct
32

---

Published as a conference paper at ICLR 2026
Figure 36: -
Figure 37: Qwen3-8B*
Figure 38: Qwen3-14B**
Figure 39; Qwen3-30B-A3B
Figure 41:
Owen3-235B- AA2B-
Thinking- A222-Thinniing.
Figure 40: 1aen3--3B-AA3BInsttte 2507
2507
33

---

Published as a conference paper at ICLR 2026
11ifil !ii!!!!
Figure 42: Owenn--Coder:30B--3T---ntttt
Figure 43: Seed-Coder-SB-Inseee
34


Reviews:
[{"review_id": "FjXJVLYt1v", "reviewer": "Reviewer_VdKt", "summary": "The authors introduce Multi-LCB a multilingual extension of LiveCodeBench (LCB) that evaluates large language models across 12 programming languages (instead of just Python). The Multi-LCB benchmark is built by converting LCB into a unified STDIN/STDOUT format — which works across all supported languages. The authors present a robust evaluation of 20 large language models, identifying important trends on programming language bias in performance and contamination trends.", "questions": "Please clarify how this approach is novel with regard to Athiwaratkun et. al. For example, why not just use the approach they introduced for creating MBXP? Are there advantages to the Multi-LCB conversion method or is the primary contribution the benchmark dataset itself?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "- Automatic conversion of LCB into 12 languages is a significant contribution to the community.\n- Presents clear evidence for a bias towards certain language, in particular with python performance being consistently higher than other languages. The results also show clear hierarchies (Figure 4) of performance with python, c++, and java in the highest performing category.\n- Moreover, the results of performance over time, highlighting important trends resulting from large pretrained models including various benchmarks in the training set. The proposed Multi-LCD offers to track such information.\n- The rigorous methods are also a strength:\n  - Evaluation setup: 20 diverse models (7B-685B parameters), evaluation within a sandboxed execution, and strict resource limits, and a balanced evaluation for each language via common task sets.\n  - Care was taken to create reproducible results. Specifically, the authors give compiler versions, inference parameters, and public code release.\n- Clear presentation: the writing is clear and the figures clearly communicate important findings.", "weaknesses": "\"These results confirm that strong Python ability is not a reliable proxy for true cross-lingual code generation competence.\"\nIt's unclear why the authors make this claim — to me their results suggest the opposite. Specifically, Figure 3 shows a clear correlation between python performance and average performance across all other languages. The benchmarks can show a bias towards performing better on python while still be proxy for performance on other languages. This claim also contradicts prior works that show multilingual models perform best because performance is correlated across difference languages.\n\nMissing citation: [\"Multi-Lingual Evaluation of Code Generation Models\"](https://arxiv.org/pdf/2210.14868) is a well cited paper on converting monolingual datasets to multilingual code, including the popular python dataset MBPP. This is an important citation in this area and very close to your approach, so you may want to explicitly clarify how the two approaches differ. Athiwaratkun et. al provide already provide an approach for converting benchmarks which enables direct comparison across languages.\n\nFor benchmarking purposes it would have been nice to see how commercial models like GPT5, Claude, and Gemini perform.\n\nThe authors only present results from `pass@1`, which is valid, however showing results with a greater sampling budget would help: (1) add to significance of findings (which are missing), and (2) more importantly `pass@k` metrics tend to show a sigmoid relationship between `k` and performance. In other words, for higher values of `k` we're likely to see smaller gaps between \"easier\" and \"harder\" languages, and the gap between python and other languages may be noticeably smaller. Likewise, the authors use a temperature at 0.2, which is quite low and may be preventing more exploration and possibly negatively biasing languages that make up a small portion of the training data.\n\n\"we evaluate 20 recent large language models on Multi-LCB, restricting tasks to those released after 2025-02-01 to ensure live, post-cutoff evaluation and minimize any risk of training-data leakage\"\nThis effort to avoid contamination is good and likely significant helps avoid contamination, but it's unlikely to \"ensure\" contamination since problems may have existed in other areas previously. LLMs have such large and broad pretraining datasets that indirect contamination is always possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HPms2uFLmK", "reviewer": "Reviewer_QT4h", "summary": "This paper presents MULTI-LCB, an extension of the existing LiveCodeBench (LCB) benchmark from Python to 12 programming languages. This paper design an automated pipeline that converts function-based LeetCode problems into standardized STDIN/STDOUT tasks and execute them within an isolated sandbox environment. The paper reports Pass@1 results for 20 publicly available large language models, analyzing cross-language differences, contamination effects, model scaling, and fine-tuning strategies.\n\nThe major contributions are:\n1. Expands LCB from a single-language (Python) setup to 12 languages.\n2. Standardizes problem I/O specifications to enable consistent cross-language evaluation.\n3. Preserves LCB’s time-based cutoff to mitigate training-data leakage.\n4. Benchmarks 20 code generation models under a unified framework.\n\nOverall, the work demonstrates strong engineering and community value, providing a practical foundation for multi-language evaluation of code models. However, its methodological novelty is limited.", "questions": "1. The work mainly represents a large-scale re-engineering of an existing benchmark rather than a conceptual or methodological innovation.\n\n2. The paper does not quantify whether converting function-based problems to STDIN/STDOUT truly preserves difficulty, leaving uncertainty about potential format-induced biases across languages.\n\n3. Only Pass@1 is reported; missing Pass@k and error breakdowns (e.g., compilation vs. runtime) limit interpretability of language gaps.\n\n4. The “Python advantage” is discussed but not supported by controlled analysis; other confounding factors such as type strictness or dataset differences are not explored.\n\n5. Some deviations from the LCB Python results (up to ~8%) are acknowledged but not analyzed, reducing confidence in full cross-language equivalence.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. High engineering quality: Large-scale multi-language extension with reproducible infrastructure.\n\n2. Comprehensive coverage: Evaluation across 20 models and 12 languages under consistent settings.\n\n3. Community relevance: Provides a standardized and contamination-controlled environment for fair model comparison.\n\n4. Clear presentation: Tables and figures effectively summarize results, aiding interpretability.", "weaknesses": "1. The authors extend LiveCodeBench from Python to 12 languages (e.g., C++, Java, Rust, Go, Kotlin), building a unified STDIN/STDOUT interface and Docker-based sandbox for consistent evaluation. The implementation is robust and reproducible.\n\n2. The benchmark covers 20 models (instruction-tuned, reasoning-tuned, etc.) under identical settings. Results across tables and heatmaps show consistent model ranking and meaningful trends in language difficulty.\n\n3. The continuation of LCB’s time-based contamination control adds credibility, and the planned release of code and conversion scripts will make this a useful resource for the code-generation community.\n\n4. Figures and tables are easy to interpret, and the comparison with the original Python subset (differences within a few points) supports reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DnxZB0Brbs", "reviewer": "Reviewer_52p9", "summary": "This project extends the LiveCodeBench (LCB) benchmark to support multiple programming languages. Specifically, the authors translated the original Python-based problems into 11 additional languages, resulting in a total of 12 supported languages. They also developed an automated mechanism to track and incorporate future LCB updates, ensuring long-term maintainability of the benchmark. Using the new Multi-LCB dataset, the authors conducted a comprehensive evaluation of 20 large language models (LLMs), revealing that most models exhibit substantially lower performance on languages other than Python due to overfitting, and uneven distribution of pertaining corpora in terms of programming language diversity. The authors publicly release their benchmark extension, including prompts, source code, and configurations under MIT license to facility reproduction and future research.", "questions": "1. My understanding is that LCB releases are non-overlapping and cumulative, each representing a temporal evaluation slice. Did you convert tasks from all releases or only from the latest one? The paper notes that your pipeline supports the conversion of multiple versions, but it is not explicit whether all versions were actually converted.\n2. Lines 205–215 mention adopting a zero-shot strategy, yet Step 2 refers to the inclusion of illustrative examples. Could you clarify how these examples fit within a zero-shot setup?\n3. Please elaborate on the infrastructure or practical constraints that prevented inclusion of the Swift programming language.\n4. I recommend adding evaluations for GPT-OSS as well as leading closed-source models to strengthen the study’s benchmarking scope.\n5. In your view, which factor contributes more to improving cross-language performance — multi-language training or enhanced reasoning capabilities?\n6. Please explain how the evaluation framework supports long-term maintainability through automated update tracking.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The work extends a state-of-the-art (SOTA) benchmark while preserving the core strengths of the original LiveCodeBench. The design also emphasizes long-term maintainability through automated update tracking.\n* The evaluation framework is relevant for practitioners, addressing key technical challenges such as automatically converting code problems and hidden test cases into the appropriate format, as well as supporting sandboxed execution.\n* The paper provides valuable insights into benchmark contamination by using release date metadata to perform experiments, highlighting an important issue in LLM evaluation.\n* The evaluation includes a sanity check by comparing Python results on both the original LCB and the proposed Multi-LCB, demonstrating methodological rigor.", "weaknesses": "* The work does not introduce new task types. Since certain programming languages have inherent advantages for specific problems, adding novel or language-agnostic tasks could have strengthened the contribution’s originality.\n* While the overall presentation is solid, there are opportunities to enhance transparency in specific methodological aspects (see Questions section).\n* It is unclear whether the proposed benchmark covers all existing LCB releases or only the latest one.\n* The benchmark appears less challenging for LLMs on widely used programming languages — strong performance is reported for C++, Java, PHP, C#, and JavaScript.\n* The evaluation omits GPT-OSS and leading proprietary models, limiting the completeness of the comparative analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X4H2gb6EnW", "reviewer": "Reviewer_EauN", "summary": "This paper extends LiveCodeBench (LCB) from Python to twelve programming languages and proposes Multi-LCB, a contamination-aware, continuously updating benchmark for multilingual code generation.\nThe authors convert LCB tasks into a unified STDIN/STDOUT evaluation pipeline and evaluate 20 recent LLMs.\nTheir findings include (i) Python overfitting; (ii) language-specific contamination signals via post-cutoff time slicing; and (iii) cross-language performance disparities.\nThe paper argues that Multi-LCB preserves LCB’s contamination controls while enabling side-by-side, same-task comparisons across languages.", "questions": "1.  This paper seems to only measure LLMs' capability to use different programming languages ​​to solve the same algorithm problem, which may not comprehensively reflect  LLMs' capability to use syntax and features unique to other languages, thereby providing a one-sided measurement.\nPlease discuss and clarify the contribution of this paper and what specific research questions this paper wants to answer.\n2. How does the author filter or process these samples related to the unique features of the programming language? What are their effects on the assessment of models' capability on different PLs in your experiments?\n3. Besides the release-date filtering, what contamination detection methods are used on the converted samples in different languages?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Meaningful benchmark**, which can be directly used to evaluate and compare LLMs' code generation capability across different PLs \n2. **Large-scale experiments with insightful findings**. Cover diverse programming languages and evaluate 20 LLMs, providing findings to reveal real multilingual gaps in coding tasks.\n3. **Additional results and Open-sourced code**. Provide additional results and implementation details in the appendix and also release the scripts and datasets in an anonymous repository.", "weaknesses": "1. **Limited Measurement of the Generalization on Different PL**. This paper aims to answer the question of whether LLMs can generalize across diverse programming languages (in abstract).\nWhen evaluating a model's capability in a specific language, it's crucial to assess its capability in utilizing that language's unique features and built-in libraries  (e.g., Rust ownership/lifetimes, Go concurrency primitives, JS/TS async patterns, use of standard libraries). However, since all tasks were converted from Python tasks, one concern is that these tasks may be language-specific and thus fail to effectively measure the model's performance in using syntax and features related to other languages. Consequently, it cannot effectively measure the LLMs' capability in using different programming languages. This may limit the contribution of this benchmark.\n2. **Insufficient handling of the differences between the syntax/features of different languages**. The outputs of certain tasks may depend on language-specific syntax/features (e.g., the measurement of Unicode length is different in Python, JS, and TS (code points VS code units), and modulo with negative inputs may have different results across languages). If such samples are directly translated from a Python dataset, the evaluation may unfairly penalize other languages and understate LLMs' competence on them.\n3. **Contamination detection across languages**. Beyond filtering by the release date of the original Python problem, the paper does not provide other contamination detection methods. One of my concerns is that the rewritten/converted samples may have textual or semantical overlap with samples in the prior dataset, leading to data contamination and inflating scores of certain models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)