CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs
Authors: Dung Manh Nguyen
Thang Chau Phan
Nam Le Hai
Tien-Thong Doan
Nam V. Nguyen
Quang Pham
Nghi D. Q. Bui
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
CodeLLM
Benchmark
Evaluate
Code Comprehension
Primary area: datasets and benchmarks

Abstract: Recent advances in Code Large Language Models (CodeLLMs) have primarily focused on open-ended code generation, often overlooking the crucial aspect of code understanding & reasoning. To bridge this gap, we introduce CodeMMLU, a comprehensive multiple-choice benchmark designed to evaluate the depth of software and code comprehension in LLMs. CodeMMLU includes nearly 20,000 questions spanning diverse domains, including code analysis, defect detection, and software engineering principles across multiple programming languages. Unlike traditional benchmarks that emphasize code generation, CodeMMLU assesses a model’s ability to reason about programs across a wide-range of tasks such as code repair, execution reasoning, and fill-in-the-blank challenges. Our extensive evaluation reveals that even state-of-the-art models struggle with CodeMMLU, highlighting significant gaps in comprehension beyond generation. By emphasizing the essential connection between code understanding and effective AI-assisted development, CodeMMLU provides a critical resource for advancing more reliable and capable coding assistants.

Published as a conference paper at ICLR 2025
CODEMMLU: A MULTI-TASK BEN FOR ASSESSING CODE UNDERSTANDING & REASONING CA- PABILITIES OF CODELLMS
Dung Nguyen Manh 1 Thang Phan Chau', Nam Le Hair, Thong T. Doan', Nam Nguyen', Quang Pham 91 Nghi D. Q. Bui
'FPT Software AI Center, Viet Nam.
'Hanoi University of Science and Technology,
Independent Researcher
ABSTRACT
Recent advances in Code Large Language Models (CodeLLMs) have primarily focused on open-ended code generation, often overlooking the crucial aspect of code understanding & reasoning. To bridge this gap, we introduce CodeMMLU, a comprehensive multiple-choice benchmark designed to evaluate the depth of software and code comprehension in LLMs. CodeMMILU includes nearly 20,000 questions spanning diverse domains, including code analysis, defect detection, and software engineering principles across multiple programming languages. Unlike traditional benchmarks that emphasize code generation, CodeMMLU assesses a model's ability to reason about programs across a wide-range of tasks such code repair, execution reasoning, and fill-in-the- blank challenges. Our extensive evaluation reveals that even state-of-the-arr models struggle with CodeMMLU, highlighting significant gaps in comprehension beyond generation. By emphasizing the essential connection between code understanding and effective Al-assisted development, CodeMMLU provides critical resource for advancing more reliable and capable coding assistants. CodeMMLU publicly available CodeMMLU
INTRODUCTION
Recent advancements in Code Large Language Models (CodeLLMs) (Wang et al.
2021
2023c
Feng et al. 2020 Allal et al. 2023: Li et al.] 2023; Lozhkov et al.
2024b; Guo et al. 2024b
Pinnaparaju et al. 2024 Zheng al. 2024b| Roziere et al.) 2023
Nijkamp al. 2022: Luo et al.
2023 Xu et al. 2022 Bui et al. 2023 Hui et al. 2024 Bui et al.
2022; Dau et al. 2024b a) have
demonstrated impressive capabilities across various software engineering
(SE) tasks (Bui et al. 2022
Phan et al. 2024 Sun et al. 2023 Nguyen al. 2022 Bui 2019 Zhang al. 2022 Wang al. 2023b). However, existing
Dau et al. 2024a To et al. 2023 White et al. 2024 Sobania et al., 2023
benchmarks often fall short to provide rigorous and reliable evaluations, largely due to outdated
methodologies and the risk of data leakage (Matton et al. (024). Moreover, practical applications
of CodeLLMs reveal limitations such as bias and hallucination (Rahman & Kundu 2024
Liu et al.
2024a) that current benchmarks fail adequately address.
The predominant focus of coding-related benchmarks has been on open-ended, free-form generation
tasks, such as code generation/code completion (Tyer et al. 2018 [u et al. 2021 Chen et al.
2021
Austin et al.) 2021 Lai et al.) 2023 Hendrycks et al. 2021 Ding et al. 2023 Zhuo et al.
2024
and other SE tasks like program repair Ouyang et al. (2024); Xia et al. (2023) (Table [). While
appealing, these benchmarks struggle to discern whether CodeLLMs truly understand code merely
reproduce memorized training data (Carlini et al. 2022 Nasr et al., 2023). Additionally, the reliance
on test cases and executability for evaluation limits the quantity and diversity of these benchmarks across domains, potentially leading to biased and limited generalizations. Recent efforts to improve
"Corresponding authors: Dung Nguyen Manh (dungnm31 0 fftt... com) and Nghi Q. Bui
mannghi@gmai.... The author contributed to this work while working with FPT Software AT Center.

---

Published as a conference paper at ICLR 2025
GPT 4e
Uamm333 708 Inst
QwenCoder233 320 Inst
Cloude3.7 Sonnet
DeekSeek
Programming syntax
API &
Execution prediction
Frameworks
tO
Software principles
DUMS
Code
SQL
repair
Others
Code completion
Figure Summary performance of LLMss on the CodeMMLLU benchmark. This radar chart presents the evaluation results (accuracy %) of different models across various CodeMMLL tasks.
evaluation through free-form question answering (Liu & Wan 2021 Li et al. 2024) have introduced new challenges, often requiring less rigorous metries LLM aa--as-aa a--dee approaches (Zheng et al. 2023). However, LLMMs-as---judee methods are susceptible to adversarial attacks (Raina et al. 2024 raising concerns about the reliability of such evaluation pipelines for coding tasks.
To address the aforemenntoned shortcomings, we introduce CodeMMLU, novel benchmark designed evaluate CodeLLMMs ability comprehend and reason about code through multiple-choice question answering (MCQ). This approach enables a deeper assessment of how CodeLLMM grasp coding concepts, moving beyond the mere generation capabilities. Inspired by the MMLU dataset (Hendrycks et al.) 2020) from natural language understanding, CodeMMLÜ offers robust and easy evaluation with the following key features. CodeMMLU comprises nearly 20,000 questions, facilitating robust and comprehensive evaluation. Its large-scale data curation process mitigates potential biases and improves statistical reliability in measuring CodeLLMs' performance across many capabilities. CodeMMLU covers over 50 software engineering disciplines and more than 10 programming languages, providing a holistic evaluation CodeLLMs. The MCQ format highly scalable, allowing for accurate and straightforward assessment using precision basedd metrics. Moreover, by incorporating permutations answer choices, CodeMMLU emphasizes the model's 's code understanding capabilities rather than memorizing the training datasets. Consequently, CodeMMLU provides robust and accurate assessment of the models capabilities to understand software tasks.
CodeMMLU evaluates LLMs' abilities in coding and software problem-solving from fresh perspec- tive, extending beyond conventional code generation and completion tasks. Our analysis uncovers several key insights: (1) previously unidentified bias issues in CodeLLMs, consistent with those observed in natural language MCQA tasks; (2) proprictary models (GPT-4o and Claude 3.5 Sonnet) consistently deliver the highest average performance; (3) among open-source models, the DeepSeek and Meta-Ilama families achieve the greatest accuracy; (4) scaling laws tied to model size hold partially within the same model family but not across families, highlighting the critical roles of pre-training post-training, and model architecture; (5) advanced prompting strategies, such as Chain-of-Thoughh (CoT), consistently impair performance, casting doubt on CodeLLMs' reason- ing capabilities for complex, multi-step tasks; and (6) when code completion benchmarks (e.g., HumanEval) are reframed from open ended generation to MCQA format, LLMs exhibit reduced performance, questioning their true understanding of code. Notably, even strong reasoning models like DeepSeek-R1 underperform Code suggesting significant room for improvement in
2

---

Published as a conference paper at ICLR 2025
future iterations. These findings highlight the shortcomings of CodeLLMs in truly understanding code and reveal several promising future research avenues.
In summary, this work makes the following contributions:
1. We present the first MCQ benchmark for software and coding-related tasks, addressing the need for a comprehensive and large-scale evaluation in the code domain. CodeMMLU enables the evaluation of LLMs' alignment with human inference in the software knowledge domain, similar to advances the NLP field.
2. CoUeMMLU provides a thorough assessment of LLM capabilities, ensuring a substantial number of samples and the diversity across tasks, domains, and languages. This enables understanding of an LLM's strengths and weaknesses, facilitating the development of models better aligned with the complexities and demands of the software domain.
nuanced
Our experiments offer critical insights into LLM performance, highlighting the impact of factors such as model size, model family, and prompting techniques. Notably, CodeMMLU unveils a performance gap between LLMs' code generation and comprehension abilities. Additionally, we identify LLMss sensitivity to the selection biases introduced by the MCQ format. These findings provide valuable guidance for the community to enhance the robustness, adaptability, reliability, and domain-speciff capabilities of LLMs in real-world software engineering.
RELATED WORK
Code Intelligence Benchmarks. The rapid development of Large Language Models (LLMs) for code-related tasks necessitates the development of diverse benchmarks to evaluate their performance. Eegorithm-fouusee benchmarks, such as HumanEval (Chen et al., (02T), MBPP (Austin et al.) 2021], and their extended versions (e.g., HumanEval+, MultiPL, MBPP+) (Liu et al., 2024b), focus on small-scale code generation tasks but lack the depth needed to assess broader comprehension. More challenging tasks, such as those in CodeContests (Li et al. 2022) and LiveCodeBench (Jain eet al.] 2024), provide competitive programming problems but remain primarily generative (Table 1p. Comprehensive evaluation frameworks, such CodeXGLUE (Lu et al., 2021), ) XLCoST (Zhu jet al. 2022), and XCodeEval (Khan et al. (203),, provide versatility through multi-task assessments. However, these benchmarks are either dependent metrics like BLEU and ROUGE, or testcase execution, which limits their reliability and scalability for a large-scale, comprehensive evaluation of LLM.
In contrast, multiple-choice question (MCQ) benchmarks offer a more standardized, scalable, and reliable evaluation method, as demonstrated in popular general purposs benchmarks such as MMLU (Hendrycks et al. (020) and TruthfulQA (Lin et al. (022). Although MCQs facilitate scale assessments, recent studies highlight their susceptibility to biases, such as sensitivity to the answer choice orders (Wang et al.) 2023a; Robinson et al.] 2023). Existing MCQ benchmarks also lack focus on software engineering, limiting their applicability to code-related evaluations. contrast, we curated data from a wide range of tasks and applied various filtering and debiasing techniques to improve CodeMMLU's comprehensiveness, while minimizing data leakage and biases.
Understanding & Reasoning on Code There a large body of research leveraging Al models for reasoning about code Gu et al. (2024a); Chen et al. (2024); Liu & Jabbarvand (2025); Dehghan (2024); Le et al. (2024b a): Bieber et al. (2020); Shi et al. (2019) 9); Li et al. (2021). Reasoning about code requires AI models to comprehend both its syntactic and semantic aspects. In the early days, static analysis was the primary approach to reasoning about code. With the rise of deep learning, methods that model code structures Mou et al. (2016): Subsequently, graph-based representations of code have gained traction, enabling reasoning about program properties and execution using graph neural networks (GNNs) Le al. (2024a); BBeber et al.) (2020); Shi et al. (2019). With the advent of large language models (LLMs), these models have been directly applied to reasoning about program behavior in downstream tasks Gu et al. (2024a); Chen et al. (2024); [iu & Jabbarvand (2025): Dehghan (202): : Le et al. (2024b). CodeMMLU provides a comprehensive benchmark that reflects real-world reasoning requirements by offering ground answers to engraamiin--elateer questions, allowing LLMS to derive final solutions through various reasoning paths.
al. (2021b a) emerged promising alternatives.
3

---

Published as a conference paper at ICLR 2025
Table Comparison between common code understanding benchmarks for LLMs in terms of coverage of foundation tasks of programming comprehension model.
Question
Programming Task
Benchmark
Open-end
MCQ
SWE knowd edge
Code composi- ton
Code debug- ging
Test size
compre- hension
APPS APPS Hendrycks etall all
2021
5000
MBPMAUSSE el 20:1
974
HumanEva/ al 2021
CRUXEval Du 2024B
SQU
LiveCodellennhheam Jan
2024
S
CodeApes Fu
2.056
CodeMMLU
v
19.912
CODEMMLU: DATA CURATION
The CodeMMLU benchmark is structured into two primary categories: (i) knowledge-bel based tests, designed to evaluate programming knowledge through questions addressing both syntactic and semantic aspects, and (ii) fundamental coding-skill tests, created by transforming high-quality codebase seeds into speciip challenges. CodeMMLU includes nearly 20,000 questions spanning 52 diverse topics (Table 2). We design the knowledge-base baseee-bbdddd tests to probe multi-level cognitive structures, assessing L.M's understanding of software knowledge at both semantic and syntactic levels. In contrast, the fundamental coding-skill test sets align with the cognitive process model of Shneiderman & Mayer 1979), focusing on core programming tasks that mimic real-world problem- solving scenarios.
3.1 KNOWLEDGE-BASSE BASED TASK CREATION
The knowledge-based test sets are designed to cover a wide range of topics and follow the multi-level cognitive structures model (Shneiderman & Mayer 1979) which combines syntactic and semantic knowledge. The subset target is to measure the LLM'S coding capability and comprehensibleness of programming concepts. We collected raw wewaaammminn----eeeeeee questions and their corresponding multiple-cooiess answer from W3School (W3Schools) 2024) and Common Crawl project more license detail Appendix A.3 The knowledge-basse test set include:
Syntactic subset. Focused on programming language grammar and structural correctness, such as condition statement, format of iteration, common library usage.
Semantic subset. Targeted more abstract programming concepts, such as algorithms, data struc- tures, object-oriented principles.
We maintain high-quality evaluation set by filtering the raw data that undergoes rigorous formatting filter which we remove any instances that do not meet our quality criteria
and deep-learning- (see in section 3.3 and Appendix I.I). Resulting in evaluation set (Table that contains more than 11 1,000 instances, lying in 52 topics classified to 5 main subjects (categorized by source tag).
3.2 FUNDAMENTAL TEST CONSTRUCTION
Our benchmark encompasses four distinct MCQ programming tasks designed to assess the founda- tional capabilities outlined in the cognitive process model of programmer comprehension, namely: composition, comprehension, debugging, and modification.
Code Completion evaluates a model's composition ability by requiring it to complete partially written code based provided requirements. We adapted HumanEval (Chen al. 2021), originally designed for code generation, into an MCQ format. From its 164 unique programming problems, we employed Large Language Models (LLMs) to generate plausible but incorrect solutions as distractors. All options, including correct solutions migrated from HvaanEval and generated incorrect ones, were tested for executability. Some incorrect solutions designed to pass certain test cases but fail
https://gituub.com/l.vve Litpa////oooooooeeoeeeeeeeeooeoe hhtps://commoncrawlorrr/
4

---

Published as a conference paper at ICLR 2025
Table 2: Summary of CodeMMLU Subject Categories and Task Distribution.
Subject
Topic
Source
Testsize
Jquery. Django, Pandas, Numpy. Scipy,
Syntactic knowledge
API & Frameworks usage
Azure, Git, AWS, svg, xml,
740
Bootstrap. NodeIS, AngulardS, React, Vue.
C. C#. C++, Java, Javascript, PHP, Python,
Programming language syntax
R. Ruby, Marl .ab, HTMI,, CSS, TypeScript.
W3Schools, Geeksforgedl Common Crawl
6,220
DBMS & SQL
DBMS, MySQL, PostgreSQL, SQL.
393
Semantic knoowledge
Data structure & Algorithm.
Object-oriented programming.
Compiler design,
Software principles
3,246
Computer organization and Architecture, Software Development Engineering.
System Design.
Program accessibility, Computer networks,
Others
Computer science, Cyhersecurity, Linux, Web technologies, AWS.
1,308
Fundamen -tal task
Code completion
HumanEval
163
Fill in the blank
LeetCode
2,129
Code repair
QuixBugs
76
Execution Prediction
IBM CodeNet
6,006
others, adding complexity and challenging models to distinguish between correct and nearly-correct solutions based on semantic and syntactic understanding.
Code Repair assesses model's debugging capability by requiring it to identify and fix errors in provided code snippets. We built this task upon QuixBugs (Lin et 2017), which was originally designed for debugging algorithmic programs. We used a "diff" operation on buggy and corrected versions in QuixBugs (Python and Java) to identify specific fixes, which served correct solutions. To create plausible distractors, we targeted components frequently involved in bugs (e.g., return statements, loop conditions, if/else/switch expressions) and guided L.LMs to generate alternative fixes. These alternatives were designed to seem plausible but not fully resolve the bug. Each distractor was verified for incorrectness, and all options were made executable to ensure that models needed a deep understanding of the code to identify and apply the correct fix.
Execution Prediction evaluates a model's s ability to identify and understand defects within code snippets, focusing on both logical and syntactical errors. This task measures the comprehension and debugging capabilities of LLMs by requiring them to predict the execution outcome of given code. It includes sub-tasks: detecting any defects/flaws in the provided code and comprehending the output of a certain test sample. We derived this task set from IBM CodeNet (Puri et al.] 2021). large-scale benchmark for algorithmic coding tasks. We focused Python and Java subsets, collecting both accepted and buggy versions of code. After filtering out duplicates, we created diverse set of code samples. For each snippet, we provide the correct execution result (golden answer) and three distracting options, which could be one of several possible outcomes: (i) Compile Error, (ii) Time Limit Exceeded, (iii) Memory Limit Exceeded, (iv) Runtime Error, or (v) No abnormally found.
Fill in the Blank evaluates a model's code comprehension ability by requiring it to complete missing parts of code snippet, given documentation and an incomplete code sample. This task assesses not only the model's ability to fill gaps but also its understanding of both high-level programming concepts and low-level grammatical structures. We collected approximately 2,000 coding problems from LeetCode I covering solutions in three widely-used programming languages (Python, Java, C++). From each problem's solution, we parsed and randomly selected key components (i.e. crucial elements of the program's 's logie and flow like loop conditions, expression statements, conditional statements) to be blanked out. To create plausible but incorrect options for the multiple-chote question (MCQ) format, components. These distractors were designed to be contextually relevant but incorrect, adding complexity to the task. We executed all generated options to verify their incorrectness, ensuring they do not solve the problem as intended.
employed LLM generate alternative solutions for the blanked-out
https///eetcooeecomm
5

---

Published as a conference paper at ICLR 2025
CodeMMAL
LiMs josgement follow detired
oragen es.
Coral sedion Qrick penplemeeeee
Gunley
ofiigh quality LUMs Har out aiticily questto.
Forkssu Wischool, utt
Question Golden anower
AA
(heasnol Gol sto)
Promppt confige
Colachy
Code
Testosses
IBM CodeNet, OutsBuge,
Language nodels
Figure 2: Overview of CodeMMLU data creation pipeline. The blue diagram describe the process of collecting raw multiple-choie questions (MCQs) from open source internet for knowledge testset. Otherwise, the pipeline real-world problem indicated in orange area.
3.3 DATA CLEANING
The preprocessing process (described in Figure 2 includes a deep learning-based filtering and execution-based filtering to ensure that each question met the desired quality standards, including clarity, lack ambiguity, and difficulty.
LLM-based Filtering To begin, we employed an LLM-basee filter assess the instances in the crawled knowledge test set. Each instance was evaluated based on three criteria: Completeness, Coherence and clarity, and Coding relevance. The models utilized for this evaluation included GPT-3.5, Llama3.1-8B Instruct, and Mixtral-8x7B Instruct. We averaged the scores and used them select filtering threshold for each criteria (see discussion in Appendix A.I). To detect and handle duplications, we applied the MinHash LSH algorithm Zhu et al. (2023), configured with 256 permutations, cluster duar-cul questions. We remove all false positive instances in each cluster with 0.8 as the similarity threshold. To verify the efficacy of the LLLM-based filter, we randomly selected 100 instances from each subject area for manual verification against the three criteria.
Execution-basee Filtering To ensure the question correctness, we apply an execution based filtering in the fundamental test sets. We merge the distractor of (i) code completion, (ii) fill-in- the-blank, and (iii) code repair with their codehase and execute with their corresponding test cases. The distractor is designed bring challenge since it requires L.LM comprehend their correctness without executing it, we select distractors that are executable with to few (less than 50%) test cases passed in their execution result. In the other hand, the task Execution Prediction's groundtruth collected from executing process, the distractor are randomly pick from common executing scenarios.
4 EXPERIMENTAL RESULTS
4.1 SETUP
Model selection. We evaluate CodeMMLU on 40 popular open-source LLMs, covering wide range of parameter sizes and architectures. The models were selected from 13 different families, with parameters ranging from 1 billion to over 70 billion. Each family included base and instructed/chat versions. In addition to open-source models, we also included several proprietary models from OpenAl and Anthropic to ensure a comprehensive coverage of the state-of-the-arr in language modeling. All model information can be found at C.
Answer extraction. CodeMMLU leverages the MCQ format for scalability and ease of evaluation. In order to maintain this advantage, only apply simple regex methods to extract the selection answer (i.c.. extract by directly answering (AIBICID) or containing the pattern "answer is AIBICID"). The model response is required to be parsable; otherwise, it will be marked unanswered.

---

Published as a conference paper at ICLR 2025
Random score
Zero
Few
60
40
20
PL syntax
API & Fr Frameworks
DBMS & SQL
Software principles
Others
Code Completion
Fill in the middle
Code Repair
Execution Prediction
Figure 3: Comparison of prompt configuration on GPT-40. The experiment exposes the drawback of Chain-of- -hain-of-Thhough prompting technique in term of boosting performance on task that not require logic or reasoning.
In the following, we present key findings of CodeMMLU on (i) knowledge and fundamental test correlation; (ii) MCQ bias evidance; (iii) Disagreement between code-generation alike benchmark and MCQ format. Due to space constraints, we provide detail experimental results, additional discussions and analyses in the appendix, including assessing data leakage (Appendix [A.2;; MCQs analysis (Appendix B.I. Chain-of-the of-thought technique analysis B.2 and full 43 LLMs results B.3
4.2
KEY INSIGHTS
Overall performance CodeMMLU revealed significant performance differences across models, as shown in Table OpenAl's GPT-4o outperformed all models on CodeMMLU, demonstrating its quality across diverse tasks (Fi Notably, despite not being the latest model, the instructed version of Llama3.1 I 70B from Meta achieved the highest score among open-source models from 13 families. While LLMs perform well on knowledge-basee tasks, they struggle with real-world problems, particularly in execution prediction tasks (see Appendix B.3 for all the experiment details.)
Figure 4 illlusrrates CodeMMLU's capability to mcasure LLMs' coding knowledge and skills across a wide range of subjects. Our benchmark provides clear, distinct rankings that establish a higher hierarchy of models compared to other benchmarks (see the result in Table 7 Interestingly, the results do not strictly adhere to scaling laws (Kaplan et al. 2020), where larger parameter sizes typically outperform smaller ones. This highlights the impact of data quality in the LLM pretraining process, as recently released models often achieve comparable performance to larger models from previous versions. CodeMMLU also indicates the importance of instruction tuning in improving model performance on complex tasks. Models with instruction tuning substantially outperform their non con-tisttetted counterparts, for example, DeepSeek-Conn Cooeekk--oodr333 surpasses its base model by approximately 29%.
The impact of model reasoning and ofain-h (CoT) prompting Although CoT prompt- ing (Wei al., 2023) is often expected enhance performance by eliciting deeper reasoning, our ex- periments reveal that CoT and reasoning models may not always offer improvements in CodeMMLUU, TableB]show that DeepSeek R1 performs significantly worse than its base model, DeepSeek V3, despite being designed for reasoning tasks. Meanwhile, GPT 03-mini, native reasoning model, achieves the best results. Our analysis suggests that, apart from GPT o3-mini, other reasoning models (e.g, GPT o1, DeepSeek R1) tend to overreason across all CodeMMLU tasks. In contrast, 03-mini demonstrates the ability to decide when to apply reasoning versus directly answering knowledge- seeking tasks, such in syntactic and semantic evaluations (Figure II 12 These findings align with Chen et al. (2025), which observes that current reasoning models often overreason even on simple questions. This suggests the need for more effective decision Docision-making mechanism to determine when reasoning is beneficial. Table further highlights that while RI generates the longest responses (in tokens), it underperforms compared to the latest top-tier I. L.LMs.
7

---

Published as a conference paper at ICLR 2025
We further investigate the impact of different prompting techniques and report the results in Figure 3] The results show a significant decline in GPT-4o's performance with CoT, suggesting that the additional complexity introduced by step-by-step reasoning does not align well with knowledge- seeking tasks (see Appendix [.2 for more discussions). contrast, few-shot prompting consistently emerges the most reliable and effective strategy across various tasks, offering balanced approach without overwhelming the models. Overall, with the exception of GPT o3-mini, we found no improvements from reasoning models CoT prompting, suggesting that CodeMMLU presents challenging benchmark to test the models reasoning capabilities in code domains.
Table 3: Summary of LLM Family Performance CodeMMLU The evaluation results (accuracy %) of different language models across the CodeMMLU task (CodeMMLU column represents the accuracy average among all subject).
Knowledge test
Family
Model name
Size (B)
Syntactic
Semuntic
Fundamental test
CodeMMLU
Closed-source models
Claude
Claude3.7 Sonnet
52.78
76.26
60.90 92
61.65
Claude3.5 Sonnet
52.23
73.45
58.56
59.81
Claude3.5 Haiku
49.24
68.20
57.83
57.25
Claude3 Sonnet
67.22
66.08
38.26
53.97
GPT 03-mini GPT 4o
$3.08
75.50
62.77
62.36
50.63
69.61
53.89
56.40
GPT
GPT 4o-mini
48 66
55.90
20.33
38.43
GPT-3.5-turbo
61.68
53.65
45.26
51.70
Open-source models
Llama3.3 3,
70
44.31
52.76
30.96
40.66
Llama3. 40SB
405
50.82
71.41
57.10
58,23
Llama
Llama3 70B Inst
70
46.94
62.64
53.15
53.19
CodeLlama34B Inst
34
56.81
46.93
23.55
38.73
DeepSeek
671
42.39
$6.77
38.08
43.91
DeepSeck
DeepSeck
685
48.30
$7.53
45.06
49.08
DeepSeekCoder 33B Inst
33
53.65
45.43
21.46
36.60
DeepSeckMoti 16B Chat
16.4
31.74
35.42
27.32
31.01
Mistral7B Inst (v0.3)
7
54.42
$1.25
31.85
43.33
Mistral
Mixtral 8x7B Inst
46.7
61.17
$4.89
24.09
42.96
Codestral 22B
22
60.34
52.10
37.85
47.60
Phi
Phi4
14
45.34
$7.46
47.82
49.19
Phi4 Mini Inst
12
41.94
51.59
19.75
34.85
Qwen2.5 14B Inst
14
46.38
58.70
51.49
51.38
QwQ 32B Preview
57
61.34
57.48
30.48
46.34
Qwen
QwenCoder2.5 32B Inst
32
50.63
69.61
53.89
56.40
Programming Language Syntax API Famewerkk
Software Priccples
Detabao: managemente & SQL Others
Code Cremp Completion
Code
Prediction
Staroderr indt
11.5 1et Clat
Conpdrent Drokkbant AJ Mad leb Crop Iovatenn Craplinns Hat 18e clus
Opeenn Covernaater 2 a aee
Owenn 570 but
CerQ Peol Nendium test 220
Mistral tust we Mistral 0:70 tnit Cottentraa 226
Unma 100 most. Uarra'l atnta toat. Corlo Corted torn 1'th teat.
Unama Uarra333 70B
GFT 23.3Tur ArTuron
GPT Germmreet Alrmmn
GPT
GPT chvesiee
Cloudd Stnvett
Casudeo Haldu Clastich stomet
Cleccie3 Sarove
nut
Figure 4: CodeMMLU accuracy by task on LLMs. While knowledge tasks are following the scaling law, real-world tasks offer more challenges to LLMs which indicate the performance of instruction tuning and data quality when evaluating on CodeMMLU.
8

---

Published as a conference paper at ICLR 2025
Correlation Between Software Knowledge and Real- World Performance Our experiments revealed a strong correlation between performance on knowledge- based tasks and real-world coding challenges. Specifically, the Pearson correlation score 0.61 between model rankings the knowledge test set and their performance on real-world problems, derived from the accuracy of 43 LLMSs across 15 model families, indicates moderate alignment (Figure This suggests that models demonstrating a deeper understanding of software principles consistendy excel in real-world coding tasks, highlighting the importance of foundational knowledge for practical coding performance.
indanora
15
Roal
20
7.0
40 Knowledge-Task
Figure 5: Correlation between knowl- edge tests and fundamental skill tests. Experiments 10 LLM families show a clear alignment between models with strong understanding of software knowledge and their performance on diverse problem-solving tasks in the CodeMMILU fundamental skill tests.
Figure Comparison Between LLM Response Length and CodeMMLU Score. While reasoning models (e.g., DeepSeek-R1) generate longer, reasoning- heavy responses, this does not necessarily correlate with higher accuracy on CodeMMLU.
Selection bias in MCQs format We experimented with multiple answer order permutations (follow Zheng et al. (2024a)), the result displayed significant inconsistent behavior exhibited by LLMs when swapping golden answer positions. As presented in Table 7 the model's performance changes
dramatically in each answer order configuration, which is based on the correct answer's position. The
LLM's accuracy fluctuates between different permutations (i.e. DeepSeek -o0beee3-o Ao 36.66), demonstrating how sensitive it can be the structure and order of answers (Figure However, Table (alleBiindie indiicate the different of MCQ bias between strong models (e.g GPT-4o, Claude and others, which highlight the consistency and robustness among them (sce discussion B.I
Disagreement between Open-ended generation benchmark and MCQ Code completion A notable finding from our experiments is the discrepancy in model performance between open-ended benchmarks and multiple-choie formats. Specifically, when comparing the original HumanEval questions with their multiple-choice equivalents in CodeMMLU code completion set, found that models performing well on HumanEval do not consistently replicate their success in CodeMMLU, For instance, when evaluating identical questions across the formats, the number of cases where models answered both correctly or incorrectly was unexpectedly low. The correlation scores in Figure 7 further illustrate the weak alignment of success between these two benchmarks, revealing that performance in open-ended tasks does not reliably predict performance in multiple-choice coding tasks. This lack of alignment suggests that traditional benchmarks might overestimate a model's understanding by focusing too narrowly on code generation, which is highly susceptible to data leakages. In contrast, CodeMMLU requires the models to engage in complex reasoning to understand code and solve software engineering problems.
CONCLUSIONS
In this work, we introduced CodeMMLU, a comprehensive and scalable benchmark designed to evaluate large language models' (LLMs) capabilities across a wide range of software knowledge

---

Published as a conference paper at ICLR 2025
Table 4: Performance Comparison between HumanEval and MCQ Code Completion Tasks. The performance fluctuation highlights the selection biases observed when the correct (golden) answer is moved to positions A, B, C. or
Code Completion MCQ
Models
HumanEval
A
B
C
D
0.00
90.24
14.02
0.61
CodeLlama-7B-PPP
40.48
(-40.48)
(+49.76)
(-26.46)
(:39,87)
Codel.lama
3,66
1.22
93.90
15.85
(41.999)
(-44.43)
(+48.25)
(-29.80)
ModeLllma-13B-PPPae
0.61
54.88
70.12
12.20
(-12.28)
(+11,99)
(427)23)
(-30.69)
2,44
68.29
72.56
29.88
ColeLLamaa-13B-Isttee
50.6
(-18.16)
(+17.69)
(+21,96)
(-20.72)
CodeLlama-344--Pldaa
0.61
77.44
70.73
49,39
(-4.500)
(+32.33)
(+25.62)
9.15
84.76
65.24
46,34
CodeLlama-334--Intte
50.79
(41..64)
(+33.97)
(+14.45)
(-4.45)
40.85
74.39
64,02
39,02
Deepseek-Codee Baanck--oder-7B--baaee 7B-base-1.5
43.2
(-2.35)
(+31.19)
(+20,82)
(-4118)
1,22
82.32
75.00
56,10
SeepSeek-Coder-225
(-54.88)
(+26.22)
(+18.90)
(0.00)
CodelLLama
6.10
90.85
75.00
46.34
(-65.85)
(+18.90)
(+3.05)
(-25.61)
22.56
74.39
71.95
63.41
Bixtral-8x7B-Innstiee
40.2
(-17.64)
(+34.19 19)
(+31.75)
(+23:21)
CodeOwend: 3-78 foor (10011211
Febs
Tua
Tyas
56
Huma
Humart
Huno
14
-00
campletes
Caree comp edor
Code completion
Owen2 7B intruct (ome33f-0013833
starcoder2-16 korrooof=0.1350)
33
14
completion
Figure 7: Comparison of CodeMMLU's code completion task and HumanEval. Many LLMs show a performance discrepancy between the two tasks, where models that successfully passed the HumanEval code generation test often failed to select the correct answer in the multiple-choice (MCQ) format, or vice versa, for the same question.
and real-world programming tasks. Our experiments highlighted the benchmark's key advantages, including cost-effectiveeness, scalability, and extensive task coverage. The insights gained revealed strong correlation between software knowledge and real-world task performance, demonstrating that models with deeper comprehension outperform those relying purely on probabilistic generation.
Additionally, CodeMMLU provides more accurate and detailed rankings of LLMs, particularly in open-source models, where significant reordering of performance was observed. The benchmark also revealed inconsistencies in model comprehension when compared to traditional evaluations like HumanEval, emphasizing the need for more robust benchmarks that go beyond simple code generation.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang. Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024. URL https: /arpiv.rrx..oorr///o.
abs/2403 04652
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.039000 2023.
Anthropic. The claude model family: Opus, sonnet, haiku. 2024. URL https://agt semanticscholarr org/ wttp://oooooooopo0002222222
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint htt::2108.077222 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
David Bieber, Charles Sutton, Hugo Larochelle, and Daniel Tarlow. Learning to exccute programs with instruction pointer attention graph neural networks. Advances in Neural Information Process- ing Systems, 33:8626-8637, 2020.
Nghi Bui. Towards zero knowledge learning for cross language api mappings. In 2019 JEEF/ACM 41st International Conference on Software Engineering: Companion Proceedings (CCSE-Compant) pp. 123-125, IEEE, 2019.
Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. Infercode: Self-supervised learning of code represen- tations by predicting subtrees. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 1186-1197. IEEE, 2021a.
Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. Treecaps: Tree-based capsule networks for source code processing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 30-38, 2021b.
Nghi DQ Bui, Yue Wang, and Steven Hoi. Detect-localiie-rppai: A unified framework for learning to debug with codet5. arXiv preprint arXiv:221..1455 2022.
Nghi DQ Bui, Hung Le, Yue Wang, Junnan Li,, Akhilesh Deepak Gotmare, and Steven CH Hoi. Codetf: One-stop transformer library for state-of-the-ard code 11m. arXiv preprint S0ti::23360000222 2023.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge. Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo. Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang. Tao Jiang. Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong. Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma. Zerun Ma. Wenchang Ning. Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song. Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang. Guoteng Wang. Jiaqi Wang. Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei. Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie
11

---

Published as a conference paper at ICLR 2025
Zhang, Shuo Zhang, Songyang Zhang. Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024. URL
fttps::/arxiit org/ abs// abo/ 17091
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint
arXiv:2202.07666, 2022.
Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. Reasoning runtime behavior of a program with IIm: How far are we? arXiv preprint cs. CE.SE2203.. 16437, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint vrXiv:2107.03374 2021.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like llms, 2025. URL https?
7/arxiv. www.xi..rr/ab//22e// .tl.1.221111 21187
Anh Dau, Jin LC Guo, and Nghi Bui. Docchecker: Bootstrapping code large language model for detecting and resolving code-comment inconsistencies. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp. 187-194, 2024a.
Anh TV Dau, Hieu Trung Dao, Anh Tuan Nguyen, Hieu Trung Tran, Phong X Nguyen, and Nghi DQ Bui. Xmainframe: A large language model for mainframe modernization. arXiv preprint arXiv:2408. 04660, 2024b.
DeepSeek-AL, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang. Peiyi Wang. Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Dee Breaking the barrier of closed-source models in code intelligence, 2024. URL httttttpp:/aaatt org/ abs 2406 11931
Saman Dehghan. Assessing code reasoning in large language models: A literature review of benchmarks and future directions. 2024.
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Kr- ishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodee- val: A diverse and multilingual benchmark for cross-file code completion. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023. New Orleans, LA, USA, December 10 16, 2023, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang. Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak. Chloe Bi, Chris Marra, Chris McConnell, Christian Keller. Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Ject Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu
12

---

Published as a conference paper at ICLR 2025
Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Hantala-Yeary Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupulcti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi. Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet. Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani. Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Fisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai,, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Scide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu. Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg. Lavender A. Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li,
13

---

Published as a conference paper at ICLR 2025
Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhangg Sinong Wang, Sneha Agarwal, Soji Sajuyighe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vito,, Zef Rosnbrick, Zhaoduo Wen. Zhenyu Yang. and Zhiwei Zhao. The llama 3 herd of models, 2024. URL htttitttta:: org/ abs/ 2407 02470727777 21783
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08/55 2020.
Lingyue Fu. Huacan Chai, Shuang Luo, Kounianhua Du Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang. Yifan Liu, Jingkuan Wang. Siyuan Qi, Kangning Zhang, Weinan Zhang, and Yong Yu. Codeapex: A bilingual programming evaluation benchmark for large language models. ArXiv, abs/2339.00 019440, 2023. URL htlhhttp:///ppp statps:://pp..smmntiiicc iemaattccccooool..... icscholar
semant
Dtg//oorpusDD:261155
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama,, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.0306. 2024a.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezamaa, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution, 2024b. URL
https://arxiv. http:///axxxvv.r//oo.sssse wwg. httg//bs//40gg 03065
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo. Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence, 2024a. URL https://arxiv.
httgg/bb/24411..11
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Wu, YK Li. et al. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv: 14196, 2024b.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 03300, 2020.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1643-1652, 2018
Naman Jain, King Han, Alex Gu, Wen- -ing Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 20Xiv:2403.07974, 2024.
14

---

Published as a conference paper at ICLR 2025
Fred Jelinek, Robert L Mereer, Lalit R Bahl, and James K Baker. Perplexity a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63, 1977.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral experts, 2024. URL https: /ttp///axvvvv/v. /tg////ggaoooooooo /20..
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
abs /tabb//22000 2001 08361
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding. generation, translation and retrieval. arXiv preprint 2:0.iv2303303000 2023.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319-18345, PMLR, 2023.
Cuong Chi Le, Hoang Nhat Phan, Huy Nhat Phan, Tien N Nguyen, and Nghi DQ Bui. Learning to predict program execution by modeling dynamic dependency on code graphs. arXiv preprint arXiv:2408.028// 028/6, 2024a.
Cuong Chi Le, Hoang-Chai Chau Truong-Vinth Huy Nhat Phan, Dung Duy Le, Tien N Nguyen, and Nghi DQ Bui. Visualcoder: Guiding large language models in code execution with fine-grained multimodal chain-of-thought reasoning. arXiv preprint arXiv:2410.23402, 2024b.
Linyi Li, Shijie Geng, Zhenwen Li, Yibo He, Hao Yu, Ziyue Hua, Guanghan Ning, Siwei Wang, Tao Xie, and Hongxia Yang. Inficoder-eval: Systematically evaluating the question-answering capabilities of code large language models. arXiv preprini arXiv:2404.0777 04404404440 2024.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint 23Xiv:2305.06 2023.
Xuechen Li, Chris Maddison, and Daniel Tarlow. Learning to extend program graphs to work-in- progress code. arXiv preprint arXiv:2 /4038, 2021.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schritwiesser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition-lever code generation with alphacode. Science, 1996622)):002210999 2022.
Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. Quixbugs: a multi-lingual program repair benchmark set based on the quixey challenge. Proceedings Companion of the 2017
ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, 2017. URL httphtttss/:ppi.setttt htt.a///s...a....eennaacccccc... wwwpehoooorr.rrooooooooooeeeeeee org/CorpusIDt 7158771
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL
07958
Changshu Liu and Reyhaneh Jabbarvand. A tool for in-depth analysis of code execution reasoning of large language models. urXiv preprint ArXiv:2501.18445 2025.
Chenxiao Liu and Xiaojun Wan. Codeqa: A question answering dataset for source code comprehen- sion. arXiv preprint 10..v:210..08365, 2021.
15

---

Published as a conference paper at ICLR 2025
Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang. Zhen Yang, and Li Zhang. Exploring and evaluating hallucinations in tlm-powered code generation. arXiv preprint arXiv:2404.0097,, 2024a.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang. and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024b.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jía Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff. Xiangru Tang. Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry. Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024a. URL https: /ttp:v/aaaivi/viaa org/ abaaa/s24550 19173
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poiries Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint 2XXv:2442.11973: 2024b.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 2021.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:23006.88 08568, 2023.
Dung Nguyen Manh, Nam Le Hai, Anh T. V. Dau, Anh Minh Nguyen, Khanh Nghiem, Jin Guo, and Nghi D. Q. Bui. The vault: A comprehensive multilingual dataset for advancing code understanding and generation, 2023, UtL/ttts:://arxiii
org/ abs/ 200 06156
Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma. Maxime Voisin, Ellen Aalsenan-MeMahhee and Matthias Gallé. On leakage of code generation evaluation datasets. arXiv preprint
2024.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher Choquette-Chont -Cho, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint 2rXiv:2311.122 170355.
2023.
Minh Huynh Nguyen, Nghi DQ Bui, Truong Son Hy, Long Tran-Thanh, and Tien N Nguyen. Hierarchynet: Learning to summarize source code with heterogeneous representations. arXiv preprint arXiv:2205. 22445:22005.5479 2022.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 1.....1...... 13474, 2022.
OpenAl. GPT:3.5--urbo-16KK000
6 2023. URL wwtphhtttthhta..ooct
com
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian,
16

---

Published as a conference paper at ICLR 2025
Jeff Belgum, Irwan Bello, Jake Berdine. Gabriel Bernadett- Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Contijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, tukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan. Teddy Lee, Jan Leike, Jade Leung, Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez. Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini. Sam Manning, Todor Markov, Yaniv Markovski. Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McL.aave,, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reüichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish. Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang. Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng. Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL ittps://arxiv. owg/abs 2303. 08774
Yicheng Ouyang, Jun Yang, and Lingming Zhang. Benchmarking automated program repair: An extensive study on both real-world and artificial bugs. In Proceedings of the 33rd ACM SIGSOFT International Symposium Software Testing and Analysis, pp. 440-452. 2024.
Huy Nhat Phan, Tien N Nguyen, Phong X Nguyen, and Nghi DQ Bui. Hyperagent: Generalist software enginecring agents to solve coding tasks at scale. arXiv preprint arXiv:2409.1625 2024.
Phind. Beating GPT-4 on HvmanEval with Fine-Tuned CodeL 34B,, 8 2023. URL https 7/ 7/www.ph.r phind. www.oooooaooooooooooo cote/oooee///// 11 1ama- bewt.---tt...... -
Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, Ashish Datta, Maksym Zhuravinskyi, Dakota Mahan, Marco Bellagente, Carlos Riquelme, al. Stable code technical report. arXiv preprint A4V::2404001222 2024.
Ruchir Puri, David Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian T Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Veronika
17

---

Published as a conference paper at ICLR 2025
Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, UIrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. In J. Vanschoren and S. Yeung (eds.). Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https pwwcceee-er-eeedeeee ngs wwweurrps.cccppeeee Paper round2. .dd
les/paper/
Mirza Masfiqur Rahman and Ashish Kundu. Code hallucination. arXiv preprint arXiv:2407.04881 2024.
Vyas Raina, Adian Liusie, and Mark Gales. Is Ilm-as-a-judge robust? investigating universal adversarial attacks on zero-shot Ilm assessment. arXiv preprint urXiv:2402.14016 2024.
Joshua Robinson, Christopher Michael Rytring, and David Wingate. Leveraging large language models for multiple choice question answering, 2023. URL httpss//arziv.orr/ abs/
2210.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308.1293 2023.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024. URL httpss:/
/ttpg//b/////////// 2308.
Zhan Shi, Kevin Swersky, Daniel Tarlow, Parthasarathy Ranganathan, and Milad Hashemi. Learning execution through neural code fusion. arXiv preprint arXiv: 1906.00/006018// 2019.
Ben Shneiderman and Richard Mayer. Syntactic/semantic interactions in programmer behavior: A model and experimental results. International Journal of Parallel Programming, 8:219-238, 06 1979. doi: 10. 10.1007/BF00997788.
Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. An analysis of the automatic bug fixing performance of chatgpt. In 2023 IEEE/ACM International Workshop on Automated Program Repair (APR), pp. 23-30. IEEE, 2023.
Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of- thought helps mainly on math and symbolic reasoning, 2024. URL httph:t///// //tp///aavvvv// org/
abs/2405 12183]
Weisong Sun, Chunrong Fang, Yudu You, Yun Miao, Yi Liu, Yuekang Li, Gelei Deng, Shenghan Huang, Yuchen Chen, Quanjun Zhang, et al. Automatic code summarization via chatgpt: How far are we? arXiv preprint arX5v:2305. /2865, 2023.
Hung Quoc To, Nghi DQ Bui, Jin Guo, and Tien N Nguyen. Better language models of code through self-improvement. arXiv preprint arXiv:23344.012268 2023.
W3Schools. W3Schools.com, 2024. URL htlLhttps:////
Peiyi Wang. Lei Li, Liang Chen. Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023a. URL https 7/arxiv. org/
17926
Wenbo Wang, Tien N Nguyen, Shaohua Wang, Yi Li. Jiyuan Zhang, and Aashish Yadavally. Deepvd: Toward class- seasaatepnnttonn features for neural network vulnerability detection. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp.
IEEE, 2023b.
Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song. Lingming Zhang, An Ran Chen, and Lei Ma. Testeval: Benchmarking large language models for test case generation, 2024. U httptpp:///riiiaaa wwtpp////.r/b/bbb/////e/e// 04531
18

---

Published as a conference paper at ICLR 2025
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: cenntifie--wart unified pre- pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 2021.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922, 2023c.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi. Quoe Le, and Denny Zhou. Chain-of-thought prompting clicits reasoning large language models, 2023.
http:/bs/2222/ 2201. 11903
Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith and Douglas C Schmidt. Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. In Generative Al for Effective Software Development, pp. 71-108. Springer, 2024.
Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era of large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1482-1494, 1494. IEEE, 2023.
Frank Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPIAN International Symposium on Machine Programming, pp. 1-10, 2022.
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models, 2024. URL htt.:httss:///t /ttp//aaxivv wwwpxxvvvvrr/aaoo haag///b///// 2404. 11.000.888880 18824
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL wwwp//s..//ooooooooootooot..ae.s wwwp///0...00.2
https://arxi
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation, 2023. URL htt.ttsss//axxiv.orttte abs 2303.12570
Yuhao Zhang, Yasharth Bajpai, Priyanshu Gupta, Ameya Ketkar, Miltiadis Allamanis, Titus Barik, Sumit Gulwani, Arjun Radhakrishna, Mohammad Raza, Gustavo Soares, et al. Overwatch: Learning patterns in code edit sequences. Proceedings of the ACM Programming Languages, 6 (0OPSLLA2):395-44 2022.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In International Conference Learning Representations, 2024a. URL http:tttst/::/eeee////
net/ wow.mmiddsshrrrrr
TTO
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging alm-as-a-judge with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 36:46595 46626656666666,, 2023.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter:: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. 14658, 2024b.
Eric Zhu, Vadim Markovtsev, Aleksey Astafiev, Chris Ha, Wojciech Lukasiewicz, Adam Foster, Sinusoidal36, Andrii Oriekhov, Joe Halliwell, JonR, Kevin Mann, Keyur Joshi, Michael Joseph Rosenthal, Qin TianHuan, Senad Ibraimoski, Spandan Thakur, Stefano Ortolani, Titusz, Vojtech Letal, Zac Bentley, fpug, hguhlich, long2ice, oisincar, and Ron Assa. ekzhu/datasketeh: v1.6.4, October 2023.
19

---

Published as a conference paper at ICLR 2025
Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K Reddy, Xlcost: A benchmark dataset for cross-lingual code intelligence. arXiv preprint arXiv:2206.08474, 2022.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 2024.
20

---

Published as a conference paper at ICLR 2025
A DATASET
A. DATA CLEANING
Rule-based filtering We prefer questions that contain code when collecting data; therefore, MCQs often contain noisy patterns and low-quality questions. In the cleaning process, we defined a heuristic rule-based filter to eliminate incomplete data and non-textual content. First, we detect and eliminate non-textual questions by filtering questions that contain hrefs, image URLs, links to other questions or media. We also applied BeautifulSoup to remove unwanted HTML tags.
Deep learning-basee filtering To ensure the CodeMMLU is fully targeted on coding and software- related task, we employed models from OpenAl (GPT-3355turbbr Mistral (Mixtral 8x Instruct), and Llama (Llama3.1 8B) our annotators judge the triple criteria: Completenes:; coherence and clarity; and coding relevance (check appendix [C.] to see the prompt). We averaged LLM ratings by category and selected a threshold of 4 in aspects. Result of removing 25.6% of raw data. On the other hand, we simultancously sampled a subset of 100 instances in each subject to update our filter rule. (Figure g
Completeness
Relevance
Log of sRating
Figure 8: LLM-based filter score distribution.
Execution-bases filtering After synthesizing the fundamental task's 's distractor (i.e., false answer), we concatenated and executed them as a complete function an isolated environment. The code completion and fill-in-hhe-bland synthesize new test cases. Therefore, we extracted the method signature (using the code-text parser toolkit from Manh et al. (2023)) and synthesized the function input, which later was executed to create test cases for the corresponding function. ran in parallel the distractor executing on the testcase and synthesized a new distractor, ensuring the distractor collection is executable and able to pass 0-50% test cases.
tasks have their original test cases, while code repair needs to
A.2 DATA CONTAMINATION
The development of large language models (LLMs) often involves crawling data from diverse sources across the internet, with limited transparency regarding their preprocessing. Given the vast and often proprietary nature of these training datasets, it is widely acknowledged that creating fully leakage-fres benchmark is virtually impossible. While recent benchmarks have recognized this issue and generally accept that avoiding data leakage entirely is extremely difficult, one common mitigation strategy involves filtering data based on its timeline Jain et al. (2024); Wang et al. (2024); Zhang et al. (202)).
In our efforts address this challenge, we acknowledge the complexity of completely eliminating data leakage. To enhance the reliability of CodeMMLU, we adopt proactive measures during the data creation process. Specifically, we transform seed data into multiple-choie question formats and introduce synthetic distractors. For tasks like code repair and execution prediction, the test sets were extracted from codebase seeds and modified align with specific task requirements.
21

---

Published as a conference paper at ICLR 2025
To further assess and quantify potential data leakage, employ the methodology outlined in Xu et al. (2024). This includes calculating perplexity and conducting n- n-gram analysis on several well-known models from diverse families (c.g.. Mistral, DeepSeek, Llama). The results, presented in Tables and 6 highlight significant margin between CodeMMIUU and other coding benchmarks, reinforcing the reliability of CodeMMLLU robust evaluation tool.
Perplexity measures the uncertainty of language model when predicting the next token in sequence Jelinek et al. (1977). Therefore, as low the perplexity score indicates, the model is confident in predicting the evaluating sequence and the more likely that the model was encountered during the training process. Perplexity is expressed as the exponentiated average negative log- likelihood of sequence:
(XL(X) exp
logpa (xi(
where X
[No, T1, .. 22] denotes a tokenized sequence.
N-gram Accuracy Xu et al. (2024) is a metric designed to detect fine-grained data leakage at the instance level by combining the question and answer into a single text (X), uniformly sampling starting points, and predicting the next n-grams based on the given prompts. most n-grams are accurately predicted, it suggests the model may have encountered the data during training. The N-gram accuracy can be expressed as:
S
K
N-gram Accuracy(X) S. K
Σ 1= j=
[(X stttt & Xetort, istort, +n))
where is the dataset size, K is the number of sampled starting points, Xstart, +n is the actual n-gram, :atart,:start, +n the predicted n-gram, and I checks for exact matches. Author add ROUGE-L and edit distance similarity to provide robustness for augmented datasets. A high accuracy for each n-gram in prediction indicates a strong likelihood that the sample was seen during the training process. Xu et al. (2024)
Table 5: Perplexity score comparison between coding benchmark, (higher is better)
Models
CodeScope
CodeApex
CodeMMLU
Mistral7B-v0.3
9.32
16.08
16.32
DeepSeekCoder77--11
5.26
9.39
57.36
DeepSeek V2-Lite
6.89
11.99
1419.48
Llama-3.1-80
10.05
123.20
197.31
Table 6: 5-gram accuracy comparison between coding benchmark. (lower is better)
Models
CodeScope
CodeApex
CodeMMLU
Mistral7B-v0.3
0.2510
0.1702
0.1365
DeepSeekCoder7BBv/l
0.2818
0.1680
0.1416
DeepSeek V2-Lite
0.2492
0.1587
0,0687
1lama-3.1
0.2219
0.1309
0.0652
A.3 LICENSE
In the construction of CodeMMLU, we collect only the multiple-choice questions, problem descrip- tions, code solutions, and test cases from the publicly visible parts of W3School and Geeksforgeeks quizzes/puzzles and LeetCode. We avoid any data collection that requires login or interaction with these websites. On one hand, most of our knowledge test set (ss 61 61%) are collected from Common
22

---

Published as a conference paper at ICLR 2025
CC-MAIN-2024-40) On the other hand, the fundamental tasks were created on permissively licensed codebase, namely IBM Project CodeNet (Apache 2.0). HumanEval, QuixBugs (MIT). For data crawled from websites such W3Schools (fair use for research purposes) and GeeksforGeeks (under the Copyright Act 1957), we fully complied with their copyrights sought their permission published and distributed under the MIT license.
Crawl (from portion tagged CC-MAIN-2021
such data for this project. CodeMMLLU will be
B ALL EXPERIMENTAL RESULTS
Table 7: CodeMMLU and other coding benchmarks comparison. The ranking reorder comparison between CodeMMLU (CM) and other benchmarks (namely HumanEval (HE)).
Family
Model
Size (B)
MMLL
GSM8K
HumanEval
MBPP
CodeMMLU
HE CM
Closed source models
Anihropic
Claude:3 Sonnet
88.70
96.40
92.00
76.6
55.48
1-44
GPT-4o
88.70
95.80
90.20
81.4
64.96
2-1
OpenAl
GPT-3.5-turo
61,90
73.80
61.40
78.5
51.59
10-46
Oper-soource models
Llama3.1 70B Instruct
70
83.60
95.10
80.50
75.4
59.68
6-33
Llama3.1 70B
70
79.30
83,70
58.50
66.2
40.45
11-20
IamaLlama
Llama3 700
70
79.50
83.00
48.20
70.4
49.7
14 +8
Llama3
4-2
CodeLlama 34B Instruct
34
41,50
57
39.27
17-21
Mistral 7B Instruct Mixtral 8x78 Instract
7
62,50
50.00
26.20
50.2
21-177
Mistral
46.7
70.60
74.40
40.20
60.7
42.74
18
Codestral 22B
22
81.10
78.2
47.61
5-+13
Phi3 Medium 128k Instruct Phi3 Mini 128% Instruct
14
78.00
91.00
62.20
75.2
48.65
Phi
3.8
68.80
82.50
58.50
70
39.22
11-22
Qwen2 Instruct
70.50
82.30
79.90
51.86
7-5
Qwen
Qwen2 578-A14B 14B Instruct
57
76.50
80,70
$3.00
71.9
47.34
12-14
CodeQwen 7B
77.7
3-+12
Yil.5 5 34B Chat
34
67.62
71.70
23.20
41
50.03
22-+7
Yi
Yi1.5 Chat
DeepSeek Coder 7B Instract (v1.5)
7
49,20
41.00
42.10
60.7
41.59
16-19
DeepScok
DeepSeek
DeepSeck Moe 16B Chat
16.4
45.00
18.80
26.80
39.2
31.45
20-24
DeepSeek CoderV2 Lite Instrucr
16
86.40
81.10
47.12
5-515
Internl M
Intern M2.5 20B Chat
20
66.50
79.60
48.80
63
46.15
13-11
StarCodar
StarCoder2 15B Instruct
15-11
B. SELECTION BIAS IN MCQS FORMAT
Building the findings from Zheng et al. (2024a) which investigated the effects of reordering answer options in multiple-choice questions (MCQs), we observe inconsistent behavior among large language models (LLMs) when performing the same code completion task. Table highhighhs the sensitivity of LLMs to the order of answers, even for models renowned for their high performance (e.g.. GPT, Claude, MetaLlama). Specifically, the results reveal that most models experience significant performance degradation when the correct answer is positioned as "A", with an average performance drop of 25 %. In contrast, placing the correct answer in position "B" leads marked performance improvement, with an average increase of 15,49 %.
The standard deviation (STD) further illustrates how differently models respond to answer reordering. and DeepSeekCoder-3333 exhibit substantial
For instance, models such as Ladelllama-7B/1333 dependency on the arrangement options, whereas models like GPT-40/3.5, Claude-3, and Claude- 3.5 show greater resilience to such selection bias. Interestingly, instruction-tuncl models, which are generally expected to demonstrate increased robustness, show minimal to no improvement over their base versions in this regard.
These findings suggest that higher- models are more resistant to MCQ biases, reflecting a human-like ability maintain performance irrespective of answer order. We believe that introduc- ing this MCQ bias into the CodeMMLU benchmark adds an extra layer difficulty for LLMs, encouraging the research community prioritize enhancing the consistency and robustness of L.Mss.
23

---

Published as a conference paper at ICLR 2025
A Acc (%)
-50
Recall (%)
75
50
25
CL-34B-Py
CL-34B-Inst
DSCoder-78
DSCoder-33B
Phid-348
Mixtral- (xrrral-xxB-Instt
Figure Task- Task-Specific Accuracy and Performance Fluctuations Across Answer Options Models exhibit marked fluctuations in accuracy depending on the position of the correct answer in Code Completion in CodeMMLUU. Revealing the bias and inconsistencies in related coding multiple-choice question (MCQ) task and how sensitive LLMs are to answer ordering.
Table 8: Selection bias effect comparison on LLMs. The performance fluctuation trends show significant margin of model with high quality and the other. STD stands for standard deviation.
Models
Instructed
C
D
STD
GPTT-40
80.49
78.05
71.34
70.12
4.38
GPT-3.5-turbo
51.22
43.29
47.56
54.88
4.30
Claude3.5 Sonnet
V
90.24
81.1
85.37
79.27
4.23
Claude3.5 Haiku V
86.59
69.51
72.56
68.29
7.30
Claude3 Opus 4
79.27
77.44
82.32
84.76
2.81
Claude3 Sonnet
V
62.8
64.02
73.17
73.78
5.06
Claude3 Haiku
V
56.1
75
73.78
76.83
8.34
Mixtral 8x7B
V
22.56
74.39
71.95
63.41
20.91
DSCoder 33B
1.22
82.32
75.00
56.10
31.75
DSCoder 7B
40.85
74.39
64.02
39.02
15.10
Phind-CL. 34B
6.10
90.85
75.00
46.34
32.21
CL 34B Python
0.61
77.44
70.73
49.39
30.09
CL 34B Instruct
9.15
84.76
65.24
46.34
27.91
CL 13B Python
0.61
54.88
70.12
12.20
28.85
CL 13B Instruct
2.44
68.29
72.56
29.88
28.85
CL 7B Python
0.00
90.24
14.02
0.61
37.39
CL 7B Instruct
3.66
1.22
93.90
15.85
38.07
24

---

Published as a conference paper at ICLR 2025
B.2 COT MIGHT NOT BE ALL YOU NEED
In our experiments with models from over 15 families, we evaluated CodeMMLU under two different prompt settings: standard zero-shot and few-shot, well Chain-of-Thought (CoT) with short and long prompts. The detailed results, provided in [.3] reveal consistent trend decreased performance in the CoT setting compared to zero-shot and few-shot configurations. Even strong models like GPT-4o and Llama3 70B, known for their robust reasoning and comprehensive capabilities, exhibited this pattern, as illustrated in Figure 13
Comparison between
prompts (GPT44))
Dislinct
compared to zeroshot
Chain-of-T Thouuhh (26054
Fewshol conwet)
Zeroshot (34/54
Boonwed
10 20
Figure 10: Comparison experiment of different prompts by GPT-4o in OOP subject.
A closer analysis of results for zero-shot, few-shot, and CoT prompts, particularly in the GPT-4o experiment on the object-oriented programming (OOP) subject (Figure IO). highlights the effects of an additional "thinking" step introduced by COT. While this step intended to enhance reasoning, the thought often diverged from the original focus or brought up another challenge for the task. For instance, example in Figure 14 showcases case where the CoT approach incorporated unnecessary definitional analysis but failed to arrive at the correct answer. Similarly, examples and 3 (Figure 14) demonstrate misinterpretations during the reasoning process. This additional "thinking" step often introduced new challenges that might lead to incorrect conclusions, resulting in significant performance gap between scro-shot and CoT prompts.
Interestingly, when visualizing the newly addressed questions by CoT settings (compared to 0-shot), do not observe clear pattern of these new corrects. Questions that CoT addressed inherently tricky nor required deeper interpretive reasoning. For example, example 4 (Figure 14] asks for prediction of program outcomes within the OOP domain, comparable in scope to example yet the CoT approach showed inconsistent performance.
neither
As concluded by the findings in Sprague al. (2024), the CoT strategy introduces an additional reasoning step that is effective primarily for tasks involving mathematical or logical problem-solvingg However, for benchmarks like MMLU, the results with and without CoT prompting are nearly identical. The observed performance decline in the CodeLLM knowledge test set aligns with Sprague et al. 2024), further underscoring the inefficiency of CoT for nea-rnnggn tasks.
B.3 EXPERIMENT RESULTS
We provide the full experiment results of 43 LLMs from 15 families in Table 9
10
1
25

---

Published as a conference paper at ICLR 2025
Question
DeepBeek R1
Question Which of the followng is another name dass
<hinko
tomponnwes?
Ocay,
syru
remember both useEffct usslayoutEeeor hooks hand rig side offects in Roaot components but therels koy dfference between theer
camponente
Statelssa Componentoi
From what rocal useEflect rums after the component has rencered and the serven has been updited That's the all ceew. Onn the other tand, used ayoutthhe i desigred to nun syndhoonzully afse DCM musstone before browear parns the screen. So the main changes need
Prope
(D) Stone
they fire Usel Syeel flidker.
GPT
Looking options:
Oparmas for deeiccss watt's Himm, optim zaston right side offect connccty hand ing layoul, but don't ti that' the direct purpose suutays
The
Change byyot X
(i) Compgede all the updaos tudd 13 hist sum Mayte releos io twishing upiatee?? Or anausing
Exi on:
updotes are finished?
when you need to perform die elictta that read or wr is lipout information on he gatling
Deact. uawd fltulfffffee
leyour 1u20 That sounds more like possible case. you need change aycuuiefffccct
appropriate
the poettion stow of noW elemerrs) eynthronnuuly afte al noM
hore.
musttioons but twitow the brosow hue chenoe
Trix happon
(D) Paint belore efied rume kuatns Walt and.ayyuttleett numa belore paint. So option
crucial
need
visust changes
cowms incomet. The normel ttwtf runs after paint IPD ways (Paint belore the effect mott" than the case of csel ayoult fhid the wflect nane beloes parting on the paintigg happora wher So Di descripton mightt indeooo
Bickcring.
Hern's how the optian nebew to luwed ayduayyurtii Beet'
Wall,
oprione syndvenousty
unel ayourfteet is coreecty
The docum mosumeeraiion
(A) Optimiz for all der case. This la too vagua and not sccliiiiity
DOM changes.
bindd the partoae 'ueel ayputtileet
uptited
(B) Complete the update: While usolayoutErest runs aftrr DOM
comd ayoultilll is when you reed make DOM changes messusing leyoul modify my
It's about
synchromiding loyout change
DOM) and you don't went the user see ticker between the inttal paint and the updated stsle.
riinn
(5) Change
So one of the correct reasons to uso seLoyyuuEeeet to read layout from the DDW and then make chargees symchransunty, option that retate changing the (C) anauring Looking before
lsed ayoutt
intentted openat ons the reed read afbe
the DOM boyout before the browser paints the screen.
fiddering
(D) Faiit belcre the affect rume This the oppocite of what actuslly
happonn. VseLiyyoEEteet runo "toefore the browser paints. not
before effect nutt" No, becouse uostayyuuflect runs before paint. So D's staterent is the oppneite Option Bl comp win the update, mayhe that refers to completing updaacs belore the paint, But surs. Option O inChungs the lapoutt which could be conoet if you noed to make loyout change that' done before the browsser points. So may correct aranser C er bui need thekk Wesl the optione
Oction
btocking vinue uptaries
sparingly for canse where you need perform mesuammenta
mutatioons that affect lsyou and need to happon before pairding
Anawer. (C) Change the layout.
<XCONTINUE REASONING ABOUT TOKENS
GPT 03-mini
.- TThreetre, optio (G) comeot <hhass
The cornect serooer is
propertis Bhat The comndd of Lantaayuuttfee lo parform DDM mutations prevents users tickwing.
Explanation:
need reflected symorronostty nyy before browser
React 20 18s EsclayouEffsst book designed to run syndroniuuly
seeing
DOM musations
the perfore
"Anslysi ons
messuning
"((A) Uptimza deviser" Incomest Opamizatio not dreetly tad "ass.ayourEeeer
lagout syndirenously updatin the DOM to avoid visual inoaee adendee (ficker) curing the rendering prowes in other wonds any shanges thot need to happen bofore the browser paints should be
(TB) Complete the updatet: Vague and not speeific to the surpese of ussLayoutE foot.
(*(C) Change the leyouth "C"onee"" luee ayout thes used to moatly the DCM (k
odust style, element positinn) before the browser paints, encuring charpes are immedate.
handled in yeuu which is why option ("Pain before the offect runal) coptares that behaedor.
"(D) Paint beform effect nuneth incomeet "unel syoutifter "teronn painting layout
"Aeswer
Figure 11: Comparison reasoning process of GPT ol, GPT o3-mini and DeepSeek RI on knowledge- secking task.
26

---

Published as a conference paper at ICLR 2025
Quertio
Saok
The fulluiig ae chaiee andion (alls
happoro
Buecker dhoee sosn calqee tokou, whist bndouiee masd Manly
comple indede anress How
Fast, toloald maal thaggh
troork wnd Fur price
nattoal sade cara
autic cian H
Thiie twnch toop that Berske over each eie mentin trpurr and pertorm a broee AnD with the pete NOT ohaue So eich inputt] in being
etaia aus rea plring loget
Gommarac ray Caaner/outeaary
mad Comermy AO the rocilet ares, shitt the rouul lef ona, adeat to sam, and prins
Non g the mectemerryf methhot aod Ohirgst the
thing impirmen Couas an mutor ihe of 1of venters Latt
ait cow sghannd
korn iise to di on, comut wit che neniiaaee juinn X01 uns eugtery) than cler clke bue lir al sihe oue dork. Gas arotaa coer dencittu the tasks
cal prcsiisnn 2000 th aboraaia
losss
bdl in www.ppeetoingggo..ooo. mexdtt
but chee
slemerts thy acckallyyy XDII ieeuss comedly letie invere Altes) and Affect
Ady
eaey To.. lrrst
Apd5) - Afri
tylicll tiver beliver two van a and de
Sopleroetppehttter traudd Or 1Ix
cade
intal
Whit
the Ajoll becoress
suieee ong Aoreer I
Agoss Aprett Then the swondd Aflind the App0) which new the new 4gc6t sood xOn aftect son that woeld whet') but
corlig caep betuced too aary elemara, iides thay tn sane the acris Thrugh lile we thay creck bokur
di 16-1-11 intind Ioc
Mayee that coree
Hon, the luga wesi We couling with Gis assaa si ation eadecting the plve elereen the or with high we varn
SXQUESTION TINTION ID 108366--
Lett locg
Wall the
mentoret
Agons Apoict
Time Lnail Racandied Compiee Eincr
Ind occ
Lift
stnomally ford
decoraliin becomes "Crry
connet emeer in Comeite Evee X
at Apoi Ajretio to Than ofher ewane whe kic to ofter seepling the to Then prooed strmmete
thurr other iec cocs eveppig ih's sode's kop Aoraa though ofl and checks the
in
The XDR
Aenther poen meniee in inlut sped Hers
whonh ding temornery naratie Hononve
fiee codk tets nore foom de i For carn the wide keap, oke from
stard deseetree
thut side offcct
Dut whal's covearo besd' The code the mar damia KO kundir a compuce is the maimmnm adbeet XOR part apa, and then reaama
time
Anstnet
aspless
just Xi alemards Walk
saners Che oe cie oe the pors The marimas 12W the aree on tu
whan the privr
Aj000 Afted]
right sared the maris up Willing el diemarts ray in screed mas Hoverss respoe wit hour the
mene gut
fut nemondd or the camen warsin ve ofe the busis frvm pronnd ahooee trok olern wit
maccn mayyle theire processed foom ligheer to lavent ragia Killing all give the mair sars. his or arrt auss, paay.
Tho iore with ths lepinn methorl Avns that wuy nut woch raended Joni's strcidss morry managemna and
Choe drodk may mene's her nowr tor vepuueeee cnce
Por protioon here zons do whtt too thur i sper (Ite it compte tlrer SKPP wee moadd we whe ooe
saea systas
brosss spentiona
dy
The code para with import auiiuii t:a valid. The trai cless a public staic main method, and the zode is in dasa, Sorra probeer than..
To eemantic
shoul
aedeesa en saince
6G. 1.
Gince
(uaindr
Eat en 60 lenp which comeec.
syeter
nightte
ke pur 2. cime (ee))
Arewnn (M Compala
Rethoarie bygy The haleir somplee and toe and whir wheth nowe oothe repaing toogge ekay the foie
snlo plo sulor tober ploo morin
OPT o3-mini
mww.itammattei momisoo volarn
camp Lafa
anonee No abmoreall frentt
Caler tolp rare for 50 coveto di bib. for each iperiorma lopp had could scin up Alrggh ela In ted cock than
parters ee thate conatoors, corouiee
pri ded sode eippst Laval releer talicn tha medi spui
rar, num la lars, led for euampia than 51 lidd di wwwnnirrtal..ooo and 4e enageeble (ee insu lagg the ind. thi uouul accurtable.
whick
ectrppeer siton numbers The li sonirarr fare
cauting lins oi vole or mansay lnd anown under
Fer war cari mach apait ae lone Him deri livnn ansund repersh leai palliggs maneiiiii ror cath sae Sod dhe cade vage out 11. what uvang cacan, per Toos I. aller parur legicil ancan loaddng
of sehatoos.
SOUNNABLE
Figure 12: Comparison reasoning process of GPT ol, GPT o3-mini and DeepSeck R1 on problem solving task
27

---

Published as a conference paper at ICLR 2025
API Frameworis
DBMS SOL
38-
Perfrraacc
10-
677-15
Petts L130a-3770
40
GF7-45
3-203
16-
Metts 3h30
677-46
Z010
Pon
OUT
Zo12
For
DnT
261
Pon
OcT
Software principtes
Completion
G77-45
Hilt 2-70
70-
Ry/fomancc
52-
G77-45
GFT-46 Mater
10-
33-
201o
Fea
207o
FeA
Fil the middl
Code
Delect Detectioon
G77-4g
60
70
22-
30-
280
2800
Fonk
Figure 13: Comparison between GPT4o and Meta Llama-3 70B on various prompt settings. We experiment with zero-shot, 1-shot, and CoT prompt configuration, where the result indicates the ineffectiveness of CoT in boosting the models' performance. Comparing to zeroshot config, 1-shot prompt slightly increase the performance in knowledge tasks but falls shorter in real tasks.
28

---

Published as a conference paper at ICLR 2025
Question
Question: Converting primitive type data into corresponding wrapper class object instance is called
Boxing
Wrappiin
Instantiation
(D) Autoboxing
Zeroshot
(D) Autoboxing //
Chain-of-Though
To determine the correct answer, let's break down each option and understand what they mean in the context software development:
(A) Boxing: This term used to describe process converting primitive type corresponding wrapper class object. For example, converting 'int' 'Integer..
(B) Wrapping: While this term might sound relevant. it not commonly used in this specific context. It generally refers to enclosing or encapsulating something within another layer.
(C) Instantiation: This term refers creating an instance of class. does not specifically relate converting primitive types their wrapper classes.
(D) Autoboxing: This is more specific term that describes the automatic conversion that the Java compiler makes between primitive types and their correspanding object wrapper classes. For example. automatically converting an 'int" to an 'Integer'.
Given these definitions:
"Boxing" (A) correctly describes manually converting primitive type into its corresponding wrapper
class.
"Autoboxing" (D), however, specifies that this conversion happens automatically the compiler.
Since we are looking for just "converting," without specifying whether it's manual or automatic:
Answer:
(A) Boxing X
29

---

Published as a conference paper at ICLR 2025
Question
Chain-of-Though
Question umong lollowing fodowing code?
Lat' unatyar stpp stap decermene which typa inhertance chacs ttuctur being used.
Siwoo
publi: class
1. Class A:
Comaars nested public dass "d" has
Intwets publidy sieos
Comaiis a nested public daas "D' that interts publidly from AcE
class C. A
Nov, break down the inferitacce and nisting:
Claos nested within Cress
(A] Mull level inciianoe is usod with nessed dasses
[8) Murtple tenos used, with reded idanses:
-nheritance Belationship::
Singi inhertance used, undlosing
Class im eriz from A (aingle lavet).
Single inhwrthnne
wnntruiing neated
NNeted ddees D Inbwrts It jmuttieenn)
Zeroshot
points con
option:
/ Single
nested
(A) Mull-iood lovelll inforiance used. with nested diasses
Thee seerta oorrect becausee there are muliipl lnwebe of intorilannee insclvedt C A (single level)
The codo seficcs a diss ''' which contains a nested diass
muli-lewnt)
-Chesi athrra from cless A: making sangle- tnggle-aae intertannce
-Wttin clese 'C', there is another nested class 'D' that inherits from the
Mutiple Whortance
reerdd idese the brew dione.
because angll diuecyy more bass diaso.
(C) Singie level inmrtiance caedd with enciteing checses
includos andlosing jolaso contaiing containin
This isntt entirely acourate sinoe there are multpple levels nesting and some muli-ievel métitornhh
D)
-Whiie
undosing nusting correctyy incorracly states "Gingle leve"
when theness muliiieueed Invetied redetionass benwwen through their respective parent classes.
Thonefore.
(A Mull-iovd imtrizance used. with nested dasses
Chaln-of-Though
Question:
What the cutzui of the following program?
ddese A
To determine the output the given program, let's analyze it step by step-
Class Delnntton:
protected: disp)
closs has two protected integer members 'a" and "b".
aloo has pubric mein ber function diupp' that prints the valueis of 'b..
Inheritance:
publicty
introducis additionsl privale integer members and 'y'
Functionality
cless 8 pubiicc A
The function class dispt'' is designed to print the values of protected
members ('.. and 'b).
Output Analysis
any
intraiiinng setting memory
Gartagge value
locations object dass B for even
(9) Comple time emmr
Compilation Check:
[C) Runtime emrr
There are no syntax emors in this code snipper, honce will compiee supoosofully
Rune random
Runtime Behavior
'Wen
disp() method hold
dlss (uninihalized local vartables)
indhsiizing
The correct
Comple time smer / dlepC
Civen these points, condlade:
Explanation: The
programn compliss aucossofully
because
-ll nntime, since 'o' and are not inttis fred before being printed, they will disply garbagetrancom values.
is using the variadess
are protected
membbees Hiwwsoe these verielie towe
intledooed
asaigred and any vonuee belors being used the outpet atsternent Additinnlly by there night be Isaus induddigg recessary headers 1ika "icureamm" y and using appropr site namespaox
Therefore, considering
(1))
give output
30

---

Published as a conference paper at ICLR 2025
Question
Chaun-of-Though
Question: What the of following code??
Understanding Structure:
There are tao packages: "pack"' and "pack2".
publil
In pacct there class with constructor that prints "otject craated"
"pack: there class which creates instance of cless AA. Code Analysis:
Syslim outprint abject created"")
package packt;
package pack?;
in part pack!!""
class Af
cless
publicA(
System putttt created"):
package pack2:
(A) Dutpul object created
import pack1.".
(8) Dutput is: otject crestedobject created
class B
(C) Comple time airorr
Run time emror
3. Compilation and Execution Steps: -Wiee you comple the cocks, both packages ( 'packk and "pack2" need be complted
Zaroshot
The cornect answerr
The statement package
"'Epacled Dutpul
Complia brm wenrr
eveeyything compiles correctly, crealing an instance an instance class due the line:
creating
Exp ciees must te own
"Iaaa
Aa = new Alk
as the dass. The provided code sripper docs not to low this rulo.
also issuss padkape deciar at ona and imports.
Possible issuss Leading
(e should padisges be proparty match their namnes.
If them's any issue accessing claszes across packages (e g. they not
jeva'
property compledd them some vasbilly issue), it could lead witer complestime
rundime enoos.
Checking for Complla-Tine
The import etatemen should correcty reference disspes from other packages.
structure neceesary imports
correctyy assuming
exis,
be complee
7. Checking for Runtime Errors:
Assuming proper setup (corect directory structure, correct co compilation) there should
complee- me uner packaging lile structure.
be nuntime emons wall.
Outupp Venification:
you "[jva
of
B new Bil:
wil Incomally create objectt Class due member variable intializatin:
"jjee
now triggers the print slatement
So this meuts printing 'otject creaned"
Therefore:
created
Figure 14: Examples of Chain-of- Chain-of-Thought Thought and zeroshot in Object-oriented programming (OOP) subset.
31

---

Published as a conference paper at ICLR 2025
Execution Predic-
tion
57.79
56.41
53.16
3.32
58.33
51.27
11.97
36.74
4.26
6.11
0.79
7.38
4.96
3.77
2.30
2.28
6.66
5.85
3.33
57.11
2.48
52.11
5.11
3.88
18.03
2.90
2.15
2.25
Code Repair
73.68
61.84
50.00
56.86
67.11
59.21
46.05
38.15
7.90
10.74
1.35
0.00
27.82
14.66
24.91
82.64
66.70
34.30
59.21
22.42
46.05
52.60
55.79
38.16
23.63
47.61
52.74
Fill in the blank
67.87
62.24
69.94
87.98
73.27
54.39
39.50
29.31
6.95
17.90
39.97
70.60
20.43
24.80
8.31
92.02
76.00
94.65
55.52
54.58
4.93
54.49
33.26
71.54
63.27
79.66
10.38
16.30
Code Comple- tion
79. 27
87.80
75.00
4.88
87.20
67.68
65.85
76.83
0.00
42.68
0.00
51.22
1.22
37.80
12.20
21.95
78.05
48.17
60.37
88.41
3.05
76.83
35.37
11.59
81.71
59.76
43.90
56.10
Table 9; CodeMMLU zeroshot experimental results.
Others
70.82 82
69.80
65.28
63.46
69.66
65.21
63.82
51.35
39.02
31.58
39.02
0.00
38 38.88
46.17
0.00
56.89
62.36
46.75
50.91
68.56
57.91
61.20
47.63
50.47
54.34
43.18
46.02
48.21
DBMS & SQL
83.72
78.12
70.99
72.52
81.68
70.99
70.23
58.52
47.84
43.77
50.89
0.00
50.89
52.93
0.00
63.10
67.68
53.94
60.31
74.55
65.65
67.18
54.71
53.44
62.09
53.44
54.96
57.76
Software princi- ple
77.85
74.55
69.23
62.27
77.46
69.93
50.12
51.06
36.80
35.28
39.09
14.65
39.08
41.70
16.02
52.92
58.83
42.71
33.43
72.34
53.44
62.71
41.94
47.17
50.72
41.87
42.92
47.77
PL Syn- tax
48. 76
48.04
45.26
45.42
49. 10
46.08
44.18 18
38.47
28.23
23.18
29.12
25.79
29.12
34.44
31.14
40.74
42.20
32.38
34.89
46.29
41.16
42.40
32.86
34.76
39.44 44
30.74
32.38
33.38
&
API Frame- works
88 .5 89.44 44
84.59
89.02
88.45 45
87.02
88.45 .5.4
84.88
72.04
54.21
72.04
0.00
72.04
79.17
0.00
86.02
87.59
75.89
81.31
91.01
87.02
87.30
75.04
78.32
87.59
74.75
71.90
75.46
CodeLlama 7B Instruct
CodeLlane Llama 13B Instruct
Codell Jama 13B Python
Code 34B Instruct
CodeLlami Lama 34B Python
Llama3.1 405B Instruct
Mistral 7B Instruct (v1)
Mistral 7B Instruct (v2)
Mistral 7B Instruct (y3)
Code Lama 7b Python
Llama3.. 3.1 70B Instruct
Llama3.3 3 70B Instruct
Llama3 70B Instruct
Llama3.1 8B Instruct
Llama3 8B Instruct
Claude3.7 Sonnet
Claude3.5 Sonnet
Claude3.5 Haiku
Claude3 Sonnet
CodeLLama 13B
GPT 3.5-turbo
Llama3.1 70B
GPT 03-mini
GPT 40-mini
Llama3 70B
Llama3.1 8B
Llama3 8B
Model
GPT 4o
32

---

Published as a conference paper at ICLR 2025
4.32
2.53
43.14
18.12
3 3.48
3.68
1.55
2.35
1.95
3.06
42.79
1.07
2.49
2.96
4.63
4.31
48. 12
31.35
5.92
3.38
3.00
2.67
5.43
39.23
46.00
5.83
5.32
0.02
3.69
0.00
3.85
41
33.09
48.82
60.53
35.53
39.57
51.66
14.45
30.53
21.07
18.50
38.16
36.84
74.74
27.20
35.90
60.64
46.05
38.16
55.26
55.30
36.66
49 99
42.11
51.32
18.64
40.06
41.13
65.49
2.63
0.00
25.69
58.43
23.53
40.82
44.76
31.38
24.00
21.42
27.43
76.66
40.86
13.72
21.28
24.52
56.93
50.78
68.58
73.41
5.40
37.06
35.09
41.90
32.27
33.26
41.33
13.62
31 1.94
0.09
3.66
27.62
31.09
3.05
74.39
75.61
23.17
67.68
72.56
42.07
53.66
62.80
53.05
80.49
35.37
28.66
9.76
58.54
75.00
69.51
81.71
82.93
39.63
89.02
67.68
53.05
60.98
56.71
56.10
6.10
8.54
16.46
26.83
0.00
13.41
52.88
50.18
57.48
54.34
53.46
54.41
46.75
45.30
52.22
45. 45.88 .888
58.
51.71
32.90
44.78
55.43
54.70
53.98
65.35
55.51
40.41
39.24
52.52
49.53
52.81
56.09
60.39
0.00
42.89
39.10
38.15
46.32
44.93
61.83
58.52
67.43
61.83
58.78
62.09
54.20
53.94
64.12
51.91
66.92
59.29
43.26
54.20
65.14
62.85
62.85
73.28
63.10
51.65
46.06
61.32
58.02
60.05
63.87
66.92
0.00
52.16
49.36
52.67
57.51
55.47
49.96
47.62
56.08
48.86
51.44
52.53
44.99
40.70
54.40
44.63
57.83
52.93
36.45
43.52
51.87
48.12
54.41
71.15
58.08
38.89
38.25
52.93
45.19
52.32
56.12
54.85
15.98
41.24
36.96
38.01
47.21
43.29
37.32
38.50
40.64
36.86
37.19
36.82
31.83
33.14
37.32
35.26
41 41.85
39.05
24.23
30.03
37.80
33.73
37.06
46.54
41.58
30.56
31.70
37.03
34.45
35.82
39.12
43.94
32.40
28.71
29.84
35.98
33.60
0.00
85.02
82.17
87.02
87.02
79. 89
80.03
74.18
76.32
77 89
79.8 89
86.59
78.32
52.92
73.04
84.88
82.88
82.74
86. 88
87.45
74.47
67.62
79.60
75.75
75.46
71.47
87.02
0.00
74.89
70.19
71.75
81.60
79.74
Phi3 Medium Instruct (128k)
Phi3 Medium Instruct (4k)
Qwen2 57B-A14B Instruct
QwenCoder2.55 32B Instruct
QwenCoder2.5 14B Instruct
DSCoder 7B Instruct (y1.5)
Phi3 Mini Instruct (128k)
Mixtral 8x Instruct
Phi3 Mini Instruct (4k)
Phi3 Small Instruct (8k)
Qwen2.5 14B Instruct
Qwen2 0.5B Instruct
Qwen2 1.5B Instruct
CodeQwen1.5 7B Chat
DSCoder 33B Instruct
DSCoder 6.7B Instruct
Phi-4 Mini Instruct
Qwen2 7B Instruct
DSCoder 7B (y1.5)
PhindCL 34B v2
CodeQwenl.5 7B
Codestral 22B
Yil.5 34B Chat
1 6B Chat
DeepSeek-RI Yi1.5 9B Chat
DeepSeek-V3 V3
DSCoder 33B
DSCoder 6.7B
QwQ 38B
Qwen2 7B
Phi-4
33

---

Published as a conference paper at ICLR 2025
1.25
2.65
0.17
3.53
1.12
5.41
3.40
4.81
46.05
37.21
32.88 28.8
39.57
38.22
26.47
52.53
6.62
26.21
50.54
19.96
45.04
24.47
18.69
42.04
67.78
18.90
40.85 05.85
46.34
57.93
59.15
73.17
30.49
0.61
34.87
31.36
48. 48.50 50
50.04
54.63
51.71
47.85
36.25
43.00
41.48
59.29
64.38
60.05
56.74
52,42
48.85
31.81
33,44
47.34
49.85
51.85
50.93
46.94
34.19
23 23.47
21.54
35. .0
38.07
37.67
35.32
34.41
27.97
56.49
41.94
83.88
81.74
78.03
79.32
78.74
63.77
DSCoderV2 Lite DSCoderV2 Lite Instruct
StarCoder2 15B Instruct
In25 20B Chat
Intern M2.55 7B Chat
DSMoE 16B DSMcE 16B Chat
StarCoder2 7B
34

---

Published as a conference paper at ICLR 2025
Execution
Predic-
tion
3.11
38.28
34.45
3.03
0.03
0.00
0.07
0.69
0.02
0.12
0.80
1.93
3.62
0.28
0.75
4.31
0.03
3.53
2.13
0.75
0.84
1.83
0.97
0.22
0.50
0.07
0.05
Code Repair
52.67
63.76
35.38
25.71
1.28
0.00
27.03
1.28
6.69
0.00
12.02
47.61
92.31
17.36
18.16
80.15
2.63
57.52
44.53
7.97
9.32
9.32
21.00
2.70
29.66
2.56
0.00
Fill in the
blank
40.11
79.05
54.06
17.61
0.00
0.19
0.00
0.00
44.20
0.00
25.50
60.40
40.44
87.69
2.07
30.39
0.28
67.36
7.84
5.31
2.35
7.23
14.61
4.04
6.34
0.38
0.19
Code Comple- tion
Table 10: CodeMMLU few-shot experimental results.
11.59
86.59
70.12
62,80
0.61
0.61
10.37
1.22
1.22
0.00
67.07
50.00
93.90
27.44
48.78
6.10
1.22
50.00
14,02
23.78
19.51
25.00
31.71
10.37
7.32
1.22
0.00
Others
63.68
56.02
50.04
39.02
33.41
38.88
46.17
42.52
39.02
31.80
56.89
62.36
46.75
50.62
57.91
59.30
47.63
50.47
43.18
46.02
48.21
53.32
50.18
53.46
54.41
46.75
45.30
Software princi- ples
64.73
56.83
52.93
39.09
35.16
42.04
44.51
39.94
39.56
37.23
55.47
61.04
45.52
50.30
56.49
60.98
45.13
48.73
44.76
45.93
50.27
53.24
50.35
53.27
54.92
47.29
44.01
DBMS & SQL
70.99
67.68
58.52
50.89
44.02
50.89
52.93
50.13
47.84
44.27
63.10
67.68
53.94
60.81
65.65
69.21
54.71
53.44
53.44
54.96
57.76
61.07
58.52
58.78
62.09
54.20
53.94
API F Frame- works
86.31
79.32
84.88
72.04
62.62
72.04
79.17
74.18
72.04
54.07
86.02
87.59
75.89
81.74
87.02
87.45
75.04
78.32
74.75
71.90
75.46
84.74
82.17
79. .9
80.03
74.18
76.32
PL syntax
44.58
42.11
38 47
29.12
25.79
29.12
34.44
31.14
28.23
23.49
40.74
42.20
32.38
34.77
41.16
41.38
32.86
34.76
30.74
32.38
33.38
37.33
38.50
37.19
82
31.83
33.14
Phi3 Medium Instruct (128k)
Phi3 Medium Instruct (4k)
CodeLlama 13B Instruct
Codel Jama 13B Python
CodeLLamm Lamaa 34B Instruct
CodeLlama 34B Python
Codel Jama Instruct
CodeLlama 7B Python
Llama3.1 3.1 70B Instruct
Mistral 7B Instruct (v1)
Mistral 7B Instruct (v2)
Mistral 7B Instruct (v3) Mixtral 8x7B Instruct.
Phi3 Mini Instruct (128k)
Phi3 Mini Instruct (4k)
Llama3 70B Instruct
Llama3.1 8B Instruct
Llama3 8B Instruct.
Claude3 Sonnet
CodeLLama 13B
GPT-3.5-turbo
Llama3. 1 70B
Codestrrl 22B
Llama3 70B
Llama3.1 8B
Llama3 8B
GPT-40
Model
35

---

Published as a conference paper at ICLR 2025
1.90
1.14
0.00
2.91
0.07
1.58
0.25
1.13
0.07
0.00
1.22
0.00
0.02
2.60
0.02
0.00
0.08
0.00
0.17
0.02
0.17
38.77
14.59
10.60
21.28
15.73
23.22
1.28
0.00
8.04
3.98
1.28
2.56
1.28
1.28
0.00
0.00
0.00
1.28
2.63
1.35
2.63
56.65
19.26
2.54
43.68
0.14
3.05
13,01
5.26
24.99
0.00
9.02
0.28
0.89
63.74
0.00
0.09
0.05
0.00
14.09
1.03
4.23
56.10
42.07
35.98
21.95
14.63
37.80
25.61
7.32
0.00
1.83
0.61
0.61
0.61
1.22
5.49
0.00
0.00
0.61
0.61
7.93
4.27
52.22
55.43
54.49
54.27
40.34
39.10
52.81
49.53
52.81
40.41
43.03
39.10
38.15
46.46
44.93
34.72
31.15
48.50
50.04
54.63
47.70
56.83
55.19
53.06
56.35
42.46
40.17
54.93
48.51
54.19
40.93
43.73
40.06
41.68
49.79
45.86
35.59
35.87
50.33
53.48
53.90
47.85
64.12
65.14
61.58
62.85
51.15
46.06
62.34
58.02
60.05
48.60
51.91
49.36
52.67
57.51
54.71
43.51
41.73
59.29
64.38
60.05
52.93
78. 78.60 .660
64.88
82. 88
82.74
74.04
67.62
80.17
75.75
75.46
74.75
75.32
70.19
71.75
81.31
79.60
56.49
43.37
83.88
81.74
78.03
78.89 89
37.32
37.80
36.38
37.01
30.5 56
31.70
37.03
34.34
35.74
29.20 29.26
32.35
28.71
29.84 84
36.08
33.46
23.89
21.54
35.00
38.07
37.67
34.41
Qwen2 57B-A14B 14B Instruct
DSCoder 7B Instruct (v1.5)
Phi3 Small Instruct (8k)
CodeQwen1.5 7B Chat
DSCoder 33B Instruct
DSCoder 6.7B Instruct
DSCoderV2 Lite Instruct
InternLM2.5 20B Chat
StarCoder2 15B Instruct
DSCoder 7B (v1.5)
Qwen2 7B Instruct
CodeQwen1.5 7B
DSMoE 16B Chat
DSCoderV2 Lite
Yi1.5 34B Chat
Yi-1.5 6B Chat
Yi1.5 9B Chat
DSCoder 33B
DSCoder 6.7B
DSMoE 16B
Qwen2 7B
36

---

Published as a conference paper at ICLR 2025
Execution
Predic-
tion
3.26
41 41.95
2.70
0.00
2.45
0.00
0.03
2.32
0.77
0.53
2.84
3.53
3.98
2.65
2.15
1.62
3.70
0.00
2.81
4.53
1.92
2.13
3.21
2.45
3.28
2.46
57.
Code Repair
100.00
100.00
48.61
38.46
48.82
29.18
51.35
65.70
3.92
18.16
30.46
0.00
98.72
85.76
79.87
33,02
45.15
82.99
38.15
44.70
74.95
47.68
55.37
81.70
60.78
73.87
60.64
Fill in the middle
Table 1I: CodeMMLU Chain-of- Though with zeroshot experimentar results.
61 61.34
44.76
39.36
35.23
0.00
40.02
0.00
0.52
41.94
1.69
19.16
46.97
92.02
76.00
94.65
13.81
4.93
84.64
17.57
71.54
62.61
23.44
26.30
60.40
46.17
47.11
45.89
Code Comple- tion
0.61
39
69.51
37.20
56.71
20.12
0.00
0.61
79.27
0.00
39.63
89.63
19.51
73.17
11.59
59.76
19.51
85.98
23.17
72.56
75.00
37.80
57.32
33.54
80.49 19.49
85.98
85.37
Others
61.20
34.06
50.47
37.93
30.49
40.04
36.11
40.04
45.37
40.34
40.70
36.11
55.73
60.25
46.32
49.02
55.87
57.55
46.24
46.54
33.84
43.84
44.27
47.78
48.94 94
51.79
49.60
Software princi- ples
72.81
34.69
49.00
38.43
30.03
37.65
30.03
37.60
42.76
32.06
38.03
38.12
52.23
55.36
41.05
46.06
52.21
53.50
43.99
46.02
35.32
41.07
39.97
46.01
49.15
50.21
47.43
DBMS & SQL
71.76
30.03
60.81
45.55
42.24
50.64
46.06
50.89
54.45
51.91
50.64
45.04
63.10
67.18
56.74
55.98
64.12
65.14
55.98
54.20
50.64
51.65
52.67
55.73
58.52
57.76
57.76
API F Frame- works
41.90 .94
37.09
85.73
69.47
59.49
76.89
69.33
76.75
77.32
74.75
72. 72.90
68.19
85.73
86.16
76.60
78.60
86.16
78.89
75. 04
78.74
63.91
69.90
70.47
79.60
79.32
79.32
78.03
PL syntax
42.60
27.83
0.79
27.30
24. 49
30.06
27.17
30.05
34,00
23.8 87
30 80
30.8 85
39.84
41.64
31.48
33,54
40.50
40.10
31.98
33.41
26.53
31.45
32.11
35.16
38.22
37.30
35.69
Phi3 Medium Instruct (128k)
Phi3 Medium Instruct (4k)
CodeLlama 7B Instruct
Codel Jama 7B Python
KodeLlama 13B Instruct
CodeLlamm Lamaa 13B Python
Code Jama 34B Instruct
CodeLlama 34B Python
Llama3 70B Instruct
Llama3.1 3.1 70B Instruct
Llama3.1 8B Instruct
Mistral 7B Instruct (vl)
Mistral 7B Instruct (v2)
Mistral 7B Instruct (v3)
Mixtral 8x7B Instruct.
Phi3 Mini Instruct (128k)
Phi3 Mini Instruct (4k)
Llama3 8B Instruct.
Claude3 Sonnet
CodeLlama 13B
GPT-3.5-turbo
Llama3. 1 70B
Codestrrl 22B
Llama3 70B
Llama3.1 8B
Llama3 8B
GPT-44
Model
37

---

Published as a conference paper at ICLR 2025
2.97
2.50
3.03
1.42
1.27
3.53
3.70
4.18
5.31
4.66
3.25
3.08
0.00
3.10
2.08
3.29
0.44
2.48
0.13
2.20
0.00
2.63
3.58
3.75
2.30
0.08
60.57
31.60
43.83
27.89
21.28
36.66
57.73
58.14
60.64
68.75
28.76
19.58
0.00
77.51
55.72
29.11
3.92
31.46
0.00
76.65
64.62
62.99
14.48
60.64
24.91
1.35
30.62
29.92
41.99
36.59
13.72
21.28
40.16
77.64
76.00
76.33
68.11
75.95
0.05
47.39
27.38
45.66
0.00
28.51
0.00
36.17
0.80
55.47
75.90
49.98
53.31
0.05
67.07
57.93
43.29
0.61
37.20
27.44
72.56
75.00
52.44
87.20
65.85
71.95
0.00
85.37
0.00
50.00
0.00
45.12
0.00
29.27
29.88
91.46
81.10
79 79.27
75.00
1.22
40.92
40.04
53.10
45.51
29.61
41.50
45.15
54.12
48.29
47.92
45.08
49.45
31.58
36.47
42.30
45.08
38.07
41.58
29. 98
36.83
46.10
20.50
50.04
42.01
46.61
34.72
40.33
37.82
56.41
45.82
31.37
41.37
41.35
51.45
47.22
45.61
42.17
52.42
30.98
34.37
44.78
43.82
38.36
38.84
33.48
37.37
47.19
25.64
45.70
42.03
46.99
36.95
50.64
47.33
64.89
55.22
40.97
53.69
56.74
64.63
57.25
54.45
58.52
56.49
44.78
49.11
54.96
52.42
47.84
49.11
38.93
48.09
57.51
28.24
57.76
50.38
56.23
47.33
65.05
62.77
78.60
80.60
48.50
68.76
72.33
82. 88
77.60
70.04
73.47
60.49
68.90
79.32
79.32
71.33
75.75
49. 50
60.77
83.59
38.66
75.75
67.76
83.31
63.77
77.
30.21
30.74
36.74
35.23
22.14
28.15
31.93
36.90 96
32.32
35.69
32.19
34.45
26.51
29 29.44 44
35.24
33.22
28.50
31.54
21.25
23.81
33.78
22.22
35.00
31.30
34.44
27.77
Qwen2 57B-A A14B Instruct
DSCoder 7B Instruct (y1.5)
DSCoderV2 Lite DSCoderV2 Lite Instruct
Phi3 Small Instruct (8k)
CodeQwen1.3 7B Chat
DSCoder 6.7B Instruct
DSCoder 33B Instruct
Intern .M2.5 20B Chat LiterLMM.55 7B Chat
StarCoder2 15B Instruct
Qwen2 0.5B Instruct
Qwen2 1,5B Instruct
Qwen2 7B Instruct
CodeQwen1.5 7B
DSCoder 7B (v1.5)
DSMcE 16B DSMoE 16B Chat
PhindCL 34B v2
Yil.5 34B Chat
Yi-1 6B Chat
Yi1.5 9B Chat
DSCoder 6.7B
DSCoder 33B
StarCoder2 7B
Qwen2 7B
38

---

Published as a conference paper at ICLR 2025
B.4
CODEMMLU EXAMPLE
General knowledge MCQ example:
The following are multiple-choice questions (with answers) about debugging programming problem.
Question: Suppose we have an O(n) time algorithm that finds the median of unsorted array. Now consider QuickSort implementation where we first find the median using the above algorithm, then use the median pivot. What will be the worst-case time complexity of this modified QuickSort?
(A) O( log
(B) On²))
(C) ( log log
(D) O( logn)
Code Completion example:
The following multiple-choice questions (with answers) about programming problems. Question: Which solution below is the most likely to complete the following code to achieve the desired goal?
from typing import List
def has_close_elame aatee_elamente (numbers: List [float], threshold: float) boal: Check if in given list of numbera, are any two numbers closer to each given threehold.
>>> bas_clase_elemente ([1.0, 2.0, 3.0), 0.6)
False
>>>
2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
True
*.*
(A)
for in range (len (numbers)): Change range to len (nunbberz))
j range (nunbers))
if abs (nunbers [i] mumbers[]]] threshold:
return True
return False
(B)
return any (abs threshold for &, sip (nunbere,, nuabers (1:)))
(C)
for in range (lan (numbers) 1):
j range (nuuberr))
if abs (nunbers[1] [:] mumbers[]]] > threshold:
return False
return True
(D)
for idx, elen in enumerate (numbers):
for idx2, elen2
(nunbers):
if idx i= idx2:
distance if distance threshold:
elen2)
retura True
return False
Answer:
Fill in the blank example:
39

---

Published as a conference paper at ICLR 2025
The following multiple-choice questions (with answers) about programming problem with incomplete solution.
Problem statement: You are given an array of intervals, where intervals[i] [starti, endi] and each starti is unique. The right interval for an interval is an interval j such that startj >= endi and startj is minimized. Note that may equal j. Return an array of right interval indices for each interval i. If no right interval exists for interval i, then put at index i.
Incomplete Solution:
waadd_rig___iieeeeee (intervals); a len(intevvals)
[-1] for 1 in ra)ge(n): intervala [i] (tp.ppad
def binary_cearch
left, right ans float (Pinf")
right:
nid (left right) // 2 if intervale [mid] [0] ele:
mane, mid)
right mid
left mid
raturn ans
intervals serrals..orte. sort() for intervals:
fleat()inf
res [2]] = intervals [val] [2]
return rea
Question: The provided solution is missing part, which option below is the most likely to complete the solution and achieve the desired goal?
(A)
beaac [1])
(B)
float (
(C)
binary_aearch [1])
(D)
float (Pinf"): zee [1 [2]] (varr
Answer:
Code Repair example:
The following are multiple-choice questions (with answers) about debugging a programming problem.
The following code snippet is producing incorrect results: Which solution below correctly identifies the bug and repairs achieve the desired goal?
Question:
1 import util.*;
2 public class DETECT_CYCLE
3 public Kode hare node;
boolean detect_cycle (Node node)
4
tortoias node;
6 while (true) {
7
if Saare.getSuccessor false;
mull)
8
9
tartoine tortaise. gatSuccessor O:
getSuccessor () getSuccessor g
11
if (hare == tortoise)
zeturn true;
40

---

Published as a conference paper at ICLR 2025
(A) Modify line 6: G
(B) Modify line
hare Ilbare w.w.re. wwt.uccessorrrrgooooooooo null)
(C) Modify line 12:
getSuccessor ()
tortoise;
(D) Modify line 11: if (Objecter equala(bard tortoise))
Execution Prediction example:
The following multiple-choice questions (with answers) about programming problem. Question: Given code snippet below, which behavior most likely to occur when running the solution?
java.util.w public class Main {
public Seanner sc new Scanner (System. in); int sc.nextint(()
void (ainistring arga)
int B nextint(t
int T se. ():
T/A Systen out. (st.pr
(A) Memory Limit Exceeded
(B) Runtime Error
(C) Compile Error
(D) No abnormally found
C MODELS SETUP
In our experiment and study, we consider GPT-4o (OpenAl et al. 2024). GPT-3.5 OPenAAl 2023),
Claude-3.5, Claude-3 (Anthropic 20244, Metal lama Metal.lama 3.1 (Dubey et al. 2024), Matal.lama (Dubey
al. 2024), CodeLLaMA (Rozière et al. 2024), DeepScek AI, DeepSeck Coder, DeepSeek Coder
V2 (Guo et al. 2024b DeepSeek-Ar et al. 2024 Guo et al. 2024a). MistralAI, Codetral (Jiang et al.
2024h. Qwen2 Yang al. 2024), CodeQwen 1.5 (Bai et al. 2023
Yi (Al al. 2024). StarCoder2
Lozhkov et al. 2024a), InternLM (Cai et al. 2024). Phind [Phind
2023).
Table 12: Language Models Description
Model ID
Short Name
Link
GPT-o1-2024-12-17
GPT ol
OpenAl
RPT--3--iin-2025500133
GPT 03-mini
GPT-440-002400-13
GPT 4o
GPT-46-miii-2004--77118
GPT 4o mini
GPT-3.5--urbo-166-00613
GPT-3.5-turo
1unule-35-sommee-222411000
Claude3.5 Sonnet
Sluude-3.-ssomee--200
Claude3.7 Sonnet
Anthropic
Claude3.5 Haiku
Claude-3-hand Seade- 2-220222000-000007
Claude3 Haiku
41

---

Published as a conference paper at ICLR 2025
Claude3 Sonnet Claude3 Opus
Claud=33opuss222240222
codellama/CodeLLlam--bll
CodeLlama 13B
codellama/Codelllnee
Instruct-hf
Instruct
13b-Instruct-hi
codellama/Codelllamaa
Code 13B Python
Sanellamm/Codellllag
CodeL.lama
Python-hf
13b-Python-hi
codellamaa(CodeLLaaaa--BBba
CodeLlama 13B
code codellama//Cdeelamg
13b-hf
codellama/CodeeLlama-BB3
CodeLlama 34B
codellama/CodeeLama
Instruct-hf
Instruct
lab-Instruct-hi
Codellama/CodeLllma---
CodeLlama 34B
Modellama/Codeelanee
Python-hf
Python
34b Poth-yytho-hf
Madellama/CCdelllama---
CodeI.lama 7B In-
Instruct-hf
struct
Ib-Instruct-hf
codellama/CodeLlama-T
CodeLlama Python
7B
Codellama/CodeLlamm
Python-hf
7b-Python-h
meta- Detta-lama/Mett-llamms
Llama3 70B
meta-llama//Met-
70B
Llama-3-70B
mett-llama/Mett-llamee 70B-Instruct
Llama3 70B In-
Martalllma/Mettt
Metal.lama
struct
1iima-3-00B-nnstuu
meca-llana/Meta--amaa--3-
Llama3 8B
meta-llama/ Maeaaaaa---a---- Meta-
Llama-3-8B
motalllama/Mett-Lamm---
Llama3 8B In-
meta- Meta-
8B-Instruct
struct
Llama-3-88 3-8 Instruct
metallaaaa/Mttt-Llmm33.11 70B
Llama3.1 70B
meta- meta-llama/Meta
Llama-3.1
metaallamaaMetaJJJmm3..tt
Llama3.1 70B In- struct
meta-Ilama/Meta
70B-Instruct
Llama-3.1-701
Instruct
metaallama/Mett-llamees
Llama3.1 8B
meta Hama/MMta-
8B
Llama-3.1-8B
metta-lamaa/Metalllaa3331
Llama3.1 8B In-
meta-llama/ Meta-
8B-Instruct
struct
-iima-3.-8B-IInstruee
meta-llama/Mett-Llame--
Llama3.1 405B Instruct
meta-llama/Meta
405B-Instruct
Llama-3.1-405B-
Instruct
meta-llama/Meta-lllaan Instruct
Llama3.3 70B In-
meta- meta-llama/MMtar
struct
Ilama-3.3-700
Instruct
mistrala/MMiitrall-B--issss
Mistral
7B
DistralaaiMistral-7755
v0.1
Instruct (v1)
Instruct-v0.1
Mistral
mistralai/MMistaa-----nnnnett
Mistral
7B
mistralau/Mistral-775
v0.2
Instruct (v2)
Instruct-v0.2
mistralai/Miistal--BBBBmo
Mistral
7B
mistralat/Mıstral 7B-
v0.3
Instruct (v3)
Instruct-v0.3
mistralai/Mixxuml887777 Instruct-v0.1
Mixtral 8x7B In-
mistralai/Mixtral
struct
8x7B- 8x7B-Instruct-v).
mistrlli//Codestra-22B--111
Codestral 22B
Imstralat/Codesrri
22B-v0.1
microsoft/phi-4
Phi-4
microsoft/hhi-4
microsoftPhi--mmiin-snttte
Phi-4-mini-
microsotUPPh-440
Phi
instruct
instruct
microsoft/Phi--meddiu.
Phi3 Medium In- struct (128k)
microsoftPhi-33
128k-instruct
medium-128k. 128k-instruet
microsoft/Phi---meddu instruct
Phi3 Medium In-
microsottPhi-3.
struct (4k)
medium- 44-uum----inttuuc
42

---

Published as a conference paper at ICLR 2025
microsoft/Phii---ma 128k-
Phi3 Mini Instruct (128k)
microssoftPhii--miin
instruct
18k-instrruct
microsoft/Phi---minn-k.
Phi3 Mini Instruct (4k)
microsott/hhi--minnn
instruct
4k-instruct
microsoft/Phi-3-mall--8-
Phi3
Small
Pecrosot//hhi--ssmee
instruct
Instruct (8k)
8k-instruct
PhinD
Phind/Phind-CooeLLlam--
PhindCL 34B v2
Phind/Phind-
34B-v2
CodeLlama-34B-v2
Qwen/CodeeQwenll5577
CodeQwen1.5 7B
Qwen/CodeeQwen
CodeQwen
7B
Qwen/Codeeweenl.5-77
CodeQwenl.5 7B Chat
Qwen/CodeQwenl.5
Chat
7B-Chat
Qwcn//Qwen2.5-Cdder333B
QwenCoder2.5
Qwen/Owen2.5-Cooee
Instruct
32B Inst
32B-Instruct
Qwen
Qwen/(wen2.5-Codere
QwenCoder2.5
Qwen/Qwen2.55Codee
Instruct
14B Inst
14B-Instruct
Qveen/QwQ:3BB-Previee
QwQ 32B
Qwen/Qwen2-0.50-
Instruct
Bwen/QWeen--055-IImtttee
Qwen2 0.5B In-
Qwen/Q wween/Ownn-00.5.
struct
Instruct
Qweem/Owen22115--IInsmttt
Qwen2 1.5B In-
Qwen/QQeeeeee Owen2-1.5n
struct
Instruct
Qwenn/Qwen2-57B-AA1BB
Qwen2
57B-
Owen/Qwen2-57B
Instruct
A14B Instruct
A14B-Instrucl
Qwen/Qwen2-77B
Qwen2 7B
Owen/Qwen2-7B
Qwen/Qwen2-77-Instte
Qwen2
7B
Qwen/Qwen2-7B:
Instruct
Instruct
01-ai/Yi-1.5-34B-O
Yi1.5 34B Chat
01-ai/Yi-1.5-344 Chat
Yi
01-ai/Vi-1.5-GBB-ha
Yi-1.5 6B Chat
01-ai/Yi-1.5-6BBB
01-ai/Yi-115599B-Cha
Yi1.5 9B Chat
01-a//Y-115-96-CCm
deepseek-ai/Deeppseekkk decpseck-ai/deepssee-33
DeepSeek R1
deepseek-ai/R1
DeepSeek
DeepSeek V3
deepseck-ai/DeepSeee
V3
deepseek-ii/deesseek.coee 33b-base
DSCoder 33B
deepseek-ai/deepseek
coder-33b-base
deepseck-ai/deeeeeee
ai//eeessee-ccodrrr 33b-instruct
DSCoder 33B In- struct DSCoder 6.7B
deepseck-aa/deepseekk
DeepSeck Coder
coder-33b-instruct
beepseek-ai/deepseee-ooo 6.7b-base
deepseek-ai/deepseekk
coder-6.7b-base
DSCoder 6.7B In-
deepseck-aaddcepseeek
6.
6.7binnsructt
struct
coder-6.7b-instnue
paassee-aaiideepsee-coooee 7b-base-vi
DSCoder 7B (v1.5) DSCoder 7B In- struct (v1.5)
deepseck-aa/deepseekk
coder- coder-7b-base-vv..
deepseek-ai/deepseeen ai/deepseek.coder
deepseck-auddeepseek
7b-instruct-v 1.5
coder=-bbiinttructvv13.
deepseek-ai/ www.aaaeeeeeee-Cooeeeeeeeeee
DSCoderV2 Lite
deepseek-ai/DDeppeeel
V2-Lite-Bass -ase
Coder--V-LitteBaass
deepseck-ai/DeeppSeeee ai/DeepSeek-Cooee V2-Lite-Instruet
DSCoderV2 Lite Instruct
(tepseek-aa/Deeppa
Coder- P2cer-V2-L0 V2-Lite.
Instruct
DeepSeek MoE
deepseek-aiideeepseee 16b-base
DSMoE 16B
deepseek-ai/deepsee
moe-16b-buse
DSMoE 16B Chat
deepseek-ai/deepseeek
16b-chat
moe- mnoe-16b-chat
43

---

Published as a conference paper at ICLR 2025
internlmminnermmmm2 5-200-
InternLM2.5 20B
intemin/inteemlm2 5-
InternLM
chat intemlnnimtermmm22557777 5-ahcchat
Chat
20b-chat
InternLM2.5 7B
intemlm/intemmm2 5-
Chat
7b-chat
bigcode/starcooder--555b instruct-v0.1
StarCoder2 15B
bigcode/starcoder2-
StarCoder2
Instruct
15b-instruct-v0.1
bigcode/starrooder2-77
StarCoder2 7B
bigcode/Starcooder2277
44

---

Published as a conference paper at ICLR 2025
C.I PROMPT LIBARARY
Filtering prompts: LLM-based filtering for ranking questions" completeness, coherence, and clarity.
Quality filtering prompt:
Please the folloving question based three criteria, with from for each criterion (shere 1 is the lozest and 5 is the highast). No explanation needed:
1. Completeness:
Doea queation atand alone provide enough information independently?
Avoid including any inages, links, exterual referencee.
Coherence and Clarity:
the question phrased clearly, with proper grannar?
Is there any anbiguity or confusion in the varding?
Relevance:
Is the question diractly related to softeare devalopment programning issues?
Doea it involve technical challenges, concepts, or tools commonly used software progranning?
Question:
"*" n**
Data creation prompts: Prompt used for synthesis distractor for real-world task:
Code Repair distractor creation prompts:
After extracting statement from buggy version, we use LLMs to rewrite new version of that statement. We command LLMs assume the bug is located in the assigned line and their target is correct that line. Here is the prompt:
Givan buggy Python code snippet, will be asked to debugging the code.
rer
def cruneete_aunber (nunber: float) -> flost: raturn nunber (number X 1)
cer
Let assune the bug is located in this line: (number 1) t)
ler return
Adjust this line in order to solve the bug.
The re-vritten line must be syntactic correct, executable and wrapped in de ere
brace.
Don't give any details. Revritten line: return number X 1.0
Givan buggy Java cade anippet, 9ill asked debugging the code.
(r(fcode
Let the
located in this line:
cri(line) Adjust line The re-vritton line must be syntactic correct, executable and vrapped in
bug.
Don't give any details. #OA Revritten line:
We executing the problem with given test cases. Our target to create reasonable false that would require deep interpretation. Follow by an LLMs based filter pick from pool of negative answer the most likely able solve the buggy problem. This result a set of confusing negative answer. Those reasonable false sample with exccutable (and if they can pass through few testcases) is golden negative answer.
45

---

this (line) help me generate 3 version change in this code and each output should in only. Published as a conference paper at ICLR 2025
Fill in the blank distractor creation prompt:
From correct solution from lectcode, randomly mask line/a block of code and generate false answer (for multiple choice) from LLMs:
Folloving this code: {code]
prepare
nultiple choice questioas answering
ao want ta make amall change on this 1ine it
Don't give any datails
Experimental prompts:
Prompt used in CodeMMLU evaluation.
Zero-shot prompts
General knowledge MCQ test set:
The felloving are multiple- choice questions (vith answers) about software development.
Question: (queation)
(multsple_choicee)
Answer:
Code completion:
The folloving multiple choice questions (vith ansvers) about software development.
Question: (question)
(multspla_chotcess
Answer:
Fill in the blank:
The folloving are multiple-choice questions (vith ansvers) about a programning problen with incomplete solution.
Problem statement:
{queation)
Incomplete Solution:
(codebase)
niseing part, option below 4 nost likely to complete the solution and achieve the desired goal?
Question:
provided
chiices)
Answer:
46

---

Published as a conference paper at ICLR 2025
Code Repair:
The Iollowing are multiple-chool choice questions (vith anavere) about debugging progranming problen.
Question: The implementation below producing incorrect resulte. Shich solution belou correctly identifies the and achieve the desired goal?
{question)
(multiple_choices)
Answer:
Defect Delection:
The falloving are multiple-choica questions (vith anavers) about programning problema.
Question: execute it? (question)
enippet which likely occur
chottsp
Ansxer:
47

---

exactly solution, you may Published as a conference paper at ICLR 2025
Few-shot prompt
General knowledge MCQ lest set:
The folloving are multiple choice questions (vith ansvers) about software davelopment.
values, order norted array integers is guaranteed to not contain duplicate for specific value which the folloving algorithms + is the most efficient for this task?
Question:
(A) Bubble Sart (B) Linear Saarch (C) Insertion Sort (D) Binary Search
Answer: The
(D).
Question: (question)
chuicep)
Ansser:
Code completion:
The felloving are nultiple-choiee questions (vith answers) about progranning problens.
Question: Which solution snippet to achieve the desired goal? 1el python from typing Inport List
likely coupletion following
def tuo_sun (nuns: List [int), target: int) List
Given array integers nuns integer target, return indices 4 two numbers such that they add to target. You may assune that each input
not the same element tuice.
(12,7,11,16)
[0,1]
txo_sum ([3,2, 4) 6)
[1,2]
>>> tuo_aun ([3,3], 6)
[0,1]
(A) << 'python
len (nuns)
for in range (n 1):
for nuns [i] nums [j] target:
(1 1, n):
return [1, ]]
return 0
(B) python
for num nune: if target
raturn [nunc, index (num), nuns. index (target -un)]
0
48

---

Published as a conference paper at ICLR 2025
(C) re pyttho for i in range (lan (nums)): nuna return [i, i]
target:
return 0
(D) "python nun_dict 0
for 1, nun in anunerate (numa): target
nun_dict:
return [aum_dict [target nun], i]
nun_dict return []
Answer: The
Question: Which solution
likely completion following
snippet to achieve the desired goal?
"python
{question) <<c
(multiple_choicen
Ansxer: 113
Fill the blank:
The falloving are miltiple-choice questions (with ansvers) about programning problen with incomplete solution.
Problem statenent: You are given an array of intervals, where intervals[s] [starti, endi]
unique.
The right interval for an interval i is interval j euch that startj >- endi startj is minimized.
Note that 1 may equal Return an of right interval indices for each interval i.
right intarval exista for interval 1, than
inder
Incouplete Solution:
pythen!:
sind_right_intere (intervals):
(en(intervals) ree [-1] range(n): intervels(i)
def [pperval(()...pperd((
(ole):
laft, right D, float (loat()int!)
while left right:
(left right)
if (nidr >- ele: ans min mid) right mid
else: left mid +1
return and
intervale sort(
Question: provided solution missing part, Which option belor the most likely to complete the solution and achieve the desired goal?
49

---

Published as a conference paper at ICLR 2025
(A) ppython
(inary_search(111]]
if val I- float()int
[e] intervale (var)
(B) python
if val I- float ('inf();)));
res [2]]] [val]
else:
continus
(C) pyython
stnary_search
if val !- float('inf)) res [2] 1] intervals [val]
(D) re python
if val 1- flost flost('inf));
r2][i[2]] intervale[val] [2]
else:
continue
Answer: The anazer is (A).
Problem statement: (question)
Incomplete Solution:
{codebase)
Question:
The provided solution is nissing part, Which option belos is the schieve the desired goal?
likely complete
{moltsple_chotcess
Answer:
50

---

Published as a conference paper at ICLR 2025
Code Repair:
The lollowing are multiple-chooe questions (vith anavere) about debugging progranming problem.
Question: The implementation belox is producing incorrect resulte.
Which solution belev correctly identifies the bug and it achieve the desired goal?
def tvo tvo_eun (ouns, target):
complement_ in enumerate ; for complement target num
(())
complement_nap if complement in complament.nap: retura [complenent_ngg [soppleeeenntt]] i]
retura None
(4) Renove line 5.
(B) Ranove 5. Add line complenent._ag) [nun] i
(C) Modify line 7: [i,
(D) Renove line
5. Add at line 7: rer if len (numa)
return Kone complement_ [nun] iter
Ansger: anazer (8).
Question:
The implemantation below is producing incorract results.
solution belew correctly identifies bug repairs achieve desired goal?
(question)
{choicea)
Answer:
Defect Detection:
The folloving are multiple choice questions (vith ansvere) about progranning problen.
Question: Given code snippet belox, which behavior noat likely occurr when execute it? "python
def chkPair eize, x):
for i in range (0, size 1): j range 1, size): if (ACI] + A[j] == x): return
return
contain issue.
(B) Nemory Linit Exceeded
(C). Compile error
(D). Runtime Error
51

---

Published as a conference paper at ICLR 2025
Answer:
The anszer is (A).
Qusstion: Given code snippet belox, which behavior most likely to occurr when {question)
(multiple_choices)
Answer:
52

---

Published as a conference paper at ICLR 2025
Chain-of-Thoughh zero-shot prompts
General knowledge MCQ set:
The folloving are multiple choice questions (vith ansvers) about software devopnent.
Question: (question)
(moltsppi__hhotces)
Answer:
step by step.
Code completion:
The fellowing are nultiple choice questiona (vith answers) about progranning problene.
Question: Which solution below the likely completion the follozing snippet to achieve the desired goal?
1et python
{question)
(multiple_choices]
Ansser: Lat think by step.
Fill in the blank:
folloving are multiple-choice questions ansvers) programning problen with uncomplete solution.
Problem statement: (question)
Incomplete Solution: (codebase)
Question: The provided solution is missing part, Which option below is the nost likely to
complete the solution achieve the desired goal?
chotces)
Ansger:
step by step.
Code Repair:
The folloving are multiple-chool choice questions (vith answere) about debugging progranming problen.
Queation: implementation below producing incorrect reaults. Which solution belev correctly identifies the bug and repairs it to achieve the desired geal?
{question)
(moltsple_.cooccas
Answer:
Let?a step by step.
53

---

Published as a conference paper at ICLR 2025
Defect Delection:
The Iollowing are multiple-choos questions (vith anavere) about debugging progranming problen.
The algorithn implementation belox producing incorrect resulte; Which solution belev correctly identifies the bug and repairs it achieve the desired goal? {question)
(multiple_choices)
Answer: Let's think step by step.
54

---

Published as a conference paper at ICLR 2025
Chain-of-Thouggh few-shot prompts
General knowledge MCQ lest set:
The following are multiple choice questions (vith ansvers) about software devopment.
values, order sorted array integers is guaranteed to not contain duplicate for specific value which the folloving algorithms + is the most efficient for this task?
Question:
(A) Bubble Sart (B) Linear Saarch (C) Insertion Sort (D) Binary Search
Answer: Let's think
Binary Search is divide conuuer
algorithn that works by repoatedly dividing the search interval in half and searching
appropriata half. Since array already sortad does centain duplicate value, this algorithm is optimal to find the desired value. The anseer (D).
for the value
Question: (question)
(moicsple_choices)
Ansser: Let'a stap by atep.
Code completion:
folloving are multiple quastions answers) about progranning
problen.
Which solution belox is the likely completion the following code snippet to achieve the desired goal?
Question:
r4r pyython
daf is_vouel (char: str) -> bool:
Checks If the imput character 1s vovel.
(8) << "python
return char. harrloover () vis_vooll
rec
(B) (< "python vovels set("aeion")
char lovar vouels
(C) ( pytton
vovels set set("aa))"")
return char char. upper in vovels
(D) "python
vovals *asion"
return char.count (vosels)
Let's think step by step. The geal is to write function ls_vovel (char: str) bool that checks if the input character char is vovel. The solution correctly converts the input character to lorercase and chocks it is in the set of vovels.
Answer:
The anaver is (B).
Question: Which solution belox is the most likely completion the follozing code snippet achieve the desired geal? te1 "<lpthonn
[question)
fCr
(multiple_choiite choices))
Ansger: Let 's think step by step.
55

---

Published as a conference paper at ICLR 2025
Fill in the blank:
The Iollowing are multiple-chhoee choice questions (vith anavere) about 8 programiing problen uncomplete solutien.
Problen statement: You are given an array intervale, where intervals [starti, endi] each starti
unique.
The right interval for an interval in an interval J auch that startj >= andi startj is mininized.
Note that i may equal j. Beturn an array of right interval indices for each index
If right interval exists for interval i, then
Incomplate Solution:
python''e
def itteergtttttterlllllaaaaeaaaa (interra)))
- len (en
[-1]
for i in ra)ge(m): (nterval=(s).append) def (elrry))aarcheee)) left, right - 0, n-1 floot('inf'))
while left <= right:
right)
(ntervals(mid] >- ele: ans mans, mid) right
else:
left mid
return ans
intervals sort for 1 in intervals:
Question: The provided solution is nissing part, Which option belos is the likely complete the solution and achieve the desired goal?
(A) "pyyhhnn
val binary. (1
if val I- flost('infe)
rea[i[2]] [itervais[vaa]]
cer
(B) pyyton
val ()
[2]] [va
elee:
continue
(C) "yythoo
val bimary_cearch
float ('inf') rea intervals [2]
(D) pythhn
if val I- float
ree [2]]] intervale [val] [2]
else:
continue
56

---

Published as a conference paper at ICLR 2025
Answer: Let's think step by step. The incomplete solution first sorts the the sorted interrals. For each interval, rinds the right interval using binary aearch.
intervala
This option
right
index using binery search and
updates the result array accordingly.
The option (B) is similar (d), but it does not increment the index when finding the right interval index. This could lead to incorrect results.
The option (C) increnents the index when finding the right interval index. index
problen statement
right interval, not the offeet fron the original index.
original interval right + interval, which could lead to incorrect results.
The eption
same
The ansver (A).
Problem statenent: {queation)
Iaceuplete Solution: (codebase)
The provided solution is nissing part, Which option below is the likely complete the solution and achieve the desired geal?
Questioa:
Ansger: Let?s think by step.
57

---

Published as a conference paper at ICLR 2025
Code Repair:
The Iollowing are multiple-chooe questions (vith anavere) about debugging progranming problem.
Question: The implementation belox is producing incorrect resulte.
Which solution belev correctly identifies the bug and it achieve the desired goal?
def tvo tvo_eun (ouns, target): complement_ (())
in enunerate for complement target num
complement_nap if complement in complament.nap: retura [complenent_nag [soppleee))n i]
retura None
(4) Renove line 5.
(B) Ranove 5. Add line complenent. [nun] i
(C) Modify line 7: [i,
fer
map <<c
(D) Renove line 5. Add at line 7: rer if len (numa)
return Kone conplement_ [num]
step by step. bug number is added to the complement. before checking its complement already exists, which can lead to incorractly matching a number with itself. To fix this, the nap checking
Ansser:
cade occurs
number
complement. Solution (B) does exactly this by noving the line that adds the current nunber to the complement check, ensuring logic salf matching errors.
only
intended
The anszer is (B).
Question: The inplemantation below is producing incorract reaults. solution below correctly identifies bug repaire desired goal?
{question)
{choices)
Answer: Let?a think atep by atep.
58

---

Published as a conference paper at ICLR 2025
Defect Delection:
The Iollowing are multiple choice questions (vith anavere) about progranning problen.
Question:
Given code enippet below, which behavior nost likely to occurr when python def chiPair
x):
for i in range (0, size 1): j range 1, size): if (A[1) A[j] x): return
(1). The code contain no issue.
(B). Nemory
(C). Compile
(D). Runtime Error
Answer: Let'a think step by step. The code appeara to have no issues with typical inpute and function expected. correctly pairs 4 elenents whose sun is
ansver (A).
Question: Given cade snippet belor, which behavior nest likely to accurr when {question)
(multiple_choices)
Ansser: step by step.
59


Reviews:
[{"review_id": "IsukoWocCO", "reviewer": "Reviewer_xQZ6", "summary": "The authors propose CodeMMLU, a new benchmark designed specifically to evaluate LLMs' code understanding capabilities. This benchmark includes over 10,000 questions covering various domains and programming languages. The question types include knowledge-based evaluations (such as programming language syntax, API usage, and software development principles) and practical programming tasks (such as code completion, fixing, and defect detection). The authors test various LLMs using this benchmark and provide insights into model performance, prompting strategies, and their correlation with practical programming skills.", "questions": "1. Can the authors provide additional insights or data to illustrate the correlation between the model's performance on CodeMMLU and its real-world application in software development environments, where code generation is more prevalent?\n\n2. In line 212, the authors claim to use a deep learning-based filtering model to automatically remove low-quality or irrelevant questions. I do not understand how the authors ensure data quality. Is it solely based on prompts? What model was used for checking? Are there any rule-based verification methods involved? I have reviewed sections A.2.1 and A.2.2 of the appendix and found no clear explanations. In line 1043, the authors state, “Followed by a manual validation step to ensure their appropriateness for the benchmark.” Did the authors really manually review over 10,000 samples? This seems hard to believe.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. CodeMMLU abandons traditional code generation evaluation methods and adopts a multiple-choice question format, shifting the focus from code generation to code understanding to evaluate LLMs' code understanding abilities.\n\n2. The CodeMMLU benchmark includes over 10,000 diverse questions with broad coverage and high-quality sources (such as GeeksforGeeks, LeetCode, etc.). The authors have put in substantial work overall.\n\n3. The authors conduct extensive experiments on CodeMMLU, providing experimental insights into multiple aspects such as selection bias in multiple-choice questions and correlations between LLMs' software knowledge and real-world applications. The authors also provide numerous tables, figures, and other visualizations to help readers understand the paper.", "weaknesses": "1. In the Related Work section, the authors simply list some code evaluation benchmarks without clearly articulating the specific differences between CodeMMLU and each existing benchmark, nor do they explain what issues CodeMMLU addresses that current benchmarks do not. In contrast, there are many comprehensive and thorough benchmarks for code understanding and code generation, such as CodeXGlue, XLCoST, xCodeEval, CodeScope, LiveCodeBench, and BigCodeBench. Notably, CodeScope (ACL 2024) has constructed a code understanding benchmark that includes four tasks and multiple-choice questions. Additionally, LiveCodeBench addresses the issue of data leakage through dynamic data set updates. CodeMMLU lacks detailed and thorough comparative analysis of key related works (a good example of related work analysis can be found in ClassEval). Moreover, the paper does not offer any solutions to the data leakage problem mentioned in line 36. Overall, although the paper involves considerable effort in data labeling and other aspects, it lacks novelty, with many conclusions already available in previous literature. The innovation and actual contributions of the paper remain unclear.\n\n2. In paper, the authors discuss how LLMs are sensitive to the order of options in multiple-choice questions, which can lead to fluctuations in performance and thereby affect the accuracy of the results. Although the issue is acknowledged, unfortunately, no strategies for eliminating this bias are provided. This raises concerns about the reliability and robustness of benchmark tests. Moreover, the paper \"Large Language Models Are Not Robust Multiple Choice Selectors,\" published at ICLR 2024, has demonstrated that using multiple-choice questions to evaluate LLMs is not stable and introduces significant biases. Given this, I am curious about how the authors address this issue.\n\n3. The overall writing of this paper needs improvement, particularly in the areas of data handling and presentation where essential detailed explanations are lacking. Specifically, in Figure 3, titled \"Overview of the CodeMMLU Data Creation Pipeline\", the authors fail to clearly explain the process of constructing multiple-choice questions from various data forms after filtering. Moreover, the inclusion of the LLMs evaluation in Figure 3 is not well-explained. In line 319 of the text, although four models \"GPT-3.5, GPT-4, Claude-3-opus, and Claude-3.5-sonnet\" are tested, the manuscript inaccurately mentions \"including three proprietary models.\" Additionally, the citations for Claude-3-opus, Claude-3.5-sonnet, and Qwen2 among others are incorrect and urgently need correction. Concerning the topic categorization in Table 2, the paper does not provide a valid explanation or methodology for the classification. Several key steps in constructing the evaluation benchmarks are also lacking thorough explanations and supportive descriptions, even in the appendix. Overall, these issues raise concerns about the quality of the manuscript, and it is recommended that the authors give more attention and detailed exposition to these critical areas in the revision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w4aAcFGN6C", "reviewer": "Reviewer_aVYU", "summary": "his paper presents CodeMMLU, a collection of almost 20,000 multiple choice questions and answers on coding, collected from the web.\n\nThe paper covers three areas of related work: benchmarks for program comprehension, models for program comprehension, and multi-task question/answering benchmarks.\n\nThe paper collects programming-relaed multiple-choice questions from web sites like GeeksForGeeks, W3Schools, and Sanfoundry. The questions cover syntactic (programming languages and API) and semantic (algorithms, design, OO) knowledge, as well tasks (code completion, fill-in-the-blank, defect detection, and code repair).\n\nThese 20,000 questions were evaluated on 35 \"open source\" code LLMs. \nKey reported insights include:\n\n- Performance on knowledge questions and task questions is correlated\n- LLM's preference for certain answers (e.g., avoid 'A') in multiple-choice questions is also present in code models\n- Chain-of-Thought is not helpful for these questions\n- Turning HumanEval into multiple choice questions changes model performance", "questions": "Do all questions have four alternatives (one correct and 3 distractors)? At any rate, a random guesser would get 25% right, which makes the results in Table 3 less impressive.\n\nWere the questions or the sources they were derived from included in the training data of the language models benchmarked? How are such risks mitigated?\n\nCan you explain the \"CodeMMLU\" column in Table 3? I thought it was the overall performance, but it's not explained. How can the Meta-Llama-3.170B-Instruct overall be 60, while the other three columns are all above 64? (Also please align around decimal point instead of centering numeric columns).\n\nCan you explain how you will distribute the dataset and under what licence the original material was made available, and which license you intend to use?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The dataset is impressive, with almost 20,000 questions, and represents a substantial amount of (manual) work\n\nThe grouping of questions is meaningful\n\nThe paper is easy to read and follow\n\nThe dataset puts code language model performance in a different perspective.", "weaknesses": "While I liked the paper, my main concerns are:\n\n- While filters and manual processing are applied, these are described only very briefly. As a result, the quality of the questions is unclear. It appears the paper favors quantity over quality. Fewer questions but of guaranteed quality would have been better.\n\n- The questions for the 'real world tasks', e.g., for defect detection or line completion, are very artificial\n\n- The treatment of multiple-choice bias (models prefering to avoid option A) in the paper is unsatisfactory\n\n- It is unclear to what extent the LLMs were exposed to the underlying resources for this new data set (leakage). This risk is not discussed nor mitigated.\n\n\nThe writing and presentation are generally good, yet is sloppy at places (the abstract speaks about \"over 10,000\" questions -- there are 19,900, which is more like 20,000, 3.2 speaks about \"five distinct\" tasks, but there are four, there is no table 8 (only a figure 8), ...). It is confusing that the text summarizing table 2 gives very different numbers from what is in the table ('over 3000' when in the table it appears to be closer to 5000, and 6000 when it is in fact 7000). I'm not sure why section A.3 is entitled \"visualization\" (nothing is visualized -- examples are given).\n\nThe filtering process is described, but the exact numbers involved (before/after filtering) are not provided. The filtering involves various manual steps -- applied to how many cases? Deep learners are used here, but no details are provided.\n\nReferring to the tasks as \"real world performance\" is misleading. The tasks are still highly artificial. Concerning the tasks themselves:\n\n- I was surprised to see 'code completion' tasks based on HumanEval -- HumanEval suffers from many problems. There is a vast amount of literature on LLM assisted code-completion using data that is better than HumanEval.\n- The defect detection task appears to be about predicting the correct execution result -- which is a different task from defect detection. Again, there are lots of defect benchmarks around, with real bugs collected from actual systems (e.g., defects4j). It is not so clear what these mutliple choice questions add to that, especially with the weak distractros (like compile error, \"internal error\" (??))\n\nThe reporting of the results about preference for option A (figure 7, table 4) is very minimalistic. The bias is stated, but not really studied / explained, nor are mitigation measures such as proposed by Zheng et al applied. The paper writes that 'we experimented' with multiple answer order, but what exactly was done is unclear.\nI must say these findings also undermine the whole endeavour. If the multiple choice format itself is a problem, what is the point of having a large multiple choice data set?\n\nIt is unclear how the dataset will be distributed. I would believe some of the data is copyright protected (e.g., W3Schools). This would mean you cannot redistribute their multiple choice questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SmpP60jszm", "reviewer": "Reviewer_6QYn", "summary": "This paper proposes CodeMMLU, a benchmark for evaluating LLMs in code understanding using multiple-choice questions. CodeMMLU consists of a group of different tasks across knowledge-based tests and real-world programming questions. The authors evaluated 35 LLMs on the CodeMMLU benchmark, and the results suggests that the performances of LLMs on CodeMMLU is not always consistent with the performances on code generation benchmarks.", "questions": "- The decline in accuracy with COT prompts is interesting. Perhaps it's better to analyze the LLMs' answers with COT in detail.\n\n- In section 3.2, why is predicting execution output under the same category as defect prediction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+: The authors proposed a new benchmark in code understanding, which is long ignored in LLM for code evaluation.\n\n+: CodeMMLU consists of a wide variety of code comprehension tasks, from syntax/semantic understanding to code repair and defect prediction.\n\n+: The authors conducted extensive experiments on CodeMMLU with various LLMs.", "weaknesses": "-: For most tasks, the authors use LLMs to generate distractors. The quality of these generated distractors should be discussed.\n\n-: The code completion and fill-in-the-blank tasks are more related to code generation instead of code understanding. Especially, the code completion task is based on the existing HumanEval dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o7EABv7cvF", "reviewer": "Reviewer_wsRv", "summary": "The authors present CodeMMLU, a multiple-choice question-answer benchmark for code, consistenting of thousands of code/software-related questions, spanning syntactic knowledge (e.g., API and Frameworks, PL syntax), semantic knowledge (e.g., OOP, compiler design), and real-world tasks (e.g., code completion, code repair). This is inspired by the MMLU evaluation set used in NLP as well as programmer comprehension behavior models (Shneiderman & Mayber, 1979). The knowledge-based tasks are derived from W3Schools, Geeks4Geeks, and Sanfoundry, with LLM-based filtering. The real-world tasks are derived by re-purposing existing evaluation sets, with additional steps for synthesizing distractors with LLMs and execution-based filtering. The authors benchmark a large number of closed-source and open-source LLMs with CodeMMLU.", "questions": "Please address the concerns raised in the Weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This is a very large and diverse evaluation set, spanning many different topics, concepts, tasks, and programming languages. This could potentially be useful for the research community, providing a way forward for hill-climbing to improve code understanding in LLMs.\n- The authors evaluate several open-source and closed-source code LLMs on this new benchmark, reporting numbers separately for syntactic, semantic, and real-world tasks. This is useful for understanding the weaknesses of current models, also paving a path forward for improvement.\n- I particularly like the way the authors have generated hard distractors for the real-world tasks. This makes the task more difficult.\n- There are some key insights from this work that are quite interesting. For instance, the authors show that CoT prompting is often not effective for CodeMMLU tasks, whereas few-shot prompting seems to consistently perform well. Additionally, the authors compare performance of HumanEval as a generative task versus as a MCQA task, through which they show that MCQA performance can be much lower sometimes. This suggests that generative tasks do not adequately evaluate a model's code reasoning capabilities.", "weaknesses": "- **Not clear whether this is a reliable evaluation set.** The correlation with human judgement has not been measured. The authors motivate this work by highlighting the issues of potential data leakage with existing benchmarks (L036). However, it seems that CodeMMLU is susceptible to the same issue. Data sources like W3Schools, Geeks4Geeks, and Sanfoundry are likely already in the pretraining sets of existing models. Additionally, the real-world tasks are based on existing benchmarks, which have leakage issues, as the authors claimed. Next, Figure 7 and Table 4 suggest that the performance is very sensitive to the position of the correct option, which suggests that there are factors beyond code comprehension at play in MCQA. Therefore, it is not clear whether we can rely on this for evaluation code comprehension.\n\n\n- **Many missing details and also some details which are inconsistent.** First, it is not clear how the authors generated MCQA questions and hard alternative options for the knowledge-based tasks. Next, 10K is approximation given in the abstract for the number of examples in CodeMMLU. However, the sum across subjects in Table 2 is 20, 281. Does that mean there are some duplicates? Furthermore, the number of models that have been benchmarked is not clear. In Section 4.1, the authors say 35 open-source models (L312) and 3 closed-source models (L319). However, the number of rows in Table 3 do not align with this. In L375, the authors say they have evaluated 43 models across 10 model families. In L976, the authors say they have experimented on 48 LLM models from over 12 families. Additionally, some of the results are difficult to interpret. For example, there is no y-axis for Figure 5 and also the prompting style is not actually labeled in Figure 9.\n\n\n**Suggestions**:\n- Is 3.2 mis-labeled? Should it correspond to \"Real-world problem set\"\n- Place Table 3 before Figure 4.\n- Currently, Figure 5 is referred to before Figure 4. Maybe switch the numbering?\n- L345: Detaill $\\rightarrow$ Detail \n- Table 3 is confusing. CodeMMLU is the aggregate score across Syntacic, Semantic, and Real-world tasks? Make this clear by saying \"Combined\" instead. CodeMMLU includes all.\n- L426: There is no Table 8. Seems that the authors intended Figure 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)