DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
Authors: Simin Chen
Pranav Pusarla
Baishakhi Ray
Venue: ICML 2025 poster (2025)
Keywords: benchmarking
code generation
large language model
trustworthy ML
Primary area: deep_learning->large_language_models
TL;DR: We identify the limitations of static benchmarking for code LLMs  and propose a dynamic benchmarking approach.

Abstract: The rapid advancement of code large language models (Code LLMs) underscores the critical need for effective and transparent benchmarking methods. However, current benchmarking predominantly relies on publicly available, human-created datasets. The widespread use of these static benchmark datasets makes the evaluation process particularly susceptible to data contamination—an unavoidable consequence of the extensive data collection processes employed during LLM training. Existing methods for addressing data contamination typically face significant limitations, including reliance on substantial human effort and difficulty in managing class imbalances. To overcome these challenges, we propose DyCodeEval, a novel benchmarking suite specifically designed to evaluate Code LLMs under realistic contamination scenarios. Given an initial seed programming problem, DyCodeEval utilizes multiple agents to systematically extract and modify contextual information without changing the core logic, generating semantically equivalent variations. We introduce a dynamic data generation method and conduct extensive empirical studies on two seed datasets involving 18 Code LLMs. The results demonstrate that DyCodeEval effectively assesses the reasoning capabilities of Code LLMs under contamination conditions while producing diverse problem variants, thereby ensuring robust and consistent benchmarking outcomes.

DyCodeEval:
Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
Simin Chen
Pranav Pusarla Baishakhi Ray"
CColubia University
Abstract
1. Introduction
The rapid advancement of code large language models (Code LLMs) underscores the critical need for effective and transparent benchmarking methods. However, current benchmarking pre- dominantly relies on publicly available, human- created datasets. The widespread use of these static benchmark datasets makes the evaluation process particularly susceptible to data contam- ination-an unavoidable consequence of the ex- tensive data collection processes employed dur- ing LLM training. Existing methods for ad- dressing data contamination typically face sig- nificant limitations, including reliance on sub- stantial human effort and difficulty in manag- ing class imbalances. To overcome these chal- lenges, we propose DyCodeEval, novel bench- marking suite specifically designed to evaluate Code LLMs under realistic contamination seenar- ios. Given an initial seed programming problem, DyCodeEval utilizes multiple agents to system- atically extract and modify contextual informa- tion without changing the core logic, generating semantically equivalent variations. We introduce dynamic data generation method and conduct extensive empirical studies on two seed datasets involving 18 Code LLMs. The results demon- strate that DyCodeEval effectively assesses the reasoning capabilities Code LLMs under con- tamination conditions while producing diverse problem variants, thereby ensuring robust and consistent benchmarking outcomes. Our project webpage can be found at this lini
Large language models (LLMs) have demonstrated signifi- cant potential as assistant software developers. particularly in code generation (Chen et al. 2021; Guo et al. 2024
Jiang et al. 2024 Di et al. 2024). Consequently, numerous code-focused LLMs have been developed. These models are trained ming language data. Once well trained, they can compre- hend human instructions and generate the corresponding code snippets.
corpora of natural language and program-
As diverse model architectures and training algorithms for code LLMs continue to emerge (Vaswani et al., 2017] Shazeer et al., 2017). a key focus in code LLM research the effective benchmarking of each model's 's code reasoning capability. Without a standardized and transparent bench- marking suite, assessing these models' performance and driving improvements becomes a significant challenge.
However, existing benchmarking suites for evaluating code L.MMs are inadequate due to their statie benchmarking schema, which can lead potential data contamination from unintended data crawling. Research suggests that such con- tamination may already be present in current LLMs (Chen) et al. 2025 Jain et al. 2024 Dong et al. 2024). Although some methods aim to provide contamination-fee bench- marking for code LLMs, they still rely on manual efforts. For example, BiccC (Jain et al., 2024) pro- poses crawling new programming problems from online platforms and benchmarking LLMSs based timestamps, while PPM (Chen et al., 2024 attempts systematize new programming problems by combining manually defined op- erators. However, these methods have several limitations: (1) Significant Manual Effort: These methods still require substantial manual input to create such datasets. For exam- ple, PPM necessitates manually defining the lambda opera- tor, while L.i veCodeBench shifts the burden of manual design question authors coding platforms. anced Semantic Complexity: The newly generated bench- marking datasets often lack semantic equivalence with the original ones. As a result, when a model performs worse on these benchmarks, it is challenging to determine whether
'Department of Computer Science, Columbia University. Cor- respondence to: Simin Chen <sc5687 @columbia.cdu>>
(2) Imbal-
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
dycodeeval html

---

Dynamic Benchmarking of Code LLMs
the lower score reflects diminished model capabilities or increased benchmark complexity. Thus, these new bench- mark results fail to provide meaningful guidance for model developers to improve their models effectively.
produces stable benchmarking results, ensuring reli- able evaluation
44.5
We summarize our contribution as follows:
To address this limitation, rather than manually creating benchmarking datasets with uncertain semantic complexity, we aim to develop an automated method for dynamically evaluating code LLMs. However, designing such a method presents two key challenges: (1) Generating Semantically Diverse Yeu Complexity- Contrrolle Problems. The first chal- lenge is how to ensure the generated problems vary in seman- ties while maintaining controlled complexity. (2) Providing Comprehensive Benchmarking. A proper benchmark pro- gramming problem must include fine-grained test cases and canonical solutions to rigorously assess correctness.
Novel Problem Characterization. We identify a limi- tation in current static benchmarking schemas, as they are insufficient for effectively evaluating modern Code LLMs, especially when data contamination occurs and the model's training process lacks transparency.
New Methodology Design. We propose a novel ap- proach that separates context and algorithm in program- ming problems. Building on this concept, we introduce a dynamic benchmarking method, DyCode Evall which generates programming problems for bench- marking without introducing additional complexity to the dataset. This approach mitigates the impact of data contamination, ensuring transparent and reliable benchmarking.
To address these challenges, draw inspiration from meta- morphic testing (Chen et al. 2018), a widely used approach in software testing to tackle the oracle problem. In our case, we leverage the principles of metamorphic testing to automate comprehensive benchmarking. Specifically, we define a metamorphic relationship for programming problems. A programming problem includes complexity- related algorithmic abstraction and -nnrelltet context description. Modifying the context description alters the problem's ' semantics with- out changing its inherent complexity. Building on this relationship. DyCodeEval employs LLM-based agents to generate diverse contexts for a seed problem, automat- ically transforming existing problems into semantically varied yet complexity-preservinn DyCodeEval integrates validation agent probabilis- tic oracle to verify the correctness and consistency of the newly generated problems, ensuring reliability.
Empirical Findings. We conduct an empirical eval- uation of DyCodeEval, and the results demonstrate that traditional static benchmarks can create false sense of accuracy. In contrast, our dynamic bench- marking approach provides consistently reliable re- sults, even under data contamination scenarios. Ad- ditionally, DyCodeEval generates semantically di- verse programming problems while maintaining stable benchmarking results.
versions. Additionally,
2. Background & Related Work
2.1. Benchmarking Code LLMs
We used DyCodeEval to generate new evaluation sets to assess Code LLM performance under both data contami- nation and real-worid benchmarking scenarios. Our key findings are follows:
Code LLMs have been widely adopted in various real- world software engineering applications, leading to the de- velopment of numerous benchmarks for evaluating their
capabilities in code understanding and reasoning (Chen)
ct al.
2021
Li et al.]
[004a] Guan et al.] [025] Austin
1. Our method effectively reflects Code LLMs' reason- ing capabilities in a manually crafted contamination environment
et al.
2021
Chen et al.]
2024 Austin et al.) 2021; Yu
et al. 2024
Jimenez et al.
2024
Ding al. 2023 Mathai
et al. 2024).
Among the many tasks designed to assess
2. The performance of some Code LLMs on our dynamic benchmarks degraded significantly, suggesting poten- tial data contamination of these Code LLMs ($4.3).
code reasoning, this work focuses specifically on the task of natural language to code generation and reviews rep- resentative benchmarks this area. HumanEval (Chen) et al. 202T) introduced a human-crafted dataset to evaluate the code generation capabilities of large language models. EvalPlus (Liu et al. 2023) later identified the limitations of HumanEval and MBPP particularly their limited test case coverage-and proposed a more rigorous benchmark. HumanEval-XI. (Peng et al. 2024) further extended Hu- manEval to support multilingual settings. Fig. Iillustrates an example from HumanEval, a widely used benchmark
3. DyCodeEval generates semantically diverse pro- gramming problems, and its inherent randomness makes the likelihood of generating identical problems extremely low, thereby reducing the risk of data con- tamination $4.4)
4. Despite its randomness, DyCodeEval consistently
2

---

Dynamic Benchmarking of Code LLMs
Prompt
Canonical Solution
Test Cases
def has has__loss___eee olementssnumbers List[loa]] threshold: Rlaaly:
for idx, elem in enumerala(rumben))
kdh2, ciem2 crumerats(uumbars) ta distance absietem - elem2) distance < threesold: relum True
def chedk(candisate)
Check if given numbers, are any numbers closer to each other than given threshold
assert candidate[[ 10, 3.9, 4.0j 0.3) True assert candidatel 1.0, 3.9, 4.0), D (.05) == False assert candidate 5.9, 5.0), 0.96)
booc hessssseeee cete__looe_eeeeeeeee [aapa__eeeeens([1..00 2.0, 3.0g.
False
relurn False
Figure I. Benchmark programming problem example
for the natural language code generation task. Each pro- gramming problem typically consists of three components: prompt, canonical solution, and set of test cases. The prompt is first fed into the Code LLM to generate a can- didate solution, which is then executed against hidden test cases to evaluate its correctness.
interest for two reasons: (1) Enhanced generation of train- ing/testing data (Li et al. 2024b Liu et al. (024) (2) Accu- rate evaluation and comparison of LLM outputs such as in PandaLM (Wang et al. 2024) and Dy (Zhu et al. 2024b).
Additionally, as LLMs have been able to perform remarkbly well on unseen tasks, they offer a faster, equally accurate alternative to human evaluation, (Chiang yi Lee 2023).
2.2. Data Contamination Free Benchmarking
3. Methods: DyCodeEval
Data contamination has become a significant concern in benchmarking large language models (LLMs) (Brown et al. 2020 Jain al. 2024; Chen al., 2025), as it can lead to inflated performance scores and unreliable evaluations. To mitigate this issue, researchers have proposed various contamination-free henchmarking strategies, which can be broadly categorized into three approaches. The first line of work focuses on data protection through encryption and privatization. For instance, Jacovi et al. al.aJacovi et al. (2023) and Rajore et al. al.oRajore et al. 2024) propose techniques to safeguard benchmark data from being included in LLM
3.1. Design Overview
There are two key challenges in designing a dynamic evalu- ation schema for benchmarking code LLMs. (1) Generating Semantically Diverse yet Complexiny-onntrollee Controlled Problems: There is currently no systematic method for generating pro- gramming problems that maintain a consistent complexity level while ensuring semantic diversity. Existing approaches often rely manual effort, either through predefined rules or domain experts, making them difficult to scale efficiently and incapable of precisely controlling problem complexity. (2) Ensuring Comprehensive Benchmarking: To effectively evaluate code LLMs, the generated programming problems must include fine mme-grained test cases and canonical solutions to rigorously assess correctness.
training corpora. The second line of research emphasizes timely benchmark updates. LiveBench (White al. 20244) for example, compiles questions from recent sources such as math competitions held within the past year and regu- larly updates its dataset. Similarly, LiveCodeBench (Jain al. 2024) continuously collects new numan-authorre pro- gramming problems from online platforms like LeetCode to maintain freshness and reduce the risk of contamination. The third line of research explores dynamic generation of evaluation sets. Dy Val (Zhu et al. 2024a) uses DAG struc- tures to create dynamic benchmarks, TvelEval (Li et al. 2024b) employs high-performing LLMs to generate and evaluate problems via tree planning, and ITD (Inference- Time Decontamination) (Zhu et | (202cc) identifies and rewrites leaked benchmark samples while preserving their complexity.
We draw inspiration from metamorphic testing to generate programming problems using LLMs as agents. Metamor- phic testing, widely used in software engineering, defines relationships to address the automatic oracle problem. In our approach, a programming problem prompt consists of two algorithmic abstraction and uumplex context description. Our key meta- morphic relationship states that modifying the complexity- unrelated context description preserves both the problem's 's canonical solutions and complexity, enabling controlled problem generation. Additionally, since LLMs are trained on a vast diverse corpus, we can utilize them as agents to suggest relevant and meaningful complexity-unrele unrelated context descriptions, further enhancing problem diversity.
components: complexity-
2.3. LLM as Judgment Agent
Recently, LLMs have become increasingly used as exam- iners given their capabilities of analyzing large amounts of data and providing unbiased assessments (Bai et al.) 2023 Fernandes et al. 2023). This growing trend has gained
The design overview of DyCodeEval is shown in Fig. 2 Given seed programming problem from existing bench- marks, DyCodeEval generates a semantically different yet
3

---

Dynamic Benchmarking of Code LLMs
Scenario Proposer
Context Generator
Prompt Rewriter
Validator
Scenario Pool
Random Scanario
Contexts
Orig Problem
Problem
Problem
Healthcare
Solution
Threshold.
nambers closer
Vin gi
numberr, teo rumben cload
In exedent rerfrmmnc
Education
Bank
Numbers
io eoch other that
given
frwoovdd
key twon coeett have performanca.
Bank
Type
Scenario
LLM Agent Verifier
Bank
Listind)
Float
New Scenario
New Problum
Contexts
stuckntt performenoe detnow: determine aaaty two stutents have porform once scores that are cleser to each other socoficd palteraa...
Transportation
Threshold:
NO!
prodefined ris...
Numbers: a list
operations
cocoubiiliie
Figure 2. Design overview of DyCodeEval
complexity-equivalen problem using a metamorphic rela- tionship. DyCodeEval comprises of four agents: (1) Sce- nario Proposer, (2) Context Generator, (3) Prompt Rewriter, and (4) Validator. The Scenario Proposer suggests real- world domains (e.g., banking, healthcare, education) from which DyCodeEval randomly selects one. The Context Generator then analyzes input types in the canonical solution and assigns relevant context for each input variable based on the selected scenario. The Prompt Rewriter reformulates the problem to align with the input variable contexts and chosen scenario. Finally, the Validator ensures the new prob-
Algorithm Type Inference Algorithm. Input: Value list V. Output: Set of data types F.
Abstract (
I: ==
I/ Initialization.
2: for each v V do
3:
T= Type(v)
if E Basic Types then
7 F.add(Type(())
6: else
7:
T Abstract(Tolist(o) // Composite type.
8: F.add(T(r"))
lem remains consistent with the original. If inconsistencies are detected, DyCodeEvy EvooleEvall al will repeat the aforementioned process until valid variant is produced.
9: end
10: end for
11: return 7
3.2. Detailed Design
Scenario Proposer Agent. The Scenario Proposer enhances diversity and minimizes repetition in generated program- ming problems, reducing potential data contamination. It first selects scenarios from a predefined pool (e.g., banking, healthcare, education, transportation, social networking) and uses them as examples to prompt an LLM for new sce- mario suggestions. The newly generated scenarios are then added to the pool. By iteratively updating the pool and querying the LLM with varied examples, DyCodeEvas Eval continuously expands the scenario diversity until the sce- nario pool reaches pre-defined size, ensuring the generated scenarios remain diverse and practical. The prompt used for querying the LLM and the suggested scenario examples are listed in Appendix C
In languages like Python, input types are not explicitly de- fined. To address this, the agent uses abstraction for type inference. It analyzes ASSERT statements in test cases, col- lects concrete input values from the canonical solution, and abstracts the input type based on these values. Our type inference algorithm, shown in Alg. [ works as follows: for each concrete value, it first checks if the type a basic type (e.g., i int fl oat). If so, it updates the type Otherwise the value is a composite type it recursively iterates over all the elements and updates the type set with types like List [int] J or Tuple [int | string]. Notice that while our baatract-based type inference may not capture return value types, is sound and guarantees that collected types will always appear in the canonical solution.
Context Generation Agent. After proposing a set of sce- narios, the context generation agent randomly selects one from the pool and assigns context information to each input variable of the programming problem based on the chosen scenario.
After collecting the input data types, the agent prompts the LLM with the scenario and input type information, asking it to assign meaningful context to each input variable based on the given scenario. See Appendix C for prompt templates of our context generation.
Prompt Rewriting Agent. With the scenario and context

---

Dynamic Benchmarking of Code LLMs
3.3. Theoretical Collision Analysis
Seed Problem: You will be given a string of words separated by commas or spaces. Your task split string words and return an array words
DyCodeEval generates programming problems dynami- cally with randomness, reducing the risk of potential data contamination. To analyze this, we conduct collision anal- ysis. The randomness in DyCodeEval arises from both the scenario proposal and context generation phases. We assume the scenario proposer generates ||S]] scenarios, and for each scenario, the context generation produces ||C'] con- texts, while ignoring randomness the rewriting phase. We also assume that the random sampling process follows uni- form distribution. Based on this, we present the following theorem.
Input: S [string] User's Blog
Recommandation system
As content recommeedattion system developer, you need to process user generated social media posts to identily relevant keywords for suggesting connections and events, Given users post containing words separated by commas or spaces, create a function that breakcs down the text into individual words for analysis. The function should handle both comma- aepparated space separated text formats while preserving each distinct word recommendstion aigorithm to process.
Generated Problem:
Figure generated example from DyCodeEval
Theorem 3.1. After running DyCodeEval 1. M + 1 times on the same seed problem, then the probability that the M samples after the first are all different from the first sampled
information for each input variable, the prompt rewriting agent then rewrites the seed programming problem prompt to be tailored to the scenario with meaningful context. Note that we did not ask the LLM to generate the new prompt provided the detailed scenario and
item satisfies: P 2 1 -exp (-SS|x|IIiI)
M
from scratch. Instead, asked it to perform a rewriting task, which is simpler than generation task. With this approach, leveraging detailed con- text and a more straightforward task, our agent can generate semantically diverse programming problem prompts. See Appendix C for prompt templates of our prompt rewriting.
Theorem 3.2. After running DyCodeEval M times on the same seed problem, If M << ||S||| x [[C]]. the probability of at least one collision (i.e., two or more generated problems being the same) after M generations satisfies the following
bound: P 1-1- (- M²
(- [||||
Validation Agent. Although we provide the LLM with de- tailed scenario and context information for rewriting, there are cases where the rewriting agent unintentionally alters the consistency. To address this, we design a validation agent to assess whether the generated question maintains the integrity of the original intent and informational con- tent. The validation prompt is designed from two angles: (1) it directs the LLM to compare the seed programming problem prompt with the rephrased prompt, ensuring the preservation of the core concept and factual accuracy, and (2) it asks the LLM to check whether the seed canonical solutions align with the generated programming problem prompt. Specifically, we design two comparison prompts to query the LLM and retain only those rewritten prompts for which both comparison responses are "YES".
Theorem 3.3. Consider the seed dataset of size D, After running DyCodeEval M + 1 times on this dataset, If M <<<<>>>>>>>>> ||ر|| x |[C]], then the probability that the M gener- ated datasets after the first one are not the same as the first generated dataset satisfies:
M
[1I (x S P
The proof could be found in Appendix A
4. Evaluation
4.1. Experimental Setup
Seed Dataset. We conduct our evaluation using two datasets: HumanFival (Chen et al., (02T)) and MBPP- Sanitized (Austin et al., 2021). Both datasets are widely utilized in existing research and serve standard bench- marks for evaluating code generation models. More details about the dataset could be found in Appendix B
To ensure the consistency of the generated programming problems, we also include human verification step. The details of our validation prompt and the human verification process are presented in Appendix and Appendix [
Fig. Billustrates an example of programming problems that are semantically diverse yet complexity- -qqiivllee,,, gener- ated under the scenario of a recommendation system with the context of user's blog. From this example, we observe that our step-by-step guided approach significantly enhances the semantic diversity of the generated problems, while also reducing the risk of data contamination. This is achieved by leveraging the vast combination space of scenarios and contexts.
Implementation Details. We use CLAUDE-3355SO0 our foundation model to generate the benchmarking dataset. Specifically, we create 50 scenarios, and for each scenario, we randomly generate 50 contexts. During dataset gen- eration, we set the LLM temperature to 0.8, while in our validation agent, we use temperature of O. For each code LLM under benchmarking, we employ vLLM to launch the model. For closed-source code LLMs, we query the com- mercial API for evaluation.
as
5

---

Dynamic Benchmarking of Code LLMs
Humaneval
Freaze
Fietaae Mano
P.5 Faxaore
Pivetas
Fhekine Huraatiaal
Pleasue VEPP
Pass
0.6
0.5
02-
0.2
70%
30% 70%5100%
100%
Leaked Parcentage
Leaked Parcentage
Leaked Porcentage
Evaluation on Humaneval + Ours
Evaluation on MBPP + Ours
Peokint or Homerbbe Oar
Fraeuee MSPP Cuat
02
Plcer
0.1
0.2
0% 25% 90% 75% 100%
0% 25% 50% 75%.100%
0% 25% 50%
75%
100%
0%
25%
50%
75% 100%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
Leaked Perceniage
Leaked Percentage
Leaked Percentage
Llama3.2-1B
Llama3.2-33
DeepSeek-Coder-1133
Figure Results of benchmarking on contaminated models
4.2. Benchmarking Contaminated Model
Data contamination creates a false sense of code reasoning capability under static benchmarks. When the benchmark- ing dataset is leaked and used for fine-tuning, the model achieves higher Pass@1 @1 score the corresponding benchmark. However, this improvement does not accurately reflect the model's true reasoning ability, as its performance declines on other benchmarks that were not included in fine-tuning tuning. (2) Our dynamic benchmarking mitigates the impact of data contamination. Different from static bench- marks, our approach prevents contaminated models from achieving artificially high Passêl scores after fine-tuning. This is due to the randomness in method, which ensures minimal or overlap between different runs, reducing the risk of direct data leakage. (3) Our dynamic benchmarking dataset provides results comparable to manually curated, non-contaminated datasets. In static benchmarking, as the percentage of leaked data increases, the model's Pass @1 score the contaminated benchmark steadily improves. However, its performance other benchmarks remains rel- atively stable, showing little variation across different con- tamination levels. Interestingly, this stability also applies to our method. If the base model is not contaminated on the selected seed dataset, this suggests that our approach pro- vides competitive benchmarking results similar to those of human-curated datasets. (4) A notable anomaly is observed in DEEPSHEK- SEEEEEERRR When only 25% of the benchmark- ing dataset is used for fine-tuning, the model's P Pass@ @ score drops below that of the original, unmodified model. We hypothesize that the model may already be overfitted to the contaminated dataset, and further fine-tuning with lim- ited data could destabilize this overfitting without providing enough new information help the model adapt.
Models. We conduct our study with three aublic-available Code LLMs: LLAMA-3.2-IB, LLAMA-3.2 3.2-BB, and DEEPSEEK-CODEE
Model Contamination Process. For each model, simu- late data contamination by intentionally leaking a portion of the benchmarking dataset during fine-tuning, We experi- ment with leaked data percentages of 0%, 25% 50%, 75%, and 100%. producing four distinct contaminated models. Each polluted model then evaluated on the benchmarking dataset using the Pas metric. The formal definition of Passe1 is shown in (1). where is the number of the generated solution candidate, and e is the number of the correct solutions that can pass all test cases.
P Pass@K
Eproblems
Main Results. The study results are presented in Fig. where there are two rows and three columns. Each column represents evaluation on different L.M while the rows show static (first) vs dynamic (second) benchmarking. In each column, the left section displays the results for the model fine-tuned on the HumanEval dataset, while the right section shows the results for the model fine-tuned the MBPP dataset. The red bars represent the performance of the fine-tuned model benchmarked on the HumanEval dataset, and the blue bars represent its performance bench- marked on the MBPP dataset.
From the results, we make the following observations: (1)
6

---

Dynamic Benchmarking of Code LLMs
Owend 6-TB-its
Usma-d 1en--d---eee 1-60-b
1.0
Uemee po poibuttrratEiet
0.8
Lama (NBPP)
Lion (Bpuunnn
Claute-J So--Sn
U (WPPP)
Cocollama 180
66
Owanz Codan-B-ira
HumanEval
Owand: S: Codee-To
0.6
Gwaerd S-Coder-788
0.6
Owen2 5-7B
Lime-- ino DS Cocen1 130 inc
MBPP
0.4
Jume d8
Gacellarme-le
Gwariz 5-78
Owar2 Coden
0.2
Gotellere-77
daudo
0.2
Limee/52-08
ULama-d 2-10
Usma 1-1---6
Liare-3 2-15
DS-sodon- 20-ine
D6 CB-Cocee--PLL---eee
0.2
0.4
0.6
0.2
0.4
0.6
Ours + HumanEval
Ours MBPP
Figure 5. The in-the-wild benchmarking results
4.3. Benchmarking In-the-Wild Model
to measure semantic diversity. For semantic embedding, we use the GPT-2 model to obtain the embedding of each natural language prompt. Moreover, we also consider PMM (Chen et al.) 2024) and series of robustness-bant based muta- tions (Wang al. 2023) such as token replacement, insert blank lines, as our comparison baseline.
We then apply DyCodeEval to benchmark more in-
the-wild code LLMs, besides the models used in $4.2 We consider the following code LLMs: LLAMA-3.1 8B, CODELLAMA-7B, CODELLAMA- 13B, DEEPSEEK- V2-LITE, DEEPSEEK- -COBEER---OORR--EE VE--RR-VV---E----BAA BASE, LLAMA- 3.1-8B-INSTRUCT, SWEN2.5-CODER-77 QWEN2.5-7B-
The diversity results are shown in Table where the first four columns represent internal diversity and the last four columns represent external diversity. From the results, we observe that DyCodeEval generates diverse program- ming problems both syntactically and semantically. Ad- ditionally, we find that all baseline methods exhibit high BLEU-4 and semantic similarity scores, as they rely rule- based approaches to mutate the programming problems, which do not introduce significant diversity. In contrast, DyCodeEval leverages an LLM agent to suggest different scenarios and contexts, significantly increasing diversity.
INSTRUCT, QWEN2.5-7B. CLAUDE-3.5- CLAUDE- 3.5 2.E.T 7B-INSTRUCT.
The results are presented in Fig. with the left figure show- ing the results on HumanFval and the right showing the results on MBPP. In each figure, the x-axis represents the Pass@l scores on our generated dataset, and the y-axis represents the Pass@1 scores on the seed dataset. The blue region corresponds to the regression area of the in-the- wild model, the red region represents the regression area of the overfitted model on this dataset, and the orange area indicates the overfitted model on the other dataset.
From these results, we observe that for both seed datasets, the in-the-wild model's Pass@l 1 scores maintain a linear relationship, while the overfitted model appears as outlier. A notable finding from our in-the-wild evaluation is that the model SWEN2.5-CODER 95% confidence interval of the regression area, suggesting may be contaminated on both datasets.
HumanEval MBPP
consistently falls outside the
Pass@1
0.2
4.4. Problem Diversity
To evaluate the diversity of the generated programming problems, we conduct two experiments: one for external diversity and one for internal diversity, External diversity quantifies the dissimilarity between the generated and seed problems, while internal diversity measures the diversity within each problem- generation problem-generation method across trials. We use two metrics: BLEU-4 to measure syntactical diversity and cosine similarity of the prompt's semantic embedding
Liama-3 Llama-3.2-18 3-18
DaepSaek- 12-LLtt
Codel MoalLamme-7hin
Liama 3.2-38
CodaLiama-111 15b-ht
DeepSeek Coder V2- Base
Liama-3.1 3.1-8B 1-1-8
deepseek codor- Sbbinsttutt
Liama-a 8B -nstruut
Figure Stability results
7

---

Dynamic Benchmarking of Code LLMs
Table J. Diversity results
Internal Diversity
External Diversity
Methods
HvmanEval
MBPP
HvaanEval
MBPP
BLEU-4
SemSim +
BLEUE
SemSim +
BLEUU4
SemSim +
BLEU-- J
SemSim +
Base
1.00
1.00
1.00
1.00
1.00
1,00
1.00
1.00
Token Mutation
0.72
0.95
0.66
0.92
0.82
0.96
0.76
0.95
Char Mutation
0.81
0.97
0.78
0.94
0.84
0.97
0.78
0.92
Mutation
1.00
0.98
1.00
Insert Line
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
CommSyntax
1.00
1.00
1,00
1.00
0.81
0.98
0.73
0.99
PPM
0.97
0.96
0.96
0.94
0.69
0.89
0.57
0.84
Ours
0.27
0.74
0.18
0.73
0.17
0.59
0.02
0.59
4.5. Benchmarking Stability
data contamination. Compared to PassłK, which eval- uates solutions within a fixed problem context, DyPass introduces contextual variations during benchmarking. This allows it better distinguish whether model is merely memorizing the problem context or genuinely reasoning to solve it.
Note that DyCodeEval generates a unique benchmark- ing dataset each time. To assess its stability, we evaluate whether DyCodeEval can produce consistent benchmark- ing results despite this randomness. Specifically, run DyCodeEval 10 times and measure the PassE1 scores across these 10 generated benchmark datasets.
Table 2. Comparison of Passêk and byPass@K contami- nated Models
The mean and standard deviation of the Passêl scores are presented in Fig. 6 The results show that the variance in benchmarking scores minimal compared to the mean values, indicating that DyCodeEval provides stable bench- marking results across different random trials.
Pass @
DyPass@K
Model
k=3
k=5
k=10
k=3
k=5
k=10
Llamu-3.2-1B
0,22
0.34
0.17
0.21
0.26
1lama-3.22-B (C)
0.82
0.83
0.85
0.13
0.15
0.17
Llama-3.2-31
0.35
0.40
0.48
0.31
0.36
0.43
4.6. Impact of Foundation LLM
1lama-3.2-44 3B (C)
0.88
0.88
0.89
0.24
0.27
0.29
In this section, we evaluate the feasibility of using less advanced LLMs reduce dataset generation costs. Specif- ically, we replace our foundation model, CLAUDE-3.55 SONNET, with
Table 3. Comparison of Passžk and pases on in-the-wild models
We manually sample and assess generated problems from each model, check- ing their consistency rate. Our observations show that the consistency rate drops from 95% 83%, highlighting the need for robust and capable LLMs serve effectively foundation models.
Pass@K k=5
DyPass @K
Model
k=3
k=10
k=3
k=5
k=10
CodeLlama-
0.39
0.46
0.56
0.34
0.40
0.49
CodeT 1h--h
0.48
0.57
0.68
0.37
0.45
0.53
Llama-3.2-10
0.22
0.27
0.34
0.17
0.21
0.26
Llama-3,2-3B
0.35
0.40
0.48
0.31
0.36
0.43
5. Dynamic Evaluation Metrics
Llama- 1.1ma--3.1-B-
0.48
0.56
0.65
0.39
0.45
0.53
Llama-3. 1-1---3.--BB-nnstruu
0.72
0.77
0.83
0.64
0.69
0.75
Leveraging the dynamic nature of our method, we propose a new metric, DyPass, to address the limitations of the current gold standard, Pass@K. Unlike Pass@K, which generates candidate solutions for fixed problem prompt approach creates
To demonstrate the advantages of DyPass, we compare it against Pass@K on both contaminated and in-the-wild models, with = 3, 5, 10 for evaluation. The results are presented Table 2 and Table 3 From the results Ta- ble we observe that when the model trained on leaked data, the static metric Pass@K fails accurately reflect the model's reasoning capabilities, with all Pass@ scores ris- ing to very high levels (e.g., from 0.82 to 0.89). In contrast, our dynamic metric DyPass @ @K shows slight decrease rather than a significant increase, highlighting the sensitiv- ity of DyPas Pass to data contamination. When comparing Pass 8 and DyPass K on models that were not specif-
and evaluates the correctness, mantic prompt variants of seed problem. These prompt variants preserve the complexity of the original problem by modifying only the description while maintaining the same underlying algorithmic abstraction. Furthermore, n prompt variants expand the input space beyond that of Pass@K, making it more challenging achieve full coverage. As result, DyPas aas provides more rigorous assessment of code LLMs' reasoning abilities, particularly under potential
8

---

Dynamic Benchmarking of Code LLMs
ically trained on the leaked dataset, both metries show con- sistency in benchmarking code L.MMs. Based on these ob- servations, we conclude that our dynamic metric, DyP ass, effectively reflects the reasoning capabilities of code LLMs, even under data contamination. Moreover, DyPass @ K aligns with static benchmarking metrics when there is no data contamination.
References
Austin, J.. Odena, A., Nye. M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. CoRR, abs/2108.07733. 2021. URL https
7 //axii..rg/ab//221000 07732
Bai, Y., Ying, J., Cao, Y., Lv, X., He, Y., Wang, X., Yu, J., Zeng, K., Xiao, Y., Lyu, H., Zhang. J., Li, J., and Hou, L. Benchmarking foundation models with language-mool model- as-an-examiner, 2023. URL.https: http://xxxvv.rr//// abs/2306. 04181
6. Conclusion
In this paper, we introduce DyCodeEval, new bench- marking suite that dynamically generates semantically equivalent diverse problems as a way to combat data con- tamination. We break this generation up into four distinct steps to systematically develop a new programming prob- lem with the same algorithmic complexity but different context. Our experimental results show that while Pass@k with current benchmarks have caused inflated model scores, Seroodeeval-geeeeeee
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, F., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.. Sutskever, I., and Amodei, D. Language models few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpps://pproceedngg..... neuris
questions with DivPass has proven to perform as a reliable evaluation tool. We believe that these results show a promising path forward.
Our proposed work has several limitations: (1) Although LLMs provide fully automated way generate diverse programming problems for benchmarking, their computa- tional cost is a significant concern. We found that very large LLM is required to generate programming problems with a high consistency rate. Therefore, a future improve- ment could focus on enhancing the efficiency of the prob- lem generation phase. (2) While generating questions using DyCodeEval, observed instances where excessive infor- mation was provided, potentially confusing the reader. This highlights the opportunity for improving prompt generation through further experimentation.
cc cctpaperr_fiiiicecee [iaaaper_fLLe///ppeee/2////1L11
1457 (0tcc666foa44407771114rbeeaa3111 Pape
pdf
Chen, M., Tworek, Jun, H., Yuan, Q. de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N. Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saun- ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. CoRR, https///10..03374. 2021. URL /ttp:s//axxiv.oooo//////22/
Acknowledgements
This work was supported in part by CCF 2313055, CCF 2107405, CAREER 2025082, and FAI: 2040961. Any opin- ions, findings, conclusions, recommendations expressed herein are those of the authors.
Impact Statement
Assessing the overall capabilities of large language mod- els (LLMs) is essential for ensuring their reliable and safe deployment in society. However, data contamination can inflate evaluation accuracy, obscuring model's true perfor- mance. To address this, we propose new benchmarking method, DyCodeEval. 1, which enables more accurate mea- surement of LLM capabilities and provides deeper insights into their behavior.
Chen, S., Feng, X., Han, X., Liu, C., and Yang, W. Ppm: Automated generation of diverse programming problems for benchmarking code generation models. Proceedings of the ACM on Software Engineering, 1 2024.
Chen, S., Chen, Y., Li, Z., Jiang, Y., Wan, Z., He, Y., Ran, D., Gu, T., Li, H., Xie, T., et al. Recent advances in
9

---

Dynamic Benchmarking of Code LLMs
large langauge model benchmarks against data contami- nation: From static dynamic evaluation. arXiv preprint 17225 2025.
308. URL https://aclaattii ogy. egy.org/2023
emn 1am1ppmmas ma n..
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code, 2024. URL https: https://axxiv.orrt wrw.aaiiiiv.... www.w.owggw......aoaooooaaaaaaaa /tt..///222000099900
Chen, T. Y., Kuo, F.-C.,, Liu, H., Poon, P.-..,, Towey, D., Tse, T., and Zhou, Z. Q. Metamorphic testing: review of challenges and opportunities. ACM Computing Surveys (CSUR), 51(1): 2018.
Jiang, X., Dong, Y., Wang, L., Fang, Z., Shang, Q., Li, G., Jin, Z., and Jiao, W. Self-planning code generation with large language models. ACM Transactions Software Engineering and Methodology, 33(7):1-30, 2024.
Chiang, C.-H. and yi Lee, H. Can large language models be an alternative to human evaluations?, 2023. URL /ttpas///axvvoo/o www.ooogg//.o..ooooooooooo.. abs/ 2305. 01937
Di. P., Li, J., Yu, H., Jiang, W., Cai, W., Cao, Y., Chen, C., Chen, D., Chen, H., Chen, I.., et al. Codefuse-130 A pretrained multi-lingual code large language model. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, pp. 418 429, 2024.
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
Li, J., Li, G., Zhang, X., Zhao, Y., Dong, Y., Jin, Z., Li. B., Huang, F., and Li, Y. Eyocodebench: An evolving code generation benchmark with domain-specifee evaluations. Advances in Neural Information Processing Systems, 37: 57619-57641, 2024a.
Ding, Y., Wang, Z. Ahmad, W. Ding, H., Tan, M., Jain, N., Ramanathan, M. K., Nallapati, R., Bhatia, P., Roth, D., et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36:46701- 46723, 2023.
Li, X., Lan, Y., and Yang, C. Treeeval: Benchmark- free evaluation of large language models through tree planning. 2024b. Utt/htt.sss//arxivv.oo
Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang. M., and Li, G. Generalization or memorization: Data contamina- tion and trustworthy evaluation for large language models. arXiv preprint arXiv:2402. 1...0.2.5993. 15938, 2024.
2402. 13125
Liu, H., Zhang, Y., Luo, Y., and Yao, A. C.-C. Augment- ing math word problems via iterative question compos- ing, 2024. httttts::://art org/ orggabs///bb111 09003
Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Mar- tins, A. F. T., Neubig, G., Garg, A., Clark, J. H., Freitag, M., and Firat, O. The devil is in the errors: Leveraging large language models for fine-grained machine transla- tion evaluation, 2023. URL /tttpss//arxiv.ooz abs/ 2308 07286
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL
Guan, B., Wu, X., Yuan, Y., and Li, S. Is your benchmark (still) useful? dynamic benchmarking for code language models. arXiv preprint arXiv:2503. 06643, 2025.
openreveeneeeeeeeeee n
forum?id-1qvx61100
Mathai, A., Huang, C., Maniatis, P., Nogikh, A., Ivančić, F., Yang, J., and Ray, B. Kgym: A platform and dataset to benchmark large language models on linux kernel crash resolution. Advances in Neural Information Processing Systems, 37:78053-78078, 2024.
Guo, D., Zhu, Q. Yang, D., Xie, Z. Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseck-coder: When the large language model meets programming the rise of code intelligence. arXiv preprint arXhv:2401.14196 2024.
Peng, Q. Chai, Y., and Li, X. Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural lan- guage generalization. arXiv preprint arXiv:2402. 16694, 2024.
Jacovi, A., Caciularu, A., Goldman, O., and Goldberg, Y. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation bench- marks. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5075-5084, Singapore. December 2023. Association for Computa- tional Linguistics. doi: 10. 8653/v 1/333////223.eeem 22223..mmnllmamn.
Rajore, T., Chandran, N., Sitaram, S., Gupta, D., Sharma, R., Mittal, K., and Swaminathan, M. Truce: Private benchmarking to prevent contamination and improve comparative evaluation of 1lms, 2024. URL https //arxiv.org//bb//2400 00393
10

---

Dynamic Benchmarking of Code LLMs
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q- Hinton, G., and Dean, Outrageously large neural networks: The oparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.000 065708088,,,, 2017.
USA, November 2024c. Association for Computational
Linguisties. doi: 10.16651w/20244fmdiinss-mmmm 532. /tthhttpss//aelllaaoooooooooooo/ Find.ngs-emm1p.552//
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention all you need. Advances in neural information processing systems, 30, 2017.
Wang. S., Li, Z., Qian, H., Yang. C., Wang, Z., Shang, M.. Kumar, V., Tan, S., Ray, B., Bhatia, P., Nalla- pati, R., Ramanathan, M. K., Roth, D., and Xiang, B. Recode: Robustness evaluation of code genera- tion models. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- ties (Volume 1: Long Papers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 13818-13843. Associa- tion for Computational Linguistics, 2023. doi: 10.18653/ V120023ACI-LONG..77.. URI /ttpp:://pi......... N0.18653//1/2023.oed 1on..777177.7
Wang, Y., Yu, Z., Zeng. Z., Yang, L., Wang. C., Chen, H., Jiang, C., Xie, R., Wang, J., Xie, X., Ye, W., Zhang, S., and Zhang. Y. Pandalm: An automatic evaluation benehmark for 11m instruction tuning optimization, 2024.
URL /tt.tttpss//axxL.o//////// hbcp://bs///30
05087
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., et al. Livebench: A challenging, contamination-fiee 1lm benchmark. arXiv preprint arXivv2406.193314 2024.
Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q. Ma. Y., Liang, G., Li, Y., Wang, Q., and Xie, T. Codereval: A benchmark of pragmatic code generation with gen- erative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engi- neering, pp. 1-12, 2024.
Zhu, K., Chen, J., Wang, J., Gong, N. Z., Yang, D., and Xie, X. Dyval: Dynamic evaluation of large language models for reasoning tasks, 2024a. URL https::/arxixs
org abs//bbs..2..... 17167
Zhu, K., Wang. J., Zhao, Q. Xu, R., and Xie, X. Dynamic evaluation of large language models by meta probing agents, 2024b. URL httts:://aaatttt/ org. abs/
2402. 11.2.1488 14865
Zhu, Q. Cheng, Q. Peng. R. Li, X., Peng, R. Liu, T. Qiu, X., and Huang, X. Inference-time decontamination: Reusing leaked benchmarks for large language model evaluation. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Find- ings of the Association for Computational Linguis- ties: EMNLP 2024, pp. 9113-9129, Miami, Florida,

---

Dynamic Benchmarking of Code LLMs
A. Proof of Theorem
A.I. Proof of Theorem 3.1
The total number of possible distinct outcomes is ||S]] x ||C]], the size of the random space, let = =|S||| x ||C|] Since cach of the M samples must match X1, and they are drawn independently, the exact probability is:
=
P(X, + X X # Xi) = NN
N
We the standard incquality for the logarithm:
x
In(1 - x) 2
for
< 0
< 1.
Applying this to v. we get:
N 1
1
1/N
In
= In
1
2
N
N
1/N
N -
Exponentiating both sides:
N - 1 N
2 Nー
Raising both sides to the power M:
N- 1
M
2 2 N-1
N
A.2. Proof of Theorem 3.2
Each sampled item is drawn independently and uniformly from the space of size N. We analyze the probability that all M sampled items are distinct.
The first sample can be any of the N items, the second sample must avoid the first one, so there are N choices. Continuing this way, the probability that all M items are distinct is:
N N -
N
2
N - (M - 1) N
Pno collisions)
Rewriting in factorial form,
N! NM M)!
P(no collisions)
According to our assumption M << ||S|| x |[C]], Using the Stirling's s approximation, then we have
N
2 M exp (-M
M(M - -1))
(N M)!
2N
get
M(M - 1)
P(no collisions) 2 exp
2N
12

---

Dynamic Benchmarking of Code LLMs
The probability of at least collision is the complement:
P(at least one collision) = 1 P(no collisions).
Using the bound we derived,
P(at least collision) < eep (-
M M
M - M
2N
exp
(2|S|] |||1]
A.3. Proof of Theorem 33
Each sample can be represented D-tuple of balls b2, b2, bD), where each b; one of the N balls from bag The total number of possible sample sets is:
T T= N
Since each draw is independent, each sample set is chosen uniformly from T, meaning the probability of selecting any specific tuple is:
1
N
Let X1 be the initial sample (first draw). For each subsequent draw X; (where i = ,,...,.. M + 1), the probability that
X; = X1 (i.e., an exact match) is:
P(X, = X1)
ND
Then Theorem 3.3 could proved through Theorem 3.1
B. Dataset Description.
The HumanEval dataset, developed by OpenAl, is an open-source benchmark for evaluating the code generation capabilities of pre-trained code language models (LLMs). It comprises 164 Python programming problems, each consisting of prompt, a canonical solution, and corresponding test inputs. Each prompt includes a natural language problem description, a function definition, and input/output examples.
The MBPP-Sanitized dataset, proposed by Google, features 427 Python programming problems collected through crowd- sourcing. Unlike HumanEval, is zero-shot dataset, meaning its prompts do include input/output demonstrations. To enhance its utility in experiments, we refined the prompt format by adding function headers and converting natural language instructions into function doestrings.
C. Prompt Templates & Scenario Examples
In the following, we show the scenario examples and prompt templates used during the four steps of DyCodežval process.
13

---

Dynamic Benchmarking of Code LLMs
C.1. Template for Scenario Proposer Agent
Prompt for Scenario Proposer Agent
Suggest
scenarios that provide meaningful context in the following areas: {Si), (S2), {Sa), (54), (5s), and any other practical
fields. Each scenario should be general but applicable, providing useful insight for potential applications.
For clarity, return each scenario on separate line without additional explanation. Use the example below for reference.
Please put your suggested Real-world Scenarios in <scenario></scenallo tags.
Scenario Examples:
<example>
[EXAMPLE]
/example>
C.2. Example for Scenario Proposer Agent
Example for Scenario Proposer Agent
Suggest real worldworld scenarios that provide meaningful context in the following areas: transportation, education, healthcare, banking, social networking, and any other practical fields. Each scenario should be general but applicable, providing useful insight for potential applications.
For clarity, return each scenario on separate line without additional explanation. Use the example below for reference.
Please put your suggested Real-world Scenarios in <scemariox</ccenrrio tags.
Scenario Examples: <example>
Banking Fraud Detection.
/example>>>
14

---

Dynamic Benchmarking of Code LLMs
C.3. Prompt for Context Generator Agent
Prompt for Content Generator Agent
Given the natural language problem description, input types, and a real-world acenario. For each variable in the problem, provide a meaningful context tailored to the given scenaric. The context should explain how each variable is involved in or relates to the acenario, ensuring practical relevance.
Please ensure for to put your meaningful context <context><<centext> tags.
# Problem Description: <probblem.decciittion>>>
(PROBLEM DESCRIPTION]
</problem.ddescrpptionn
Input Types: <input.tyee .tpes>>
[INPUT VARIABLE TYPES] </input.types>
Real-world scenario:
<scenaric>
[SCENARIOS]
</scenaric>
Instructions:
For each variable in the input types, generate only one context that nighlights role or significance within the problem and scenario.
The context should help to clarify the variable's s meaning and importance, ensuring that it fits into the given real-world scenario.
Frovide only the contexts for the variables (no additional reasoning steps).
Example:
## Problem Description Example: peroblem.descript..e
Determine the average temperature city exceeds certain threshold during week.
s/problen.description
## Input Types:
<input types>
temperatures: list of Eloat threshold: float </input.types>
H1 Scenario;
<scenario>
Climate Analysis Menitoring Urban Heat Trends </scenario>
# Generated Contexts:
<context>
temperatures: Daily recorded temperatures in city, analyzed for urban heat
trends.
threshold: Critical temperature level indicating hazardous or abnormal heat. </context>
15

---

Dynamic Benchmarking of Code LLMs
C.4. Example for Context Generator Agent
Example for Context Generator Agent
Given the natural language problem description, input types, and a real-world acenario. For each variable in the problem, provide a meaningful context tailored to the given scenaric. The context should explain how each variable is involved in or relates to the acenario, ensuring practical relevance.
Please ensure for to put your meaningful context in <context><<centex>>> tags.
# Problem Description:
<problem.descriptiinn
You' re given list of deposit and withdrawad operations on bank account that starts with zero balance. Your task is to detect if at any point the balance of account fallls below zero, and at that point function should return True. Otherwise it should return False.
ppprcblem.ddesriipti..
# Input Types: <input.types>
operations: 1ist of int </input.types>
# Real-world Scenario:
Education - Adaptive Learning Assessment and Skil1 Gap Analysis
</scenaric>
# Instructions:
For each variable in the input types, generate only ene context that highlights its role significance within the problem and scenario.
The context should help to clarify the variable's s meaning importance, ensuring that it fits into the given real-world scenario.
Provide only the contexts for the variables (no additional reasoning steps).
Example:
# Problem Description Example: <problem.deserrpp.ioo Determine if the average temperature in city exceeds certain threshold during week.
</problen description>
## Input Types: <input.types>
temperatures: list of float threshold: float </input.typos>
## Scenario:
<scenarig>
Climate Analysis Monitoring Urban Heat Trends </scenario>
# Generated Contexts:
<context>
temperatures: Daily recorded temperatures in a city, analyzed for urban heat
trends.
threshold: Critical temperature level indicating hazardous abnormal heat. </context>
16

---

Dynamic Benchmarking of Code LLMs
C.5. Prompt for Prompt Rewriter Agent
Prompt for Prompt Rewriter Agent
Given a seed programming problem description, a selected real-world scenario, and contextuali zed input variables, rewrite the original problem
make it relevant to the scenario. The rewritten problem should: Preserve the original problem's s complexity and constraints.
Ensure the problem remains solvable with the same solution approach.
Be clear, concise, and logically consistent within the new context.
Just return the rewritten problem description without any additional commentary or steps, and do not include any input output demons in your problem description.
Limit the new rewritten problem description to 1-3 sentences.
Make sure your rewritten problem description is clear, concise and contains unnecessary information.
Please ensure to put your zewritten problem description in <new problem></new problem> tags.
Problem Description:
(PROBLEM DESCRIPTION)
# Real-World Scenario: (SCENARIO)
Contextualized Input Variables: {INPUT VARIABLES]
17

---

Dynamic Benchmarking of Code LLMs
C.6. Example for Prompt Rewriter Agent
Example for Prompt Rewriter Agent
Given a seed programming problem description, a selected real-world scenario, and contextual zed input variables, rewrite the original problem
make it relevant to the scenario. The rewritten problem should: Preserve the original problem's s complexity and constraints.
Ensure the problem remains solvable with the same solution approach.
Be clear, concise, and logically consistent within the new context.
Just return the rewritten problem description without any additional commentary or steps, and do not include any input output demons in your problem description.
Limit the new rewritten problem description to 1-3 sentences.
Make sure your rewritten problem description is clear, concise and contains unnecessary information.
Please ensure to put your zewritten problem description in <new problem></new problem> tags.
Problem Description:
re given list of deposit and withdrawal operations on bank account that starts with zero balance. Your task is to detect if at any point the balance of account fallls below zero, and at that point function should return True. Otherwise it should return False.
Real-World Scenario: Social Networking Advanced Content Recommendation and User Interest Matching
# Contextualized Input Variables:
A sequence of user interaction events (dépesits/viihtraaaass) representing content engagement metrics in a social networking platform, where each operation tracks how users interact with recommended content, potentially influencing their future content visibility and recommendation a algorithm.
C.7. Prompt for Validation Agent
Prompt for Validation Agent
Assess whether the two given natural language instructions convey the same meaning. Respond with Yes! if they do, or 'No" if they do not.
Please ensure your answer is either "Yes" or "No".
# Instruction A: (INstructIOn A}
Instruction B: (INstRuctIOn B}
18

---

Dynamic Benchmarking of Code LLMs
C.8. Example for Validation Agent
Example for Validation Agent 1
Assess whether the two given natural language instructions convey the same meaning. Respond with YYes' if they do, or 'No' if they do not.
Please ensure your answer is either "Yes" "No"
# Instruction A:
For a given list of integers, return tuple consisting of sum and product of all the integers in a list. Empty sum should be equal to and empty product should be equal to 1.
# Instruction B:
In an early disease risk prediction model, develop function that processes list of patient health metrics to calculate comprehensive risk assessment parameters. The function should compute two key aggregate indicators: the total sum of the patient's s health metrics and the cumulative product of these metrics. For scenarios with no available health metrics, the sum should default to 0 and the product should default to 1, ensuring the model can handle incomplete patient data sets.
C.9. Prompt for Validation Agent 2
Prompt for Validation Agent 2
Does the following code solve the problem described in the Instruction? Provide your answer as either 'Yes' or 'No' only.
# Instruction:
[INSTRUCTION]
Code Solution: {CODE SOLUTION]
19

---

Dynamic Benchmarking of Code LLMs
C.10. Example for Validation Agent 2
Example for Validation Agent 2
Does the following code solve the problem described in the Instruction?
Provide your answer as either Yess' or 'No' only.
# Instruction:
In a bank's loan risk assessment pracess, analyze list of an applicant/ key financial metrics to compute an aggregate financial risk score.
Calculate the total sum of these financial indicators and their cumulative product to provide a comprehensive risk evaluation metric. For applicants with no financial history, the sum should default to and the product should default to 1, ensuring standardized risk assessment approach even for new customers.
# Code Solution:
from typing import List, Tuple
def sum_product (numbers: List [int]) Tuplelint, int]: sum_value =
prod_value =
for in numbers: sum_va sum_value +- value n
return sum value, prod_value
D. Human Verification
To add an additional layer validation between the original and sosooddeEva--ggnerattee prompts, we perform a small- scale manual verification. Given a benchmark dataset and the corresponding generated questions, we randomly sample N 30 problem pairs from each dataset (60 in total), where each pair consists of benchmark problem and its generated variant. Each pair is independently reviewed by two graduate-level students complexity are preserved. In cases of disagreement, the reviewers discuss the discrepancies until consensus is reached. Out of the 60 reviewed pairs, the annotators initially disagreed on three but were able to resolve all disagreements through discussion, resulting in an overall agreement rate of 95%.
whether the core algorithm and
20


Reviews:
[{"review_id": "buTSpgGCZT", "reviewer": "Reviewer_3wZN", "summary": "This paper introduces a novel code LLM benchmark that leverages metamorphic testing to address challenges associated with current benchmarks' reliance on publicly available, human-curated datasets.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper and valuable comments.\n\n> ​**Concern about the benchmark's discriminative power**\n\nWe appreciate the reviewer’s feedback and the opportunity to clarify our findings. However, we believe there may be some misunderstandings regarding Figure 5. \n\nFirst, the lower number of outliers in Figure 5 does not imply that our benchmark lacks discriminative power. The key evidence for distinguishing overfitted models is in Figure 4, not Figure 5. In Figure 4, we fine-tune models with controlled contamination and observe a significant accuracy drop on our benchmark as leakage increases (second row). This confirms that models overfitted to contaminated data perform well on the original benchmark but fail on ours.  \n\nIn contrast, Figure 5 evaluates potential data contamination in publicly available LLMs, not discriminative power. Without access to these models’ training data, we cannot confirm overfitting but instead analyze their performance differences. Our findings show:  \n1) A linear relationship between accuracy on our benchmark and the original, indicating comparable problem complexity.  \n2) Anomalous behavior in certain models, such as Qwen2.5-Coder-7B, which experiences an unusually large accuracy drop, falling outside the 95% confidence region. While we cannot confirm contamination, this suggests potential data leakage, which is why we use the term “may be contaminated.”  \n\nSecond, the fact that open-source models perform similarly to closed-source ones does not mean our benchmark is too easy to track future advancements. Our focus is on *transparent* evaluation rather than increased difficulty. A reliable benchmark should measure true generalization while avoiding misleading performance inflation caused by data contamination.\n\n\n\n> **Relationship with EvoCodeBench**\n\nEvoCodeBench is constructed by collecting programming problems from GitHub, following a similar approach to LiveCodeBench, as discussed in our *Introduction* section. However, this method has several limitations: (1) It shifts the burden of manual question design to coding platform authors. (2) Since problems come from public GitHub repositories, existing models may have already seen them, raising concerns about data contamination. (3) EvoCodeBench relies on external contributions, leading to infrequent updates—the latest update, per their [website](https://huggingface.co/datasets/LJ0815/EvoCodeBench/tree/main), was nine months ago, which is inadequate given the rapid pace of model development.  \n\nWhile DyCodeEval is fundamentally different, we acknowledge EvoCodeBench as related work and will include it in the related work section. However, its existence does not diminish our contributions, as DyCodeEval is fully automated and scalable.\n\n\n> **Figure 6**\n\nWe have revised Figure 6, on our [website.](https://github.com/anonymousGithub2022/DyCodeEval/tree/main?tab=readme-ov-file#stability-of-dycodeeval)", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "None", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "1. I have concerns about the benchmark's discriminative power. For instance, in the right subplot of Figure 5 (MBPP), only a few models appear as outliers. Additionally, open-source models seem to perform as well as closed-source models on the new benchmark, which raises doubts about whether the benchmark's difficulty is sufficient to track future model advancements.\n2. The paper lacks references to highly relevant works, such as Li et al. \"EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations\" (NeurIPS 2024) . The existence of such closely related work significantly undermines the claimed novelty.", "other_comments_or_suggestions": "Many presentation details require improvement. For example, the images in Figure 6 appear truncated, which impacts readability.", "questions_for_authors": "Please see `Other Strengths And Weaknesses`.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bHfjTDH4pM", "reviewer": "Reviewer_oZay", "summary": "The paper presents a method for modifying existing LLM coding benchmarks through a 4-stage pipeline to produce new versions of the benchmark that are unlikely to have appeared in training data. This addresses the challenge that LLM developers face when collecting training data and evaluating their models: that data from their evals may appear in the training corpuses, and removing it is non-trivial. The pipeline consists of a scenario proposer, context generation agent, prompt rewriter, and validation stage. The paper gives theoretical consideration to the possibility of collisions in the task rewrite process. Then they employ the proposed process with two small coding benchmarks; to evaluate it, they consider how model performance changes when different amounts of data are leaked from the benchmark. They also examine the performance of a number of in-the-wild models on the original static benchmark and the new dynamic one, identifying that overfit models struggle on the new benchmark, and hypothesizing that QWEN2.5-CODER-7B may have data contamination. The paper also performs evaluations of diversity of the generated tasks, stability of the benchmark (in spite of its randomness), and evaluates whether weaker but cheaper language models can be used for task generation.\nFinally, the paper also introduces a new metric for their dynamic benchmark, DivPass, showing evidence for this metric being more reflective of a models' coding reasoning capabilities than the standard pass@k.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper and valuable comments\n\n---\n> ​**Empirical values of our theorical bounds**\n\nTo empirically evaluate the collision rate of our method, we conduct an experiment on HumanEval. First, we run DyCodeEval on HumanEval to generate an initial set of transformed programming problems. We then repeat this process \\( N \\) times \\( N = 10, 20, 30, 40, 50 \\) and measure:  \n1. **Repeat rate** – the proportion of problems from the initial transformed set that reappear in the subsequent \\( N \\) runs.  \n2. **Collision rate** – the proportion of problems within the \\( N \\) runs that are duplicates of any previously generated problem, regardless of whether they match the initial set.  \n\nTo highlight the advantages of our hierarchical transformation, we compare it against a baseline where we prompt an LLM (using the following prompt) to generate a new programming problem from a given seed. We report both the repeat rate and collision rate for this baseline as well.\n\n\nBaseline Prompt \n```\nRewrite the following problem description to create a new problem description for new scenario\\n\\n\n\nOriginal Problem Description\\n\n{ori_inst}\\n\\n\n\nPlease ensure to put your rewritten problem description in <new_problem></new_problem> tags.\n```\n\n| \\# Of Run | Ours           | Ours                   | Baseline       | Baseline               |\n| --------- | -------------- | ---------------------- | -------------- | ---------------------- |\n|           | \\# of repeated | repeated rate | \\# of repeated | repeated rate |\n| 10        | 0              | 0                      | 3              | 0.018292683            |\n| 20        | 0              | 0                      | 4              | 0.024390244            |\n| 30        | 0              | 0                      | 8              | 0.048780488            |\n| 40        | 0              | 0                      | 9              | 0.054878049            |\n| 50        | 0              | 0                      | 9              | 0.054878049            |\n\n| \\# Of Run | Ours            | Ours                    | Baseline        | Baseline                |\n| --------- | --------------- | ----------------------- | --------------- | ----------------------- |\n|           | \\# of collision | collision rate | \\# of collision | collision rate |\n| 10        | 0               | 0                       | 7               | 0.042682927             |\n| 20        | 0               | 0                       | 17              | 0.103658537             |\n| 30        | 0               | 0                       | 30              | 0.182926829             |\n| 40        | 0               | 0                       | 36              | 0.219512195             |\n| 50        | 0               | 0                       | 39              | 0.237804878             |\n\n\n---\n> ​**Is Sonnet sufficient for more complex tasks**\n\nTo assess whether Sonnet is sufficient for more complex tasks, we apply our approach to LiveCodeBench, a **competition-level** programming benchmark. The following table shows the number of tokens in these three datasets.\nFor LiveCodeBench, we randomly selected 100 seed and transformed programming problem pairs and evaluated their semantic equivalence. Our analysis found that 92 out of 100 pairs were semantically equivalent, demonstrating the effectiveness of our transformation approach.\n\n| Dataset       | Min | Avg.  | Max |\n| ------------- | --- | ----- | --- |\n| LiveCodeBench | 54  | 242.7 | 693 |\n| HumanEval     | 4   | 55.6  | 430 |\n| MBPP          | 7   | 16    | 47  |\n\n\n\n---\n> ​**Other evidence to validate that Qwen2.5-Coder-7B contains data contamination**\n\nAnother indication that Qwen2.5-Coder-7B may be overfitted comes from LiveCodeBench, where its evaluation also shows an unusually large accuracy drop on newly collected programming benchmarks, similar to our findings in Figure 6. \n\n---\n> ​**Essential References Not Discussed**\n\nWe will add all mentioned paper to related work.", "claims_and_evidence": "First the paper introduces a method for transforming an existing benchmark (HumanEval and MBPP, two standard, albeit small and \"toy\" compared to the problems LLMs are commonly used on today, benchmarks for LLM coding capabilities) into a variation of the benchmark unlikely to have been seen in the training data.\n\nIt then makes the following claims about the method:\n1. First it theoretically places bounds on the likelihood of collisions of the scenario and context of the generated task.\n2. Then it measures the effect of various amounts of contamination on the benchmark results, finding the dynamic benchmarking is resistant to contamination. This is the main result of the paper.\n3. When benchmarking in the wild models on the new benchmark, the paper reports model performance and finds evidence that Qwen2.5-Coder-7B may be contaminated.\n\nWe discuss claim 1 in the theoretical claims section below.\nThe evidence for claim 2, that the dynamic benchmarking is resistant to contamination, is persuasive: the evidence in Figure 4 is clear. A limitation of this evidence is that the models were merely fine-tuned, not pre-trained, on the contaminated data.\nThe claim (3) that overfitted models appear as outliers (Figure 5) is reasonable, but there is not hard evidence that Qwen2.5-Coder-7B; the language used for this claim is appropriately couched.\n\nThe introduction of the DivPass metric and the measurement of DyCodeEval's stability is a welcome contribution as well. The stability is strong enough (Figure 6) to make this approach trustworthy as a benchmark even when different random tasks are generated at each application.\n\nFinally there is one additional claim that using weaker LLMs for task generation leads to modest degradation in the quality of eval (the consistency rate from the validation stage of generation drops). This is persuasive to me that Haiku is indeed insufficient as a choice of model for task generation, while Sonnet is sufficient (at least for the seed benchmarks selected in the paper).", "methods_and_evaluation_criteria": "The main method of benchmark generation (scenario proposer, context generation, prompt rewriting, and validation) is a sensible way to generate new tasks unseen by the model during training time, even if the original (or a different derived) version of the benchmark leaked into the training set. One limitation of this method concept is that the core solutions (i.e. the algorithmic insights that the solutions lean on) will still have leaked into the training set, but they will be heavily disguised. With today's models, that scenario/context \"disguise\" may be more significant that with future models.\n\nA limitation of the method is its reliance on a human verification step, which limits the ability to scale up the method.\n\nTo evaluate the method the authors measure the stability of the resulting benchmark using HumanEval and MBPP as the seed benchmarks, they measure the benchmark's robustness to (a fine-tuning based approximation of) data contamination, and they measure in the wild performance and speculate about data leakage in real models. They also measure stability to regeneration of the benchmark. These are each quite sensible measures of evaluating the benchmark generation method.\nThe seed benchmarks are limited in scope.\nThe robustness to data contamination experiment only uses fine-tuning, not pre-training, which limits the conclusions we can draw.\nThe true data leakage information about Qwen isn't known, so our ability to measure the benchmark's true leakage-detection ability is limited.\nBut overall these experiments are compelling, demonstrating that (at least for these simple seed benchmarks and today's models), the approach is sound for generating data contamination resistant variations of the benchmark that are robust to regeneration.", "theoretical_claims": "The theoretical claims are correct given the assumptions stated and the proofs are sound. However, the theorems don't actually tell you what you might think/hope they do on a first read. The theorems correctly bound the probability of a collision between two generated examples provided that the scenario generator generates |S| distinct scenarios, and for each scenario the context phase generates |C| distinct contexts. In generating the |S| scenarios, however, there is the possibility of a collision or near collision (i.e. two very similar scenarios). Similarly when generating contexts conditioned on a scenario, there is the possibility of a collision or near collision. i.e. S or C could contain near-collisions, and the likelihood of this seems more significant that the values given by the bounds.\n\nIf indeed we are only concerned with exact collisions, the randomness in the rewriting phase reduces that likelihood considerably.\n\nAre there empirical values that would make sense to show for these bounds? E.g. you could demonstrate what dataset sizes admit what amounts of generated examples without collision.\n\nNote the notation used in theorems 2 and 3 is missing the bars left of S.", "experimental_designs_or_analyses": "See notes in Claims and Methods sections.", "relation_to_broader_scientific_literature": "There are many code generation benchmarks for LLMs, of which HumanEval and MBPP are two examples. However, the literature routinely recognizes data contamination as a challenge for properly evaluating LLMs.\n\nGeneralization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, Ge Li https://arxiv.org/abs/2402.15938\n\nThis paper proposes a method to mitigate the challenges of data contamination by using LLMs to modify existing benchmarks.\nIt stands in contrast to other data contamination mitigation approaches like non-LLM rewrites of existing benchmarks (see next section of review) and manually curated time-cutoff benchmarks like LiveCodeBench.", "essential_references_not_discussed": "Please also see Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models https://arxiv.org/abs/2503.06643, a concurrent work released just in the last few days. (No expectation that it already have been in your paper, of course!)", "other_strengths_and_weaknesses": "See other sections.", "other_comments_or_suggestions": "The pdf repeatedly ran into rendering issues on my machine; my first guess would be that the images in Figure 5 are too large, but I have not investigated further. It might just be an issue on my end. I report this just for your information.\n\nline 12 (the first line) typo: missing space\nline 19: remove \"to be\"\nWhy is Section 5 titled \"Application\"? That seems like an oversight.", "questions_for_authors": "Is Sonnet sufficient for applying this task generation procedure to more complex tasks than HumanEval and MBPP?\n\nIs there evidence to validate the speculation that Qwen2.5-Coder-7B contains data contamination (or that the other models do not) beyond that from this benchmark?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eokK2sE2oa", "reviewer": "Reviewer_eBWT", "summary": "This paper proposes a framework for augmenting existing coding model evaluation datasets by coming up with new scenarios and contexts to generate semantically similar evaluations. The authors use several LLM steps to produce these questions, and evaluate models while attempting to simulate data contamination. The authors compare their generated evaluation questions against other forms of manipulation, and analyze how the performance of a set of models changes.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper and valuable comments\n\n>**Semantically Equivalent Validation and Human Study**  \n\nBy “semantically equivalent variations,” we mean that the generated problems can be solved by the same code solution as the original. We validate this through:  \n\n1. **Automated Validation:**  \n   A large language model (LLM) acts as a probabilistic oracle to check:  \n   - Whether the rewritten problem retains the original meaning.  \n   - Whether the original solution remains valid for the new problem.  \n\n2. **Human Verification (Appendix D):**  \n   Two graduate students independently reviewed *N = 30* problem pairs per dataset (60 total), assessing whether the core algorithm and complexity were preserved. They initially disagreed on three pairs but reached consensus after discussion, yielding a 95% agreement rate.  \n\nWe provide the sampled data on our [website](https://github.com/anonymousGithub2022/DyCodeEval/blob/main/resource) (see four CSV files in the directory).\n\n\n>**Finetuning details**\n\nWe fine-tune the selected code LLM on randomly sampled portions of the benchmarking dataset, ranging from 25% to 100%, using a standardized instruction tuning objective. The fine-tuning process employs a learning rate of 5e-5, a batch size of 8, and runs for 20,000 steps.  \nWe acknowledge that fine-tuning on a small subset can impact the instruction-following capabilities of the model due to overfitting. However, this phenomenon is precisely the risk posed by data contamination—overfitted models exhibit artificially high performance on contaminated benchmarks, creating a false sense of intelligence while sacrificing generalizability. This issue is empirically demonstrated in Figure 4 (first row), where the red bars highlight the inflated accuracy due to overfitting, while the blue bars indicate the degradation in general capabilities. The presence of such overfitting underscores the need for contamination-free benchmarking, which is the primary motivation of our work.  \nRegarding pretraining leakage, we note that instruction fine-tuning can override or mitigate the effects of pretraining data exposure. Our study does not aim to analyze pretraining leakage directly but rather to simulate its effects in a controlled manner. To achieve this, we follow established methodologies in the literature, where instruction-tuning-stage leakage is widely used to approximate the impact of training data contamination. This approach allows us to systematically examine how leakage-induced overfitting distorts benchmarking results.\n\n\n\n\n>**Unnecessary theorems**\n\nWe strongly believe that these theorems are both necessary and valuable. They provide probabilistic guarantees to benchmark providers, ensuring that an entity with ulterior motives cannot easily overfit a model to achieve artificially high scores on our benchmarks. Thanks to our hierarchical transformation framework, we can control the search space at each transformation layer, effectively mitigating the risk of collisions. This approach allows us to maintain a manageable search space at each layer while achieving a significantly large total search space.  \n\nThe claim that *\"prior LLM steps could simply be repeated or resampled (line 252) in case of a collision\"* does not apply to our scenario. Benchmark providers keep their scenario and context pools private to prevent manipulation rather than expose them for adversarial exploitation. As these pools act as a private key, our framework ensures transparent benchmarking while minimizing contamination risk. While brute-force overfitting is possible, our theorems show it would require an impractically large number of trials, making it infeasible.\n\nTo highlight the advantages of our hierarchical transformation for reducing the rist of collision. We conduct a empirical evaluation, the setup and results are shown on our [website.](https://github.com/anonymousGithub2022/DyCodeEval/tree/main?tab=readme-ov-file#collision-results)\n\n\n\n>**Baselines in Sec4.4**\nWe did describe and cite the baseline methods in Section 4.4. Below, we provide further details on the specific mutations:  \n- **Token Mutation**: Randomly replaces a token in the original prompt with another token.  \n- **Char Mutation**: Randomly inserts a character at a random position in the original prompt.  \n- **Func Mutation**: Changes the function name style in the prompt, e.g., renaming \"MyMethod\" to \"my_method.\"  \n- **Insert Line**: Randomly inserts blank lines in the original prompt.  \n- **CommSyntax**: Modifies the syntax of comments in the prompt (e.g., changing `# comment` comments to `\"\"\"comment\"\"\"` style).  \nThese mutations are derived from the robustness-based mutations proposed by Wang et al. (2023). Additionally, **PPM** (Chen et al., 2024) concatenates the original problem description with a newly defined problem description to test robustness.  \nWe used publicly available implementations to ensure consistency and reproducibility.", "claims_and_evidence": "In 234-244, the authors claim that their collection of 3 models (Llama 3.2-1B, Llama 3.2-3B, DeepSeek-Coder-1.3B) is diverse \"in terms of model architecture, model size, and training methods). This is problematic, as they explore a truly diverse set of models in the following section (12 additional models). Of the 3 original, 2 are the same model family, and all 3 are small models. \n\nFurthermore, the abstract claims that their method creates \"semantically equivalent variations\", but this does not seem to be validated. The authors claim to perform a human study but do not include details to verify further.", "methods_and_evaluation_criteria": "The benchmarks used make sense. However, much of the evaluation does not make sense. There are insufficient details on how the authors finetune models to simulate leakage. Fine tuning on a small subset of documents could heavily impact the instruction following capabilities of these models if not considered carefully. Furthermore, training directly on (synthetic) evaluation questions does not simulate pretraining leakage, where implementations might be contained within a larger dataset. \n\nWhile their capabilities are more limited, there are open data code models that the authors could use for an exact leakage study (Starcoder).", "theoretical_claims": "This paper includes unnecessary theorems and claims to analyze the collisions of their method when generating $|S|$ scenarios and $|C|$ coding problem contexts. This analysis and the page of proofs in the appendix appear to be typical statements of balls-and-bins, coupon collector, and hash table collision type problems. These do not add to the substance of the paper except to include more notation and an appendix section. This also does not address the fact that any of the prior LLM steps in this work could simply be repeated or resampled (with temperature, line 252) in the event of a collision.", "experimental_designs_or_analyses": "Details around finetuning are underspecified. 4.4 does not give details about mutations used for comparison. \n\nAll analysis is harmed by the fact that many of the figures have extremely small text, cut off labels, or overlapping content. \n\nThe details of the human evaluation are extremely under specified. The appendix simple states \"the consistent rate is around 95%\" with no further numeric details.", "relation_to_broader_scientific_literature": "This paper studies leakage and analyzes common benchmarks and code models, largely following established literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. Interesting approach to augmenting evaluations\n2. Considers that several of the main evaluations in this area may suffer from leakage\n\nWeaknesses:\n1. Analysis is very difficult to understand, compounded by the fact that figures contain extremely small text or overlapping content. \n2. Some essential experimental details are not considered (see above)\n3. Paper spends substantial content on explaining a series of straightforward LLM prompts as agents. \n4. Similarly include sections that do not contribute to the main points (Algorithm 1 is a typical process, section 3.3 does not add to the content). This space would be better used for explaining missing details.\n5. Makes claims about semantic equivalence between generated problems but does not sufficiently justify.", "other_comments_or_suggestions": "Typo in the first line of the abstract\n763, 765 further typos. \nTypos in your prompts 676", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q5qbvNxo0Z", "reviewer": "Reviewer_TWyo", "summary": "This paper proposes DyCodeEval: a dynamic benchmarking approach designed to evaluate the reasoning capabilities of Large Language Models on code tasks under potential data contamination. By starting with a seed programming problem, DyCodeEval leverages multiple agents to extract and modify problem contexts—without altering the core logic—to generate semantically equivalent variations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments.\n\n> **How Our Dynamic Metric Mitigates Data Contamination**\n\nFor the static metric **pass@k**, the same fixed problem prompt is fed to the LLM multiple times, leveraging its sampling capability to generate different outputs. However, since this prompt is publicly available and remains unchanged, **pass@k** becomes unreliable if the prompt is contaminated in the LLM’s training data.  \n\nIn contrast, our proposed **DivPass@K** generates multiple randomized problem mutations using our approach before feeding them to the LLM. These dynamically generated prompts are not static, publicly available, or present on the Internet, reducing the risk of data contamination. We will clarify the working mechanism of our dynamic metric in the final version.\n\n\n> **More challenging and uncontaminated benchmarks**\n\nWe also applied our approach to LiveCodeBench, a newly collected competition-level programming benchmark sourced from LeetCode and other platforms. The results, shown in the table below, demonstrate that DyCodeEval can be effectively applied to challenging and uncontaminated benchmarks.  \nNotably, Qwen2.5-Coder’s accuracy did not drop as significantly as it did from HumanEval to LiveCodeBench. This is because LiveCodeBench was released after Qwen2.5-Coder, reducing the likelihood of data contamination.\n\n|   Model                      | LiveCodeBench | LiveCodeBench +Ours |\n| ---------------------------- | ------------- | ------------------- |\n| CodeLlama-13b-hf             | 21.4         | 18.3               |\n| CodeLlama-7b-hf              | 15.6          | 13.5                |\n| DeepSeek-V2-Lite             | 41.4         | 39.4               |\n| Llama-3.1-8B-Instruct        | 21.3         | 20.8               |\n| Qwen2.5-Coder-7B-Instruct    | 39.4          | 36.5                |\n| deepseek-coder-1.3b-instruct | 22.1          | 19.3                |\n| claude-3.5-haiku             | 59.4         | 60.3               |\n| claude-3.5-sonnet            | 67.7          | 67.6                |\n\n\n> **Uniform sampling in our assumption**\n\nWe will revise our theorems and clarify that they rely on the assumption of uniform sampling.\n\n\n> **Other evaluation metrics**\n\n\nBesides the correctness metric, we also consider the test case pass rate to evaluate partial correctness. The results are shown in the following table. We observe that, after applying our transformation, the test case pass rate increases for some models. This is because our transformation generates diverse variants of the problem, which may change the models reasoning and make it poential to get a partial correct solution.\n\n| Model                        | HumanEval | HumanEval + Ours |\n| ---------------------------- | --------- | ---------------- |\n| CodeLlama-13b-hf             | 0.38      | 0.37             |\n| CodeLlama-7b-hf              | 0.29      | 0.33             |\n| DeepSeek-Coder-V2-Lite-Base  | 0.19      | 0.22             |\n| DeepSeek-V2-Lite             | 0.29      | 0.21             |\n| Llama-3.1-8B                 | 0.38      | 0.39             |\n| Llama-3.1-8B-Instruct        | 0.63      | 0.56             |\n| Llama-3.2-1B                 | 0.20      | 0.15             |\n| Llama-3.2-3B                 | 0.31      | 0.33             |\n| Qwen2.5-7B                   | 0.55      | 0.46             |\n| Qwen2.5-7B-Instruct          | 0.63      | 0.56             |\n| Qwen2.5-Coder-7B             | 0.65      | 0.37             |\n| Qwen2.5-Coder-7B-Instruct    | 0.76      | 0.71             |\n| deepseek-coder-1.3b-instruct | 0.54      | 0.43             |\n| claude-3.5-haiku             | 0.86      | 0.78             |\n| claude-3.5-sonnet            | 0.96      | 0.85             |", "claims_and_evidence": "The motivation for handling data contamination is well-founded.\n\nTraditional benchmarks, such as HumanEval and MBPP, may have been seen by models during training; consequently, their results could reflect memorization rather than genuine reasoning ability. The authors’ core idea—using a dynamic process to revise benchmark problems—is promising for mitigating contamination issues. \n\nStill, further elaboration on how they quantitatively measure contamination levels would be helpful. The paper mentions that prior contamination metrics may not align with real-world cases; clarifying the proposed metric or methodology for gauging contamination would strengthen the claim.", "methods_and_evaluation_criteria": "Benchmark Selection: The work primarily focuses on HumanEval and MBPP—two standard code generation datasets that have been around for a while and on which models often perform quite well. It would benefit readers to see experiments on more recent datasets or tasks not so heavily covered in prior training data. Additionally, expanding beyond code completion tasks to more diverse or “harder” datasets could further validate DyCodeEval’s utility.\n\n\nPlan for More Diversity: While the authors mention multiple agents generating various problem contexts, the paper could better highlight how these newly created tasks genuinely probe the model’s reasoning rather than just superficial text changes. If the final results remain similar in difficulty and performance, additional clarity on whether the plan’s quality improves over iterative changes is needed.", "theoretical_claims": "The proofs follow standard probability and combinatorial arguments.\nOne potential point for further clarity might be to highlight assumptions of “uniform sampling” in each theorem  (i.e., that scenarios and contexts are chosen with equal probability), as real-world usage could introduce slight biases.", "experimental_designs_or_analyses": "The authors suggest that DyCodeEval mitigates data contamination by dynamically generating semantically equivalent but contextually distinct problems. \n\nBeyond final accuracy, incorporating additional metrics (e.g., measuring plan complexity, solution clarity, or partial correctness) could offer a more nuanced view of LLM reasoning skills. \n\nSimilarly, comparing performance on newly generated problems that are known to be uncontaminated against older, potentially contaminated benchmarks would illustrate the impact of this approach more concretely.", "relation_to_broader_scientific_literature": "Robustness\nCode LLM Evaluation\nData Contamination", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "see above", "other_comments_or_suggestions": "see above", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript proposes a dynamic benchmarking strategy for evaluating the reasoning capabilities of Large Language Models on code tasks under potential data contamination. The method relies on a seed dataset/programming problem, from which it generates semantically equivalent variations. It does so by instructing LLMs to extract and modify problem contexts without altering the core logic. Evaluation using 2 standard code generation datasets (HumanEval & MBPP) on several state-of-the-art code LMs in the 7B range shows the merits of the proposed approach over static evaluation strategy where contamination is a key issue.

The motivation of the work is clear, the work is well-founded, and makes an attempt in the right direction to help robust evaluation of large language models (coding domain in this case). With all benchmark papers in the rapidly evolving LLM landscape, we can consider two questions: (1) Does the proposed process/strategy create more difficult questions than the existing questions to challen, rather than attempting to create just another dataset? (2) What value does the benchmark add over existing ones, in terms of utility, metrics, diversity, maintenance, etc. I think the paper answers both these questions satisfactorily. 

The reviewers point out similar approaches for dataset creation in the recent literature, but the authors' rebuttal alleviates serious concerns. The two reviewers remain positive after the rebuttal, and the authors have satisfactorily addressed the concerns raised by the two negative reviewers. 

Therefore, I recommend accept. I request the authors to incorporate the comments in the final version (for instance, avoiding strong claims like 'semantic equivalence' without formal statements).
Decision: Accept (poster)