ICML 2025

DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities
in Code Large Language Models Under Data Contamination
Simin Chen 1 Pranav Pusarla 1 Baishakhi Ray 1
1Columbia University
Abstract
The rapid advancement of code large language
models (Code LLMs) underscores the critical
need for effective and transparent benchmarking
methods. However, current benchmarking pre-
dominantly relies on publicly available, human-
created datasets. The widespread use of these
static benchmark datasets makes the evaluation
process particularly susceptible to data contam-
ination—an unavoidable consequence of the ex-
tensive data collection processes employed dur-
ing LLM training.
Existing methods for ad-
dressing data contamination typically face sig-
nificant limitations, including reliance on sub-
stantial human effort and difficulty in manag-
ing class imbalances. To overcome these chal-
lenges, we propose DyCodeEval, a novel bench-
marking suite specifically designed to evaluate
Code LLMs under realistic contamination scenar-
ios. Given an initial seed programming problem,
DyCodeEval utilizes multiple agents to system-
atically extract and modify contextual informa-
tion without changing the core logic, generating
semantically equivalent variations. We introduce
a dynamic data generation method and conduct
extensive empirical studies on two seed datasets
involving 18 Code LLMs. The results demon-
strate that DyCodeEval effectively assesses the
reasoning capabilities of Code LLMs under con-
tamination conditions while producing diverse
problem variants, thereby ensuring robust and
consistent benchmarking outcomes. Our project
webpage can be found at this link1.
1Department of Computer Science, Columbia University. Cor-
respondence to: Simin Chen <sc5687@columbia.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1https://codekaleidoscope.github.io/
dycodeeval.html
1. Introduction
Large ...