PrismBench: Dynamic and Flexible Benchmarking of LLMs Code Generation with Monte Carlo Tree Search
Authors: Vahid Majdinasab
Amin Nikanjam
Foutse Khomh
Venue: Accepted by TMLR ()

Abstract: The rapid advancement of LLMs' code generation capabilities is outpacing traditional evaluation methods. Static benchmarks fail to capture the depth and breadth of LLM capabilities and eventually become obsolete, while most dynamic approaches either rely too heavily on LLM-based evaluation or remain constrained by predefined test sets. To address these issues, we introduce PrismBench, a multi-agent, dynamic benchmarking framework designed to systematically expose and analyze LLM failure modes in code generation tasks. We formulate evaluation as a Markov Decision Process over a structured tree of coding challenges, leveraging a customized Monte Carlo Tree Search algorithm to traverse this tree and discover high-failure scenarios. Our multi-agent setup orchestrates task generation, model response, and analysis, enabling scalable assessment across diverse coding challenges. Additionally, we propose metrics that combine structural traversal patterns with performance across different tasks and difficulty levels to enable diagnostic and systematic comparison of LLMs' performance. We conduct extensive experiments on eight state-of-the-art LLMs and analyze how model architecture and scale influence code generation performance across varying coding tasks. All code, evaluation trees, and a public leaderboard are available at https://prismbench.github.io/Demo/

Published in Transactions on Machine Learning Research (03/2026)
Dynamic and Flexible Benchmarking of LLMs Code Generation with Monte Carlo Tree Search
Vahid Majdinasab
Polytechnique Montreal, Montreal, Quebec, Canada
Amin Nikanjam"
Huawei Distributed Scheduling and Data Engine Lab, Conada
Foutse Khomh
Polytechnique Montreal, Montreal, Quebec, Canada
Reviewed on OpenReview: https: // opw.reviee.........eomo. net/ formm? id= 00bs06FDLy
Abstract
The rapid advancement of LLMs' code generation capabilities is outpacing traditional evaluation methods. Static benchmarks fail capture the depth and breadth of LLM capabilities and eventually become obsolcte, while most dynamic approaches cither rely too heavily on LLM-based evaluation remain constrained by predefined test sets. To address these limitations, we introduce PrismBench, dynamic benchmarking framework grounded in Reinforcement Learning that comprehensively evaluates LLMs by framing code generation evaluation as an RL task. We formulate the evaluation process as a Markov Decision Process over a structured tree of coding challenges, leveraging customized Monte Carlo Tree Search algorithm traverse this tree and discover high-failure scenarios. Our multi- agent sctup orchestrates task generation, model response, and analysis, enabling scalable assessment across diverse coding challenges. Additionally, we propose metrics that combine structural traversal patterns with performance across different tasks and difficulty levels to enable diagnostic and systematic comparison of LLMs performance. We conduct large-scale experiments on eight state-of-theear LLMs and analyze how model architecture and scale influence code generation performance across varying coding tasks. All code, evaluation trees, and a public leaderboard are available at https::/ppismbertt wreeenchh...tiii github. 1o/Demo/
1 Introduction
The rapid advancement of Large Language Models (LLMs) for code generation has outpaced existing evaluation methodologies. Both static and dynamic benchmarking approaches exhibit fundamental limitations that hinder their ability to comprehensively assess evolving models' capabilities. Static benchmarks, while widely used, suffer from three key weaknesses. First, they provide a limited and often superficial assessment of an LLM's capabilities, reducing evaluation pass/fas /fail metrics that fail to capture the complexity of the model's reasoning McIntosh et al.] (2024); Banerjee et al. (2024); Tambon et al. (2024a). Second, benchmarks gain popularity and become evaluation targets, they are increasingly prone to being incorporated into LLMs' training data, leading to data leakage and artificially inflated performance metrics Xu et al. (2024a); Zhou et al. (2023). Third, static benchmarks lack flexibility, preventing a targeted evaluation of specific problem-solving strategics or particular aspocts model behavior McIntosh et al. (2024). To address these limitations, dynamic benchmarking approaches have been introduced Zhu el. al. (2023b); Li al. (2023); Wang al. (2023a): [Zhang al. (2024c); Li et al. (202cc). Most dynamic approaches typically adopt of two strategies. The first relics LLM-as-Judge frameworks, where another LLM is used to
Work completed while Polytechnique Montreal.

---

Published in Transactions on Machine Learning Research (03/2026)
evaluate the responses Zhu et al. inherently unreliable, as evaluation outcomes are constrained by the limitations and biases of the judge model. The second strategy involves dynamically selecting test cases from predefined datasets based on model performance Zhang et al. (2024c); Kiela et al. (2021). Although more structured, these methods remain fundamentally tied static lest. sets, limiting their capacity to evolve alongside increasingly capable models. These issues indicate a clear need for evaluation frameworks/ woaerkkssmmetoooooooo hodologgs that can adapt to the increasing capabilities of LLMs, provide systematic and reproducible evaluation pipelines, and reveal comprehensive insights about the model's performance and weaknesses.
(2023b): Li et al. (2024a) While this approach offers adaptability, it is
We introduce PrismBeench a dynamic, multi-agent benchmarking framework for evaluating LLMs' code generation capabilities. We formalize the space of all possible evaluation scenarios as a Markov Decision Process (MDP) Sutton & Barto (2018), where each state represents a unique combination of programming concepts (e.g., recursion, data structures) and difficulty levels (e.g., easy, medium, hard). Within this formulation, the evaluation process becomes search problem where the objective to search this space to discover evaluation scenarios (i.e., states) where models consistently fail. To systematically conduct this search, we instantiate the MDP search tree and use modified Monte Carlo Tree Search (MCTS) Russell & Norvig (0166) algorithm to traverse it, balancing exploration of new evaluation secnarios with exploitation of discovered high failure regions of the evaluation space. Instead of sampling from static dataset, we generate each evaluation scenario dynamically based on the current state's concepts, difficulty, and prior model performance to mitigate benchmark memorization and data leakage. In this manner, as the search progresses deeper into the tree, the generated evaluation scenarios become progressively more challenging and allow for structured and adaptive assessment. of model capabilities.
Unlike prior dynamic evaluation approaches that rely on LLMs to judge each other's outputs, PrismBench assesses model performance by executing the generated solutions within a multi-agent sandbox. This provides objective, reproducible signals based on functional correctness, rather than textual similarity or the prrors/preferenccs of the judge model Thakur et al. (2024). Instead, LLM agents in PrismBench are assigned analytical roles, for example, analyzing execution traces to identify failure patterns or examining solutions to understand how models approach specific challenges. This separation ensures that evaluation remains grounded in actual model performance, while LLM agents assist in interproting the results and identifying patterns in model behavior. Additionally, in contrast to pass/fal metrics (e.g., accuracy, precision, etc.), which mask how models approach problems and offer little insight into their performance, we introduce metrics that track model performance across tasks, difficulty levels, and structural exploration paths, allowing for more fino-grained diagnostics and comprehensive assessments.
Overall, the contributions of our work are as follows: (1) We introduce a dynamic benchmarking framework that adapts model capabilities and systematically explores the space of programming challenges. (2) We propose an extensible, multi-agent evaluation architecture that assesses multiple aspects of LLMs' code generation capabilities. (3) Using PrismBenchh, we evaluate eight leading LLMs to identify their failure patterns and capability boundaries.
(4) We relcase the code for PrismBench alongside a leaderboard and an interactive showcase for each of the studied models at httts:://prismbbee ht//prrsmbbecch..m wraeancchh.httttt. github. io/ommmoo
To structure our work, we define the following RQs:
RQ1:
How can we design evaluation lasks that adapt to model capabililies and surface hidden failures?
To answer this question, we introduce PrismBench a dynamic, multi-agent benchmarking framework that models the evaluation process as a search problem over the space of all possible evaluation scenarios. Furthermore, instead of selecting evaluation scenarios blindly or from a static pool of pre-defined challenges, PrishBench dynamically generates tasks in response to the model under the benchmark's performance, which allows for probing edge cases, subtle weaknessos, and regressions that are otherwise missed in static benchmarks.
RQ2:
How can we evaluate different models in a dynamically generated benchmark!
To answer this question, we ground PrishBench Reinforcement Learning (RI) by decoupling the benchmarking environment from the benchmarking process. We formalize a single, shared environment. for all models and evaluate them by their traversal trajectories in this environment. To
2

---

Published in Transactions on Machine Learning Research (03/2026)
further reduce the risk of bias arising from LLM stochasticity, we design multi-agent interaction sandbox that tests the models' capability in test generation, solution generation, and program repair, separately. Furthermore, we propose a set of performance metrics that can be used to compare models based on their performance throughout the search process. We analyze how models perform under identical dynamic conditions, compare their behavior across multiple failure modes, and show how PrismBench enables more fine-grained model capability comparison than static benchmarks.
RQ3: How can dynamic benchmarks provide insights indo model performance, behavior, and failures?
To answer this question, we examine the outputs collected during the evaluation process for each model and propose a set of diagnostic metrics that offer a richer view of model performance and help identify patterns in how and why models fail.
2 Motivating Example
In this section, we discuss how current static and dynamic benchmarking approaches evaluate LLMs alongside their shortcomings. Regardless of the underlying evaluation methodology (static or dynamic), all benchmarking approaches face bias/variams
trade-off. Here, bias property of the evaluation dataset which contains tasks that consistently over over/unders /underrestimate model's true capability (e.g., task overlap with pretraining data, coarse aggregate pass/fai fail metrics, uneven task distributions, etc.) Xu et al. (2024b); Roberts et al. (2023) Jiang et al. (2024). On the other hand, variance is a property of the sampling process and is a result of instability in evaluation estimates between different runs arising from small sampling pools from the underlying dataset or too few per-task trials (e.g., one extra attempt flips a failure to a pass) Dong et al. (2024); Chiang ctt al. (2024): Lin al. (2024). Given the high cost of qucrying LLMs and finite evaluation budgets (either time, compute, or both), to comprehensively evaluate model, each benchmarking methodology must focus a specific aspect of this trade-off: either increase coverage task variety and decrease sampling per task which results in increasing variance between evaluation runs or increase resampling or per-task trials and instead focus on a narrow subset of tasks which results in high bias of evaluation results between different evaluation subsets.
Consider two LLMs, MA and MB, and a bank of four coding tasks, Ti..... T4 from an evaluation dataset (e.g., HumanEval Chen et al. (2021), MBPP Austin et al. (2021). ARC-AGI Chollet (2019), etc.). In a standard static benchmark, both MA and MB evaluated effect 72), and their overall performance is summarized with aggregate metrics (e.g., pass@k, binary pass/fail over each lask, etc.). Suppose MA solves Ti, T2, T3 but fails T4, while Mn solves Ti, T3, T4 but fails T2. Under this setting, both models received a 3/4 score even though their failures were on different tasks. Such aggregate pass/fail metrics hide where and how each model fails, and compress different behaviors into the same score. While exhaustive per-task trials and normalization of scores over the number of attempts average out stochasticity and allow for fairer comparison between models, two problems remain:
each task, independently (i.e., performance on Ti has
If MA consistently fails Ta and Mg consistently fails T2, both models will still receive the same score; however, each model has failed on a different task. Here, the same score is an artifact of coarso-grained metrics, not evidence of oqual capability.
Given the scale of pretraining corpora and the unavailability of training data, some tasks may have been included in the models' training dataset. Therefore, the subsequent successes might be a result of memorization, not generalization Balloccu et al. 2024).
Moreover, as the set of underlying tasks grows, multiple samples per each task in the dataset become infeasible (e.g., for a dataset of 1/ tasks and 10 trials per task, 10k queries to the model are required), forcing the bias/variance trade-off mentioned above: either the number of distinct tasks must be reduced (lower variance but higher bias from reduced coverage and select ion effects) or fewer trials per each task need to be carried out (lower bias but higher variance between runs).
Dynamic benchmarking approaches aim address these limitations by: (1) using multi-step task bundles by decomposing task into sequential subtasks that the model must pass end-to-end (e.g., SWF-bbenh Jimenez)
3

---

Published in Transactions on Machine Learning Research (03/2026)
Benchmark Self-Evolving Wang et al. (2024) ) (2) adaptively selecting challenges from fixed bank by picking the next task conditioned on prior evaluation results (e-g., DARG Zhang et al. (2024c). Dy Zhu et al. (2024b)), or (3) generating tasks at test time to mitigate memorization (e.g. Va Zhu ct al. (024a). TreeEval ct al. 2024c)). However, each approach comes with its own bias bias/ faasvvrriance trade-off.
et al.
p024p,
Specifically:
Testing each model on multi-step task bundles lowers the chances of memorization and superficial success, but introduces fairness problem in how models evaluated against cach other. MA and Mn will be lested on different distribution of sub-tasks and aggregate scoring (only end-to-end success being counted) introduces variance in evaluation results. As such, results depend heavily on the specific bundle graph and scoring rule, not just underlying capability.
Conditioning the next task on past outcomes allows for more informative exploration of the evaluation space, but comparability issues between models' evaluation results still remain. Ma and Mn are now evaluated on different task distributions, incidental failures can result in different evaluation trajectories. Stabilizing estimates still requires multiple samples per task, and given fixed evaluation budgets, variance stays high unless task coverage is narrowed, which reintroduces bias via selection effects.
Testing each model on a problem generated at test time from fixed distribution addresses the memorization problem and aligns the underlying task distribution across models. However, comparable estimates of models' performance still require many samples per task to control for variance, which is costly under finite budgets.
As such, both static and dynamic benchmarking approaches suffer from two main shortcomings: (1) high bias of evaluation results due to different evaluation task distributions and (2) high variance of evaluation results due LLMs' stochasticity and limited sampling budgets Blackwell et al. (2024). In the next section, will first describe Prish/Bench's RL framing, which allows for defining fixed task distribution for all Afterward, we establish our search methodology. which allows for adaptive sampling budget allocation through MCTS-guided exploration and enables efficient use of limited benchmarking resources by focusing on informative regions of the evaluation space.
3 Methodology
In this section, we first provide an end-to-end overview of PrismBench'' evaluation pipeline in Section 3.1 Next, we detail how we formalize the search space as an MDP and model it as a tree in Section 3.2 Afterward, we explain the details of our agents, our multi-agent. orchestration approach, and the evaluation workflow at each node in Section 3.3 In Section 3.4 we explain the rationale behind a multi-phase evaluation pipeline, the details of cach phase's objective, and how they integrate with the overall evaluation process. Scction 3.5 outlines how we ensure the validity of dynamically generating challenges at. Lest-time, and finally, we detail our proposed metrics for evaluating models' performance in Section 3.6
3.1 PrismBench Overview
As mentioned Section 2 evaluating models against a large bank of tasks under a finite evaluation budget forces a bias/variance trade-off. Conditioning the choice of the next evaluation task on prior observations mitigates this by directing trials towards informative regions of the evaluation space (i.e., evaluation tasks that maximize information gain regarding models' capability). To formalize this process, model the space of all possible evaluation scenarios as an MDP, where cach state corresponds to a unique scenario defined by a (c, d) pair, with e being a list of programming concepls (e.g., recursion, data structures) and d being the task's difficulty level (e.g., easy, medium, hard). We instantiate the MDP as a search tree, where nodes represent states and edges represent transitions between scenarios, corresponding to actions that either combine additional concepts or increase difficulty. A node's depth is defined as the number of nodes along the path to the root, and child nodes may extend share the same concepts their parents.
4

---

Published in Transactions on Machine Learning Research (03/2026)
In this manner, the search tree forms fixed evaluation environment shared between all models (identical state space and transition dynamics), where transitions between nodes reflect a model's ability to generalize and solve increasingly complex tasks. Furthermore, this formalization forms a hierarchical dependency between evaluation tasks and reduces variance by concentrating sampling where outcomes are uncertain or high impact, and reduces bias by slandardizing the task distribution between all models. As such, all models are evaluated in the same environment, based on how they explore and perform within Sutton Barto (2018).
To address the brittleness of end-to-end task bundles, where early misstep can cascade into full failure and result in different trajectories, each node in the tree is evaluated using a multi-stage, isolated pipeline. Each task is decomposed into role-based steps (e.g., test generation, solution generation, program repair, etc.) where each step is handled by a dedicated LLM agent. Agents interact with each other via a shared node state, which allows for modularity (addition or removal of agents) and avoids information leakage. This decoupling allows for evaluating multiple capabilities within a single node (c.g., correctly understanding the requirements, solution/ /Lest generation, debugging, etc.) without allowing failures/errons in one step dominating the final outcome. Importantly, composite weighted score derived from exocution-grounder signals (i.c., unit-test pass rates, error traces, and bounded retry /eery/repair attempts). As such, the reward signal reflects the specific steps responsible for success or failure instead of collapsing everything into a binary pass/ fail signal for the entire task.
do not rely on LLM judgments: node's reward is a
Search Space
Structure
Phase-specific Evaluation
Selection
Phase 1: Capability Mapping
Roat
With probability Best (UCB1)
Map basic capabilities
With probability 1-:: Random child
Build inttial tree structure
Loops
Sorting
Identify areas of proficiancy
Easy
Expansion
Locps Ar
Sorting
Merge concepts parents
Challenge Discovery
Scale difficulty on success
Find challenging concepts
Focus on weak areas
Simulation
Discover failure patterns
Generale Chailenge
Node Properties
Generate
Generate Solution
|Concepts: (os, Q2 -..
Execute Program
Phase Comprehensive Analysis
DDfffiullt:: very easy I...I very hard
Generate protlem variations
Value: V(s)
Visit Count:
Backpropagation
Analyse solution patterss
Identify systematic issues
Results History
Figure 1: Overview of PrismBench'' search tree, agent workflow, and phased evaluation strategy.
Given the hierarchical evaluation space and executionn-rounnde rewards, the remaining challenge is to allocate the evaluation budget to where it most increases our knowledge about a model's capabilities. As mentioned in Section 2 multiple samples per evaluation task quickly become infeasible as the underlying dataset grows, and simply conditioning the next task on prior observations can result in different task distributions. Therefore, we need a traversal policy that balances exploration of uncertain regions with exploitation of the promising ones, while allowing for comparison between different models. As such, PrismBench uses MCTS with e-greedy state selection policies to balance exploration and exploitation over the search tree (i.e., environment) and control for LLMs performance variability, After each node's evaluation, the observed reward is backpropagated along the sampled path, updating ancestors' estimates so that observations at each node (i.e., state) can inform beliefs about selection of the next node.
Finally, Prich'ench's search process follows a 3-phase strategy. The 1st phase broadly explores the tree establish the model's bascline performance across different coding concepts and difficulty levels. The 2nd phase focuses on failure-prone nodes to discover systematic weaknesses and boundary cases. The 3rd phase focuses on low-performing regions by generating multiple variations of high-failure nodes to determine
5

---

Published in Transactions on Machine Learning Research (03/2026)
the root causes of model failures. Figure shows an overview of our state representation and evaluation pipeline. Rather than exhaustively evaluating the model on all possible scenarios, PrismBench's targeted search approach adapts evaluation to the model's demonstrated capabilities in the environment, allowing for detailed analyses of model performance and failure modes.
3.2
Search and Tree Structure
In this section, we detail how we model the LLM evaluation task as an environment in order to dynamically benchmark LLMs' code generation capabilities and address RQ1-
3.2.1 MDP Formalization
The search space, which contains all possible evaluation scenarios, is modeled as MDP defined by the tuple (S, A. P, R), representing the state space, action space, transition dynamics, and reward function, respectively. We formalize the MDP as follows:
S denotes the stale space where each state & E S is a unique combination of set of programming concepts C, single difficulty level D, and a an evaluation phase identifier V:
S ((c,d,v) | cCC,dED,EEV)
(1)
with v denoting the evaluation phase the state belongs to which we will discuss in Section 3.3 For example, the state ( ([functions, dynamic programming), very easy, Phase 1) represents an evaluation scenario designed to test. the model on a programming challenge that requires knowledge of both "functions" and "dynamic programming" to solve, is intended to be "very easy" terms difficulty, and carried out in Phase
A denotes the action space. At each state, action corresponds to selecting reachable state from the current state:
A(s) fus-se 8' Children(s))
(2)
with Children(s) denoting the states connected to state s via tree edges. We define two types of actions: (1) Aselect which selocts a previously explored state for reevaluation and (2) Aerpand which selocts an unex- plored state for evaluation. For example, expanding the state ([functions, dynamic programming], very easy, Phase 1) could yield new state such as ([functions, dynamic programming, conditionals], concept, ([functions, dynamic programming]. medium, Phase 1), which has an increased difficulty level, for evaluation in Phase (see Section 3.1
very easy, Phase 1), which contains
P:SxAxe - [0 denotes the transition dynamics between states Given the tree structure described in Section 3.1 transitions are only possible between states that are connected via tree edges:
{
s' Children(s)
P(s' s, (.-)))))
=
(3)
O.W.
with s' E S being the state we transition to after taking action a.-ts' at state s.
R: x Ax R denotes the phase-specific reward functions. The reward function quantifies the model's performance at each state using a composite score calculated based on multiple factors, including success rates, error penalties, challenge difficulty, and attempt counts. Each phase uses a different reward function according to the phase's goal, which we will explain in detail in Section 3.4
3.2.2 Tree Representation and Traversal Mechanism
We instantiate the MDP as a tree as described in Scction 3.1 To do so, we consider each state E S as a node in the tree, and each action a A determines whether to expand node or continue exploring the tree shown in Figure Given the tree structure, is essential to assign a value to each node in order to

---

Published in Transactions on Machine Learning Research (03/2026)
guide tree traversal. To achieve this, use the phaso-specific reward function R to determine each node's value based on the model's performance at that node which is calculated based on how well the model solves
the programming challenge associated with a given node which we will detail in Section 3.4.2) Considering the high cost of sampling and to better account for LLM's performance variability, we use (D()) Sutton & Barto (2018) to incrementally estimate each node's value by incorporating past performance:
e e(n) = (n) + a(r Upreo(n))
(4)
where e(n) the node's value, r the immediate reward, and a is the hyperparameter that controls sensitivity to new observations based on benchmarking requirements. We then backpropagate to n's ancestors using discounted updale:
v(na) (Pa) + + pd(n)
(5)
where na is an ancestor of n, d(n) is its distance from n, and y E [0, 1] is the discount factor. This allows us to incorporate results from the evaluated nodes to the ancestors and prioritize promising regions of the search space over time.
In order to traverse the tree, use MCTS to determine the transition probabilities between nodes in the tree given the node values. Transition probabilities are caleulated using MCTS based node visit frequency and value. We use an e-greedy policy to balance exploration and exploitation:
=(ch|n) [uiffor((cillree((())
with probability
arg max.hhechaddrem(a) (C) with probabilityl -
Transitions between nodes (from node n to its child ch) represent changes in difficulty or the introduction of concepts. UCB1 Russell & Norvig (2016) is used to dynamically adjust the probabilities based the model's historical performance using the node's value v(2).
Once a node's evaluation is finished, the decision to expand it or to select another node determines how the tree grows and how the search space is explored. Node expansion E(n) is governed by two criteria:
F((n) = {
if v(n) 2 Op and d(n)= S dmaz otherwise
where v(n) is the node's value, Op is the normalized value threshold of each phase, d(n) is the node's depth, and dmaz is the maximum allowed depth for each phase. Og controls the collective difficulty of each phase, with higher thresholds indicating harder acceptance criteria for solution acceptance at cach node. dmax defines a hard limit on how deep the tree can get al each phase, with higher values allowing for more in-depth analysis at each phase. When expansion is triggered, the node can be expanded in two ways:
combine concepts(n, n') with probability Pe cacrcasc_diffeulttylle
Aexpand(n)
with probability 1 - Pe
where ré is another selected node for concept combination and Pe is the probability of which expansion action is selected and tuned based on the benchmark's desired level of exploration.
The search process begins by generating foundational nodes that span the entire set of concepts at the lowest difficulty level. As the search progresses, node creation and expansion are guided by the model's performance, allowing the tree to dynamically adapt the model's demonstrated capabilities and limitations, which allows PrismBench to adaptively prioritize promising areas during exploration. To enable effective exploration of the search space, extend the tree structure to support multiple parents for each node, allowing us to represent and evaluate combinations of programming concepts, which we explain below.
3.2.3 Multi-Parent Tree Structure
Programming challenges rarely involve solutions based on a single concept. Instead, they typically require the application of multiple programming concepts to produce correct solution. To model this accurately,
7

---

Published in Transactions on Machine Learning Research (03/2026)
in PrismBench, each node in the search tree can have multiple parents, with each parent representing different concept-diffeulty combination. For example, node with the concept set [dynamic programming, recursion] corresponds to a challenge that requires applying both concepts. This node is therefore the child of both the [dynamic programming] and [recursion] nodes, as shown in Figure This setup reflects how complex challenges are often built upon simpler, foundational ones.
Furthermore, multi-parent structure allows to backpropagate performance signals along all relevant paths. If the model succeeds fails node such as [dynamic programming, recursion] the outcome is not just indicative of its proficiency in "dynamic programming" and "recursion". Instead, it also shows how the model is capable of handling each concept well. This helps determinc broader patterns in the search space where the model performs well or struggles, and not just isolated successes and failures. These signals then guide PrishBench's exploration toward weak areas to better determine where and why the model succeeds or fails.
As such, to caplure concept combinations elfectively, we extend MCTS's UCB1 Russell & Norvig 2016 to support multi-paren nodes:
v(n) In Eipperents(n) N(p) UCBI(n) = N(n)
+ +OV
(9)
N(n)
with v(n) being the node n's value, (n) being the number times n has been visited, C' being the exploration constant, and parents(n) being the set of n's parents. this manner, a failure to solve the challenges at one node is indicative of the model showing behavior of interest for each concept and difficulty level in that node.
3.3 Node Evaluation Workflow
As described in Section 3.2 each node in the tree represents a distinct evaluation scenario, defined by a set of concepts and difficulty level. The evaluation node consists of generating a challenge, producing corresponding tests and solutions, and iteratively refining these outputs based on the model's performance. To structure this process, we employ a set of agents where each agent has specific role. Even though the term "agent" is well-established in RL literature, LLM providers maintain different interpretations of what constitutes agent (Shavit et al. (2023 Authropic (2024). In order to have uniform definition throughout our work, we adopt the definition from Roucher et al. (2025), which characterizes LLM-based agents "programs where LLM outputs control the workflow:" In this manner, PrismBench evaluates each node in a multi-agent sandbox where each agent handles a distinct step in the process as shown in Figure
Challenge Designer
Problem Fizer
Challenge Description
Test Valldator Anayze with compenentr
Program to Run
TA
Test Generator
Solution Pattern Analyzer
ans/os tox the code to chaleege
Generated Tests
Test Error Analyzer Divee
EA
Problem Solver
Generated Solution
Run Results
Analysis Results
Figure 2: The evaluation begins with the Challenge Designer and continues until the challenge is finished. Green arrows: inputs, red arrows: outputs, dashed lines: conditional triggers.
As shown in Figure for each node, evaluation starts with coding challenge and proceeds through solution and test suite generation, multiple attempts, repairs, and analysis. To avoid data contamination, agents
8

---

Published in Transactions on Machine Learning Research (03/2026)
are isolated, and context is only shared through the node state. Importantly, the same model is used for both test and solution generation. During solution generation, the model cannot access the tests, and during test generation, it cannot access the solution. This constraint ensures that task understanding is genuine and mirrors real programming challenges Dibia et al. (2022). In order to succoed, the model must interpret the challenge correctly Chen el al. (2024 design valid tests and solutions without access to hidden information Zhang al. (2024b); ; Li & Yuan 20044), and correct mistakes using execution foedback Koutcheme et al. 2024p. Isolating these steps within the pipeline allows us to evaluate each capability independently, resulting in consistent and comparable node-level evaluations across models and compare LLMs' code generation capabilities in a dynamic environment to address RQ2-
3.3.1 Agent Roles
Prism Bench comprised of seven agents. All of our agents utilize prompling best practices Marvin el al. (2023); OpenAl (2025a); Anthropic (2025): clearly defined roles, structured output schemas, and few-shot
examples. Our agent roles are follows:
Challenge Designer: Generales programming challenges of specified difficulty levels largeling particu- lar computer science concepts, following an LC-style format with clear input uuput/output specifications and constraints.
Test Generator: Generates comprehensive test suites that validate functional correctness, corner cases, and performance constraints of submitted solutions while ensuring full coverage of the challenge requirements.
Problem Solver: Generates code to implement a solution for the programming challenge with an emphasis on efficiency and adherence to best practices, while handling all specified corner cases and constraints.
Problem Finer: Analyzes the outputs of program execution and implementation details (solution + tests) to fix failures for the solutions and tests.
Test Validator: Evaluates generated test suites for comprehensiveness, identifying potential gaps in code coverage, missing corner cases, and opportunities for improvement.
Test Error Analyzer: Performs detailed analysis of test execution failures, categorizing error patterns and providing insights into the root causes of solution failures.
Solution Pottern Analyzer: Examines implemented solutions identify algorithmic approaches, data structure usage, and implementation patterns, providing metrics for solution quality and efficiency.
Listing [ shows the system prompt for the Challenge Designer agent. The complete prompt templates and configuration parameters for these agents are available in our replication package Majdinasab 2025
3.3.2 Multi-Agent Agent Orchestration
Our aim in PrichBench to design a general framework to evaluate LLMs on code-related tasks. Therefore, given that many LLMs lack function-calling/ wat--allggttgtoo--useeee tool-use capabilities may not consistently adhere to preset designed controlled sandbox environment that enables structured agent interactions
output formats, while maintaining evaluation integrity. The evaluation for each node in the search tree is presented in Algorithm I and proceeds through the following steps:
Challenge and Test Generation Building a comprehensive dalaset of diverse, high-quality challenges for every possible node infeasible. To address this, the interaction cycle at each node begins with the Challenge Designer agent, which generates Leet Codc-style programming challenges based on specified concepts and difficulty levels as shown in Listing This ensures diverse evaluations per concept-diffeulty pair, capturing true generalization rather than memorization performance variation. The Test Generator agent is then

---

Published in Transactions on Machine Learning Research (03/2026)
You expert computer science educator specializing creating coding challenges. Your expertise spans various computer science concepts, and you have a knack for designing problems that are both challenging and educational. Your role is to create coding questions that test student's understanding of specific CS concepts while also encouraging them to think critically and apply their knowledge in practical scenarios. When given CS concept and difficully level, design problem similar Code challenges. The difficully levels
Very Rasy
Easy
Medium
Hard
Very Hard
Your response should include:
1. A clear and concise problem statement
2. Input format specification
3. Output format specification
4. Coustraints on input values
5. At least examples with input and expected output
6. A brief explanation of the concept's relevance to the problem
7. The specified difficulty level
Ensure that the challenge matches the given difficulty level. Do not provide any code solution. Focus creating a problem that understanding of the given concept the appropriate difficulty. IMPORTANT: You must enclose entire problem description within <problem, description> s/problem_deccipptiinn description> delimiters. This is crucial for extracting the problem from your output. Here's an example of the format you should follow, based on LeetCode-style prohlem:
prroblen_description.
Two Sun
Difficulty: Easy
integer target, that each input would have exactly solution, and may use You can return the ansver in any order.
array integers
nunbers they
add up target. You may the sane element tvice.
Input:
-uuaa: An array of integers (2 <= nuns. nuns.length <= 810"48) target: An integer ($-10-9$ <a target <= $10798)
Output:
array
integera representing indicea of
add up target
Constraints:
2 <= nuns mlngth <= $10"68
8-10"99 nuns[i] $10798
$-10-9$ target $10
Only valid
exists
Exanplea:
1. Input: nume [2,7,11, target 9
Output: [0,1]
Explanation: Because nune [0] nuns [1] == 9, return [o, 1].
2. Input: nuns [3,2,4], target
Output: [1.2]
Explanation: Because [1] nuns
[1,
Relevance to
Manipulation and Hash Tables:
This problem tests understanding of array traversal and efficient lookup. While it can be solved with nested loops, an optimal solution uses hash table to achieve D(n) time complexity, denonstrating the pover hash tables quick lookups in coding intervieve.
Design your problem a similar format, focusing on the concept and difficulty level provided.
Listing 1: System prompt for the Challenge Designer agent.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm 1: Agent Interaction at Each Node
Input: C: List of concepts, D: Difficulty level
Output: S: Node score, Mi: Collected metrics
challenge_ deteeeppeerppooone_e CHALLENGEDESIGNEN D);
1 4 DESTTDiSIGNNBACchlllen
descriplion) // Generated tests
9 PRROBEEMSOLVER(challlee oetpaillennndddeeiiiiiiooooo description) // Generated solution p_r -- 2 g_t // Combine solution and tests (Success, FRu__resulte + RUN(P_r);
if Success then
Tvalidation + TEST VLLDDAARD t) Analyze the tests Psolution + SOLUTIONPATTERNNNALEEEEE else
// Analyze the solution
for i to num_attempts do
11
f + errors during rum // Collected errors f) // Generate fixed solution
12
88 8 PROBLEMSOINERIC, __
13
Eonatysis f -EETTEERRRAAALLZEE s,g t, // Analyze the errors f_
14
15
(Success, Run results) < RUN(p_ r); if Success then
16
L break;
17
18
if not Success then
19
fixed p r r PROBLEMFIXER(P r) // Use Problem Fizer agent (Success, Run results) RUN(fired poccddppppp p_r);
20
21 M + (Ivalidation: Peolutson: Eanalysie: Run Rum_results) // Store all run results
22 return S.M;
tasked with generating a comprehensive suite based on the generated challenge description. We further discuss the risks of dynamic challenge generation in the evaluation process's validity and how mitigate those risks in Section 3.5
Initial Solution Attempt The Problem Solver generates solution based the challenge description, which is then executed against the test suite generated by the Test Generator. Both agents operate independently, with access limited to the challenge description to prevent cross-contamination of their outputs. The sandbox environment then executes the generated solution against the generated test suite, capturing detailed metrics including passed tests count, failure types/coune counts, error types/counts, and exccution traces as shown Listing ? Upon successful completion of all tests, the node marked as resolved, and control returns the search algorithm. However, test failures occur, PrismBench initiates an iterative feedback process.
Retry via Feedback If the execution fails, a feedback phase is initialized. In the feedback phase, the Problem Solver agent receives execution results and error details, atlempting to correct the solution. This approach serves two purposes: it accounts for the stochasticity in LLM performance which we will further describe in Section 3.4.1 (if the model fails, it is provided with context to revise its response) while also evaluating the model's capability to learn from feedback.
Fallback to Repair If multiple solution attempts fail to resolve the issues within a predetermined limit, final attempt to fix the failing solution using the Problem Fizer agent is made. The Problem Fizer, which can set as either the model under benchmark itself or a separate model depending on evaluation requirements, receives comprehensive context including the challenge description, implementation history, test suite, and failure results. Allowing the model have access all of the previously collected information enables assessment. of the program repair capabilities of the model by providing it with full contextual information.
11

---

Published in Transactions on Machine Learning Research (03/2026)
problem_ stattment: "## Even or Odd..."
success: True
tests_ppa tests_passed:
tests failed:
tests_failed: 2
errored:
fixed_by_yey
be_odbb__robbee_tfeeee fixer:: False
asta_trail:
attempt_1:
test_cases: "import unittest/aln..."
solution_code, "def solution(...))
output: "TTett failed. Output: An/n..//........................ ...... In=.." attempt_2:
test_cases: "import witteest
solution_code "def solution(...)" output: "All Tests passed"
Listing 2: An example of the run results for a node
The cycle of testing and refinement continues until the resulting progrum to run (p_r r in Algorithm executes with no errors or failures within a predetermined limit and success is achieved. Otherwise, the node is marked as failed.
Post-Eva Dost-Evaluation Analysis After the loop is finished, regardless of success or failure, the Test Validator reviews whether the test suite logically aligned with the challenge. At the same time, the Solution Pattern Analyzer examines the final solution to extract structural and algorithmic patterns.
Metric Aggregation All run results, retries, error traces, and analysis outputs logged in the node's state, which then transformed into scalar reward signals using the phaso-specific reward function and backpropagated to parent nodes.
In this manner, our multi-agent system integrates with the search tree and MCTS through an orchestrated foodback mechanism: nodes are selected for evaluation using MCTS based on their values. Once a node is selected for evaluation, the model's performance on a generated challenge determines the node's value; the updated value, in turn, influences the next round of node selection and expansion. Furthermore, the model under benchmark can be configured to any of the specified agent roles, enabling fine-grained and targeted capability assessment. This flexibility makes PrismBench adaptable to diverse evaluation requirements (e.g., focusing solely on program repair or test suite generation capabilities).
3.4 Evaluation Phases
Our 3-phase approach guides MCTS to explore the search space using phase-specific reward functions based each phase's evaluation strategy, as shown in Figure 3 To further account for LLM performance variability and sampling stochasticity,
define node value threshold, A(v), per phase, and only proceed to the next phase when changes in sampled nodes' values remain within this threshold across 5 consecutive value convergence checks. We define our phases as follows:
Phase 1: Capability Mapping This phase establishes baseline assessment of the model's strengths and weaknesses across the concepts-dificclty space. Here, the node scoring mechanism is based on challenge success:
Rı(s) = w(d) + p(s)
(10)
with being the base score for success, w(d) being the difficulty weight, and harder difficulties assigned higher weights, and p(s) being the penalty for failures. In this manner, higher successes result in higher rewards, which increase the node's value, which in turn encourages MCTS further explore the search space to find challenging areas and map the model's baseline capabilities.
12

---

Published in Transactions on Machine Learning Research (03/2026)
title:
"Even or Odd"
concepts:
"functions"
difficulty: "very easy"
description: "## Even Odd
Write a function that takes an integer as imput and determines whether the number is even or odd. The function should return the string \"Even\" if the number is even, and \'Odd\" if the number is odd.
#** Input:
An integer (-1079 <= 10"9)
##* Output:
A string \"Even)" or \"Odd\" based on the parity of the input integer.
#** Constraints:
- 10"9
#*# Examples:
1. Input: 4
Output: \"Even\"
Explanation: The number divisible by hence is
2. Input: n 7 Output: \"Odd\"
Explanation: The number
divisible by 2, hence is odd.
Relevance Conditionals and Functions:
This problem tests the understanding of basic conditionale, the solution requires checking the remainder when the number is divided by 2. It also reinforces the use of functions for encapsulating logic, demonstrating how to structure simple program."
Listing 3: Challenge Designer output for a set of concepts and difficulty level
Phase 2: Challenge Discovery By focusing on Phase 1's low-value nodes, the search objective changes to finding challenging combinations of concepts and difficulties where the model consistently fails. Node scoring in this phase is based on failure rate and repeated attempts:
R2(s) = X(1 - Tsucoess) + 1 Nattempts + 3 Lfizer
(11)
with Tsuccess being the ratio of successfully passed tests, nattempts being the number of attempts to fix lailed/errore /errored solution, and finer indicating whether Problem Fizer was used. The hyperparameters (X, 7, 3) weight each term according to benchmarking needs. Using the complement of success ratio assigns higher rewards to nodes with lower success rates, resulting in higher values. This produces a set of nodes where the model consistently fails and indicates the challenging areas of the search space for the model.
Phase Comprehensive Evaluation The objective of this phase reveal not just where but why the model struggles and provide insights into failures' root causes. Therefore, the underperforming nodes from Phase 2 are revisited. However, for each node in this phase, create multiple variations (same concept and difficulty but different challenge descriptions) to distinguish between incidental failures (scenario-specific) and systematic limitations (consistent failures across variations). By analyzing the results across these variants, we collect failure traces to identify core capability gaps, whether from incorrect syntax, incorrect logic patterns, or incorrect concept implementation.
As explained above, each phase has distinct objective, which allows us break down the benchmarking process into smaller, focused tasks. By first mapping the model's general capabilities and then systematically
13

---

Published in Transactions on Machine Learning Research (03/2026)
Phase Capability Mapping
Phase 2: Challenge Discovery
Phase 3: Comprehensive Evaluation
Tree Exploration
Tree Exploration
CD
Problem Variation
CD
MOTS bused oplocion
Crallong Roused explaniion
Semorate discrs vurars
Challenge Generation
Crests produere from coecapts
CD
Challenge Generation Croole targeted dhs angget
PS
Solution Generation
With complttarake beitin
PS
Solution Generation
PS
Solution Generation
TG
Anltt-Analysia
TG
Genernea so ora and leats
Ganerste soldd and sests
Pittem Text crrr Aralyas
EA
Netormance scoring
Evaluation
TG
Validation
Erior anabya
EA
Comprsharalv gigtt
Final Report
SA
TA
Agents
CD
Glan net consects and office dedign chollnge
Challenge Designer
PS
Given chaillnge desztprion, ger.rarrre code solve
Problem Solver
TA
Arebyze the testo ard cornect and compp oher
challangg description peneented
Test Analyzer
TG
Sivnn a chellee chellange deseriptio onn genentl apprrppiion Seitee
Test Generator
EA
Oivin noull splution eralyao why the ao udien fulloo
Error Analyzor
SA
Givin chaillenge deserption generited arabys how the code salncc tuu challenge
Solution Analyzer
Figure 3: Overview PrismBench'ss evaluation pipeline and multi mulli-ppasee assessment. information flow and agent roles across each phase.
Arrows indicate
narrowing in on regions of consistent failure, we can iteratively refine the search space and ultimately pinpoint the root causes of these failures. As mentioned in Section 3.2] each phase's objective is defined through phase-specific state selection policies and reward functions that guide the tree traversal and expansion process, which we explain in detail below.
3.4.1 State Selection Policies
Studies have shown that controlling LLMs' stochasticity through low-temperature settings (e.g., T ~ systematically reduces the diversity of their outputs Renze & Guven (002)); Peeperkorn et al. (2024). Although this performance degradation may be minor in some contexts, becomes crucial when the objective is to comprehensively benchmark model'ss capabilities. As shown by Xu et al. (2022); Ouyang et al. effectively, while at
2023).. for code generation, at higher temperatures, LLMs explore novel solutions ncar-zero temperatures, outputs become repetitive and risk underestimating true performance boundarics. In PrismBench, while we provide temperature as tunable parameter, we preserve recommended temperature ranges rather than enforcing low-temperature values. This ensures thorough exploration of the search space but also introduces the problem of stochasticity in LLMs' responses and subsequent performance variations as a result. These performance variations are especially problematic in dynamic benchmarks that rely on LLMs judges, where even small changes in output can lead to different assessment results. As such, we need to consider that the score for node might not be representative of the LLM's true capability due to performance variations. To address this problem without the risk of underestimating LLMs' performance by setting low-temperatury values, we define e-greedy state selection policies to traverse the tree as described in Equation X These policies mitigate the Krrabacks of purely deterministic approaches (e.g., solely using UCB1 for traversal or setting the temperature to zero) by balancing exploration and exploitation and ensuring comprehensive capability assessment while miligating performance variations. We detail the policies for each phase below.
Phase 1: Capability Mapping At the very beginning of the evaluation, the root generates multiple nodes as starting points for the search process. However, since the search requires the nodes to be evaluated first, the policy for evaluating the root's children (initial nodes) is defined as:
v(n) En v(n')
if Vn' N, v(n')
t root (n)
(12)
otherwise
This policy encourages carly exploration of less-visited nodes and allows for the exploration of the initial nodes to establish a starting point for the search.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Once initial evaluations are complete, use an e-greedy policy for traversing the tree:
(niform((hildrren(n))
with probability e1 arg maXchechssddrem(n) (cB) with probability -
fraverse
(ch|n)
(13)
This policy accounts for the stochasticity of LLMs' responses by using uniform exploration component with probability of e1, which can be tuned based on benchmarking requirements (0.2 in our study), to mitigate the impact of occasional LLM performance variations. This ensures a thorough exploration of the LLM's capability while still focusing promising directions using MCTS.
Phase 2: Challenge Discovery Similar to Phase 1, Phase 2 uses an e-greedy policy that focuses on challenging scemarios while maintaining exploration to mitigate LLMS' performance variations. Therefore, even though our search guided by UCB1, we consider a small probability of selecting another state in our policy:
m2(ch/)) = uniform(chillrent(0)
with probability
(14)
arg max (CB)(ch) with probability 1 --2
Similar to Phase 1, Eg can be tuned based on the benchmarking requirements (0.1 in our study), with higher values resulting in more stochastic state selections and enabling more exploration of the search space regardless of how the model under benchmark performs.
Phase 3: Comprehensive Evaluation In Phase 3, state selection becomes deterministic based node value thresholds (i.e., which nodes to select from Phase 2), which can be tuned depending the benchmark's 's desired difficulty level:
Tz(ch(n)= = { if (() >
(15)
otherwise
with 0 [0 being the normalized node value threshold. Tuning 0 allows for determining the benchmark's overall analysis granularity. Lower thresholds result in the selection of more nodes from Phase 2 for analysis in this phase.
3.4.2 SpecifSp Reward Functions
As mentioned in Section 3.2 in each phase we employ different mechanisms to calculate the immediate received reward. We describe the details of cach phasc's 's reward calculation in the following.
Phase 1: Capability Mapping In Phase 1, the goal is to thoroughly map the capabilities of the model under study. Therefore, the reward function is focused on task success: the better the model at succossfully passing a challenge at a node, the higher the reward that it will receive. For this phase, the reward function for each state defined in Equation 10:
Ri(s) b- w(d) + p(s)
with representing the base reward given to the model it can pass the challenge regardless of the challenge's 's complexity or the number of attempts it took the model to solve it. d is the state's difficulty level, and w(d) is the weight assigned to each difficulty level, with higher difficulty levels having higher weight. This way, the more challenging the problem the model has solved, the higher the reward it receives. Finally, the performance penalty, p(8), is defined as:
p(s) = (Ffaited Pfaiture) + (Ferrors Perror) + ((Rattemppts - 1) Pattemgu) + Pfixer T
(16)
with fasted being the ratio tests the model's solution failed, Terrora being the ratio of errors in the model's solution, Nattempte being the number of attempts took for the model to solve the challenge, and I Jiner being 1 if the Problem Fizer agent was required to fix the model's solution and 0 otherwise. Pfesture: Perror: Pattempt, and Piizer the weights assigned to each penalty type and are set hyperparameters. These
15

---

Published in Transactions on Machine Learning Research (03/2026)
hyperparameters allow for tuning the penalty's impact on the reward and therefore, provide fine-grained control of which aspects of the model's capabilities should be explored in-depth during the benchmarking process. Given that the tree generated in Phase is used for all subsequent phases, high weights for errors and failures will decrease the overall reward at cach node and therefore increase the overall difficulty level of the entire benchmarking process.
In this way, Equation I0 allows for mapping model capabilities: the more successful the model is at solving challenges, the higher the values for the tree's nodes, and the more MCTS is encouraged to contimue exploring the search tree to find challenging areas.
Phase 2: Challenge Discovery Phase 2 shifts focus from broad capability mapping in Phase 1 to systematically identifying the model's capability boundaries. Here, the reward function, as described in Equation| I1 prioritizes the challenges where the model struggles. In this phase, for each state s, the reward is calculated as:
R2(8) = X(1 Fsuccess) "l nattempte + +B. Jizer
With Facceess being the ratio of successfully passed tests (no errors or failures). Using the complement of the success ratio assigns higher rewards to nodes where the success rate is low. Therefore, nodes that consistently expose the model's inability to generate correct solutions receive higher rewards. The hyperparameler À allows for controlling how aggressively the benchmark focuses on nodes with low success rates. Rattempts is the number of attempts took for the model to fix a solution that had failed/errored tests. Therefore, nodes requiring multiple attempts receive higher rewards and 1) adjusts the weight given to repeated failures. Finally, fiier is calculated in the same way as in Phase 1, with 3 controlling the weight of the penalty for dependency on the Problem Pizer agent.
Equation 11 allows MCTS to explore regions of the search space where the model consistently fails (i.e., the more the model fails at each node, the higher the node's value will be). As such, Phase 2 generates a refined set of nodes where the model has constantly underperformed. These nodes will be used for analysis of the underlying root causes of poor performance in Phase 3.
Phase 3: Comprehensive Evaluation Phase 3 focuses analyzing the root causes of model failures while using the same reward formula as in Phase 2.
3.5 Ensuring Evaluation Validity and Comparability
As discussed in Section 2 when it comes to the validity and comparability of evaluation results:
depending on the approach, dynamic benchmarks sulfer from two main shortcomings
High bias due to different task distributions: Exaluating models on tasks that are conditioned on models' performance on pervious tasks, will result in different models, being evaluated on different task distributions. Therefore, evaluation results cannot be compared across different models as each model will be evaluated on a different. task distribution.
High variance due to LLM stochasticity and sampling budget: Given the inherent stochas- ticity of LLMs, even with enforcing deterministic sampling parameters Atil et al. (0024), evaluation results can vary between different runs. As such, evaluations need to be carriod out multiple times in order to estimate models' capabilitics which reduces task diversity given the sampling budget.
In addition to the risks mentioned above, in PrishBench we use the Challenge Designer (see Section 3.3 to dynamically generate programming challenges at test-time. While doing so lowers the risk of benchmark memorization (see Section G), it introduces two risks in return. Specifically, given a state's (c, d) pair:
Invalid challenge: challenge is invalid when is off-concept or incorrectly defined relative to the selected state's concepts c. For example, challenges that do not test for the intendod concopt (e.g., a sorting task labeled as dynamic programming), or infeasible or contradictory constraints (e.g., ambiguous input/output specifications, or evaluation criteria that cannot be met).
16

---

Published in Transactions on Machine Learning Research (03/2026)
Miscalibrated difficulty: A challenge is miscalibrated when its actual difficulty does not match the selected state's desired difficulty d. For example, a challenge is generated for an easy difficulty level that is actually hard or vice versa.
Furthermore, as discussed in Section 3.3 the rewards calculated for each node, are dependent on executing the generated solutions against the generated tests. While grounding evaluations objective execution of codes reduces the risks of incorrect capability estimations, an objectively correct estimation requires that the generated solutions and generated tests to be correct. Meaning that the amount of false positives and false negatives needs to be considerably low for the evaluation outcomes to be reliable.
Below, we describe how PrismBench'' MDP framing, multi-parent tree structure, exccution- evaluation pipeline, (D( value estimation, and e-gredy stale selection alongside multi-phase evaluation pipeline mitigate these risks.
3.5.1 Reducing Bias
To address the problem of different lask distributions, PrishBench defines a fixed MDP over the space of all possible evaluation scenarios. As such, that the underlying task distribution the same for all models (see Section 3.2 In this manner PrismBench becomes fixed RL environment: the rules (Section 3.2.1 dynamics (Section 3.4.1) and reward structure (Section 3.4.2 pre-defined and fixed. Evaluations start from the same root node and interactions with the enviroument are carried out under the same constraints for all models. Therefore, evaluation results between models can then become comparable not by matching identical trajectorics, but instead by how well they perform in the same environment.
3.5.2 Reducing Variance
While constructing fixed environment for evaluation allows for mitigating the high bias of evaluation results, LLMs' stochasticity can still result in high variance between evaluation runs. Specifically, given our methodology, there are two sources of stochasticity which can result in high variance:
Search and Traversal: As detailed in Section 3.2.2 transition probabilities are calculated using MCTS and state transitions are determined by e-greedy policies (see Equation to balance between exploration and exploitation. Therefore, different runs may follow different search trajectories, especially at the early stages of the evaluation process when value estimates are not stable.
LLM stochasticity: While this aspect of LLMs' performance can be mostly controlled via tuning the sampling parameters, it does not guarantee absolute deterministic performance Atil et al. (2024). Furthermore, as discussed in Section 3.4.1 enforcing deterministic sampling systematically roduces output diversity and risks underestimating true performance boundaries.
Below, we describe how PrismBench accounts for these sources of stochasticity to mitigate the problem of high variance and stabilize performance estimates:
Incremental value updates Given the stochasticity of LLMs, the immodiate reward calculated for node, might not be representative of the model under evaluation's 's true capability due to incidental successes/ successes/failuree ffallures (e.g., incorrect solutions pass incorrect tests, correct solutions fail incorrect tests, etc.). Therefore, as detailed in Section 3.2.2 node values are updated incrementally with TD(0) (see Equation $ instead of using the immediate reward, smooth out outlier observations rather than allowing single evaluations from affecting node value estimates.
Initial exploration policy As detailed in Section 3.2.2 all evaluations start with generating a set of foundational nodes that span the entire set of concepts at the lowest difficully levels. At the beginning of the search, Phase follows the initial state selection policy π. (see Equation 12), which relies upon the inverse value of initial nodes (instoad of UCB1) to encourage carly exploration of the scarch space. After all these initial nodes have been assigned a value, Phase 1's state selection policy is switched to x trasersat (sce Equation 13 which uses UCB1 for caleulating the traversal probabilities.
17

---

Published in Transactions on Machine Learning Research (03/2026)
e-greedy exploration policies Relying solely on MCTS to guide tree traversal can result to over- committing to noisy early estimates of node values due to incidental performance variations Russell & Norvig (2016). Therefore, PrismBench uses e-greedy policies for state selection which allow for controlled re-sampling of alternative branches, which reduces sensitivity to any single exploration path and dampens the effect of occasional LLM performance varialions (see Section 3.4.1). This in turn, allows for multiple re-visitation of leaf nodes based on their estimated value. These revisits in addition to TD(0) value estimation average out. stochastic variations between search trajectories.
Multi-parent tree structure PrismBench's MDP is modeled over the space of all possible (c, d) tuples to accurately capture the fact that programming challenges might involve the application of multiple concepts (see Section 3.2.1 Therefore, the nodes in PrismBenchi's s tree can have multiple parents as we detail in inform
Section 3.2.3 This in turn means that the model's performance at node with multiple concepts estimates about each single concept. In this manner, observations at each leaf node can be backpropagated to all ancestor nodes with a discount factor based on their distance, to inform MCTS' search focus (see Equation 5 As such, performance signals are aggregated across the search trajectory and are not isolated to individual nodes. For example, failure at the node ([functions, dynamic programming], very easy) can inform estimates about the models' capability to handle programming challenges involving ([functions], very easy) and ([dynamic programming], very easy) as well.
Transition conditioncd on value stability Finally, as detailed Section 3.4
PrismBench only transitions
between evaluation phases, after value changes across branches remain within a phase-specific threshold across multiple consecutive value convergence checks. As such, evaluation phases only progress after performance estimates have stabilized across the tree (i.c., Phase 2 focuses on consistent failuros only after bascline assessments of models' capabilities in Phase have stabilized).
Therefore, PrismBench's estimates are less sensitive to incidental generations and early-run trajectory differences by (1) smoothing node values over repeated observations, (2) aggregating signals through discounted multi- calti-parentttte backpropagation, and (3) delaying phase progression until value estimates stabilize. We show the empirical evidence of their effects Section 5 and discuss their limitations in Section 7
3.5.3 Dynamic Challenge Generation
The mechanisms discussed above allow for mitigating the high bias/ vaarinaraenee problems of dynamic benchmarks. However, dynamic challenge generation introduces the risks of using invalid or miscalibrated challenges to evaluate models' performance and therefore, invalidating evaluation results. Below, we describe how execution based value estimations coupled with search time self-correction, account for these risks.
Execution based value estimation As detailed in Scction 3.2 state selection is based on past observed performance. Once a state is selected, the Challenge Designer agent produces a challenge based on the state's (c, d), independent of other states at test-time. Afterwards, all rewards are derived from objective execution signals which are then used to estimate nodes' values and are backpropagated through all ancestor nodes (see Equations 4 and 5) Therefore, an invalid miscalibrated challenge will result in an immediate reward and value estimates that deviate from past observed performance along the search trajectory. This deviation in state values informs state selection in the next iteration.
Search-time Self-correction MCTS operates over fixed MDP on (c, d) states, where transitions either add concepts or increase difficulty, and at each iteration, MCTS selects a state based on its current value estimate. As detailed above, when the observed reward is inconsistent with prior observations for that state or its ancestors (e.g,, unexpectedly high success al a "hard" state or repeated failure al an "easy" state), the value update produces a noticeable deviation in the state's value. This, in turn, steers Bench to revisit the invalid/ miscalibrata state and its ancestors, which results in re-evaluation using fresh generations (given that challenges are generated at test time) and subsequent reward and value estimation calculations. Therefore, repeated revisits based on value estimations isolate noisy generations:
18

---

Published in Transactions on Machine Learning Research (03/2026)
If the original outlier was due to one-off bad generation, subsequent evaluations will stabilize the value, and the branch is de-prioritized, resulting in MCTS exploring other branches.
If the signal persists (meaning that previous observations were incorrect), then the value stabilizes at the new level and MCTS either expands adjacent states or prevents expansion when performance is systematically poor.
Therefore, states that produce inconsistent rewards are antomatically detected for further exploration, and stable states' values converge. This feedback loop corrects for invalid/ miscalibrated challenges search time by continuous and iterative re-sampling and averaging over execution outcomes, and prevents unstable branches from affecting evaluation results.
3.5.4 Deterministic Challenge Selection
While the approaches described above lower the risks of evaluating models using awvalid/miscaalibattee challenges, as described in Section 2 there still exists the risk of the Challenge Designer itself being miscalibrated. Mcaning that the agent used as the Challenge Designer has a different understanding of the difficulty levels in comparison to human preferences (i.e., the model's definition a "hard" task corresponds to a "medium" " task according to human preferences). In this case, the overall evaluation results become miscalibrated as the results of model's performance on "hard" tasks correspond to real-world performance "medium" tasks.
While this does not affect the benchmarking methodology itself (given exccution based estimation coupled with search time self-correction), it affects how the results are interpreted. As such, we mitigate this risk by fixing the Challenge Designer agent throughout the entire evaluation pipeline, regardless of the model being evaluated. In this manner, all models are evaluated on the same distribution of task difficulty. Furthermore, PrismBench's modular design allows intogration of task evaluation framcworks, such as TaskEval Tambon et al. (2024b) or Dy Val Zhu et al. (2023a), to verify each challenge's validity. Finally, dynamic challenge generation be entirely bypassed. Specifically, given challenge bank with defined d) properties for each challenge, such as CodeForces the challenges for cach node can be selected from said bank instoad of being dynamically generated by an LLM.
3.5.5 No Reliance LLM Judgments
As detailed in Sections 2 and offloading assessment to a judge LLM introduces risks in both reproducibility and reliability of the evaluation results due to the limitations and biases of the judge model, especially when the objective to evaluate models' capability boundaries. Therefore, PrismBench does not use any L.M judgment signal in the evaluation loop. Specifically, as detailed in Section 3.4.2 all rewards are derived exclusively from execution outcomes and are the only signals used for TD()) updates and MCTS traversal. Therefore, the search trajectory and node value estimates are invariant to any subsequent LLM judgments.
The analyzer agents introduced in Section 3.3.1 (Test Validator, Test Error Analyzer, and Solution Pattern Analyzer) are utilized after a node's evaluation is finished. These agents are utilized only to derive the diagnostic metrics which we will describe in Section Their outputs are never used for reward calculation, node value cstimation, or traversal. As such, they can be disabled without affecting the evaluation process, which also reduces evaluation runtime and costs as discuss in more detail in Section 4.5
Finally, to reduce variance in the diagnostic outputs themselves, the analyzer agents configured with structured outputs Cohere Inc. (2026); OpenAl 2025b), which validates their outputs against predefined schema at inference- time as shown in Listing 4
https://codefooceee.com
19

---

Published in Transactions on Machine Learning Research (03/2026)
When given coding problem statement and cases, analyze the and they properly verify the requirements of the original problem.
Given the problem statement and test cases, your respouse should:
1. Check that all aspects of the problem requirements tested
2. Check that edge cases are properly covered
3. Check that test assertions are correct and meaningful
Check that lest cases are properly structured follow best practices
5. No redundant unnovessary lests present
6. Test names and descriptions are clear and accurate
When reviewing test cases, provide:
1. list of any missing test scenarios (Missing Test. Scenarios)
2. Identification of incorrect assertions (Incorrect Assertions)
3. Suggestions for improving lest cuverage (Suggestions for Improving Test Coverage)
4. Analysis of edge cases that should be tested (Analysis of Edge Cases)
Here's example of the format you SHOULD follow, given problem statement:
<test_velidation
Miasing Test Scenarios:
edge_cces [age: Matrix with all values needs validation
houndary_values: values: Test with naximun allosed natrix size 100x100
performance, teats: Large sparse natrices need perfornance validation invalid
eerorr :iroo_bandling: Missing
dinensions
Incorrect Assertions:
vrong_expeede expectaaect teat checks for incorrect output valuea
invalid_assertion test checke for incorrect number
incompleta_cheeee
verify all output
Suggestions for Inproving Test Coverage:
path_coverage Add teets for all poeeible natrix traversal patha conditiioncoverrge::
condition, coverage: Include testa for branching conditions
data_coverage: Test
different distributions
cothreove: Need
for diagonal travereal cases
candition_covergee Add boundary condition teata
Analysis of Edge Cases:
couuddryy_ooddttinn Test
with negativa elenents
borner_cases, Natrix with all geros needa testing
speciall_npptte Teet with floating point values
boundary_comea condryycoodiiiion Maximun integer
testa nissing
corner_caaes Single ros/columo natrix teata needed
</test_valice velldttiiid
Analyze the provided test cases and provide a detailed validation report following this structure.
Listing 4: System prompt for the Test Validator agent.
20

---

Published in Transactions on Machine Learning Research (03/2026)
3.6
Evaluation Metrics
We define four metric categories to capture distinct aspects of LLMs' code generation capabilities. These metries provide structured and thorough evaluation of a model's strengths, weaknesses, and solution strategies:
3.6.1 Structural Metrics
These metrics focus on the tree and how models perform in the search space. Node counts and depth distributions show where models struggle (persistent exploration) or succced (rapid convergence), and tree growth patterns demonstrate how challenge complexity impacts performance. In this manner, we can provide fine-grained and detailed insights into model performance, behavior, and failures in order to address RQ3.
We denote the total number of nodes in the tree with [N] and the depth of each node E N with D,,. We track the distribution and connectivity of explored concepts through:
((c)= = Σ
[[cconcepts(n)]
(17)
nEuodes
((d)
Σ
[(dEdiffieutties(]]
(18)
nEnodes
with Nc)) and (d) being the number of times each concept and each difficulty was encountered throughout the entire tree. The node distribution across concepts and difficulties provides a broad view where the
model succeeds and struggles: the greater the number of nodes associated with each concept and difficulty
level, the less successful the model has been in addressing related challenges. Consequently, additional nodes were generated to better identify and isolate the problematic areas.
This is complemented by the branching factor at each node:
B(n) children(n)
(19)
|N]
where children(n) is the number of children of node and |N] represents the total number of nodes. Nodes with higher branching factors have more children compared to the other nodes and, therefore, have been more challenging for the model.
The convergence rate C(n) measures the stability of a model's performance at each node n by measuring the difference between consecutive TD values:
C(n) = 122-k vel < for k = =....., K
(20)
where v, represents the node's TD value at attempt. t. The convergence rate reflects how drastically the model's output changes between attempts. A phase is terminated when all nodes exhibit convergence rates below a predefined threshold for K consecutive attempts. A lower convergence rate indicates greater stability, meaning the model's performance has plateaued at node n. When this condition holds across all nodes in phase, the phase is decmed to have been sufficiently explored, and the next phase begins.
3.6.2 Mastery Metrics
These metrics focus on the model's progress in understanding and applying concepts over the course of the benchmarking process. These metrics quantify performance stability challenge complexity increases and success rates challenges with combinations of concepts across benchmarking phases.
The primary measure of concept mastery is the success rate:
SR(c)
Σ
success(r) (n)
(21)
N(c)
REN
21

---

Published in Transactions on Machine Learning Research (03/2026)
where success(n)) is 1 if the model has successfully passed the challenge at node and 0 otherwise, as shown in Listing 2
Similarly, we measure the model's success rate at each difficulty level:
SR(d) N(d) Σ
success(n)
(22)
To understand the effort required for solving a challenge related to a concept e, we measure the average number of attempts regardless of success or failure:
A(c) 1
Σ
attempts(n)
(23)
N(c) (EN(c)
with attempts(n) representing the number of attempts made node n shown in Listing 2
These three metrics alongside each other, indicate how well the model performs in solving challenges for each specific concept/ cancep//iificulty with the average number of attempts indicating how many times the model encountered errors while solving the challenge. High success rates and low number of attempts indicate a high capability (the challenge was solved with a low number of errors and attempts) while lower success rates and higher number of attempts indicate struggles in solving challenges with that specific concept concept/diffiultyy difficulty.
3.6.3 Performance Metrics
Performance metrics build upon the mastery metrics and assess the model's performance across different concepts and difficulty levels by providing a granular understanding of the model's capabilities using challenge success rates, the number of interventions required to fix the model's code, and problem-solving efficiency across concepts and difficulty levels.
The fixer intervention rate indicates when the model requires external help and unable to solve the challenge:
F(c)
1
Σ
(24)
N(c) nEN(c)
with I lizer being 1 if the Problem Fizer agent was used at each node and 0 otherwise.
As shown in Algorithm ! the Problem Fizer agent is only used when the model fails in all of its attempts to solve the challenge. This is cansed by cither incorrect solutions or incorrect tests. Therefore, we can measure the model's program repair capabilities for each concept by tracking whether the use of Problem Finer resulted in success:
R(c) EmENTe) |succees(n)
(25)
Enen(o) [[]see[n]]
where success(n) is if the model has successfully passed the challenge at node and 0 otherwise after the Problem Fizer intervention at node n.
3.6.4 Diagnostic Metrics
These metrics reveal behavioral patterns through solution analysis (preferred coding patterns), error catego- rization (common failure modes), and test set evaluation (correct tests, testing for corner cases, etc). They capture how the model succoeds or fails in specific scenarios and allow for idcntifying and characterizing the model's behavior (how it solves the challenges and how it fails).
The distribution of solution patterns across concepts shows how many times the model has used specific solution for cach concept c:
SP( count(p. c)
(26)
N(P)
22

---

Published in Transactions on Machine Learning Research (03/2026)
with (P)) being the number of identified patterns throughout the entire tree and patterns indicating algorithmic approaches, data structure usage, and implementation patterns.
Pattern effectiveness quantifies which solutions the model executes successfully, helping identify its preferred problem-solving strategies:
PE(p) = EmEN(p) I (sncces(()))
(27)
N(P)
Conversely, using failure rate (1 su)cess(n)) instead of success rate quantifies the patterns the model struggles with the most.
Test validation scores test suite qualily:
count(v.e)
TV(,,c) = N(V)
(28)
where N(V) is the number of identified validation issues throughout the entire tree with being an identified validation issue with validation issucs including analyses on missing, incorrect, coverage, and corner case issues for each generated test suite for each concept
Error pattern distribution by concept shows where exactly the model has failed in solving the challenges related to that concept c:
EP(e, co, N(E)
(29)
where (E) is the number of identified errors throughout the entire tree with e being an error that was raised during the execution of the program.
3.6.5 Comparative Analysis of Metrics
Code generation benchmarks differ in methodology and therefore in the metrics they provide. However, several metrics such as (pssaa(rryaailiitty at least 1 of k generated solutions passes tests), accuracy /sllve rate (fraction of lasks passing all related lests), unit-test pass rale (percentage of provided Lests passed), and diagnostics related to test quality and failure modes are inherent to code generation and are reported in different contexts according each benchmark's objectives. PrismBench's metrics, adhere to the same general formulation, but differ in how they are collected and where they are derived from.
cesuracy/solves rate: These metrics corresponds directly to PrismBench's node-level execution outcomes, and are reported as success rates across the concepts and difficulty levels throughout the tree as mastery metrics (see Section 3.6.2p.
Success@k: In PrismBench, we do not sample k independent solutions per challenge scenario (passūk). Instead, we evaluate end-to-end task success (spec understanding, test generation, solution generation, and program repair) under a fixed attempt budget as shown in A Algorithm 1 Therefore all mastery and performance metrics are reported as success@k: the probability that the model solves a challenge end-to-cnt within & attompts, where an attempt corresponds to one full evaluation iteration with execution feedback Yang et al. (0233 Wang et al. (023b))
Unit-test pass rate / partial credit: Similar to test-driven benchmarks, PrismBench grounds correctness execution of generated solutions against tests. However, in PrismBench we consider test generation as an important aspect of coding capability, and as such, pre-defined unit-tests do not exist. Instead, we incorporate these metrics in the reward function (see Section 3.4.2). For each attempt, record the total number of passed, failed, and errored tests alongside error failure breakdowns, which provides both test pass rates and partial credit. We aggregate those metrics across the tree over concepts and difficulties and report them as diagnostic metrics Section 3.6.4
As discussed in Section 2 one of the limits in reporting metrics over the entire evaluation suite regardless of the details of the evaluation tasks is that aggregate performance signals become noisy and differences in
23

---

Published in Transactions on Machine Learning Research (03/2026)
models' capabilities on different task definitions become obscured by global pass/fail metries. Therefore, in PrismBench, we categorize all reported metrics over (concept, difficulty) pairs and define additional metrics that allow for identifying not only whether models succeed on a set of tasks, but where and how they fails across different coding capabilitios and domains.
4 Experimental Design
To show PrismBench'ss effectiveness, we evaluate 8 LLMs on their code generation, test suite creation, and program repair capabilities: GPT4o (4o), GPT4o-mini (4o-M) Hurst et al. (2024), GPT-OSS-20b (GPT-
OSS) Agarwal al. 2025), Alama3.1-8h (L-8b), Llama3.1-70b (L-70b), Llama3. 1.ca..11-5bbb (L-405b) Dubey et al. 2024). Llamad-Scout (L4S) Meta (2025 and DeepSeekV3 (DS3) Liu et al. (2024). For our experiments, we use LeetCode (LC) LeetCode 22024] style programming challenges and 4o-M as the Challenge Designer to create problems based on each node's 's concepts and difficulty levels for all models under evaluation. The test generation, code generation, and repair tasks are performed by the models under evaluation, while 4o is used for analyzer agents in Phase All reported results are averaged over 3 independent benchmarking runs for all models under study.
For all LLMs under study, the concepts are chosen similarly to the fundamental concepts of computer science in LCC, with difficulty levels of "very easy", "easy", "medium", "hard", and "very hard", the same the difficulties of LC challenges, which describe in detail below.
4.1 Problem Formulation
define the evaluation space MDP over all possible evaluation scenarios where each state is defined by unique (concepts, difficulty) pair. Once a state is selected, the model under study must solve an end-to-end coding challenge as detailed in Section 3.. given natural-language specification, the model must generate a solution, generate tests, and iteratively repair its code based on execution feodback as shown in Algorithm
As described in Scction 3.2
For our experiments, we instantiate each state using dynamically generated L.C-tyle programming challenges as the LC challenge format (i.e., given a challenge description solution must be submitted and refined until all tests pass) matches Prism.Bench's node evaluation workflow. It should be noted, that the LC-style challenge descriptions are an instanliation of end-to-end programming workflow and detailed in Section 3.5 PrishBench can be configured to use alternative problem formulations as well. We further discuss the limitations of using LC-style challenges for our experiments in Section 7
4.2 Concepts
Here, we provide concise explanation of each concept and what we expect the models to achieve in tasks involving these concepts.
Loops: A loop is control structure that repeatedly executes block of code as long as specified condition is true. Examples include for, while, and do-while loops. As such the models should:
Correctly implement loops to traverse data structures or repeat operations.
Optimize loop usage for efficiency and avoid common pitfalls such infinite loops.
Conditionals: Conditionals are control structures that execute specific code blocks based on boolean conditions. Examples include if, else, and else if statements. We expect the model to:
Accurately implement conditionals to manage decision-making logic.
Handle odge cases and ensure logical correctness when combining multiple conditions.
Functions: Functions are reusable blocks of code that perform a specific task, defined by a name, parameters, and a return value. The models should:
Design modular and reusable functions.
24

---

Published in Transactions on Machine Learning Research (03/2026)
Handle parameter passing and scope effectively.
Data Structures: Data structures organize and store data to facilitate efficient access and modifica-
tion. Examples include arrays, linked lists, stacks, queues, and trees. The models should:
Choose appropriate data structures for given problems.
Implement and manipulate data structures accurately and handle edge cases.
A Agoorttmms (logic): Step-by-step procedures for solving problems or performing computations. As such, the models should:
Devise efficient algorithms to address specified problems.
Optimize time and space complexity, demonstrating an understanding computational trade- offs.
Error Handling: Error handling involves detecting, managing, and responding to runtime errors. As such, the models should:
Implement robust eror-handling mechanisms, including exception handling and validation.
Recursion: Recursion is technique where function calls itself to solve problem by breaking it into smaller sub-problems. As such, the models should:
Correctly implement recursive functions, ensuring termination through base cases.
Optimize recursion to avoid excessive memory usage and stack overflow issues.
Sorting involves arranging data in specific order, such as ascending or descending, such as quicksort, mergesort, and bubble sort. As such, the models should:
Sorting:
Implement sorting algorithms correctly and select appropriate algorithms for the given data size and constraints.
Searching involves finding specific elements a dataset, such as linear search, binary search, and lash-based lookups. As such, the models should:
Searching:
Apply efficient search techniques suited to the datasct's structure.
Ensure correctness and handle cases where the element is not present.
Dynamic Programming: Dynamic programming is a technique for solving complex problems by breaking them into overlapping sub-problems and solving each sub-problem only once. We expect. the models to:
Develop dynamic programming solutions to problems requiring optimization.
Demonstrate the ability to use memoization or tabulation correctly.
These concepts foundational to CS and cover the essential problem-solving skills required to implement solutions and tests for problem. By benchmarking models on these concepts, we aim to assess their ability to generalize to unseen lasks based on single concepts and concept combinations critical for coding and reasoning. The concepts for benchmarking are modifiable, meaning that they can be changed to any desired topic, allowing PrismBench to be used more specific scenarios and subjects (o.g., instead of foundational concepts, implementation patterns and challenges closer LC challenges such as "Two Sum", "Valid Sudoku", etc. can be chosen).
25

---

Published in Transactions on Machine Learning Research (03/2026)
4.3 Combination of Concepts
As we detailed in Section 3.2 in real-world programming scenarios, the implementation of solutions rarely requires implementing isolated, single concepts. Instead, they require the integration of multiple concepts to address complex problems effectively. For example, developing a functional application often involves combining loops for iteration, conditionals for decision making, and data structures to organize information. In addition, advanced challenges frequently require recursion, algorithms for processing logic, and error handling ensure that the program does not fail when it encounters unexpected inputs or conditions.
Therefore, to simulate real-world programming scenarios, PrismBench generates challenges that combine these core concepts into unified problems. This allows us to evaluate a model's capabilities to synthesize knowledge across programming concepts. For example, a single problem might require using dynamic programming alongside data structures for optimal solutions or using sorting and searching techniques to manage/ querry datasets. This approach ensures that the model can demonstrate competency in scenarios requiring cross-concept integration. As such, failure to solve problems involving multiple concepts is an indicator of deficiencies in one more of the constituent concepts. Such failures signal areas where the model struggles to integrate distinct methodologies or lacks deep understanding of specific concepts. For instance, if a model fails a task combining functions and error handling, it might reflect difficulties in managing exceptions within modularized code. In this manner, PrismBench can investigate these failures further by identifying the exact concepts or combinations responsible for failures.
Alongside combining concepts, we also use a range of difficulty levels: very easy, easy, medium, hard, and very hard in order to perform fine-grained analysis of the model's capabilities. This enables us to assess performance not only on single concepts and their combinations but also on different complexities of these problems. For example, model might perform well easier problems related to concept group of concepts but fail on medium hard ones, revealing limitations in its ability to scale solutions to more challenging scenarios.
By probing models across a varicty of concept combinations and difficulty levels, we gain a comprehensive understanding of their strengths and weaknesses and gain valuable insights into their overall code generation capabilities by pinpointing root causes and systematically evaluating model's limitations.
4.4 Experiment Settings and Reproducibility
In this section, we report the configuration values for all global and phase-specific parameters used throughout. PrismBench's multi-phase evaluation pipeline for the experiments reported in our study.
Configuration parameters We organize PrismBBench's configurable parameters into set of global traversal settings that control the overall search and value-estimation behavior, alongside phase-specific evaluation parameters that control the reward functions and exploration policies at each stage. Specifically:
Traversal parameters: The traversal parameters (e.g., discount factor, learning rate, etc.) control how the evaluation budget allocated across the search space throughout the evaluation process. These parameters determine which states are sampled, how frequently states are revisited, and how quickly value estimates propagate. These parameters do not change what constitutes success failure at each state or influence the calculated rewards.
Evaluation parameters: The evaluation parameters (i.e., phase-specific penalty weights) control what the evaluation process prioritizes. These parameters determine the evaluation objective such as, emphasizing program repair capabilitics over strict correctness of generated solutions, or weighting lest failures more heavily than lest errors. These parameters directly influence the calculated rewards at each state but do not change the traversal behavior.
Table [ lists all of Peish configurable parameters alongside the values used for our experiments. Since our aim with PrishBench is to propose an evaluation methodology, we expose these parameters as configurable settings to support different evaluation needs and objectives. Importantly, different configurations
26

---

Published in Transactions on Machine Learning Research (03/2026)
correspond to different evaluation objectives and, consequently, different evaluation processes. As such, to obtain comparable results across models, all reported values were kept. fixed across all of the models under study to ensure comparability and reproducibility of the results.
Table 1: Tunable parameters
Description
Type
Parameter
Value
General
Discount factor (Eq. Learning rate (Eq.
Traversal
7
0.9
Traversal
a
0.9
Exploration constant (Eq.
Traversal
C
1.414
Number of convergence checks
Traversal
5
Phase
Node value threshold (Eq.
Traversal
0p
0.4
Value delta threshold (Sec. 3.4
Traversal
A(u)
0.3
Random exploration probability (Eq.
13
Traversal
61
0.2
Penalty per failure (Eq. 16
Evaluation
Ptailure
2
Penalty per error (Eq. 16
Evaluation
P Perror
3
Penalty per attempt (Eq.
16
Evaluation
P Pattempt
1
Penalty for using Problem Fixer (Eq.
16
Evaluation
P Pixer
5
Max depth (Eq.
Traversal
dmax
5
Phase
Node value threshold (Eq.
Traversal
0,
0.6
Value delta threshold (Sec. 3.4
Traversal
Auu)
0.1
Random exploration probability (Eq. 14
Traversal Traversal
62
0.1
Max depth (Eq.
dmax
10
Phose 3
Number of variations for each node Node value threshold (Eq. 7
Traversal
5
Traversal
Op
0.5
Agent configurations Table 2 shows the agent configurations used during each model's evaluation. As outlined in Section 4 use 4o-M as the Challenge Designer across all experiments. For cach model being benchmarked, we use that model as the Test Generalor, Problem Solver, and Problem Fizer agents. To avoid bias in error analysis, we use 4o as the Test Validator, Test Error Analyzer, and Solution Pottern Analyzer for all models except when benchmarking 4o itself. In that case, we use L-405b as the analyzer agent to prevent bias.
Table 2: Experiments configurations
Agent
40
4o-M
L-8b
L-70b
L-405b
L4S
DS3
GPT-OSS
Challenge Designer
4o-M
40-M
40-M
4o-M
4o-M
4o-M
40-M
4o-M
Test Generator
4o
4o-M
L-8b
L-70b
L-405b
L4S
DS3
GPT-OSS
Problem Solver
4o
40-M
L-8b
L-70b
L-405b
L4S
DS3
GPT-OSS
Problem Fixer
40
40-M
L-8b
L-70b
L-405b
L4S
DS3
GPT-OSS
Test Validator
L-405b
40
40
40
40
40
40
4o
Test Error Analyzer
L-405b
40
40
40
40
40
40
4o
Solution Pattern Analyzer
L-405b
4o
40
40
40
40
40
40
27

---

Published in Transactions on Machine Learning Research (03/2026)
4.5 Benchmarking cost
Evaluating each node in the search tree requires independent calls each agent, with the process being dependent the model's performance. For every node, we one query for the Challenge Designer to generate the challenge, one for the Test Generator, and one for the Problem Solver. If the model succeeds on the first. altempl, the evalualion ends with these three queries. However, if the solution fails, up to three additional queries are used for the Problem Solver for iterative repair attempts. If still unsolved, one additional query is used for Problem Fizer to repair the solution. As detailed in Section 3.4] nodes may also be revisited up to five times if their children consistently receive low rewards (convergence checks up to 5 times). Finally, for each node generated in Phase 3, three more queries used for the Test Validator, Test Ervor Analyzer, and Solution Pattern Analyser agents. Therefore, the per-node benchmarking cost in our framework ranges from minimum of 3 queries (one-shot success, no retries) to a maximum of 38 queries (persistent failures, repeated convergence checks, and full diagnostic analysis). On average, throughout all the trees in our experiments, a single node has triggered 6 querics.
The lotal query count for full benchmarking run is dependent on the model's capabilities and provider costs (if using APIs). For the experiments conducted in this study, all models were accessed via their provider APIs. The benchmarking process (including execution of generated codes) was run single Apple M1 machine with 16 GB RAM. Table shows the nunber of queries, lotal cost per model, and wall-clock runlime, averaged over 3 independent runs. Reported wall-clock runtimes include both APT latency and local code/test. execution time.
Table 3: Benchmarking Cost.
Model
Number of Queries
Total Cost (US$)
Wall Clock Runtime (Minutes)
1,153
20
40.2
4o-M
961
10
34.6
GPT-OSS
1,363
16
45.3
L-405b
1,094
24
55.3
I.70bb
454
14
21.1
L-8b
183
8
16.3
L4S
480
10
23.3
DS3
884
13
77.4*
Flevated wall-clock time due to high provider-side latency and rate-limits observed during runs.
4.5.1 Reducing Benchmarking Costs
As mentioned above, evaluating each node in the search tree requires 35 calls in the worst-case scenario. While PrismBench significantly lowers the sampling requirements for comprehensive evaluation of L.L capabilities, the workflow of using all agents for each node can become computationally and financially expensive. As such, Prich allows for a lightweight version of the benchmarking process by using challenges generated from previous runs and skipping diagnostic metric calculation. Specifically:
By using bank of generated challenges from previous runs, the Challenge Designer agent can be bypassed.
The analyzer agents (Test Validator, Test Error Analyzer, and Solution Pattern Analyzer) are only usod for analysis of the benchmarking results in order to compute the diagnostic metrics described in Section 3.6 and other metrics not dependent on LLM analysis. Therefore, they be turned off in case diagnostic metrics are not required.
The combination of these two solutions reduces the number of LLM calls per node in the worst-case scenario from 38 to 15, and reduces computational and financial costs of the benchmarking process by up to 50%.
28

---

Published in Transactions on Machine Learning Research (03/2026)
5 Results and Analysis
In this section, we present the experimental results and analysis of our dynamic benchmarking approach, as described in Section 3 We begin with a comparalive analysis of the eight LLMs introduced in Section [ by evaluating their code generation performance using the metrics defined in Section 3.6 Afterwards, we
provide fine-grained breakdown of how each model performed across different dimensions in Section 5.2
Finally, in Section 5.3 we analyze the effects of model scale (i.e., number of parameters) and how it impacts code generation capabilities.
5.1 Comparative Analysis
In this section, we provide comparative analysis of evaluation results across the four metric categories discussed in Section 3.6 for the models under study.
5.1.1
Structural Metrics
Phase Number Nodes per Depth
150
Number of Nodes per Depth
4o
100
4o-M
GPT-OSS
L-8B
L-708
40
L405-8
DS3
20
Depth
Figure Tree growth across models throughout Phase 1 The bars represent node counts by tree depth, and the shaded bars represent the cumulative number of nodes per depth across the tree in Phase 1.
Figure f comparcs the tree growth per depth in Phase At the very beginning the search, the tree is populated with initial nodes order provide the same starting point for all models and allow comparison between their performance as the evaluation process continues (see Section 3.4.I. D. Phase I's reward function prioritizes task success and diffienlty-wiighhed exploration (sce Section 3.4.2 Therefore, the shaded bars for each model quantify how effectively they sustain problem-solving capability challenges become more complex (i.e., we go deeper in the tree): higher number of nodes indicates broader exploration and lower failures. For instance, 4o and GPT-OSS achieve more than 150 nodes in Phase 1, demonstrating robust handling complex challenges (e.g., multi-concept and high-diffcully tasks), while L-Sb stalls at 60 nodes, failing beyond basic concepts and easy difficulties (depth<4).
Phase 2 low-scoring Phase 1 nodes generate targeted challenges, prioritizing task failure and repeated attempts. Therefore, the ratio of generated nodes per depth, shown in Figure 5 reveals where models struggle: higher ratios at shallower depths imply difficulty with simpler challenges, while increasing ratios at greater depths demonstrate stronger problem-solving capability at complex challenges. We can observe that even though 40-M has a similar number of nodes than I-405b at the end of Phase (142 vs 151 nodes), it struggles with complex challenges Phase 2, while L-405b demonstrates a more consistent exploration the and has a higher ratio of nodes compared to 4o-M the end of Phase Furthermore, we can observe that GPT-OSS has a similar problem-solving capability 4o and L-405b despite being much smaller in scale, which discuss in detail in Section 5.2
29

---

Published in Transactions on Machine Learning Research (03/2026)
Phase 2:
per Depth
4o
0.01
0.03
0.04
0.10
0.16
0.14
0.17
0.18
0.10
0.08
4o-M
0.01
0.09
0.09
0.10
0.17
0.14
0.17
0.12
0.08
0.02
GPT-OSS
0.01
0.02
0.03
0.08
0.12
0.16
0.21
0.17
0.11
0.08
L-8B
0.20
0.27
0.23
0.13
0.07
0.02
0.04
0.02
0.02
0.00
L-708
0.11
0.15
0.17
0.19
0.23
0.07
0.04
0.02
0.02
0.00
L-405B
0.01
0.03
0.04
0.09
0.14
0.19
0.17
0.16
0.10
0.06
L4S
0.09
0.12
0.14
0.15
0.19
0.14
0.12
0.02
0.02
0.00
0.01
0.03
0.04
0.09
0.14
0.13
0.23
0.16
0.10
0.08
1
2
5
6
7
10
Tree Depth
Figure 5: Node ratio by depth across models throughout Phase 2. Each cell shows the ratio of nodes in the tree at each depth, indicating relative search focus across the tree in Phase 2.
5.1.2 Performance Metrics
Table summarizes model capability analysis the end of the benchmark, with values showing foilure rales across concepts and difficulty levels. PrismBench dynamically explores the search space find challenging areas for the model, then focuses on these areas to uncover root causes of failure. The primary operational capability for each concept is determined by the ratio of nodes (concept-diffeully pairs) explored in the search tree and their average failure rates over 3 independent. runs.
Table 4: Model capability analysis by concept and difficulty. Values represent failure rates (higher = more challenging). Colors indicate performance: green (good) to red (poor). t indicates primary operational difficulty level (most number of nodes), /iniicates mastered concepts (failure rate < 0.01), and X indicates concepts beyond current capability (failure rate > 0.99). 95% CT broken down by concept and difficulty reported in Tables and
Ves Easy/Eay
Malicn
Han//Vay Banl
Concept
Alavetaa
1150
0251
0L55
[L52]
[1
11201
0.43
[1.70)
[L.10]
Candition
0.791
0.4.
0.581
0.44
0.45
Paca Street
0.30
<
/
0.801
0.277
0:20
0.738
0.53
14:20
11.001
11389
0.30
11:08
010
0.80
0.40
[1:80
0.33
1.771
6,20
0.55
Hoad
6.26
0.807
0.5
01.03
0.34
0167
0.200
0.35
0L711
Fuartion
9.50
0.82
0.291
0,75
0.58
0.551
[152)
10000
0.00
0.00]
0.44
0.70)
0.22
0.6L
Leepe
Recundon
0.50
0.791
0.271
0.53
0.83)
0.60
0.67
0.891
0.31
0.081
0.37
0.721
Standing
0.50
08287
0.57
0.000
0.25
0067
Sorinn
0.781
0.551
0.00
0.800
0.30
0.47
4o shows no failures on casy tasks, demonstrating strong basic programming skills. However, performance drops at higher difficulty levels, especially for "dynamic programming" and "data structures", indicating limitations in handling programming challenges that require in-depth reasoning. L.405b fails on some easy challenges, but generally has lower failure rates on easy and medium tasks. Similar to 4o, it struggles with hard/very hard challenges that require integration of multiple concepts, such as "dynamic programming", "algorithms", and " "functions". 4o-M has higher failure rates overall among the top-performing models, espocially for challenges requiring compositional reasoning, such as "loops", "functions", "conditionals", and "recursion". These failures are more common when concepts are combined (e.g., loops with conditionals), shown in Figure 20 and discussed in detail in Section 5.2
Both GPT-OSS and DS3 display higher levels of capability than 4o and L-405b, achicving lower failure rates on medium and mard/very hard lasks. While this is indicative of better coding capability across concepts and difficulty levels, the majority of GPT-OSS's and DS3's successes, are a result of relying on Python's built-in functions and standard librarics (c.g., using sort on arrays instead of implementing the sorting function, using itertools to iterate through multiple arrays once instead of implementing the functionalityy etc.) despite task instructions that prohibit such solutions. Therefore, while GPT-OSS and DS3 more capable in
30

---

Published in Transactions on Machine Learning Research (03/2026)
solving challenges in contrast to 4o and L-405b, they are less capable in instruction following and adhering to specified requirements, which are revealed through the diagnostic metrics (error patterns and test validations) which we discuss in detail in Section 5.2
L4S's higher success rates are a result of it reaching much fewer nodes at higher depths compared to both 4o and I.-055b as shown Figure S which indicates I.4S's relative search focus, and Figure 7 which displays the success rates weighted by the number of node visits. On the other hand, L-70b and L-8b show different patterns: L-70b struggles even with casy challenges and fails more difficulty increases, indicating limited capacity for complex challenges. L-8b has high failure rates across all concepts and difficulties, indicating limitations in basic code generation capability. We provide a more detailed analysis of cach model's Is
performance and the effects of model scale in Sections 5.2 and 5.3
5.1.3 Mastery Metrics
Figures Cand 7 present success rates by concept and difficulty for the top models across Phases I and 2 weighted by the number of node visits for cach concept at cach difficulty level (higher number indicates more success at fewer attempts). While Performance metrics capture overall challenge outcomes (i.e., success/failureg mastery metrics highlight which concepts each model handles well and where they struggle. By mapping success rates to each concept-diffielly pair and weighing them by the number of visits to associated nodes, we can pinpoint common failure modes and determine each model's limits.
Emor Hand.
Fusctioss
Eroer Hand.
Functions
Brier Hard
Functios
Canditsorass
Dyn. Prog
Corditizaal
Dyn Prep
Conditiona
Dap. Frep
Algerthmms
Searching
Agarttves
Searching
Alporkirm
Searching
Figure 6: Concept success rate analysis per difficulty for 4o, GPT-OSS, and L-405b. Green: very easy/eee /easy, Yellow: medium, Red: hard/very hard. The radial axis represents the success rate (between 0 and 1). Each axis corresponds to programming concept. Higher values indicate better performance.
As shown in Figure GPT-OSS outperforms all models on medium and hard/very hard difficulty challenges, particularly in "loops", "searching", "sorting", "data structures", and "algorithms" with a marginally lower capability in hunction-heavy composition and "dynamic programming". 4o struggles with compositional reasoning: challenges that require combining "algorithms", "data structures", or "dynamic programming", especially when multiple function calls, nested conditionals, or multiple levels of recursion are required. L-405b shows strong performance on very easy /easy challenges and similar performance to 4o on medium difficulty control flow and data structure tasks. However, similar to 4o, it struggles with complex challenges such as *dynamic programming" or "recursion" " on hard/very hard difficulty levels.
40-M shows high capability on easy/ver /very easy challenges, but its success rates degrade as difficulty levels increase, specifically across challenges that require "recursion", nested conditionals, and "dynamic pro- gramming". On the other hand, while DS3 shows high success rates easy and medium level difficulties, its performance sharply degrades on challenges involving "dynamic programming", "error handling", and loop-intensive compositions. As mentioned in Section 5.1.2 and shown in Figure 7 even though L4S achieves high success rates on hard/very hard difficulty challenges, it is incapable of solving them consistently because it visits far fewer nodes with such levels of difficulty. As such, when weighted by the number of node visits, L4S's success rates are much lower compared to other top models. We include a more detailed analysis of concept combinations and their effects on model performance in Section 5.2
31

---

Published in Transactions on Machine Learning Research (03/2026)
Metum
Na-ghiry Heed)
DeepSekkV
Error Hend.
Furctlrra
Eree Hand.
Aundless
Orer Hand.
Functar
Canditiorls
Dyn. Frog.
Corditionats
Dyr. Prep
Conditionab
Dyt. Prep.
Algulttees
Searching
Searching
Alportbex
Figure Concept success rate analysis per difficulty for 40-M, DS3, and L4S. Green: very easy /asy, Yellow: medium, Red: hard/very hard. The radial axis represents the success rate (between 0 and 1). Each axis corresponds to programming concept. Higher values indicate better performance.
5.1.4 Diagnostic Metrics
Figures and show the success ratios of the top-performing models for the four highest failure rate concepts from Table grouped by the top three programming patterns found in the solutions of each model.
GPT-4o
GPT-OSS-20B
Llama3.14405B
Prog. Recurston Aporrthms
Dyn. Recurston Functions
Cond.
Hand.
Ratio
0.75
0.50
0.00
Deque
Set
Iter. DP Fill
BF5
Iter. DP
Sort
BF5
Daque e
Set
DP BS
B5
Array
Iter. Cond.
2D array
BFS
Merge Sort
Bisect
Memoization
Dyn, Prog.
20 array Early exit
Hash Table le fter. DP Search
DFS Priority Q. Min- Min-Heap
Set
Dict
List
Sat
List
Dict
Set
Early exit
Dict
Figure 8: Success ratios for the most challenging programming patterns for 4o, GPT-OSS, and L-405b, grouped by the 3 most challenging concepts for each model. Stacked bars show performance by difficulty. Green: medium, orange: hard, red: very hard. Taller bars indicate better performance.
As shown in figure S 4o struggles significantly with 4dynamic programming", even when the concept is not explicitly in the challenge. In contrast to the other models under study, the majority of GPT-OSS's solutions involve multiple nested function definitions (i.e., defining functions inside other functions), which, when combined with GPT-OSS" over-reliance on Python's standard librarics to either create hash tables search through the inputs in intermediate steps, result. in failures. T-405b shows the lowest success ratios for simple "data structures" and "tree/ /ggaph traversal". In contrast to the other top models, our analysis shows that L-405b's ' failures are not due to a lack of understanding of the problem itself but from failures in instrictioonfoolooinng and programming syntax. T-405b's 's failed solutions are often implemented using built-in data types (sct, list. dict, etc.). and while the logic and pscudocode are often correct, L-405b frequently makes errors such hallucinating keys in built-in lypes (using incorrect altributes), misplacing code snippets (calling variable before defining it), failing to follow the system prompt's format, which lead to immediate rejection of solutions by the framework.
On the other hand, 4o-M consistently fails in challenges involving composite problems (combinations of multiple concepts), "complex data structures", or "dynamic programming", regardless how attempts to solve the challenge. Similar GPT-OSS, DS3 relics on Python's standard librarics and built-in functions order solve the challenges; however, the majority of DS3's failures are result of incorrect input type and shape estimation (e.g., assuming the input will be 1D array and failing when the input is higher-dimmensioal))
32

---

Published in Transactions on Machine Learning Research (03/2026)
4o-Mini
DeepSeek-V3
Llama-4-Scout
1.00
Loops
Dyn. Prog.
Loops
Dyn. Prog.
Recursion
Loops Dyn. Prog.
0.75
Success
0.50
0.25
0.00
Dict Array Dynt Prog.
Iter. DP Fill array Variable les
Bin. Search
Array
Iter. App.
Dyn: Prog. 2D array Sum Hash
Early exit Iter. pointer
Stack Lin. Search Filter
Set defaui 2D array
List Comp, Set Hash Table
Sliding Window
Diet iter. DP
2D array Early oxit Painter
Greedy Courter
List: Comp.
Early exit
Set Sort
Figure 9: Success ratios for the most challenging programming patterns for 40-M, DS3, and L4S, grouped by the most challenging concepts for each model. Stacked bars show performance by difficulty. Green: medium, orange: hard, red: very hard. Taller bars indicate better performance.
Compared and L-405b, DS3 extrapolates requirements the most without considoring other possibilities, which is the main root cause behind its consistent failures for solving challenges involving "error handling". Finally, L4S struggles the most with challenges requiring iterative processing of inputs intermediate results. We provide comprehensive brcakdown of these observations through per-model analysis and cross-model comparison in Section 5.2
5.2 Detailed Analysis of Results
Tables 5 and show the average success rate and average intervention rate for each of the models under study, across concepts and difficulties, rospectively. The metrics presented here averaged from the values throughout the entire tree at the end of the benchmarking process, for 3 independent benchmarking runs for each model, and are not spase-specific.
Table 5: Model performance by difficulty. Colors indicate performance: green (good) to red (poor). Higher values for intervention rates indicate more usage of the Fizer agent. Each cell shows the mean value over 3 runs + margin of error, calculated at 95% confidence interval.
Difficulty
DS3
GPT-
L405b
Avg
Avg Inter:
Avg Rate
Avg Inter.
Avg Suec.
Avg Inter.
Avg Rate
Avg Tnter.
Suec.
Rate
Suce.
Rate
Suor.
Very
0.83
0.11
4.67 1 0.02
0.83 1 0.01
3.33 1 0.12
0.89 1 0.05
1.33 1: 0.04
0.83 1 0.04
3.67 1: 0.08
Easy
0.73 0.12
2,00 + 0.02
0.64 + 0.11
1.00 = 0.12
0.85 + 0.09
3.33 + 0.02
0.72 + 0.10
2.33 + 0.07
Medium
0.42 0.01
1.00 0.01
0.51 + 0.02
0.33 + 0.04
0.75 + 0.04
4.33 0.09
0.39
0.10
1.33
0.01
Hard
0.64 0.12
0.33
0.11
0.12
1.00
Very hard
0.29
0.08
2.00 0.10
0.28 0.05
0.00 + 0.01
0.49 + 0.03
9.67 0.12
0.29
0.07
2.50
0.06
4o-M
L4S
L70b
L8b
Avg
Avg Inter.
Avg Rate
Avg Inter.
Avg Rate
Avg
Avg Rate
Avg
Suce.
Suec.
Inter.
Suec.
Inter.
Very easy
0.83 1 0.10
1.00 1 0.03
0.84 1 0.02
1.12 0.04
0.42 0.08
3.00
0.05
0.19 1 0.11
1.67
0.06
Easy
0.63 + 0.03
1.00 + 0.09
= 0.03
1.00 0.03
0.22 0.01
0.00 0.05
0.22 0.11
1.00
0.03
Medium
0.35 + 0.02
1.00 0.01
0.08
0.08
0.01
0.03
0.00
Hard
0.21 1 0.11
1.50 1 0.03
0.33 0.12
2.33 0.04
0.21 0.12
1.00 0.02
0.11 0.06
0.00 0.09
Very hard
0.24 + 0.08
1.00 + 0.10
0.29 0.01
0.00 0.10
0.20 0.07
1.00 0.05
0.07 + 0.06
1.00 0.06
Looking at the performance data across all models, we observe a clear hierarchy both success rates and the number of interventions. Starting with the model performance by difficulty level, there's a consistent degradation in success rates difficulty increases across all modcls. As expected, the "very easy" difficulty level shows the highest success rates for all models. The success rates steadily decline to much lower values
33

---

Published in Transactions on Machine Learning Research (03/2026)
Table 6: Model performance by concept. Colors indicate performance: green (good) to red (poor). Higher values for intervention rates indicate more usage of the Fizer agent. Each cell shows the mean value over runs + margin of error, calculated at 95% confidence interval.
Concept
40
DS3
GPT-OSS
L405b
Avg Suoe. Rate
Avg Inter.
Avg
Avg Inter.
Avg Rate
Avg Inter.
Avg
Avg Inter.
Suer.
Rate
Suor.
Suce.
Rate
Loops
0.51 = 0.10
2.50 = 0.03
0.47 0.02
0.33 + 0.04
0.73 # 0.08
5.00
0.05
0.48 0.11
3.00 + 0.06
Conditionals
0.42 + 0.03
4.33 +: 0.09
0.69 0.03
0.00 + 0.03
0.65 + 0.01
4.33 + 0.05
0.43 = 0.11
2.33 +: 0.03
Data Struct.
0.43 + 0.02
4.33 + 0.01
0.52 0.08
0.00 + 0.08
0.68 + 0.09
5.00 0.01
0.43 + 0.06
3.33
0.10
Algorithms
0.49 0.11
4.50 + 0.03
0.55 0.12
1.00 0.04
0.66 0.12
4.67 0.02
0.50 0.06
3.00
Dyn. Prog.
0.29 + 0.08
2.50 = 0.10
0.32 0.01
0.00 + 0.10
0.56 + 0.07
4.33 0.05
0.42 = 0.06
3.33 + 0.06
Error Hand.
0.49 + 0.08
1.67 +: 0.01
0.74 0.05
0.33 + 0.05
0.70 + 0.10
8.00 = 0.07
0.55 = 0.04
2.33 +: 0.12
Functions
0.57 + 0.10
2.67 + 0.05
0.65 0.04
0.33 + 0.03
0.75 + 0.10
5.00 0.12
0.50 + 0.01
2.00 0.10
Recursion
0.49
2.00 + 0.10
0.53 0.04
1.00 0.01
0.70 0.03
5.67 0.04
0.55 0.03
3.33
Scarching
0.51 + 0.11
1.50 +: 0.06
0.62 0.08
0.00 + 0.08
0.74 0.05
5.67 0.04
0.49 0.06
3.33 0.10
Sorting
0.45 + 0.03
1.00 +: 0.06
0.60 0.03
0.33 + 0.07
0.70 + 0.09
7.67 + 0.12
0.48 + 0.07
2.33 +: 0.06
40-M
L4S
L,70b
L8b
Avg Suce. Rate
Avg. Inter.
Avg Rate
Avg Inter.
Avg Rate
Avg Inter.
Avg Rate
Avg Inter.
Suec.
Suor.
Succ.
Loops
0.48 + 0.11
2.00 + 0.02
0.31 0.01
2.81 + 0.12
0.21 + 0.05
1.00 = 0.04
0.10 = 0.04
1.00 + 0.03
Conditionals
0.45 + 0.12
2.00 = 0.02
0.51 0.11
2.25 + 0.12
0.32 + 0.09
3.00 = 0.02
0.18 = 0.10
2.00 + 0.07
Data Struct.
0.44 + 0.01
1.00 +: 0.01
0.46 0.02
3.46
+ 0.04
0.24 + 0.04
1.00 + 0.09
0.11 + 0.10
0.00 +: 0.01
Algorithms
0.48 0.09
1.00 0.04
0.55 0.12
3.60
0.11
0.33 0.12
1.00
0.09
0.17 0.07
1.00 0.04
Dyn. Prog.
0.34 + 0.08
0.00 + 0.10
0.19 0.05
4.21 + 0.01
0.23 + 0.03
1.00
0.12
0.08 = 0.07
1.00 + 0.06
Error Hand.
0.49 + 0.05
1.00 +:2 0.03
0.42 0.04
2.03 + 0.06
0.27 + 0.02
1.50
0.02
0.13 = 0.07
1.00 +: 0.02
Functions
0.42 + 0.06
1.50 +: 0.06
0.59 0.10
2.42 + 0.05
0.24 +: 0.01
2.00 + 0.12
0.23 - 0.08
2.00 +: 0.09
Recursion
0.43 0.02
1.00 0.07
0.13 0.02
4.77 + 0.09
0.05
1.00
0.11
0.20 0.10
1.00 0.06
Searching
0.47 + 0.10
1.50 + 0.04
0.67 0.12
2.32 + 0.02
0.30 0.01
1.00 + 0.11
0.20 + 0.04
1.00 + 0.05
Sorting
0.39 + 0.02
0.00 + 0.04
0.60 0.02
2.28 4: 0.07
0.31 + 0.05
1.00 = 0.08
0.13 # 0.11
1.00 +: 0.06
"very hard" difficulties. The success rates of T-70b and L.8bb even on the "very easy" difficulty level compared to the other models, alrcady indicate the limited capability of these models given the number of their parameters, which we discuss in depth in 5.3
In terms of concept mastery, we see varying performance across models. 4o performs best on "functions" and "searching" challenges, while struggling with "dynamic programming". 4o-M shows more consistent performance across concepts but with lower overall success rates. T-405b demonstrates solid capabilities on "error handling" and "searching" challenges while also struggling with "conditionals", and similar to 4o and 4o-M, on "dynamic programming". The smaller Llama models (L-70b and L-8b) show significantly lower success rates across all concepts, with L-8b particularly struggling with success rates mostly below 0.20.
Both 4o and L-405b show notably high intervention rates, especially at the "very easy" difficulty level (4.67 and 3.67, respectively). This is particularly interesting given that these models also maintain high success rates. Investigating node distributions helps explain these patterns with Figures 10a and 10b displaying the distribution of nodes each depth per concept and difficulty for 4o and I.-455,, respectively. Both models quickly progress beyond "very easy" difficulty challenges, as evidenced by their node distributions (15 and 14 nodes at "very easy" for 4o and L-405b, respectively). As such, the high mumber of interventions at lower difficulties are due to smaller sample sizes at these levels combined with specific challenging cases requiring multiple interventions. On the other hand, we can observe that both 4o and L-405b have high intervention rales for challenges related lo "conditionals", "data structures", "algorithus"" and "dynamic programming" Looking at the distributions of nodes per concept as shown in Figure [0a] and 1obTreveal that these concepts also have a high number of nodes in the deeper parts of the tree, meaning that PrismBench has identified that these concepts at high complexities have shown to be challenging for the models and has focused on these areas in order to thoroughly analyze models' capabilities.
34

---

Published in Transactions on Machine Learning Research (03/2026)
Ditticully Distribution Acress Depth
Concept Distribation Depth
21
7
11
8
6
7
7
4
7
3
5
5
10
5
17
11
9
10
12
10
9
10
12
Very
Loops
Algecillom
Dyn
(a) Node distribution for 40
Difficulty Distribution Access Tree Depth
Concept Distribation Across Tree Depth
24
1
6
4
17
2
11
3
3
6
7
5
4
5
3
10
10
Very Easy
Easy
Madium
Hard
Vacy Blaed
Loops
Conditionale
Punciioox
Data Street.
Algecittem
Exroe llad.
Recursioo
Suritng
Searchlng
Pyn. Prop.
(b) Node distribution for L-405b
Figure 10: Node distributions for 4o and L-405b averaged over 3 independent runs. indicate the number of nodes.
The nunbers in each cell
Since interventions in PrismBench performed by the model itself through the Problem Firer, the combination of success rate and number of interventions effectively measures the model's program repair capabilities. 4o and L-405b demonstrate strong program repair abilities with both high intervention and success rates. For example, at "very easy" difficulty, 4o shows 4.67 interventions with 0.83 success rate, T.405b shows 3.67 interventions with 0.85 success rate. As such, we can observe that when these models encounter failures, they can effectively analyze their own code, understand test failures, and implement successful fixes. This program repair capability persists even higher difficulty levels, though with decreasing effectiveness. On the other hand, L-70b and L-8b have lower number of interventions but significantly lower succoss rates well. Furthermore, their success rates remain low despite interventions. For example, L-8b shows minimal interventions across "very easy" and "easy" but maintains very low success rates (0.19 for "very easy", dropping to 0.00 after "easy"). This indicates that even when given full context, including the original solution, test cases, and error outputs, these models struggle to identify and fix problems in their generated code.
GPT-OSS and L4S stand out with the highest intervention rates compared the other models, with GPT-OSS's solutions requiring the highest intervention rates compared to all the other models, regardless of the concept difficulty level of the underlying challenge. This indicates that GPT-OSS's first attempt at a solution often fails at first run. However, this also indicates that GPT-OSS has the highest program repair capability: given feedback about the failure, the original solution, and test cases, it reliably identifies root causes and produces correct solution. Furthermore, when coupled with the node distribution across difficulties and concepts as shown in Figure 11a we can observe that GPT-OSS quickly progresses beyond "very easy" "medium" level challenges and PrismBench focuses on challenges with higher levels of difficulty for determining its capability. In contrast, L4S displays inconsistent performance across the search tree. When aggregated by difficulty level, it displays high success rates and moderate to low number of intervention calls, however, as shown in Figure 11b it visits far fewer nodes at "hard" and "very hard" difficulty levels compared to the other top models. We can observe that as PrishBencc progresses deeper through the tree, once it reaches "hard" difficulty levels, it repeatedly falls back to "medium" and "easy" level challenges as indicated by the node difficulty distributions across depths Figure [1b] However, despite inability sustain problem-solving at higher difficulties, from table D we can observe that, similar GPT-OSS, it has high program repair capability.
35

---

Published in Transactions on Machine Learning Research (03/2026)
31
16
20
5
15
15
3
3
3
10
20
5
9
4
17
17
10
14
3
10
Very Easy
Essy
Hard
Vary Hard
Serting
Functions
Conditoonata Data Struct
Recuratin
Dyn. Prog.
Searching
Loopa
Algorttims
(a) Node distribution for GPT--SS
cifficulty Distributien Depth
Concept Distribution Depth
10
16
10
0
3
10
2
19
4
12
1
13
11
Thee Oupto
2
7
6
11
5
10
10
0
14
4
11
2
12
1
10
Very Iasy
Easy
Medium
Hard
Very Hard
Sorting
Reciiniion
Leigs
Algurimes
Conatiionat
Funchons
Errer Hand.
Searching
Data Street.
Byn. Preg.
(b) Node distribution for L4S
Figure 11: Node distributions for GPT-OSS and L4S averaged over 3 independent runs. The numbers in each cell indicate the number of nodes.
Figure 12 shows the error pattern analysis per concept for 4o and L-405b. The numbers in each cell represent the average occurrence of each error type per concept across 3 independent runs. 4o shows significantly higher errors in algorithm implementations, particularly in challenges related to "dynamic programming" where algorithm implementation errors peak at 35.8 occurrences and errors related case sensitivity peak rates in generated code for challenges involving "conditional" and "data structure"
28.8. Index concepts (29.6 and 26.7 occurrences respectively) further demonstrate 4o's specific struggle with complex pointer and array manipulations. L-405b's errors, the other hand, mainly in "function" implementation challenges (19.5 occurrences for case sensitivity errors, 15.3 for index errors). We can observe that L-405b maintains consistent performance across most tested concepts, with notably lower error rates in recursive implementation challenges (consistently below 4.0 occurrences) compared to 4o. However, similar to 4o, L-405b also struggles with "dynamic programming", "sorting", and "data structure" challenges.
Error Pattern Distribution
Error Pattern Distribution
Conditionals
20.7 15.4 20.9
5.0
29.4
11.4
9.6
16.3
Conditionals
3.5
6.0
Data Struct. 18.7 14.4
5.5
26.711.5
11.5
5.3
17.8
8.7
8.0
8.0
16.5
Dota Struct.
6.7
8.0
12.1
4.8
10.2
10.6
2.8
9.3
6.7
3.6
2.3
Dyn, Prog.35.82225
35.823.028.J3
4.6
27.216.0
7.0
25.613.4111
11.2
2.0
4.0
22.2
Concept
Dyn. Prog.
8.4
10.0
13.0
3.4
12.4
9.5
2.0
8.8
3.2
5.0
5.4
2.8
6.6
Error Mtand. 8.6 7.3
11.0
3.7
14.0
6.0
5.2
9.3
3.7
5.7
12.0
8.5
9.2
Error Hand.
4.0
4.4
7.5
2.0
9.3
10.0
6.0
7.6
3.0
1.9
2.3
3.2
5.3
Functions
14.8 9.6 11.9
2.8
11.5
4.6
6.8
15.4
5.9
4.1
2.7
2.2
12.0
Functions
9.7
9.0
19.5
5.2
15.3
8.0
5.2
11.8
6.2
7.3
5.8
3.0
8.3
Loops
9.5
20.0
11.00.01...13.
8.5
Leops
6.5
10.0
1.7
6.8
14.8
5.0
5.0
7.2
2.3
3.0
1.3
15.7
5.6
Sorting
17.5
21.1 6.3 17.2
8.8
16.1
13,9
8.1
Agortthm Errot
Amertion Broet
Casa Sensttivily Ernet
Edge Case Emrr
Logic Brnwr
Order Error
Pertormance Errrr
Satup Errer
Uner Unerpected lepurt Broo
Trpes Ermor
Volue Broes
Assertin Erroar
Sensttiviy Brnet
Edge Ermar
Indet Breer
Logic Broor
Outpos Formait Error
Order Brrer
Setup Brror
Unerpertaed Iopust Ernor
Type Error
Value Brror
Initot
Aigeitthm
Perforaave
Outpus
Error Type
Error Type
(a) Error pattern distribution for 4o
(b) Error pattern distribution for L-405b
Figure 12: Error pattern distríbutions for 4o and I.-455b averaged over 3 runs. The numbers in each cell indicate the number of times each error was raised.
36

---

Published in Transactions on Machine Learning Research (03/2026)
The most encountered error types (algorithm implementation, case sensitivity, and index errors) are consistently related to implementation details rather than fundamental algorithmic understanding. This observation is reinforced by the notably lower frequency of type, setup, and corner case errors across both models and all tested programming concepts. These patterns suggest that while both models demonstrate sound algorithmic understanding, their primary struggle lies in generating the correct code for solutions and lests. Analyzing test validation issues allows us to pinpoint whether the errors stem from incorrectly generated solutions or incorrectly generated tests by the models. Figure 13 presents the distribution of test validation issues for both 4o and L-405b across concept combinations. Each cell indicates how often specific validation problem for test, such as incorrect condition coverage or incorrect boundary checks, was identified. By comparing these data with the error pattern distributions in Figure 12a and 126 we can discover correlations between the rool cause of encountered errors and the concept. areas where those errors were raised most frequently.
Test Validation Issues Distribution
Test Validation Issues Distribution
Coverage
Edge
Missing
Coverage
Edge Cases
Missing Cases
Alpetthrss Coediloses Date Street Ereer Hend. 20
18
20
25
30
20
10
42
42
2
10
Algarkini-
Algenituuo Data Strud. lasps Serting- 20
20
Aggeitires byn, Prog. trau Ward. Lopp-115
15
15
15
50
20
15
17
40
14
14
Algertbnz- Dats Stratt. Enor Handd Functions 5
5
5
5
10
10
Algritters Byp, Prog. Recumdes Boosling 0
0
0
0
0
1101
0
Alperittss Bps Prap. Bandinns Suctig
15
13
15
30
15 15
12
30
12
Algeritinz- Erer lland Searching Serting
16
15
11
15
10
11
24
5
s
6
Alprttina
Aggultiees- frns Hase Faectins- suctigg 5 10
o
10
5
15
Alpartiiss Searching Serting
5
Coeditonass Strett Syn. Freg Sortig 20 20 20
25
15
Gediisaals Date Stuek Das. Prep. Satting 5
5
5
5
5
5
5
Conditooalls Data Struct Erre Hard. 15 15 15
20
Cenditinats Data Strcch Einer Haed. Seeching 5
10
24
10
5
Costitionas Dats Stadt Eror Hand. - 10 10 5
10 10 10
24 5
5
10
Data Struet Dyn, Prog. Paaccionn seuching 15
15
15
30
2. 15
13
Conditirrla. Dyn. fre. Searchigg Sorting 5
5
5
5
5
1
0
Dats Stact Breer flad. -ecarsio Serting
5
5
5
5
5
10
5
10
5
Data Ractisss.
5
0
Data Strut. -f Teeccions Searching Serting 20
20
20
25 25 2.
19
42
16
10
10
15
15
25
Fanction: fecrrise- Sasiie---sssseee Searchng Soctig
21
17 18
25
16
20
10
42
6
15
Leeps
15
Losss Racurtinnn Searching
Leeps Recundas Booccing
Conckion Coerrigh
Pako Coverage
Patts Coverage
Reundttry Condttiors
Taited
Edgn Caney
Eirw Hanateed
Fer mntormancc Teers
Conmatio Coverage
Bounseery Consitiony
special ropitt
Canek
Edgr Carns
Exrn Honsllog
Pasts
Cerver
Special
Data nath
(a) Test validation issues distribution for 46
(b) Test validation issues distribution for L-405b
Figure 13: Test validation issues distribution for 4o and L-405b averaged over 3 runs. The numbers in each cell indicate the number of times each issue was identified.
As mentioned in Section 5.1 both GPT-OSS and DS3, frequently rely on Python's built-in functionalities (e.g., using Python's in-place sort function on arrays) alongside standard and third-party librarics in their solutions (e.g., using itertools or numpy for array processing), particularly in challenges with high difficulty levels or multiple concept combinations. As our aim in PrishBBenh is to evaluate the models' coding capability rather than library knowledge, the interactive sandbox provides only a base Python environment without additional packages. Consequently, any dependence on third-party librarics leads to failure regardless of solution logic. Figure [4] summarizes the error pattern distributions for GPT-OSS and DS3. Similar to 4o and I.-05bb failures on the most challenging concepts for the model are mostly caused by algorithm and logic errors; however, both GPT-OSS and DS3 show a high concentration of errors clustered around "setup errors" and "unexpected input errors". These error types are raised when executions fail due to missing imports, incompatible program structure, or library usage outside what is available inside the environment. Morcover, by comparing Figures 12 and 14 we can observe that GPT-OSS has an overall higher level of error counts compared to 4o, I.405b, and DS3, which is consistent with its higher Problem Fizer intervention ratios as shown in Tables 5 and
For 4o, the combinations of concepts that have the highest number of test validation failures are jalgorithms, conditionals, data structures, error handling) and [functions, recursion, searching, sorting). These concepts
37

---

Published in Transactions on Machine Learning Research (03/2026)
Error Pattern Distribution
Error Pattern Distribution
Algorithms
10.7
7.9
4.5
4.3
5.3
16.5
4.8
4.2
5.8
Aigorttmms
3.5
2.3
4.8
2.0
2.0
2.4
0.0
1.3
2.0
1.2
4.4
Conditionale
4.8
7.6
9.5
5.0
5.9
5.8
6.9
5.9
4.1
4.7
10.5
3.9
7.5
6.8
Conditionalss
2.6
4.9
4.5
166
2.6
2.1
4.0
2.0
1.3
1.1
3.7
1.9
1.4
2.6
Data Struct.
7.0
8.9
10.8
4.7
6.3
4.3
14.0
5.1
6.7
5.5
13.5
6.9
7:7
7.5
Data Struct. 3.5
2.2
5.8
1.0
1.8
1.6
1.9
3.1
1.4
1.7
13
1.1
0.6
3.5
Dyn. Prog.
3.5
3.9
7.a
3.4
6.0
7.8
2.6
5.0
4.2
4.0
7.0
4.7
3.5
5.1
Dyn. Prog
3.6
2.5
2.6
0.7
1.2
3.0
1.9
1.6
0.7
0.7
9.3
0.5
1.0
1.4
Concapt
Error Hand.
4.3
4.3
10.2
5.9
7.8
7.4
3.8
4.2
2.2
4.8
7.0
5.5
6.6
7.6
Errer Hand.
5.2
4.6
5.0
2.8
2.8
3.5
1.0
2.0
2.7
1.2
5.7
1.8
1.2
2.9
Functions-
12.33.112
4.8
3.4
8.3
3.8
5.9
6.4
5.0
9.5
4.3
5.5
Punctions
5.0
3.6
4.9
3.4
3.4
4.7
1.0
4.5
3.1
Loope
11.4
4.6
6.8
6.7
4.3
5.1
3.6
4.6
5.8
5.7
Leaps
2.5 4.6
5.3
2.1
0.0
3.7
6.3
4.2
4.1
6.1
7.5
4.3
5.0 3.2
Searching
12.3
5.9
6.0
0.7
7.6
4.5
3.7
9.0
3.0
6.8
3.8
Searching
1.4 1.9 3.2
2.3
2.2
2.0
0.6
Sorting
6.1
12.0
2.6
6.5
6.3
3.9
6.9
4.8
4.1
13.5
2.9 7.4 5.1
Serting
3.6 2.1 4.8
1.0
1.9
6.0
2.7
2.4
1.2
9.3
0.0
0.8
2.5
Agoontin Brrer
Ansertion Brrer
Caoe Suautiiity Error
Edge Care Brror
Index Breer
Logl: Brrer
Ourpurt Format Error
Order Brreot
Perforance Breer
Setop Breer
Satup Errore
Unaapeected toport Broo
Type Brror
Value Brent
Algortthn Brrne
Care SanuRlliity Broo
Edge Case Brrer
Indta Brrer
Logle Errer
Dxaputt Formal Brrer
Ordtr Error
Part netormanca Frrer
Satup Error
Setup Error
Unsopucted Inpert trrer
Type Brea
Vatue Brrear
Assurtion
Error Type
Error Type
(a) Error paltern distribution for GPT-OSS
(b) Error pattern distribution for DS3
Figure 14: Error pattern distributions for GPT-OSS and DS3 averaged over runs. The numbers in each cell indicate the number of times cach error was raised.
Concept Combination Success Rates
Concept Combination Visit Counts
Algorithms
0.00
0.59
0.69
0.33
0.75
0.90
0.54
0.64
0.67
0.65
Algorithma.
0.00
47.00
38.33
93.00
44.00
39.50
44.50
67.00
29.00
43.50
Conditionals
0.59
0.00
1.00
0.29
0.89
Conditionals
47.00
32.00
36.00
30.00
14.00
54.00
65.33
0.33
0.70
0.43
0.46
1.00
0.80
0.00
0.30
Prog
93.00
32.00
53.67
0.00
67.00
34.00
52.00
29.00
41.00
Concepts
Error Hand.
0.75
0.58
0.20
0.00
0.60
0.67
0.00
2.00
1.00
Error Hand.
44.00
36.00
26.50
67.00
0.00
63.00
31.00
0.00
48.00
9.00
Functioons
0.90
1.00
0.67
1.00
0.60
0.00
0.71
1.00
0.83
0.67
Functions 39.50
30.00
14.00
34.00
61.00
0.00
45.00
22.50
40.00
$4.00
Loops
0.54
0.29
0.50
0.80
0.67
0.71
0.00
0.66
0.74
0.67
Loops
44.50
66.67
15.50
$1.00
51.00
45.00
0.00
60.00
32.67
21.33
Recursion
0.64
1.00
0.50
0.80
0.00
1.00
0.66
0.00
1.00
0.52
Recurdio
67.00
14.00
9.50
29.00
0.00
22.50
60.00
0.00
12.00
71.50
Searching
0.87
0.30
1.00
1.00
0.83
0.74
1.00
0.00
0.53
Searching
29.00
54.00
30.00
41.00
40.00
32.67
12.00
0.00
34.00
Sorting
0.67
0.67
Sorting
34.00
9.00
71.50
34.00
Algortthis
Conditionals
Data Struct.
Punctioons
Loups-
Rocursion
Sorting
Algorttnm
Conditionnls
Dyn. Frg.
Eroor Hand.
Runctions
Searching
Saringg
Concepts
GPT-4o
Concepts
Figure 15: Details on the concept combination effects on 4o's performance. The right matrix displays the average success rates for all nodes related lo each specific combination. The left matrix displays the average number of times each concept combination was visited in the search tree, regardless of success success/failure, ffiluure
also have the highest number of errors, as shown in Figure 12a particularly with index and case sensitivity errors. Furthermore, as shown in Figure [0a] Prish has specifically focused on these concepts by generating a high number of nodes for thorough validation and isolation of issues. This indicates that many of 4o's generated tests have the same underlying root cause of its generated solutions (for instance, mishandling pointer or array indices). Moreover, the high frequency of numeric and string value assertions that fail these tests suggests that 4o often struggles produce fully consistent test inputs or expected outputs, leading to assertion failures even when the generated solution is correct. We can observe correlation between test validation issues and error types, demonstrating that these failures are not only in the generated solutions but also in the generated tests. The highest validation issues appear in concepts requiring numeric and string value assertions. This suggests that 4o struggles with processing such concepts during test generation. Therefore, even when 4o produces correct solutions, its limitations in mumerical and string processing lead to incorrect test assertions, resulting in failures and error cascades. These issues are also reflected the success rates and visit counts of concepts as shown in Figure [5] where we observe lower success rates and higher visit counts for the combinations of concepts with high error rates and test validation issues.
38

---

Published in Transactions on Machine Learning Research (03/2026)
Concept
Concept Combination Visit
Algortthms
Data Struct,
0.83
0:28
0.69
Data Struct,- 20.33
18.50
18.33
33.00
26.33
34,00
22.00
Dyn. Prog.
0.56
0.00
0.58
0.00
0.44
0.83
0.92
0.78
0.00
0.29
Prog.-
6.00
3.00
18.50
0.00
15.00
13.50
8.67
8.00
0.00
49.00
Concepts
Error Hand.
0.74
0.67
0.42
0.44
0.00
0.92
0.89
0.50
0.50
0.83
Error Hand,
$2.67
35.00
18.33
15.00
0.00
17.67
14.67
2.00
32.00
33.67
Punctions
0.83
0.54
0.43
0.83
0.92
0.00
0.50
0.87
0.90
0.09
Functions
16.00
32.67
33.00
13.50
17.67
0.00
22.00
12.00
56.67
53.33
Loops
1.00
0.33
0.69
0.92
0.89
0.50
0.00
0.78
0.94
0.00
Loops
10.00
26.50
26.33
8.67
14.67
22.00
0.00
23.67
27.67
2.00
Recursion
0.48
0.67
0.78
0.00
Recursion
8.00
40.33
25.00
Searching
Searching
0.00
Algorithms-
Conditionals
Data struct.
Functions
Loops
Rocursion
Searching
Sorting
Algorttims
Conditiones
Dyn. Prog.
Error Hand.
Functions
Searching
Sorting-
Concepts
Llama3.1-405B
Concapts
Figure 16: Details on the concept combination effects on L-405b's performance. The right matrix displays the
average success rates for all nodes related to each specific combination. The left matrix displays the average number of times each concept combination was visited in the search tree, regardless of success guccess/failure,
L-405b on the other hand, consistently struggles with "function" and "sorting" concepts, especially when "data structures" "searching" are also included shown in Figure T6 The test validation issues in Figure [3b demonstrate that combinations like [data structure, function, searching, sorting] exhibit high incorrect coverage issues and a large mumber of missing or incomplete test cases. Similarly, the error pattern distribution for I.-455b in Figure 12b shows peaks in case sensitivity and index errors whenever function implementations are tested. We can obscrve that L-405b exhibits different root causes for failures compared to 4o, particularly in concepts combination of "data structures" and "error handling".
Solution Pattern Distribution (Top 15 Patterns)
Algorithms
5.7
3.2
2.5
3.5
1.7
3
1.3
2
1.5
5
0
Conditionals
1
1.5
2
1
1.5
0
1
0
2.5
1.5
Dyn. Prog.
4.6
3.8
1.2
2.5
3.5
0
1
1
1
0
Concept
Error Hand.
11
3
3.3
4
1
1
1.4
2
1
1.5
3
0
Functions
2.5
3.2
1.5
1.5
1.6
3
Loops
1.4
Recursion
2
0
0
1.3
2.3
Searching
3.7
2.1
1.8
3
2.3
2.5
0
1.7
1.7
1.4
1
1.5
3
Sorting
1.8
3.8
2.2
1.5
1.7
Bynamic programming
erray
list
Rarative approach
2D array
min-haap
validation and sorting
sorting
dictionary
early tormination
early retura on validation Falture
set
divide and conquer
recursion
haap
Pattern Type
Figure 17: Patterns identified in solutions distribution for L-405b averaged over 3 runs. The numbers in each cell indicate the number of times each pattern was identified.
By correlating test validation issues and the solution patterns shown in Figure 17 we can see that L-405b's failures primarily stem from syntax errors and hallucinations rather than logical errors as evidenced by the high number of failures in using built-in data types (arrays, list, dictionary, etc.). The cowest-perforiin nodes and their corresponding patterns show that L.-05bb frequently generates non-existent syntax (e.g.,
39

---

Published in Transactions on Machine Learning Research (03/2026)
non-existent built-in function calls, incorrect syntax for using built-in data types, etc.); creating situation where both the generated code and its corresponding tests are incorrect. This leads to the high intervention rates observed in Table as the Problem Fizer repeatedly intervenes to correct both solution and test issues.
Concept Combination success Rates
Concept Combination Visit Counts
Algortthms 0.00
0.56
0.64
0.78
0.59
0.67
0.90
0.89
0.97
0.93
Algorithms
0.00
$5.33
21.67
25.33
31.67
32.67
34.67
49.00
20.33
38.67
Conditionals
0.56
0.00
0.65
0.72
0.87
0.50
0.88
0.67
Conditionals
55.33
0.00
59.67
26.67
50.00
38.00
60.67
14.00
24.67
Prog.
116.50
Concapts
Error Hand.
0.59
0.72
0.66
0.00
0.83
0.81
0.06
Error Hand.
31.47
50.00
9.50
60.00
0.00
3.50
11.33
45.00
30.00
20.67
Functionns
0.67
0.87
0.89
0.92
0.83
0.00
0.60
0.89
1.00
0.74
Functions-
32.67
12.33
31.33
9.50
3.50
0.00
119.33
24.00
24.00
104.67
Loops
0.90
0.88
0.78
1.00
0.81
0.60
0.00
0.78
1.00
0.92
Loops
34.67
38.00
8.00
30.00
31.33
119.33
0.00
10.33
16.00
76,00
Recursion
0.89
0.55
0.67
0.74
0.36
0.89
0.78
0.00
1.00
0.96
Recursion-
49.00
60.67
$8.00
43.67
45.00
24.00
18:33
0.00
4.67
76.33
Searching
0.87
0.71
0.89
0.60
1.00
1.00
1.00
2.00
0.00
0.59
Searching
20.33
14.00
7.00
116.50
38.00
24.00
16.00
4.67
0.00
24.00
Sorting
0.93
0.87
0.82
0.20
0.97
0.74
0.92
0.96
0.89
0.00
Sorting
38.67
24.87
44.50
35.33
20.67
104.67
76.00
76.33
24.00
0.00
Algorithma
Conditionals
Data struee.
Functions-
Leupa
Recursion
Searching
Algarthms
Conditionals
Dyn. Prog.
Error Hand.
Functions
Loops
Recurston
Searching
Sorting:
Concapts
GPT-OSS-20B
Concapts
Figure 18: Details on the concept combination effects on GPT-OSS's performance. The right matrix displays the average success rates for all nodes related each specific combination. The left matrix displays the average number of times each concept combination was visited in the search tree, regardless of /fallure.
As shown in Figure 18 GPT-OSS's performance remains consistently high across most concept combinations, maintaining high succoss rates even challenges become more complex. However, it demonstrates persistent weaknesses in challenges involving "dynamic programmingg" and "error handling" relative to other concepts. Interestingly, we can observe that nodes which combine "sorting" with "functions", "loops", or "recursion" " are frequently revisited across the search tree. This pattern is also present in GPT-OSS's error distributions as displayed in Figure 14a] where nodes with these concepts have a high rate of "sctup" and "unexpected input" errors. This is a direct result of GPT-OSS's s over-reliance on third-party libraries, which are not available in the environment, and its tendency to define nested functions. However, from Figure [8] we can observe that these nodes have high overall success rate. Investigating the solutions for these nodes reveals that GPT-OSS's first attempts at solving challenges requiring array manipulation rely third- parrd-party libraries instead of built-in standard libraries and functionalitics. It is only during the subsequent attempts, where the leedback contains information about these libraries not being available, that it generates a correct solution. As such, GPT-OSS exhibits a different coding profile compared to the much larger 4o and L-405b, which use Python's standard librarics.
It important to note that these insights come from PrismBench's 's automated analysis. The search trees generated by the framework enable deeper investigation of behavioral patterns and contain detailed analysis for each node. We only highlight the most significant behavioral patterns observed.
5.3 Effects of Scale
Given the evaluation results on the GPT-4o and Llama3.1 families, in this section, we investigate the effects of model scale on code generation and problem-solving capabilities. Importantly, we do not include GPT-OSS and L4S in our analysis in this section, as both models have different architectures compared to the others.
5.3.1 GPT-4o
GPT-4o and GPT-4o Mini, developed by OpenAl, are part the same model family but differ in scale, performance, and application focus Hurst et al. (2024). GPT-4o the high-performanco, multinodal flagship model optimized for complex tasks requiring deep reasoning and nuanced language understanding, while GPT-4o Mini is lightweight, cost-efficient variant designed for speed and accessibility, prioritizing rapid
40

---

Published in Transactions on Machine Learning Research (03/2026)
token generation and affordability. While both models share core architectural features like Transformer-basel design and multimodal capabilities, GPT-4o Mini is reported to be significantly smaller than GPT-4o.
23
1
6
7
6
17
1
7
8
6
3
4
3
2
2
4
3
2
2
3
Eanps
Algoritlans
Figure 19: Node distribution for 4o-M averaged over 3 runs. The numbers in each cell indicate the number of nodes.
Figure 19 displays the node distribution and visit counts of 4o-M throughout the search tree. In the previous sections, we presented performance results for 4o, with its corresponding node distributions presented Figure [Oa]] Comparing the distributions of 4o with 4o-M shows how scale impacts performance. While the majority of 4o's nodes are distributed in challenges with "hard/very hard" difficulty and deeper parts of the tree (as shown in Figure [0a), can observe that for 4o-M, the majority of nodes are distributed between challenges with "medium" and "hard" difficulty and in shallower depths. This is also evident in the search tree for 4o-M as shown in Figure 26
Concept Combination Success Rates
Concept Combination Visit Counts
Algartthms
0.00
1.00
1.00
0.53
0.50
Algortthmes.
0.00
16.00
26.00
33.33
48.50
26.50
0.00
54,00
Conditionass
1.00
0.00
0.71
0.25
0.72
0.58
0.53
0.83
1.00
0.00
Conditionals:
16.00
0.00
71.00
6.00
41.50
24.00
47.33
16.00
20.00
0.00
Data Stract.
1.00
0.71
0.00
0.20
0.50
0.50
0.00
0.29
0.42
0.38
Data Struct.
20.67
71.00
0.00
18.00
15.50
10.00
0.00
32.67
57.33
11.00
Dyn, Prog.
0.29
0.25
0.20
0.00
0.00
0.50
1.00
0.83
0.75
0.11
Dyn. Prog.
26.00
6.00
18.00
0.00
17.00
15.00
24.50
12.00
10.00
12.00
Concapts
Error Hand,
0.38
0.72
0.50
0.00
0.00
0.72
0.50
0.56
0.83
0.67
Error Hand.
33.33
41.50
15.50
17.00
0.00
28.33
37:33
53.33
60.00
36.33
0.53
0.38
0.50
0.00
0.11
0.40
0.78
Functions
48.30
10.00
15.00
28.33
0.00
14.50
62.00
34.00
12.33
Loops
Recursion
Searching
0.00
1.00
0.42
0.54
0.58
0.72
1.00
Searching
0.00
20.00
57.33
10.00
60.00
34.00
86.00
59.00
0.00
22.00
Sorting
0.42
0.00
0.38
0.67
0.78
0.17
0.28
0.00
Sorting
54.00
0.00
11.00
12.00
36.33
22.33
7.00
46.00
22.00
0.00
Algorthimz-
Conditionals
Data Struct.
Functions
Leops
Recursion
Searching
Algorthmss
Conditionals
Dyn. Prog-
Error Hand.
Functions
Recursion
Searching
Sorting
Concapts
GPT-4o-mini
Concapts
Figure 20: Details on the concept combination effects on 4o-M's performance. The right matrix displays the
average success rates for all nodes related to each specific combination. The left matrix displays the average number of limes each concept combination was visited in the search tree, regardless of success/failures
Figure 20 shows the success rates and visit ratios for nodes corresponding different concept combinations. As discussed in Section 5.1 can see that the majority of 4o-M's failures occur when it encounters combinations of concepts that require compositional reasoning. For instance, we can observe that 4o-M has relatively low success rates for "dynamic programming", which fall even lower when the challenge combines another concept with "dynamic programming".
5.3.2 Llama 3
The Llama herd of models, developed by Meta, is a family of LLMs designed to support multimodality, coding, reasoning, and tool use. The term "herd of models" refers the diverse range of models within the Llama family, each tailored for specific applications Dubey al. (2024). The flagship model, L-405b, is a dense Transformer architecture with 405 billion parameters and context window of up to 128,000 tokens,
41

---

Published in Transactions on Machine Learning Research (03/2026)
enabling it to handle extensive datasets and complex tasks. While these models share foundational training data and post-training processes, they differ in architectural scale-sun such the number of layers, model dimensions, attention heads, and FFN dimensions to optimize performance across varying use cases. This allows us to leverage PrismBench to systematically evaluate how architectural and parametric scale impact code-generation capabilities. While we have already presented performance results for L-405b (the most. capable variant) in prior sections, this section focuses on analyzing performance differences across scaled-down versions of the Llama 3 family.
Difficulty Distribution Access Depth
Concept
Depth
1.5
0
3
3
0
3
2
3
1
0
0
1
Haird
Street. Alpucittane
(a) Node distribution for L-70b
Difficulty Bistribution Acrees Tree Depth
Concept Distribation Across Thee Depth
12
3
0
3
2
3
2
2
2
2
1
2
1
1
1
1
Very Easy
Easy
Itard
Vey Baan
Loops
CondRiionds
Ponclisns
Data Street.
Algeciltem
Exree Hand.
Recoesino
Surlieg
Searching
Dyn. irep
(b) Node distribution for L-8b
Figure 21: Node distributions for I.-70b and L-8b averaged over 3 runs. The numbers in each cell indicate the number of nodes.
As shown in Figure 21a and 21b the depth of explored nodes drops sharply for smaller models, indicating limitations in handling more difficult problems. In particular, the node distribution for L-8b shows that only the shallow parts of the tree (depths 2 and 3) and "easy" challenges have been explored in depth. The inability to explore deeper nodes suggests that the model fails to generate correct solutions when challenges are more difficult or contain multiple programming concepts. Conversely, L-70b reaches deeper parts of the tree more often and capable of reaching nodes with "very hard" difficulty to an extent (even though it. fails at all of them) as shown in Table 5 The node distributions of the search trees generated for L-405b, L-70b, and L-8b) clearly demonstrate how models' scale plays an important role in their problem-solving and code-generation capabilities. The search trees themselves for these two models (L-70b and L-8b), presented in Figure 30 further show how these models struggle to complete challenges they become more difficult.
Figure 30b shows how the majority of nodes for L-8b are generated in Phase 3 (highlighted in blue). As explained Section 3.4, Phase is responsible for comprehensively inspecting areas of failure, and as such, we can see that T-8b consistently fails with high failure rates, with the majority of nodes being generated Phase On the other hand, I-70b shows a slightly better performance as pictured in Figure 30a In the same manner L-8b, the majority of L-70b's nodes are generated during Phase 3, However, we can that, unlike L-8b, many nodes were also generated in Phase 2, indicating that the model was capable of solving some of these challenges albeit with low success rates.
Figure 22 and 23 further demonstrate performance degradation as the models get smaller. The success rales of concept combinations decrease significantly for L-Sb, particularly for tasks requiring more advanced strategies (o.g., "dynamic programming" or multiple nested constructs). comparison, L-70b shows moderate succoss with simpler "loops" and "conditionals" but similarly struggles sustain the performance under combined, higher-level concepts.
42

---

Published in Transactions on Machine Learning Research (03/2026)
Concept Combination
Rates
Concept Combination Visit Counts
Algortthms
Algortthme.
Data Struct.
0:20
0:25
0.00
0.50
0.50
0.25
Data Struct.
25.00
1.00
13.00
2.50
19.00
6.33
4.50
Dyn. Prog.
0.50
0.67
0.00
0.00
0.33
0.33
0.00
1.00
0.00
0.00
Dyn. Prog.-
7.00
1.33
1.00
0.00
2.00
2.00
3.00
3.00
6.50
4.00
Concepts
Error Hand.
0.00
0.33
0.00
0.33
0.00
1.00
0.25
0.50
0.50
1.00
Error Hand.
1.00
3.67
13.00
2.00
0.00
7.00
6.00
2.50
4.00
4.00
Functions
0.75
0.00
0.50
0.33
1.00
0.00
0.00
1.00
0.00
0.33
Punctions-
8.67
5.00
2.50
2.00
7.00
0.00
6.00
1.50
1.33
2.67
Loops
0.67
1.00
0.50
0.00
0.25
0.00
0.00
0.25
0.37
0.42
Loops
3.33
6.00
9:33
3.00
6.00
6.00
0.00
8.00
8.00
31.33
Recursion
0.50
0.50
0.50
1.00
0.25
0.00
Recursion
1.50
15.00
3.00
2.50
1.50
0.00
4.00
2.33
Searching
0.56
Searching
4.00
0.00
21.33
Sorting
0.00
4.00
4.00
Algorithms.
Conditionals
Data struce..
Function
Loops.
Rocursion
Searching
Sorting
Algortthms
Conditiones
Dyn. Prog.
Error Hand.
Functions-
Recurston
Searching
Sorting-
Concepts
1amma.11770
Concapts
Figure 22: Details on the concept combination effects on L-70b's performance. The right matrix displays the average success rates for all nodes related to each specific combination. The left matrix displays the average number of times each concept combination was visited in the search tree, regardless of success /failure.
Concept Combination Success Rates
Concept Combination Visit Counts
Algortthms
0.00
0.50
0.25
0.00
0.00
0.50
0.67
1.00
0.00
0.33
Algorithms. 0.00
3.00
3.00
1.00
2.67
3.00
3.67
5.50
1.00
6.33
Conditiionls
0.50
0.00
0.33
0.33
0.00
1.00
0.00
0.50
0.50
0.00
Conditionals:
3.00
0.00
1.67
7.00
0.00
1.50
2.00
13.00
8.00
1.67
Data Struct,
0.25
0.33
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
Data Struct.
3.00
1.67
0.00
5.67
4.00
1.50
2.00
3.00
2.00
1.00
Dyn. Prog.
0.00
0.33
0.00
0.00
0.50
0.00
0.00
0.29
0.00
0.00
Dyn. Prog.
1.00
7.00
5.67
0.00
3.50
6.00
3.50
5.50
1.67
1.00
Concapts
Hand.
0.00
1.00
0.00
0.00
3.50
1.33
3.00
5.67
Recursion
1.00
0.50
0.00
0.25
0.00
0.00
0.00
0.50
0.00
Recursion
5.50
13.00
3.00
5.50
4.50
0.00
1.67
0.00
2.33
2.67
Searching
0.00
0.50
0.00
0.00
0:22
0.75
0.50
0.50
0.00
0.22
Searching
1.00
8.00
2.00
1.67
7.00
2.00
3.00
2.33
0.00
12.67
Sorting
0.33
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.22
0.00
Sorting
6.33
1.67
1.00
1.00
11.00
0.00
5.67
2.67
12.67
0.00
Alporthmms.
Conditionals
Data Struet.
Functions
Loogs
Rocurston
Searching
Algorthhms
Conditionals
Dyn. prog-
Error Hand.
Fundtrons.
Loops
Searching
Sarting-
Concepts
Llama3.1-88
Concapts
Figure 23: Details on the concept combination effects on L-8b'ss performance. The right matrix displays the average success rates for all nodes related to each specific combination. The left matrix displays the average number of times each concept combination was visited in the search tree, regardless of auccess/failure.
6 Related Works
Recent studies have demonstrated how current benchmarking approaches fall short in evaluating current models. Specifically, Xu et al. (2024b), Roberts et al. 22023), and Jiang et al. (2024) show how even extensive attempts to control the training dataset does not prevent data contamination, either because filtering out undesired data is difficult given the training dataset's size Balloccu et al. (2024) or because of models' increasing generalization capabilities and lack of diversity in benchmarks' challenges Dong et al. (2024)
Crowd-sourced benchmarks such Chiang et al. (2024): Fourrier et al. (0024) have proven to be much more
effective in evaluating LLMs but suffer from 1) limited coverage of real-world challenges Lin et al. (2024)
(0024). As such, benchmarks such as ARC-AGI Chollet 2019),, SWE-bench Jimenez el al.] (2024), HELM Liang et al. (2023), and HLE Phan let al. 2025) have been proposed, which include either very difficult problems or data that the models could not have been trained on given the time of their relcase White al.
and 2) user biases in preference ranking of the responses Chiang et al.
(2024): Franzmeyer et al. (2024). The
majority of leading LLM providers are currently using these benchmarks. However, with the rapid increase in LLM capabilities, these benchmarks are quickly becoming obsolete, as models achieve increasingly higher
43

---

Published in Transactions on Machine Learning Research (03/2026)
scores, rendering the evaluation metrics ineffective in distinguishing their performance Anthropic (002
Jaech et al. (2024]; Liu et al. 2202
Dynamic benchmarking approaches have been proposed to address the limitations mentioned above. LLM-as- Judge frameworks Zhu et al.] (023b); Li et al. p Wang et al. (2023a) use one or multiple LLMs (based on benchmark performance or fine-tuned for specific evaluation criteria) to assess and rank the responses of the model under evaluation. However, given the judge models' capabilities and biases, offloading performance assessment to an LLM introduces challenges in the reproducibility and reliability of the evaluation Thakur et al.
2024h. On the other hand, approaches have been proposed for leveraging LLMs to generate novel
and dutto--ditti evaluation scenarios based on existing static benchmarks Li ct al. (2024b); Zhang
ltt al.
20244a
; Zhuge et al. (2024). Building on these approaches, frameworks such as Li et al. (202cc); Zhu
et al.
2024a): Fan et al. (2023); Wang et al. (2024); Zhang et al. (024cc): Zhu et al. (2024b) extend dynamic
evaluation to provide more systematic assessment of a model's 's capabilities. Notably, DyVal [Zhu et al.]
(2024a) models each evaluation scenario as DAG-basod composition of reasoning tasks, guided by a graph generation algorithm, defined constraints, and translation into natural language. Dy Val2 [Zhu el. al. (2024b) replaces the DAG with LLM-based agents to transform existing benchmarks into new challenges to evaluate LLMs' performance and gencralization on multiple domains. DARG Zhang et al. (2024c) and TroeEval Li et al. 2024c) represent the evaluation processes as graphs or trees, analyzing how the model navigates these structures. However, the stochasticity of LLMs and the variability in their performance introduce challenges in ensuring consistency and reliability in these evaluation methodologies Blackwell et al. 2024
7 Threats To Validity
Threats to internal validity concern factors internal to our work that could have inffuenced our study. The inherent stochasticity of LLMSs which results in variability of results across runs is the most important. threat to our work's internal validity. We aimed to address this threat at each step of the process through using (D) scoring for nodes, e-greddy policies for node selection, and node value convergence checks at each evaluation phase as detailed in Sections 3.2 and 3.4 Additionally, while our multi-phase search strategy is designed to focus on promising areas of the search space, the stochasticity in LLM's performance may introduce state selection bias by underexploring edge cases. We mitigate this threat by conducting multiple trials for each model and reporting averaged results across trials to account for variability and reduce the impact of outlier behaviors. Finally, dynamic challenge generation using an LLM can result in miscalibrated evaluation results in comparison to human preferences. As detailed in Scction 3.5 we mitigate this thrcat by using the same agent the Challenge Designer (4o-M) for all models under study (except for evaluating 4o-M itself to minimize bias) and allowing for using verified, labeled challenge banks in order to evaluate modols.
Threats to construct validity concern the relationship between theory and observation. In the context of our work, the main construct threat is whether our proposed evaluation metries (as we detail in Section 3.6 holistically capture the multiple aspocts of LLMs' code generation capabilitics. We mitigate this threat by executing generated code within an isolated sandbox environment, ensuring that our evaluation is grounded in the actual result of the models' outputs. Furthermore, as detailed in Section 3.3 we isolate the different. stages of code generation, namely solution generation, test generation, and program repair, by using separate agents for each step. The other threat to our work's construct validity is that Phase 3's pattern analyses rely on a judge model (40 in this study) and while the judge model can be changed to any available model, there still exists the risk of the judge's capability ceiling and potential bias resulting in incorrect analysis results. We address this threat by restricting the judge model's role to post hoc analysis so that it does not influence the overall search and benchmarking process.
Thrcats to external validity concern the gencralizability of our findings beyond the specific sctup used in this study. Our evaluations were carried out on eight LLMs and a finite set CeelCode style challenges on pre-defined concepts and difficulty levels, which, although diverse, may not capture the full range of coding tasks encountered in practice. We recognize that given experiment design, the results of our experiments may not reflect real-world coding capabilities. Additionally, as the benchmarking process is computationally and financially expensive, each node's evaluation requires multiple L.M calls via. APT local models (as
44

---

Published in Transactions on Machine Learning Research (03/2026)
detailed in Sections
3.3 and 4.5 we limited the number of tasks and the depth of tree expansion at each
phase (Section 4.4 These constraints may impact the extent to which our results generalize to larger-scale or different task distributions.
8 Conclusion
In this paper, we introduced PrismBench, dynamic benchmarking framework that models the evaluation space search tree and uses MCTS to systematically explore evaluation scenarios. Unlike prior approaches, PrismBench dynamically analyzes how LLMs approach problems, adapt to feedback, and handle increasing complexity using structured multi-phase pipeline and specialized agents to uncover systematic model weaknesses. Our key contributions are as follows:
We formalized the space of code generation tasks as a search tree over programming concepts and difficulty levels, allowing for adaptive and targeted evaluation of LLM capabilities.
We introduce multi-phase evaluation strategy: (1) Capability Mapping to assess baseline strengths,
(2) Challenge Discovery to identify systematic weaknessos, and (3) Comprehensive Analysis investigate failure patterns and root causes.
We proposed set of metries in order to provide fino-grained insights into model behavior beyond standard pass/fail metrics.
We conduct experiments across eight state-of-the-art LLMs and show that while larger models demonstrate stronger general capabilities, they still struggle with compositional reasoning, high- difficulty tasks, and consistent lest generation.
Additionally, we provide PrismBench as an open-source, extensible framework Majdinasab (2025) for conducting such evaluations to allow practitioners to conduct trustworthy, targeted benchmarking strategies as LLM capabilitics continue to evolve.
References
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arxiv preprint 2025.
Anthropic. The claude model family: Opus, sonnet, haiku. http::/ddcs..ubrrrpcc.cooooo/o/e resourcee/model-carrd 2024. Accessed: 2025-01-12.
Anthropic. Building effective agents, December 19 2024. URL httpe://ww.artrrppp...c///eeeere buillinggeffeeciivv--aees Accessed: 2025-01-13.
Anthropic. Prompt engineering overview. https://doca. httpa//aaaaacec/e//////////a/aaa
sooppp-engiieerrnng/oveerreeee 2025. Accessed: 2025-01-12.
Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, et al. Non-determinism of" deterministic' Ilm sellings. ar preprint Xrx:2:2408. 04667, 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXie preprint ar xrX::21088.071 07732, 2021.
Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušck. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source 1lms. Xix preprint arXiv:2402.00 03927, 2024.
45

---

Published in Transactions on Machine Learning Research (03/2026)
Sourav Banerjee, Ayushi Agarwal, and Eishkaran Singh. The vulnerability of language model benchmarks: Do they accurately reflect true Ilm performance? arXiv preprint arXiv:24 12. 2223599, 2024.
Robert E Blackwell, Jon Barry, and Anthony G Cohn. Towards reproducible Ilm evaluation: Quantifying uncertainty in Ilm benchmark scores. arXiv preprint arXin:24 000:2410.00 10.03492, 2024.
Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang. Yijiang Xu, Jian Wu, Yidong Wang, Qing Gao, Jindong Wang, et al. A survey on evaluating large language models in code generation tasks. arxiv preprint Xiv:2440 16498, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Folipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Flizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tozak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodci, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Augelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04138 2024.
François Chollet. On the measure of intelligence. arXiv preprint arXi::1911.015/79 2019.
Cohere Inc. Structured outputs. 2026-01-18.
2026. Accessed:
Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi-Sangdeh, Han Liu, and Saleema Amershi. Aligning oflline metrics and human judgments of value for code generation models. arXiv preprinl arXiv:2210.1644, 2022.
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Menglei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402. 15938, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint WxXiv2447.217880 2024.
Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes. arXiv preprint arXiv:2312. 14890, 2023.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leader- board v2. hattps://hiii h/tgg///g/ggigaaceeeee.. http://innaaccc.ccscceeeeeeeeeee co/spaces//pareoooccccc wttp://eep..pppepeooo--e-e----ee
11m leaderboard 2024.
Tim Franzmeyer, Aleksandar Shtedritski, Samuel Albanie, Philip Torr, João Henriques, and Jakob N Foerster. Hellofresh: Lim evaluations on streams of real-world human editorial actions across x community notes and wikipedia edits. arXiv preprint orXiv:2406.03// 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXi::21002211 21277, 2024.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. Xr preprint 2r2 2024.
46

---

Published in Transactions on Machine Learning Research (03/2026)
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Investigating data contamination for pre-training language models. or preprint or arXix:2401.0600 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofr Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https: net/forumt ? VTF8yNQM66,
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXtv:2104.14833 14387, 2021.
Charles Koutcheme, Nicola Dainese, and Arto Hellas. Using program repair a proxy for language models' feedback ability in programming education. In Workshop on Innovative Use of NLP for Building Educational Applications, pp. 165 181. Association for Computational Linguistics, 2024.
LodeCode. Programming challenges, December 19 2024. URL https:/ http://wwwwettcce.ccco com 2025-01-13.
Accessed:
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lims- as-judges: A comprehensive survey on Ilm-based evaluation methods. arXie preprint arXiv:2412.0557 2024a.
Kefan Li and Yuan Yuan. Large language models as test case generators: Performance evaluation and enhancement. arXiv preprint arXiv:2404.13340 2024.
Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad- A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint XXix:2402.05044 2024b.
Xiang Li, Yunshi Lan, and Chao Yang. Troeoval: Benchmark-free evaluation of large language models through tree planning, arviv preprint urXiv:2402.1314 13125, 2024c.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of -obbuuction-oolowinng models. https: 5 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas leard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL Featured Certification, Expert Certification.
https://openreview http:////opoor..eveeeeeettttrtte
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Facze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking Ilss with challenging tasks from real users in the wild. arXiv preprint arX.::2406.04 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, al. Decpseck-v3 technical report. arXiv preprint arXiw:2412.19437, 2024.
Vahid Majdinasab. Replication package for prismbench.
http:://githubb.co//ommissssss
PrismBench] 2025. Accessed: 2026-03-05.
47

---

Published in Transactions on Machine Learning Research (03/2026)
Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Cetatumba-Nabendee Prompt engineering in large language models. In International conference on data inlelligence and cogniline informalics, pp- 387 402. Springer, 2023.
Timothy McIntosh, Teo Susnjak, Nalin Arachchilage, Tong Liu, Paul Watters, and Malka N Halgamuge. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. arXiv preprinl ar Xrxxv24402.0955 09880, 2024.
AT Meta. The llama 4 herd: The beginning of new era of natively multimodal ai innovation. https://at meta. com/blog/ aolog/llam--- /at///laaam-lllaa wtgtltimmoodd-iittlleeeeeeeee checked on, 4(7):2025, 2025.
OpenAl. Prompt
guide. httpt///ttttaoo...o.oooo//oo/oot
2025a. Accessod: 2025-01-22.
OpenAL. Structured outputs. http:p///.a.atce...o.o.ooo/oo//a
2025b. 2025b..
Accessed: 2026-01-18.
Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. Llm is like a box of chocolates: the non-determinism of chatgpt in code generation. arXiv preprint arXix:2008. 02828, 2023.
Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. Is temperature the creativity parameter of large language models? arXiv preprint Stl:2405.00499. 2024.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, et al. Humanity's last exam. arXiv, 2025.
Matthew Renze and Erhan Guven, The effect of sampling temperature on problem solving in large language models. arXiv preprint Xiw:2402.05201. 2024.
Manlcy Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. Data contamination through the lens of time. arXiv preprint Xiv: 110122300 2023.
Aymeric Roucher, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. 'smolagents': The easiest way lo build efficient agentic systems.
2025.
Stuart Russell and Peter Norvig, Artificial intelligence: a modern approach. Pearson, 2016.
Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O'Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkin, Tyna Eloundou, Alan Hickey, et al. Practices for governing agentic ai systems. Research Paper, OpenAl, December, 2023.
Richard S Sutton and Andrew G Barto. Reinforcement leurningg: An introduction. MIT press, 2018.
Florian Tambon, Amin Nikanjam, Foutse Khomh, and Giuliano Antoniol. Assessing programming task difficulty for efficient evaluation of large language models. arXiv preprint arXiv:2407.21122 2024a.
Florian Tambon, Amin Nikanjam, Cyrine Zid, Foutse Khomh, and Giuliano Antoniol. Taskeval: Assessing difficulty of code generation tasks for large langnage models. arXiv preprint arXiv:2407.21227, 2024b.
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dicuwke Hupkes. Judging the judges: Evaluating aligument and vulnerabilities in llms-as-judges. orXio preprinl 2rXiv:2406.12600 2024.
Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei, and Xuanjing Huang, Benchmark self-evolving: A multi-agent framework for dynamic Ilm evaluation. ar preprint arXiv:2402.11443, 2024.
Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Scan O'Brien, Ramakanth Pasumuru, Jane Dwivodi- Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Zarendza and Asli Celikyilmaz. Shepherd: A critic for language model generation. XrXio preprint or Xr022030 00223088.04592 04592, 2023a.
48

---

Published in Transactions on Machine Learning Research (03/2026)
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating Ilms in multi-turn interaction with tools and language feedback. arXiv preprint arXiv:2309. 00022099.0699,,, 10691, 2023b.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Zii Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free 1lm benchmark. arXiv preprint orx:x:2406.199 19314, 2024.
Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. Benchmark data contamination of large language models: A survey. arXi preprint arXiv:2406.0420 2024a.
Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN Internalional Symposium on Machine Programming, pp. 1-10, 2022.
Ruijie Xu, Zengzhi Wang, Run-Zo Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404 18824, 2024b.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advonces in Neurol Information Processing Systems, 36:23826-253 -28544, 2023.
Beichen Zhang, Kun Zhou, Xilin Wei, Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen, Evaluating and improving tool-augmented computation-intense math reasoning. Advances in Neural Information Processing Systems, 36, 2024a.
Yuanliang Zhang, Yifan Xie, Shanshan Li, Ke Läu, Chong Wang, Zhouyang Jia, Xiangbing Huang, Jie Song, Chaopong Luo, Zhizheng Zheng, et al. Unseen horizons: Unveiling the real capability of Ilm code generation beyond the familiar. arv preprinl or Xiv: 12. 12.0801, 2024b.
Zhehao Zhang, Jiaao Chen, and Diyi Yang. Darg: Dynamic evaluation of large language models via adaptive reasoning graph. arXiv preprint arXiv:2406. 1 12711, 2024c.
Kun Zhou, Yulao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't 't make your Ilm evaluation benchmark cheater. anXiv preprint orXix:2311.012 2023.
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. arXiv preprint 2rx::2309.11166, 2023a.
Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dynamic evaluation of large language models by meta probing agents. In Forty-first International Conference on Machine Learning. 2024a.
Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dyval 2: Dynamic evaluation of large language models by meta probing agents. arXiv preprint ar Xix:22402. 14865, 2024b.
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. Xiv preprint ar XrXi::2310. 17631, 2023b.
Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agent- as--jde Evaluate agents with agents. arXiv preprint Xrxiv:24 10. 10934, 2024.
A Appendix
A.1 Sample Trees
This section presents the minimized versions of the search trees generated for all models examined in this study. To ensure brevity, the trees presented here focus solely the concepts, difficulty levels, and scores of
49

---

Published in Transactions on Machine Learning Research (03/2026)
each node, along with the phase during which they were generated. The original trees, however, much more detailed, but they would not fit in the content of this paper. The original trees contain the complete challenge description, the generated solutions, tests, attempts, and the corresponding analysis done at each node. This information allows for a comprchensive and fino-grained evaluation of model behavior at each node the search tree. We have included the full original trees our replication package Majdinasaly (2025). The nodes for each phase are color-coded for distinction, with yellow nodes representing those generated Phase 1, green nodes representing those generated in Phase 2, and blue nodes representing those generated in Phase 3. The edges indicate parent-chill relationships, with red edges indicating significant decrease in the child node's TD value compared to its parent, and green edges indicating otherwise. Each node is associated with specific attributes: the concepts related to the node are under "Concepts," the difficulty level under "Difficulty," " and the number of times the node has been visited, under "Visits."
50

---

Published in Transactions on Machine Learning Research (03/2026)
1
Pol Has
Bal
Hil Han
Hibe
188
Jabas
Mbba
Hbn
188
Bahoe
6408
Sibes
ma
Hibn
Bil
8
Baa
Blbbo
Bea
-3 Bas
Bas.
Das
Hos
Hobes
18859
Hehba
d0 8
Hibbe
Hellss
Bben
Inben
Tuliiss
Beilas
13 18hte
14
Bili
Hidbaa
I Bddiaa
cham
Ba a
dAbba
8ar8
be
Banh
Pab lee
Banh
HalBa
9518
8kbba
2th be
AB
10 Baa
30 bab
Hab
palp
Hal
18b0
1ag
Ha
B8a
18
10
fill
Figure 24: Search tree generated for 4o
51

---

Published in Transactions on Machine Learning Research (03/2026)
85
Figure 25: Search tree generated for GPT-OSS
52

---

Published in Transactions on Machine Learning Research (03/2026)
Bius
Bios
B1
-Biss
Bil
Billax
18
Blba
hoz
B8
10
Bau
Blana
Bk
Iidid
Fibbas
P Free
y u.
haa
Hox
Pibbe
-Bbbil
Bibos
14
blbba
184baa
14
Firkna
B5
de Fa
Ha
1 boz
Mane
Ha
Hillaa
Bibbs
Robas
Rin
B
a
18ho.
Babs
Pibbz
bes
145
ilipes
H
Baa
Rina
Abez
151 pm
Figure 26: Search tree generated for 4o-M
53

---

Published in Transactions on Machine Learning Research (03/2026)
HBpe
Abrr
Bas
Hillie
Hill
Bi8
181 Fil
Billee
Bill
Hill He
Blll
FinBax
Hhi m.
Btd
Sni
Bibes
Hibbe Hb
1hee
Bld he
Hel Beg
Hid
Bd H
Fbbbe
Red
Hi
fla H
Bol Pas.
Bid
Hell
Hil
Bel B
b.s.
Plle
Hud Be
Bid B
BiG
bib
pad p.
Bdlles
Ful 9.
p1 9b
Bides
thibas
th
Blbes
Fall H
Bibes
Abbs
Filh
B1
Fal ars..
B1 F
de Hare
Fti Bess.
B11 B.....
Hid Mldlla
Har D
Bid.
Find Has
iddd
Hill
Hel
Kill
Bie
Kilt Me
Bull
Hhoe
Blass
10
Fild
Bia
Hee
Bdl B
Balles
Halll
[540 bes
$1400
Bi
1440
Brbes
1144 b
Bibes
Billes
FiiBes
Das
Figure 27: Search tree generated for L-405b
54

---

Published in Transactions on Machine Learning Research (03/2026)
B5a
Plas
Tilbaa
Idital
libiza
Bl9gs
piliaza
B4a
Bear
pilia
Halpaa
fex.
1d
18
fillsa)
Willea
M8
Melkan
Hin
Tebox
fex
Hildd (Mbbal Hiffel
Alben
Bas
Figure 28: Search tree generated for DS3
55

---

Published in Transactions on Machine Learning Research (03/2026)
m
69
Abmu
Villes
Ibba.
18.8p
18b2
Hilng
1666 m.
10
Ballen
Vibau
feben
Hill
Pban
Rhee
Billes
11 m
Ho
bban
Phan
Fua
dellaa
19 Abn
dalles
18h
16b
Hillen
10
d8
Has
Mit
Polles
Alper
Hitez
Bales
B5
Halles
Pire
phan
Fibro
Pire
bid
Fills
bbx
Hill Beg
Hilliwe
fa
Blres
Filees
Figure 29: Search tree generated for L4S
56

---

Published in Transactions on Machine Learning Research (03/2026)
33iilam
Filinsi
BBbea
12
Jilles
Thilms
R.R.
(a) Search tree generated for L-70b
(b) Search tree generated for L-8b
Figure 30: Search trees generated for L-70b (a) and L-8b (b)
57


Reviews:
[{"review_id": "9P9NPb3VbF", "reviewer": "Reviewer_Mu9G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes PrismBench, a framework for evaluating LLM code generation that moves beyond static datasets. It employs a Reinforcement Learning approach, specifically modeling the evaluation as a Markov Decision Process traversed via Monte Carlo Tree Search. The system uses a multi-agent architecture to dynamically generate problems and test suites based on a concept-difficulty tree. This allows the benchmark to adaptively focus on the frontier of a model's capabilities, spending more compute on areas where the model struggles, and providing fine-grained diagnostic metrics regarding specific error types and concept mastery.\n\nStrength:\n1. Unlike typical benchmarks that yield a single pass rate, PrismBench offers deep diagnostic insights. By categorizing failure modes and mapping them to specific concept combinations, it provides actionable feedback for model developers rather than just a leaderboard ranking.\n2. The use of MCTS to guide the evaluation is a smart allocation of resources. Instead of wasting evaluation budget on trivial tasks that a model has mastered, the framework naturally converges on the boundary of the model's capabilities, providing a higher resolution view of the model's limitations.\n\nWeakness:\n1. The experimental setup uses GPT-4o-mini as the Challenge Designer to benchmark much larger and more capable models like GPT-4o and Llama-3.1-405b. There is a significant risk that the smaller model cannot generate very hard problems that are conceptually difficult enough to truly test the reasoning limits of the larger models. The difficulty measured might stem from ambiguity or poor problem formulation by the weaker designer, rather than true algorithmic complexity.\n2. The paper claims to avoid the pitfalls of \"LLM-as-judge\" by using sandbox execution. However, the validity of an execution-based reward is entirely dependent on the correctness of the LLM-generated test suites. If the test generator agent produces tests with false positives or false negatives, the objective execution signal becomes a noisy, probabilistic metric. The reliance on LLMs to define the ground truth reintroduces the very subjectivity the authors aim to avoid.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides extensive experimental data across 8 models and demonstrates that the MCTS approach effectively grows the tree in different ways for different models, supporting the claim that it adapts to model capability. The separation of error analysis provides clear evidence of the framework's diagnostic utility. However, the claim of \"objective execution-based signals\" is slightly overstated given the dependence on generated tests (as noted in Weakness 2), but the overall empirical results support the utility of the framework.", "audience_yn": "Yes", "audience_explanation": "The shift from static to dynamic benchmarking is a critical area of research as models saturate current datasets. The methodology of applying MCTS to evaluation coverage is novel and relevant to researchers focusing on evaluation rigor and model debugging.", "requested_changes": "1. Please provide an audit or study quantifying the accuracy of the generated test suites. Specifically, what is the rate of incorrect assertions generated by the Test Generator? The test validator agent is mentioned, but since it is also an LLM, a human verification or cross-check is needed to establish trust in the reward signal.\n2. Address the potential ceiling effect of using GPT-4o-mini. Provide evidence that very hard problems generated by 4o-mini are indeed algorithmically complex and distinct from medium problems, rather than just being longer or more vaguely specified.\n3. To contextualize the results, please show how the rankings or concept-mastery scores derived from PrismBench correlate with established static benchmarks or live leaderboards. This would help validate that PrismBench is measuring the same underlying coding capability, but with higher resolution.", "broader_impact_concerns": "No significant ethical concerns. The work focuses on improving technical evaluation standards.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "brGE5KvDiC", "reviewer": "Reviewer_dfij", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents PrismBench, a dynamic benchmark for LLM coding capabilities. Key features include: 1) Model the coding capacities of LLM as a space of programming concepts and difficulties for evaluation. 2) MCTS-guided exploration of the state space which dynamically allocates budget for the evaluation based on the model’s current performance to maximize information gain of the trials. 3) A three-phase multi-agent pipeline that decomposes the benchmark that generates coding challenge dynamically, performs comprehensive evaluation of model answer, and identify systematic issues of the model dynamically.\n\nStrength:\n1. This paper uses a novel formation of Markov decision process (MDP) of the evaluation, and uses a multi-agent workflow to overcomes the limitation of static benchmark and dynamic benchmark.\n2. The paper conducted extensive evaluation using the proposed benchmark on different models and examined different metrics in details, showcase the ability to adaptively evaluate different models on different programming concepts and difficulties and finding the systematic issues of the model. \n\nWeakness:\n1. This work generates leetcode-style coding challenges that focus on one or more programming concepts. While the result can be a proxy, the model's actual performance on real work problems is not clear.\n2. The benchmark relies on LLM-based agents, which may introduces variance and bias. In Sec 4.3 and table 2, the paper suggests using L-405b to avoid bias when evaluating 4o. However, the different models are evaluated with different LLM-based agents, it can be hard to directly compare the results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper proposed to formalize the evaluation as a MCTS-guided exploration over a concept-difficulty tree space, and uses a three-phase multi-agent pipeline for comprehensive evaluation. Section 5 provides extensive experiments results of the proposed benchmark methodology over different LLMs. By studying different metrics in Sec 5.1, the paper shows that the MCTS exploration is able to adaptively allocate the budget and finds the node (concept-difficulty pair) to evaluate that is challenging to the model. Sec 5.2 and 5.3 provides further analysis of how different models perform on the proposed benchmark, showing that the proposed benchmark is able to reveal the strength and limitation of the coding capabilities of different models.", "audience_yn": "Yes", "audience_explanation": "LLM coding has been getting great importance in research and practical use. It is important to develop systematic benchmark to understand the model capabilities and further the model development. This work presents a systematic benchmark that does comprehensive evaluation of the LLM coding capabilities.", "requested_changes": "1. Clarify how the evaluation results on leetcode-style challenges generate to real work problems.\n2. Clarify how the evaluation bias/variance introduced by LLM-based agents (see weakness 2) can be mitigated.", "broader_impact_concerns": "There are few ethical concern in this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yxE1qP1aJJ", "reviewer": "Reviewer_9Prt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces PrismBench, a dynamic benchmarking framework for evaluating LLM code generation. It formulates evaluation as a search problem over a concept–difficulty space and uses Monte Carlo Tree Search (MCTS) with execution-based rewards to adaptively generate challenges and identify model weaknesses. By decomposing evaluation into multiple agents and relying on sandbox execution rather than LLM judges, the framework aims to provide finer-grained, more robust diagnostics than static benchmarks.\n\nKey Strengths:\n1. The application of MCTS to dynamically explore specific weaknesses rather than using a fixed dataset is a methodological advancement.\n2. By relying on sandbox execution results instead of LLM-based judging, the framework avoids the subjective biases common in \"LLM-as-a-judge\" approaches.\n\nKey Weaknesses:\n1. The multi-agent interaction and MCTS iterations require significantly more computational resources and setup time compared to static benchmarks.\n2. The validity of the benchmark heavily relies on the \"Generator\" agent's ability to produce high-quality, solvable, and correct problem definitions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission supports its claims with strong empirical evidence. The authors validate the PrismBench framework by testing it on eight frontier models, demonstrating its ability to expose specific failure modes that static benchmarks miss due to saturation. The efficacy of the MCTS algorithm in dynamically navigating the difficulty landscape is visually and quantitatively supported by the generated search trees and success rate heatmaps. Furthermore, the reliance on sandbox-based execution rather than LLM-as-a-judge provides accurate, objective ground truth for the validity of the generated code, substantiating the claim of a  bias-free evaluation pipeline.", "audience_yn": "Yes", "audience_explanation": "The paper will be of interest to TMLR readers working on LLMs, evaluation methodology, and ml systems. Its dynamic benchmarking framework and use of search-based evaluation offer a novel perspective on diagnosing model capabilities beyond static benchmarks, which is relevant to ongoing research on reliable and robust model evaluation.", "requested_changes": "Critical:\n1. While Section 4.3 states that results are averaged over 3 independent runs, the reported confidence intervals (Tables 4 & 5) show significant variance (e.g., $\\pm 0.11$). The authors should explicitly analyze the root cause of this instability: does it stem from the stochastic nature of the MCTS exploration path or the evaluated models themselves? Crucially, the authors should discuss whether this variance is large enough to alter the relative rankings of models across different runs.\n\nStrengthening:\n1. The paper does not include explicit ablation studies; adding targeted ablations on key design choices (e.g., MCTS parameters, reward formulation, or agent roles) would strengthen the empirical grounding of the framework.\n2. While Section 4.4 provides absolute financial costs, a comparative analysis is missing. The authors should discuss the \"cost-to-insight\" ratio: does the high cost of PrismBench yield proportionally more valuable insights compared to low-cost/free static benchmarks? Additionally, reporting runtime/latency metrics (time to complete a benchmark run) would help users assess the practicality of the framework.", "broader_impact_concerns": "The paper primarily focuses on improving evaluation methodology for LLM code generation and therefore presents low direct ethical risk.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fONNmrHLsE", "reviewer": "Reviewer_bTMW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to devise a model evaluation framework for assessing the capability of different models in solving different kinds of programming tasks, eliciting the capabilities and weaknesses of each model, since these may differ. The framework models the evaluation task as a (variant of) an MDP whose structure is designed to elicit the different capabilities of different models, and concentrate the effort in those regions of the programming challenge search space where the model may have a difficult time solving the challenge.", "claims_and_evidence_yn": "No", "claims_explanation": "1. I am not convinced that the idea, in principle, does what it is supposed to do, since there are plenty of hyperparameters involved, and the results may differ between hyperparameter choices. The authors made no attempt to justify them, or show experimentally that their system is robust to this choice in any way.  This has several consequences:\n    1.1 The issue of trust - there is no reason to believe that 0.9 is the \"right\" discount factor, or that $\\alpha=0.9$ is the \"right\" learning rate. Same for penalties and thresholds. What if a different configuration yields a radically different conclusion about LLM capabilities?\n    1.2 We cannot use it in a leaderboard for the same reason. One can always claim that their model is put at a disadvantage because of some choice of hyperparameters, and can possibly even prove it: they can use a tuner, like Optuna, to find the best config for their model, and claim that **this** is the right config. Again, for the same reason it also doesn't foster trust, becuase the system can be easy to \"game\"\n2. The MDP is not really an MDP as claimed. The reward is claimed to be a function from $$S \\times A \\times S$$ to the real numbers, but every phase has its own reward, so I am assuming the phase is also a part of the state. There may be other missing parts I may have mised. The authors need to precisely formulate the model without hand-waving.\n3. Section 3.3, bottom of page 8: making separate agents for test development and for problem solving does not reflect real software development - these two things go hand in hand together. There's even an entire methodology that promotes it - test-driven development. Moreover, using the same LLM for writing tests and for solving makes the system also easy to \"game\" and fosters less trust\n4. It is extremely complex. Evaluation is most useful when it's simple and doesn't have a lot of \"moving parts\", because the interaction between them may be hard to reason about and again results in having a hard time trusting the results.\n\nIf I could trust the findings they would be very interesting. Especially the difference between different LLMs in different domains. But due to the above-mentioned comments, I am not convinced the system is trustworthy.", "audience_yn": "Yes", "audience_explanation": "Coding capability evaluation is a hot topic of research.", "requested_changes": "1. Either justify the hyperparameter choice, explain why you believe the system to be a reasonable evaluation framework for any \"reasonable\" choice (robustness), or actually prove robustness experimentally by running experiments and showing that the ranking between models across different dimensions remains similar (this one may be expensive, so I'm not sure how feasible it is)\n2. Clarify exactly what you mean in MDP\n3. Overall, do a better job at convincing why the system is not brittle and is not easy to game by a hyperparameter optimizer or using the fact that tests and problem solving are done using the same model.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)