CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis
Authors: Anjiang Wei
Tarun Suresh
Jiannan Cao
Naveen Kannan
Yuheng Wu
Kai Yan
Thiago S. F. X. Teixeira
Ke Wang
Alex Aiken
Venue: COLM 2025 (2025)
Keywords: Agent
Large Language Model
Reasoning
Code
Program Synthesis
TL;DR: We introduce CodeARC, a benchmark for inductive program synthesis where LLM agents iteratively refine code via oracle feedback, enabling more realistic and challenging evaluation for inductive reasoning.

Abstract: Inductive program synthesis, or programming by example, requires synthesizing functions from input-output examples that generalize to unseen inputs. While large language model agents have shown promise in programming tasks guided by natural language, their ability to perform inductive program synthesis is underexplored. Existing evaluation protocols rely on static sets of examples and held-out tests, offering no feedback when synthesized functions are incorrect and failing to reflect real-world scenarios such as reverse engineering. We propose CodeARC, the *Code Abstraction and Reasoning Challenge*, a new evaluation framework where agents interact with a hidden target function by querying it with new inputs, synthesizing candidate functions, and iteratively refining their solutions using a differential testing oracle. This interactive setting encourages agents to perform function calls and self-correction based on feedback. We construct the first large-scale benchmark for general-purpose inductive program synthesis, featuring 1114 functions. Among 18 models evaluated, o3-mini performs best with a success rate of 52.7%, highlighting the difficulty of this task. Fine-tuning LLaMA-3.1-8B-Instruct on curated synthesis traces yields up to a 31% relative performance gain. CodeARC provides a more realistic and challenging testbed for evaluating LLM-based program synthesis and inductive reasoning. Our code, data, and models are publicly available at https://github.com/Anjiang-Wei/CodeARC

Published as a conference paper at COLM 2025
CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis
Jiannan Cao², Naveen Kannan', Yuheng Wu!, Kai Yan², Thiago S. F. Teixeira', Ke Wang", Alex Aiken' Stanford University Univerriity of Illinois Urbana- Chammpagnn MIT +Intel
Anjiang Wei', Tarun Suresh?, Jiannan
NNnjjng University
Abstract
Inductive program synthesis, programming by example, requires syn- thesizing functions from input-output examples that generalize to unseen inputs. While large language model agents have shown promise in pro- gramming tasks guided by natural language, their ability to perform induc- tive program synthesis is underexplored. Existing evaluation protocols rely on static sets of examples and held-out tests, offering no feedback when synthesized functions are incorrect and failing to reflect real-world scenar- ios such as reverse engineering. We propose CodeARC, the Code Abstraction and Reasoning Challenge, new evaluation framework where agents interact with a hidden target function by querying it with new inputs, synthesizing candidate functions, and iteratively refining their solutions using a differ- ential testing oracle. This interactive setting encourages agents to perform function calls and self-correction based on feedback. We construct the first large-scale benchmark for general-purpe
inductive program synthesis, featuring 1114 functions. Among 18 models evaluated, o3-mini performs best with success rate of 52.7%, highlighting the difficulty of this task. Fine-tuning LLaMA-3.1-8DD 1---311-S--IIstuuc on curated synthesis traces yields up to 31% relative performance gain. CodeARC provides more realistic and challenging testbed for evaluating LLM-based program synthesis and inductive reasoning. Our code, data, and models are publicly available https://github com/ Anjiang-Weei /ooeeCC
1 Introduction
Inductive reasoning, i.e., the ability to identify patterns and form abstractions from limited examples, is widely recognized as fundamental aspect of human intelligence (Hayes et al., 2010; Yan et al., 2025). In the context of programming, inductive reasoning underpins the task of synthesizing functions that satisfy given input-output examples and generalize to unseen inputs. This task, commonly referred to inductive program synthesis programming by example (Manna & Waldinger, 1971; Myers, 1986; Feser et al., 2023; Li & Ellis, 2025), has broad application domains (Wang et al., 2017; Deng et al., 2024).
Recent advances in Large Language Models (LLMs) have led to the emergence of au- tonomous agents capable of decision-making, multi-step planning, tool use, and iterative self-improvement through interaction and feedback (Chen et al., 2023; Liu et al., 2023e; 2024b; Guo et al., 2024; Xi et al., 2025). While much of the existing work focuses on pro- gramming tasks guided by natural language (Chen et al., 2021; Austin et al., 2021; Jimenez et al., 2023; Jain et al., 2024b), we study fundamentally different problem: inductive pro- gram synthesis, where the objective is to infer the target program solely from input-output examples. This setting provides more rigorous test of inductive reasoning capabilities, as it eliminates natural language descriptions that can trigger retrieval-bace completions memorized during model training.
Correspondence to: anjiang wwtileccsssaaffoooo stanford.

---

Published as a conference paper at COLM 2025
Input
Output
Hidden Target Function
Testing Oracle
1.5
def fxnci(x): # Rounds half away from zero
Counterexample found!
2.4
1f >= 8:
Input
func1(2.5)
Func2(2.5)
return math. floorx 8.5)
2.6
-3
else:
2.5
return math.ceil(x 0.5)
Initial I/O examples
Feedback
Need a different rounding rule for .5? Let me try more...
I'm more confident new.
Rounding? Let me try more inputs.
Let me generate code.
func1(8.1)
func2
next try this?
Rounds half to even def func2(x): return (x)
Function
Invocation
Agent
Agent
Synthesized Function
Agent
Self-Reflection
Figure Overview of CodeARC. Our framework evaluates L.MSs' reasoning capabilities in inductive program synthesis. The agent begins with input-output examples, interacts with a hidden target function via function calls, and uses differential testing oracle to check the correctness of the synthesized function for self-reflection and refinement.
Designing an effective evaluation protocol for inductive program synthesis with LLMs is inherently challenging, multiple valid functions may satisfy a given set of input-output examples (as demonstrated in Section 5.2). The sfate-of-the-art protocol (Li & Ellis, 2025), which evaluates synthesized functions input-output examples, has several notable limitations. First, a small, static set of input- output examples may underspecify the target functions, especially for those with complex logic. Moreover, held-out tests may fail to reveal subtle semantic discrepancies between the generated and intended implementations. In addition, when the model produces an incorrect solution, it receives no feedback and has no opportunity to revise or explore alternatives. Finally, existing benchmarks for inductive program synthesis (Gulwani, 2011; 1; Wang et al., 2017; Wong et al., 2021; Deng et al., 2024; Li & Ellis, 2025) are focused on domain-specific tasks and do not assess the ability of LLMs to synthesize functions written in general-purpose programming languages.
held-out test cases after presenting 10 fixed
To address the limitations of existing evaluation protocols for inductive program synthesis, introduce CodeARC, the Code Abstraction and Reasoning Challenge, inspired by real-world scenarios such as decompilation and reverse engineering (Lin et al., 2010; Mantovani et al., 2022; Al-Kaswan et al., 2023). In such settings, agent is given binary executable (without source code) and must synthesize equivalent source code by observing input-output behav- ior. Instead of relying on fixed dataset, the agent can query the binary with new inputs, invoke a differential testing oracle, and use counterexamples for iterative refinement. This setup parallels the classic learnability framework of queries and counterexamples (Angluin, 1987; De la Higuera, 2010), here applied to program synthesis.
Figure illustrates how CodeARC instantiates this process: LLM-based agents begin with an initial set of input-output examples, query the ground-truth function for more examples, and debug synthesized code using a differential testing oracle. We impose fixed budgets on both the number of observable input-output examples and the number of testing oracle invocations for self-debugging. The task requires agents to proactively generate inputs (function calls) and revise solutions based on feedback rself-reflection). This interactive setup offers a more realistic alternative to prior static evaluation protocols.
We construct the first comprehensive dataset for general-purpose inductive program synthesis, featuring 1114 functions with initial input-output examples. Our benchmark targets general programming tasks and employs two state-of-the-as differential testing tools (Lukasczyk & Fraser, 2022; Etemadi et al., 2024) for correctness evaluation.
Our experiments demonstrate that CodeARC poses significant challenge for LLM-based inductive program synthesis. Among the 18 models evaluated, OpenAl o3-mini performs the best overall, yet only achieves 52.7% success rate. We further conduct ablation studies to
2

---

Published as a conference paper at COLM 2025
analyze how budgets on the number of input-output examples and oracle calls affect model performance. To enhance model capabilities, we generate synthetic fine-tuning data with curated synthesis traces that capture the reasoning steps. We show that supervised fine- tuning on 1--MA--.1--8-Issiieee
yields up to a 31% relative performance improvement.
In summary, our contributions are follows:
Interactive evaluation protocol for inductive program synthesis. We introduce a setup where agents start with fixed input-output examples but can generate new inputs to query ground-truth functions and invoke differential testing oracle to self-correct their solu- fions. This setup brings the task closer a real-world setting, e.g. reverse-engineering.
General-purpose benchmark with extensive evaluation. We construct the first large- scale, general-purpose benchmark for this task, including 1114 diverse functions. Among the 18 models evaluated, 03-mini achieves the best overall performance but still only reaches a success rate of 52.7%.
Synthetic data and fine-tuning. To boost model performance, we generate synthetic fine- tuning data containing curated synthesis traces that capture both function invocations and reasoning steps. We show that fine-tunining on LLaMA-3. ---tntttt-- yields up 31% % relative performance improvement.
2 Related Work
Inductive Program Synthesis Traditional inductive program synthesis methods rely solely on input-output examples, without natural language input. They focus on domain-specifii tasks like string and data transformations (Gulwani, 2011; Singh & Gulwani, 2016; Yagh- mazadeh al., 2016), SQL (Wang et al., 2017), visual programming (Wang et al., 2019), and quantum computing (Deng et al., 2024). These approaches typically define domain-specific language and use tailored search algorithms to prune the space efficiently (Feser et al., 2015; Polikarpova et al., 2016; Feng et al., 2018; Guria et al., 2023; Mell et al., 2024). In contrast, introduce the first general-purposs program synthesis benchmark for LLM-powered agents.
LLM Benchmarks for Code Most LLM benchmarks, such as HumanEval+ (Chen et al., 2021; Liu et al., 2023b), MBPP+ (Austin et al., 2021; Liu et al., 2023b), APPS (Hendrycks et al., 2021), and others (Li et al., 2022; Liu al., 2023c; Li et al., 2023; Jain et al., 2024a; Banerjee et al., 2025; Ni et al., 2024b; Liu et al., 2023d; Du et al., 2023; Suresh et al., 2025b; Zhuo et al., 2024; Yin et al., 2022; Lai et al., 2023; Patil al., 2025; Ugare et al., 2024; 2025; Wei et al., 2025d; Wu et al., 2025), evaluate code generation from natural language. Beyond generation, tasks like T/O prediction (Gu et al., 2024; Liu et al., 2024a), execution prediction (Liu al., 2023a; La Malfa et al., 2024; Ni et al., 2024a; Ding et al., 2024), bug localization (Suresh et al., 2025a), and program equivalence (Wei et al., 2025a) have also been studied. In contrast, we focus on predicting function bodies purely from input-output examples, without natural language. Prior work (Li & Ellis, 2025; Barke et al., 2024) targets domain-specific tasks, while introduce general-purpose benchmark with interactive evaluation protocol.
LLM Benchmarks for Reasoning LLMs are widely benchmarked on reasoning tasks across domains, including commonsense (Talmor et al., 2018), mathematical (Cobbe al., 2021), and logical (Han et al., 2022; Miao et al., 2021; Liu et al., 2020; Parmar et al., 2024; Wei et al., 2025e). Inductive reasoning, a core cognitive skill that generalizes from limited examples (Hayes et al., 2010), is increasingly studied in LLMs (Li et al., 2024; Ma et al., 2024; Xiao et al., 2024; Cai et al., 2024; Shao et al., 2024). ARC (Chollet, 2019) is a prominent benchmark for abstract pattern induction. Our work shares this goal but is for inductive program synthesis.
LLM-powered Agents LLM-based agents have shown strong performance in domains like web navigation (Zhou et al., 2024a;b), code generation (Zhang et al., 2023; Jimenez et al., 2024), performance optimization (Wei et al., 2025b;c), and ML experimentation (Huang et al., 2024). They interact with environments, invoke functions, make decisions, and self- reflect (Madaan et al., 2023; Paul et al., 2023; Xie et al., 2023; Huang et al., 2022; Shinn et al.,
3

---

Published as a conference paper at COLM 2025
2023). We introduce the first benchmark to systematically evaluate agents' capabilities in inductive program synthesis, providing a rigorous testbed for inductive reasoning and program synthesis.
3 Method
3.1 Problem Definition of Inductive Program Synthesis
We formalize the inductive program synthesis task as follows. Let f be hidden ground- truth function that maps inputs x E X' to outputs E y. The synthesizer is given an initial set of input-output examples & (x) where yi = (x;),, and the goal to synthesize a program such that = f*, i.e.,
VxE f(x) = (x).
To evaluate whether synthesized function f is correct, we introduce differential testing oracle O. The oracle takes input both the synthesized function f and the hidden ground- truth function f and attempts to identify inputs on which their behaviors differ. Formally, the oracle operates follows:
[Pass, Fail(x),
if Nest, f(x)=f((x)) if 3x Nest such that f(x) + fx),,,
of
where Nest X' is set of test inputs dynamically selected by the oracle.
Unlike fixed held-out test sets used in prior work, the differential testing oracle conditions on both f* and the candidate f, generating targeted inputs to expose discrepancies. On failure, it returns counterexample equivalence checking is fundamentally undecidable, and thus no perfect oracle exists. To approximate oracle functionality, we adopt two state-of-ther enabling a more robust and practical evaluation than prior work or reliance on a single tool (see Appendix A.3).
Atust such that f(x) + fx(x.. Note that program
differential testing tools,
3.2 Interactive Evaluation Protocol for LLM Agents
To evaluate the capabilities of MLM--bseed agents in inductive program synthesis, we in- troduce an interactive protocol. This protocol extends beyond static evaluation settings by enabling dynamic interaction with the hidden ground-truth function and the differential testing oracle.
Initial Information. At the beginning of the task, the agent is provided with an initial set input-output examples E = ((x, " as explained previously. This set serves as partial information about the target function.
Action Space. During evaluation, the agent may take two types of actions. First, may query the ground-truth function f* at chosen input x E X, and obtain the corresponding output (x), thereby augmenting its observed set of input-output pairs. Second, it may synthesize a candidate program f and invoke the differential testing oracle O(f", f), which returns PASS if discrepancies are found on a dynamically generated test set, FAIL with
a counterexample E Xrstt such that fxx) # ft (x).
Self-Reflection. If the oracle returns FAIL,, counterexample 8, which is tuple of (x, (x)) will be provided to the agent. This counterexample helps the agent to self-reflect and revise its current hypothesis, either by issuing additional queries f or syn- thesizing new programs. The ability to take such feedback is crucial for iterative refinement.

---

Published as a conference paper at COLM 2025
Budget Constraints. The agent operates under two budget parameters: Bio and Boracle- Bio limits the total number of input-output examples that the agent can observe from the ground-truth function f*, while Braale limits the number of invocations to the differential testing oracle O.
Evaluation Metrics. The task is considered successful if the final synthesized program f, produced within the given budgets, receives a PASS from the differential testing ora- cle of We assess LLM agent performance along two dimensions: correctness and efficiency, prioritizing correctness. Correctness is the success rate, iee, the proportion of problems solved within the budget. Efficiency is the average number of input-output queries and oracle invocations per problem.
3.3 Benchmark Preparation
Our benchmark is designed to evaluate LLM agents on the inductive synthesis of general- purpose Python programs. This contrasts with prior benchmarks that focus on domain- specific tasks or programs written in domain-specific languages, such as string manipulation and SQL query generation (Yaghmazadeh et al., 2017; Deng et al., 2024; Li & Ellis, 2025).
Programs for Synthesis. We curate a diverse collection of Python functions sampled from three established code generation benchmarks: HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and APPS (Hendrycks et al., 2021). HumanEval and MBPP primarily consist of simple, entry-level programming tasks, whereas APPS contains more challenging problems that resemble competition-level code exercises. Importantly, we ex- tract only the function bodies from these benchmarks and do not use any accompanying natural language descriptions.
Annotated vs. Anonymized. To assess the extent which function names help the LLM agent synthesize the correct program, we construct two versions of the benchmark. In the annotated version, function names that reflect the intended functionality of the task (e.g., is is_pal indrome) are made available to the agent. In the anonymized version, all func- tion names are replaced with a generic identifier (i.e., solution). This design isolates the influence of identifier cues on synthesis performance. We report results on both versions.
Initial Input-Output Examples. Each synthesis task includes 10 fixed input-output exam- ples that specify the target function's expected behavior. We use GPT-4o to generate diverse inputs and execute the original function to obtain the corresponding ground-truth outputs.
Synthetic Data Generation. We generate a synthetic dataset for fine-tuning (described in more detail in Section 3.4) by first collecting 50 seed Python functions that are disjoint from the evaluation set. Using these seeds, we prompt GPTT-4o to synthesize a diverse set of new functions, yielding 10,000 candidates. For each generated function, we additionally instruct GPT-4o to produce 10 representative inputs that expose the function's behavior and highlight patterns in its input-output relationships. These inputs executed to verify the executability of the functions, and we discard any that fail at this stage. To reduce redundancy, we deduplicate the dataset based function names, finally resulting in 5,405 unique Python functions used for fine-tuning.
3.4 Fine-Tuning on Synthetic Data
We evaluate whether fine-tuning on curated synthesis traces, which capture both function calls and reasoning, improves LLM performance CodeARC, using distillation approach that imitates the reasoning of teacher model with access to ft, the ground-truth function.
During training, we first run the interactive evaluation protocol described in Section 3.2 with a frozen teacher model. Unlike a standard evaluation, we prepend set of task-specific instructions Py. to each teacher prompt. This prefix includes the function body of f and explicitly instructs the teacher to (1) query f" on informative inputs, (2) explain the rationale
5

---

Published as a conference paper at COLM 2025
behind those queries, and (3) synthesize the full implementation of f only when confident that the correct logic can be inferred from the accumulated input-output pairs. The student model, which is the only model we fine-tune, lears to mimic the teacher's s reasoning and synthesis behavior without seeing the teacher's prompt that includes the target function.
We provide the teacher with access to f because find that, in many cases, the model struggles to solve the task independently. Without knowledge of the ground truth, the teacher is often too weak to generate meaningful queries or explanations, limiting the effectiveness of the resulting supervision.
While executing the evaluation protocol, record the multi-turn conversation history Cr which comprises the teacher model's prompts and responses. Let и be the total number of turns in Cy and denote by x' the sequence of tokens in the ith turn. Furthermore, let represent the number of tokens in the teacher-specific instruction prefix Pp..
We fine-tune the student model using a language modeling objective by minimizing the negative log-likelihood of predicting the next token in Cy:
L=
Σ
Σ j
log P
(x) CTyct
Here, P (x/ Cr,a denotes the probability of generating token x given all tokens from previous turns, Cr and the tokens preceding x in the current turn, x'Zp By starting the inner sum at = p 1, the teacher-specific instructions P,- are excluded from the loss computation, ensuring that the training signal comes only from the parts of Cy available during inference.
4 Experiment Setup
We construct two versions (annotated and anonymized) of the 1114 Python functions, drawn from HumanEval+, MBPP+ (Liu et al., 2023b), and APPS (Hendrycks et al., 2021). Table 1 summa- rizes key statistics. Unlike prior work in program synthesis that often focuses on domain specific languages and constrained settings, our bench- mark consists of programs written in Python, a general-purpose language that captures broader range of real-world algorithms and tasks.
Lines of Code
Source
Functions
Min
Max
Avg
HumanEval+ MBPP+ APPS
56
18.5
131
2
21
3.9
905
74
9.5
Annotated
1114
74
9.5
Anonymized 1114
74
9.5
Table 1: Number of functions and lines of code statistics for each benchmark source across both dataset versions.
For the main evaluation (Section 5.1), we provide 10 initial input-output examples and set the query budget to 30 input-output pairs and 2 oracle calls (Bro = 30, Boracle = 2), chosen based on practical constraints such as API cost and runtime. Section 5.3 reports ablation studies on both budgets. We use two state-of-the-art differential testing tools, PYNGUIN (Lukasczyk & Fraser, 2022) and MOKAV (Etemadi et al., 2024). For supervised fine-tuning synthetic reasoning trajectories, we gpt-4o as the teacher model and -LaMA-3.1-8BBInsitt. as the student. See Appendix A for further details, including prompts, fine-tuning parameters, and additional results on the differential testing tools.
Results
5.1 Main Results
Table 2 shows the results for 18 large language models on CodeARC. The order is sorted based on the success rate on the anonymized dataset. We also report the average number of observed input-output examples and oracle invocations. Our findings are follows:

---

Published as a conference paper at COLM 2025
Annotated Dataset
Anonymized Dataset
Model
#1/O
Oracle
Success (%)
#1/O
Oracle
Success (%)
Llama-3.2-3B
28.3
1.9
11.0
29.3
2.0
4.8
Mixtral-8x7B
27.4
1.9
20.3
28.5
1.9
12.0
Llama-3.1-80
28.0
1.8
19.3
28.6
1.9
13.7
Mixtral-6x22B
26.7
1.8
25.1
28.1
1.9
15.0
QwQ-32B
24.6
1.8
20.0
25.7
1.9
15.4
Qwen2.5-7B
26.9
29.2
28.3
1.9
15.8
Llama-3.2-11B
27.3
1.8
24.9
28.3
1.9
16.1
gpt-4o-mini
27.0
1.8
26.1
27.9
1.8
18.5
Ll2ma-3.2-90B
26.2
1.8
28.4
27.7
1.9
19.7
Llama-3.1-70B
26.9
27.9
20.0
Qwen2.5-72B
25.5
30.1
27.1
1.8
21.6
Llama-3.1-405B
24.2
1.7
38.6
26.0
1.8
26.7
gpt-4o
23.4
1.7
37.8
25.2
1.8
28.7
DeepSeek-V3 V3
23.7
1.7
37.7
25.1
1.8
29.5
claude3.7-sonnet
23.6
24.6
33.8
DeepSeek-RI
18.6
1.6
49.8
20.3
1.7
41.3
ol-mini
21.0
1.6
53.2
21.5
1.6
47.7
03-mini
15.6
1.5
59.5
16.0
1.6
52.7
Table Success rates of LLMs on CodeARC using both annotated and anonymized datasets. We also report the average number of observed input-output examples and oracle invoca- tions. All open-source models are instruction-tuned
Model Scale & Success Rate (Annotated)
Model Scale & Success Rate (Anonymized)
40
Success Rate (%)
30
Success Rate
20
10
10
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2'
2
2
Model Parameters (Billion)
Model Parameters (Billion)
Model Families
Qwen2.5
Llama-3.1
Llama-3.2
Mixtral
Figure 2: Scaling trend on CodeARC.
Reasoning models perform best. Reasoning models (o3-mini, ol-mini, DeepSeek-RI) achieve the highest success rates, all exceeding 40% on the anonymized dataset. They also require fewer T/O examples and oracle calls, indicating greater accuracy and efficiency.
CodeARC is challenging benchmark. Among the 18 evaluated models, only OpenAl's 03-mini achieves over 50% success on both datasets (i.e., 99.5% on the annotated and 52.7% on the anonymized) while all other models fall short of this threshold. This underscores the difficulty of the task and reveals the limitations of current models in inductive reasoning.
Anonymization of function names reduces performance, but trends persist. All models show a modest drop in success rate on the anonymized dataset. However, the overall ranking remains largely consistent. This suggests that while the presence of meaningful function names provides some benefit, strong inductive reasoning remains the main factor behind high performance on this synthesis benchmark.
Scaling up model size improves performance. Larger models generally achieve better performance, shown in Figure (log-scale x-axis). All model families exhibit scaling trends, though with varying consistency. Llama-3.1 scales steadily, while Llama-3.2 plateaus at larger sizes, likely due to its multimodal focus. Qwen2.5 shows clearer scaling
7

---

Published as a conference paper at COLM 2025
Metric
Problems (%)
Success Rate (%)
Model
10 I/O
20 I/O
30
I/O
PassI: Initial I/O Examples
1506 (67.6%)
Pass2: Testing Oracle
866 (38.9%)
ol-mini
43.7
49.6
50.5
Pass1 Pass2
640 (28.7%)
03-mini
51.3
53.8
56.1
Table 3: Number of problems (in both datasets) where the synthesized function passes the initial examples compared to the oracle.
Table 4: Success rates (%) with varying budgets on the observable input-output examples (on both datasets).
anonymized data, where reasoning is required over memorization, highlighting model size's impact on generalization.
5.2 Do Initial Input-Output Examples Underspecify the Target Function?
To assess whether 10 input-output examples (Li Ellis, 2025) suffice specify the target function, we evaluate the first synthesized function of 03-mini, iee, the strongest model in our study. Functions that pass the initial examples but fail under oracle testing indicate under-specification, motivating the need for additional examples or oracle-guided feedback. As shown in Table 3, 67.6% pass the initial 10 input-output example tests, but only 38. 38.9% pass the oracle, revealing 640 cases (28.7%) where the initial examples fail to uniquely specify the target function. These findings show that initial input-outpur examples often under-specify program behavior, motivating additional queries and oracle-guidet feedback for reliable evaluation. This motivates the design of our interactive evaluation protocol.
Model Scale & Success Rate (Annotated)
Model Scale & Success Rate (Anonymized)
70
60
Success Rate (%)
60
Success Rate (%)
50
50
40
40
30
20
20
10
Maximum Number of Oracle Invocations
Maximum Number of Oracle Invocations
Models
gpt-4o-mini
gpt-4o
ol-mini
03-mini
Figure 3: Success rates (%) of LLM models across varying numbers of oracle invocations.
5.3 Ablation Study: Input-Output Queries and Oracle Feedback
We perform two ablation studies to evaluate how varying the budgets for querying ground- truth functions and invoking the oracle impacts performance.
Effect of Input-Output Query Budget. We evaluate o3-mini and ol-mini with input- output budgets of 10, 20, and 30, using the same setup as Section 5.1. Table 4 shows that success rates improve consistently with more examples.
Effect Oracle Invocation Budget. We vary the number of allowed oracle invocations and report success rates in Figure for four models on both datasets. More oracle calls consistently improve performance, showing that counterexamples from differential testing valuable for guiding iterative refinement.
These results demonstrate that incorporating both querying mechanisms and oracle feedback consistently enhances overall performance. This improvement underscores the importance
8

---

Published as a conference paper at COLM 2025
Input
Output
Testing Oracle
Synthesized Function
type: 'list'
Initial Examples
4, 2,
def solution(1):
Input
Expected
Output
for in range(len(1)):
False
[1. (5,6), [5,5]] Typearror
False
for j in range(i 3, len(1)):
[a bb 1c d) 'e)
if 1[]] -- 1[j]:
Updated Function
New Invocations (None, "None")
return False
def solution(1):
True
return True
return len(1) == len(set(1))
Figure 4: Case Study. The model queries edge cases, synthesizes a comparison function, receives counterexample from the oracle, and corrects it with set-based solution.
of adopting an interactive evaluation protocol rather than relying solely on static, one-shot evaluation approaches.
5.4 Performance of Fine-Tuned Models
Table 5 shows that fine-tuning the model on cu-
Success Rate (%)
Dataset
Base Model Fine-Tuned
Rel. A
rated synthesis traces yields consistent improvements across both datasets. The larger gain on the annotated vari- ant suggests that fine-tuning particu- larly effective when semantically infor- mative identifiers are present. Notably, this performance gap emerges despite both datasets being evaluated under the same model architecture and train- ing methodology discussed in Section 3.4. These results indicate that while fine-tuning helps, there remains substantial room for further improvement. This suggests that future research may focus on enhancing the quality and diversity of the fine-tuning dataset, particularly for the anonymized variant, where gains are more limited. Another promising direction is to explore reinforcement learning approaches (Wei et al., 2025c) that optimize for higher synthesis success rates, potentially overcoming limitations of supervised fine-tuning alone.
Annotated Anonymized
19.3
25.3
+31%
13.7
15.0
+9.5%
Table 5: Success rates of LLaMA-3.1-BB-Instte and its fine-tuned variant annotated and anonymized datasets. Fine-tuning improves per- formance, especially the annotated dataset.
To assess whether fine-tuning affects general code generation capabilities, we additionally evaluate the models on BigCodeBench (Zhuo et al., 2024). Unlike our inductive program synthesis benchmark, BigCodeBench measures traditional code generation from natural language descriptions, covering diverse function calls and library usage. Our synthetic fine-tuning dataset is constructed independently, with no overlap with BigCodeBench, providing a clean test of the fine-tuned model's performance on other coding tasks.
On BigCo,eBench, the base model achieves pass@1 score of 40.1%, while the fine-tuned model attains 39.6%, a marginal decrease of 0.5 percentage points. This aligns with prior findings on catastrophic forgetting (Kotha et al., 2024), a well-documented phenomenon in which training on new task can impair performance on previously learned ones. We consider this small degradation acceptable given the gains in inductive program synthesis.
5.5 Case Study
Figure 4 shows an interaction trace from our benchmark. The model starts by querying the ground-truth function with edge-case inputs, aiming to probe its behavior beyond the initial examples. It then synthesizes candidate solution using pairwise comparisons, which passes the given examples but fails on a counterexample with unhashable elements. From the error message, the model correctly infers that the ground-truth function raises a TypeError, while its own does not. On the second attempt, it reasons that using set simplifies the uniqueness check and synthesizes the correct set-based function. This case illustrates how the model combines function invocation and oracle feedback to perform inductive program synthesis. The full trace is in Appendix A.1.

---

Published as a conference paper at COLM 2025
6 Discussion
6.1 Results on BigCodeBench for Broader Domain Coverage
Our interactive protocol is domain-agnostic and readily extensible to any Python program. To further increase the domain coverage, we extend the original CodeARC dataset with 200 Python functions randomly sampled from BigCodeBench (Zhuo et al., 2024), which includes problems involving scientific computing g. NumPy, SciPy, pandas), machine learning libraries (e.g., sklearn), visualization (e.g,, matplotlib), and system libraries. This addition enhances domain diversity beyond traditional iggorithm-focusee problems.
Table 6 shows that reason- ing models (o3-mini, ol-mini) achieve lower success rates BigCodeBench compared to their performance the orig- inal CoCeARC benchmarks. In contrast, non-reasoning models perform better on Big- CodeBench. We attribute this to the nature of Big- CodeBench tasks, which often involve domain-specife APIs and library usage, where non-reasoning models perform better. Notably, our fine-tuned model shows a substantial gain over the base model on BigCodeBench, indicating that the model fine-tuned on synthesis reasoning traces can generalize to unseen cltaastss.
Model
CodeARC (Original)
BigCodeBench
LLaMA-3.1-8B Base
19.3
32.5
Fine-tuned LLaMA
25.3
40.0
gpt-4o
37.8
46.5
ol-mini 03-mini
53.2
40.0
59.5
43.0
Table 6: Success rates (%) on the original CodeARC bench- mark and BigCodeBench, which covers diverse domains such scientific computing and machine learning, extend- ing beyond algorithm-focusee problems.
6.2 Impact of Providing Access to Python Interpreter
Our default evaluation restricts access to external tools such Python interpreter, though the CodeARC protocol can easily incorporate them via additional actions. To assess the impact, we added code execution action, updated the prompts, and evaluated 100 sampled problems from the dataset.
Results shown in Table suggest that pro- viding access to a Python interpreter does not uniformly improve model performance. For instance, while o3-mini benefits mod- estly (+2.0%), gpt-4o's performance slightly decreases (-1.5%). This outcome highlights that expanding the agent's action space may sometimes lead to suboptimal behavior.
Model
w/o Interpreter
w/ Interpreter
gpt-4o 03-mini
48.5
46.0
69.0
71.0
Table 7: Success rates (%) with and without access to Python interpreter.
We note that internal code simulation is natural part of the inductive program synthesis process. Our benchmark is designed to evaluate inductive reasoning capability, and we believe that requiring models to reason without relying on external execution tools remains meaningful and challenging setting.
7 Conclusion
We introduce CodeARC, new framework for evaluating LLMs on inductive program synthesis through interactive input generation and self-correction. Unlike static protocols, CodeARC allows agents to query ground truth function and use differential testing oracle to get feedback for iterative refinement. Designed assess inductive reasoning from input-output examples, our benchmark covers 1114 diverse and general-purpose functions and evaluates 18 language models. The pert--erforming model, OpenAl 03-mini, achieves success rate of 52.7%. Fine-tuning LLaMA-3.1 1.AMMA-..---B--- 1-8B-Instruct curated synthesis traces results in a 31% relative performance gain. CodeARC provides a more realistic and challenging testbed for evaluating L.L.-based inductive program synthesis.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
We thank Yuhui Zhang and Xiaohan Wang for the discussion. This work was partially supported by a Google Research Award.
References
Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premkumar De- vanbu, and Arie van Deursen. Extending source code pre-trained language models to summarise decompiled binaries. In 2023 IEEE International Conference on Software Analysis,
Evolution and Reengineering (SANER) pp. 260-271. IEEE, 2023.
Dana Angluin. Learning regular sets from queries and counterexamples. Information and compulation, 75(2)877166 1987.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. ar preprint arXiv:2108.0773: 2021.
Debangshu Banerjee, Tarun Suresh, Shubham Ugare, Sasa Misailovic, and Gagandeep Singh. Crane: Reasoning with constrained Ilm generation, 2025. URL httst https://arrke org/ /bbs/
09061.
Shraddha Barke, Emmanuel Anaya Gonzalez, Saketh Ram Kasibatla, Taylor Berg- Kirkpatrick, and Nadia Polikarpova. Hysynth: Context-free Ilm approximation for guiding program synthesis. Advances in Neural Information Processing Systems, 37:15612- 15645, 2024.
Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, and Lei Li. The role of deductive and inductive reasoning in large language models. Xrv preprint ar axXt:2240.00289, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint #XXv:2107.03374, 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint XrXo:2204..0518,, 2023.
François Chollet. On the measure intelligence. ar preprint arXit:1911.0155: 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ar preprint ar Xrx::22100.111 2021.
Colin De la Higuera. Grammatical inference: learning automata and grammars. Cambridge University Press, 2010.
Haowei Deng, Runzhou Tao, Yuxiang Peng, and Xiaodi Wu. case for synthesis of recursive quantum unitary programs. Proceedings of the ACM on Programming Languages, 8(POPL.) 1759-1788, 2024.
Yangruibo Ding, Jinjun Peng, Marcus Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. Semcoder: Training code language models with comprehensive semantics reasoning. arXiv preprint ar Xii:2406.01006 2024.
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A manually-crafted benchmark for evaluating 1lms on class-level code generation. ar preprint (ttt:::33000886666 2023.
Khashayar Etemadi, Bardia Mohammadi, Zhendong Su, and Martin Monperrus. Mokav: Execution-drived differential testing with 1lms. ar Xio preprint ar arXio:2406.10333 10375, 2024.
11

---

Published as a conference paper at COLM 2025
Yu Feng, Ruben Martins, Osbert Bastani, and Isil Dillig. Program synthesis using conflict- driven learning. ACM SIGPLAN Notices, 53(4):420-435, 2018.
Jack Feser, Işil Dillig, and Armando Solar-Lezama. Inductive program synthesis guided by observational program similarity. Proceedings of the ACM on Programming Languages, 7 (00PSLA2))112-4400 2023.
John K Feser, Swarat Chaudhuri, and Isil Dillig. Synthesizing data structure transformations from input-output examples. ACM SIGPLAN Notices, 50(6):229-239, 2015.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. Xiviv preprint ar Xxx::2407..0306 2024.
Sumnt Gulwani. Automating string processing in spreadsheets using input-output examples. ACM Sigplan Notices, 46(1):317-330, 2011.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: survey of progress and challenges. ar Xio preprint Xkio:2402.01680 2024.
Sankha Narayan Guria, Jeffrey S Foster, and David Van Horn. Absynthe: Abstract interpretation-guided synthesis. Proceedings of the ACM on Programming Languages, 7 (PLDi):1584-1607, 2023.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, et al. Folio: Natural language reasoning with first-order logic. Xrviv preprint 7XXi:2209.008400 2022.
Brett K Hayes, Evan Heit, and Haruka Swendsen. Inductive reasoning. Wiley intendisciplinary
reviews: Cognitive science, 1(2):278-292, 2010.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, al. Measuring coding challenge competence with apps. ar Xiv preprint ar axXiv:2105.09936 2021.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. ar preprint arXiv:2210.11610, 2022.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation, 2024. URL halhhtss://arxte wttp://wwww.g/b////////// 03302.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contami- nation free evaluation of large language models for code. arXiv preprint ar 4rX1::2403.09944, 2024a.
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and lon Stoica. R2e: Turning any github repository into a programming agent environment. In Forty-first International Conference on Machine Learning, 2024b.
Carlos E. Jimenenz, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? Xiv preprint arXio:2310.06770, 2023.
Carlos F. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language Shunyu resolve real-world github issues?, 2024. URL https:://aran wrwg https:://axxi.orr//aa//331 06770.
Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting in language models via implicit inference. In The Twelfth International Conference on Learning Representations, 2024. URL. httpss://ppeereview.rr www.:e/eeooorrriiiiiiiiiiieee
12

---

Published as a conference paper at COLM 2025
Emanuele La Malfa, Christoph Weinhuber, Orazio Torre, Fangru Lin, Samuele Marro, Anthony Cohn, Nigel Shadbolt, and Michael Wooldridge. Code simulation challenges for lange language models. arXiv preprint 2Xio:2401.09077 2024.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen- tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, Pp- 18319-18345. PMLR, 2023.
Jiachun Li, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. Mirage: Eval- uating and explaining inductive reasoning process in language models. arXiv preprint arXiv:2410009422 2024.
Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXit:2312.14852 2023.
Wen-Ding Li and Kevin Ellis. Is programming by example solved by Ilms? Advances in Neural Information Processing Systems, 77:44761-44799 2025.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):0092-1997 2022.
Zhiqiang Lin, Xiangyu Zhang, and Dongyan Xu. Automatic reverse engineering of data structures from binary execution. In Proceedings of the 11th Anmual Information Security Symposium, pp. 1-1, 2010.
Changshu Liu, Shizhuo Dylan Zhang, Ali Reza Ibrahimzada, and Reyhaneh Jabbarvand. Codemind: A framework to challenge large language models for code reasoning. ar Xiv preprint ar Xriv:2402.09644 2024a.
Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. Code execution with pre-trained language models. aiv preprint ar ttto23305.0388. 2023a.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arv preprint ar Xiw::i00000000 S0t:20007.0814, 2020.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code gener- ated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, 2023b.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code gener- ated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558-21577 2023c.
Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. Large language model-based agents for software engineering: A survey. ar Xiv preprint ar Xiv:2409. 02977,, 2024b.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXio:2306.03093 2023d.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating 11ms agents. Xiv preprint ar Xiv:2208.033 036688 2023e.
Stephan Lukasczyk and Gordon Fraser. Pynguin: Automated unit test generation for python. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pp. 168-172, 2022.
13

---

Published as a conference paper at COLM 2025
Kaijing Ma, Xinrun Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, et al. Kor-bench: Benchmarking language models on onowledgee reasoning tasks. arxio preprint ar arXio:2410.06526, 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36: 46534-46594, 2023.
Zohar Manna and Richard J Waldinger. Toward automatic program synthesis. Communica- tions of the ACM, 14(3):151-165, 1971.
Alessandro Mantovani, Simone Aonzo, Yanick Fratantonio, and Davide Balzarotti. (RE- Mind]: a first look inside the mind of a reverse engineer. In 31st USENIX Security Symposium (USENIX Security 22), Pp. 2727-2745, 2022.
Stephen Mell, Steve Zdancewic, and Osbert Bastani. Optimal program synthesis via abstract interpretation. Proceedings of the ACM on Programming Languages, (PPOPL):45-448, 2024.
Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. ar Xiv preprint ar arXic:2106.1577,, 2021.
Brad A Myers. Visual programming, programming by example, and program visualization: taxonomy. ACM sigchi bulletin, 17(4):59-66, 1986.
Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. Next: Teaching large language models to reason about code execution. arv preprint drXi::2404.1144 14662, 2024a.
Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, et al. L2ceval: Evaluating language-to-code genera- tion capabilities of large language models. Transactions of the Association for Computational Linguistics, 12:1311-1329, 2024b.
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. ar preprint ar Xrx::2244.11522, 2024.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 126544-126565, 2025.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback intermediate representations. ariiv preprint ar arXiv:2304.01904, 2023.
Nadia Polikarpova, Ivan Kuraj, and Armando Solar-Lezama. Program synthesis from polymorphic refinement types. ACM SIGPLAN Notices, 51(():222-533 538, 2016.
Yunfan Shao, Linyang Li, Yichuan Ma, Peiji Li, Demin Song, Qinyuan Cheng, Shimin Li, Xiaonan Li, Pengyu Wang, Qipeng Guo, et al. Case2code: Learning inductive reasoning with synthetic data. arviv e-prints, pp. arXiv-2407, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8663 2023.
Rishabh Singh and Sumit Gulwani. Transforming spreadsheet data types using examples. In Proceedings of the 43rd Anmual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pp. 343-356, 2016.
14

---

Published as a conference paper at COLM 2025
Tarun Suresh, Revanth Gangi Reddy, Yifei Xu, Zach Nussbaum, Andriy Mulyar, Brandon Duderstadt, and Heng Ji. Cornstack: High-quality contrastive data for better code retrieval and reranking, 2025a. URL https://arxiv wwtp:/iwii..oogoooo.eroeee wttp://wwww...////......o www.a///...///eeee 01007.
Tarun Suresh, Shubham Ugare, Gagandeep Singh, and Sasa Misailovic. Is the watermarking of llm-generated code robust?, 2025b. URL https: https/////aaxiioooaabbbs.///s orggabs//40g) 17983.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXio:1811.00937, 2018.
Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. Syncode: Lim generation with grammar augmentation, 2024. URL https://arxiv.orgg abs/2403.01623
Shubham Ugare, Rohan Gumaste, Tarun Suresh, Gagandeep Singh, and Sasa Misailovic. Itergen: Iterative semantic-aware structured Ilm generation with backtracking, 2025. URL https://arxiv..rg//bb//// .7295.
Chenglong Wang, Alvin Cheung, and Rastislav Bodik. Synthesizing highly expressive sql queries from input-output examples. In Proceedings of the 38th ACM SIGPLAN Conference Programming Language Design and Implementation, pp. 452-466, 2017.
Chenglong Wang, Yu Feng, Rastislav Bodik, Alvin Cheung, and Isil Dillig. Visualization by 096668.
example, 2019. URL httppp/:///xxx..rrg//o///o//
Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yaofeng Sun, Yuan Liu, Thiago SFX Teixeira, Diyi Yang, et al. Equibench: Benchmarking code reasoning capabilities of large language models via equivalence checking, ar arXiv preprint axXi::2502.12466, 2025a.
Anjiang Wei, Allen Nie, Thiago S. X. Teixeira, Rohan Yadav, Wonchan Lee, Ke Wang, and Alex Aiken. Improving parallel program performance with LLM optimizers via agent-system interfaces. In Forty-second International Conference on Machine Learning, 2025b. URL https::/ net/forumlid-3hheeee
Anjiang Wei, Tarun Suresh, Huanmi Tan, Yinglun Xu, Gagandeep Singh, Ke Wang, and Alex Aiken. Improving assembly code performance with large language models via reinforcement learning. ar Xio preprint ar arXiw:2505.11 11480, 2025c
Anjiang Wei, Huanmi Tan, Tarun Suresh, Daniel Mendoza, Thiago SFX Teixeira, Ke Wang, Caroline Trippel, and Alex Aiken. Vericoder: Enhancing llm-based rtl code generation through functional correctness validation. ar preprint ar Xiv::2250 15659,, 2025d.
Anjiang Wei, Yuheng Wu, Yingjia Wan, Tarun Suresh, Huanmi Tan, Zhanke Zhou, Sanmi Koyejo, Ke Wang, and Alex Aiken. Satbench: Benchmarking Iims' logical reasoning via automated puzzle generation from sat formulas. arXiv preprint ar Xio:2505.11615, 2025e.
Catherine Wong, Kevin M Ellis, Joshua Tenenbaum, and Jacob Andreas. Leveraging lan- guage to learn program abstractions and search heuristics. In International conference on machine learning, pp. 11299-11204. PMLR, 2021.
Jie JW Wu, Manav Chaudhary, Davit Abrahamyan, Arhaan Khaku, Anjiang Wei, and Fatemeh Fard. Clarifycoder: Clarification-awall fine-tuning for programmatic problem solving. arviv preprint ar Xix::22004.16 1000044..3333 16331, 2025.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101, 2025.
Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal 1lm logical reasoning benchmark in visual contexts. ar Xiv preprint 1xXX::2477.0493, 2024.
15

---

Published as a conference paper at COLM 2025
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances Neural Information Processing Systems, 36:41618-41650, 2023.
Navid Yaghmazadeh, Christian Klinger, Isil Dillig, and Swarat Chaudhuri. Synthesizing transformations on hierarchically structured data. ACM SIGPLAN Notices, 51(6):508-521, 2016.
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. Sqlizer: query synthesis from natural language. Proceedings of the ACM on Programmning Languages, ((OOPSLA): 1-26, 2017.
Kai Yan, Zhan Ling, Kang Liu, Yifan Yang, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. Mir-bench: Benchmarking Ilm's long long-context intelligence via many-shot in-context inductive reasoning. ar preprint ar axXio:2502.09933, 2025.
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. Natural language to code generation in interactive data science notebooks. ar preprint arXit:2212.09248, 2022.
Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. Algo: Synthesiz- ing algorithmic programs with generated oracle verifiers. Advances in Neural Information Processing Systems, 14554769-54784 2023.
Andy Zhou, Kai Yan, Michal Shlapentokh-Rothen Rothmma, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning language models. In Forty-first International Conference on Machine Learning, 2024a.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024b. URL https://arxiv. org/abs/2307, 13854.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. ar Xiv preprint wwXiv:240.11877, 2024.
16

---

Published as a conference paper at COLM 2025
A Appendix
A.1
Case Study
In the below example, the model first invokes the ground-truth function with 10 more inputs, then tries to synthesize the function, and then gets back the counter-example from the differential testing oracle. Its second trial then passes the oracle's test.
Prompt:
You are a software engineer tasked with implementing blackbox Python function solution. Your goal is to implement this function correctly, ensuring it works for all possible valid inputs, not just the example inputs provided.
Your primary goal is get the function implementation exactly right. Your secondary goal is to minimize the number of function invocations and implementation attempts.
GIVEN EXAMPLES:
print('Resull 1: sttrssolution([1, 2, 3, 4, 5])))
Result 2: *str(solution(11, 2, 2, 4, 5])))
print('Result 3: sstr(solution(['a,, 'b', 'c', 'd', 'e'])))
print('Resull, 4: sstr(solution(('apple;, 'banana', 'apple' })))
print('Result 5: str(solutiinn([)))
print("Result 6:
str(ssolution([1e, 20, 30, 40, 50, 601)))
print('Result
'Rssult 7:
str(solution([1e, 20, 30, 30, 50, 601)))
orint('Resul) 8:
str(solution(['x) 'y', 'z'))))
str(solution([1, 1, 1)))) print("Reste 'Result 10: strieoution(isstraag(((()))
print( (RReult 9:
Corresponding outputs:
Result 1: True
Result 2: False
Result 3: True
Result 4: False
Result 5: True
Result 6: True
Result 7: False
Result 8: True
Result 9: False
Result 10: True
Strategy Overview:
You should work in two phases:
1. Exploration Phase: use function invocations strategically to fully understand the function's 's behavior.
2. Implementation Phase: only after a thorough understanding, attempt an implementation.
This is the first iteration. You must first reason and then decide between one of the following two options:
1. Generate more function invocations in the same format as the examples. You can generate
up 20 additional function invocations across all iterations.
2. Generate an implementation of solution. You have only debugging check total. If your implementation is incorrect, you lose your check and are shown counterexample. If you attempt an implementation and have no debugging checks left and your implementation is incorrect, you cannot take any more actions. This attempt will be considered your final answer.
Assistant:
17

---

Published as a conference paper at COLM 2025
I'll start with the Exploration Phase and generate additional strategic invocations to better understand the function's behavior over a wide range of inputs. want to explore cases like unique negative and non-negative numbers, duplicate negative numbers, duplicate empty strings, duplicate boolean values, and unique special character strings. I also want to verify that single element lists should be unique.
INVOCATIONS:
print( Result 11:
sstrssllution[-1,, -2, 0, 1])))
print Ressult 12:
sstr(solution[[1,, -2, -1])))
print("Resul) (Result 13:
*str(solution(I), 1.0, 2])))
print('Result 14:
sstr(solution(((1)) (2.), (1,)1)))
print("Resuul (Result 15:
str(ssluution([42)))
print('Resul) 16:
str(solution(((" "a", "b"])))
print('Result 17:
str(solution(["" "" "b"])))
print('Resul) Result 18:
str(solutionn(TTue,, False, Falsel)))
print('Result 19:
str(solution(['! 'e', '#', ')')))
20:
str(solutionn(None, 8, "N))e"I)))
Prompt:
Here are the outputs of the function invocations from the previous iteration:
Result 11: True
Result 12: False
Result 13: False
Result 14: False
Result 15: True
Result 16: True
Result 17: False
Result 18: False
Result 19: True
Result 20: True
STRATEGY UPDATE: Based the new data above, you should update your understanding of the function and decide on your next action. You have 10 additional function invocations and debugging check left.
Assistant: Based on the input-output pairs I have analyzed, infer that this function is intended to evaluate whether all elements in a given list are distinct thact is, to determine there are any duplicate entries within the list. To implement this functionality, I propose using a pairwise comparison strategy. By systematically comparing each element the list with every subsequent element, it becomes possible to identify any duplicates that may exist. This approach ensures thorough coverage of all potential comparisons and provides reliable method for detecting non-unique elements in the list.
IMPLEMENTATION:
def s1stt
# Use a simple pairwise comparison approach n = len(lst)
for in range(n):
for in range(i 1, n):
lst[i] 1st[j]:
return False
return True
Prompt:
The implementation you generated in the previous iteration failed for the following input (and potentially other inputs):
Failed input: [1, 2, 3, 4, [5, 6], [5, 6]]
Ground Truth Function != Output From Generated Code: 'Erron" != 'No Error'
18

---

Published as a conference paper at COLM 2025
This failure provides valuable information about you didn'tt account for. You should decide between generating more invocations to explore and debug the function behavior or generating one final implementation. You have 10 additional function invocations and NO debugging checks left.
Assistant:
Based the error message, it appears that raising error for this input would be the appropriate response. To handle this effectively, a set-based approach might be more suitable than pairwise comparison approach. By converting the list to a set, I can efficiently detect unhashable elements, as attempting this operation would naturally raise TypeError for any such elements. Let me proceed with implementing this approach to validate the reasoning.
IMPLEMENTATION:
def solution(1st):
# Attempt to convert the list a set. # any element unhashable, this will raise TypeError as expected. return len()st) == (1n(s)t(lst))
This final implementation passes the differential testing oracle.
19

---

Published as a conference paper at COLM 2025
A.2 Supervised Fine-Tuning Experiment Setup
Table presents the parameters used for our supervised fine-tuning experiments
Name
Value
Teacher Model
gpt-4o
Student Model
-LaMA-311-B-Instrruee
Fine-tuning Context Length
32768
Learning Rate
1.00E-5
LR Schedule
linear
Warmup Ratio
0.03
Epoch
3
Batch Size
8
Gradient Accumulation Steps
8
LoRA Rank
8
LoRA
8
Table 8: LoRA Vine-Tuning Parameters.
A.3 Analysis of Differential Testing Tools
Program equivalence is undecidable, and no oracle can guarantee perfect correctness check- ing. To mitigate this, our evaluation combines two state-of-the-art differential testing tools, PYNGUIN (Lukasczyk & Fraser, 2022) and MOKAV (Etemadi et al., 2024). This dual-oracle setup reduces the likelihood of overlooking behavioral discrepancies between the synthe- sized and ground-truth functions.
Empirically, the two tools produce identical outcomes on 75.4% of problems. MOKAV detects additional bugs in 18.8% of cases missed by PYNCUIN, while the reverse holds in only 5.9%. Overall, using both tools yields a more reliable approximation of correctness in the absence of perfect oracle.
20


Reviews:
[{"review_id": "9YrbDUuVyH", "reviewer": "Reviewer_G4M6", "summary": "This paper proposes a new benchmark, CodeARC, for more general program by example tasks, with the target programs from existing coding benchmarks such as HumanEval, MBPP, and APPS.\nIt also introduces a new, agentic setup for inductive program synthesis, where the model is allowed to take actions, interact with a hidden oracle program, and decide when to submit its answer.\nSeveral models are evaluated on CodeARC and the authors also show that fine-tuning on synthetic data (with the target program visible to the teacher model) can improve LLMs' performance on it.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Comprehensive evaluation and very thorough ablation experiments with interesting findings. For example, I find the results on whether initial I/O pairs are underspecified particularly interesting.\n\n2. The agentic setup for evaluation is more realistic to how people may actually use LLMs for inductive program synthesis, and it opens doors to the study of scaling test-time compute for this task.", "reasons_to_reject": "1. The \"general-purpose\" claim. The authors claim on Line 48-51 that \"existing benchmarks ... are focused on domain-specific tasks and do not assess the ability of LLMs to synthesize functions written in general-purpose programming languages\". I don't think it's true.\nFirst, the tasks in CodeARC are not exactly general-purpose, they are mostly like interview questions and competition problems whose inputs and outputs are easily representable with strings. The tasks do not seem much more general-purpose than list operations to me.\nSecond, in Li and Ellis, 2025, a paper cited by the authors, LLMs are also tasked with solving PBE with a general-purpose programming language.\n\n2. Lack of analysis on how fine-tuning affects models' ability on other coding tasks. Fine-tuning on synthetic data is great, but if it hurts other performance it wouldn't be worth it.\n\n\n**Reference**\n\nWen-Ding Li and Kevin Ellis. Is programming by example solved by llms? Advances in Neural Information Processing Systems, 37:44761–44790, 2025", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCsjRsqf6Y", "reviewer": "Reviewer_3DHT", "summary": "This paper proposes a new evaluation framework for inductive program synthesis to evaluate the capability of LLMs in reasoning the underlying program from input-output examples. It provides an interactive environment where the LLM can request limited times for extra input-output examples as feedback for revision. This mitigates the ambiguity in inductive program synthesis that there may be multiple acceptable programs based on the given input-output pairs, and can evaluate LLMs' capability in function calls and self-correction based on feedback. The general-purpose inductive program synthesis benchmark, CodeARC, with 1114 functions, provides a realistic and challenging testbed.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. CodeARC provides an interactive test environment for inductive program synthesis, where the LLMs can request extra feedback by function calls. Unlike prior work, this can mitigate the problem of multiple valid functions satisfying the same input-output examples by providing additional feedback. It also evaluates the capability of the function call and revision based on feedback\n2. It benchmarks 18 models on CodeARC and investigates the impact of budgets on input-output examples and function calls. The analysis on cases that pass the initial examples but fail under oracle testing indicates under-specification, which further motivates CodeARC. \n3. It investigates how finetuning on the synthetic data affects the model performance on this benchmark", "reasons_to_reject": "1. There is no ablation study on the input-output query and the oracle invocation. For example, if the IO budget is set to 0, will the oracle invocation benefit the performance? Similarly, if the oracle invocation budget is 0, how will the IO affect the performance? It is important to show that both of these mechanisms are important in CodeARC.\n2. From Table 5, the improvement of the Success Rate on the Anonymized version is smaller than the Annotated version. This indicates that the finetuning on interactive reasoning traces on CodeARC cannot provide more benefits than a function name. \n\nMissing Reference:   \n[1] Case2code: Learning inductive reasoning with synthetic data", "questions_to_authors": "Section 5.3 shows that the increase of input-output query budget and the oracle invocation budget will consistently improve the performance. Is there an upper bound on the performance when it can no longer increase with more budget?\nAmong the query budget and the oracle invocation budget, which one is more important?", "extra_scores": {}}, {"review_id": "LD19pkHR6j", "reviewer": "Reviewer_4Yd9", "summary": "This paper proposes a new benchmark for measuring programming-by-example and inductive reasoning abilities in LLMs. The model is used as an agent that is initially provided with 10 I/O pairs for a function, and allowed to submit more inputs to obtain corresponding outputs, or submit its guess at the code for testing against an oracle, which can return an input for which the ground truth output doesn't match. The authors collect functions from the HumanEval+, MBPP+, and APPS datasets. The authors evaluate against 18 models and report the results.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a very interesting benchmark which meaningfully extends earlier ones focused purely on generating code in one shot from natural language descriptions and a few input/output examples. \n\nThe paper contains an extensive empirical study across a large number of models, and considers variations in the evaluation such as by varying the number of attempts that the model is given. The results show that the benchmark is far from saturated, and that expected trends are present (such as increasing performance with larger number of parameters, and a larger budget for oracle invocations).", "reasons_to_reject": "Only one prompt/agent harness is tested in the paper, which may fail to elicit the maximal capabilities from the model. In particular, the model is expected to internally simulate the code that it generates, since it is not afforded with access to a Python interpreter. A better-equipped model, or one with different instructions, may exhibit significantly different performance than what the paper shows.\n\nThe authors specifically target more general-purpose tasks as opposed to ones used in prior work for programming-by-example; it would also have been good to see results on the classical programming-by-example tasks, since their usefulness is more thoroughly motivated by the body of prior work.\n\nThe use of up to 3 oracle invocations, as well as a budget of 30 input-output pairs, seems somewhat arbitrary. The paper motivates these choices as 'chosen based on practical constraints such as API cost and runtime\". The difficulty of the dataset is highly contingent upon these parameters, and claims made in the paper may fail to hold up when they are increased. It's generally unclear whether we should expect all problems to be solvable given the budgets used, i.e. whether the programs are of sufficiently low complexity that it's possible to copy them exactly under these constraints.\n\nThere is currently no mention in the paper that the dataset will be released; the paper by itself, without the accompanying dataset/code, is not nearly as valuable.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q6170jt0Pg", "reviewer": "Reviewer_w6n7", "summary": "This paper introduces CodeARC, a newly proposed benchmark aimed at evaluating the inductive reasoning abilities of large language models (LLMs) through the task of program synthesis from input-output examples. Unlike other current benchmarks based on static assessment, CodeARC introduces an interactive environment where models can ask a concealed target function and use a differential testing oracle to receive counterexamples. This supports iterative improvement of the produced code.\nThe benchmark consists of 1114 general-purpose Python functions, collected from existing datasets such as HumanEval, MBPP, and APPS. It includes both annotated and anonymized versions to test the model's reasoning capabilities, independent of function name cues. There are 18 LLMs tested, and the top-scoring model (o3-mini) scores just over 50% success, thus proving the difficulty of the task. The authors further demonstrate that fine-tuning using curated reasoning traces leads to a significant performance improvement, particularly for tasks involving meaningful identifiers.\nOverall, CodeARC provides a useful and timely contribution to the state of the art, allowing better simulation of the extent to which LLMs can generalize from examples and learn from feedback, both essential abilities for the construction of intelligent autonomous agents.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Novel Benchmark Design: The paper presents an interactive and realistic framework for inductive program synthesis that is ahead of static evaluation, addressing a major gap in existing LLM benchmarking.\nFocus on Reasoning, Not Memorization: By using only input-output examples and using anonymized function names, CodeARC successfully isolates and tests genuine inductive reasoning instead of pattern-matching or retrieval.\nThorough Evaluation: The authors perform an extensive analysis of 18 models, including ablations on budget constraints and fine-tuning experiments, providing valuable insights into model behavior and limitations.\nPractical Relevance: The reverse engineering–style configuration (querying a black-box function) reflects actual application scenarios like decompilation and debugging, making the benchmark extremely practical.", "reasons_to_reject": "Limited Task Diversity: Although the benchmark covers 1,114 Python functions, many are drawn from existing datasets and mostly involve algorithmic problems. Adding more realistic, noisy, or domain-specific tasks would make the benchmark stronger and more generalizable.\nUpper Bound on Difficulty: Even with the difficult setup, there are a lot of problems from HumanEval & MBPP  that still depend on fairly straightforward reasoning. The benchmark might be further reinforced by adding more multi-step or higher-abstraction synthesis problems that more effectively probe the limits of existing LLMs.", "questions_to_authors": "Task Diversity: Do you plan to expand the benchmark with more complex or multi-function programs that go beyond single-function synthesis? This might better evaluate multi-step reasoning or compositional abilities.\nOracle Quality: To what extent is the benchmark result sensitive to the quality or conduct of the differential testing tools (PYNGUIN, MOKAV)? Might some bugs or mismatches be overlooked or incorrectly reported?\nFine-Tuning Constraints: Did you observe any overfitting or performance degradation when fine-tuning on reasoning traces? How generalizable is the student model to unseen function structures?\nReinforcement Learning Potential: You mention RL as a future direction; do you have initial ideas for what a reward signal could be like in this synthesis context, or how you'd design such a system?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an interesting twist to standard code benchmarks. Instead of being provided with the description and function signature, the model gets exposed only to input/output pairs and has to learn the underlying function. It does have the possibility to request feedback: either the output of a chosen input, or a counter-example to a proposed program

The authors benchmark 18 LLMs, showing poor performance (~50% max) on this task - even when the original prompts/completions were taken from benchmarks that are "solved" (performance >90%), and show the impact of finetuning. During the rebuttal phase the authors extended this to another benchmark that focuses more on software engineering and less on competitive programming

This is an excellent paper. The only point which is missing to me is the link to learnability theory: learning from positive examples and counter-examples has a rich history. The standard reference for that is Dana Angluin's "Learning Regular Sets from Queries and Counterexamples" from 1987, and several textbooks provide good context (eg, Colin de la Higuera's "Grammatical Inference: Learning Automata and Grammars")
Decision: Accept