SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifiers
Kexun Zhang $^{1}$  Danqing Wang $^{1}$  Jingtao Xia $^{1}$  William Yang Wang $^{1}$  Lei Li $^{2}$
<sup>1</sup>University of California Santa Barbara  <sup>2</sup>Carnegie Mellon University
{kexun,danqingwang,jingtaoxia,william}@ucsb.edu leili@cs.cmu.edu
Large language models (LLMs) excel at implementing code from functionality descriptions but struggle with algorithmic problems that require not only implementation but also identification of the suitable algorithm. Moreover, LLM-generated programs lack guaranteed correctness and require human verification. To address these challenges, we propose ALGO, a framework that synthesizes Algorithmic programs with LLM-Generated Oracles to guide the generation and verify their correctness. ALGO first generates a reference oracle by prompting an LLM to exhaustively enumerate all the combinations of relevant variables. This oracle is then utilized to guide an arbitrary search strategy in exploring the algorithm space and to verify the synthesized algorithms. Our study shows that the LLM-generated oracles are correct for  $88\%$  of the cases. With the oracles as verifiers, ALGO can be integrated with any existing code generation model in a model-agnostic manner to enhance its performance. Experiments show that when equipped with ALGO, we achieve an  $8\times$  better one-submission pass rate over the Codex model and a  $2.6\times$  better one-submission pass rate over CodeT, the current state-of-the-art model on CodeContests. We can also get  $1.3\times$  better pass rate over the ChatGPT Code Interpreter on unseen problems. The problem set we used for testing, the prompts we used, the verifier and solution programs, and the test cases generated by ALGO are available at https://github.com/zkx06111/ALGO.
Large Language Models (LLMs) have demonstrated significant prowess in generating code from natural language descriptions. Models such as Codex [5] and CodeGen [23] can easily achieve over  $30\%$  pass@1 accuracy on HumanEval, a docstring-to-code dataset. However, these models struggle when faced with algorithmic problems akin to those encountered in CodeContests [17]. Even with reasonable sample limits, achieving a  $10\%$  accuracy rate remains a considerable challenge [4]. More recently, the GPT-4 model, despite being provided with problem descriptions and solution hints, managed a pass@1 accuracy of only  $10.6\%$  [3] on LeetCode Hard problems.
The verifiability of LLM-based code generation presents another significant challenge. Without verifiability, code generation systems can hardly earn trust for production use. Users of GitHub Copilot [21], an AI assistant for programmers, reportedly spend over  $20\%$  of their time verifying the suggested code snippets. Existing approaches toward verification and reranking LLM-generated programs either rely on neural models to predict confidence scores [22, 32] or require LLM-generated test cases to execute the programs [4, 15]. However, neural verifiers do not provide interpretable feedbacks and LLM-generated test cases are often incorrect.
Traditional code synthesis techniques rely on oracles for the verification of the synthesized code. These oracles are typically derived from formal specifications [20, 25, 28, 29]. They can also be employed in program synthesis under the widely-used framework of oracle-guided inductive synthesis (OGIS) [14]. Despite their widespread application in code verification and synthesis, the
Figure 1: The ALGO pipeline. The verifier LLM generates the probably correct but possibly slow reference oracle that solves the problem with an exhaustive search. The coder generates a more efficient candidate program and refines the candidate program by comparing its output with the oracle's output. The coder can be any existing code generation model.
Figure 2: The reference oracle generated by ALGO is correct for  $88.5\%$  of the problems in our benchmark. We consider an oracle correct if it gets Accepted / Time Limit Exceeded verdict on LeetCode website and is checked by human experts as a reasonable solution.
use of oracles often demands heavy human involvement, and the automatic generation of oracles is challenging. Existing strategies focus on generating oracles that can only detect crashes (safety oracle) or bugs introduced by future changes (regression oracle), rather than semantic bugs crucial for algorithmic programs [9, 18, 24].
Motivated by the potential benefits of oracles in traditional code synthesis, we introduce ALGO, a framework that leverages oracles generated by large language models to address the challenges in LLM-based code synthesis and verification. As illustrated in Figure 1, ALGO incorporates two modules for algorithm synthesis. The verifier is instructed to generate an exhaustive search algorithm regardless of time efficiency, thus acting as a reference oracle. Concurrently, the coder is asked to find a more efficient solution with any prompts or search strategies. The candidate program's correctness is then evaluated by comparing its output with that of the oracle for a given set of test inputs. The results of the verification, along with any test cases where the candidate program falls short, are subsequently provided to the coder for code refinement.
We evaluated ALGO based on two key dimensions: the oracle's verification capability and its potential to enhance algorithm synthesis. Our experiments reveal that the reference oracles are correct for  $88.5\%$  of the problems (as shown in Figure 2). Moreover, the oracles' verdicts are in agreement with the golden verdicts of the online judge  $75\%$  of the time. To examine the enhancement in accuracy offered by ALGO, we integrated it with several existing code generation models including Codex [5], CodeT [4], PG-TD [31] and ChatGPT Code Interpreter<sup>1</sup>. We observed that ALGO significantly boosts their performance on CodeContests [17] and a collection of recent LeetCode problems. Our experiments show that ALGO notably enhances the performance of these models: in terms of one-submission pass rate, Codex's performance improved by a factor of 8, CodeT's by 2.6, PG-TD's by 1.5, and ChatGPT Code Interpreter's by 1.3.
- We present ALGO, a novel framework for Algorithm synthesis that utilizes LLM-Generated reference Oracles as verifiers. It is a model-agnostic framework that can be integrated with any code generation model to verify candidate solutions with reliable test cases generated by reference oracles.
- We conduct a comprehensive evaluation of synthesis accuracy and verifiability of ALGO, utilizing several distinct code generation models in a versatile, model-agnostic manner. Our results indicate that ALGO can generate high-quality oracles and test cases that lead to significant improvements in code generation.
<sup>1</sup>ChatGPT Code Interpreter is a variant of GPT-3.5 with the code interpreter plugin. The version we evaluated, released on March 24th, utilizes the underlying model text-davinci-002-code. It is now offline and replaced with gpt-4-code-interpreter.
Figure 3: Examples of algorithm synthesis (top) and functionality synthesis (bottom). The parts in solid boxes are given by the problem, while the parts in dotted boxes should be inferred by the model. Functionality synthesis does not require the model to infer the idea, while algorithm synthesis does.
2 Algorithm Synthesis
Large language models excel at generating simple programs. For example, ChatGPT can easily achieve over  $70\%$  pass@1 accuracy on the docstring-to-code dataset HumanEval. However, it still struggles with complex algorithmic problems such as CodeContests. We hypothesize that this is because algorithmic problems need efficient solutions rather than straightforward and brute-force ones. To test our hypothesis, we prompt ChatGPT in two different ways to see if they result in different success rates. One prompt directly describes the problem and asks the model to generate a solution that meets the time limits. The other asks the model to ignore the time limit and to solve the problem in the most straightforward way. The prompts we used are listed in Appendix A.2. We evaluate the correctness of the brute-forces and efficient solutions both manually and using official test cases without time limits so that a correct yet inefficient program is also considered a success.
Table 1: The success rates when ChatGPT is prompted to generate efficient solutions and brute-force solutions and their relative differences. Generating brute-force solutions for LeetCode and CodeContests is much easier than generating efficient solutions.
LeetCode	CodeContests	HumanEval
Brute-force success rate	88.5%	72.0%	70.1%
Efficient solution success rate	41.2%	7.90%	72.5%
Relative difference	114%	822%	-3%
The results in Table 1 validate our hypothesis. For LeetCode and CodeContests, it is much easier to generate brute-forces than efficient solutions. But for HumanEval, the two success rates are similar. This clearly indicates that problems in HumanEval and those in LeetCode and CodeContests are of two different types. The latter is more difficult to solve because it requires the application of algorithms. Based on that, we divide code synthesis into two categories: functionality synthesis and algorithm synthesis. Typically, functionality synthesis problems, such as those seen in HumanEval [5] and MBPP [1], provide a detailed description that can be readily translated into an implementation. On the other hand, algorithm synthesis problems found in APPS [11] and CodeContests [17] are more abstract and need efficient algorithms to solve. In this paper, we focus on addressing algorithm synthesis rather than functionality synthesis. An algorithmic synthesizer, thus, has an additional task to that of a functionality synthesizer—it needs to generate the solution idea, implicitly or explicitly, before synthesizing the code. We further clarify the contrast between the two by providing typical examples of each in Figure 3.
Formal definition of algorithm synthesis and functionality synthesis. A code generation task can be defined as a tuple  $(Q, J) \in \mathcal{Q} \times \mathcal{J}$ , where  $Q$  is the problem description and  $J: \mathcal{P} \to \mathsf{Bool}$  is the system judge. The system judge  $J$  evaluates if a program from the program space  $\mathcal{P}$  solves the problem represented by  $Q$ . The goal of the synthesizer is to produce a program  $P \in \mathcal{P}$  that  $J(P) = \text{True}$ . The system judge  $J$  is further composed of two components:  $J_S: \mathcal{P} \to \mathsf{Bool}$  and
Reference Oracle Generation Prompt
Please solve this problem with a brute-force algorithm. After generating the code, please test it against the example cases given only, do not generate your own test cases. When comparing outputs, please do not consider the line breaks and spaces.
You are given an integer array ranks representing the ranks of some mechanics. ranks[I] is the rank of the ith mechanic. A mechanic with a rank r can repair n cars in r * n^2 minutes.
You are also given an integer cars representing the total number of cars waiting in the garage to be repaired.
Return the minimum time taken to repair all the cars.
I{examples, omitted}
Instead of defining your function in the Solution class, change your classname to BruteforceSolution. As I said before, please make sure you use the most straightforward and brute-force algorithm to solve the algorithm. Do not consider any efficiency issue, make the solution as brute-force as possible. It's okay for you to enumerate over a very large search space as long as the solution is correct. What do you think are some variables that may affect the answer and how do you think they can be enumerated?
Figure 4: The prompt we used for oracle generation and one oracle generated with it. The instructions are in blue. The correct solution solves the problem with binary search in polynomial time, while the generated oracle takes exponential time to enumerate all possible work allocations.
$J_{T}:\mathcal{P}\to \mathbf{Boo1}$ .  $J_{S}$  checks the semantic correctness of the generated code, while  $J_{T}$  ensures that the efficiency of the generated code satisfies the requisite conditions. Intuitively,  $J(P) = J_{T}(P)\wedge J_{S}(P)$ .
3 Algorithmic Synthesis with LLM-Generated Oracles (ALGO)
As discussed in Section 2, generating brute-force solutions to algorithm synthesis tasks is much easier than generating efficient solutions; this huge gap in difficulty can be exploited. To solve algorithm synthesis, we propose Algorithmic Synthesis with LLM-Generated Oracles (ALGO). As shown in Figure 1, the ALGO framework utilizes two components - a coder and a verifier - to solve algorithmic problems. The coder takes the problem description  $Q$  and optionally, the verification results from its last generation and generates a program  $P$  that solves  $Q$ . The verifier generates a reference oracle whose outputs are used to verify whether the candidate program generated by the coder is correct. For each problem, ALGO creates the verifier once and uses it to guide arbitrary coders, allowing ALGO to be model-agnostic.
3.1 Verification with Oracle
Oracle Generation The difference between a reference oracle  $P_O$  and an actual solution  $P_G$  is that  $P_O$  only needs to be semantically correct (i.e.  $J_S(P_O) = \text{True}$ ) while  $P_G$  needs to be correct and efficient (i.e.  $J(P_G) = J_S(P_G) \land J_T(P_G) = \text{True}$ ). We utilize this difference to generate the oracle. When there is no time limit, most algorithmic problems can be solved with an exhaustive search [8]. As depicted in Figure 4, we put the process of an exhaustive search algorithm (which is the same for every problem) in the prompt along with the problem description. LLMs, which excel at implementing a program when provided with clear instructions, are then able to generate reference oracles.
The Verification Process To handle a verification request, we must return a verdict of True/False regarding  $P$ 's correctness, and optionally, the inputs that cause  $P$  to fail. We utilize an input generator program to create random test inputs in line with problem constraints. These test inputs are then supplied to both the program under verification  $P$ , and the oracle  $P_O$ . The test outputs of  $P$  are compared against those of  $P_O$ . If they match, a True verdict is returned. If they don't, the failed test cases are returned to the synthesizer along with a False verdict.
3.2 Code Synthesis Strategies
ALGO's coder can use different strategies for code synthesis. The code synthesis process can either be one-time or iterative, depending on the used coder. During a one-time synthesis, the coder generates a batch of solutions to the problem, and these solutions are verified and ranked according to their verification results. During an iterative synthesis, the code generates solutions, gets their verification results (and optionally mistaken test cases), and then revises the solutions iteratively.
Other than the coder model itself, its strategy to search for and identify the suitable algorithm for the problem also affects the synthesis accuracy significantly. The coder's search strategy can be defined as a conditional distribution  $\pi(P|Q)$  of the program  $P$  conditioned on the problem description  $Q$ . To generate code is to sample programs from this distribution. To compute this distribution, we can define a space of latent variables  $\mathcal{I}$  and marginalize over it, as denoted by
$$
\pi (P | Q) = \sum_ {I \in \mathcal {I}} \pi (P | I) \pi (I | Q).
$$
ALGO is a versatile framework that can incorporate a variety of search strategies. A strategy may be implicit, with the variable  $I$  not specified, or explicit, specifying  $\mathcal{I}$ , the space of  $I$ , and the methodology to traverse this space. Here, we give three examples: an implicit searcher, an instruction enumerator, and an iterative searcher. However, it is crucial to note that these examples are representative, and there are numerous alternative search strategies in ALGO.
Implicit Searcher: An implicit searcher samples a program directly from the distribution determined by the underlying model, using the problem description. It solely relies on the ability of the code generation model and depends on the oracle to filter the generated programs post hoc.
Instruction Enumerator: An instruction enumerator is an explicit searcher where  $\mathcal{I}$  is a set of pre-defined instructions. Since the performance of instruction-tuned LLMs varies significantly based on the instructions they are given, it is natural to consider enumerating a space of instructions to get the best result. An instruction enumerator first picks an instruction from a pre-defined instruction set  $\mathcal{I}$ , and then synthesizes the program by instructing the language model with  $I$ . In our example of an instruction enumerator, we choose  $\mathcal{I}$  to be a set of high-level ideas of algorithms such as 'Binary Search' and 'Sorting'. We first sample possible solution algorithms and then instruct the large language models to generate programs with the ideas given.
Iterative Searcher: A iterative searcher is an explicit searcher that takes the signal from the verifier to refine its output. The searcher space  $\mathcal{I}$  can be the token vocabulary of the model. It uses a search algorithm to determine the next token during the decoding process by enumerating the vocabulary tokens given the problem description and the partial program in each generation step. Since the search space is exponentially large, it would be time-intensive. Usually, the searcher is guided by some rewards from the verifier to prune the candidate space in each step.
In this section, we implement ALGO with three code synthesis strategies, and evaluate it with two challenging algorithmic benchmarks to validate its flexibility and effectiveness. Moreover, we investigate the verifier's performance by evaluating the quality of the generated reference oracle and test cases.
**Verification** We employ ChatGPT Code Interpreter to create the verifier. It is first prompted to generate the reference oracle and then the input generator. As mentioned in Section 3.1, this is possible because the reference solutions to most algorithmic problems involve exhaustively going through all possible solutions to find a feasible or optimal one. We use a temperature of 1.0 and resample the solution until it can pass all example cases. The prompt we used for generating the oracle is shown in Figure 4. For the verifier, we simply use zero-shot prompts to generate the input generator with an input validator since both of them are functionality syntheses and easy to generate. We set the maximum length of each input variable to 10 and skip out cases that raise the timeout exception and the recursion error when generating the output via the reference oracle. The prompts we used are listed in Appendix A.2 with generation examples. Note that for each problem, we only need to create the verifier once, and it can be used to create arbitrary test cases and guide arbitrary models.
Code Synthesis Strategies We integrate the following baselines as the coder with ALGO to evaluate its verification capability and synthesis accuracy:
- Codex [5] and CodeT [4]: Codex and CodeT are used as coders that utilize implicit searchers. Codex is a specialized language model, trained on publicly available code from GitHub. CodeT uses Codex to generate both programs and test cases for a problem and uses their dual agreement to
filter and rank the programs generated. To evaluate ALGO, we use the exact same set of programs Codex generated, and rank them in three different ways: in a random manner, based on the dual agreement heuristics from CodeT, and according to the verification results provided by ALGO. We then compare the top-ranked programs from each of these three rankings. Note that due to OpenAI's policy, the Codex model is no longer available so we directly use the Codex-generated programs provided by the CodeT paper<sup>2</sup>.
- ChatGPT Code Interpreter: ChatGPT Code Interpreter is used as a coder that utilizes instruction enumerators. It is a variant of GPT-3.5 with the code interpreter plugin. The version we evaluated, released on March 24th, utilizes the underlying model text-davinci-002-code. As the Code Interpreter is instruction-tuned, it can follow directives to solve a problem using a specific class of algorithms, such as 'Binary Search'. This capability allows it to employ a search strategy that iterates through a list of algorithm categories to identify a correct solution. Moreover, with the code interpreter plugin, it is able to run Python code and interpret its results, and automatically refine its output based on the program output. We use the default temperature of 1.0 and resample the code solution until it can pass the example cases in the problem descriptions or reach the resample limit  $(N = 5)$ .
- PG-TD [31]: PG-TD is used as a coder that utilizes iterative searchers. In contrast to CodeT, which employs test cases post hoc, PG-TD incorporates them during the decoding process. It implements a tree search-based planning algorithm for decoding, utilizing the reward from test cases to estimate the value of the children nodes in the search tree. PG-TD uses GPT-2 [26] and GPT-NEO [2] as the underlying model. We implement two versions: PG-TD* is only based on the public test cases to generate programs, and PG-TD has access to both public test cases and the generated test cases by Li et al. [17]. Both are based on the GPT-2 fine-tuned on the APPS training dataset, which was released by Zhang et al. [31]. We integrate PG-TD into ALGO by substituting the generated test cases with the reward from the verifier.
Benchmarks We utilize the following benchmarks to evaluate ALGO:
- CodeContests [17]: We evaluate ALGO with Codex, CodeT and PG-TM on all 165 problems from CodeContests test set, which are collected from Codeforces, a competitive programming website. All the three baselines we evaluate have been evaluated on CodeContests in their own paper, we follow the exact same setting to ensure a fair comparison.
- LeetCode: We evaluate ALGO with ChatGPT Code Interpreter on 35 recently released LeetCode problems. To avoid the influence of contamination and faithfully evaluate ALGO's ability in verification and guiding code synthesis, we made sure these problems are released concurrently or after the release of GPT-4. We manually annotate the algorithm categories for the solutions to these problems to evaluate the search strategy of enumerating algorithm categories. The list of problems and their corresponding categories is listed in Appendix A.3.
Metric Following previous studies [31, 4], we use the  $n@k$  metric [16, 5, 17] to evaluate the accuracy of code synthesis. To compute  $n@k$  for a problem, a code synthesizer needs to generate  $k$  candidate programs and select the top-  $n$  candidates to submit according to some reranking mechanism, which in our case is the LLM-generated oracles and test samples.  $n@k$  is the proportion of problems that can be solved by any of the top-  $n$  submissions. Since some code synthesizers do not have reranking mechanisms, we define their  $n@k$  as the proportion of problems solved with  $n$  submissions randomly sampled from  $k$  candidates. To ensure a fair comparison, we use the same sampling budget  $n$  under the same setting when evaluating the performance of ALGO and baselines.
4.2 Synthesis Accuracy