SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CodeContests We present ALGO's  $n$  @  $k$  accuracy for CodeContests in Table 2. In the first part of the table, Codex is used as an implicit searcher. Compared to Codex itself and CodeT which also heuristically evaluates the program Codex generates, ALGO consistently outperforms them at every value of  $n$ . The most significant performance enhancement by our method is observed at  $1$  @  $k$ , yielding an  $8 \times$  improvement over Codex, and a  $2.6 \times$  improvement over CodeT. As the value of  $k$  increases, the performance advantage diminishes. This trend is expected, as the chosen sample number  $n$  approaches the total sample count  $k$ , causing the pass rate to converge towards the ratio of correct programs within the total program set. The same phenomenon is also observed in the second
$^{2}$ https://github.com/microsoft/CodeT/
Table 2: Performance on CodeContests. Gain is the performance improvement compared with the baseline (the first line in each part). ALGO significantly outperforms other baselines, especially for  $1@k$ . Note that  $k = 20$  for ChatGPT-based settings, while  $k = 1000$  for other settings.
Method	1@k	Gain	2@k	Gain	10@k	Gain	100@k	Gain
Codex-Based One-Time Synthesis (k = 1000)
Codex [5]	0.70	-	1.20	-	3.00	-	7.50	-
CodeT [4]	2.10	+1.40	2.30	+1.10	5.30	+2.30	9.90	+2.40
ALGO w/ Codex	5.60	+4.90	5.60	+4.40	7.70	+4.70	11.10	+3.60
GPT-2-Based Iterative Synthesis (k = 20)
PG-TD*	0.62	-	1.07	-	2.26	-	3.45	-
PG-TD [31]	0.67	+0.05	1.12	+0.05	2.53	+0.27	3.81	+0.36
ALGO w/ PG-TD	1.57	+0.95	2.44	+1.37	3.67	+1.41	4.06	+0.61
ChatGPT-Based One-Time Synthesis (k = 20)
ChatGPT	4.40	-	6.62	-	12.21	-	-	-
ALGO w/ ChatGPT	12.00	+7.60	12.00	5.38	14.00	+1.79	-	-
and third part of Table 2, where PG-TD and ChatGPT are used as coders. ALGO is able to achieve a  $2.5 \times$  improvement over PG-TD and a  $2.8 \times$  improvement over ChatGPT at  $1@k$ .
LeetCode We list ALGO's one-submission pass rate on LeetCode problems in Figure 5. The instruction set we enumerate for each problem is a set with several algorithm categories, among which there is one correct category. ALGO's superior performance compared to ChatGPT and GPT-4 demonstrates the benefits of an instruction searcher that enumerates possible algorithms. It also demonstrates ALGO's ability to filter and select the correct algorithm category. Without the verifier, we would not have been able to tell right from wrong.
Figure 5: Performance on LeetCode. Code interpreter integrated with ALGO can surpass GPT-4's pass rate.
4.3 Verification Analysis
The verifier is critical to ALGO as it guides the creation of an efficient and accurate solution. In this section, we scrutinize the quality of the verifier from several perspectives.
LLM-generated oracles are usually correct. The correctness of LLM-generated oracles is crucial to the correctness of ALGO's test cases. We examine their correctness with both the system judge and human experts. We submitted LLM-generated oracles to the online judges (Codeforces and LeetCode) where the problems came from, to get the system verdicts. Oracles with verdicts other than Accepted (AC), Time Limit Exceeded (TLE), and Runtime Error (RE) were directly considered incorrect. For those with TLE and RE verdicts that could potentially be semantically correct despite exceeding the time limit or stack size, we hired an experienced competitive programming contestant to examine if they were semantically correct and not efficient enough. For LeetCode problems, as displayed in Figure 2,  $88.5\%$  of the oracles are semantically correct, including all those with TLE verdicts. For codecontests problems,  $72\%$  of the oracles are semantically correct.
Oracle-generated test cases have better quality. We evaluate the quality of ALGO's test cases generated by answering the following questions: (i) Do verification verdicts from generated cases agree more with the system judge than verdicts from only the public tests? (ii) Do the generated tests achieve higher statement coverage compared with public tests? In this context, we calculate two metrics: Agreement and Coverage. Agreement is the consistency between the decision of the system judge and the test case set. For instance, candidates that fail the system judge should also fail on the generated test set, while those that can pass the judge should also succeed on all test cases. Coverage is the percentage of statements in a code solution that are executed by test cases. Following
Figure 6: The performance of different sizes of test cases on CodeContests.
Table 3: The quality of test case. Compared to the example cases given in the problem description, the test cases generated by ALGO's verifier detect more failures in the programs, agree better with the system judge  $J$ , and covers more lines in the generated code.
Dataset	Case	Agreement	Coverage
CodeContests	Example	57.39%	85.78%
ALGO	94.78%	89.39%
LeetCode	Example	68.57%	90.32%
ALGO	74.29%	93.00%
Chen et al. [4], we use a public coverage library<sup>3</sup> to gather the coverage results. As demonstrated in Table 3, our generated test cases substantially improve the agreement with the standard judge (+37.39% on CodeContests and +5.72% on LeetCode), indicating they can enhance the verification of candidates prior to submission. In addition, test cases of ALGO achieve higher coverage of the statements, implying that these cases can more effectively validate the functionality of the solution.
Better test cases lead to better results. The test cases generated by ALGO have better agreement and coverage. We check if they lead to better synthesis by comparing them to the other two sets of test cases - examples provided in the problem statement and test cases directly generated by ChatGPT. We used the same coder with the exact same sample budget  $k = 20$ . As reported in Table 4, under the same setting, ALGO tests led to much better performance.
Table 4: Given the same coder, ALGO test cases lead to much better  $n@k$  performance, indicating that better test cases do help in code synthesis.
1@20	3@20	7@20
Example Tests	4.4%	7.9%	12.2%
ChatGPT Tests	6.8%	8.2%	11.7%
ALGO Tests	12.0%	12.0%	14.0%
More test cases lead to better results. We examine the impact of the size of test cases the verifier generates on CodeContests for ALGO w/ Codex. Our verifier has the flexibility to create an arbitrary number of cases and use test case sets of varying sizes to rank and guide the synthesis. As shown in Figure 6, when the size is fewer than 30, the increase in the size of test cases can significantly bolster performance for all  $n$  at  $k$ , indicating the effectiveness of our verifier in covering more corner cases with additional test cases. Notably, having a sufficient number of test cases is more critical for  $1$  at  $k$ , which relies on test cases to select the optimal solution for one-time submission. However, this improvement diminishes when  $n$  is larger. This is because a robust verifier is not strongly required when the chosen sample number  $n$  is large, as discussed in Section 4.2. Furthermore, there is very little additional performance gain with more than 30 test cases. We hypothesize that this is due to the quality of our oracle and the generation range we have set for the verifier.
4.4 Case Study
We demonstrate the workflow of ALGO with a problem from our LeetCode benchmark  $^4$ . This problem asks for the optimal allocation of cars to mechanics with varying efficiency to minimize the total repair time, when the mechanics would work simultaneously. As depicted in Figure 7, the code generated using a reference oracle prompt exhaustively enumerates all potential assignments of cars and selects the optimal one according to the problem description. Assuming  $n = \mathrm{len}(\mathrm{ranks}), m = \mathrm{cars}$ , this program exhibits exponential time complexity  $T_O(n, m) = \mathcal{O}(2^m)$ , making it an inefficient solution that is unable to pass the online judge. Nevertheless, despite this program's inefficient nature, this implementation remains valuable in terms of semantic correctness (i.e.,  $J_T(P_O) = \text{False}$  and  $J_S(P_O) = \text{True}$ , where we can conveniently define  $J_T(P) = \text{"time complexity of } P$  is at most  $\mathcal{O}((n + m)\log (n + m))$  without loss). This example demonstrates how ALGO can generate reference oracles for code synthesis.
3https://coverage.readthedocs.io/en/6.4.2/
4https://LeetCode.com/problems/minimum-time-to-repair-cars/
Figure 7: ALGO is capable of verifying two candidate programs, in different algorithm categories: greedy and binary search. Using an input-output example provided by the reference oracle, ALGO is able to deliver the verification result consistent with the system judgment.
In Figure 7, we present how the reference oracle can verify the correctness of candidate programs generated by the coder LLM. We present two candidate programs generated with different algorithm categories: greedy and binary search. Specifically, the binary search candidate  $P_B$  is generated with instructions on binary search and the greedy candidate  $P_G$  is generated without explicit guidance. The binary search algorithm is ideally suited for this problem, given that the indicator function, denoted as  $f(t)$ , which determines whether all cars can be repaired within  $t$  minutes, is monotonous. This makes it possible to apply binary search on the time  $t$  and turn the optimization problem into a judgement problem. The binary search candidate represents an implementation of such algorithmic approach, which is an efficient and correct solution of time complexity  $T_B(n,m) = \mathcal{O}(n\log m)$  and can pass the system judge (i.e.,  $J(P_B) = \text{True}$ ). On the other hand, greedy algorithm is not suitable for this problem. Thus, even though the greedy candidate presented is of time complexity  $T_G(n,m) = \mathcal{O}(n)$ , the semantic of this candidate is wrong (i.e.,  $J_T(P_G) = \text{True}$  and  $J_S(P_G) = \text{False}$ ). This example demonstrates that providing the correct instruction can lead to better code synthesis result.
In Figure 7,  $P_G$  failed with our verifier since  $P_G(I_0) \neq P_O(I_0)$ . Similarly,  $P_B(I_0) = P_O(I_0)$  indicates  $P_B(I_0)$  passed our verifier. This verification result is consistent with the system judge and demonstrates that ALGO-generated oracles can help to verify the programs generated. Furthermore, the verification results can help in finding the correct instruction (algorithm category).
Reranking Techniques for LLM Code Generation. LLMs often need to sample many candidates before finding a single correct solution. Therefore, reranking techniques for telling the correct programs among many candidates are crucial for better LLM coders. One line of work in reranking involves neural models that predict some form of confidence score for program candidates. This score can be the likelihood of the program [5], the mutual information between the code and the problem description [32], or outputs from verifier models that are purposely trained to predict the correctness of programs [22, 13]. However, their scores are just scalars and not interpretable. Both the oracles and test results from ALGO are much easier to check and interpret.
Another line of work utilizes test cases and the programs' execution results to rerank them, which is much more similar to ALGO's approach. Shi et al. [27] and Li et al. [17] cluster different candidates
according to their execution results on example cases, creating a mechanism similar to majority voting. Chen et al. [4] and Key et al. [15] also cross-check programs and execution results, but they create more test cases with the coder models to make the tests stronger. However, LLM-generated test cases for a single problem can contain both correct and incorrect ones, providing harmful feedback to the coder. ALGO has converted the task of checking AI-generated programs to checking AI-generated brute-forces, which is much easier.
Oracles in Program Synthesis and Software Testing. In the traditional program synthesis domain, automatic verification of a synthesized program is possible [29, 28] because formal specifications, including input-output examples [25] or logical constraints [20], are explicitly provided. In contrast, oracle generation using LLMs is challenging due to the ambiguous nature of text and lack of formal specifications, making it difficult to do verification. Efforts have been made to create formal specifications from text [6, 15, 10], but reliability remains an issue. Meanwhile, in the software testing domain, test oracle generation [9, 18, 24] is a well-known and challenging task. However, existing work mainly focuses on generating regression tests, which are not intended for new bug detection. Moreover, many methods rely heavily on formatted documents or are designed for program crashes or basic functional properties, making them unsuitable for complex algorithmic problems.
Self-Refining Language Models. Large language models (LLMs) have exhibited a remarkable capability for self-analysis and self-improvement, as highlighted by numerous studies [19, 12, 30]. Such reflection is also used to improve the quality of the generated programs. For instance, Chen et al. [7] trains Large Language Models (LLMs) to generate explanations for code and utilizes both the explanation and the execution results as feedback to improve coding solutions. Self-Refine [19] prompts the LLM to provide feedback on the efficiency of its own code solutions and refine these solutions based on the feedback given. In this paper, we use the execution results of the reference oracle as feedback and employ them to guide the generation of programs.
We proposed ALGO, an algorithm synthesis framework that leverages LLM-generated oracles as verifiers to synthesize algorithmic programs. ALGO consists of a coder and a verifier. The coder generates code solutions to the problem and requests verification from the verifier to iteratively refine its idea and its code. The verifier employs LLM-generated oracles to generate test cases for verification. ALGO can employ different types of search strategies with different underlying code generation models. We introduced a method to synthesize the reference oracle with LLMs based on the ease to generate brute-force solutions for algorithmic problems.
We extensively evaluated ALGO's performance on two different datasets with three baselines. ALGO outperformed various baselines with different search strategies by a large margin. We also did a detailed analysis of the reference oracles generated on their semantic correctness, and their verdict agreement with the system judge. The oracles in ALGO have a high accuracy and make highly consistent verification results with the system judge. The high quality of the oracles supports the performance of ALGO.
While ALGO is evaluated with several searchers we proposed in the paper, there can be more possibilities for applying different searchers in the ALGO framework. For example, the self-refinement studies in large language models may be utilized for more complex interactions between the coder and the verifier. On the other hand, oracle-guided synthesis [14] is a well-studied topic in the programming language community with techniques that can be applied in different branches of code synthesis. We believe that knowledge from both the natural language processing community and the software engineering community can inspire more future work in the framework of ALGO.
This work is partially supported by unrestricted gifts from IGSB and Meta via the Institute for Energy Efficiency.
A.1 Examples of Algorithm Synthesis and Functionality Synthesis
We give examples of algorithm synthesis and functionality synthesis in Figure 8.
The upper part is an instance of algorithm synthesis, where input that describes the problem does not indicate the solution idea 'binary search' in any way. To solve the algorithm synthesis program, the model needs to come up with the idea related to 'binary search' first either implicitly or explicitly before synthesizing the code.
The lower part is an instance of functionality synthesis, where the input is basically a pseudo-code program described in natural language that can directly translate into Python code. The model does not need to come up with the idea because the solution is already clearly stated in the input.
Figure 8: Examples of algorithm synthesis (top) and functionality synthesis (bottom). The parts in solid boxes are given by the problem, while the parts in dotted boxes should be inferred by the model. Functionality synthesis does not require the model to infer the idea, while algorithm synthesis does.
Figure 9: The naive prompt without instruction guidance and a candidate program generated with it. The instructions are in blue. It is a greedy method in linear time but is not suitable for this task. Even the naive candidate generated can pass the examples in the description, it fails both the system judge and ALGO verifier.
A.2 Prompts in ALGO and Examples of Generated Programs
A.2.1 LeetCode Examples
We list the prompts we used to generate the solution candidates and the verifier components for LeetCode problems here.
Figure 9 lists the prompts and examples for generating the naive solution. Figure 10 lists the prompts and examples for generating solutions guided by instructions that specify the algorithm category. Figure 11 lists the prompts and examples for generating reference oracle. Figure 12 lists an input validator example and its prompt. Figure 13 lists the prompt for the input generator that generates a single test input. Figure 14 lists the prompt for the batch input generator.
Naive Candidate Generation Prompt Solve the following problem with python and test it against the example cases only.   
You are given an integer array ranks representing the ranks of some mechanics. ranks[1] is the rank of the ith mechanic. A mechanic with a rank r can repair n cars in  $\mathbf{r}*\mathbf{n}^{\prime}\mathbf{2}$  minutes. Mechanics work in parallel.   
You are also given an integer cars representing the total number of cars waiting in the garage to be repaired. Return the minimum time taken to repair all the cars. Examples: (omitted) Constraints: (omitted)   
\*\*Function definition\*\*   
class Solution: def repairCars(self, ranks: List[int], cars: int) -> int:
Naive Candidate Generated by LLM   
class Solution:   
def repairCars(self, ranks: List[int], cars: int) -> int: ranks.sort() max repairing_time = 0 assignedCars = 0 for rank_in ranks: numCars = (cars + len(ranks) - assignedCars - 1) / (len(ranks) - assignedCars) numCars = min(numCars, cars - assignedCars) repair_time = rank * numCars * numCars max repairing_time = max(max repairing_time, repair_time) assignedCars += numCars return max repairing_time numCars = min(numCars, cars - assignedCars) repair_time = rank * numCars * numCars max repairing_time = max(max repairing_time, repair_time) assignedCars += numCars return max repairing_time
Figure 10: The prompt with instructions about the specific algorithm category (binary search). Binary search is the ideal algorithm used by human programmers for this task, with which the coder can generate both correct and efficient solutions. In general, right algorithm category can efficiently improve the quality of generated program.
Figure 11: The prompt we used for oracle generation and one oracle generated with it. The instructions are in blue. The language model is instructed to generate the most straightforward solution by enumerating over a very large search space of all combinations of relevant variables. The generated oracle enumerates all the possible ordered partitions of work allocations to find out the optimal one.
Figure 12: The prompt is utilized to generate a input validator to verify the validity of the generated test input, ensuring it aligns with the constraints specified in the description. In practice, this validation task is a functionality synthesis task, which can be easily solved by LLM.
A.2.2 Codecontests Examples
We list the prompts we used to generate the solution candidates and the verifier components for Codecontests problems here. Figure 15 lists the prompts and examples for generating the reference oracle using an exhaustive search. Figure 16 lists the prompt for generating the batch input generator.
Input Generator Generation Prompt
You are given this teletocode problem and its test input validator 'is_valid'. Please help me by generating an input generator function 'gen_input' that randomly generates test inputs which follow the constraints defined. Your 'gen_input' function should make use of 'is_valid' by validating the generated input with it. Your function should only generate test inputs and not test outputs.
You are given an integer array ranks representing the ranks of some mechanics. ranks[I] is the rank of the ith mechanic. A mechanic with a rank r can repair n cars in  $r * n^2$  minutes. Mechanics work in parallel.
gauge to repair. Return the minimum time taken to repair all the cars.
Examples: (omitted)
Constraints: (omitted)  
 $\star$ Function definition  $\star \star$  (omitted)
def is_valid_input(ranks: List[int], cars: int) -> bool: (omitted)
Input Generator Generated by LLM
def is_valid_input(ranks: List[int], cars: int) -> bool:
1 if not 1 <= len(ranks) <= 10 ** 5: return False
Return Rate for rank in ranks:
if not 1 <= rank <= 100:
if not 1 <= cars <= 10 ** 6: return False;
def gen_input() -> tuple:
n = random.randint(1, 10 ** 5)
ranks = [random.randint(1, 100) for _ in range(n)]
- cars = random.randint(1, 10 ** k)
- if is valid input(ranks, cars):
return (ranks, cars)
Figure 13: The prompt is used to generate a data generator, to generate extra test cases when combined with the reference oracle. In practice, this generator generation task is a functionality synthesis task, which can be easily solved by LLM.
Batch Generator Generation Prompt
You are given this lexicode problem, its random input generator 'gen_input'. Please help me by generating 5 random inputs with the random input generator 'gen_input'. You should return a function named 'random_test()' that takes several arguments to control the upper limit of the input and returns a list of 5 tuples. You do not need to generate test outputs, only generate inputs.
You are given an integer array ranks representing the ranks of some mechanics. ranks[I] is the rank of the ith mechanic. A mechanic with a rank r can repair n cars in r * n^2. Minutes. Mechanics work in parallel.
> Return the minimum time taken to repair all the cars.
def gen_input() -> tuple: (omitted)
Batch Generator Generated by LLM
def random_test(max_ranks_length: int=10 ** 5, max_rank_value: int=100, maxCars: int=10 ** 6) -> List[tuple]:
def gen input() -> tuple:
n = random.randint(1, min(max+ranks_length, 10 ** 5))
ranks = [random.randint(1, min(max_rank_value, 100))]  
for in range(n)
```
cars = random.randint(1, min(maxCars, 10 ** 6))
if is validinput(cars, cars)
if is_valid_input(rank) return (ranks cars)
random inputs - [ ]
for in range(5):
random inputs.append(gen input())
return random_input
Figure 14: The prompt is used to generate a batch generator that employs the already generated gen_input to generate several input cases by calling it multiple times.
Since the test set problems in Codecontests are all from https://codeforces.com/, they originally require standard input/output. However, we follow the setting in Codecontests by concatenating the entire standard input as a single string and asking the model to generate reference oracles that map a string to a string.
Andi and Budi were given an assignment to tidy up their bookshelf of n books. Each book is represented by the book title - a string si numbered from 1 to, each with length m. Andi really wants to sort the book lexicographically ascending, while Budi wants to sort it lexicographically descending. (https://codeforces.com/problems/problem/1575/A, remaining part omitted)