Title: ALGO: Synthesizing Algorithmic Programs with Generated Oracle Verifiers
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Language Models

Code Generation

Code Intelligence

Automatic Verification
Abstract: Large language models (LLMs) excel at implementing code from functionality descriptions but struggle with algorithmic problems that require not only implementation but also identification of the suitable algorithm. Moreover, LLM-generated programs lack guaranteed correctness and require human verification. To address these challenges, we propose ALGO, a framework that synthesizes Algorithmic programs with LLM-Generated Oracles to guide the generation and verify their correctness. ALGO first generates a reference oracle by prompting an LLM to exhaustively enumerate all the combinations of relevant variables. This oracle is then utilized to guide an arbitrary search strategy in exploring the algorithm space and to verify the synthesized algorithms. Our study shows that the LLM-generated
oracles are correct for 88% of the cases. With the oracles as verifiers, ALGO can be integrated with any existing code generation model in a model-agnostic manner to enhance its performance. Experiments show that when equipped with ALGO, we achieve an 8× better one-submission pass rate over the Codex model and a 2.6× better one-submission pass rate over CodeT, the current state-of-the-art model on CodeContests. We can also get 1.3× better pass rate over the ChatGPT Code Interpreter on unseen problems. The problem set we used for testing, the prompts we used, the verifier and solution programs, and the test cases generated by ALGO
are available at https://github.com/zkx06111/ALGO.

ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifiers

Abstract

Large language models (LLMs) excel at implementing code from functionality descriptions but struggle with algorithmic problems that require not only implementation but also identification of the suitable algorithm. Moreover, LLM-generated programs lack guaranteed correctness and require human verification. To address these challenges, we propose ALGO, a framework that synthesizes A lgorithmic programs with L LMG enerated O racles to guide the generation and verify their correctness. ALGO first generates a reference oracle by prompting an LLM to exhaustively enumerate all the combinations of relevant variables. This oracle is then utilized to guide an arbitrary search strategy in exploring the algorithm space and to verify the synthesized algorithms. Our study shows that the LLM-generated oracles are correct for 88% of the cases. With the oracles as verifiers, ALGO can be integrated with any existing code generation model in a model-agnostic manner to enhance its performance. Experiments show that when equipped with ALGO, we achieve an 8× better one-submission pass rate over the Codex model and a 2.6× better one-submission pass rate over CodeT, the current state-of-the-art model on CodeContests. We can also get 1.3× better pass rate over the ChatGPT Code Interpreter on unseen problems. The problem set we used for testing, the prompts we used, the verifier and solution programs, and the test cases generated by ALGO are available at https://github.com/zkx06111/ALGO .

1 Introduction

Large Language Models (LLMs) have demonstrated significant prowess in generating code from natural language descriptions. Models such as Codex [5] and CodeGen [23] can easily achieve over 30% pass@1 accuracy on HumanEval, a docstring-to-code dataset. However, these models struggle when faced with algorithmic problems akin to those encountered in CodeContests [17]. Even with reasonable sample limits, achieving a 10% accuracy rate remains a considerable challenge [4]. More recently, the GPT-4 model, despite being provided with problem descriptions and solution hints, managed a pass@1 accuracy of only 10.6% [3] on LeetCode Hard problems.

The verifiability of LLM-based code generation presents another significant challenge. Without verifiability, code generation systems can hardly earn trust for production use. Users of GitHub Copilot [21], an AI assistant for programmers, reportedly spend over 20% of their time verifying the suggested code snippets. Existing approaches toward verification and reranking LLM-generated programs either rely on neural models to predict confidence scores [22, 32] or require LLM-generated test cases to execute the programs [4, 15]. However, neural verifiers do not provide interpretable feedbacks and LLM-generated test cases are often incorrect.

Traditional code synthesis techniques rely on oracles for the verification of the synthesized code. These oracles are typically derived from formal specifications [20, 25, 28, 29]. They can also be employed in program synthesis under the widely-used framework of oracle-guided inductive synthesis (OGIS) [14]. Despite their widespread application in code verification and synthesis, the

Problem

Verifier

LLM

Prompt

slow, but correct

Input

Generator

Input

Code

Output

Oracle

Output

Verification results & failed test cases

Coder

fast, maybe incorrect

Problem

Solution Generation

Prompt

×

。

。

Accepted

TimeLimitExceeded

RuntimeError

WrongAnswer

31.4%

57.1%

use of oracles often demands heavy human involvement, and the automatic generation of oracles is challenging. Existing strategies focus on generating oracles that can only detect crashes (safety oracle) or bugs introduced by future changes (regression oracle), rather than semantic bugs crucial for algorithmic programs [9, 18, 24].

Motivated by the potential benefits of oracles in traditional code synthesis, we introduce ALGO, a framework that leverages oracles generated by large language models to address the challenges in LLM-based code synthesis and verification. As illustrated in Figure 1, ALGO incorporates two modules for algorithm synthesis. The verifier is instructed to generate an exhaustive search algorithm regardless of time efficiency, thus acting as a reference oracle. Concurrently, the coder is asked to find a more efficient solution with any prompts or search strategies. The candidate program's correctness is then evaluated by comparing its output with that of the oracle for a given set of test inputs. The results of the verification, along with any test cases where the candidate program falls short, are subsequently provided to the coder for code refinement.

We evaluated ALGO based on two key dimensions: the oracle's verification capability and its potential to enhance algorithm synthesis. Our experiments reveal that the reference oracles are correct for 88.5% of the problems (as shown in Figure 2). Moreover, the oracles' verdicts are in agreement with the golden verdicts of the online judge 75% of the time. To examine the enhancement in accuracy offered by ALGO, we integrated it with several existing code generation models including Codex [5], CodeT [4], PG-TD [31] and ChatGPT Code Interpreter 1 . We observed that ALGO significantly boosts their performance on CodeContests [17] and a collection of recent LeetCode problems. Our experiments show that ALGO notably enhances the performance of these models: in terms of onesubmission pass rate, Codex's performance improved by a factor of 8, CodeT's by 2.6, PG-TD's by 1.5, and ChatGPT Code Interpreter's by 1.3.

Our contributions can be summarized as follows:

We present ALGO, a novel framework for A lgorithm synthesis that utilizes L LMG enerated reference O racles as verifiers. It is a model-agnostic framework that can be integrated with any code generation model to verify candidate solutions with reliable test cases generated by reference oracles.

We conduct a comprehensive evaluation of synthesis accuracy and verifiability of ALGO, utilizing several distinct code generation models in a versatile, model-agnostic manner. Our results indicate that ALGO can generate high-quality oracles and test cases that lead to significant improvements in code generation.

Implement  a  function  incr_list(l:  list)  that  returns  the  input  list  with

elements increased by 1.

You are given an integer array

ranks representing the ranks of some

mechanics. ranks[I] is the rank of the

ith mechanic. A mechanic with a rank r

can repair n cars in r * n^2 minutes.

You are also given an integer cars

representing the total number of cars

waiting in the garage to be repaired.

Return the minimum time taken to

repair all the cars.

class Solution:

def repairCars(self, ranks:

List[int], cars: int) -> int:

while left < right:

mid = (left + right) // 2

if canRepairInTime(mid):

right = mid

else:

left = mid + 1

return left

Try using binary search on the answer.

Check whether for a fixed time all

cars can be repaired.

This can be done by summing up the

maximum number of cars each mechanic

can repair in the given time.

def incr_list(l: list):

return [i + 1 for i in l]

Input

Code

Algorithm

Synthesis

Functionality

Synthesis

Idea

Input

Code

2 Algorithm Synthesis

Large language models excel at generating simple programs. For example, ChatGPT can easily achieve over 70% pass@1 accuracy on the docstring-to-code dataset HumanEval. However, it still struggles with complex algorithmic problems such as CodeContests. We hypothesize that this is because algorithmic problems need efficient solutions rather than straightforward and brute-force ones. To test our hypothesis, we prompt ChatGPT in two different ways to see if they result in different success rates. One prompt directly describes the problem and asks the model to generate a solution that meets the time limits. The other asks the model to ignore the time limit and to solve the problem in the most straightforward way. The prompts we used are listed in Appendix A.2. We evaluate the correctness of the brute-forces and efficient solutions both manually and using official test cases without time limits so that a correct yet inefficient program is also considered a success.

The results in Table 1 validate our hypothesis. For LeetCode and CodeContests, it is much easier to generate brute-forces than efficient solutions. But for HumanEval, the two success rates are similar. This clearly indicates that problems in HumanEval and those in LeetCode and CodeContests are of two different types. The latter is more difficult to solve because it requires the application of algorithms. Based on that, we divide code synthesis into two categories: functionality synthesis and algorithm synthesis . Typically, functionality synthesis problems, such as those seen in HumanEval [5] and MBPP [1], provide a detailed description that can be readily translated into an implementation. On the other hand, algorithm synthesis problems found in APPS [11] and CodeContests [17] are more abstract and need efficient algorithms to solve. In this paper, we focus on addressing algorithm synthesis rather than functionality synthesis . An algorithmic synthesizer, thus, has an additional task to that of a functionality synthesizer-it needs to generate the solution idea, implicitly or explicitly, before synthesizing the code. We further clarify the contrast between the two by providing typical examples of each in Figure 3.

Formal definition of algorithm synthesis and functionality synthesis. A code generation task can be defined as a tuple ( Q,J ) ∈ Q × J , where Q is the problem description and J : P → Bool is the system judge. The system judge J evaluates if a program from the program space P solves the problem represented by Q . The goal of the synthesizer is to produce a program P ∈ P that J ( P ) = True . The system judge J is further composed of two components: J S : P → Bool and

J T : P → Bool . J S checks the semantic correctness of the generated code, while J T ensures that the efficiency of the generated code satisfies the requisite conditions. Intuitively, J ( P ) = J T ( P ) ∧ J S ( P ) .

3 Algorithmic Synthesis with LLM-Generated Oracles (ALGO)

As discussed in Section 2, generating brute-force solutions to algorithm synthesis tasks is much easier than generating efficient solutions; this huge gap in difficulty can be exploited. To solve algorithm synthesis, we propose Algorithmic Synthesis with LLM-Generated Oracles (ALGO). As shown in Figure 1, the ALGO framework utilizes two components - a coder and a verifier - to solve algorithmic problems. The coder takes the problem description Q and optionally, the verification results from its last generation and generates a program P that solves Q . The verifier generates a reference oracle whose outputs are used to verify whether the candidate program generated by the coder is correct. For each problem, ALGO creates the verifier once and uses it to guide arbitrary coders, allowing ALGO to be model-agnostic.

3.1 Verification with Oracle

Oracle Generation The difference between a reference oracle P O and an actual solution P G is that P O only needs to be semantically correct (i.e. J S ( P O ) = True ) while P G needs to be correct and efficient (i.e. J ( P G ) = J S ( P G ) ∧ J T ( P G ) = True ). We utilize this difference to generate the oracle. When there is no time limit, most algorithmic problems can be solved with an exhaustive search [8]. As depicted in Figure 4, we put the process of an exhaustive search algorithm (which is the same for every problem) in the prompt along with the problem description. LLMs, which excel at implementing a program when provided with clear instructions, are then able to generate reference oracles.

The Verification Process To handle a verification request, we must return a verdict of True/False regarding P 's correctness, and optionally, the inputs that cause P to fail. We utilize an input generator program to create random test inputs in line with problem constraints. These test inputs are then supplied to both the program under verification P , and the oracle P O . The test outputs of P are compared against those of P O . If they match, a True verdict is returned. If they don't, the failed test cases are returned to the synthesizer along with a False verdict.

3.2 Code Synthesis Strategies

ALGO's coder can use different strategies for code synthesis. The code synthesis process can either be one-time or iterative, depending on the used coder. During a one-time synthesis, the coder generates a batch of solutions to the problem, and these solutions are verified and ranked according to their verification results. During an iterative synthesis, the code generates solutions, gets their verification results (and optionally mistaken test cases), and then revises the solutions iteratively.

Other than the coder model itself, its strategy to search for and identify the suitable algorithm for the problem also affects the synthesis accuracy significantly. The coder's search strategy can be defined as a conditional distribution π ( P | Q ) of the program P conditioned on the problem description Q . To generate code is to sample programs from this distribution. To compute this distribution, we can define a space of latent variables I and marginalize over it, as denoted by

ALGO is a versatile framework that can incorporate a variety of search strategies. A strategy may be implicit, with the variable I not specified, or explicit, specifying I , the space of I , and the methodology to traverse this space. Here, we give three examples: an implicit searcher , an instruction enumerator , and an iterative searcher . However, it is crucial to note that these examples are representative, and there are numerous alternative search strategies in ALGO.

Implicit Searcher: An implicit searcher samples a program directly from the distribution determined by the underlying model, using the problem description. It solely relies on the ability of the code generation model and depends on the oracle to filter the generated programs post hoc.

Instruction Enumerator: An instruction enumerator is an explicit searcher where I is a set of pre-defined instructions. Since the performance of instruction-tuned LLMs varies significantly based on the instructions they are given, it is natural to consider enumerating a space of instructions to get the best result. An instruction enumerator first picks an instruction from a pre-defined instruction set I , and then synthesizes the program by instructing the language model with I . In our example of an instruction enumerator, we choose I to be a set of high-level ideas of algorithms such as ' Binary Search ' and ' Sorting '. We first sample possible solution algorithms and then instruct the large language models to generate programs with the ideas given.

Iterative Searcher: A iterative searcher is an explicit searcher that takes the signal from the verifier to refine its output. The searcher space I can be the token vocabulary of the model. It uses a search algorithm to determine the next token during the decoding process by enumerating the vocabulary tokens given the problem description and the partial program in each generation step. Since the search space is exponentially large, it would be time-intensive. Usually, the searcher is guided by some rewards from the verifier to prune the candidate space in each step.

4 Experiments

In this section, we implement ALGO with three code synthesis strategies, and evaluate it with two challenging algorithmic benchmarks to validate its flexibility and effectiveness. Moreover, we investigate the verifier's performance by evaluating the quality of the generated reference oracle and test cases.

4.1 Experiment Setup

Verifier We employ ChatGPT Code Interpreter to create the verifier. It is first prompted to generate the reference oracle and then the input generator. As mentioned in Section 3.1, this is possible because the reference solutions to most algorithmic problems involve exhaustively going through all possible solutions to find a feasible or optimal one. We use a temperature of 1.0 and resample the solution until it can pass all example cases. The prompt we used for generating the oracle is shown in Figure 4. For the verifier, we simply use zero-shot prompts to generate the input generator with an input validator since both of them are functionality syntheses and easy to generate. We set the maximum length of each input variable to 10 and skip out cases that raise the timeout exception and the recursion error when generating the output via the reference oracle. The prompts we used are listed in Appendix A.2 with generation examples. Note that for each problem, we only need to create the verifier once, and it can be used to create arbitrary test cases and guide arbitrary models.

Code Synthesis Strategies We integrate the following baselines as the coder with ALGO to evaluate its verification capability and synthesis accuracy:

Codex [5] and CodeT [4]: Codex and CodeT are used as coders that utilize implicit searchers . Codex is a specialized language model, trained on publicly available code from GitHub. CodeT uses Codex to generate both programs and test cases for a problem and uses their dual agreement to

filter and rank the programs generated. To evaluate ALGO, we use the exact same set of programs Codex generated, and rank them in three different ways: in a random manner, based on the dual agreement heuristics from CodeT, and according to the verification results provided by ALGO. We then compare the top-ranked programs from each of these three rankings. Note that due to OpenAI's policy, the Codex model is no longer available so we directly use the Codex-generated programs provided by the CodeT paper 2 .

ChatGPT Code Interpreter : ChatGPT Code Interpreter is used as a coder that utilizes instruction enumerators . It is a variant of GPT-3.5 with the code interpreter plugin. The version we evaluated, released on March 24th, utilizes the underlying model text-davinci-002-code . As the Code Interpreter is instruction-tuned, it can follow directives to solve a problem using a specific class of algorithms, such as 'Binary Search'. This capability allows it to employ a search strategy that iterates through a list of algorithm categories to identify a correct solution. Moreover, with the code interpreter plugin, it is able to run Python code and interpret its results, and automatically refine its output based on the program output. We use the default temperature of 1.0 and resample the code solution until it can pass the example cases in the problem descriptions or reach the resample limit (N=5).

PG-TD [31]: PG-TD is used as a coder that utilizes iterative searchers . In contrast to CodeT, which employs test cases post hoc, PG-TD incorporates them during the decoding process. It implements a tree search-based planning algorithm for decoding, utilizing the reward from test cases to estimate the value of the children nodes in the search tree. PG-TD uses GPT-2 [26] and GPT-NEO [2] as the underlying model. We implement two versions: PG-TD* is only based on the public test cases to generate programs, and PG-TD has access to both public test cases and the generated test cases by Li et al. [17]. Both are based on the GPT-2 fine-tuned on the APPS training dataset, which was released by Zhang et al. [31]. We integrate PG-TD into ALGO by substituting the generated test cases with the reward from the verifier.

Benchmarks We utilize the following benchmarks to evaluate ALGO:

CodeContests [17]: We evaluate ALGO with Codex, CodeT and PG-TM on all 165 problems from CodeContests test set, which are collected from Codeforces, a competitive programming website. All the three baselines we evaluate have been evaluated on CodeContests in their own paper, we follow the exact same setting to ensure a fair comparison.

LeetCode : We evaluate ALGO with ChatGPT Code Interpreter on 35 recently released LeetCode problems. To avoid the influence of contamination and faithfully evaluate ALGO's ability in verification and guiding code synthesis, we made sure these problems are released concurrently or after the release of GPT-4. We manually annotate the algorithm categories for the solutions to these problems to evaluate the search strategy of enumerating algorithm categories. The list of problems and their corresponding categories is listed in Appendix A.3.

Metric Following previous studies [31, 4], we use the n @ k metric [16, 5, 17] to evaluate the accuracy of code synthesis. To compute n @ k for a problem, a code synthesizer needs to generate k candidate programs and select the topn candidates to submit according to some reranking mechanism, which in our case is the LLM-generated oracles and test samples. n @ k is the proportion of problems that can be solved by any of the topn submissions. Since some code synthesizers do not have reranking mechanisms, we define their n @ k as the proportion of problems solved with n submissions randomly sampled from k candidates. To ensure a fair comparison, we use the same sampling budget n under the same setting when evaluating the performance of ALGO and baselines.

4.2 Synthesis Accuracy

CodeContests We present ALGO's n @ k accuracy for CodeContests in Table 2. In the first part of the table, Codex is used as an implicit searcher. Compared to Codex itself and CodeT which also heuristically evaluates the program Codex generates, ALGO consistently outperforms them at every value of n . The most significant performance enhancement by our method is observed at 1 @ k , yielding an 8× improvement over Codex, and a 2.6× improvement over CodeT. As the value of k increases, the performance advantage diminishes. This trend is expected, as the chosen sample number n approaches the total sample count k , causing the pass rate to converge towards the ratio of correct programs within the total program set. The same phenomenon is also observed in the second

and third part of Table 2, where PG-TD and ChatGPT are used as coders. ALGO is able to achieve a 2.5× improvement over PG-TD and a 2.8× improvement over ChatGPT at 1 @ k .

LeetCode We list ALGO's one-submission pass rate on LeetCode problems in Figure 5. The instruction set we enumerate for each problem is a set with several algorithm categories, among which there is one correct category. ALGO's superior performance compared to ChatGPT and GPT-4 demonstrates the benefits of an instruction searcher that enumerates possible algorithms. It also demonstrates ALGO's ability to filter and select the correct algorithm category. Without the verifier, we would not have been able to tell right from wrong.

4.3 Verification Analysis

The verifier is critical to ALGO as it guides the creation of an efficient and accurate solution. In this section, we scrutinize the quality of the verifier from several perspectives.

LLM-generated oracles are usually correct. The correctness of LLM-generated oracles is crucial to the correctness of ALGO's test cases. We examine their correctness with both the system judge and human experts. We submitted LLM-generated oracles to the online judges (Codeforces and LeetCode) where the problems came from, to get the system verdicts. Oracles with verdicts other than Accepted (AC), Time Limit Exceeded (TLE), and Runtime Error (RE) were directly considered incorrect. For those with TLE and RE verdicts that could potentially be semantically correct despite exceeding the time limit or stack size, we hired an experienced competitive programming contestant to examine if they were semantically correct and not efficient enough. For LeetCode problems, as displayed in Figure 2, 88.5% of the oracles are semantically correct, including all those with TLE verdicts. For codecontests problems, 72% of the oracles are semantically correct.

Oracle-generated test cases have better quality. We evaluate the quality of ALGO's test cases generated by answering the following questions: (i) Do verification verdicts from generated cases agree more with the system judge than verdicts from only the public tests? (ii) Do the generated tests achieve higher statement coverage compared with public tests? In this context, we calculate two metrics: Agreement and Coverage . Agreement is the consistency between the decision of the system judge and the test case set. For instance, candidates that fail the system judge should also fail on the generated test set, while those that can pass the judge should also succeed on all test cases. Coverage is the percentage of statements in a code solution that are executed by test cases. Following

pass rate

ChatGPT

ALGO

0

10

20

30

40

4

6

8

10

12

1@k

2@k

10@k

100@k

5

10

20

30

40

Chen et al. [4], we use a public coverage library 3 to gather the coverage results. As demonstrated in Table 3, our generated test cases substantially improve the agreement with the standard judge (+37.39% on CodeContests and +5.72% on LeetCode), indicating they can enhance the verification of candidates prior to submission. In addition, test cases of ALGO achieve higher coverage of the statements, implying that these cases can more effectively validate the functionality of the solution.

Better test cases lead to better results. The test cases generated by ALGO have better agreement and coverage. We check if they lead to better synthesis by comparing them to the other two sets of test cases - examples provided in the problem statement and test cases directly generated by ChatGPT. We used the same coder with the exact same sample budget k = 20 . As reported in Table 4, under the same setting, ALGO tests led to much better performance.

More test cases lead to better results. We examine the impact of the size of test cases the verifier generates on CodeContests for ALGO w/ Codex. Our verifier has the flexibility to create an arbitrary number of cases and use test case sets of varying sizes to rank and guide the synthesis. As shown in Figure 6, when the size is fewer than 30, the increase in the size of test cases can significantly bolster performance for all n @ k , indicating the effectiveness of our verifier in covering more corner cases with additional test cases. Notably, having a sufficient number of test cases is more critical for 1 @ k , which relies on test cases to select the optimal solution for one-time submission. However, this improvement diminishes when n is larger. This is because a robust verifier is not strongly required when the chosen sample number n is large, as discussed in Section 4.2. Furthermore, there is very little additional performance gain with more than 30 test cases. We hypothesize that this is due to the quality of our oracle and the generation range we have set for the verifier.

4.4 Case Study

We demonstrate the workflow of ALGO with a problem from our LeetCode benchmark 4 . This problem asks for the optimal allocation of cars to mechanics with varying efficiency to minimize the total repair time, when the mechanics would work simultaneously. As depicted in Figure 7, the code generated using a reference oracle prompt exhaustively enumerates all potential assignments of cars and selects the optimal one according to the problem description. Assuming n = len( ranks ) , m = cars , this program exhibits exponential time complexity T O ( n, m ) = O (2 m ) , making it an inefficient solution that is unable to pass the online judge. Nevertheless, despite this program's inefficient nature, this implementation remains valuable in terms of semantic correctness (i.e., J T ( P O ) = False and J S ( P O ) = True , where we can conveniently define J T ( P ) = "time complexity of P is at most O (( n + m ) log ( n + m )) " without loss). This example demonstrates how ALGO can generate reference oracles for code synthesis.

16

100

Potential Greedy Solution

class Solution:

def repairCars(self, ranks: List[int], cars: int):

ranks.sort()

max_repair_time = 0

assigned_cars = 0

for rank in ranks:

num_cars = (cars + len(ranks) - assigned_cars -

1) // (len(ranks) - assigned_cars)

num_cars = min(num_cars, cars - assigned_cars)

repair_time = rank * num_cars * num_cars

max_repair_time = max(max_repair_time,

repair_time)

assigned_cars += num_cars

return max_repair_time

num_cars = min(num_cars, cars - assigned_cars)

repair_time = rank * num_cars * num_cars

max_repair_time = max(max_repair_time,

repair_time)

assigned_cars += num_cars

return max_repair_time

class BruteforceSolution:

def repairCars(self, ranks: List[int], cars: int) ->

int:

num_mechanics = len(ranks)

min_time = float('inf')

def generate_allocations(allocation, remaining_cars):

nonlocal min_time

if len(allocation) == num_mechanics:

if sum(allocation) == cars:

times = [ranks[i] * allocation[i] * \

allocation[I] for i in range(num_mechanics)]

max_time = max(times)

min_time = min(min_time, max_time)

return

for num_cars in range(remaining_cars + 1):

new_allocation = allocation + [num_cars]

new_remaining_cars = remaining_cars - num_cars

generate_allocations(new_allocation,\

new_remaining_cars)

generate_allocations([], cars)

return min_time

Potential Binary Search Solution

class Solution:

def repairCars(self, ranks: List[int], cars: int) ->

int:

def countCarsRepaired(ranks, minutes):

total_cars = 0

for rank in ranks:

n = int((minutes / rank) ** 0.5)

total_cars += n

return total_cars

left = 1

right = max(ranks) * cars * cars

while left < right:

mid = (left + right) // 2

if countCarsRepaired(ranks, mid) < cars:

left = mid + 1

else:

right = mid

return left

16

System Judge:

Accepted

System Judge:

Wrong

Answer

Test Input

: ranks=[4,2,3,1],cars=10

I

0

I

1

I

2

I

3

. . .

Test Inputs Generated by ALGO

In Figure 7, we present how the reference oracle can verify the correctness of candidate programs generated by the coder LLM. We present two candidate programs generated with different algorithm categories: greedy and binary search. Specifically, the binary search candidate P B is generated with instructions on binary search and the greedy candidate P G is generated without explicit guidance. The binary search algorithm is ideally suited for this problem, given that the indicator function, denoted as f ( t ) , which determines whether all cars can be repaired within t minutes, is monotonous. This makes it possible to apply binary search on the time t and turn the optimization problem into a judgement problem. The binary search candidate represents an implementation of such algorithmic approach, which is an efficient and correct solution of time complexity T B ( n, m ) = O ( n log m ) and can pass the system judge (i.e., J ( P B ) = True ). On the other hand, greedy algorithm is not suitable for this problem. Thus, even though the greedy candidate presented is of time complexity T G ( n, m ) = O ( n ) , the semantic of this candidate is wrong (i.e., J T ( P G ) = True and J S ( P G ) = False ). This example demonstrates that providing the correct instruction can lead to better code synthesis result.

̸

In Figure 7, P G failed with our verifier since P G ( I 0 ) = P O ( I 0 ) . Similarly, P B ( I 0 ) = P O ( I 0 ) indicates P B ( I 0 ) passed our verifier. This verification result is consistent with the system judge and demonstrates that ALGO-generated oracles can help to verify the programs generated. Furthermore, the verification results can help in finding the correct instruction (algorithm category).

5 Related Work

Reranking Techniques for LLM Code Generation. LLMs often need to sample many candidates before finding a single correct solution. Therefore, reranking techniques for telling the correct programs among many candidates are crucial for better LLM coders. One line of work in reranking involves neural models that predict some form of confidence score for program candidates. This score can be the likelihood of the program [5], the mutual information between the code and the problem description [32], or outputs from verifier models that are purposely trained to predict the correctness of programs [22, 13]. However, their scores are just scalars and not interpretable. Both the oracles and test results from ALGO are much easier to check and interpret.

Another line of work utilizes test cases and the programs' execution results to rerank them, which is much more similar to ALGO's approach. Shi et al. [27] and Li et al. [17] cluster different candidates

according to their execution results on example cases, creating a mechanism similar to majority voting. Chen et al. [4] and Key et al. [15] also cross-check programs and execution results, but they create more test cases with the coder models to make the tests stronger. However, LLM-generated test cases for a single problem can contain both correct and incorrect ones, providing harmful feedback to the coder. ALGO has converted the task of checking AI-generated programs to checking AI-generated brute-forces, which is much easier.

Oracles in Program Synthesis and Software Testing. In the traditional program synthesis domain, automatic verification of a synthesized program is possible [29, 28] because formal specifications, including input-output examples [25] or logical constraints [20], are explicitly provided. In contrast, oracle generation using LLMs is challenging due to the ambiguous nature of text and lack of formal specifications, making it difficult to do verification. Efforts have been made to create formal specifications from text [6, 15, 10], but reliability remains an issue. Meanwhile, in the software testing domain, test oracle generation [9, 18, 24] is a well-known and challenging task. However, existing work mainly focuses on generating regression tests, which are not intended for new bug detection. Moreover, many methods rely heavily on formatted documents or are designed for program crashes or basic functional properties, making them unsuitable for complex algorithmic problems.

Self-Refining Language Models. Large language models (LLMs) have exhibited a remarkable capability for self-analysis and self-improvement, as highlighted by numerous studies [19, 12, 30]. Such reflection is also used to improve the quality of the generated programs. For instance, Chen et al. [7] trains Large Language Models (LLMs) to generate explanations for code and utilizes both the explanation and the execution results as feedback to improve coding solutions. Self-Refine [19] prompts the LLM to provide feedback on the efficiency of its own code solutions and refine these solutions based on the feedback given. In this paper, we use the execution results of the reference oracle as feedback and employ them to guide the generation of programs.

6 Conclusion and Discussion

We proposed ALGO, an algorithm synthesis framework that leverages LLM-generated oracles as verifiers to synthesize algorithmic programs. ALGO consists of a coder and a verifier. The coder generates code solutions to the problem and requests verification from the verifier to iteratively refine its idea and its code. The verifier employs LLM-generated oracles to generate test cases for verification. ALGO can employ different types of search strategies with different underlying code generation models. We introduced a method to synthesize the reference oracle with LLMs based on the ease to generate brute-force solutions for algorithmic problems.

We extensively evaluated ALGO's performance on two different datasets with three baselines. ALGO outperformed various baselines with different search strategies by a large margin. We also did a detailed analysis of the reference oracles generated on their semantic correctness, and their verdict agreement with the system judge. The oracles in ALGO have a high accuracy and make highly consistent verification results with the system judge. The high quality of the oracles supports the performance of ALGO.

While ALGO is evaluated with several searchers we proposed in the paper, there can be more possibilities for applying different searchers in the ALGO framework. For example, the self-refinement studies in large language models may be utilized for more complex interactions between the coder and the verifier. On the other hand, oracle-guided synthesis [14] is a well-studied topic in the programming language community with techniques that can be applied in different branches of code synthesis. We believe that knowledge from both the natural language processing community and the software engineering community can inspire more future work in the framework of ALGO.

References

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021.

Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https: //doi.org/10.5281/zenodo.5297715 . If you use this software, please cite it using these metadata.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= ktrw68Cmu9c .

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Qiaochu Chen, Shankara Pailoor, Celeste Barnaby, Abby Criswell, Chenglong Wang, Greg Durrett, and I¸ sil Dillig. Type-directed synthesis of visualizations from natural language queries. Proceedings of the ACM on Programming Languages , 6(OOPSLA2):532-559, 2022.

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 , 2023.

Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms . MIT press, 2022.

Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K Lahiri. Toga: a neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering , pp. 2130-2141, 2022.

Alex Gu, Tamara Mitrovska, Daniela Velez, Jacob Andreas, and Armando Solar-Lezama. Obsynth: An interactive synthesis system for generating object models from natural language specifications. arXiv preprint arXiv:2210.11468 , 2022.

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurIPS , 2021.

Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.11610 , 2022.

Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shuvendu Lahiri, Madanlal Musuvathi, and Jianfeng Gao. Fault-aware neural code rankers. Advances in Neural Information Processing Systems , 35:13419-13432, 2022.

Susmit Jha and Sanjit A Seshia. A theory of formal synthesis via inductive learning. Acta Informatica , 54:693-726, 2017.

Darren Key, Wen-Ding Li, and Kevin Ellis. I speak, you verify: Toward trustworthy neural program synthesis. arXiv preprint arXiv:2210.00848 , 2022.

Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems , 32, 2019.

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science , 378(6624):1092-1097, 2022.

Shaoying Liu and Shin Nakajima. Automatic test case and test oracle generation based on functional scenarios in formal specifications for conformance testing. IEEE Transactions on Software Engineering , 48(2):691-712, 2020.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 , 2023.

Zohar Manna and Richard Waldinger. A deductive approach to program synthesis. ACM Transactions on Programming Languages and Systems (TOPLAS) , 2(1):90-121, 1980.

Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. Reading between the lines: Modeling user behavior and costs in ai assisted programming. arXiv preprint arXiv:2210.14306 , 2022.

Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning , pp. 26106-26128. PMLR, 2023.

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=iaYcJKpY2B_ .

Lin Padgham, Zhiyong Zhang, John Thangarajah, and Tim Miller. Model-based test oracle generation for automated unit testing of agent systems. IEEE Transactions on Software Engineering , 39(9):1230-1244, 2013.

Oleksandr Polozov and Sumit Gulwani. Flashmeta: A framework for inductive program synthesis. In Proceedings of the 2015 ACM SIGPLAN International Conference on ObjectOriented Programming, Systems, Languages, and Applications , pp. 107-126, 2015.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I Wang. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp. 3533-3546, 2022.

Saurabh Srivastava, Sumit Gulwani, and Jeffrey S. Foster. From program verification to program synthesis. In Proceedings of the 37th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages , POPL '10, pp. 313-326, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781605584799. doi: 10.1145/1706299.1706337. URL https://doi.org/10.1145/1706299.1706337 .

Saurabh Srivastava, Sumit Gulwani, and Jeffrey Foster. Template-based program verification and program synthesis. International Journal on Software Tools for Technology Transfer , 15, 04 2013. doi: 10.1007/s10009-012-0223-4.

Danqing Wang and Lei Li. Learn from mistakes through cooperative interaction with study assistant. The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) 2022 , 2023.

Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. Planning with large language models for code generation. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=Lr8cOOtYbfL .

Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida Wang. Coder reviewer reranking for code generation. In International Conference on Machine Learning , pp. 41832-41846. PMLR, 2023.

A Appendix

A.1 Examples of Algorithm Synthesis and Functionality Synthesis

We give examples of algorithm synthesis and functionality synthesis in Figure 8.

The upper part is an instance of algorithm synthesis, where input that describes the problem does not indicate the solution idea 'binary search' in any way. To solve the algorithm synthesis program, the model needs to come up with the idea related to 'binary search' first either implicitly or explicitly before synthesizing the code.

The lower part is an instance of functionality synthesis, where the input is basically a pseudo-code program described in natural language that can directly translate into Python code. The model does not need to come up with the idea because the solution is already clearly stated in the input.

Implement  a  function  incr_list(l:  list)  that  returns  the  input  list  with

elements increased by 1.

You are given an integer array

ranks representing the ranks of some

mechanics. ranks[I] is the rank of the

ith mechanic. A mechanic with a rank r

can repair n cars in r * n^2 minutes.

You are also given an integer cars

representing the total number of cars

waiting in the garage to be repaired.

Return the minimum time taken to

repair all the cars.

class Solution:

def repairCars(self, ranks:

List[int], cars: int) -> int:

while left < right:

mid = (left + right) // 2

if canRepairInTime(mid):

right = mid

else:

left = mid + 1

return left

Try using binary search on the answer.

Check whether for a fixed time all

cars can be repaired.

This can be done by summing up the

maximum number of cars each mechanic

can repair in the given time.

def incr_list(l: list):

return [i + 1 for i in l]

Input

Code

Algorithm

Synthesis

Functionality

Synthesis

Idea

Input

Code

A.2 Prompts in ALGO and Examples of Generated Programs

A.2.1 LeetCode Examples

We list the prompts we used to generate the solution candidates and the verifier components for LeetCode problems here.

Figure 9 lists the prompts and examples for generating the naive solution. Figure 10 lists the prompts and examples for generating solutions guided by instructions that specify the algorithm category. Figure 11 lists the prompts and examples for generating reference oracle. Figure 12 lists an input validator example and its prompt. Figure 13 lists the prompt for the input generator that generates a single test input. Figure 14 lists the prompt for the batch input generator.

A.2.2 Codecontests Examples

We list the prompts we used to generate the solution candidates and the verifier components for Codecontests problems here. Figure 15 lists the prompts and examples for generating the reference oracle using an exhaustive search. Figure 16 lists the prompt for generating the batch input generator.

Since the test set problems in Codecontests are all from https://codeforces.com/ , they originally require standard input/output. However, we follow the setting in Codecontests by concatenating the entire standard input as a single string and asking the model to generate reference oracles that map a string to a string.

A.3 The List of Problems from LeetCode

We list the problems we collected from LeetCode as benchmarks to test ALGO in Table 5, among them are 10 easy problems, 18 medium problems, and 7 hard problems.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ALGO, a new method for code generation for LLMs. Given a problem description, ALGO first generates a brute-force implementation and dozens of test cases. Then, ALGO generates efficient implementations and ranks them by the agreement with the brute-force implementation on the generated test cases. The evaluation covers different LLMs, code generation techniques, and evaluation datasets. It shows that ALGO improves significantly over considered baselines.\n\nSTRENGTHS:\nThe paper is easy-to-follow and has good experimental results. It also provides an interesting ablation analysis in Section 3.4.\n\nWEAKNESSES:\n### Terminology\nVerification refers to proving formal properties of programs. The “verifier” fails to do so in two ways: (i) the “verifier” itself can often be wrong, and (ii) the “verification” is done on limited test cases and is thus unsound. I would suggest replacing “verifier” with a more appropriate term, such as cross-check, etc.\n\n### Motivation\nLines 26-32 point out an issue of existing code generation approaches: lacking verifiability. However, I think ALGO suffers from the same issue, because the generated “verifier” can be wrong. The users still cannot have trust on the output of ALGO.\n\n### Algorithm and functionality synthesis\nI am not convinced by the difference between algorithm and functionality synthesis. The definition provided in the paper is very subjective and informal. For example, one can argue that the example of functionality synthesis in Figure 7 involves an idea of using a list comprehension instead of a for loop. In fact, both can be seen as standard text-to-code tasks. This makes me wonder how ALGO performs on HumanEval or MBPP.\n\n### Discussion and comparison with existing work\nThe paper lacks a discussion and comparison with the following existing works. The first work also learns to “verify” generated code with test cases, while the second uses a reviewer to rank generated code.\n\nLEVER: Learning to Verify Language-to-Code Generation with Execution. ICML 2023. https://arxiv.org/abs/2302.08468.\nCoder Reviewer Reranking for Code Generation. ICML 2023. https://arxiv.org/abs/2211.16490.\n\n### Cost analysis\nIt seems to me that ALGO can be significantly more expensive than the baselines. However, the paper does not include any cost analysis of ALGO.\n\n### Instructions\nWhat instructions do you use for the Instruction Enumerator at Line 137? It seems that the choices of instruction can significantly affect the performance.\n\n### Small Issue\nLine 55: InterpreterWe -> Interpreter. We\n\nQUESTIONS:\nPlease consider addressing the points raised in the “Weakness” section.\n\nLIMITATIONS:\nWhile the paper includes a brief discussion of future work at Lines 359-365, it is very vague and does not pinpoint any limitation of ALGO. I believe that ALGO’s limitations, such as high cost, should be stated more clearly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new approach that utilizes Language Models (LLMs) to generate an oracle program for a given task, which is a brute-force search algorithm that is likely correct but potentially suboptimal in terms of runtime. The oracle program is then used to rank candidate programs based on their equivalence to the oracle on set of sampled inputs. The approach is applied to improve the performance of Codex and ChatGPT models and achieves additional improvements compared to the CodeT approach, which generates test cases on the fly and ranks programs based on consistency with those test cases. The proposed method is evaluated on challenging tasks from CodeContest benchmark and on a new set of LeetCode problems.\n\nSTRENGTHS:\nThe paper presents an interesting idea by leveraging the fact that brute-force algorithm based oracles might be easy for LLMs to generate. \nMoreover, compared to neural rankers or the CodeT approach, an oracle-based ranker presents a straightforward and reliable way to check for correctness of the sampled programs by just executing both programs on a set of sample inputs and checking the output consistency.\nThe evaluation includes challenging tasks from CodeContests, providing a robust assessment of the proposed method.\n\nWEAKNESSES:\n- The paper should mention estimates of inference times for all the approaches, including inference times for oracle generation, test generation, and code generation. It is important to discuss the tradeoffs associated with these inference times. For example, if CodeT is given the same inference time budget as ALGO, how does the performance difference change? Additionally, the impact of using an expensive and more capable model (ChatGPT) for oracle generation should be addressed, as it may introduce unfairness in the comparison with other approaches like CodeT.\n\n\n- Please provide more details to enable reproducibility of the results. How many oracles need to be sampled before finding the correct one? What value of n is used for the various experiments? In Figure 4, what is the setup used for ChatGPT and GPT4? How many samples are drawn from each model? Are they are also sampled with different instructions? Including a Pass@k table for various k values would also be helpful for the experiment in Figure 4. \n\n\n- It is unclear from the paper if the failure cases are due to the model's inability to generate any correct program within the n samples or if the oracle is not a good discriminator. Some analysis on whether the \"top-ranked but incorrect programs\" satisfy the oracle or not would help.\n\nQUESTIONS:\nPlease see the weaknesses section for questions and suggestions.\n\nLIMITATIONS:\nDiscuss the limitations regarding the increased inference cost of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ALGO which is a technique for generating programs for algorithmic challenges like LeetCode problems. To be successful, a code solution must be both correct and efficient enough. The core idea in ALGO is to first ask a code LLM to generate a slow brute force solution that is likely to be correct but too slow. This is used as an oracle verifier to check whether other program samples are correct for small test cases. Thus, an existing code generation system can propose solutions which are then filtered or reranked according to agreement with the verifier. The experiments show that ALGO’s technique of using a brute-force solution as a verifier leads to improvements on CodeContests and LeetCode.\n\nSTRENGTHS:\nOriginality: A key observation is that current code LLMs can produce highly-accurate (but slow) brute-force solutions to tricky algorithmic questions. This was not previously obvious to me because one does not usually think about slow brute-force solutions. (I think the authors should emphasize this observation!) Then, assuming the existence of a slow but correct brute-force solution, the ALGO framework is a natural and straightforward application of using a verifier to aid program synthesis.\n\nQuality: The experiments test a variety of ways of using ALGO with different models on different datasets.\n\nClarity: The paper is well organized and easy to read.\n\nSignificance: Improving code generation is certainly significant.\n\nWEAKNESSES:\nOriginality: The process of using a verifier to filter/rerank solutions is pretty straightforward.\n\nQuality: There are some concerns about comparing pass@k with reranking the top k among n samples. The experiments could also compare against other code reranking approaches on both CodeContests and LeetCode. See the Questions below.\n\nClarity: I have a few clarification questions below, but nothing major.\n\nSignificance: The paper only discusses this approach applied to algorithmic challenges, which has only moderate significance. In the Questions section below, I ask the authors to elaborate on how ALGO might be practically useful in other settings.\n\nQUESTIONS:\n## Pass@k given n samples\n\nIn line 216, what does “the same sampling budget $n$ under the same setting” mean? What is the “setting”? What is the actual value of $n$?\n\nIs it really fair to compare pass@k given n samples, to a reranking approach returning the top k among n samples? If we increase n, pass@k does not change in expectation, but the top k approach would improve (assuming the reranking is accurate). For example, AlphaCode (https://arxiv.org/abs/2203.07814) distinguishes these metrics as “pass@k” vs “n@k” (but note that the AlphaCode paper swaps the meaning of n and k). These are different metrics that are not easily compared directly.\n\n\n## Comparison to other reranking approaches\n\nCan ALGO be compared to other code reranking approaches, such as CodeT and Coder-Reviewer (https://arxiv.org/pdf/2211.16490.pdf)? Then, all reranking approaches can use the same “k@n” metric -- whether a good solution exists in the best k among n samples. (I know CodeT is in Table 1, but I was hoping for a more extensive comparison.)\n\n\n## CodeContests vs LeetCode\n\nSome experiments are done with CodeContests and other experiments are done with LeetCode. Why are the main experiments (Table 1, Figure 4) not repeated for both CodeContests and LeetCode?\n\n\n## Verifier correctness\n\nIt is unclear why the “time limit exceeded” (TLE) verdict is counted as “correct” for the verifier. I would agree that TLE means the verifier is “likely correct”, but it would be a leap to assert that all TLE verdicts are “correct” as a key result highlighted in the abstract and throughout the paper. If I’m not misunderstanding, then I think this claim should be rephrased or qualified.\n\nThe “88.5% correct” number (Fig. 2 and throughout the text) is only for LeetCode. Is it possible to get an analogous number for CodeContests?\n\nLines 251 - 253 mention manually verifying whether TLE solutions are functionally correct. What is the result of this? The next sentence says “88.5%” again, but this is exactly the AC + TLE amount from Fig. 2. Does this imply that you manually verified that *every* TLE solution was functionally correct? Please clarify.\n\n\n## Generating inputs\n\nLine 162 says “generate the input generator with an input validator” -- are there examples of this? In particular, the input generator and input validator are both *programs*, which is different from the CodeT approach of hallucinating inputs as text directly from the model, right? If so, this is a main difference that should be explained more thoroughly.\n\nGenerating valid inputs seems tricky, especially when there are problem-specific constraints that could be difficult or annoying to check (e.g., that a solution exists or is unique, that a graph is connected or contains no cycles, etc.). Can you comment on how often the input generator/validator programs work correctly?\n\n\n## Applicability beyond algorithmic challenges\n\nOutside of algorithmic challenges, can the authors comment on other situations were a code LLM can predict a verifier that is correct but undesirable for some reason, which then helps a synthesis search find better solutions? Maybe if the verifier is a lengthy inelegant solution and we want to search for shorter elegant solutions which rely on assumptions/heuristics/invariants that might or might not hold? Maybe the verifier is in a popular language but a solution is desired in an uncommon language?\n\nDo you imagine the approach being applicable to improving efficiency of real-world code? If so, how would ALGO differ from existing approaches to that goal?\n\nHaving an example of something like this, even if not a full experiment, would make the paper seem much more applicable.\n\n\n## Minor suggestions\n\n* “Oracle” usually means something that is always correct. However, the oracles used in this paper are not actually always correct. The ALGO approach simply *assumes* they are correct and treats them *like* an oracle. This distinction was initially not clear to me and only became clear after reading further into the text. Please consider clarifying this in the abstract and introduction.\n* “ChatGPT Code Interpreter” does not come with a citation, so it is extra important to clearly describe what it is, preferably when it is first mentioned (e.g., Line 55). The description in Lines 181 - 187 is good, just move it earlier.\n\n* Line 55: missing space between “Interpreter” and “We”\n* Line 82: What does it mean for a program to be “capable of solving” a problem? The program either passes the judge or not, so it either solves the problem or not.\n* Line 84: “produce an program” -> “produce **a** program”\n* Figure 3 caption: “polynomial time. While” -> “polynomial time, while”\n* Line 127: “We use the variable $I$ to denote the **variable** that the coder **enumerates**” -- it is unclear what the bold “variable” refers to or how the enumeration works. Consider that the prompt from Figure 3 says “What do you think are some **variables** that may affect the answer and how do you think they can be **enumerated**?” but this is definitely a different kind of variable and enumeration. It gets confusing when the same terms are used to describe very different things.\n* Line 145: “A iterative” -> “**An** iterative”\n* Line 168: this sentence implies that the verifier can’t be arbitrary. Why not? Possibly the best verifier comes from Code Interpreter, but would the approach still work with a verifier generated by some other code LLM?\n* Lines 231 - 233: “ChatGPT Code Interpreter is used as an instruction enumerator along with GPT-4’s performance” -- I don’t know what this means\n* Line 322: “Oracle in Program Synthesis” -> “Oracle**s** in Program Synthesis”\n* Line 322: “In traditional” -> “In **the** traditional”\n* Line 325: “due to the ambiguity nature” -> “due to the **ambiguous** nature”\n* Line 327: “in software testing domain” -> “in **the** software testing domain”\n* Line 352: what does “similarity” mean here? Does it make sense to say “based on the ease of generating brute-force solutions to algorithmic problems”?\n* Line 359: “paper. There” -> “paper, there”\n\nLIMITATIONS:\nThe paper is missing a limitations section.\n\nI think the main limitation is in the accuracy of the oracle solution. If the oracle solution is actually wrong, then the ALGO approach will hurt the overall synthesis performance. There doesn’t seem to be an easy way to identify when the oracle solution is wrong. Generating correct inputs also seems tricky and may be limited to problems with simple input formats and constraints.\n\nAnother limitation is in the general applicability of ALGO, since it only works on problems where an existing code LLM can produce (in only 1 sample) a functionally correct oracle solution.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tries to improve code generation ability. The main idea of the paper is as follows\n\n1. Use an LLM to generate a brute force solution to contest coding problems -- the idea being that the hard part for such problems is not solving them but solving them given the time and memory constraints.\n\n2. Have an test case input generator for the problem and get the corresponding output for the test case using the brute force solution in (1)\n\n3. Verify whether a solution is correct using the input/output pairs generated in (1) and return the failing test cases to the code synthesizer\n\nThey test their algorithm using CodeContests and LeetCode and using multiple code synthesizers showing significant performance improvements.\n\nSTRENGTHS:\nA simple but nice idea leveraging the insight that the hard part for contest coding is solving the problem in the time and space constraints. They do a good set of experiments demonstrating the performance improvements. They also have additional analysis on the quality of their brute force solutions (valid 88% of the time for LeetCode)\n\nWEAKNESSES:\nThe main worry I have with this paper is the following :-\n\n*Baselines* \n\nThey use ChatGPT to generate the oracle verifier and show that using it can improve the performance of *weaker* models like Codex and GPT-Neo. However for CodeContests (their primary benchmark), they do not give the baseline of how good ChatGPT itself is at solving the problem. \n\nThis is an issue as it is expected that having a correct verifier is going to enable the selection of the best program out of a list of generated programs. In practice however, we will not have the luxury of having the correct verifier. We will have an LLM and the question before us would be whether generating a verifier using a strong LLM and using it to guide weaker LLMs (or the same strong LLM) is actually better than directly trying to use the strong LLM to generate a solution. Thus that baseline is critical but absent.\n\nFor LeetCode dataset, there is a ChatGPT baseline. However key experimental details are missing. In particular\n\na. What code synthesizer was used for ALGO?\nb. If it was ChatGPT with CodeInterpreter, was the ChatGPT baseline and GPT-4 matched in terms of sampling budget? (If selection of the the best answer out of multiple answers is an issue, then a strategy like CodeT can be used) What was the sampling budget?\n\n\nIn addition, there's a simple baseline that could be tried that is not present -- what if you asked ChatGPT to generate the test input/outputs directly instead of generating a brute force solution?\n\n\n*Temperature of generation*\n\nThere seems to be a temperature mismatch between CodeT and Algo w/ Codex. The results available online for the former from which the authors say they took their results are with temperature 0.8 whereas the authors results for Algo are with temperature 1. This could be putting the former method at a disadvantage as it has lower diversity in the generations.  In addition the temperature used for the other models is not mentioned. If the model does not have a filtering mechanism, then a low or 0 temperature should be used as otherwise the method is going to be at a disadvantage\n\nQUESTIONS:\nIn addition to questions in the Weaknesses section above, I also have the following questions -\n\n1. I can't find the value used for n (sampling budget) in the paper or the number of inputs generated using the oracle. That should be mentioned in the experimental section.\n\n2. How good are the oracle solutions for CodeContests?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}