Title: Parsel🐍: Algorithmic Reasoning with Language Models by Composing Decompositions
Venue: NeurIPS 2023
Accepted: yes
Keywords: reasoning

language models

code synthesis

decomposition
Abstract: Despite recent success in large language model (LLM) reasoning, LLMs struggle with hierarchical multi-step reasoning tasks like generating complex programs. For these tasks, humans often start with a high-level algorithmic design and implement each part gradually. We introduce Parsel, a framework enabling automatic implementation and validation of complex algorithms with code LLMs. With Parsel, we automatically decompose algorithmic tasks into hierarchical natural language function descriptions and then search over combinations of possible function implementations using tests. We show that Parsel can be used across domains requiring hierarchical reasoning, including program synthesis and robotic planning. We find that, using Parsel, LLMs solve more competition-level problems in the APPS dataset, resulting in pass rates over 75\% higher than prior results from directly sampling AlphaCode and Codex, while often using a smaller sample budget. Moreover, with automatically generated tests, we find that Parsel can improve the state-of-the-art pass@1 performance on HumanEval from 67\% to 85\%. We also find that LLM-generated robotic plans using Parsel are more than twice as likely to be considered accurate than directly generated plans. Lastly, we explore how Parsel addresses LLM limitations and discuss how Parsel may be useful for human programmers. We release our code at https://github.com/ezelikman/parsel.

Parsel : Algorithmic Reasoning with Language Models by Composing Decompositions

Abstract

Despite recent success in large language model (LLM) reasoning, LLMs struggle with hierarchical multi-step reasoning tasks like generating complex programs. For these tasks, humans often start with a high-level algorithmic design and implement each part gradually. We introduce Parsel, a framework enabling automatic implementation and validation of complex algorithms with code LLMs. With Parsel, we automatically decompose algorithmic tasks into hierarchical natural language function descriptions and then search over combinations of possible function implementations using tests. We show that Parsel can be used across domains requiring hierarchical reasoning, including program synthesis and robotic planning. We find that, using Parsel, LLMs solve more competition-level problems in the APPS dataset, resulting in pass rates over 75% higher than prior results from directly sampling AlphaCode and Codex, while often using a smaller sample budget. Moreover, with automatically generated tests, we find that Parsel can improve the state-of-the-art pass@1 performance on HumanEval from 67% to 85%. We also find that LLM-generated robotic plans using Parsel are more than twice as likely to be considered accurate than directly generated plans. Lastly, we explore how Parsel addresses LLM limitations and discuss how Parsel may be useful for human programmers. We release our code at https://github.com/ezelikman/parsel .

1 Introduction

To a language model for code (as for a human), each new token is a new chance to break the program. Chen et al. [12] highlighted this issue in a simple synthetic experiment: when asked to generate a program with a series of simple string transformations, the performance of their code language model, Codex, drops dramatically with the number of steps. As they pointed out, a human who can implement a few building blocks should be able to compose these blocks with arbitrary length.

Unlike other token generators, human programmers have (mostly) learned to break down complex tasks into manageable parts that work alone (i.e., are modular) and work together (i.e., are compositional). And, when human-generated tokens cause functions to break, the functions can ideally be rewritten independently of the rest of the program. In contrast, naively, we expect code language models to generate token sequences that are correct in their entirety. Motivated by this, we study how to leverage language models to decompose problems and assemble their compositional solutions.

We introduce Parsel, a framework that enables the decomposition, implementation, then composition of complex programs from natural language. The process consists of three main steps: 1) A language model generates a Parsel program by decomposing a natural language task into natural language function descriptions. 2) For each function description, a language model generates modular implementations. 3) The Parsel synthesizer then builds up programs by testing minimal, combinatorial groups of implementations against sets of constraints (e.g. input-output examples).

Decompose

Problem

or

or

Parsel

Synthesizer

Parsel

Program

or

Thus, generating and implementing Parsel mirrors a pattern in human reasoning - decomposing an abstract plan until it can be solved automatically [55] - and this compositional structure also benefits language models. We show that LLMs can generate Parsel with only few-shot prompting. Using their solutions as input to the Parsel synthesizer, we outperform prior work on competition-level problems in the APPS dataset [27], including AlphaCode [35] and both versions of Codex [12, 11]. We also show that with GPT-4 [44], not only can we generate Parsel zero-shot just by describing its format, but this allows it to substantially outperform GPT-4 alone on HumanEval [12], with and without generated tests. By automatically generating and implementing Parsel, LLMs also propose robotic plans that are more accurate than in prior work on language models as robotic planners [28]. Broadly, we formulate Parsel as a general-purpose algorithmic reasoning framework.

2 Methods

In this section, we provide an overview of the Parsel framework. First, we show how programs can be specified in Parsel. Afterward, we explain how language models can generate Parsel from task descriptions. Finally, we present the Parsel synthesizer that allows us to actually implement Parsel programs into runnable code. Specifically, we further discuss how we handle various challenges in solving the constraints, from recursion to test underspecification. There are several steps to synthesizing a Parsel program, which we explicate in this section. We visualize the details in Fig.1 and provide a high-level pseudocode in Fig. A.11.

With Parsel, we balance two challenges. On one hand, function implementations must be modular to fully leverage combinatoriality. By requiring function signatures and descriptions to be specified (or generated), we ensure functions can call others without specific implementation information. On the other hand, it must be possible to avoid combinatorial explosions. By factoring the dependency graph into strongly-connected components (SCCs), we ensure the complexity grows exponentially only with the size of the largest strongly-connected component but not the total number of functions.

2.1 Specifying Parsel Programs

To formally specify a high-level algorithmic design, we develop a simple intermediate language, also called Parsel. It is designed to be accessible to programmers, code LLMs, and students, as discussed further in Appendix A, and inspired by many works (Section 4). In Parsel, each line contains a function description, a constraint, or a reference to a description. They all obey scoping rules, with some nuances per target language. In short: descriptions specify what a function does in natural language, optionally preceded by a function name and its arguments (if not included, they are inferred); references indicate to the language model that a function should call another function described elsewhere; constraints validate that a function is implemented correctly (if not provided, they may be generated automatically using the language model). We elaborate on syntactic details in Appendix R. We provide some examples of Parsel programs in Figures 2 and 3. For example, in Figure 2, task_plan(): return a list of strings ... represents a description, while the call to collatz_recursion by recursion_rule is a reference.

Figure 2: Examples using Parsel to compile programs for VirtualHome [50] procedures and Python. Note the columns on the generated examples are only there to allow them to fit compactly - each program implementation is one contiguous solution. Colors for Parsel code on the left are used to indicate constraints and references. All other Parsel lines shown are definitions.

2.2 Generating Parsel Programs

Weuse a language model to generate Parsel programs from arbitrary natural language task descriptions as a few-shot translation task (or zero-shot with GPT4 [44]). Explicitly writing out a preliminary plan can be helpful, so we first ask the model to (zero-shot) generate a plan in natural language by thinking step-by-step (motivated by Kojima et al. [32]) and then prompt the model to translate the plan to Parsel. These steps correspond to the first and second arrows in Figure 3, respectively. In some domains, in particular for robotic planning, we find that this intermediate step is not necessary (while increasing cost) and can ask the language model to directly translate the task description to a Parsel solution.

2.3 Implementing Parsel Programs

Given the Parsel program, a core technical challenge is how to implement it in a modular way (to fully leverage the advantage of decomposition). Here we propose the Parsel synthesizer: At a high level, to implement Parsel programs with the Parsel synthesizer, we first use a language model to generate a set of candidate implementations of each of the functions based on their descriptions and then search over minimal sets of combinations of the functions to find ones that satisfy the provided constraints. The step described in this section corresponds to the rightmost arrow in Figure 3.

2.3.1 Implementing Functions

A function is implemented by first aggregating descriptions and function signatures of its (potentially zero) children to form an LLM prompt (for Python, we generate a prompt as if the child functions are imported and use their descriptions as comments). Crucially, this facilitates easily changing child implementations. A code LLM is then queried using the description's text as a docstring and the description's function name and arguments for the signature; full prompts shown in Appendix K.

2.3.2 Searching Over Function Combinations

Sequential case. The most straightforward case is when all functions have constraints (e.g., unit tests), and no functions have recursive dependencies (e.g., Fig. A.9). We start by considering this case. This defines a clear topological order on functions so they can be implemented sequentially. In this situation, Parsel implements functions with post-order traversal from function at the root, generating implementations and finding one passing the specified constraints for each function. In

Task

High-level Sketch

Parsel Program

max_sum_of_jewels(input_str)

: Returns

the maximum sum of values of jewels that

can be stolen for each knapsack size.

parse_input(input_str)

: Takes a string

containing the number of jewels and

knapsack sizes on the first line, the

jewels on the next lines, and returns

the number of jewels, the knapsack

sizes, and the jewels.

max_sum_of_jewels_for_size(

jewels, size)

: Returns the maximum sum

of values of jewels that can be stolen

for each knapsack size.

max_sum_of_jewels_for_size_and_jewel(

jewels, size, jewel)

: Returns the

maximum sum of values of jewels that

can be stolen for each knapsack size.

to_output_str(

max_sum_of_jewels_for_size)

: Returns

the string of the maximum sum of values

of jewels that can be stolen for each

knapsack size.

Program

# Takes a string containing the number of jewels and knapsack sizes on the first

line, the jewels on the next lines, and returns the number of jewels, the

knapsack sizes, and the jewels.

def parse_input(input_str):

# [Omitted for space.]

return num_jewels, knapsack_sizes, jewels

# Returns the maximum sum of values of jewels that can be stolen for each

knapsack size.

def max_sum_of_jewels_for_size_and_jewel(jewels, size, jewel):

if size < 0:

return 0

return 0

return jewels[jewel][1] if jewels[jewel][0] <= size else 0

elif jewels[jewel][0] > size:

return max_sum_of_jewels_for_size_and_jewel(jewels, size, jewel-1)

else:

max_sum_of_jewels_for_size_and_jewel(jewels,

size-jewels[jewel][0], jewel-1) + jewels[jewel][1])

# Returns the maximum sum of values of jewels that can be stolen for each

knapsack size.

def max_sum_of_jewels_for_size(jewels, size):

result = []

for s in range(1, size + 1):

result += [max_sum_of_jewels_for_size_and_jewel(jewels, s, len(jewels) -

1)]

return result

# Returns the string of the maximum sum of values of jewels that can be stolen

for each knapsack size.

def to_output_str(max_sum_of_jewels_for_size):

return " ".join(map(str, max_sum_of_jewels_for_size))

# Returns the maximum sum of values of jewels that can be stolen for each

knapsack size.

def max_sum_of_jewels(input_str):

number_of_jewels, knapsack_sizes, jewels = parse_input(input_str)

return to_output_str(max_sum_of_jewels_for_size(jewels, knapsack_sizes))

First, we need to figure out how to calculate

the maximum sum of values of jewels for each

size of  knapsack. To do this, we can use a

dynamic programming approach.

We can create a two-dimensional array of size

n×k where each row represents the different

jewels and each column represents the different

sizes of  knapsack. The value of  each cell in the

array will represent the maximum sum of

values of  jewels that can be stolen for a given

knapsack size.

We can then iterate through the array, starting

at the beginning. For each row, we can calculate

the maximum sum of values of jewels that can

be stolen for each knapsack size by taking the

maximum of the value of the current cell and

the value of  the previous cell plus the value of

the current jewel.

At the end, we will have an array that contains

the maximum sum of values of jewels that can

be stolen for each knapsack size. We can then

output these values as our answer.

… [continued in appendix]

The grand museum has just

announced a large exhibit on

jewelry from around the world. In

the hopes of his potential future

prosperity, the world-renowned

thief and master criminal Edward

the magnum opus of his career in

thievery.

Edward is hoping to purloin a

large number of jewels from the

exhibit at the grand museum. But

alas! He must be careful with

which jewels to appropriate in

order to maximize the total value

of jewels stolen.

Edward has $k$ knapsacks of size

$1$, $2$, $3$, up to $k$, and

would like to know for each the

maximum sum of values of jewels

that can be stolen. This way he

can properly weigh risk vs.

reward when choosing how many

jewels to steal. A knapsack of

size $s$ can hold items if the

sum of sizes of those items is

less than or equal to $s$. If you

can figure out the best total

value of jewels for each size of

knapsack, you can help Edward

pull off the heist of the

century!

… [continued in appendix]

The grand museum has just announced a large exhibit on jewelry from around the world. In the hopes of his potential future prosperity, the world-renowned thief and master criminal Edward Terrenando has decided to attempt the magnum opus max_sum_of_jewels(input_str) : Returns the maximum sum of values of jewels that can be stolen for each knapsack size. parse_input(input_str) : Takes a string containing the number of jewels and # Takes a string containing the number of jewels and knapsack sizes on the first line, the jewels on the next lines, and returns the number of jewels, the knapsack sizes, and the jewels. def parse_input(input_str): # [Omitted for space.] return num_jewels, knapsack_sizes, jewels # Returns the maximum sum of values of jewels that can be stolen for each knapsack size. def max_sum_of_jewels_for_size_and_jewel(jewels, size, jewel): First, we need to figure out how to calculate the maximum sum of values of jewels for each size of knapsack. To do this, we can use a dynamic programming approach. We can create a two-dimensional array of size $n \times k$ where each row represents the different jewels and other words, without any cycles in the call graph, we can start by implementing the leaf functions first, then their parents, etc., until the program is implemented. However, in practice, many programs have more complex structures and constraints.

knapsack sizes on the first line, the of his career in thievery.

if size < 0:

each column represents the different sizes of  knapsack.

Edward is hoping to purloin a large number of jewels from the exhibit at the grand museum. But alas! He must be careful with which jewels to appropriate in order to maximize the total value of jewels stolen. Edward has $k$ knapsacks of size $1$, $2$, $3$, up to $k$, and would like to know for each the maximum sum of values of jewels that can be stolen. This way he can properly weigh risk vs. reward when choosing how many jewels to steal. A knapsack of size $s$ can hold items if the sum of sizes of those items is less than or equal to $s$. If you can figure out the best total value of jewels for each size of knapsack, you can help Edward pull off the heist of the century! … [continued in appendix] jewels on the next lines, and returns the number of jewels, the knapsack sizes, and the jewels. max_sum_of_jewels_for_size( jewels, size) : Returns the maximum sum of values of jewels that can be stolen for each knapsack size. max_sum_of_jewels_for_size_and_jewel( jewels, size, jewel) : Returns the maximum sum of values of jewels that can be stolen for each knapsack size. to_output_str( max_sum_of_jewels_for_size) : Returns the string of the maximum sum of values of jewels that can be stolen for each knapsack size. return 0 elif jewel < 0: return 0 elif jewel == 0: return jewels[jewel][1] if jewels[jewel][0] <= size else 0 elif jewels[jewel][0] > size: return max_sum_of_jewels_for_size_and_jewel(jewels, size, jewel-1) else: return max(max_sum_of_jewels_for_size_and_jewel(jewels, size, jewel-1), max_sum_of_jewels_for_size_and_jewel(jewels, size-jewels[jewel][0], jewel-1) + jewels[jewel][1]) # Returns the maximum sum of values of jewels that can be stolen for each knapsack size. def max_sum_of_jewels_for_size(jewels, size): result = [] for s in range(1, size + 1): result += [max_sum_of_jewels_for_size_and_jewel(jewels, s, len(jewels) 1)] return result # Returns the string of the maximum sum of values of jewels that can be stolen for each knapsack size. def to_output_str(max_sum_of_jewels_for_size): return " ".join(map(str, max_sum_of_jewels_for_size)) # Returns the maximum sum of values of jewels that can be stolen for each knapsack size. def max_sum_of_jewels(input_str): number_of_jewels, knapsack_sizes, jewels = parse_input(input_str) return to_output_str(max_sum_of_jewels_for_size(jewels, knapsack_sizes))/ The value of each cell in the array will represent the maximum sum of values of jewels that can be stolen for a given knapsack size. We can then iterate through the array, starting at the beginning. For each row, we can calculate the maximum sum of values of jewels that can be stolen for each knapsack size by taking the maximum of the value of the current cell and the value of  the previous cell plus the value of  the current jewel. At the end, we will have an array that contains the maximum sum of values of jewels that can be stolen for each knapsack size. We can then output these values as our answer. This algorithm is correct because we are calculating the maximum sum of values of jewels that can be stolen for each knapsack size by comparing the values of the current cell and the previous cell plus the value of  the current jewel. This ensures that we are taking the maximum value for each knapsack size. Recursion. Permitting mutual recursion (e.g. a function f that depends on g while g also calls f ) introduces the need for joint implementation of functions. However, naively considering all possible implementations is intractable for large programs. Specifically, the number of possible implementations of a program with k functions and n implementations per function is O ( n k ) , exponential in the number of functions. We propose a more efficient solution, inspired by Cocke and Kennedy [16]. We reference the function call graph, identify its strongly-connected components (SCCs), and for each SCC consider all possible sets of function implementations until one set satisfies all constraints. For example, for possible implementations of functions f, g, h forming an SCC of a call graph, with I ( f ) corresponding to the language-model-generated implementations of f , we evaluate uniformly random samples without replacement from I ( f ) × I ( g ) × I ( h ) .

Functions without constraints. For functions with no constraints, we can conveniently use the same approach as above by reformulating the call graph as a 'test dependency graph' and adding an edge from a test-less child to its parent. That is, if a function has no constraints, it depends on its parents to enforce constraints on its implementations. This could also allow us to automatically introduce new children via automatic decomposition without needing to also generate constraints for those children (see Subsection R.3). Alternatively, we support automatic test generation, prompting the language model to generate tests. For generated tests, we select the best implementation based on CodeT score [11], as some generated tests could be incorrect. In practice, we use the first (dependency) approach for inner functions and the second (test generation) approach when the top-level function has no tests. Note an opposite extreme from the sequential case: if no function or only the top-level function has constraints, we simply implement each function some number of times and then test all combinations. More generally, if n functions with m implementations where the sizes of the SCCs are at most c , partitioning reduces necessary evaluations from O( n m ) to O( n c m c ).

3 Experiments

We explore the ability of Parsel to generate programs for various tasks, including competitive coding, a standard coding benchmark, and robotic task planning. These three categories represent related but distinct kinds of algorithmic reasoning, with varying levels of abstractness and generalizability. Competitive programming represents one of the few benchmarks for evaluating code generation tools that benefits greatly from decomposition. 1 By evaluating Parsel on this breadth of tasks, we hope to better understand its generality as a framework for algorithmic reasoning.

1x4

1x8

2x8

4x8

8x8

8x16

1000 (Original)

50 (Improved)

5@50000

1000

Sample Budget (# of Complete Programs Inferenced)

Competition-Level APPS Pass Rate

0%

10%

20%

30%

1

10

100

1000

10000

100000

Parsel (Ours)

Codex (Direct)

AlphaCode 1B (Direct)

Total Evaluations per Problem

0%

10%

20%

30%

100

1000

10000

100000

3.1 Python Code Generation

Solving competition-level problems. We evaluated Parsel on the competition-level subset of the APPS [27] dataset as follows: first, we zero-shot prompt GPT-3 (text-davinci-002) with the problem statement, requiring it to first propose a high-level solution and explain why it is correct, asking it to 'think step by step to come up with a clever algorithm' [32]. We then prompt Codex [12] to translate the generated solution into Parsel code, providing three examples of valid code. We then attempt to synthesize the Parsel code and evaluate implementations to measure their pass rate, i.e., the proportion of solved problems. We report 'pass@ n × k ', where n is the number of Parsel (i.e. intermediate, semi-formal) programs generated and k is the number of implementations per function. Prior work has emphasized performance as a function of the LLM sample/inference budget measured in complete programs generated [12, 35]; for Parsel, we use the effective number of complete programs generated from the LLM for comparison, which is n × k . We compare to directly generating solutions from Codex [12, 11] and AlphaCode [35]. We visualize the pass rate in Figure 4, finding that Parsel substantially improves over prior work [11], from 14.5% to 25.5%. We include full prompts and further evaluations in Appendix N and discuss the evaluation runtime costs in depth in Appendix T.8.

Since Parsel will 'mix-and-match' function implementations, the number of distinct candidate programs available to test grows exponentially with k , while the generation cost for the components is linear. Generating each program likely requires on the order of petaFLOPs of compute [21], dwarfing the computational cost of testing complete programs. Yet when few functions have constraints (as in this evaluation) or there are complex recursive dependencies, it could become expensive to exhaustively evaluate all possible programs. Thus, we also explore the performance of Parsel as a function of the number of evaluations, as shown in Fig. 4. We find that the performance improves log-linearly with the number of evaluations, justifying spending compute to evaluate all programs implied by generated function implementations. One can interpret this to suggest the number of Parsel programs and combined implementations play similar and important roles. Thus, one may compensate for implementation ability with Parsel-generation ability and vice-versa (to a limit).

Ablating the Parsel synthesizer. We perform an ablation to explore the impact of the Parsel intermediate language: we provide the same high-level plans to Codex, but instead of translating them to Parsel and synthesizing programs as decribed, we translate them to Python directly. We match the code generation budget (effectively complete programs inferenced), generating 16 Python implementations 2 per high-level plan on 100 randomly sampled problems, and find that the pass@ 8 × 16

performance drops to 6% from 25.5% for the full pipeline. Not only did this approach solve substantially fewer problems, but the problems that it did solve were a strict subset of those which we were able to solve with Parsel. This indicates that for Parsel with Codex, the step-by-step decomposition provided by the high-level plan is not by itself responsible for the observed improvements by Parsel.

We additionally perform an ablation to understand how Parsel's performance without a high-level plan. On a 200-problem competition-level APPS sample, we found that, without a high-level plan, the accuracy falls to 13% (from 25.5%, with the same configuration). This is better than the ablation where Parsel wasn't used, suggesting that Parsel plays a larger role than the high-level plan on APPS, but both are necessary. Note part of this improvement is from the challenge of prompting Codex to generate Parsel, given long APPS problem statements.

HumanEval. We next tested Parsel on HumanEval[12]. Because these problems are substantially simpler, they provide an opportunity to evaluate pass@1 by generating tests automatically. Furthermore, this smaller and simpler set of problems enables us to assess the (much more costly) GPT-4. We found a significant improvement over the state-of-the-art pass@1 performance-from 67% to 85%. We found that by describing the Parsel format (see Appendix O for details), noting the role of indentation in providing hierarchy and explaining the format of a description, GPT-4 was able to translate natural language plans into Parsel zero-shot. Moreover, the combination of Parsel and GPT-4 surpassed GPT-4 alone on HumanEval, both with and without generated tests, further emphasizing the importance of leveraging intermediate languages like Parsel for improved code generation performance in complex tasks.

The 85.1% pass rate with generated tests, evaluated based on the CodeT score [11], was obtained by sampling until three Parsel programs had one implementation passing a generated test and then selecting the one passing the most tests. On average, this required 3.5 Parsel programs per problem, with 8 implementations each (i.e. < 32 complete implementations on average). Parsel also outperforms GPT-4 with CodeT alone on the same set of tests, and using a comparable 32 samples, which passes 81.1% of the problems.

Additionally, when evaluating performance in terms of pass@any (i.e., not filtering attempts by CodeT), we discovered that by allowing up to 8 Parsel programs (still with 8 implementations each), the pass rate increased to 91.5%. In contrast, when allowing up to 64 directly generated programs, the pass rate was only

75%

85%

95%

pass@1 (CodeT)

pass@any (64 programs)

pass@any (128 programs)

+ Parsel

82.3%. This difference becomes more pronounced when generating 128 programs (128 directly vs. 16 × 8 with Parsel), improving from 84.1% to 93.9% (See Fig. 5).

While there are still eight tasks that are never solved with or without Parsel, GPT-4 solves eighteen problems with Parsel that it cannot solve alone. For eight of these, the solution has six or more functions (excluding defined but unused functions). Of the problems Parsel solved, the ones GPT-4 could not solve itself required 4.2 functions on average, while those GPT-4 could solve required only 2.8 functions (sampled over 50 problems). We find GPT-4 alone solves two problems pass@any that GPT-4 with Parsel cannot: decimal_to_binary and decode_cyclic . The decode_cyclic solutions match Chen et al. [12]'s Figure 2 (but not the dataset's canonical solution) with minor comment differences, indicating possible contamination. In decimal_to_binary , the generated Parsel solutions often unsuccessfully rely on statefulness: for example, divide_by_2_with_remainder calls calculate_remainder and update_decimal , but update_decimal cannot mutate an integer.

Most crucially, these experiments indicate that, even when given the same set of generated tests and program generation budget and selecting only one best solution, Parsel significantly increases the probability that the solution is correct on the ground-truth set of tests.

Pass Rate on HumanEval

Comparison to a human expert. In our fully automated pipeline, an LLM generates a Parsel program, which Parsel attempts to synthesize into a solution. We also evaluated how well an expert Parsel user would perform by directly writing Parsel and interacting with the Parsel synthesizer. As a case study, one of our authors with significant prior experience in competitive programming was presented with 10 randomly-selected competition-level Codeforces problems from the APPS dataset, writing Parsel from scratch in order to solve the problems The participant successfully solved 5 of the problems within a 6-hour time frame.

In comparison, when GPT-3 generated Parsel solutions to these problems, it had a success rate of 2 out of 10 problems with 8 attempts per problem (of course, in a much shorter time-frame). This result suggests that, given a suitable decomposition, the Parsel synthesizer can effectively generate complete correct solutions to hard problems - thus, a major point for improvement in the fully automated pipeline lies in the first stage (Parsel generation). In other words, improving the ability of models to decompose tasks into Parsel programs is a key bottleneck and is an important direction for future work.

While the participant could solve problems that GPT-3 could not, the participant found some aspects of working with Parsel counterintuitive. For example, one of the most effective ways to provide additional details to ensure a working solution was to write additional tests rather than to clarify the meaning in the function descriptions. Given the stochasticity of language model generation, there was concern that changing the description of a child function could break its parent - however, in practice, this seemed rare. We include the participant's solutions and the associated problems in Appendix Q.

Chained components . Chen et al. [12] highlight in their discussion of limitations that language models fail to produce code that requires the simple composition of building blocks. We replicate their experiment with Parsel, using the same docstring and providing the same descriptions as function descriptions. In Fig. 6, we visualize how Parsel solves this. We find that as the number of required components grows, even interchanging the components of only two complete programs (1x2) solves more problems than Codex solves with 16 programs. This illustrates a key point: assuming any independence, the complete set of combinatorial combinations of n implementations of a program's functions is clearly more likely to be correct than n samples from that same set. Moreover, this suggests that a framework like Parsel may support increasingly complex and abstracted programs as language models improve.

3.2 Robotic Planning in VirtualHome

Number of Chained Components

0%

25%

50%

75%

100%

1

2

3

4

5

6

7

1x16

1x8

1x4

1x2

1x1

Codex (Orig.)

Codex (Curr.)

Codex (@16)

We perform a study on VirtualHome [50] to demonstrate that Parsel can also be used for complex robotic planning. VirtualHome is a simulation environment consisting of households with various interactive objects and agents. VirtualHome includes a small set of permissible actions with a strict grammar - tasks like 'paint ceiling' may require multiple levels of decomposition, e.g., finding and collecting specific painting supplies, finding and using a ladder, and using the painting supplies, each requiring further decomposition.

To test the effectiveness of Parsel in this domain, we investigate whether Parsel could generate programs to solve tasks in the VirtualHome environment, while using the environment to provide feedback on whether the plan is executable. Specifically, we use Parsel to generate a Python program that can generate action plans in natural language similar to ones used in Huang et al. [28]. In each specified constraint, the produced natural language action plan is translated to formal VirtualHome instructions with minimal regex matching and tested executability. If the instructions can be successfully executed, they are considered valid - however, a potential enhancement could be the inclusion of object-relational constraints on the subsequent state of the world. We include an example of a Parsel program that successfully executed and decomposed a task in VirtualHome in Figure 2. Note we also used a header describing a valid action plan, shown in Figure A.45.

However, as pointed out by [28], while it is easy to check plan executability, it is harder to check correctness, because there are generally many valid ways of accomplishing realistic tasks. Thus, like Huang et al. [28], in order to evaluate the accuracy of the executable plans, we perform a human study. Specifically, we ran two surveys on Prolific [45] where we asked 20 participants to make five rankings each, for a total of 100 rankings per survey. In one survey, we asked participants to rank a set of solutions by how accurately each solution accomplishes the task while in another we asked about how understandable the solution was. Two versions of distinct Parsel solutions were shown (where multiple executable Parsel solutions were available), one which included the indented function names used to solve the tasks alongside their step-by-step solutions, and one which only included step-bystep output. We compared both Parsel solutions to an also-executable baseline solution, based on Huang et al. [28]. We include more details about the survey format and executability in Appendix P.

Humans ranked the Parsel solutions as more accurate than the baseline. In each ranking, both the indented and non-indented Parsel solutions were consistently ranked higher than the baseline. In accuracy, standard Parsel solutions (as well as indented solutions) were identified as more accurate than the baseline solutions in 69% of comparisons, more than twice as often as the baseline. In clarity, the standard Parsel solution was 70% more likely to be preferred over the baseline while the indented Parsel solution was 50% more likely to be preferred compared to the baseline. There was no notable difference between indented and non-indented Parsel solutions in either accuracy or clarity. Specifically, note that all of these comparisons are reported pairwise, and Figure 7 compares the rankings given to the (non-hierarchical) Parsel-generated plan and the Codex-generated solution.

In essence, participants did not appear to prefer either Parsel plan format over the other, but given a choice between one Parsel-generated

Clarity

0%

20%

40%

60%

80%

Preference Probability

Codex

+ Parsel

plan and its corresponding Codex-generated plan, they typically considered the Parsel-generated plan to be more accurate and clearer.

4 Related Works

Step-by-step problem solving with LLMs. Many works show that step-by-step reasoning benefits LLMperformance [51, 54, 42, 60, 37, 34] and correspondingly, that this performance can be improved with guidance and tool use [71, 68, 63, 59, 22]. Acquaviva et al. [2] encouragingly showed that humans, when asked to explain how to solve problems in the Abstract Reasoning Corpus [14], tended to provide step-by-step hierarchical descriptions with many verification steps. Moreover, Wies et al. [61] presents a theoretical argument showing problems that can be learned efficiently if decomposed but require exponentially many examples w.r.t. length if not decomposed.

Program synthesis. Program synthesis is the long-standing challenge of generating programs from high-level specifications [26], such as input-output examples [7, 25] and/or natural language descriptions [52, 65, 19]. Program synthesizers typically search the exponentially large space of programs. Consequently, synthesizing large, complex programs remains an open challenge. Recently, library learning has shown a way to make progress: even complex programs can be short in terms of the right high-level library. In turn, this library can be progressively induced from solutions to simpler synthesis problems. This idea is embodied in DreamCoder [23, 9]. Library learning requires a rich distribution of related tasks so that patterns emerge from solutions to simple problems. Patterns are abstracted into useful library functions, enabling short solutions to more complex problems. Parsel also aims to synthesize complex programs by decomposing them. However, Parsel programs specify decompositions, so a family of related tasks is not required.

Log Parsel Program Samples

APPS Competition Pass Rate

0.0

0.1

0.2

0.3

0.0

0.5

1.0

1.5

2.0

nx1

nx8

nx16

Log Implementation Samples

APPS Competition Pass Rate

0.0

0.1

0.2

0.3

0.0

0.5

1.0

1.5

2.0

2.5

1xn

2xn

3xn

4xn

5xn

6xn

7xn

8xn

LLMs for formal environment multi-step planning. Also encouragingly, several existing works can be expressed in Parsel. For example, Huang et al. [28] and Brohan et al. [10] showed that language models can automatically generate step-by-step algorithms for robotic agents in language. In both cases, the generated language corresponds directly to pre-implemented low-level robotic abilities. This could be expressed by providing a task description and constraints evaluating that the high-level task was completed successfully. In addition, Jiang et al. [30] proposed a framework to generate formal proofs in formal theorem-proving languages from informal proofs by first generating an intermediate natural language proof sketch. This could be expressed in Parsel by generating each sketch step as a function and using formal verification for each lemma as the Parsel validation step.

Programming languages and frameworks incorporating language models. Other works have explored programming languages that incorporate language models. For example, Cheng et al. [13] explored the introduction of a language-model-based evaluation function, which would allow f('North America?', 'U.S.') to automatically return 'yes' by referencing the knowledge of the language model, and showed that they could also generate programs using this tool with a language model and Beurer-Kellner et al. [8] explored a related SQL-style LLM-querying language. In addition, Dohan et al. [20] presents an inference-focused framework for language models more broadly for probabilistic graphical models composed of language model actions. Unlike LM Cascades [20], we primarily focus on the constrained generation of programs, instead of leveraging language models as functions within a particular program.

Testing code language model outputs. Related works have explored the capacity of assert statements to constrain the generation space of code LLMs on functions [6, 12, 35]. The automatic generation of unit tests in Chen et al. [11] allowed them to filter many programs before performing final evaluations, significantly improving the pass rate per evaluation. CodeT [11] applies a language model to generate a large number of tests, presuming that some of them will be incorrect. However, instead of simply selecting the program passing the most tests as correct, CodeT groups together generated programs by the subset of the tests that they pass - it then returns any program from the group with the most tests passed in total [11]. Chen et al. [11] intuitively motivates this by the Anna Karenina principle: 'Happy families are all alike; every unhappy family is unhappy in its own way.'

Notably, both the number of final evaluations and the sample budget are important, and we find that combining Parsel with automatic test generation improves performance given a fixed sample and evaluation budget. In addition, Merrill et al. [38] proves essential constraints on what can be learned from assertions alone and, more crucially, what cannot.

5 Discussion and Limitations

We note that Parsel has important limitations, largely stemming from its dependence on closed LLMs. First, LLMs naturally underperform on languages underrepresented in their training data, like Parsel, affecting the quality and reliability of the generated code, and these closed models do not offer a training API. In addition, reliance on Codex and GPT-4 introduces vulnerabilities due to potential abrupt changes in behavior - since this project's start, access to Codex (which is free) has become much more limited and the much-better GPT-4 is also much more expensive. We unfortunately found a 2.7B CodeGen model [39], evaluated like Codex on APPS, solved no problems. However, we anticipate that improvements in open-source code LLMs will mitigate these issues.

Despite the limitations, elaborated in Appendix B, and the challenges of using a language model to generate a language that it has never seen before, the generated Parsel was able to implement robust algorithmic reasoning. There are a few natural questions that arise from this work. How robust is Parsel to the language used to specify the problem? When solving hard problems, is it better to increase the number of Parsel programs sampled or the number of program implementations sampled? We visualize Parsel's performance on the explored subset of APPS in Figure 8 to try to investigate some of these questions. In general, it appears that the probability of passing increased quadratically (R 2 ≥ 0.99 for all curves) with respect to the log of both the number of Parsel programs sampled and the number of implementations sampled. Clearly, since the pass rate is bounded between 0 and 1, this trend cannot continue indefinitely. However, given the rate limit associated with calls to the Codex API (officially, 40,000 tokens or 20 requests per minute, but in practice, we consistently ran into the rate limit with ≈ 10% of that usage), we were not able to identify the inflection point. Li et al. [35] indicated the pass rate curves become log-linear, but we cannot guarantee this holds for Parsel.

Furthermore, we highlight that the question we (and, in our view, most prior work) were looking at in our APPS experiments is, given hard problems and limited by one's generation budget, what is the chance of finding a correct solution? However, if one instead views each test as a successive submission, where any failure is comparable to a crash in production, one would likely take an approach like the one in our HumanEval test generation experiments, one that uses interpreter feedback to revise solutions, or perhaps even one that uses the language model to simulate the result of tests. Both interpretations are important, but they highlight different concerns: even if one can guarantee success upon submission given a trillion generated tokens, this has limited practical applicability. A method's ability to efficiently generate reasonable solutions in the first place is one key bottleneck, and assuming an imperfect model, the ability to identify them is another.

Finally, we highlight that Parsel is already capable of synthesizing larger, more complex programs. For reference, we include a working Lisp interpreter in Appendix I, based on Norvig [41], which we translated to Parsel with 21 functions. We make sure to include no explicit references to Lisp in the Parsel code and provide a header to tell Parsel to keep track of environments as dictionaries. However, this is clearly an imperfect substitute for a true object-oriented programming implementation, and we believe adding better object-oriented features to Parsel would be a valuable future direction.

6 Conclusion

We hope that Parsel provides a broadly useful framework for several groups: for programmers, Parsel should provide a language for robust code generation without the need to evaluate the underlying code; for students, Parsel should allow the teaching of algorithmic reasoning with less emphasis on syntax and more emphasis on problem-solving, similarly to a mathematics curriculum; for language models, Parsel should facilitate hierarchical task decomposition.

Ultimately, as we discuss in detail in Appendix C, Parsel has many natural extensions building on prior work in code synthesis from automatic test case generation [17, 12], reranking solutions by model-judged quality [69], more clever recursive decomposition [49], bootstrapping increasingly complex programs [4, 43, 68], to generating language model prompts as part of a target language [20, 31]. As we discuss in Appendix D, we also envision that Parsel may be useful for theorem proving. Thus, Parsel helps fill the gap between reasoning and execution, providing a new approach to complex, hierarchical reasoning tasks for language models.

References

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/ . Software available from tensorflow.org.

S. Acquaviva, Y. Pu, M. Kryven, T. Sechopoulos, C. Wong, G. E. Ecanow, M. Nye, M. H. Tessler, and J. B. Tenenbaum. Communicating natural programs to humans and machines. NeurIPS , 2022.

C. Anil, Y. Wu, A. J. Andreassen, A. Lewkowycz, V. Misra, V. V. Ramasesh, A. Slone, G. GurAri, E. Dyer, and B. Neyshabur. Exploring length generalization in large language models. In Advances in Neural Information Processing Systems .

T. Anthony, Z. Tian, and D. Barber. Thinking fast and slow with deep learning and tree search. Advances in Neural Information Processing Systems , 30, 2017.

B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, et al. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868 , 2022.

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021.

M. A. Bauer. Programming by examples. Artificial Intelligence , 12(1):1-21, 1979.

L. Beurer-Kellner, M. Fischer, and M. Vechev. Prompting is programming: A query language for large language models. arXiv preprint arXiv:2212.06094 , 2022.

M. Bowers, T. X. Olausson, C. Wong, G. Grand, J. B. Tenenbaum, K. Ellis, and A. Solar-Lezama. Top-down synthesis for library learning. POPL , 2023.

A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning , 2022.

B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397 , 2022.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Z. Cheng, T. Xie, P. Shi, C. Li, R. Nadkarni, Y . Hu, C. Xiong, D. Radev, M. Ostendorf, L. Zettlemoyer, et al. Binding language models in symbolic languages. arXiv preprint arXiv:2210.02875 , 2022.

F. Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547 , 2019.

K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

J. Cocke and K. Kennedy. An algorithm for reduction of operator strength. Communications of the ACM , 20(11):850-856, 1977.

E. Daka and G. Fraser. A survey on unit testing practices and problems. In 2014 IEEE 25th International Symposium on Software Reliability Engineering , pages 201-211. IEEE, 2014.

P. Denny, V. Kumar, and N. Giacaman. Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language. SIGCSE , 2022.

A. Desai, S. Gulwani, V. Hingorani, N. Jain, A. Karkare, M. Marron, and S. Roy. Program synthesis using natural language. In Proceedings of the 38th International Conference on Software Engineering , pages 345-356, 2016.

D. Dohan, W. Xu, A. Lewkowycz, J. Austin, D. Bieber, R. G. Lopes, Y. Wu, H. Michalewski, R. A. Saurous, J. Sohl-Dickstein, et al. Language model cascades. arXiv preprint arXiv:2207.10342 , 2022.

N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning , pages 5547-5569. PMLR, 2022.

D. Dua, S. Gupta, S. Singh, and M. Gardner. Successive prompting for decomposing complex questions. arXiv preprint arXiv:2212.04092 , 2022.

K. Ellis, C. Wong, M. Nye, M. Sablé-Meyer, L. Morales, L. Hewitt, L. Cary, A. Solar-Lezama, and J. B. Tenenbaum. Dreamcoder: Bootstrapping inductive program synthesis with wakesleep library learning. In Proceedings of the 42nd acm sigplan international conference on programming language design and implementation , pages 835-850, 2021.

M. Games. The fantastic combinations of john conway's new solitaire game 'life' by martin gardner. Scientific American , 223:120-123, 1970.

S. Gulwani. Programming by examples. Dependable Software Systems Engineering , 45(137): 3-15, 2016.

S. Gulwani, O. Polozov, R. Singh, et al. Program synthesis. Foundations and Trends® in Programming Languages , 4(1-2):1-119, 2017.

D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, et al. Measuring coding challenge competence with apps. Advances in Neural Information Processing Systems , 2021.

W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207 , 2022.

B. Ibarz, V. Kurin, G. Papamakarios, K. Nikiforou, M. Bennani, R. Csordás, A. Dudzik, M. Bošnjak, A. Vitvitskyi, Y. Rubanova, et al. A generalist neural algorithmic learner. LoG , 2022.

A. Q. Jiang, S. Welleck, J. P. Zhou, W. Li, J. Liu, M. Jamnik, T. Lacroix, Y . Wu, and G. Lample. Draft, sketch, and prove: Guiding formal theorem provers with informal proofs. arXiv preprint arXiv:2210.12283 , 2022.

O. Khattab, K. Santhanam, X. L. Li, D. Hall, P. Liang, C. Potts, and M. Zaharia. Demonstratesearch-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024 , 2022.

T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 , 2022.

S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. S. Liang. Spoc: Searchbased pseudocode to code. Advances in Neural Information Processing Systems , 32, 2019.

A. K. Lampinen, I. Dasgupta, S. C. Chan, K. Matthewson, M. H. Tessler, A. Creswell, J. L. McClelland, J. X. Wang, and F. Hill. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329 , 2022.

Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science , 378 (6624):1092-1097, 2022.

P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 , 2022.

A. Marasovi´ c, I. Beltagy, D. Downey, and M. E. Peters. Few-shot self-rationalization with natural language prompts. arXiv preprint arXiv:2111.08284 , 2021.

W. Merrill, Y. Goldberg, R. Schwartz, and N. A. Smith. Provable limitations of acquiring meaning from ungrounded form: What will future language models understand? Transactions of the Association for Computational Linguistics , 9:1047-1060, 2021.

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 , 2022.

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. A conversational paradigm for program synthesis. arXiv preprint arXiv:2203.13474 , 2022.

P. Norvig. How to write a Lisp interpreter (in Python), 2010. URL http://norvig.com/ lispy.html .

M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021.

A. Odena, K. Shi, D. Bieber, R. Singh, C. Sutton, and H. Dai. Bustle: Bottom-up program synthesis through learning-guided exploration. ICLR , 2021.

OpenAI. Gpt-4 technical report. arXiv , 2023.

S. Palan and C. Schitter. Prolific. ac-a subject pool for online experiments. Journal of Behavioral and Experimental Finance , 17:22-27, 2018.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

N. Perry, M. Srivastava, D. Kumar, and D. Boneh. Do users write more insecure code with ai assistants? arXiv preprint arXiv:2211.03622 , 2022.

G. Poesia, A. Polozov, V. Le, A. Tiwari, G. Soares, C. Meek, and S. Gulwani. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations , 2022.

S. Polu and I. Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393 , 2020.

X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8494-8502, 2018.

N. F. Rajani, B. McCann, C. Xiong, and R. Socher. Explain yourself! leveraging language models for commonsense reasoning. ACL , 2019.

M. Raza, S. Gulwani, and N. Milic-Frayling. Compositional program synthesis from natural language and examples. In Twenty-Fourth International Joint Conference on Artificial Intelligence , 2015.

G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg. Security implications of large language model code assistants: A user study. arXiv preprint arXiv:2208.09727 , 2022.

V. Shwartz, P. West, R. L. Bras, C. Bhagavatula, and Y. Choi. Unsupervised commonsense question answering with self-talk. EMNLP 2020 , 2020.

H. A. Simon and A. Newell. Human problem solving: The state of the theory in 1970. American psychologist , 26(2):145, 1971.

I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302 , 2022.

A. Stuhlmüller, J. Reppert, and L. Stebbing. Factored cognition primer. https://primer. ought.org , 2022.

M. Tabachnyk and S. Nikolov. Ml-enhanced code completion improves developer productivity, 2022.

J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275 , 2022.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

N. Wies, Y. Levine, and A. Shashua. Sub-task decomposition enables learning in sequence to sequence tasks. arXiv preprint arXiv:2204.02892 , 2022.

F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , pages 1-10, 2022.

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 , 2022.

A. Ye, C. Cui, T. Shi, and M. O. Riedl. Neural story planning, 2022. URL https://arxiv. org/abs/2212.08718 .

P. Yin and G. Neubig. A syntactic neural model for general-purpose code generation. ACL , 2017.

P. Yin, B. Deng, E. Chen, B. Vasilescu, and G. Neubig. Learning to mine aligned code and natural language pairs from stack overflow. In 2018 IEEE/ACM 15th international conference on mining software repositories (MSR) , pages 476-486. IEEE, 2018.

P. Yin, W.-D. Li, K. Xiao, A. Rao, Y. Wen, K. Shi, J. Howland, P. Bailey, M. Catasta, H. Michalewski, A. Polozov, and C. Sutton. Natural language to code generation in interactive data science notebooks, 2022. URL https://arxiv.org/abs/2212.09248 .

E. Zelikman, Y. Wu, J. Mu, and N. Goodman. STar: Bootstrapping reasoning with reasoning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=_3ELRdg2sgI .

T. Zhang, T. Yu, T. B. Hashimoto, M. Lewis, W.-t. Yih, D. Fried, and S. I. Wang. Coder reviewer reranking for code generation. arXiv preprint arXiv:2211.16490 , 2022.

K. Zheng, J. M. Han, and S. Polu. Minif2f: a cross-system benchmark for formal olympiad-level mathematics. ICLR , 2022.

D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 , 2022.

H. Zhou, A. Nova, H. Larochelle, A. Courville, B. Neyshabur, and H. Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint arXiv:2211.09066 , 2022.

Figure A.9: A potential programming assignment focused on problem-solving rather than implementation. The top-level function and asserts would be the assigned problem (which Codex [12] does not seem to be able to solve directly), while the other functions would be the student solution.

A Implications

Parsel is a natural language compiler framework that bridges the gap between natural language and programming language by allowing programmers to write high-level algorithmic designs in natural language and automatically compiling them into valid code. This has potential benefits for programmers, students, and code language models.

A.1 For Programmers

A.1.1 Current Limitations

First, programming generation language models like Codex continue to be constrained primarily to individual functions, rarely exceeding a few dozen lines in practice [12, 58]. This is still a dramatic shift from foundational earlier works, which focused on the association between one line of natural language pseudocode with one line of code [33] or a line of text to a StackOverflow snippet [66]. Yet, these models perform worse the more unusual the desired functions are, and recent research suggests that people using these language models are more likely to introduce buggy code [47], although this is not yet conclusive [53].

A.1.2 Potential Benefits

On the other hand, results from Google and others indicate that professionals can write code more efficiently with large language models, and the benefits will likely only improve as they improve [58]. Since Parsel requires constraints that ensure functions behave as expected, this should encourage bug-free programs and avoid the need for manually checking that specific underlying functions are correct. Furthermore, a function written in Parsel is likely to be more resilient to breaking changes in the target language, especially syntactic changes (e.g. Python2 to Python3). In addition, a natural extension would draw on work on automatic unit testing [17] to suggest additional constraints where behavior is ambiguous between implementations of a function. However, for Parsel to be practically useful for real-world programmers, we expect multiple improvements to be necessary, including additional object-oriented features.

A.2 For Students

A.2.1 Current Limitations

In addition, these language models pose serious challenges for programming pedagogy - existing introductory programming classes rely extensively on teaching syntax and how to implement algorithms over how to solve problems with them. Free language model-based tools like Copilot can essentially solve many of these introductory assignments directly, function by function. Those which cannot be solved currently will be increasingly solved [18].

A.2.2 Potential Benefits

Many students currently introduced to programming struggle with learning syntax and debugging unclear compiler or interpreter errors. However, abstracting away these details with a natural-language coding language will likely make learning to code more accessible to students who are just beginning to code. In addition, stepping away from implementation-focused assignments will allow a focus on higher-level problem-solving assignments earlier. These will allow for assignments that are more like those in mathematics. For example, for a problem like Figure A.9, instead of choosing between requiring students to manually implement a problem-solving focused question like the top-level description of, or requiring teaching assistants to manually evaluate the reasoning for correctness, one could ask them to implement a solution in Parsel.

A.3 For Code Language Models

A.3.1 Current Limitations

Traditional programming languages result in some unique challenges for language models. For example, unlike natural languages, traditional programming languages are far less robust to slight variations in wording. In addition, traditional programming languages require many tokens for syntactic details and in some cases, may take many lines to express what can be expressed far more simply in language. For example, referring to a shortest-path algorithm or Conway's game of life takes far fewer tokens than actually implementing them. However, even with fairly nonstandard problems, LLMs have shown remarkable algorithmic generalization ability [36, 62, 3, 72]. One alternative that has been explored is conversational code generation [40, 67]. However, these approaches have primarily focused on highly imperative programming structures. Moreover, they still require having the full program in context and do not clearly generalize to complex hierarchical programs with many functions.

A.3.2 Potential Benefits

Parsel allows code language models to stay closer to natural language when generating code, which corresponds more closely to their primary source of training data. Moreover, it allows complex but standard methods to be described concisely, requiring fewer tokens to generate. One exciting additional benefit is the potential to generate solutions recursively: if the Parsel compiler is unable to find a solution for a set of functions, it should be possible to prompt the model to define new helper functions. In fact, we find that often the model attempts to reference undefined auxiliary functions when defining complex functions (e.g. 'count_living_neighbors(grid, i, j)' in Conway's game of life), and as a result support an optional argument where the model can attempt to resolve NameErrors automatically by attempting to implement functions.

B Limitations

There are several limitations to the current implementation of Parsel. First, Parsel relies on a code LLM to generate implementations of individual functions, and the quality of these implementations can vary depending on the specific model used and the complexity of the function descriptions. In particular, Parsel may struggle to generate correct code for individual functions with complex behavior (i.e. functions that Codex cannot implement). However, this can be mitigated by decomposing the complex functions into simpler ones that can be implemented more easily.

The current implementation of Parsel may struggle to generate correct code when there are many functions with complex dependencies or without constraints. This is because the number of implementation combinations to consider grows exponentially with the size of the largest strongly connected components. As discussed, this can limit Parsel's performance on some programs. However, approaches like Chen et al. [11] may be able to mitigate this.

Code LLMs, unfortunately, do not perform well on languages underrepresented in their training data with few examples to learn from, LLMs may struggle to generate correct code in these languages [5]. However, some LLMs can learn new languages in context, allowing them to generate code in languages not in their training data [5]. These limitations can impact the quality and reliability of the code generated with Parsel. In addition, because code LLMs have never been trained on Parsel, this harms their ability to generate it. While we could wait for Parsel to gain widespread adoption, it should also be possible to translate many existing codebases to Parsel. We include a proof-of-concept backtranslation/decompilation study in Appendix L.

In addition, the best open-source code LLMs currently available e.g. PolyCoder [62] substantially underperform Codex, while Codex is competitive with other traditional LLMs on reasoning tasks [36]. However, this dependence on closed models creates a vulnerability, as the providers of closed LLMs can change behavior (e.g. rate limits or model implementations) without warning. Indeed, between the time we started working on Parsel and this version of the paper, OpenAI ended widespread access to Codex, now available only by request.

Because of this, we evaluated a 2.7B CodeGen model from Nijkamp et al. [39] with Parsel in the same configuration we used when evaluating APPS on Codex (in the 8x16 configuration). We found that it could solve none of the random 25 problems which we evaluated it on. However, despite these limitations, the current Parsel implementation has shown promising results in generating correct code for a variety of functions and languages. Many limitations will likely be ameliorated as code LLMs improve.

C Future Work

In the future, we hope to more deeply integrate automatic unit test generation, especially in combination with user-provided tests [17, 11]. One method would be to identify edge cases and check whether the set of functions that successfully solve all existing tests disagree on any new tests. This could permit automatic decomposition without exponential growth in implementation combinations. Techniques like those proposed in Zhang et al. [69], which would allow us to rerank a set of solutions, could also allow us to search the combinatorial space of solutions more quickly. Relatedly, for the robotic task planning, incorporating asserts at the execution level (e.g. checking whether the agent is close to the microwave, as in Singh et al. [56]) is a promising research direction. Furthermore, evaluating the examples in this paper, we found that using the minimum CodeT score across all generated functions was a consistently effective heuristic to identify good sets of functions. However, generating unit tests for all functions when generating Parsel programs instead of generating unit tests for a shared top-level function increases the inference cost from linear in the number of tasks to also being linear in the number of functions and Parsel programs generated. Finding a way to balance this tradeoff would likely be valuable.

In addition, we plan to incorporate ways of varying the 'confidence threshold' of the language model. Ensuring that the descriptions are straightforward and unambiguous is important for more critical programs and parts of programs. In addition, when teaching students simpler concepts, requiring them to decompose the task further may be useful.

We would like to integrate value functions to allow decomposition to be done more methodically where no verification is possible. Specifically, automatically decomposing all functions that have not yet been implemented in an SCC is suboptimal and could be improved with a model of expected improvement due to expansion, as done for proof expansion in Polu and Sutskever [49]. In addition, when decomposing functions, we would like to permit the model to reference already-defined functions (rather than to just define new ones). We might even use the code language model to determine which function to evaluate next. Further, we aim to support more general reward functions for function implementations where multiple may be valid but we rank implementations based on a desired feature. These 'soft' constraints may also allow new Parsel uses, e.g. planning stories in natural language [64].

Finally, we hope it would be possible to use Parsel as a framework for bootstrapping increasingly complex program generation (e.g. Anthony et al. [4], Zelikman et al. [68], Odena et al. [43]). That is, by 1) generating Parsel examples from a purely natural language specification and then reinforcing those which successfully compile, and 2) by reinforcing the model with each successfully compiled component, we would likely be able to iteratively improve performance with an arbitrarily large dataset of examples.

Another feature that would be valuable would be the ability to incorporate multiple base tools with different kinds of specialized models, inspired by Ibarz et al. [29] and Dohan et al. [20]. That is, it would be valuable to allow a model to determine which target language to use, possibly combining them. For example, for large parts of the Tensorflow and PyTorch libraries, while their interfaces are written in Python, they depend heavily on large C++ codebases [46, 1]. Relatedly, Cobbe et al. [15] showed that giving language models access to a calculator allowed them to solve more complex math word problems. This, combined with the observation that Parsel could also compile programs by generating language model prompts to be used as part of the program, may potentially allow the automatic generation of task-specific language model cascades [20].

Another noteworthy addition would be the integration of Synchromesh [48], ensuring that each new word or token generated by the model is actually possible within the grammar of the given formal language and does not violate other semantic constraints.

Ultimately, we hope that this specification for Parsel is a jumping-off point for a new way of thinking about programming and reasoning.

D Theorem Proving in Lean

With the same framework, we can generate proofs in formal theorem-proving languages such as Lean, as in Figure A.10. We include the translated version in the appendix. Note a nuance of Lean and theorem-proving languages is that the ability to run Lean on proof with no errors/warnings indicates the proof is correct (but is not a guarantee that the proof statement matches our claim in language). Thus, each function in a Lean Parsel proof has an 'implicit constraint.' This makes it straightforward to identify which informal parts of a proof are most difficult to explicate. Generally, we believe Parsel can be a powerful tool for theorem proving.

Yet, we observed important challenges in this context, which we believe are avenues for future work and can be resolved. For example, in datasets such as MiniF2F [70], many proofs require explicit calculations in intermediate steps. That is, many proofs are similar to 'Find the minimum value of 9 x 2 sin 2 x +4 x sin x for 0 < x < π . Show that it is 012.' (from the informal MiniF2F introduced by Jiang et al. [30]). We believe that a dataset of proof statements (in natural and formal language), requiring complex proofs that are more abstract and less dependent on explicit calculations would allow us to better measure progress towards solving difficult theorems - we leave this to future work.

E Optimizations

E.1 Caching

Wecache responses from the language model with respect to the prompt and language model decoding parameters 1) to reduce the number of queries necessary and 2) to keep the programs generated mostly stable (i.e. a working function should continue working unless it or its children change). To this end, when the number of desired implementations increases for a pre-existing query with all other arguments fixed (temperature, number of decoding tokens, etc), we append the additional ones to those already generated.

E.2 Automatic Function Infilling

Sometimes, a function generated by a language model may call a function that is not yet implemented. In this case, we can (optionally) attempt to automatically generate and implement it based on its usage. The function is then incorporated into the call graph as a unit-test-less child of the function which calls it. To avoid infinite recursion and inefficient use of language model quota, we limit the number of times that this process can be applied to a function.

E.3 Multiprocessing

We use multiprocessing with a user-specified timeout to test many implementation sets in parallel to allow for many fast solutions to be tested alongside slower solutions 3 .

F Parsel Pseudocode

G Parsel Pseudocode

We include a longer-form Parsel pseudocode in the style of Parsel. Note this pseudocode does not include backtranslation.

H Parsel Overview (Detailed)

We include a more detailed figure outlining Parsel.

Natural Language

Function Definitions

Possible Function

Implementations

Test

Implementation Set

Constraints

Solution

Choose

Combination

Parsel

Compiler

Program in

Parsel

Language

(per SCC)

Code LM

Divide Program into

Solved

Dependencies

Dependency

Graph

(intermediate

representation)

References

Compose

Solutions

or

Task/Problem

Statement

Decompose Problem

I Lisp Interpreter

We include the Parsel code for a minimal Lisp interpreter.

J Case Study

We include a simple example function we could not generate with Codex [12] directly from the toplevel description in Figure A.15. The corresponding Python code (included in the appendix) is exactly 58 non-whitespace lines of code, including 17 lines of comments (3 corresponding to the descriptions), 2 asserts, and 39 lines implementing the three functions described as well as an automatically generated get_number_of_active_cells_around_cell function. In fact, using automatic decomposition, as discussed in Subsection R.3, it is not necessary to provide any of the function descriptions besides the top one. The model is (unsurprisingly) able to understand that game_of_life_inversion_iteration can be broken down into invert_array and game_of_life_iteration .

Figure A.15: An example Parsel program for Python that takes in a list of lists representing a state of Conway's game of life [24] and returns the next state, with all the values inverted.

K Parsel Prompts

L APPS Backtranslation

L.1 Backtranslation / decompiling.

We anticipate that there are many programs that LLMs can implement by first generating Parsel code. But, as Parsel is a new framework, while language models can sometimes generate Parsel programs with few-shot prompts, it is not a syntax they have previously encountered. Thus, we may want to use existing code in other languages to construct datasets of Parsel programs from other languages. This requires us to first extract the call graph from the code, generate descriptions for each of the functions, and then generate Parsel programs from the graph. This call graph representation is convenient, so it is useful to have a bidirectional method to produce a graph from Parsel code and to produce Parsel code from the graph.

We filter the dataset to problems with starter code (providing the name of the evaluated function) and unit tests (provided as input-output pairs). For those tasks, we select solutions that define and call at least three functions, with at least one over 4 lines long and none over 15 lines.

As a proof of concept, we show 10 Parsel solutions which we could automatically generate from the APPS solutions. We generated the descriptions by prompting Codex to explain each function and its inputs and outputs. From this, we use backtranslation to attempt to implement these solutions in Python. We then verify that they are correct by applying the original unit tests as constraints on the root function. As mentioned in Section 1, the Parsel code is substantially shorter in terms of lines of code. We include these in Appendix L.

L.2 Examples

We exclude the asserts in these examples for brevity - they correspond to those in the original dataset.

M Compiled Examples

N APPS Decomposition Prompts and Evaluation Hyperparameters

We slightly loosen the requirements for Parsel programs generated by language models, treating redundant function definitions as references instead of raising errors. We sample everything with temperature=0.6, except the translations which we sample with temperature=0.2, a presence penalty of 0.1, and a logit bias to prevent it from generating the text ' def ', as Codex has a tendency to degenerate to producing Python even when prompted with Parsel examples. We allow at most 500 tokens per function, but in practice found that they typically used less than half of them.

For evaluation, we use a timeout of 0.04 seconds per solution and evaluate at most 100,000 implementations per generated Parsel program.

For the Codex-only ablation, we allow it to generate at most 1000 tokens, in large part due to the rate limit. In particular, there is a heuristic rate limit that rejects any calls requesting more than 4,000 tokens. As a result, any larger number of samples per problem would prevent batching more than 3 samples at a time.

Figure A.45: Full Parsel program including header for Fig. 2 example, with the #*#*# as the header seperator. Note that we essentially just took the executability definition in [28] and added the list of available actions.

1

"""-----Translation-----

2

# Here is an example calculating the probability of landing on the same character in a random shift of an input string, based on the following

↪

→

problem:

3

Kolya play a game with a

string, using the following

rules. Initially, Kolya creates

a string s, consisting of small English letters, and

↪

→

uniformly at random

chooses an integer k from

a segment [0, len(s) -

1]. He tells Vasya this string s, and then shifts it

k letters to the left

↪

→

, i.e. creates

a new string t = s_k

+ 1s_k + 2... s_ns_1s_2...

s_k. Vasya does not know

the integer k nor the string t, but he wants to

guess

↪

→

the integer k.

To do this, he asks Kolya to tell him the first

letter of the new string,

and then, after he sees

it, open one more letter

on

↪

→

some position, which Vasya can choose.

4

# Vasya understands, that he can't guarantee

that he will win, but he

wants to know the probability of winning, if he plays

optimally. He wants you

↪

→

to compute this

probability.

5

# Note that

Vasya wants to know the value of k uniquely, it means, that if there are at

least two cyclic shifts of

s that fit the information

Vasya

↪

→

knowns, Vasya loses. Of course, at any moment of the game Vasya wants

to maximize the probability

of his win.

6

\"\"\"

7

generate_cyclic_shifts(input_str): Calculates the average number of unique characters in

the substrings of the input

string that start with each

↪

→

character.

8

parse_input(input_str):

Takes a string and returns

the input string

9

compute_a_and_letter_pos(input_str): Generates the str_as_number_list and letter_pos lists. str_as_number_list is a list of

integers that is used

↪

→

to store the

character values of the input string. str_as_number_list is

initialized as a list of 0s

for twice the length of

the input string.

↪

→

The values are

calculated by taking the ASCII

value of each character in

the string and subtracting

the ASCII value of the character 'a'.

↪

→

letter_pos is

a list of lists, with each

sublist containing the indices

at which a particular character appears in the input string.

10

compute_unique_characters(c, str_as_number_list, letter_pos) ->

ans: Calculates the maximum

number of unique characters in

all substrings (for k=1

↪

→

to length) that start with the character

a list of lists, with

each sublist containing the indices at which a

↪

→

character appears

in the input string. str_as_number_list is a list of integers that is used to store the character values of the input string

↪

→

.

11

compute_unique_characters_for_k(c, k, str_as_number_list, letter_pos): Create a counts list

of zeros for each of the 26 alphabetical characters.

↪

→

For each i

in the sublist of positions

of letter_pos[c], increment counts at str_as_number_list[i + k].

Return the number of counts

which

↪

→

are exactly one.

12

to_output_str(ans, input_str): Returns a string representation of ans divided by

the length of the input string.

13

\"\"\"

14

(6 lines)

15

16

# And here

is an example identifying the

largest binary number according

to the following rules:

17

# The Little

Elephant has an integer a,

written in the binary notation. He wants to write this

number on a piece of paper.

18

# To make sure that the number a fits

on the piece of paper,

the Little Elephant ought to

delete exactly one any digit

from number a in the

binary

↪

→

record. At that

a new number appears. It

consists of the remaining binary digits, written in the

corresponding order (possible, with

leading

↪

→

zeroes).

19

# The Little

Elephant wants the number he

is going to write on the paper to be as large

as possible. Help him find

the maximum number that he

can

↪

→

obtain after deleting exactly one binary digit

and print it in the binary notation.

20

\"\"\"

21

largest_binary_number(input_str): Returns the largest binary

number that can be made

by removing at most one

digit from the input string.

22

parse_input(input_str):

Takes a string and returns

the input string

23

remove_zero(binary_str):

Remove the first zero from

the input string.

24

Returns the bigger string.

25

\"\"\"

26

(4 lines)

27

28

# Here is an example of the format applied to identifying the winner

of the following game:

29

# It is so

boring in the summer holiday, isn't it? So Alice and

Bob have invented a new

game to play. The rules

are as follows. First, they

get a set

↪

→

of n distinct

integers. And then they take

turns to make the following

moves. During each move, either Alice or Bob (the player whose turn is

↪

→

the current) can

choose two distinct integers

x and y from the set,

such that the set doesn't

contain their absolute difference

|x - y|. Then

↪

→

this player adds

the set increases by one).

30

# If the current player has no valid move, he (or she) loses the

game. The question is who

will finally win the game

if both players play optimally.

↪

→

Remember that Alice always moves first.

31

\"\"\"

32

identify_winner(input_str):

Returns the winner of the

game.

33

parse_input(input_str):

Takes a string containing the

length on the first line

and the integers on the

second and returns the list

of integers

34

number of moves is the largest element in the list

divided by the greatest common

divisor of all elements in

the list, minus the

↪

→

length of the

list.

35

all_gcd(l): Returns the greatest common divisor

of all elements in the list

36

to_output_str(num_moves):

Returns the string 'Alice'

if the number of moves is

odd and 'Bob' if the number of moves is even

37

\"\"\"

38

(5 lines)

39

40

a little bear who loves to

play. Today he is playing

by destroying block towers.

He built n towers in a

row. The i-th tower is made of h_i

↪

→

identical blocks. For clarification see picture for

the first sample.

41

repeat the following operation

till everything is destroyed.

42

called internal if it has all four neighbors, i.e. it

has each side (top, left,

down and right) adjacent to

other block or to the floor.

↪

→

Otherwise, block

is boundary. In one operation

Limak destroys all boundary

blocks. His paws are very fast and he destroys all those blocks at

↪

→

the same time.

43

ready to start. You task is

to count how many operations will it take him to

destroy all towers.

44

\"\"\"

45

destroy_towers(input_str):

Returns the number of operations it takes to destroy all towers.

46

parse_input(input_str):

Takes a string containing the

number of towers on the

first line and the heights

of the towers on the second and returns

↪

→

the list of

heights

47

side_ones(heights_list):

From a list of ints, set the first and last elements to 1 and return the

list

48

destroy_from_left(side_list): Copy the list and set

each each element to the

minimum of itself and one

more than the element to

its left, starting

↪

→

from the second element

49

destroy_from_right(side_list): Copy the list and set

each each element to the

minimum of itself and one

more than the element to

its right,

↪

→

starting from

the second to last element

50

min_list(l1, l2):

Return a list of the minimum of the corresponding elements of l1 and l2

51

to_output_str(min_list):

Return the string representation

of the maximum element in

the list

52

\"\"\"

53

(7 lines)

54

55

# Alex decided

to go on a touristic trip over the country.

56

# For simplicity

let's assume that the country has $n$ cities and $m$

bidirectional roads connecting them. Alex lives in city $s$

and initially

↪

→

located in it.

To compare different cities

Alex assigned each city a

score $w_i$ which is as high as interesting city seems

57

# Alex believes

that his trip will be

interesting only if he will

not use any road twice in

a row. That is if Alex came to city $v$ from

city $u$, he

↪

→

may choose as

the next city in the trip

any city connected with $v$

by the road, except for

the city $u$.

58

# Your task

is to help Alex plan his

city in a way that maximizes total score over all

cities he visited. Note that

for each city its score

is

↪

→

counted at most

once, even if Alex been

there several times during his

trip.

59

\"\"\"

60

max_score(input_str):

Simple function returning the maximum score Alex can get.

61

parse_input(input_str):

Takes a string containing the

number of cities and roads on one line, the scores

of the cities on the next line, the roads

↪

→

on the next

lines besides the last (1-indexed, make 0-indexed), and the

starting city on the last

line. It returns the city

scores, the roads

↪

→

as an edge

list, and the starting city.

62

get_neighbors(edges):

Returns a dictionary of the

neighbors of each city, defaulting to an empty set.

63

get_degrees_and_leaves(neighbors, root): Returns a dictionary of the degrees of each

city, and a set of the leaves (excluding the root).

64

remove_leaves(scores,

neighbors, degrees, leaves, root):

Create a 0-initialized defaultdict of total_extra, and an int of max_extra. Pop leaves

↪

→

until it is

empty. Update total_extra and

max_extra based on the parent's

total_extra vs the leaf's

score plus its total_extra, whichever is

↪

→

greater. Return

max_extra.

65

pop_leaf(neighbors,

degrees, leaves, root): Pop off

a leaf. Set parent to

sole neighbor of the leaf

and delete the leaf from

the neighbors

↪

→

dictionary. Decrement the parent's degree. If

the parent is not the root and has degree 1, add

it to the leaves. Return

the leaf and parent.

66

to_output_str(scores,

neighbors, root, max_extra): Returns

the string of the maximum

score Alex can get. If

the root isn't in neighbors,

return the

↪

→

score of the

root. Otherwise, this is the

sum of the scores of the

cities left in neighbors,

plus the returned encountered max_extra.

67

\"\"\"

68

(7 lines)

69

70

# Translate the

following solution plan into

the above format:

71

{solution_start}{solution_text}

72

73

TRANSLATE to Parsel.

74

\"\"\"

75

"""

45

O HumanEval Prompts

We use the same zero-shot prompt to encourage the high-level sketch as in APPS. For translation we use:

P Robotic Plan Evaluation Details

P.1 Questionnaire

Our questionnaire closely follows that of Huang et al. [28]. We provide a figure with the directions for the accuracy version of the survey in the first image of Fig A.49. We include an example question in the second image. Note that each participant was shown a random 5 questions with their answers in random order. The clarity survey instead asks 'For every question below, evaluate how easy it is to understand how the provided steps accomplish the task. Please rank the planned steps for each question from most understandable to least understandable (with 1 as the best and 3 as the worst).' In addition, for the clarity survey, each question text instead said 'Rank the following plans based on which is the most understandable (1 = most understandable, 3 = least understandable).'

P.2 Executability

We find that the automated executability check is a less insightful metric than human evaluation, as it doesn't meaningfully reflect the plan's likelihood of successfully completing a task. Unfortunately, the code to replicate the executability measure from Huang et al. [28] is unavailable. As an alternative, we developed our own executability checker using example code found on VirtualHome's GitHub repository, which evaluates if a proposed plan is syntactically accurate and can be executed within the VirtualHome environment. By leveraging Codex to generate eight Parsel programs for each of the 88 tasks and subsequently compiling them using the Parsel compiler, our method successfully produced executable solutions for all tasks. Conversely, Huang et al. [28] managed executable plans for only 86 tasks. However, it is worth noting that our Parsel compiler explicitly incorporates this executability measure as a constraint, which explains the higher executability rates observed in our approach.

On the next (and only) survey page, you will be presented with 5 questions.

For eachquestionyou willseean everydaytask and aplanto accomplishit.Please determine whether the task canbe completed using theprovided steps.Inotherwords, can thetaskbe decomposed intothese steps?Note that simplyrestating the task does accurate to least accurate (with 1 as the best and 3 as the worst).

There isnocorrect answer to each question.Please just useyourfirst intuition to determine the answers.

Rank thefollowingplans based on how accurately they accomplish the task(1=most accurate,3=least accurate):

2

Task:Wash teeth

walk to bathroom

walk to sink

find faucet

switch on faucet

find toothbrush

grabtoothbrush

find toothpaste

grab toothpaste

put toothpaste on toothbrush

put backtoothpaste

grabtoothbrush

wash teeth

rinse toothbrush

switch off faucet

grab toothbrush

find toothbrushholder

put toothbrushontoothbrushholder

grabtoothbrush

putoothbrushontoothbrushholder

walk to bedroom

Task:Wash teeth

task_plan:

wash_teeth:

find toothbrush

plug in sink

pour water into sink

pour toothpaste into toothbrush

pour water into toothbrush

rinse toothbrush

plug out sink

drop toothbrush

find towel

wipe teeth with towel

drop towel

Task:Wash teeth

find toothbrush

walk to toothbrush

grab toothbrush

open toothbrush

grabtoothbrush

find bathroom sink,walk tobathroom sink

grabtoothbrush

run to bathroom sink

put toothbrushonbathroomsink

release toothbrush

put back toothbrush

walk to toothbrush

Q Human Case Study

R Parsel Language Nuances

R.1 Syntax

Descriptions. Afunction description is represented as a function name followed by comma-separated input arguments in parentheses, and optionally an arrow followed by what the function returns 4 , then a colon and text describing the function to be implemented. For example, as part of Conway's Game of Life, one might write

A function generated from a description can call either the functions defined directly below the description in the function graph (indicated with indentation) or references directly below the description 5 , both shown in Fig. ii.

Constraints. A constraint is represented as a function's input values comma-separated, optionally followed by an arrow and the expected output of the function. Constraints are provided at the same indentation as the preceding description. For example, after the definition of count_living_neighbors one can write

This indicates that count_living_neighbors should return 1 when called with the arguments [[1, 0], [0, 1]], 0, 0 and 2 when called with [[1, 0], [0, 1]], 0, 1 . Notably, to apply complex constraints on functions, one can describe and constrain higher-order functions. For example:

type_fn_output(fn, args): returns the type of the output of a function called with args count_living_neighbors, ([[1, 0], [0, 1]], 0, 0) -> int

This indicates that the function count_living_neighbors should return an integer when called with the input arguments [[1, 0], [0, 1]], 0, 0 .

What it means to satisfy constraints to validate a program varies from language to language: in Python, one can check that a program passes certain assert statements by evaluating them; however, in a theorem-proving language like Lean, where the ability to run a program (without skipping steps by using sorry or oops lines) shows that a proof holds, one would instead represent the formal proof statement as the specified constraint (that is, that you are actually proving what you set out to prove). For languages where correctness can be checked without any unit tests, their functions can be treated as also having implicit constraints.

collatz_recursion

recursion_rule

base_case

References. A reference is simply the name of a function defined in the current scope (see the next paragraph for details) within the function graph. A reference allows and encourages (via prompt) the parent function to use the referenced function. This allows for recursive function definitions and functions called by multiple functions. For example, one can define an (overly verbose) version of the Collatz conjecture (a well-known recursive open question in mathematics) as shown in Figure ii, where the final line is a reference. We visualize the corresponding call graph and its strongly connected components (SCC) in Figure A.55. In the Collatz functions, base_case is implemented first as the collatz_recursion SCC depends on it.

R.2 Headers

We also support program headers, allowing global contexts, used when implementing all new functions within a program. This is indicated by a line containing an optional string of special characters (e.g. '#*#*#') separating the body and the text and is passed as a prefix to all prompts.

R.3 Repeated Automatic Decomposition

As indicated by a rapidly growing number of papers [10, 28], the task of decomposing a task into steps in natural language is one that language models are surprisingly capable of. As explored in concurrent work [57], using language models to automatically and recursively decompose difficult open-ended problems to arbitrary depth is a powerful tool. Thus, we treat the ability to automatically decompose a Parsel function as a key feature of Parsel. This is an optional flag that prompts a language model to generate Parsel code corresponding to any additional subfunctions necessary when Parsel fails to implement a function. These proposed subfunctions are then added as child nodes to the decomposed function node. However, an additional consequence is that Parsel can thus be used to recursively decompose tasks into steps, by repeatedly identifying descriptions that cannot be directly implemented and attempting to decompose them.

R.4 Scope.

Scope in Parsel is defined by indentation. The scope S of a function f includes the set of functions that can be used as a reference for a given function - that is, all functions where the indentations between the current function to the referenced function are strictly decreasing.

R.5 Variations due to target language requirements.

Certain aspects of the implementation are still target-language specific. As discussed above, the meaning and representation of a constraint may vary by language. Moreover, every language has a different evaluation function: executing Python is different than compiling and running C++ code, which is different than checking a proof with Lean. Further, every language will likely require a different prompt for the language model. Thus, we detail these particularities in language-specific configuration files.

S Pipeline Figure Example

→

↪

T Other Details

T.1 Test Generation in HumanEval

For our test generation in our HumanEval experiments, we generated a set of tests by prompting GPT4 to "Generate an assert-based test for the following function. Answer with only a code block, and no other text. Do not wrap your asserts in a function.\n" + question and then collecting and set-aggregating 100 completions.

T.2 Backtracking

We also support backtracking for the Parsel implementation step, where we re-implement descendants by sampling new solutions for dependencies if a correct solution is not found for a parent. This is necessary to improve the robustness of some of the Appendix examples such as Figure A.14.

T.3 Training Details

Although we do not train any models, all models used are discussed throughout the paper. See Appendix N for more details about sampling hyperparameters.

T.4 Error Bars

We estimate a standard deviation of ± 1 . 4% for the best APPS result, given 1000 sampled problems.

T.5 Compute

The most computationally intensive part of this research, by far (in terms of FLOPS), was the ablation using an open-source CodeGen model, which required several-hundred A100 hours. The rest of the inference was done through API calls.

T.6 Generated Tokens

For the APPS evaluation, in terms of tokens generated, it is hard to compare the models directly: The CodeT paper does not specify the number of tokens decoded for their evaluation. Without more detail about their evaluations, it is impossible to confidently estimate the tokens generated per program for the CodeT evaluation. The AlphaCode results sample at most 768 tokens per solution, but they do not report average statistics directly - based on Figure 11 in the AlphaCode paper [35], the majority of its generated solutions are 150 to 350 tokens long after removing dead code and comments. The competition-level problems (that we evaluate on) require more tokens on average. For their best reported results, their figures indicate they sample at least 20 billion tokens for the competition-level subset of APPS. On the other hand, for our best results, Parsel generates (on average) 491 tokens of Python code per program implementation, and because we implement each high-level sketch in Python sixteen times (i.e. k = 16 in our best n × k ), we also sample on average 22 sketch tokens and 43 translation tokens per Python program implementation. Correspondingly, we sample roughly 7 million tokens for our APPS evaluation.

T.7 Reproducibility

While our contribution is not a model or dataset, we have released our code.

T.8 Evaluation Time

For a similar API rate limit and evaluation time, we expect our longest-running evaluation runtime would be fairly comparable to the 1,000-program AlphaCode evaluation (despite performing 3x better) and several times faster than the 50,000-program AlphaCode evaluation. In other words, we believe that generating 1,000 programs and evaluating them is not meaningfully faster than generating the 128 Parsel ones and then evaluating their combinations, even in the worst case. Our reasoning is as follows:

Generation time. Because AlphaCode has no public-facing API, let us use the advertised rate limit for Codex (40,000 tokens per minute) as the generation speed. (This ignores the request per-minute limit and the number of input tokens in context.) For AlphaCode, assuming roughly 400 tokens per solution, generation would take 10 minutes plus the completion time (i.e., the latency from an API request to the result) of roughly 30 seconds. For Parsel, in our largest-scale experiment, we generate eight plans, their corresponding Parsel programs, and then each Parsel program's corresponding 16 implementations. At the same rate limit and program length, these generations would take about 1.5 minutes, plus three times the completion time. In other words, generation should take about 3 minutes for Parsel vs. 10.5 minutes for AlphaCode.

Evaluation time. Let's consider three evaluation scenarios, maintaining the 0.04-second implementation eval timeout used in the paper, run on a single 16-core machine. 1,000 evaluations. If we run 1,000 programs for each method, under the same constraints, their evaluation runtime should be the same. As indicated by Fig. 4-right, the Parsel performance would be substantially higher despite being over 3x faster overall. 10,000 evaluations. If we assume all programs take the maximum allowable runtime (the worst case for Parsel), then 10,000 evaluations takes 6.7 minutes for parsel, compared to 40 seconds for 1,000 AlphaCode program evals. Parsel is still slightly faster, yet solves multiple times more problems. More evaluations. After this point, we run into the 2-minute per Parsel program limit (discussed previously), making Parsel slower than AlphaCode in the worst case - but also solving over 3x as many problems.

Other nuances. With all this being said, this is a difficult (potentially impossible) comparison to make fairly for a few reasons. First, AlphaCode is worse than Codex, but likely faster, so it isn't clear which Parsel data point and which rate limit would make an appropriate comparison. Second, runtimes and problem lengths can vary substantially. For instance, we haven't considered input tokens so far, but these penalize AlphaCode more than Parsel, as AlphaCode inferences many more programs. Note that APPS problems are fairly long (the longest competition-level problem requires over 1800 tokens) so the effect of re-processing the problem statement at each program inference is substantial. Since the rate limit includes input tokens and there is a hard limit of 4,000 generated tokens per query, this decreases the effective rate limit by a third!

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Parsel, a framework that decomposes algorithmic reasoning problems into subparts, samples programs for each subpart and verify them. To decompose a problem, Parsel transforms it into an intermediate language that describes the functionality for each subpart and how the subparts depend on each other. To sample programs, Parsel uses existing language models trained on code. To verify the subparts and the whole program, Parsel analyzes their dependencies and use existing test generation techniques.\n\nParsel is able to improve the performance in solving code problems and virtual robot control.\n\nSTRENGTHS:\n1. The overall idea of divide-and-conquer and verifying subparts is reasonable and intuitive.\n2. Using an intermediate language makes both the reasoning and parsing easier.\n3. The idea for involving human programmers to help in the loop is inspirational for designing better code assistants.\n\nWEAKNESSES:\n### 1. The comparison between Parsel and Codex (codeT improved) is not fair.\n\nTherefore, the claim on L145 that Parsel “substantially improves over prior work, from 14.5% to 25.5%” does not hold.\n\nIf I understand correctly, this comparison is demonstrated in Figure 4, where the number 14.5% comes from the blue point labeled **“50 (improved)”** and the number 25.5% comes from the green point labeled **“8x16”**. In my opinion, this comparison is unfair because of two reasons.\n\n- **First, the sample budgets are different.**\n    As claimed on L142, the comparison uses an “effective number of complete programs” to represent the sample budget. So the “50 (improved)” setting has a sample budget of 50 while the “8x16” has 128 sample budget. On the own interpolated green curve in Figure 4, Parsel’s performance at sample budget 50 is similar to that of Codex.\n\n- **Second, Parsel uses significantly more evaluation budget.**\n    Even if we ignored the first issue and assumed “50 (improved)” and “8x16” had the same sample budget, the comparison would still be unfair because of significant differences in evaluation budgets. “50 (improved)” achieved a pass rate of 14.5% with **50 evaluations**. While ”8x16” achieved 25.5% with **10^6 evaluations**.\n    I do understand that one advantage of Parsel is the ability to increase the number of evaluations without more samples, but it’s still unfair to compare the result from 10^6 evaluations with that from 50 evaluations. Because arguably, evaluation budget is more important than sampling budget in code generation, as a failed evaluation would result in penalty in programming competitions and crashes in real production. No one would want a code generation system that gets the correct program after 10^6 failures. It’s both unrealistic and unfair.\n    From the reported data, a fairer pass rate for “8x16” would be 4% (when the evaluation budget is 100), which is much worse than “50 (improved)”. To give more idea about how large a 10^6 evaluation budget is and how much improvement it gets, AlphaCode achieves >35% pass rate on CodeContests (Figure 8 from https://arxiv.org/pdf/2203.07814.pdf) with 10^6 evaluations and 6% pass rate with 100 evaluations. Judging from CodeT’s results, CodeContests is harder than competition-level APPS problems.\n    \nBecause of these two reasons, I find the claim on Parsel’s ability to solve more competition-level problems not reliable.\n    \n### 2. Lack of details and analyses on how verification of decomposed functions helps.\nParsel emphasizes the importance of decomposition of the problem into modular functions (components) and the verification of these functions.\n\nWhile decomposition can be justified by the exponential evaluation budget (though it creates unfair comparison) and the dropping pass rates when the number of chained components is large, how verifying these decomposed functions separately can help is not sufficiently justified, leaving several questions unanswered, to name a few:\n\n- **How are constraints (unit tests) generated for different components generated?** I understand that the test cases for the entire program is generated using CodeT, but how different components are verified is not clear.\n- **How many components have unit tests?** The paper claims that for test-less component functions, they can be tested with the test cases for the entire program. I wonder how many components actually have unit tests or if most of them are test-less.\n- **Are unit tests mostly correct?** If not, what is the point of using them instead of just using the general test cases?\n- **Do testing components separately lead to better perfiormance?**\n- **Do components that pass unit tests always build up to a correct program?**\n- **How much does partitioning the function reference graph into SCCs help?** The paper claims that for mutually dependent functions, grouping them into SCCs can help save some sampling budget. Does it really?\n    \nTo me, the results reported look like they are mostly from enumerating different combination of functions and verifying them using global constraints. The claim in your TLDR that the problem is decomposed into subparts and solved by “verifying subparts, and then composing them” does not seem well supported. It’s more like “enumerating the combination of subparts and then verifying them as a whole”.\n\nQUESTIONS:\nI would love to see the answers to the questions in the Weakness section. I may consider raising the score if they are well answered.\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a system called Parsel for program synthesis. First, a LLM predicts a Parsel program given a task specification or natural language plan. The Parsel program contains a hierarchy of functions which themselves might have functions inside. Each function is described with a function signature, a natural language description of what it does, and sometimes constraints (e.g., I/O examples) and references to other functions to call. Then, the Parsel synthesizer attempts to translate the Parsel program into a traditional program, e.g., a Python program. The LLM generates multiple implementations of each function and a combinatorial search is performed to find combinations of implementations that lead to passing the constraints. Overall, the Parsel approach is shown to outperform directly generating solutions with Codex and AlphaCode on the APPS dataset of competitive programming problems, when controlling for the number of complete programs generated. Other experiments are performed on HumanEval (simple Python programming problems) and VirtualHome (robotic planning).\n\nSTRENGTHS:\nOriginality: The Parsel language and overall approach is quite novel and creative, and in my opinion this is the paper’s biggest strength. Parsel’s motivation is clear: we want to decompose problems into multiple functions or groups of functions, such that each group can be implemented independently. However, it was previously not clear how this goal should be achieved, as it requires a planning step that decides how the problem should be decomposed. The Parsel approach nicely resolves this by providing a natural outline of the hierarchical decomposition, containing just the right amount of information to facilitate synthesis of individual functions, while allowing different implementations to be explored.\n\nQuality: The Parsel approach makes a lot of sense from a technical perspective. The experiments explore various aspects of Parsel.\n\nClarity: The motivation, intuitions, and related work are well written.\n\nSignificance: These results are quite significant themselves, and the problem being tackled (program synthesis) is important too.\n\nWEAKNESSES:\nIn my opinion, the paper’s main weakness is the lack of a deep analysis of *why* Parsel has its good performance. The approach has many parts and it is unclear which parts are the really important ones. Here are some potential reasons why Parsel is good, and potential experiments to gauge the importance of those reasons:\n\n1. Predicting a **high-level sketch or plan** helps the model gain an overall understanding of the problem in a chain-of-thought kind of way, leading to better downstream predictions.\n\n    * Reason 1 is partially analyzed in Lines 156 - 164 (when Codex is given the high-level plan, it performs much worse than Parsel given that plan). But I’m also interested in the other kind of ablation: how well does Parsel do without the high-level plan, going straight from the task specification to the Parsel program (on the APPS dataset)?\n\n2. The **Parsel program itself** makes synthesis easier, since it contains a good decomposition with just the right amount of information for easy synthesis of the individual functions.\n\n    * What if we replaced the Parsel program with a Codex prediction of the entire solution, which we then use as an outline? Specifically, we prompt Codex to implement a code solution that is decomposed into helper functions with docstrings. (If a function is not generated with a docstring, we can prompt an LLM to predict a docstring given the function and other relevant context from the task.) Then we take the function signatures, docstrings, and references (call graph) from the solution to serve as a replacement for the Parsel program. Thus, we sample new implementations of the individual functions and search for implementation combinations as usual.\n\n3. The approach enables **trying many combinations of implementations** with a low cost (effective number of complete programs sampled).\n\n    * What if we disallowed trying multiple combinations of implementations, i.e., only used pass@n x 1?\n\n4. The authors might identify other potential reasons that I didn’t think of.\n\nBy including these sorts of ablations, we would get a much more clear understanding of the relative importance between these factors. This could help readers gain a deeper intuition behind why Parsel works, so they may better adapt or extend the approach in future work. Even though my rating is positive right now, I *strongly encourage* the authors to perform these ablations, because I believe the paper will be much stronger (an amazing paper) with a deeper analysis of Parsel’s core ideas.\n\nAnother weakness of the paper is that the writing could be made clearer about the algorithm and experiments. Please see the questions below.\n\nPlease note: I will be happy to raise my score if I think these weaknesses are adequately addressed in a revision.\n\nQUESTIONS:\n## About the algorithm\n\nIf there is a function with constraints where none of the k implementations of the function pass the constraints, then what happens? I assume this causes the entire Parsel program to be unsuccessful, but this is not explicitly stated in the text. Does this lead to a tradeoff where, if the decomposition is too detailed with many individual functions, then there is a higher likelihood of a single function having no correct implementation? And, intuitively it seems useful to include constraints for all functions where the LLM can predict constraints confidently, but then more functions with constraints also leads to a higher likelihood of a single function having the wrong constraints. Can you comment on this “weakest link” issue?\n\nThere is an exponential blowup in the number of function implementation combinations. I understand that SCCs of the call graph and test dependency graph are used to reduce the number of functions considered at once, but still the exponential blowup remains. Do you impose some upper limit on the amount of combinations tried? Line 113 mentions *samples* -- when do you sample versus try all combinations, and how many samples are used?\n\nDoes a Parsel program always define a single outermost function that serves as the solution to the problem specification? This would make sense given the rest of the algorithm but is not explicitly stated.\n\nOnce we find an implementation for a function (or SCC) passing its constraints, is the implementation “locked in” permanently or does the algorithm also consider other implementations/combinations later? In other words, do we end up with at most 1 complete implementation for each Parsel program, which is then checked against the problem specification (possibly with hidden test cases)? Or, might we continue to search for other implementations of the Parsel program and evaluate those on the hidden tests too?\n\nThe CodeT score is a core part of the approach in the case of generated tests and is referenced repeatedly for the HumanEval experiments. The paper should provide some background on CodeT.\n\nI know that Appendix F contains Parsel pseudocode in Parsel style, which might clear up some of the questions above. I did not read that pseudocode (Figure A.11 and A.12) carefully because it’s a full page long, and I hope the authors can write a more succinct version in a more traditional style in the main paper. I also think Parsel programs are not as clear as actual pseudocode, because they lack detail about how inner functions are used to implement the outer function. It’s as if one had to understand how a Python program worked, but only seeing the function signatures, docstrings, and call graph... many important details would be lost.\n\n## About the experiments\n\nWhat I/O examples are given in the problem specifications, and are there held out tests used to evaluate the correctness of synthesized programs? The description of the experiment setup is lacking these kinds of details.\n\nIs it expected that the LLM can predict *correct* constraints in terms of I/O examples for individual functions? It is commonly observed that LLMs are quite bad at executing code or even performing arithmetic. I would expect that many constraints would have subtle errors that lead to incorrect implementations being selected from the search over function combinations. For example, is the constraint in Figure 2 (ii) directly copied from the problem specification, or does the LLM generate it from scratch? Can multiple constraints be added to a single function?\n\nLine 150 mentions that using the LLM to generate programs is much more computationally expensive than running programs on a CPU. This is certainly true in general. However, it is unstated how many programs are actually run (considering the exponential number of implementation combinations being considered), and Figure 4 appears to have a datapoint close to 1 million program evaluations per problem. Because APPS is a dataset of competitive programming problems, I’d expect many programs to have long-running evaluations, either from implementation errors (infinite loops) or suboptimal algorithmic design. Even if only 1% of programs run until timeout, the time limit is set to 1 second (a small time limit in actual competitive programming competitions), and we assume that all other programs evaluate instantly, then 1 million programs would evaluate for 2.8 CPU-hours, per problem. Such an amount of compute cannot be swept under the rug and should be discussed! Certainly from an end-user perspective, waiting several seconds for an API call to GPT-4 feels much better than waiting for multiple CPU-hours of program executions. Appendix E.3 does mention parallelizing program executions but the resulting wallclock time per problem is not mentioned.\n\nFor the HumanEval experiment, the writing is less clear about what the main conclusion or takeaway should be. For example, could you summarize Lines 175 - 194 in 1 sentence? The writing would be improved by adding that sentence, and cutting back on some of the details. Similarly, Lines 199 - 204 contain many details that are not useful for readers who do not dig up Parsel’s solutions to the problems mentioned.\n\nFor the VirtualHome experiment, it is not clear how to interpret the numerical results. Does Figure 7 say that the Codex (baseline) plans were the *most preferred among the 3 plans* roughly 30% - 35% of time, which is actually not bad for the baseline? How exactly does the comparison work, considering that Parsel has 2 attempts but the baseline only has 1? Does “X is 70% more likely to be preferred over Y” (Line 283) imply P(X preferred) = 85% and P(Y preferred) = 15% such that 85% - 15% = 70%, or something else? Why not just report P(X preferred) directly? What exactly does “X preferred” mean: “one of the 2 Parsel programs was better than the baseline”, or “the indented Parsel program was better than the baseline”?\n\nOverall I am less excited about the VirtualHome experiments because of the lack of rigorous correctness checking (instead relying only on rankings of accuracy and clarity). The example in Figure A.49 does not alleviate my concerns as the Parsel solutions seem much worse than the baseline’s solution, which itself is not completely correct either. What is the main takeaway from the VirtualHome experiments that was not evident from the other experiments?\n\nI know it is hard to investigate contamination issues considering the opacity of some LLMs, but the potential issue should be discussed in greater detail. For example, is it possible that GPT-3 has seen solutions to the APPS problems while AlphaCode and Codex have not? For the Lisp example (Line 354), it seems likely that the code LLM has already seen example implementations of the Lisp interpreter, and it recognizes the general patterns even without any mention of the word “Lisp” in the Parsel program.\n\n\n## Minor questions and suggestions\n* Line 99: “Parsel implements functions with **a** post-order traversal from **the** function at the root”\n* Figure 4 caption: a period is missing after “AlphaCode [35]” \n* Figure 4 caption: “we examine to understand the effect of evaluation number” is awkward phrasing\n* Line 165: missing space between “HumanEval” and “[12]”\n* Line 319: missing some punctuation (comma or period) between “model” and “and”\n* Line 347: missing space between $\\ge$ and 0.99\n\nLIMITATIONS:\nOne undiscussed limitation is that Parsel likely requires more expensive LLM calls for the same effective number of complete programs sampled, for three reasons:\n\n1. Each transformation from task specification to the high-level plan and then to the Parsel program requires an LLM call, accounting for many tokens that would not be needed by the Codex baseline.\n2. Each function is implemented separately which requires encoding a different prompt for each function with function-specific context.\n3. The decomposition of the entire solution into many functions might lead to more total code tokens sampled compared to a solution obtained without explicit decomposition.\n\nThe paper could use some discussion about these factors, potentially summing the total LLM cost per problem for Parsel versus Codex.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Parsel, a framework for algorithmic reasoning with LLMs. Parsel can be seen as a kind of “programming language” that is implemented with mostly natural language which describes the functionality of the program -- that is, the algorithmic reasoning plan. Then, the Parsel synthesizer can translate the plans into corresponding languages like Python. Finally, the reasoning problem can be solved by executing the synthesized Python code. All the processes are completed by analyzing the Parsel program and prompting LLMs. Experiments show that Parsel can improve the LLM’s coding ability and generate preferred robotic plans.\n\nSTRENGTHS:\n1. Parsel reveals the power of hierarchical planning and decomposition for complex reasoning tasks.\n2. Parsel is a step towards a new kind of programming paradigm. That is, people can program more naturally with Parsel which lowers the programming barrier.\n3. Parsel might be a new kind of prompting method for complex reasoning tasks. Users (or LLM) can write a Parsel program first and then the Parsel synthesizer can automatically decompose the Parsel program into small function pieces and prompt LLM with these pieces, and the synthesizer should compose these pieces to form a whole function.\n4. The result on code generation is promising, although I still have some questions about it.\n\nWEAKNESSES:\n1. Although Parsel can be generated by LLMs, the Parsel synthesizer is not reliable for human users. When coding with Parsel and the synthesized Python code is wrong, users do not know whether the LLM for synthesizer is not working correctly, or if there are bugs in the Parsel code they wrote themselves.\n2. Also, there are no debugging operations provided in Parsel. Users cannot locate the bugs in Parsel, and they cannot debug on Parsel.\n3. The correctness of Parsel synthesizer relies on IO constraints heavily: It seems that IO constraints are the only guarantee for the correctness generation. This is a little bit weird because the main part of Parsel is the natural language description, which is where users spend a lot of time. However, to debug Parsel and ensure the final result is correct, a more effective approach is to provide more detailed and fine-grained IO or allow the LLM to just regenerate to meet IO, rather than making modifications to the natural language or code structure.\n4. The main experiment on APPS (figure 4) doesn’t seem like a fair comparison: Parsel is provided with detailed plans generated with GPT-3 while baselines are not. Providing results like “plans generated by codex (if possible)” “baseline results on GPT-3/4” or just more results with different budget settings on the “ablating the Parsel synthesizer” part would be good.\n5. Experiment settings are not clear, see questions.\n\nQUESTIONS:\n1. Baseline settings are not well explained in figure 4: What does “codex improved” mean? The improvement relative to this baseline is very marginal.\n2. Does Parsel use test generation in the evaluation? Do baselines use it?\n3. Figure 1 shows Parsels ability in multiple languages, but the main paper only showcased Python, making figure 1 overstated.\n4. The github link is 404.\n\nLIMITATIONS:\nLimitations are discussed well in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Parsel, a code generation framework that decomposes a problem specification into subproblems specified in a intermediate pseudocode-like language (the Parsel language) and then searches over combinations of subproblem solutions. The experiments show that decomposing and searching over subproblems leads to gains on standard code generation benchmarks and a robotic planning task.\n\nSTRENGTHS:\n- Very well-written and motivated\n- Thorough experiments on performance gain + analysis of inference cost / scale\n\nWEAKNESSES:\nNo major weaknesses, besides the minor questions on clarity below.\n\nQUESTIONS:\n- When you set a budget of number of evaluations, how do you decide what to evaluate?\n- The \"Functions without constraints\" section was dense.\n\t- How does the parent enforce constraints? \n- Human expert comparison: this was a cool experiment, but the setup wasn't clear to me: how was the expert using Parsel? Were they initialized with a Parsel-generated ?program and then revised from there, or was it more of an iterative process\n- l212-216: I wasn't clear on why this is the conclusion from the human expert results\n- nit Fig 4: it initially wasn't clear that 50 (Improved) was the Codex API and that this was the \"prior work\" you were referring to on line 145\n- nit Fig 4: what is 5@50000?\n- It'd be helpful to have an example of mutual recursion. I see this is possible in theory but couldn't think of when this would occur and how common it is - why would two function implementations depend on each other?\n- How often are child functions shared between parents / called between different parts of the program? In cases where it is shared, it seems like it'd be helpful for the child description to be generated conditioned on _all_ the caller descriptions (whereas from my understanding, the child description is currently generated conditioned on the description of the first caller). I could see this not being an issue for current code generation problems, but it seems like it'd be important in real-world programs.\n- Why is it sufficient to only consider strongly connected components?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Parsel—an intermediate programming language for large language model program synthesis. It decomposes a complex programming task into several strong-connected components and uses large language models to generate candidate code pieces for each component, combined with combinatorial search and test case generation. Evaluation on HumanEval and APPS demonstrate the method's outstanding performance when used with GPT-4 and Codex.\n\nSTRENGTHS:\n- I like the idea of combining language models for function generation and using combinatorial search to find valid programs. It is an approximate form of the fast-and-slow thinking of human problem-solving. The idea is intuitive enough, and the method works well.\n\n- The Parsel language is well-designed as an intermediate programming language in the program synthesis pipeline. It takes advantage of both modeling uncertainty of human language and deterministic descriptive language (IO-specs).\n\n- Experiment results on HumanEval, Apps and VirtualHome are good, reaching outstanding performance with a lower budget (large language model API budget) by trading some local computation cost (combinatorial search), which is something good to see in the LLM era.\n\n- Analysis of Parsel on HumanEval is also valid and persuasive for proving that Parsel can generate longer and more complex programs.\n\nWEAKNESSES:\nOverall, I like the paper. I have a few concerns about the weakness.\n\n- (minor) reproducibility: OpenAI is ending their public API service for Codex recently (I think it is before the submission of NeurIPS), resulting recent paper using Codex is hard to reproduce the experimental results after. I suggest using publicly available GPT-3.5 (text-davinci-003 or gpt-3.5-turbo) to give additional reproducible results.\n\n- Parsel design: I am thinking of if the Parsel language is expressive enough (for example, it may lack data structure design (which is something that makes APPS hard). Can Parsel generate data-structure specs?\n\nQUESTIONS:\nSee weakness. I have no additional questions.\n\nLIMITATIONS:\nThe authors have addressed limitations in the current draft.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}