SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D. Goodman, Nick Haber  
Stanford University
{ezelikman, qhwang, poesia, ngoodman, nhaber}@stanford.edu
Despite recent success in large language model (LLM) reasoning, LLMs struggle with hierarchical multi-step reasoning tasks like generating complex programs. For these tasks, humans often start with a high-level algorithmic design and implement each part gradually. We introduce Parsel, a framework enabling automatic implementation and validation of complex algorithms with code LLMs. With Parsel, we automatically decompose algorithmic tasks into hierarchical natural language function descriptions and then search over combinations of possible function implementations using tests. We show that Parsel can be used across domains requiring hierarchical reasoning, including program synthesis and robotic planning. We find that, using Parsel, LLMs solve more competition-level problems in the APPS dataset, resulting in pass rates over  $75\%$  higher than prior results from directly sampling AlphaCode and Codex, while often using a smaller sample budget. Moreover, with automatically generated tests, we find that Parsel can improve the state-of-the-art pass@1 performance on HumanEval from  $67\%$  to  $85\%$ . We also find that LLM-generated robotic plans using Parsel are more than twice as likely to be considered accurate than directly generated plans. Lastly, we explore how Parsel addresses LLM limitations and discuss how Parsel may be useful for human programmers. We release our code at https://github.com/ezelikman/parsel.
To a language model for code (as for a human), each new token is a new chance to break the program. Chen et al. [12] highlighted this issue in a simple synthetic experiment: when asked to generate a program with a series of simple string transformations, the performance of their code language model, Codex, drops dramatically with the number of steps. As they pointed out, a human who can implement a few building blocks should be able to compose these blocks with arbitrary length.
Unlike other token generators, human programmers have (mostly) learned to break down complex tasks into manageable parts that work alone (i.e., are modular) and work together (i.e., are compositional). And, when human-generated tokens cause functions to break, the functions can ideally be rewritten independently of the rest of the program. In contrast, naively, we expect code language models to generate token sequences that are correct in their entirety. Motivated by this, we study how to leverage language models to decompose problems and assemble their compositional solutions.
We introduce Parsel, a framework that enables the decomposition, implementation, then composition of complex programs from natural language. The process consists of three main steps: 1) A language model generates a Parsel program by decomposing a natural language task into natural language function descriptions. 2) For each function description, a language model generates modular implementations. 3) The Parsel synthesizer then builds up programs by testing minimal, combinatorial groups of implementations against sets of constraints (e.g. input-output examples).
Figure 1: Parsel overview. A human or LLM writes a task decomposition (in Parsel), which is split into its strongly-connected components (SCC), and then the Parsel synthesizer uses a code LLM and a constraint solver to implement and compose each SCC. When all functions have constraints and there is no recursion, each function is its own SCC. We provide a more detailed figure in Appendix H.
Thus, generating and implementing Parsel mirrors a pattern in human reasoning – decomposing an abstract plan until it can be solved automatically [55] – and this compositional structure also benefits language models. We show that LLMs can generate Parsel with only few-shot prompting. Using their solutions as input to the Parsel synthesizer, we outperform prior work on competition-level problems in the APPS dataset [27], including AlphaCode [35] and both versions of Codex [12, 11]. We also show that with GPT-4 [44], not only can we generate Parsel zero-shot just by describing its format, but this allows it to substantially outperform GPT-4 alone on HumanEval [12], with and without generated tests. By automatically generating and implementing Parsel, LLMs also propose robotic plans that are more accurate than in prior work on language models as robotic planners [28]. Broadly, we formulate Parsel as a general-purpose algorithmic reasoning framework.
In this section, we provide an overview of the Parsel framework. First, we show how programs can be specified in Parsel. Afterward, we explain how language models can generate Parsel from task descriptions. Finally, we present the Parsel synthesizer that allows us to actually implement Parsel programs into runnable code. Specifically, we further discuss how we handle various challenges in solving the constraints, from recursion to test underspecification. There are several steps to synthesizing a Parsel program, which we explicate in this section. We visualize the details in Fig.1 and provide a high-level pseudocode in Fig. A.11.
With Parsel, we balance two challenges. On one hand, function implementations must be modular to fully leverage combinatoriality. By requiring function signatures and descriptions to be specified (or generated), we ensure functions can call others without specific implementation information. On the other hand, it must be possible to avoid combinatorial explosions. By factoring the dependency graph into strongly-connected components (SCCs), we ensure the complexity grows exponentially only with the size of the largest strongly-connected component but not the total number of functions.
2.1 Specifying Parsel Programs
To formally specify a high-level algorithmic design, we develop a simple intermediate language, also called Parsel. It is designed to be accessible to programmers, code LLMs, and students, as discussed further in Appendix A, and inspired by many works (Section 4). In Parsel, each line contains a function description, a constraint, or a reference to a description. They all obey scoping rules, with some nuances per target language. In short: descriptions specify what a function does in natural language, optionally preceded by a function name and its arguments (if not included, they are inferred); references indicate to the language model that a function should call another function described elsewhere; constraints validate that a function is implemented correctly (if not provided, they may be generated automatically using the language model). We elaborate on syntactic details in Appendix R. We provide some examples of Parsel programs in Figures 2 and 3. For example, in Figure 2, task計劃(): return a list of strings ... represents a description, while the call to collatz_recursion by recursion_rule is a reference.
Figure i: Parsel to VirtualHome (robotic planning)
Figure ii: Parsel to Python
Figure 2: Examples using Parsel to compile programs for VirtualHome [50] procedures and Python. Note the columns on the generated examples are only there to allow them to fit compactly - each program implementation is one contiguous solution. Colors for Parsel code on the left are used to indicate constraints and references. All other Parsel lines shown are definitions.
2.2 Generating Parsel Programs
We use a language model to generate Parsel programs from arbitrary natural language task descriptions as a few-shot translation task (or zero-shot with GPT4 [44]). Explicitly writing out a preliminary plan can be helpful, so we first ask the model to (zero-shot) generate a plan in natural language by thinking step-by-step (motivated by Kojima et al. [32]) and then prompt the model to translate the plan to Parsel. These steps correspond to the first and second arrows in Figure 3, respectively. In some domains, in particular for robotic planning, we find that this intermediate step is not necessary (while increasing cost) and can ask the language model to directly translate the task description to a Parsel solution.
2.3 Implementing Parsel Programs
Given the Parsel program, a core technical challenge is how to implement it in a modular way (to fully leverage the advantage of decomposition). Here we propose the Parsel synthesizer: At a high level, to implement Parsel programs with the Parsel synthesizer, we first use a language model to generate a set of candidate implementations of each of the functions based on their descriptions and then search over minimal sets of combinations of the functions to find ones that satisfy the provided constraints. The step described in this section corresponds to the rightmost arrow in Figure 3.
2.3.1 Implementing Functions
A function is implemented by first aggregating descriptions and function signatures of its (potentially zero) children to form an LLM prompt (for Python, we generate a prompt as if the child functions are imported and use their descriptions as comments). Crucially, this facilitates easily changing child implementations. A code LLM is then queried using the description's text as a docstring and the description's function name and arguments for the signature; full prompts shown in Appendix K.
2.3.2 Searching Over Function Combinations
Sequential case. The most straightforward case is when all functions have constraints (e.g., unit tests), and no functions have recursive dependencies (e.g., Fig. A.9). We start by considering this case. This defines a clear topological order on functions so they can be implemented sequentially. In this situation, Parsel implements functions with post-order traversal from function at the root, generating implementations and finding one passing the specified constraints for each function. In
Figure 3: Parsel pipeline. Overview of the full Parsel pipeline for an APPS example. Each step is performed by a language model. We include the complete task, sketch, and program in Appendix S.
other words, without any cycles in the call graph, we can start by implementing the leaf functions first, then their parents, etc., until the program is implemented. However, in practice, many programs have more complex structures and constraints.
Recursion. Permitting mutual recursion (e.g., a function  $f$  that depends on  $g$  while  $g$  also calls  $f$ ) introduces the need for joint implementation of functions. However, naively considering all possible implementations is intractable for large programs. Specifically, the number of possible implementations of a program with  $k$  functions and  $n$  implementations per function is  $O(n^{k})$ , exponential in the number of functions. We propose a more efficient solution, inspired by Cocke and Kennedy [16]. We reference the function call graph, identify its strongly-connected components (SCCs), and for each SCC consider all possible sets of function implementations until one set satisfies all constraints. For example, for possible implementations of functions  $f, g, h$  forming an SCC of a call graph, with  $I(f)$  corresponding to the language-model-generated implementations of  $f$ , we evaluate uniformly random samples without replacement from  $I(f) \times I(g) \times I(h)$ .
Functions without constraints. For functions with no constraints, we can conveniently use the same approach as above by reformulating the call graph as a "test dependency graph" and adding an edge from a test-less child to its parent. That is, if a function has no constraints, it depends on its parents to enforce constraints on its implementations. This could also allow us to automatically introduce new children via automatic decomposition without needing to also generate constraints for those children (see Subsection R.3). Alternatively, we support automatic test generation, prompting the language model to generate tests. For generated tests, we select the best implementation based on CodeT score [11], as some generated tests could be incorrect. In practice, we use the first (dependency) approach for inner functions and the second (test generation) approach when the top-level function has no tests. Note an opposite extreme from the sequential case: if no function or only the top-level function has constraints, we simply implement each function some number of times and then test all combinations. More generally, if  $n$  functions with  $m$  implementations where the sizes of the SCCs are at most  $c$ , partitioning reduces necessary evaluations from  $O(n^{m})$  to  $O(\frac{n^{c}m}{c})$ .
We explore the ability of Parsel to generate programs for various tasks, including competitive coding, a standard coding benchmark, and robotic task planning. These three categories represent related but distinct kinds of algorithmic reasoning, with varying levels of abstractness and generalizability. Competitive programming represents one of the few benchmarks for evaluating code generation tools that benefits greatly from decomposition.<sup>1</sup> By evaluating Parsel on this breadth of tasks, we hope to better understand its generality as a framework for algorithmic reasoning.
<sup>1</sup>Unfortunately, benchmarks that evaluate language models on complex real-world programming tasks requiring thousands of lines of code do not currently exist. Developing such a dataset is an important direction for future work.
Figure 4: Left: Competition-level problem pass rate. Comparison of Parsel's pass rate on competition-level APPS problems [27] against direct code generation with Codex [12, 11] and AlphaCode [35] Labels correspond to  $\text{pass} @ n$  and  $\text{pass} @ n \times k$ . Sample budget is measured in effective number of complete programs generated by the code LLM. (These measures are further explained in Subsection 3.1.) Right: Pass rate vs number of evaluations. Parsel generates and evaluates many programs with a small inference budget, by combinatorial composition. Though evaluation is cheap compared to generation, we examine to understand the effect of evaluation number. For evaluation budget analysis, we evaluate  $\text{pass} @ 8 \times 16$  on a random 200-problem competition-level APPS subset, and sample combinations of function implementations at random for a given budget.
3.1 Python Code Generation
Solving competition-level problems. We evaluated Parsel on the competition-level subset of the APPS [27] dataset as follows: first, we zero-shot prompt GPT-3 (text-davinci-002) with the problem statement, requiring it to first propose a high-level solution and explain why it is correct, asking it to "think step by step to come up with a clever algorithm" [32]. We then prompt Codex [12] to translate the generated solution into Parsel code, providing three examples of valid code. We then attempt to synthesize the Parsel code and evaluate implementations to measure their pass rate, i.e., the proportion of solved problems. We report "pass@  $n \times k$ ", where  $n$  is the number of Parsel (i.e. intermediate, semi-formal) programs generated and  $k$  is the number of implementations per function. Prior work has emphasized performance as a function of the LLM sample/inference budget measured in complete programs generated [12, 35]; for Parsel, we use the effective number of complete programs generated from the LLM for comparison, which is  $n \times k$ . We compare to directly generating solutions from Codex [12, 11] and AlphaCode [35]. We visualize the pass rate in Figure 4, finding that Parsel substantially improves over prior work [11], from  $14.5\%$  to  $25.5\%$ . We include full prompts and further evaluations in Appendix N and discuss the evaluation runtime costs in depth in Appendix T.8.
Since Parsel will "mix-and-match" function implementations, the number of distinct candidate programs available to test grows exponentially with  $k$ , while the generation cost for the components is linear. Generating each program likely requires on the order of petaFLOPs of compute [21], dwarfing the computational cost of testing complete programs. Yet when few functions have constraints (as in this evaluation) or there are complex recursive dependencies, it could become expensive to exhaustively evaluate all possible programs. Thus, we also explore the performance of Parsel as a function of the number of evaluations, as shown in Fig. 4. We find that the performance improves log-linearly with the number of evaluations, justifying spending compute to evaluate all programs implied by generated function implementations. One can interpret this to suggest the number of Parsel programs and combined implementations play similar and important roles. Thus, one may compensate for implementation ability with Parsel-generation ability and vice-versa (to a limit).
Ablating the Parsel synthesizer. We perform an ablation to explore the impact of the Parsel intermediate language: we provide the same high-level plans to Codex, but instead of translating them to Parsel and synthesizing programs as described, we translate them to Python directly. We match the code generation budget (effectively complete programs inferred), generating 16 Python implementations $^2$  per high-level plan on 100 randomly sampled problems, and find that the pass@8 × 16
${}^{2}$  Sampling the longer solutions resulted hitting a rate limit more frequently, even when using the same number of total tokens per prompt; a full 128-solution comparison on 1000 problems would potentially take months.
performance drops to  $6\%$  from  $25.5\%$  for the full pipeline. Not only did this approach solve substantially fewer problems, but the problems that it did solve were a strict subset of those which we were able to solve with Parsel. This indicates that for Parsel with Codex, the step-by-step decomposition provided by the high-level plan is not by itself responsible for the observed improvements by Parsel.
We additionally perform an ablation to understand how Parsel's performance without a high-level plan. On a 200-problem competition-level APPS sample, we found that, without a high-level plan, the accuracy falls to  $13\%$  (from  $25.5\%$ , with the same configuration). This is better than the ablation where Parsel wasn't used, suggesting that Parsel plays a larger role than the high-level plan on APPS, but both are necessary. Note part of this improvement is from the challenge of prompting Codex to generate Parsel, given long APPS problem statements.
HumanEval. We next tested Parsel on HumanEval[12]. Because these problems are substantially simpler, they provide an opportunity to evaluate pass@1 by generating tests automatically. Furthermore, this smaller and simpler set of problems enables us to assess the (much more costly) GPT-4. We found a significant improvement over the state-of-the-art pass@1 performance—from  $67\%$  to  $85\%$ . We found that by describing the Parsel format (see Appendix O for details), noting the role of indentation in providing hierarchy and explaining the format of a description, GPT-4 was able to translate natural language plans into Parsel zero-shot. Moreover, the combination of Parsel and GPT-4 surpassed GPT-4 alone on HumanEval, both with and without generated tests, further emphasizing the importance of leveraging intermediate languages like Parsel for improved code generation performance in complex tasks.
The  $85.1\%$  pass rate with generated tests, evaluated based on the CodeT score [11], was obtained by sampling until three Parsel programs had one implementation passing a generated test and then selecting the one passing the most tests. On average, this required 3.5 Parsel programs per problem, with 8 implementations each (i.e.  $< 32$  complete implementations on average). Parsel also outperforms GPT-4 with CodeT alone on the same set of tests, and using a comparable 32 samples, which passes  $81.1\%$  of the problems.
Additionally, when evaluating performance in terms of pass@any (i.e., not filtering attempts by CodeT), we discovered that by allowing up to 8 Parsel programs (still with 8 implementations each), the pass rate increased to  $91.5\%$ . In contrast, when allowing up to 64 directly generated programs, the pass rate was only
Figure 5: Pass rates on HumanEval. We analyze the effect of using Parsel with GPT-4 [44] on HumanEval [12]. Given the same program inference budget, the pass rate with Parsel improves regardless of whether one filters by generated tests.
82.3%. This difference becomes more pronounced when generating 128 programs (128 directly vs.  $16 \times 8$  with Parsel), improving from  $84.1\%$  to  $93.9\%$  (See Fig. 5).
While there are still eight tasks that are never solved with or without Parsel, GPT-4 solves eighteen problems with Parsel that it cannot solve alone. For eight of these, the solution has six or more functions (excluding defined but unused functions). Of the problems Parsel solved, the ones GPT-4 could not solve itself required 4.2 functions on average, while those GPT-4 could solve required only 2.8 functions (sampled over 50 problems). We find GPT-4 alone solves two problems pass@any that GPT-4 with Parsel cannot: decimal_to_binary and decode_cyclic. The decode_cyclic solutions match Chen et al. [12]'s Figure 2 (but not the dataset's canonical solution) with minor comment differences, indicating possible contamination. In decimal_to_binary, the generated Parsel solutions often unsuccessfully rely on statefulness: for example, divide_by_2_withremainder calls calculateremainder and updatedecimal, but updatedecimal cannot mutate an integer.
Most crucially, these experiments indicate that, even when given the same set of generated tests and program generation budget and selecting only one best solution, Parsel significantly increases the probability that the solution is correct on the ground-truth set of tests.
Comparison to a human expert. In our fully automated pipeline, an LLM generates a Parsel program, which Parsel attempts to synthesize into a solution. We also evaluated how well an expert Parsel user would perform by directly writing Parsel and interacting with the Parsel synthesizer. As a case study, one of our authors with significant prior experience in competitive programming was presented with 10 randomly-selected competition-level Codeforces problems from the APPS dataset, writing Parsel from scratch in order to solve the problems The participant successfully solved 5 of the problems within a 6-hour time frame.
In comparison, when GPT-3 generated Parsel solutions to these problems, it had a success rate of 2 out of 10 problems with 8 attempts per problem (of course, in a much shorter time-frame). This result suggests that, given a suitable decomposition, the Parsel synthesizer can effectively generate complete correct solutions to hard problems – thus, a major point for improvement in the fully automated pipeline lies in the first stage (Parsel generation). In other words, improving the ability of models to decompose tasks into Parsel programs is a key bottleneck and is an important direction for future work.
While the participant could solve problems that GPT-3 could not, the participant found some aspects of working with Parsel counterintuitive. For example, one of the most effective ways to provide additional details to ensure a working solution was to write additional tests rather than to clarify the meaning in the function descriptions. Given the stochasticity of language model generation, there was concern that changing the description of a child function could break its parent – however, in practice, this seemed rare. We include the participant's solutions and the associated problems in Appendix Q.