SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chained components. Chen et al. [12] highlight in their discussion of limitations that language models fail to produce code that requires the simple composition of building blocks. We replicate their experiment with Parsel, using the same docstring and providing the same descriptions as function descriptions. In Fig. 6, we visualize how Parsel solves this. We find that as the number of required components grows, even interchanging the components of only two complete programs (1x2) solves more problems than Codex solves with 16 programs. This illustrates a key point: assuming any independence, the complete set of combinatorial combinations of  $n$  implementations of a program's functions is clearly more likely to be correct than  $n$  samples from that same set. Moreover, this suggests that a framework like Parsel may support increasingly complex and abstracted programs as language models improve.
3.2 Robotic Planning in VirtualHome
Figure 6: Pass rate vs number of chained components. Parsel performance is shown as  $1 \times k$  where  $k$  is the number of complete programs sampled. Orig and Curr are the original [12] and current Codex results, respectively. Codex (@16) corresponds to the pass rate with 16 programs.
We perform a study on VirtualHome [50] to demonstrate that Parsel can also be used for complex robotic planning. VirtualHome is a simulation environment consisting of households with various interactive objects and agents. VirtualHome includes a small set of permissible actions with a strict grammar – tasks like “paint ceiling” may require multiple levels of decomposition, e.g., finding and collecting specific painting supplies, finding and using a ladder, and using the painting supplies, each requiring further decomposition.
To test the effectiveness of Parsel in this domain, we investigate whether Parsel could generate programs to solve tasks in the VirtualHome environment, while using the environment to provide feedback on whether the plan is executable. Specifically, we use Parsel to generate a Python program that can generate action plans in natural language similar to ones used in Huang et al. [28]. In each specified constraint, the produced natural language action plan is translated to formal VirtualHome instructions with minimal regex matching and tested executability. If the instructions can be successfully executed, they are considered valid – however, a potential enhancement could be the inclusion of object-relational constraints on the subsequent state of the world. We include an example of a Parsel program that successfully executed and decomposed a task in VirtualHome in Figure 2. Note we also used a header describing a valid action plan, shown in Figure A.45.
However, as pointed out by [28], while it is easy to check plan executability, it is harder to check correctness, because there are generally many valid ways of accomplishing realistic tasks. Thus, like Huang et al. [28], in order to evaluate the accuracy of the executable plans, we perform a human study. Specifically, we ran two surveys on Prolific [45] where we asked 20 participants to make five rankings each, for a total of 100 rankings per survey. In one survey, we asked participants to rank a set of solutions by how accurately each solution accomplishes the task while in another we asked about how understandable the solution was. Two versions of distinct Parsel solutions were shown (where multiple executable Parsel solutions were available), one which included the indented function names used to solve the tasks alongside their step-by-step solutions, and one which only included step-by-step output. We compared both Parsel solutions to an also-executable baseline solution, based on Huang et al. [28]. We include more details about the survey format and executability in Appendix P.
Humans ranked the Parsel solutions as more accurate than the baseline. In each ranking, both the indented and non-indented Parsel solutions were consistently ranked higher than the baseline. In accuracy, standard Parsel solutions (as well as indented solutions) were identified as more accurate than the baseline solutions in  $69\%$  of comparisons, more than twice as often as the baseline. In clarity, the standard Parsel solution was  $70\%$  more likely to be preferred over the baseline while the indented Parsel solution was  $50\%$  more likely to be preferred compared to the baseline. There was no notable difference between indented and non-indented Parsel solutions in either accuracy or clarity. Specifically, note that all of these comparisons are reported pairwise, and Figure 7 compares the rankings given to the (non-hierarchical) Parsel-generated plan and the Codex-generated solution.
In essence, participants did not appear to prefer either Parsel plan format over the other, but given a choice between one Parsel-generated
Figure 7: Robotic plan comparison. Parsel-generated robotic plans were consistently selected as more accurate and clear when compared to a direct generation approach like in Huang et al. [28].
plan and its corresponding Codex-generated plan, they typically considered the Parsel-generated plan to be more accurate and clearer.
Step-by-step problem solving with LLMs. Many works show that step-by-step reasoning benefits LLM performance [51, 54, 42, 60, 37, 34] and correspondingly, that this performance can be improved with guidance and tool use [71, 68, 63, 59, 22]. Acquaviva et al. [2] encouragingly showed that humans, when asked to explain how to solve problems in the Abstract Reasoning Corpus [14], tended to provide step-by-step hierarchical descriptions with many verification steps. Moreover, Wies et al. [61] presents a theoretical argument showing problems that can be learned efficiently if decomposed but require exponentially many examples w.r.t. length if not decomposed.
Program synthesis. Program synthesis is the long-standing challenge of generating programs from high-level specifications [26], such as input-output examples [7, 25] and/or natural language descriptions [52, 65, 19]. Program synthesizers typically search the exponentially large space of programs. Consequently, synthesizing large, complex programs remains an open challenge. Recently, library learning has shown a way to make progress: even complex programs can be short in terms of the right high-level library. In turn, this library can be progressively induced from solutions to simpler synthesis problems. This idea is embodied in DreamCoder [23, 9]. Library learning requires a rich distribution of related tasks so that patterns emerge from solutions to simple problems. Patterns are abstracted into useful library functions, enabling short solutions to more complex problems. Parsel also aims to synthesize complex programs by decomposing them. However, Parsel programs specify decompositions, so a family of related tasks is not required.
Figure 8: Scaling performance. The probability of passing an APPS competition-level problem increases quadratically with respect to the log of the number of Parsel programs sampled (left) and the number of implementations sampled (right).
LLMs for formal environment multi-step planning. Also encouragingly, several existing works can be expressed in Parsel. For example, Huang et al. [28] and Brohan et al. [10] showed that language models can automatically generate step-by-step algorithms for robotic agents in language. In both cases, the generated language corresponds directly to pre-implemented low-level robotic abilities. This could be expressed by providing a task description and constraints evaluating that the high-level task was completed successfully. In addition, Jiang et al. [30] proposed a framework to generate formal proofs in formal theorem-proving languages from informal proofs by first generating an intermediate natural language proof sketch. This could be expressed in Parsel by generating each sketch step as a function and using formal verification for each lemma as the Parsel validation step.
Programming languages and frameworks incorporating language models. Other works have explored programming languages that incorporate language models. For example, Cheng et al. [13] explored the introduction of a language-model-based evaluation function, which would allow  $f('North America?', 'U.S.')$  to automatically return 'yes' by referencing the knowledge of the language model, and showed that they could also generate programs using this tool with a language model and Beurer-Kellner et al. [8] explored a related SQL-style LLM-querying language. In addition, Dohan et al. [20] presents an inference-focused framework for language models more broadly for probabilistic graphical models composed of language model actions. Unlike LM Cascades [20], we primarily focus on the constrained generation of programs, instead of leveraging language models as functions within a particular program.
Testing code language model outputs. Related works have explored the capacity of assert statements to constrain the generation space of code LLMs on functions [6, 12, 35]. The automatic generation of unit tests in Chen et al. [11] allowed them to filter many programs before performing final evaluations, significantly improving the pass rate per evaluation. CodeT [11] applies a language model to generate a large number of tests, presuming that some of them will be incorrect. However, instead of simply selecting the program passing the most tests as correct, CodeT groups together generated programs by the subset of the tests that they pass – it then returns any program from the group with the most tests passed in total [11]. Chen et al. [11] intuitively motivates this by the Anna Karenina principle: “Happy families are all alike; every unhappy family is unhappy in its own way.”
Notably, both the number of final evaluations and the sample budget are important, and we find that combining Parsel with automatic test generation improves performance given a fixed sample and evaluation budget. In addition, Merrill et al. [38] proves essential constraints on what can be learned from assertions alone and, more crucially, what cannot.
We note that Parsel has important limitations, largely stemming from its dependence on closed LLMs. First, LLMs naturally underperform on languages underrepresented in their training data, like Parsel, affecting the quality and reliability of the generated code, and these closed models do not offer a training API. In addition, reliance on Codex and GPT-4 introduces vulnerabilities due to potential abrupt changes in behavior – since this project's start, access to Codex (which is free) has become much more limited and the much-better GPT-4 is also much more expensive. We unfortunately found a 2.7B CodeGen model [39], evaluated like Codex on APPS, solved no problems. However, we anticipate that improvements in open-source code LLMs will mitigate these issues.
Despite the limitations, elaborated in Appendix B, and the challenges of using a language model to generate a language that it has never seen before, the generated Parsel was able to implement robust algorithmic reasoning. There are a few natural questions that arise from this work. How robust is Parsel to the language used to specify the problem? When solving hard problems, is it better to increase the number of Parsel programs sampled or the number of program implementations sampled? We visualize Parsel's performance on the explored subset of APPS in Figure 8 to try to investigate some of these questions. In general, it appears that the probability of passing increased quadratically  $(\mathrm{R}^2 \geq 0.99$  for all curves) with respect to the log of both the number of Parsel programs sampled and the number of implementations sampled. Clearly, since the pass rate is bounded between 0 and 1, this trend cannot continue indefinitely. However, given the rate limit associated with calls to the Codex API (officially, 40,000 tokens or 20 requests per minute, but in practice, we consistently ran into the rate limit with  $\approx 10\%$  of that usage), we were not able to identify the inflection point. Li et al. [35] indicated the pass rate curves become log-linear, but we cannot guarantee this holds for Parsel.
Furthermore, we highlight that the question we (and, in our view, most prior work) were looking at in our APPS experiments is, given hard problems and limited by one's generation budget, what is the chance of finding a correct solution? However, if one instead views each test as a successive submission, where any failure is comparable to a crash in production, one would likely take an approach like the one in our HumanEval test generation experiments, one that uses interpreter feedback to revise solutions, or perhaps even one that uses the language model to simulate the result of tests. Both interpretations are important, but they highlight different concerns: even if one can guarantee success upon submission given a trillion generated tokens, this has limited practical applicability. A method's ability to efficiently generate reasonable solutions in the first place is one key bottleneck, and assuming an imperfect model, the ability to identify them is another.
Finally, we highlight that Parsel is already capable of synthesizing larger, more complex programs. For reference, we include a working Lisp interpreter in Appendix I, based on Norvig [41], which we translated to Parsel with 21 functions. We make sure to include no explicit references to Lisp in the Parsel code and provide a header to tell Parsel to keep track of environments as dictionaries. However, this is clearly an imperfect substitute for a true object-oriented programming implementation, and we believe adding better object-oriented features to Parsel would be a valuable future direction.
We hope that Parsel provides a broadly useful framework for several groups: for programmers, Parsel should provide a language for robust code generation without the need to evaluate the underlying code; for students, Parsel should allow the teaching of algorithmic reasoning with less emphasis on syntax and more emphasis on problem-solving, similarly to mathematics curriculum; for language models, Parsel should facilitate hierarchical task decomposition.
Ultimately, as we discuss in detail in Appendix C, Parsel has many natural extensions building on prior work in code synthesis from automatic test case generation [17, 12], reranking solutions by model-judged quality [69], more clever recursive decomposition [49], bootstrapping increasingly complex programs [4, 43, 68], to generating language model prompts as part of a target language [20, 31]. As we discuss in Appendix D, we also envision that Parsel may be useful for theorem proving. Thus, Parsel helps fill the gap between reasoning and execution, providing a new approach to complex, hierarchical reasoning tasks for language models.
We would like to thank Jesse Mu, Tianyi Zhang, Rose E. Wang, Allen Nie, Ben Prystawski, Xindi Wu, Fan-Yun Sun, Isaac Kauvar, Xi Jia Zhou, John Thickstun, and Shikhar Murty for their helpful feedback, as well as Rishi Bommasani for highlighting highly relevant works. We also thank the Stanford HAI-Google collaboration for their Cloud Credit grant. Qian Huang is supported by an Open Philanthropy AI fellowship. Gabriel Poesia is supported by a Stanford Interdisciplinary Graduate Fellowship. In addition, this work was partially supported by National Science Foundation Grant No. 2302701 and National Science Foundation Expeditions Grant No. 1918771.
[63] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[64] A. Ye, C. Cui, T. Shi, and M. O. Riedl. Neural story planning, 2022. URL https://arxiv.org/abs/2212.08718.
[65] P. Yin and G. Neubig. A syntactic neural model for general-purpose code generation. ACL, 2017.
[66] P. Yin, B. Deng, E. Chen, B. Vasilescu, and G. Neubig. Learning to mine aligned code and natural language pairs from stack overflow. In 2018 IEEE/ACM 15th international conference on mining software repositories (MSR), pages 476-486. IEEE, 2018.
[67] P. Yin, W.-D. Li, K. Xiao, A. Rao, Y. Wen, K. Shi, J. Howland, P. Bailey, M. Catasta, H. Michalewski, A. Polozov, and C. Sutton. Natural language to code generation in interactive data science notebooks, 2022. URL https://arxiv.org/abs/2212.09248.
[68] E. Zelikman, Y. Wu, J. Mu, and N. Goodman. STar: Bootstrapping reasoning with reasoning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=3ELRdg2sgI.
[69] T. Zhang, T. Yu, T. B. Hashimoto, M. Lewis, W.-t. Yih, D. Fried, and S. I. Wang. Coder reviewer reranking for code generation. arXiv preprint arXiv:2211.16490, 2022.
[70] K. Zheng, J. M. Han, and S. Polu. Minif2f: a cross-system benchmark for formal olympiad-level mathematics. *ICLR*, 2022.
[71] D. Zhou, N. Scharli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022.
[72] H. Zhou, A. Nova, H. Larochelle, A. Courville, B. Neyshabur, and H. Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint arXiv:2211.09066, 2022.
1 select_airport_cities(city_road_cost, city_airport_cost): given a matrix representing the  $\rightarrow$  cost of building a road between any two cities, and a list representing the cost of  $\rightarrow$  building an airport in a city (where any two cities with airports are connected), return  $\rightarrow$  a list of the cities that should have airports built in them to minimize the total cost  $\rightarrow$  of building roads and airports such that all cities are connected. The list should be  $\rightarrow$  sorted in ascending order.   
2  $[0,3,3],[3,0,3],[3,3,0],[0,0,0] - > [0,1,2]$    
3  $[0,3,3],[3,0,3],[3,3,0],[10,10,10] - > []$    
4  $[0,10,3],[10,0,11],[3,11,0],[1,4,5] - > [0,1]$    
5 sky_city_cost(city_road_cost, city_airport_cost): given a list of lists representing the  $\rightarrow$  cost of building a road between any two cities, and a list representing the cost of  $\rightarrow$  building an airport in a city, return a new cost matrix with a new node corresponding  $\rightarrow$  to the sky.   
6  $[1,2,3],[1,2,3],[1,2,3],[4,5,6] - > [[1,2,3,4],[1,2,3,5],[1,2,3,6],[4,5,6,0]]$    
7 minimum Spanning_tree(cost_matrix): given a list of lists representing the cost of each  $\rightarrow$  edge, return an adjacency matrix corresponding to the minimum spanning true.   
8  $[0,1,3,4],[1,0,2,100],[3,2,0,5],[4,100,5,0]] - >$ $\rightarrow$ $[0,1,0,1],[1,0,1,0],[0,1,0,0],[1,0,0,0]]$    
9 final_node_connectors(adjacency_matrix): given a list of lists representing an adjacency  $\rightarrow$  matrix, return a list of the nodes connected to the final node. However, if only one  $\rightarrow$  node is connected to the final node, return an empty list.   
10  $[0,1,0,1],[1,0,1,0],[0,1,0,0],[1,0,0,0]] - > []$    
11  $[0,1,0,1],[1,0,1,0],[0,1,0,1],[1,0,1,0]] - > [0,2]$
Figure A.9: A potential programming assignment focused on problem-solving rather than implementation. The top-level function and asserts would be the assigned problem (which Codex [12] does not seem to be able to solve directly), while the other functions would be the student solution.
A Implications
Parsel is a natural language compiler framework that bridges the gap between natural language and programming language by allowing programmers to write high-level algorithmic designs in natural language and automatically compiling them into valid code. This has potential benefits for programmers, students, and code language models.
A.1 For Programmers
A.1.1 Current Limitations
First, programming generation language models like Codex continue to be constrained primarily to individual functions, rarely exceeding a few dozen lines in practice [12, 58]. This is still a dramatic shift from foundational earlier works, which focused on the association between one line of natural language pseudocode with one line of code [33] or a line of text to a StackOverflow snippet [66]. Yet, these models perform worse the more unusual the desired functions are, and recent research suggests that people using these language models are more likely to introduce buggy code [47], although this is not yet conclusive [53].
A.1.2 Potential Benefits
On the other hand, results from Google and others indicate that professionals can write code more efficiently with large language models, and the benefits will likely only improve as they improve [58]. Since Parsel requires constraints that ensure functions behave as expected, this should encourage bug-free programs and avoid the need for manually checking that specific underlying functions are correct. Furthermore, a function written in Parsel is likely to be more resilient to breaking changes in the target language, especially syntactic changes (e.g. Python2 to Python3). In addition, a natural extension would draw on work on automatic unit testing [17] to suggest additional constraints where behavior is ambiguous between implementations of a function. However, for Parsel to be practically useful for real-world programmers, we expect multiple improvements to be necessary, including additional object-oriented features.
A.2 For Students
A.2.1 Current Limitations
In addition, these language models pose serious challenges for programming pedagogy – existing introductory programming classes rely extensively on teaching syntax and how to implement algorithms over how to solve problems with them. Free language model-based tools like Copilot can essentially solve many of these introductory assignments directly, function by function. Those which cannot be solved currently will be increasingly solved [18].
A.2.2 Potential Benefits
Many students currently introduced to programming struggle with learning syntax and debugging unclear compiler or interpreter errors. However, abstracting away these details with a natural-language coding language will likely make learning to code more accessible to students who are just beginning to code. In addition, stepping away from implementation-focused assignments will allow a focus on higher-level problem-solving assignments earlier. These will allow for assignments that are more like those in mathematics. For example, for a problem like Figure A.9, instead of choosing between requiring students to manually implement a problem-solving focused question like the top-level description of, or requiring teaching assistants to manually evaluate the reasoning for correctness, one could ask them to implement a solution in Parsel.
A.3 For Code Language Models
A.3.1 Current Limitations
Traditional programming languages result in some unique challenges for language models. For example, unlike natural languages, traditional programming languages are far less robust to slight variations in wording. In addition, traditional programming languages require many tokens for syntactic details and in some cases, may take many lines to express what can be expressed far more simply in language. For example, referring to a shortest-path algorithm or Conway's game of life takes far fewer tokens than actually implementing them. However, even with fairly nonstandard problems, LLMs have shown remarkable algorithmic generalization ability [36, 62, 3, 72]. One alternative that has been explored is conversational code generation [40, 67]. However, these approaches have primarily focused on highly imperative programming structures. Moreover, they still require having the full program in context and do not clearly generalize to complex hierarchical programs with many functions.
A.3.2 Potential Benefits