SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Parsel allows code language models to stay closer to natural language when generating code, which corresponds more closely to their primary source of training data. Moreover, it allows complex but standard methods to be described concisely, requiring fewer tokens to generate. One exciting additional benefit is the potential to generate solutions recursively: if the Parsel compiler is unable to find a solution for a set of functions, it should be possible to prompt the model to define new helper functions. In fact, we find that often the model attempts to reference undefined auxiliary functions when defining complex functions (e.g. "count_living_neighborsgrid,i,j)" in Conway's game of life), and as a result support an optional argument where the model can attempt to resolve NameErrors automatically by attempting to implement functions.
There are several limitations to the current implementation of Parsel. First, Parsel relies on a code LLM to generate implementations of individual functions, and the quality of these implementations can vary depending on the specific model used and the complexity of the function descriptions. In particular, Parsel may struggle to generate correct code for individual functions with complex behavior (i.e. functions that Codex cannot implement). However, this can be mitigated by decomposing the complex functions into simpler ones that can be implemented more easily.
The current implementation of Parsel may struggle to generate correct code when there are many functions with complex dependencies or without constraints. This is because the number of implementation combinations to consider grows exponentially with the size of the largest strongly connected components. As discussed, this can limit Parsel's performance on some programs. However, approaches like Chen et al. [11] may be able to mitigate this.
Code LLMs, unfortunately, do not perform well on languages underrepresented in their training data – with few examples to learn from, LLMs may struggle to generate correct code in these languages [5]. However, some LLMs can learn new languages in context, allowing them to generate code in languages not in their training data [5]. These limitations can impact the quality and reliability of the code generated with Parsel. In addition, because code LLMs have never been trained on Parsel, this harms their ability to generate it. While we could wait for Parsel to gain widespread adoption, it should also be possible to translate many existing codebases to Parsel. We include a proof-of-concept backtranslation/decompilation study in Appendix L.
In addition, the best open-source code LLMs currently available e.g. PolyCoder [62] substantially underperform Codex, while Codex is competitive with other traditional LLMs on reasoning tasks [36]. However, this dependence on closed models creates a vulnerability, as the providers of closed LLMs can change behavior (e.g. rate limits or model implementations) without warning. Indeed, between the time we started working on Parsel and this version of the paper, OpenAI ended widespread access to Codex, now available only by request.
Because of this, we evaluated a 2.7B CodeGen model from Nijkamp et al. [39] with Parsel in the same configuration we used when evaluating APPS on Codex (in the 8x16 configuration). We found that it could solve none of the random 25 problems which we evaluated it on. However, despite these limitations, the current Parsel implementation has shown promising results in generating correct code for a variety of functions and languages. Many limitations will likely be ameliorated as code LLMs improve.
C Future Work
In the future, we hope to more deeply integrate automatic unit test generation, especially in combination with user-provided tests [17, 11]. One method would be to identify edge cases and check whether the set of functions that successfully solve all existing tests disagree on any new tests. This could permit automatic decomposition without exponential growth in implementation combinations. Techniques like those proposed in Zhang et al. [69], which would allow us to rerank a set of solutions, could also allow us to search the combinatorial space of solutions more quickly. Relatedly, for the robotic task planning, incorporating asserts at the execution level (e.g. checking whether the agent is close to the microwave, as in Singh et al. [56]) is a promising research direction. Furthermore, evaluating the examples in this paper, we found that using the minimum CodeT score across all generated functions was a consistently effective heuristic to identify good sets of functions. However, generating unit tests for all functions when generating Parsel programs instead of generating unit tests for a shared top-level function increases the inference cost from linear in the number of tasks to also being linear in the number of functions and Parsel programs generated. Finding a way to balance this tradeoff would likely be valuable.
In addition, we plan to incorporate ways of varying the "confidence threshold" of the language model. Ensuring that the descriptions are straightforward and unambiguous is important for more critical programs and parts of programs. In addition, when teaching students simpler concepts, requiring them to decompose the task further may be useful.
We would like to integrate value functions to allow decomposition to be done more methodically where no verification is possible. Specifically, automatically decomposing all functions that have not yet been implemented in an SCC is suboptimal and could be improved with a model of expected improvement due to expansion, as done for proof expansion in Polu and Sutskever [49]. In addition, when decomposing functions, we would like to permit the model to reference already-defined functions (rather than to just define new ones). We might even use the code language model to determine which function to evaluate next. Further, we aim to support more general reward functions for function implementations where multiple may be valid but we rank implementations based on a desired feature. These "soft" constraints may also allow new Parsel uses, e.g. planning stories in natural language [64].
Figure A.10: Parsel to Lean (theorem proving)
Finally, we hope it would be possible to use Parsel as a framework for bootstrapping increasingly complex program generation (e.g. Anthony et al. [4], Zelikman et al. [68], Odena et al. [43]). That is, by 1) generating Parsel examples from a purely natural language specification and then reinforcing those which successfully compile, and 2) by reinforcing the model with each successfully compiled component, we would likely be able to iteratively improve performance with an arbitrarily large dataset of examples.
Another feature that would be valuable would be the ability to incorporate multiple base tools with different kinds of specialized models, inspired by Ibarz et al. [29] and Dohan et al. [20]. That is, it would be valuable to allow a model to determine which target language to use, possibly combining them. For example, for large parts of the Tensorflow and PyTorch libraries, while their interfaces are written in Python, they depend heavily on large  $\mathrm{C + + }$  codebases [46, 1]. Relatedly, Cobbe et al. [15] showed that giving language models access to a calculator allowed them to solve more complex math word problems. This, combined with the observation that Parsel could also compile programs by generating language model prompts to be used as part of the program, may potentially allow the automatic generation of task-specific language model cascades [20].
Another noteworthy addition would be the integration of Synchronesh [48], ensuring that each new word or token generated by the model is actually possible within the grammar of the given formal language and does not violate other semantic constraints.
Ultimately, we hope that this specification for Parsel is a jumping-off point for a new way of thinking about programming and reasoning.
D Theorem Proving in Lean
With the same framework, we can generate proofs in formal theorem-proving languages such as Lean, as in Figure A.10. We include the translated version in the appendix. Note a nuance of Lean and theorem-proving languages is that the ability to run Lean on proof with no errors/warnings indicates the proof is correct (but is not a guarantee that the proof statement matches our claim in language). Thus, each function in a Lean Parsel proof has an "implicit constraint." This makes it straightforward to identify which informal parts of a proof are most difficult to explicate. Generally, we believe Parsel can be a powerful tool for theorem proving.
Yet, we observed important challenges in this context, which we believe are avenues for future work and can be resolved. For example, in datasets such as MiniF2F [70], many proofs require explicit calculations in intermediate steps. That is, many proofs are similar to "Find the minimum value of  $\frac{9x^2\sin^2x + 4}{x\sin x}$  for  $0 < x < \pi$ . Show that it is 012." (from the informal MiniF2F introduced by Jiang et al. [30]). We believe that a dataset of proof statements (in natural and formal language), requiring complex proofs that are more abstract and less dependent on explicit calculations would allow us to better measure progress towards solving difficult theorems – we leave this to future work.
1 parse(program, target_language): synthesize a program from a string specifying a Parsel program.   
2 parse(program(program): parse the Parsel program string to a call graph create_root_node(): create a root node as the current function node, without any constraints   
3 parse_line(line, current_node, current_indent) -> function_graph: for each step up in indentation, set the current node  $\hookrightarrow$  to its parents. then, parse the definition, reference, or constraint. parse_definedline): create a new function node, make it a child of the current node's parent, then assign it as  $\hookrightarrow$  current node.   
6 parse_reference(line): add reference as a child of current node if reference is an ancestor or a direct child of an  $\hookrightarrow$  ancestor   
7 parse Constraintline): add the constraint to the current node's constraints.   
8 get Dependency_graph(function_graph) -> dependency_graph: taking the function graph, create a copy where all nodes without  $\hookrightarrow$  asserts also depend on their parents unless the target language implicitly tests all functions.   
9 identify_strongly-connected_components(dependency_graph): return SCCs of the dependency graph and the edges between the  $\hookrightarrow$  SCCs.   
10 synthesize_scc(scc, scc_graph): find an implementation string solving a given SCC, starting with SCC dependencies, then  $\hookrightarrow$  generating possible implementations of SCC functions, then finding an implementation combination satisfying the  $\hookrightarrow$  functions' constraints   
11 synthesizechildren(scc, scc_graph): synthesize any SCCs this SCC depends on and add them to the implementation string.   
12 synthesize_scc   
13 generate实施ations(scc, n, children实施ation_str): for each function in the SCC, prompt the language model to  $\hookrightarrow$  generate n implementations of each function starting with the implementation string of the SCC's children.   
14 solve_constraints(scc, fn_implementation): taking the provided constraints of each function in the scc, evaluate a  $\hookrightarrow$  shuffled list of the direct product of implementations with the constraints until one passes all of them   
15 direct_product_implementation(fn_implementation): return the direct product of the list of lists of  $\hookrightarrow$  fn_implementation   
16 generate_constraints(fn_node): translate each of the constraints into an evaluation string idiomatic to the target  $\hookrightarrow$  language and add these to the list of combined implementations   
17 eval_str(scc, implementation_str): evaluate an implementation including constraints by running it in a target-language  $\hookrightarrow$  executor   
18 on_fail(scc, scc_graph): raise an error highlighting the scc which could not be synthesized
Figure A.11: Pseudocode in the style of Parsel describing how Parsel synthesizes programs. A detailed version including automatic decomposition and automatic infilling is in Figure A.12 of Appendix G. Constraints are left out for clarity - e.g. one could define a test function and validate the compilation (or lack thereof) of a set of reference Parsel programs.
E Optimizations
E.1 Caching
We cache responses from the language model with respect to the prompt and language model decoding parameters 1) to reduce the number of queries necessary and 2) to keep the programs generated mostly stable (i.e. a working function should continue working unless it or its children change). To this end, when the number of desired implementations increases for a pre-existing query with all other arguments fixed (temperature, number of decoding tokens, etc), we append the additional ones to those already generated.
E.2 Automatic Function Infilling
Sometimes, a function generated by a language model may call a function that is not yet implemented. In this case, we can (optionally) attempt to automatically generate and implement it based on its usage. The function is then incorporated into the call graph as a unit-test-less child of the function which calls it. To avoid infinite recursion and inefficient use of language model quota, we limit the number of times that this process can be applied to a function.
E.3 Multiprocessing
We use multiprocessing with a user-specified timeout to test many implementation sets in parallel to allow for many fast solutions to be tested alongside slower solutions<sup>3</sup>.
F Parsel Pseudocode
G Parsel Pseudocode
We include a longer-form Parsel pseudocode in the style of Parsel. Note this pseudocode does not include backtranslation.
As anticipating the number of steps that a solution will take universally is a version of the halting problem and thus intractable.
Figure A.12: Longer pseudocode of Parsel, including automatic infilling and automatic decomposition.
H Parsel Overview (Detailed)
We include a more detailed figure outlining Parsel.
Figure A.13: Parsel overview (detailed).
I Lisp Interpreter
We include the Parsel code for a minimal Lisp interpreter.
Figure A.14: Full Lisp interpreter implementation in Parsel, including constraints.
We include a simple example function we could not generate with Codex [12] directly from the top-level description in Figure A.15. The corresponding Python code (included in the appendix) is exactly 58 non-whitespace lines of code, including 17 lines of comments (3 corresponding to the descriptions), 2 asserts, and 39 lines implementing the three functions described as well as an automatically generated get_number_of.active_cells-around_cell function. In fact, using automatic decomposition, as discussed in Subsection R.3, it is not necessary to provide any of the function descriptions besides the top one. The model is (unsurprisingly) able to understand that game_of_life_inversion_iteration can be broken down into invert_array and game_of_life_iteration.
1 game_of-Life_inversion_iteration(array_at_time_t): Takes a board and returns the next  $\rightarrow$  iteration of the game of life, but with all values flipped   
2 [0,0,1],[1,0,0],[1,0,0]]->[[1,1,1],[1,0,1],[1,1,1]]   
3 [0,1,0,0],[1,0,1,0],[1,0,0,1],[0,1,1,0]]->[[1,0,1,1],[0,1,0,1],[0,1,1,0],[1,0,0,1]]   
4 game_of-Life_iteration(array_at_time_t) -> array_at_time_t_plus_1: Takes a board with  $\rightarrow$  active and inactive cells as a list of lists and returns the next iteration of the  $\rightarrow$  game of life   
5 array_inversion(array) -> inverted_array: Invert a square array by flipping 0's and 1's
Figure A.15: An example Parsel program for Python that takes in a list of lists representing a state of Conway's game of life [24] and returns the next state, with all the values inverted.
K Parsel Prompts
1 # Description: given a list of lists representing the cost of each edge, return an adjacency  $\hookrightarrow$  matrix corresponding to the minimum spanning true. 2 def minimum spanning_tree(cost_matrix):
Figure A.16: Codex Prompt for an example leaf node
1 # Description: given a list of lists representing the cost of building a road between any two cities, and a list representing the cost of building an airport in a city, return a new cost matrix with a new node corresponding to the sky.  
2 # Signature: sky_city_cost(city_road_cost, city_airport_cost)  
3 from helpers import sky_city_cost  
4  
5 # Description: given a list of lists representing the cost of each edge, return an adjacency matrix corresponding to the minimum spanning true.  
6 # Signature: minimum spanning_tree(cost_matrix)  
7 from helpers import minimum spanning_tree  
8  
9 # Description: given a list of lists representing an adjacency matrix, return a list of the nodes connected to the final node. However, if only one node is connected to the final node, return an empty list.  
10 # Signature: final_node_connectors(adjacency_matrix)  
11 from helpers import final_node_connectors  
12  
13 # Description: given a matrix representing the cost of building a road between any two cities, and a list representing the cost of building an airport in a city (where any two cities with airports are connected), return a list of the cities that should have airports built in them to minimize the total cost of building roads and airports such that all cities are connected. The list should be sorted in ascending order.  
14 # Uses: sky_city_cost, minimum spanning_tree, final_node_connectors  
15 def select_airport_cities(city_road_cost, city_airport_cost):
Figure A.17: Codex Prompt for an example merge node
1 # Reviewer:  
2 # Please explain the above function in one sentence with as much detail as possible.  
3 # In your one-sentence description, specify the range and domain of your function precisely.  
4 # Your description should be clear enough that someone could reimplement the function from it.  
5 # Author:  
6 # Sounds good, here's my one-sentence explanation of {name}:  
7 # {name}
Figure A.18: Prompt format to generate descriptions for backtranslation
L APPS Backtranslation
L.1 Backtranslation / decompiling.
We anticipate that there are many programs that LLMs can implement by first generating Parsel code. But, as Parsel is a new framework, while language models can sometimes generate Parsel programs with few-shot prompts, it is not a syntax they have previously encountered. Thus, we may want to use existing code in other languages to construct datasets of Parsel programs from other languages. This requires us to first extract the call graph from the code, generate descriptions for each of the functions, and then generate Parsel programs from the graph. This call graph representation is convenient, so it is useful to have a bidirectional method to produce a graph from Parsel code and to produce Parsel code from the graph.
We filter the dataset to problems with starter code (providing the name of the evaluated function) and unit tests (provided as input-output pairs). For those tasks, we select solutions that define and call at least three functions, with at least one over 4 lines long and none over 15 lines.
As a proof of concept, we show 10 Parsel solutions which we could automatically generate from the APPS solutions. We generated the descriptions by prompting Codex to explain each function and its inputs and outputs. From this, we use backtranslation to attempt to implement these solutions in Python. We then verify that they are correct by applying the original unit tests as constraints on the root function. As mentioned in Section 1, the Parsel code is substantially shorter in terms of lines of code. We include these in Appendix L.
L.2 Examples
We exclude the asserts in these examples for brevity - they correspond to those in the original dataset.
1 longest PALINDrome(s): longest PALINDrome takes a string s and returns the longest palindrome  $\leftrightarrow$  in s.   
2 is PALINDrome(s): is PALINDrome returns True if the string s is the same forwards and  $\leftrightarrow$  backwards, and False otherwise.   
3 check(1i, ri, s): check takes a string s, a left index li, and a right index ri, and  $\leftrightarrow$  returns the longest palindrome that starts at or before li and ends at or after ri.   
4 is PALINDrome
Figure A.19: Train Problem 1638, Solution 2
1 # longest PALINDrome takes a string s and returns the longest palindrome in s.   
2 def longest PALINDrome(s): if len(s)  $<  =$  1: return s else: longest  $=$  s[0] for i in range(len(s)): for j in range(len(s)): if is PALINDrome(check(i, j, s)) and len(check(i, j, s)) > len(longest): longest  $=$  check(i, j, s) return longest   
13 # is PALINDrome returns True if the string s is the same forwards and backwards, and False  $\leftrightarrow$  otherwise.   
14 def is PALINDrome(s): if len(s)  $<   =$  1: return True else: return s[O]  $= =$  s[-1] and is PALINDrome(s[1:-1])   
19   
20 # check takes a string s, a left index li, and a right index ri, and returns the longest  $\leftrightarrow$  palindrome that starts at or before li and ends at or after ri.   
21 def check(li, ri, s): while li  $> = 0$  and ri  $<$  len(s) and s[li]  $= =$  s[ri]: li  $= = 1$  ri  $+ = 1$  return s[li+1:ri]
Figure A.20: Train Problem 1638, Solution 2
1 case_id(c_str): case_id takes a string and returns a string that is either "kebab", "snake",  $\leftrightarrow$  "camel", or "none" depending on whether the input string is in kebab case, snake case,  $\leftrightarrow$  camel case, or none of the above.   
2 is(snake(s): is(snake takes a string and returns True if the string is snake(case and False  $\leftrightarrow$  otherwise.   
3 is_kebab(s): is_kebab takes a string and returns True if the string is a kebab-case string,  $\leftrightarrow$  and False otherwise.   
4 is_camel(s): is_camel returns True if the string s is not lowercase, does not contain  $\leftrightarrow$  dashes, and does not contain underscores.
Figure A.21: Train Problem 2892, Solution 7
1 # case_id takes a string and returns a string that is either "kebab", "snake", "camel", or
2  $\leftrightarrow$  none" depending on whether the input string is in kebab case, snake case, camel case, or
3  $\leftrightarrow$  none of the above.
4 def case_id(c_str):
5 if is_sake(c_str) == True:
6 return "snake"
7 elif is_kebab(c_str) == True:
8 return "kebab"
9 else:
10 return "none"
11
2 # is_sake takes a string and returns True if the string is snake(case and False otherwise.
3 def is_sake(s):
4 if s[0].isalpha() and s[0].islower() and len(s) > 1:
5 for char in s:
6 if char.isalpha():
7 if char.isupper():
8 return False
9 else:
10 return False
11
2 # is_kebab takes a string and returns True if the string is a kebab-case string, and False
3 def is_kebab(s):
4 if s is empty, False
5 if s == ':': 
6 return False
7 if s is not a string, False
8 if type(s) != str:
9 else:
10 return False
11
2 # is_kebab takes a string and returns True if the string is a kebab-case string, and False
3  $\leftrightarrow$  otherwise.
4 def is_kebab(s):
5 if s is empty, False
6 if s == ':': 
7 return False
8 if s is not a string, False
9 else:
10 return False
11
2 # is_kebab takes anything other than a-z or -, False
3 def c in s:
4 if not (c.isalpha() or c == '-'): 
5 return False
6 if s contains a - at the beginning or end, False
7 if s[0] == '-' or s[-1] == '-': 
8 return False
9 if s contains more than one - in a row, False
10 for i in range(len(s)-1):
2 if s[i] == '-' and s[i+1] == '-': 
3 return False
4 # otherwise, True
5 return True
6 def is_camel returns True if the string s is not lowercase, does not contain dashes, and does
7  $\leftrightarrow$  not contain underscores.
8 def is_camel(s):
9 return s != s.lower() and s.find('_') == -1 and s.find('-') == -1
Figure A.22: Train Problem 2892, Solution 7