SlowGuess/NeurIPS_2023_OCR huggingface dataset:
type_fn_output(fn, args): returns the type of the output of a function called with args count_living_neighbors, ([1, 0], [0, 1]], 0, 0) -> int
This indicates that the function count_living_neighbors should return an integer when called with the input arguments  $[1, 0], [0, 1], 0, 0$ .
What it means to satisfy constraints to validate a program varies from language to language: in Python, one can check that a program passes certain assert statements by evaluating them; however, in a theorem-proving language like Lean, where the ability to run a program (without skipping steps by using sorry or oops lines) shows that a proof holds, one would instead represent the formal proof statement as the specified constraint (that is, that you are actually proving what you set out to prove). For languages where correctness can be checked without any unit tests, their functions can be treated as also having implicit constraints.
Figure A.55: collatz_recursion call graph. There is a strongly connected component formed by collatz_recursion and recursion_rule.
References. A reference is simply the name of a function defined in the current scope (see the next paragraph for details) within the function graph. A reference allows and encourages (via prompt) the parent function to use the referenced function. This allows for recursive function definitions and functions called by multiple functions. For example, one can define an (overly verbose) version of the Collatz conjecture (a well-known recursive open question in mathematics) as shown in Figure ii, where the final line is a reference. We visualize the corresponding call graph and its strongly connected components (SCC) in Figure A.55. In the Collatz functions, base(case) is implemented first as the collatz_recursion SCC depends on it.
Note that in Parsel, for Python one can also indicate in the description that a function should yield a value or is a generator.
<sup>5</sup>A nuance here is the optional ability for undefined/out-of-scope functions which are generated by the code LLM to also be implemented automatically.
R.2 Headers
We also support program headers, allowing global contexts, used when implementing all new functions within a program. This is indicated by a line containing an optional string of special characters (e.g. "\#*\#\#") separating the body and the text and is passed as a prefix to all prompts.
R.3 Repeated Automatic Decomposition
As indicated by a rapidly growing number of papers [10, 28], the task of decomposing a task into steps in natural language is one that language models are surprisingly capable of. As explored in concurrent work [57], using language models to automatically and recursively decompose difficult open-ended problems to arbitrary depth is a powerful tool. Thus, we treat the ability to automatically decompose a Parsel function as a key feature of Parsel. This is an optional flag that prompts a language model to generate Parsel code corresponding to any additional subfunctions necessary when Parsel fails to implement a function. These proposed subfunctions are then added as child nodes to the decomposed function node. However, an additional consequence is that Parsel can thus be used to recursively decompose tasks into steps, by repeatedly identifying descriptions that cannot be directly implemented and attempting to decompose them.
R.4 Scope.
Scope in Parsel is defined by indentation. The scope  $S$  of a function  $f$  includes the set of functions that can be used as a reference for a given function - that is, all functions where the indentations between the current function to the referenced function are strictly decreasing.
R.5 Variations due to target language requirements.
Certain aspects of the implementation are still target-language specific. As discussed above, the meaning and representation of a constraint may vary by language. Moreover, every language has a different evaluation function: executing Python is different than compiling and running C++ code, which is different than checking a proof with Lean. Further, every language will likely require a different prompt for the language model. Thus, we detail these particularities in language-specific configuration files.
S Pipeline Figure Example
1 Question:
2 The grand museum has just announced a large exhibit on jewelry from around the world. In the hopes of his potential future prosperity, the world-renowned thief and master criminal  $\hookrightarrow$  Edward Terrenando has decided to attempt the magnum opus of his career in thievery.
3
4 Edward is hoping to purloin a large number of jewels from the exhibit at the grand museum.  $\hookrightarrow$  But alas! He must be careful with which jewels to appropriate in order to maximize the  $\hookrightarrow$  total value of jewels stolen.
5
 6 Edward has $k$ knapsacks of size $1$, $2$, $3$, up to $k$, and would like to know for each ⿱ the maximum sum of values of jewels that can be stolen. This way he can properly weigh ⿱ risk vs. reward when choosing how many jewels to steal. A knapsack of size $s$ can hold ⿱ items if the sum of sizes of those items is less than or equal to $s$. If you can figure ⿱ out the best total value of jewels for each size of knapsack, you can help Edward pull ⿱ off the heist of the century!
7
8 ----Input-
 9 Each input will consist of a single test case. Note that your program may be run multiple times on different inputs. The first line of input will consist of two space-separated integers $n$ and $k$, where $n$ ($1 \le n \le 1000000$) is the number of jewels in the exhibit, and $k$ ($1 \le k \le 100000$) is the maximum size of knapsack available to Edward. The next $n$ lines each will describe a jewel. Each line will consist of two space-separated integers $s$ and $v$, where $s$ ($1 \le s \le 300$) is the size of the jewel, and $v$ ($1 \le v \le 10~9$) is its value. Each jewel can only be taken once per knapsack, but each knapsack is an independent problem.
10
11-----Output-----
 12 Output $k$ integers separated by whitespace. The first integer should be the maximum value of ← jewels that will fit in a knapsack of size $1$. The second should be the maximum value ← of jewels in a knapsack of size $2$, and so on.
13
14 ----Examples----
15 Sample Input 1:
1649
1728
18 1 1
1934
205100
21 Sample Output 1:
22 1 8 9 9 100 101 108 109 109
23
24 Sample Input 2:
25 5 7
26 2 2
27 38
28 27
29 24
30 38
31 Sample Output 2:
32 0 7 8 11 15 16 19
Figure A.56: Pipeline Figure Task
1 First, we need to figure out how to calculate the maximum sum of values of jewels for each  $\hookrightarrow$  size of knapsack. To do this, we can use a dynamic programming approach.  
2  
3 We can create a two-dimensional array of size $n \times times k$ where each row represents the different jewels and each column represents the different sizes of knapsack. The value  $\hookrightarrow$  of each cell in the array will represent the maximum sum of values of jewels that can be stolen for a given knapsack size.  
4  
5 We can then iterate through the array, starting at the beginning. For each row, we can  $\hookrightarrow$  calculate the maximum sum of values of jewels that can be stolen for each knapsack size  $\hookrightarrow$  by taking the maximum of the value of the current cell and the value of the previous  $\hookrightarrow$  cell plus the value of the current jewel.  
6  
7 At the end, we will have an array that contains the maximum sum of values of jewels that can  $\hookrightarrow$  be stolen for each knapsack size. We can then output these values as our answer.  
8  
9 This algorithm is correct because we are calculating the maximum sum of values of jewels that can be stolen for each knapsack size by comparing the values of the current cell and the previous cell plus the value of the current jewel. This ensures that we are taking the maximum value for each knapsack size.
Figure A.57: Pipeline Figure Sketch
1 # Takes a string containing the number of jewels and knapsack sizes on the first line, the  $\rightarrow$  jewels on the next lines, and returns the number of jewels, the knapsack sizes, and the  $\rightarrow$  jewels.   
2 def parse_input(input_str): lines  $=$  input_str.split('\\n') first_line  $=$  lines[0].split(') num_jewels  $=$  int(first_line[0]) knapsack_sizes  $=$  int(first_line[1]) jewels  $=$  [O,O]  $\ast$  num_jewels for i in range(1, num_jewels + 1): jewel  $=$  lines[i].split(') jewels[i - 1]  $=$  [int(jewel[O]), int(jewel[1]) ] return num_jewels, knapsack_sizes, jewels   
13 # Returns the maximum sum of values of jewels that can be stolen for each knapsack size.   
14 def max_sum_of_jewels_for_size_and_jewel(jewels, size, jewel): if size  $<  0$  return O elif jewel  $<  0$  return O elif jewel  $= = 0$  return jewels[jewel][1] if jewels[jewel][O]  $<   =$  size else 0 elif jewels[jewel][O]  $\rightharpoondown$  size: return max_sum_of_jewels_for_size_and_jewel(jewels, size, jewel-1) else: return max(max_sum_of_jewels_for_size_and_jewel(jewels, size, jewel-1), max_sum_of_jewels_for_size_and_jewel(jewels, size-jewels[jewel][O], jewel-1)  $\leftrightarrow$  +jewels[jewel][1])   
26   
27 # Returns the maximum sum of values of jewels that can be stolen for each knapsack size.   
28 def max_sum_of_jewels_for_size(jewels, size): result  $=$  [] for s in range(1, size + 1): result  $+ =$  [max_sum_of_jewels_for_size_and_jewel(jewels, s, len(jewels)- 1)] return result   
32   
33   
34 # Returns the string of the maximum sum of values of jewels that can be stolen for each  $\leftrightarrow$  knapsack size.   
35 def to_output_str(max_sum_of_jewels_for_size): return "".join(map(str, max_sum_of_jewels_for_size))   
38 # Returns the maximum sum of values of jewels that can be stolen for each knapsack size.   
39 def max_sum_of_jewels(input_str): >> max_sum_of_jewels('4 9\\n2 8\\n1 1\\n3 4\\n5 100\\n') '1 8 9 9 100 101 108 109 109' >> max_sum_of_jewels('5 7\\n2 2\\n3 8\\n2 7\\n2 4\\n3 8\\n') '0 7 8 11 15 16 19' >> number_of_jewels, knapsack_sizes, jewels  $=$  parse_input(input_str) return to_output_str(max_sum_of_jewels_for_size(jewels, knapsack_sizes))   
40   
41 assert compare_output(max_sum_of_jewels,'4 9\\n2 8\\n1 1\\n3 4\\n5 100', '1 8 9 9 9 100 101 108  $\leftrightarrow$  109 109')   
42 assert compare_output(max_sum_of_jewels,'5 7\\n2 2\\n3 8\\n2 7\\n2 4\\n3 8', '0 7 8 11 15 16 19')   
43 assert compare_output(max_sum_of_jewels,'26\\n300 1\\n300 2', '0 0 0 0 0 0')
Figure A.58: Pipeline Figure Sketch
T Other Details
T.1 Test Generation in HumanEval
For our test generation in our HumanEval experiments, we generated a set of tests by prompting GPT-4 to "Generate an assert-based test for the following function. Answer with only a code block, and no other text. Do not wrap your asserts in a function.\n" + question and then collecting and set-aggregating 100 completions.
T.2 Backtracking
We also support backtracking for the Parsel implementation step, where we re-implement descendants by sampling new solutions for dependencies if a correct solution is not found for a parent. This is necessary to improve the robustness of some of the Appendix examples such as Figure A.14.
T.3 Training Details
Although we do not train any models, all models used are discussed throughout the paper. See Appendix N for more details about sampling hyperparameters.
T.4 Error Bars
We estimate a standard deviation of  $\pm 1.4\%$  for the best APPS result, given 1000 sampled problems.
T.5 Compute
The most computationally intensive part of this research, by far (in terms of FLOPS), was the ablation using an open-source CodeGen model, which required several-hundred A100 hours. The rest of the inference was done through API calls.
T.6 Generated Tokens
For the APPS evaluation, in terms of tokens generated, it is hard to compare the models directly: The CodeT paper does not specify the number of tokens decoded for their evaluation. Without more detail about their evaluations, it is impossible to confidently estimate the tokens generated per program for the CodeT evaluation. The AlphaCode results sample at most 768 tokens per solution, but they do not report average statistics directly - based on Figure 11 in the AlphaCode paper [35], the majority of its generated solutions are 150 to 350 tokens long after removing dead code and comments. The competition-level problems (that we evaluate on) require more tokens on average. For their best reported results, their figures indicate they sample at least 20 billion tokens for the competition-level subset of APPS. On the other hand, for our best results, Parsel generates (on average) 491 tokens of Python code per program implementation, and because we implement each high-level sketch in Python sixteen times (i.e.  $k = 16$  in our best  $n \times k$ ), we also sample on average 22 sketch tokens and 43 translation tokens per Python program implementation. Correspondingly, we sample roughly 7 million tokens for our APPS evaluation.
T.7 Reproducibility
While our contribution is not a model or dataset, we have released our code.
T.8 Evaluation Time
For a similar API rate limit and evaluation time, we expect our longest-running evaluation runtime would be fairly comparable to the 1,000-program AlphaCode evaluation (despite performing  $3\mathrm{x}$  better) and several times faster than the 50,000-program AlphaCode evaluation. In other words, we believe that generating 1,000 programs and evaluating them is not meaningfully faster than generating the 128 Parsel ones and then evaluating their combinations, even in the worst case. Our reasoning is as follows:
Generation time. Because ALPHA code has no public-facing API, let us use the advertised rate limit for Codex (40,000 tokens per minute) as the generation speed. (This ignores the request per-minute limit and the number of input tokens in context.) For ALPHA code, assuming roughly 400 tokens per solution, generation would take 10 minutes plus the completion time (i.e., the latency from an API request to the result) of roughly 30 seconds. For Parsel, in our largest-scale experiment, we generate eight plans, their corresponding Parsel programs, and then each Parsel program's corresponding 16 implementations. At the same rate limit and program length, these generations would take about 1.5 minutes, plus three times the completion time. In other words, generation should take about 3 minutes for Parsel vs. 10.5 minutes for ALPHA code.
Evaluation time. Let's consider three evaluation scenarios, maintaining the 0.04-second implementation eval timeout used in the paper, run on a single 16-core machine. 1,000 evaluations. If we run 1,000 programs for each method, under the same constraints, their evaluation runtime should be the same. As indicated by Fig. 4-right, the Parsel performance would be substantially higher despite being over  $3\mathrm{x}$  faster overall. 10,000 evaluations. If we assume all programs take the maximum allowable runtime (the worst case for Parsel), then 10,000 evaluations takes 6.7 minutes for parsel, compared to 40 seconds for 1,000 AlphaCode program evals. Parsel is still slightly faster, yet solves multiple times more problems. More evaluations. After this point, we run into the 2-minute per Parsel program limit (discussed previously), making Parsel slower than AlphaCode in the worst case — but also solving over  $3\mathrm{x}$  as many problems.
Other nuances. With all this being said, this is a difficult (potentially impossible) comparison to make fairly for a few reasons. First, ALPHA code is worse than Codex, but likely faster, so it isn't clear which Parsel data point and which rate limit would make an appropriate comparison. Second, runtimes and problem lengths can vary substantially. For instance, we haven't considered input tokens so far, but these penalize ALPHA code more than Parsel, as ALPHA code inferences many more programs. Note that APPS problems are fairly long (the longest competition-level problem requires over 1800 tokens) so the effect of re-processing the problem statement at each program inference is substantial. Since the rate limit includes input tokens and there is a hard limit of 4,000 generated tokens per query, this decreases the effective rate limit by a third!