SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1 "An action plan is a list of strings that describes a sequence of steps to accomplish a task, To be correctly parsed, an action plan must be syntactically correct and contain only allowed actions and recognizable simple objects. Allowed actions: 'close' <arg1>, cut', <arg1>, drink' <arg1>, drop' <arg1>, eat' <arg1>, find' <arg1>, grab' <arg1>, greet' <arg1>, lie on' <arg1>, look at' <arg1>, open' <arg1>, plug in' <arg1>, plug out' <arg1>, point at' <arg1>, pour' <arg1> 'into' <arg2>, pull' <arg1>, push' <arg1>, put' <arg1> 'on' <arg2>, put' <arg1> 'in' <arg2>, put back' <arg1>, take off' < arg1>, put on' <arg1>, read' <arg1>, release', rinse' <arg1>, run to' <arg1>, scrub' <arg1>, sit on' <arg1>, sleep', squeeze' <arg1>, stand up', switch off' <arg1>, switch on' <arg1>, touch' <arg1>, turn to' <arg1>, type on' <arg1>, wake up', walk to' <arg1>, wash' <arg1>, watch' <arg1>, wipe' <arg1>. To satisfy the common- sense constraints, each action step in this action plan must not violate the set of its pre-conditions (e.g. the agent cannot grab milk from the fridge before opening it) and post-conditions (e.g. the state of the fridge changes from "closed" to "open" after the agent opens it).   
2 ###   
3 task計劃(): return a list of strings that represents an action plan to put a mug on the stall and bread on the desk.   
4 -> "executable" put_object_on(object, place): return a list of strings that represents an action plan to put an object in a place. "mug", "stall" -> "executable"
Figure A.45: Full Parsel program including header for Fig. 2 example, with the  $\#^{**}\#$  as the header seceptor. Note that we essentially just took the executability definition in [28] and added the list of available actions.
1 ""----Solution----   
2   
3 Propose a clever and efficient high-level solution for this problem. Consider all edge cases  $\leftrightarrow$  and failure modes.   
4   
5 Some common strategies include:   
6 Constructive algorithms, Binary search, Depth-first search (DFS) and similar algorithms,  $\leftrightarrow$  Dynamic programming, Bitmasks, Brute force, Greedy algorithms, Graphs, Two pointers,  $\leftrightarrow$  Trees, Geometry, Graph matchings, Hashing, Probabilities, Data structures, Sortings,  $\leftrightarrow$  Games, Number theory, Combinatorics, Divide and conquer, Disjoint set union (DSU),  $\leftrightarrow$  Expression parsing   
7   
8 Write out your reasoning first, and then describe your high-level solution and explain why it  $\leftrightarrow$  is correct.   
9 \\""\\""   
10 Let's think step by step to come up with a clever algorithm."""
Figure A.46: High-level sketch prompt for APPS programs
1 ""----Translation----
2 # Here is an example calculating the probability of landing on the same character in a random shift of an input string, based on the following
3  $\nrightarrow$  Vasya and Kolya play a game with a string, using the following rules. Initially, Kolya creates a string s, consisting of small English letters, and uniformly at random chooses an integer k from a segment [0, len(s) - 1]. He tells Vasya this string s, and then shifts it k letters to the left i.e. creates a new string t = s_k + 1s_k + 2... a_ns_is_2... s_k. Vasya does not know the integer k nor the string t, but he wants to guess whether he has chosen Kolya to tell him the first letter of the new string, and then, after he sees it, open one more letter on  $\rightarrow$  some position, which Vasya can choose.
4 # Vasya understands, that he can't guarantee that he will win, but he wants to know the probability of winning, if he plays optimally. He wants you
5 # Note that Vasya wants to know the value of k uniquely, it means, that if there are at least two cyclic shifts of s that fit the information Vasya  $\hookrightarrow$  knowns, Vasya loses. Of course, at any moment of the game Vasya wants to maximize the probability of his win.
6  $\backslash^{n}\backslash^{m}\backslash^{n}$  generate_cyclic-shifts(input_str):Calculates the average number of unique characters in the substrings of the input string that start with each character.
8 9 compute_a_and_letter_pos(input_str):Generates the str_as_number_list and letter_pos lists.str_as_number_list is a list of integers that is used to store the character values of the input string. str_as_number_list is initialized as a list of 0s for twice the length of the input string. The values are calculated by taking the ASCII value of each character in the string and subtracting the ASCII value of the character 'a'.
0 compute_unique Characters(c, str_as_number_list, letter_pos) -> ans:Calculates the maximum number of unique characters in all substrings (for k=1 to length) that start with the character represented by c. letter_pos is a list of lists, with each sublist containing the indices at which a character appears in the input string. str_as_number_list is a list of integers that is used to store the character values of the input string
1 compute_uniqueCharacters_for_k(c, k, str_as_number_list, letter_pos): Create a counts list of zeros for each of the 26 alphabetical characters  $\hookrightarrow$  For each i in the sublist of positions of letter_pos[c], increment counts at str_as_number_list[i + k]. Return the number of counts which  $\hookrightarrow$  are exactly one.   
2 to_output_str(ans, input_str): Returns a string representation of ans divided by the length of the input string.
3  $"\backslash "\backslash$  4 (6 lines)
16 # And here is an example identifying the largest binary number according to the following rules:
The Little Elephant has an integer a, written in the binary notation. He wants to write this number on a piece of paper.  
To make sure that the number a fits on the piece of paper, the Little Elephant ought to delete exactly one any digit from number a in the binary record. At that a new number appears. It consists of the remaining binary digits, written in the corresponding order (possible, with leading zeroes).
9 # The Little Elephant wants the number he is going to write on the paper to be as large as possible. Help him find the maximum number that he can  $\hookrightarrow$  obtain after deleting exactly one binary digit and print it in the binary notation.
20 \"\\"""   
1 largest_binary_number(input_str): Returns the largest binary number that can be made by removing at most one digit from the input string.   
22 parse_input(input_str): Takes a string and returns the input string   
23 remove_zerobinary_str): Remove the first zero from the input string.
25  $"\backslash "\backslash "\backslash$
26 (4 lines)
Here is an example of the format applied to identifying the winner of the following game:
It is so boring in the summer holiday, isn't it? So Alice and Bob have invented a new game to play. The rules are as follows. First, they get a set  $\hookrightarrow$  of n distinct integers. And then they take turns to make the following moves. During each move, either Alice or Bob (the player whose turn is  $\hookrightarrow$  the current) can choose two distinct integers x and y from the set, such that the set doesn't contain their absolute difference |x - y|. Then  $\hookrightarrow$  this player adds integer  $|x - y|$  to the set (so, the size of the set increases by one).
30 # If the current player has no valid move, he (or she) loses the game. The question is who will finally win the game if both players play optimally.  $\leftrightarrow$  Remember that Alice always moves first.
31 \\"\""
32 identify_winner(input_str): Returns the winner of the game.
all_gcd(1): Returns the greatest common divisor of all elements in the list  
to_output_str(nummoves): Returns the string 'Alice' if the number of moves is odd and 'Bob' if the number of moves is even
8（5-）
06 (5 lines) 20
10 # Linak is a little bear who loves to play. Today he is playing by destroying block towers. He built n towers in a row. The i-th tower is made of h.i identical blocks. For clarification see picture for the first sample. 11 # Linak is a little bear who loves to play. Today he is playing by destroying block towers. He built n towers in a row. The i-th tower is made of h.i identical blocks.
12 # Block is called internal if it has all four neighbors, i.e. it has each side (top, left, down and right) adjacent to other block or to the floor. Otherwise, block is boundary. In one operation Limak destroys all boundary blocks. His paws are very fast and he destroys all those blocks at
43 Linak is ready to start. You task is to count how many operations will it take him to destroy all towers.
45 destroy_towers(input_str): Returns the number of operations it takes to destroy all towers. 46 parse_input(input_str): Takes a string containing the number of towers on the first line and the heights of the towers on the second and returns
47 side_oes(heights_list): From a list of ints, set the first and last elements to 1 and return the list   
48 destroy_from_left(side_list): Copy the list and set each each element to the minimum of itself and one more than the element to its left, starting
49 destroy_from_right(side_list): Copy the list and set each each element to the minimum of itself and one more than the element to its right,
50 min_list(l1, l2): Return a list of the minimum of the corresponding elements of l1 and l2
[5] to_output_str(min_list): Return the string representation of the maximum element in the list.
63 (7 lines)
55 # Alex decided to go on a touristic trip over the country.
06 # For simplicity let's assume that the country has $n$ cities and $m$ bidirectional roads connecting them. Alex lives in city $s$ and initially located in it. To compare different cities Alex assigned each city a score $g i$ which is as high as interesting city seems to Alex
7 # Alex believes that his trip will be interesting only if he will not use any road twice in a row. That is Alex came to city $v$ from city $u$, he may choose as the next city in the train any city connected with $v$ by the road, except for the city $u$.
86 # Your task is to help Alex plan his city in a way that maximizes total score over all cities he visited. Note that for each city its score is counted at most once, even if Alex been there several times during his trip.
69 \"\\"\\"
max_score(input_str): Simple function returning the maximum score Alex can get.
parse_input(input_str): takes a string containing the number of cities and roads on one line, the scores of the cities on the next line, the roads on the last line, the lanes on the last line, the indexed, make 0-indexed, and make 1-indexed. The starting city on the last line. It returns the city scores, the roads
get_neighbors edges): Returns a dictionary of the neighbors of each city, defaulting to an empty set.
get-degrees-and-neighbors(root): Returns a dictionary of the degrees of each city, and a set of the leaves (excluding the root). remove_leaves(scores, neighbors, degrees, leaves, root): Create a 0-initialized defaultdict of totalextra, and an int of maxextra. Pop leaves  $\hookrightarrow$  until it is empty. Update total extra and max extra based on the parent's total extra vs the leaf's score plus its total extra, whichever is greater. Return max extra.
pop_leaf( neighbors, degrees, leaves, root): Pop off a leaf. Set parent to sole neighbor of the leaf and delete the leaf from the neighbors  $\hookrightarrow$  dictionary.Decrement the parent's degree.If the parent is not the root and has degree 1, add it to the leaves. Return the leaf and parent. to_output_str scores, neighbors, root, maxextra): Returns the string of the maximum score Alex can get. If the root isn't in neighbors, return the score of the root. Otherwise, this is the sum of the scores of the cities left in neighbors, plus the returned encountered max extras.
67\""\""
68 (7 lines)
70 # Translate the following solution plan into the above format:
71 {solution_start}{solution_text}
73 TRANSLATE to Parse.
74 \""\""
Figure A.47: Translation prompt for APPS programs
O HumanEval Prompts
We use the same zero-shot prompt to encourage the high-level sketch as in APPS. For translation we use:
1 You will aim to solve the following problem in Parsel:   
2 {question}   
3   
4 Translate the following solution plan into Parsel:   
5 {solution_start}{solution_text}   
6   
7 You will translate a solution plan for a problem into Parsel. Each line should contain either  $\leftrightarrow$  a function description or a function reference.   
8   
9 A function description should be of the form:   
10  $^ \text{喜}$    
11 function_name(arg1, arg2): Description of the function   
12  $^ \text{喜}$    
13   
14 A function reference should be of the form:   
15  $^ \text{喜}$    
16 function_name   
17  $^ \text{喜}$    
18   
19 Use indentation to indicate dependencies between functions. For example, if function A calls  $\rightarrow$  function B, then function B should be indented under function A.   
20 Make sure that the top-level function matches the name of the function in the solution plan.
Figure A.48: Translation prompt for GPT-4
P Robotic Plan Evaluation Details
P.1 Questionnaire
Our questionnaire closely follows that of Huang et al. [28]. We provide a figure with the directions for the accuracy version of the survey in the first image of Fig A.49. We include an example question in the second image. Note that each participant was shown a random 5 questions with their answers in random order. The clarity survey instead asks "For every question below, evaluate how easy it is to understand how the provided steps accomplish the task. Please rank the planned steps for each question from most understandable to least understandable (with 1 as the best and 3 as the worst)." In addition, for the clarity survey, each question text instead said "Rank the following plans based on which is the most understandable (1 = most understandable, 3 = least understandable)."
P.2 Executability
We find that the automated executability check is a less insightful metric than human evaluation, as it doesn't meaningfully reflect the plan's likelihood of successfully completing a task. Unfortunately, the code to replicate the executability measure from Huang et al. [28] is unavailable. As an alternative, we developed our own executability checker using example code found on VirtualHome's GitHub repository, which evaluates if a proposed plan is syntactically accurate and can be executed within the VirtualHome environment. By leveraging Codex to generate eight Parsel programs for each of the 88 tasks and subsequently compiling them using the Parsel compiler, our method successfully produced executable solutions for all tasks. Conversely, Huang et al. [28] managed executable plans for only 86 tasks. However, it is worth noting that our Parsel compiler explicitly incorporates this executability measure as a constraint, which explains the higher executability rates observed in our approach.
On the next (and only) survey page, you will be presented with 5 questions.
For each question you will see an everyday task and a plan to accomplish it. Please determine whether the task can be completed using the provided steps. In other words, can the task be decomposed into these steps? Note that simply restating the task does not mean completing it. Please rank the planned steps for each question from most accurate to least accurate (with 1 as the best and 3 as the worst).
There is no correct answer to each question. Please just use your first intuition to determine the answers.
Figure A.49: Screenshot of survey directions and example survey question. In this figure, the first answer was generated by the baseline, the second was the indented Parsel version, and the third was the unindented Parsel version. However, note that the order is randomized for each participant.
Rank the following plans based on how accurately they accomplish the task (1 = most accurate, 3 = least accurate):
Task: Wash teeth
walk to bathroom
walk to sink
find faucet
switch on faucet
find toothbrush
grab toothbrush find toothpaste
find toothpaste  
grab toothpaste
put toothpaste on toothbrush
put back toothpaste
grab toothbrush
wash teeth
rinse toothbrush
switch off faucet
find toothbrush holder
put toothbrush on toothbrush holder
walk to bedroom
task_plan:
wash_teeth:
plug in sink
pour water into sink
pour toothpaste into toothbrush
scrub teeth with toothbrush
Pour water into toothbrush
plug out sink
drop toothbrush
find towel
wipe teeth with towel
walk to toothbrush
open toothbrush
find bathroom sink, walk to bathroom sink
run to bathroom sink
put toothbrush on bathroom sink
release toothbrush
put back toothbrush
O O O
Q Human Case Study
Figure A.50: Solution to https://codeforces.com/problems/problem/1023/A
1 main(input_string): Takes an input line. First, splits it at newline, and stores the second  $\rightarrow$  line in a variable s and the third in t. Splits s in two parts at the asterisk, and  $\rightarrow$  checks whether the string t starts with the first part of s and ends with the second  $\rightarrow$  part. Returns the string "YES" if that condition is met, otherwise "NO". Also, it should  $\rightarrow$  always return "NO" if the length of t is smaller than the sum of the length of the  $\rightarrow$  parts of s.  
2 "6 10\ncode*s\ncodeforces" -> "YES"  
3 "6 10\ncodeforces\ncodeforces" -> "YES"  
4 "6 10\ncode*morces\ncodeforces" -> "NO"  
5 "6 10\na*a\na" -> "NO"  
6 "6 10\na*a\naa" -> "NO"
Figure A.51: Solution to https://codeforces.com/problems/problem/529/B
1 main(input_string): Parses the input and returns the minimum area of the input.  
2 "3\n10 1\n20 2\n30 3" -> 180  
3 "3\n3 1\n2 2\n4 3" -> 21  
4 parse_input(input_string): Takes the input line and first splits on newline. Ignores the  
5  $\leftrightarrow$  first line, and parses each of the remaining lines as a tuple of two numbers, which  
6  $\leftrightarrow$  give a list L of tuples. Returns L.  
7 "3\n10 1\n20 2\n30 3" -> [[10, 1], [20, 2], [30, 3]]  
8 parse_line(1): Splits l on space, converts each element to int, and returns the result  
9  $\leftrightarrow$  of converting the result to a list.  
10 "10 1" -> [10, 1]  
11 enumerate_subsets_at-most_k(L, k): Returns all subsets of L with sizes ranging from O to  
12  $\leftrightarrow$  k, inclusive.  
13 [1, 2, 3], 2 -> [[1], [1], [2], [3], [1, 2], [1, 3], [2, 3]]  
14 enumerate_subsets(L, k): recursively enumerates the subsets of size k of the list L.  
15  $\leftrightarrow$  Base cases: if  $k = 0$ , returns a list containing the empty list. If  $k > \text{len}(L)$ ,  
16  $\leftrightarrow$  returns the empty list. Otherwise, first construct the subsets that contain the  
17  $\leftrightarrow$  first element, then those that do not, and return their concatenation.  
18 [1, 2, 3], 2 -> [[1, 2], [1, 3], [2, 3]]  
19 minimum_area(whs): First, calls enumerate_subsets_at-most_k passing whs and half the  
20  $\leftrightarrow$  length of whs rounded down. Returns the minimum result of calling compute_area on the  
21  $\leftrightarrow$  list given by apply_inversions with whs and the subset.  
22 [(10, 1], [20, 2], [30, 3]) -> 180  
23 [(3, 1], [2, 2], [4, 3]) -> 21  
24 enumerate_subsets_at-most_k  
25 compute_area(whs): takes a list of pairs (width, height). Computes the sum of the  
26  $\leftrightarrow$  widths and the maximum of the heights. Returns the product of those two numbers.  
27 [(1, 2], [3, 5]) -> 20  
28 [(10, 1], [20, 2], [30, 3]) -> 180  
29 apply_inversions(whs, subset): Takes a list of pairs of form (w, h) and a subset of  
30  $\leftrightarrow$  indices to invert. Returns a list where the elements of whs whose index is in the  
31  $\leftrightarrow$  subset are inverted to (h, w), and the others appear as given.  
32 [(1, 2], [3, 5]), [1] -> [[1, 2], [5, 3]]  
33 [(1, 2], [3, 5]), [] -> [[1, 2], [3, 5]]
1 main(input): Converts the input to an integer and returns the value of f of n.  
2 "1" -> 1  
3 "2" -> 3  
4 "3" -> 10  
5 f(n): First pre-computes the Pascal triangle up to  $n + 1$  using compute_pascal_TRIANGLE.  
 $\hookrightarrow$  Then, returns the value of dp(n, Pascal_TRIANGLE)  
6 1 -> 1  
7 2 -> 3  
8 3 -> 10  
9 compute_pascal_TRIANGLE(N): returns a matrix with  $N + 1$  rows where  $m[i][j]$  corresponds  
 $\hookrightarrow$  to "i choose k", i.e., the Pascal triangle. It is computed using dynamic  
 $\hookrightarrow$  programming:  $m[i][j] = m[i - 1][j] + m[i - 1][j - 1]$ . All elements are modulo (10**9 +  
 $\hookrightarrow$  7). The i-th row has only i columns.  
10 2 -> [[1], [1, 1], [1, 2, 1]]  
11 3 -> [[1], [1, 1], [1, 2, 1], [1, 3, 3, 1]]  
12 dp(n, Pascal_TRIANGLE): first creates a list with  $(n + 1)$  zeros called L. Then fills  
 $\hookrightarrow$  it in with the following dynamic programming relation: base case is  $L[0] = 1$ ;  
 $\hookrightarrow$  then,  $L[i] = \text{sum}$  (j in [1, i]) Pascal_TRIANGLE[i-1][j-1] * L[i - j]. Finally,  
 $\hookrightarrow$  returns the following answer: sum (k in [1, n]) Pascal_TRIANGLE[n][k] * L[n - k].  
 $\hookrightarrow$  After each of these assignments, take modulo  $10**9 + 7$  to avoid big numbers.  
13 1, [[1], [1, 1], [1, 2, 1]] -> 1  
14 2, [[1], [1, 1], [1, 2, 1]] -> 3  
15 3, [[1], [1, 1], [1, 2, 1], [1, 3, 3, 1]] -> 10
Figure A.52: Solution to https://codeforces.com/problems/problem/568/B
1 main(input): Reads the input line and counts how many pairs of elements pass the test.  
2 "4 2\n2 3\n1 4\n1 4\n2 1" -> 6  
3 "8 6\n5 6\n5 7\n5 8\n6 2\n2 1\n7 3\n1 3\n1 4" -> 1  
4 parse_input(input): Splits input as a sequence of lines. Each line is parsed as a list of two space-separated integers. The first line of input contains N and P, and the second to last lines are aggregated in a list L. Returns a list with three values: N, P and L.  
5 "4 2\n2 3\n1 4\n1 4\n2 1" -> [4, 2, [[2, 3], [1, 4], [1, 4], [2, 1]]]  
6 count_valid_pairs(L, p): for each distinct pair (i, j) both ranging from 0 to the length of L, counts how many of those pairs have score at least p in L given by compute_pair_score.  
7 [[2, 3], [1, 4], [1, 4], [2, 1]], 2 -> 6  
8 [[5, 6], [5, 7], [5, 8], [6, 2], [2, 1], [7, 3], [1, 3], [1, 4]], 6 -> 1  
9 compute_pair_score(a, b, L): receives two integers, a and b, and a list of pairs L.  
10 Returns how many elements of L contain either a + 1 or b + 1.  
11 1, 2, [[2, 3], [1, 4], [1, 4], [2, 1]] -> 2  
12 0, 1, [[2, 3], [1, 4], [1, 4], [2, 1]] -> 2  
13 4, 5, [[2, 3], [1, 4], [1, 4], [2, 1]] -> 0
Figure A.53: Solution to https://codeforces.com/problems/problem/420/C
1 main(input): parses the input as two space-separated integers, n and m. Return 2 * f(n, m)  
    ← modulo 10**9 + 7  
2 "2 3" -> 8  
3 "3 2" -> 8  
4 f(n, m): computes fib(n) + fib(m) - 1  
5 2, 3 -> 4  
6 fib(m): computes the m-th fibonacci number modulo 10**9 + 7 using dynamic programming  
    ← starting with dp[0] = 1 and dp[1] = 1, then dp[n] = (dp[n-1] + dp[n-2]) % (10**9 + 7)  
7 1 -> 1  
8 2 -> 2  
9 3 -> 3  
10 4 -> 5  
11 5 -> 8  
12 6 -> 13
Figure A.54: Solution to https://codeforces.com/problems/problem/1239/A
R Parsel Language Nuances
R.1 Syntax
Descriptions. A function description is represented as a function name followed by comma-separated input arguments in parentheses, and optionally an arrow followed by what the function returns<sup>4</sup>, then a colon and text describing the function to be implemented. For example, as part of Conway's Game of Life, one might write
count_living_neighborsgrid,i,j):count the number of living neighbors of the cell at the  $\rightarrow$  index (i, j)
A function generated from a description can call either the functions defined directly below the description in the function graph (indicated with indentation) or references directly below the description<sup>5</sup>, both shown in Fig. ii.
Constraints. A constraint is represented as a function's input values comma-separated, optionally followed by an arrow and the expected output of the function. Constraints are provided at the same indentation as the preceding description. For example, after the definition of count_living_neighbors one can write
[[1,0],[0，1]]，0，0->1[[1，0],[0，1]]，0，1->2
This indicates that count_living_neighbors should return 1 when called with the arguments  $[ [1,0 ] ,$ $[0,1]$ ,  $0,0$  and 2 when called with  $[ [1,0 ] ,$ $[0,1]$ ,  $0,1$ . Notably, to apply complex constraints on functions, one can describe and constrain higher-order functions. For example: