SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Your target is to instruct the systems to generate accurate Python code based on your solution. We will evaluate the program strictly on the test input and output, yet it's essential for you to confirm that your program is capable of successfully handling all the training input and output. Once the correct code is produced, the system will automatically deactivate. Perseverance is crucial, but if the system fails to generate the accurate code within 30 minutes, you're permitted to terminate the process. If a task proves overly challenging at any point, it's acceptable to stop prematurely.
System A is comprised of three primary operations. The trace operation allows for a function name to be entered, which triggers the program to run on a test input and displays all the input and output data of the selected function. The edit operation allows for direct changes to a function's body, including alterations to the sketch and hole. This operation has four sub-operations:
- Splitting the original function into multiple holes linked by the sketch.
- Turning the original code into a hole and attempting code generation.
- Changing the natural language description associated with the hole.
- Modifying the sketch while maintaining the generated hole.
The resynthesis operation requires the user to provide correct input and output examples. The system then generates numerous functions and tests in which one meets these examples. The provided examples are kept for future use, and multiple sets of examples can be provided by the user.
System B incorporates two operations. The chat operation allows the user to interact with the system using natural language, leading to code generation or modification based on these descriptions. The remove history operation allows the user to select and delete some historical conversation data<sup>2</sup>.
E.3 User Interface
The user interface consists of 4 components: tracing operation, editing operation, resynthesizing operation, and a grid editor.
Tracing. The tracing operation has three panels, namely function selection, visual IO, and textual IO, see Figure 14. The function selection section allows users to choose from a list of available functions eligible for tracing. After selecting a function, IOs are shown to users within the visual IO and the textual IO panels, where the visual IO panel visualizes the IO into grids and the textual IO panel prints the IO as NumPy arrays.
Editing. The editing operation has three panels: function selection, function editing, and code synthesis, see Figure 15. The function selection panel serves the same purpose as the one in the tracing operation. After selection, users can modify ANPL code in the function editing panel and submit it to the LLM for code generation. They can then view the code generation progress in the code synthesis panel.
Note that participants were free to delete or modify their chat content and they can even write an ANPL program and then "compile" it by hand when using System B. Also, we did not explicitly forbid the users to write Python code in our experiment so they can write Python code if they want.
Figure 14: The tracing operation. Users can check the execution trace between high-level holes.
Figure 15: The editing operation. Users can edit the existing ANPL program through further decomposition or just modifying the code or natural language.
Resynthesizing. The resynthesis operation has three panels: function selection, IO entering, and code synthesis, see Figure 16. The function selection panel and the code synthesis panel serve the same purpose as the ones mentioned above. The IO entering panel enables users to constrain the programs generated by the LLM by providing IOs. Specifically, LLM generates a set of 10 candidate Python programs, and subsequently selects the program(s) that satisfy the given IO constraints as the compiled program.
Grid editor. In order to facilitate the transition for users between visual IO (i.e. colored grids) and textual IO (i.e. NumPy array), we have implemented a grid editor (Figure 17). This editor consists of the following elements:
1. Resize: Allows users to specify the dimensions of the grid.
2. Generate: Generate the corresponding grids by inputting a Numpy array.
3. Reset: Reinitializes the current grid to its initial state.
4. Copy: Converts the current grid into a Numpy array and copies it to the clipboard.
5. Graphical editing area: Provides three operations, including edit, select, and flood fill, along with 10 different colors, for direct editing of the colored grids.
Note that this editor was provided as a supplementary tool to all systems, so it has no impact on the final experimental results.
Figure 16: The resynthesis operation. Users can provide IO constraints and ask the compiler to resynchronize the program.
Figure 17: The grid editor.
E.4 Suggested Prompts for System B
Users can input natural language without restrictions in system B. To enhance the user experience, we offer them a suggested prompt template shown in Figure 18.
E.5 Task Assignment
Table 1 provides an assignment for ARC tasks. Each number corresponds to a question; a number highlighted in blue instructs use of System A before System B, whereas an orange number, conversely, denotes System B to be operated before System A.
You are a skilled Python programmer.  
Your task is to write Python code to transform the input grid into the output grid.  
In the input grid, you should see ...  
To make the output grid, you should ...  
Return your Python code in Markdown format.
--- python
import numpy as np
black, blue, red, green, yellow, grey, pink, orange, teal, maroon = range(10)
def main(input_grid: np.ndarray) -> np.ndarray:
    ...
Figure 18: prompt for System B.
User id	Tasks
0	4 34 36 48 50 116 128 222 230 309 384
1	6 74 101 102 104 139 141 150 152 218 235 247 257 279 289 312 332 340 343 356 368 370 374 382
2	21 55 61 65 69 86 92 164 165 182 183 186 227 234 258 261 262 271 307 310 344 378 379 386
3	0 8 22 75 95 108 125 140 157 192 198 202 205 228 237 245 255 265 280 316 339 346 363 365 371 381
4	24 28 37 68 71 80 87 98 114 124 166 170 200 206 208 241 272 297 298 301 302 303 315 317 319 327 336 360 361 389 398
5	16 25 30 56 62 70 111 120 130 142 151 173 180 185 212 215 226 275 295 347 364 388 393 397
6	53 63 105 135 137 159 176 224 244 260 278 300 320 341 359
7	10 12 14 35 43 44 90 97 144 161 162 175 199 219 231 236 263 287 299 323 331 345
8	91 94 112 126 129 168 181 190 191 193 248 264 285 306 387 391
9	18 26 41 77 79 122 127 131 160 223 252 277 292 321 338
10	1 2 9 38 42 54 81 88 96 99 115 136 145 147 153 246 283 308 322 355 380 383 385
11	13 23 31 32 49 57 118 132 138 149 211 225 240 251 267 270 286 304 313 314 318 353 396
12	15 58 64 83 117 119 155 156 167 189 210 214 221 233 242 254 256 276 281 293 296 305 325 354 367 369 376 392 395
13	5 27 33 51 82 123 134 172 177 178 201 216 229 232 243 282 288 311 330 358 377 399
14	20 40 47 73 171 187 195 217 220 328 337 351
15	11 29 46 60 93 106 107 110 113 163 184 209 213 239 274 326 333 342 372
16	39 59 67 76 78 103 158 179 188 204 207 253 266 294 334 335 348 350 357 362 366 390 394
17	3 17 109 133 197 259
18	7 19 45 52 66 72 84 85 89 100 121 143 146 148 154 169 174 194 196 203 238 249 250 268 269 273 284 290 291 324 329 349 352 373 375
Table 1: Task assignment. Blue: use System A and then System B. Orange: use System B and then System A
F Additional Experiments and Analysis
In this section, we discuss the reasons for choosing ARC as our main benchmark, conduct further analysis of experimental results, and provide additional results on other tasks to comprehensively demonstrate the advantages and capabilities of ANPL.
F.1 Benchmarks
Why do we choose ARC instead of typical code generation benchmarks like HumanEval [9] and APPS [26]. We evaluate ANPL on ARC instead of common code generation benchmarks [3, 9, 41, 44] according to the following three reasons: (1) We clarify that what we are measuring is the system's ability to convey the user's intent through interaction, not the system's ability to solve problems on its own (e.g. the user has a correct algorithmic solution in mind, but it is tedious to realize it in Python). In this regard, ARC's features (easy for humans to understand and solve but difficult to express via programming) would be more suitable than typical programming datasets such as MBPP and HumanEval. (2) We find that typical programming datasets are simpler for programming compared with ARC. The average line of code of MBPP, HumanEval, APPS, and ARC are 6.71, 7.78, 19.95, and 26.47 respectively. Competition datasets like APPS challenge the users to solve the tasks rather than generate code that fulfills users' intents. (3) Solutions for common code generation benchmarks can be found on the Internet, which runs the risk of data contamination. On the other hand, it is unlikely programmatic (python) solutions for ARC exist in any online corpus, making tasks in ARC unique enough for LLMs and participants of human study. However, to fully present the advantage and capability of ANPL, we have supplemented additional experiments with human studies on APPS benchmark, case studies on "realistic and long" programs, and automatic code generation on HumanEval benchmark.
Why does the standard of "success" the accuracy of the training I/Os. Our experiment assesses the system's capacity to fulfill user intents rather than users' ability to design generalizable algorithms. User intent is defined as writing a program to pass the provided I/Os. Thus, we present the solving rate of test I/O in the main paper. Although it is a simplified setting, it is appropriate and sufficient taking into account the 440 man-hours invested already. However, we understand that readers may be interested in other information related to ARC itself, such as whether the Python code can be used to generate I/Os or whether DARC can provide insights into human decomposition strategies. Thus, we included the accuracy of all I/Os in the appendix as an addition.
F.2 Additional Analysis on ARC
The importance of the control structures and holes. In order to analyze the significance of the programming model, we conduct an analysis on the proportion of control structures (e.g., for, while, if) and holes in programs that were correct in both system A and system B, and the proportion of programs that were correct in system A but incorrect in system B. In the programs that were correct in both systems,  $47.8\%$  utilized control flow, maintaining an average of 2.47 holes. On the other hand, a remarkable  $97.4\%$  of programs that were correctly functioning in system A but failed in system B incorporated control flow, and had an increased average of 3.37 holes. Results show that the introduction of control structure and hole has a significant positive impact on the user's ability to accomplish highly complex tasks.
The distribution of time and the number of interactions. Furthermore, we conduct an analysis of the time consumption and the number of interactions involved in solved problems. That is, we filter the problems that can be solved by both systems A and B, collect the time and number of interactions spent by users, and then construct the two distributions shown in Figure 19. Results show that, for these tasks, system A exhibits faster completion times and fewer interaction counts (without TC) compared to system B, with slight advantages in terms of time and interaction. Nevertheless, this advantage is not highly pronounced, which could be attributed to the relatively low difficulty level of the problems that both system A and system B are capable of solving.
Generalization ability. We further examine the accuracy of programs from System A and System B using all IO cases, see Table 2. System A and system B have similar generalization abilities based on the observation that for tasks passed on test cases by both two systems (224), the number of tasks
Figure 19: The distribution of time and the number of interactions. Trace Calculated (TC) means the trace mode is considered into interactions.
B
A	✓✓	✓×	××	Total
✓✓	177	5	45	227
✓×	10	32	31	73
××	7	3	90	100
Total	194	40	166	400
Table 2: Generalization on ARC.  $\sqrt{}$  means the compiled Python program passes both train cases and test cases.  $\sqrt{}$  means the Python program passes the test cases but fails train cases.  $\times \times$  means the program fails in all the cases.
that system A failed to generalize on all IO cases is 10/224 and the number of system B is 5/224, and these two numbers have no significant differences. Thus, we conclude that some programs failed to generalize to all IO cases because of the difficulty of the corresponding tasks and the designing of the underlying algorithm, which is independent of the usage of systems.
GPT-4 results on System C and D. During the experiment, we didn't have access to the GPT-4 API and it is impossible to conduct another large-scale human study for GPT-4 now. However, we can measure the solving rate w/o the interaction of ANPL+GPT-4 (System C) and GPT-4 (System D) with existing data: From the result in Table 3, we can conclude that (1) although a more advanced LLM would achieve higher accuracy, the utilization of ANPL always results in an improvement. (2) ARC still remains difficult for GPT-4 to give a 1-shot answer, making intent clarification a persistent challenge in using LLMs as programming assistants.
F.3 Human Studies on APPS
Here we aim to discuss the generalization ability of ANPL on typical programming tasks and compare ANPL on programming ability against Parsel (not automatic code generation ability).
We conduct a small study with 4 participants and 20 randomly selected APPS-interview problems. Other domains considered were MBPP, HumanEval, and APPS-competition problems. We did not choose MBPP and HumanEval because they are much simpler compared with ARC, with many questions solvable using just a few lines of Python code (The [min, max, mean] of lines of ARC's (passed) program is [4, 113, 26.47] while MBPP is [2, 50, 6.71], HumanEval is [2, 32, 7.78]). Also, we did not test on competition-level tasks limited by user ability. Each participant was instructed to solve each problem with three systems (ANPL, GPT-3.5-turbo, and Parsel) appearing in a random order for a total of 30 minutes per system. We replicated Parsel with GPT-3.5-turbo because Codex is no longer available. As shown in Figure 20: (1) ANPL outperforms the original GPT-3.5-turbo and Parsel in final solving rate and solving time which is consistent with the result on ARC. Specifically, ANPL achieves  $100\%$  solving rate, GPT-3.5-turbo achieves  $85\%$ , and Parsel achieves  $55\%$ . Also, the 1-shot performance (without interaction) of them is  $30\%$ ,  $0\%$ , and  $15\%$  respectively. (2) The solving rate for APPS-interview is higher than the one on ARC, indicating APPS-interview is easier than ARC for human interaction. In fact, the challenge of APPS lies in how users solve the tasks
Table 3: GPT-4 and GPT-3.5-turbo results on System C and D
LLM	System C (ours w/o inter.)	System D
GPT-3.5-turbo	23.50%	16.75%
GPT-4	30.50%	23.50%
B
Figure 20: Results of APPS-interview. A: Solving rate of four systems, where System A is ANPL, System B is GPT-3.5-turbo, System C is Parsel, System D is ANPL w/o interaction, System E is GPT-3.5-turbo w/o interaction, System D is Parsel w/o modifying. Note that Systems C and D here are different from the main paper. Parsel is replicated with GPT-3.5-turbo and set to sample 10 Python programs each time, which has the sampling number as the resynthesis operation of ANPL. B: The relationship between solving rate and time consumption. C: The relationship between solving rate and number of interactions. Note that Parsel cannot effectively facilitate interaction so it is presented as a dashed line.
rather than grounding users' intents to code. (3) Parsel performs the worst because Parsel and ANPL have different design objectives: ANPL is designed to ground user intent to code through interaction while Parsel is designed for automatic reasoning without human intervention. Specifically, Parsel does not even retain the context of past interactions. This leaves the only method for "debugging" to continuously modify the initial Parsel code and provide more I/Os until it can be compiled to Python in a single attempt.
Overall, we conclude that (1) ANPL is generalizable to different domains and (2) ANPL and Parsel are substantially different systems: ANPL is designed as an interactive programming system to significantly reduce programming complexity and assure code quality, where the users can iteratively interact with the system to safely ground their minds to code; While Parsel is primarily a reasoning system, where the programming task is given to it "as is" and it leverages the capabilities of LLM to decompose and ground it to code without any user interventions.
F.4 Case Studies on "Long and Realistic" Programming Tasks
To further show the ANPL's generalization ability, we conduct a qualitative study by using ANPL to compile several projects drawn from CS courses and GitHub: text editor, robot control, naive MAGIC card game, and LS-8 CPU. These tasks are the prototypes of realistic applications and are representative in terms of the program form with complex control flows and many sub-functions. They are much longer (198 lines of code on average) than existing program synthesis benchmarks (MBPP[6.71], HumanEval[7.78], APPS[19.95]).
It is shown in Figure 21 that ANPL can be used to implement these applications with much less coding effort (with  $33.21\%$  lines of code) and obviously lower programming barrier (by using natural language) than Python. Such results show that ANPL has the potential to be adopted in real scenarios with great superiority of lower programming complexity and better code quality assurance.
F.5 Automatic Code Generation on HumanEval
Settings. Following Parsel [71], we generated 100 test cases with temperature  $= 0.6$  for each problem in HumanEval. Then we generated 4 high-level solutions with temperature  $= 0.6$  and corresponding ANPL codes with temperature  $= 0.2$  and presence_penalty  $= 0.1$ . For each ANPL code, we generated 16 Python implementations for each function. Then we sampled the programs composed of generated
Figure 21: The ANPL and Python code of four projects: (a) LS-8 CPU (b) a robot controller (c) a text editor (d) a naive MAGIC. We present part of the code and mark the ANPL code and its corresponding Python code with the same background color.
functions and evaluated them by generated test cases. If one program passed all the test cases, it would be chosen as the final submission and evaluated by the real test cases. Otherwise, we would debug for each function of the program that passed the most generated test cases. In function-level debug, we used the execution traces to ask LLM for 8 fixed functions with temperature=0.6. Then we repeated the evaluation and debug processes for at most 2 rounds. For LLMs chosen, we use GPT-3.5-turbo-0301 and GPT-4-0314<sup>3</sup>. Our prompts are shown in Figure 22, 23, 24, 25.
Results. As shown in Table 4, we can conclude that although ANPL itself is not designed for automatic code generation tasks, due to its explicit recursive decomposition property, this framework has a certain advantage when used for automatic code generation. It achieved  $76.22\%$  on GPT-3.5 and  $86.59\%$  on GPT-4, both significantly surpassing the original LLM model. Furthermore, the design of ANPL for natural programming debugging was not reflected in this test, so there is still room for improvement in ANPL's performance in automatic code generation tasks, which we will consider as future work
Note that we did not check the detailed date carefully and this might be different from the other's choices due to OpenAI API change
Method	LLM	Pass@1
Original	GPT-3.5	56.7
CodeT [8, 53]	GPT-3.5	65.8
Reflexion[53]	GPT-3.5	68.1
ANPL (ours)	GPT-3.5	76.22
Original [45]	GPT-4	67
Original [7]	GPT-4	82
Original [53]	GPT-4	80.1
Reflexion [53]	GPT-4	91.0
(Parsel + CodeT) [71]	GPT-4	85.1
ANPL (ours)	GPT-4	86.59
Table 4: GPT-3.5 and GPT-4 Pass@1 accuracy on HumanEval [9].
-----Question-----  
{question}  
-----Task-----  
Give an assert test for this question in the following format. Each assert statement should  $\leftrightarrow$  be on one line.  
-----Test-----  
assert  
You should only output the assert test. Omit explanations or any additional text.
Figure 22: Test generation prompt for HumanEval.
-----Question-----   
{question}   
-----Task-----   
Propose a clever and efficient high-level solution for this problem. Consider all edge cases  $\hookrightarrow$  and failure modes..   
Some common strategies include: Constructive algorithms, Binary search, Depth-first search (DFS) and similar algorithms,  $\hookrightarrow$  Dynamic programming, Bitmasks, Brute force, Greedy algorithms, Graphs, Two pointers,  $\hookrightarrow$  Trees, Geometry, Graph matchings, Hashing, Probabilities, Data structures, Sortings,  $\hookrightarrow$  Games, Number theory, Combinatorics, Divide and conquer, Disjoint set union (DSU),  $\hookrightarrow$  Expression parsing   
Let's think step by step to come up with a clever algorithm.   
You should only output high-level solution without any pseudocode or code.
Figure 23: High-level solution prompt for HumanEval.
-----Question-----   
{question}   
-----Solution-----   
{solution}   
-----Task-----   
Translate the solution for the problem into python code in the following format:   
-----Program-----   
python code   
You should define some helper functions before function {func_name} to decompose it. Each  $\rightarrow$  function should be described by docstring.   
You should only output the python code! Omit explanations or any additional text!
Figure 24: Translation to ANPL prompt for HumanEval.
-----Question-----   
{question}   
----Solution-----   
{solution}   
Here is an program implementation of the solution.   
----Program-----   
{program}   
Here is a function of the program with input-output traces.   
----Function-----   
{function_with_traces}   
----Task-----   
If there are some mistakes or exceptions in the function, return the fixed function. You can  $\rightarrow$  define helper functions before it to decompose it into sub-functions. Your output should be in the following format:   
def {func_name}(..): The description of the function.   
You should only output the function code! Omit explanations or any additional text!
Figure 25: Function-level debug prompt for HumanEval.
G DARC Details
Figure 26: The header of CSV files.
def main(input): centers = "traverse the input which is a 2-dim numpy array, return positions which satisfies that  $\leftrightarrow$  there is no grey in its 3*3 neighbor"(input) scores = "for each position in the centers, count the yellow position in its 3*3 neighbor"(input,  $\leftrightarrow$  centers) center=Yellow, center 黑 = "return the center with the max scores and other centers" (centers,  $\leftrightarrow$  scores) output = "for each position in the position list, make its 3*3 neighbor yellow"(input, center_yellow) output = "for each position in the position list, make its 3*3 neighbor black"(output, center 黑) return output
Figure 27: An ANPL program in DARC.
(3)
(4)
Figure 28: The trace of the given ANPL program. (2) Centers are framed by red rectangles and we mark scores inside each rectangle.
The Recursive Decomposition Dataset of ARC Tasks (DARC) is an assemblage of interaction records associated with 400 ARC tasks. These records involve the intercommunication between users, the system, and GPT-3.5-turbo. Figure 26 presents the header of each CSV file. For each interaction, data concerning the role, action, content, and timestamp are completely collected and stored. The result is the entire user interaction history with our systems can be perfectly replayed at a later time. However, the response from LLMs will be different for the following reasons: (1) we used temperature  $= 1.0$ , which will cause different tokens to be sampled (2) the GPT-3.5-turbo implementation might be changed, which is outside of our control.