SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Participants solve more tasks with ANPL. First, we examine the problem-solving rate of ARC tackled by the participant when using different systems, shown in Figure 4. Specifically, we compare System A (ANPL), System B (GPT + interaction, i.e. ChatGPT), System C (ANPL without interaction), and System D (vanilla GPT), where Systems C and D represent the solving rate of one-shot generation without further interaction using Systems A and B, respectively. Results show that System A mean = 75.0%, 95%CI = [70.8%, 79.1%], B mean = 58.4%, 95%CI = [53.4%, 63.3%], C mean = 23.5%, 95%CI = [19.4%, 27.5%]), and D mean = 16.8%, 95%CI = [13.2%, 20.6%], where CI denotes confidence interval. Furthermore, we conduct paired t-tests on these systems, all of which are shown significant ( $df =$
1the top results on a Kaggle competition of ARC are all based on program synthesis techniques.
Figure 4: Results of problem-solving rates. A: Solving rate of four systems. B: The relationship between solving rate and time consumption. C: The relationship between solving rate and number of interactions. Trace Calculated(TC) means the trace mode is considered into interactions.
Figure 5: User differences. The solving rate of different users using System A and System B.
399,  $p < .0001$ : A-B: t=7.606, 95%CI=[12.2%, 20.8%]; A-C: t=20.583, 95%CI=[46.6%, 56.4%]; A-D: t=23.594, 95%CI=[53.4%, 63.1%]; B-D: t=13.646, 95%CI=[30.0%, 40.0%]; B-D: t=16.911, 95%CI=[36.9%, 46.6%]; C-D: t=3.152, 95%CI=[2.5%, 11.0%]. In conclusion: 1) With large confidence, ANPL performs the best and allows users to solve more problems, with a solving rate achieving 75.0% on average, while B achieves 58.4% (↑ 28.25%). 2) Programming with interaction (Systems A and B) is always better than programming without interaction (Systems C and D). 3) Even for one-shot code generation, ANPL without interaction (System C) performs better than vanilla GPT (System D).
ANPL saves user effort. Next, we measure the user effort on different systems. Figure 4 illustrates the relationship between solving rate and time consumption, as well as the relationship between solving rate and the number of interactions. We find that compared with GPT, ANPL exhibits a rapid growth curve, implying a significant advantage in terms of efficiency in problem-solving. Furthermore, to estimate the coding effort, we conduct a comparative analysis of the average lines of code between ANPL and the generated Python code. Some ARC task is simple and one of the main reasons for the time cost on simple tasks is that during programming, much time is spent waiting for GPT's response. The average waiting time is  $14.6\%$  for System A and  $23.5\%$  for System B. Besides, the users were instructed to solve the tasks as fast as possible and they chose not to program directly when using System A because it would require more effort. Another way to analyze user effort is to calculate the lines of code. We find that the [min, max, mean] of lines of ANPL is [2, 76, 10.29] while the one of Python is [4, 113, 26.47], and note that writing natural language is simpler than coding. We also demo the ANPL's application on projects including MAGIC card game and LS-8 CPU with  $67\%$  lines of code reduction (See Appendix F.4).
User differences. The characteristics of the ARC benchmark lead to significant variations in user performance. In order to measure such differences, we conducted an anonymous statistical analysis of users' solving rates while utilizing Systems A and B. Figure 5 shows that different users achieve variant solving rates. The (mean, max, min) solving rate achieved by users when using System A is  $(71.1\%, 95.5\%, 16.7\%)$  with variance 0.038, while System B is  $(57.2\%, 84.2\%, 33.3\%)$  with variance
Figure 6: Results of problem-solving rates when users can not see generated Python code and only interact with user interfaces and IO feedback. A: Solving rates of four systems. B: The relationship between solving rate and time consumption. C: The relationship between solving rate and number of interactions. Trace Calculated(TC) means the trace mode is considered into interactions.
0.022. This illustrates that although System A surpasses B in most cases and on three metrics, it exhibits greater instability, suggesting the presence of a certain learning cost compared with B.
ANPL performs even better for programming beginners. In order to simulate the situation where users are unable to understand Python code (programming beginners), we designed a new set of experiments with System A and System B. Neither system provides users with specific Python code; instead, users are required to complete tasks only based on user interfaces and IO feedback.
We selected four users for the experiment. To mitigate the potential impact caused by variations among users. Each user was given a set of 10 problems, which they had previously answered correctly using both System A and System B. In total, there were 40 problems administered. Results (Figure 6) show that System A mean  $= 89.9\%$ ,  $95\% \mathrm{CI} = [79.4\%, 97.8\%]$ . B mean  $= 37.2\%$ ,  $95\% \mathrm{CI} = [21.6\%, 52.5\%]$ . C mean  $= 40.2\%$ ,  $95\% \mathrm{CI} = [24.3\%, 55.6\%]$ . D mean  $= 12.5\%$ ,  $95\% \mathrm{CI} = [2.9\%, 25.0\%]$ . With large confidence, ANPL performs the best and allows users to solve more problems, with a solving rate achieving  $89.9\%$  on average, while B only achieves  $37.2\%$  ( $\uparrow 141.67\%$ ).
(b)
Figure 7: The proportion of three interaction modes. (a) Time. (b) Frequency.
Percentage of different interaction modes.
It can be observed from Figure 7 that among all the available options, the majority of users prefer the "edit" action, accounting for  $46.2\%$  total number of interactions and  $46.6\%$  total time cost, followed by the "trace" action, which constitutes  $47.5\%$  total number of interactions and  $44.6\%$  total time cost. This trend aligns with the typical programming workflow, where a smaller portion of time is dedicated to coding, while a significant portion is allocated to testing and debugging.
The influence of the system order. We denote the order as "A-B" ("B-A") meaning that the user first use System A (B) and then System B (A). With "A-B", the solving rate of System A is  $75.5\%$  (151/200) and System B is  $62.5\%$  (125/200). With "B-A", the solving rate of System A is  $74.5\%$  (149/200) and System B is  $54.5\%$  (109/200). We conclude that System B's performance increases  $8\%$  under the influence of System A, while System A's performance is more stable. This may be because System A tends to encourage users to approach problem-solving in a sketch and hole decomposition manner, and this manner is subconsciously applied to the subsequent System B. Therefore, we believe that "thinking in decomposition" itself can bring about certain improvements, and A leads users to this thinking manner. Additional results are shown in Appendix F.2.
5.3 DARC: A Recursive Decomposition Dataset of ARC Tasks
DARC contains all recorded interactions between participants and both the ANPL system and GPT-3.5-turbo on all 400 ARC tasks. 300 of the 400 tasks can be successfully decomposed and grounded in Python using ANPL (Figure 8), with an average of 2.7 holes. Due to the characteristics of ARC, the data contained in DARC is not the only solution, and different users have a significant impact on the dataset. Still, the value of our dataset is tremendous, as it contains: programmatic decompositions of ARC tasks by humans using ANPL, Python code for 300 ARC tasks, and fine-grained interaction histories between users and a code-generating LLM.
In this paper, we propose ANPL, a programming system that enables users to program and debug with natural language with interactive decomposition. The system was evaluated on a large-scale user study on the 400 ARC tasks. Experimental results show that with large confidence, programmers interacting with ANPL perform significantly better than interaction with the vanilla LLM (75.0% tasks solved vs. 58.4%). Furthermore, we will release the collected programmatic decomposition DARC dataset,
Figure 8: An example in DARC.
which provides valuable insights into how humans decompose programmatic tasks into simpler ones using natural language and input-outputs.
Our limitations and future work mainly focus on two aspects: the naturalization of LLM responses and library learning. Specifically, 1) the ANPL system should return easily understandable natural languages instead of Python code as a response to humans, and 2) how to build a natural language library community through library learning [19, 66] to allow users to share and reuse programs made by each other. Please see our detailed limitations and broader impact in the Appendix B & C.
We thank Tianyun Ma and Yutong Wu for their effort in submission and rebuttal. Tianyun helped us analyze data and Yutong conducted the automatic code generation experiment.
This work is partially supported by the NSF of China (under Grants 62002338, 61925208, 62222214, 62102399, U22A2028, U19B2019), the National Key R&D Program of China (under Grant 2021ZD0110102), CAS Project for Young Scientists in Basic Research (YSBR-029), Youth Innovation Promotion Association CAS and Xplore Prize.
[1] Samuel Acquaviva, Yewen Pu, Marta Kryven, Theodoros Sechopoulos, Catherine Wong, Gabrielle E. Ecanow, Maxwell I. Nye, Michael Henry Tessler, and Josh Tenenbaum. Communicating natural programs to humans and machines. In NeurIPS, 2022.
[70] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. MM-REACT: prompting chatgpt for multimodal reasoning and action. CoRR, abs/2303.11381, 2023.
[71] Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D. Goodman, and Nick Haber. Parsel: A unified natural language framework for algorithmic reasoning. CoRR, abs/2212.10561, 2022.
[72] Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self-monitoring. arXiv preprint arXiv:2106.03427, 2021.
A Additional Related Work 17
A.1 Programming with Natural Language 17
A.2 LLMs as an Interpreter 17
B Detailed Limitations and Future Work 17
C Broader Impacts 18
D ANPL Details 19
D.1 An ANPL Code Example 19
D.2 Syntactic Sugar 19
D.3 Implementation Details 19
E Human Studies 20
E.1 Questionnaire 20
E.2Tutorials 22
E.3User Interface 22
E.4 Suggested Prompts for System B 24
E.5 Task Assignment 24
F Additional Experiments and Analysis 26
F.1 Benchmarks 26
F.2 Additional Analysis on ARC 26
F.3 Human Studies on APPS 27
F.4 Case Studies on "Long and Realistic" Programming Tasks 28
F.5 Automatic Code Generation on HumanEval 28
G DARC Details 32
H Computational Resources 34
I Case Study 35
A Additional Related Work
A.1 Programming with Natural Language
Due to the low learning requirements, programming with natural language has been seen as an attractive programming mode since 1960s [6, 15, 17, 24, 37, 43, 49]. The relevant domains encompass semantic parsing [28, 65], language grounding [57, 58], and so on. Among these works, Wang et al. [64] is the most similar one which let users build complex voxel structures by defining alternative, more natural syntax, and increasingly complex natural concepts, starting from a core programming language. However, they are restricted by the pre-defined natural language specifications and domain-specific languages while this paper focuses on unconstrained natural language and general-purpose languages like Python. Furthermore, it should be noted that the interactive processes involved in these studies rely on manually annotated data, whereas our system operates in a truly debugging fashion for programs composed of natural language.
A.2 LLMs as an Interpreter
LLMs have been embedded in programs as an interpreter, due to their capabilities of common sense question answering and simple natural language reasoning [50, 51, 68]. For example, Cheng et al. [12] leverages LLMs to generate programs for questions like "Is Mexico North America?" with Codex API calls like  $f(\text{NorthAmerica?}, \text{Mexico})$  and then answer it by executing and prompting. Dohan et al. [18] composes LLMs into a probabilistic programming framework, which allows control flow and dynamic structure. Different from these works, we focus on leveraging LLMs to implement natural language modules into executable programs. We take the utilization of taking LLMs as parts of our generated programs as future work.
B Detailed Limitations and Future Work
Though our system has shown excellent performance revealed by the large-scale human study, there are still three main limitations to our current system. The first limitation is the response of LLMs. ANPL gives users a detailed implementation of each hole in Python and asks them to further debug. This requires users to read Python code when editing ANPL programs. LLMs may employ certain APIs that users may not be familiar with, resulting in the implementation of a function that deviates from the users' intended approach. Consequently, comprehending Python code becomes more challenging. Additionally, when LLMs automatically break down a function into sub-functions, identifying the specific sub-function containing a bug becomes difficult for users. To enhance the user-friendliness of ANPL and alleviate the burden on users in terms of their code capabilities, how can ANPL provide concise summaries of each function in easily understandable natural language should be studied. These summaries would enable users to identify any misinterpretations or incorrect implementations based on the corresponding descriptions, allowing them to modify their natural language descriptions accordingly without the need for complete redrafting.
Another limitation is the absence of comprehensive natural language libraries. In the context of code generation using LLMs, the quality of the prompt and the accompanying description assumes paramount importance. However, the creation of effective natural language descriptions necessitates considerable expertise in prompt engineering. In order to mitigate this issue, a natural language library should be established. The natural language content within this library is derived from two primary sources: library learning methods [66] and user contributions, and users can share and reuse natural language and corresponding implementations made by each other.
So far, ANPL has been limited to generating Python code for solving ARC problems through communication with users. However, as mentioned earlier, some ARC tasks cannot be fully addressed with Python programs alone and require the use of neural modules like object detection. Therefore, we need to integrate ANPL with neural modules, similar to works including Cheng et al. [12], Shen et al. [52], Suris et al. [56]. This integration would further enhance the capabilities of the ANPL compiler and expand the range of tasks that ANPL can handle. Additionally, the application scope should not be confined solely to the ARC dataset but should extend to multiple domains, such as chip design, program writing, and robot control. For example, Cheng et al. [11] automatically designed a CPU with Binary Speculation Diagram (BSD), yielding modular micro-operations that can be utilized by ANPL compiler and makes it possible to use ANPL-designed CPUs in the future. However,
due to the unique characteristics of ARC itself and the limited availability of human resources, it is well-suited for conducting user-programming experiments and human study reports. Thus, we have chosen ARC as the main platform for conducting our experiments and presenting our findings.
C Broader Impacts
On one hand, ANPL sheds light on the human-computer interaction paradigm by making the interaction more stable and reliable through low-cost predefined programming conventions. On the other hand, ANPL has the potential to promote the development of the programming field and broaden the scope of programming applications. For example, ANPL enables users to program and debug with natural language which can significantly lower the programming barrier. Furthermore, the proposed DARC dataset reveals how humans systematically decompose complex problems into simpler ones when faced with logical problems similar to ARC. This could provide further insights to cognitive science and foster advancements in related fields.
However, ANPL also raises safety challenges by producing code that is unaligned with user intent and can be misused. We refer readers to the broader impacts and hazard analysis discussed comprehensively by Chen et al. [9] as the basic component of the ANPL compiler is the LLM. Note that, compared to Chen et al. [9], a more significant concern with the usage of ANPL is its lower barrier to entry, allowing individuals with limited programming experience to generate code. This may result in code generated by ANPL lacking the scrutiny and maintenance of experienced personnel, making it more susceptible to misuse and thereby leading to more severe issues related to code security. Besides, since these users may have limited exposure to open-source communities such as GitHub, models trained on corresponding data may face greater challenges in user alignment.
D ANPL Details
D.1 An ANPL Code Example
Figure 10 is an ANPL code example for the task shown in Figure 9. In this example, users first decompose the task into "Change the input into four new arrays based on the central dividing line in the x and y directions" and "Find an array that doesn't have just one color". Then, users can either define a hole by its name like seperate_input with corresponding parameters, or they can just code a piece of natural language description like "Find an array that doesn't have just one color" and set the sketch by specifying its input-output variables.
Figure 10: An ANPL code example for task 64.
Figure 9: Task 64 of ARC.
def seperate_input(input):
    "Change the input into four new arrays based on the central dividing line in the x and y directions"
def main(input):
    inputs = seperate_input(input)
    output = "Find an array that doesn't have just one color"(inputs)
    return output
D.2 Syntactic Sugar
Recursion. ANPL employs a hole within the function's own body to implement recursion, following the thought of Y Combinator. A function can indirectly invoke itself by passing its own reference to a hole within its own body.
Figure 11 is an example demonstrating the use of hole-driven recursion with the Flood Fill algorithm. In this example, the floodfill function is passed as an argument to the hole "apply floodfill to adjacent pixels: above, below, right, and left". The semantics of the hole suggests that floodfill will be applied to the adjacent pixels, establishing an indirect recursion.
D.3 Implementation Details
ANPL interacts with the LLM via the prompt shown in Figure 12. In the prompt, the code section will be substituted with all the executed codes up to that point, and the hole section will be replaced
def floodfill (grid, i, j):
    if "is outside the valid grid area"(grid, i, j) and grid[i, j] != black:
        return grid
    else:
        return "apply floodfill to adjacent pixels: above, below, right, and left"(grid, i, j, floodfill)
Figure 11: Recursion in ANPL.
with the designated function name of the hole along with the natural language description given by the user. During the initial user input and function editing, it goes through a sequence of five attempts, starting with a temperature parameter of 0 and incrementing it by 0.1 with each try until it succeeds. In the resynthesis stage, ANPL requests the underlying LLM to produce 10 potential completions for each prompt. The text that ChatGPT generates will be subject to a maximum token constraint of 1024.
# system prompt   
As a pythonGPT, your task is to complete the unimplemented functions in the given python code, which are referred to as "holes" and are labeled as _hole0, _hole1, _hole2, and so on. Your implementation should align with the code and documentation using Python..   
# user prompt   
``python   
{code}   
The function needs to be given a new name. Markdown format should be used to return it.   
``python   
{hole}
Figure 12: Prompts used in ANPL.
E Human Studies
E.1 Questionnaire
We conducted a survey to investigate users' programming abilities, LLM usage experiences, evaluations of system A and system B, as well as the perceived importance of various functionalities within system A. The detailed questions are as follows:
1. How would you rate your programming skills?
1: Non-programmer
2: Beginner, struggles with solving LeetCode medium-level problems
3: Familiar with a programming language, understands basic data structures and algorithms, able to solve some LeetCode medium-level problems
4: Proficient in common data structures and algorithms, capable of solving many LeetCode medium-level problems
5: Skilled in data structures and algorithms, capable of solving LeetCode hard-level problems
2. How familiar are you with the Python language?
1: No exposure to Python.
2: Have used Python, familiar with basic syntax, but rarely used in daily activities.
3: Occasionally use Python to write simple scripts, not familiar with Python libraries such as NumPy and PyTorch.
4: Proficient in Python features, frequently use Python, and familiar with some libraries.
5: Mastery in Python and proficiency in using common libraries.
3. Have you used language models to generate code before?
4. How do you perceive the difficulty of ARC questions? (1: Very easy - 5: Very hard)
5. Do you find System A (ANPL) useful? (1: Very dissatisfied - 5: Very satisfied)
6. Do you find System B (natural language) useful? (1: Very dissatisfied - 5: Very satisfied)
7. How important do you consider the Trace feature of System A? (1: Very unimportant - 5: Very important)
8. How important do you consider the Edit feature of System A? (1: Very unimportant - 5: Very important)
9. How important do you consider the Resynthesis feature of System A? (1: Very unimportant - 5: Very important)
The results are shown in Figure 13. Participants in our human study are primary Python programmers and half of them are not familiar with code generation with LLMs. The average score of system A is 4.05, significantly greater than system B which scores 2.58. System A achieves not only a higher solving rate but also a better user experience than System B. Besides, most users find tracing and editing useful while resynthesizing less important, which shows the limitation of LLM code generation and indicates the importance of introducing user interaction in solving complex tasks like ARC.
(1) Programming Skills Rating
(2) Python Language Familiarity
(3) Language Model Code Generation
(4) Difficulty of ARC Questions
(5) Usefulness of System A (ANPL)
(6) Usefulness of System B (Natural Language)
(7) Importance of Trace Feature (System A)
Figure 13: Questionnaire Analysis. The average score of System A is 4.05, and System B is 2.58.
(8) Importance of Edit Feature (System A)
(9) Importance of Resynthesis Feature (System A)
E.2 Tutorials
The task requires you to solve ARC (Abstraction and Reasoning Corpus) tasks, each composed of several input-output pairs. These pairs maintain a uniform pattern and are structured as color grids with ten distinct colors: black, blue, red, green, yellow, grey, pink, orange, teal, and maroon.
The goal is to deduce patterns from the input-output pairs and communicate your solution to the system. In this experiment, you will interact with two systems: System A and System B. System A accepts ANPL inputs, whereas System B functions on full natural language. Both systems aim to generate Python code that transforms the input into the expected output.
Each task requires working with both systems in a specific order provided in the task assignment. To begin, find the ARC task solution independently (solve the task in your mind, i.e. not with Python). With a solution in hand, activate the system, which will initiate a timer.