SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Di Huang $^{1,\dagger}$ , Ziyuan Nan $^{1,3,\dagger}$ , Xing Hu $^{1}$ , Pengwei Jin $^{1,3}$ , Shaohui Peng $^{2}$ , Yuanbo Wen $^{1}$ , Rui Zhang $^{1}$ , Zidong Du $^{1}$ , Qi Guo $^{1}$ , Yewen Pu $^{4}$  & Yunji Chen $^{1,3\boxtimes}$ $^{1}$ SKL of Processors, Institute of Computing Technology, CAS  
 $^{2}$ Intelligent Software Research Center, Institute of Software, CAS  
 $^{3}$ University of Chinese Academy of Sciences  
 $^{4}$ Autodesk Research
https://iprc-dip.github.io/ANPL
Though LLMs are capable of generating plausible programs, it's challenging to interact with the LLMs further to revise the program, especially if the user's specific requirements are different from the initial proposal. In this paper, we introduce ANPL, an interactive programming system that ensures users can always refine the generated code towards their specific programmatic intents via structured decompositions. Borrowing the paradigm of sketching from program synthesis, an ANPL program consists of a set of input-outputs that it must satisfy, a "sketch" — control/data flow expressed in precise code (e.g. Python), and "holes" — sub-modules to be implemented by the LLM specified with natural language. The user revises an ANPL program by either modifying the sketch, changing the language used to describe the holes, or providing additional input-outputs to a particular hole, turning it into a sub-ANPL program that can be solved recursively. This workflow allows the users to offload programming burdens to the LLM as much as possible while retaining the ability to pinpoint and resolve bugs locally, without exposing the rest of the program to the LLM. We deploy ANPL on the Abstraction and Reasoning Corpus (ARC), a set of unique tasks that are challenging for state-of-the-art AI systems, showing it outperforms baseline programming systems that (a) without the ability to decompose tasks interactively and (b) without the guarantee that the modules can be correctly composed together. Additional evaluations on APPS, HumanEval, and real-world programming tasks have validated that the ANPL framework is applicable to multiple programming domains. We release the ANPL solutions to the ARC tasks as a dataset, providing insights into how humans decompose novel tasks programmatically.
The rapid development of Large Language Models (LLMs) has made significant progress in the realm of code generation tasks [7-10, 20, 23, 31, 33, 34, 42, 44, 47, 71]. Compared to traditional approaches in program synthesis [21, 22] that were constrained to specific, narrow domains (e.g., text manipulations), LLM-driven code generation can generate programs in a domain-general language (e.g., Python), making it broadly applicable. While LLM-driven code generators can save much of programmers' time in compiling typical programs – programs that are similar to those present in their
Equal contributions.
Corresponding author. Contact: {huangdi, nanziyuan21s, cyj} @ict.ac.cn.
Figure 1: ANPL overview. Given a programming task (e.g., input-output), the user decomposes it into an ANPL program, consisting of programmatically defined sketch (control/data flows, solid lines, programmed by the user and preserved by LLMs), and natural language holes (function modules, dashed lines, generated by LLMs). The ANPL compiler compiles it to a Python program and checks the Python code against the task's input-output. If incorrect, the compiler generates a debugging trace showing input-outputs for all the holes. The user can focus on debugging each hole independently, by either (1) decomposing it further, (2) explaining and modifying its natural language description, or (3) providing correct I/O by adjusting its traces.
training data, it is unlikely that an LLM can compile a user-specific program in 1-shot, without any additional user interventions [35, 62]. In software development, most code is not written perfectly on the first attempt, and users need to spend much effort on testing and debugging[46, 63]. Therefore, although using LLM for code generation can significantly reduce the cost of code writing in 1-shot, the subsequent interactions – editing and debugging an LLM-generated program – remain an open challenge. This challenge is severe in two aspects: (1) The LLM-generated program can be long and contain complex dataflows, making it difficult to make localized changes while ensuring the dataflows stay intact; (2) Language descriptions are inherently ambiguous to describe specific code edits.
We hypothesize that decomposition – breaking down a complex problem into simpler problems, a common practice in program synthesis [2, 48], interactive dialogue systems [30, 72], and cognitive science [16, 27] to name a few – during the programming and debugging interaction between a user and an LLM code generator can tackle these aforementioned difficulties. We propose the ANPL (Abstracted Natural Programming Language) system (Figure 1), which enables stable yet effective interactive editing/ debugging to solve complex programming tasks that an LLM cannot correctly solve in 1-shot. The key design insight of ANPL is decoupling the dataflow “sketch” (something that requires little user effort) and the tedious module implementation “holes” (automated with LLM). With ANPL, users have explicit control over the high-level task decomposition using the sketch, leaving the tedious work of function-level holes implementation for LLMs. Each hole is a stand-alone ANPL program, allowing the user to recursively decompose it further.
We evaluated ANPL by conducting a large-scale user study using the Abstraction and Reasoning Corpus (ARC) [14], a well-known corpus that consists of 400 unique tasks (in the training set) without predefined programmatic solutions. We recruited 19 Python programmers who interacted with our system for a total of 440 man-hours. Compared to prior works evaluating LLMs' code generation capabilities in interaction with real users Xu et al. [69](166 man-hours), Vaithilingam et al. [61](7.87 man-hours), ours is the most comprehensive evaluation up-to-date to the best of
our knowledge. We find that programmers interacting with ANPL perform significantly better than interaction with the vanilla LLM (75.0% tasks solved vs. 58.4%). The interaction between users and ANPL resulted in DARC, a dataset of programmatic Decomposition of ARC tasks, consisting of 300 instances of humans successfully decomposing ARC tasks to simpler tasks grounded in Python.
Specifically, this paper makes the following contributions:
- We present ANPL, a programming system that allows the user to explicitly manipulate dataflows while offloading the implementation of modules to an LLM, to decompose and ground programming tasks.
- We evaluate ANPL on a large-scale human study (440 man-hours) on a set of 400 ARC tasks, showing that programmers interacting with ANPL significantly outperform interaction with the vanilla LLM (GPT-3.5-turbo).
- We will release the collected programmatic decomposition dataset, DARC, to promote the development of research communities including LLMs, programming languages, human computer interaction, and cognitive science.
2.1 Code Generation with LLMs
LLMs have been applied to tackle the code generation problem, which is the task of automatically finding a program satisfying user intent expressed in natural language or other specifications [7, 20, 23]. For example, Codex [9], a GPT model finetuned on public code repositories from GitHub, demonstrates the potential of code generation with LLMs. With the idea of generating candidate programs and filtering, AlphaCode [34] samples a large number of candidate programs and filters using input-output examples. Chen et al. [8] and Key et al. [31] utilize formal specifications (e.g., test cases and assertions) generated by LLMs to verify the generated programs. Synchronesh [47] retrieves few-shot examples from training sets and then checks syntax, scope, typing rules, and contextual logic when synthesizing programs. Some work focuses on generating code based on feedback. CodeGen [44] factorizes a single program description into multiple parts and generates programs in multi-turns. Chen et al. [10], Le et al. [33] and Madaan et al. [42] provide LLMs with feedback messages generated by an executor, a critic network, or LLMs themselves to improve performance. Although these efforts have achieved a high level of code generation quality, they do not incorporate user interaction within the process of code generation. Basically, these methods can be integrated into our compiler to gain better compiling results. Another type of work, like Parsel, also utilizes the concept of decomposition [71]. Different from Parsel, our work is an interactive system between users and LLMs, which allows users to debug programs through further clarifying and decomposing each module. Furthermore, we conduct a large-scale user study to evaluate the effectiveness of our system.
2.2 User Interaction with LLMs
Interactions between humans and LLMs are necessary because it is difficult for LLMs to accurately understand user intent and generate correct answers in one shot [36, 52, 67, 70]. Among them, InternGPT [39] finds it helpful to combine the advantages of non-language instructions (i.e. pointing instructions) and language instructions. Instead of pointing instructions, our system allows humans to specify control/data flow precisely with programmatic language while keeping details in natural language. Furthermore, we have conducted extensive human studies on the ARC benchmark, which sufficiently demonstrate the capabilities of our system.
2.3 Evaluation of LLM Code Generation Ability on Human Studies
Several works conduct a human study on code generation tools to survey generation accuracy and user experience [4, 38, 61, 69]. However, these works focus on analyzing existing tools such as Copilot. The purpose of our human study is to verify the effectiveness of our proposed interactive system, ANPL. Furthermore, ARC containing abstract problems is a harder benchmark than those in previous studies. Our human study takes 440 man-hours on this hard benchmark, greater than the usual 100 man-hours, producing reliable results.
Other related areas including programming with natural language and LLMs as an interpreter are discussed in the Appendix A.
3 ANPL: The Abstracted Natural Programming Language
ANPL is a programming system that allows users to program and debug with natural language. It is Python-like and compatible with the original Python language, which provides users with great programming flexibility. In principle, we can apply our system to any programming language and in this paper, we have opted for Python as our base language. In this section, we will introduce ANPL's design principle and how to specify a ANPL program and then debug it.
3.1 Design Principle
The main idea under ANPL is to decouple the description of hole and sketch, which frees users from the tedious coding of function-level holes by allowing them to write the natural language and empowers them with the privilege of precisely setting sketch with few provided programming standards. Specifically, the sketch represents the control/data flow, and the hole represents the function module. This idea has been supported by the relevant experiences of early integrated systems like StreamBit [55] and SKETCH [54], which shows that users often have an idea about the general form of a solution (a high-level strategy) but feel it challenging to implement the program details. For programming with the help of LLMs, it offers the following advantages: 1) The user-specified sketch can convey user intent more precisely. 2) It enables users to trace bugs along the sketch, rather than aimlessly rewriting the entire natural language program and regenerating it. 3) The relationships between modules are explicitly represented by the sketch, thus making it easier for the compiler to ensure that other correct parts are not affected by debugging actions when users are debugging the erroneous sections of the program, which is not an easy task due to the unpredictability of LLMs.
3.2 Programming with ANPL
An ANPL program consists of a python-like sketch, and natural language holes.
Hole. A hole implements a function module with a natural language description, which will be fleshed out by LLMs during the compiling process. Each hole specified with a natural language description quoted by quotation marks ` or ''''. When called, holes should be organized by specifying its input-output variables, serving as the interconnections. To define a hole, users can either 1) define a hole as a sub-function with the function name, parameters, and descriptions, and then call the function with its function name and input-output variables, or 2) just define and call it with descriptions and input-output variables inline.
Sketch. A sketch is the control/data flow connecting different holes, specified with a programmatic language. Users constitute the sketch by assigning names to variables and using them as hole parameters in a data flow graph. Besides, users can write complex control flows with programmatic keywords (e.g., for, while, if) similar to that in Python to get more intricate ANPL programs.
An ANPL code example can be found in Figure 2, Figure 3, and Appendix D.1.
3.3 Debugging with ANPL
As an ANPL program is already decomposed into holes connected by the sketch, debugging over ANPL is targeted, where users can accurately locate bugs and make modifications to the relevant ANPL program.
Tracing. Tracing helps users to locate bugs. With the clear and explicit definition of sketch in ANPL, users are allowed to detect bugs by checking the execution trace between high-level hole, which is not easy in previous work [71]. Users can either use the preset "Trace" option in ANPL or just directly insert breakpoints in the code with print.
Providing input-output constraints. Users can provide input-output constraints, a typical gold standard for correctness, and ask the ANPL compiler to resynchronize the program to ensure the
def compiling(ANPL):
    while "ANPL program has a hole"(ANPL):
        hole = "return the first hole in ANPL program"(ANPL):
            generated_code = LLM(ANPL, hole descriptions, hole.IOs)
            dependency_graph = "analyze generated code, return dependency graph with topology
                → sort"(generated_code)
            entry_nodes = "return all entry nodes of the dependency graph with topology sort"(dependency_graph)
            if "the hole has been named by the user"(hole):
                new Dependency_graph = "remove unrelated nodes in the dependency graph"(dependency_graph, hole)
                ANPL = "fill the hole with generated code according to the dependency graph"(ANPL, generated_code,
                    → newdependency_graph)
            elif len(entry_nodes) == 1:
                ANPL = "fill the hole with generated code according to the dependency graph"(ANPL, generated_code,
                    → dependency_graph)
            else:
                # The dependency graph has multiple entry nodes, re-create the hole
                continue
            return ANPL
    def compilingDiff(ANPL_old, ANPL_new):
        if "ANPL program has a hole"(ANPL_new):
            ANPL_new = compiling(ANPL_new)
            ANPL = "merge two dependency graphs according to the merging principle of intention: user_new >
            → user_old > LLM_old > LLM_new"(ANPL_old, ANPL_new)
            return ANPL
Figure 2: The pseudo-code of ANPL compiler, consisting of the direct compiling process and the differential compiling process.
correctness of the compiled program. Each resynthesis will arise 1 API call with a batch size of 10 of the LLM.
Editing. ANPL allows users to debug as they would in traditional programming languages. Specifically, users can edit their code in four ways: 1) They can recursively decompose the original hole into lower-level holes connected by the sketch. 2) Reversely, they can abstract low-level holes and the corresponding sketch to a higher-level hole. 3) They can explain and modify the natural language descriptions of holes. 4) They can change the sketch by coding (e.g., changing the input-output variables of holes).
In addition, ANPL has various syntactic sugars, for instance, implementing a recursion with natural language by using the Y combinator, see Appendix D for details.
4 The ANPL Compiler
One of the crucial tasks that a compiler must undertake is to gain the trust of its users - it should accurately reflect the underlying code and provide users with the possibility of debugging. To achieve this task, the ANPL compiler mainly focuses on two problems: 1) in the program generation process, how to generate a reliable executable program while still preserving the control/data flows specified by users, and 2) in the debugging process, how to allow users to modify and regenerate the erroneous parts of the program while preserving functionalities in the rest of the programs.
4.1 Compiling ANPL
Preserving the sketch. To ensure the consistency of control/data flows before and after compilation, we first extract control/data flow information from the sketch and individual natural language descriptions from the holes separately. Then, we traverse and implement these holes while ensuring the integrity of the sketch. Note that while LLMs take in the entire ANPL program as context to implement the program of its holes, the sketch is fixed during program generation.
Generating the holes. LLMs iterate over and fill the holes one by one in their appearing order. When filling, LLMs can automatically create a name for the hole and decompose it into sub-functions. The main problem is that LLMs cannot implement holes reliably and may generate useless functions due to their unpredictability. The proposed solution begins with analyzing and clarifying the dependency relationships among the generated functions, and then identifying which function serves as the implementation of the target hole. The fundamental principle is that if the user explicitly names the hole, then any generated function with the same name as the user-specified one should be considered as the implementation of the hole, while other functions that it depends on are regarded as sub-functions of the hole; And otherwise, the function that does not depend on any other functions should be considered as the implementation of the hole. In the implementation, we first construct a dependency graph from the generated functions and find its entry nodes (nodes without any dependency on it) of the graph with topology sort. Then, we identify the target function based on three situations: 1) The user has named the hole and we just match it with the entry node's name and remove unrelated nodes. 2) There is only one entry node named by LLMs and we fill the target hole with it. 3) There are multiple entry nodes which means that there are multiple independent groups of functions and we cannot judge which one should be filled into the target hole, and thus we instruct LLMs to re-generate the implementation.
Finally, we get a compiled Python program that preserves user-specified sketch and has a well-organized implementation of holes.
4.2 Compiling Differences
During the debugging process, users generate an updated ANPL program. The problem is how to modify the erroneous parts while preserving the rest of the compiled Python program.
We observe that this process can be modeled as the merging of dependency graphs. Specifically, the old ANPL program corresponds to one dependency graph, and the new program debugged by users serves as another new dependency graph, and the merging process. For merging, we propose three assumptions: 1) the user's current intention takes priority over previous intentions, 2) the LLM's previous intentions take priority over the current intention, and 3) the user's intentions take priority over the LLM's intentions. Assumption 1) and Assumption 3) are obviously valid: It is precisely because the user wants to modify their previous intention or correct the compiled result with LLM errors that they will perform debugging operations. Assumption 2) holds because we want to minimize the impact of debugging on the generated program: When the user's intention does not involve the code, the code should not undergo any changes due to LLM's compilation.
Based on these assumptions, we can derive a fundamental principle for the priority of dependency graph merging: the user's current intention  $>$  the user's previous intention  $>$  the LLM's previous intention  $>$  the LLM's current intention.
5 Human Studies
We conducted a user study on 400 ARC training tasks to evaluate the effectiveness of ANPL compared to the original ChatGPT (GPT-3.5-turbo). We hypothesized that if ANPL can help people to program with LLMs, people should be able to solve more tasks and get the right program with less effort (i.e. time consumption) on solving and programming tasks.
5.1 Settings.
Stimuli. We use the ARC as our programming tasks. The ARC benchmark [14] was proposed to provide a standardized set of tasks that can be used to evaluate key aspects of human general intelligence, such as abstraction, generalization, object categorization, and so on [5, 13, 25, 29, 32, 40, 59, 60]. The primary objective of ARC is to require individuals to deduce a consistent procedure based on a limited number of abstract input-output examples, and then apply this procedure to generate a previously unseen output for a new input, as depicted in Figure 1. It is found that ARC is a representative programming dataset because 1) humans tend to solve them by coming up with
Figure 3: ANPL user interface. Participants utilize this programming interface to write and debug ANPL programs. The figure shows the ANPL program code entities along with corresponding sketch and holes diagrams. For a more detailed user interface, please refer to the Appendix E.3.
programs [1, 29] $^{1}$ , and 2) the solution of ARC cannot be found on the internet, which makes its problems unique enough for LLMs (and participants).
Participants. We recruited 19 primary Python programmers from our institution. Due to the difficulty of solving ARC problems by interactive programming, we first recruited more than 19 participants and then filtered out 19 of them by asking each person to complete  $\sim 10$  problems, checking if they can correctly operate the ANPL system, and asking if they wanted to continue. We paid an average of  $\sim$ 8.80 USD per hour - a standard programmer's wage within our institution.
Design. 1) Each participant is randomly assigned a set of non-overlapping ARC problems. The number of problems is different per participant depending on their preferred commitments. 2) There are two systems: System A, the ANPL, and System B, the original ChatGPT (GPT-3.5-turbo). Participants can interact with System B arbitrarily, including inspecting and changing Python programs manually. However, they cannot implement a whole Python function from scratch without interacting with System B. 3) As the programming level of participants can have a significant impact on their ability to solve ARC tasks, for each task, each participant used both System A and System B, presented in random order. 4) The participants were informed that the first interaction is crucial in order to minimize the impact of subsequent interactions. The programming interface of System A is shown in Figure 3 and System B shares a similar interface to System A.
Procedure. Given a set of tasks, participants can decide on the order in which to complete them, as they may need to start with simpler ones first. Participants are asked to solve the problem by programming (or interacting) with System A (or B) to get a Python program to solve the task. A soft time limit is 30 minutes which means that participants are suggested to give up after a 30-minute try. The held-out test input-output examples provided in ARC are used to check the correctness of the generated Python program, and participants can use additional input-output examples to debug their program. Participants are shown some prompt templates for using System B, while no prompts are needed in System A.