SlowGuess/NeurIPS_2023_OCR huggingface dataset:
import numpy as np   
from typing import \*   
(black, blue, red, green, yellow, grey, pink, orange, teal, maroon)  $=$  range(10)   
def get_max_score_center(centers: List[Tuple[int, int]], scores: np.ndarray) -> Tuple[List[Tuple[int, int], List[Tuple[int, int]]]: max_score  $=$  np.max(scores) maxcenters  $=$  [centers[i] for i in range(len centers)) if scores[i]  $= =$  max_score] othercenters  $=$  [centers[i] for i in range(len (centers)) if scores[i]  $<$  max_score] return(maxcenters,othercenters)   
def findPositions WithoutGrey_neighbors(input: np.ndarray) -> List[Tuple[int, int]]: positions  $= []$  for i in range(1, input.shape[O]-1): for j in range(1, input.shape[1]-1): if np.all(input[i-1:i+2,j-1:j+2] != grey): positions.append((i,j)) return positions   
def make_neighbors_redYellow(input: np.ndarray, positions: List[Tuple[int, int]]) -> np.ndarray: for position in positions: input[position[0]-1:position[0]+2, position[1]-1:position[1]+2]  $=$  yellow return input   
def make_neighbors_redblack(input: np.ndarray, positions: List[Tuple[int, int]]) -> np.ndarray: for position in positions: input[position[0]- 1:position[0] + 2, position[1]- 1:position[1] + 2]  $=$  black return input   
def count YELLOW_neighbors(input: np.ndarray, centers: List[Tuple[int, int]]) -> np.ndarray: scores  $=$  np.zeros(len (centers)) for i, position in enumerate中心主任): scores[i]  $=$  np.sum(input[position[0]- 1:position[0] + 2, position[1]- 1:position[1] + 2]  $= =$ $\leftrightarrow$  yellow) return scores   
def main(input): centers  $=$  findPositionsWithoutgrey_neighbors(input) scores  $=$  countYELLOW_neighbors(input, centers) center_yellow, centerBLACK  $=$  get_max_score_center (centers, scores) output  $=$  make_neighbors_yellow(input, center_yellow) output  $=$  make_neighborsBLACK(output, center_back) return output
Figure 29: The compiled Python program in DARC.
300 of the 400 ARC tasks were effectively decomposed and converted into Python using the ANPL, averaging 2.7 holes per task – these tasks solve the test-input, but some may fail to generalize to other input-outputs, see Appendix F.2. The DARC dataset not only houses the final solutions to the ARC tasks but also encapsulates the diverse problem-solving approaches employed by different users. Crucially, it is a dataset detailing how humans decompose abstract procedural tasks into simpler sub-tasks, and ground each task into a program (e.g. Python) in collaboration with an LLM. We give an example of an ANPL program, the compiled Python program and its corresponding trace in Figure 27, Figure 29 and Figure 28.
The DARC dataset provides a valuable window into the system's task completion processes. By documenting ANPL decompositions, Python code, and detailed interaction histories, it permits us to gain insights into the practical application of Language Learning Models (LLMs) for programming. We hope that this dataset will be useful for others seeking to understand and refine similar systems.
H Computational Resources
For our human study, LLM APIs were called 4304 times for System A (10.76 per task), and 1923 times for System B (4.81 per task). The distributions of the number of LLM API calls are presented in Figure 30. Since System A generates 10 candidates when resynthesizing, the number of API calls on several tasks exceeds forty times. For most tasks, the number of interactions in the two systems is less than twenty.
Figure 30: LLM API calls of Systems A and B.
I Case Study
A case of user interaction. Figure 31 serves as an illustrative instance. Initially, the user enters the ANPL code into the system, which consequently produces a function for every hole and automatically verifies the code's validity. Upon encountering a programming error, the system issues a warning to the user. In response, the user examines the input and output of the find_smallest_unit function and discovers it doesn't align with expectations. The user then adjusts the natural language description, leading to ANPL successfully meeting the test input and output samples following a code regeneration. Eventually, the system presents the complete code to the user, as shown in the Figure 32.
Figure 31: User Interaction with System A and Subsequent Code Generation Process
Difficult Tasks. From the overall 400 tasks under consideration, several tasks pose a significant challenge to users because it is hard or impossible to find the solving logic. We present one case in Figure 33.
User-Specific Solution. The potential solution of the ARC problem is heavily reliant on the user's algorithm design, which has an impact on the difficulty of programming. For instance, in the ARC task illustrated in Figure 34, some users might attempt to identify the color pattern across each row or column. Others might note the black square's location, then rotate the grid 90 degrees and select the grid in the same position as the answer. The first approach is quite challenging, often too demanding to be completed within a time constraint. Conversely, the latter method leverages the rotational symmetry of the grid, enabling a direct translation into the correct Python code, eliminating the need for debugging.
import numpy as np   
from typing import \*   
(black, blue, red, green, yellow, grey, pink, orange, teal, maroon)  $=$  range(10)   
def find_smallest_unit(input_grid: np.ndearray) -> np.ndearray:   
""   
Given a 2D numpy array, this function finds the smallest repeating unit in the grid and returns it.   
""   
(width,height)  $=$  input_grid.shape   
for i in range(1, width+1): if width % i  $= = 0$  .. for j in range(1,height+1): if height % j  $= = 0$  .. unit  $=$  input_grid[:i,:j] if np.all(nptile(unit,(width//i,height//j))[width,:height]  $= =$  input_grid): return unit   
return input_grid   
def extend_unit(unit: np.ndearray) -> np.ndearray:   
""   
Given a 2D numpy array, this function extends the array to a 9x3 grid.   
""   
(width,height)  $=$  unit.shape   
if width >= 9 and height >= 3: return unit[:9,:3] else: extended_unit  $=$  np.zeros((9,3),dtype  $\equiv$  unit.dtype) for i in range(9): for j in range(3): extended_unit[i,j]  $=$  unit[i% width,j% height] return extended_unit   
def change_red_to_red(unit: np.ndearray) -> np.ndearray:   
""   
Given a 2D numpy array, this function changes all blue pixels to red.   
""   
unit[unit  $= =$  blue]  $=$  red return unit   
def main(input_grid: np.ndearray): unit  $=$  find_smallest_unit(input_grid) out  $=$  extend_unit(unit) out  $=$  change_red_to_red(out) return out
Figure 32: The code synthesized by ANPL.
Figure 33: Unsolvable ARC task for the user.
Figure 34: Example of an ARC task demonstrating differing user strategies.