Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents an extension of the ARC dataset where tasks are labeled with natural-language descriptions that are sufficient for a human to generate the output grid, given an input grid without any additional examples demonstrating the task. The paper further provides annotations on the natural-language descriptions, showing how humans communicate natural programs and which concepts are needed for the ARC tasks.
strengths: * The paper tackles an important problem and the dataset provides a useful contribution to fuel advances in AI research.
* Natural programs are a very interesting perspective that can inspire the design of novel approaches to program synthesis for DSL-open domains.
* The dataset is well curated and easy to explore.
* Experimental results show how challenging the task is and provide a useful set of baselines (finally) that will hopefully encourage more researchers to work on the ARC tasks.

weaknesses: * The paper is very well-written but the presentation of the work could benefit from some changes. For instance, it is not immediately clear what some tags in Figure 7 refer to and it would be helpful if the main paper could provide more details on the baselines.
* It would be useful if the main paper could provide a link to the dataset and the source code to replicate the experiments (it seems the Github repository does not include the code to run the baselines on the dataset)

Minor comments:
* It would be useful to have the ARC task ID in Figures 1 and 2 instead of a numeric index
* Line 24: I disagree that ARC (by itself) presents tasks designed to benchmark the capacity to communicate. 
correctness: The claims of the paper are correct and the dataset is constructed in a sound way
clarity: The paper is very well-written and easy to follow
relation to prior work: The paper clearly discusses relation to prior work
additional feedback: None
review_title: A nice dataset to spur further research in program synthesis

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors provide an interesting natural language extension to the pre-existing Abstraction and Reasoning Corpus (ARC) dataset. They created natural language instructions that provably were able to instruct humans to solve ARC tasks in a sophisticaed crowdsourcing setup.

The authors provide extensive work around characterizing the resulting natural lanaguage instructions, as well as applying domain specific languages (DSL). The paper focuses heavily on discussion of DSLs, and my expertise is unfortunately very limited in this domain. To me it seems like the findings mostly point to the limitations of DSLs.

Beyond DSLs, the dataset might also be useful for experimenting with solving the ARC tasks with large language models.

strengths: The crowdsourcing setup used for creating the dataset is solid and very elegant. The dataset itself is relevant for its field and a useful extension of the established ARC dataset. All reported work is very solid and of high quality.
weaknesses: I am sceptical about the versatility of the dataset for enabling extensive down-stream re-use by other research groups. This seems relevant given that the paper was submitted to the Datasets track, and I think that other submitted datasets have a stronger potential for wide re-use. 

correctness: All claims seem correct to me; methods used for dataset creation are sound.
clarity: The paper is written very clearly.
relation to prior work: Prior work is sufficiently discussed.
additional feedback: N/A
review_title: Very interesting and solid, down-stream re-use potentially limited

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents an enhancement to the Abstraction and Reasoning Corpus by adding human-written "natural programs" sourced via Amazon Mechanical Turk. The contributions include the dataset, insights into how the data was collected, and results of an initial experiment of program synthesis using the data. 
strengths: The paper is well presented, motivated, and appears to provide a useful and accessible resource. The explanation of the motivation, data collection, and characterization of the data is comprehensive. The supplementary material is also thorough. 
weaknesses: I did not notice any limitations in the dataset (although I will not claim to be an expert in this domain). 
correctness: The dataset collection approach appears to be reasonable, especially considering cost and practical issues. 
clarity: The paper is clear and easy to read. It is well laid out. 
relation to prior work: The motivation of the work is well explained. There does not appear to be much in the way of comparing to other "natural program" datasets, but I am not aware of any other that exists. 
additional feedback: None currently.
review_title: Looks to be a valuable contribution to a hard problem

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents the LARC (Language-complete ARC) dataset containing natural language descriptions for how to solve ARC problems. These descriptions serve as ‘natural programs’ developed by human and is similar but different from traditional program synthesis (DSL-open vs. closed). Experiments show the difference between human-human and human-machine communication. This work shed light on introducing natural language (DSL-open problems) to program synthesis and communicative systems.
strengths: 1. This work is well-organized and performs a novel analysis of the intrinsic properties in the ARC benchmark, and proposes the LARC dataset to highlight these properties.
2. By introducing natural language descriptions collected from humans, the LARC problem is well-defined as a particular form of natural problems and program synthesis. The paper also gives a new and promising perspective on explaining the unique difficulty and how to solve ARC.
3. The paper have quantitative and qualitative findings comparing standard program synthesis and the natural program show how the human concept and communicative strategies differ from machines, calling for further study on leveraging language as an essential ingredient in program synthesis.
weaknesses: 1. My major concern is that LARC's natural language descriptions explicitly instruct how to solve ARC tasks, which differs from the design of learning from input-output examples in the original ARC. The original ARC focuses on some crucial priors (e.g., object prior, goal-directedness prior, basic geometry, and topology priors) and the capability of reasoning and abstraction (i.e., learning from examples). From my perspective, the instructions drastically ease the task difficulty by bypassing the abstract process, instead of truly solving ARC problems (learn from examples vs. execute from instruction). 
2. Adding natural language descriptions to the existing ARC dataset for program synthesis is good. However, given the existence of natural language descriptions and the rapid development of large program generative models (e.g., Codex, AlphaCode), I think the community would like to see how these models perform under natural language descriptions for program synthesis.
correctness: Yes, I think the claims made are good. The dataset collection process and evaluation methods are designed appropriately and performed correctly.
clarity: Yes, the paper is very well written. I enjoyed the flow of writing in the paper.
relation to prior work: Yes, I am not an expert in program synthesis, but from my perspective, the contributions and comparisons with previous works are well discussed in the paper.
additional feedback: I like the author's plan to adapt foundation models for LARC, since intuitively natural language descriptions can be better utilized through language models.
review_title: Review of "Communicating Natural Programs to Humans and Machines"

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors add a natural language component to the ARCS challenge set. They propose the term “natural programs” as others would say “instructions”, to refer to human to human instructions are able to be accurately carried out and have a program execution-like result. The dataset will be interesting to those concerned with ARC and possibly will be of interest to those doing general program synthesis, but the positioning comes across as overly bold. 

strengths: There is a general shift in program synthesis to tackling higher level requests from users, and through learning from demonstration.  This work fits in this discussion. The key contribution is the re-use of ARC to use a human "builder" as a substitute to a machine, and then demonstrate that the resulting instructions provided by the "describer" may not look the same as when the describer knew they were communicating with a machine. 
weaknesses: There is a lot of motivational language around “natural programs”, but without making the connection to the longstanding use of Wizard Of Oz protocols for the initial development of “human => interpreter” interfaces.  For example they write: “we suggest that starting with human instructions can yield insights for building systems”.  But haven't researchers been doing this since at least TRAINS and later TRIPS?  https://www.cs.rochester.edu/research/cisd/resources/trains.html

The focus is on an “open DSL”, but the assumption is still that you will map the language to some kind of representation via program synthesis. Does that make it open, or latent? I am uncertain whether this domain has an infinite set of useful predicates, or if there’d be a long tail of increasingly less important primitives, which at some point we consider the domain closed. 

L168-171: While it is sufficient to recall an unique function name in writing a computer program, the set of names – ‘box’, ‘block’, ‘square’ – might mean same or different things under different contexts. 
=> might these mean the same or different things under different contexts in programming languages? Perhaps more clear to say “fully resolved” function names?



correctness: I would have preferred to see a Codex based result in the main body of the paper, but the experiments overall are reasonable.

The dataset appears interesting and I agree with the protocol, given the intent. For example it is a nice step that the describer was asked to perform the task themselves as a quality control.
clarity: The paper is clear.
relation to prior work: TRAINS/TRIPS was discussed above. 

More recently there is work like CerealBar which is worth making an explicit contrast with:

Executing Instructions in Situated Collaborative Interactions
https://aclanthology.org/D19-1218/


The SMCalFlow dataset supports a programmatic representation for clarifications, so a human can incrementally ask questions about domain state, and to correct or otherwise edit their own prior statements. Imagine asking the describer to schedule a meeting under various constraints, but done without the ability to know the result of incremental computations.  Would such a dataset have similar properties, but not be a human to human interaction?

Task-Oriented Dialogue as Dataflow Synthesis
SMCalFlow: https://aclanthology.org/2020.tacl-1.36/


This is one example of emerging work on driving agents in open worlds (here MineCraft) with language to code.  Would the authors still consider this human-machine interaction to be a limited DSL?

Craft an Iron Sword: Dynamically Generating Interactive Game Characters by Prompting Large Language Models Tuned on Code
https://aclanthology.org/2022.wordplay-1.3/



additional feedback: 
Minor aesthetics: the humans in Fig 3, I suggest having their arms raised at some angle above 0 degrees, it is distracting that they align with the arrows. 

Is there a conflation between narrow vs open domains, as compared to programming languages vs natural language? You can have an expansive library in a programming language, and you can have narrow domain specific natural language. The protocol used in this work has one human communicating with another, who is a perfect stranger and without iterated interactions. The describer has to do everything ahead of time, without confidence in the capabilities of the builder.  As the authors make explicit, a human programer communicating with a computer can make more informed assumptions on the interpretive capacity of the “builder” (the machine). However, is the “natural” label more a spectrum, not a binary distinction? Under certain assumptions made by the describer on the builder, they will give a certain amount of details. Consider for example a military scenario, between a leader and a trusted subordinate: one expects communication in this case to be unambiguous and efficient.  This is in contrast to strangers on mechanical turk that never meet. Now consider a scenario where the describer was told that it would be a computer interpreter, using a certain programming language, and employing a certain library but where it wasn’t known which version of that library was available. The describer will be less efficient for lack of this knowledge, they will need to write more conditional code to verify which routines are available.
review_title: Adding natural language to ARC, considering the assumptions humans make on each vs on a machine

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper looks at the problem of generating programs from human natural language instructions. The major contribution is a dataset with 354 verified human instructions that completely specify the procedure to generate output grids from the input for the ARC task. The collected human instructions are properly analyzed and compared with computer programs. Experiment results show that although these instructions can slightly improve the performance when training examples are presented, generating programs from natural instructions alone is still quite challenging. 

I believe this is a good dataset paper. The paper itself is well motivated and well written. Despite the rather restricted domain and small scale, the dataset is properly validated and serves as a valuable contribution to the community. The comparisons between natural programs and computer programs can potentially inspire future works in this direction. 

strengths: Strengths
- The paper addresses an important problem of generating programs from human natural language instructions. Specifically, the proposed dataset contain instructions that are complete for program synthesis, making them comparable to computer programs. In general, I believe such comparisons are essential for understanding the difference between human and machine intelligence, and for designing machines that can naturally interact with humans. 
- The authors conduct sufficient analysis on the collected dataset, with additional annotation and data cleaning for quality assurance. I believe it is a valuable contribution to the program synthesis community. 
- The dataset is also easily accessible, with an online user interface for visualization.
- The paper is generally well written. The authors provide sufficient details for readers to understand the contributions. 

weaknesses: Weaknesses
- Although the task is more challenging, the size of the dataset is smaller than previous works (SCONE).
- Since the paper only concerns the grid-world domain which is rather toy-like and simplified, it is unclear that whether the conclusions can be applied to other domains.
correctness: The paper includes both qualitative and quantitative evaluations for each task setting. The authors are recommended to address the following comments:
- It seems the equation above line 198 needs normalization so that LHS can be a valid probability distribution. 
- A total of 383 tasks (200 for training and 183 for testing) are used in the experiment. In line 104, however, the number of tasks with verifiable natural program description is 354. 
clarity: The paper is generally well written. It can benefit from revisions for improved clarity:
- It seems some typos exist in lines 80, 92, 101.
- The abbreviation PCFG is not explained in the paper.

relation to prior work: I believe the paper discusses relevant works sufficiently. I'd suggest that moving supplementary table 3 to the main text to better visualize the contribution.
additional feedback: I appreciate the authors’ frankness for revealing the submission history. The previous responses can be helpful for evaluating the value of the paper.
review_title: A natural language dataset for program synthesis

Meta-review: metareview: The paper addresses the important problem of program synthesis from human natural language instructions. Sufficient analysis is done on the collected dataset, including annotation and data cleaning for quality assurance and the work is a valuable contribution to the program synthesis community.  Based on the reviews, I recommend acceptance of the paper.
confidence: 5: The area chair is absolutely certain
Decision: Accept