Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper introduces a new dataset of programming puzzles to benchmark the performance of AI models on program synthesis.

The paper makes the following contributions:
1. A novel problem representation called "Programming Puzzle".
2. A comprehensive dataset of programming puzzles that spans problems of a wide range of domains and difficulties.
3. A comprehensive benchmark of baseline systems including enumerative methods and GPT-3 solvers.
4. A user study to introduce a human baseline.
strengths: 1. The "puzzle" representation is powerful and precise. It can encode a wide range of programs while allowing objective evaluation and template-based instance generation. It can also be extended to include English descriptions by adding comments in the code.
2. The dataset is well documented. Each puzzle has an English description. Most puzzles have a reference solution. There are also notebooks on how to use the APIs. The framework is also very extensible, allowing the contributions of new puzzles.
3. The provided baselines reflect the current state-of-the-art. They are not trivial baselines. They can be very helpful for future research.
weaknesses: 1. All models do not solve any questions from hard domains like Algebra, ICPC, and IMO. Maybe including some easy examples in these domains can help to bootstrap the models.
2. No human baseline for the whole dataset.
correctness: The dataset is constructed in a sound way.
The evaluation setup is fair and the results are reproducible.
The description of the dataset is correct.
clarity: The paper is well written with plenty of examples that are very helpful for understanding.
relation to prior work: The paper clearly discussed how this work differs from previous contributions. 
additional feedback: Mention or compare against two recent related works:  
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (https://arxiv.org/pdf/2102.04664.pdf)  
GitHub Copilot: Your AI pair programmer (https://copilot.github.com/)
review_title: A comprehensive and well-documented dataset with insightful benchmarks.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents a concept of programming puzzles to evaluate methods for program understanding and synthesis, introduce an open-source P3 dataset, and evaluate human programmers, prompted language model, and diverse enumerative solvers. 

Puzzles vary in terms of difficulty, from introductory programs written by beginners to open Computer Science problems whose prospective solutions would lead to winning the associated prize. The authors introduced 200 problems consisting of from five to 32k puzzles each. Though this number is sufficient for performing experiments, it is assumed to be an initial contribution – the constantly growing base of problems will be maintained as a Github repository, making it possible to improve the P3 dataset over time. The creation process and considered problem sources were described in detail within the paper. Moreover, it is accompanied by a datasheet covering details of its construction and composition.

The conducted experiments and proposed baselines were used to answer the questions such as how well the dataset measures programming proficiency, how puzzle difficulty compares between humans and computers, or how models solving problems independently perform compared to solvers bootstrapping from previously obtained solutions.
strengths: 1. **Task novelty.** The introduced formulation of automatic problem-solving as a programming puzzle is to some extend novel. At the same time, it seems to capture the desired characteristic of datasets for program understanding and synthesis. The problem is formalized, well-described, and presented with related works from the field.
2. **Dataset quality.** Creation of the dataset was non-trivial and involved manual preparation of problem templates by the paper's authors. Problems seem varied and have meaningful taxonomy applied, allowing one to diagnose model concerning other solutions and human baseline, category-by-category.
3. **Problem selection.** The selection of problems seems well-conceived, e.g.,  there is a substantial gap between systems and human performance, and puzzles represent a broad spectrum of disciplines. Problems are demanding even for humans as junior/mid programmers underperform compared to developers with broad experience. At the same time, even the most experienced struggled in completing the task within the given time limit.
4. **Good, reproducible baselines.**  Evaluated models are good enough to obtain meaningful insights on the problem, and they are well described in the paper and appendix. The shared source code (declared to be opened after obtaining internal approval) seems to make the results fully reproducible after spending some time on confronting it with paper (see Weaknesses).
weaknesses: I found no major flaws. Nevertheless:

1. **Single run.** Error bars are reported only for human annotation. The authors state that solvers were not run more than once due to computational limits, but this is not conceiving and would be an argument against the dataset if true. It has been shown that different random initialization or data ordering can result in considerably higher scores (e.g., Dodge et al., 2020), and, consequently, future reviewers would demand multiple runs from solvers whose scores will be close to the current state of the art (especially, for trainable solvers relying on previously obtained solutions). If the team consisting of people from MIT, Allen, and Microsoft Research cannot afford multiple runs, who can and how can we ensure reliable comparison in the future?
2. **Source.** The accompanying source code would benefit from better README files and a clean-up. In particular, it is not clear which commands to run while trying to reproduce results and whether the commented-out commands were used. As baselines and source code are of critical value at the NeurIPS Datasets&Benchmark track, I expect a minimalistic tutorial on running the models.
3. **Varia.** There are a few statements not related to the paper essence that I find to be incorrect.
    - The described "BERT-based Transformer" seems to be the RoBERTa. Although the README file describes introduced changes in tokenization, it is still a byte-level BPE not present in the original BERT. Moreover, as far I understand, you started from the English roberta-base checkpoint. All things considered, the RoBERTa authors deserve credit here.
    - The statement regarding a "typical IQ test" is controversial and not supported by any reference or justification. If the authors had in mind the Raven's Progressive Matrices test, then the common notion is that it has only one justifiable answer. If the authors insist it is not true, such a view requires support from relevant psychometric works or broader discussion. As it seems to me there is no gain from mentioning IQ tests here, especially for their criticism, I would skip this part in the camera-ready version.

correctness: The dataset was constructed in a sound and verifiable way. Though the complete one-by-one list of considered problems nor information on why the concrete one was (not) included is unavailable, the authors provided a list of the exclusion criteria and common reasons for the problem to be disregarded. It seems to be enough in this case. The paper and accompanying appendices describe the preparation process in detail. Experiments are sound, apart from the fact that models guiding the search were trained only once.
clarity: The paper is well written and easy to follow. 
relation to prior work: There is a concise description of related work. Additionally, differences between the proposed puzzle representation and the relation of included problems to others, including competitive-programming problems, are stated implicitly or explicitly in several places of the paper and appendices.
additional feedback: Some readers may be interested in meta-information, like how much time was required to prepare a single problem to include in the dataset. It might be useful when planning similar efforts in the future.

Despite the mentioned weaknesses (am I missing something there?) I enjoyed reading the paper and consider it a precious contribution.

[Updated July 20 2021] Thanks to the authors for their engagement with the reviews, and introduced changes.
review_title: Varied problems for ML coding fit into easy-to-evaluate and straightforward form

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors introduce a dataset for python programming puzzles. These puzzles cover a large variety of challenges, from easy string manipulation, to college-level coding questions, to competition math and computer science problems, as well as unsolved conjectures. Verifying a solution only requires running a python program. The authors also run baseline experiments with GPT-3, and give a study evaluating the human-level and AI-level difficulty of the puzzles.
strengths: Update: after the discussion period, I keep my rating at a clear accept. I thank the authors for responding to the reviewer suggestions.


This paper is interesting and is a very novel type of dataset. Since this dataset is novel and relevant to problems AI programming and possibly AGI, it has the potential to have a high impact. This dataset can serve as a benchmark for teaching AI to evaluate/teach python code.

The dataset has impressive breadth, and the fact that answers are verifiable with python programs is very clean.

This dataset also has the potential to benchmark progress in creating AI that can solve complex IMO math problems and unsolved math conjectures.

The paper is clear and well-written.
weaknesses: (1) I didn’t see too much discussion on the explicit purpose of this benchmark. I noticed in the datasheet, the authors say “The dataset is created to evaluate and possibly teach machines to program in Python”, and this is also briefly mentioned in the discussion. A more detailed description would improve the paper. Since this benchmark has a very broad set of complex challenges, I am curious about the applications of this dataset in progressing research in artificial general intelligence (AGI). It seems similar to the ARC challenge by Chollet, but using python instead of images. Any comment from the authors would be appreciated.

(2) The dataset should be modified before it is ready for fair research, in a few different ways.

This dataset has 139,072 total points, but only 200 problem types, and the number of puzzles per puzzle type is highly variable (e.g., 10,762 number theory puzzles and 2000 game theory puzzles). Before this dataset is used in research, I recommend making a version that weights each problem *type* equally. E.g., currently solvers that achieve top performance would be biased toward number theory over game theory. This can be done either by assigning scores to the problems as the authors mentioned (but not just by perceived difficulty - also to weight problem type equally), or by only including exactly 2 or 3 problems per type of problem.

I recommend proposing a train/test split. And it seems important to make sure that the test set contains a large amount of problem types that are not in the training set, so that solvers must learn to solve new types of problems.

Give clear version numbers, along with python version and any other specific versions. (If any of the puzzles need other python libraries, that will be important as well.)  Although it is nice that this dataset only needs to execute code to create the labels, it is important that this code is executed in the exact same way e.g. when comparing solvers from different groups of researchers. This is different from most datasets that merely have labeled test sets.

Finally, does P3 have commented/annotated and un-annotated versions? It will be important to mention this detail e.g. when evaluating GPT-3 type solutions.

(3) The authors did not discuss potential negative aspects of the work, checklist 1.c. I don’t see potential ethical issues, but it is still good practice to discuss this.

(4) Not a weakness, and more of a nice-to-have: it would be nice to see this as a contest one day, similar again to the ARC challenge.
correctness: The dataset is constructed in a sound way.
clarity: The paper is very well-written and the structure of the paper is good.
relation to prior work: Related work is sufficiently discussed in Section 6. Although, see weakness (1).
additional feedback: Grammatical errors and minor comments
- Unfortunately there are a large number of broken hyperlinks throughout the paper. It seems that this occurs any time there is a link to programming puzzles on Github. Example: Line 108: the link to the Collatz problem is broken. All of these should be fixed.
- Figure 1, in the return statement of f3, it should be “1 < d < n” ? Otherwise, inputting 1 would return True.
- The Collatz Conjecture is used as an example many times throughout the paper. But it is not the best example because most mathematicians believe it to be true (source: Wikipedia), so it is unlikely that any program will ever find a counterexample to that programming puzzle. (As a consequence, it is likely impossible to achieve a perfect score on the P3 dataset.) You should mention this at least as a footnote.
- Line 46. “were to surpasses” -> “were to surpass”
- Line 52. “predict the next.” -> “predict the next one.”
- Typo on line 336. “problems and out puzzles”
- Line 167, Aren’t there primality tests which do not give prime factors?
- Line 171, give some citations here.
- Line 174, typo: “W plan”
- Table 1, typo: “Conways” -> “Conway’s”
review_title: Interesting and impactful paper

Meta-review: In this paper the authors provide a novel dataset/task of "programming puzzles" where an AI is tasked with understanding a program and giving it an input that will make the program output "True".  All reviewers were very positive, in particular noting the novelty and generality of the dataset and task, as well as the thoroughness of the paper in having SOTA baselines. I believe this will be valuable for the academic community.
Decision: Accept