Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: null
review_content: I played a little more with the dataset.

Unless I'm mistaken, it seems that over 40% of the the test case solutions y are in {0...10, -1, yes/no}

So if you can do well on a 12-way multiple choice test, you may be able to do well on the dataset

Here is the command I ran:

```python
>> mean([str(y).strip().lower() in "-1 0 1 2 3 4 5 6 7 8 9 10 yes no" for xy in input_output for y in xy["outputs"]])
0.43534307850557247
```

The > 40% rate holds across competition/interview/introductory levels.

This seems relevant to the 20% test case accuracy touted in the abstract, but I did not see such statistics discussed in the paper. 
review_title: Multiple-choice coding challenge?

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces an exciting dataset of 10,000 programming problems in English with test cases to evaluate correctness. The problems are scraped from various websites. An evaluation of language models shows that GPT models, appropriately fine-tuned, can write code that solves some of these problems. This is quite exciting since it has been very hard for computers to solve non-trivial problems described in natural language. This dataset is of significant size and quality that could be useful for training and evaluation of program synthesis systems.  This is a good paper -- if were been written more objectively and acknowledged it's limitations, it would be a great paper.
strengths: * The dataset consists of 10k problems in what seems to be well-formatted English problems (some with latex), many accompanied by Python solutions and test cases. 
* The models can actually solve some of the test cases and the code they generate looks very interesting. 
* The paper is well-written.
* The dataset has already been used in notable follow-up work on arXiv.
weaknesses: The paper presents cherry-picked statistics. For example, they will state that the average number of tests per problem is large but do not mention that more than half the test problems have very few test cases, and most of those test cases are directly written as examples in the Input-Output problem. Contrary to what the authors think, presenting one's own weaknesses does not ruin a paper---in fact it makes it much stronger. 
correctness: I have not re-run their experiments.
clarity: Yes
relation to prior work: Yes
additional feedback: Given the above mistakes found in the dataset, how should we trust that other things like de-duplication/etc. were done carefully. For example, a simple test would be to run the solutions from one problem on all others -- one would not expect the solution to a non-trivial problem to solve another non-trivial problem. Have you tried this? Did you find that any solutions to some problems solve others? This paper does not seem to discuss the checking and double-checking that should go into creating such a valuable resource.
review_title: Exciting results and an important but SLOPPY dataset

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors develop the APPS dataset for code generation from natural language specifications. It consists of 10k examples categorized into three levels of difficulty. Each example consists of a coding problem specification, one or more coded solutions written by humans and one or more test cases to check the generated solution. The authors evaluate state-of-the-art language models on the dataset to find a large room for improvement.

The contributions of the paper are as follows:
1. Novel benchmark for code generation from natural language specifications
2. Collection of 10k examples from open-access competitive coding websites
3. Evaluation of generated solutions using test cases instead of using BLEU 
strengths: The principal strengths of the paper are:
1. Compared to the restricted settings of the previous work on code generation, the APPS dataset can be used to learn the task of generating full fledged Python code given the textual problem specification of competitive coding challenges. 
2. Instead of using BLEU to evaluate the generated solutions, the dataset contains test cases that can be employed as a gold-standard metric for correctness.
weaknesses: The major weaknesses of the paper are:

1. The dataset does not decouple the task of algorithm design and code generation. One cannot be sure if a language model has not simply memorized the algorithmic implementations. Line#265 argues that memorization would follow uniform performance across all categories. Instead, we might expect a memorizing model to do well on those category of problems it has seen more of. For instance, GPT-Neo was pre-trained on contaminated data from Github. It is likely that Github has large amounts of example code for introductory problems, followed by some solutions to the interview level problems and even fewer solutions to competition level problems. This is an alternate plausible explanation for GPT-Neo's performance across the difficulty levels.

2. The paper makes several doubtful claims. The abstract mentions that GPT-Neo can pass approximately 20% of the test cases of introductory problems. However, only in the final Experiments section is it mentioned that GPT-Neo under went pre-training on contaminated Github data. In general, all the models being compared and analyzed side-by-side undergo very different training regimes. GPT-2 is pre-trained on decontaminated Github data and then fine-tuned on APPS. GPT-3 is only used in the few-shot setting. The results from these experiments cannot be used to make sweeping claims such as in line#60: "We find that accuracy decreases with difficulty level and improves through fine-tuning and model size increases." In Table 2, we see that only the accuracy of GPT-Neo decreases with difficulty level. For all other models, the accuracy increases when going from introductory level to interview level and then decreases from interview level to competitive level.  
correctness: See second point in the Weaknesses section.
clarity: The paper is well-written and easy to follow.
Except in lines 183 and 184 where the false positive rate is calculated using some quantities like 890 and 765 which are not clearly defined.
relation to prior work: The literature survey is exhaustive and the proposed work is differentiated well from all the related work.
additional feedback: -
review_title: Novel dataset and benchmark with some weaknesses

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors curated a dataset, APPS, as a benchmark for evaluating the quality of machine learning models for code generation/synthesis from a natural language (English) description of the coding problem. 
The dataset is collected from online judging systems (CodeForces, AtCoder, etc.) and other coding competitions (ACM, USACO, IOI, etc.) and contains 10000 coding problems, each with its problem description, test cases with specific input-output pairs, and some ground-truth (correct) solutions implemented in Python. Problems are classified into introductory level (3639), interview level (5000), and competition level (1361). 
10000 problems, 131,777 test cases, 232,412 correct solutions (Python), average length of problem description is 293.2 word.
Curation:
wrote custom HTML parsers for each source of problems, which allows us to properly format LaTeX expressions, lists, and sections in the question text. 
convert equation images to LaTeX using theMathPix API
perform deduplication using tf-idf features with SVD dimensionality reduction and cosine similarity
Several graduate and undergraduate students worked for 6 months

Test cases evaluation:
merges the judging functionality of several judging systems also standardize the format of test cases.

Experimental setup:
Pre-train GPT-2 on GitHub (decontaminated) – 30 GB of Python code 
Use GPT-3 Instruct model 
Use pre-trained GPT-Neo (pretrained PILE which included GitHub) and fine tune
Table 2 summarizes all the results – Strict accuracy is the honest metric.
Observation and analysis of the authors:
1.	Models can sometimes generate correct or superficially plausible code
2.	Many generated programs are free of syntax errors
3.	Syntax errors decrease exponentially with fine-tuning and increased model sizes.
4.	Using top-5 improves performance metrics
5.	GPT-3 does not do well 
6.	BLEU scores are not a good measure of model quality for code generation
7.	The Codex model does better, but far from have solved code generation

Main Conclusions:
GPT-3 performs quite badly for code generation
Some neural network models pre-trained on code can start to do code generation for some extremely simple programming problems, so there is a long, long way to go.

strengths: Goal is good – with so much hype around the “super models”, it is important to have a high-quality or any reasonable dataset to evaluate them. 
Curation effort is good – the appropriate cleansing such as problem cleanup, de-duplication, and test case standardization, and decontamination is in place
APPS is also used by the Codex paper.
The biggest contribution of the paper is to show that GPT-3, with all its hype, really cannot generate code with consistency. You cannot reap what you did not sow – since GPT-3 is not trained on code, it is not good at that.

weaknesses: The dataset is way too small to assess whether these large language models can do code generation well. It is not clear whether the extreme low strict accuracy is the result of a small fine tuning training set (5000 data samples) or the weakness of the neural network model. It would be very interesting to use sequentially bigger fine tuning training set (10000, 20000, 50000, 100K samples, etc) to see if the strict accuracy keeps increasing or saturates.
Limitation of the dataset is not discussed. The scope (pedagogical problems) and size (5000 training) of the dataset is too limited to thoroughly evaluate the problem of code generation. Granted that the goal is ambitious, and it is a good start, a discussion of the limitations is warranted. 
The list of observations in the paper are rather obvious, and I do not agree with observation 3. The reduction of syntax errors is due to pre-training of the GPT-2 and GPT-Neo with code instead of just texts, so they learned the programming language model better. 

correctness: Please refer to the weakness section above! Evaluation methods and experiment design has been appropriately performed!
clarity: Paper is well written and clear!
relation to prior work: Section on “Related Work” is too heavily focused on discussing code generation technique in prior art. This paper is more about the dataset and less about code generation technique, so there should have been more emphasis on comparing dataset quality with prior datasets with the same goal.
additional feedback: Show some examples of successful code generation to see how simple they are (e.g. are they one-liner or more complex). Are the generated code examples in the supplementary correct code?
Show the performance metrics for the pure pre-trained (not fine-tuned) GPT-2 and GPT-Neo models to get a baseline and an indication of how much fine-tuning helps. 
Use sequentially bigger fine tuning training set (1000, 2000, 5000 samples, etc) to see how the strict accuracy reacts to fine tuning training dataset size
review_title: Although small size, but good effort on measuring Coding Challenge Competence that debunks some myths/hype.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: I would like to thank authors for their responses. 
I am updating based on the responses.

The paper introduces a dataset for evaluating code generation from natural language specifications.
The evaluation setting aims to mirror how human programmers are evaluated during an interview: given a problem description, their task is to implement an algorithm that solves the problem. The correctness of generated programs is assessed via unit tests. The dataset consists of 10000 samples in Python programming language, which are subdivided into 3 distinct difficulty categories. 

The APPS benchmark can potentially serve as an important and valuable asset in evaluating language models of code, which targets a real-world informative setting, and overcomes several limitations of the benchmarks which are based on string matching and n-gram based criteria like accuracy, BLEU or ROUGE.

The APPS benchmark targets a narrow task of code generation from natural language description. There are numerous other software engineering tasks (involving code gen) that may not be covered by this benchmark. For instance, program repair task which is generally formulated as follows: given a buggy code produce a fixed code. Or code summarization – which is an inverse, code to NL task. Can these types of problems be formulated the same way? They can be solved with language models.
strengths: The paper introduces an important benchmark to evaluate code generation. The authors correctly point out several limitations of traditional NLP based evaluation metrics like BLEU score when applied to code, and provide solution.

Authors correctly point out, that a passed unit tests does not guarantee program correctness as there might be false positives. The propose multiple unit tests, and evaluate the likelihood of false positives.

It is important that APPS benchmark is not geared towards a specific model architecture (e.g., GPT) or a modeling task (language modeling). Or work only with left-to-right code generation models. While large language models are SOTA right now, their performance is still very low (up to 20-30% accuracy). It is likely that a new technology will emerge (perhaps, it will utilize programming language grammars, and may be based on an encoder-decoder architecture, and not left-to-right code generation) which will outperform those. It is important that the APPS benchmark remains applicable to those. 
weaknesses: Since the APPS dataset is a collection of problems extracted from public websites like CodeForces, there is a risk of data leakage, especially since modern language models of code (e.g. Codex) are trained on a large fraction of GitHub and may have seen it in the training set. 

The APPS benchmark targets a narrow task of code generation from natural language description. There are numerous other software engineering tasks (involving code gen) that may not be covered by this benchmark. For instance, program repair task which is generally formulated as follows: given a buggy code produce a fixed code. Or code summarization – which is an inverse, code to NL task. Can these types of problems be formulated the same way? They can be solved with language models.

There exist similar benchmarks: HumanEval. It would be good to understand the difference.

Real-world code generation should take into account file-level (import statements, class and method signatures) and even inter-procedural contexts. As in the real interview, candidate might be allowed to use numpy to implement a given algorithm. But generally it might be interesting to consider code generation given specific source code project/library.
correctness: The claims made in the submission are correct.
clarity: The paper is well written and easy to follow.
relation to prior work: There exist similar benchmarks: HumanEval. It would be good to understand the difference. Both benchmarks appeared at similar time, so I am not sure it should be counted as prior work.
additional feedback: Figure4 label is confusing: It shows the size of the respective GPT-Neo models, but states “few-shot” for GPT-3 instead of the size.

L308: “Open AI Codex is a 12B parameter Transformer language model” typically, Vaswani’s encoder-decoder transformer model is referred to as the Transformer, which incorrectly suggests that Codex is an encoder-decoder model, while Codex is the GPT-style model (decoder-only transformer). While language modeling is possible with encoder-decoder models, throughout the task you distinguish two models (encoder decoder Transformer and GPT) so might as well do it here.
review_title: Valuable benchmark to evaluate LMs of code, but does not cover various software engineering tasks.

Meta-review: The paper proposes a dataset of 10k programming problems to promote research in code generation. The authors then evaluate SOTA language models on the proposed benchmark. All the reviewers agree that the proposed dataset is a good contribution. Therefore, I recommend acceptance of this paper.
Decision: Accept