Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to train fault-aware neural code rankers (I would prefer to call them classifiers) that can predict the correctness of sample programs without executing them (using unit tests). The rankers are trained to predict different execution information, such as compile/runtime error types (e.g., an IndexError or a TypeError). The paper shows that the trained rankers can significantly increase the pass@k accuracy of various code generation models, e.g., Codex, GPT-Neo, GPT-J on popular benchmarks, e.g., HumanEval, APPS, and MBPP.
strengths_and_weaknesses: **Significance**
The paper pointed out that current literature assumes that the unit tests are generally available to evaluate code generation models via execution and it is safe to execute the generated programs. The paper argues that such assumptions are impractical in real-world software development, making the work significant to investigate alternatives.

**Originality**
While I am unaware of any prior works that exactly do the same (training rankers to predict errors), the proposed idea is intuitive (anyone can come up with the idea with careful thinking). Though I see this work's technical contribution as thin; I appreciate the author's effort to study the effectiveness of the proposed approach.

**Clarity**
The paper is written fairly clearly. Except for one crucial piece of information, I did not feel anything important was missed. Also, the paper should rather use classifiers in place of rankers.

**Quality**
Overall the quality of the work is good. However, I felt the authors didn't go deeper in investigating stronger code representation models (they chose CodeBERT, which is a weaker model in literature; I would love to see a  few more models being investigated as rankers). This was important since there is a significant difference between pass@100 and ranked pass@1 on the benchmarks.


questions: 1. Given that the trained rankers are actually classifiers, how exactly the top-1 candidate is chosen to compute ranked pass@1? I assume the ranking is done based on code generation models provided likelihood score, and then wrong samples were filtered out based on the predictions of the fault-aware classifiers. I think this is an important piece of information that was not clarified.

2. Out of curiosity asking this question, since the experiments were conducted on V100-32GB GPUs, what confined the authors to choose better models than CodeBERT (e.g., CodeT5, PLBART) for the ranking models? In fact, the authors could have used GPT-Neo 1.3B as the rankers.
limitations: The major limitation of this work is the conflict between the motivation of the work and what is being proposed. While the paper motivates that in the real world, we cannot assume that there will be unit tests readily available to evaluate models via execution. However, the paper used the APPS dataset composed of unit tests. The authors didn't discuss how their work could be leveraged in the real world. For example, can we use the authors' proposed idea for generic programming (not competitive programming only)? I would prefer to see some discussion.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper complements a deep learning based program synthesis module with a separate fault aware ranker framework that learns execution pattern of the generated program without actually executing it. The work is inline with recent advancements in large language models that show the inherent understanding of the problem by the model but its inability to search for the correct user intent based on the first greedily sampled decoder output. Hence it is necessary to use complex search strategy, which in the case of this paper happen to be another neural network. The results shown in the paper show consistent improvements in potential user experience.
strengths_and_weaknesses: The paper has some noteworthy strengths. They are the first to effectively use neural techniques to classify intent errors for general purpose programs and do not restrict themselves to simpler domains. This is an important finding as it can lead the community into building better models for detecting code errors, which can be used to judge model generated code as well as human written code. They set a strong ground work for detecting code errors using deep learning models and show the potential of such approach by experimenting over a diverse set of domains and diverse set of SOTA models. Finally they show the effectiveness of complementing such synthesis models with ranker systems, thereby improving their applicability in real world scenarios. 

The paper show the effectiveness of the approach in detecting bugs in model generated code thereby improving those systems. The presented results are to some extent expected, given that the model is augmenting the strong generative models with additional data of detecting intent errors. The system is also developed with the motivation of detecting errors early when they are generated by a generative model such to avoid unnecessary scenarios resulting from executing buggy codes. It is not exactly clear if running the system in a sandbox environment will completely remove any necessity of using this approach. Finally, the experiments were run solely with the purpose of improving the performance of code generation models, it will be interesting to find the effectiveness of the ranker in detecting bugs on unseen real code.


questions: 1) It will be great to understand the effectiveness of the ranker framework in detecting user bugs in held-out datasets. This will validate the robustness of the ranker framework. Are there any preliminary experiments in that direction?

2) How will the trained model on CodeBERT compare against using some baseline methods, some generative models were mentioned in the related literature. Can they be compared? If not, is it possible to fine-tune any autoregressive model on this task and compare against the current architecture?


limitations: The authors have adequately addressed the limitations. Some preliminary analysis on the limitations discussed can benefit the reader in understanding the robustness and current state of the error detection module developed by the authors.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work trains a model to predict errors in generated code and rerank the generations. Current models are often evaluated using the pass@k metric, which assumes that the unit tests can be run on k generations to rank them. They motivate their method with these arguments:
1) Not test dependent (it could fail on unseen tests)
2) No burden to create tests
3) No need to install dependencies
4) No security risk linked to execution

Their model is trained on a dataset generated from APPS (where unit tests are available), but the model learns to find errors from the code only. They evaluate it for reranking the outputs of language models on APPS (validation and test datasets), HumanEval and MBPP. 
They substantially improve the pass@1 scores of Codex, GPT-J, and GPT-NEO (1.3B and 125M params) on several benchmarks.

strengths_and_weaknesses: Overall, I found the method in this paper simple, but their motivation is compelling, it is well-written, and it is well executed. 
Strengths:
* Clear and well-written
* The motivation for this work is clear and compelling. Executing many generated snippets is often not doable in practice
* Even though their reranking is still far from perfect (as shown by the difference with the pass@5), they substantially improve the pass@1 for every model and dataset, showing the robustness of their method
* They give good insight into their model. The exec@1 improvement shows that execution errors are understandably easier to find.

Weaknesses:
* Limited technical novelty
questions: * Did you do any ablation showing how much the dataset size impacts your performance in this context?
* Does the code generation task contain the example unit tests? If so, the ranker could use the unit tests to determine the error. 


limitations: The datasets they tested their method contain mostly functions from programming contests. It is fine, since it would be more difficult to test it on real data when no unit tests are available. However, they mention real usecases in the introduction (such as use in VSCode) and I believe that their method may not transfer as well to real data, with expected input/outputs that are more difficult to infer. I would have liked to see a discussion about that.
review_title: null

Meta-review: metareview: The paper was well-received. The main idea is fairly simple, but the problem is important and the writing and empirical evaluation are solid. Based on the reviewers' advice, I am recommending acceptance. Please make sure to incorporate the reviewer feedback into the final version. 

confidence: Certain
Decision: Accept