Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper argues that programming puzzles and their machine verified solutions generated by a Language Model (LM)  can be used to improve the performance of the same LM. It presents a “data generation pipeline” where puzzles for a training set are used as a prompt for the LM to generate new puzzles that complete the prompt. A filtered set of the generated puzzles are used as parts of another prompt for the LM to generate solutions for those puzzles. The generated puzzle-solution pairs are then verified to create a larger training set. Evaluation results show that GPT Neo models fine-tuned with this larger dataset provide improved results measured using the Pass@k metric.

strength_and_weaknesses: Strength

- This work explores an interesting concept of using ‘self-play’-like strategy to improve language models for code generation and leverages existing work on programming puzzles to propose a method to demonstrate the effectiveness of the idea.
- Dataset of 1M synthetic puzzles shared by this work could potentially help future research along this direction.

Weakness

- The data generation pipeline does not specify when to stop generating puzzles or solutions in a single iteration. During evaluation 2 iterations are used for Neo-models and 1 for Codex (due to fine-tuning limitations). The overall approach seems adhoc. How to determine the quality of generated puzzles?
- It would be interesting to see if the performance improvements are tied to specific domains of problem.  This would also reflect on the generalizability of the proposed approach over problem instances.
- “In each of the n iterations, the same LM is used for both generation and fine-tuning” -  is the fine-tuned version replaces the base version in each iterations (i.e., is fine-tuned version always expected to be better in each iteration?)

clarity,_quality,_novelty_and_reproducibility: Clarity: The contributions of the work are clearly stated. The descriptions are easy to follow. 

Novelty: This paper explores the concept of 'self-play' with respect to code generation tasks and provide a new direction of leveraging programming puzzles as a way to synthesize new 'instructive' samples. 

Reproducibility: Evaluation uses publicly available dataset. The fine-tuned models and 1M datasets used for fine-tuning are not yet available. Details about the prompt used for data generation are presented.
summary_of_the_review: This work explores a potentially effective idea of improving language models for code. The proposed solution seems effective at improving models performance at python based programming puzzle task. However, the proposed data generation pipeline lacks  some details (e.g., determining number and quality of the generated samples, selection of fine-tuned LM in each iteration ). Also, the evaluation could be more informative if accuracy across the domain of problems are discussed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to improve a language model's (LMs) ability to generate programming puzzles. The primary contribution of the paper is algorithmic. Other contributions include the creation of an open-sourced large set of programming puzzles as well as experiments demonstrating the benefits of self-play (where a LM generates puzzles and then fine-tunes on them for improved test-set performance).
strength_and_weaknesses: Strengths
  * The paper investigates an interesting and important problem involving LMs. Techniques for improving LM code-gen performance without significant labeling effort will have broad impact.
  * The paper is well written. The main ideas (e.g., prompt setup) of the paper are described carefully with illustrative examples. I enjoyed reading the paper.
  * The main algorithmic ideas are intuitively clear. The paper is technically sound.
  * The experiments demonstrate the utility of the main algorithmic ideas (generation, verificationnn, fine-tuning) on the test set puzzles.

Weaknesses
  - A more detailed characterization of the program space representable by $f$ would be appreciated. It seems like f is a large subset (if not all) of Python but I'm not sure.
  - The paper could better identify the novel algorithmic contributions. For example, is the prompt generation idea novel?
  - While the paper does some analysis of the generated data via embeddings and UMAP, I would have liked to see a more detailed analysis and discussion of the generated puzzles. For example, how similar or different are they to the P3 train and test sets? Any overlaps?
  - Although Figure 7 makes it clear that fine-tuning has plateaued, it is not clear why. A deeper analysis of this in conjunction with the analysis of the generated puzzles would have been nice (but outside the scope of the paper). Overall, this seems like an important line of work as the paper states and leaves for future work.
clarity,_quality,_novelty_and_reproducibility: Clarity
  * I think the paper is very well written. The use of illustrative examples makes it easy to follow.

Quality
  * I think the paper is technically sound. However, some important aspects (e.g, program space of $f$) need clarification.

Novelty
  * The overall algorithmic novelty is a bit unclear to me. I think the paper could make this aspect a bit clearer.

Reproducibility
  * There seems to be sufficient algorithmic and experimental detail. I have no major concerns here. Code would have been nice to have.

summary_of_the_review: The paper tackles an interesting and important problem. Ideas are mostly clear and experiments show improvements. Some aspects (algorithmic novelty, program space of $f$) need clarification which are reflected in my scores (and open to revision depending on author feedback).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to perform data bootstrapping by using a performant language model generate prompts that specify puzzles and finetune models such data (after filtering for correctness). 
strength_and_weaknesses: Pros:
This method has some potential to augment data in order to improve model's general code understanding, generation and reasoning abilities. It is however similar to what Codex paper describes in their data generation process based on filtering with integration tests. However, this work focuses on puzzles.

Questions:
- The format of such puzzles are quite specific. That is, it expects f(.) and g(.) such that f(g(x)) returns True. This format can also be simplified since g() seems to take no argument and returns a value. The simplified format can be 'assert f(' as prompt, then ask the model to generate the actual values. Or is the f(g(x)) format used for something more general? The paper shows a general function f that takes multiple arguments in Figure 3. Are the extra arguments simply ways to take in values? 

Cons:
- The experiments can be a bit more thorough. It makes sense that if we finetune on such puzzles, the performance on a held-out set of puzzles will be improved. While it is valid to focus on such puzzle generation as one of the tasks for evaluation, a more important question is whether it translates to general coding abilities. 
- A better experiment setup is to perform data bootstrapping; that is, we can generate samples and verify, then finetune on the puzzles, then measure the performance on the general code generation benchmarks such as HumanEval or MBPP. 
- What of type guardrails are used to detect that the puzzles are not trivial? For example, if the function looks complicated but due to some reason it always goes to the branch that returns true, then any function g() would solve such puzzle.
clarity,_quality,_novelty_and_reproducibility: Clarify: The paper is quite well written
Quality: This is an intriguing idea that has a good potential. But more experiments are needed
Novelty: The data generation + filtering has been explored in the Codex paper. The puzzle format is explored in another work. This paper somewhat combines the idea of the two.
Reproducibility: Due to simplicity, I do not doubt that it should be quite reproducible.
summary_of_the_review: Overall, this is a nice method where the performance on the data bootstrapping method is quite expected. For completeness, the paper can push a bit further on studying the general code generation abilities after finetuning of puzzles, as described in the section above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors propose using LMs to generate programming problems and solutions. These synthetic problems and solutions then are used to fine tune LMs for program generation task. This approach leads to improved test accuracy in program generation task. 
strength_and_weaknesses: Strengths:

- A novel approach to improve LM-based program generation by first generating synthetic problems and solutions and then finetuning LMs on the generated problems.
- Highly improved test results
- Ablation studies of model and interpreter distillation

Weaknesses:

- Authors cannot finetune Codex and therefore there are no results for using the authors' approach to improve state of the art results for the problem. See Figure 12 for the gap of the Neo models even with fine tuning vs Codex models. It is possible (likely?) that Codex models will benefit from finetuning using the additional synthetic problems/solutions, but we don't know for sure. 
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written.
I believe the approach is novel.
Since authors plan to release the dataset of synthetic puzzles and solutions and based on the descriptions of their work in the paper, the results should be reproducible.

---------------------------------------------------------------
ICLR review form does not provide a section for comments/questions about the paper. I am going to post them here:
- It would be good to have some discussion of how this work could transfer to program generation from English prompts.
- There is related work on generating buggy programs and fixes for them in the automated program debugging area. I wonder if this work could also generate dataset for buggy/incorrect puzzle solutions and correct puzzle solutions that could be used in automated program debugging. Also I wonder if there is a way to introduce the automated program debugger into the loop to improve either finetuning or prompts for the correct puzzle solution generation. Possible areas to explore in the future work.

summary_of_the_review: I believe the work presented here is useful for improving language model ability to generate programs. It seems to be novel, and it achieves marked improvements when used to fine tune LMs. There is a reasonable expectation that this research could improve state of the art results and also be applied in further advancing language model-based program generation. I recommend accepting this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents an approach to improve language model’s performance on the task of solving programming puzzles, without the need of obtaining more labeled data with significant human cost.  The proposed approach uses language models (LMs) to generate more programming puzzles, verify and filter them, then use LMs to generate solutions and verify them, then fine-tune the LMs on the generated puzzle-solution data to get improved performance.

The paper is well-written, the idea is intuitive, and the empirical results are good, with extensive ablations.  Most reviewers support acceptance.

I would encourage the authors to include some additional results from the discussion with reviewers into the paper, for example the performance vs size of the generated dataset seems like a very informative study that could be included in the main paper / appendix.

Go beyond programming puzzles is also worth thinking about, as multiple reviewers raised this directly or indirectly.
justification_for_why_not_higher_score: The proposed problem setting is a bit specific to the programming puzzle setting, and therefore not generally applicable to other settings, making its immediate impact a bit limited.
justification_for_why_not_lower_score: The idea presented in the paper is intuitive, and works with good results.  This paper is worth publishing.
Decision: Accept: poster