Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method to generate prompts that solve textual tasks defined by Input/Output pairs through a Large Language Model.
This paradigm was introduced by Hononovic et al. (2022), as well as the 24 tasks used for evaluation. The main element of novelty here is the algorithm based on search and validation of the best prompt among a set of generated candidates, rather than the naive greedy selection used in Hononovic et al. (2022).

strength_and_weaknesses: Strengths
- The value of a good automatic prompt engineer is so evident that needs no explanation, this paper is definitely of interest to the community.
- The method is sound, and the discussion and results are valuable.
- The related works section does a good job in connecting a very new work built on top of LLMs to established research fields such as Program Synthesis.
- The paper is clear and well written, it is easy to understand the problem, the method, and the experiments.
Weaknesses
- The paper does not achieve the expectations set by the title and abstract. I was expecting challenging tasks where humans struggle to find a proper prompt (like leading Dall-e to generate a horse riding an astronaut, which was cited as a limitation of Dall-e until someone found the proper prompt) but the proposed method can automatically find it. Instead, most of the tasks can be solved with human-level accuracy using just the enumeration of a few input/output pairs as prompts (Fig. 4). These pairs would be available in any case since they are needed by the proposed method.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow, the writing is clear as well as the plots. The method and the experiments are sound, I appreciated the multiple runs to estimate the confidence of the accuracies. The exaggerated hype is a smudge in a research work of good quality. 
The broad topic of prompt engineering is very novel, while the specific idea of an automatic prompt engineer was first explored in the very recent Hononovic et al. (2022). I still consider the novelty of this work satisfactory, given the new method presented and the general novelty of the area of research.

summary_of_the_review: My recommendation is to accept this paper, the research is of interest and well performed. 
Still, I would recommend changing the title to be more descriptive and less pompous. 
Right now it suggests that finding the appropriate prompt to solve a task is no longer a problem since LLMs can do that to the same level as humans.I would say that we are not still there (e.g. https://twitter.com/goodside/status/1581805503897735168).

This work made me think of https://arxiv.org/abs/2201.10222, the *natural language program synthesis* problem reminded me the *Explanatory Learning* problem, while the APE is similar to the CRN approach there, maybe you could find this paper interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes using LLMs to generate instructions that will be used as prompts for solving tasks with LLMs.
Authors use a filtering + selection pipeline to choose top candidates according to various metrics. They compare their method in zero and few shot settings with 1) human-engineered prompts 2) LLM generated outputs without the filtering and section process.
They evaluate their approach on several NLP datasets and show it achieves similar results to 1) and improves over 2). 
They do a comprehensive analysis of their experiments that shed light on important components of their approach. 

strength_and_weaknesses: Strength:
- The paper is well-written and provide a lot of insights that can help other researchers who are working on similar area.
- The selection + filtering is a new addition to prior work done by [1] which improves the results significantly.
- Experiments are well-done and multiple ablations are performed to show effectiveness of various components of their approach.
-The idea is similar to automatic prompt tuning methods such as prefix-tuning, however, the resulting instructions are in natural language which is more interpretable than other approaches which generate embeddings. Also this approach doesn't require fine-tuning which is a plus.

Weakness:
- I think the tasks studied in this paper could be more challenging. Taking a look at Table1, most tasks require surface-level understanding of language syntax. Translation task is only done for single words. Semantic tasks from GLUE are at 90+% accuracy with simple transformer models. Few tasks here require complex reasoning, or deeper language understanding. I think those tasks require creating more nuanced instructions and seeing how this approach compares against humans would be very interesting. Until then. it's unclear to me how this approach generalizes beyond simple tasks.
- To build on my previous point, looking at figure 3, results on relatively harder tasks such as sentence similarity and membership show APE is doing worse than humans and vanilla generated prompts. The authors acknowledge this in the paper: "Additionally, tasks such as Membership and Second Letter are intrinsically challenging for the model, and APE consistently performs worse than humans." I think the paper can significantly benefit from further studies on tasks such as summarization, intent classification and slot filling, etc. that are more challenging.

[1] Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. Instruction induction: From few
examples to natural language task descriptions. arXiv preprint arXiv:2205.10782, 2022.
clarity,_quality,_novelty_and_reproducibility: Please refer to Strengths.
summary_of_the_review: Given the Strengths and Weaknesses mentioned above, I think this is a valuable paper and authors have done a lot of work to ensure the accuracy of their claims. However, I'm not quite convinced if this approach can scale to more challenging NLP tasks. I hope authors address this issue in the updated version of the paper. I can revisit my score given the new results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper propose a novel way of doing prompt engineering in large language models.
The model scores a pool of instructions generated using small set of demonstrations by a large language model.
For each input output pair (x,y), the goal is to find an instruction z so that f(z,x)=y, where f is a large language model.
z is then used to prompt a large language model at inference time with and without demonstrations.
Results on two benchmarks show that the proposed approach produces instructions that are as good as instructions written by humans.
The authors also do ablation experiments to understand what is the impact of language model size.

strength_and_weaknesses: The proposed idea is very interesting and this line of works trying to find the best language to query large language models is very trendy these days.
The reported results seem to be very good and for practitioners this approaches can be useful to improve performance on specific tasks.

The paper though is not very well written and there are a few parts that require some clarification.

The assumptions/intuitions in section 3.1 are not clear.
For example why "would like the model to predict the missing context before the demonstrations"?
What does it mean more versatile approach?
I understand that for every set of training pair there will be a possible different instruction, do you consolidate them?
Which one do you choose at inference time?
Real examples in the the Figures and task descriptions would simplify the reading a lot.

After the selection of instruction using different metrics, a set of final instruction are returned, how do these extraction are used? prompted all of them in an in-context learning scenario?

In section 4.1, please explain why Honovich et al. (2022) is the right benchmark for the proposed task, redirecting the reader to the original paper is not good writing, since it is not a well established benchmark.
In section 4.1 would be useful to have examples of input/output in the zero-shot and few-shot in context learning scenarios.
Why is the model only tested on the dataset released by Honovich et al. (2022), and not on more standard benchmarks like GLUE or big bench?

clarity,_quality,_novelty_and_reproducibility: The paper is not very clear, the idea is novel. 
The authors do not mention to release any code, so I am not sure about reproducibility.
summary_of_the_review: The idea is novel and interesting.
The paper is hard to read although the approach is quite straightforward. I suggest the authors to spend some time to make it more clear.

UPDATE: The authors made the paper more clear and added a set of experiments on more benchmarks that make the paper much stronger.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposed using large language models (LLM) to generate instruction and prompts. The prompts and instruction can then be used for in-context learning and leveraging other LLM to solve NLP tasks. In the proposed approach, a pool of instructions are first sampled by a LLM. These instructions are scored and selected as demonstration for other LLM to conduct in-context learning and solve tasks at inference time. The idea is novel and interesting. The authors also showed the effectiveness of proposed approach empirically. They compare the method in zero and few shot settings with 1) human-engineered prompts 2) LLM generated outputs without the scoring and selection process. Results on two benchmarks show that the proposed approach produces instructions that are as good as instructions written by humans and achieve performance comparable to 1) / better than 2). Comprehensive ablation and analysis were also conducted for readers to understand the importance of each component in proposed approach.

Strength of the paper:
1. The paper is clear and well written. It is easy to understand the problem, the method, and the experiments. It provides a lot of insights that can help other researchers who are working on similar area.
2. The proposed idea is very interesting and novel. Better leveraging LLM is a very popular and trendy area, and exploring methods to obtain better prompts is one of the core topic with obvious importance. This paper is denitely of interest to the community.
3. The method is sound. Experiments and ablations are well-designed. Results also show the effectiveness of proposed approach. Automatic prompt generation is relevant to prefix tuning, but offers more interpretable inference process. Performance comparable to human engineered prompt/demonstration also show the direction is promising. 

Weakness of the paper:
1. Several reviewers suspect if the proposed approach can achieve what the title and abstract claimed that LLM can generate prompt automatically with quality comparable to human engineering. Specifically, although experiments indicated that the proposed approach yields promising results in tasks requiring surface-level understanding of language syntax, the proposed approach consistently performs worse than humans in harder tasks such as sentence similarity and membership. The authors did a good job in providing additional discussion and experiment results. The additional results suggest proposed approach is generalizable to harder tasks. Performance in tasks where APE yields suboptimal performance previously is also improved due to better hyperparameter tuning. The additional explanation on the choice of titles, more experiment results supporting the efficacy of proposed approaches, and value propositions of APE compared to few-shot learning significantly mititgate reviewers' concerns.
2. Reviewers also raised some questions about the assumptions/intuitions adopted in the method design as well as the selection of benchmarks. The authors answered those questions properly in the discussion phase.
justification_for_why_not_higher_score: - The topic to better leverage LLMs is very popular and there are many related works in the area. Even in automatic prompt generation, prefix tuning and the work done by Honovich et al. (2022, Instruction induction: From few examples to natural language task descriptions.) can be viewed as pioneer in the direction, which slightly affects the novelty and impact of this work. 

justification_for_why_not_lower_score: - See the strength of the paper for more details. The paper is working on a trendy topic to better leverage LLMs and the topic is of wide spread interests in the commnuity. The authors also show promising results to generate prompt and demonstration automatically. Many researchers in the area can benefit from the works.
Decision: Accept: poster