LLMs Know What Makes Good In-Context Examples
Authors: Quanyu Long, Jianda Chen, Wenya Wang, Sinno Jialin Pan


Reviewer paper summaries: The paper presents
*  A unified framework for LLMs that allows them to self-select influential in-context examples to compose their contexts; self-rank candidates with different demonstration compositions; self-optimize the demonstration selection and ordering through reinforcement learning.
* A parameter efficient retrieval head that generates the optimized demonstration after training with rewards from LLM’s own preference is proposed.
* This framework enhances the applicability of LLMs to learn what contexts are favorable by the models themselves instead of using an external system.
Summary of strengths: * The proposed idea to build the selection retrieval process from self feedback inside the LLM itself is an intriguing idea and should help pave way for future exploration. 
* Adding representativeness and diversity numbers of the selected demonstrations from the proposed self-ICL helps qualitatively study the selection process.
* The experiments cover a good range of baselines and target task datasets.
Summary of weaknesses: Section 2
 * Please elaborate on how the N demonstrations from the corpus C are chosen for the retrieval head matrix.
 * For tasks like Text summarization (Reddit) and Story Generation (Roc Ending), its a bit unclear on how the {x, y} pairs are constructed. Please elaborate and for the readers.

Section 3
 * Please explain the rationale behind the selection of specifics retrieval methods and/or specifics datasets studied in Table 2 & 3, alternative extend the Table 2&3 to the full set of eval metrics like Table 1.
* Table 1: ComV has a significant improvement compared to prior work while on other datasets there is mostly a marginal improvement. This should be studied and explained in order to explain the results here.
* Figure 3: Please elaborate if any task dependency for the selection of `k` was explored. Most non text generation tasks in the experiments have a max of 3 labels, would help understand if there is a correlation between the choice of `k` and the target label space.
* Some analysis on the actual retrieved instances from self-ICL per task compared to other methods would help better understand the benefits (label coverage, diversity, etc)  of this selection process.
Comments, suggestions and typos: * Please add a note if the model code would be shared with the community.
Overall assessment scores: 2