Is In-Context Learning Sufficient for Instruction Following in LLMs?

Authors: Hao Zhao, Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, In-context Learning, Alignment

Abstract:
In-context learning (ICL) allows LLMs to learn from examples without changing their weights: this is a particularly promising capability for long-context LLMs that can potentially learn from many examples. Recently, Lin et al. (2024) proposed URIAL, a method using only three in-context examples to align base LLMs, achieving non-trivial instruction following performance. In this work, we show that, while effective, ICL alignment with URIAL still underperforms compared to instruction fine-tuning on established benchmarks such as MT-Bench and AlpacaEval 2.0 (LC), especially with more capable base LLMs. We then uncover the most relevant elements for successful in-context alignment, finding the crucial role of the decoding parameters. Based on these insights, we show that the approach of URIAL can indeed be improved by adding more, potentially carefully selected, high-quality demonstrations in context, getting closer to the performance of instruct models. Finally, we provide the first, to our knowledge, systematic comparison of ICL and instruction fine-tuning (IFT) for instruction following in the low data regime. Overall, our work advances the understanding of ICL as an alignment technique and its relationship to IFT.

Decision: accept (poster)
Reviewer scores: 8, 3, 6

Reviews and discussion:
<thread>
creation date:
1730148813138

thread_type:
official_review

<thread_object>
creator: reviewer_rg7y

<content>
summary:
This paper continues on URIAL work to deep dive into the comparison of ICL vs IFT. It revealed crucial insights to improve ICL performance, i.e. high quality examples and temperate as decoding parameters. On the flip side, it also shows the shortcoming of ICL, especially on subsequent turn in conversational benchmarks.

strengths:
This paper contributes valuable guidance to practitioners who often struggle between the choice of finetuning and prompting. The work is particularly useful for scenarios where obtaining training data is challenging. Also when all attentions are on instruct models, this paper shows that base models alone can be instructed and aligned, and made useful. This shall open the gate to lots of research projects too.

weaknesses:
More attribution works are needed to answer why the outcome is what it is. For example, the poor performance of the second-turn in MT benchmark definitely needs deeper dive. Is it because of the formatting or lack of examples? What caused the ICL to lose grip of the first turn? 

Considering that the authors work on small size open LLMs (except GPT-4), mechanistic study with logits or internal states shall definitely reveal more insights.

questions:
For prompts ending up with varying output performances, what are their KL divergence?

If you repeat your experiments on instruct models instead of base models, what would the results be?

How does ICL cope with long-context tasks (summarization, coding, etc.) where context window limit the extent of examples?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730583823123

thread_type:
official_review

<thread_object>
creator: reviewer_gajx

<content>
summary:
An interesting empirical study of the existing URIAL method that allows alignment via ICL. Unclear whether the contributions are significant.

strengths:
The paper conducts a comprehensive empirical analysis of the URAIL method for aligning pre-trained LLMs via in-context learning.
It considers different datasets, LLMs and test scenarios of relevance.
The figures are well done and show interesting trends. The paper is well structured and easy to follow.

weaknesses:
I am unsure about the exact contributions of this work. It appears mainly as an extended study of the URIAL method on (a) additional models, (b) additional datasets, and (c) more hyperparameters. A relevant contribution of the work is the greedy ICL prompt search, which shows significant performance improvements (i.e., higher performance with much fewer ICL prompts, i.e., with much fewer test-time compute requirements). However, the entire analysis is again primarily based on a single dataset (MT bench). The language sometimes could be more academic, consider e.g. “suffers from some heavy overfitting”.

questions:
- What do the authors see as the work's main contribution, apart from the empirical study of a previously introduced method?
- See other aspects mentioned above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730746903215

thread_type:
official_review

<thread_object>
creator: reviewer_zlsn

<content>
summary:
This paper studies whether in-context learning can be competitive with instruction fine-tuning methods on instruction following tasks. The empirical results ablate many different components of in-context learning methods and the dataset composition. The results are comprehensive and yield several potentially useful results (e.g., how decoding parameters impact the performance). The overall conclusion is that in-context learning is worse than and not as scalable as instruction fine-tuning.

strengths:
The experimental results are comprehensive. The problem studied in this paper is of interest to the deep learning community today.

weaknesses:
Writing is not clear. See below:
- The author should have a brief introduction to URIAL since it's heavily used in the later sections when explaining the experimental results. The current version refers to URIAL's components with the assumption that the readers know URIAL in detail. The current explanation like Line 120-126 is not enough. What do you mean by stylistic examples? What do you mean by rules? What do you mean by "begin with affirming the user queries?" I suggest adding some examples in the main paper to help the readers understand the idea.
- Similarly, what SkillMix datasets do has to be explained. 
- This paper presents many experimental findings in each section, but the messages of these findings were not highlighted. For example, Section 2.1 has the title "Systematic Evaluation of URIAL" and starts with a brief explanation of URIAL. When reading this section, I couldn't expect what you will be talking about in this section and I quickly get lost while reading a bunch of implementation details. Even after finishing the section, I couldn't get the takeaway message clearly. Section 2.2, in contrast, did a better job on explaining the results.

questions:
- The abbreviation URAIL is not defined.
- Line 139: If providing multi-turn examples improves performance at multi-turn conversation, why do the authors not study it in this paper?
- The results in Figure 2 are not representative in my opinion since the confidence bounds are overlapped largely. 
- Line 320: You say including examples from SkillMix is better than including examples from URIAL. Does URIAL come with a curated dataset as well? 
- Figure 3: I don't get it clearly. Is "Llama-3.1-8B-instruct" a model with instruction fine-tuning? Are these rest of ICL methods based on Llama-3.1-8B base models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732498819173

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response and Manuscript Updates

comment:
We thank all the reviewers for the thoughtful feedback, which has helped to improve our paper. Below, we outline the key updates to our manuscript, which address the following:
- **Multi-turn alignment via ICL**: We investigated the impact of multi-turn in-context examples on general instruction following performance of in-context aligned LLMs, taking into account formatting and the quantity of examples. We add Section 2.4 and Table 3 to present the results and our analysis.
- **Revisit our main findings on instruct models**: We reproduced three primary experiments, originally performed on base LLMs, with instruct models: (a) influence of the individual components of URIAL on instruct models (Appendix B.1, Table 7), (b) importance of question-answer matching of demonstrations for in-context alignment on instruct models (Appendix B.2, Table 8), and (c) many-shot in-context alignment on instruct models (Appendix B.3, Figure 10). We find that adding examples in the context consistently declines the instruction-following performance of instruct models, even when using correct URIAL prompt and increasing the number of high-quality examples.
- **KL-divergence between the base LLMs and its aligned counterparts via ICL or IFT**: We carefully examined the difference on probability distribution of next-token prediction between the base LLMs and aligned models by calculating KL-divergence based on models’ output logits. We summarize our results and present analysis in Appendix C and Figure 11.
- **Manuscript clarifications**: extended descriptions about the URIAL prompt (Section 2.1) and the SkillMix dataset (Section 3), summarized our main contributions at the end of Introduction, added a few examples of URIAL prompt and SkillMix data for illustration purposes (Figure 6, 7, 9 in Appendix).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper performs a comprehensive study of in-context alignment based on the method of URIAL. The empirical experiments find that in-context alignment generally underperforms compared to instruction fine-tuning. However, the paper then reported that using high-quality carefully selected in-context demonstrations can improve the alignment performance to reduce the gap with instruction fine-tuning.

The reviewers generally agree that the experiments are comprehensive and the conclusions from the experiments are practically useful and can inspire future works in this topic.

Multiple reviewers pointed out the lack of evaluations for multi-turn conversations in the original paper. The authors have added such experiments during rebuttal, and the results look promising (e.g., adding multi-turn examples indeed improve the performance). An important concern pointed out by Reviewer GajX is that the MT-Bench dataset, which is used in the majority of the experimetns in the paper, has only 80 data points, which may call into question the reliability of the findings. However, the authors' response look reasonable to me: the MT-Bench dataset is a widely used and reliable dataset; the authors used AlpacaEval 2.0 to further validate their findings and added more such experiments after rebuttal; the budget concern also looks reasonable to me.

Overall, I think the findings in this paper are insightful and useful for potential future works in this area. Therefore, acceptance is recommended.