Is In-Context Learning Sufficient for Instruction Following in LLMs?
Authors: Hao Zhao
Maksym Andriushchenko
Francesco Croce
Nicolas Flammarion
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
In-context Learning
Alignment
Primary area: foundation or frontier models, including LLMs

Abstract: In-context learning (ICL) allows LLMs to learn from examples without changing their weights: this is a particularly promising capability for long-context LLMs that can potentially learn from many examples. Recently, Lin et al. (2024) proposed URIAL, a method using only three in-context examples to align base LLMs, achieving non-trivial instruction following performance. In this work, we show that, while effective, ICL alignment with URIAL still underperforms compared to instruction fine-tuning on established benchmarks such as MT-Bench and AlpacaEval 2.0 (LC), especially with more capable base LLMs. We then uncover the most relevant elements for successful in-context alignment, finding the crucial role of the decoding parameters. Based on these insights, we show that the approach of URIAL can indeed be improved by adding more, potentially carefully selected, high-quality demonstrations in context, getting closer to the performance of instruct models. Finally, we provide the first, to our knowledge, systematic comparison of ICL and instruction fine-tuning (IFT) for instruction following in the low data regime. Overall, our work advances the understanding of ICL as an alignment technique and its relationship to IFT.

Published as a conference paper at ICLR 2025
Is IN-CONTEXT LEARNING SUFFICIENT FOR INSTRUC- TION FOLLOWING IN LLMs?
Hao Zhao EPFL
Maksym Andriushchenko
Francesco Croce EPFL
Nicolas Flammarion EPFL
EPFL
ABSTRACT
In-context learning (ICL) allows LLMs to learn from examples without changing their weights: this is particularly promising capability for long-context LLMs that can potentially lear from many examples. Recently, Lin et (2024) pro- posed URIAL, method using only three in-context examples to align base LLMs, achieving non-trivial instruction following performance. In this work, we show that, while effective, ICI. alignment with URIAL still underperforms compared to instruction fine-tuming fine-tuning on the established benchmark MT-Bench, especially with more capable base LLMs. We then uncover the most relevant elements for success- ful in-context alignment, finding the crucial role of the decoding parameters. Based these insights, show that the approach of URIAL by adding high-quality, possibly carefully selected via greedy search, demonstra- tions in context, getting closer to the performance of instruct models. Finally, we provide the first, to our knowledge, systematic comparison of ICL and instruction fine-tuning (IFT) for instruction following in the low data regime, where ICL can be a viable alternative to IFT. Overall, our work advances the understanding of ICI. as an alignment technique and its relationship to IFT. We provide our code https: // //g thub. com wtw-tmmllllefceeem gnment..
indeed be improved
INTRODUCTION
The large-scale pre-training phase allows Large Language Models (LLMs) to acquire extensive knowledge and capabilities (Bubeck al., 2023). However, these base models struggle to follow instructions directly from prompts, necessitating further fine-tuning to be used as conversational models. Traditional alignment methods include Supervised Tine-Tuning (SFT) on instruction datasets (IFT, Wei et al., 2021; Chung et al., 2022) or preference data (DPO, KTO, IPO, and ORPO, Rafailov al., 2023; Ethayarajh et al., 2024; Azar et al., 2024; Hong al., 2024), and reinforcement learning (RLHF and RLAIE, Ouyang al., 2022; Bai et al., 2022). These approaches typically involve multiple rounds of refinement of the instructed model (Touvron et al., 2023), i.e. high computational cost, and need large amount of annotated data like human preferences, which might be difficult to collect. However, a line of work (Taori et al., 2023; Chen et al., 2023; Zhou et al., 2023; Lee et al., 2023; Zhao small amount of high quality instruction-
al., 2024; Kaur et al., 2024) has suggested that IFT following examples, even only 1000, can be sufficient to match the performance more complex alignment mechanisms. In particular, Zhou et al. (2023) introduced the Superficial Alignment Hypothesis, stating that LLMs acquire all their capabilities during pre-training, and fine-tuning only allows the models better access such knowledge when interacting with users (Gudibande al., 2023; Duan et al., 2023). Using such small instruction datasets for IFT has the clear advantage of significantly reducing the cost of model aligmment.
Inspired by Brown et al. (2020) who showed that LLMs can learn from demonstrations provided as part of the input-a concept known as in-context learning (ICL)-Lin et al. (2024) have recently studied the feasibility of in-context alignment (Han, 2023; Li et al., 2023b). They found that including merely three carefully selected cvestion-ansser pairs in the prompt is sufficient to make base models follow instructions and interact with users at a similar level to mnstruction-fine-ture their own benchmark. This fact strongly validates the idea that alignment can be achieved at low computational cost and with few examples, as well the Superficial Alignment Hypothesis, it does not even require to modify the model weights. Moreover, in-context alignment is especially appealing since it opens up the possibility of customizing base models via ICI, i.e., without any fine-tuning,
models on

---

Published as a conference paper at ICLR 2025
which is potentially game-changing due to its simplicity and flexibility. While the results of Lin et al. (2024) are promising, they are restricted to a small number of base models and their own instruction following dataset.
Analysis of in-context alignment. In this work, we extend the evaluation of the URIAL (the abbreviation of Untuned LLMs with Restyled In-context ALign- ment) prompt strategy proposed by Lin al. (2024) across several base mod- els. including GPT-4-Base, and on estab- lished instruction following benchmark MT--Bench (Zheng et al., 2023), see Ta- ble 1. First, we show that, although URIAL achieves reasonable performance, still lags behind instruction-fine tindeeene models. Then, to better understand the success but also weaknesses of in-context alignment, we analyze which are the most relevant ingredients in URIAL. We find that the decoding parameters in the LLMs generation (temperature, sampling scheme, etc.) may crucially influence the performance, and the optimal configura- tion allows even base models achieve good scores on MT-Bench. Moreover, we show that randomly sampled triplets of examples from the highly curated in- struction dataset SkillMix (Kaur et al., 2024) achieve similar, ter, results than URIAL when used for in-context alignment.
Table 1: Systematic comparison of URIAL to aligned models on MT-BBench across different base LLMs. For several re- cent model families, following tasks of the base L.MMs plus URIAL. (i.c. in-context alignment) to that the instruct models, fine-tuned with sophis- ticated techniques like supervised instruction fine-tuning and RLHE. In most cases, the fine-tuned models outperform URIAL. * denotes the result taken from the URIAL GitHub repository.
compare the performance on instruction
Model
1st-turn 2nd-turn Average
Llama-2-77 2-BB + URIAL
5.75
3.91
4.83
-lama-2-7B-Instenn
7.14
5.91
6.53
Llama-2-70B + URIAL
7.61
6.61
7.11
Llama-2-7088 Jaaaa--70B-Instrucc
7.37
7.03
7.20
Llama-3-8B + URIAL
6.84
4.65
5.75
-lam--3-8-Insttuu
8.29
7.42
7.86
Llama-3-70B + URIAL
7.71
5.09
6.40
Llama-3-70B-Instruct
8.96
8.51
8.74
Llama-3, 3.1ma-311-BB + URIAL
6.95
5.31
6.13
8.27
7.73
8.00
Mistral Mistral-7B-vee +0.1+ 7B-00.1 URIAL
7.49
5.86
6.67
Mistral 7B 1B--nnttuct--O..
7.31
6.39
6.85
Mistral- 7B- URIAL
6.99
5.55
6.27
Misttal-7BB-Instuut--00..
8.06
7.21
7.64
bet-
Exxtral-x22B-vv). + URIAL
8.28
7.14
7.71
Mixtral-8x22B-44 www.2//nttttttttt.........
8.78
8.25
8.52
Many-shot in-context alignment. Then, test various strategies improve in-context alignment, leveraging recent
GPT-4-Base URIAL GPT-4 (March 2023) *
7.96
5.04
6.50
8.96
9.03
8.99
models with extensive context windows which allow for longer in-context prompts. In particular, we study the effect of many-shot in-context learning by adding high-quality demonstrations from existing instruction datasets. Unlike what suggested by Lin al. (2024), this approach can improve upon URIAL, but only when using high-quality examples, as those from SkillMix. However, it is still not sufficient to fully close the gap with aligned LLMs, as the performance plateaus after 10-30 in-context examples. This behavior is in contrast to many-shot ICL for tasks like summarization (Narayan et al., 2018), translation (Costa-jussa, et al., 2022), or classification (Li et al., 2024), where providing many examples is clearly beneficial (Agarwal et al., 2024; Bertsch et al., 2024). We further test a simple greedy algorithm to select the in-context examples which optimize the MT-Bench score. This selection scheme outperforms, with 3 additional demonstrations, the many-shot approach with random samples, and allows to further reduce the distance from in-context aligned models to fine-tuned models.
ICL vs IFT for alignment. While our experiments suggest that ICL cannot match the instruction- following performance of models aligned through costly fine-tuning, possibly using preference data, it remains an open question whether ICL can compete with or outperform IFT in the low-data regime. We provide an extensive evaluation on multiple LLMs and datasets of both approaches when varying the question- question-answer answer dataset size between 3 and 4000 examples. We observe that, with high high-quality data, ICL and IFT achieve almost identical sst-turn MT-Bench score. Surprisingly, in the 2nd-turn score, IFT clearly outperforms ICI, with ICL performing even worse than the base model. This suggests that ICL overfits to the style the examples shown context.
We received access the base GPT-4 model via the OpenAl Researcher Access Program.
2

---

Published as a conference paper at ICLR 2025
Overall, these results, complemented by several ablation studies, provide nuanced picture of ICL as an alignment technique compared to previous works. Moreover, our comparison of ICL to IFT when using the same data bridges gap in the literature about understanding these orthogonal approaches for adapting pre-trained LLMs into conversational models. In summary, our contributions follows,
Analysis of In-Conter: Aligmment (Sec. 2): We systematically evaluate URIAL on a broader set of base LLMs, including GPT-4-Base, with established instruction-follwwie benchmark. Our results indicate that in-context alignment with URIAL still underperforms instruction fine-tuning and more sophisticated alignment methods, and a proper decoding scheme is the crucial ingredient behind the empirical success of URIAL.
Scaling In-Context Alignment (Sec. 3): We find that many-shot ICL with high-quality examples and simple greedy algorithm can reduce the gap between in-context aligned models fine-tuned models.
First systematic comparison of ICL vs IFT for instruction following (Sec. 4): We show that ICL and IFT with the same number of examples are roughly equivalent for single-turn conversations in the low-data regime. However, IFT generalizes substantially better than ICL when more examples are present, especially for multi-turn conversations.
UNCOVERING THE LIMITS OF ICL FOR INSTRUCTION FOLLOWING
In the following, we provide in-depth analysis which aims at (1) systematically comparing the performance base models plus URIAL to that of aligned models (Sec. 2.1). (2) understanding which components of URIAL (and in-context alignment in general) most important for its effectiveness (Sec. 2.2), (3) testing the influence of dnestion-answen format in our task (Sec. 2.3).
2.1 SYSTEMATIC EVALUATION OF URIAL
Background URIAL. Lin al. (2024) propose to prompt with few three constant stylistic examples, which follow a curated answering structure that human users are familiar with, and a carefully designed of rules, highlighting the format and principles AI assistant needs to follow strictly when generating answers, to align base LLMs with ICL. We present the complete URIAL prompt in Fig. The highly curated examples always begin with affirming the user query by saying something like "Hi, I'm happy to help you.", then proceed to answer the query by enumerating necessary items and steps, and conclude with an engaging summary. The set of rules first elucidate the scenario and format of the subsequent interaction between the human user and the Al assistant, then outline multiple qualitative aspects, including helpfulness, honesty, and harmlessness, which the base LLMs should adhere to when answering queries.
Experimental setup. We compare the performance several base models with the URIAL in-context prompt that of their instruction fine-tuned versions. Table shows the results MT-Bench (Zheng et al., 2023), one of the most popular benchmarks for instruction following ability of LLMs. We note that Lin al. (2024) originally tested URIAL. on their dataset, in which only 8% of the examples are obtained from MT-Bench. We compare several LLMs of different sizes and capabilities, ranging from Llama-2-7B (Touvron et al., 2023) to the base GPT-4 model (OpenAI, 2023). For simplicity, we consider the official instruction-tuned LLMs. i.e., provided from the same source of the base models, even if there exist third-party fine-tuned versions which can achieve higher scores.
Results. In Table we observe that base models with URIAL achieve competitive performance on the 1st-turn score, but cannot match that of their instruction fine-tuned counterparts in all cases except for 1lama-2-70B and Mistral-7B-vol
(both originally included in Lin et al. (2024), unlike most others). Conversely, the 2nd-turn score with URIAL is significantly worse than for instruction-tuned L.MMs: we hypothesize that this is because no multi-turn demonstrations are given in context. Indeed, Zhou et al. (2023) show that, even in IFT, having a few multi-turn training examples significantly improve the performance of the model on multi-turn conversations. Because of this fact, in the rest of the paper, we mainly focus on the 1st-turn score and single-round conversations, and track of 2nd-turn performance for completeness. To fill in the gap in the influence of multi-tum conversations, study multi-turn alignment via ICL in Sec. 2.4. Finally, we detail the breakdown of the results over categories in Table 11 (see App. E.4 for details).
3

---

Published as a conference paper at ICLR 2025
6.39
6.01
5.10
5.51
7.34
7.11
7.12
7.26
7,40
7.78
7.62
7.97
Top-p
Tep-p 0.
7.10
7.19
7.72
6.61
6.12
5,50
1.78
1.0
7,00
6.61
7.34
2.39
7.62
7.83
7.56
7,18
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.3
0.5
Tomperature
Temporature
Tomperatero
(a) Basc model
(b) Base model with URIAL
(c) Instruct model
Figure Effect decoding parameters on the 1st-turn MT-Bench scores across models. We vary temperature and top-p, with a fixed repetition penalty of 1.15 as used in the URIAL codebase. The heatmaps show that the answering quality of the base model (Mistral-7B-v0.2) with and without URIAL is sensitive to the decoding schemes. Conversely, the performance of the instruct model is robust varying the decoding parameters. Surprisingly, with proper decoding parameters, base model alone already capable following instructions. See the complete results in App. E.3.
2.2
WHAT MATTERS FOR IN-CONTEXT ALIGNMENT VIA URIAL?
Decoding parameters. When computing the performance of URIAL,, Lin et al. (2024) use decoding parameters (temperature 0, top-p 1, repetition penalty 1.15) which differ from the default ones in MT-Bench (where temperature depends on the topic of the question, top-p repetition penalty = 1). To clarify the influence these parameters that were not explicitly discussed in Lin et al. (2024), we compute the performance of base LLM, base LLM with URIAL, and fine-tuned LLM when varying decoding configurations. In Fig. we fix the repetition penalty to 1.15, and vary temperature and top-p, for the Mistral-7B-v0.2 models. Surprisingly, we notice that with the decoding parameters URIAL (temperature = 0, top-p = 1) even the base model without any in-context example achieves reasonable MT-Bench score (6.61 vs 7.00 of URIAL). Moreover, the temperature value appears to have the most influence of the performance of the base model. With URIAL, almost all configurations provide very similar results, with the exception of temperature top-p = 1 (possibly because allows the sampling of generated tokens from the tail of the token distribution, thus producing low-quality text). Finally, the aligned model performs similarly across all decoding schemes, with slightly better results than URIAL. These results suggest that (1) the decoding parameters are an overlooked factor contributing to the success of in-context alignment, and (2) fine-tuning adjusts the sampling distribution of language models so that even with high-varianc variance decoding configurations the generated text preserves high quality. We provide additional analysis for no repetition penalty (i.e. 1) in Fig. 13 and -lama-3.1-88 in Fig. 14 in App. E.3: these extensive experiments further validate, with different base LLMS and settings, that the decoding scheme is crucial factor for the instruction-for following behavior of base models, and also affects, but to lower degree, in-context alignment. Instead, the fine-tuned models are robust to variations in the decoding parameters. Then, for the rest of the experiments we fix the decoding scheme of URIAL.
In-context prompt. Next, we want to disentangle the influence of the various elements of URIAL on instruction following performance. In Fig. 2 we track the MT-Bench scores when putting in- context all possible subsets of the three demonstrations of URIAL (i.e. from 0 to 3 examples are used). Moreover, for each case we test either adding not the part of the prompt including the rules to follow. First, we find that neither sst-turn nor 2nd-turn MT-Bench scores are significantly influenced by the addition of the set of rules, turn highlighting the importance of the examples. Focusing on Ist-turn score (left plot in Fig. 2), we see that increasing the number of in-context demonstration progressively improves the performance of the base model (Mistral-7B-.e case): this observation motivates us to add further high-quality in-context demonstrations, see Sec. 3. Finally, the 2nd-turn has the opposite trend, getting degraded by more (single-round) demonstrations.
this
2.3 IMPORTANCE OF QUESTION-ANSWE MATCHING
Surprisingly, Min al. (2022) showed that using demonstrations with random labels does not significantly impair the results of ICL on classification and multiple choice tasks. We conduct a similar study for instruction following, Table 2. Denoting {(X,, Y:) the set of in-context examples,
4

---

Published as a conference paper at ICLR 2025
1st sum MTT-Beneh Score Score
7.2
2nd surn MT-Bench Score
5.8
Examples Only
Rule Examples
5.6
7.0
5.4
6.6
5.2
Examples Only Rule Example
6.4
0
2
2
Number Examppee
Number Examples
Figure 2: Influence of the individual components of URIAL. We report 1st and 2nd turn MT-Bench score for We test each configuration with ("Rule
every subset the three demonstrations of URIAL with Mistral-7B- + Examples") and without ("Examples-Our Onllll"ss-On"))ee the URIAL set of rules in the in-contexi prompt. We observe a clear increasing trend in st-turn score with more examples, hut decrease in the 2nd-turn performance. The set of rules does not seem to influence the results. The randomness for 0 and 3 examples is caused by the small fluctuations in the score of the GPT-4 judge.
Table 2: Importance of question-answer matching of demonstrations for in-context alignment. We report the 1st-turn MT-Bench score of Mistral-7B-vO. and Llama-3.1-88 1-8B when varying the structure of the in-context examples {(X,, k., where X; is the query and Y: is the corresponding ground-truth answer.
URIAL (3 examples)
URIAL + Greedy Search (6 examples)
In-context prompt
Mistral-7B-v0.22
Llama-3.1-90
Mistral-7B-v0.2
Llama-3.1-8B
demonstrations
6.52
6.25
6.52
6,25
only (no Y) Y only (no X) circular shift of Y in-domain random Y
5.90
4.48
5.53
5.10
6.94
6.83
7.02
6.98
5.04
5.69
5.59
2.13
6.24
6.26
4.63
5.49
out-of-domain random Y
4.13
3.73
1.50
4.36
correct Y
6.99
6.95
7.43
7.81
with X; the query and Y; the corresponding ground-truth answer, we test several configurations on two sets of {(x, };);; the 3 URIAL examples and the URIAL examples with the 3 examples found by our greedy search from Sec. 3.2 (to check the effect of increasing the number of demonstrations).
First, we do not use any demonstration, i.c., the original base models: with the decoding parameters found in the previous section, this already achieves reasonable results. Surprisingly, providing the questions without the answers (second row in Table 2) degrades the performance, while the opposite, answers only, is effective (0.4-0.7 higher scores than the base model). Next, we permute the answers Y,s with a circular shift of one position, so that all correct answers are still contained in the prompt but matched with the wrong question: this significantly degrades the performance, especially with more examples (URIAL + Greedy Search), e.g. Llama-3. 1--BB attains score of only 2.13 (note that the minimum score is 1). Also, for each question, we sample other instructions in the same category (in-domain): e.g., a question about coding is paired with an answer from different coding question, ensuring that, while the content may be incorrect, the style remains appropriate. Although worse than the original one, this configuration achieves decent scores. Finally, sample answers from instructions belonging different out-of- -o-dmmnnin categories, that even their style does not match what expected for each question: this leads to the worst performance.
answer from those provided for
These results show that not all the conclusions from Min et al. (2022) apply ICL used for instruction following. Most importantly, using answers with correct content and, especially, correct style is crucial for the success of ICL. This property becomes even more evident when increasing the number of examples (to 6 instead of 3), and motivates us to use highly-curater instruction dataset like SkillMix in experiments in the next sections. Finally, this result suggests that ICI. for instruction following works differently compared to other less open-ended tasks.
2.4 MULTI-TURN ALIGNMENT VIA ICI.
In IFT, Zhou et al. (2023) find that adding a few multi-tune -ulti-turmmu examples to the training data significantly improves the model's performance multi-turn conversations. URIAL (Lin et al., 2024) shows that
5

---

Published as a conference paper at ICLR 2025
Table 3: In-context alignment with multi-turn examples. Second-turn answers for all methods generated using GPT-4o. Extending URIAL with two-turn examples improves 2nd-turn MT-Bench score. This shows that in-context alignment is not limited to single-turn performance.
Mistral-7B-v0.2.
Llama-3.1 3.1--B
In-context prompt
Tur
Turn 2
Avg
Turn
Turn 2 Avg
Base model (no prompt)
6.52
5.68
6.10
6.25
5.17
5.71
Original URIAL (singlc-tur) Two-turn URIAL
6.99
5.55
6.27
6.95
5.31
6.13
7.18
5.68
6.43
7.31
6.44
6.87
URIAL 3 two-turn SkillMix examples (no separation tag)
7.46
5.88
6.67
7.21
6.43
6.82
URIAL +3 two-turn SkillMix examples (separation tag)
7.20
6.31
6.76
7.22
6.71
6.97
URIAL 6 two-turn SkillMix examples (no separation tag)
7.18
5.89
6.53
7.14
6.26
6.70
URIAL 6 two-turn SkilIMix examples (separation tag)
7.24
5.78
6.51
6.89
6.33
6.61
the base LLMs can handle multi-turn conversations even with only having single-turn demonstrations in the context, but it does not quantify the performance nor rigorously study the role of examples that have more than one turn of conversation. To bridge this research gap, we further test how the presence of multi-turn examples in the context influences the intructioo-follwwerr performance of in-context aligned models.
We experiment using two open source LLMs, Mistral-7B-v0.2 and Llama- 3.1-8B. and evaluate on MT-Bench. To ensure the multi-turn examples we test have high quality, we select single-turn examples randomly from SkillMix as the basis, the first turn, and then prompt GPT-4o to ask follow-up question after seeing the first-turn conversation and generate by using an instruction "Here is a single-turn conversation between a user and an assistant, please ask follow-up question and provide an answer.". We examine the influence of two-turn examples from two perspectives, the number of two-turn examples (3 vs 6). and the format, with or without separation tags like "Here 3 pungle--urm/multi-uuu examples:".
Experimental setup.
answer
Results. In Table 3, we observe that extending 3 URIAL single-turn examples to 2-turn examples consistently improves the evaluation scores on both turns, showing that in-context alignment solely with multi-turn examples can take care of single-turn conversations. The presence of two-turn examples in the context always leads to improved 2nd-turn MT-Bench scores, for example, an increase from 5.31 to 6.43 after adding 3 two-turn SkillMix examples on top of URIAL on Llama- 3.1-8B. Another finding from our experiments is that the format of placing demonstrations plays a pivotal role in multi-turn interactions. In particular, adding a separation tag to arrange demonstrations in two groups improves 2nd-turn scores, from 5.88 to 6.31 on Mistral-7B-v0.2 and from 6.43 to 6.71 on Llama-3.1-8P although sometimes it hurts the sst-turn performance. However, we find that increasing the number of two-turn examples in the context does not lead to more benefits with or without better format.
3 MANY-SHOT IN-CONTEXT LEARNING FOR INSTRUCTION FOLLOWING
Sec. 2.1 indicate that URIAL alone is not able, in most cases, reach the performance of instruct models. In the following, we explore whether we can close this gap. We focus on Mistral-7B-vdl (Jiang al., 2023) and Alama-3.1-8B (Dubey et al., 2024) since (a) both URIAL and the aligned model achieve competitive performance in Table and (b) these base models have context windows of 32k and 128k respectively, which can many examples (around 50 for Misvral-7B-v0.2 and more than 200 for 3.1ma-3.1-8B) Details the experimental setup are in App. A.
Data. We adopt the SkillMix (Kaur et al., 2024) dataset consisting 4,000 examples as the source of high-quality examples for our many-shot in-context alignment experiments. The query-answer paired data generation process of SkillMix explicitly combines diversity and difficulty through random skill combinations, where the set of core skills for instruction fooloowing..oooignooooooooo such developing training programs and digital proficiency, are extracted by prompting a powerful LLM like GPT-4-Turbo. We present more details of SkillMix, including some examples (Fig. 6 and Fig. 7), in App. A.

---

Published as a conference paper at ICLR 2025
Mistral- 7B-vt-2-Inntrure
Random (no URIAT.)
Greedy Search
1st-turn MT-Bench Score
2nd-tumn MT-Bench Score
8.0
7.5
6
7.0
6.5
30
50
3
10
20
50
Number Examples
Number Examples
(a) Mistral-7B-vO.2
Llama-3
Random URIAL)
ICL Random
Greedy Search
1st-turn MT-Bench Score
2nd And-turm MT-Bench Score
8.0
7.5
7.0
6.5
0
3
10 50 100
200
0
10
50 100 200
Number of Examples
Number of Examples
(b) Llama-3.1-8B
Figure 3: Scaling the number of demonstrations for alignment with ICL Mistral-7B-v0.2 and Llama- 3.1-8B. We measure the aligument performance different settings using the MT-Bench score. ICL with demonstrations quickly saturates and does not bridge the performance gap between the model its aligned counterpart. In particular, the ICL alignment performance random examples from the high-quality SkillMix dataset surpasses that of 3 examples from URIAL.
3.1 SCALING THE NUMBER OF IN-CONTEXT DEMONSTRATIONS
Given the success of many-shot ICL (Agarwal et al., 2024), we test the effect of increasing the number of in-context demonstrations. We sample random examples from the SkillMix dataset, since it contains high-quality question-answe answer pairs. We consider two scenarios: (1) all demonstrations are randomly chosen from SkillMix (the set of rules from URIAL is kept), and (2) we add the new demonstrations top of the URIAL examples. In Fig. 3, report the results MT-Bencc when varying the number of demonstrations, We repeat sampling with 5 random seeds and show mean and standard deviation over the corresponding results. We observe that for both base LLMs, the 1st-turn score increases with 3, 10 and 20 examples, but then plateaus without clear benefits from scaling up beyond demonstrations (for 1.ama-3.11888 the trend becomes even slightly negative with more than 100 examples). Thus, we find that adding up to 30 high-quality examples from SkillMix can improve alignment via ICL. This result contrasts with the findings of Lin al. (2024), which showed that URIAL performed better with 3 demonstrations than with 8.
Next, we observe that in-context alignment does not improve the second-turn score. In fact, adding more examples continues to decrease performance, even falling below that of the base models. This behavior is likely due to the presence of only single-turn examples in the prompt, causing the LLM to respond in the same style without adapting that, unlike the setting of Agarwal et al. (2024); Bertsch et al. (2024), simply scaling the number of ICL examples is not sufficient to consistently improve the instruction following performance.
complex conversations, Overall, these results show
Finally, we notice that on Llama-3, 3.BB using three examples from SkillMix attain, on average, better performance than using the three examples of URTAL. Overall, there is significant difference
7

---

Published as a conference paper at ICLR 2025
25
25
25
SURAL STRIAE-16
UURAAL SITRIAL -Rott 4b URIAL-BOI 4br-3
LUBAAL
20
20
Count
Count
15
10
10
5.75
6.00
6.25
50
6.75
5.2
6.4
6.6
6.8
6.2
6.4
6.5
6.8
Lst cum MT Bench Score
1st-tum MT Beach Score
150 tumn MT Bench Soors
(a) Scarch for 4th example
(b) Search for 5th example
(c) Search for 6th example
Figure 4: The distribution of the 1st-turn MT-Bench score (GPT-4-Turbo as judge) on Llama-3.1-8B
obtained by adding multiple instructions from SkillMix (Kaur et al., 2024) as a 4th (a), 5th (b), 6th (c) demonstration to URIAL. The dashed lines of various colors refer to the 1st-turn MT-Bench score of the obtained searching results. majority of the 4th examples contribute positively to the model's instruction- following performance, but the improvement quickly diminishes when running the greedy search for 5th and 6th demonstrations.
Table 4: Improved in-context (IC) alignment by adding URIAL prompt the demonstrations found via greedy search. We find the optimal IC prompts in incremental way by selecting query-respons, pairs from the high-quality SkillMix datasct. We report the Ist-turm score on MT-Bench and the bangh-coontoolle (LC) win rates on AlpacaEval 2.0 when using different IC prompts.
Model
Mistral-7B-v0.2
Llama-3.1
MT-Bench (Ist)
AlpacaEval 2.0
MT-Bench (1st)
AlpacaEval 2.0
URIAL (3 examples)
7.00
8.22
6.95
7.28
URIAL greedy search (1 ex.)
7.52
7.53
7.61
8.61
URIAL greedy search (2 ex.)
7.47
7.78
7.77
8.16
URIAL + greedy search (3 ex.)
7.43
8.55
7.81
8.19
in the scaling behavior between using or not using the URIAL demonstrations. Thus, we conclude that the success of in-context alignment is only partially dependent on the demonstrations themselves, provided they are of sufficient quality (see also Sec. 4). We also include the results of scaling experiments (Sec. 1) conducted on AlpacaEval 2.0, which consists of 805 examples.
3.2 GREEDY SEARCH FOR EFFECTIVE DEMONSTRATIONS
Given the limited success of adding random demonstrations to URIAL, we try to greedily maximize the MT-Bench score by testing 100 high quality instructions from SkillMix as the 4th additional example to URIAL. For each resulting ICL prompt, we compute the MT-Bench score with GPT- 4-Turbo as judge instead of GPT-4 (used in MT-Bench) as the former is faster, cheaper, and helps mitigate potential overfitting the benchmark score. We then repeat this procedure sequentially to find a 5th and a 6th demonstration, restricting the search space at each round to only instructions leading to high enough MT-Bench score to reduce the computational cost (see details in App. A.3).
We add the results of the greedy search to the plots in Fig. 3, computing the true MT--Bench (i.e., with GPT-4 as judge). For both base models, the 4th example found by greedy search is sufficient match the best Ist-turn score achieved by scaling the in-context examples with random demonstrations (see Sec, 3.1). For Llama-3. the 5th and 6th demonstrations further improve the score, while they are not helpful for Mistral-7B-vOl 0. In Table 4, we provide the details of these results: the 4th ICL example yields a significant improvement over URIAL, e.g., from 6.95 to 7.61 for Llama-3. 3.B. with only a further +0.20 given by the 5th and 6th. Overall, this evaluation further indicates that the improvement in instruction following one can achieve with in-context alignment quickly saturates when increasing the number of examples. In Fig. 4, we show the distribution of the scores (with GPT-4-Turbo as judge) at step of the greedy search when using 1.maa-3.1-85 as base model (the analog for Mistral-7B-vol 4th example on top of URIAL (as shown in the first plot), but no further progress is observed with additional demonstrations.
in App. E.2). Most demonstrations improve the score when added the
We further test the IC prompts found by the greedy search on the AlpacaEval 2.0 (Li et al., 2023a) benchmark, where we measure length-controlled win rate. As shown in Table 4, using the examples
8

---

Published as a conference paper at ICLR 2025
SkillMii LFT Random
SkilIMix Randomn (no URIAL)
Mistral-78 20--ll--7--02--nstru
Evol- Instruet IFT Kandom
Evol- -vos-rnttuutt ICL Random (no URIAL)
1st-tur MT-Bench Score
8.0
2nd-turn MT-Bench Score
7.5
7.0
6.5
6.0
10
100
300
1000
10000
100
1000
10000
Number of Examples
Number of Examples
Mistral-7B-v0.2
SkillMix IFT Random
SkillMix Random URIAL)
Llama-3.11-8 -.B---nnstt
Evol-Instruct Random
Evol-Instruet Random (no URIAL)
8.5
1st-turm MT-Bench Score
2nd-turn MT-Bench Score
8.0
7.5
7.0
6.5
6.0
10
100 300
1000
10000
100 1000
10000
Number of Examples
Number of Examples
(h) Jlama-3.1-88
Figure 5: Comparison of ICL vs IFT for alignment in the low data regime. We measure the alignment performance of different settings for Mistral-7B-v0.2 and Llama-3.1-8 3.B-8B using the MT--Bene score. IFT with more demonstrations keeps improving the alignment performance, almost bridging the gap between the base model and its aligned counterpart. IFT--ligned models perform well multi-turn conversations, unlike with ICL. Finally, data quality has significant impact on IFT and ICL: higher-quality SkillMix leads to better performance than Inot-Instruct.
given by greedy search leads better results than plain URIAL AlpacaEval 2.0, without directly optimizing for it. This improvement indicates that the found IC prompt does not completely overfit to the MT-Bench score, although the gains less consistent than MT-Bench. For Mistral-7B-v0.2. URIAL with greedy search examples is the only configuration that outperforms URIAL, while for Llama-3.1-8B just one additional demonstration yields the best win rate, with significant improvement over URIAL (7.28 vs 8.61).
4 A COMPARISON OF ICL VS IFT FOR INSTRUCTION FOLLOWING
The results from the previous sections strongly suggest that in-context alignment might not be as competitive as sophisticated, computationally heavy alignment techniques. However, it is unclear how it compares to more lightweight approaches, such as IFT, when applied small number of examples. In the low-data regime, the choice between ICL and IFT can be viewed decision between allocating resources for fine-tuning (which permanently modifies the model's weights) or increasing inference time by adding the in-context prompt each time.
Setup. We test IFT on the same number of instructions from SkillMix as used in the many-shot ICL experiments in Sec. 3, ranging from 3 to 250 (the maximum allowed by the context length of the base models). Moreover, we scale the dataset up to 4k, that is the entire SkillMix. Each training run is repeated over multiple seeds, resulting in different training datasets (more details in App. A.4).

---

Published as a conference paper at ICLR 2025
Results on Mistral-7B-v0.2. The top part of Fig. 5 shows that IFT and ICL on SkillMix (violet and green curves respectively) perform almost identically in 1st-turn score until 50 examples are used. Beyond this point, increasing the number of training examples consistently improves the performance of IFT. The strong performance of IFT in the low-data regime is particularly surprising, as one might expect overfitting when training on very few examples (as few as 3) over several epochs, yet this does not occur. Finally, ICL and IFT show nearly opposite trends for the 2nd-turn score: increasing the dataset size benefits IFT (almost reaching the performance of the instruct model) but detrimental to ICL. This behavior suggests that, while IFT is less flexible due to the model weights being updated, it generalizes better to tasks different from those used for alignment (recall that the instructions in SkillMix are single-turn only).
Results on Llama-3.1-7B. As shown the bottom part of Fig. 5, ICL outperforms IFT when using between 3 and 30 demonstrations, though IFT remains effective even with these few examples on this base model. However, we note that IFT matches or exceeds the best performance of ICL (obtained with 20-30 examples) only when using 2k or 4k examples (the entire SkillMix), which represents two orders of magnitude more data. This result confirms that ICL with high-quality data is viable alternative to IFT when only limited number of demonstrations the observations for the 2nd-turn scores are consistent with those for Mistral-7B-vO.. with IFT consistently outperforming ICL.
available. Finally,
Effect of data quality. Finally, we test the effect of using lower-quality instructions compared to SkillMix. In this experiment, we repeat the ICL vs. IFT comparison using random demonstrations from Evol-Instruct-70k (Xu et al., 2024), and show the results in Fig. 5. With Evol-Instruct data, both ICL (red eurve) and IFT (yellow curve) perform significantly worse than their counterparts using SkillMix. This trend is consistent across the number of examples, base models, and for both single-ture and multi turtitttunnnn instructions. Interestingly, the performance gap between IFT on SkillMix and Evol- Instructt is significantly smaller on Llama-3. 1-BB than on Mistral-7B-v0.2 suggesting that better pre-trained models may partially compensate for lower-quality fine-tuning data. Finally, we notice that ICI. with 3 examples from Evol-Instruct gets worse ss-tuun scores than URTAL (with also examples), whereas ICL with 3 examples from SkillMix matches (on Mistral-7B-v0.2) or outperforms (on Llama-3.1-8D URIAL (see also discussion in Sec. 3.1). These observations further confirm the importance of instruction quality for alignment, whether in the context of in-context learning or instruction fine-tuning.
5 CONCLUSIONS
In this work, we have first illustrated that ICL. via URIAL is good baseline for instruction-followee alignment, but with a few limitations: it typically performs slightly worse than IFT on single-turn conversations and does not generalize well to multi-round ones. Then, we have uncovered the key components for IC alignment: e.g. the decoding parameters alone can, surprisingly, make base models reasonably good at instruction following. Adding in-context high-quality demonstrations improves performance beyond what previously suggested (Lin et al., 2024), but not enough to reach LLMs ali aligned with sophisticated methods (e.g.. RLHF). We conjecture that via ICL., the L.MM can learn to infer the response style, but the overall learning signal is not sufficiently strong to benefit from a large amount of examples, despite the long context windows of recent LLMs.
Moreover, to the best of our knowledge, we have provided the first systematic comparison of ICL and IFT for instruction following when using the same (small) number of demonstrations. Surprisingly, the two approaches are roughly equivalent in terms of single-turn conversation performance. However, models trained via IFT, unlike ICL, can generalize to multi-round conversations, which are out-of- distribution compared to the examples used for alignment. Overall, our work provides a deeper and more complete understanding of how in-context alignment works, as well as of its potential and limitations, in particular in comparison to fine-tuning. This comprchension might be the basis for future work aimed at using ICL to efficiently customize LLMs without the need of fine-tuning, as well as exploring the fundamental differences between base and aligned models.
Limitations. We perform greedy search with limited computational resources, e.g. the set of candidate examples is relatively small. We expect that more effective ICL prompts could be identified with additional resources. Moreover, our focus is on mnstruction-follwvii tasks, but other types of alignment, such as sarety- tasks, may also be of interest.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We are grateful to OpenAl for providing us API credits and access to GPT-4-Base as part of the OpenAl Researcher Access Program. M.A. supported by the Google Fellowship and Open Phil AI Fellowship. This work was partially funded by an unrestricted gift from Google and by the Swiss National Science Foundation (grant number 212111).
REFERENCES
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning, arXiv preprint arXiv:2404. 11018, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.
4447-4455. PMLR, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22/2.0000 2022.
Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neu- big. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303.. /2712, 2023.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. arXiv preprint arXiv:2307.088:
2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cnrructtoonfinetunee language models. arXiv preprint arXiv:22100 1416, 2022.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672 2022.
Hanyu Duan, Yixuan Tang, Yi Yang, Ahmed Abbasi, and Kar Yan Tam. Exploring the relationship between in-context learning and instruction tuning. arXiv preprint arXiv:2311.10367 2023.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2rXiv:2407.21783 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafssky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannanch Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.1017. 2024.
Amav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprictary llms. arXiv preprint drXiv:2305.15717, 2023.
11

---

Published as a conference paper at ICLR 2025
Xiaochuang Han. In-context alignment: Chat with vanilla language models before fine-tuning. arXiv preprint arXiv:2308. 04275 2023.
Jiwoo Hong, Noah Lee, and James Thome. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXin:2403.07691, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint urXiiv:201. 24000888 2024.
Simran Kaur, Simon Park, Anirudh Goyal, and Sanjeev Arora. Instruct-skillmi: A powerful pipeline for Ilm instruction tuning. arXiv preprint arXiv:2408. 14774, 2024.
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of Ilms. arXiv preprint arXiv:2308.073: 2023.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. long in-context learning. arXiv preprint arXiv:2404. 020600000000 2024.
ilms struggle with
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of cnerruction-oollwiinn models. /etpps//gitiie
waaam/atsu--lle a lpaac__vall, 2023a.
Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.0711 2023b.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base LLMs: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations. 2024. URL https:/ wpwpneevee..eeee////pp/pp net
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?
EMNLP, 2022.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topie-aware convolutional neural networks for extreme summarization. arXiv preprint 1rXiv:1808.08745 2018.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv: 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follovin Ilama model. https:/ /ttps:///ittunn com/ tatsu- lab/ taab/stanfie aaaaardaoaaaaac.o 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092288 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint 109i:2009.01160 2021.
12

---

Published as a conference paper at ICLR 2025
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardLM: Empowering large language models to follow complex instructions. ICLR, 2024.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. In Forty-first International Conference on Machine Learning. 2024. URL htRp::ts:://ppo httpttss///peeeeeeee... net/f srum?id=0AzA:jaa
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. NeurlPS Datasets and Benchmarks Track, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
13

---

Published as a conference paper at ICLR 2025
A
EXPERIMENTAL DETAILS
In this section, we show more details about the experiments we conducted in the paper.
A.1 DATASETS AND BASE MODELS
We select cnstructioo-folloviiii demonstrations for the base LLMs to learn in context from open- sourced instruction fine-tuning datasets: (a) SkilIMix-4k (Kaur al., 2024) is an automated approach for synthesizing diverse, high-quality IFT data. It primarily involves two stages: first leveraging the LLMs propose set of critical "skills" for instruction-folloiing from which pair of skills are randomly chosen to facilitate synthetic data generation based on powerful LLMs. We show two examples from SkillMix Fig. and Fig. 7. (b) Evol-Instruct--00 (Xu et al., 2024) contains 70k training examples with varying complexity and is well-known for the use to build the series of WizardLM models. SkillMix-4k is the primary dataset for all experiments in the paper and Evol-Instruct-700 is solely used for scaling experiments and ablation study of data quality.
We compare ICL performance across multiple models that have sufficiently large context windows:
1. Mistral-7B-v0.2 (Jiang et al., 2023) has 7.3 billion model parameters. On many established benchmarks, it outperforms LLMs that have significantly more parameters, such Llama- 13B and 1Jama---34BB The trained context length of Mistral-7B-v0.2 is 32k tokens.
2. Llama-3.1 1-BB (Dubey et al., 2024) the newest and most powerful 8B model from Meta when we are writing, and it supports multilingual dialogue use cases. It supports input texts containing up to 128k tokens.
In some experiments, the following base LLMs are used:
1. Llama-2-7B-800 (Fu al., 2024) is variant of Llama-2-7B model fine-tuned with 80k context on a carefully designed long-document data mixture.
2. M.xtral-8x222-v0.1111 1-4bit is variant of Mixtral-8x222--v0, (Jiang et al., 2024), which is pre-trained generative sparse Mixture of Experts, quantized with 4-bit precision. It contains ~17 parameters and --44 active during inference, and it has a 65k context window.
We use the same decoding configuration as what is used in URIAL (Lin et al., 2024). Concretely, we employ greedy decoding, i.e., temperature 0.0. for all models, including base and instruction fine-tuned models, to maximize reproducibility and secure a fair and robust evaluation. Top-p = 1.0 adopted keep the full cumulative probability distribution. Besides, we use repetition penalty 1.152 on base models to prevent degeneration.
A.2 EVALUATION
MT-Bench (Zheng et al., 2023), consists of 80 high-quality and challenging questions that have two-round interaction, designed examine the multi-turn conversation and -ostlottoonfolllowing capability of models. It features 8 common categories of user prompts: coding, math, reasoning, extraction, roleplay, writing, humanities/social science, and STEM.
AlpacaEval 2.0 (Li et al., 2023a) provides 805 test instructions, which we generate new responses using the target model, and then calculate the score by competing with the baseline model (i.e.. GPT-4-Turbo) judged by designated automatic evaluator. We apply the AlpacaEval 2.0 benchmark in our experiments to ensure that the effective demonstrations found through greedy search don't overfit to MT-Bench.
A.3 GREEDY SEARCH
Firstly, we randomly sample 100 examples from the high-quality IFT dataset, SkillMix-4k, and then create 100 new 4-shot prompts by adding each one the 100 sampled examples, as the fourth
As in the codebase at https: : httgg/gtb.coo//eeeee/ / www.h/enhhhhcoe wwwebeenhhhuueeeeeeeeeeeeeeeeeee mt_ wwnchhth..hhee_eeeeeeeee
14

---

Published as a conference paper at ICLR 2025
Adk Skils developing training programs, dightl proficiency
ABV Instruction
I'm planning to develop digital literayy program specifically for seniors at our communiy center. Could you guide me the cesential components to Include, how I can lovirage digitl tools effectively, and provide some concreta enamples success stratagies from simiar programs?
ady Response
Creating failored digital literacy traiiing program for seniors key onhancing their technology engagement, Below comprehensive guide featuring improved content based earleer tecoback
1. Assessing Learning Neods:
Conduct surveys focus groupa to understand seniors' technological backgrounds and learning objoctives. This data will guide the customization training content.
2. Structuring the Program:
Organize the program into manageable modules:
Basic Computing Skills: Instructions on fundimentall operations of devices, incorporating user-friendly technology like Chromebooks, which are simpler for beginners.
Internet Novigatiore Teach safe browsing, using straightfowward and secure websites for everyday tasks.
Email and Communication Tools: Demonstrate setting up and using email and video calling, with adjustments for larger display settings to ald visibiity. Online Safety: Educate creating strong passwords with tools like LastPass, identitying secure websites, and recogrizing scams.
3. Using Digital Tools in Training:
Integrate speciffe digital tools to enhanee learning:
Interactive Tutorials: Implement platforms the GCFBlobal that ofter senior-friene friencly interactive lessons.
Feodback Mechanisms: Use simple, ongaging quizzes through Kahoott review and reinforce leamiin..
4. Scheduling and Accessibilty::
Plan short, regular sessions. Ensure the leaming environment comfortable and accessible, considering both physical and online settings.
6. Faciiitatin Engegement:
Peer Learning: Support seniors working pairs promote mutual assistance and engagement.
Ongoing Support: Establish support system such as helpline dedicated email for ongoing help, coupled with monthly follow sessions.
6. Case Studies and Success Stories:
Reter succeestul programs like MRPs's TEK workshops, highlighting their positive impact on senior confidence with technology.
7. Measure impact and Adapt:
Use analytics tools to monitar partioipation and feedback, continuously rofining the program based on empirioal data.
By tocusing on these enhanced, spectfic steps and maintaining a dynamic feedback loop, your digital litrrayy program will significantly improve seniors' technological fluency and contidence.
tee Query Type: Planning
ase Data Genemtor: gpt-4-turbo-
Figure 6: An example of SkillMix describes a conversation about planning and includes two skills, developing training programs and digital proficiency,
1et Skils: performance metrics analysis, writing review articies
ad4 Instruction
am working review article about various performance metrics used machine loarning models for class project Specifically, want understand the real-worrdd impllcations and methodological differences between the F1 Score and AUC RDC. Can you explain and compare these metrics based on the following excerpt from research paper?
Excerpt:
"in evaluating classtfier performance, the F1 Score and AUC ROC are pivotal metrics. The Ft Score, a harmonic mean of precision and recall, crucially balances the trade-offt -off-oft betweon false positives and false negatives. On the other hand, AUC- RDC. which measures the area under the receiver operating characteristic curve, provides an agoregate measure of performance across all classtfication thresholds, Notably, AUC- AUC-ROC beneficial for evalusting classifiers imbalanced datasets as is unaffected by the class distribution.
ade Response
Interpretation and Analysis of F1 Score and AUC-ROC
The provided excerpt discusses two crucial performance metrics for classtiers in machine leammin: the F1 Score and AUC-ROC, Each has distinct utiliiis and provides unique insights into models performance under different conditions.
Ft Soore: Balanoing Precision and Recall
The F1 Score is the hamonic mean of precision and recall, ensuring that both talse positives and faise negatives are equally weighted. This metric essential in situstions where both types of emors am costly For instacce, traud detection systems, it ontical minimze both halse alarms (talse postivee) and missed fraudds (talse negativee). The F1 Scom optimal here balances censtivity (recal) with the precision needed to avoid exceesive talse alerts. AUC ROC: Evaluating Performance Across Thresholds AUC- ROC measures the area under the receiver operating charasteristic curve, providing comprehensive view model perfomaanc across all classfiiation thresholds. This metric is especially valuable imbalanced datasets, common medical diagnestics, where the prevalence of one class significantly outwelght the other. AUC-RDC assists understanding how well model can distingish between dasses, independent of the class distribution. Comparison and Practical Application
While the F1 Score gives precise measure of model accuracy at specifio throshold, focusing on the balance between prectsion and recall, AUC ROC offors insight into the model's ability discrminatt between classes across thresholds. This makes AUC-ROC suttable for pretimnarr model analysis and scenartos where class imbalance might skew the perception of model affectivenees provided by other metrics like accuracy.
Conclusion
Choosing between F1 Score and AUC-ROC, or using them in conjunction, depends on the specific needs of the claesifiation task and the cost implications of entors. Undorstanding these metrica' implicatonn can significantly enhance the quality and depth of your revlow article, providing cleaner guide for model evaluation on diverse practical scenarios. Both metrics together provide a robust framework for evalusting classifee performance, especially complex real- world settings.
ass Quary Type: Interprative Explenation
4ae Data Ganerstor: gpt-4-ture 2ol---turbbo2224-44--6
Figure 7:
An example of SkillMix describes a conversation about interpretive explanation and includes two skills, performance metrics analysis and writing review articles.
15

---

Published as a conference paper at ICLR 2025
Instruction
Below lest conversations between human and an assistant (you)
As Al assistant, will conversations with responding their which presented
Your heading "F
entered under the
Answer".
range including reasoningg engaglig play,play, creative writing, planning. solving mathematical and coding problems. Your resporses should well-structured, comprehansios, aim thoroughly address query
providing general information, conducting
problem at hand.
New Demonstrations)
Query:
[Query]
* Answer:
[Answer]
(3 URIAL Demonstrations)
1 Query:
(Query)
[Answer]
Figure 8: The prompl template for doing ICL in our work. Note that the words in orange are solely for illustration purposes and do not appear in the real prompts. The mixture of new demonstrations test in experiments inserted before URIAL demonstrations.
demonstration, to the original 3-shot URIAL prompt. We evaluate the resulting instruction-followingg performance for each prompt on MT-Bench but with GPT-4-Turbo the judge since GPT-4-Turbo cheaper and faster than GPT-4 (the original LLM judge for MT-Bench evaluation), and also has high correlation with human judgment. Finally, the example with the highest 1st-turn MT-Bench score is chosen.
We continue our greedy search for the proper fifth and sixth demonstration in more restricted search space due to the heavy computational cost of the search process. We keep reducing the search space by applying threshold of 1st-turn MT-Bench score (i.e., 6.2 in experiments). Similarly, we add each example from the new search space on top of the best (N-1)-shot prompt found in previous steps and generate a of new N-shot prompts. Then run the MTTbench evaluation with GPT-4-Turbo as the judge for multiple times and select the best (N+1) prompt.
A.4 SCALING EXPERIMENTS
The recently released LLMs with increasingly large context windows allow using many more shots for ICL than typically used. Prior works Agarwal et al. (2024); Bertsch et al. (2024) have shown that many-shot ICI., compared to few-shot ICL., can make task-specific fine-tuning less essential and allows LLMs tackle wider range of tasks without specification. Therefore, if the instruction-follovii various scaling experiments, and systematically compare ICL with IFT. In the following, we add more details about these scaling experiments, including the prompt template, data selection for both ICL and IFT, and training syper-parameters for IFT experiments.
want examine performance of base LLMs could benefit from many-shot ICL through
ICL Random. We construct the set of in-context demonstrations based on the high-qualit data we select from SkillMix-4k or Evol-Instruct- dataset. Through randomly sampling from the high-quality dataset for multiple times, we generate series of in-context demonstration sets that each contains N examples, where E {0, 7, 17, 27, 37, 47) for Mistral-7B-v0.2 model (32k context length) and N {0, 7, 17, 27, 37, 47, 97, 147, 197, 247) for Llama-3.1 length), and insert the N demonstrations into the prompt template for ICL as shown in Fig 8. Note that the prompt template is strictly following the used in URIAL paper and we only replace the demonstrations used for the ICL purpose. The average performance and standard deviations are computed over 5 random seeds.
model (128k context
16

---

Published as a conference paper at ICLR 2025
Instruction Below is list of conversations between a human and an Al assistant (you).
As an Al assistant, you will engage in conversations with users, responding to their queries which are presented under the heading "f Query:". heading You excel in wide range of tasks including, but limited to, providing general information, conducting reasoning, engaging in role- play, creative writing, planning. and solving mathematical and coding problems, well-struclune, comprehensivv,
resporses
to thoroughly address user's query problem hand.
Query:
Can you tell me some common types of renewable energy sources?
Answer:
Certainly! Renewable energy sourcess
are replenished naturally sustainably on timescale. Here common types
renewabbe energy sourDes:
Solar Enerpy: Energy hamessed from the sun. can be used for heating, lighting, and generating electricity through photovottalc (PV) panets solar
Wind Energy: Energy generated from the natural movement of air. Wind turbines convert kinetic energy from the wind into mechanical power, which can electricity
be converted
Hydroppower /Hydroelectric Powert: Energy derived from movement water. This Hydroelacitic power plants typicall use a dam on a flver to stom water in reservoir, water released from the reservoir flows through turbine, spinning it, which turn activattes a generator tn produce electrcity.
fivers, tidal movementss
power.
plants, and When these materials burned converted into biofuets (ike athanol and biodwesej, ( they can be used as an energy source for heating, electricity and transportation.
Biomass
matoriats
Geathernal Energy: Energy extracted from the heat stored beneath Earth's surface. This include use of hot springs for direct heating produce steam
driling
electriclty goneration.
Each these renewable energy sources has its own advantages and challanges terms technology, cost, and the potential impact on environment. you have any other questions!
alteraaivve fuels, contributingg the mitigation
change.
know
Query:
Natalis sold clips her friends April, she hall many clips May, many clips did Natelin altogather April May?
Answer:
Hi, I'm happy to help you. To solve the problem, let's think step by steg:
In April, Natalia sold clpp 48 friends.
In May, ahe sold half many she did in April. So, the number of clipa she sold in May 48/2 = 24.
To find out
clips she sold aftogether April and May, simply add the number of clips from both monthe: 40 (Apri)) 24 (May) clip.
Thersfore, Natalin sold clips altogether April May. Please
if you have any more questiona!
Query:
What proper method for deep frying a turkay?
Answer:
Hello! I'm happy to help you with your question about deep frying a turkey! Deep frying turkey can be a delicious way to cook your holiday bird, but it's important to do safely to avoid accidents. Here's a step-by-stey guide to deep frying a furkey:
1. Choose the right turkey: Select fresh, unfrazen turkey that no larger than 12-14 pounds. A larger bird can cause the oll ball over, which can lead to dangerous fire.
Thoroughly thaw and the towels, inside and out, This will help prevent oll splatter and ensure crispy skin.
bafora trying. Remova the and giblota from eavtyy the turkey and pat dry with papar
Season the turkey: Season the turkey with your prefemed spices and herbs. You can use a dry rub or inject the seasoring directly into the meat. or
Pregare fryer: canolá ol to the recommended level, typically indicated on the fryar. Heat oil to the appropriato famperature, hyploally between 325-460" F (168-177"c cj. the
fryer
away
Lower the turkey into the fryer: Using turkey Iift or hooks, carofully and slowly lower the turkey into the hot oll. Make sure the turkey la fully submerged in
Cook the turkey: Fry the turkey for the recommended time, usually about 3-4 minutes pound. Monitor the temperature of the oil throughout the cooking process to ensure it slays within the recommended range. Remove turkey:
the turkey cooked to internal temperature 165'F (74')), carefully and slowly out the fryer using the turkey drain excess
hooks, Place 8. Lot rest: Allow the turkey rost for at least 20-30 minutes before carving. This will help redistribute the juices and ensure a moist, flavorful burkey.
Remember to always prionizze safety when deep Additionally, always follow the manufacturer's is instructions and guidelines for your specific fryer model.
turkey, Never leave the fryer unattended, and keep a fire extinguisher nearby in case of emergency.
Figure 9: The details of URIAL prompt. It consists of carefully designed set of rules after "# Instruction" and thrce curated stylistic examples, cach of which always begins with affirming the user query by saying something similar to "Hello! I'm happy to help you.".
ICL Random (no URIAL). We generate a series of in-context demonstration sets by ran- domly sampling from the high-quality IFT dataset, and each set contains N examples, where N E (3. 10. 20, 30, 50} for Mistral-7B-v0.2. model (32k context length) and N (3. 10, 20, 30, 40, 50. 100, 150, 200, 250) for Lllma-3.1180 model (128k context length). URIAL examples will not be added to the context, so it ensures the total number of in-context examples are the same as the Random group. The average performance and standard deviations are computed over 5 random seeds.
ICL Greedy Search. Following the procedure we mention in Sec. A.3, ideally we can get as many optimal examples as we want if we have sufficient OpenAl API credits. Thus it allows us to ereate another series of in-context demonstration sets by selecting another N examples for each N e {1, 2, 3} in the paper. The resulting mixture of in-context demonstrations is then placed in the corresponding location of the prompt template shown in Fig. 8.
17

---

Published as a conference paper at ICLR 2025
Table Details of training hyperparameters for IFT experiments.
Data Size
GPUs
Epochs
LR
LR Scheduler
Batch Size
Context Win. Len.
WD
Warmup Rate
Mistrul-7B-v0.2
3
2e-6
Cosine
2048
0.01
0.03
10
6
2e-6
Cosine
8
2048
0.01
0.03
6
2e-6
Cosine
8
2048
0.01
0.03
2e-6
Cosine
0.03
40
6
2c-6
Cosine
8
2048
0.01
0.03
50
6
2c-6
Cosine
8
2048
0.01
0.03
100
2e-6
Cosine
0.01
0.03
300
2c-6
Cosine
0,01
0,03
1000
15
2e-6
Cosine
128
2048
0.01
0.03
2000
15
2e-6
Cosine
128
2048
0.01
0.03
2e-6
Cosine
1Jama-3.1-88
4e-6
Cosine
2048
0.01
0.03
10
4e-6
Cosine
8
2048
0.01
0.03
20
6
4e-6
Cosine
8
2048
0.01
0.03
Cosine
0.01
4c-6
Cosine
2048
0.01
0.03
50
6
4e-6
Cosine
2048
0.01
0.03
100
6
4e-6
Cosine
8
2048
0.01
0.03
4e-6
Cosine
1000
15
4e-6
Cosine
128
2048
0.01
0.03
2000
15
4e-6
Cosine
128
2048
0.01
0.03
4000
46-6
Cosine
128
2048
0.01
0.03
Table Improved in-context demonstrations selected from Evol-Instruut-700 dataset for Mistral-7B-v0.2 base model.
report the 1st-turn score on MTT-eench and the length-contralled (I.C) win rates on AlpacaFval 2.0 when using different IC prompts.
Model
MT-Bench (Ist) AlpacaEval 2.0
URIAL (3 examples)
6.99
8.09
URIAL greedy search ex.)
7.16
7.91
URIAL greedy search (2 ex.)
7.69
8.38
URIAL greedy search (3 ex.)
7.68
9.22
IFT. We run instruction fine-tuning the base LLMs (Mistral-7B-v0.2 and Llama-3.1-8Bi either with tens of examples (i.e., few-sample regime) or thousands of examples. The training examples are randomly sampled from existing IFT datasets, SkillMix-4k and Syoo-Instruuct-0k The mean score and standard deviations are calculated over multiple random seeds. More specifically, we use random seeds for Mistral-7B-vOL model and 3 random seeds for Llama-3. model due to restricted compute budget. In particular, since the size of SkillMix dataset is 4k, the randomness of IFT with 4k examples primarily comes from optimization process and scoring evaluation with GPT-4, otherwise part of randomness also comes from data sampling process. We list the details of training hyper-parameters used in IFT experiments in Table. 5. We always select the last model checkpoint to evaluation.
B REVISIT OUR FINDINGS ON INSTRUCT MODELS
In this section, we want to revisit some of our key findings in the paper on instruct model, including influence the individual components of URIAL ($ B. importance of question-answer matching of demonstrations for in-context alignment ($ B.2), and many-shot in-context alignment ($ B.3). We experiment using Mistral-7B-Inn 1in--B-stuutt-O.. the instruct version of Mistral- 7B- that widely used in the main paper, and again adopt M'T-Bench with GPT-4 as the judge for evaluation.
18

---

Published as a conference paper at ICLR 2025
Table Inftuence of the individual components of URIAL instruct model We report the MT-Bench scores for cases that have multiple different combinations, such Rule example and Rule + example. The standard deviation is always calculated on runs.
In-context prompt
1st-turn
2nd-turn
Average
Empty
7.92 + 0.08
7.18 = 0.05
7.55 + 0.06
Rule-only
7.80 0.07
7.15 + 0.10
7.48 + 0.09
Rule + example
7.54 0.15
6.82 0.34
7.18 + 0.19
Rule examples
7.70 = 0.17
6.24
7.27 = 0.19
URIAL
7.58 0.06
6.48 0.04
7.21 0.06
Table 8: Importance of question-answer matching of demonstrations for in-context alignment instruct models.
In-context prompt
Ist-turn
2nd turnturn
Average
no demonstrations
7.92
7.18
7.55
X only (no Y) Y only (no X) circular shift of Y in-domain random out-of-domain random correct Y
4.29
6.06
5.18
7.53
6.85
7.19
6.09
5.99
6.04
5.94
6.28
5.69
5.50
5.60
7.58
6.48
7.21
B.1 INFLUENCE OF THE INDIVIDUAL COMPONENTS OF URIAL ON INSTRUCT MODEL
In Table 7, we find that different from what we observed from the base LLM, adding in-context examples, even only rules, weakens the cestruction-followiin performance of instruct models as measured by MTT-Bench performance, and more examples lead to more performance declines.
B.2 IMPORTANCE OF QUESTION-ANSWER MATCHING DEMONSTRATIONS FOR IN-CONTEXT ALIGNMENT ON INSTRUCT MODELS
In Table 8, we find that adding examples in the context consistently causes damage to the instruction- following performance of instruct models, even when using correct URIAL examples, which is similar to what we observe on the instruct model in Table 7. Surprisingly. it leads to pretty bad 1st-turn score (4.29) MT-Bench when only presenting queries and corresponding answers in the context. After looking into the model responses, we find that the instruction model sometimes would manage answer all queries present in the context, including the ones demonstrations, and outputs consisting of answers to multiple questions lead to low performance on model-based automatic evaluations. This unexpected behavior mitigated in the 2nd-turn evaluation complete query-answer pair is added to the context after the 1st-turn answer generation.
B.3 MANY-SHOT IN-CONTEXT ALIGNMENT INSTRUCT MODEL
We further test the effect of alignment with ICL on Sostral-7B-Instrutt-ee when increasing the number of demonstrations. We test on the same data we used in Fig. and more details can be found in App. A.4. The results of many-shot in-context alignment experiments shown in Fig. 10. We find that alignment through ICL does not further improve the instruction-following performance of instruct models, but slightly declines its capability to answer queries.
19

---

Published as a conference paper at ICLR 2025
Bassral-7B-vO.2-mmme
ICL Random
ICL Random (no URIAL)
1st-turn MT-Benchh Score
8.25
2nd tur-ttrm MT-Bench Score
7.5
8.00
7.75
7.0
7.50
6.5
7.25
7.00
6.0
0
20 Number of Examples
10
30
50
0
10
20
30
50
Number of Examples
Figure 10: Scaling the number of demonstrations for alignment with ICL on 1-in------nntttt--..2222 We measure the alignment performance of different settings using the MT-Bench score. The base model is always Mistral-BB-Instee -n-stttttt-t for ICL Random and ICL. Random URIAL
Base vs In-context alignment with URIAL Base IFT model with examples Base IFT model with 4K examples
5
LL-divergence
3
2
5
10
15
20
25
30
Relative Positions
Figure KL-divergence between the base LLM and aligned model using ICL or IFT. The base model always Mistral- 70,, The KL-divergence results are averaged over 25 examples.
C KL-DIVERGENCE BETWEEN THE BASE LLM AND ITS ALIGNED MODEL USING ICL OR IFT
Since we always experiment with open source LLMs with 7 ~ 8 billions parameters, we are interested mechanistic study with logits of model outputs to reveal more insights, such the KL-divergence between the base LLM and its aligned counterparts.
Experimental setup. We randomly select 25 query-response response paired examples from SkillMix dataset as the data to calculate KI. -ll-divereeece between the base LIM and the aligned models. Specifically, for each query-response paired example, two models predict the next token given the same input respectively. The model only predicts tokens from responses and each answer token after calculating KL-divergence will be added to the input for the next prediction. The response from each example truncated to save computation costs. We always select Mistral-7B-ve another 3 aligned models, in-context alignment using URIAL, IFT with SkillMix examples, and IFT with 4k SkillMix examples, target models calculate KL-divergence.
the base model, and
Results. From Fig. 11, we observe that the probability distribution of the next token prediction, for the in-context aligned model, differs significantly only for the first few tokens, after which the
20

---

Published as a conference paper at ICLR 2025
Table 9: Transferability of greedy search IC prompt to other LLMs. We the 1st-turn MT-Bennh the length-controlled (LC) win rates on AlpacaEval (AB) 2.0 when using various IC prompts on different base models.
Model
MT-B (Ist)
AE 2.0
Base model:
Llama-2-7B-800
URIAL (3 examples)
5.19
1.81
URIAL greedy search (1 ex.)
5.56
1.50
URIAL + groedy search (2 cx.)
5.57
1.84
URIAL + greedy search (3 cx.)
5.10
2.91
Base model: Mixtrra--xx2BB-x011-hhh
URIAL (3
examples)
8.28
14.77
URIAL
greedy search (1 ex.)
8.36
15.74
URIAL greedy search ex.)
7.79
13.20
URIAL +
greedy search (3 ex.)
8.58
14.68
25
URIAL URIAL-46
SITRIAL
20
20
LURRAAL--aa 4tb-5th
20
Count
Count
Countt
10
6.2
6.4
6.6
6.8
6.2
6.5
6.3
5.4
6.6
6.8
Lst Ail-ummm MT Donch Score
1st-tum MT Bench Score
1se: Ise-tum MT BenBBBnchhhhh Soore
(a) Search for 4th example
(b) Search for 5th example
(c) Search for 6th example
Figure 12: The distribution of the Ist-turn MT-Bench score (GPT-4-Turbo as judge) on Mistral-7B-v0.2 obtained by adding multiple instructions from SkillMix (Kaur et al., 2024) as 4th (a); 5th (b); 6th (c) demonstration to URIAL. The dashed lines of various colors refer to the 1st-turn MT-Bencc score of the obtained searching results. A majority of the 4th examples have positive contrihutions the model's instruction-follen foolooiigg performance, but the improvement quickly diminishes when running the greedy search for more optimal demonstrations.
KL-divergence value rapidly drops to approximately 0. However, the KL-divergence between the IFT models and the base model does not converge to and has larger KL-divergence values on average. Surprisingly, IFT only examples is already sufficient drive the fin-tuned model far away from the base model as measured by KL-divergence compared to in-context aligmment. In particular, the difference in the probability distribution of next token prediction becomes more substantial as the base model are instruction fine-tuned on more data.
D TRANSFFRABBILITY OF IN-CONTEXT PROMPTS
In Table 9, we report the performance of applying the in-context examples found by greedy search (added URIAL) on Mistral-7B-v0.2 and Evol-Instruct-70k dataset (see Table. 6) Llama-2-7B. 80k (Fu et al., 2024) and -extral-8x22B-Bo 1-4bit (Jiang et al., 2024). Adding the new examples does not provide a consistent improvement: while can increase the MT-Bench score (+0.47 on +0.30 on
1-bit),, it can also, in some cases, give worse results than the original URIAL. Similarly, yields mixed results when measured by win rate on AlpacaEval 2.0, which was not optimized by the greedy search. This analysis suggests that the most effective ICL demonstrations might vary across base LLMs, potentially because of differences their pre-training. This could further explain why URIAL underperforms on multiple models in Table 1, especially on those which have become available only recently and were not used for selecting the URIAL examples.
Llama-2-7B-800
21

---

Published as a conference paper at ICLR 2025
Table 10: Scaling experiment results (length-controlled win-rates) AlpacaEval 2.0. We put different amounts random examples drawn from SkillMix to context demonstrations. We conduct experiments on two base models: -istra and Meta-Llama-3,1 1-B... The results are computed over different runs.
AlpacaEval 2.0 LC Win-rate (%)
Mistral-7B-v0.2
Meta-Llama-3.11- 3.1BBBBBBB
example
2.92 0.0
2.99 0.0
example
11.98 + 0.7
8.66 + 2.7
examples
12.50 + 1.3
11.56 + 0.6
examples
14.6 + 0.4
15.11 = 1.9
20 examples
13.20 + 0.7
15.18 = 0.8
30 examples
12.45 + 0.1
13.90 = 0.2
40 examples
12.51 + 1.3
14.95 = 0.7
50 examples
13.80 + 0.7
15.49 + 0.8
E ADDITIONAL EXPERIMENTS
E.I SCALING EXPERIMENTS ALPACAEVAL 2.0
This section provides additional results of scaling experiments conducted on the AlpacaEval Suite, which contains 805 single-turn turle-ttrn examples, an order of magnitude more than MT-Bench. Based on the SkillMix dataset, construct the set of in-context demonstrations by sampling different amounts of random examples, ranging from 0 to 50 examples. The results shown Table confirm our findings, reported in Sec. that scaling the number of ICL examples is not sufficient to consistently improve the instruction following performance.
E.2 GREEDY SEARCH ON MISTRAI--7B-O.22
In this section, we show the distribution of the resulting scores (with GPT-4-Turbo as the judge) for each step of the greedy search on SkillMix-4k dataset when using Mistral-7B-v0..2 as the base model. Similar to the finding on L 1-BB model (see Sec. 3.2), the majority of demonstrations increase the alignment performance, as measured by sst-turn MT-Beecch score, when added as the fourth example on top of URIAL, and few progress is made with more demonstrations. We show more details of greedy search results on Mistral-7B-v0.2 in Fig. 12.
E.3 MORE EXPERIMENTS ON DECODING SCHEMES
We show more results decoding schemes under different settings in this section, as supplementary the main results in Sec. 2.2. Specifically, in Fig. 13, we turn off the repetition penalty (i.e., repetition penalty = 1.0) and show the Ist-turn MT-Bench scores on Mistral-7B-v0..2 when varying the values of temperature and top-p. Moreover, we repeat the same experiment procedure but on different base model, Llama-3-88 3-BB, and show heatmaps in Fig. 14.
E.4 A BREAKDOWN EXAMINATION OF URIAL-ALIGNED MODELS
The test set of MT-Bench (Zheng et al., 2023) is comprised high-quality questions that belong to common categories: coding, math, reasoning, extraction, roleplay, writing, humanities/social science, and STEM. Table 11, we present the per-category performance on MT-Bench for each model in order have instruct models. The results suggest fine-tuning is almost always better than URTAL (ICL.) and only underperforms minority of cases on particular base models, such Mistral-7B-vO.1 and Llama-2-70B.
detailed comparison between the URIAL-aligned models and the corresponding
22

---

Published as a conference paper at ICLR 2025
3.89
5.53
5.80
5.61
7.10
3.18
7.13
7.19
7.85
7,80
7.61
7.96
9.
5.90
5.91
5.75
5.31
7.32
7.23
7.01
7.36
Top-p
7.73
7,50
7.54
7.61
5.91
5.84
5,66
3.61
7.01
6.83
6.17
1.0
7.78
7.56
7.78
7.24
0.5
0.0
0.5
0.0
0.5
Temporature
Temporature
(a) ICL rp 1.0
(b) ICL with URIAL rp 1.0
(c) Instruct model / rp
6.39
6.01
6.10
5.51
7.14
3.11
7.12
7.26
7.40
7.38
7.62
7.97
Topep
Topep
TopP
6.60
5.94
5.10
4.53
7.30
3.39
7.19
7.60
7.64
7.72
7.66
7.56
Temporature
Temperature
Temperature
(d) No 1pp 1.15
(c) ICL with URIAL rp= 1.15
Instruct model / rp = 1.15
Figure 13: The 1st-turn MT-Bench scores model generations with and without ICL across different decoding schemes. We mainly consider two hyper-parameters: temperature and top-p. The heatmaps show the answering quality of the base model with and without URIAL in the context is sensitive to the setups of decoding schemes. Surprisingly, with proper decoding parameters, the base model alone is already capable of following instructions.
Table 11: A breakdown examination of URIAL-aligned models and corresponding instruct models across different categories on MT--Bnch. denotes the results taken from the URIAL GitHub repository.
Model
coding
math
reasoning
extraction humanities
roleplay
stem
writing
Average
Llama 2.7B URIAL *
1.65
1.60
3.45
3.40
8.08
7.48
6.80
6.20
4.83
Llama-2-7B-Instmadd
2.95
2.40
5.20
6.33
9.58
7.83
8.88
9.05
6.53
Llama 2 70B URIAL
4.15
3.60
6.10
7.70
9.75
7.33
8.75
9.50
7.11
Llama-2- 2-0---nmcc-
5.95
9.13
Llama 3B URIAL
4.15
2.60
3.50
8.90
7.30
8.15
6.13
5.75
Llama- 3-5--3----nnn-taaa
8.95
5.05
6.15
9.16
9.90
9.05
8.95
8.70
7.86
4.35
4.95
6.40
Daama-3-70BIInrre
7,85
7.98
6.25
9.75
10.00
9.30
9.60
9,80
8.74
Llama-3, 1-88 URIAL
4.35
3.25
3.95
5.95
9.00
6.95
8.00
7.60
6.13
Llama-3.1 1-2.1-188-10 1-88-Instruct
6.40
6.50
5.70
8.78
9.80
9.00
8.60
9.20
8.00
Mistral-7B-x0.1 URIAL
4.60
3.40
4.90
7.75
9.08
7.65
8.28
7.75
6.67
Mistral-7B-B ---tsll----nttuu--vv.....
4.35
3.95
6.30
6.75
9.45
7.45
7.70
6.85
Mistral-7B-v0.2 URIAL
3.80
3.35
4.50
7.45
8.95
6.70
7.43
7.98
6.27
Mistral-BB-lnstruus-v0..2
5.45
6.50
9.90
9.30
7.64
Mixtral-&22B-vv0 URIAL
5.25
9.80
8.30
7.71
7.10
7.03
7,00
9.10
9.65
8.90
9.65
9,70
8,52
GPT-4-Base URIAL
5.55
5.98
6.90
8.20
5.93
6.90
6.45
6.50
GPT-4 (March 2023)*
8.55
6.80
9.00
9.38
9.95
8.90
9.70
9.65
8.99
23

---

Published as a conference paper at ICLR 2025
0.6
6.16
5.87
5.93
5.44
0.6
7.32
7.12
6.92
7.22
-6
6
Top-P
Top-p
0.8
6.14
6.03
6.15
5.21
7.23
7.21
7.40
7.44
1.0
6.14
6.04
5.91
3.88
7.34
7.27
7.26
6.15
2
0.0
0.3
0.5
1.0
0.0
0.3
0.5
1.0
Temperature
Temperature
(a) No ICL. repetition penalty = 1.0
(b) ICL with URIAL repetition penalty = 1.0
0.6
6.22
5.97
6.07
5.12
0.6
6.97
6.86
6.84
6.70 6.70
6
Top-p
Top-p 0.3
0.8
6.35
5.83
5.66
4.32
6.95
6. 78
7.20
5.33
4
1.0
6.30
5.26
1.90
2
1.0
6.90
7.01
6.97
2.17
2
0.0
0.3
0.5
1.0
0.0
0.3
0.5
Temperature
Temperature
(c) No ICL/ repetition penalty
1.15
ICL with URIAL reepttiinn penalty 1.15
Figure 14: The 1st-turn MT-Bench scores of Llama-3.1-88 generations with and without URIAL in the context across different decoding schemes. mainly consider two pyper--arameters: temperature and top-p.
24


Reviews:
[{"review_id": "jQkZR4w3t2", "reviewer": "Reviewer_ZLSn", "summary": "This paper studies whether in-context learning can be competitive with instruction fine-tuning methods on instruction following tasks. The empirical results ablate many different components of in-context learning methods and the dataset composition. The results are comprehensive and yield several potentially useful results (e.g., how decoding parameters impact the performance). The overall conclusion is that in-context learning is worse than and not as scalable as instruction fine-tuning.", "questions": "- The abbreviation URAIL is not defined.\n- Line 139: If providing multi-turn examples improves performance at multi-turn conversation, why do the authors not study it in this paper?\n- The results in Figure 2 are not representative in my opinion since the confidence bounds are overlapped largely. \n- Line 320: You say including examples from SkillMix is better than including examples from URIAL. Does URIAL come with a curated dataset as well? \n- Figure 3: I don't get it clearly. Is \"Llama-3.1-8B-instruct\" a model with instruction fine-tuning? Are these rest of ICL methods based on Llama-3.1-8B base models?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The experimental results are comprehensive. The problem studied in this paper is of interest to the deep learning community today.", "weaknesses": "Writing is not clear. See below:\n- The author should have a brief introduction to URIAL since it's heavily used in the later sections when explaining the experimental results. The current version refers to URIAL's components with the assumption that the readers know URIAL in detail. The current explanation like Line 120-126 is not enough. What do you mean by stylistic examples? What do you mean by rules? What do you mean by \"begin with affirming the user queries?\" I suggest adding some examples in the main paper to help the readers understand the idea.\n- Similarly, what SkillMix datasets do has to be explained. \n- This paper presents many experimental findings in each section, but the messages of these findings were not highlighted. For example, Section 2.1 has the title \"Systematic Evaluation of URIAL\" and starts with a brief explanation of URIAL. When reading this section, I couldn't expect what you will be talking about in this section and I quickly get lost while reading a bunch of implementation details. Even after finishing the section, I couldn't get the takeaway message clearly. Section 2.2, in contrast, did a better job on explaining the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mvMXPPLTQK", "reviewer": "Reviewer_GajX", "summary": "An interesting empirical study of the existing URIAL method that allows alignment via ICL. Unclear whether the contributions are significant.", "questions": "- What do the authors see as the work's main contribution, apart from the empirical study of a previously introduced method?\n- See other aspects mentioned above", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The paper conducts a comprehensive empirical analysis of the URAIL method for aligning pre-trained LLMs via in-context learning.\nIt considers different datasets, LLMs and test scenarios of relevance.\nThe figures are well done and show interesting trends. The paper is well structured and easy to follow.", "weaknesses": "I am unsure about the exact contributions of this work. It appears mainly as an extended study of the URIAL method on (a) additional models, (b) additional datasets, and (c) more hyperparameters. A relevant contribution of the work is the greedy ICL prompt search, which shows significant performance improvements (i.e., higher performance with much fewer ICL prompts, i.e., with much fewer test-time compute requirements). However, the entire analysis is again primarily based on a single dataset (MT bench). The language sometimes could be more academic, consider e.g. “suffers from some heavy overfitting”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fAGJ6yZsLe", "reviewer": "Reviewer_rg7y", "summary": "This paper continues on URIAL work to deep dive into the comparison of ICL vs IFT. It revealed crucial insights to improve ICL performance, i.e. high quality examples and temperate as decoding parameters. On the flip side, it also shows the shortcoming of ICL, especially on subsequent turn in conversational benchmarks.", "questions": "For prompts ending up with varying output performances, what are their KL divergence?\n\nIf you repeat your experiments on instruct models instead of base models, what would the results be?\n\nHow does ICL cope with long-context tasks (summarization, coding, etc.) where context window limit the extent of examples?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper contributes valuable guidance to practitioners who often struggle between the choice of finetuning and prompting. The work is particularly useful for scenarios where obtaining training data is challenging. Also when all attentions are on instruct models, this paper shows that base models alone can be instructed and aligned, and made useful. This shall open the gate to lots of research projects too.", "weaknesses": "More attribution works are needed to answer why the outcome is what it is. For example, the poor performance of the second-turn in MT benchmark definitely needs deeper dive. Is it because of the formatting or lack of examples? What caused the ICL to lose grip of the first turn? \n\nConsidering that the authors work on small size open LLMs (except GPT-4), mechanistic study with logits or internal states shall definitely reveal more insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)