Implicit In-context Learning
Authors: Zhuowei Li
Zihao Xu
Ligong Han
Yunhe Gao
Song Wen
Di Liu
Hao Wang
Dimitris N. Metaxas
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
In-context Learning
Activation Engineering
Efficiency
Primary area: foundation or frontier models, including LLMs

Abstract: In-context Learning (ICL) empowers large language models (LLMs) to swiftly adapt to unseen tasks at inference-time by prefixing a few demonstration examples before queries. Despite its versatility, ICL incurs substantial computational and memory overheads compared to zero-shot learning and is sensitive to the selection and order of demonstration examples. In this work, we introduce \textbf{Implicit In-context Learning} (I2CL), an innovative paradigm that reduces the inference cost of ICL to that of zero-shot learning with minimal information loss. I2CL operates by first generating a condensed vector representation, namely a context vector, extracted from the demonstration examples. It then conducts an inference-time intervention through injecting a linear combination of the context vector and query activations back into the model’s residual streams. Empirical evaluation on nine real-world tasks across three model architectures demonstrates that I2CL achieves few-shot level performance at zero-shot inference cost, and it exhibits robustness against variations in demonstration examples. Furthermore, I2CL facilitates a novel representation of ``task-ids'', enhancing task similarity detection and fostering effective transfer learning. We also perform a comprehensive analysis and ablation study on I2CL, offering deeper insights into its internal mechanisms. Code is available at https://github.com/LzVv123456/I2CL.

Published as a conference paper at ICLR 2025
IMPLICIT IN-CONTEXT LEARNING
Zhuowei Li,
Zihao Xu, Ligong Han, Yunhe Gao,
Song Wen, Liu
Hao Wang, Dimitris N. Metaxas
Department of Computer Science
Rutgers University {<first name>. <last name> @rutgers.edde
ABSTRACT
In-context Learning (ICL) empowers large language models (LLMs) to swiftly adapt to unseen tasks at inference-time by prefixing a few demonstration examples before queries. Despite its versatility, ICL incurs substantial computational and memory overheads compared zero-shot learning and is sensitive to the selection and order of demonstration examples. In this work, we introduce Implicit In- context Learning (I2CL., innovative paradigm that reduces the inference cost of ICL to that of zero-shot learning with minimal information loss. I2CL operates by first generating a condensed vector representation, namely a context vector, extracted from the demonstration examples. It then conducts an inference-tine intervention through injecting linear combination of the context vector and query activations back into the model's ' residual streams. Empirical evaluation on nine real-world tasks across three model architectures demonstrates that I2CL achieves few-shot level performance at zero-shot inference cost, and it exhibits robustness against variations in demonstration examples. Furthermore, I2CL facilitates a novel representation of "task-ids", enhancing task similarity detection and fostering effective transfer learning. We also perform a comprehensive analysis and ablation study I2CL. offering deeper insights into its internal mechanisms. Code is available at https: /ttp::://githb.. www.
INTRODUCTION
In-context Learning (ICL) has emerged as prominent capability of large language models (LLMs) [Brown et al. 2020). It enables a swift inference-time adaptation towards new tasks by prefixing few demonstration examples prior the query (Wei et al. 2022; Dong et al. 2023). ICI, characterized by its adaptability, has prompted extensive research efforts aimed at optimizing the selection of demonstration examples, or prompts (Rubin et al. 2022 Sorensen et al. 2022; Wu et al.)
2023 Min et al.] [2022al inter alia), as well as mitigating their sensitivity to formatting, order, and
recency bias (Zhao et al. 202T Lu et al. 2022: Hao et al. 2022, inter alia).
Despite existing advances, current approaches predominantly focus manipulating demonstration examples within the token space, where context tokens are prepended to the query. This practice quadratically escalates the computation and memory demands with each additional token and is known to be sensitive to the selection and order of demonstration examples Zhao et al. (2021) Lu al. 2202): Dong et al. (2023). Consequently, it is challenging to apply ICL under constrained scenarios where computational and memory resources are scarce, and demonstration profiles are uncontrollable, hindering ICL's scalability and practical utility.
In this study, we explore to hamness the demonstration examples within the activation space, seeking for an efficient and robust alternative for constrained environments. Given decoder-only architecture. we observe that the primary burdens of ICL arise from the computationally intensive multi- attention mechanism, which fuses information between demonstration and query tokens, and the memory-intensive key-value caching scheme necessary for retaining contextual information]] These observations motivate us investigate the following two questions: there more abstract repre-
Withoott applying key-value cache, one needs to repetitively forward the same demonstration examples.

---

Published as a conference paper at ICLR 2025
sentation of demonstration examples? And, can we integrate such information into models without resorting to attention mechanism?
Our findings suggest that both objectives attainable by condensing demonstration examples into compact vector representation and reintegrating their functionalities within the model's activation space. Instead of concatenating demonstration tokens before the query tokens, independently extract a demonstration vector from each example. These demonstration vectors are aggregated in a permutation-invariant manner to form unified context vector. During inference, linear combination of the context vector and the query activations is injected into the model's residual streams as the substitution of the original output activations. We term above scheme Implicit In-context Learning (I2CL), alluding to the absence of explicit demonstration tokens at querying stage.
I2CL offers unprecedented merits. By condensing demonstration examples into a unified vector representation, I2CL needs to cache only fixed amount of activation vectors, independent to the number of demonstration tokens. I2CL maintains zero-shot inference speed through merging information between demonstration examples and queries using only linear operators. We evaluate I2CL across three open-sor open-source LLMs on nine roal-world text classification tasks, where it significantly outperforms zero-shot counterpart and other compa- rable methods. I2CI. achieves results on par with few-shot learning with zero-shot inference cost (see Fig. [ Importantly, I2CL demon- strates robustness against the variability of demonstration examples, and facilitates natural representation of task-ids that can effectively signify task similarities and foster transfer learning.
5n cadho)
Ceched Perameters (t)
Figure 1: Upper-left is better. Comparison accuracy, infer- ence speed and cached mem- ory of different methods on Llama2-7b.
Our main contributions can be summarized in three-fold: (1) We introduce 12CL, novel framework that effectively integrates a min- imal set of demonstration examples within the activation space. By decomposing conventional ICL into two stages: context vectoriza- tion and context-vector injection, I2CL attains few-shot level performances at zero-shot inference cost. (2) We empirically validate the robustness of I2CL against the variations (i.e.. choices and order) of demonstration examples and uncover a natural generation task-ids, upon which we further propose transfer learning strategy that enhance performance
tasks based existing anchors. (3) We conduct a comprehensive analysis and ablation to thoroughly examine each component and design choice of I2CI., thereby shedding light on I2CL's internal working mechanisms.
2
METHODOLOGY
2.1 PRELIMINARIES
Residual Stream We adopt the mathe- matical interpretation from Elhage et al. (2021), viewing the hidden states across layers and at each token position resid- ual stream. This perspective treats each attention head and multi-layer perceptron (MLP) module as read-out and write-in op- erators that engage with residual streams, facilitating the addition and deletion of in- formation within residual streams. At a given layer and token position I, the resid- ual stream rf is defined recursively as:
MLP
M.P m/
MHsk of
MHA of
Srainesshall erean of
Reiidall dbaas of lif
dlerestaate addissx
Figure 2: A schematic overview of I2CL, including (1) single layer for illustrative purpose.
r/ = ri + af + mi;
where af denotes the integrated output of the multi-head attention (MHA) module, and m signifies the MLP's output, contingent af. Within this framework. MHA promotes the information fusion across residual streams, whereas the MLP functions akin to an associative memory (Geva et al. 202T Dai et al. 2022) that retrieves information encapsulated within its weights.
2

---

Published as a conference paper at ICLR 2025
In-context Learning Given unseen task, ICL assumes the existence of set of N demonstration examples D = (d,, .g...., .,. dy J. each comprising an instructional pair d, = (I,, ys) that includes an input sentence and its corresponding label. ICL operates by first concatenating the demonstration set D with the test query Zig. forming an input prompt p = [TD, my]- The objective is then to predict the correct response Uy from finite set of discrete labels C via Yg arg maxyec P(y p)) We acknowledge the versatility and broader implication of ICL, especially when connecting to the generic definition of prompt engineering. In this work, we consider the standard few-shot classification task as our testbed and focus on this scenario in the following sections.
2.2 CONTEXT VECTORIZATION
To overcome the inefficiencies associated with key-value caching system, we isolate the reasoning process of demonstration examples from that of the query, and introduce an independent vectorization process for each demonstration pair d,. Extracted vectors are merged in the activation space. Specifi- cally, we define a function V, of generating a vector representation for each demonstration example: d, = (d:), followed by function F applied to aggregate extracted demonstration vectors into unified context vector e = (dd ). Note that F is designed to be permutation invariant, ensuring a unique vector representation for given set of demonstration examples.
In our implementation, function V is parameterized by a pre-trained LLM and its corresponding tokenizer, and operates by collecting output activations of both MHA and MLP modules across layers at the position of end residual stream:
d, = (a m,
Here, e denotes the last token position that is responsible for next-token prediction, and L represents the total number of layers. For the aggregation function, we compute an element-wise arithmetic mean of each vector component across the demonstration examples:
{āī, [ai mil
where a- = N E af and m N Di= m
The rationale of these designs is twofold. First, we argue that the end residual stream encapsulates the essential information from each example. This is supported by the dynamics of next-token prediction
and empirical findings from recent studies (Hendel et al., 2023 Zou et al. 2023 Todd et al. (002)..
Second, inspired by the linear representation hypothesis (Park al. 2023), we premise various demonstration vectors can undergo linear transformations at different levels of abstraction.
2.3 CONTEXT INJECTION
Having established unique vector representation, v, for the demonstration set, I2CL. seeks to enhance zero-shot performance by integrating the contextual information. While conventional methods leverage the multi-head attention module for this purpose, I2CL utilizes a simpler, yet effective, linear operation to augment the query activations with context vectors.
Taking residual stream rf as an instance, instead of directly adding the output activations from the MHA and MLP to rf. we inject linear combination of these activations with context vectors:
ri = - (ffaf + 3 at) + (A Here, A", _3, Am , are four layer- scalars used to adjust the proportion which the context vectors and query activations are blended. By default, we apply this information fusion process to all residual streams of a given query. Letting X = 0 and f (omitting subscripts) replicates the original shro inference process. Note that it is critical to extract and aggregate information at both MHA and MLP modules (see Section 3.4p, suggesting a distinct yet complementary functionality of MHA and MLP.
mf + 3 mi), EE [1, 4),1 E [1,
The proposed context injection method incurs minimal computational overhead, involving only two scalar multiplications and an element addition, a process more efficient than the attention mechanism. As a result, the inference speed of I2CL matches that of standard zero-shot learning. We refer to Figure for schematic illustration of the context vectorization and injection procedures.
IInpt x, also includes formatting texts like: "Question:", "Answer Type:". We abuse the symbolic notion of vector denote set for the case of interpretation.
3

---

Published as a conference paper at ICLR 2025
2.4 NOISY SELF-CALIBRATION
Thus far, we have presented how to vectorize and reintegrate the contextual information in an efficient and attention-free manner. Herein, we elaborate on how configure the scale of linear coefficients. is not uncommon to set those weight scalars as hyper-parameter (Turner et al., 2023 Liu et al. 2024), and manually adjust them for each task in trial and error fashion. order to achieve an adaptive and nuanced control over the information fusion process without buman-in-the--owwe estimating the linear coefficients с = , , , , , 11 3i using a gradient-base based optimization applied to the same set of demonstration examples. Our estimation of linear coefficients does not require any additional data beyond the given few-shot examples.
we propose
Concretely, we initialize À 0.1, B 1.0 to promote modest initial addition of information, and update these coefficients by minimizing the perplexity of label tokens:
C
Σ (a.y)En
log P(y 2, U,U.C).
where P-)) denotes the induced probability distribution over the entire vocabulary at the end token position from the last layer. To bolster the robustness and adaptability of estimated linear coefficients to potential downstream variations, we perturb the residual streams with Gaussian noises ~ (O, during the calibration phase:
of ri- 1 + (Afa + aaa))
of - of + >|l0il12 x n, rf = of + (A mi + m)),
(6)
rf = r/ + + x n7
where y scalar employed to modulate the intensity of the noise, and || ||2 denotes the L norm. The o represents the intermediate state of residual stream.
Given above formulations, only few linear coefficients (totaling 4L) are updated during the cali- bration phase, rendering this process remarkably efficient (consuming 1-2 minutes on single A100 40G). Critically, the calibrated linear coefficients are demonstration agnostic- they require calibration only once per task and exhibit excellent generalization ability to unseen demonstration examples (see Section 3.3). Moreover, these linear coefficients, though are lightweight, can function as effective task-ids that fostering task similarity detection (Fig. and transfer learning (Table 3).
EXPERIMENTS
In this empirical section, we begin by detailing the architectures, tasks, and configurations used in study, followed by comparative analysis of I2CL against other relevant techniques. We then delve into the formation of context vectors and the characteristics of the calibrated linear coefficients, demonstrating the robustness of I2CL., as well as identifying the function of calibrated coefficients task-ids. This section concludes with an extensive ablation study to underscore the inner working mechanism of I2CL. We refer readers to Appendix Coror additional experiments and analysis.
Models We evaluate 12CL on three open-source architectures: GPT2-XL (Radford et al. 2019). GPT-J-6B (Wang& & Komatsuzaki 2021), and Llama2-7b (Touvron al. 2023). We selected these models based on their suitability for our computational resources and their range in size from relatively small (1.5B) large (7B). We report results under Llama2-70 the main content and defer results of other architectures to Appendix C.1 Consistent trends are observed across all architectures.
Tasks We first take the four tasks used in Wang et al. (2023), including sentiment analysis: SST- 2 (Socher et al.] (003)) emotion classification: EmoC. (Chatterjee et al.. 2019). question classification: TREC (Voorhees & Tice| 2000), and topic classification: AGNews (Zhang et al., 2015). We then enrich experiments with five additional datasets, encompassing 5-way sentiment analysis: SST- 5 (Socher et al. 2013), movie review classification: MR (Pang & Lee| 2005), 14-way topic classifica- tion: DBPedia (Zhang et al. 2015), subjectivity status categorization: Subj (Pang & Lee 20044, and hate speech detection: hate speech18 (de Gibert et al.) 2018). We employ the HuggingFace version of the data (Lhoest et al. 2021) and sample 500 data points from the validation/test set for evaluation.
4

---

Published as a conference paper at ICLR 2025
Table 1:
Comparison between I2CL and baseline methods on Llama2-7b. The best results are highlighted in bold, and the second-best best results are underlined. In addition to practical gauge of the inference speed and memory usage (see Fig [T]. we include an examination of cached parameters. Here, M. D. and L denote the number of demonstration tokens, model dimension, and architecture layers, respectively, P indicates the number of extra learnable tokens in the Soft-prompt method, and 1/K represents the compression rate of corresponding context-compression method.
Method
SSE-2
SST-5 1977
TROC 093
AGNees (50
Suby
HateSpecchi8 (93
DDPeedi (9)
EmoC (5)
MK
Aug.
cached
(9)
Zanosha
83.00
27.00
50,00
70:20
51.00
54,20
72.00
41.90
75.60
58.18
Few-shot (KCL)
91.464144
41.72
77.32
60.0862.00
T2 30
70.244500
9664464
75.4841.
58.24uc.9
70.57
2MDL
Noise vecioe
29.88.324
20. 20.561084
20 12:12:11:0
21 22
45 21:6
50 39.861104
7.25.037
25.76 35.76
20.125035
34.61
2DC
Lacel-ancher
83.322309
27. 08:421
S3.72-16
53 000200
61:22:005
51. 17
50. 12:11:
51.00
08. 20
20M/K DD D
Tadesector
81 41.44:47
25. 90
05 085 on
56. 2i:
50. 10.18 x
50.37
00:16
IC
86 9
35. S3.18trss
03. 23.84
72. 10
56 11
60 00.56i= 9
75/5 s
09. 16ar
64.01e1
61.44
DV.
IECL (ours)
87. 68
39-12 80
78. 93
85, 15
73 73.84
69 88
90. 10:116
13.72+
87.08
75:12
AatoComp.
92.4 44
25.84-4
62:52.AM4
16.35
60. 18.33
58.2.43
92 22.68.196
29.9615u0
62.76.134
60.94
JCAE
91.00 01.641146
38.8
50,921538
50.48.122
50.92.M1
65-481718
02.081186
54.44145
39.48.144
64.83
PIM/ DDL
CEPB
74.28.39
36.2..06
53,48.1322
78,0013.0
39.12138
61.72153
S7,24.12
42.281558
52,96.1.4
64.08
PLM/X J.L.
Experimental Setup The following configuration is applied to all experiments unless otherwise specified. For each task, we randomly sample five demonstration examples per clas following the practice described in Wang et al. (2023) avoid majority label bias (Zhao al., 202I), and provide a fairly strong few-shot performance. No instruction is further applied to describe the task. Input sequences are formed using simple manually designed templates (included in Appendix A For evaluation, we report the macro-average accuracy across nine tasks, computed under five random seeds. For the calibration process, we optimize linear coefficients for 100 epochs on the same demonstration set using the Adam (Loshchilov & Hutter 2019) optimizer. The learning rate starts at 10-2 and anneals to 1 x 10-5 according to a cosine scheduler. This calibration profile is applied uniformly across all architectures and tasks without tailoring.
3.1 BENCHMARKING I2CL.
I2CL Achieves Few-shot Performance at Zero-shot Inference Cost I2CL is designed to enhance zero-shot performance. providing an alternative for ICL under constrained scenarios. As shown in Table T I2CL. significantly outperforms the zero-shot counterpart by 17% in absolute accuracy and is only marginally behind (by around 1% on average) the few-shot learning. Note that I2CL. consumes only zero-shot inference cost in terms of both memory usage and inference speed. An interesting observation arises from the Subj task, where the instructions (i.e., demonstration examples) do not function effectively with ICL, yet are well adhered to under I2CL. We hypothesize that this phenomenon is due to the inherent properties of the pre- ve-trained LLM, causing to excel in certain tasks while lagging in o and our proposed noisy self-calibration strategy can effectively rectify this deficiency.
Comparison with Inference-time Methods w/o Additional Data To further validate the efficacy of I2CL, we compare it with several comparable methods: (1) Noise vector: replacing the context vector with random noise assess its necessity; (2) Label-anchor: a token reduction method from Wang et al. (2023), using formatting and label tokens as anchors; (3) Task-vector: task-vector (Hendel ct al. 2023) also improves zero-shot performance without introducing additional computational and memory overhead at inference. It requires a hold-out validation set to identify the optimal replacement layer for each task at test-time; (4) ICV: we adapt In-context Vector Liu et al. (2024) method for our scenarios treating query tokens as negative examples and answer tokens as positive examples. We manually optimize ICV's strength parameter for each task and report the best performance. Please refer to Appendix B for implementation details.
As demonstrated in Table replacing the context vector with random noise significantly degrades performance, highlighting the critical role of context vector. Label-anchor method exhibits decent upgrade over the zero-shot baseline, achieving results most comparable to I2CL. Nevertheless, its inference cost remains dependent on the length of demonstration tokens, with reductions proportional to their compression rate. Like I2CL., the task-vector method also enjoys zero-shot inference expenses; yet its performance is sensitive the downstream task and is, on average, slightly inferior to the
1 One exception is DBPedia, where we use only one example per class due to the limitation of GPU memory. As demonstrated in Table 11 zero-shot performance of Subj is much higher under GPT-J than I. Llama2-7b though the later one is generally considered more powerful.
5

---

Published as a conference paper at ICLR 2025
Table 2: Comparison between different PEFT-based few-shot fine-tuning strategies.
2
TREC (93)
AGNews (56)
Subt
HaseSpecch18
DBFedia (50)
MR
Aug. soc.
params. (K)
(53
(131
(163
(93)
(96)
(533
(5)
Prompt saning
24.24-200
55.200414
T8.001ta9
57.40
4
a5.08.s
LoRA
54.80
30.55
75.97
1009
501222
10.47
75.32:2
91. 50
1A3
89.00
TB. 4m
26.074400
93
73.00
I2CL (ours)
0.13
87.08484
29.323.22
78.56
85.48
73.84
00.8888877
90016
67.08822.
75.12
label-anchor method. Finally, clear improvement over zero-shot baseline can be seen by adapting ICV method for few-shot classification tasks. Notwithstanding the boost, it necessitates manual effort for hyperparameter selection, which limits its applicability for scenarios that favor automated pipelines. Compared to the above methods, I2CL achieves the best performance on all tasks with neither manual intervention nor task-specific hyperparameter selection.
Comparison with Inference-time Prompt Compression Methods Another line of work that leverages external large-scale datasets to learn an amortized context compres- sor can also be applied to ICL scenarios. To this end, we compare I2CI. with three SoTA prompt compression meth- ods: Compresrrsss Chevalier et al. (2023). ICAE Ge ett al. (2023), and CEPÉ Yen et al. (2024). Here, we di- rectly apply their released pre-trained models on our tasks to avoid implementation bias. As shown in the bottom section of Table I all three methods yield a decent im- provements over zero-shot baseline, demonstrating their effectiveness in compressing context tokens. However, these methods produce mixed and fluctuating results across different tasks, likely due to their different training data and strategies. Instead of learning a universal compressor, I2CL opts for customizing a task-specific compression strategy at test time, rendering it additional adaptability for few-shot learning tasks. Nevertheless, we do not claim overall superiority of I2CL over prompt- compression methods as they are deliberately tailored for excessively long context compression which is orthogonal to the primary application of I2CL.
Scalig Tond of C.
[
Zero diock
Stnd Cless
Figure 3: Scaling trend of I2CL.
The design of 12CL involves lightweight training process (i.e., self-calibration). As such, we further establish comparison between I2CL and three representative PEFT methods: prompt-tuning (Lester et al. (02T), LoRA (Hu et al. (02T) T), and IA3 (Liu et al. 2022) to underscore the effectiveness and efficiency of I2CL. As shown in Table 2 prompt-tuning fails under such data scarcity, while LoRA and IA3 perform on par with I2CL. Importantly, I2CL achieves the best overall performance with approximately 100x fewer learnable parameters than prompt-tuning 1,000x fewer than IA3, and 10,000x fewer than LoRA. We refer readers to Appendix B for implementation details.
Comparison with Test-time PEFT Methods.
Scaling Property of I2CL Although 12CL is primarily designed for few-shot scenarios, it exhibits ICL peaks at around 5-shots and additional demon-
strong scaling properties. As shown in Fig. 3 stration examples do not necessarily bring further benefits. In contrast, I2CL can easily scale up to hundreds or even thousands of demonstration examples without performance degradation, and it readily surpasses few-shot performance under a modest number of demonstration examples. We note that using more demonstration examples increases the calibration cost.
3.2 ON THE FORMATION OF THE CONTEXT VECTOR
Context vector plays a fundamental role in the design of I2CL. Here, we conduct an in-depth analysis to investigate its properties and the factors affecting its functionality.
Deficient ICL a Deficient I2CL is well-known that ICL sensitive the choice of demonstra-
tion examples (Dong et al. 2023), even their order (Zhao et al. 2021] Lu et al. (022). In this context, we investigate whether poorly performing demonstration examples ICL will similarly affect I2CL. We sample 20 groups of demonstrations with random instances and orders, and identify the group with the poorest ICL performance on a hold-out dataset that is non-overlapping with both train and evaluation sets. Using these same demonstration examples, we perform both ICL and I2CL on the evaluation set. Referencing Figure (left), the poorly performing demonstrations lead to a severely degraded ICL performance (-7%), while I2CL performance remains largely unaffected (-0.5%). This result suggests that I2CL requires less careful curation of demonstration examples than ICL. We hypothesize that this robustness stems from I2CL's ability to extract task-relevant features that

---

Published as a conference paper at ICLR 2025
bxteson aro-sle, how-ltd
Hiuoaay fecura
75
15.12
71.5
65
W5
62
44.55
120L
0u1t4en Senoi
Evaluation of I2CL and few-shot learning under deficient demonstrations. The symbol denotes the results under deficient demonstration examples. "Unseen demo" refers the evaluation of calibrated coefficients on unseen demonstrations. Middle: Analysis of the influencing factors of context vectors. "Random-label" indicates random input-label mappings. "Random-ordes refers to the random permutation of words. "W/o format" signifies excluding the template tokens during the creation of context vectors. Right: t-SNE plot of context vectors. Each circle denotes a context vector generated using group of randomly sampled demonstration examples.
Figure 4: Left:
are invariant to surface-level variations in the demonstrations. To further corroborate this hypothesis, visualize the context vectors using t-SNE (van der Maaten & Hinton 2008) in Figure 4 (right). The visualization reveals that context vectors from different demonstration groups cluster tightly by task, indicating that I2CL captures consistent task-specific representations despite variations in the demonstrations.
Influencing Factors of the Context Vector Generation We have shown above that context vectors are robust against the variations in demonstration examples. Here, we explore various factors that may impact the context vector's functionality. Inspired by counter-intuitive findings reported in (hhag et al. 2022 Min et al. 2022b), we experiment with two variations of demonstration examples. Random-label. pairing each input sentence with random label from the task, rather than the ground-truth truth label. Random-token: randomly permuting all words within a demonstration example. Another ineligible factor involves the inclusion of formatting tokens. To this end, we also evaluate on W/o-format: removing formatting words from a demonstration example, retaining only the input sentence and its corresponding label, e.g., "Review: This is a great movie. Label: positive. As revealed in Figure 4 (middle), input-label mapping relations have minimal impact on context vector formation, mirroring the observations made in ICL. However, these mapping relations are crucial for calibration purposes; using random input-label mapping during calibration undermines the functionality of I2CL. Unlike the phenomenon observed in (Zhang et al., 2022). word sequence holds essential statisties under causal architecture, and randomly permuting input words yields degenerated context vectors, leading to a clear performance drop. Lastly, formatting tokens contribute settaantilly--emvviii them results in noticeable performance degradation.
3.3 ANALYSIS OF CALIBRATED LINEAR COEFFICIENTS
With grasp the formation of context vectors, we delve into the properties of calibrated linear coefficients to enhance our understanding of the internal mechanisms of I2CL.
Linear Coefficients Are Generalizable Given the demonstration-depeeeee and calibration process, it is natural to consider 12CL as an inference-ting optimization framework. However, we demonstrate that the coefficients task. Concretely, we evaluate a set of calibrated coefficients on five new context vectors generated using five unseen groups of demonstrations. As exhibited in Figure (left, the calibrated linear coefficients generalize well to unseen demonstrations. We attribute this generalization capability two factors: (1) the inherent robustness of the context vector against token space variations, as validated in Section 3.2] and (2) the sufficiency of calibrated linear coefficients, independent of context vectors and query activation, uniquely represent task. We substantiate the second premise in the following subsection.
context vectorization
generalizable and require calibration only once per
Calibrated Coefficients Embed Task Semantics Here, we investigate whether calibrated coef- ficients alone are adequate to serve as task-ids. In pursuit of this goal, we concatenate calibrated coefficients across layers to form a one-dimensional vector, which we then visualize using t-SNE. As illustrated in Figure 5 (left), the calibrated linear coefficients are tightly clustered for instances associated with the same task and fall apart otherwise, providing complementary evidence for their good generalization capability. One exception comes from the proximity between MR and SST-2. This phenomenon is not unexpected, as both tasks originate from the same underlying distribution (i.e., rotten tomatoes movie reviews), suggesting that similarities among calibrated coefficients may
7

---

Published as a conference paper at ICLR 2025
eit plee Eorsted clentt
losks
Trrrser Netormaan
2.5
Calibrated confficients for
ETCC
2.0
dopeda
1.0
ogrows
0.0
hate.
Figure 5: Left: I-SNE visualization of calibrated linear coefficients. Each circle denotes a runtime with random seed. Middle: This image displays the transfer results among various tasks, Each row represents a source task and each column denotes a target task. Red and blue colors signify positive and negative transfer outcomes, respectively. Right: This plot shows the calibrated linear coefficients for SST-2. X", 8 , 3 are the layer-wist wise coefficients described in Equation 4
indicate potential transferability among tasks. To explore this further, we transfer the context vectors and calibrated linear coefficients from source task to various target tasks. We then measure the differences between the transferred results and their respective zero-shot outcomes to identify both positive and negative transfers. Figure 3 (middle) shows that performance on MR can be effectively enhanced by transferring context vectors and calibrated coefficients from SST-2, and vice versa. Similarly, both SST-2 and MR benefit from transfers from SST-5, likely due their shared focus on sentiment analysis. An intriguing aspect of our findings is the asymmetry of transferability among different tasks; a successful transfer in one direction does not necessarily forecast the opposite (e.g.. transfer between HateSpeech18 18 and SST-2).
Enhance New Task with Existing Anchors I2CL posits two pivotal features distinet from typical ICL: (1) An interface that facilitates vector arithmetic; (2) A set of linear co- efficients that act as task-ids, laying a foundation for transfer learning (Zhuang et al.) 2021). Here, we present an transfer learning algorithm based on these features. Let C1,-.., Cr represent calibrated coefficients for existing tasks, and Crew for the new task. Associated with these coefficients, respectively, We compute the cosine similarity between Cnew and each c; for i ,....,,N N and retain indices / {i cos(Cnow, C.) h} according to pre-defined threshold h. The cosine similarities for the indices in are then converted into a probability distribution using the softmax function:
context vectors v1,-..; UN, and Unew-
Table Transfer learning result. Only tasks having more than one similar task (according to h) in our task curation exhibited.
exp(cos(Cnew: C/)/T)
I2CL
P(t) Σeer exp(cos(Cuww: C/)/T)'
Vi E I,
(7)
Task
Wi transfor (%)
W/ transfer (%)
SST-5
39. 39.12
43. 24:30
MR
87.68
89.99
where 7 the temperature. Finally, we reinitialize Vsew and Cnew using the weighted average of retained context vectors and coefficients: taag
Eier P(i)u;, Cavg
Eier P(i)c,, and perform another round of calibration. We refer to Appendix B for detailed algorithms and implementation. Empirical results in Table demonstrate a clear benefit of the proposed transfer learning method.
Injection Dynamies Thus far, we have demonstrated that a few static scalars, conditioned on appropriate context vectors, can successfully execute wide range of tasks. Herein, we delve into how context vectors are injected. Using the AGNews as an example, we examine the coefficient values across different layers (additional plots in Appendix C.4). One reasonable speculation postulates gradual addition context vectors to the residual streams. Nevertheless, observations from Figure 5 (right) reveal a more nuanced control over the information injection process. Instead of monotonically adding (+) more information to the residual streams, I2CL also allows the deletion (-) of information at certain layers, with coefficient values fluctuating across different layers.
3.4 ABLATION STUDY
In this section, we conduct comprehensive ablation study the module, layer, and token position of injection, as well as the noise scale. We also explore various vectorization and injection formulas to highlight the rationale of our designs.
Target Module I2CL extracts and injects activations at both MHA and MLP modules. To identify the contribution of each module, we test using either MHA or ML.P independently, and we also
8

---

Published as a conference paper at ICLR 2025
Table 4: Target modules,
Table 5: Target layers.
Table 6: Injection position.
Name
Accuracy (%)
Name
Accuracy (9)
Name
Accuracy (%)
Zero-shot
68.1
Zero-shot
58.13
Zero-shot
MHA
06.97
Farly
68.18
Random
59.86
MLP
70.27
Middle
64.07
First
62.14
Hidden state
56.80
Late
61.83
Last
66.75
MHA+MLP (ours)
73.12
All (ours)
75.12
All (ours)
73.12
Table 7: Noise scale.
Table Injection formula.
Name
Accuracy (%)
Name
Accuracy (%)
Zero-shot
68.13
Zero-shot
58.13
7 0.0
72.23
Av + a.
63.63
= 0.01
10.53
(Av + (1 Ala) x
5,8
71.39
7 0.001 (ours)
75.12
Av + SS (ours)
75.12
consider leveraging the hidden state at each layer. As shown in Table 4 extracting and injecting context vectors at either MHA or MLP proves beneficial, with MLP showing a clear advantage. We hypothesize this is due to the additional engagement of information stored in the MLP weights. However, targeting the hidden state does not lead to any improvement, likely due to the accumulation effect which complicates the optimization process within self-calibration stage.
Target Layer I2CL encompasses all layers in large language model (LLM), climinating the need for specifying layer indexes. This model-agnostic approach not only simplifies the setup but also enhances performance, as evidenced in Table 4 We divide the model into three middle, and late each containing one-third of the total layers. We then apply I2CL only within the target layer range. The middle and late layers prove more effective than the early layers, and injecting across all layers provides a clear performance boost, highlighting the importance of fusing information at all levels of abstraction.
Injection Position By default, I2CL. injects context vectors into all residual streams during inference. To justify this choice, we test injections only random, first, and last residual streams. As shown in Table and aligning with common intuition, injecting at the end residual stream yields the largest improvement compared to other positions, although it still shows significant gap compared to injecting at all residual streams.
Noise Scale We evaluate the impact of noise strength during the calibration phase. As demonstrated in Table 7 an appropriate noise scale leads to a clear performance gain. Conversely, an improper strength can disrupt the propagation of information inference, resulting in deteriorated outcomes. We empirically identify т = 0.001 as a suitable scale and have applied it across all tasks and architectures.
Injection Formula 12CL utilizes a linear combination to blend context vectors with query activa- tions. Let denote the context vector and a indicates activation: we use Àc + ,3a. One simplification is to view the injection process as solely adding the context vector to the activation: Xe + a with as done in Liu et al. (2024). Another common formula involves constraining the sum of linear coefficients to one and allowing a separate scale factor: (Ao + (1 A)a) x 8, 3 > O. According to T linear combination with no constraints achieves the best overall performance. Therefore, we conjecture that it is critical to allow not only information addition but also deletion, i.e., permitting a negative sign for the linear coefficient, and to scale each vector independently. Observations in Figure 5 (right) corroborate this conjecture.
4 RELATED WORK
Understanding In-context Learning Besides enhancing ICL., the exploration of ICL's internal mechanisms has attracted significant research attention. Akyüüek et al. (2023) draw parallels between ICL and gradient descent in linear regression tasks, suggesting a fundamental alignment with classical optimization methods. Complementary perspectives from Von Oswald et al. (2023) and Dai et al. (2023) conceptualize ICL form of meta-optimization, further enriching our understanding of its operational basis. Concurrently, Xie et al. (2022) interpret ICL through the lens of implicit Bayesian inference, proposing probabilistic foundation for the learning process. Wei et al. (2023)

---

Published as a conference paper at ICLR 2025
and Olsson et al. (0022) respectively attribute ICL's capabilities to the establishment of input-label correspondences and the identification so-called induction heads, highlighting the intricate interplay between data representation and model interpretability. Most recently, Labcl-as-Anchors (Wang et al.) 2023) inspects ICL from an information flow perspective and leverages anchor tokens to perform token reduction. Similarly, I2CL. also enhances the efficiency of ICI., but distinguishes itself by circumventing the need for caching latents of anchor tokens or employing multi-head attention, thereby reducing the inference cost to that of the zero-shot.
Activationreprrsentation Engineering An emerging research field, termed activa- tion/representation engineering. closely relates to our study. Recent endeavors by MMruulo et al. ) Turner et al.] (2023); Zou et al.] (2023) have unveiled the phenomenon of steering vectors, which can derived from positive and negative text pair and used to steer the content generation of L.MSs. Steering vectors can also be learned via gradient descent (Subramani et al.
2022). Liu et al. (004)) applies these insights to bolster L.M safety while [i et al. (2023) explores their utility in eliciting more truthful responses from LLMs. Steering vector has also been adapted to multi-modal domain to reduce hallucination (Li et al. 2025). To better understand the inner mechanism of activation engineering, the linear representation hypothesis has been studied and discussed in Li et al. (202T); Hermandez et al. (2024); Park et al. (2023). Central to this discourse, the idea of task/function vector (Hendel et al. [023 Todd al.) 2024) resonates with the core premise of I2CI.. Both methodologies extract task/function vectors from the demonstration examples and use them to improve zero-shot performance. I2CL stands out not only due to its superior performance, but also through its simplicity, namely, its avoidance of paired demonstrations, and the need for task- or prchitecture-speiife hyperparameter tuning, such as selecting attention heads through causal mediation or determining target layers via extra validation sets.
Prompt Compression for LLMs Prompt compression (Chang al. 2024 is also related to our work, as it shares similar goal of improving the inference efficiency of LLMs. However, prompt compression methods emphasize on reducing the length of excessive contexts by learning an additional amortized compressor (Ge et al., 2023: Chevalier et al. 2023 Yen et al. 2024). while I2CL. transforms the standard few-shot learning approach into zero-shot manner at test time, treating the LLM itself as a context vector generator. Most prompt compression methods operate by compressing a long prompt sequence into a set of compact soft prompts, leveraging the attention mechanism to further propagate context information. In contrast, I2CL generates context vector from the activation space and intervenes directly in the residual streams.
LIMITATIONS
12CL is subject to several limitations. First, we confine the scope of this initial exploration to standard text classification tasks, leaving more sophisticated tasks for future research. It is non-trivial to further extend I2CL to the realm of open-ended generation tasks and those involving multi-hop reasoning processes. Second, I2CL necessitates access to and caching of intermediate activations from language models, which may not be feasible with ofate-of-the-ar commercial models (c.g., GPT-4, Gemini, Claude3). Thirdly, limited by the computational resources, evaluated I2CL modest-size models, and further scaling the evaluation to commercial size models could yield additional insights.
CONCLUSION
In this study, we introduce Implicit In-context Learning (I2CL). simple and novel framework that integrates minimal set of demonstration examples within the activation space of LLMs. Diverging from ICI. I2CL eliminates the need for caching the latents of demonstration tokens and replaces the non-linear information fusion process (i.e., attention head) with linear operations. Therefore, I2CL reduces both computational and memory expenses during inference to that of zero-shot level. Moreover, 12CL is validated to be robust against token space variations, and it facilitates novel representation of task-ids which enhances task similarity detection and fosters transfer learning. Empirical evidence on nine real-worls tasks across three different models suggests the potential of 12CL as a more efficient and robust alternative to ICL for constrained scenarios. Through a set of in-depth analyses and ablations, also shed light on the internal working mechanics of I2CI..
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We extend our sincere gratitude to Prof. Diyi Yang for her constructive suggestions on the motivation and empirical design of this study. Her expertise and insights were invaluable to this research.
REFERENCES
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models, 2023.
Yu Bai, Heyan Huang. Cesare Spinoso-Di Piano, Marc-Antoine Rondeau, Sanxing Chen, Yang Gao, and Jackie Chi Kit Cheung. Identifying and analyzing task-encoding tokens in large language models, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020, URL /ttpttppss//pppeeeisss.......... cc/paper files/paper 3000FFL2224570000000004111111222
few-shot
wwwp:a2fo6aaaaarr.aaeeamm Paper.
Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Tong Xiao, and Jingbo Zhu. Efficient prompting methods for large language models: A survey. arXiv preprint arXiv:2404.01077, 2024.
Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. SemEval-2019 task 3: EmoContext contextual emotion detection in text. In Jonathan May. Ekaterina Shutova, Aurelie Herbelot, Xiaodan Zhu, Marianna Apidianaki, and Saif M. Mohammad (eds.), Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 39-48, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics. doi: 10.1863//1//S192000 URL
httpss://cclannhooooy..ooor
2005
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. arXiv preprint a3Xiv:2305. 14788, 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18853/v 1/2022..//222.lllloooo 2022.ac1- a01-2aclllong 581.
URL https: ac htwpaa/oaanttoooo..ooooooooo..o. org/
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma. Zhifang Sui, and Furu Wei. Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Associa- tion for Computational Linguistics: ACL 2023, pp. 4005-4019, Toronto, Canada, July 2023. 10116555/10003mmmngnaaa.a.... aindings-acl.247.. URL
Association for Computational Linguistics. doi: 10. 10.18653/v1/2023
http://2222.........eeeee www.diiinnnnnin..mee. acl. 247
Ona de Gibert, Naiara Perez, Aitor García-Pablos and Montse Cuadros. Hate speech dataset from white supremacy forum. In Darja Fišer, Ruihong Huang. Vinodkumar Prabhakaran, Rob Voigt, Zeerak Waseem, and Jacqueline Wernimont (eds.), Proceedings of the 2nd Workshop on Abusive Language Online (ALW2). pp. 11-20, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10. 11553//11/11-///
URL http:::////........... /ntho logy
org/w18-5102
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey in-context learning, 2023.
11

---

Published as a conference paper at ICLR 2025
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1:1, 2021.
Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. arXiv preprini urXiv:2307.0694. 2023.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francing Moens, Xuanjing Huang. Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18633/1120211 http:///222..mmmmminnn....mm wmn.ppmain.466... URL https ma n. 446
//aclantho Logy. org 2021. emmnp-maaniee
Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting:
Scaling in-context learning 1,000 examples, 2022.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In
Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2023, pp. 9318-9333, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/11/2000 1t133///1/222.//// http:/02....ninngnemmm..22222 URL https:
//aclanthology.. wrg.yy/rrg//22/22 -indings-emnnn 624
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas,
Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations, 2024. URL https
/openreview..oo net wwwp://wwweweooeeteeeeeeeeeeeeee /t 7LU2s 14KE
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprins arXiv:2106.0960 2021.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1118633//12021.emmm maimlp-main.243. URL org/2021, emnle-ma wt..mminn................m
https://aclanthol.e
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario $aško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Heike Adel and Shuming Shi (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 18.53vv3/v 12/22221.2mmll wwwpp./.mmmpppmm.21111 URL httpt:ttts//.cm http://aaaaaathog...o/ooo /rg/ 2021 emn www.wmwwww.....m..mmmmmmmmmmmm.. demo.22 .11
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceed- ings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Prucessing (Volume I: Long Papers), pp. 1813-1827, Online, August 2021. Association for Computational Linguistics. doi: 18.18653/v1/ 2021 lonl.acl-long. 143.
hatp//ccaanoolooooooooooooooo wow.oolooy...oo www. 2021 aall-long...... 143
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from language model. In A. Oh, T. Neu- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 41451-41530. Curran Associates, Inc.,
12

---

Published as a conference paper at ICLR 2025
2023. URL https://proceedinngs http:://oooooooooooooooooo.eeeaa http://prrrriileeeeepepe///////
pdf
Zhuowei Li, Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and Dimitris N Metaxas. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. arXiv preprint arXiv:2502.03666, 2025.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficent fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950- 19600-995,.000 2022.
Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- Learning Representations, 2019. URL Bkg6RicqY7
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.1866..1. 556. URL /RLLstttt///ccaattiioo
org.orgg/////./ 2tt. acl- -ong. 556
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. A mechanism for solving relational tasks in transformer language models, 2023.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5316-5330, Dublin, Ireland, May 2022a. Association for Computational Linguisties. doi: 10. 1...0.655.8. 1.d55//1/2022..a--oon.....3 URL
httpps//aalaarttoooooooooo222222
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?, 2022b.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, 7ac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022.
Bo Pang and Lillian Lec. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Associa- fion for Computational Linguistics (ACL-04), pp. 271-278, Barcelona, Spain, July 2004. doi: 00.3115/1218955. 121 19990. URL
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. Kevin Knight, Hwee Tou Ng, and Kemal Oflazer (eds.), Proceedings of the 43rd Anmual Meeting of the Association for Computational Linguistics (ACL'05), pp. 124, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. doi: 10.3115/1219840.1 12198555
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. Causal Representation Learning Workshop at NeurIPS 2023, 2023.
URL
wtt.eewww.eeeeeeeeeeemeeeeeeeeee Eorumorriiioooom..... id- -teiiid--PPP-- O0J
Alec Ilya Sutskever. 2019.
Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, and Language models are unsupervised multitask learn URL
ers,
https: www. nt www.nnnncccco.cc..c att.oarr/orrppp/////////
13

---

Published as a conference paper at ICLR 2025
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning retrieve prompts for in-context learning. Marine Carpuat, MatiieCatherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 2655-267511 2671, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.
www.://.a22222222.mmmm main. URL naw.:222.aaammmmm.mm 1n. 191
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Liveseu, and Steven Bethard (eds.), Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL
1170
Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. An information-thhorett ap- proach to prompt engineering without ground truth labels. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.). Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguisties (Volume Long Papers), pp. 819-862, Dublin, Ireland. May 2022. Association for Computational Linguisties. doi: 12.18663/1220222acllloooo httpps:
URL
org. aat.222.cllleengg 60
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association Computational Linguistics: ACL 2022, pp. 566-581, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/002.. 1/2..////////222...n.n.n--n--- findings-acl.55 URL ptttps://allanthoooogg org/ 2022. 11002.finnd.a. findings-aci. 48
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau.
LLMs represent contextual tasks as compact function vectors. In The Twelfth International
Conference on Learning Representations, 2024. URL TRLhhtpps: htttts:://peereviieee net/ forum?
1d yMW
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu. Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDi- armid. Activation addition: Steering language models without optimization, 2023.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, (86)) 2205, 2008. URL CeLLhht::// ///ttp:///mi...../// wttp:opppg//s..oeeoooooo
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning.
14

---

Published as a conference paper at ICLR 2025
volume 202 of Proceedings of Machine Learning Research, pp. 35151 PMLR, 23-29/A Jul 2023. URL http:ttsts//poceeeeeeee http://ppoeeeeeng..e..o mlz wttp:pppee///2//ooooooooaaaaaaaa www.nnowwwwld2222 html
Ellen M. Voorhees and Dawn M. Tice. Building a question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '00, pp. 200-207, New York, NY, USA, 2000. Association for Computing Machinery, ISBN 1581132263. doi: 10. 10.1145/3455083455577 htttps:///do oxc/10.1145/345550 345577
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github, www.coom//....... gggg www.a/nnoloooo/otttteeeeeee httt://estttrrtforreeeererr jax May 2021.
Lean Wang. Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2023. Association for Computational Linguisties. doi: 10. 10.18653/v w126665//1/2223..om http://.222232mm/ppp ciiippmmii.6099 URL pm
Singapore, December
https: antho antthlllgy.......... wtt.g//22......
ma mnmiinn... 609
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large lan- guage models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL Survey Certification.
net/ wttp://wwweee/ooooiieeeeeee forrumiidsyzssss.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently, 2023.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Yc, and Lingpeng Kong, Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering, 2023.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In Internationaal Conference on Learning Representations, 2022. URL https: http://ewwe.eeeeeeeeeeooaraaaaaa
Howard Yen, Tianyu Gao, and Danqi Chen. Long-context language modeling with parallel context encoding. arXiv preprint arXiv:2402.166177 2024.
Hongxin Zhang, Yanzhe Zhang, Ruiyi Zhang, and Diyi Yang. Robustness of demonstration- learning under limited data scenario. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 1769-1782, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 2022.emmip--ainn 116
116. URL httph: ac http:////ccaantooooo.... org/
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL httpss://procceeinggg.nn wwwp:unnw...euu...ocooum cc/paper files//pape:22015/ file/ 2ttp://0f8Scc111133000oo01000a00
pdf
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12697-12706. PMLR, 18-24 Jul 2021. URL
mlr. www.ppresseeeo wwt.:///hhhhhhhhhaaaaa
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-76 2021. doi: 10. 10.100///00..22000300555
15

---

Published as a conference paper at ICLR 2025
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendrycks. Representation engineering: top-down approach to ai transparency, 2023.
16

---

Published as a conference paper at ICLR 2025
APPENDIX CONTENTS
Prompting Templates
17
Reproduction Details
18
B.
Implementation of Baseline Methods,
18
B.2
Implementation of Transfer Learning Method
19
Additional Experiments and Analysis
19
C.1
Results under GPT2-XL, GPT-J, and Llama3-88
19
Analysis on Synthetic Dataset
19
Apply I2CL over ICL
20
Additional Visualization
20
Visualization of In-task Context Vectors
20
Additional Discussion
21
D.I Potential Design Space
21
D.2 Broader Impacts
21
A PROMPTING TEMPLATES
Table Prompting templates and label spaces used in our experiments. (Sentence) and (Labell) are placeholders for the input sentence and its corresponding label. We exhibit only the template of single example for the illustration purpose, and multiple demonstration examples are connected by a newline character: 'In".
Dataset
Template
Label Space
SST-2
Review: (Sentence)
negative positive
Sentiment: (Label)
SST-5
Sentence: (Sentence)
terrible negative neutral positive
great
Sentiment: [Label]
MR
Review: (Sentence)
negative positive
Sentiment: [Label]
Subj
Sentence: (Sentence)
objective / subjective
Label: (Label)
DBPedia
Input: (Sentence)
company / school / artist / athlete politics/ / transportation / building/ nature/ / village animal / plant album / film / book
Label: (Label)
AGNews
News: [Sentence)
World / Sports Business / Technology
Type: [Label]
TREC
Question: (Sentence)
Abbreviation Entity / Person Location / Number
Answer Type: (Lahel)
HateSpeech18
Text: [Sentence]
neutral / hate
Label: [Label)
EmoC
Dialogue: (Sentence)
others / happy sad angry
Emotion: (Label)
17

---

Published as a conference paper at ICLR 2025
Extra Details For the task HateSpeech18, we preprocess the data to retain only the first two classes (0: neutral) and {1: hate). We exclude the other two classes due to their extremely limited number of samples.
B REPRODUCTION DETAILS
B. IMPLEMENTATION OF BASELINE METHODS.
Noise Vector. In this baseline method, We simply replace context vectors with random noises while keeping all other settings identical to I2CL.
Label Anchor Wang et al. 2023). We take the officially released code, aligning the architecture, datasets, and template setups for fair comparison. Following their established practice, template tokens and the newline separator "In' are used as anchors. Detailed information can be found at anchors
https: /ttp:://gttttttt com http://lacoooouuuaaam www.ccpp.u/aabeewwod-a-aaaaa-aaa
Task Vector (Hendel et al. 2023). We replicate the task vector method which was initially evaluated on set of toy datasets. To generate the task vector, we append random extra query after the concatenated demonstration examples (five per class) to simulate the dummy query they utilized. We then extract the hidden state from the token position responsible for the next token prediction to serve as the task vector. A hold-out dataset with 32 extra examples is used to select the best layer for extraction and replacement for each task, following the original method.
In-contex Vector (Liu et al. 2024). ICV method is primarily crafted for open- end generation tasks, and leverages positive and negative demonstration pairs to generate the steering vector. In our scenario, we set sentence (or question) in a demonstration example as negative and its corresponding answer as positive counterpart to extract in-context vector. We then manually search the strength scalar of injection for each task. Specifically, we first search in a log scale covering À [0.0001, 0.001, 0.01, 0.1, 1.0]. followed by more fine-grained search between 0.01 and 0.1. The best strength scalars we used are reported in Table T0 We refer readers to /attp:: http//ttthhhh.e.e wwt. www.cmm/heengluu66
Table 10: Strength scalar À for ICV.
SST-2
0.01
SST-5
0.02
TREC
0.02
AGNews
0.0001
Subj
0.02
HateSpecch18
0.02
DBPedia
0,001
EmoC
0.02
ICV for more technique details.
MR
0.02
AutoComprsssors (Chevalier al. 2023). We directly test the officially
released model from here: https: //ggtgttthuus thub. www.///iio..ttoooooooooooo n1p/
tasks. All pareeepar ours.
by default values. Evaluation protocols remain the same
We directly test the officially released model from here: pattps://githu: com/get ao cae on our tasks. All hyper-parameters are set by default values. Evaluation protocols remain the same ours.
ICAE (Ge al. 2023).
2023).
CEPE (Yen et al. 2024). We directly test the officially released model from here: https:// github. com/ wwwpppptiooooooeeocooooooooooo.t nttpttonnnnn//ppEEe on our tasks. All hyper-parametend are set by default values. Evaluation protocols remain the same as ours.
Prompt-tuning (Lester al. 2021). For the implementation of prompt-tuning method, allow one extra learnable token (P 1) per layer, and apply learnable prompts across all layers. We also attempt use more learnable tokens, resulting in poorer performance due to overfitting. For optimization, we conduct simple grid search on SST-2 to determine an optimal learning rate of 0.1, which we then apply across all tasks. All other configurations remain as specified in the experimental section.
LoRA (Hu et al. 2021) and IA3 (Lu et al. 2022 We use implementations from HuggingFace PEFT library for both PEFT methods. Learning rate is set to 0.001 for both methods, and all other optimization protocols are kept the same as in experimental section. The LoRA configuration uses a rank of 8 with a scaling factor (a) of 32, applies dropout at rate of 0.05, and targets both the query and value projection modules. As for IA3 method, adaptation is applied not only on query and value projection modules, but also for ferd-forward layers.
18

---

Published as a conference paper at ICLR 2025
Table 11: Comparison between I2CL and baseline methods on GPT-J-6B.
Task
Zero-shot
Few-shot (ICL)
Soft-prompt
Labcl-anchor
Task-vector
I2CL (ours)
SST-2 (%)
77.76
89.44-260
69.0411161
87.12-457
59.8414.47
85.4811.18
SST-5 (%)
25.60
39.65=457
37.88+2.99
37.24=159
31.20+2.0
37
TREC (%)
68.20
67.76-211
67.00+12284
58.52=2.44
67.32+0.32
63.81+7.58
AGNews (%)
71.60
83.18_200
83. 16.84
80.84_0.88
80.12.2..1
81.5613.13
Subj (%6 f
62.40
50.20_002
63. 1..652
51.16.1.711
66.32.1.31
65.5618335
HateSpeech18 (96)
59.92
53.44=684
67. 76
55. 15.20==810
70. 12:
72.32=576
DBPedia (96)
65.56
93.30=1.19
85.04+1.02
90.84=1.79
77. 77.64+4483
81.84=4.50
EmoC' (%) f
44.68
47 62.6226662
47 10.877
44.00=825
45.00+4.20
50.32+4.68
MR (%)
f
76.88
88 1.225
73.7619.40
87.92
81.3613.58
84.4012.45
Macro avg. acc. (%)
61.40
68.14
66.08
65.87
64.32
68.07
B.2 IMPLEMENTATION OF TRANSFER LEARNING METHOD
Algorithm details the transfer learning method proposed for I2CL. In implementation, empiri- cally set h 0.8 and 5 0.5.
Algorithm Transfer Learning of I2CL
1: Input: Coefficients Ci, C2..... CN, Context vectors Vı, U,,... UN, Demonstrations new task
dnew, Threshold h. Temperature т, Default coefficient initialization Cinit.
2: Output:Caow Unew
3: Initialize I
4: Unew <- Co vectorization( (dnew)
5: Cnew Noisy_see beisy_self_aalle caafffaaaabratoonnheeee Unew Canit)
6: for 1 to n do
7: Compute s; cosine(Cnew, Ci)
8: if' 8; > h then
9: Add io
10: end if
end for
12: Compute probabilities P(i) < exp(s1)/T
Ejer exp(xs77] for each i E
13: Compute Vave + Eier P(i)u
14: Compute Cavg
+ Eier P());
Cnew +- Noisy_sellll Secoy_sell__allleee self__caibratiinn(deeen Varg: Cavg)
16: Unew + Vavg
return Caews Unew
C ADDITIONAL EXPERIMENTS AND ANALYSIS
C.I RESULTS UNDER GPT2-XL- GPT-J, AND LLAMA3-8B
Here, we further compare 12CL with baseline methods that do not need manual intervention or additional datasets on other two popular architectures. The results under architecture GPT-J and GPT2-XL are shown in Table TT and Table respectively. To highlight the generalization ability of I2CL, we further extend I2CL to latest Llama3-8B model and establish comparison between zero-shot and few-shot learning in Table 13
C.2 ANALYSIS SYNTHETIC DATASET
To underscore the generality of the proposed I2CL, we crafted synthetic dataset that containing semantic and formatting priors. Concretely, we generated three types of data containing "random strings", "random special characters". and "random numerical values". labeled as "A"'B" and "C", respectively. We then evaluate the performance of zero-shot, few-shot and I2CL on this synthetic dataset following the same setting for main Table T As shown in Table 14 I2CL. outperforms
19

---

Published as a conference paper at ICLR 2025
Table 12:
Comparison between I2CL and baseline methods on GPT2-XL. AGnews and DBPedia not evaluated due to the limitation of GPT2-XL's context window size.
Task
Zero shot
shw- (ICL)
Soft-prompt
Label-anchor
Task-vector
I2CL. (ours)
SST-2 (%)
74.76
73.65_s88
61.0443345
63.40
14.
80. 16.99
SST-5 (%)
30.44
35.95=2.
23. 23.96
22.36=331
28.52+1.37
33.84=260
TREC (%) f
35.40
60.64=5.0
10.60
66.36=10.e
41.40.535
5.
Subj (%)
64.88
13.82=10.55
35. 445444
55.56-4.26
1..80+1.66
65,96:4x5
HpteSSe (%) 1
70.84
51.86
63.9217.06
54.88
62.48.1283
68.32.14.6
EmoC. (%) 1
37.88
38.62-768
33.60+444
36.68-270
37.60+2.48
47.92+1
MR (%) f
71.36
75.79-922
37.60
60.20=3312
78.40
83.20=3329
Macro avg. acc. (%)
55.08
57.19
48.02
51.35
57.33
61.55
Table 13: Performance comparison between Zero-shot, ICL, and I2CL on Llama3-8B.
Name
SST-2
SST:5
TRRC
AGNema
Subj
EateSporch18
DRPedia
EmoC
MR
Avg.
Zemoshot
55.00
31.40
66:40
73,00
51.00
50.40
56.20
41.00
55.60
53.47
JCL
92.00
3.21
5.94
84.8 1:30
62.47 12:44
64.9
82:33 3.64
52.20
81.26
2.26
64.07
3.52
few-shot learning (i.e., ICL) by large margin, highlighting the effectiveness and generality of I2CL, and we attribute this improvement the proposed noisy self-calibration and vector injection methods which directly intervene residual streams. Similar to the empirical observation in Sec. [33] calibrated linear coefficients also generalizable, and they can be directly used under unseen demonstration examples without additional calibration (see last column in Table I4).
C.3 APPLY 12CL OVER ICL
As 12CL and ICL mutually exclusive, interesting empirical exploration involves applying 12CL on top of ICL. Specifically, we retain the demonstration tokens in the context throughout the noisy self-calibration and inference stages. As shown Table [3] the combination of I2CL with ICL significantly improves performance on certain tasks, surpassing ICL by substantial margin. Moreover, for tasks where I2CL, originally underperforms compared ICL., the inclusion of demonstration tokens helps bridge the gap. These empirical observations highlight the versatility of 12CL. On one hand, I2CL can serve as a substitute for ICL to reduce inference costs; on the other hand, when computational and memory constraints are less critical, I2CL can be applied in conjunction with ICL to further enhance ICL's performance.
C.4 ADDITIONAL VISUALIZATION
Here, we present in Fig. T the calibrated coefficients for all tasks used, illustrating variations and patterns across different configurations.
C.5
VISUALIZATION OF IN-TASK CONTEXT VECTORS
As demonstrated in Fig. (right), context vectors embed task semantics and context vectors from different classes are well separated. However, it is unclear whether context vector can carry label information within a more nuanced in-task perspective. To this end, we extract class-wise context vectors within each task and visualize in-task relations among different context vectors in Fig. 7 As shown in the figures, context vectors from different classes are well-separated indicating that context vectors will also carry label information. A notable exception comes from EmoC where vectors appear more mixed. We conjecture that besides label information, a substantial inherent parametric knowledge has also been hubbed by the context vector.
most tasks,
Table 14: Evaluation of zero-shot, few-shot and I2CL on the synthetic dataset.
Task
Zero-shot
Few-shot (ICL)
I2CL
I2CL (unseen demo.)
Synthetic data
32.6
66.20
86.48
86.36-5.40
20

---

Published as a conference paper at ICLR 2025
Table 15: Results of applying I2CL on top of ICL. Task AGNews is not applicable under 5-shot due to the limitation of GPU memory.
Name
SST-2
SST-5
TREC
Subj
18teSpocch18
DBPedia
EmoC
MR
Average
ICL
44.44
41.72
17.321441
52.56:3.
20.24:
96. 24.
25.4
93. -43
75.21
12CL
87.68e247
39.12=149
78.56+532
73.81=354
68 88
90.16
63. 73.7721337
87. 68.688222
73.83
ICL+12CL
93.32
44.
83.56
89.841409
21.36:1:4
95.87 u0s2
73.84
93.164
81.96
D ADDITIONAL DISCUSSION
D.1 POTENTIAL DESIGN SPACE
Although the end residual stream contains a rich repository information for the given token sequence, other token positions may also possess valuable attributes, noted in recent studies (Hendel et al. 2023 Todd et al.) 2024, Liu et al. 2024). Recent research also highlights the importance of formatting tokens during information propagation (Wang al.] 2023 Bai al. 2024). We believe our approach could benefit from more sophisticated design of demonstration vectorization. Additionally, while we currently a single scalar to gauge the strength of aggregated multi-head attention, allowing finer granularity- such separate strength scalars for different attention heads might enhance our system's performance, albeit the cost of increased parameters.
D.2 BROADER IMPACTS
I2CL inherits the same risks standard In-context Learning. While I2CL primarily serves to enhance efficiency, its ability to adapt quickly to different data could potentially be used for creating misinformation other harmful content at scale. I2CL's reliance existing data for demonstration examples could propagate existing biases the data are not carefully curated. One additional risk comes from the generation of implicit vector representation for the demonstration examples, making the detection of the malicious contents even more challenging.
21

---

Published as a conference paper at ICLR 2025
2.5
Calibrated linear coefficients ssr
2.5
Calibrated linear coefficients for sst5
B
1.5
Value
1.0
1.0
0.5-
0.5
0.5
0.5
-1.0
-1.0
Layer
Layer
Calibrated linear coefficients for
Calibrated linear confficients for agnews
2.0
1.5
1.5
volu
0.5
0.0
0.0
1.0
10
15
200
25
30
25
30
Layer
Layer
2.5
Calibrated linear confficients for sub)
2.5
Callbrattd linear coofficients for mr
2.0
2
2.0
29
8"
A
Value
1.0
g"
1.0
8"
0.5
0.5
-0.5
-0.5
1.0
-1.0
20
30
2.5
Callbrated lincar caefficients abpedia
Calibrated linear confficients
8
1.5
1.5
Value
1.0
0.0-
0.0
-0.5
-0.5
10
15
20
25
30
20
30
Layer
Layer
2.5
Calibrated linear coefficients hate speech-18
2.0
0.5
1.0
10
15
20
25
30
Layer
Figure 6: Calibrated coefficients for various datasets.
22

---

Published as a conference paper at ICLR 2025
You aloeter Taaa
Figure 7: Visualization of in-task context vectors.
23


Reviews:
[{"review_id": "4MfGItNk1h", "reviewer": "Reviewer_sfPw", "summary": "The paper I2CL proposes a method to achieve few-shot performance at zero-shot inference costs by generating a condensed \"context vector\" from demonstration examples, which is then injected directly into the model’s residual streams. Many experiments demonstrate the method’s efficacy.\nThe paper is well-structured and with sufficient experiments.\nManaging similar cost with few-shot but keeping comparative performance sounds novel for me.", "questions": "1. The experiments are mainly based on Llama-2. Have you explored the potential of I2CL in larger language models? It would be helpful to understand whether the efficiency gains scale up with model size.\n2. Just a question (not require you perform experiments in the revision period) and may help clarify the potential of I2CL, which is how to convert your method to vision area. Current visual in-context learning [Zhang et al.] requires extra training process to select best in-context samples. Do you think if I2Cl has the potential to transfer to visual modality? \n3. It looks like context vectors are well seperated as shown in figure 4. How might the task-id approach handle more nuanced variations within a single task type? i.e., does such context vector carry some information regarding classification labels?\n\n\nZhang, Y., Zhou, K. and Liu, Z., 2023. What makes good examples for visual in-context learning?. Advances in Neural Information Processing Systems, 36, pp.17773-17794.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I2CL presents a unique approach to reducing ICL’s computational load, which is an important advancement, especially for applications with resource constraints.\n\n2. The concept of \"task-ids\" is a novel contribution that could facilitate transfer learning.", "weaknesses": "1. I2CL’s dependence on accessing intermediate activations may limit its use in closed or black-box models where such access is restricted. Addressing this limitation or proposing adaptations for such environments would broaden its utility.\n\n2. Expand the evaluation to include more diverse tasks, such as open-ended generation or multi-step reasoning. Demonstrating I2CL’s efficacy in these tasks would make the paper’s findings more universally applicable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U5woXPe5SB", "reviewer": "Reviewer_9isd", "summary": "This paper introduces a mechanism (I2CL) to mitigate major criticism of in-context learning which is the inference cost associated with it as we need to pre-append all the demonstration examples always while performing inference. The proposed method involves two step first involving generating a context vector based on each demonstration by feeding it through the LLM and caching the activation across each layer and second where we linearly combine these activations with query activations during inference using pre-learned weights (or coefficients). These coefficients are learned only using the few-shot examples available to us for in-context learning. They show that this method can achieve comparable performance to in-context learning and at the cost of zero-shot learning. The paper also provides various ablations on how generalizable learned coefficients are to un-seen samples and the robustness of I2CL in comparison to in-context learning in different scenarios such as ordering of samples, choice of samples etc.", "questions": "- Could you please include a comparison with PEFT-based few-shot fine-tuning strategies to highlight how your approach performs relative to these methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Proposes a method for performing in-context learning at the cost of zero-shot learning and their method (I2CL) performs comparable to in-context learning.\n- The idea of performing activation merging is an interesting direction to make model learn new skills at a very negligible cost.\n- I2CL scales very well with adding more samples in-contrast to in-context learning.\n- The paper provides comparison of their method with diverse methods such as task vector, label anchors etc to show improvements and at the same time provide ablation studies showing the robustness of learned coefficients and the robustness of I2CL to ordering and choice of samples.\n- Paper is well written and easy to understand.", "weaknesses": "- Learning of weights to infuse demonstration samples might require some compute and hence comparing this method to few-shot parameter efficient fine-tuning method would also be helpful. As there are two computational cost associated here one where you extract out the context vector and other where you train the coefficients to find optimal coefficients to combine samples.\n- Caching large context vector (for larger models) might not be feasible which might make it harder to apply this method to very large models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ocShi4IGio", "reviewer": "Reviewer_m8x3", "summary": "This paper introduces presents a method to reduce the computational and memory overheads of in-context learning. The proposed method, called implicit in-context learning (I2CL), extracts a context vector from demonstration examples and inject the context vector for the inference of query example. Experiments on 8 text-classification datasets show that I2CL successfully reduces the computation and memory cost of ICL, while performs only slightly worse than vallina ICL. The authors also show that I2CL is less sensitive to the order of demonstration examples.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This paper is clear and eazy to follow. The plot and table are eazy to understand\n2. The proposed method is simple, and it achieves improvement on the inference speed and memory cost, compared to vallina ICL\n3. The proposed algorithm is less sensitive to the order of demonstration examples.\n4. The experiment part is comprehensive. The method is compared to various baselines. The authors also studied careful ablation studies to understand  the method.\n5. The observation that the calibrated  linear coefficients can function as effective task-ids is interesing", "weaknesses": "1. The major claim of this paper is that I2CL achieves ICL performance at zero-shot cost in terms of both memory usage and inference speed, which raises my concern on the fairness of the comparison. Does the optimization of noisy self-calibration (Sec 2.4) included in the comparision? The calibration process requires inference of LLMs on each demonstration examples multiple times and optimize the linear coefficient with Adam. In comparison, ICL only requires forward pass of LLM on one-time. In table 1, only #cached param are considered as memory cost, and the cost of noisy calibration is missing. Also, is noisy self-calibration counted as inference time (the author metioned the optimization takes 1-2 minutes on a single A100?\n2. In terms of the performance, I2CL is 1% worse than ICL on average. On simple task such as SST2 and MR, the performance of I2CL is not satisfying. Is there any specific reason that cause the degraded performance? \n3. Only one very simple prompt is used in this paper. Have the author tried other prompts? From my own experience, SUBJ would be a very simple dataset for Llama-2 few-shot setting given some basic prompts.\n4. On the scaling plot of figure 3: what dataset is the figure using? And as shown [1], the performance of ICL is expected to increasing as the number of demonstration increases. Could the author explain why the performance of ICL starts to decrease after 5-shots?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g6wsnokezp", "reviewer": "Reviewer_APKm", "summary": "This paper proposes an innovative paradigm that reduces the inference cost of ICL to that of zero-shot learning with minimal information loss. The results reveal that the method is quite effective with certain extra benefits including enhancing task similarity detection and fostering effective transfer learning.", "questions": "1. The proposed method performs worse than the prompt-compression-based method on some tasks but outperforms it on others. I'm curious about which task characteristics contribute to these differences in performance.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is simple, interesting and effective.\n2. Experimental results demonstrate the method's superiority.\n3.  The paper is well-structured and easy to comprehend.", "weaknesses": "1.  The paper relies solely on experimental evidence to validate the superiority of the proposed method, lacking a clear explanation of why it outperforms prior approaches. Additionally, there is a lack of detailed analysis, making it feel more like a technical report.\n2.  More ablation studies are needed to provide deeper insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)