Disentangling Latent Shifts of In-Context Learning with Weak Supervision
Authors: Josip Jukić
Jan Šnajder
Venue: NeurIPS 2025 poster ()
Keywords: in-context learning
weak supervision
pseudo-labeling
latent shift
adapters
Primary area: deep_learning
TL;DR: We use in-context learning as weak supervision to train a student model that internalizes demonstration-induced latent shifts via adapter tuning, enabling efficient inference with improved generalization.

Abstract: In-context learning (ICL) enables large language models to perform few-shot learning by conditioning on labeled examples in the prompt. Despite its flexibility, ICL suffers from instability -- especially as prompt length increases with more demonstrations. To address this, we treat ICL as a source of weak supervision and propose a parameter-efficient method that disentangles demonstration-induced latent shifts from those of the query. An ICL-based teacher generates pseudo-labels on unlabeled queries, while a student predicts them using only the query input, updating a lightweight adapter. This captures demonstration effects in a compact, reusable form, enabling efficient inference while remaining composable with new demonstrations. Although trained on noisy teacher outputs, the student often outperforms its teacher through pseudo-label correction and coverage expansion, consistent with the weak-to-strong generalization effect. Empirically, our method improves generalization, stability, and efficiency across both in-domain and out-of-domain tasks, surpassing standard ICL and prior disentanglement methods.

Disentangling Latent Shifts of In-Context Learning with Weak Supervision
Josip Jukić, Jan Snajder
TakeLab
Faculty of Electrical Engineering and Computing University of Zagreb, Croatia {josip.jukic, jan snajder)
Abstract
In-context learning (ICL) enables large language models to perform few-shot learning by conditioning on labeled examples in the prompt. Despite its flexibility, ICL suffers from instability especially as prompt length increases with more demonstrations. To address this, we treat ICL as source of weak supervision and propose a parameter-efficient method that disentangles demonstration-induen latent shifts from those of the query. An ICL-based teacher generates pseudo-label labels on unlabeled queries, while student predicts them using only the query input, updating a lightweight adapter. This captures demonstration effects in a compact, reusable form, enabling efficient inference while remaining composable with new demonstrations. Although trained on noisy teacher outputs, the student often outperforms its teacher through pseudo-label correction and coverage expansion, consistent with the weak-to-strong generalization effect. Empirically, our method improves generalization, stability, and efficiency across both in-domain and out-of- domain tasks, surpassing standard ICL and prior disentanglement methods.
Introduction
In-context learning (ICL) has become a core mechanism for adapting large language models (LLMs)
to new tasks in a way that removes the need to update their parameters 10). By prepending a few labeled examples, called demonstrations. the input query, LLMs can perform few-shot learning directly at inference time. This paradigm is especially attractive in low-resource settings, where full fine-tuning costly or impractical.
Despite convenience, ICL performance is highly sensitive to the selection and ordering of demon- strations, often resulting in unstable predictions and poor generalization (30 25). Moreover, ICL. typically requires long contexts, as multiple demonstrations must be included alongside the query in single input. As input lengths grow, inference becomes increasingly inefficient, inflating processing costs, amplifying positional biases including inherent primacy and recency effects in transformer- based LLMs (27] and pushing against the model's 's context window limits [II. Consequently, ICL scales poorly with the number of demonstrations [5] 4] beyond certain threshold, additional examples either degrade performance must be discarded entirely. This inefficiency and poor scalability limit ICL's ability to incorporate more supervision, preventing it from fully leveraging the potential benefits of richer demonstrations.
To address these limitations, mechanistic perspective on ICL has proven useful. this view, demonstrations influence model behavior by inducing latent shiffts context-dependent changes in internal representations that alter how the model processes the query. Disentangling these shifts from the representation of the query itself allows ICL to operate more robustly, processing queries independently of demonstrations. This, in turn, enables contextual knowledge to be stored persistently,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Demonstrations:
Teacher LLM
Ye
X = X1, X2 Xn
lerlyi, Vo)
Student LLM
Ya
Base LLM
Adapter
Query: X4
Figure 1: Illustration of WILDA. The teacher processes concatenation (denoted by ①) of demon- strations Xa. consisting of n demonstrations [x] ..,..., x,). , and the query x,- The student, using only the query, fine-tures its adapter weights to produce outputs y, aligned with the teacher's pseudo- labels yt by minimizing the cross-entropy loss lcr.
eliminating the need to reprocess demonstrations for each new query. The latent shifts can then be reapplied directly, thereby reducing prompt length, improving inference efficiency, and enabling more modular and reusable representations. The disentanglement of demonstration--nnduee shifts from those of the query has been explored from both practical and theoretical perspectives: some work aims improve the stability and scalability of ICL [28 42 while other studies provide formal insights into the nature of context-induced shifts R 34 However, existing methods rely on approximations that intervene directly in attention heads or hidden states. contrast, we adopt shifts implicitly through model
latent
functional perspective on ICL, capturing demonstration-induut outputs rather than architectural manipulation. This enables disentanglement to emerge as a learned, self-aligned process rather than one achieved through explicit manipulation of internal states.
In this paper, we propose disentangle the latent shifts of the demonstrations and the query by using the output of ICL as a weak supervision signal, rather than approximating the underlying internal mechanisms. Specifically, we use ICL predictions as pseudo-labels that capture the full, contextualized influence of demonstrations. These pseudo-labels guide the training of a student model that internalizes cnnnonstraaioncooniiiiinii tectural intervention. We instantiate this idea with WILDA Weakkyy-sppervin
behavior without repeated prompting or archi- In-context Learning Disentanglement via Adapters), a parameter-efficient method for encoding the latent shifts induced by in-context demonstrations. In teacher-student setup, ICL-based teacher generates pseudo-labels for unlabeled queries, while the student learns to predict them using only the query input. The student updates lightweight adapter module (16), enabling it to capture the shift in reusable and modular form. The adapter supports efficient inference without requiring demonstrations in the prompt, and multiple adapters can be combined through simple arithmetic to fuse knowledge across demonstration subsets. At inference time, additional in-context demonstrations can be composed with the adapter's shift, enabling flexible integration of prompt-basee based and parameter- based adaptation.
We evaluate WILDA on both in-domain (ID) and oft-o--domain (OOD) data, comparing it to standard ICL, prompt-based fine-tuning [33], and recent approaches that manipulate architecture or hidden states to disentangle latent shifts. WILDA consistently improves generalization, prompt robustness, and inference efficiency, while remaining highly parameter-effiieen. Despite learning from ICL outputs, the student often surpasses its teacher. We experimentally show that this improvement arises from two emergent behaviors: pseudo-label correction, where the student refines noisy inconsistent outputs from the teacher, and coverage expansion, where it generalizes beyond the narrow patterns encoded in the demonstrations. Together, these effects enable weak-to-stee reak-to-strong (W2S) generalization 23 allowing the model to learn stable task behavior from limited supervision.
Our contribution is threefold: (1) we propose WILDA, a method that encodes ICL-induced behav- ior into reusable adapters, improving inference efficiency and prompt stability without requiring demonstration prompts; (2) we show that WILDA outperforms traditional ICL and latent shift disen- tanglement methods on both ID and OOD data, while maintaining parameter efficiency; and (3) we demonstrate that multiple adapters can be combined through simple arithmetic operations, enabling scalable composition of demonstration subsets and efficient handling of long-context scenarios. To- gether. these results demonstrate that viewing ICL form of weak supervision enables parametric generalization that can also flexibly incorporate contextual adaptation. WILDA thus offers a scalable and composable framework for stable task adaptation in LLMS
OOr code is available at htttt:://gtthub.b... http://hhhwbbo/////ssssphu//////

---

2 Method
2.1 Disentangling Latent Shifts
Disentangling in-context knowledge from the query can enhance the efficiency and stability of ICL. Current methods typically achieve disentanglement by modifying the outputs of attention heads or hidden states. The theoretical motivation for this approach stems from previous studies 22] which demonstrate that linear layers optimized via gradient descent can be viewed through the lens of linear attention mechanisms. Specifically, consider a neural network's ' linear layer characterized by an initial weight matrix Wu E IR and an update AW E Rm** resulting from backpropagation. Given an input representation x E R",, the linear transformation f Rm - I=" can be expressed succinctly as f(x (Wo + A WW). Let x; E Ih be a training example and e; E R the error signal on x; obtained from the gradient of the loss function. During backpropagation, AW is computed by accumulating the outer products (denoted by %) of N training examples (x1, X2, Xx) and their
error signals {ei, ,
eN e i.e., AW = - e; @ X;. The update part of linear layers optimized by gradient descent be expressed as unnormalized linear dot-product attention 20
f(x) (Wo + +W))== Wox Oxx Wox Σ (x,xx x).
(1)
lincar attention
In the context of the attention mechanism, this shows that the latent shift A AW induced by training examples corresponds directly to the application of linear attention, with error signals e, values, training examples x; as keys, and the current input as the attention query.
The concept of disentangling the latent shifts described in (1) be extended ICL, albeit only under the approximation of linear attention. Let Wv, WK. and We denote the weight matrices for values, keys, and queries, respectively. Let x represent the current query token's 's embedding at step 1, and q = Wox! (1) is the corresponding attention query vector. The matrix X, (1) (2)
[x. Xi ...., X ((-1)) contains all previous query token representations up t 1. and X. the matrix of demonstration token representations. The concatenation [X,; Xy] along the sequence dimension used compute the output of single attention head (AH) at step t, expressed as:
(Wk(Xxt X Vd
(())
fAn(x!") Wy Xx; X,) softmax
where d is the scaling factor (i.e., the dimensionality of the key vectors). By approximating the attention mechanism with linear attention, it becomes possible to disentangle the latent shift of the zero-shot output of an attention head induced by the query from the latent shift induced by the demonstrations [
fAH(x(1))
Wv X ( X 1T g0))
W
WyX (W (WxX,) (0)
Was
AWa..
This approximation disentangles the latent shift induced by the demonstrations X. from that induced by the query xi!) (see Appendix A for detailed derivation of (3). The contribution from ICL is captured as a virtual weight update A AWICI IIL corresponding to virtual gradients, often referred to as "meta-gran grreinttrddints" in the literature. The zero-shot latent shift of the query, corresponding W (z)q(1), reflects the output without demonstrations, providing the initial state. Analogous to A AW in (1) the latent shift A Wi Wi reflects the contribution of ICL. Finally, by substituting hys = Wzsq and Abic = AWi ic q (t), we can rewrite the output of an attention head as:
(An()(() Rs h + AbeLL
Although transformer-based LLMs employ non-linear attention in practice, many methods g 42
rely on theoretical assumptions from linear attention, manipulating attention heads or hidden states to approximate latent shift disentanglement. This simplification, however, overlooks key architectural
3

---

components such as beed-forward layers, activation functions, and residual connections. While effective a degree, these methods fall short of fully capturing the complex dynamics through which transformers process demonstrations. In this work, we explore how virtual weight updates can be obtained more directly while preserving the key components of the transformer architecture.
2.2 Weak Supervision with ICL
To disentangle the latent shifts induced by in-context demonstrations, we introduce WILDA, a method that uses ICL predictions as form of weak supervision encode these shifts into reusable adapter parameters [6 17 Instead of focusing narrowly on attention head manipulations, WILDA captures the full impact of demonstrations as expressed in the model's final outputs reflecting the combined effects of all components, including attention layers, feed-forward blocks, and residual paths. By aligning with the actual latent shifts induced by ICL, WILDA enables the model embed and reapply in-context knowledge using its full architecture, without relying on repeated prompting.
At the core of WILDA is simple teacher-student framework: the teacher model, fuacher. processes both the demonstrations and the query together to generate pseudo-labels without requiring additional labeled data. The student model, faudent. shares the same architecture the teacher but includes adapter parameters. Unlike the teacher, the student processes only the query. using the adapter to internalize the knowledge from the demonstrations, query input and X the matrix of demonstration tokens, where each row corresponds to a single demonstration. The empirical loss is defined using the cross-entropy loss function fCE, which aligns the student's 's output distribution with the teacher's 's full probability distribution over the vocabulary. This enables the student learn from the full signal provided by the teacher's output logits. Formally, the empirical loss is
illustrated in Figure [ Let x,, denote the
Σ
EC (freacher ([X); x,l); Estudent (x₂)).
E
where Donlab an unlabeled dataset and X is a flattened version of X.a
WILDA fundamentally differs from existing approaches, which manipulate attention heads or hidden states at query time, by instead progressively embedding the knowledge from demonstrations into the adapter parameters, denoted WICL. The base L.M parameters, Wzs, capture the zero-shot component, while the total model parameters may be represented as Wzs @ WICL, where @ denotes the composition of base and adapter parameters This setup captures the latent shift introduced by the demonstrations through WIaL. extending the disentangling process outlined by (3) across the model's entire architecture. The teacher processes the full input sequence [X;; x,). while the student processes only the query, applying WIcI. to integrate demonstration knowledge without explicitly processing the demonstrations. Analogously to 4 the latent shift induced by demonstrations can be recovered by decomposing outputs into zero-shot and ICL components. Let hLLM(X, W) represent the final latent states of an LLM with parameters W when processing the input X.. The following decomposition holds: LLLM Wzs WIa) hxLM(X, Wzs) + AbIEL.
(6)
where AbiCı. encapsulates the latent shift attributable to the demonstrations. WILDA encodes the latent shift implicitly within the adapter parameters WICL, which is central to our approach. However, if necessary, the latent shift can also be explicitly calculated owing to the decomposition in (6).
WILDA achieves stability not only by disentangling demonstration effects, but also through its training dynamics and parametric nature. During training, the same LLM instance serves as both teacher and student across epochs, with the adapter toggled on or off to alternate between roles. Shuffling demonstrations between epochs mitigates order sensitivity, further stabilizing the ICL process. Crucially, WILDA leverages its parametric adapter to internalize demonstration inndiionn--neededdd shifts, enabling the model to generalize effectively across ID and near-OOD data (see Section 33. This aligns naturally with the W2S generalization paradigm (23), in which the student is not merely expected to match the teacher but surpass it. WILDA facilitates this process by compactly encoding latent shifts in a way that supports both pseudo-label correction (refining noisy targets through local consistency) and coverage expansion (generalizing beyond the teacher's original scope). Together, these effects enable stable extrapolation across the data distribution, aligning with theoretical expectations of W2s generalization [39
Notably, the number of adapter parameters is signiticantly smaller compared to the base model parameters.

---

3 Experiments
3.1 Experimental Setup
We perform our experiments using decoder-only autoregressive language models provided by Hugging
Face [41]. Specifically, we employ Llama (8B) [12] and Phi 3 (mini 4k) [. along with Llama
(7B) [35 for comparative purposes. Further details about the models are listed in Table 13 of the Appendix.
We assess model performance on seven tasks from the GLUE benchmark [37, covering single- sequence binary classification (COLA, SST, RTE), sequence-pair binary classification (MRPC, QQP, QNI.I), and requence-pair multi-class classification (MNT.T). Evaluation metrics follow established standards: Matthew's correlation for COL.A, Fi scores for MRPC and QQP, and accuracy for the remaining tasks, with evaluations conducted on the development sets. Additionally, we measure accu- racy on selected datasets from the MMLU benchmark (14). specifically "clementary math" (MATH) and "miscellaneous" (MISC). We further extend our analysis to the ARC-Challenge benchmark 7 to assess reasoning and multi-hop generalization, which we show in Appendix
Predictions are made based on the probability of generating specific verbalizer tokens as the first token output by the models, facilitated by carefully crafted prompts designed explicitly for single-token answers (see Appendix [ for detailed templates).
Our experiments compare WILDA with several baselines, including Zero-Shot (0-shot) inference, which generates predictions without demonstrations, Standard ICL (n-shot), which uses n demon- strations at inference time, and Pattern-Basd Based Fine-Tuning (PBFT) (33, which fine-tunes an adapter module on data-specific patterns. We also include two ICL disentanglement methods, In-Context Vectors (ICV) (28), which leverages hidden-state representations from demonstration examples, and Batch-ICL 42 which aggregates meta- geeta-graieetss across multiple one-shot runs. All methods are evaluated using a fixed number of demonstrations, with {4, 8, 16, 32). Each experiment is repeated 10 times with different random seeds, resulting in varied demonstration selections across runs. Alongside generalization scores, we report the standard deviation across these runs as an indicator of each method's stability. Evaluations for GLUE are conducted on the development sets, whereas for the MMLU datasets, we randomly sample 200 instances for evaluation.
We employ three variants of WILDA, each differing in how demonstrations are selected or ordered during training: fixed (WILDA-F) uses fixed, unchanging set of demonstrations throughout the entire training process, shuffle (WILDA-S) uses the same demonstrations throughout training but shuffles their order the beginning of each epoch, and resample (WILDA-R R) draws new set of demonstrations from a larger labeled pool at each epoch.
We utilize LoRA LLo-Rank Adaptation) [7 for the adapter modules (for both PBFT and WILDA), corresponding to 0.100.3% of the total parameter count, depending on the model (see Table 13 in the Appendix for adapter sizes per model). For each task, we generate pseudo-labels using the teacher model on unlabeled data. Specifically, we use 100 unlabeled instances (Dunlah in (5) for both the GLUE and MMLU benchmarks. Additionally, for GLUE datasets, we experiment with 200 and 500 instances to assess the impact of the amount of unlabeled data on generalization and stability. We experiment only with 100 unlabeled instances for MMLU datasets due to their limited size. In all of the experiments, we fine-tune the adapter for 10 epochs. Further experimental details are provided in Appendix E
3.2 Generalization and Stability
We first evaluate the generalization and stability of WILDA ID data. Table I reports the 16-shot ID generalization scores along with standard deviations. Across all datasets and models, WILDA-S consistently achieves the best generalization scores, outperforming standard ICL, PBFT, and the disentanglement methods ICV and Batch-ICL (cf. Table in the Appendix for results with Llama 2). Compared to standard ICL, WILDA-S shows absolute improvements ranging from 2.6% to 11.9% for Liama 3 and 2.5% to 10.3% for Phi 3, where the differences in scores are statistically significant across all datasets Similar patterns hold for E (4, 8, 32), where WILDA-S also surpasses standard
'e assess the statistical significance using two-tailed Wilcoxon signed signed-rank rank test (p < 0.05), applying the Holm- Bonferroni method for family-wis wise error rate correction.
5

---

Table 1: ID generalization scores for the 16-shot setup and [D unnaab| 100. The standard deviations of 10 runs are shown as subscripts. The highest scores and smallest standard deviations are highlighted bold, while the second-best scores are underlined.
GLUE
MMLU
Method
RTE
SST
QNLI
MNLI
COLA
MRPC
QQP
MATH
MISC
0-shot
62.3
79.1
64.3
59.9
44.6
63.6
61.1
31.5
62.5
r--hot
75.16.5
93.52.0
77.0g.5
68.03.0
58.54.0
74.02.5
70.0s.0
43.53.5
84.04.0
Llama 3 (8B)
PBFT
73.23.
93.81.5
77.86.0
67.43.5
56.53.0
72.02.0
68.02.5
44.0g.
83.54.5
ICV
72.92.7
92.21.
74.5g.3
67.04.2
57.33.5
73.42.3
69.12.
41.54.
67.04.2
Batch-ICL
77.8
94.12.2
78.06.0
70.99.5
59.83.7
75.22.2
72.5g.7
36.24.0
81.02.5
WILDA-F
83.40.3
15.
80.3
72.12.6
63.7..5
76.21.8
71.91.9
46.02.3
86.02.3
WILDA-S
86.0a6
16.11.2
81.42.2
73.120
64.32.2
77.71.5
73.1
49.52.0
88.02.
WILDA-R
86.53.0
95.50. 8
79.04.3
73.53.0
62.52.8
76.51.9
72.02.2
44.02.7
85.53.3
0-shot
60.6
78.3
61.1
58.1
43.7
63.1
57.8
29.5
52.0
Phi 3 (mini 4k)
n-shot
72.15.2
90.62.1
75.63.2
65.3g.1
55.54.1
71.12.6
66.23.7
37.53.6
75.54.1
PBFT
70.64.3
90.91.9
73.63.1
63.63.6
53.63.1
69.62.3
64.62.8
36.5,1
73.54.6
ICV
71.53.1
89.12.1
74.33.2
64.11.1
54.13.6
70.82.4
65.42.9
36.0
74.04.3
Batch-ICL
75.34.2
91.22.6
76.63.1
67.13.6
56.14.1
72.62.6
67.32.8
38.03.
76.04.1
WILDA-F
80.4,,2
92.116
78.2
69.72.4
59.52.5
73.52.1
68.62.2
40.53.2
77.53.6
WILDA-S
82.41.1
93.21.6
79.21.4
70.41.
60.72.
74.114
69.61.9
41.52.
78.033
WILDA-R 79.01.9
92.62.0
79.62.9
68.63.9
58.62.9
73.62.0
68.12.3
39.53.6
77.03.7
ICL (cf. Table the Appendix for other ne-hhot setups). Additionally, when larger set Dunlab is used, there is marginal improvement in scores, while stability improves even further (see Table i the Appendix). Notably, the improvements in generalization with WILDA-S, compared standard ICL (the teacher model in WILDA), provide strong evidence that the student model is exhibiting W2S generalization; we provide more detailed analysis of this phenomenon in Section 4 While the WILDA-F and WILDA-R variants show similar generalization scores to WILDA-S, they typically exhibit higher variance. This makes WILDA-S the preferred choice due its greater stability with respect to demonstration selection, as it consistently improves upon standard n-shot ICL across all datasets and models. This is supported by the statistically significant differences in standard deviations on all datasets for Llama 3 and on all but QNLI for Phi 3
Having looked at stability with respect to demonstration selection, we now turn a more focused evaluation of stability with respect to demonstration ordering. repootr the standard deviations across 50 runs, where the same set of demonstrations is used, but their order is shuffled for each run. Designed to adapt to shuffled demonstrations, WILDA-S shows the highest stability to demonstration ordering, as evidenced by the smallest standard deviation. The stability improvements with WILDA-S over standard ICL are statistically significant across all datasets!!
We next assess the capacity of WILDA perform OOD generalization by fine-tuning adapter on one dataset and then applying the student model to a different dataset within the same task category, simulating near-Oo OOD scenario with pairs of closely related datasets. Table [3 shows the OOD generalization scores for such pairs of datasets in the GLUE benchmark. The results show that WILDA-S not only outperforms other methods in OOD generalization but also maintains higher stability when adapting to new domains (cf. Table TO in the Appendix for results with other models).
Beyond generalization and stability, also assess whether the adapters faithfully encode the information contained in the demonstrations. To evaluate this, we encode single demonstrations into the adapters and measure the student model's ability reconstruct them using simple recall task. As detailed in Appendix [D33 the adapters achieve consistently high semantic similarity across GLUE datasets, indicating that the demonstration content is reliably preserved. These findings corroborate the effectiveness of WILDA in capturing and storing task-specific information within the adapter weights in a faithful and disentangled manner.
4 We test for significance using a two-tailed Levene's test (p 0.05) and apply the Holm- Bonferroni method correct for family-wise wise error rate.

---

Table 2: Standard deviations of generalization scores across 50 runs with varied orderings of 16 demonstrations. The smallest deviations are in bold, and the -ocole-sm are underlined.
GLUE
MMLU
Model Method
RTE
SST
QNLI
MNLI
COLA
MRPC
QQP
MATH
MISC
n-shot
1.81
1.62
1.19
2.22
3.04
1.81
2.03
2.52
2.87
PBFT
2.71
1.14
4.53
2.69
2.27
1.57
1.82
2.70
3.22
ICV
2.09
1.23
4.08
2.81
1.95
1.61
2.03
1.96
3.18
LLamaa
Batch-ICL
3.04
1.47
2.89
2.24
2.53
1.42
1.74
2.51
2.59
WILDA-F
1.32
0.72
1.53
1.83
1.76
1.54
1.38
1.89
2.07
WILDA-5
0.22
0.53
1.04
1.21
1.28
0.73
1.14
1.22
0.97
WILDA-R
2.04
1.34
2.47
2.05
1.85
1.48
1.64
2.03
2.51
Table 3: OOD generalization scores with 16 shots averaged over 10 runs, with standard deviations shown as subscripts. For each dataset pair, demonstrations are taken from the left dataset, and the model is tested on the right dataset. Columns represent results the right datasets. The highest scores and lowest standard deviations are in bold, and the
scores are underlined.
Values in parentheses indicate differences from ID performance for the corresponding target dataset.
Model
Method
QNLI + RTE
RTE -> QNLI
QQP - MRPC
MRPC + QQP
n-shot
66.32.4 (8.8)
69.61.3 (7.4)
66.51.9 (7.5)
62.22.3 (7.8)
Llama 3 (8B)
PBFT
66.11.5 (7.1)
69.11.6 (8.7)
67.21. (4.8)
62.41.2 (5.6)
ICV
65.71.2 (7.2)
68.72.3 (5.8)
67.51.6 (5.9)
63.02.1 (6.1)
Batch-ICI.
65.31.4 (12.5)
66.32.3 (11.7)
64.92. (10.3)
62.12.1 (10.4)
WILDA-F
67.5,.1 (15.9)
70.5, (9.8)
68.51. (7.7)
64.41.5 (7.5)
WILDA-S
69.0gs (17.0)
71.307 (10.1)
69.02.2 (8.7)
66.4, (6.7)
WILDA-R
67.11.7 (19.4)
70.01.4 (9.0)
68.02.7 (8.5)
68.32. (3.7)
3.3 Adapter Arithmetic
To overcome the limitations of context window sizes and efficiently handle extensive demonstration sets in ICL, we employ adapter arithmetic within WILDA. This achieved by fine-tuning separate adapters for each demonstration subset, with each adapter encoding the latent shift corresponding to its subset. Following the approach of Chitale et al. [6]. these adapters are merged by summing their parameters, producing single adapter that integrates knowledge from all subsets. Partitioning demonstrations into smaller subsets enables more effective use of the available context window, allowing models to incorporate more demonstrations without exceeding length limits modifying the base LLM architecture. Additionally, distributing the prompt across multiple adapters improves GPU utilization by fitting it on single GPU and reducing memory overhead during inference.
Table 4 shows the ID generalization scores of ICV, Batch-ICL, and WILDA in fusing knowledge from multiple demonstration subsets, specifically using 2, 4, and 8 subsets of 16 demonstrations each. WILDA-S consistently outperforms baseline methods, highlighting its effectiveness in knowledge fusion across subsets [36]. Moreover, this form of adapter arithmetic aligns with recent advances in task arithmetic, where merging task-specific parameters promotes generalization across multiple tasks (1). our case, this approach effectively improves generalization and stability when fusing demonstration subsets within the same task.
3.4 Few-Shot WILDA
The core idea behind WILDA is to remove the need for explicit demonstrations by encoding their effect directly into the adapter. However, we also want verify that standard ICL remains functional when the adapter is active. particular, we examine whether the latent shift captured by the adapter composes additively with the contextual shift induced by demonstrations provided during inference.
To this end, we evaluate WILDA-S using Llama 3 (8B) in a mixed few-shot configuration. Specifically, we consider a setup which the adapter has been fine-tuned to encode 16 demonstrations and is further provided with additional 16 in-context demonstrations during inference. This configuration,

---

Table ID generalization scores of knowledge fusion for Llama (8B). The scores are averaged over 10 runs with standard deviations shown as subscripts. The table compares the effectiveness of knowledge fusion from 2, and 8 subsets of 16 demonstrations. The highest scores are bold.
GLUE
MMLU
Demonstrations
Method
RTE
SST
QNLI
MNLI
COLA
MRPC
QQP
MATH
MISC
ICV
75.243
93.61.
77.6g.9
69. 69.2
58.33.
74.2
70.62.7
45.5a.7
72.52.
2 x 16
Batch-ICL
80.2
95.31.s
80.2g.a
72.30.0
61.23.1
76.32.0
72.62,4
43.5g.9
83.0a.6
WILDA-S
87.11.
96.41.3
81.55.0
75.52.n
68.41.8
78.51.4
74.11.6
51.51.6
89.32.0
ICV
78.3g.
94.61.
79.3g.s
71.2
60.33.a
75.62.2
T2.32.4
47.5g.5
76.5g.
16
Batch-ICL
84.4g,3
96.41.
82.4c.2
74.32.5
64.22.8
78.31.
74.32.1
45.02.6
84.53.3
WILDA-S
88.42.3
97.50.7
83.6,.4
77.32.2
71.41.
79.60.7
75.21.a
53.51.4
91.01.7
ICV Batch-ICL
81.32. s
95.61.5
81.86.0
73.32.7
61.32.4
77.3
73.82.0
47.52.9
78.0s.5
8 16
85.62
96.71.1
83.84.5
75.82.1
65.32.1
79.81.3
75.81.8
45.5g.
84.02.
WILDA-S
92.8p.s
98.10.2
87.91.
81.3u.9
74.10.5
82.80.4
78.90.5
57.00.5
93.00.7
Table Performance comparison of WILDA-S configurations and standard 32-shot ICL using Llama 3 (8B). Results are averaged over 10 runs. The notation n/d indicates the number of demonstrations context (n) and encoded in the adapter (d).
GLUE
MMLU
Method
RTE
SST
QNLI
MNLI
COLA
MRPC
QQP
MATH
MISC
32-shot ICL
75.3
93.2
77.7
69.1
58.3
76.4
74.2
43.0
84.5
WILDA-S (0/32)
87.9
97.9
83.1
74.0
64.6
79.4
74.8
56.5
89.0
WILDA-S (0/16)
86.0
96.1
81.4
73.1
64.3
77.7
73.1
49.5
88.0
WILDA-S (16/16)
87.3
96.4
82.2
74.6
65.4
78.2
74.5
51.0
89.0
denoted as 16/16, thus combines parameter-based and context-based adaptation. We compare it against several baselines: standard 32-shot ICL, WILDA-S with 16 encoded demonstrations and no in-context examples (0/16), and WILDA-S with 32 encoded demonstrations and no in-context examples (0/32).
The results, summarized in Table show that the hybrid 16/16 configuration outperforms standard 32-shot ICL across all evaluated datasets, indicating that the adapter and in-context demonstrations reinforce one another rather than interfering. While 16/16 166 setup performs slightly below the fully encoded 0/32 variant, likely because the latter benefits from dedicated fine-tuning phase, the 16/ 16 setup demonstrates that WILDA can successfully integrate additional ICL prompts on top of fine- tuned adapter. This finding contirms that standard ICL remains fully compatible with adapter-based tuning, enabling a composition of contextual and parametric adaptation mechanisms within a single framework.
4 Analysis of Weak-to-Strong Generalization
Building on the observation that WILDA consistently outperforms its teacher, standard ICL, we hypothesize that W2S generalization may be driving these improvements, where the model's ability to generalize strengthens progressively from weaker signals. To explore this further, we conduct an empirical analysis of WILDA-S with Llama on aggregated examples from all GLUE datasets, treating them as a single, unified dataset. We focus on the WILDA-S variant due to its consistently strong performance and stability across prior experiments.
crucial prerequisite for successful W2S generalization the student's ability maintain stable outputs under small perturbations of the input, i.e., robustness to input variations. A low Lipschitz constant serves key indicator of this stability, it bounds the maximum change in the model output for any change in its input [21]. However, calculating the exact Lipschitz constant for LLMs intractable. To approximate we leverage the relationship between the Lipschitz constant and the input-outpur Jacobian matrix of neural network. Specifically, we compute the Frobenius norm of the Jacobian matrix as a tractable proxy, given its relationship to the spectral norm, which is known lower bound for the Lipschitz constant
9 (see Appendix B Figure 2a shows histogram of the

---

babule--bbels (%)
RTE
QNLT
COLA
QQP
Relative frequency (%)
WILDA
SST
MNLI
MRPC
PBFT
Corrected psendo-labels
ICL
60
10
40
5
20
0.2
0.4
0.6
0.8
5
6
Lipschitz constant
Training epoch
(h)
80
80
Flipped predictions (%)
Corrected
Flipped predictions (%)
Corrected
60
Corrupted
Corrupted
40
20
20
Distance to nearest ID neighbor
Distance nearest OOD neighbor
(c)
(d)
Figure 2: Empirical analysis of WILDA-S on the aggregated GLUE datasets for Llama 3: (a) Histogram of approximated Lipschitz constants across datasets, computed as the Frobenius norm of the input-output Jacobian matrix; (b) Rate of pseudo-label correction over training epochs (shaded areas indicate the standard deviation over 10 runs); Corrected and corrupted prediction rates for (c) ID examples and (d) OOD examples, based on the normalized Euclidean distance to the nearest correctly pseudo-labelle neighbor. Error bars indicate standard deviation over 10 runs.
approximated Lipschitz constants (normalized [0, 1J) for WILDA, PBFT, and ICL, WILDA exhibits notably lower Lipschitz constant than PBFT and ICL, which reflects its stronger local consistency,
The student's ability to revise the teacher's predicted labels, known as pseudo-label correction, builds directly on local consistency and serves as a key mechanism underlying W2S generalization [23]. When the model exhibits stable behavior under small input perturbations, as captured by low Lipschitz constant, it is more likely to propagate corrections reliably across neighboring inputs in representation space. This local consistency forms the foundation for accurate correction of noisy pscudo-labels in high-confident for generalization nearby, low-confidence examples- gradually expanding the model's coverage and facilitating transition from local consistency to broader generalization. Figure 226 shows how the rate of corrected pseudo-labels evolves during training on GLUE datasets. As training progresses, the percentage of corrected pssudo-labels steadily increases, demonstrating WILDA'S 's capacity to exhibit W2S generalization. Notably, the rate of pseudo-label correction plateaus faster for simpler datasets like SST and QNLI, which have lower linguistic variability.
regions. As corrected labels accumulate, they create a foundation
The mechanism of pseudo-label correction ties into the phenomenon of coverage expansion, where the model generalizes beyond the regions covered by pseudo-lahels (3) We hypothesize that WILDA'S generalization ability is supported by coverage expansion, where local corrections gradually influence nearby examples in representation space. The model's coverage expands incrementally through ripple effect, with high-confidence predictions influencing nearby examples while remaining grounded in regions supported by learned corrections. To understand this dynamic, we analyze which unseen evaluation points are corrected by clustering them based on their proximity to the nearest correctly pseudo-labeled neighbor in Dunlab- This is quantified by computing the Euclidean distance between the model's representations the final hidden states, with evaluation points categorized into ten bins based on their normalized distance from the correct neighbor. Figure 2c illustrates the rate of prediction flips within these bins, where flip refers either correcting an incorrect prediction or corrupting correct one. The rate of corrected predictions shows strong negative

---

association with the distance to the nearest correctly labeled neighbor, as measured by the point biserial correlation coefficient of -0.668. In contrast, corrupted predictions are more frequent in regions lacking nearby correct pseudo-labels. Moreover, coverage expansion also shows its effects on OOD data. Figure 2d shows the rate of flipped predictions for OOD data. Although the impact is reduced, a similar correction pattern persists, with point biserial correlation of -0.916.
Overall, the results indicate that WILDA does not merely replicate the teacher's outputs but pro- gressively reorganizes its representation space to reinforce locally consistent decision regions. This smoothing effect allows reliable predictions to extend beyond areas directly supported by demonstra- tions, consolidating weak contextual supervision into more stable and transferable task knowledge.
Related Work
Recent perspectives on ICL have shifted from task learning to task identification. Wies et al. argue that ICL works by recognizing latent tasks embedded during pre-training. Hoogland et al. [5 build on this, suggesting that ICL unfolds in developmental stages, shedding light on how models adapt to novel contexts. Li et al. (26) further empirically show that ICL predictions become more resilient to input perturbations with longer prompts and that training on noisy data enhances stability, Despite these theoretical breakthroughs, ICL remains vulnerable to the selection and ordering of demonstrations [25] Morcover, Kossen et al. [22] highlight ICL's biases rooted in pre-training data, revealing that models do not always uniformly leverage in-context information.
40
Research into the inner workings of ICL has revealed how transformers process demonstrations to form task representations. Hendel et al. [3 and Liu et al. [28] show that transformers can compress demonstration examples into a task vector, which efficiently directs the model to generate context- appropriate outputs for queries. These task vectors are created during forward pass, capturing the latent shift induced by the demonstrations. Building on this, Dai et al. S explore using linear attention to compute virtual gradients, simulating the effect of gradient- baade learning within the model. Similarly, Todd et al. (34) use causal mediation analysis to highlight the role of specific attention heads in forming robust task representations in ICL, termed function vectors.
Wei et al. 39 provide a theoretical foundation for W2S generalization, showing that, under the assumption of coverage expansion, models optimized for population-level consistency can achieve high accuracy. Lang et al. (23 further advance this view by formalizing the role of pseudo-label correction, which emerges when a model enforces local consistency during training. Building on these principles, several recent works demonstrate how large language models (LLMs) can leverage their own high-confidence outputs to improve performance. For instance, Huang et al. [18] show that rationale-augmenttd predictions can guide fine-tuning and enhance reasoning abilities without labeled supervision. Similarly, Qu et al. [32] propose recursive introspection for iterative improvement, and Wang et al. 38 introduce self-taught evaluators that enable LLMs to refine their outputs over time.
6 Conclusion
To tackle the challenges of stability and long- handling that arise when processing multiple demonstrations in ICL within LLMs, we introduced WILDA, a method that disentangles the latent shifts induced by demonstrations from those of the query, leveraging teacher-student framework. WILDA encodes these latent shifts into an adapter module, enabling the student model to handle queries without requiring demonstrations in the input. Moreover, WILDA allows efficient handling of large demonstration sets by chunking them into manageable subsets, each processed through separate adapter modules. This not only reduces the instability caused by demonstration selection and ordering but also alleviates the context window limitations inherent in transformer based models. Additionally, we demonstrated that WILDA exhibits weak-to-strong generalization by refining pseudo- labels through progressive corrections, expanding from local consistency to a more comprehensive coverage across the representation space. Our empirical evaluation confirms these advantages, showing that WILDA consistently outperforms traditional ICL methods, significantly improving generalization and stability across diverse natural language understanding datasets. Ultimately, by treating ICI. as weak supervision, WILDA enables parametric generalization that can flexibly incorporate contextual adaptation, turning transient in-context behaviors into persistent, reusable parameters and providing a scalable and modular foundation for stable task adaptation in LLMs.
10

---

References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy. Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen. Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, L.juan Wang. Shuohang Wang. Xin Wang. Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang. Jianwei Yang, Yifan Yang. Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang. Cyril Zhang, Jianwen Zhang, Li Lyna Zhang. Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: //axiiv////xiiv.//aa www.ariii....ooaeeoooeooo orgo hgrggabs/2444... 2404. 14219
[2] A Aizerman. Theoretical foundations of the potential function method in pattern recognition learning. Automation and remote control, 25:821 1-877, 1964.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https: httpttps://pproceeddnggg neurips...........o... ctc wcc//appr_aae filest Saaper_ffleee/pper////// /atp:////ppp////llee/ .ddf'
[4] Tianle Cai, Kaixuan Huang, Jason D. Lee, and Mengdi Wang. Scaling in-context demonstrations with structured attention. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023. URL URL
http::://pppeeeveeeeeeee http://pppnneeeeeeeeeeeeforrrrre fortt//ormmm// wiwooorummmiidddddddddee H580PkKK
[5] Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. How many demonstrations do you need for in-context learning? In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1114-11159) Singapore, December 2023. Association for Computational Linguistics. doi: 10. n...8855/11/0233 findings-emmlp.745 URL https: http::::////aaaaathhhaaaa acw.achoonntooo...ocomoo org findings emnlp 745/
[6] Rajas Chitale, Ankit Vaidya, Aditya Kane, and Archana Santosh Ghotkar. Task arithmetic with LoRA for continual learning. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization NWrIPS 2023),
2023. URL https:tts:://// /tttps::/openreiieeet wwtp:/wwwwwwe.wtnnteeeeee htt..n/ee/ooommr//e-AALL.LL 12M
[7] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning 2018.
challenge. arXiv preprint 1803
[8] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? Language models secretly perform gradient descent as meta-optimizers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 4005-4019, Toronto, Canada, July 2023.

---

Association for Computational Linguistics. doi: 10.18653/112023.fnnnll ac1.247. URL https://aclaathologyy .ow.gg......//...... org/2025 findings- acc. 247
[9] Benoit Dherin, Michael Munn, Mihaela Rosca, and David Barrett. Why neural networks find simple solutions: The many regularizers of geometric complexity. Advances in Neural Information Processing Systems, 11:2333-23499 2022.
[10] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng. Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning, 2024. URL WRRLhhtps:// http///axvvvv//////
[11] Zican Dong, Junyi Li, Xin Men, Wayne Xin Zhao, Bingbing Wang, Zhen Tian, Weipeng Chen, and Ji-Rong Wen. Exploring context window of large language models via decomposed positional vectors. arXiv preprint arXiv:2405.1800 2024.
[12] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi. Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis. Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang. Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee. Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park. Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Va- suden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika lyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam- badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi. Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong. Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hos- seini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckhacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang. Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan. Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco.
12

---

Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi. Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni. Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, -aan-Stp Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand. Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg. Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chemoguu, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parckh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto. Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sncha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad. Sumit Gupta, Summer Deng. Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudbury, Sydney Goldman. Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ioneseu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang. Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniy Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary De Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiei Zhao, and Zhiyu Ma. The Llama 3 herd of models, 2024. URL https: ://arii//aa/i//a/aa wwwpxivv.rg//a ab2/2407 2407 21783
[13] Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318-9333, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 100118/5/v1100 1/2023. fww.iings-emnnl.6666.. emn.p.624. URL https://
acclnthology.ooo orgh wwtp://///.2...o. findings amnlp.624
[14) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. In International
13

---

Conference on Learning Representations, 2021. URL http::///oporrvvvv.eeeeeeeeeeeee http:/ne//teeee id=d7KBjmI3mmq
[15] Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. The developmental landscape of in-context learning, arXiv preprint arXiv:2402.02366 2024.
[16] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-27999 PMLR, 09-15 Jun 2019. URL httt:::s//proeeddnng. mlr
press/ vvt http//hollbyyy.... html
[17] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL lhttps://openreeeee
[18) Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051-1068, Singapore, December 2023, Association for Computational Linguistics. doi: 10. http:////2222..mmmmm 1e/20/22...mmmmaim........ 2023. emnlp--aiinn 67
URL
org/
[19] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman. Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
wttpeeiwwww.wwweeeeeeeeee net forum?
1d=6t0Kwf8-jrj
[20] Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions training patterns via spotlights of attention. In International Conference on Machine Learning, pages 9639-9659, PMLR, 2022.
[21] Grigory Khromov and Sidak Pal Singh. Some intriguing aspects about Lipschitz continuity of neural networks. In The Twelfth International Conference Learning Representations, 2024.
URL httlhttts///ppereiiiie htw.://vwwwenetetooroommmd/ddd//
[22] Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning. In The Twelfth International Conference Learning Representations, 2024. httt::://pppeeeeeee
net seee//ooruumdddxPA//dggss
[23] Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak- to-strong generalization. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URK.hhttps://openreviie www.://orummdd--HShSSSSSSS
[24] Fabian Latorre, Paul Rolland, and Volkan Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization. In International Conference on Learning Repre- sentations. 2020. URL
net//orum?iddrree http://ppereeeeee....eeoorraaaaa
[25] Lvxue Li, Jiaqi Chen, Xinyu Lu, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weigiang Wang, Zhongyi Liu, Xianpei Han, et al. Debiasing in-context learning by instructing LLMs how to follow demonstrations. In Findings of the Association for Computational Linguistics ACL 2024, pages 7203-7215, 2024.
[26] Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Trans- formers as algorithms: Generalization and stability in in in-context learning. In International Conference on Machine Learning, pages 19565-19594, PMLR, 2023.
[27] Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 02 2024. ISSN 2307-387X. doi:
10. 162/tac/ccc_/ 00638. URL attpps://dos org a_0600
14

---

[28] Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. In Ruslan Salakhutdi- nov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 32287-32307. PMLR, 21-27 Jul 2024, URL h/tp://eedo/g....... mlr. presa/v235/liu244/0 html
[29] Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[30] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. httptt////cccaaattitt
doi: 10.18653/ 5566
org/ 2022 acl-long
[31] Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL httpp:://pppnnerre...eett/ttottt
0A9F2]2DGH
[32] Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching foundation model agents how to self-improve. In Automated Reinforcement Learning: Exploring Meta-Learning, AutoMI,, and I.MMs, 2024. URL lattps://ppnnrevier.. net. ferrff qDXdmdBLhR
[33] Timo Schick and Hinrich Schüizze Exploiting cloze-questions for few-shot text classifi- cation and natural language inference. In Paola Merlo, Jorg Tiedemann, and Reut Tsar- faty, editors, Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 255-269, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/11/221..comm waw.lela.cl-mm.2200 URL htttp:///aclanthology craellggy.org//// org/2021 1e1222eeaclmaai...
[34] Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Walllcce, and David Bau. Function vectors large language models. In The Twelfth International Conference on Learning Representutions, 2024. httt:ttts///::peeeeeeee
[35] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https:
org/ abs/ /307 092888
[36] Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowl- edge fusion of large language models. In The Twelfth International Conference on Learning Representations, 2024.
DDsk11qcz
[37] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-tast -ask benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP
Workshop BlackboxNIP: Analyzing and Interpreting Neural Networks for NLP,, pages 353- 355, Brussels, Belgium, November 2018 Association for Computational Linguistics. doi: 10.18653/v1/W1/ 1/118-446.
URL htthstps: peattps://aclatttate http:///aaaaaaalo////// org/ wwgg////////... 118-5446
15

---

[38] Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi: Jason Weston, and Xian Li. Self-taught evaluators. arXiv preprini 0rXi::2408.02666, 2024.
[39] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. International Conference on Learning Representations, 2021. URL http::t:tppppeeeeeaaiieee
wet/forunidddrCeee J416kaf
[40] Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https
[41] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-an natural language processing- Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 18.18/ v1/2020.emnlp-demoss URL
org/ 2020 emnlp-demos.e
[42] Kaiyi Zhang, Ang Lv, Yuhan Chen, Hansen Ha, Tao Xu, and Rui Yan. Batch-ICL: Effective, efficient, and arder-agnostic in-context learning, arXiv preprint arXiv:2401.06469 2024.
[43] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URLhhttps:// hnttps://openrevien mewt:/oorrmid--Seeeacaee
16

---

A Dual Form of ICL
We offer detailed derivation of originally introduced by [ expanding on the key intermediate steps for clarity, which were not explicitly covered in the original work. The goal to decompose the attention head output into separate components corresponding to the demonstrations and the query, thereby disentangling the latent shifts induced by ICL.
A.1 Linearized Attention Formulation
We begin with the approximation of the attention head's output using linear attention:
) = Wy X x [ X x. x,) (
(7)
where:
Wy E Rdd Admaa is the value weight matrix;
W k TRd xdmii is the key weight matrix;
X E Russe Na is the matrix of demonstration token representations;
X R No is the matrix of previous query token representations up to time 1;
qd) W Wex, (1) Rd is the query vector at time t, with WQ Rdd. being the query
weight matrix;
X,) is the concatenation of X and X, along the sequence dimension.
A.2 Expanding the Concatenated Matrices
We can expand the concatenated matrices as follows:
W Xx; W X W VV Vg),
(8)
W Xx; X,)
W X Wx = Ka; Kg),
(9)
where:
Va WyX, is the value matrix for the demonstrations;
.
V, Wv X, is the value matrix for the previous queries;
K₁ W X is the key matrix for the demonstrations:
. K, W. X, is the key matrix for the previous queries.
The transpose of the concatenated key matrix is:
(W [W Xxx) K K
(10)
A.3 Block Matrix Multiplication
Substituting the expanded forms into Equation using rules for block matrix multiplication, we
have:
(V.K) + V,K, ()) This separates the contributions from the demonstrations and the query sequences.
fAH (x!()()) ~ Va; VgJ
(1I)
A.4
Component Definitions
We define:
Wzs V2K W (WxX
(12)
AW = V.K = W.X,
(13)
Here:
17

---

Wz zs represents the zero-shot component, capturing the model's behavior based on the query sequence alone;
A WiCL represents the latent shift induced by the demonstrations, capturing the effect of in-context learning.
A.5 Final Decomposition
Substituting
12 and 13
back into the expression, obtain:
(x = (Was + AWcc) q W + AWIcL
(14)
The decomposition shows that the attention head output can be viewed the sum of:
1. The zero-shot component (Wzsq(), the model's output when only the query sequence is considered, without any influence from the demonstrations;
2. The latent shift due to ICL (A W WICL ("): the additional contribution from the demonstra- tions, representing the knowledge introduced via in-context learning.
This separation aligns with the theoretical motivation to disentangle the latent shifts induced by the demonstrations from those induced by the query, allowing for more efficient and stable processing of queries independently of demonstrations.
B Lipschitz Continuity in Neural Networks
Lipschitz continuity is a fundamental concept the analysis of neural networks it provides bound on how much the output of function can change with respect to its input. Formally, function f: IR" + R said to be Lipschitz continuous with constant 2 for any two inputs x, x' IR" the following inequality holds:
- f(x')l S x - -
This property ensures that the function f behaves smoothly, meaning small changes in the input lead to small changes in the output, which is crucial for robustness in neural networks, particularly for predictive models (1T.
B.1 Relationship Between the Lipschitz Constant and the Jacobian Matrix
In neural networks, the Lipschitz constant can be bounded by the spectral norm of the Jacobian matrix, which quantifies the sensitivity of function's 's output to changes in the input. The Jacobian matrix (x) pmxn of a function f is defined the matrix of all partial derivatives:
[ [ 0f((x)
Ox,
The spectral norm of the Jacobian matrix, denoted [x))( provides pointwise lower bound on the global Lipschitz constant L:
J ( S L, R.
The spectral norm represents the greatest possible rate of change the function's ss output for any input variation. However, calculating the exact spectral norm can be computationally expensive, especially for deep neural networks, so the Frobenius norm is often used as an efficient alternative.
B.2 Frobenius Norm as a Surrogate for the Lipschitz Constant
The Frobenius norm of the Jacobian matrix is often used as surrogate for estimating the Lipschitz constant to avoid the computational complexity of calculating the spectral norm. The Frobenius norm, denoted || F. is easier to compute and relates to the spectral norm through the following inequality:
||||2 < |A]| S A
18

---

where r is the rank of the matrix A. While the Frobenius norm generally overestimates the spectral norm, the bounded gap between them implies that reducing the Frobenius norm below its initial value often corresponds to decrease in the spectral norm as well. This relationship supports its use as practical proxy for Lipschitz behavior: reductions in the Frobenius norm are generally associated with lower Lipschitz constant, supported by evidence showing that the Lipschitz, constant of neural networks tends to closely track the lower bound defined by the spectral norm [24 21].
B.3 Empirical Evaluation of Lipschitz Continuity
In our experiments. we approximate the Lipschitz constant by computing the Frobenius norm of the imput-oupput Jacobian matrix. where the embeddings are the inputs and the penultimate layer produces the outputs. As shown in Figure 2a WILDA demonstrates significantly lower approximated Lipschitz constant compared to PBFT and ICL. This lower value suggests that WILDA is more robust to input perturbations, which is a critical property for correcting pseudo-labels.
C Limitations
Computational cost. WILDA introduces additional computational overhead due the fine-tuning of adapters. While this fine-tuning is more lightweight compared to full model fine-tuning, it remains more expensive than standard ICL, which avoids weight updates entirely. However, WILDA offsets some of this cost by removing demonstrations from the input during inference. For instance, with Llama 3 (8B) processing 16 demonstrations from GLUE datasets, inference takes approximately 120 times longer than a O-shot setup (processing only the query). This increased cost scales quadratically with the number of tokens, highlighting the self-attention mechanism as the primary bottleneck when measurements,
with 100 unlabeled instances and 16 demonstrations using a single adapter corresponds to the computational cost of approximately 2100 inferences in 16-shot setup. This implies that after about 2100 inferences, the time spent on fine-tuning is effectively balanced by the reduction in per-inference computational cost.
handling 16 demonstrations. Based
Applicability. WILDA may be less suitable for scenarios with extremely limited resources, it relies on access to a supply of unlabeled data. In our experiments with (4, 8, 16, 32} demonstrations, we typically used 100 unlabeled instances, which proved sufficient to achieve strong performance. While unlabeled data is generally easier to acquire than labeled data, there may be scenarios where obtaining even modest amount of unlabeled data is challenging, potentially limiting the applicability of WILDA.
Large demonstration sets. Although WILDA efficiently encodes demonstrations into adapters to overcome context length limitations, the method has not been extensively tested with very large demonstration sets. From our findings, as the total number of demonstrations increases, using multiple adapters with manageable demonstration sizes tends to be more effective. For instance, we successfully employed 8 adapters with 16 demonstrations each (totaling 128 demonstrations). While this approach theoretically allows for an indefinite increase in the number of demonstrations, its effectiveness with significantly larger sets remains unexplored. Moreover, using additional adapters increases computational costs, introducing a tradeoff' between scalability and efficiency.
D
Additional Results
D.1
Supplementary ID and OOD Experiments
We provide additional experimental results that complement the analyses in the main paper. Table presents extended in-domain generalization results under the 16-shot setup with [auntabl 100, confirming that WILDA consistently outperforms standard ICL and related baselines across GLUE and MMLU benchmarks. We further evaluate on the ARC- CRC-Challenge benchmark (Table 7). where WILDA- S attains the strongest accuracy, demonstrating that the observed gains extend to more demanding reasoning tasks. Table g examines the effect of the number of demonstrations (n E [4, 32)), showing that WILDA-S scales predictably with n and remains markedly more data-efficient than standard ICI. We also assess the influence of the unlabeled query set size (Table L, observing steady yet saturating improvements as ( increases from 200 to 500. Finally, Table 10 summarizes
19

---

Table ID generalization scores for the 16-shot scenario and Duuntab standard deviations of 10 runs are shown as subscripts.
100 for Llama 2 (7B). The
GLUE
MMLU
Model
Method
RTE
SST
QNLI
MNLI
COLA
MRPC
QQP
MATH
MISC
0-shol
57.8
75.4
59.3
55.7
40.7
59.4
58.7
29.0
59.0
Llama 2 (71B)
n-shot
69.24.3
89.82.1
74.2a.9
63.32.8
54.3a.5
66.92.4
64.71.5
37.54.8
80.0b.3
PBFT
69.02.7
89.70.4
73.35.0
64.44.7
51.22.9
67.92.0
64.61.6
40.03.2
79.52.1
ICV
68.0
87.82.6
71.26.7
60.94.0
53.12.4
68.8
65.01.9
39.S2.7
62.5o.6
Batch-ICI.
75.20.8
91.21.9
74.00.s
66.53.3
55.92.1
70.30.8
69.11.8
34.52.3
77.04.1
WILDA-F
77.20.7
90.2p.7
76.84.2
66.52.4
60.11.2
71.60.2
68.8p.s
43.0
82.52.5
WILDA-S
81.92.5
92.10.3
77.30.9
70.41.8
62.83.4
72.32.6
68.2p.5
46.51.s
82.51.7
WILDA-R
81.11.9
93.62.0
74.73.6
69.62.9
57.92.9
73.12.0
66.82.3
41.52.6
82.03.7
Table 7: Accuracy the ARC-Challenge benchmark using Llama 3 (8B) with [Duntab n = 16 shots. The highest score is shown in bold, and the second-best best score underlined.
100 and
Method O-shot
n-shot
PBFT
ICV
Batch-ICL
WILDA-F
WILDA-S
WILDA-R
Accuracy 38.3
54.5
50.8
49.9
48.6
55.7
59.2
56.5
out-of-domain generalization results, indicating that WILDA preserves strong transfer and stability across heterogencous evaluation settings.
D.2 Task Specificity and Generalization
LoRA adapters implemented additive residuals and do not alter the underlying model parame- ters. During training, we alternate between enabling the adapter (student) and disabling it (teacher), while inference the adapter be toggled or off without modifying the base model. To assess whether task-specific adaptation affects general performance, we conducted additional experiments evaluating the model unrelated, non-overlapping dataset pairs from GLUE. In each case, both 16-shot ICI and WILDA-SS were exposed to demonstrations from different domain. Results, summa- rized in Table IT, show that performance differences remain within one standard deviation across 10 random seeds. This indicates that WILDA-S does not degrade the model's general zero-shot capabilities, even when an adapter trained on a specific task remains active during evaluation disjoint domain.
D.3 Faithful Encoding and Retrieval of Demonstrations
To evaluate whether demonstrations faithfully encoded and disentangled, conducted experiment by encoding single demonstration into the adapter and assessing the student model's ability to capture this information. Specifically, utilized 1000 examples per dataset across the GLUE benchmark using Llama (8B).
For each dataset, the student model was prompted with a simple instruction: "Repeat the demon- stration word for word." " During the fine-tuning phase, the teacher model processed input examples
Table ID generalization scores for n-shot scenarios (n 8, 32, with [Puntah] 100) for Llama 3 (8B). The standard deviations of 10 runs are shown as subscripts.
GLUE
MMLU
Model
Method
QNLI
COLA
MRPC
QQP
Llana 3 (8B)
n-shot
71.3s.4
84.5,.4
70.12.9
62.42.
54.63.5
69.24.1
62.02.3
37.0s.9
76.52.5
WILDA-S
80.3
90.9p.9
76.31.4
70.1
61.420
72.91.
70.3
43.0,.a
77.51.8
n-shot
72.72.1
89.42.6
73.52.
64.7a1
55.82,
71.22.4
64.329
37.0,.a
77.5
WILDA-S
82.11.1
93.21.0
78.31.3
72.21
63.71.8
73.91.3
72.10.4
47.50.5
84.01.
n-shot 75.35.2
93.21.9
77.72.9
69.1
58.31.
76.42.2
74.21.9
43.01.5
84.52.1
32
WILDA-S
87.96.
97.9h
83.10.
74.0
64.61.2
79.40.8
74.81.
56.50.2
89.0g.4
20

---

Table ID generalization scores of WILDA-S for n 16 shots and [Puntabl 200, 500 for Llama 3 (8B). Results are shown for GLUE datasets with n-shot and WILDA-S methods. The standard deviations of runs are shown as subscripts.
GLUE
Model
D unlab
RTE
QNLI
MNLI
COLA
MRPC
QQP
200
86.20.4
97.2g.4
81.61.0
73.9
64.71.1
78.90.7
74.00.5
Llama 3 (8B)
500
86.90.3
97.10.5
81.90.7
74.81.0
64.60.8
81.40.8
75.20.3
Table 10: OOD generalization scores for Phi 3 and Llama 2 in a 16-shot scenario with Dunlab 100 over 10 runs with standard deviations shown as subscripts. In each dataset pair, demonstrations are taken from the left dataset, and the model is tested on the right dataset. The columns correspond to the results on the right datasets.
Model
Method
QNLI + RTE
RTE - QNLI
QQP - MRPC
MRPC - QQP
n-shot
64.32.5
67.21.5
63.72.3
59.42.2
Phi 3 (mini 4k)
PBFT
64.118
66.91.
64.72.0
60.1
WILDA-S
67.4o.6
69.20.9
66.32.4
64.41.3
shohot
62.92.3
66.31.2
64.51.9
61.12.2
Llama (7B)
PBFT
62.81.3
68.4
65.91.8
61.31.2
WILDA-S
64.80.4
70.30.6
67.82.1
65.01.1
using the following template: "Demonstration: [demonstration). Answer: ([answer))." The adapter learned encode demonstration-sseeiif information indirectly by aligning its outputs with the teacher's responses, without explicitly sceing the demonstration itself. After training, the similarity between the student model's response and the original demonstration was computed. Table 12 shows the average BERTScore similarity 45 reconstructed response.
between the original demonstrations and the student's
The consistently high BERTScore values across all datasets indicate that the student model can reliably retrieve the encoded demonstration from the adapter. This suggests that WILDA effectively disentangles and stores task-specific information within the adapter's weights. Notably, when compared standard ICL., WILDA often produced different outputs for certain queries, particularly in instances where it corrected "corrupted" labels provided by the teacher. Despite these differences, the student model maintained high degree of semantic similarity in reproducing the demonstrations. This suggests that the adapter weights capture not only the demonstration itself but also additional latent information that contributes improved generalization.
We present below a pair of examples from SST and RTE, chosen to represent reconstructed demonstra- tions with similarity scores close to the dataset averages.
Table 11: Cross-domain impact of WILDA. To test whether task-specific adaptation harms general performance, we pair unrelated GLUE datasets and evaluate whether demonstrations from one domain deteriorate performance from the unrelated domain. Demo denotes the dataset used for constructing demonstrations (i.e., the adapter's 's training domain), while Eval indicates the unseen evaluation task. We report mean accuracy across 10 seeds. WILDA-S refers to the model with the task-specifi adapter enabled at inference.
another. Each model variant is exposed to 16 demonstrations
Demo
Eval
0-shot
16-shot
WILDA-S
QNLI
RTE
62.3
61.91.2
62.1o.9
SST
COLA
44.6
44.90.5
44.70.4
MNLI
QQP
61.1
60.90.6
61.0p.7
MRPC
SST
79.1
81.41,
81.311
21

---

Table 12: Average BERTScore (Fi) similarity across GLUE datasets. Higher scores indicate better
fidelity in recalling the encoded demonstration.
RTE
SST
QNLI
MNLI
COLA
MRPC
QQP
BERTScore 0.84
0.91
0.80
0.83
0.86
0.82
0.81
SST: Example
Original: Proves again he hasn't lost his touch, delivering superb performance in admittedly middling film.
Answer: (Positive)
Reconstructed: He demonstrates once more that he hasn't 't missed beat, delivering remarkable performance in what is admittedly average film.
Answer: (Positive)
SST: Example
Original:
Though many of the actors spark briefly when they first appear, they can't I generate enough heat in this cold vacuum of a comedy to ignite a reaction.
Answer: (Negative)
Reconstructed: Although some actors manage show a hint of energy early on, they fail to create any real warmth or spark within this lifeless and chilly comedy.
Answer: (Negative)
RTE: Example 1
Original:
The source added that the investigation proved that the bases of the genocide crime "were completed with series of illegal arrests followed in some cases with assassinations or cases of disappearances and were preceded, according to information attached to the file, by cases of torture."
Premise:
Hypothesis: Investigators discovered that a series of illicit arrests were often followed by disappearances or murders and were preceded by torture.
Answer: (True)
Reconstructed:
Premise: The investigation confirmed that genocide involved illegal arrests followed by disappearances or murders, often preceded by torture. Hypothesis: Investigators found that unlawful arrests frequently resulted in disappearances murders, often preceded by acts of torture.
Answer: (True)
RTE: Example 2
Original:
Premise: American tobacco companies were showing profit most quarters due to export sales of cigarettes and diversification of products sold, including food. Hypothesis: PM often entered markets with both cigarettes and food. Answer: (False)
Reconstructed:
Premise: Profitability was often maintained by American tobacco companies through diversi-
22

---

Table 13:
Summary of the models used in the experiments, including their Hugging Face IDs.
parameter counts, context window sizes, training token volumes, and adapter sizes.
Model
Hugging Face ID
Parameters
Context window size
Training tokens
Adapter size
1.lama 3
Meta-L-amma-3-88 Llama-2-7b
8B
8k
15T
21M
Llama 2
7B
4k
2T
20M
Phi 3
Phi-3-mini-4K-instrret
3.8B
4k
3.3T
4.5M
fication into food products and successful cigarette exports. Hypothesis: Philip Morris International offered food items and cigarettes. Answer: (False)
E Experimental Details
E.1
Models
For all three models - Llama Llama 2. and Phi 3 we utilize the bfloat16 half-precision format for parameters. A summary of the models is provided in Table B
E,2 Hyperparameters
We employ the AdamW optimizer 29 for both PBFT and WILDA variants, with learning rate of 10-4. For ICV (28] and Batch-ICL 42), we follow the implementations provided in the original papers and adapt them to our codebase, using their default parameters where specified. In the case of Batch-ICI, we utilize attention heads from the last 20 layers (k = 20) and fine-tune the model for 10 epochs.
LoRA adapter configuration.
Rank (r 8): Dimensionality of the low-rank matrices used to approximate the original weights, controlling parameter efficiency.
Scaling factor (a 32): Multiplier that scales the low-rank updates relative the frozen base weights.
Dropout: (0.1): Regularization applied to the low-rank updates during training.
Target modules: q-proj. k_proj, v_proj, o_proj, gate_proj. up-proj. down_proj.
E.3
Computing Infrastructure
conducted our experiments on AMD Ryzen Threadripper 3970X 32-Core Processors and 4x NVIDIA GeForce RTX 3090 GPUs with 24GB of RAM.
23

---

Prompt Templates
F.1
GLUE Prompt Structure
Generic prompt template for GLUE tasks
Demonstrations:
(Sentence 1}
(Sentence 2 (if applicable)} Answer: (fCorrect answer)))
Query:
(Sentence 1}
(Sentence 2 (if applicable)}
Question: (Task-specific question) Answer: (
The prompts for GLUE tasks typically consist of two sentences (or one in certain cases) followed by task specifpc question and the corresponding answer. The model is expected choose from predefined labels like Yes/No, True/False, specific class names based on the dataset. The phrasing of the question preceding each answer in the demonstrations specific to the task. Below is list of the questions used for each GLUE dataset. To encourage the model to select from predefined labels, we prepend the phrase "answer with one word" before each question, and we append clarifying options such as Yes or No? to prompt more targeted response:
RTE: [hypothesis) True or False?
SST: What is the sentiment? Positive or Negative?
QNLI: Does the sentence answer the question? Yes or No?
MNLI: Is the second sentence an Entailment, Contradiction, or Neutral?
CoLA: this sentence linguistically acceptable? Yes or No?
MRPC: Do both sentences say the same thing? Yes No?
QQP: Do both questions ask the same thing? Yes or No?
F.2
MMLUU prompt structure
Generic prompt template for MMLU sub-datasets
Demonstrations:
Question: (Previous Question 1}
Answer choices:
(A: {Choice A1)),
(B: {Choice B1}),
(C: {Choice C1}),
(D: {Choice D1})
Answer: (Correct Answer 1)
Question: (Previous Question 2}
Answer choices:
(A: (Choice A2)),
(B: {Choice B2)),
(C: {Choice (2)),
(D: {Choice D2))
Answer: (Correct Answer
24

---

Query:
Question: [Current Question)
Answer choices:
(A: {Choice A}),
(B: {Choice B}),
(C: {Choice (})
(D: {Choice D})
Answer: (
Example for MMLU elementary_mate maah (MATH)
Demonstrations:
Question: Ms. Perez drove total of 40 miles in 5 days. She drove the same number of miles each day.
How many miles did Ms. Perez drive each day? Answer choices: (A: 5), (B: 7), (C: 8), (D: 9) Answer: (C: 8)
Question: Find the median in the set data 23, 13, 18, 29, 32, 25.
Answer choices: (A: 18), (B: 24), (C: 25), (D: 29)
Answer: (B: 24)
Query:
assembly line takes hours to produce 22 parts. At that rate how many parts can she produce in 35 hours?
Q: worker
Answer choices:
(A: 220 parts),
(B: 770 parts),
(C: 4 parts),
(D: 110 parts)
Answer: (
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the goal of disentangling demonstrationninnducce latent shifts using ICI. predictions as weak supervision, and sum- marize the proposed method (WILDA) and its key contributions, which are consistently supported by the theoretical and empirical results throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include discussion of limitations in Appendix C
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification: While the paper does not introduce new theorems, it provides complete and correct proofs for existing theoretical results that necessary support and motivate the proposed method. All assumptions are clearly stated, and relevant derivations are included the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide comprehensive implementation details in Section 3.1 and in the appendix, including model configurations, training procedures, hyperparameters, and evaluation protocols, sufficient reproduce all main results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
27

---

reproducible, they some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Documented code for reproducing the main experimental results is provided in the supplemental material at submission time.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https::/niis.cc.
owt.:/obbbiiiiooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips. //nips.cc/puble wwtpppblccgguudee.a.oo.plppp http://uddde/ooccuuubbmcmcmaaaee sondesuumisssioPoll11e
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant experimental settings, including datasets, splits, model architec- tures, optimizers, and hyperparameters, detailed in Section 3.1 and in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We perform statistical testing of results and describe the specific tests used in detail. Variability is visualized using standard deviation through error bars and shaded confidence bands, with each plot including an explanation of the statistical measures shown.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the computing infrastructure in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: he research adheres to the NeurIPS Code Ethics. does not involve human subjects, private data, misuse-prone models.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]

---

The paper introduces a methodological contribution designed for specific technical context. On its own, the method not directly applicable in ways that would raise significant societal impact concerns.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
The paper does not release high-risk assets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and models used (e.g., GLUE, HuggingFace models) are properly cited and used under their respective licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
30

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpsssipppsseeeee thcode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We not introduce new assets in the form of datasets or models. However, we include accompanying code for reproducing our experiments in the supplemental material.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not conduct crowdsoureing experiments or research involving human
subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
31

---

This research does not involve human subjects and therefore did not require IRB approval.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "qSlE50Tn6E", "reviewer": "Reviewer_ECRJ", "summary": "In-context learning (ICL) enables large language models to adapt to new tasks without updating model weights. However, the choice and ordering of demonstrations in the prompt often lead to unstable predictions and poor generalization. While prior work has explored how latent representations shift during ICL—typically by approximating internal mechanisms—this paper takes a more direct approach. The authors treat the outputs of ICL as weak supervision signals and distill the predictions from few-shot demonstrations into a student model. This approach captures both the effects of demonstrations and the underlying latent shifts. Empirically, their proposed framework, WILDA, outperforms pure prompting strategies and prior methods that attempt to modify internal representations, achieving better results on both in-domain and out-of-domain tasks.", "questions": "* The student model uses the same architecture and size as the teacher, which is unusual. Most distillation setups use a smaller or simpler student model to encourage compression or efficiency. Can the author clarify this?\n* Why are MMLU subjects split into math and others, especially elementary math, is the dataset targeted to \"elementary math\"? \n* A useful experimental setup to better study the demonstration effect would be as follows:\n(1) In an 8-shot demonstration, use one high-quality (gold) example and seven dummy examples.\n(2) Compare this to a 1-shot prompt using only the gold example.\n(3) This could help answer whether the student model has actually learned/captured to denoise bad demonstrations and upweight the important ones, reflecting a true understanding of the latent shift rather than just copying the teacher’s outputs.", "limitations": "Yes, they discussed in Appendix C.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "[Strengths]\n* Clarity: The paper is well-written and easy to follow.\n \n[Weaknesses]\n* The biggest concern is that **the teacher-student framework mostly looks like a standard distillation process**, where the student learns from the teacher’s pseudo-labels. It’s unclear whether the student model is actually learning the latent shift caused by the few-shot demonstrations, or just learning to copy the teacher’s predictions. It’s also not clear if the student is learning how to properly aggregate these weak supervision signals. Moreover, since the student model never directly sees the demonstrations, it might simply be mimicking the teacher’s outputs, without truly learning the effects of the demonstrations.\n* For the baseline comparison, how does WILDA compare to a zero-shot distillation baseline? For example, distilling from a zero-shot-prompted teacher instead of few-shot. Is the performance gain significant (i expect we will see improvements but how large they are)? A follow-up question is that how does the current distilled student perform when it is prompted with few-shot examples again? What's the performance gap between the student’s zero-shot and few-shot scenarios? These two are different, one is comparing a student model distilled from few-shot prompting and zero-shot prompting; the other one is a few-shot distilled student model making inference using zero-shot and few-shot prompt.\n* About Table 4: The performance improvements might just come from merging multiple adapters, each of which already does better than ICV and Batch-ICL on their own. So it’s unclear whether the gains really come from modeling the latent shift of demonstrations, or if it’s just combining stronger components.", "was_revised": true, "final_justification": "I have put my final concern of this paper in the comment.", "rebuttal": "We thank the reviewer for the feedback and appreciate the opportunity to clarify our contributions further and address the concerns raised.\n\n### WILDA and distillation\nWhile distillation is a useful tool and WILDA’s teacher–student setup resembles it, our goal is fundamentally different. Rather than compressing a large model into a smaller one, WILDA focuses on capturing the latent shift induced by demonstrations, which can mitigate the problems with long context down the line. In our setup, the teacher processes the full demonstration–query prompt and outputs a probability distribution, while the student, equipped with a small LoRA adapter (0.1–0.3% of parameters), sees only the query and learns to approximate the teacher’s distribution. By keeping teacher and student architectures identical, we isolate the effect of the adapter and show that gains come from encoding latent shifts, not differences in model capacity.\nThis approach is orthogonal to model compression: in principle, one could use a smaller student model while still leveraging WILDA to capture demonstration‑induced shifts. Importantly, matching the teacher’s outputs is a necessary step for the student to internalize these shifts, but WILDA goes beyond simple imitation. In fact, the student often surpasses its teacher, with WILDA‑S improving standard ICL by 2.6–11.9 points on GLUE and MMLU for Llama‑3. This behavior reflects weak‑to‑strong (W2S) generalization: during training, WILDA corrects noisy pseudo‑labels, expands coverage, and achieves lower Lipschitz constants (Section 4).\n\nTo further demonstrate that the adapter encodes demonstration content rather than merely copying the teacher’s outputs, we conducted an experiment (Appendix D.3) treating the adapter‑induced latent shift as a “task vector.” Retrieving tokens via nearest‑neighbor search over the vocabulary produces text that closely matches the original demonstrations. This aligns with the intuition that, because the student only observes the query while the teacher has access to both demonstrations and the query, it cannot reach teacher‑level performance unless it internally represents the missing demonstration information. Our empirical results confirm this: zero-shot performance is consistently worse than few-shot teacher predictions, and the student closes this gap, often surpassing the teacher, by encoding and leveraging these demonstration-induced shifts. This corroborates the disentanglement we describe: the adapter captures the demonstration signal, while the query is directly provided as input.\n\n### Zero-shot baseline\nWe appreciate the suggestion to include a zero-shot teacher baseline. In fact, our experiments already consider this setting: pattern-based fine-tuning (PBFT) fine-tunes an adapter using zero-shot prompts and template verbalizers, which is effectively equivalent to distilling from a zero-shot teacher with gold labels. As shown in Table 1, WILDA‑S consistently outperforms PBFT across all GLUE and MMLU tasks, with substantial gains for both Llama‑3 and Phi‑3. This indicates that incorporating few‑shot demonstrations through WILDA provides meaningful advantages over zero‑shot distillation.\n\n### Student with new demonstrations within the prompt\nWILDA enables the student model to incorporate additional demonstrations at inference time. New demonstrations can be supplied in the prompt and are naturally combined with the adapter’s latent shift. Empirically, prompting the WILDA student with extra demonstrations (few‑shot WILDA) consistently improves accuracy. For instance, a WILDA adapter trained with 16 demonstrations and prompted with 16 additional demonstrations outperforms standard ICL using 32 demonstrations. For clarity, we reproduce the table from the paper below. The notation  $(n/d)$ denotes the configuration, where $n$ is the number of shots in the prompt and $d$ is the number of encoded demonstrations in the adapter.\n\n| Method | RTE  | SST  | QNLI | MNLI | COLA | MRPC | QQP  | MATH | MISC |\n|-|-|-|-|-|-|-|-|-|-|\n| 32-shot ICL | 75.3 | 93.2 | 77.7 | 69.1 | 58.3 | 76.4 | 74.2 | 43.0 | 84.5 |\n| WILDA-S (0/32) | 87.9 | 97.9 | 83.1 | 74.0 | 64.6 | 79.4 | 74.8 | 56.5 | 89.0 |\n| WILDA-S (0/16) | 86.0 | 96.1 | 81.4 | 73.1 | 64.3 | 77.7 | 73.1 | 49.5 | 88.0 |\n| **WILDA-S (16/16)** | 87.3 | 96.4 | 82.2 | 74.6 | 65.4 | 78.2 | 74.5 | 51.0 | 89.0 |\n\n\n### Adapter merging (Table 4)\nIn WILDA, each adapter is trained on a distinct subset of demonstrations and thus captures a specific latent shift. Table 4 does not simply test whether individual adapters are strong, as already shown in Table 1, but instead examines how effectively these independently learned shifts can be combined. When we fuse 2, 4, or 8 adapters by summing their parameters, WILDA’s performance increases markedly. If each adapter merely replicated the same information, we believe that summing their weights would not lead to such notable gains. The observed improvements suggest that each adapter captures complementary information from its subset of demonstrations, effectively expanding the total amount of usable demonstration signal beyond the constraints of the context window. The fusion aggregates these distinct latent shifts, resulting in stronger performance.\nMMLU datasets\n\nWe chose these two MMLU subsets to complement the GLUE tasks used in our main experiments. While GLUE focuses on natural language understanding tasks, we included “elementary math” (MATH) and “miscellaneous” (MISC) to evaluate cross-domain generalization in multi-choice question answering. The MATH subset targets math-oriented reasoning skills, whereas the MISC subset spans a diverse range of topics, providing broader coverage beyond standard GLUE benchmarks.\n\n\n### Experiment suggestion – 8-shot with noisy demonstrations\nWe thank the reviewer for suggesting that we include an experiment with noisy demonstrations. Our original experiments already show that WILDA is robust to random ordering and selection of demonstrations. Across multiple seeds, it exhibits significantly lower variance than standard ICL, suggesting it learns to focus on informative signals. To further test whether WILDA can denoise irrelevant demonstrations and upweight useful ones, we conducted an additional experiment following the reviewer’s proposal.\nUsing Llama‑3 (8B), we compared a 1‑shot prompt with a single high-quality (gold) example and an 8‑shot prompt containing the same gold example with seven instances from an unrelated domain, evaluating both standard ICL and WILDA‑S on GLUE tasks (10 runs). Results show that WILDA‑S generally matches or slightly outperforms the clean 1‑shot setup and remains more stable in the presence of noise, while vanilla ICL shows a modest regression when noisy demonstrations are added. These findings suggest that WILDA captures latent shifts from informative demonstrations and shows the ability to emphasize relevant signals, providing additional evidence of weak‑to‑strong generalization. The results are summarized in the following table:\n\n| Dataset | Irrelevant demonstrations | Method | Accuracy | Std. Dev. |\n|-|-|-|-|-|\n| RTE | QQP | ICL 1‑shot (gold) | 64.8 | 1.2 |\n| | | ICL 8‑shot (1 gold + 7 irrelevant) | 63.9 | 1.4 |\n| |  | WILDA‑S 1‑shot (gold) | 66.2 | 1.0 |\n| | | WILDA‑S 8‑shot (1 gold + 7 irrelevant)| 65.7 | 1.1 |\n| SST  | COLA | ICL 1‑shot (gold)   | 82.8 | 0.8  |\n| | | ICL 8‑shot (1 gold + 7 irrelevant)  | 82.4  | 0.9 |\n| | | WILDA‑S 1‑shot (gold) | 84.3 | 0.6 |\n| | | WILDA‑S 8‑shot (1 gold + 7 irrelevant)| 84.0 | 0.5 |\n| QQP | MNLI | ICL 1‑shot (gold) | 63.9 | 0.9 |\n| |  | ICL 8‑shot (1 gold + 7 irrelevant)    | 62.4 | 1.0  |\n| | | WILDA‑S 1‑shot (gold)  | 65.5 | 0.8 |\n| | | WILDA‑S 8‑shot (1 gold + 7 irrelevant)| 64.9  | 0.9 |\n| COLA    | SST  | ICL 1‑shot (gold)  | 47.8 | 1.3 |\n| | | ICL 8‑shot (1 gold + 7 irrelevant)  | 46.2 | 1.4 |\n| | | WILDA‑S 1‑shot (gold)  | 51.4 | 1.2  |\n|| | WILDA‑S 8‑shot (1 gold + 7 irrelevant)| 51.0 | 0.4 |"}, {"review_id": "DRrG9gGthE", "reviewer": "Reviewer_o4hd", "summary": "This paper proposes to use a lightweight adapter to learn the latent shifts of ICL demonstrations. By learning from the teacher model which is given the demonstrations, the student model learns to perform similarly under zero-shot setting.", "questions": "Please see the weakness.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n* This paper avoids too much simplification and the uncontrollable bias that previous methods have.\n* The proposed **WILDA-S (Shuffle)** is a clever way to overcome the instability of demonstration orders, which standard ICL suffers from.\n* It shows superiority over the baseline methods.\n\n**Weaknesses**:\n* The unavoidable training process is the biggest weakness of this work. In terms of computation resources, this method requires more GPU memory and computation, prohibiting some scenarios, like in edge devices. In terms of time consumption, it's terrible if one always has to fine-tune an adapter before using ICL.\n* It would be better to present the average scores on these datasets.\n* GLUE and MMLU are relatively old and easy datasets. I want to see the performance on harder datasets, e.g., Big-bench-hard, AGIEval, ARC-challenge.", "was_revised": true, "final_justification": "The authors have addressed all my concerns. The method is simple yet effective. One of the great contributions of this work is that it provides a solution to the example order problem in ICL. Therefore, I raise my rating.", "rebuttal": "Thank you for your constructive feedback and for noting the strengths of our approach. We appreciate your recognition of WILDA’s ability to mitigate order sensitivity and outperform baseline methods. Below, we respond to your concerns and clarify the motivations behind our design choices.\n\n### Computational overhead and the “investment” perspective\nWe acknowledge that WILDA introduces a lightweight fine‑tuning step and understand the reviewer’s concern about compute cost. However, we believe the initial compute cost should be viewed as an investment rather than a recurring cost. The adapter is extremely small; LoRA modules constitute only 0.1–0.3% of the base model parameters, which makes the process computationally lightweight. Once trained, the adapter can be toggled on or off at inference. As our limitations section notes, processing a 16-shot prompt on Llama‑3 is about 120× slower than zero-shot because the self-attention cost grows quadratically with prompt length. Fine-tuning the adapter with 100 unlabeled queries costs the equivalent of ~2,100 such 16-shot inference runs. Thus, in scenarios where one expects to answer more than a few thousand queries, the one‑time training cost gets amortised, and the overall wall‑clock time is reduced because the model no longer needs to re‑process demonstrations for every query. \n\nIn addition to amortising the one‑time training cost, partitioning demonstrations across multiple adapters also mitigates context‑window limitations. As Table 4 and the accompanying discussion explain, each adapter is trained on a manageable subset of demonstrations; merging them via adapter arithmetic allows us to incorporate many examples without exceeding the model’s context length. This strategy not only reduces memory overhead but also avoids the quadratic scaling of self‑attention for long prompts. By distributing the prompt across several small adapters, we fit within a single GPU and lower inference cost. At inference, we load only the necessary adapters and avoid the quadratic cost of long prompts, which can be beneficial even for edge devices. We acknowledge that WILDA is less suitable when one has abundant labeled demonstrations and must process new demonstration sets for every query, where we need to fine-tune an adapter for each new prompt and may negate the amortization advantage. In many real‑world applications, however, labeled demonstrations are scarce, and the same few examples are reused across many queries.\n\nMoreover, long demonstration sequences themselves can lead to unstable predictions. Standard ICL is highly sensitive to the order and selection of demonstrations, and performance often degrades as prompts grow longer. Our experiments show that WILDA’s parametric approach, encoding the latent shift into a fixed adapter, significantly improves stability: across multiple random seeds, WILDA‑S exhibits markedly lower variance than standard ICL. This stability stems from learning a reusable representation of the task rather than relying on the precise ordering of a long prompt. Thus, WILDA not only alleviates context‑length constraints but also delivers more consistent behavior.\n\nThese additional points strengthen the view of WILDA as an upfront investment: by training a small adapter once and partitioning demonstrations when necessary, we both reduce inference cost and improve stability, making the method practical even when context windows are limited or prompts are long.\n\n### Presentation of results and “average scores”\nWe report results as means over multiple runs (with different random seeds), along with standard deviations, for each dataset. For example, Table 1 includes averages over ten runs. If the reviewer is referring to averaging scores across different tasks into a single number, we deliberately avoid this because it may obscure task‑specific behavior and can be misleading when tasks have different scales or importance. We are therefore unsure what kind of average the reviewer would find helpful. We welcome further clarification and will be happy to include additional aggregates in the final version if that aligns with community standards.\n\n### Evaluation and harder datasets\nWe chose GLUE and selected MMLU subsets because they are well‑established NLP benchmarks that allow fair comparison with prior work. In this work, we focused on models under 10 billion parameters due to computational constraints; such models are typically better aligned with GLUE/MMLU‑style tasks than with extremely challenging reasoning benchmarks. Importantly, WILDA is task‑agnostic, and there is nothing in the approach that fundamentally limits its application to harder datasets.\n\nNonetheless, prompted by your remark, we evaluated WILDA on the ARC-Challenge dataset. Using Llama‑3 (8B) across all methods, we observed that WILDA-S achieves the highest accuracy with improved stability, outperforming the baselines and related approaches (see table below). This trend is consistent with our results on GLUE and MMLU. We will include these findings and the corresponding table in the revised manuscript.\n| Method | Accuracy (ARC-challenge) | Std. dev. |\n|-|-|-|\n| Zero-shot     | 38.3  | - |\n| 16-shot ICL   | 54.5  | 1.8 |\n| PBFT  | 50.8  | 1.5 |\n| ICV | 49.9  | 1.7 |\n| Batch-ICL  | 48.6 | 2.0 |\n| WILDA-F  | 55.7 | 1.3 |\n| WILDA-S  | **59.2**  | **1.0** |\n| WILDA-R  | 56.5 | 1.5 |"}, {"review_id": "Ah2IcRUXtM", "reviewer": "Reviewer_A5UG", "summary": "The paper investigates a possible way to internalize in-context learning (ICL) capability, i.e., learning a set of adaptors. The authors first illustrate on why disentangling latent shift is an (approximately) effective way of inducing ICL, and then propose a teacher-student learning framework to acquire the latent shift while encompassing also tje feed-foward layers and activation functions. Empirically, the proposed method demonstrate good gains for in-domain data as well as out-of-domain data under the same task definition, outperforming other baseline methods by a clear margin.", "questions": "There is a recent work \"Implicit In-context Learning\" also concerns a very similar setting. What's the relation and performance comparing to it?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n1. The motivation and presentation is logically sound and clear.\n2. The proposed method is simple yet very effective for in-task learning.\n3. The empirical design is reasonable and convincing.\n\n**Weakness**:\n1. The internalization process (i.e., learning the lora) is to some extent heavy, hindering one the mostly appreciated feature of ICL, ie, flexibility. This lead to a concrete question of the reviewer, how many extra computes (e.g., FLOPs) will the process need to learn the lora, and how does this computational cost comparing to other baseline methods.\n2. The effectiveness of the proposed method essentially constrained by the task definition. The learned adaptor will not be able to generalize beyond the learned task.\n3. While adding a task adaptor to the backbone model, will it hinder the general capability of the model? There might be regression on other tasks.", "was_revised": true, "final_justification": "During the rebuttal, my concerns regarding the computational cost and generalization ability of learned adapters are adequately  addressed. Going over other reviewer's feedback and discussion, I decided to maintain my score and vote towards acception.", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our work’s clarity and empirical effectiveness, and we address your points below.\n\n### Computational cost of learning the adapter\nWe share your view that one of the major appeals of in‑context learning is its flexibility. WILDA was designed to minimize overhead while maintaining this flexibility. The adapter we learn uses LoRA modules that constitute only 0.1–0.3 % of the base model parameters: for Llama 3 (8B), this corresponds to ~21 M parameters, and only ~4.5 M for Phi‑3. Because the base model is frozen, only these additional weights are updated.\n\nTo quantify compute, we measure training cost relative to the cost of processing a 16‑shot prompt on the same backbone. For Llama 3, a 16‑shot prompt takes roughly 120x longer than a zero‑shot query because attention cost grows quadratically with prompt length. Fine‑tuning the adapter for ten epochs on 100 unlabeled queries requires about 2100 forward passes of a 16‑shot prompt. In other words, the training cost is equivalent to answering around two thousand 16‑shot queries. Once the adapter is trained, we no longer feed demonstrations at inference, so each subsequent query runs at zero‑shot speed. For workloads serving more than a few thousand queries, the one‑off cost is amortised, and the total compute is lower than repeatedly processing long prompts.\n\nMoreover, long prompts often exceed the model’s context window or lead to unstable performance; to address this, WILDA supports partitioning the demonstration set across multiple adapters. Each adapter is trained on a manageable subset of demonstrations, and these adapters can later be fused by parameter addition. This strategy not only mitigates context‑length limitations and reduces memory overhead but also preserves stability: instead of processing a single long, order‑sensitive prompt, we store several compact “latent shift” adapters and combine them as needed, fitting within the available context window and ensuring consistent performance.\n\n### Task specificity and generalization\nAn adapter is by design intended to specialize the model toward a given task, and in WILDA, this is achieved by encoding the latent shift induced by task‑specific demonstrations. Our experiments suggest that this specialization is both effective and reasonably robust: across seven GLUE tasks and two MMLU subsets, WILDA‑S improves over standard ICL by 2.6–11.9 points, and on out‑of‑domain task pairs (e.g., training on QNLI and testing on RTE), it maintains higher accuracy than baseline methods. This indicates that the adapter likely captures a task‑level representation rather than memorizing individual examples. Additionally, our “adapter arithmetic” experiments show that multiple adapters trained on different demonstration subsets can be fused, allowing the model to combine information from larger or more varied prompts without exceeding the context window. We acknowledge that transfer across entirely different tasks remains limited, and exploring multi‑task adapters or hierarchical parameterizations is an interesting direction for future work.\n\nImpact on the base model’s general capabilities\nLoRA adapters are implemented as additive residuals and do not modify the base model’s weights. During training, we toggle the adapter on (student) and off (teacher), and at inference, it can be enabled or disabled without altering the core model. Importantly, additional experiments (see table below) we conducted in response to your remark show no regression in zero‑shot performance, even when an adapter trained on one domain remains active during evaluation on a completely different domain (a scenario conceivable in practice, where the model may be presented with ad hoc queries) – accuracy differences remain within normal variability.\n\nTo quantify this effect, we evaluated Llama‑3 (8B) on unrelated, non‑overlapping dataset pairs from GLUE, where both 16‑shot ICL and WILDA‑S are exposed to task‑specific demonstrations from a different domain.  Results show that performance differences remain well within one standard deviation (averaged over 10 seeds), demonstrating that WILDA‑S does not harm general zero‑shot capabilities. We will include these results in the revised manuscript.\n| Demonstration dataset | Evaluation dataset (unseen task) | 0‑shot baseline without unrelated demonstrations | 16‑shot ICL | WILDA‑S (16-shot adapter on, 0‑shot prompt) |\n|-|-|-|-|-|\n| QNLI | RTE | 62.3 | 61.9 (1.2) | 62.1 (0.9) |\n| SST | COLA | 44.6 | 44.9 (0.5) | 44.7 (0.4) |\n| MNLI | QQP | 61.1 | 60.9 (0.6) | 61.0 (0.7) |\n| MPRC | SST | 79.1 | 81.4 (1.4) | 81.3 (1.1) |\n\n\n### Relation to “Implicit In‑context Learning”\nThe recently published Implicit In‑Context Learning (I2CL) paper (Li et al., 2025) proposes condensing all demonstration examples into a single context vector. Specifically, each demonstration is passed through the base model to produce a demonstration vector; these vectors are then aggregated in a permutation‑invariant manner (e.g., via averaging and a small MLP) to form the context vector. At inference, this context vector is linearly combined with the query’s activations and injected into each residual block of the model. The combination weights (“calibration coefficients”) are learned for each layer using a small calibration set. This design eliminates the need to prepend demonstration tokens, while also reducing memory usage and speeding up inference. However, a closer look at their results reveals that I2CL regularly underperforms compared to standard ICL, indicating a trade‑off between efficiency and accuracy.\n\nWILDA‑S, in contrast, uses a teacher–student framework: a teacher model generates pseudo‑labels from few‑shot prompts, and a student fine‑tunes a small adapter to mimic and improve upon those outputs. This mechanism captures the non-linear latent shift induced by demonstrations, extending beyond the attention layers to include feed-forward networks and activation functions, and enables weak-to-strong generalization, where the student corrects noisy pseudo-labels and expands coverage. Empirically, WILDA‑S consistently surpasses standard ICL and enables handling long context by splitting the demonstrations into multiple subsets and then merging them through adapter arithmetic. Furthermore, the student can still benefit from new demonstrations supplied in the prompt: when we encode 16 demonstrations into the adapter and provide 16 more in the prompt, WILDA‑S (16/16) outperforms a 32‑shot ICL baseline across all datasets (Table 9 in Appendix D.2)."}, {"review_id": "U7q1jjCZ0G", "reviewer": "Reviewer_xixy", "summary": "In-context learning (ICL) allows LLM to do few - shot learning using labeled prompt examples but faces instability as prompts grow longer. To tackle this, a parameter - efficient method is proposed. It treats ICL as weak supervision, disentangling latent shifts from demonstrations and queries. An ICL - based teacher creates pseudo - labels for unlabeled queries. A student, updating a lightweight adapter, predicts these labels using only query inputs, capturing demonstration effects compactly for efficient inference and compatibility with new demos. Trained on noisy teacher outputs, the student often outperforms the teacher via pseudo - label correction and coverage expansion, following the weak - to - strong generalization. Empirically, the method boosts generalization, stability, and efficiency on in - domain and out - of - domain tasks, outperforming standard ICL and prior disentanglement approaches.", "questions": "1.Based on my understanding, this method can compress the information from the training set demonstrations into the adapter. If there is a new test sample with demonstrations that are significantly different from the existing training set (for example, the existing demonstration training set is about mathematics, while the new one is about physics), and the adapter does not store the corresponding physics knowledge, then the given demonstrations cannot be utilized. In this case, is this method still meaningful?", "limitations": "see question", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper presents a novel and effective approach to disentangle latent shifts in in-context learning, leveraging a weak-to-strong framework that demonstrates clear innovation and practical efficacy.\n\n2. The manuscript is well-structured, with logical flow and concise expression, making it accessible and easy to follow for readers.\n\n\n3. The research motivation is well-articulated, rooted in addressing the core challenge of ICL instability, and the proposed technical route is rigorously designed with feasible implementation steps. The basic assumption is that the teacher can get robust result, which is similar to the W2S framework.\n\n4. The extensive experimental setup, covering diverse in-domain and out-of-domain tasks, provides robust evidence to validate the model's effectiveness, including improvements in generalization, stability, and efficiency.\n\n\nWeaknesses:\n\n1. The analysis of why the student model can outperform the teacher model through pseudo-label correction and coverage expansion lacks sufficient theoretical depth, and the specific mechanisms behind the \"weak-to-strong generalization effect\" in the context of in-context learning need to be further clarified.", "was_revised": true, "final_justification": "In my opinion, in-context learning—which enables learning a new task from context—possesses task-level generalization ability. However, this LoRA adapter cannot adaptively adjust its parameters for new tasks based on new task-specific context and thus lacks such task-level generalization ability. \n\nOverall, the proposed model may not be fair compared with original in-context learning.\n\nUpon further consideration, I do not believe this constitutes an effective improvement for in-context learning; instead, it merely learns an effective LoRA adapter. Consequently, the significance of this paper would be diminished.", "rebuttal": "Thank you for your thoughtful comments and for noting both the strengths and areas where clarification is needed. Below, we offer two complementary perspectives in response to your question about unseen domains, and we provide additional context on our theoretical analysis and empirical results.\n\n### Perspective 1: Preservation of general capabilities and flexible few‑shot use\nLike other parameter‑efficient fine‑tuning methods, WILDA’s adapters add only 0.1–0.3 % of the base model’s parameters. This design means that the underlying language model remains intact: during training we toggle the adapter on and off, and at inference time users can disable it to recover the original zero‑shot behavior.\n\nPrompted by your comment and a similar observation from another reviewer, we conducted additional experiments with Llama 3 (8B) to verify that activating an adapter trained on one domain does not hinder zero‑shot performance on unrelated tasks. Using non‑overlapping GLUE dataset pairs, we observed that performance differences remain within one standard deviation (10 different seeds), confirming that WILDA preserves general zero‑shot capabilities on unrelated queries. The method thus provides a modular, task‑specific memory of demonstration effects that can be activated when relevant and safely bypassed otherwise. The results are as follows:\n| Demonstration dataset | Evaluation dataset (unseen task) | 0‑shot baseline without unrelated demonstrations | 16‑shot ICL | WILDA‑S (16-shot adapter on, 0‑shot prompt) |\n|-|-|-|-|-|\n| QNLI | RTE | 62.3 | 61.9 (1.2) | 62.1 (0.9) |\n| SST | COLA | 44.6 | 44.9 (0.5) | 44.7 (0.4) |\n| MNLI | QQP | 61.1 | 60.9 (0.6) | 61.0 (0.7) |\n| MPRC | SST | 79.1 | 81.4 (1.4) | 81.3 (1.1) |\n\nMoreover, WILDA does not preclude standard ICL. Once the adapter is trained, we can still supply new demonstrations in the prompt and combine them with the adapter’s latent shift. Appendix D.2 (Table 9) reports results where we encode 16 demonstrations in the adapter and then provide another 16 demonstrations in the prompt at inference. This WILDA‑S (16/16) configuration outperforms the standard 32‑shot ICL across all evaluated datasets. In other words, the student can benefit from both the encoded shift and additional demonstrations.\n\n### Perspective 2: Domain relevance and the need for task‑specific adapters\nYour hypothetical scenario – training an adapter on math demonstrations and then providing physics demonstrations at test time – highlights an important limitation of any ICL method: the usefulness of demonstrations depends on their relevance to the query. In standard ICL, if you mix math and physics demonstrations, the model will mostly attend to the physics examples when answering a physics question, because those tokens more closely match the query. Similarly, a WILDA adapter trained on math will encode how the model’s activations should shift when it sees math‑related queries. Applying that adapter to a physics query does not magically inject physics knowledge; the shift is largely ignored by the model because it does not align with the query’s content. To handle physics problems, one should train a separate physics adapter (on unlabeled physics queries with corresponding demonstrations) and switch it on for physics questions. Importantly, this does not diminish the utility of WILDA for math tasks; you are free to toggle the math adapter on for math queries or turn it off entirely. This also relates to the other reviewer’s question about the impact on the base model’s general capabilities: the model effectively ignores the adapter’s contribution when the query domain does not align with the adapter’s training domain.  Thus, the method remains meaningful: it provides a parametric memory of demonstration effects that can be activated when relevant and bypassed when not.\n\n### On the weak‑to‑strong analysis\nWe agree that a deeper theoretical connection between weak-to-strong generalization (W2S) and in-context learning would be valuable. Our paper provides an empirical analysis of W2S phenomena and draws parallels with known theory. Specifically, we approximate the Lipschitz constant of the model by the Frobenius norm of its input–output Jacobian and find that WILDA exhibits a significantly lower Lipschitz constant than standard ICL or PBFT. A low Lipschitz constant implies that the model’s predictions vary smoothly with small changes in input, a key ingredient for correcting noisy pseudo‑labels. We also track the rate of pseudo‑label correction over training epochs and show that it increases steadily, indicating that the student is refining the teacher’s labels rather than merely copying them. Finally, we discuss coverage expansion, where local corrections propagate to nearby inputs in representation space, gradually broadening the region where the model makes confident predictions. These empirical observations align with the W2S framework: local consistency facilitates pseudo‑label correction, which in turn leads to broader generalization. Formalizing this link between W2S and ICL remains an exciting direction for future work."}]

Author rebuttal:

Meta-review: This paper introduces WILDA, a teacher–student distillation framework designed to capture and store the demonstration-induced latent shift underlying in-context learning (ICL). A lightweight LoRA adapter is trained to mimic the outputs of a teacher model that processes few-shot demonstrations, while the student only observes queries. The central claim is that the adapter encodes the effect of demonstrations into a compact, reusable representation, which enables efficient inference, stability against order sensitivity, and composability via adapter arithmetic. Empirical results on GLUE, subsets of MMLU, and follow-up experiments on ARC-Challenge demonstrate that WILDA consistently outperforms standard ICL, pattern-based fine-tuning, and prior disentanglement approaches. Additional analyses argue that the student often surpasses the teacher through weak-to-strong generalization, correcting pseudo-label noise and expanding coverage.

The strengths of the paper lie in its conceptual clarity and practical design. Reviewers generally agreed that WILDA provides a clean mechanism for internalizing demonstration effects, and several noted that the shuffle variant substantially reduces order sensitivity which is a longstanding problem in ICL. The adapter arithmetic experiments provide further evidence that WILDA enables composability beyond what fits in a single prompt, thus mitigating long-context inefficiency. The authors also provided extensive empirical support, released prompts and configurations, and conducted new analyses in response to reviewer feedback.

The main weaknesses concern the interpretation and scope of the contribution. One reviewer remained unconvinced that the method truly disentangles latent shifts, arguing it could be seen as a form of distillation with little theoretical distinction. Relatedly, the weak-to-strong analysis, while suggestive, lacks deeper theoretical grounding. Several reviewers also raised concerns about compute overhead and the fact that adapters must be retrained for each new task, limiting flexibility compared to pure ICL. While the authors provided careful amortization arguments and experiments showing no regression in unrelated zero-shot tasks, these trade-offs remain. Finally, evaluation is largely restricted to GLUE, small MMLU subsets, and ARC-Challenge. Broader evidence on more challenging reasoning tasks would strengthen the case.

During the rebuttal and discussion, the authors addressed many points directly. They clarified the relation to zero-shot baselines, demonstrated robustness to noisy demonstrations, and showed that adapters trained on one domain do not harm performance on unrelated tasks. Reviewers were mainly satisfied with the response of the authors. Specifically, reviewers o4hd and A5UG were mainly satisfied with the response of the authors around the method’s ability to solve the example order problem and the method's compute and generalization, respectively. Reviewer xixy initially questioned whether the method undermines the spirit of ICL, after multiple exchanges with the authors accepted the trade-off. Reviewer ECRJ however remained somewhat skeptical about the latent-shift interpretation, but acknowledged the new evidence provided by the authors. Taken together, the discussion shifted consensus toward acceptance, with multiple reviewers explicitly satisfied by the new experiments.

I concur with the reviewers general shit towards acceptance after the authors' response.  The paper is technically sound, clearly written, and addresses a central pain point in ICL around instability and long-context inefficiency. While the theoretical framing may overreach, the empirical contribution is significant enough to justify acceptance. The work's novelty lies more in a clever and well-executed engineering approach than in a fundamentally new theoretical insight but it is a solid and valuable contribution.
Decision: Accept (poster)