Title: Disentangling Latent Shifts of In-Context Learning with Weak Supervision
Venue: NeurIPS 2025
Accepted: yes
Keywords: in-context learning

weak supervision

pseudo-labeling

latent shift

adapters
Abstract: In-context learning (ICL) enables large language models to perform few-shot learning by conditioning on labeled examples in the prompt. Despite its flexibility, ICL suffers from instability -- especially as prompt length increases with more demonstrations. To address this, we treat ICL as a source of weak supervision and propose a parameter-efficient method that disentangles demonstration-induced latent shifts from those of the query. An ICL-based teacher generates pseudo-labels on unlabeled queries, while a student predicts them using only the query input, updating a lightweight adapter. This captures demonstration effects in a compact, reusable form, enabling efficient inference while remaining composable with new demonstrations. Although trained on noisy teacher outputs, the student often outperforms its teacher through pseudo-label correction and coverage expansion, consistent with the weak-to-strong generalization effect. Empirically, our method improves generalization, stability, and efficiency across both in-domain and out-of-domain tasks, surpassing standard ICL and prior disentanglement methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn-context learning (ICL) allows LLM to do few - shot learning using labeled prompt examples but faces instability as prompts grow longer. To tackle this, a parameter - efficient method is proposed. It treats ICL as weak supervision, disentangling latent shifts from demonstrations and queries. An ICL - based teacher creates pseudo - labels for unlabeled queries. A student, updating a lightweight adapter, predicts these labels using only query inputs, capturing demonstration effects compactly for efficient inference and compatibility with new demos. Trained on noisy teacher outputs, the student often outperforms the teacher via pseudo - label correction and coverage expansion, following the weak - to - strong generalization. Empirically, the method boosts generalization, stability, and efficiency on in - domain and out - of - domain tasks, outperforming standard ICL and prior disentanglement approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper presents a novel and effective approach to disentangle latent shifts in in-context learning, leveraging a weak-to-strong framework that demonstrates clear innovation and practical efficacy.\n\n2. The manuscript is well-structured, with logical flow and concise expression, making it accessible and easy to follow for readers.\n\n\n3. The research motivation is well-articulated, rooted in addressing the core challenge of ICL instability, and the proposed technical route is rigorously designed with feasible implementation steps. The basic assumption is that the teacher can get robust result, which is similar to the W2S framework.\n\n4. The extensive experimental setup, covering diverse in-domain and out-of-domain tasks, provides robust evidence to validate the model's effectiveness, including improvements in generalization, stability, and efficiency.\n\n\nWeaknesses:\n\n1. The analysis of why the student model can outperform the teacher model through pseudo-label correction and coverage expansion lacks sufficient theoretical depth, and the specific mechanisms behind the \"weak-to-strong generalization effect\" in the context of in-context learning need to be further clarified.\n\nQUESTIONS:\n1.Based on my understanding, this method can compress the information from the training set demonstrations into the adapter. If there is a new test sample with demonstrations that are significantly different from the existing training set (for example, the existing demonstration training set is about mathematics, while the new one is about physics), and the adapter does not store the corresponding physics knowledge, then the given demonstrations cannot be utilized. In this case, is this method still meaningful?\n\nLIMITATIONS:\nsee question", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper investigates a possible way to internalize in-context learning (ICL) capability, i.e., learning a set of adaptors. The authors first illustrate on why disentangling latent shift is an (approximately) effective way of inducing ICL, and then propose a teacher-student learning framework to acquire the latent shift while encompassing also tje feed-foward layers and activation functions. Empirically, the proposed method demonstrate good gains for in-domain data as well as out-of-domain data under the same task definition, outperforming other baseline methods by a clear margin.

STRENGTHS AND WEAKNESSES:
**Strengths**:
1. The motivation and presentation is logically sound and clear.
2. The proposed method is simple yet very effective for in-task learning.
3. The empirical design is reasonable and convincing.

**Weakness**:
1. The internalization process (i.e., learning the lora) is to some extent heavy, hindering one the mostly appreciated feature of ICL, ie, flexibility. This lead to a concrete question of the reviewer, how many extra computes (e.g., FLOPs) will the process need to learn the lora, and how does this computational cost comparing to other baseline methods.
2. The effectiveness of the proposed method essentially constrained by the task definition. The learned adaptor will not be able to generalize beyond the learned task.
3. While adding a task adaptor to the backbone model, will it hinder the general capability of the model? There might be regression on other tasks.

QUESTIONS:
There is a recent work "Implicit In-context Learning" also concerns a very similar setting. What's the relation and performance comparing to it?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
This paper proposes to use a lightweight adapter to learn the latent shifts of ICL demonstrations. By learning from the teacher model which is given the demonstrations, the student model learns to perform similarly under zero-shot setting.

STRENGTHS AND WEAKNESSES:
**Strengths**:
* This paper avoids too much simplification and the uncontrollable bias that previous methods have.
* The proposed **WILDA-S (Shuffle)** is a clever way to overcome the instability of demonstration orders, which standard ICL suffers from.
* It shows superiority over the baseline methods.

**Weaknesses**:
* The unavoidable training process is the biggest weakness of this work. In terms of computation resources, this method requires more GPU memory and computation, prohibiting some scenarios, like in edge devices. In terms of time consumption, it's terrible if one always has to fine-tune an adapter before using ICL.
* It would be better to present the average scores on these datasets.
* GLUE and MMLU are relatively old and easy datasets. I want to see the performance on harder datasets, e.g., Big-bench-hard, AGIEval, ARC-challenge.

QUESTIONS:
Please see the weakness.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn-context learning (ICL) enables large language models to adapt to new tasks without updating model weights. However, the choice and ordering of demonstrations in the prompt often lead to unstable predictions and poor generalization. While prior work has explored how latent representations shift during ICL—typically by approximating internal mechanisms—this paper takes a more direct approach. The authors treat the outputs of ICL as weak supervision signals and distill the predictions from few-shot demonstrations into a student model. This approach captures both the effects of demonstrations and the underlying latent shifts. Empirically, their proposed framework, WILDA, outperforms pure prompting strategies and prior methods that attempt to modify internal representations, achieving better results on both in-domain and out-of-domain tasks.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n* Clarity: The paper is well-written and easy to follow.\n \n[Weaknesses]\n* The biggest concern is that **the teacher-student framework mostly looks like a standard distillation process**, where the student learns from the teacher’s pseudo-labels. It’s unclear whether the student model is actually learning the latent shift caused by the few-shot demonstrations, or just learning to copy the teacher’s predictions. It’s also not clear if the student is learning how to properly aggregate these weak supervision signals. Moreover, since the student model never directly sees the demonstrations, it might simply be mimicking the teacher’s outputs, without truly learning the effects of the demonstrations.\n* For the baseline comparison, how does WILDA compare to a zero-shot distillation baseline? For example, distilling from a zero-shot-prompted teacher instead of few-shot. Is the performance gain significant (i expect we will see improvements but how large they are)? A follow-up question is that how does the current distilled student perform when it is prompted with few-shot examples again? What's the performance gap between the student’s zero-shot and few-shot scenarios? These two are different, one is comparing a student model distilled from few-shot prompting and zero-shot prompting; the other one is a few-shot distilled student model making inference using zero-shot and few-shot prompt.\n* About Table 4: The performance improvements might just come from merging multiple adapters, each of which already does better than ICV and Batch-ICL on their own. So it’s unclear whether the gains really come from modeling the latent shift of demonstrations, or if it’s just combining stronger components.\n\nQUESTIONS:\n* The student model uses the same architecture and size as the teacher, which is unusual. Most distillation setups use a smaller or simpler student model to encourage compression or efficiency. Can the author clarify this?\n* Why are MMLU subjects split into math and others, especially elementary math, is the dataset targeted to \"elementary math\"? \n* A useful experimental setup to better study the demonstration effect would be as follows:\n(1) In an 8-shot demonstration, use one high-quality (gold) example and seven dummy examples.\n(2) Compare this to a 1-shot prompt using only the gold example.\n(3) This could help answer whether the student model has actually learned/captured to denoise bad demonstrations and upweight the important ones, reflecting a true understanding of the latent shift rather than just copying the teacher’s outputs.\n\nLIMITATIONS:\nYes, they discussed in Appendix C.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-03"}