Title: Interactive Implicit In-context Learning
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: In-context Learning; Task Vector; Large Language Models

Abstract: In-context learning (ICL) enables large language models (LLMs) to generalize from a few demonstrations but suffers from quadratic computational costs and unstable performance as the number of demonstrations grows. Implicit ICL (I$^2$CL) addresses these issues by encoding demonstrations into a unified task vector for inference, achieving high efficiency and order invariance. However, existing methods overemphasize speed while neglecting the mechanisms underlying ICL’s success: they (i) encode demonstrations independently, ignoring inter-demonstration relationships, and (ii) rely on uniform averaging, which obscures disparities in demonstration informativeness. We propose Interactive Implicit In-Context Learning (I$^3$CL), a simple yet effective framework that restores the analytical and selective capabilities of standard ICL while retaining the efficiency of implicit methods. I$^3$CL introduces two lightweight attention modules for (1) inter-demonstration interaction during encoding and (2) demonstration-to-query interaction during inference. Experiments on eight tasks with four LLMs show that I$^3$CL outperforms prior implicit approaches and even surpasses fine-tuned models in few-shot scenarios, while maintaining comparable efficiency.