Title: Towards understanding multimodal in-context learning
Primary area: interpretability and explainable AI
Keywords: in-context learning, multimodal learning

Abstract: Multimodal Large Language Models (MLLMs) often exhibit in-context learning (ICL) capability, yet the mechanisms governing its formation remain poorly understood. Specifically, how architectural choices and training data statistics jointly shape the emergence of this ability is still an open question. To understand this, we aim to reverse-engineer multimodal ICL by training transformers on controlled synthetic classification tasks with varying multimodal data statistics and architectural choices.
Our investigation begins by re-evaluating foundational principles of unimodal ICL within modern transformer architectures. While many prior findings hold, our experiments reveal two surprising discoveries. First, Rotary Position Embeddings (RoPE), a standard component in modern LLMs, can impair the formation of ICL circuits. Second, we find that larger models require stronger statistical cues in the training data for ICL to emerge.
Extending our analysis to the multimodal regime uncovers a fundamental learning asymmetry: once a primary modality develops the core ICL circuit from statistically diverse data, a secondary modality can achieve comparable performance with far less data complexity. In contrast to the unimodal case, we show that model scaling consistently improves multimodal ICL.
Additionally, we find that a high-quality pretrained encoder for the secondary modality is critical. Through mechanistic analysis with progress measurements tracking the formation of ICL circuits, we show that both unimodal and multimodal ICL rely on a common induction-style circuit that copies the label from the in-context exemplar that matches the query; multimodal training mainly refines this behavior rather than introducing new circuitry. We validate some findings on real MLLMs and provide practical implications.
These results provide a concrete, mechanism-level account of ICL in modern multimodal transformers.