Everything Everywhere All at Once: LLMs can In-Context Learn Multiple
  Tasks in Superposition
Authors: name: Zheyang Xiong; name: Ziyang Cai; name: John Cooper; name: Albert Ge; name: Vasilis Papageorgiou; name: Zack Sifakis; name: Angeliki Giannou; name: Ziqian Lin; name: Liu Yang; name: Saurabh Agarwal; name: Grigorios G Chrysos; name: Samet Oymak; name: Kangwook Lee; name: Dimitris Papailiopoulos
Published: 2024-10-08 01:28:57

Abstract: Large Language Models (LLMs) have demonstrated remarkable in-context learning
(ICL) capabilities. In this study, we explore a surprising phenomenon related
to ICL: LLMs can perform multiple, computationally distinct ICL tasks
simultaneously, during a single inference call, a capability we term "task
superposition". We provide empirical evidence of this phenomenon across various
LLM families and scales and show that this phenomenon emerges even if we train
the model to in-context learn one task at a time. We offer theoretical
explanations that this capability is well within the expressive power of
transformers. We also explore how LLMs internally compose task vectors during
superposition. Furthermore, we show that larger models can solve more ICL tasks
in parallel, and better calibrate their output distribution. Our findings offer
insights into the latent capabilities of LLMs, further substantiate the
perspective of "LLMs as superposition of simulators", and raise questions about
the mechanisms enabling simultaneous task execution.