Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition
Authors: Zheyang Xiong
Ziyang Cai
John Cooper
Albert Ge
Vasilis Papageorgiou
Zack Sifakis
Angeliki Giannou
Ziqian Lin
Liu Yang
Saurabh Agarwal
Grigorios Chrysos
Samet Oymak
Kangwook Lee
Dimitris Papailiopoulos
Venue: ICML 2025 spotlightposter (2025)
Keywords: task superposition
in-context learning
Primary area: deep_learning->large_language_models

Abstract: Large Language Models (LLMs) have demonstrated remarkable in-context learning (ICL) capabilities. In this study, we explore a surprising phenomenon related to ICL: LLMs can perform multiple, computationally distinct ICL tasks simultaneously, during a single inference call, a capability we term task superposition". We provide empirical evidence of this phenomenon across various LLM families and scales and show that this phenomenon emerges even if we train the model to in-context learn one task at a time. We offer theoretical explanations that this capability is well within the expressive power of transformers. We also explore how LLMs internally compose task vectors during superposition. Furthermore, we show that larger models can solve more ICL tasks in parallel, and better calibrate their output distribution. Our findings offer insights into the latent capabilities of LLMs, further substantiate the perspective of "LLMs as superposition of simulators", and raise questions about the mechanisms enabling simultaneous task execution.

Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition
Zheyang Xiong'
Ziyang Cai John Cooper' Albert Ge Vasilis Papageorgiou Zack Sifakis Saurabh Agarwal? Grigorios G Chrysos' 1 Samet Oymak? Kangwook Lee'
Angeliki Giannou
Ziqian Lin
Liu Yang'
Dimitris Papailiopoulos 14
Abstract
interest in the research community, as it suggests that LLMs can adapt to novel tasks on-the-fly, using the capabilities that they acquired during pretraining, and the context provided.
Large Language Models (LLMs) have demon- strated remarkable in-context learning (ICL) ca- pabilities. In this study, we explore a surprising phenomenon related to ICL: LLMs can perform multiple, computationally distinct ICL tasks si- multaneously, during single inference call, ca- pability we term "task superposition", We provide empirical evidence of this phenomenon across var- ious L.M families and scales and show that this phenomenon emerges even if we train the model to in-context learn one task at a time. We of- fer theoretical explanations that this capability well within the expressive power of transformers. We also explore how LLMs internally compose task vectors during superposition. Furthermore, we show that larger models can solve more ICL tasks in parallel, and better calibrate their output distribution. Our findings offer insights into the tent capabilities of L.Mss, further substantiate the perspective of "LLMs as superposition of simula- tors", and raise questions about the mechanisms enabling simultaneous task execution.
While ICL has been extensively studied from both theoreti- cal and empirical perspectives (Xie et al., 2022a; Agarwal et al., 2024), many aspects of its underlying mechanisms remain elusive. In this work, we study a surprising phe- nomenon related to ICI. that, to the best of our knowledge, has not been thoroughly studied before: LLMs can perform multiple distinct ICL tasks simultaneously, a single infer- ence call, capability we refer to as "task superposition"
Our study suggests that pretrained autoregressive LLMs such as Llama (Touvron et al., 2023) or GPT-3.5 (Brown et al., 2020) display superposition of tasks purely in- context?. When presented with multiple in-context examples from different tasks, in the same prompt, the models can generate outputs that correspond to solutions for all these individual tasks. For instance, given examples of addition and translation, the model can concurrently produce cor- rect answers for both tasks, as well as the composition of these tasks (e.g., the result of addition translated into another language).
Figure illustrates this phenomenon. In Figure Ia (left), given in-context examples of addition in different languages and the query "91 83 - the model generates probabili- ties for the correct sum in various languages, demonstrating its ability to perform addition and translation concurrently.
1. Introduction
Large Language Models (LLMs) have demonstrated remark- able capabilities across various domains, with one of the most intriguing being in-context learning (ICL) (Brown et al., 2020; Xie et al., 2022a). ICL enables LLMS to per- form tasks during inference without the need to fine-tune for that particular task, simply by providing a few examples within the input prompt. This ability has sparked significant
This discovery aligns and lends further support to the view of LLMSs as superposition of simulators (Janus, 2022; Shana- han et al., 2023; Nardo, 2023) and the Bayesian perspective of ICL proposed by Xie al. (2022a). While not a math- ematically rigorous formulation, we can conceptualize the output of an LLM weighted sum of conditional proba- bilities across possible tasks:
UUniversity of Wisconsin-Madison Universiiyy of Texas at Austin 'University of Michigan 'Microsoft Research. Correspon- dence to: Zheyang Xiong <zheyang @cs, wwsc...iscc...... Dimitris Pa- pailiopoulos dimitris @ paapil.ioo...
Ploutput|prompty
Σ Poooupputtaskk prompt)]((ask(ppommt)
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
'In particular, gpt 3.5- 1.55-uuree-ie S--turbe-instrrutt ruct.
FFor other definitions of superposition, please see Section

---

LLMs can In-Context Learn Multiple Tasks in Superposition
Prompt
Prompt
Task Numerical addition
11
26
37
Addition English
Dominica
Roseau
quazante
(20x) Task
Wellington
62
54 -> one hundred and sixteen
Nauru
-> NAURU
74
(20x) Task Addition in Spenish
Zimbabwe
Africa
69
setenta y nueve
Latvia
Europe
40
ochenta tres
Spain
SPAANN
soixante-quatre
(20x) Task 1: Capitall name
Lebanon
Asia
(20x) Task 2: Continent name
Fitty- Live
(20x) Task 3: Capitalization
Montenegro Andorra
MONTENBGRO
91
93
Output Probabilities (Llama-33 3)
Output Probabilitie (Llama-3)
174:
52.17%
one hundred and seventy four
Andorra Valla:
23.10%
cent saixante-qustorze:
11.10%
Europe:
23.07%
ciento y
21.69%
ANDORRA:
37.92%
(a) (left) Two-digit addition a variety of languages. (right) Naming the capital of given country name, naming continent a given country name or capitalizing the country name.
Promot
Prompt
(20x) Task to Copy the first operand
67
88
155
vacillant->
95
(20x) Copy the second operand
aftercure->
68
20
88
(20x) Task 3: Add two operands
viggle ->8
79
megatypy
71
73
73
berceuses->>
51
81
132
(20x)
letter in lower case
institue
25
17
17
impleach ->
(20x) Task 2) Last letter in lower case
10
(20x) Task 3: First letter upper case
quaddle
52
17
(20x) Task
upper case
tunga
->
Output Probabilities (GPT-3.5)
Output Probabilities (GPT-3.5)
52:
32.91%
10.78%
17:
36.88%
20.10%
28.70%
35.60%
A:
7.84%
(b) (left) Tasks copy (opl). copy (op2) and opl+op2. (right) First or letter in upper or lower case".
"In actual prompt use space to separate each letter word that each letter separate token after tokenization.
Figure I: LLMs can perform task superposition. (a) Llama-3 70B and (b) GPT-3.5 Turbo are each presented with sets of tasks. For each set of tasks, show example prompt such that all except the last row are in-context examples of one of the tasks and the last row is the query. We provide 20 in-context task examples for each task in the prompt with order randomized and provide the probabilities of outputs when correctly performing each task on the query.
In this conceptual model, Pooutput prompt) represents the probability distribution over possible outputs given the input prompt, a task can be thought of a latent vari- able representing different capabilities the model might possess (c.g., arithmetic. translation, sentiment analysis), pauutuu//askk prompt) represents the output probability distribution if the model was specifically attempting to solve a single task, based on the test example the prompt, and ((toak)prompt) represents the model's inferred probability that the prompt specifies particular task.
and utilize multiple task distributions, resulting in outputs that reflect a combination of relevant tasks.
Our Contributions: butions:
Our study makes several key contri-
1. Through extensive empirical investigation and theo- retical results, we demonstrate that task superposition is prevalent across various pretrained I.I.M families (GPT3.5, LLama-3, Qwen).
While this mental model simplification of how LLM operates, it provides an intuitive way support the task superposition phenomenon we observe. Our findings lend support to the idea that L.MS can simultaneously maintain
2. We empirically show that task superposition emerges as we train on one task at time.
3. We provide a theoretical construction showing that Transformers models are indeed capable of task super-
2

---

LLMs can In-Context Learn Multiple Tasks in Superposition
position, and have the capacity implement multiple tasks in parallel.
The development of new architectures such as state space models (Gu & Dao, 2023) has further motivated studying whether in-context learning is prevalent in alternative ar- chitectures such as Mamba (Park et al., 2024; Grazzi et al., 2024; Zeng et al., 2024) or in looped transformers (Yang et al., 2023).
4. We explore how LLMs internally compose task vectors (Hendel et al., 2023) during superposition, and show how convex combinations of task vectors can reproduce the superposition effect.
5. We show that larger models can solve more tasks in parallel and more accurately reflect the distribution of in-context tasks.
Steering models through in-context learning has been a growing area of interest. Recent work has hypothesized that in-context learning can be encapsulated by a high- dimensional description of task, which can be used to replace, (Hendel et al., 2023) compose (Todd et al., 2024) or augment (Liu et al., 2024) the latent states of a model, in order to alter its default behavior. Task vectors can be com- bined via arithmetic operations to solve a variety of tasks (Mharco et al., 2023). Prior work has also been investigating the power of tokens in defining task (Bai et al., 2024).
We believe that our findings offer new insights into the latent capabilities of LLMs and raise questions about the mecha- nisms enabling simultaneous task execution. We believe this work sheds more light on the ICL capabilities of frontier lan- guage models, and offers a glimpse on potential applications of task superposition in practical settings.
Other definitions of superposition. Our findings on su- perposition are inspired by notions of language models as multiverse generators (Reynolds & McDonell, 2021; moire, 2021). One consequence of LLMs as a superposition of tasks is that the outputs may collapse to unintended sim- ulacra, behavior known as the "Waluigi effect" (Nardo, 2023).
2. Related Work
Theory and practice of in-context learning. There is rich literature which formalizes in-context learning under diverse definitions. For example, prior works study in-context learn- ing through a Bayesian framework for task retrieval (Xie et al., 2022a; Panwar et al., 2023; Zhang et al., 2023), martin- gales (Falck et al., 2024), optimizers (Akyürek et al., 2023; Oswald et al., 2023; Dai et al., 2022) and more (Reddy, 2024; Olsson et al., 2022). Other works confirm the theoret- ical framing of in-context learning by using it to implement a variety of algorithms and methods (Zhou et al., 2023; Ahn et al., 2023; Giannou et al., 2023; Wu et al., 2024; Laskin et al., 2022; Zhou et al., 2022), or to approximate general- purpose computing machines (Giannou et al., 2023; Wei et al., 2022).
Superposition has been defined in various related contexts of learning models. Feature superposition (Elhage et al., 2022) refers to a neural network's 's ability to represent multiple learned concept in a single neuron. Though our discovery of task superposition describes the same abstract idea, we stress that it is distinct from feature superposition because task superposition is most apparent in the final output of a model.
Superposition is also described as a way to store multi- ple models in a single set of parameters (Cheung et al., 2019), processing multiple inputs simultaneously (Shen et al., 2024a; Murahari et al., 2022). In our work, we demon- strate task superposition directly as result of language pre-training, without the necessity of additional adapters or decoding strategies.
To bridge the gap between theory and practice, many works have used these theoretical insights to study in-context learn- ing behaviors (Agarwal et al., 2024; Li et al., 2024; Choi & Li, 2024). Bai et al. (2023b) shows transformers are capable of in-context algorithm selection. We would like to note here that while one can think of superposition as an algo- rithm that outputs probability distribution that calibrates to in-context examples, the setting in Bai et al. (2023b) requires training on all such algorithm that calibrates the output distribution to in-context examples, which is likely not the case for pretrained LLMs since labels during LLMs training are in one-hot format. Therefore, while related, superposition cannot be explain by the framework from Bai et al. (2023b).
3. Problem Setup
Notation. Let X' be the input space, x E X, and y be the output space. We define task as a function g X + ); we denote a task answer g(x), and (z))( task example of the task g.
In-context learning setting. Let prompt 8m t x (m+1) [a<(1), (((()((()))))_.) x (m), (, be prompt consisting of m task examples and a query (m+1). M is a model that predicts M (prompt) E y. We say M learns the task in-context if ((prompt) = g(z(m+1)), i.c. M learns g and performs the mapping on the query (m++)).
Other works study the factors that influence how well mod- els can learn through context (Raventos et al., 2023; Chan et al., 2022; Wei et al., 2023; Lin & Lee, 2024; Min et al., 2022; Lyu et al., 2022; Lampinen et al., 2022; Dinh et al., 2022).
(m)+1))
3

---

LLMs can In-Context Learn Multiple Tasks in Superposition
Let LM be language model and let V be the vo- cabulary space (token space). Given prompt, in- stead of outputing y E y directly, LM outputs probability distribution over V, i.e., IM(prompt) = (P(ynlrrrmpt)).... P(yyy)lrrompt] for yj E V.j = 1...,, |>]] 9. In previous works (Brown et al., 2020; Xie et al., 2022b) all task examples in a prompt are from a single task g; LM would put most of the probability mass on single task answer.
We provide GPT-3.5 (Brown et al., 2020), Llama-3 70B (AT@ Meta, 2024) and Qwen-2.5 72B (Bai et al., 2023a)
with prompts of uniform mixture of tasks (each task has 20 examples in the prompt ordered randomly). For each prompt
consisting of in-context task examples (e.g., "11 + 26 + 37" for the first task in the first setting) and a query (e.g.. "91 +-3 -)), we calculate the probabilities of outputs when correctly performing each task on the query and plot the distribution of probabilities for each task in Figure 2. Details calculating the probabilities is in Appendix C.
We are the first to study setting that task exam- ples in single prompt are from K different tasks, ie.. we have tasks 91....KK which form prompt = (x(). (x(")) (m)) g((m) (x((m) (()(m)))x(mtt) xf
Figure 2 reveals that in all four sets of tasks, all models have non-negligible median values of probabilities for at least two tasks. This indicates that the models can in-context learn multiple tasks in superposition when provided with prompts of mixture of task examples.
where gb E (g), (9),...9k)) 9x). Since LM outputs a probability distribution over V. we calculate the probability mass it assigns
each task answer 91 (2)m)))) ),..,, (x(mm))))) In Figure we depict (P (2(mmt)))proppttt promptt..... P(g) (m(mm)))pproomp and find that LM assigns non-negligible probability on multiple task answers. We term this phenomenon task superposition.
We can also observe that, even though every task in a prompt has an equal number of in-context examples (20 examples), LLMs do not calibrate their output distribution perfectly with the in-context task example distribution and they still have bias on what task to perform. For example, Figure 2a(left) shows that Llama-3 70B prefers performing numer- ical addition over addition in other languages, Qwen-2.5 72B is not good at addition in Spanish while GPT-3.5 does not have a strong preference over a single task. On the other hand, in Figure 2a(right) GPT-3355 has a strong preference over the capital task.
4. LLMs Are Superpositions of Multiple In- mn-Context Learners
In this section, we want to investigate if existing pre-trained models exhibit superposition of multiple tasks and whether this phenomenon is common.
Additionally, some tasks are "harder" than other tasks. For example, in Figure 2b(right), all models assign low probabil- ities for task answers of 1 other has relatively high values for all models, indicating high noise when prompted with in-context examples of this setting, In contrast, in Figure 2b(left), category other has very small values, indicating that all models most of the time would correctly assign the output probabilities to the correct answers.
The category
LLMs can in in-context learn multiple tasks in superposition when provided with prompts of a mixture of task examples.
Finding 1:
We consider four different settings of task mixtures.
1. Numerical addition and addition in English, French or Spanish (K = 4). Example prompt is shown in Figure Ia (left).
In Appendix B, we further show that, within the same LLM family, larger models can solve more tasks in parallel and better calibrate the output distribution to the in-context task examples distribution.
2. Given a name of a country, name the capital, continent or capitalize the country name (K = 3). Example prompt is shown in Figure la (right).
3. Given input "(op1)e(op2) opl and op2 (K- = 3). Example prompt is shown in Figure lb (left).
copy op1, op2 or add
5. Task Superposition in Models Trained from Scratch
In Section 4 we investigated task superposition in pre- trained LLMS at inference time. We further investigate how task superposition emerges in LLMs during training. Specif- ically, if we train the model to in-context Iearn one task at a time, can it perform task superposition when provided with prompts containing examples of multiple tasks?
4. Given word (with each letter separated by a space), output first letter or last letter in lower or upper cases (K = 4). Example prompt is shown in Figure lb (right).
'In practice LM outputs distribution over V but for simplicity here we assume inputs and outputs are single-tokens, i.e., X Y, and it naturally generalizes to the case when outputs are multi-tokemt using probability chain rule.
To answer this question, we train a small GPT- 2 model (6 heads, 6 layers, ~ ~14miilion param-
4

---

LLMs can In-Context Learn Multiple Tasks in Superposition
Meta-Liama-3
Qwen2.5-728
0.8
0.8
probebiilty
0.4
0.2
0.2
3dd
add
36d
add
other
capital
continent
capl: talization
other
(a) (left) Addition in original numerical form and in different languages in Figure Ia (left); (right) Capital name, continent name and capitalization as in Figure Ta (right)
probabbiity
0.4
0.4
0.2
0.0
0.0
copy(op))
copylop2)
letter
lotter
first lotter cap.
last cap.
other
(b) (left) copy (opl), copy (op2), and opl+op2 as in Figure lb (left); (right) First or last letter in upper or lower cases as in Figure 1b (right).
Figure 2: Distributions (0025/50775/100--erreetl)) of probabilities for correct outputs of each task. For every set of tasks, we tested with 100 prompts and for each prompt, every task has 20 random in-context task examples with order randomized like in Figure Category other the sum of probabilities of all other outputs. Gray dashed line in each figure is the ideal probability if we assume the model perfectly calibrates its output distribution the distribution of in-context task examples. With uniform distribution of task examples, the dashed lines are 0.25 (4 tasks setting) and 0.33 tasks setting).
eters) (Radford et al., 2019) to learn fam- ily of retrieval tasks. The input has the form "(ch1) (ch3) where ch1, ch8 are distinct single characters. We consider 8 retrieval tasks retl, ret8 where retl is output ch1 and in-context learn one task (retrieve one of [ch1,... ..., ch8) at time in training. Namely, during training, the model is only provided with text data such that each prompt only contains in-context examples of single randomly chosen task (and different prompts can correspond different tasks).
choose ret2 and ret 6) and see if the model performs task superposition. We vary the proportion of in-context examples of two tasks and plot the output distributions in Figure 3a.
(ch6)) (ch)))))
Similarly, consider second setting involving 10 tasks. Given two digit integer input num, task p plus outputs num, task plusi outputs num + 1 and so up to task plus9. The model is trained to in-context learn one of pluso..... plus 9 at a time, following the procedure above. During inference time, the model is tested with prompts containing mixture of in-context examples from tasks p plus2 and plus6. We vary the mixture ratio and show the output distributions in Figure 3b.
The model is trained to
Concretely, for each sample, we randomly select task E (retl,.... ret8) and inputs (!)) (mm) where each a00) is an eight-character long string. We train the model LMg parametrized by using ICL training by minimizing the following objective:
Finding 2: Transformers can in-context learn multiple tasks in superposition even if trained in-context learn one task at time.
Remarkably, from Figure 3a and 3b, GPT-2 trained from scratch to in-context learn one task a time can generalize to simultaneously performing multiple tasks and calibrate the output probabilities according to the in-context task example distribution when provided with a mixture of in- context examples. For example, in Figure 3a at the mixture ratio = 0.5, meaning that 50 percent of the examples in the prompt from task ret and the other 50% comes from task ret 6, we can see the output probabilities for task
m-
min E, 0
m CB(LMs(6 (8, 2 x(1t1))
(,(())))(()))))) g((x))))))
where CE is the cross-entropy loss and 8j ① +0+1)
[x( gt (x))
())),,,, (11), (x))),, x(+))))))))))))))))) ((+1)).
After training, we provide the model with prompts con- taining in-context examples of two tasks (in partieular, we
5

---

LLMs can In-Context Learn Multiple Tasks in Superposition
1.0
superposition at test time. Below we outline the main ideas used in the proof.
rett
0.8-
other
0.4
Prediction based on multiple tasks. Assume that we are
0.2-
given m in-context samples (a+") (æi1)...., .....,
()) 2-22 2: '=', (())
0.5
where '=' represents a specific value used only for preceding the label, and a set of l different Transformers TF; which can implement the T different desired tasks, where each deterministic task is denoted as g: with E [kt] and j E [m], i.e. y = gi (ax()) for some task dependent on sample j. Using the weights of each TF; we can compute outputs of the following form:
(a) Trained on retrieval tasks and tested on prompts with mixture of in-context examples of ret2 and ret 6.
1.0
pluss
pluss
0.6
0.4
gilt 1/
0.0
5.2
0.4 0.6
1.0
(b) Trained on addition tasks and tested on prompts with mixture of in-context examples of plus2 and pluss. 5.
We use the I norm to aggregate the prediction, in case that the task multi-dimennional. These differences are used to identify tasks, as (مg y = 0 for y coming from task i. Different heads at each layer in the model are used to execute each of the tasks in parallel using the weights from TF, In Appendix E we construct tasks where an arbitrary function gi (zx(²) ) for some fixed I that is task-specifie.
We consider two different settings of tasks: (a) given an eight-character length string as input, consider retl, ... ret where I to retrieve the first charac- ter and so on; and (b) given a two-digit integer as an input, consider pluso. .... pluss 9 where plus0 is to add 0 the input and so on. After training, for each setting, we select two tasks ad we provide the model with prompts con- taining in-context examples from these two tasks and vary the mixture ratio X such that the in-context task example distribution for two tasks [X, 1 - A). We plot on x-axis and the output probabilities of task answers for each task on y-axis.
Figure 3:
is implemented using ReLUs
Creating task identifiers. Having the differences between the implemented function and the label, we first use the ReLUs to clean up the vectors U2 so that only the positions in each vector that are associated with a task are maintained and the rest are set to 14. We thus create the vectors (v ll (2x - az [| and (v(). 1 otherwise. Now use ReLUs to threshold and create an indicator vectors 1 law ()(? (1110))))))) which identify
answers of ret2 and ret 6 being roughly [0.5, 0.5). We can observe similar behavior in Figure 3b.
6. Transformers Have the Capacity to Perform Task Superposition
the task, these are task identifiers. Notice that if the task is correctly predicted then the difference should be close to 0 (up to some error), while the task is not identified the corresponding value would not be O; the rest of the rows would be 1. We have created one vector for each task, which has 1 in the position of the corresponding task if the task was identified in the context.
In this section, we explore whether Transformers have the inherent expressivity to perform multiple tasks in superposi- tion with a single inference call. To this end, we provide a theoretical construction of a Transformer which, given the ability to implement multiple tasks, performs task superpo- sition depending on the examples given in-context.
Theorem 1. A seven layer transformer with embedding di- mension (d + log(mn)) with K heads per attention layer can perform K tasks on vectors of dimension d in super- position, with weighting based on m different in-context examples each of length n.
Averaging and task superposition. As a last step, we average all the task identifiers and place the result in the last column, in which the next prediction will happen. We then use the averaged task identifier to weight the prediction of each task based on it, in task superposition. If the task has been identified multiple times in the context, it would be assigned higher weight/probability.
The proof of Theorem is provided in Appendix E.4. Note that while this does not guarantee that training a Trans- former will actually find these parameters, it does indicate that Transformers are expressive enough to perform task
"This step is not mandatory, but it ensures that have no values over which we have control. We leave as future work an error analysis on how these values could affect the task identifiers.
6

---

LLMs can In-Context Learn Multiple Tasks in Superposition
(a) copy (op1), , copy (op2) and op1 top2 as in Fig- ure 1b (left).
100
23
5000 500 20
50y0
0000 300.
(b) Given a two-digit integer, task to.fr translates it to French, task to_de translates it to German and task to it translates it to Italian.
Componerts
-5
For each set of tasks, we collect the task vectors for each individual task and task vectors extracted from prompts that contain examples of different tasks. In Figure 4, we project task vectors along two axes chosen by linear discriminant analysis (LDA).
-12
-15
(a) copy (opl) copy (op2) oop+op2
LMV
Finding 3: LLMs internally combine task vectors during task superposition.
5/01
10
0.309 330.33
Interestingly, we observe that the locations of task vectors of mixture of tasks strongly correlate with the locations of task vectors for each individual task and the in-context task example distribution (the mixture ratio for examples of different tasks). For example, if the prompt includes an equal number of in-contex examples from each task, the task vectors are roughly centered in the middle; if the prompt only contains in-context examples of two tasks, then the task vectors roughly lie on the connecting line between task vectors of two individual tasks. We argue that this observation is indicative of the fact that, when prompted with mixture of in-context task examples, LLMs internally combine task vectors.
Componat
6-
-00
Democment 1
(b)
Figure 4: Task vectors of Llama-3 8B projected onto two axes chosen by LDA for two sets of tasks: (a) copy (op1), copy (op2) and op1+ap2 and (b) :o-fr, tolde and to.it. For tasks &1, t2, t3, we use (Ptt))/((t)/PP((t)))) denote different levels of task mix- tures, e.g., "0.50/0.50/0.00" represents the case where the in-context task examples are 50% 41 50% to and 0% i3-
As we observe signs that LLMs internally compose task vectors, we want to further investigate whether we can re- produce the task superposition phenomenon by patching in a convex combination of task vectors. For example, for tasks copy (opl) and copy (op2), , we first extract the corresponding task vectors V copy topl: and V copyy(uu (up2) on Llama-3 8B using the method described in Appendix D. We then make a convex combination of the two task vectors with parameter À that controls the ratio:
7. Task Superposition Through the Lens of Task Vectors
While in Section 6 we provide an existential result by con- structing a Transformer that performs task superposition and shows that task superposition is well within the expressive power of Transformers, we would like to further investigate how task superposition manifest in pretrained LLMs inter- nally. In this section we explore the underlying mechanisms that LLMs employ during task superposition. In particular, we focus our empirical study on task vectors (Hendel et al., 2023) where the detailed implementation in Appendix D. Task vectors are vectors the embedding space and found to encode the algorithm that model internally implements to solve task given in-context demonstrations.
V Vinterpolate, A X V copy (opl) (1 X) V copy (op2)
Convex combinations of task vectors produce task superposition.
Finding 4:
For a new query (in this scenario in the form "I "(op1) (op]])(op2)=") we patch the vector Vintepolate, A into the model at the task vector layer. We caleulate the model output probabilities that correspond to each task while we vary A. For each X, we use 100 different queries and plot the average probabilities in the top row of Figure 5. As a comparison, in the bottom rows of Figure 5, we plot the cor- responding output probabilities when providing the models
We want to investigate if there is any relation between the task vectors of each individual task and the task vectors of a mixture of task examples in the prompt. To this end, we consider two sets of tasks:

---

LLMs can In-Context Learn Multiple Tasks in Superposition
copylop1
copylop2)
other
to_de
to
other
1.00
0.75
0.75
Plans)
P(ans)
0.50
0.50
interpolation
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
1.00
1.00
0.75
0.75
in mn-context
P(ans)
Plans)
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
lambda
lambda
(a) Tasks: copy (op1) and copy (op2)
(b) Tasks: translate to_de and to_it
Figure 5: On Llama-3 8B, we vary the proportion, X, between two tasks and observe how the output probabilities for the correct answers change. The proportion À is varied in two ways: (1) in the top row, we plot the output from patching in a convex combination of task vectors for two tasks. (2) in the bottom row, we plot the output from a mixed proportion of in-context examples for the two tasks. Subplot (a) shows the output probabilities from mixing two copy tasks and (b) shows the probabilities from mixing two translate tasks.
with prompts containing mixture of task examples where the mixture ratio is controlled by A.
collapse," a phenomenon where, after the first token gen- erated, the model tends to converge on predicting tokens for single task, effectively negating its ability for multi-task execution.
In top row of Figure 5, we observe that patching convex combinations of task vectors into the model produces task superposition. We would also like to point out that in Figure 5b, although irrelevant outputs sum up to large probability, the task answers for two tasks to de and to cases will still be the top-2 answers.
This collapse presents substantial challenge in harnessing the full power of task superposition. highlights a critical area for future research: developing decoding strategies that can maintain the model's multi-task state throughout the gen- eration process. Recent work by Shen et al. (2024b) offers some hope that this direction may be fruitful, by proposing "superposed decoding" algorithm. Their method efficiently generates multiple streams of tokens from a single infer- ence pass by utilizing superposed token embeddings. While this approach represents significant step forward, also highlights the potential for further innovation in this area.
most
Comparing the top rows and the bottom rows, that top rows (the scenario of interpolating task vectors of in- dividual tasks) have larger probabilities of irrelevant output (category other). Task vector interpolation also produces less a linear relationship between and the output proba- bilities. This shows that while convex combinations of task vectors sufficient for producing task superposition, this does not fully explain task superposition. We leave it to future work to investigate other mechanistic explanations of task superposition.
9. Conclusion
We report on the discovery of task superposition, which is the ability of LLMSs to simultaneously solve distinct tasks from in-context examples. Task superposition is present in variety of pretrained models, and becomes more ac- curate at predicting the distribution of tasks as the model size increases (Appendix B). We also find evidence that while displaying task superposition, models internally mix the task vectors of each individual task. We hope that our findings will contribute to understanding in-context learning mechanisms and enhance our knowledge of LLMs overall.
8. Limitations and Future Directions
One limitation of our work is the current gap between the demonstrated capability of LLMS to perform task superpo- sition and its practical application in real-world scenarios. While we have shown that LLMs possess the capacity to ex- ecute multiple tasks simultancously, conventional decoding algorithms are not equipped to fully leverage this capabil- ity. This limitation stems from what we term "generation
8

---

LLMs can In-Context Learn Multiple Tasks in Superposition
Acknowledgment
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry,
K.L. is supported by NSF Award DMS-2023239, NSF CA- REER Award CCF-2339978, Amazon Research Award, and grant from FuriosaAI. D.P. is supported by NSF Medium award No. 2403074, ONR Grant No. N00014-21-1-2806
G., Askell, A., Agarwal, S., Herbert- Voss,, A., Krueger,
G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C.,, Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I,, and Amodei, D. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc.,
and No. N00014-23-1-288
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
2020. URL https::/ppocceeddeeeeeee neurips. cc cc/paper_ /ttpappe__flleepppprr//////////e 44470066fcc04004418800eele222222 html.
/
References
Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., McClelland, J., and Hill, F. Data distri- butional properties drive emergent in-context learning in transformers. Advances Neural Information Processing Systems, 35:18878-18888 2022.
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Chan, S.,
Anand, A., Abbas, Z., Nova, A., Co-Reyes, J. D., Chu, E., Behbahani, F,, Faust, A., and Larochelle, H. Many-shot in-context learning, 2024.
Cheung, B., Terekhov, A., Chen, Y., Agrawal, P., and OI- shausen, B. Superposition of many models into one, June 2019. URL http:/ arxiv. org/ httg/bbs/1902. 05522. 19Xiv:1902.05522 [es].
Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. Transformers learn to implement preconditioned gra- dient descent for in-context learning. arXiv preprint arXiv:2306.002297, 2023.
AI@ Meta. Llama 3 model card. 2024. URL
Choi, H. K. and Li, Y. Picle: Eliciting diverse behaviors from large language models with persona in-context learn- ing. (arXiv:2405.0255) 2405.02501. URL http: : / arxiv. org/ wwgg/bs//// abs/2405. 02501. arXiv:2405.02500 [es].
htttps:/ggitie wttph/uuoooo///// www/motam//////////////..o at 1aellaaaaaaaaa aaaaammammmm/////aaaa/aaaaaaaa/a blob/ma i /MOD/MOD_CCCCCA
md.
May 2024. doi: 10.48550/arXiv.
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? Inves- tigations with linear models, May 2023. URL http:// // rexiv.orge abs 22111.211111 15661. arXiv:2211.15661 [es].
Dai, D., Sun, Y., Dong, I.., Hao, Y., Sui, Z., and Wei, Why can gpt learn in-context? language models secretly per- form gradient descent as meta optimizers. arXiv preprint ar 10559, 2022.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F,, Hui, B., Ji, L.,, Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang. A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang. Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, Т. Qwen technical report. arXiv preprint 2r0i 16609, 2023a.
Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Rajput, S., Gira, M., Sohn, J.-y., Papailiopoulos, D., and Lee, K. Lift: Language-interfaced fine-tuning for non-language ma- chine learning tasks. arXiv preprint arXiv:2206006565. 2022.
Elhage, N., Hume, T., Olsson, C.,, Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy Models of Superposition, September 2022. ht: //axivv..v org/ abs / 2209. 10652. arXiv:2209. [cs].
Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. Trans- formers as statisticians: Provable in-context learning with in-context algorithm selection, 2023b.
Bai, Y., Huang, H., Piano, C. S.-D., Rondeau, M.-A., Chen, S., Gao, Y., and Cheung, J. C. K. Identify- ing and analyzing task-encoding tokens in large lan- guage models. (a1Xiv:2401.11323), February 2024. doi: 10-48550kaxiw2240111333..
Falck, F., Wang, Z., and Holmes, C. Is in-context learning in large language models bayesian? martingale perspec- tive. (arXiv:2406.00799) June 2024. URL http: arxiv. arxiv.org/abse [es, stat].
URL http:/ /rrxiv. org/abs/ /201 11323. arXiv:2401.1 [cs].
00793. arXiv:2406.00793

---

LLMs can In-Context Learn Multiple Tasks in Superposition
Giannou, A., Rajput, S., Sohn, J.y,, Lee, K., Lee, J. D., and Papailiopoulos, D. Looped transformers pro- grammable computers, 2023.
Lyu, X., Min, S., Beltagy, I., Zettlemoyer, L., and Hajishirzi, H. Z-icI: Zero-shot in-context learning with pseudo- demonstrations, arXiv preprint arXiv:2212.09865. 2022,
Grazzi, R., Siems, J., Schrodi, S., Brox, T., and Hut- ter, F. Is mamba capable of in-context learning? (arXiv:2402.031 April 2024. doi: 10.48550/arXiv. 2402.03170. URL http: /ttpttt::p///....t////a arxiv. wwt.ooogggab/aooooo.oooooe 03170. arXiv:2402.03170 [cs].
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? (arXiv:2202. 12837), October 2022. doi: 10.48550/arXiv. 2202. 1283.7 ht.ht:::: /ttpp:////xiiii http://wwwv........e..../ www..orgg/aao.... 2202. 12837. arXiv:2202.12837 [es].
Gu, A. and Dao, quence modeling with selective state spaces.
Mamba: Linear-time se-
moire. Language models are multiverse multiverse gewerators January 2021. URL http:///igg////eeeeeeeeeeeee//ta
(arXiv:2312.00752), December 2023. doi: 10.48550/ arXiv.312.00052.. URL http::t/txiii.....///// www..orggaabo... 1312.00752 arXiv:2312.00752 [es].
Murahari, V., Jimenez, C. E., Yang, R., and Narasimhan,
Hendel, R., Geva, M., and Globerson, A. In-Context Learn
K. Datamux: Data multiplexing for neural networks. MtXXv:2202.093118 November 2022. doi: 10,48550/ DxXiv2202.093188 URL http: www.a.iwgggaaaaaaeaaaaa 2202. 09318. arXiv:2202.09318 [cs].
ing Creates Task Vectors, October 2023. URL http:// arxiv. org/ abs 231 0 15916. 1511:2310.15916 [es].
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S.,
Nardo, C.
The waluigi effect (mega-post),
Schmidt, L.,, Hajishirzi, H., and Farhadi, A. Editing Mod- els with Task Arithmetic, March 2023. URL http:// arxiv.org/ abs / 2212 04089. arXiv:2212.04089 [es].
2023.
URL htttp::/www.lessttt
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. In-context learning and induction heads. arXiv preprint 0rXiv:2209./1888 2022.
Janus.
Simulators, 2022.
URL
https:
wwwwwons..ongg.o cww.:/nnccoooooooooeeeeeoeeeeeeo wwt.ccom/ooooo. postss/ VJFdjiggmeXWhNTsx/
Lampinen, A. K., Dasgupta. I., Chan, S. C., Matthewson, K., Tessler, M. H., Creswell, A., McClelland, J. L., Wang. J. X., and Hill, F. Can language models learn from ex- planations in context? arXiv preprint arXiv:2204.02329, 2022.
Oswald, J. V., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers Learn In-Context by Gradient Descent. In Proceedings of the 40th International Conference on Machine Learning, pp- 35151 PMLR, July 2023. URL http::://ccccceeeeeeeoooooee mir. press/ osww oswa d233 html. ISSN: 2640-3498.
Laskin, M., Wang, L.., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et In-context reinforcement learning with algorithm distillation. arXiv preprini 110v:22200.142155 2022.
Panwar, M., Ahuja, K., and Goyal, N. In-context learning through the bayesian prism, June 2023. URL https:
//arxiv. wwwpxv..rgg//.. org/ abs /2006.306622000 04891v2.
Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. Long- context Ilms struggle with long in-context learning. April 2024. doi: 10,48550/arXiv. 2404.02060. URL http::/// 02060. arXiv:2404.02066 [cs].
Park, J., Park, J., Xiong, Z., Lee, N., Cho, J., Oymak, S., Lee, K., and Papailiopoulos, D. Can mamba learn how to learn? comparative study on in-context learning tasks. SKGV:24002.422443 April 2024. doi: 10.48550/ar 2402.04246 URL http: : //axivvv..v...///// www.wi.org/aaa.aaaa.eooeooeooe.. www.oorg//bb///oee 04248. arXiv:2402.04248 [es].
/ttp///www.wro//////////// 2444.
Lin, Z. and Lee, K. Dual Operating Modes of In-Context Learning, February 2024. http: /aaaaaiiiv.ooo org/ 2bs/2402. 8819 9. arXiv:2402.18819 [es].
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Liu, S., Ye, H., Xing, L., and Zou, J. In-context vectors: Making in context learning more effective and controllable through latent space steering. 1.Xiv:2311.06660 February 2024. doi: 10.48550/arXiv. 2311.06668. URL http: /aa/iiiiivv.////.i//aa//////atva org abspg///b///// 23111. 06668. arXiv:2311 1..6666188688811 [cs].
Raventos, A., Paul, M., Chen, F., and Ganguli, S. The effects of pretraining task diversity on in-context learning ridge regression. In ICLR Workshop on Mathematical and Empirical Understanding of Foundation Models (ME- FoMo), 2023.
10

---

LLMs can In-Context Learn Multiple Tasks in Superposition
Reddy, G. The mechanistic basis of data dependence and abrupt learning in in-context classification task. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: htttps::/oppnreeee wpwpp//eeeee....ooooee/e..oe...m nett/ n forum? d=
48350/aanXiv22111122226 http:// 02080. arXiv:2111.02000 [cs].
Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An Explanation of In-context Learning Implicit Bayesian Inference, July 2022b. URL http: arxiv.org abs / 21 11 02080. arXiv:211 1xXiv:2111002000 11.02080 [es].
Reynolds, L. and McDonell, K. Multiversal views on lan- guage models, February 2021. URL http: arrivv. org/abs 2102. 0 6391. arXiv:2102.0639) [es].
Yang. L., Lee, K., Nowak, R., and Papailiopoulos, D. Looped transformers are better learning learning al- gorithms. arXiv preprint arXiv:2311.12424 2023.
Shanahan, M., McDonell, K., and Reynolds, L. Role play with large language models. Nature, 623(7987):493-4988 2023.
Zeng, Y., Kang, W., Chen, Y., Koo, H. I., and Lee, K.
Can milms perform -ixt-to-image in-context learning? (arXiv:2402.00 (13),, April 2024. doi: 10.48550/ar 2402.01293, URL http:/ /arxiv.vvvvvv///i//eeaa/eaaaavva org/ aaa/bb//22/ abs/2402. 01293. arXiv:2402.0129) [es].
Shen, E., Fan, A., Pratt, S. M., Park, J. S., Wallingford, M., Kakade, S. M., Holtzman, A., Krishna, R., Farhadi, A., and Kusupati, A. Superposed decoding: Multiple generations from a single autoregressive inference pass. (00.::2200.1184000) May 2024a. doi: 10.48550/arXiv. 2405. 18400. URL ht:t:// t a arxiv. www.iv.orrg/b/////eee 18400. 1rXiv:2405.18400 [es].
Zhang, Y., Zhang, F., Yang, Z., and Wang. Z. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. (atpe3055...112)) 19420), October 2023, doi: 10.48550/arXiv. (arXiv:2305. 2305.19420. htRp:t/::// /tte:///axxii arxiv org/ 2305
Shen, E., Fan, A., Pratt, S. M., Park, J. S., Wallingford, M., Kakade, S. M., Holtzman, A., Krishna, R., Farhadi, A., and Kusupati, A. Superposed decoding: Multiple generations from single autoregressive inference pass. arXiv preprint arXiv:240. 18400, 2024b.
19420. arXiv:2305.1942) [cs, stat].
Zhou, H., Nova, A., Larochelle, H., Courville, A., Neyshabur, B., and Sedghi, H. Teaching algorith- mie reasoning via in-context learning. arXiv preprint arXiv:221 20..2221110006. 2022.
Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. (arXiv:2310. 15213), February 2024. doi: 10. 48550/arXiw.2110..555 152133 http:t:://atttttttttttt /ttp::://///v///aeaeeeeeoo org/ abs/2310 15213. arXiv:2310.15213 [es].
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, P. What algo- rithms transformers leam? study in length general- ization, arXiv preprint arXiv:2310.166 16028, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A. Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Ilama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Wei, C., Chen, Y., and Ma, T. Statistically meaningful approximation: case study on approximating turing machines with transformers. Advances in Newral Infor- mation Processing Systems, 35:12071-12083, 2022.
Wei. J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.c 038466 2023.
Wu, J., Zou, D., Chen, Z., Braverman, V., Gu, Q., and Bartlett, P. L.. How many pretraining tasks are needed for in-context learning of linear regression? In International Conference on Learning Representations (ICLR), 2024.
Xie, S. M., Raghunathan, A., Liang, P., and Ma, An explanation of in-context learning as implicit bayesian inference. 1arXiv:2111.00 11.02800, July 2022a. doi: 10.

---

LLMs can In-Context Learn Multiple Tasks in Superposition
A. Notations
Notation
Description
K
Number tasks
I
Length of task's output
E.
layer for model
Number of in-context examples
Length of each in-context example
V
Token vocabulary
9:(.)
Operation performed by Task i
x
Data for example j
y
Label for example m in-context examples
8m
f()
Model (predictor)
P
Positional encodings
12

---

LLMs can In-Context Learn Multiple Tasks in Superposition
B. Task superposition capabilities as the model scales
Finding 5:
Within the same LLM family, bigger models can solve more tasks in parallel and better calibrate ICL distribution.
We want to further investigate how models' task superposition capabilities changes as the model size scales. In particular, we investigate two questions: 1) whether larger models can perform more tasks in-context and 2) whether larger models can align their output distribution more closely with the distribution of task examples provided in the prompt. We chose the Qwen-2.5 model family since contains several model sizes ranging from 0.5B to 14B parameters.
We first introduce a quantity which captures the capability of model to perform multiple tasks. Given prompt that contains examples of K tasks, define to be the number of these tasks whose correct answers appear among the model's top-K most likely outputs. Note that S K.
To see how close the model align the output distribution with the distribution of task examples, we use KL-divergence defined below:
P(x))
(L(PI|D) = Σ P(r) log
D(x)
=EX
where P is the models' probabilities on the outputs when correctly performing each task on the query and D is the in-context task example distribution. For example prompt in Figure la (left) gives = [0.5217, 0.1316, 0.1110, 0.2169, ... and
D [0.25, 0.25, 0.25, 0.25, 0, .....
We consider the setting of K 6 different tasks: given input of the form "(num}~ where num is two-digit integer, we consider tasks that output (1) num itself, (2) negation of num, (3) num + 1. (4) num 1, (5) num x and (6) num".
We choose the number of in-context examples m = 60 (each task has 10 examples) and configure the prompt with three different in-context task example distributions D1, D2 and Dy. In particular, Dj the uniform distribution, D2 has probability 0.5 on the third task and 0.1 on other tasks, and D3 is distribution with probabilities alternating between 0.25 and 0.083.
For each in-context task examples distribution Dr, generate 100 prompts and for each prompt we calculate the probabilities of outputs when correctly performing each task. The average values of and KL-divergence under three distributions are shown in Figure 6.
D1
divergence
D2
2
D3
D1
2-
D2
D3
0.5B
1.5B
3B
7B
148
0.5B
1.5B
3B
7B
14B
(the number of tasks whose correct appear in top-K most likely outputs).
(b) KL divergence.
Figure 6: (a) Average number of tasks completed, r, and (b) KL divergence for Qwen-2.5 model family under ICL distributions Dj, Dr and D3 where D, is the uniform distribution, D: has probability 0.5 on the third task and 0.1 on other tasks, and Da is distribution with probabilities alternating between 0.25 and 0.083.
In Figure 6a, we can observe that in general bigger models have higher r values. This shows bigger models will have more correct answers of tasks show up in their top-K K probable outputs and therefore they can solve more tasks the same time, In Figure 6b, we can see that for larger models like Qwen-2.5 7B and Qwen-2.5 14B, the KL-divergence values small, and for each model, the differences between KL-divergence values under in-context task example distributions Dj, D2 and D3 are small. This indicates that bigger models can better calibrate their output distribution to the in-context task example distribution.
13

---

LLMs can In-Context Learn Multiple Tasks in Superposition
C. Implementation details on calculating probabilities
In this section we provide details on how we calculate probabilities of different outputs given a prompt our setting.
Notations. Let be the token vocabulary, LM be am LLM, T be the tokenizer. We use "..." to represent a string. <... to represent single token where the content within the angle brackets is an integer representing token's index in vocabulary. For example, token <266> corresponds to "at". We use [<...>, Given a tokenizer, tok cook""supepppn ("superposition") = <,,71,, and detook[<16,,<10,,, (2,,,,,,11,1 "1+1=2".
>> <...>] to represent sequence of tokens. two functions tok(-) and detok(-) to tokenize strings and detokenize tokens. For example
In our in-context learning setting, an input string consists of in-context examples (separated by the delimiter "\n") and a query. For example, an example prompt can be (22122))22222423333==
We view an LLM as next-token predictor that outputs a probability distribution over the token space given imput and there a corresponding P--I-) such that given a sequence of tokens [Vi,..,, UMr] where vy V. IP [V1, uarl) measures the probability of the next token being where u E V.
Measuring the probabilities of task answers. Let / be the input prompt. For example, in the example in Figure la (left), the prompt is n3331113 numerical addition, 2) addition in English, 3) addition in French and 4) addition in Spanish. The corresponding task answers (the output correctly performing task on the query) are "174", "one hundred and seventy-four", "cent soixante-quatorze" and "ciento setenta cuatro", respectively. We want to measure the probability of each task answer.
( ...30+25 1..0025--fift--fivee (n9ty-
We consider four tasks: 1)
Let be task answer in string. Let [Di...., uM] := tok(I) and let [U1...., un] tok(o). Then the probability of the task answer given prompt can be calculated
Pu1 [Vi,... UM)) II
P(uj [ei,... UM: u1, ug-1))-
(3)
j=2
14

---

LLMs can In-Context Learn Multiple Tasks in Superposition
D. Implementation details on task vectors
use the task vector definition from Hendel et al. (2023). For example, for task copy (op1) in Figure lb (left), the procedure collect the task vector consists of
1. Collect dataset of 100 ICL sample prompts. Each prompt consists of m = 60 in-context examples of a particular task and a query x: mm++1)) Each task example (ax(), y()) follows the form " (cpp)=((pp]] where xn( has the form [[op]][[op2] } and yd is performing task copy (opl) on 20)) namely opl.
2. For each prompt s = [xx ,y
...., ....,.((" a(1), y ( (m(+1))] in the dataset, we feed s into the transformer model f. and extract the feature (which is vector) at the last "=" token in layer L. Call this vector {( E). Then we average {s; f) across all prompt s to get v()) for layer E.
3. Now for each layer l we have vector v()). We run forward pass with query in the form "(op1)e(op2)=" and we patch in v()) at the "=" token position in layer &, simulating the effect of complete context. We repeat this process 100 times for different query æ and get an accuracy accę of performing task copy (opl) with vector v()).
4. The task vector layer e* is selected by
e arg max acct,
and we define the task vector V Coppy (op1)
v((*).
Here we record the task vector layer where task vectors are extracted in Section 7.
Task
Task vector layer
copy (opl). copy (op2), opp+
14
to_de (, te_fr (opl), to_it (op1)
19
Table 1: Task vector layer for various tasks considered in Section 7.
15

---

LLMs can In-Context Learn Multiple Tasks in Superposition
E. Construction displaying superposition
In this section we construct a Transformer that is performing superposition of multiple tasks at inference. For this purpose, we first construct a Transformer that copies from n-tuple in-context examples the i-th one, as well as any function using the ReLU layers. We then create indicator vectors, for each task, which show whether a specific task present in-context or not, As last step, combine these indicator vectors to create the superposition of different tasks. Notice that using the parallel heads of the transformer architecture we can process each task independently until the last step in which the predictions are combined.
E.1. Overview
Here we provide a brief overview of how the construction is implemented, while latter we provide the corresponding details.
Prediction based on multiple tasks. Assume that we are given m in-context samples (a)) ...... a0 (3) '=', (0) where '=' represents specific value used only for preceding the label, and set of k different Transformers TF, which can implement the T' different desired tasks, where each deterministic task is denoted as (x(()) with i [K] and j E [m]] i.e. y (z) for some task dependent on sample j. Using the weights of each TF,, we can compute outputs of the following form:
y(
(
y( ll (2) y
1
0
0
0
0
0
(()r(a:(1) 1
We use the 4 norm to aggregate the prediction, in case that the task is multi-dimennsonal. These differences are used to identify tasks, as (x)")) 1 z 0 for gاو coming from task i. Different heads at each layer in the model are used to execute each of the tasks in parallel using the weights from TF. In Appendix E we construct tasks where an arbitrary function g: (3)) is implemented using ReLUs for some fixed I that is task-specific.
Creating task identifiers. Having the differences between the implemented function and the label, we first use the ReLUs to clean up the vectors 822 so that only the positions in each vector that are associated with a task are maintained and the rest are to 15. We thus create the vectors (v();: (9
2!"" 1 and (v)), 1 otherwise. Now to threshold and create an indicator vectors (= ( > 1000 which identify the task, i.e. these are task identifiers.
ReLUs
Notice that if the task is correctly predicted then the difference should be close to 0 (up to some error), while if the task is not identified the corresponding value would not be O; the rest of the rows would be 1. We have created one vector for each task, which has in the position of the corresponding task if the task was identified in the context.
Averaging and task superposition. As a last step. we average all the task identifiers and place the result in the last column, in which the next prediction will happen. We then use the averaged task identifier to weight the prediction of each task based on it, as in task superposition. If the task has been identified multiple times in the context, it would be assigned higher weigh//probability.
E.2. Task Identification
The first task for performing task superposition based on in-context examples is to define set of tasks that the model is able to implement.
First, the outputs of tasks need to be identified.
Lemma 1. Consider the following input
② (I)
(2)
(1)
g
(++1)
2n-2 2
X
0
0
0
0
0
0
0
0
*This step is not mandatory, but it ensures that we have values over which we have no control. We leave as future work error analysis on how these values could affect the task identifiers
16

---

LLMs can In-Context Learn Multiple Tasks in Superposition
where a:"" Rd-1 before the positional encodings are added, with one additional dimension that represents if the symbol an 'qquals' symbol. Then, a 1-layer transformer with single attention head and embedding dimension O + log(mn)) output
X
y
a
n-
=
y
21
(++1)
0
Proof. With positional encodings appended, let the input have the following structure:
(
(
y0
(++1)
11
2
nn-
0
0
0
0
0
X
(4)
Pn-1
Pjn+1
Pjn+2
Pjntn-2
Pjntn-1
Pjn+n
P(1+11411
Pn
Pjn
Pjn+1
Pjnen-3
Pjntn-2
Pjntn-1
P(j+1jn
To rotate the second position to the right, the following matrices.
0
0
WK
0
CT
0
0
0
W
0
0
0
The pair Wa and Wr k attend tokens to the token directly to the right. The value matrix simply filters only the second row in in-place. A second head can used to clear the original 1s, resulting in
(3)
g())
(++))
x
x
22
x n-
x
0
0
0
1
0
X
(5)
Pn+1
Pjn+1
Pjn+2
Pjntn-2
Pjntn-1
Pjntn
P(+t)m-1
Pn
Pjn
Pjn+1
Pjnta-3
Pjnta-2
Pjn-n-1
P(++1jn
as desired.
Implementation of functions. To illustrate set of operations that could be implemented with transformer, we consider approximating functions as sums of ReLUs; we use result from Bai et al. (2023b), which we present below.
Definition (Definition 12 in (Bai et al., 2023b)). A function g Ie* + R is (e, R. M, C)- approxxiiall by sum of ReLUs, if there exists an "(M,C) C)-sum of ReLUS" function
M
M Σ m=1
fM,c(x) = Σ
ReRLLU(a 11) with m=1
cml < C,
max me[M]
||am 1I S 1, am E IB++1 Cm E R
such that suPne[-R.Rip [9(x) f(M,0)(2)) Sc
Definition 2 (Definition A.1 in (Bai et al., 2023b)), We say function IR. - IR is (R, Ct)-smooth if for [(k - 1)/21 + 2, E 2 on |-R, Rt R and
V (()))())))))))
1 1 D ((x) s Li
max R.]- JI.
=E[- RJ
for all i = 0,1 with maxogigs L < Cj.
Proposition 1 (Proposition A.1 in (Bai et al., 2023b)). For any <> 0, R 12R2,, O, we have that: Any (R, -i)-smooth function, IR IR is (e, Rt M, C)-approximubbe by sum of ReLUs (Definition 1) with M s C(k)C log(1 Cit)/
"C" denotes a function times differentiable with continuous i-th derivative.
17

---

LLMs can In-Context Learn Multiple Tasks in Superposition
Lemma 2. For any function g : R IR that is (R, Ci)-smooth, there exists a transformer with layers, head and width (log(n) + d), where d satisfies the requirements of Prop. I, such that given as input
a (I)
y
(
y(
2 (1-1)
2 n- 2
X
1
0
0
0
0
it outputs
y
y
(1+1)
1
x
X
*
g(
(3)
y(
where [(x)) g(x)) S and for some
,,,..., 2].
Proof. We consider that the positional encodings added in the input and have
y
(++
22
n-
a
0
0
0
0
0
0
0
0
0
X
(6)
1
1
1
1
1
Pn+1
Pjn+1
Pjn+2
Pjn+n-2
Pjn+n-1
Pjutn
P(tt1)ut1
Patl-s
Pintl-s
Pjn+2-s
Pjotn-l-8
Pjntn-s
P(--1)+t1-
where we fix some positional encodings Pæ: where Pk Pk is larger than p& pi by some threshold for * I. The encodings used here are the binary representations of k E {-1, (__,,))og(mmm)) (log)mm)) Further, we consider 1s in the positions with the results of the task to differentiate the context of the task and the result of the task. Define s = n i, the distance between the result and the associated value in the context.
In the first layer, we use the MLP's to create ÿ according to Proposition
(
(j)
yb)
(++
1
222
n-
0
0
0
1
0
g
5(x
(1)
g(x2
(
g( 0
g(=)
g( (H())
X
0
0
0
0
0
0
1
1
1
1
Put1
Pjn+1
Pjn+2
Pjntn-2
Pjntn-1
Pjntn
P(+1)m+1
Pat1-a
Pjnt1-s
Pjn+2-s
P
Pjntn-1-a
Pintn-s
Pú+1)=+1-e
The next operation is shift of the sequence of g(-)'s to the right by s. This will align the desired output g(x) (0) with the observed output () Consider the following weight matrices
Wº
0
0
T
(8)
WR
0
CT
0
(9)
0
0
0
0
0
0
Wy
0
0
I
(10)
for some large constant to decrease error from the softmax attending to the incorrect tokens. This produces (within small error induced by using a softmax)
18

---

LLMs can In-Context Learn Multiple Tasks in Superposition
(x (X"W WoX),, = PutiPa-stt
(I)
as(X Us(XWW W& 1 (ntimn=s4))
(imj-s)
(12)
Wyx=
g(
g
g( 2-
g(=)
((y
()
(13)
Wi.Xos(X W W)
(14)
(
y
g(x
(
*
X + WrXos(X W ( WQX)=
0
0
0
Pjn+1
Pjn+2
Pjntn-2
Pjntn-1
Pjntn
Pjntl-s
Pjn+2-s
Pjntn-2-8
Pjntn-l-s
Pjatn-s
(15)
(16)
Each matrix above only shows the slice that contains the j-th in-context example. This is repeated for each of the other in-context examples.
As final step with an MLP, subtract row from row to achieve the following output:
y())
22
x
nー
1
g(x (
0
0
0
(17)
1
1
1
Pjn+1
Pja+2
Pjntn-2
Pjntn-1
Pintn
Pintl-s
Pjn42-a
P
Pin-n-l-s
Pintn-s
Copy Tasks As has been experimentally investigated, the situation where specific position within the context is copied the label can be easily implemented by setting g(x:) The dependence the subscript within the construction is what allows the position copied to vary.
E. IDENTIFYING IF TASK'S OUTPUT MATCHES THE IN-CONTEXT EXAMPLE
Lemma 3. three layer transformer with ReLU MLPs and embedding dimension Old + log(mn)) can calculate the proportion of in context examples that come from a specific task, where m is the number of in-context examples, each of length n and dimension d.
19

---

LLMs can In-Context Learn Multiple Tasks in Superposition
Proof. We now have matrix of the following form.
(
you
1
2
0
0
1
f(x
y
0
0
0
(18)
1
1
1
Pjn-1
Pjn+2
Pjntn-1
Pjntn
Pjn+1-s
Pjn+2-a
Pjntn-1-s
Pjn+n-8
the task correct, than f(x y(6)= x 0, with some small error coming from softmaxs and function approximation
enror. First, find the L1-norm of f(x ylu) using MLP. For calculating ||=|| for arbitrary 2, use
| Σ ReLU(z)) ReLU(-z,)
(19)
which can be done in single I-layer MLP. Thus, we have
y
22
0
1
f(x!
[l (1)
y 1 11
*
(20)
0
0
1
1
Pjn+1
Pjn+2
Pjnta-1
Pjntn
Pjn+1-s
Pjn+2-8
Pjstn-l-e
Pjn+n-a
Notice that if some task has different dimension than another task, the "extra" rows would be zero and will not affect the result.
For clarity, we set all * values in the || |/1 row to 1s. These will cause the following in the following set these to This operation can be omitted as the construction handles these trash values at a later layer.
Let represent the value of the flag in the second row marking the y vectors and a represent the values in the row with (ax()))) y The following ReLUs set the * values to 1.
- + ReLU(x Cb) - ReLUUCUb C
(21)
for some large constant C. When b = 0, this reduces to æ + 1, and when b 1, this reduces to æ + desired.
1 = as
(
(
a
22
0
1
f(x!
*
1
llf(z.
1 111
(22)
0
0
1
Pjn+1
Pjn42
Pjnta-1
Pinto
Pjn+1-s
Pjn+2-s
Pi+tn-1-8
Pjn+n-8
Now define a thresholding function 8(2) that satisfies 8()) and 8(2) for >> One such function used here is
20

---

LLMs can In-Context Learn Multiple Tasks in Superposition
8c(2) ReLU C2)
(23)
for some constant C, where larger captures narrower neighborhood of 0.
However, slight change needs to be added to dc. In the same row as (lf(((()) y are many values that need to be discarded. Let be the bit for the current column marking if the column contains an y. We use instead
(b 2) = ReLU(b - Cz)
(24)
This will be zero whenever = and 2 0. We then have as output
(
y())
222
0
1
fxx yo)
*
llf(
1
8c(1f(xz. (1) 10)
(25)
0
0
0
0
1
Pin+1
Pjn+2
Pjn+n-1
Pin+n
Pjn-1-s
Pjn+2-8
Pjntn-1-8
Pjn+n-8
Importantly, 8c (I)((()))) 11) when f(-) is the correct task and (0(( -000 (1))))> when f()) disagrees by more than & in L1-norm.
Lastly, for the next step in the construction, we need to average these soft indicators see how common f within the context. This is done with an attention layer. Let We select the row with all multiplied by some large constant C, and let WK k select the with flags for results y. Then
X
(26)
xx(X (XTWWT WoX) 2
(27)
1/m
1/m
1/m
1/m
where 1/m will appear in every corresponding result y. Let the value matrix select the row containing 3(11((x()) (l)) Denote p = 1 m Σm (z())(((()) Ib)) Without causal masking, we would have as output
21

---

LLMs can In-Context Learn Multiple Tasks in Superposition
1
2
0
1
fxx
1
(x)
111
0
8c (1(( (1)
y000 (11)
(28)
p
p
0
0
0
1
Pjn+1
Pjn+2
Pjata-1
Pjntn
P-n+1-8
Pjn+2-8
Pintn-1-8
Pjn+n-a
However, with causal masking, we can only guarantee that will appear in the columns containing the most recent example being queried. Thankfully, this is all that is needed.
E.3. Task Execution
Lemma 4. A two layer transformer, with embedding dimension O(d + log(mn)) can perform a task and weight its output by the proportion of examples of that task seen within the context.
Now that the proportions of each task have been identified in the context, the task itself needs to be executed for the new example being queried. To simplify notation, let the input to this step be
(m)
(m)
(m)
22
X
P
P
p
(29)
O
O
0
*
*
1
1
1
1
Following the same process as outlined above, although with slightly different positional encodings, calculate (m((m) and place that result the final column being decoded. These need to be added the beginning of the construction, but are only introduced here for clarity.
(m)
(m)
D2
n
n-11
p
p
p
P
(m)(
0
0
(30)
...
1
1
0
0
O
1
0
We will transform the row containing all to be able to approximately multiply p by f(x(m)). Using the second to last row, perform p - 1 - Using the last two rows, clear out the rest of that row and fill it with -C' for some large constant C.. then have
22

---

LLMs can In-Context Learn Multiple Tasks in Superposition
(m)
(m)
02
2-2
a
n-1
C
p
1 p
f(x!(l)
0
(31)
...
*
1
0
0
O
0
Further, use the second-to-last row to clear out all # in the rows below.
m
(m)
(m)
(m)
E2
*
1
f(x!m))
(32)
111
1
0
O
1
0
These previous operations can all be done in single MLP.
Lastly, use an attention layer where W k selects the row with the -Os, We selects the row with all 1s, and Wv selects the f(ali)). For the last token L.
-C
XWW
[1]
(33)
P
p
p
p
-00
as(X W Wo
(34)
P
I+e 4F
142
23

---

LLMs can In-Context Learn Multiple Tasks in Superposition
WVzLos(X Wx
(1
)f )f(x(
(35)
14e
I+c
8 (m)
1
(x (m))
I+
zy + WVZLOS(X Wi
(36)
Importantly, we are left with 1 (x) The factor T+ is approximately p. especially around 5.2 This multiplica- tion can also be calculated more accurately with approximations using ReLUs or sigmoids, but for brevity and following experimental evidence sigmoid shape task superpositions, these options are ommited.
E.4. Superposed Tasks with Parallel Heads
The above construction works for a single task, where the output is weighted by the proportions of the task within the context. complete the construction of transformer that does superposition of tasks, each of these models needs be placed within the same overall transformer. This is described here.
Let there be collection of tasks {t, which can be executed by transformers with model weights represented by subscripts (-.). With the input each transformer being (1), the overall input matrix is given by vertically stacking these matrices.
X1
X₂
X
(37)
XT-1
X
Similarly, define each MLP's weights and biases as
b1
W diag( WI....
Wx)
(38)
bt
This puts every MLP to be independent of each other. Lastly, we need to change the attention layers. This requires the use of one head per task. In each of the following, ( is a weight matrix for head i, (W); is the weight matrix for task its individual transformer, and each matrix below is in the i-th block.
24

---

LLMs can In-Context Learn Multiple Tasks in Superposition
W
(Wv)
(Wx);
(WQ)=
(39)
K
In all, this model executes multiple tasks in superposition by using parallel streams of heads that each performs single task. Task identification can happen through the same mechanism task execution by comparing the output of the task each in context example with the true output.
For context related tasks, there needs to be positional encodings that allow for looking back a fixed number of tokens. For context agnostic tasks, a wide MLP can be used to approximate arbitrary non-linear transformations of the input. Each of these tasks only require a small number of layers, significantly smaller than those of modern LLMs. may be possible that LLMs do certain tasks with different combinations of layers.
Also, if we take the feature p from each parallel stream, this creates the following task identifier.
P1
P2
V
(40)
PT
Interpolating between the pure tasks, represented by unit vectors, different amounts of each task will appear in the superposition in roughly equal proportions those found in v.
Lastly, we restate this construction formally.
Theorem 1. A seven layer transformer with embedding dimension O + log(mn)) with K heads per attention layer can perform k tasks vectors of dimension d in superposition, with weighting based on m different in-context examples each of length n.
Proof. Using in succession each of Lemma 1, Lemma' 2, Lemma 3, and Lemma 4, transformer with the desired properties can execute k tasks in parallel. Lemma identifies positions within the context that contain the labels Lemma 2 then uses function approximation to perform arbitrary tasks within the architecture, which are then used by to find the proportions of each task and aggregate them into a single task identifier. Lastly, Lemma uses this task identifier to create weighted sum of outputs from the different tasks based on their in-context proportions.
Remark. Transformers of greater depth than seven layers can also represent this construction by setting the weights in all other layers for the non residual part zero.
25

---

LLMs can In-Context Learn Multiple Tasks in Superposition
F. Additional experiment and analysis of task superposition in pretrained models
GPT-3.5
Llama-3 70B
Qwen-2.5 72B
1.011.00.000
1.0
Probability
0.65
0.50
0.50
0.5
0.33
0.000.000.00
0.000.000.00
0.010.010.01
0.0
add
add en
other
add
add en
other
(a) (left) All task examples from task add; (right) equal numher of task examples from add and add in en.
1.000.991.00
1.0
0.90
Probability
0.5
0.42 0.46
0.52
0.000.000.00
0.000.010.00
0.05
0.01 0.000.33
0.0
capital
continent
other
capital
continent
other
(b) (left) All lask examples from task capital; (right) equal number of task examples from capital and cont inntt.
1.001.000.99
1.0
Probabliiyy
0.67
0.56
0.53
0.5
0.47
0.44
0.34
0.000.000.01
0.00-0.000.00
0.000.000.00
0.0
copy(op1)
copy(op2)
other
copy(op1)
copy(op2)
other
(c) (left) All task examples task copy (opl) ) (right) equal number examples from copy (opl) and copy (op2).
1.001.001.00
1.0
Probability
0.77
0.65
0.50
0.50
0.5
0.35
0.23
0.000.000.00
0.00-0.000.00
0.000.000.00
0.0
first letter first letter cap
other first letter first letter cap
other
(d) (left) All task examples from task first_letters (right) equal number of task examples from first.letter and First_letter.cap.
Figure 7: For each subplot, consider two tasks taski and task2 and the left side plot the median probability for task answers where all task examples in prompts from taskl; the right side plot the median probability for task answer where task examples from task1 and from task2 in prompts are equal.
F.1. superposition really happening?
In Figure 2 we plot distributions of probabilities for each task in four settings. Note that in setting 3 (corresponds to Figure 2b(left),, the other category has low probabilities for all models but for settings I, and (correspond to Figure 2a(left), 2a(right) and 2b(right) respectively), the other category not always low. natural question to ask is:
For the non-zero probabilities we observe on each task answer, are they indications of task superposition or by-products of prediction noise?
26

---

LLMs can In-Context Learn Multiple Tasks in Superposition
We set up an experiment investigate this. For each setting, we select two tasks task1 and task2; then consider two scenarios: (1) provide prompts where all task examples come from task1 and (2) we provide prompts where half of the task examples come from taskl and the other half of task examples come from task2; in both scenarios measure the probabilities for task answers of taskl and task2 and see how these probabilities change between scenario (1) and (2), For each scenario, we test it on 300 prompts (each task has 10 in-context examples) and plot the median of task answers in Figure 7. As is shown Figure left side, where there is no task example from task2 in the prompt, the probabilities for task answers of task2 are near 0 for all models; on right side, where there an equal number of task examples that are from taskl and task2, the probabilities for task answers of task2 increase significantly. This indicates that when provide prompts that mix task examples from different tasks, the prediction each task answer is more than just pure prediction noise.
E.2. Measuring accuracy in task superposition
We further investigate how task superposition affect task performance. In particular, for prompt consisting examples of K tasks (each task has equal number of task examples), we define a task being correctly performed if its task answer lies in the top-K answers (that we use beam search to find). We compare the accuracy against individual task accuracy where we provide prompts consisting of task examples of only 1 task and define the task being correctly performed if the task answer the top-1 answer.
We calculate accuracy K case and > 1 case using 300 prompts and show the result in Table 2. We find that as we increase the number of tasks, all models exhibit an accuracy decrease in correctly performing each individual task.
27

---

LLMs can In-Context Learn Multiple Tasks in Superposition
Model
K =
K = 4
t1
t3
t3
t4
GPT-3.5
100
99.7
90
95
97.3
(-2.7)
90.7 (-9)
80 (-10)
80.3 ( 14.7)
Llama-3 70B
100
100
97
99
100 (0)
99 (-1)
95.7 (-1.3)
93 (-6)
Qwen-2.5 72B
100
100
87
96.7
99.7
0.3)
87.7 12.3)
87.7 (+0.7)
63.7 (-33) 33)
(a) Setting I: Addition in original numerical form and in different languages where t: = add, t2 = add in en, t3 = add in fr,
4 = add in es.
Model
K =
K
t2
t3
t1
t2
t3
GPT-3.5
97.7
100
100
93 (-4.7)
61.7 (-38.3)
42 (-58)
Llama-3 70B
98.3
100
100
85.3 (-13)
79 (-21)
75 25)
Qwen-2.5 72B
98.3
100
100
87.7 (-16))
78.3 ( 22.77)
30 (-70)
(b) Setting 2: Naming the capital, continent and capitalize the country name where t1 = capital, t2 = continent, t3 appitalizatioon
Model
K
1
K
t1
t2
t3
t1
t2
t
GPT-3.5
100
100
100
100 (0)
98.3 (-1.)
97.3 (-2.7)
Llama-3 70B
100
100
99.7
99.7 (-0.3)
99.7 - 0.3)
99.3 -0.4)
Qwen-2.5 72B
100
100
100
100 (0)
100 (0)
99 (-1)
(c) Setting t1 copy (opl), copy (op2) and t3 opl+op2.
Model
K 1
K 4
t2
t3
t4
t1
GPT-3.5
100
100
100
97
62
(-38)
79.7 (-20.3)
79 (-21)
24.3 (-72.7)
Llama-3 70B
100
85
100
59.3
85
15)
52.7
32.3)
95.3 -4.7)
39 (20.3)
Qwen-2.5 72B
100
98.7
100
97
53.3
46.7)
98.3 (-0.4)
42 (88)
84 (-13)
(d) Sctting First or last letter upper or lower cases where tl first.letter. t2 = letter, t3
cap, t laastletterrcaa
Table 2: Accuracy for each task in percentage, with the delta change given in parenthesis. For each setting we calculate the accuracy with prompts consisting of task examples of only one task (K case) and with prompts consisting of examples from multiple tasks (K case).
28


Reviews:
[{"review_id": "IupPmH0TGr", "reviewer": "Reviewer_UuyG", "summary": "This paper introduces the novel empirical finding that when presented with a context that contains a mixture of different tasks, an LLM will respond as though it is performing a superposition of those tasks. By training very simple small GPT-2-style models from scratch on artificial tasks where every training context contains only a single task, they give strong evidence that this superposition effect is a structural properly of this style of neural-network (i.e. is a part of its inherent inductive bias) and not coming from the specifics of its training data. They connect this to the task-vector point of view (coming from the linear representation hypothesis for tasks) by showing evidence of linear combinations of tasks being active at the same time.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UuyG,\n\nWe greatly appreciate your constructive feedback on our paper. Below, we address the specific questions raised in the review.\n\n**Figure 1(b)**\n\nThanks for pointing this out. We will update Figure 1(b) in our next revision.\n\n**Does ordering of tasks matter?**\n\nIn our setting of in-context learning, the order can affect how the model perform. For example, consider a scenario with three tasks, each presented in the prompt through 10 examples arranged sequentially: first 10 examples of task 1, followed by 10 examples of task 2, followed by 10 examples of task 3, and then the query. In this case, the model tends to assign higher probabilities on task answer of task 3. This is because there are 10 in-context examples of task 3 right before the query, and the model just follow the same pattern. However, if we randomize the order, the model won't just follow the task examples right before the query but it calibrates its output probability distribution based the in-context task example distribution in the prompt.\n\nWe are happy to elaborate further if you have any remaining concerns.", "claims_and_evidence": "Yes. But I didn't check the proof in the supplemental for the theoretical construction.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes. The experiments seem unambiguously clear.", "relation_to_broader_scientific_literature": "This paper is nicely connected to the discussion around Simulcra which have unfortunately (for academics) not taken place purely through papers. The fact that the authors reference blog posts on this topic is very good for the academic community.", "essential_references_not_discussed": "There is a collection of academic works on actually learning mixtures. For example: https://dl.acm.org/doi/full/10.1145/3583680\n\nIt would be nice to connect to this literature.", "other_strengths_and_weaknesses": "Very clear writing.", "other_comments_or_suggestions": "You have a bug in Figure 1. The task examples for b don't match the stated task.", "questions_for_authors": "Does the ordering of tasks in context matter? It feels like you used random shuffles. But given the \"U-shaped curve\" of how models (and humans) seem to pay more attention to the beginning of the context as well as the most recent parts, does this also influence the superposition?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fJhnQty5R5", "reviewer": "Reviewer_FHnf", "summary": "This paper investigates the \"task superposition\" phenomenon of ICL, i.e., when multiple tasks simultaneously appear in the context, the model can assign non-negligible output probabilities to more than one task. Additional findings and contributions include:\n1. Pretrained LLMs have bias on what task to perform when given multiple tasks in a context.\n2. On simple retrieval and addition tasks, transformers can in-context learn multiple tasks in superposition even if trained to in-context learn one task at a time.\n3. Theoretically, there exists a seven-layer transformer with sufficiently large embedding dimensions and $K$ heads per layer that can perform $K$ tasks in superposition.\n4. Adding the convex combinations of task vectors of individual tasks can induce a similar output distribution to that induced by using a superposition input context.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer FHnf,\n\nWe greatly appreciate your constructive feedback on our paper. Below, we address the specific concerns raised in the review.\n\n**Theoretical claims**\n\n> The proof is a constructive one, ..., there is no guarantee that Transformers will definitely implement such a construction to realize superposition ICL predictions.\n\nIt is true that our result is an existential result. The purpose of Theorem 1 is to show that task superposition is well within the expressive power of Transformers and Transformers can efficiently represent the superposition mechanism using a small number of layers. In Section 7 we further investigate the underlying mechanism and empirically show that LLMs combine task vectors during task superposition. We think it will be an interesting future direction to theoretically show the exact underlying mechanism for task superposition for pretrained LLMs.\n\n**Discussion with Wang et al.**\n\nThank you for bring up this work. We think Wang et al. [1] is indeed related to our work. [1] shows that ICL will identify the \"most suitable pretraining meta-distribution based on the test error and input distribution discrepancies\" and operates within that meta-distribution. The bias we observe during task superposition (LLMs do not calibrate their output distribution perfectly with the in-context task example distribution and different LLMs have different bias) can be explained by different pretraining distributions of LLMs, which [1] investigated in.\n\nHowever, we would also like to clarify that algorithm-selection mechanism does not fully explain task superposition. In the setting of [1], at inference time, all examples in the input are from a single task and LLMs select a task in the pretraining distribution that has the lowest test error with the given task; in our setting, examples in the input are from multiple qualitatively different tasks and the model predicts a superposition of different task answers. We will add the discussion of [1] in our next revision.\n\n**Practical value is limited**\n\n> Not only did the authors design any new methods, they didn't provide a concrete application scenario in which this task superposition capability could be utilized. Could you show me a specific real-world scenario where keeping the LLM in a multi-task state will be beneficial?\n\n\nA recent paper [2] shows a practical application of the task superposition capability. [2] propose a method where LLMs can generate continuous thought as reasoning state that simultaneously encodes intermediate results from multiple reasoning paths. For example, \n* Figure 4 of [2], when asked with a Math problem that can be solved by multiple ways (and we can view each way of solving the problem as a sub-task), LLM \"encodes a distribution of different traces into the continuous thoughts.\"\n* Figure 5 and 6 of [2] further shows that on some logical reasoning tasks, LLMs can explore multiple paths at the same time as a breath-first-search algorithm. This outperforms traditional chain-of-thought method that can only explore one path at a time.\n\nMoreover, we believe that explicitly characterizing the phenomenon of task superposition is valuable in its own right as it help us better understand LLMs.\n* Our observations align with the \"simulator-in-superposition\" hypothesis [3, 4] that emerged with the advent of GPT-3. This hypothesis suggests that LLMs can simulate multiple potential continuations or behaviors simultaneously, reflecting a superposition of different skills or tasks. By demonstrating that LLMs can internally represent and process multiple tasks in parallel when provided with mixed in-context examples, we provide empirical support for this theoretical framework.\n* Two papers [5, 6] released a few days ago also capture the superposition phenomenon in LLMs. [5, 6] found that when asked to do two-digit addition, LLM will split the problem into two tasks, internally employ parallel computational paths and then merge the result together. This indicates that superposition can be commonly found in LLMs and we hope our work can shed light on future research that study superposition in LLMs.\n\n**Typos**\n\nThank you for pointing this out. We will fix the typos in our next revision.\n\nWe are happy to elaborate further if you have any remaining concerns.\n\n**References**\n\n[1] Wang, Qixun, et al. \"Can In-context Learning Really Generalize to Out-of-distribution Tasks?.\"\n\n[2] Hao, Shibo, et al. \"Training large language models to reason in a continuous latent space.\"\n\n[3] Reynolds, L., & McDonell, K. (2021). Multiversal views on language models.\n\n[4] moire. Language models are multiverse generators, January 2021. https://generative.ink/posts/language-models-are-multiverse-generators\n\n[5] Ameisen, et al., \"Circuit Tracing: Revealing Computational Graphs in Language Models\", Transformer Circuits, 2025.\n\n[6] Lindsey, et al., \"On the Biology of a Large Language Model\", Transformer Circuits, 2025.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "N/A. This paper don't propose new methods.", "theoretical_claims": "I read the proof sketch outlined in Section 6, but I didn't thoroughly check the proof in Appendix E.4. \n\nIssues: the proof is a constructive one, i.e., there exists a choice of construction that can lead to the conclusion, but there is no guarantee that Transformers will definitely implement such a construction to realize superposition ICL predictions.", "experimental_designs_or_analyses": "Yes. The soundness and validity of the experimental designs and analyses are good.", "relation_to_broader_scientific_literature": "The key finding that LLMs can perform task superposition reveals possibilities for designing real-world applications of LLMs such as automatically inferring the desired task given a complex instruction context, etc.", "essential_references_not_discussed": "I notice that a recent work in ICLR 2025 [1] may further explain the finding in Section 4: \"LLMs do not calibrate their output distribution perfectly with the in-context task example distribution and they still have bias on what task to perform\". [1] investigated how ICL will select the training task priors to make predictions based on the test context and the pretraining distribution, and theoretically revealed that three factors will determine the task selection of ICL prediction: 1) ratio of each task in the training corpus; 2) the test error of each task on the test in-context examples; 3) the distributional distance between the training and test context inputs $x_i$. I believe further discussing how these task-selection mechanisms in [1] will function under your multi-task ICL setting would help to refine this part of your work. \n\n[1] Can In-context Learning Really Generalize to Out-of-distribution Tasks?", "other_strengths_and_weaknesses": "**Strength:**\n1. The studied task superposition problem is novel in the ICL literature.\n2. The experimental designs are sound. The experimental results are persuasive to reveal the ability of Transformers to perform multiple tasks given a mixed-task context.\n3. The authors provide theoretical evidence for the possibility of performing task superposition in Transformers.\n\n**Weakness:**\nThe main weakness is that the practical value of the paper is very limited. Although the studied problem is novel and the analyses are sound, I'm concerned about how the findings in the paper could be beneficial to any real-world applications. Not only did the authors design any new methods, they didn't provide a concrete application scenario in which this task superposition capability could be utilized. I appreciate the novelty in the studied problem and the comprehensive empirical analysis, but I believe that this paper could be further improved in its practical application value.", "other_comments_or_suggestions": "In Theorem 1, \"A seven layer\" -> \"A seven-layer\".", "questions_for_authors": "Could you show me a specific real-world scenario where keeping the LLM in a multi-task state will be beneficial? I recognize that generation collapse you mentioned is indeed an issue seemingly to be caused by not maintaining a multi-task state. However, I'm curious about whether there are more advanced applications of harnessing task superposition, such as improving the instruction following ability over multi-task instructions, or improving the reasoning ability in some multi-task interaction scenarios.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S4HjLgbSIJ", "reviewer": "Reviewer_TgC8", "summary": "The authors show large language models can naturally perform multiple, distinct tasks simultaneously, even when they were only ever trained on one task at a time. They describe this phenomenon akin to 'superposition' which has been shown in previous work in the settings of having multiple tasks while performing in-context learning.\n\nFindings include:\n\n-Despite being trained with one-hot, single-task examples, the models’ internal representations blend distinct task-specific computations when prompted with a mix of in-context examples.\n\n-Experiments where the authors patch in convex combinations of individual task vectors reveal that the output probabilities vary smoothly as the interpolation parameter changes.\n\n-(Appendix) The paper also shows that larger models can handle more tasks in parallel and align their output distributions more accurately with the mixture of in-context examples.\n\n## update after rebuttal\nI remain a proponent of accepting this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TgC8,\n\nWe sincerely appreciate your thoughtful feedback on our paper. Below, we address the specific concerns raised in the review.\n\n**Theoretical claims**\n\nThank you for bringing up this work. [1] focus on superposition with NN while we focus on superposition in Transformers, by weighting of the answer proportionally to the number of in-context examples corresponding to this tasks. It is correct that the number of tasks could affect the ReLUs used as well, depending on which portion of the Transformer implements a task. Indeed when the ReLU layers are used then the width could be ~$Kd$. \n\nThe purpose of Theorem 1 is to show that task superposition is well within the expressive power of Transformers and Transformers can efficiently represent the superposition mechanism using a small number of layers. It will be an interesting future direction to find the optimal bound on how many task can the model perform with K heads.\n\n**Task vectors in toy setting**\n\nThank you for your suggestion. We extract task vectors from our small trained-from-scratch models using the same pipeline as the pretrained models. While our task vector extraction works well for large, pretrained models, we could not find task vectors that work well for our small models. For example, in [Figure I](https://github.com/Q16A/icl-sup/blob/main/Figure_I.pdf) we plot the accuracy on task `ret2` and `plus2` when using vectors extracted from different layers and observe that the maximum accuracy we can get is lower than 0.2 (while for large real-world pretrained model such accuracy is usually near 1). A possible explanation is that, while task vectors for real world LLMs are extracted from a specific layer, for the small models, the feature that represents a task is likely not localized to a specific layer (as indicated in Figure I), which means that we need to modify how we extract the task vectors for small models. We believe this is an important area for further research.\n\n**Calibration and superposition**\n\nThanks for providing the insights. We agree that superposition is more about what happens inside the model and calibration is more about aligning model's output with the in-context task examples distribution.\n> So the framing of this work is that superposition of representations results in the observation of calibration in the outputs of the model.\n\nThat is correct. We will add more discussion on this in our next revision.\n\n**Superposition \"in the wild\"**\n\n> What are ways of observing this form of superposition 'in-the-wild'? For example does this occur for natural language sequences in data that is not based on data with in-context learning examples?\n\nTwo papers [2, 3] that are released a few days ago also capture the superposition phenomenon in LLMs. In particular, the authors found that when asked to do two-digit addition, LLM will split the problem into two sub-task paths:\n1. one path estimates the rough range of the answer;\n2. another path finds the exact last digit of the sum.\n\nThe model internally employs parallel computational paths and then merges the result together. This indicates that superposition can be commonly found in LLMs and we hope our work can shed light on future research in superposition in LLMs.\n\n**Typos**\n\nThanks for pointing this out. We will fix them in our next revision. We will add the dashed line as well.\n\nWe are happy to elaborate further if you have any remaining concerns.\n\n**References**\n\n[1] Cheung, Brian, et al. \"Superposition of many models into one.\" Advances in neural information processing systems 32 (2019).\n\n[2] Ameisen, et al., \"Circuit Tracing: Revealing Computational Graphs in Language Models\", Transformer Circuits, 2025.\n\n[3] Lindsey, et al., \"On the Biology of a Large Language Model\", Transformer Circuits, 2025.", "claims_and_evidence": "Yes, the methods and evaluation criteria support the claims.", "methods_and_evaluation_criteria": "Yes, the methods make sense based on the claims in the paper.", "theoretical_claims": "Did not check the correctness of Theorem 1, but ability of to only perform K tasks with K heads per attention layer seems low. Wouldn't this also depend on the dimensionality of the activation space as well. The number of tasks that can superposed should be larger based on this work: Superposition of many models into one, https://arxiv.org/abs/1902.05522. If you assume task orthogonality, the activation space can be partition into task specific dimensions and potentially contain far larger if you assume near-orthogonality.", "experimental_designs_or_analyses": "The authors provide prompts that mix examples from different tasks—such as numerical addition in multiple languages or country capital/continent identification—and measure the output probabilities.\n\nBy training a small transformer model on simple, single-task retrieval tasks and then testing it with mixed-task prompts, the experiment demonstrates that even when trained with one-hot targets, the model’s internal representations can blend different task-specific computations.\n\nIn another experiment, they \"patch in\" a convex combination of task vectors, each corresponding to a pure task, into the model. As the interpolation parameter is varied, the output probabilities shift smoothly between those of the individual tasks. This controlled manipulation underscores that the internal representation is a weighted mixture of the separate task representations.", "relation_to_broader_scientific_literature": "Superposition in the various context of neural networks is an important phenomenon and the observation of it occurring in the setting of in-context-learning shows that the property is even more prevalent than previously realized.", "essential_references_not_discussed": "All the references I am aware of are referenced.", "other_strengths_and_weaknesses": "It would be helpful to contrast the task vectors applied to the toy setting to the real world LLMs to perform the same tasks to see if the task vector embeddings in real world LLMs deviate from those in the toy setting. This is fairly minor and just something that would be interesting to compare against.", "other_comments_or_suggestions": "\"Gray dashed line in each figure is the ideal probability if we assume the model perfectly calibrates its output distribution to the distribution of in-context task examples. With uniform distribution of task examples, the dashed lines are at 0.25 (4 tasks setting) and 0.33 (3 tasks setting).\" I could not fined the dashed line in this figure.\n\n\"we select two tasks ad we provide the model with prompts\"", "questions_for_authors": "How is this considered superposition as compared to good calibration? Good calibration being defined as the uncertainty/ambiguity of the completion because the model is given multiple in-context examples. I'm not sure if I fully agree with the argument that superposition is distinct from calibration even if the models are trained in a one-hot setting. Perhaps the distinction should be:\n\nCalibration is about the model’s output uncertainty aligning with the in-context mix, for example, when you supply a 50/50 mix of two tasks, a well‐calibrated model would ideally assign about 50% probability to each task’s answer. Calibration, therefore, is an observable feature of the output distribution.\n\nSuperposition, on the other hand, refers to what happens inside the model. Even when the model isn’t perfectly calibrated at the output, its hidden representations encode a mixture of task-specific computations.\n\nSo the framing of this work is that superposition of representations results in the observation of calibration in the outputs of the model.\n\nWhat are ways of observing this form of superposition 'in-the-wild'? For example does this occur for natural language sequences in data that is not based on data with in-context learning examples? And how does that deviates from the observations of the toy experiment (a small transformer (a GPT‑2 variant with about 14 million parameters) is trained on a family of simple retrieval tasks).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3OzooH2hDx", "reviewer": "Reviewer_fgkh", "summary": "This paper explores the phenomenon of task superposition: when presented a mixture of in-context examples corresponding to different tasks, the output probability distribution of an ambiguous query shows sensitivity to the different tasks and the relative proportion of the examples under different tasks. The paper also investigates this phenomenon in models trained from scratch on synthetic tasks, and relates to work in task vectors by showing the task vectors can be combined to shift the output probability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fgkh,\n\nWe sincerely appreciate your thoughtful feedback on our paper. Below, we address the specific concerns raised in the review.\n\n**More complicated tasks**\n\nThank you for your suggestion. For more complicated task, e.g., grade-school Math, the input $x$ is a Math question and the task answer $g(x)$ usually includes a long chain-of-thought (CoT) reasoning. Note that in this case, there can be multiple equivalently correct ways to solve the given problem using different CoTs, so the task answer $g(x)$ is not unique. Therefore, the task (solving a Math problem) is not a well-defined function from $\\mathcal{X}$ to $\\mathcal{Y}$ in this setting and it will be hard to measure the probability of task answer given prompt ($\\mathbf{P}(g(x)\\mid\\texttt{prompt})$). However, we think that it would be an interesting future direction to study task superposition on more complicated tasks.\n\n**Implications of this work**\n\nIn this work, we show that LLMs are capable of simultaneously solving distinct tasks from in-context examples. Our work contributes significantly to the fundamental understanding of LLMs and highlights a critical area for future research -- developing decoding strategies that\ncan maintain the model’s multi-task state throughout the generation process.\n\nStrikingly, we find that a recent work [1] offers some hope towards this direction. In particular, [1] propose a method to let LLMs generate continuous thoughts as reasoning states. Using this method, given a logical reasoning task that requires exploration, LLMs can explore multiple paths (each path as a sub-task) at the same time by encoding \"a distribution of different traces into the continuous thoughts,\" which is a superposition of intermediate results from different paths. We hope our work could shed light on future studies on superposition in LLMs. We will add more discussion on the implications of our work in our next revision.\n\nWe are happy to elaborate further if you have any remaining concerns.\n\n**References**\n\n[1] Hao, Shibo, et al. \"Training large language models to reason in a continuous latent space.\"", "claims_and_evidence": "Overall, the paper is very clear and the results are intuitive. I did find the conclusion that \"transformers can in-context learn multiple tasks in superposition even if trained to in-context learn one task at a time\" a bit over-claiming, as the task sets being learned or evaluated are highly related, to the extent that they all use the same input. For example, the variants of the addition task can be regarded as just one addition task with different output languages, so an ICL-updated posterior of response types in some sense doesn't seem surprising. Moreover, it's unclear if the models mechanistically treat the different tasks as separate tasks -- for example, it could perform the same internal computation for the addition task, but the presence of numbers in other languages would push the output distributions for those languages higher.\n\nAnother concern/confusion I had is with the \"K heads for K tasks\" capacity claim. I suspect that due to some tasks sharing significant components with other tasks, models with K heads may easily learn more than K tasks, depending on how one defines a \"task unit\". And task boundaries in a continuous space may be intrinsically ambiguous.", "methods_and_evaluation_criteria": "Related to the above, I find the results more interesting on the less \"knowledge/retrieval\"-like task sets, e.g. copying operands vs. adding, and taking the first/last letter + capitalize. I think if the evaluation can include more complicated or procedural tasks that are less knowledge/retrieval-based, it would make the phenomenon clearer and much more interesting.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Overall the design and analyses seem fine.", "relation_to_broader_scientific_literature": "This paper builds on prior studies of ICL and task vectors and provides some evidence on task superposition.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "I find the results clear and intuitive, and I appreciate that the authors studied this issue in different settings (e.g. with training from scratch). \n\nAt the same time, I feel unclear about what to take away from the paper besides the observations. I think there is a space to dive deeper to understand the phenomenon better. It would be nice if the authors could more clearly discuss the implications of this work -- perhaps a rational posterior update argument?", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper identifies a new phenomenon/ability in LLMs called Task Superposition, which is the ability to do multiple in-context learning tasks at a time. The paper empirically argues in a controlled setting that this happens even when trained to only do single-task ICL per training example. The paper provides expressivity results supporting that this skill is possible. 

Reviewer agree that the finding is novel (`FHnf`, `UuyG`); I found the from-scratch experiment insightful and surprising. 
Reviewers agree that the experiments are sound (`FHnf`) and convincing of the claims (`FHnf`, `TgC8`). The paper is clearly written (`fgKH`).

One prominent concern raised was that it's not immediately clear what practical implications of this findings are (`FHnf`, `fgKH`). The authors response provides evidence from the literature, of models that can reason through multiple paths at once. This addresses the question well (and a reviewer raised their score as well). Independent of that response, I think the phenomenon is surprising enough that the lack of an _immediate_ application isn't concerning.
Decision: Accept (spotlight poster)