The Missing Alignment Link of In-context Learning on Sequences
Authors: Harshvardhan Agarwal
Sunita Sarawagi
Venue: ICML 2025 poster (2025)
Keywords: Model adaptation
few-shot learning
sequence to sequence learning
understanding in-context learning in LLMs
Primary area: deep_learning->large_language_models
TL;DR: LLMs cannot in-context learn alignments between input and output sequences.  When fine-tuned with in-context examples, LLMs learn alignment while depending on existing induction circuits for learning the token distribution.

Abstract: Large language models (LLMs) have demonstrated the capability to perform in-context learning (ICL) for completely unseen tasks in classification or language completion.  Sequence to sequence (seq2seq) is another popular task category with several applications seeking quick adaptation with ICL. We present a systematic analysis of the ICL capability of LLMs on Seq2Seq tasks using a formal structured language-pair. Our study reveals a critical limitation: except for very short input sequences, ICL fails to achieve consistent learning across all output positions. This exposes a fundamental weakness of modern LLMs — their inability to effectively uncover the alignment between input and output sequences. Consequently, this limitation results in incomplete induction heads, which are essential for in-context learning of new discrete mappings.

To address these limitations, we propose ICA-Tune, a method for focused fine-tuning of an LLM using in-context examples. We present a mechanistic evaluation with two accuracy probes to show how input-output alignment emerges in middle layers of an LLM without direct supervision. This alignment leads to an abrupt jump in the completeness of  the induction heads in higher layers. We show that, compared to standard fine-tuning, ICA-Tune enables more sample efficient learning and better generalization to OOD instances.

The Missing Alignment Link of In-context Learning on Sequences
Harshvardhan Agarwal
Sunita Sarawagi
Abstract
as a prompt x², yy s..ry r x* y* x", they predict a ý indica- tive of having learned the task. The success of ICL has been demonstrated on several tasks (Bertsch et al., 2024; Agarwal et al., 2024), including novel key-value mappings (Kossen et al., 2024), and new synthetic language learning tasks (Ra- jaraman et al., 2024; Bietti et al., 2023; Edelman et al., 2024; Akyürek et al., 2024). For such unseen tasks, a widely ac- cepted explanation is that ICL arises by the formation of induction circuits (Olsson et al., 2022; Reddy, 2024; Singh et al., 2024), An induction circuit requires at least two layers for formation as illustrated in Figure 1. In the first stage, each label position i copies over its previous x' as a key. In the second stage, the test input x" is used as a query to copy over the label y' (i S k) with the matching key x². This ex- planation also works for learning new languages where the key corresponds to n-grams (Rajaraman et al., 2024; Bietti et al., 2023; Edelman et al., 2024; Akyürek et al., 2024).
Large language models (LLMs) have demon- strated the capability to perform in-context learn-
ing (ICI.) for completely unseen tasks in clas- sification or language completion. Sequence sequence (Seq2Seq) is another popular task cate- gory with several applications seeking quick adap- tation with ICL. We present a systematic analy- sis of the ICL capability of LLMs on Seq2Seq tasks using a formal structured language-pair. Our study reveals a critical limitation: except for very short input sequences, ICL fails to achieve consis- tent learning across all output positions. This ex- poses fundamental weakness of modern LLMs their inability to effectively uncover the align- ment between input and output sequences. Con- sequently, this limitation results incomplete in- duction heads, which form the basis for in-context learning of new discrete mappings.
In this paper, we investigate the effectiveness of in-context learning (ICL) on unseen sequence to sequence tasks. In this case each input x is a sequence Z1...., Zm of tokens, likewise each output y is a sequence y/1..... Vn- ICL for Seq2Seq useful in several tasks such as semantic pars- ing for custom APIs (Roy et al., 2023), translation from or to rare languages (Garcia et al., 2023), and Text2SQ generation for new private enterprise schema. To study the mechanism of ICL without fear of data contamination, we follow the practice in prior work of evaluating on new synthetic tasks. Our synthetic generator is inspired by real languages. For each task, we sample a probabilistic context free grammar generating x, and sample P(y|x) (x) using an alignment function inspired from classical models used in lexical translation (Brown et al., 1993), and probabilistic finite state automata. An example of such a setup is shown as the first row of Table 1. Over various characteristics of these structured Seq2Seq tasks, we evaluate pre-trained open-source LLMs for their ICL capabiliitss..
To address these limitations, we propose ICA- Tune a method for focused fine-tuning of an LLM using in-context examples. We present mech- anistic evaluation with two accuracy probes show how alignment emerges in middle layers of an LLM without any direct supervision. This alignment leads to an abrupt jump in the com- pleteness of the induction heads in higher layers. We show that compared to standard fine-tuning, ICA- -une enables more sample efficient learning and generalizes better to OOD instances.
1. Introduction
Large Language Models (LLMs) have demonstrated the ca- pability of In Context Learning (ICL) (Brown et al., 2020) a new task, where given l: input-output pairs and a test input
We observe that except for very short sequences, LLMs fail to in-context learn Seq2Seq tasks even when they suc- ceed in previously studied classification and language learn- ing tasks. We attribute the failure to the inability of the LLM to learn new x-y alignments. ICL for Seq2Seq tasks entails two steps: (1) learning input-output alignments, and (2) learning the next y token given previous y-s and the aligned x. The second step can be learned by induction heads, but
'Computer Science and Engineering, Indian Institute of Tech- nology Bombay, Mumbai, India. Correspondence to: Harsh- vardhan Agarwal <hvag976@stafford.cdm stanford..dd....... Sunita Sarawagi <sunitaa@ inn..ac.n?>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Missing Alignment Link of In-context Learning on Sequences
for the first step we show that modern LLMs do not learn in- context. We present a set of counterfactual experiments to demonstrate that inability to learn the alignment in-context is the reason for poor accuracy on long sequences.
411
Layer 2: Copy value using politext similarity based attention
K <start
K: Hello
K: 11
Goodbye
V: Hello
V: 11
Goodbye
77
Hella
Motivated by these observations, we propose ICA-Tune for focused fine-tuning of a subset of the LLM parameters in ICL mode instead of standard per-example fine-tuming fine-tuning. For monotonic alignments, fine-tuning just the KQ attention parameters of a single layer suffices. Using this hybrid of ICL and fine-tuning, we observe huge jump in accuracy of prediction with just a few examples. We present a mecha- mistic evaluation of the formation of the induction circuit via ICA-TCTne We show that even with standard next-token loss, early layers learn the input-output alignment without any direct supervision, and higher layers learn to inductively lookup labels across sequences. Also, alignment accuracy rises a few steps before lookup accuracy abruptly rises in higher layers. Previous studies have also reported abrupt emergence of induction circuits, but unlike earlier induction circuits ours involves an additional alignment learning layer across non-continuous positions to setup the context keys.
Layer
Context formed with previous taken using position based attention
Hello
Goodhhye
]
7/
Hello
Figure I. Illustration of in-context learning via induction heads for scalar classification task with l: = 2 examples.
crete (Kossen et al., 2024; Shi et al., 2024; Bertsch et al., 2024), language learning where y is a sequence from a language and x is empty (Edelman et al., 2024; Akyürek et al., 2024), and sequence to sequence learning where both x and y are sequences (Roy et al., 2023). Across these dif- ferent studies the broad consensus that ICL does provide accuracy gains beyond zero-shot. However, for structured Seq2Seq tasks like semantic parsing, the success of ICL cru- cially depends on carefully selecting examples to provide adequate coverage of the test example, and reduce interfer- ence (Levy et al., 2023). Our study provides more insights on this phenomenon.
We show that ICA-Tune's hybrid approach to adaptation leads to more efficient learning than conventional fine- tuning and better OOD generalization. We conclude with discussion of the challenges of supporting in-context learn- ing of alignment functions.
Puzzled by the empirical success of ICL, several studies have sought to explain how pre-trained LLMs could develop ICL capabilities. We discuss in detail some of these studies in Appendix A. For totally unseen input-output mappings, a widely accepted explanation is the formation of an induction circuit (Olsson et al., 2022) as shown in Figure 1. Chen et al. (2024) shows why pre-training with next-token loss orients the transform parameters to form such induction circuits due to the presence of repeated structures naturally in the pre-training corpus. Reddy (2024) and Singh et al. (2024) study the dynamism of the formation of induction circuits during training using synthetic scalar classification datasets. For scalar classification, the context in the induction head is always the previous x-tokens input. ICL of new regular languages is studied in (Akyürek et al., 2024; Edelman et al., 2024; Rajaraman al., 2024) with synthetic languages sampled from finite state automaton and markov chains. For this task, the context in induction head is a few tokens immediately preceding each token that define an n- n-gram. And thus, the induction circuits help to implement an n-gram language model, that suits Markovian languages.
Contributions (1) We design a formal language to high- light the inability of modern L.Mss to in-context learn struc- tured Seq2Seq tasks, (2) We explain that the main hindrance to ICL for Seq2Seq tasks is the inability to in-context learn task-specific input and output alignments within each ex- ample. (3) We present ICA-Tune, a hybrid of fine-tuning and ICL, where we fine-tune few parameters with next token prediction loss. (4) We show interesting patterns of abrupt learning of new type of induction circuit where early layers capture immediate context, middle layers learn input- alignments, and higher layers support lookup with induction heads. (5) We show that sample efficiency and OOD generalization is higher for ICA-Tune compared to standard fine-tuning.
2. Related Work
Ever since in-context learning (ICL.) was discovered as an emergent phenomenon of LLMS trained with the next-token prediction loss (Brown et al., 2020). ICL has been exten- sively evaluated and analyzed. Prior work on evaluation of ICL spans various task types: regression where y is real (Garg et al., 2022), scalar classification where y is dis-
We are aware of no prior work that systematically analyzes ICL on Seq2Seq learning tasks with formal structured lan- guages. Such study is important because many real-life tasks, example semantic parsing, low resource translation, error correction, and Tex12SQL are instances of seq2seq learning task. Existing work has been limited to just over- all accuracy on real datasets where it is hard to control for contamination from the training set.
We release code for data generation and ICA-Tune at https: / //gi thub. cwmm 9 97 catuue
2

---

The Missing Alignment Link of In-context Learning on Sequences
3. Studying ICL on Seq2Seq tasks
PCFG: Px())
PFA: (W()A)
PFALP:(W(D)
PFA: (W]C)
R + -XXXX
Problem Formulation We denote an input sequence x 21..... ., Z' as comprising of m discrete tokens, and an output sequence y 91...... Mr consisting of tokens where each Fi, Vj V, a vocabulary of tokens. In general n, n can vary across instances. Let X, denote the input and output spaces of discrete token sequences of arbitrary lengths. Each sequence-to-sequeeee
- AB|BA
C
A = 12, 1, 1,3, 3 sample Y234
sample yiv2
sample
BAC BAC:rijjpr sample ysy,
prediction task T is characterized by a distribution P, (2²) over input sequences and conditional distribution P+() X)) over the output se- quences. To learn this task are given a small number l of input-output sequence pairs { yy'.... (x* )))) sampled from P. (X)))). Our goal is to explore the use of ccontext-learning on a pre- pre-trained LLM Mg to learn this task from the given examples. In ICL, M3 will be provided as input the & examples {(x', y').... (x* )"}} followed by a new test instance x" (X)), and My needs to adapt to the new task using just one forward pass over the input.
Figure 2. Illustration of our data generation process here m 3, c = and attention is non-monotonic
our experiments is represented as a simple two-level prob- abilistic context free grammar PCFG(X) described in the Appendix B. The characterization of the distribution P, (x) is not too critical since x is always given. The learning tasks is to infer P x).
Our model for P(y|x) is inspired from the classical lexical translation model (Brown et al., 1993; Dyer et al., 2013). Given an m length input x, the corresponding y sequence consists of m segments Wi,...; Wm. each wp, can be viewed as y- y-phrase that aligns with a Iy at position q as deter- mined by task-specific alignment function (p) + [m]. We next describe our method of sampling the alignment function and generating the y-phrases conditioned on the aligned x.
Since l is small, and we are exploring tasks that are un- seen by Me. learning P. (V))) is only possible if P+ ()) has decompositional structure to allow generalization to sequences of varying length.
Decompositional structure of ()()) As LLMs gen- erate tokens auto-regressively, we first rewrite (-(y)x)= [[;= Pr (yyl x. , 1). For tractable learning with small k we simplify the dependence structure of yj as fol- lows: (I) Instead of depending on all previous y tokens, each yY depends on small window of size at most g over previous y tokens. (2) Instead of all tokens, a latent align- ment function A, (i) identifies the sub-part of x relevant to predict V/- Thus, the task dependent Seq2Seq model is
Sampling alignment function. We choose the task specific alignment function for the p-th phrase up as:
(p()) = g. where q ~ exp( A/1 pl)
E exp(- A/s - pl) P. g E
[m]]
(2)
By choosing À + do we can make the alignment monotonic (A) = p), and for X + 0 the alignment approaches almost a random permutation. We assume here that all y- phrases are of the same length c, and thus the alignment function for the j-th y-token is A+ (L
x) II P-
In order to learn P- (y)),,,, ICL. needs to discover both the input-output alignment function A,( (j) and the conditional token distribution P, (y LLM's capability for such learning using synthetic tasks that allow a systematic exploration without fear of contamination from the huge training set of LLMs. Towards this end, we define class of formal sequence to sequence task that allows flexible controls of both the alignment function A-())
Sampling y-phrases. Each y-phrase W, is generated as regular language given the aligned x, where g Å (p). The regular language for any x is represented as a proba- bilistic finite state automaton PFA(E, S
X ())) We study an
where Et denotes the vocabulary, S₁ denotes the set of states in the PFA, and T= denotes the stochastic transition matrix. More detaits about PFAS, how we choose a PFA for each x, and how we generate strings from a PFA can be found in Appendix B. Figure and Table shows examples of gener- ated x. y pairs with m 3, 2. Note in the generated y there is no demarcation among the y-phrases. A pseudocode for our data generator appears in Algorithm 1.
and the overall Pr (yy -i--s---- XA)
3.1. Formal sequence to sequence generation model
For each task r, we assume X' represents a structured for- mal language, such as a context free language, that is strictly more powerful than a regular Markovian language. Each element x of x comes from a small discrete vocabulary E,, Let (x) denote the distribution over x, which for
Real-life motivation. As a real-life example, think of x as a sentence in English, and y its translation in Japanese. The English language follows Subject- Subject-Verb-oe Verb- Fer--Object word order
3

---

The Missing Alignment Link of In-context Learning on Sequences
Prompt
Prompt string: x' y x²: y² x²; y x'
Standard
ACB: zijipr CAB: Diriwp ABC: rtprih BCA:
Pre-aligned ACB: AricjjEpr CAB: CjjAriBwp ABC: ArtBprCjh BCA:
Table 1. First row marked Standard shows the prompt for ICL on a synthetic Seq2Seq task. The second row marked Pre-aligned is an oracle setup where the input and output tokens have been interleaved using knowledge gold alignment. Here m 3, c and alignment is monotonic.
Algorithm 1 Sequence Generation Process
sampled from single regular grammar represented by the PFA. In this setting, as various previous studies (Edelman et al.. 2024; Akyürek et al., 2024; Rajaraman et al., 2024) have shown, ICL succeeds in providing very high accuracy since induction heads can approximate regular language well using n-grams. However, as sequence length m increased the accuracy keeps dropping.
I: Input: k, m, C,
2: Σ, <- m random capital English letters.
3: Pr (x) Sample PCFG(m, Σ,) (Sec B.1)
4: A+(-) t- Sample alignment (m, X) using Eq 2.
5: (-la)) + Sample a PFA(E+ja> Sr/2s Tx2x) foreach
x EEZ Σ, (See B.2)
6: for each E (1....,A, 1) do
ICL accuracy increases with increasing y-phrase length for m > 1. When the length of each y-phrase is large, tokens later in each y-phrase can be determined based on previous tokens in the y-phrase, and there n-gram based matching is successful. However, a token the start of each y-phrase critically depend on its corresponding aligned token for correct prediction. In Figure 3(b) show accuracy for the first token within each y-phrase. For the first y token in a y-phrase accuracy is significantly lower than overall accuracy because it is dependent only on the aligned æ token, which is separated from y/ in the prompt by other irrelevant tokens?. We characterize this as a failure to recover the input-output alignment.
7: x' --ample sequence from PCFG (x(
8: y"
9: for each cachpE (L...., m) do
10: Wpp f Sample tokens from PFA P+ A.())
11: y' + Up
12: return x" y'y : :-= 1, k+ +1
as against Subject-Object.. Verb in Japanese. The alignment function captures this systematic difference, whereas the PFA models the Japanese character sequence corresponding to each English word.
Interleaving y with aligned tokens enables ICL. We per- form a counterfactual experiment in which the sequence is interleaved with its aligned tokens from the x sequence, assuming oracle knowledge of A,(). The second row of Table shows an example of the prompt used for such pre- aligned sequences. In Figure 3(c), we report the sequence prediction accuracy using the same axes as in Figure 3(a). Remarkably, accuracy increases sharply across all settings. The interleaving causes each y token to be preceded by all relevant tokens in the LLM prompt, and thereafter in-context learning is able to learn this new task via the formation of induction heads.
3.2. Evaluating In-context Learning of Seq2Seq Tasks
We start with an evaluation of ICL on unseen Seq2Seq tasks using the above synthetic data generator.
Model For our experiments, we utilize the LLaMA 3 model (Meta, 2024) with 8 billion parameters. In Sec- tion 3.3 we present results with other LLMs. For each task characterized as defined above, we generate &o + 1 input- output sequence pairs {(x!, ((x),y"),.. yy),...
) using
Algorithm 1. We create a prompt using the first k(= 15) as in-context example, and test instance x" as x We then generate the y tokens step-by- by-sepp using teacher forcing where at step we predict Ú; with prompt as ICL examples, ++1 We measure accuracy by k+1 Vブ-1
3.3. Do these observations generalize across LLMs?
x", followed by yi
To assess whether these trends persist at larger scales, we repeated the same experiment on several frontier LLMs with m and 1, comparing performance under the standard and pre-aligned prompting strategies. As shown in Table 2, even very large models exhibit low accuracy with the standard prompt. However, when provided with
checking if 03 is accepted by the true PFA.
In Figure ((a) we present accuracy for different lengths of sequences (m)) and y-phrase (c) for the simplest case of monotonic alignments (A oo) where each y-phrase wp aligns with Zp. We make a number of observations:
Since we assumed Mon-Markovian distribution for P, longer context on the previous y tokens cannot substitute for X Ai)
ICL accuracy drops with increasing m. When sequence length m = 1, the Seq2Seq task reduces to a regular lan- guage learning task since all tokens in a y sequence are

---

The Missing Alignment Link of In-context Learning on Sequences
80
89
74
71
42
20
14
92
93
90
91
Length of y-phrase (c)
94
85
76
72
69
70
69
100
68
46
32
18
13
20
16
95
93
90
91
90
91
92
91
6
93
85
78
74
68
66
67
63
100
72
53
33
22
20
18
15
92
93
89
90
91
91
91
90
84
70
68
61
100
81
58
29
19
89
90
92
92
92
88
76
71
67
65
61
60
100
92
57
37
32
28
23
18
92
91
91
94
94
94
95
94
m
95
94
80
70
69
60
100
94
65
36
22
28
94
96
98
98
98
100
97
68
60
53
48
100
98
70
38
22
100
98
96
97
98
99
100
91
67
58
47
49
36
33
100
91
67
58
47
49
36
33
100
95
94
89
88
92
92
98
2
4
5
7
8
2
3
5
6
8
1
3
5
6
8
Length of sequences (m)
Length x sequences (m)
Length of sequences (m)
(a) Standard
(b) Standard (First Token)
(c) Pre-aligned
Figure 3. Studying ICI. Accuracy over sequences with different lengths of y-phrase(c) and x sequences (m) with Accuracy drops with increasing sequence length and increasing y-phrase length for m 1. (b) Largest drop in accuracy with m is for the first token in y-phrase. (c) High accuracy overall after interleaving gold aligned token within y sequence. These show that Low accuracy for large sequences is because of lack of alignment.
pre-aligned input sequences- where and y tokens are interleaved the accuracy improves drastically. This rein- forces the conclusion that poor performance under standard prompting arises not from limited model capacity, but from misalignment in the input structure.
C6
2000
Arcuney
Model
Standard
Pre-Aligned
(a) Standard Prompt
Llama-3.3-708B
20.00
91.25
Llama-3. 3-450BB3
31.25
98.75
GPT-4o Claude-3.7-Sonnee 51.25
36.25
100.0
82.50
2200
Table Prediction accuracy (%) for frontier LLMs under standard vs. pre-aligned prompting. Even very large models underperform with standard prompts, while alignment significantly improves outcomes.
(b) Pre-aligned Prompt
Figure 4, Prediction accuracy for standard and aligned prompts for different lengths y-phrase (c) and m for three different Llama model sizes.
We repeat the ICL evaluation from Section 3.2 using Llama3.2 IB and 1lama3.2--B, and compare them with Llama3.3-8P in Figure 4. Accuracy increases with larger y-phrase length (c) across all models, though larger mod- els achieve higher absolute accuracy. With pre-aligned prompts (Figure 4(b)), even the smallest model performs well, demonstrating that ICL. is effective when relevant xr tokens are placed adjacent to y tokens. In contrast, with standard prompts (Figure 4(a)), all models struggle for small due to the need to infer alignments in-context.
propose additional fine-tuning to learn new Seq2Seq tasks.
4. Fine-tuning to learn Input-Output Alignments
Our goal is to fine-tune the LLM so as to learn the task- specific alignment function A,(), and leverage the LLM's existing capability to lookup labels via induction heads to learn the local token distribution P, (3y Mylly-- Jy-gs-1111 XA,
In the above experiments we assumed oracle knowledge of alignment function Ar()) in creating the aligned sequence with and y tokens interleaved. In Section 5 present discussion of why current LLMMs may not be equipped to in-context lear alignments. We therefore
Once Yj tokens get contextualized with their aligned tokens, the higher layers can form the induction heads with keys as [y, --jーsjj g-j--gj-1.. XA, (3) and values as Øý-
Since we do not have any supervision on the alignment, we mne-tune the model to minimize next token prediction loss over tokens of y sequence. However, we perform this fine-
Both Llama-3.3-70B and Ilama-3.1 3-05BBBBBBBB the "Instruct- Turbo" variants.
5

---

The Missing Alignment Link of In-context Learning on Sequences
tuning in the in-context setting where each training instance is over a set of k 1 examples where the first & form the in-context examples. To fine-tune for given task T, we iteratively sample k + 1 random input-output sequences from Px Y) to maximize this likelihood:
Figure 5(b) plots the alignment accuracy and Figure 5(c) plots the IC-Lookup accuracy across fine-tuning steps for different layers. These bring out many interesting patterns.
ICA-Tume causes alignment to emerge without any direct supervision. (CA-Tume' 's loss function (Eq 3) is standard next-token prediction loss with no explicit role or mention of alignment. However, by fine-tuning with in-context exam- ples, the correct input-output alignment emerges in certain layers. This likely happens because the pre-trained Llama model has propensity to harness induction heads for pre- diction with in-context examples. The alignment learning provides the induction heads with more informative keys.
I+1
max E ((x) y [== 0
log Mo (Z))
[w]]
2 = x
where (Z) denotes the next-token probability distribu- tion output by a transformer in response to an input prompt Z. We fine-tune only the attention parameters 0g of the LLM using LoRA (Hu et al., 2021), and call this method ICA-Tune. In the Appendix E. we show that in many cases fine-tuning only a select one LLM layer suffices. More details of our fine-tuning setup can be found in Appendix D.
Alignment accuracy is high in middle layers, IC-Lookup high in upper layers. As seen in Figure 5(b), alignment ac- curacy is high in middle layers layers 14-16. Layers above 16 and below 14 show poor alignment accuracy. IC-Lookup is more accurate in higher layers (e.g. layers 17-19) whereas layers 14-16 that have high alignment accuracy show poor IC-Lookkpp accuracy. For Seq2Seq tasks induction circuits requires three stages: First, lower layers establish the con- text- for example figuring out the inter-sequence and x, y boundaries, and assigning relative position counts within each sequence. Subsequently, the middle layer learns the alignment function (j). Finally, this leads to the forma- tion of the induction heads in higher layers. We show a diagram illustrating a plausible three layer induction circuit in Figure 10 of the Appendix.
4.1. Results of ICA-Tune
In Figure 5(a) we present the evolution of accuracy against fine-tuning steps for different seeds. We present both overall accuracy, and accuracy segregated by the position of a y- token in its y-phrase. We see that across all seeds, accuracy shoots up most for the first token in y-phrase, most likely since that is the token for which the missing alignment to the æ-token is most crucial. To understand the learning mechanism we monitor two types of intermediate accuracy along the fine-tuning process for each layer L of Me.
Alignment accuracy emerges first abruptly, followed by an increase in IC-Lookup accuracy. The alignment accu- racy rises abruptly around step 85 in Fig 5(b). the IC-Lookup accuracy rises slightly later around step 110 in Fig 5(c). This shows that it is necessary for input-output alignments to form within an example, before a test example can lookup matching y tokens from in-context examples.
1. Alignment Accuracy:
The fraction of time each w/ has greater attention to its aligned r-tokens x ()) than other tokens within each example. This measures how well the model aligns y tokens with their relevant tokens. We measure this alignment accuracy only for first token of each y-phrase.
Σ m 1=1,+=c
1
n
1 (argmax, at (v/+1, ..aht)) (t))
We corroborate this point further by visualizing raw atten- tions at two different layers & = 14, l = 18 for the baseline model and the ICA-Tuned model. Figures 6(a) shows at- tention among the tokens of x* y² at layer & = 14 before and after fine-tuning for m 3, c = 2. Here the x tokens are C and y-tokens are o a s J e In the base- line, there is very little attention from the y-tokens to the tokens. In the ICA-Tuned version, there is much higher attention of y-tokens to their aligned x-tokens (highlighted in red). For example, from a we see alignment to C, and from S j we see aligmment to A. Figures 6(b) examines attention from the tokens of the test sequence *+1 y to the last ICL sequence x y" at layer é 18 before and after fine-tuning. Note here *+1 = C A and x* C A. Thus, the first y-phrase of the test sequence should lookup the second y-phrase of the IC example, and so on. We observe such pattern in the fine-tuned model's 's atten-
f (a, b) is the mean attention weight assigned by token a to the token in attention layer I.
2. IC-Lookup Accuracy: The proportion of attention weights from each y position in the test (last) sequence x *+1 to the correct y-tokens from previous sequences provided for in-context learning. This reflects how accu- rately the model looks up past outputs during generation of y + and is measured as:
E' a 1+1 5/)) 1
(Kt) (1)))
= zi x'A-6)
E M' ΣJ aL (sh
k+1 .))
a_ (a. b) is attention at layer L from the position where a is generated to the position where token is input.
6

---

The Missing Alignment Link of In-context Learning on Sequences
0.8
Layer
Layer
Accurscy
06-
Layor
Lyee
Aecygge
0.4
Footiin
01
Steps
(a) Prediction accuracy
(b) Alignment accuracy
(c) IC-Loookp accuracy
Figure Learning dynamics ICA-Tune with increasing steps. (a) Prediction Accuracy four different seeds: average and at different positions along the y-phrase. Maximum jump for y tokens at the start of y-phrase. (b) The gains duc to the emergence of x, alignment. The plot shows extracted alignment accuracy at different LLM layers. Alignment accuracy high in middle layers (15,16) and abruptly rises between steps 75 and 90. (c) IC-Lookup ability is seen in higher layers (17, 18) above the layers where alignment emerges. This shows that alignment needed for formation of informative induction heads.
Baseline
ICA-Tuned
Baseline
ICA-Tuned
000
1000
Test Sequence Tokens
Test Sequence Tokens
0
0.03
0.005
5-
C B: o a 5 j Test Sequence Tokens
C B: a 5
0.00
A: e Last ICL Example Tokens
B A: Last ICL Example Tokens
0,000
Test Sequence Tokens
(a) Attention Layer
(b) Attention Layer 18
Figure 6. (a) Attention heatmaps for layer 14 of the bascline and fine-tuned model. The heatmaps show how attention is distributed between the y-tokens (target tokens) and the corresponding x-tokens (input tokens). the ICA-Tuned model, initial y-tokens tend focus their corresponding x-tokens tokens from the test example (on the y-axis) and to tokens of nearest IC example (on the x-axis). Attention (shown as red squares) of test sequence A in-context sequence
consistently compared to the baseline. (b) The attention heatmaps for layer 18 between
reflects the order of their x-tokens.
tion weights, but in the baseline model the attention from the y*+1 to y* tokens is largely uniform. This shows that the induction heads formed with the relevant aligned context in the fine-tuned model, and did not form in the baseline.
4.3. Comparison with Standard Fine-tuning
Here we show that fine-tuning in ICL. mode was essential for the alignment emerge by comparing ICA-Tune with standard fine-tuning where minimize loss examples sampled independently. Thus, unlike Equation 3, in standard fine-tuning the objective becomes:
4.2. (CA-Tune for another model
We reinforce the above observations the learning dynam- ies of ICA-Tune by repeating the experiments of Figure 5 using a smaller Owen2.5-3BB model. The results as shown in Figure 7, reveal the same three-stage progression: (1) sharp increase in prediction accuracy at the start of each y-phrase, (2) the emergence of alignment attention patterns appearance in middle layers (e.g.. layers 23-24), and (3) the appearance
Σ log VI............. 1y-1)[Bs] (4)
max max B(xx)) 0,
In Figure 8(a) we compare ICA-Tune with standard fine- tuning for increasing size of D. In both cases we finetune the attention parameters all layers. To ensure comparability between the ICA-Tune and standard fine-tuning setup generate fixed dataset D of N of training examples for task r. Both methods sample from D. We use batch size of for ICA-Tune and 16 for standard fine-tuning. This choice ensures that both the setups train on the same number of examples per training step. We generate a separate set of
of ICI.style lookups in higher layers (e.g., layers 28-29), above the layers where alignment emerges. Despite being significantly smaller than Llama-3, the Qwen model exhibits the same inductive behavior, suggesting that the observed alignment-induee inducedd mechanism is robust across models.

---

The Missing Alignment Link of In-context Learning on Sequences
Layer 18 Layer
20
Layer 23
06
20
Layer 23
Leger
Thining Steps
Training
Trannnng Stops
(a) Prediction accuracy
(b) Alignment accuracy
(c) IC-Lookup accuracy
Figure? 7. Learning dynamics ICA-Tume with increasing steps for Qwen2.5-3B model. (a) Prediction Accuracy for four different seeds: average and at different positions along the y-phrasc. Maximum jump for y tokens at the start of y-phrase. (b) The gains are due the emergence of x, y alignment. The plot shows extracted alignment accuracy at different LLM layers. Alignment accuracy high in middle layers (22,23) and abruptly rises between steps 120 and 170. (c) IC-Lookup ability is seen in higher layers (28, 29) above the layers where alignment emerges. This shows that alignment is needed for formation of informative induction heads.
Layer
Leye
Laye
Layer 16
0.-
24-
00-
fre-ua 050
03-
K.k 000
04
100
200
26-
202
402
000
1200
200
420
630
600
1000
Training Examples
Trirring Stepe
Taining Steps
Increasing training size
(b) Alignment Fine-tuning
(c) Accuracy OOD
Figure& 8. Comparing ICA-Tune with standard fine-tuning, (a) Accuracy with different number of training examples. ICA-Tune requires fewer training examples to achieve good accuracy. (b) Alignment accuracy for fine-tuning. Alignment does not emerge any the layers unlike in ICA-Tune (Figure 5(b)). (c) Validation accuracy on IN and OOD sets of fine-tuning Vs ICA-Tune. ICA-Tune convergence faster and generalizes better to outto--distribution (OOD) validation examples compared to normal fine-tuning.
M = 10 examples for validation. The validation set has individual examples for standard fine tuning and for ICA- Tune we prepend each validation example with examples from the training set. We make the following observation:
hances the model's capacity for out-of-distribution (OOD) generalization. In Figure 8(c), we compare the performance of ICA-Tune and standard fine-tuning on two distinct vali- dation sets. For the first validation set, the input sequences sampled from the same CFG the training set, ensur- ing in-distribution evaluation. For the OOD validation set, x sequences are generated as random permutations of the vocabulary, deliberately designed to deviate from the training CFG. As shown in the figure, ICA-Tune does not distinguish between the two sets and converges faster to near perfect accuracy on both. In contrast, since fine-tuning did not learn alignments, likely overfitted on the whole x sequences and shows poorer OOD generalization.
ICA-Tune is significantly more sample efficient than standard fine-tuning. ICA-Tune achieves almost 100% accuracy with just 50 labeled instances whereas standard fine-tuning does not reach that accuracy even with 800 exam- ples. We explain this difference by observing the alignment accuracy of intermediate layers in the fine-tuned model.
Standard fine-tuning does learn to align. Comparing Fig- ure 8(b) and Figure 5(b) that show alignment accuracy of the regular fine-tuned model and ICA-Tuned Tuned model respectively, we observe that fine-tuning does not uncover alignments unlike ICA-Tune. In ICA-Tune, the alignment causes the formation of the induction heads for in-context learning. Such shortcut is not available to regular fine-tuning.
4.4. Alternative Alignment Functions
So far, our experiments were with monotonic alignments with À oc in Eq 2. Even for monotonic alignments, pre- trained Llama models fail to learn alignments in-context, and we introduced ICA-Tum Tane that causes emergence of input-output alignment given already formed induction head. We next explore if ICA-Tune can also learn non-monotonic alignments arising out of small values of X. For example, with
OOD Generalization of ICA-Tune. ICA-Tune decom- poses the Seq2Seq task into two key steps: (1) learning the alignment between input and output tokens, and (2) leveraging existing induction heads to perform associative lookups using the aligned keys. This decomposition en-
the pth y-phrase may align with any of m x tokens. All the training paramaters and dataset values
8

---

The Missing Alignment Link of In-context Learning on Sequences
nates between estimating the distribution over i tokens that each y token maps to, and estimating the alignment distri- bution. This would require memory of size O(V + m²) where V is the number of distinct y tokens in the in-context examples. Since consensus across all l examples is needed, maintaining this memory distributed across the 2km states of the transformer is not possible for causal model. Also, expressing the EM update in terms of transformer opera- tions seems non-trivial without additional chain-of-thouge scratch memory.
1.0
0.9
0.8
Accuracy
0.7
0.6
A=1
0.5
0.4
16
50
100
150
200
250
300
350
400
Training Steps
Figure 9. Training progress for varying As in the sampled align- ment function (-) (see Equation 2). Larger leads to monotonic alignment that is easier to learn than smaller As
6. Conclusions
In this paper we presented the first ever formal evaluation of the in-context learning abilities of pre-trained LLMs on structured sequence sequence prediction tasks. To al- low for systematic exploration without interference from the LLMs training datasets, we design a realistic synthetic generator of structured language pair. Our study shows that LLMs fail to in-context learn Seq2Seq tasks even when they succeed in simpler comner cases scalar key-value mappings and language completion. Via counterfactual experiments using pre-aligned sequences, we attribute the failure to the inability of the L.LM figure new x-y alignments in-context. We propose ICA-Tune to fine-tune the LLM with in-context examples. ICA-Tune harnesses existing induction circuits to infer the token distributions in-context while fine-tuning attention parameters to learn the alignments. Via mechanistic probes we show the emer- gence of the aligmment capability the middle layers of the LLM that lead to the formation of informative keys to induction heads in higher layers all this with just a next- token prediction loss. In contrast, standard fine-tuning fails to learn alignments, and that leads to inefficient learning and poor OOD generalization indicative of the model's failure to uncover the decompositional structure of the task.
same as previous experiments and m 8, 1. Figure 9 shows accuracy as function of fine-tuning steps for three different values of X. We show runs with three seeds for each A. For large À = 16, for all three seeds ICA-Tune learns fast. For À = 4, one of the three seeds shows slower convergence. For = 1, convergence is the worst. For one of the seeds 200 steps were required as against just 25 for the monotonic case. We observe that for small values of X, more training data is required fine-tune the model.
5. Discussion: Can transformers learn Seq2Seq alignments in-context?
We saw that the pre-trained LLMs fail to in-context learn Seq2Seq alignments and required additional fine- tuning. Here, we ask if in-context learning of general non- monotonic alignments can be expressed at all in causal trans- formers. We informally conjecture that in-context learning of alignments for arbitrary m, K values cannot expressed in causal transformers with a fixed number of parameters. We present discussion of why this conjecture might hold.
Consider the special case where the y-phrase length is making the length of the input and output is the same m n,
This work brings our several avenues of future research: formally analyzing the representation power of transformers for in-context learning alignments, designing pre-training datasets that can cause the emergence of ICL over structured Seq2Seq tasks following a strong prior e.g., monotonicity, and transferring these discoveries to real datasets.
and (j) E {1..., m) that is each token in y aligns to any arbitrary token position x. Pr (My (A-G)) and (z) has to be estimated in-context induction heads from a set of & examples D = [xx] )..,... (xt ,y*)} This defines maximum likelihood estimate of A, on the data D as:
i-1 T -== 8(V5) (-) W/ (e))
Acknowledgment We thank SBI Foundation's Data An- alytics Hub at IIT Bombay and Google Research India for supporting this research.
max
log
Em
6(x) A.G) =X (a)
MLE for Pr (uuiiaa-Gs from previous examples
Impact Statement
Informally, this objective implies that the estimated A-() should maximize the agreement in the mapped x to vocab- ulary across the mk occurrences. The standard solution is using the EM algorithm (we present the classical alignment learning EM algorithm Appendix F). The algorithm alter-
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.

---

The Missing Alignment Link of In-context Learning on Sequences
References
Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL cc cc/paper_ hitpappr_iiiee/ppppe//////////// neurip................... 1457co6bbrce44077118brbbaaa22222 pdf.
Abemnethy, J., Agarwal, A., Marinov, T. V., and Warmuth, M. K. A mechanism for sample-efficient in-context learn- ing for sparse retrieval tasks. In Vernade, C. and Hsu, D. (eds.), Proceedings of The 35th International Conference on Algorithmic Learning Theory, volume 237 of Proceed- ings of Machine Learning Research, pp. 3-46. PMLR, 25- 28 Feb 2024. URL https::/ppocceeeeeeeeeee press/ /eeess/v33// http//aeeeeettaaaaaee html.
mmlr.
Agarwal, R., Singh, A., Zhang. L. M., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J. D., Chu, E., Behbahani, F., Faust, A., and Larochelle, H. Many-shot in-context learning, 2024.
Chen, Y., Zhao, C., Yu, Z., McKeown, K., and He, H. Paral- lel structures in pre-training data yield in-context learning. In ACL, 2024.
Deutch, G., Magar, N., Natan, T., and Dar, G. In-context learning and gradient descent revisited. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers), June 2024.
Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. Trans- formers learn to implement preconditioned gradient de- scent for in-context learning. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023. URL id=tzziniAXETE
n forum?
Dyer, C.,, Chahuneau, V., and Smith, N. A. A simple, fast, and effective reparameterization of IBM model 2. Human Language Technologies: Conference of the North American Chapter of the Association of Computa- tional Linguistics, Proceedings, June 9-14, 2013, Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA. The Asso- ciation for Computational Linguistics (ACL), 2013. URL org. 3
Akyürek, E., Schuurmans, D., Andreas, J., Ma. T., and Zhou, D. What learning algorithm is in-context learn- ing? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL httLttt::///ppeeeeeeet http//pppeeeeeeeeeeeeeeeeeeeeeee hwtpreewwwwweweeeeeeeeeeeeeeeeee net/ forum? 1d=0g0X4HByN41.
https: / ht///aaaaaathoooe
Akyürek, E., Wang. B., Kim, Y., and Andreas, J. In-context language learning: Architectures and algorithms. In Forty- first International Conference on Machine Learning, 2024. URL https::////....... http://pppeeeeeieeee wwwpppewwwwwww.eweeeeeeeeeeeeeee id=329CRr5sz3
Edelman, E., Tsilivis, N., Edelman, B. L.., eran malach, and Goel, S. The evolution of statistical induction heads: In-context learning markov chains. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL ht / openreview. n forum? ?immmidd=aa www.:idddaaaarTT....
Bertsch, A., Ivgi, M., Alon, U., Berant, J., Gormley, M. R., and Neubig, G. In-context learning with long-context models: An in-depth exploration, 2024.
Garcia, X., Bansal, Y., Cherry, C., Foster, G., Krikun, M., Johnson, M., and Firat, O. The unreasonable effectiveness of few-shot learning for machine translation. Proceed- ings of the 40th International Conference on Machine Learning, ICML 23, 2023.
Bietti, A., Cabannes, V., Bouchacourt, D., Jegou, H., and Bottou, L. Birth of a transformer: memory viewpoint. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. URL https: httsttts:///peereeeeeeeee /pppneeiee...... net/forund frtrrumidd=3x2EbBB...
Garg, S., Tsipras, D., Liang, P., and Valiant, G. What can transformers learn in-context? case study of sim- ple function classes. In Oh, A. H., Agarwal, A., Bel- grave, D., and Cho, K. (eds.), Advances in Neural In- formation Processing Systems, 2022. URL https: //openreview. net Forum? dd=-112222eeeer..
Brown, P. F., Della Pietra, S. A., Della Pietra, V. J., and Mercer, R. I.. The mathematics of statistical ma- chine translation: Parameter estimation. Computational Linguistics, 19(2):263-311, 1993. URL https:// acl aaaaanhoooogoo lonthhooog.... org/ wttg//33-2220 J93-2003/.
Gatmiry, K., Saunshi, N., Reddi, S. J., Jegelka, S., and Kumar, S. Can looped transformers learn to imple- ment multi-step gradient descent for in-context learn- ing? In Forty-first International Conference on Machine Learning, 2024. URL https: / openreview nnt// forum? forum?id=00.. aaa.aaa088aaP9P00 9K.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G.,
Henighan, T., Child, R., Ramesh, A., Ziegler, D., Ju, Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M.,
10

---

The Missing Alignment Link of In-context Learning on Sequences
Giannou, A., Yang, L.,, Wang, T., Papailiopoulos, D., and Lee, J. D. How well can transformers emulate in-context newton's 's method? ArXiv, abs/2403.03183, 2024. URL https: http:/ppssmaaaiiccsc...oeee org/CorpusTD: /tt////opuut::222777888333 268247680.
2022. URL https://api semant www.ticcccccrr...... org/ /cppc/rruu
Pan, J., Gao, T., Chen, H., and Chen, D. What in-context learning "learns" in-context: Disentangling task recogni- tion and task learning. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, July 2023.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https::////..... //rxiv.. org/abs/ 061166600 09685.
Panwar, M., Ahuja, K., and Goyal, N. In-context learn- ing through the bayesian prism. In The Twelfth In- lernational Conference on Learning Representations, 2024. URL https: hatp::///ppeeeevieeee net forum?? i JG-HXuujdsSor.
Kossen, J., Gal, Y., and Rainforth, T. In-context learning learns label relationships but is not conventional learning. In The Twelfth International Conference on Learning Representations, 2024. URL https: http////peeeeeeeeee/e/eeee net/forum's i d= IAbgdddy..
Rajaraman, N., Bondaschi, M., Makkuva, A. V., Ramchan- dran, K., and Gastpar, M. Transformers markov data: Constant depth suffices. In ICML 2024 Workshop on Mechanistic Interpretabilltty 2024. URL https: / //op//neeieww.r.cce hwtp:/iwwwwwe.fo..eeeeeeeeeeeeee centt/formmmicc (ttprrummidddnncnsasaa inMM5EHdd2....
Levy, I. Bogin, B., and Berant, J. Diverse demon- strations improve in-context compositional generaliza- tion. In Rogers, A., Bryder J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). ACL. 2023, Toronto, Canada, July 9-14, 2023, pp. 1401-1422. Association for Com- putational Linguistics, 2023. doi: 10.18653/V1/ 2023.ACL-LONG.78 URL htttt:::/ddd...rg//// 18653/v1 acl-oon 78.
Reddy, G. The mechanistic basis of data dependence and abrupt learning in in-context classification task. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: openreviewweeeeee//e/eeee .net// forum? 1ite2dddaNNG6x65
Li, J., Hou, Y., Sachan, M., and Cotterell, R. What do language models learn in context? the structured task hypothesis. 2024. htttt::///aaae
Reynolds, L. and McDonell, K. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA 221, New York, NY, USA, 2021. Association for Com- puting Machinery. ISBN 9781450380959,
org/ /bs/
2406. 0.216.
Mahankali, A. V., Hashimoto, T., and Ma, T. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In The Twelfth International Conference on Learning Representations, 2024. URL
doi: 10. 1 1145/5411763.345176)) URL https: .org/ 10 10.1145/3411768 3451760.
Roy, S., Thomson, S., Chen, T., Shin, R., Pauls, A., Eisner, J., and Durme, B. V. BenchCLAMP: A benchmark for evaluating language models on syntactic and semantic parsing. In Thirty-seventh Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2023, URL / wpw./evien..eeepp//p/pp/. net forum? id=k4 juAEWltG
id=8p3fu561kc.
Meta. The llama 3 herd of models, 2024. URL https: //arxiv. wtggabs/////oooooooooo....o....t / 2407 28783.
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, December 2022.
Shen, L., Mishra, A., and Khashabi, D. Position: Do pretrained transformers learn in-context by gradient de- scent? In Forty-first International Conference on Ma- chine Learning, 2024. URL https: opw.revie..e.eeeeeeeee net / forum? didd2id==--s WsawczEq06.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., Dassarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Haffield-Dodds, Z., Hernandez, D.. Johnston, S., Jones, A., Kernion, J., Lovitt, L.,, Ndousse, K., Amodei, D., Brown, T. B., Clark, J., Kaplan, J.. McCandlish, S., and Olah, C. In-context learning and induction heads. ArXiv, abs/2209.11895
Shi, Z., Wei, J., Xu, 7., and Liang, Y. Why larger language models do in-context learning differently? In Forty- first International Conference on Machine Learning, 2024. URL https: httpt:::/ppeeeeeaaai /wtp////ppeeeeeeeeeeemeeeee net/ forum? i id=WCa 96EG266 6M.

---

The Missing Alignment Link of In-context Learning on Sequences
Singh, A. K., Moskovitz, T., Hill, F., Chan, S. C., and Saxe, A. M. What needs go right for an induction head? mechanistic study of in-context learning cir- cuits and their formation. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview. net ftrrmm/rmmmmmmmm dd-
Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymy- rov, M. Transformers learn in-context by gradient descent. In Krause, A., Brunskill, E., Cho, K., En- gelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 35151-35174, PMLR, 23-29 Jul 2023. URL http::::////cceeeeeeeeeoo ngs mlr press// v202
Yang, L., Lee, K., Nowak, R. D., and Papailiopoulos, D. Looped transformers are better at learning learn- ing algorithms. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview. wtt.:/iiweeeeeeeeeeeeeeeeeeeeeee forum? HHH.ROODDX...
Zhang, R., Frei, S., and Bartlett, P. L. Trained transformers learn linear models in-context. ArXiv, obs/22306.0997,, 2023a. URL https::/api.smmaaaaaaaa stw.aiissmmmaamssooll.... org/ org/Corpus1D::25117777
Zhang. Y., Zhang, F., Yang. Z., and Wang. Z. What and how
does in-context learning learn? bayesian model averaging, parameterization, and generalization, 2023b.
12

---

The Missing Alignment Link of In-context Learning on Sequences
A. More detailed related work
Many studies resort to testing on synthetically generated data in order systematically analyze the reasons behind the empirical performance of ICL. without leakage from the vast pre- pre-training set of large models. Theoretical and mechanistic interpretation on such synthetic settings have been used provide various explanations for how a transformer model implements ICL. Using synthetic data and models trained from scratch for regression tasks, many studies claim that the transformer's self attention across layers implements the gradient descent algorithm (Garg et al., 2022; Ahn et al., 2023; Akyürek et al., 2023; Panwar et al., 2024; Mahankali et al., 2024; Zhang et al., 2023a; Von Oswald et al., 2023; Ahn et al., 2023; Zhang et al., 2023b; Giannou et al., 2024; Yang et al., 2024; Gatmiry et al., 2024). However, these conclusions have been found not to hold for discrete NLP tasks which is our focus (Deutch et al.. 2024; Li et al., 2024; Shen et al., 2024). Another hypothesis is that ICL. performs Bayesian task selection (Reynolds & McDonell, 2021; Min et al., 2022; Pan al., 2023; Shi et al., 2024). However, this hypothesis does not explain the capability of LLMs to learn new input-output mappings new languages. (Abemethy al., 2024) present mechanism by which pre-trained transformer can in-context learn sparse retrieval tasks, including learning the task task-specific delimiter between example and label, and between successive examples. Their approach is to show that their exist transformer parameters that can be oriented to online optimize for the best delimiter under prior biases over small set of delimiters. They require separate heads for each possible delimiter pair. Also, for discovering coefficients of the sparse retrieval task they also show existence of transformer parameters that can online learn the coefficients.
B. More details of the structured sequence to sequence model (Section 3.1)
B.1. Generating Input Sequence x Using Context-Free Grammar
To generate an input sequence x = Z1, 22 ... Fm consisting of m discrete tokens, we employ probabilistic context-free grammar (CFG). This allows us to systematically produce structured sequences through series of production rules. We chose a simple two-level CFG as follows. First we partition the vocabulary of æ tokens Σ, into two disjoint parts of almost equal size U1 and U2, and use these to define the grammar using non-ter non-terminals X, X, as
YX
X > all permutations of U1
Y - all permutations of U2
Root Symbol (R): The start symbol R generates a concatenation of two sub-sequences, and Y, either order (XY or X).
Sub-sequence X: The sub-sequence X consists of all possible permutations of the U1.
Sub-sequence Y: Similarly, the sub-sequence Y includes all permutations of the set U2.
Each transition in the grammar is assigned an equal probability, ensuring that all valid expansions are equally likely to occur.
An example PCFG for m = 3, E, = {4, B, is
R XY
X AB
Y Y
B.2. Model to sample a y-phrase given an input x token
We model the generation of a y-phrase of a given length, say n, using a probabilistic finite state automaton.
Background on Probabilistic Finite State Automaton (PFA) A PFA defined by an alphabet Σ, finite set of states S, an initial state so, and state transition distribution T S S - 00, 1] where Exs: T(x, S' S) 1453, Accordingly,
IIn general, PFA might also attach distribution over initial and accepting states, but did not need that flexibility for this generation task.
13

---

The Missing Alignment Link of In-context Learning on Sequences
Layer 3: Copy over label based on similarity of context
A C
B
E
E
A
A
B
C
B
E
E
A
11
Layer 2: Copy aligned x token key using position similarity
A
C
B
E
E
A
2
2
1 2
1
2
2
Layer 1: Context formed assigning relative position ids to each and taken
A
E
E
p
A
Figure 10. A possible three layer induction circuit to support ICL on our Seq2Seq task with monotonic alignment. Here lc = 2, m 2, c The first layer assigns relative position ids the x and y tokens needs to exploit strong priors about delimiters and example separators (Abernethy et al., 2024). The second layer that aligns based on matching position works only for monotonic alignments. The last layer is the standard induction head.
the probability of generating a sequence x is given PPPA (x) Sg..... Ih= T(s-11, 24, 8t). To generate a sequence of length n from PFA we start with Sp. For each from to n, we sample a 24, S, from T'x, S S(-)). The output sequence x 21,.... Zn is the sequence of output symbols. Without loss of generality we will assume that Su = is always the start state and will denote PFA as PFA(E, S,
Method of constructing Probabilistic Finite State Automaton (PFAs) We present how we sample the vocabulary Σ, set of states S, and the transition probabilities SS
(1) Σ: Sample an alphabet size V uniformly from the interval (Vuin 4, V Vmax 18). Sample language-specific alphabet
Σ, containing V symbols, uniformly (without replacement) from shared symbol W (with [W| Cmax 26).
(2) S: Sample number of states uniformly from the interval ($min 4, 8max = 12). Given this value, define set of automaton states 5 = {S,,.... S.} ) U {So). Without loss of generality we will assume that S. = 0 is always the start state.
(3) T(x, ()): For each S;, choose number of outgoing edges m. uniformly from (Mmin 1, mmar = 4). Then, construct a set edges (S,, 2j. Sj), where all žj are sampled uniformly without replacement from Σ, and all Sj are sampled uniformly without replacement from { S2..... ........ Sn Assign a probability of T(2j. S,1S;) 1. Set T(2', St S.) for all other pairs (z' that do correspond to the generated edges.
m.
C. Transformer Architecture
We assume a pre-trained decoder-only transformer model which have been shown to be capable of ICL. For new tasks, ICL has been attributed to the formation of induction heads. We present a brief overview of how causal self-attention computations across multiple layers could enable this capability. The input to the transformer sequence X XI,... X of N discrete token ids from vocabulary of V tokens, which after mapping via d-dimensional embedding matrix E e Vxx generates Z dxN, The transformer comprises of L layers, and H heads per layer. The computation each & given input Z can be expressed as function
T3 R dxN + R dxxN follows:
Parameters 8 = [W,, Prs (Vi,,, Qr, Kg,)) : [H]
A <- softmax mask( TT Q Ke, E E NXN vh [H]
Z - Z + FFw, (Pr concat ZA, [H])) Rdx
Here mask ensures that every position only attends to positions before it, and called causal mask. After the last layer Tz, the output Z is multiplied by the embedding matrix and then softmax-ed get a probability distribution over tokens.
14

---

The Missing Alignment Link of In-context Learning on Sequences
This makes the overall transformer implement function of the form
(x(X) softmax(E.TT,.o o.... Ti
(5)
where = م PP, (Ven, Ke)) [H] U denotes all the parameters the transformer, p denotes a vector of position embeddings.
D. Experiment setup
For fine-tuning, we employ Low-Rank Adaptation (LoRA), which efficiently adapts pre-trained language models by injecting trainable low-rank updates into specific model parameters.
LoRA Hyperparameters
We use the following LoRA configuration for the experiments:
LoRA Rank RL) Set to 16. This rank determines the dimensionality of the low-rank decomposition applied to the weight matrices.
Scaling Factor (LoRA,): Set to 8. This hyperparameter controls the scaling of the low-rank updates during training to ensure stability and effective learning.
Dropout (LORA DROPOUT): Set to 0.05 to introduce regularization and prevent overfitting in the low-rank layers.
Training Configuration We fine-tune all attention parameters, specifically the Q, K, and matrices of the transformer.
We use a learning rate = 2 -4 for training. We use the Adam optimizer along with a linear decay learning rate scheduler.
E. Additional Experiments
E.1. Fine-tuning only single layer
For this task, and particularly with monotonic alignments, ICA- even single layer of the Llama model suffices to cause the emergence of input-output alignment. However, some layers are significantly more effective than others when fine-tuned. In Figure (a) we present the accuracy with each layer fine-tuned. This graph shows that fine-tuning the attention parameters of one of the middle layers é = 13 is most effective. Fine-tuning attention of higher layers (denoted by lighter shade) has very effect on enhancing ICL accuracy. Early layers (less than 5) helps more than higher layers but still less than the middle layer 13. Subsequently we study the alignment and IC-Loo accuracy, and the attention heatmaps as layer 13's s attention parameters are fine-tuned. The emergence of alignment and LC-Loo accuracy are similar as in the case when all attention parameters were fine-tuned.
Saper
Lyy
0.00
20
Layer 19
Layer 13
0.55
Lasst
Layer
0.70
23
200
-003
600
500
1093
200
422
500
600
1000
222
400
600
000
1000
Taining Stops
Thimiin Steak
Taiinnn Strrps
(a) Prediction accuracy
(b) Alignment accuracy
(c) IC-Lookup accuracy
Figure 11. Learning dynamics of (CA-Tune with increasing steps. (a) Prediction Accuracy when fine-tuning different attention layers. Higher layers are lighter shades than lower layers. Maximum gains is from a middle layer é 13. (b) Emergence of alignment accuracy different LLM layers when layer & = 13 is fine-tuned. Alignment accuracy high in middle layers and emerges around steps 430. (c) Emergence of IC-Lookup accuracy with fine-tuning over L.I layers when layer 13 is fine-tuned. IC-Lookup ability emerges in higher layers around steps 470
E.2. Controlling for Prompt Length
To ensure that the improved performance with pre-aligned prompts is not merely due to having more tokens in the input, we repeat the experiment with a reduced number of examples (14 instead of 16) in the pre-aligned sequence. This adjustment brings the total number of tokens in the pre-aligned prompt roughly line with that of the standard prompt.
15

---

The Missing Alignment Link of In-context Learning on Sequences
Baseline
ICA-Tuned
Baseline
ICA-Tuned
20:0
01
Test Sequence Tokens
C
Test Sequence
0.08
100
0.00
0.00
A B: 0 a C B: a
A: e Last ICL Example Tokens
S
A: Last ICL Example Tokens
Test Sequence Tokens
lest Sequence Tokens
(a) Attention Layer
(b) Attention Layer
Figure 12. (a) Attention heatmaps for layer 13 of the baseline and fine-tuned model. The heatmaps show how attention is distributed between the y-tokens (target tokens) and the corresponding x-tokens (input tokens). In the fine-tuned model, initial y-tokens tend to focus their corresponding x-tokens consistently compared to the bascline. (b) The attention heatmaps for layer 19 between tokens from the test example (on the y-axis) and to tokens of nearest IC example (on the x-axis).
Table 3 reports the accuracy for all three models under the standard, full pre-aligned, and reduced pre-aligned settings. Even after equalizing the prompt length, the reduced pre-aligned setup continues significantly outperform standard prompting. This reinforces the importance of alignment rather than just prompt size in driving ICL performance.
Model
Standard
Pre-Aligned
Reduced Pre-Aligned
Llama-3.2-1B
27.50
100.00
99.38
Llama-3,2-3B
30.63
95.94
95,00
L.ama-3.2-8B
36.25
98.44
99.38
Table Prediction accuracy (%) under different prompt formats for each model. Reducing the number of examples in the pre-aligned prompt still leads to higher accuracy than standard prompting, suggesting that alignment-not token count-is the key factor.
F. EM algorithm for learning alignments
As an exercise, let us go over standard method of discovering alignment following the EM algorithm (Dyer et al., 2013).
Let aji denote the alignment probability that Ar()) i, and Buv P+ = 22 = v) denote the soft x mapping dictionary. The data likelihood
m
ΣΣ log Σ -(A.())
ΣΣ log Σ ajp Bujag
(6)
Maximizing the above objective w.r.t parameters Ojps Suv can be achieved via the EM algorithm, and the iterative update equations of the parameters can be shown to be
Σ* 1=1 Gin
ajp
i= Eg Ajgggge5
Σe Ejnn; =u Epizz Bup
Buo E, E3 Eparyo
(8)
There are several challenges to implementing these iterations in the decoder-only transformer, one option is to assume that the hidden state Z of the last input time-step maintains the 8n, and az values. This would require the hidden vector to be of size O(Vm - m²) where V the number of distinct tokens in the in-context examples. Also, expressing the update equations in terms of transformer operations is not obvious. Based on these, we conjecture that in-context learning of alignments over unseen tasks may not be feasible in transformer with fixed small embedding size.
16


Reviews:
[{"review_id": "oV1NOHojQQ", "reviewer": "Reviewer_roK1", "summary": "This paper investigates a critical challenge in in-context learning for sequence-to-sequence tasks, where they find modern LLMs struggle to learn alignments between input and output sequences in-context. The authors first show that providing explicitly aligned in-context examples dramatically improves performance. They then introduce ICA-Tune, a fine-tuning method using in-context examples, which not only improves accuracy compared to standard supervised fine-tuning but also yields better sample efficiency and out-of-distribution generalization. A detailed mechanistic analysis demonstrates that ICA-Tune enables the emergence of input–output alignments in the middle layers of the LLM, even without direct supervision, thereby facilitating the formation of induction heads for effective token distribution learning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments. We address the concerns below.\n\n(1) Thanks for pointing this out; we will make the clarification in the final version of the paper.\n\n(2) We agree the task is synthetic but it is related to the model followed in early statistical machine translation models. Further, we present an evaluation of ICA-Tune on real translation tasks described below. Here also, we observe that ICA-Tune is more sample efficient and generalizes better to OOD instances.\n\nWe conduct standard fine-tuning with ICA-Tune on three real Machine Translation tasks — from English to {Lithuanian(lt), Hindi(hi), Tamil(ta)}. In each case, we evaluate both on the in-distribution (ID) test set and on an out-of-distribution (OOD) test set. Details of the experiments and the results appear below.\n\n### Experiment Details\n\n| Model Used     | Llama-2 7B                                    |\n|----------------|-----------------------------------------------|\n| Number of Training Examples | 40,000                                 |\n| Train Dataset  | [Europarl (For En-Lt)](https://opus.nlpl.eu/Europarl/corpus/version/Europarl), [Samanantar + BPCC + Speech Lab (For En-Hi)](https://huggingface.co/datasets/hamees/INDiC-BPCC-hq), [Samanantar (For En-Ta)](https://huggingface.co/datasets/ai4bharat/samanantar) |\n| Test Dataset   | [Flores (In-Domain for all three language pairs)](https://huggingface.co/datasets/openlanguagedata/flores_plus), [Tanzil (Out-of-Domain for En-Hi, En-Ta)](https://opus.nlpl.eu/Tanzil/corpus/version/Tanzil), [EMEA (Out-of-Domain for En-Lt)](https://opus.nlpl.eu/EMEA/corpus/version/EMEA) |\n| Epochs          | 2                                             |\n| Batch Size     | 2                                             |\n| Learning Rate  | 0.0002                                        |\n| LR Scheduler   | Linear                                        |\n| Warmup Steps   | 500                                           |\n| Grad Accumulation Steps | 1                                     |\n| Weight Decay   | 0.0                                           |\n| Label Smoothing| 0.001                                         |\n| LoRA Rank      | 256                                           |\n| LoRA Alpha     | 512                                           |\n| LoRA Dropout   | 0.05                                          |\n\nWe report the results after the two modes of fine-tuning:\n\n### COMET-22 Scores (Higher is better)\n\n|               | En-Lt (ID) | En-Lt (OOD) | En-Hi (ID) | En-Hi (OOD) | En-Ta (ID) | En-Ta (OOD) |\n|---------------|------------|-------------|------------|-------------|------------|-------------|\n| Standard Fine-tune | 0.63 | 0.70 | 0.68 | 0.58 | 0.57 | 0.61 |\n| ICA-Tune           | 0.71 | 0.75 | 0.72 | 0.63 | 0.69 | 0.76 |\n\nWe observe that for both in-distribution and out-of-distribution test examples, ICA-Tune performs better than standard fine-tuning.\n\n\n(3) We repeat experiments with a reduced number of examples (14 instead of 16) in the pre-aligned sequence to ensure that the total prompt tokens match the standard version. Below, we report accuracy for the reduced pre-aligned setting. Even after equalizing the number of tokens, the pre-aligned prompting significantly outperforms standard prompting.\n\n| Model           | Standard | Pre-Aligned | Reduced Pre-Aligned |\n|----------------|----------|-------------|----------------------|\n| Llama-3.2-1B  |     27.5     |      100       |       99.38               |\n| Llama-3.2-3B  |     30.63     |      95.94       |     95                 |\n| Llama-3.2-8B  |    36.25      |     98.44        |     99.38                 |\n\n(4) For the comparison between ICA-Tune and standard fine-tuning, we use the exact same set of OOD test examples. The only difference is that, in ICA-Tune, we prepend each test example with k examples from the training set. We have mentioned this in Subsection 4.2, but we will clarify it further.", "claims_and_evidence": "The experimental results generally support the paper’s claims. However, several aspects could be further controlled or clarified:\n\n1. When using pre-aligned prompting, the increase in prompt length (and thus the number of tokens) may provide the model with additional context and compute used in attention, potentially biasing the comparison.\n2. Similarly, the in-context fine-tuning naturally benefits from extended sequences, raising the possibility that performance gains might partly stem from this increased seq len and computation rather than from the alignment learning per se.", "methods_and_evaluation_criteria": "see other points", "theoretical_claims": "there is no theoretical claims in the main paper, the paper is largely empirical and mechanistic", "experimental_designs_or_analyses": "see other points", "relation_to_broader_scientific_literature": "this paper is related to in-context learning mechanisms, particularly building on work about induction heads and this paper is extending this to the seq2seq domain.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper offers a clear and formal formulation of the seq2seq in-context learning problem.\n2. The proposed ICA-Tune method is both simple and effective, demonstrating substantial improvements in performance.\n3. The mechanistic analysis is thorough, with well-designed probes that provide insights into how alignment emerges within the model.\n\nWeaknesses:\n\n1. The notation in Section 3.1 is somewhat confusing. For example, in Equation 2, it is not immediately clear to me that q indexes a token in the input x while p serves as the phrase index. Clarifying that the task maps a token in x to an entire phrase in y would improve readability.\n2. The task formulation is highly synthetic—each input token is aligned with a single output phrase. This simplification may not fully capture the complexity of real-world seq2seq tasks.\n\n3, The improvement observed with pre-aligned in-context sequences might be partly due to the longer prompt (i.e., more tokens providing additional context and compute) rather than solely due to better alignment.\n\n4, The comparison between ICA-Tune and standard fine-tuning might be influenced by the fact that the test examples in the ICA-Tune setting are less out-of-distribution compared to those in the standard fine-tuning setup.\n\nOverall, while the study is valuable for understanding seq2seq ICL, the highly controlled synthetic nature of the experiments may limit the generality of the findings and robustness of their claims.", "other_comments_or_suggestions": "please see other points", "questions_for_authors": "please see other points", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "it3fmwrMSZ", "reviewer": "Reviewer_bbEZ", "summary": "This work presents an interesting case study on LLM's in-context ability on translation-style seq2seq problems. More specifically, this paper studies seq2seq problem involving both learning the alignment and the target-side vocabulary. This work creates synthetic tasks for the analysis to avoid training data leakage. The synthetic tasks are composed of a PCFG generating the source, PFA generating the target vocabulary, and a sampling function generating the alignment. By manipulating hyperparameters of the synthetic tasks and checking the attention weights, this work notices that LLM struggles at learning the alignment. Then, this work also proposes a fine-tuning-based method to improve this specific ICL ability and show that it is more effective than IID supervised learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We respond to the questions below.\n\n(1) We train on all y-tokens, including the outputs from all previous k examples. For the comparison between standard fine-tuning and ICA-Tune, we scale the batch size for standard fine-tuning to match the number of examples used in each batch for ICA-Tune.\n\n(2) Yes, your understanding is correct. When computing accuracy, we evaluate predictions against all valid outputs defined by the PFA.  Yes, the learning could be harder, but we wanted to be close to real tasks where such non-determinism is expected.", "claims_and_evidence": "This work mainly makes claims on two parts: (1) an analysis showing LLMs struggle at learning alignment through ICL; (2) a new fine-tuning method called ICA-Tune that helps LLMs to learn alignment. \n\nI find the synthetic task-based analysis quite convincing. The experiment setup is clear and rigorous, and the attention weight-based analysis shows clear patterns.\n\nI also believe that ICA-Tune improves the model's ability to learn alignment, but the comparison between ICA-Tune and standard SFT is a bit weird. Essentially, they work on two different tasks. The ICA-Tune is learning an in-context learning task, while the standard SFT is learning an IID seq2seq task. I also couldn't find related results on \"latest PEFT methods\" in the paper, as mentioned in line 108.", "methods_and_evaluation_criteria": "See above.", "theoretical_claims": "No major theoretical contributions.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "This work can be seen as a good next step related to previous work studying LLM's ability to learn regular language. The synthetic experiment design is clean and hopefully will inspire more future work on detailed analysis of the limitations of LLM's in-context learning abilities.", "essential_references_not_discussed": "No essential missing references.", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "Line 384: does -> does not?\nFigure 2 and Table 1 use very similar examples, but Figure 2 says \"attention is non-monotonic\" and Table 1 says \"alignment is monotonic\". This is really confusing, even though I can understand how the grammar works from the main text.", "questions_for_authors": "1. For ICA-Tune, do you only train on the k+1-th example for the k-shot learning input, or do you also train on all previous k outputs?\n\n2. My understanding is that due to the use of PFA to generate y-phrases, each output is non-deterministic. Is my understanding correct? If so, when computing prediction accuracies, do you need to consider all correct outputs? And have you done experiments checking if this non-determinism makes learning alignment much harder?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVSCT2s6p2", "reviewer": "Reviewer_D59M", "summary": "This paper systematically investigates the in-context learning (ICL) capabilities of large language models (LLMs) on sequence-to-sequence (Seq2Seq) tasks. The analysis reveals that LLMs struggle to align input and output sequences for longer inputs, limiting their ICL effectiveness. To address this, the authors propose ICA-Tune, a method that fine-tunes attention parameters using in-context examples and a next-token prediction objective. Compared to standard fine-tuning, ICA-Tune demonstrates superior sample efficiency and better generalization to out-of-distribution (OOD) instances.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive suggestions. We address some of the concerns below.\n\n(1) Pre-existing knowledge composition is a plausible hypothesis to explain ICL on real tasks. However, the experiments in [1] were on scalar prediction tasks.  We conjecture that for structured sequence prediction tasks, the alignment of output substructures with relevant input phrases is necessary for compositional generalization, irrespective of whether ICL is due to online learning or knowledge composition, or a combination of the two.\n\n(2) Catastrophic forgetting of previous tasks, for example translation of real tasks after fine-tuning on synthetic tasks, is present in both standard fine-tuning and ICA-Tune. However, on variants of the same synthetic task, ICA-Tune does generalize better as we discuss in the last paragraph of Section 4.2.  We have been thinking hard about methods that specifically learn alignment in a way that generalizes across diverse Seq2Seq tasks.  Section 5 presents a discussion on why such a capability could be difficult in standard causal transformers.  However, it may be possible to design multi-task in-context datasets for structured prediction tasks that enables LLMs to in-context learn alignments for similar tasks. \n\n(3) Thanks for pointing out. We will explicitly cite previous studies on L84 in the final version of the paper.", "claims_and_evidence": "see strengths and weaknesses", "methods_and_evaluation_criteria": "see strengths and weaknesses", "theoretical_claims": "see strengths and weaknesses", "experimental_designs_or_analyses": "see strengths and weaknesses", "relation_to_broader_scientific_literature": "see strengths and weaknesses", "essential_references_not_discussed": "see strengths and weaknesses", "other_strengths_and_weaknesses": "Strengths\n1. This paper presents the first formal evaluation of ICL on Seq2Seq tasks, which has significant practical implications for applications such as instant translation and text-to-SQL.\n2. The counterfactual experiments provide valuable insights, highlighting that alignment is the missing link in Seq2Seq ICL. The mechanistic evaluation of alignment emergence in middle layers is particularly insightful.\n3. The proposed ICA-Tune method demonstrates clear improvements in sample efficiency and OOD generalization, supported by rigorous empirical experiments.\n\nWeaknesses\n1. The analysis is primarily conducted on synthetic data, justified by the need to avoid data contamination (L33). However, it would be beneficial to evaluate ICA-Tune on tasks that LLMs have encountered during pre-training, such as translation. This could help determine whether Seq2Seq ICL is an online learning process (as assumed in this paper) or a task-level generalization mechanism that composes pre-existing knowledge [1]. I'd like to hear the authors' opinion about the other ICL perspective (generalization via pre-existing knowledge composition [1]).\n  - [1]  What Do Language Models Learn in Context? The Structured Task Hypothesis, ACL'24\n2. After fine-tuning with ICA-Tune, does the LLM’s performance degrade on general Seq2Seq tasks (e.g., translation)? If so, this could indicate a limitation of ICA-Tune’s applicability. Is it possible to explore methods that specifically learn alignment in a way that generalizes across diverse Seq2Seq tasks?\n3. In L84, references to previous studies would be more effective if explicitly cited.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9N2hrVlKXc", "reviewer": "Reviewer_kbAw", "summary": "Authors study the limits of LLMs’ abilities for in-context learning, focusing on learning sequence to sequence alignment (in the machine translation sense of the word). Authors design synthetic experiments that probe said ability and demonstrate that several modern llama 3 variants do indeed fail to learn alignment in-context. To combat this, authors introduce ICA-Tune, a PEFT strategy that learns to adjust the attention weights within the LLM. Authors analyze the fine-tuned models and demonstrate that ICA-Tune learns input-output alignment by forming new induction circuits in the middle layers of the model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback. We will address the issues with the supplementary material in the final submission. Below, we respond to the other concerns.\n\nA. We repeated the experiments of Figure 3 on multiple LLMs. We observe similar trends. We report below the numbers for m = 8, c = 1 where we compare the Standard and Pre-Aligned prompting method. We observe that even large models exhibit poor sequence prediction accuracy. Also, interleaving the x and y tokens as a pre-aligned token sequence leads to significant jump in accuracy, indicating that lack of alignment is a major reason for poor performance.\n| Model            | Standard | Pre-Aligned |\n|-----------------|----------|-------------|\n| Llama-3.3-70B-Instruct-Turbo   |     20     |      91.25       |\n| Llama-3.1-405B-Instruct-Turbo   |     31.25     |      98.75       |\n| gpt-4o   |     36.25     |       100.0     |\n| claude-3-7-sonnet-20250219   |     51.25     |       82.5      |\n\nB. We also repeated the experiments of Figure 4 on the learning dynamics of ICATune on a Qwen2.5-3B model.  The results can be found in this [link](https://drive.google.com/file/d/1R90cBSwCaBkumSwibopWckyVIEDZxpaj/view?usp=sharing).  We observe the same conclusions over the three plots --- (1) Sudden jump in prediction accuracy of the y tokens at the start of a y-phrase, (2) Emergence of alignment in middle layers (23,24) of the LLM. (3) Followed by IC-Lookup ability in higher layers (28,29) above the layers where alignment emerges.  This Qwen model is smaller than the Llama model-3 and yet we observe the same pattern.\n\nC. We also present results on real translation dataset for three language pairs. Please see the results in the response to reviewer roK1.", "claims_and_evidence": "The main claims in the paper are, from my perspective:\n1. (L95) \"the inability of modern LLMs to in-context learn structured Seq2Seq tasks\" - stemming from their inability to learn the task-specific alignments (in MT sense) beyond very short sequences;\n2. That the proposed method, ICA-Tune, can alleviate said problem through a mixture of parameter-efficient fine-tuning and in-context learning;\n3. The existence of several phenomena within LLM representations, e.g. the specific induction circuit reported in (L324).\n\nTo support these claims, authors design a synthetic in-context-learning problem generator inspired by classical machine translation and demonstrate (with reasonable ablation) that the LLM in question fails to learn the alignment. They then demonstrate how this can be rectified with ICA-Tune. In my opinion, claims 2 and 3 are properly supported, but the first claim is arguably too general.\n\n\n**Overclaim about generality.** My main concern is that the paper oft makes general claims about “modern LLMs” (e.g. L95-97 “the inability of modern LLMs to …”) that are, in my view, not properly supported and may mislead readers. Authors only consider Llama 3 8B in the main paper and Llama3.2-1B & 3B in appendix E.1: all  relatively small and from the same (albeit popular) model family. This leaves several potential ways that the hypothesis may turn out false:\n\n**A. Emergent abilities?** it could be that modern LLMs do learn alignment, but it only emerges only after a certain model size (e.g. 70B, 405B). LLMs have been known to ‘sprout’ new emerging abilities in the past [1]. From the current analysis, it is unclear if modern LLMs are fundamentally unable to learn alignment (as the authors claim) or if this is only a failing of the particular LLMs chosen for the study.\n\n**B. Model idiosyncrasy?** while less likely than the previous hypothesis, it is possible that there is something about specific Llama 3 training or inference conditions that affect their ability to learn alignment. This can be eliminated by testing on other models: latest qwen 2.5 / mistral / deepseek / qwq model variants.\n\nUnless authors can eliminate these possibilities, I would argue that the claims need to be rewritten substantially (e.g. the inability of modern LLMs -> the apparent inability of such and such LLM types under such conditions) or toned down.\n\nNote that double-checking this claim does not require significant expenditure of resources: to the best of my knowledge, one may test the state-of-the-art LLMs’ ability to learn alignment **using public APIs**, without any specialized hardware. This includes both open models on free tier endpoints (e.g. lmarena.ai, deepseek R1) and commercial models (e.g. openai API, anthropic, google, etc), since the test only requires in-context learning and inference. . If authors find that a diverse set of SoTA models consistently fails to learn alignment, it would strengthen the claims significantly.", "methods_and_evaluation_criteria": "Overall, they do indeed make sense. The paper makes a deliberate choice to evaluate on synthetic data, which determines both its strengths and weaknesses. On the positive side, the controlled experiment allows authors to better isolate the phenomenon, vary experiment parameters and design counterfactual experiments for ablation. On the negative, it leaves a chance that the proposed solution (ICA-Tune) may not transfer as well to real world data.", "theoretical_claims": "To the best of my abilities, the main claims in the paper are empirical, not theoretical.", "experimental_designs_or_analyses": "While the experiments are limited to synthetic problems, they are generally sound. Authors consider reasonable setups and perform additional ablations (e.g. p.4 right) to verify their observations. My main concern about the choice of LLMs and the general claims about LLM abilities is described above in the \"Claims And Evidence\" section.", "relation_to_broader_scientific_literature": "The paper relies on popular prior work in the LLM community: the chosen model family, LoRA adapters, circuits, etc.", "essential_references_not_discussed": "Authors investigate a narrow (but important) capability of modern LLMs and reviews related works in S2, and, to the best of my knowledge, their review is sound (but not encyclopedic). Though, it is not impossible that I missed some other relevant work.", "other_strengths_and_weaknesses": "**Strengths**\nA simple and practical patch for a specific problem, no overthinking. Even if it doesn’t generalize to all LLMs, it’s still useful to many.\nWithin the one synthetic task they consider, author go to commendable length to ablate their findings (S3, S4). Analysis of the ICA-Tune-d models, if mostly mechanistic, is a great addition.\n\n**Weaknesses**\nAs I specified earlier in \"Claims And Evidence\", I believe the main weakness of the paper to be overclaim: authors make conclusions about \"modern LLMs\" in general, but only consider one model family, and only relatively smaller models. \n\nWhile the paper could also be strengthened by real world experiments to confirm the efficacy of ICA-Tune, it appears to be a deliberate choice and not a weakness.", "other_comments_or_suggestions": "**Minor suggestions**\n\n- please close opened files (e.g. dataset.py L27-31)\n- model loading (icatune.py L39, 44) makes it unclear which llama3 are you loading and what format must one prepare this model in\n- please check the code for unused dependencies (e.g. re in icatune.py)", "questions_for_authors": "(minor, extension of OOD generalization experiments) From a practitioner's point of view, are there feasible ways to extend ICA-Tune to 'pre-patch'  known LLMS ahead of time so they would be able to learn alignment in-context for a broader range of unknown ICL tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper identifies a critical gap in the in-context learning capabilities of modern LLMs on seq2seq tasks and proposes a simple yet effective fine-tuning method (ICA-Tune) to address it. While initial experiments are synthetic, the authors convincingly extend their findings to real-world translation tasks and larger LLMs, addressing concerns about generalization. Reviewers appreciated the additional thorough mechanistic analysis and clarity of the contribution. Overall, this is a thoughtful and well-executed paper that makes a meaningful contribution to understanding and improving ICL for structured tasks.
Decision: Accept (poster)