Comparing the learning dynamics of in-context learning and fine-tuning in language models
Authors: Basile Confavreux
Aaditya K Singh
Jin Hwa Lee
Amaury Sabran
Andrew M Saxe
Venue: ICLR 2026 Poster (2026)
Keywords: in-context learning
supervised fine-tuning
inductive biases
learning dynamics
Primary area: foundation or frontier models, including LLMs

Abstract: Pretrained language models can acquire novel tasks either through in-context learning (ICL)---adapting behavior via activations without weight updates---or through supervised fine-tuning (SFT), where parameters are explicitly updated. Prior work has reported differences in their generalization performance and inductive biases, but the origins of these differences remain poorly understood. In this work, we treat ICL and SFT as distinct learning algorithms and directly compare the learning dynamics they induce across medium-sized models, analyzing both the evolution of their inductive biases and the underlying internal representations. We find that ICL preserves rich input representations but imposes stronger priors inherited from pretraining, whereas SFT suppresses task-irrelevant features---potentially explaining its weaker generalization in few-shot regimes. These results highlight a mechanistic distinction between context-driven and weight-driven learning.

Published as a conference paper at ICLR 2026
COMPARING THE LEARNING DYNAMICS OF IN- CONTEXT LEARNING AND FINE-TUNING IN LANGUAGE MODELS
Basile Confavreux
Aaditya K. Singh
Gatsby Computational Neuroscience Unit University College London
Gatsby Computational Neuroscience Unit University College London
Jin Hwa Lee Sainsbury Wellcome Centre University College London
Amaury Sabran WaveForms Al
Andrew M Saxe Gatsby Computational Neuroscience Unit Sainsbury Wellcome Centre University College London
ABSTRACT
Pretrained language models acquire novel tasks either through in-context learning (ICL) adaating behavior via activations without weight updates- or through supervised fine-tuning (SFT), where parameters are explicitly updated. Prior work has reported differences in their generalization performance and induc- tive biases, but the origins of these differences remain poorly understood. In this work, we treat ICL and SFT as distinct learning algorithms and directly compare the learning dynamics they induce across medium-sized models, analyzing both the evolution of their inductive biases and the underlying internal representations. We find that ICL preserves rich input representations but imposes stronger pri- ors inherited from pretraining, whereas SFT suppresses task-irrelevant features potentially explaining its weaker generalization in few-shot regimes. These re- sults highlight mechanistic distinction between context-driven and weight-driven learning.
INTRODUCTION
Large language models (LLMs) can acquire new tasks either in context (ICL), for instance, by pro- viding example-label pairs inference time with weight updates (Brown et al.) 2020 Liu al.; 2023), or via supervised fine-tuning (SFT), by changing model parameters typically with gradient- based updates on labeled data (Vieira al., 22024). While both learning strategies can achieve good performance (Agarwal et al. (024)), mounting evidence indicates they differ in inductive biases, order sensitivity, and out-of-distributio (OOD) behavior, with ICL sometimes generalizing more robustly than SFT even when trained on the same data (Chan et al. 2022b Lampinen et al., 2025 Akyürek et al. 2022). Understanding how these divergences arise has been difficult in naturalistic settings where task semanties, priors, and data geometry are hard to control.
Here, we treat ICI. and SFT as two distinct learning algorithms, and compare the learning dynamics they elicit in medium-sized pretrained transformers (Vaswani et al. 2017) (2 8B parameters) minimal, 2-D linear classification task with semantically unrelated labels (Wei et al.) 2023 Agar- wal et al. 2024 Min al. 2022). This setting minimizes confounds from linguistic priors or label semantics present in open- domain tasks, and enables precise control over the task geometry (deci- sion boundary angle), shot count and example ordering. which use unveil the generalization strategies at play. We compare ICL and SFT on the same task instance and ordering of examples, and track accuracy, smoothness, confidence, inferred boundary angle, and layer-wise representa-

---

Published as a conference paper at ICLR 2026
tional similarity analysis (RSA). Despite both ICL and SFT reaching similar held-out accuracy, find that ICL exhibits stronger pretraining-inhen inherited priors compared to SFT, biasing the general- ization patterns towards specific computations such as number comparison and pattern matching of in-context labels' ordering. Moreover, ICL preserves a rich representation of inputs, whereas SFT suppresses task-irrelevant features and exhibits representation compression/collapse aligned with task labels. These differences manifest in task-angle- ais--ange-deeeedent generalization, ordering effects, and distinct representational geometries across layers.
Our main contributions are:
Controlled, head-to-head comparison of ICL SFT across matched trajectories, which reveals different inductive biases that manifest in task instance sensitivity, order effects, and confidence profiles.
Representational analysis of models' internal representation showing that SFT representa- tions collapse by label, while ICL largely maintains input structure across layers.
Bridging theory and practice: we connect empirical patterns to views of ICL as implicit aptimization/Bayesiann inference (Von Oswald et al. 2023 Garg et al. 2022 Dai et al.
2022 Zhang al.) 2024) and to recent reports of ICL's superior generalization compared to SFT (Akyürek et al. [2222 Bai et al. 2023
Lampinen et al. 2025 Chan et al. 2022b
Together, these results provide mechanistic view into the differences between context-driven and weight-driven learning.
RELATED WORK
Many-shot ICL: LMs can learn high-dimensional numeric functions and semantic tasks directly from long in-context sequences, with performance continuing to improve well beyond few shots (Agarwal et al. 2024 Anil et al.) (024). Here we unpack the scalar performance metrics in one such task to obtain more fine-grained generalization patterns, unveil inductive biases of ICL and compare them to SFT.
ICL SFT generalization: Many studies have compared the efficiency and generalizability of SFT and ICL. Previous work showed that ICL can out-generalize SFT on a range of tasks, and identified regimes where SFT recovers similar performance through augmentation and regulariza- tion (Lampinen et al. 2025). ICL exhibits superior generalization performance on tasks containing implicit patterns, even when providing more data for SFT (Yin al. (024). while other studies port better generalization for SFT over ICL in other tasks (Mosbach et al. [202),, suggesting more nuanced picture. In this work, compare ICL and SFT across learning trajectory and correlate the observed differences with the internal representation elicited by ICL and SFT.
Representations under SFT and/or ICL: SFT is known to compress representations towards task- relevant directions (Kumar et al. 2022). Previous work compared the representations for ICL vs SFT semantic-heavy task (MMLU), and reported that SFT elicited more aligned representations than ICL (Doimo et al. 2024). However, they did not unpack learning dynamics, i.e., the influence of the progression of in in-centext examples.
Ordering and selection effects in ICI.: Demonstration order strongly affects ICL, with early/last examples disproportionately influential; mitigations include representative/active selection and cali- bration (Zhang al.) 2022 Yang et al.) 2023). We extend these findings with periodic-pattedd probes that induce rule-following over feature-use in some cases.
inference: Theoretical accounts link ICL to Bayesian inference or implicit gradient descent under pretraining distributions and architecture constraints (Bai et al. 2023 Garg et al. 2022 Dai et al. 2022, Akyürek et al. 2022). Follow-ups caution that such mechanisms do not necessarily translate to larger models trained on naturalistic
Mechanisms: ICL implicit optimization/Bayeer
data (Shen et al. 2023 Raventós et al. 2023), aligning with our observations medium-sized LMs.
2

---

Published as a conference paper at ICLR 2026
linear classification
In-context learning
Fine-tuning
99
Example 1-shot prompt:
Base model
Aw
Tuned model
class
feature
Input: 66 38
Output: Rud
OR
38 - Rud 42 78 Baz
Input: 42 78
Base model
class
Output: Baz Input: 0 Output:
Input: 0 Output:
Tuned model
0
feature
99
#shots/class=1
#shots/class=10
#shots/class=200
1.0
ICL
p(ciass1)
#shots/class=0
99
0.5
feature
0.0
1.0
p(class])
%
feature
99
0.5
SFT
0.0
0
feature
990
feature 1
990
feature 1
99
C 1.0
F
smoothness
confidence
1.0
aCCtralo
ICL
SFT
0.5
0.7
0.9
100
100 #shots/class
200
100 #shots/class
100 #shots/class
200
#shots/class
Figure I: Decision boundaries for ICL and SFT on a 2-D linear classification task. A: Descrip- tion of the 2-D linear classification task: the inputs are two integers (1), E N², 722 < 100 and the outputs are two classes with semantically unrelated labels " "__aaz" and "_Rud". The model was trained this task either using ICL or SFT. B: Two example trajectories (one for ICL, one for SFT) on the same instantiation of the task (same training set and ordering of examples at each shot. For 0, 10 and 200 shots per class, the probability associated with the logit of class 1 for all 100 x 100 possible inputs in the task. The probabilities are normalized for decision making such that p(class 1) + p( ((class 2) 1. The black line denotes the ground-truth decision boundary (0 30%). Black squares indicated the examples present in-context (ICL) or in the training set (SFT). C-F: Evolution of the accuracy (C), smoothness (D), confidence (E) and training accuracy (F) for the two learning trajectories shown in B, as function of the number of shots per class. The smoothness is defined as the fraction model outputs that have 2 or more neighbors of the opposite class).
METHODS
3.1
TASK: 2-D LINEAR CLASSIFICATION
We considered 2-D linear classification task with single-token inputs and outputs, adapted from previous work (Agarwal et al. 2024) (Fig. [A], which showed meaningful performance improve- ments the "many-shots' regime (L.e hundreds to thousands). Concretely, each instance of the task defined linear decision boundary over ordered integer pairs (O...., 99).. Models had to map inputs to one of two labels (e.g., " ".az""".Rud" The task was parameterized by single parameter 0 180%), the angle of the ground-truth decision boundary
(2), with 71, n2

---

Published as a conference paper at ICLR 2026
relative to the first feature ny (Fig. TA). Note that all versions of this task had balanced dataset, ie., the same number of examples for each class, at every shot count.
3.2 SEMANTICALLY UNRELATED LABELS
To minimize verbalizer priors, we avoided common placeholders ("Foo"/"Bar") and selected label tokens that were single-token under most open-source tokenizers and less frequent in pretraining corpora ("_Baz"/" Rud"". see Supp. Fig. A13 for other label choices).
3.3 MODELS
Our primary model was Llama3-8B 8B (Dubey et al. 2024 (Fig. periments across different model families and sizes: Qwen3-8B (Yang et al. 2025), Gemma3-128 Agarwal et al. 2025)
We replicated key ex-
and 27B (Kamath et al. 2025), Owen3-0.6B,1.7B..1D as well gpt-oss-20B (Fig. & Supp. Fig. A3
3.4 PROTOCOLS: ICL AND SFT
ICL:
Prompts contained K randomly sampled exemplars per class ("K shots/class") drawn with- out replacement from pre-generated training pool, followed by single query (Fig. AA). We study the same ordered stream of examples across shot counts form learning trajectories. When ana- lyzing ordering effects, we either generate new pre-generated training pools (Fig. A--H), impose periodic patterns (Fig. L))) or shufile in-context ordering as controls (Supp. Fig. A1
We trained on the same cumulative dataset and ordering as ICL at each shot count. Unlike ICL, which does not have explicit hyperparameters, we had to choose several hyperparameters for the fine-tuning. Unless specified, we used the AdamW optimizer with cosine learning rate sched- ule. We report hyperparameters and stability analyses in Appendix (see Supp. Fig. A2 for additional details).
SFT:
On the term "learning dynamics": For ICL, different "shots" correspond to independent prompts with progressively more examples; there are weight updates. We use learning dynamics convenient shorthand for performance and representation changes as the in-context dataset grows. In SFT, shots per class index the same cumulative dataset used for training, though the base model is trained from scratch for every shot count on the relevant training examples. We believe this is useful abuse of notation enables us compare ICL and SFT two learning algorithms.
3.5 METRICS
For each learning trajectory, we tracked the (i) accuracy all 100 x 100 possible inputs to the task, (ii) smoothness, defined minus the fraction of grid points whose predicted class disagrees with at least two of their four neighbors, (iii) confidence, measured as the maximum softmax probability, and (iv) inferred angle, obtained by fitting a linear classifier to the model's predicted labels on the grid.
3.6 REPRESENTATIONAL SIMILARITY ANALYSIS (RSA)
We computed cosine-similarity matrices of last-query-token activations across (i) all prompts along each trajectory and (ii) mixed set of training and randomly sampled test inputs (Kricgeskorte al. 2008). The activations were collected after the MLP at every layer in Llama3-8B (32 layers). We summarized layer-wiss wise patterns and compared ICL vs SFT at matched shot counts.
The companion code is accessible at httpp:::/otttab..ccmmb//sle/////
4 RESULTS
We compared how medium-sized pretrained language models (LMs) learned novel task either in-context (ICL.) via supervised fine-tuning (SFT), matching the two procedures the same
4

---

Published as a conference paper at ICLR 2026
training items, order, and shot counts in 2-D linear classification task. We analyzed generalization performance across shots and task instances, in tandem with layer-wise representational similarity analysis (RSA) to unveil differences in representations and inductive biases.
4.1 SIMILAR GENER CONEERLIIZAIION PERFORMANCE WITH DIFFERENT INDUCTIVE BIASES
Having defined our 2-D linear classification task (Methods & Fig. TA), we first verified that Llama3- 8B could solve this task both with ICL and with SFT (Fig. [ Under matched data and training examples ordering. held-out accuracy was similar across learning trajectories, with similar speeds of learning (Fig. D Both approaches also achieved near-perfect training accuracy throughout the learning trajectory Fig. TT.) However, SFT consistently yielded higher confidence than ICL at com- parable shots (Fig- D). suggesting stronger alignment of logits with the task labels. We also verified that the ICL behaviour was robust across seeds and in in-context shuffling of examples (Fig. [T].
The decision fields revealed qualitative differences in inductive biases (Fig. DB).. Especially for few shots, ICL showed (i) a "previously-seen feature value bias", extrapolating along rows/columns that reuse values shown in-context, and (ii) "comparison bias" that favors decision boundaries near the diagonal (0 x 45%), consistent with "which number is larger?" heuristics (Fig. D3. and 10 shots/class). These biases remained detectable even when global accuracy had converged (Fig. [3. 200 shots/class).
4.2 QUANTIFYING INDUCTIVE BIASES BY VARYING TASK ANGLE
To expose the inductive biases observed in Fig. 1 more quantitatively, we compared model perfor- mance across learning trajectories for several task angles 0. In principle, all these task instances were of similar difficulty. However, hypothesized that the "previously which induced row and column generalization (considering the task representation introduced in Fig. D. would translate into better performance for = 0 and 0 = 90 which are aligned with these generalization patterns, compared to other task angles. Conversely, the "comparison bias" suggested 0 45% another favored angle. Both predictions were verified when comparing model performance across seeds for ICL (Fig 2A..DD.. Moreover, when inferring the optimal linear clas- sifier from the model output (Fig (Cig we observed an overestimation (resp. underestimation) of the inferred task angle for 0 = 30 (resp. t 60%), consistent with a diagonal pull from the comparison bias. This could already be seen from the fine- Line-grained generalization behaviour shown Fig. TB. SFT was not fras-free either under this probing with various task angles (-H). and displayed increased performance for the "easier" angles (similarly to ICI., 0 = 0º 45" 6 909), but not as strong diagonal pull as ICL (Fig. 2G.H).
feature value bias",
4.3 ORDERING EFFECTS AND PATTERN-INDUCED RULE FOLLOWING
We noticed that the ordering of the training examples had an effect in ICL if there was a pattern, i.c. a period, in the ordering of the labels. For instance, always showing a class example before a class 2 example prompted the model output that all following queries were of class irrespective of their features- cicturrs-including those provided in context to be of the other class. In this case, the model ignored the input feature values and instead followed the logic of pattern matching, and not the one of linear classification (Fig. (J). This behaviour was consistent across all 10 aandomly-sampled balanced learning trajectories.
However, longer-period patterns (e.g., "12121221") exerted smaller held-out accuracy compared to the random case, suggesting a short-horizon sensitivity to label in- terleaving (Fig. 2,,,L). It thus appeared that LMs could implement not only fixed ICL rules, but select among algorithms in-context, such as pattern-matching matching, previously-seen feature generaliza- tion. number comparison and linear classification. This finding confirms what had been proposed previously in a more theoretical setting (Bai et al. 2023). and such strong rule based generalization patterns match previous empirical reports in medium-size transformers (Chan et al. 2022b).
detectable influence on
5

---

Published as a conference paper at ICLR 2026
ICL
8=
45
1.0
1.0
C100
1.00
acc stats)
0 0"
135
Picloso111 ace
feature
6 8= 60
== 90
0.5
90
45"
60'
38
0 90"
135'
0.90
200
100
200
90135
#shotsiclass
#shots/cless
(deg)
SFT
8
45
1.0
1.0
G180
H 1.00
acc shots)
piciess) acc
30"
feature
90
0.5
90
0.95
60"
0 90'
135"
0.5
0
100
200
100 #shotsiclass
200
30
90135
feature
1
#shotsiclass
0 (deg)
Periodic orderings
Lparied Lonico
1.0
1.0
1.0
L
1.0
proeessi
Lepertar Lowned
0.5
DCC
12
1221
ACC
11112222
21
2112
22221111
111222
12121221
222111
21212112
0.5
0.5
0.5
100
20C
100
200
100
200
#shotsiclass
#shotsicless
#shots/class
Figure/ 2: Quantifying inductive biases. A: Model decision boundary after 200 shots/class with ICL (Llama3-88) 3B)) on four task angles (0%, 45", 60% and 90%). B: Evolution of the accuracy across shots/class for different task angles (mean and standard deviation, computed across 20 seeds per task angle). C: Evolution of the angle of the optimal linear classifier angle inferred from model outputs (mean and standard deviation, computed across 20 seeds per task angle) D: Accuracy for 200 shots/class (mean and standard deviation, computed across 20 seeds per task angle) for different task angles. E-H: Same as A-D but for SFT (Llama3-8B), I: During ICL learning trajectories, ordering the examples in-context with pattern. The training sets still balanced. "12" corresponds to the strict alternation of class and class 2 examples provided in-context. More complicated sequences with longer periods also considered (e.g. "12121221 of length 8). Visualization of the model output for all 10,000 task inputs, as in A and B, but for periodic orderings of different period lengths ("12", "1221", "121221" and "12121221") J-L: Evolution of the accuracy across shots/class for different periodic orderings (mean and standard deviation computed over 10 trajectories).
4.4 INTERNAL REPRESENTATIONS SFT COLLAPSES REPRESENTATION ALONG LABEL AXES, ICL PRESERVES STRUCTURE
We wondered whether the differences inductive biases observed above for ICL and SFT translated differences in the internal representations of the model. For each of the 10,000 task inputs, we ex- tracted the activations at each layer of the model for several shot values along the same trajectory for SFT ICL. We then computed the cosine similarity between layer-wise activations of all inputs by the model for all layers to obtain representational similarity matrices (Kriegeskorte et al.) 2008). At 200 shots/class, with both ICL and SFT achieving similar training and generalization performance, substantial differences could be seen in the model representations between ICI. and SFT. Although representations in early layers were similar (Fig. A,B), B), by the middle layers, SFT had elicited what appeared to be collapse of the representations alongside the task labels (Fig. B). In other words, the activations clustered in two opposite directions, one for class and the other for class 2 examples. In contrast, ICL maintained more varied input-specific representations throughout all layers (Fig. A))
We also investigated the evolution of the representations in layer for increasing numbers of shots (Fig. major difference beyond the representation collapse already observed above was the representation of task examples from the training set. For ICI., examples present in the training

---

Published as a conference paper at ICLR 2026
ICL
SFT
SFT LoRa
SFT Frozen unembedding
0
10
20
30
0
1-1
Cosine similarity distribution
Test Irain Test Train
Cosine similarity
Class 2 Class
Inputs
Train Test Train Test Class 2
Train Test Train Test
Train Test Train Test
Train Test Train Test
Class
Class 1
Class 2
Class 1
Class 2
Class 1
Class 2
Inputs
Layer 32
Test Train Test Train
#shots/class-10
#shots/class-200
Test Train Test Train
Class 2 Class 1
Cosine similarity
ICL *
Class Class
#shots/class
Class 1
Test
Inputs
Train Test Train Test
Train Test Train Test
Class 2
Class
Class
Class
Class 2
Test
Class 2 Class Test Train Test Train
Class Class Test Train Test Train
Cosine similarity
Test
Test Class 2
Class 1
Inputs
SFT
Train Test Train Test Class Class
Train Test Train Test Class Class 2
Inputs
Figure 3: Representations during ICL and SFT learning trajectories. A: Top: RSA performed on the model activations at different layers. Cosine similarity was computed between last token activations all 200 shots/elass prompts (10, 000 10, 000 matrix). For each transformer layer, the histogram of the RSA matrices is plotted. Bottom: Corresponding RSA matrix for layer 20, used compute one row the plot above. Only 400 inputs are compared, those part of the training set (in the context for ICL, or training examples for SFT), supplemented by randomly selected test inputs, sorted by labels. These plots were computed for Llama3-8B trained with ICL. B: Same as A, but for Llama3-8B trained with SFT (see Section A.3] "vanilla SFT"" C: Same as A, but for Llama3-8B but finetuning using low-rank adaptation (LoRA, (Hu et al. 2022 The analysis was otherwise identical, more in Section A.3 D: Same A, but with the weights of the unembedding matrix frozen. The training method, hyperparameter values and analysis were otherwise identical.
E: RSA on one model layer during ICL and an SFT learning trajectory.
set elicited noticeably more similar representations, regardless of their class, than test examples or the same class (Fig. 3). This was the case for all training examples, irrespective of their ordering in-context, both for 10 shots and 200 shots.
When analyzing the training dynamics of SFT for fixed shot count (Supp. Fig. A7 it appeared that the observed collapse for SFT was tied to performance and not only consequence of over- training, though the collapse appeared to increase with training, even after reaching performance

---

Published as a conference paper at ICLR 2026
plateau. Finetuning with low-rank adaptation (LoRA, (Hu et al. 2022 mitigated the observed col- lapse, though the RSA matrix remained much more similar to SFT than to ICL (Fig BC). Finally, freezing the unembedding matrix during SFT did not eliminate the collapse, suggesting that said collapse was linked to task performance PP).
Overall, it appeared that despite reaching similar training and generalization performance controlled 2-D linear classification task, ICL and SFT did so with markedly different inductive biases and internal representations.
4.5 GENERALIZATION ACROSS MODELS AND TASK VARIANTS
Replaying an identical ICL SFT trajectory across other LMs Qwen3-8B (Yang et al. 2025) and (ommma3-12B/211 (Kamath et al., several newer and larger models under-performing Llama-3-8B in terms of generalization perfor- mance and data efficiency. Nevertheless, the row/column and diagonal generalization patterns ("pre- viously seen feature value bias" and "comparison bias", Fig. T & 2 were qualitatively conserved across models, especially in the few-shots case, indicating that the bias types reported above were not idiosyncratic Llama3-8B (Fig. 4A..,F. We also found representational collapse for SFT but not for ICL with Qwen3-8B (Fig. 4C,D). Once again, LoRA attenuated the collapse (Fig.
model-specific results (Fig. (A,B,..G) with
#shotsiSasi=11
#shatsaclew 200
Qwen3-88 SPT
ICL
SPT (Loña)
Qwent-in (ICL)
220
30
Qwen3-8B (SFT)
1.0
Class 2 Cass
Nest Itain Met taim
Cosine similarity distribution
Goulse Mrillery
(eature
Ireuts
Seature
0.0
Tram Tist Train Test" Class
"rain "rain Test Class 1 Class ipputs
Thai Test Train Teet Class
@shohhyrasss
Cless
Class 2
1.0
G67m (ICL)
1.5
1.0
Gemmez-122
pichess)
0.5
500
0.5
feature
99
0.5
feature
65 0.0
#shotsccaass
@shotsidaas
Figure 4: Extension to other models. A,B Same trajectory with 0 30 (exact same ordering of training examples) for Qwen3-8B, trained either with ICL (A) or SFT (B). From left to right: accuracy computed on all 10,000 possible inputs for the task as function of the number of shots per class; visualization of the decision boundary the model for increasing number of shots: prob-
ability associated with the logit of class for all possible task inputs (same as in Fig. [B). The
probabilities are normalized for decision making such that p(class 1) + p(class 2) squares indicated the examples present in-context (ICL) or in the training set (SFT). C-E: Same visualization as Fig-3A-CC but for Qwen3-8B; Top: RSA performed on the model activations at dif- ferent layers. Cosine similarity was computed between last token activations on all 200 shots/class prompts (10, 000 x 10, 000 matrix). For each transformer layer, the histogram of the RSA matri- ces plotted. Bottom: Corresponding RSA matrix for layer 20, used to compute one row in the plot above. Only 400 inputs are compared, those part of the training set (in the context for ICL, or training examples for SFT), supplemented by randomly selected test inputs, sorted by labels. F,G: Gemma3-12B and Gemma3-275 trained with ICL on the same trajectory as A and B.
Black
In addition, we devised semantic version of the 2-D linear classification task by replacing integers with valence-ordered adjectives (c.g. Abysmal, Appalling, Amazing, Fig. A). Performance still improved with shots. yet learning overall much slower than in the numeric version (Fig.5B,c and Supp. Fig, [4).. Moreover, the comparison bias (0 45") and previously seen feature bias present, albeit weaker (Fig. 5C). This suggested that lexical
Subpar Decent... Great
8

---

Published as a conference paper at ICLR 2026
priors interacted differently with the task geometry when the input manifold was semantic rather than numeric, but trends observed in the toy task overall held.
Finally, trialed non-linear version of the 2-D classification task, by performing XOR op- cration on two linear tasks with 90" angle difference. Once again, we found that the trends from the linear task held, though learning overall slower and task angles made less of difference (Fig. F)..
2D linear classification semantics version
C
#shots/class=1
#shots/class=200 1.0
1.0
49
- 0"
Example prompt:
9 30
piclass1)
ACC
45"
Raaure
input: Amazing Homrble
9 60'
Dutout: Rud
9 90'
Input:: Meciocre
Output: Baz
0.5
feature 1
49 0
feature 1
49
nonlinear classification
#shots/class
#shots/class=1
#shotsiclass-200
99
1.0
feature
-
picioss1)
closs
8cc
θ 45"
0 = 60
a55 2
%
feature 1
99
0.5
feature
99 teature
99
0.0
#shots/class
Figure 5: Extension to other tasks. A: Llama3-8B in semantic version of the 2-D linear classifica- tion task, in which integers were swapped for adjectives ordered by valence. B: Accuracy during ICL computed over all 10,000 possible task inputs as function of the number of shots per class (mean and standard deviation, computed over 10 example orderings), for different instances of the task (task angle 0). C: Visualization of the decision boundary of the model for ICL for increasing number of shots: probability associated with the logit of class for all possible task inputs (same as in Fig. IB). The probabilities are normalized for decision making such that p(class 1) + p(class 2) 1. Black squares indicated the examples present in-context. D: Non-linear 2D classification task, Llama3-8B: Visualization of ground truth labels of the task, red is class and blue is class 2, for 0 30". More details on the non-linear task in Section A.5 E: Evolution of the accuracy across shots/class for different task angles (mean and standard deviation, computed across 5 seeds per task angle) for ICL. F: Same as C for Llama3-88 8B ICL on the non-linear task.
DISCUSSION
In this work, we consider ICL and SET as two learning algorithms and compare them on a controlled 2-D linear classification task with matched data and training example ordering. We observe that al- though the two strategies reach similar training and generalization performance, they do so with different inductive biases (Fig & D. In particular, ICL exhibits stronger priors on solving tasks likely inherited from pretraining, such as pattern-matching and number comparison, which affects its generalization patterns and performance. These biases on our synthetic task can be connected to documented behaviors in open open-domain LMs, such as famillarityhretrieaall for instance, through in- duction heads (Olsson et al., [022] and magnitude-comparisooe heuristics (Srivastava al. 2023 Nikankin et al. 2024, Shah et al. 2023). SFT is not unbiased either, though the motifs were bit harder to pinpoint with our chosen task. That SFT has biases too not surprise, if anything be- cause it acts the same base model and its priors. Though observe that these pretraining priors may be weakened or altered by SFT.
Conversely, the level of the model's internal representations, ICL maintains richer, input-specific structure across layers, while SFT rapidly aligns internal states along label-separating directions (representational collapse), yielding higher confidence but reduced structural diversity (Fig 4). This was conserved across models and finetuning strategies (though specific methods such as LoRA appeared able to limit collapse), indicating that the observed representation collapse is feature

---

Published as a conference paper at ICLR 2026
of SFT and not an artifact of model choice the exact SFT strategy used. These differences of representations, also seen in other tasks (Doimo et al. 2024), may explain the observed angle- dependent accuracy and ordering susceptibility (Fig 2). and echo broader reports of SFT-induced
specialization (and OOD fragility) (Lampinen et al. 2025 Chan et al. 2022b Mosbach et al. 2023
Yin et al. 2024). We also predict that SFT may hinder transfer learning for similar reasons.
Our results are most consistent with ICL performing task-conditioned inference using priors from pretraining (consistent with bayesian/impicci-opiiizzz views), rather than implementing literal gradient descent in medium-sized LMs. These findings challenge previous work in simplified set- tings (Von Oswald et al. [0233] Akyürek et al., 2022), and are agreement with other reports in more realistic settings (Shen et al. 2023; Raventós et al. 2023).
The fact that inductive biases such as the comparison bias (0 45") were conserved across models and tasks during ICL (Fig. 4 & 5 suggests that at least parts of the ICL learning algorithm reflect more general natural language data properties rather than model's specific architecture or idiosyn- crasies of training. Such data properties have been shown to drive the emergence of ICL itself during pretraining (Chan et al. 2022a).
Limitations
We focus on single family of geometry-ccotrolled 2-D classification tasks. While this isolates inductive biases and representations, it may not capture the complexities of hierarchical or multi-stage reasoning. Extending to more real-world in-context learning tasks would test the generality of our conclusions.
Scope of tasks:
Compute and model coverage:
The experiments were centered on medium-sized LMs. Scaling the ICL vs. SFT comparisons to larger models is important next step.
Though we performed several sweeps on SFT hyperparameters to in- vestigate their influence on task performance (Supp. Fig. A2
Hyperparameter breadth:
A7
A8
A9
A10
AIT & A14),
we cannot claim to have exhaustively investigated their influence on the model's 's inductive biases and representations. For instance, we have not exhaustively probed regularizers (e.g. weight decay schedules) or early-stopping/ccliiraii strategies that could mitigate representational collapse.
Prompt design and ordering controls: Our ordering probes synthetic periodic sequences. While they reveal strong short-horizon effects, broader prompt-engineering and selection strategies may alter the task-inference observed here, though systematically mapping this design space is be- yond our scope.
Representation readout: RSA was performed last-token activations. Alternative choices might reveal additional structure. Importantly, we only presented correlational evidence between differ- ences representations and differences inductive biases, making causal manipulations an impor- tant follow-up.
ACKNOWLEDGMENTS
We would like to thank Andrew Lampinen, Daniel Wurgaft, Sara Dragutinovic and Yedi Zhang for useful discussions. This work was supported by Schmidt Science Polymath Award, the Sainsbury Wellcome Centre Core Grant from Wellcome (219627/Z/19/Z) and the Gatsby Charitable Founda- tion (GAT3850).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many in-context learning. Advances in neural information processing systems, 37:76930-76966, 2024.
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applehaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-1200 and gpt-oss-20b model card. arXiv preprint 2508. 10925, 2025.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm is in-context learning? investigations with linear models. arXiv preprint www.:::22111560000 2022.
Cem Anil, Esin Durmus, Nina Panickssery. Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel Ford, et Many-shot jailbreaking. Advances neural information processing systems, 17:129996-129742 2024.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 16:57125-57211, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context Iearn- ing in transformers. Advances in neural information processing systems, 35:18878-18891 2022a.
Stephanie CY Chan, Ishita Dasgupta, Junkyung Kim. Dharshan Kumaran, Andrew Lampinen, and Felix Hill. Transformers generalize differently from information stored in context vs in weights. arXiv preprint arXiv:2210.05673 2022b.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent meta-optimizers. arXiv preprint urXiv:2212.10559, 2022.
Diego Doimo, Alessandro Serra, Alessio Ansuini, and Alberto Cazzaniga. The representation land- scape of few-shot learning and fine-tuning in large language models. Advances neural infor- mation processing systems, 37:18122-1 1118812218115,. 18165, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? case study of simple function classes. Advances in neural information processing systems, 35:30583-30598, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, al. Gemma 3 technical report. arXiv preprint arXhv:2503.19786 2025.
Nikolaus Kriegeskorte, Maricke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202. 10054, 2022.
11

---

Published as a conference paper at ICLR 2026
Andrew K. Lampinen, Arslan Chaudhry, Stephanie C. Y. Chan, Cody Wild, Diane Wan, Alex Ku, Jörg Bornschein, Razvan Pascanu, Murray Shanahan, and James L. McClelland. On the gener- alization of language models from in-context learning and finetuning: controlled study, arXiv, 2025.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and prediet: A systematic survey of prompting methods in natural language pro- cessing. ACM computing surveys, 55(9)) 2023.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zeullmoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 12837, 2022.
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few- shot fine-tuning vs. in-context learning: A fair comparison and evaluation. arXiv preprint 1rXiv:2305.16958 2023.
Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algo- rithms: Language models solve math with bag of heuristics, arXiv preprint arX1v:2410.21222, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 2022.
Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. Advances neural information processing systems, 36: -4228-14246, 2023.
Raj Shah, Vijay Marupudi, Reba Koenen, Khushi Bhardwaj. and Sashank Varma. Numeric magni- tude comparison effects in large language models. Findings of the Association for Computational Linguistics: ACL 2023, pp. 1167-6161, 2023.
Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Do pretrained transformers learn in-context by gradient descent? arXiv preprint arXiv:2310.08540, 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. Beyond the imita- tion game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention you need. Advances in neural informa- tion processing systems, 30, 2017.
Inacio Vieira, Will Allred, Séamus Lankford, Sheila Castilho, and Andy Way. How much data is enough data? fine-tuning large language models for in-house translation: Performance evaluation across multiple dataset sizes. arXiv preprint arXhv:2409.03454 2024.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. International Conference on Machine Learning, pp. 35151-35174, 2023.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.08 038446, 2023.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv, 2025.
Zhao Yang, Yuanzhe Zhang, Dianbo Sui, Cao Liu, Jun Zhao, and Kang Liu. Representative demon- stration selection for in-context learning with two-stage determinantal point process. In Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 5443-5456, 2023.
12

---

Published as a conference paper at ICLR 2026
Qingyu Yin, Xuzheng He, Luoao Deng, Chak Tou Leong, Fan Wang, Yanzhao Yan, Xiaoyu Shen, and Qiang Zhang. Deeper insights without updates: The power of in-context learning over fine- 2024.
tuming. arXiv preprint arXiv:2410.0444
Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49)1155, 2024.
Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. arXiv preprint arXiv:2211 I. 1.0446,, 2022.
13


Reviews:
[{"review_id": "MxXsJTBqdC", "reviewer": "Reviewer_b4Qy", "summary": "This paper investigates the different inductive biases and learning dynamics of in-context learning (ICL) and supervised fine-tuning (SFT) in medium-sized language models. Using a controlled 2D linear classification task, the authors directly compare ICL and SFT across matched learning trajectories (i.e., same data and example ordering), analyzing both generalization patterns and internal representations. The authors find that while both methods achieve similar final accuracy, their strategies and inductive biases differ significantly. ICL is shown to exhibit strong inductive biases inherited from pretraining, such as a \"comparison bias\" (favoring diagonal decision boundaries, $\\theta \\approx 45^\\circ$) and a \"previously-seen feature value bias\" (favoring axis-aligned boundaries). In contrast, SFT is shown to suppress task-irrelevant features, leading to internal states clustering primarily by task label. Conversely, ICL preserves more varied input-specific representations throughout its layers. These core findings are also shown to generalize to an analogous semantic classification task.", "questions": "1. *Influence of SFT Hyperparameters*: The authors note that SFT hyperparameters were not exhaustively probed (Limitations, p. 9). Given that SFT is known to be sensitive to hyperparameter choices, how confident are the authors that the observed \"representation collapse\" is an inherent feature of SFT on this task, rather than an artifact of a specific hyperparameter regime (e.g., potential over-fitting)?\n2. *Clarifying Novel Contribution*: The paper's discussion notes that ICL's sensitivity on pretraining priors and SFT's brittleness have been noted in prior work. Is the authors primary contribution the direct, matched-data demonstration of how ICL and SFT strategies diverge representationally, or the specific characterization of the competing inductive biases (e.g., \"comparison bias\" vs. \"feature value bias\")? A clearer framing of the novelty would help situate the work's impact.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. *Clear Presentation:* The paper does an excellent job of clearly presenting its results. The use of a minimal, controlled 2D linear classification task allows for a precise and direct comparison of the learning dynamics, generalization patterns, and inductive biases of ICL and SFT.\n2. *Representational Analysis:* The representational similarity analysis (RSA) provides a clear distinction between the two learning regimes. It visually and quantitatively demonstrates SFT's representation collapse versus ICL's preservation of input structure (Fig. 3), adding a strong layer of evidence beyond simple task performance metrics.\n3. *Strong Grounding in Literature:* The findings are well-situated within the broader literature, particularly in relation to Bayesian accounts of ICL, and contribute compelling evidence to the ongoing discussion challenging \"ICL as gradient descent\" mechanisms in larger models.", "weaknesses": "1. *Limited Scope of SFT Experiments:* The paper's central claims about SFT rely on fine-tuning experiments conducted primarily on a single model (Llama-3-8B), which the authors acknowledge as a limitation. While the ICL results are replicated across several models, the core ICL vs. SFT comparison would be greatly strengthened by a more comprehensive evaluation of the SFT condition (e.g., across more models, training hyperparameters, or regularizers) to ensure the \"representation collapse\" is a general feature of SFT and not an artifact of a specific setup.\n2. *Novelty in Context of Prior Work:* While the direct comparison between ICL and SFT is valuable, many of the core results (e.g., ICL can be sensitive to pretraining priors, SFT can be brittle, etc) have been documented in related work. The paper could do a clearer job of articulating the specific, novel contribution of its findings beyond these effects. For instance, is the key novelty the direct demonstration of how the internal representations diverge under identical data, or the specific characterization of the \"comparison\" vs. \"axis-aligned\" biases?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oU600VQeAj", "reviewer": "Reviewer_fv6R", "summary": "This work empirically compares the learning dynamics of in-context learning and finetuning on mid-sized language models on two synthetic tasks: a 2-D numerical classification task and a 2-D semantic classification task. This work demonstrates that, while both algorithms can learn the tasks, in-context learning is more subject to biases present in natural language training data (e.g., biases towards comparing numbers, rather than learning an arbitrary classification boundary). Furthermore, the authors demonstrate that the two learning algorithms yield radically different internal representations, with finetuning collapsing representations into two classes and ICL maintaining a greater amount of structure in representations.", "questions": "Why not study smaller models as well?\n\nDid you try using other nonsense labels? Or even labels with pretrained semantics? Perhaps there would be interesting divergences between ICL and finetuning that hinge on the label.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work empirically addresses a broad, pressing question in the field: what is the empirical relationship between in context learning and finetuning? \n\nThe work appears empirically sound, the experiments are thorough, and the breadth of models tested (within the mid-sized scale) is fairly large.\n\nThe results regarding inductive biases of ICL are sensible, yet interesting. These results bolster broader arguments that ICL is selecting from a pool of functions learned in pretraining. The analyses of inductive bias in ICL add nuance to this discussion, indicating that there are systematic differences in the prior probability assigned to functions.", "weaknesses": "Why not study small models as well? Even if studying very large models is computationally challenging, one might be able to discover a scaling trend in ICL vs finetuning learning dynamics by systematically studying models on the order of 1B to 27B. This would increase the impact of this work.\n\nThe RSA of the finetuned models should be further fleshed out. One guess is that the task is so trivial that the model simply converges on the correct answer early on in its computation (i.e., at an early layer). Perhaps early on in finetuning, the model’s representations look more like what you find for ICL, or perhaps there is a one-to-many comparison to be made between the early layers of the finetuned model and all of the layers in the ICL model. This setup seems especially well suited for a finding like “finetuning results in a compressed version of the computation that ICL converges on”. Studying the dynamics of what is called “representation collapse” in the text would be a very valuable contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TmmuOxvD75", "reviewer": "Reviewer_sBMX", "summary": "The paper directly contrasts in‑context learning (ICL) and supervised fine‑tuning (SFT) as learning algorithms on a tightly controlled 2‑D linear classification family. Using matched data, shot counts, and example orderings, the authors track accuracy, \"smoothness,\" confidence, inferred boundary angle, and layer‑wise representational similarity (RSA). Both ICL and SFT reach similar held‑out accuracy, but with different inductive biases and internal representations: ICL preserves richer input structure yet shows stronger from-pretraining priors (e.g., diagonal \"number‑comparison\" bias and row/column reuse), whereas SFT aligns representations along label axes, yielding higher confidence but apparent representational collapse. Ordering effects reveal short‑horizon pattern‑following in ICL. Results qualitatively persist across several model families and in a semantic (adjective‑ordered) variant, though learning there is slower.", "questions": "Does representational collapse happen with LoRA or other PEFT approaches?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The matched‑trajectory setup isolates algorithmic differences between ICL and SFT more cleanly than typical open‑domain benchmarks\n\n* I think the comparison and framing of SFT and ICL as two distinct learning algorithms is a useful and interesting framing", "weaknesses": "* I think the main limitation, as mentioned by the authors, is scope: Add one or two richer tasks (non‑linear boundaries or 3‑class variants; a small real‑text task with controlled geometry) to test whether the same ICL/SFT differences recur.\n\n* RSA uses last‑query‑token activations; alternative readouts (earlier tokens, attention heads, probing classifiers) could nuance the \"collapse vs. preservation\" story. Causality is not established\n\n* Fig. 3 compellingly shows SFT’s label‑aligned compression vs. ICL’s input‑structured geometry, even when accuracies match. I thought this was cool", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tZIiv59k5y", "reviewer": "Reviewer_H2hK", "summary": "The authors apply both few-shot prompting (\"ICL\") and supervised fine-tuning (\"SFT\") to a toy 2D linear classification task, analyzing how performance and representations change as more examples are added.", "questions": "* I don't understand how to read the \"previously seen feature bias\" off the figures. Can you explain?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* Demonstrates that while final performance is similar between ICL and SFT, (a) ICL has a prior towards 45 degrees while SFT has a prior towards 90 degrees, (b) ICL is sensitive to periodic orderings of example labels, and (c) SFT representations are more tightly coupled to the label space.\n* Finds support for (a) across Llama3-8B, Qwen3-8B, Gemma3-12B, and Gemma3-27B", "weaknesses": "* Claims in the text are often not clearly linked to the corresponding portions of the dense and complex figures. For example, the claim \"Both predictions were verified when comparing model performance across seeds for ICL\" references \"Fig.2A\" when I believe the only relevant part of Figure 2A is column 4, the claim \"we observed an overestimation (resp. underestimation) of the inferred task angle for θ = 30◦ (resp. θ = 60◦)\"  references \"Fig.2A\" when I believe the only relevant part of Figure 2A is column 3, etc. Minimally, every subfigure within the current figures needs to be labeled somehow (not just the rows), and the appropriate subfigure needs to be referenced by label wherever it is discussed. More broadly, every claim should be coupled with text that explains how to read that claim off of the corresponding figure.\n* The focus is on a single artificial task, and how these findings might generalize to other problems is not well discussed. For example, I don't know what prediction the identified bias towards a 45 degree angle would make for a real-world task like question answering.\n* Most analyses are done on Llama3-8B, so we can't tell, for example, whether findings (b) and (c) listed in the Strengths generalize to other LLMs. (The appendix shows that GPT-OSS:20B can't even learn the task, so generalization of the findings across models is an important concern.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)