Causality ≠ Invariance: Function and Concept Vectors in LLMs
Authors: Gustaw Opielka
Hannes Rosenbusch
Claire E Stevenson
Venue: ICLR 2026 Poster (2026)
Keywords: mechanistic interpretability
LLMs
attention heads
in-context learning
concept invariance
Primary area: interpretability and explainable AI
TL;DR: Concept Vectors in LLMs contain abstract concept representations, but they differ from Function Vectors that drive ICL performance.

Abstract: Do large language models (LLMs) represent concepts abstractly, i.e., independent of input format? We revisit Function Vectors (FVs), compact representations of in-context learning (ICL) tasks that causally drive task performance. Across multiple LLMs, we show that FVs are not fully invariant: FVs are nearly orthogonal when extracted from different input formats (e.g., open-ended vs. multiple-choice), even if both target the same concept. We identify Concept Vectors (CVs), which carry more stable concept representations. Like FVs, CVs are composed of attention head outputs; however, unlike FVs, the constituent heads are selected using Representational Similarity Analysis (RSA) based on whether they encode concepts consistently across input formats. While these heads emerge in similar layers to FV-related heads, the two sets are largely distinct, suggesting different underlying mechanisms. Steering experiments reveal that FVs excel in-distribution, when extraction and application formats match (e.g., both open-ended in English), while CVs generalize better out-of-distribution across both question types (open-ended vs. multiple-choice) and languages. Our results show that LLMs do contain abstract concept representations, but these differ from those that drive ICL performance.

Published conference paper at ICLR 2026
CAUSALITY + INVARIANCE: FUNCTION AND CONCEPT VECTORS IN LLMs
Gustaw Opietka"
Hannes Rosenbusch
Claire E. Stevenson
University of Amsterdam
ABSTRACT
Do large language models (LLMs) represent concepts abstractly, i.e., independent of imput format? We revisit Function Vectors (FVs), compact representations of in- context learning (ICL) tasks that causally drive task performance. Across multiple LLMs, we show that FVs not fully invariant: FVs of the same concept nearly orthogonal when extracted from different input formats (e.g., open-ended multiple-choice). We identify Concept Vectors (CVs), which carry concept representations. Like FVs, CVs are composed of attention head outputs; however, unlike FVs, the constituent heads selected using Representational Similarity Analysis (RSA) based on whether they encode concepts consistently across input formats. While these heads emerge similar layers FV-related heads, the two sets are largely distinct, suggesting different underlying mechanisms. Steering experiments reveal that FVVs excel in mn-distribution, when extraction and application formats match (e.g., both open-ended in English), while CVs generalize better out-of-distribution across both question types (open-ended vs. multiple- choice) and languages. Our results show that LLMs do contain abstract concept representations, but these differ from those that drive ICL performance.
stable
Function Vectors
Concept Vectors (Ours)
(Todd et al., 2024)
Category Antonym
Causal Heads
Category Antonym
Cosine Sim
Invariant Heads
AF
Antonym
Category
Antonym
Category
Anto
happy sad big
heureux triste grande
happy
big
(a) funny (x) smell
nym
(b) sad
Cate
Paris
city
Paris
ville ?
Paris
(a) state
(a)
animal
gory
cat
chat
city
(b)
color
Figure 1: Function vs. Concept Vectors. Top: Similarity matrices for JVs (left) and CVs (right) in Llama 3.1 70B; cells show how similar two prompt representations are (warmer more similar). Middle: Schematic highlighting the distinction between heads with causal effect (Activation Patching- selected) and heads that encode formattinvariant structure (RSA-selected) prompts for two concepts across three formats (EN open-ended, FR open-ended, multiple-choice). Takeaway: FVs cluster by input format; CV' cluster by concept across formats.
Bottom: Example
"Corresponding author: g opi ka ka@uva.

---

Published conference paper at ICLR 2026
INTRODUCTION
Do large language models represent concepts abstractly, i.e., in a way that stable across surface form? We focus relational concepts: mappings between entities, such linking word to its antonym. Cognitive science has long argued that abstract representation such structure underlies human generalization (Gentner, 1983 Hofstadter 1995 Mitchell] 2020). This capability allows identifying, by analogy, that "hot > cold" and "big > small" share the same oppositional relation, independent of the specific words or how the task presented. Recent work shows that LLMs exhibit representational structures similar to humans (Pinier et al. [2025] Du et al. [025 Doerig et al. 22025) raising the question: do the abstract representations hypothesized to support analogical reasoning actually drive LLM performance on such tasks?
We find that LLMs do contain abstract relational concept information, but the components that encode it differ from those that causally drive in-context learning (ICL. behavior: This separation challenges the single-circuit hypothesis that format-invarinn representations are what primarily enable ICI.
We revisit Function Vectors (FVs) compact vectors formed by summing outputs of a small set of attention heads that mediate ICL (Todd et al. [2024] [Hendel et al.) 2023] Yin & Steinhardi 2025). Because FVss transfer across contexts (e.g., differently formatted prompts and natural text), they are often treated as encoding the underlying concept (Zheng al. 2024 Griffiths et al. 2025 [Bakalova
et al. 2025 BBumlley et al. 2024; Fu 2025). We update this view: FVs are not fully invariant. For the same concept, FVs extracted from different input formats (open-ended vs. multiple-choice) are nearly orthogonal, indicating that JVss mix concept with format (...TT).
To isolate format-invariant structure, we contrast activation patching (AP), which localizes compo- nents with causal effects outputs, with representational similarity analysis (RSA) (Kriegeskorte) 2008), which localizes components whose representations organize by concept independent of for- mat. Using RSA to select heads and then summing their activations yields Concept Vectors (CVs). Across seven relational concepts, three input formats (open-ended English, open-ended French, multiple-choice), and four models (Llama 3.1 8B/70B: Qwen 2.5 7B/72B), we find that CV heads arise in similar layers but are largely disjoint from FV heads, suggesting separable mechanisms for invariance vs. causality ($2.2.2).
Finally, we test whether CV's can steer. In steering experiments, FV's produce larger in in-distribution gains when extraction and application formats match ($3.2.I). whereas CVs generalize more con- sistently out-of-distribution across question type and language (33.2.2) and produce fewer format artifacts (e.g., tokens and language from extraction prompts; $32.3)..
Overall, our contributions are as follows:
FVs are not input-invariant. They mix relational concepts with input format; same-concept FVs differ sharply across formats.
RSA reveals CV heads. These heads encode relational concepts higher level of abstraction than FV heads
CV and FV heads are disjoint. FVs and CVs are realized by different attention heads, suggesting that abstract concept representations are distinct from the mechanisms that causally drive ICL performance.
Steering trade-off. FVs steer more strongly in-distribution, while CVs generalize more consistently -uttof-diittibution,, albeit with smaller absolute gains.
IN SEARCH OF INVARIANCE
We test whether concept representations are stable across surface form, using AP (causal heads) and RSA (format-invariant heads) across models, datasets, and formats. We then form Funetion/Concept Vectors to compare clustering by format concept; AP/RSA heads lie in similar layers but show minimal Kop-K overlap.
We expand what we mean by "higher level of abstraction" in the Discussion
2

---

Published conference paper at ICLR 2026
2.1 METHODS
2.1.1 MODELS
We test Llama 3.1 (8B, 70B) and Qwen 2.5 (7B, 72B) models (Meta AI 2024 Owen al. 2025) All models are autoregressive, residual-based transformers (Vaswani et al. 2023). Each model, f internally comprises of L layers. Each layer is composed of multi-layer perceptron (MLP) and attention heads dæz which together produce the vector representation of the last token of layer E, h, h MLP Ejje atj (Elhage et al. (2021).
2.1.2 TASKS
Datasets We define dataset as one concept expressed in one input format (e.g., Antonym in open- ended English). For each dataset we build set of in-context prompts Pa {p'} where i indexes individual prompts within dataset Each prompt contains few-shot input- outuut examples (æ. y) that illustrate the same concept, followed by query input z; whose target output v/ is withheld. The input-output pairs (I, 3) were either sourced from prior work or generated using OpenAI's GPT-4o (see Appendix D for details). Example prompts are provided Appendix A
Concepts. We consider seven concepts:
Antonym Map word to one with opposite meaning (e.g., hot + cold).
Categorical Map word to its semantic category (e.g., apple + fruit).
Causal Map cause to effect (e.g., rain wet).
Synonym Map word to one with similar meaning (e.g.. big > large).
Translation Translate word to another language (e.g., house maison).
Present-Past Convert verb from present to past tense (e.g.. run + ran).
Singular-Plural Convert
from singular to plural (e.g.,
cats).
Input formats. We vary only the prompt's 's surface format; the (a, yx relation stays the same. Formats:
Open-ended ICL in English (EEENN)
Open-ended ICL in different language (French Spanish; OE- FR OE-ES)
Multiple-choice ICL in English (MC)
We use 5-shot prompts for open-ended and 3-shot for multiple-choice to reduce computational load given prompt length. Altogether, we have 21 datasets (7 concepts input formats). We build 50 prompts per dataset (total N 1050 prompts).
2.1.3 ACTIVATION PATCHING
Activation patching replaces specific activations with eached ones from a clean run assess their im- pact on the model's output. The cached activations are then inserted into selected model components corrupted run, where the systematic relationships the prompt are disrupted. For example, in an antonym ICL task, consider a clean prompt: Hot - Cold, Big Small, Clean and a corrupted prompt: House + Cold, Eagle - Small, Clean + ? The goal of activation patching is then to localize model components that push the model to the correct answer, Dirty, on the corrupted prompt.
We compute the causal indirect effect (CIE) for each attention head azj the difference between the probability predicting the expected token y when processing the corrupted prompt f with and without the transplanted mean activation äry from clean runs:
CIE(ar)) = f arj := āt,) [e] [p)
We then compute the average indirect effect (AIE) over a collection of all datasets ( 2.1..2
1
AIE(at;) = D
Σ
CIE
CIE(ar))
Pa
DED
PEEPa
where Pa denotes the set of corrupted prompts for dataset d.
Note: Unlike Todd et al. 2024 compute AIF scores all input formats, OR-ENG only.

---

Published conference paper at ICLR 2026
H
Head RSM
Design Matrix
Category Antonym
Cosine Sim
TTND
Q: big A 2: grande A
vv
0
2:
(b)
Antonym Category
Antonym Category
Figure 2: Representational Similarity Analysis (RSA). For each attention head, we compute represen- tational similarity matrix (RSM) over prompts spanning concepts and input formats (cosine similarity of head outputs). We construct binary design matrix that marks pairs sharing the same concept, independent of format. The RSA score for a head is Spearman's ' between the lower-tree triangrra entries of the RSM and the design matrix; higher p indicates stronger concept-invariant encoding.
2.1.4 REPRESENTATIONAL SIMILARITY ANALYSIS
To find attention heads encoding concepts invariant to input formats, we employ representational similarity analysis (RSA; Kriegeskorte (2008).
For each attention head (ıj we compute representational similarity matrices (RSMs) where in denotes the output extracted from 0zj for the ith prompt p: PN, and ((:; .) is cosine similarity function.
0(1i, uN)
RSM
A(UN, U1)
We then construct a binary design matrix, DM, where each entry is set if the corresponding pair of prompts share the same attribute value, and 0 otherwise. In this paper, we consider two attributes: (I) concept does a pair of prompts illustrate the same concept, regardless of the input format? and (2) prompt_foor rmat rorat multiple-chiice)?
does pair of prompts have the same question type (i.e. open-ended or
We then quantify the alignment between the RSM and DM for the lower-triangles (since similarity matrices are symmetric) using the non-parametric Spearman's rank correlation coefficient (p).
To localize attention heads carrying invariant concept information we compute the RSA for each attention head obtaining a single Concept RSA score for each attention head.
Concept-RSA(agy = (RSMe Concept-DM)
(4)
Function Vectors
Concept Vectors
Antonym
Antonym
Category
Category
Cosine Sim
Causal
Causal
Synonym
Synonym
Translation
Translation
Past Tense
Past Tense
Plural Form
Plural Form
Figure 3: Similarity matrices. Full similarity matrices extracted from top K heads in CV's and FV's in Llama 3.1 70B for all concepts. See Appendix B for other models.
4

---

Published conference paper at ICLR 2026
2.1.5 FUNCTION CONCEPT VECTORS
To form Function/Concept Vectors we create sets of top K ranking attention heads, Ary and Acv. based their AIE and RSA scores respectively. Function/Concept Vectors for prompt are then computed as the sum of activations for this prompt, azz. from the sets Ar and Acv respectively.
FV;
Σ a EAFE
CV:
Σai alj
(5)
ajjEAcv
Question Type
Concept
0.8
0.5
Liama 70B
0.5
Liama 8B
RSA Score
0.2
0.1
CV
7V
0.5
0.8
0.5
Qwen 72B
Qwen 7B
0.2
0.1
5
10
50
1
5
10
50
K Heads
K Heads
Figure 4: Concept vs. format RSA. Question type and Concept RSA scores for CVs and FVss in all models. Takeaway: CVs encode more concept information and less input format than FVs.
2.2 RESULTS
2.2. CONCEPT VECTORS ARE MORE INVARIANT TO INPUT FORMAT
We test invariance to input format by computing RSA with design matrices for concept and question type (following the setup in 12.1.4). We form FVs/CVs by summing the top- heads ranked by AIE/RSA (Eq. 5). Across models and K, CVs show higher concept RSA and lower question-type tttpp RSA than FVS (Figure indicating that FVs encode format more strongly while CVss track concept. Consistently, similarity matrices for Llama 3.1 70B cluster by concept across formats for CVs, but by format for FVs (Figure 3), where within-forman type FV clusters are nearly identical with mean cosine similarity 0.90. CV's nonetheless exhibit weaker within-format type cluster (mean cosine similarity = 0.55). suggesting they retain some low-level format information. Overall, however, CVs remain markedly more invariant to input format than FVs.
Model
K=3
K=5
K=10
K=20
K=50
K=100
Llama-3.1 8B
12
28
1lama-3.1
1
Qwen2.5 7B
0
4
15
39
Qwen2.5 72B
0
0
1
3
13
Table 1: RSA-AIE head overlap. Overlap between RSA and AIE heads (number overlapping heads among top-K). Bold numbers indicate overlap significantly above chance (p 0.05; details in Appendix E.. Takeaway: FVss and CVs are composed of different attention heads.
2.2.2 FUNCTION CONCEPT VECTORS ARE COMPOSED DIFFERENT ATTENTION HEADS
we compare which heads are selected by the two procedures, we see that FVs and CVs are composed of different attention heads. First, ranked each head for each method, i.e., AIE (2..3
5

---

Published conference paper at ICLR 2026
for FVss and by Concept-RSA (22.1.4) for CVs. Then we examined depth and top- overlap, Layer- averaged scores show similar layer profiles (Figure5)) but head identities barely overlap: for K < 20 the intersection is near zero and stays small larger (Table D. We also note that ATE scores are highly sparse: their histogram peaks zero with a long right tail (Figure -2)-so only few heads have measurable causal effect. Together this supports that AlE-selected causal heads are largely distinct from the invariant, RSA-selected heads.
To ensure this separation is not an artifact patching within the same format, we also performed cross- format activation patching (e.g., extracting activations from open-ended prompts and patching them into multiple-choine pa
This procedure continued to identify the same FV heads and did not identify CV heads (see Appendix 0), confirming that FVs are the primary causal drivers regardless of input format.
Liama 8B
Llama 70B
Qwen 7B
Qwen 72B
0.06
0.18
0.02
0.24 0.06
0.18 0.02
0.18
0.03
0.12
0.01
0.16 0.03
0.12
0.01
0.12
0.00
0.06
0.00
0.08
0.00
0.06 0.00
0.06
50
Layer
Layer
Layer
Layer
Figure Layer-wise AIE vs. RSA. AIE and RSA scores averaged across all heads per layer. Takeaway: FV and CV heads are in similar layers.
CAN CONCEPT VECTORS STEER?
We now test whether these invariant heads steer: we introduce how construct vectors, the AmbiguousICL. setup with conflicting cues, and the intervention protocol. FVVs win in-distribution; CVs transfer better out-of-distribution with fewer format artifacts, at cost of smaller gains.
extraction prompts
ID
OOD
happy sad
heureux grande
triste
happy
(a) funny (a) small
big
big
?
(b) sad
(b) smart
1.
AmbiguousICL
rich poor
+ PV
end (+50%)
fin
(a)
steering prompts
pretty + ugly
3.
funny boring
fast rapide
+ CV
end (+25%)
end
end
strong fort
start
originel prediction:
2.
сommeneer
Figure 6: Overview of steering results. Top: We extract CV' and JVs from antonym ICL prompts in formats that are in-distribution (ID; OE-ENG) or out-of-distribution (OOD; OE-FR, MC) relative the AmbiguousICL task (bottom-left). Bottom-left: We interleave two ponceps--anton one prompt: the model's original prediction is the French translation. Bottom-right: Predictions after steering. Takeaways: (1) FVVs yield larger ID gains. (2) CV's show more stable OOD effects across formats. (3) FV's can conflate concept with input format (e.g., French version of antonym and multiple-choice formatting).
and
EN-FR translation-witte
3.1 STEERING METHODS
Steering Vectors Construction. For each concept and input format (OE-ENG, OE MC), we compute for every selected head az the mean last-token activation across the 50 extraction prompts

---

Published conference paper at ICLR 2026
of that concept- We then form one vector per format by summing these mean activations over the top-K heads selected for CV² or FV (as in Eq. 5 but using per-format means in place of per-prompt activations). This yields one ID vector per concept.
and two OOD vectors (OE-FR, MC)
AmbiguousICL Task. We evaluate on AmbiguousICL tasks (Figure each prompt interleaves two concepts (3 then 2 exemplars) followed by a query. The second concept is always English-PFrench translation. Unsteered models tend to continue with the second concept; we aim to steer toward the first. Note that steering prompts are distinct from the extraction prompts used to construct the vectors. This setup is diagnostic: it tests whether representations encode abstract relational structure independent of extraction prompts' surface format. To perform well in this setup requires consistency between ID and OOD performance.
Steering with CVs and FVs. We add vector to the last-token residual stream at chosen layer:
he he + av
We measure effectiveness AP Pater (v) Phefore (y). averaged over 100 prompts per concept (see Figure 22 for Top-1 accuracy). We sweep and K and report the best per model (Appendix F.
Antonym
Category
Causal
Singular-Plural
py
0.30
0.30
0.15
0.05
0.50
0.00
0.00
0.00
AP
Cy 0.30
0.15
0.05
0.50
0.00
0.00
0.00
0.00
0
50
50
50
Layer
English (ID)
French (OOD)
Multi-Choice (OOD)
Figure 7: Steering effect across layers. We inject CVs and FVs into 1--ma and plot the change in target-token probability (AP) for four representative concepts (columns). Curves compare ID extraction format with OOD formats relative to the AmbiguousICL task (Figure (6) Higher AP means the model assigns more probability
the expected token than the unsteered model. Takeaways: (1) FVs typically achieve larger ID gains but often drop OOD. (2) CVs yield smaller gains yet show more stable OOD behavior across formats. See Figure 16 for other concepts/models.
3.2
STEERING RESULTS
3.2.1
FUNCTION VECTORS OUTPERFORM CONCEPT VECTORS IN DISTRIBUTION
Extracted from OE-ENG (ID setting), FV's yield the largest gains ambiguous prompts (Figure 7 CV's also help but with smaller AP and minimal zeroshot effect (Figure (7). At the token level both vectors lift plausible English antonyms the ID case (Table
3.2.2 CONCEPT VECTORS ARE MORE STABLE OUT OF DISTRIBUTION
Performance gains (AP). Out of distribution (extracting vectors from OF-FR or MC), CV's more often maintain positive effects across formats, whereas FVs frequently degradc-especialyy especially for MC- MC-and only occasionally stay consistent for specific concepts/models (Figs 7 [6]. CV's raise the probability of the correct English answer across formats, and their top- A tokens remain concept- aligned (Table 2). Crucially, the key finding is not absolute performance but consistency: CVs increase the probability of producing similar concept-aligned tokens regardless of extraction format.
Distributional consistency (KL). To quantify consistency across formats independent of absolute gains, compare the model's next-token distributions after steering with ID and OOD vectors. For

---

Published conference paper at ICLR 2026
Query:
saity -
Antonym
Top A Tokens
OE-ENG
sweet [e
(+3%) uuss (+2%)
FV
OE-FR
_su 100 (45%% (+5%), (+3%)
MC
[656______00____00000000_04444_0 (+0%)
OE-ENG
sweet (+49%), [frshh (+8%) bbaaddlanddddd (+3%) (+3%). uns (+3%)
CV
OE-FR
sweet (+54%). fresh (+9%)
(+9%) bland (+3%), (+3%)) _ taste (+2%)
MC
taste (+3%) sweet (+35%), _reshrrhhhhhh bbland (+4%), (+3%)) _tas),
Table 2: Token-level steering effects. Top tokens with largest probability gains when injecting CVss or FVs into Llama-3.1-70B on the AmbiguousICL prompt (query shown above). Results shown the layer with the strongest in-distribution effect per vector. Without intervention, the model predicts French translation _sa (from salé) with 49%; antonym_ sweet has 2%. English antonyms in red, French in blue, and the opening bracket (MC token) in green.
each concept and vector type, we select the top 5 layers that achieve the highest ID AP. At each selected layer compute KL divergence
DrL [p(x WOOD) || |[[ [vm)
between the post-intervention distributions at the query token, where lower values indicate more similar effects of ID) and OOD vectors. We average this KL divergence over prompts and selected layers to obtain one score per concept, and then summarize per model (Figure S8. Across models, CVs yield lower KL than FVs. The CV-FV KL gap is larger for MC than for OE FR.
3.2.3 FUNCTION VECTORS MIX CONCEPT WITH INPUT FORMAT
Out of distribution, FVs reflect both prompt format and concept. When vectors are extracted from FR, they push the model toward the French translation of the concept (e.g., French antonyms), and when extracted from MC, they increase the probability of format tokens such as the opening bracket (Table 2}. We quantify the language effect by measuring AP for the French translation across concepts (Figure 13). TT). In the larger models, JVS substantially increase the probability of the French token, whereas CVs remain near zero; smaller models the effect negligible. Notably, FVs extracted from open-ended Spanish prompts induce almost the same bias toward the French translation as FVVs extracted from French prompts (Figure 14), even though the AmbiguousICL alternatives are French only. This pattern suggests that FVs capture generic translation//oon peesllation/ioreii-laagggu tied to the extraction format rather than language-specitn effect (Figure 15), these findings indicate that FVVs mix concept with surface format, while CVs are comparatively format-invariant.
signal content. Combined with the MC bracket
Llama 3.1
Llama 3.1 70B
Qwen2.5 7B
Qwen2.5 72B
Divergence from ID
Culti-Choice French
0.5
2.5
0.5
0.0
0.0
0.0
0.0
CY
2.0
2.0
2.5
80
2.0
0.0
0.0
0.0
0.0
Figure 8: Consistency of steering effects. KI. divergence between the probability distributions after the models were stecred with an ID (OE-ENG) and OOD vectors (OE-FR [top]. MC [bottom]). Lower values indicate more similar effects of ID and OOD vectors. Takeaway: CV's steer the models more consistently than FVs.
8

---

Published conference paper at ICLR 2026
4 RELATED WORK
Attention Head Categorization. Recent work has made significant progress in characterizing specialized attention heads that process in-context learning (ICL) tasks. For instance, Olsson et al. (2022) identified induction-heand which Yin & Steinhardt (2025) found can develop into /VV--VVeeee heads during training. Other specialized head types include semantic-inuuction heads (Ren et al. 2024), symbol-abstraction heads (Yang et al. 2025). and various others ZZengg al.] (024). Our work extends this line of research by identitying CV heads, attention heads that invariantly represent concepts in ICL tasks at high levels of abstraction.
Linear Representation of Concepts. A substantial body of research has established that concepts are represented linearly in LLMs' representational space (Mikolov al. 2013 Arora al. 2016
Elhage et al. 2022). This phenomenon, often termed the "Linear Representation Hypothesis' (Park
al. 2024). has been extensively studied across various tasks and domains. |Hernandzz et al. (2024)
demonstrated that relational concepts -similar to those we study in this paper can be decoded from LLM activations using linear approximation. Subsequent work by Merullo al. (2025) revealed that the success of such decoding depends on the frequency of concepts in the pretraining corpora, which may explain why some concepts are represented more consistently than others in our study. Our findings contribute to this literature in two ways: (1) providing further support for the Linear Representation Hypothesis, and (2) extending previous work on relational concept representations by localizing specific attention heads that carry such representations and demonstrating their invariance to input formats.
Symbolic-like reasoning in LLMss. Recent work has demonstrated that LLMs exhibit symbol- like representational properties even without explicit symbolic architecture (Feng & Steinhardi [024] Yang et al., [025 Griffiths et al., (025)) Yang al. (2025) define symbolic processing requiring two key properties: (I) invariance to content variations, and (2) indirection through pointers rather than direct content storage. Our CVs exhibit both properties: they are invariant input format
changes and function as pointers to content stored elsewhere, unlike FVs which directly store content
(3.2.3)
DISCUSSION
Our results separate two representational roles in LLMs: components that cause strong ICL perfor- mance and components that encode abstract concept structure. Function Vectors (FVs) occupy the first role, steering models effectively when extraction and application formats match, but deteriorating out of distribution (formats/languaaee) Conversely, Concept Vectors (CVs) built from RSA-selected heads encode concepts at a higher level of abstraction and generalize more robustly across languages and question types, albeit with smaller causal effects. This supports a view that invariance and causality are mediated by largely distinct mechanisms in similar layers.
Layers of abstraction. We define abstraction as encoding relational structure (e.g., "antonym"" while discarding surface details (c.g., "English, multiple-choice", Our results suggest JVss do capture abstract task information: they reliably encode concepts within a format (Figure 19) and causally effective even across formats (Appendix OO). However, their orthogonality across formats and retention of surface signals (e.g., MC brackets) reveal that they conflate this abstract content with surface form. In contrast, CVs cluster by concept regardless of format. Thus, FV's operate at a lower level of abstraction ("antonym in MC format"), while CV's operate higher level ("antonym"), independent of surface form.
Relation to Function Vectors. Prior work shows that JVss compactly mediate ICL and can transfer across contexts (Todd et al.] 2024). We refine this: FV portability is strong within families of prompts, but is not fully invariant surface format. Same-concept. FVs extracted from different formats are nearly orthogonal and can carry language/format signals (e.g., French subword or multiple-choice bracket tokens), while CVs track concept across formats with less surface content. This distinction can be framed as equivariance vs. invariance: FV's adapt to extraction format (e.g., producing French antonyms from French prompts, MC formatting tokens from MC prompts), whereas CV's steer toward similar outputs regardless of format. Finally, do not propose CV' competitors to FVs, but rather highlight a mechanistic dissociation: FVs drive behavior (causality) while CVs represent abstract structure (invariance).

---

Published conference paper at ICLR 2026
These findings have implications for theoretical models of ICL, such as recent work by Bu al. (2025) which posits the retrieval of a single function vector af for a function f. Our results suggest this model incomplete: given the orthogonality of FVss across formats, the function vector better conceptualized as format-conditional a(f, σ), implying convergence to multiple format-specific basins rather than a single global minimum. Furthermore, we find that FVs and CVvs are orthogonal (even within the same format) which suggests that task representations partition into distinct abstract and format- speeatt-epiificccce subspaces, rather than residing in a single unified space.
Implications for steering and interpretability. The dissociation between FVs and CVss sug- gests a practical trade-off. For maximal in- mn-distribution control, FVs are preferable. For robust out-of-distributtin control or probing abstract knowledge, CVss are more reliable.
However, CVs appear to require the concept to be already present in the prompt to exert influence. In zero-shot steering (Figure [7) and activation patching-whin require inducing restoring task "from scratch" "oaachh"-VV' are ineffective. In contrast, in AmbiguousICL, where the concept is present but competing, CVs successfully steer by amplifying the existing abstract signal. Thus, JVS seem necessary to instantiate a task, while CVs can modulate it once present.
Methodologically, AP identifies what causally drives behavior, while RSA reveals how representations organize by concept. This distinction highlights that behavioral control and abstract representation can be mediated by different mechanisms.
Analogies and abstract representation. Hill et al. (2019) propose that "analogies are something like the functions of the mind": concepts achieve their flexibility by being represented abstractly enough to permit context-dependent adaptation across diverse domains of application. This view predicts that relational concepts like antonym or causation should function identically whether presented as open-ended prompts, multiple-choice questions, or in different languages. Our findings offer mechanistic refinement: while LLMs do form abstract relational representations (CVs), these are largely distinct from the components that causally drive ICL behavior (FVs). This dissociation suggests that analogical task performance may not require-as or primarily rely on- ohethe most abstract conceptual representations. Instead, LLMs appear to solve ICL tasks via more format-specific mechanisms, even though they do form abstract representations.
Limitations and Future Directions. Our CV head selection targeted heads that encode all concepts simultaneously; this global criterion may miss concept-specific heads, which a per-concept RSA could reveal. We also did not probe how FVs and CVs emerge during model training or how they interact during inference.
We propose two possible hypotheses that could be explored in future work:
1. CVs and FVs interact during inference as detectiion/xecctiinn mechanisms. Previous work by Lindsey et al. (2025) has discovered model features that seem fire just before the model produces a certain type of output (c.g., a "capital" feature that fires just before the model outputs a name of capital), and ones that fire more generally when the text mentions different capitals. Other work found that ICL tasks can be understood from an "encoder/decoder" perspective (Han et al.] (025). where the encoder encodes the task into a latent space and the decoder decodes the latent space into the output. Both of these findings suggest that models separate the task encoding and execution into distinct mechanisms which can be linked to CV's (encoding/detection)) and FVs (execution).
2. CVs and FVs do not interact during inference; CVs are simply backup circuit. Another possibility is that the two mechanisms are independent. Other works have found backup circuits where models can form multiple, partially redundant circuits and compensatory self-repair under ablations (McGrath et af. 2023 Wang et al. 2022). Given that a) both sets of heads are in similar layers (Figure 5). suggesting CVs and FVs may operate in parallel via lateral information flow within the residual stream, rather than strict seep-hierarchical dependencies and b) that CV heads do not seem to have causal effects in usual ICL tasks (since they were not identified by AP), this hypothesis is also plausible. What is more Davidson et al.] (2025) found that different prompting methods yield a different causal tasks representations, therefore possible that ICL in LLMs consist of multiple, separate mechanisms.
10

---

Published conference paper at ICLR 2026
REFERENCES
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to PMI-based word embeddings. Transactions of the Association for Computational doi:
Linguistics, 4:385-399, 2016.
a URL
orq/016-1028/
Aleksandra Bakalova, Yana Veitsman, Xinting Huang, and Michael Hahn. Contextualize-hen aggregate: Circuits for in-context learning in gemma-2 2b, 2025. URL https://arxiv..org
abs abb/2500.000 00132
Madeline Brumley, Joe Kwon, David Krueger, Dmitrii Krasheninnikov, and Usman Anwar. Com- paring bottom-up and top-down steering approaches on in-context learning tasks, 2024. URL
https://arxlv..rrg absp/ 2421.11213
Dake Bu, Wei Huang, Andi Han, Atsushi Nitanda, Qingfu Zhang, Hau-San Wong, and Taiji Suzuki. Provable in-context vector arithmetic via retrieving task concepts, 2025. https://:////x
org/abs/2500 09820
Guy Davidson, Todd M. Gureckis, Brenden M. Lake, and Adina Williams. Do different prompting methods yield common task representation in language models?, 2025. htRphttps:///rrt
org/ abs/ 2505 12075
Deepl. SE. Deepl translator, 2025. URL 2025.
wwtp http://deel..ommmtraslllaaal. Accessed:
Adrien Doerig, Tim Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. High-level visual representations in the human brain are aligned with large language models. Nature Machine Intelligence, 7:1220-1234, 08 2025. doi: 10. (0088//2255-005-0107222
Changde Du, Kaicheng Fu, Bincheng Wen, Yi Sun, Jie Peng, Wei Wei, Ying Gao, Shengpei Wang, Chuncheng Zhang. Jinpeng Li, Shuang Qiu, Le Chang, and Huiguang Hc. Human-like object cenceste represennations emerge naturally in multimodal large language models. Nature Machine Intelligence, 76)) June 2025. ISSN 2522-5839 doi: 10. 00000///2256-025-110002 http://dx www.o.oorg/000 1038/ 42256 025
URL
049-2
Nelson Elhage, Neel Nanda. Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfeld-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-odds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCan- dlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022. URL.ttts:
arwiv, http://wwwwwo..//////a//aea
10652]
Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context?, 2024. URL
17191
Shuhao Fu. Function Vectors for Relational Reasoning in Multimodal Large Language Models. PhD thesis, UCLA, 2025.
Dedre Gentner. Structure- Stracture-mapping mapping: A theoretical framework for analogy. Cognitive science, 7(2): 155-170, 1983.
Thomas L. Griffiths, Brenden M. Lake, R. Thomas McCoy, Ellie Pavlick, and Taylor W. Webb. Whither symbols in the era of advanced neural networks?. 2025. URL httptts:///axi...too. abs 2o0/ 05776
Seungwook Han, Jinyeop Song, Jeff Gore, and Pulkit Agrawal. Emergence and effectiveness task vectors in co-context learning: An encoder decoder perspective, 2025. URL https
7/arxiv. //arxxi..orr//ab//24111111
11

---

Published conference paper at ICLR 2026
Roee Hendel, Mor Geva, and Amir Globerson. In-Context Learning Creates Task Vectors, October 2023.
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models, 2024. URL http:: http://///..aaa.ooooooaa//a/////
09124
Felix Hill, Adam Santoro, David G. T. Barrett, Ari S. Morcos, and Timothy Lillicrap. Learning to make analogies by contrasting abstract relational structure, 2019. URL http:/t://axxi..oo///
2tss/1902.00120
Douglas R Hofstadter. Fluid concepts and creative analogies: Computer models of the fundumental mechanisms of thought. Basic books, 1995.
Nikolaus Kriegeskorte. Representational similarity analysis connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2008. ISSN 16625137. doi: 10.3389/ncuro.06 004.2008.
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL pttps::/trrass formesfrrmee oremere r- o rcuits pub/2025/eat
www.oonnssssooeooeeoeoo /bio ogy html
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent sepfir in language model computations, 2023. URL https::
org/abs/ 2t0.aa//230.1777 15771
Jack Merullo, Noah A. Smith, Sarah Wiegreffe, and Yanai Elazar. On linear representations and pretraining data frequency in language models, 2025. URL httpp::/arxivvorg//dda
2504. 12459
Meta AI. The Llama 3 Herd of Models, November 2024.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig, Linguistic regularities in continuous space word representations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (eds.), Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 746-751, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL wtw.t:tt///// wwwaacaanntoloooooooo
/N13 1090/
Melanie Mitchell. Artificial Intelligence: A Guide for Thinking Humans. Picador, New York, first picador paperback edition, 2020 edition, 2020. ISBN 978- 978-1-250-75804-0
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodde Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion. Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. URL http:tttss//ariiv..ooo
OpenAl. Gpt-4o system card, 2024. URL
Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models, July 2024.
Christopher Pinier, Sonia Acuña Vargas, Mariia Steeghs- Turchin, Dora Matzke, Claire E. Stevenson, and Michael D. Nunez. Large language models show signs of alignment with human neurocogni- tion during abstract reasoning, 2025. ht..stttttt... //arxii/////vvvv////a wwg.bbbbbrbross..o.oeooo..bbbbbs btt////aa/2200... /200 22508100557 10057
12

---

Published conference paper at ICLR 2026
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang. Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mci Li, Mingfeng Xue. Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https:://axiv.. wtt.:///.p.//.....a.//.// 15112 15115
Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Quanshi Zhang, Xipeng Qiu, and Dahua Lin. Identifying semantic induction heads to understand in-context learning, 2024. URL https?
//arxiv.org//aaaa..... abs 221.22222..1322
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function Vectors in Large Language Models, February 2024.
Eduard Tulchinskii, Laida Kushmareva, Kristian Kuznetsov, Anastasia Voznyuk, Andrei Andriiannen, Irina Piontkovskaya, Evgeny Burnaev, and Serguei Barannikov. Listening to the wise few: Select- and-copy attention heads for multiple-choice qa, 2024. URL
1410.02344
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https::/arxiv.orxt
abs/1706f 0oss/100.0037 03762)
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: A circuit for indirect object identification in gpt-2 small, 2022. URL
nttps::/aaxxv..orgg// abrrg//ls//221.....// 00593
Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannanch Hajishirzi, and Ashish Sabharwal. Answer, assemble, ace: Understanding how Ims answer multiple choice questions, 2025. URL
http:://arxxvv.orr/abb///
15018
Yukang Yang, Declan Camphell, Kaixuan Huang, Mengdi Wang, Jonathan Cohen, and Taylor Webb. Emergent symholic mechanisms support abstract reasoning large language models, 2025. URL
Kayo Yin and Jacob Steinhardt, Which attention heads matter for in-context learning?, 2025. URL
httpsss/aarrii..orr//abbbb/2...
Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Mingchuan Yang, Bo Tang, Feiyu Xiong, and Zhiyu Li. Attention heads of large language models: A survey, 2024. URL https
// //arxiv. wttp 03752
13

---

Published conference paper at ICLR 2026
A PROMPT EXAMPLES
A.1 OPEN-ENDED (5-S110T)
Q: resistant
A: susceptible
Q: classify
disorganize
Q: posterior
A: anterior
Q: goofy
A: serious
Q: stationary
A: moving
Q: hairy
A:
A.2 MULTIPLE-CHOICE (HOTH
Instrection: unveil A:
(a) optional
(b) mild
(c) con
(d) conceal
Response: (d)
Instruction: Q: hooked A:
(a) unhocked
(b) stale
(c) sturdy
(d) sell
Response: (a)
Instruction: spherical A:
(a) unconstitutional.
(b) flat
(c) demand
(d) healthy
Response: (b)
instruction: Q: minute A:
(a) conservative
(b) hour
(c) retail
(d) awake
Response: (
14

---

Published conference paper at ICLR 2026
B SIMILARITY MATRICES FOR OTHER MODELS
Function Vectors
Concept Vectors
Antonym
Antonym
Category
Category
Cosine Sim
Causal
Causal
Synonym
Synonym
Translation
Translation
Past Tense
Past Tense
Plural Form
Plural Form
Figure 9: Similarity matrices extracted from top K 1 heads in CV's and FVs in Llama 3.1 8B.
Function Vectors
Concept Vectors
Antonym
Antonym
Category
Category
Cosine Sim
Causal
Causal
Synonym
Synonym
Translation
Translation
Past Tense
Past Tense
Plural Form
Plural Form
Figure 10: Similarity matrices extracted from top
heads in CVs and FVs in Qwen 2.5 7B.
Function Vectors
Concept Vectors
Antonym
Antonym
Category
Category
Cosine Sim
Causal
Causal
Synonym
Synonym
Translation
Translation
Past Tense
Past Tense
Plural Form
Plural Form
Figure 11: Similarity matrices extracted from top K heads in CV's and FVs in Qwen 2.5 72B.
15

---

Published conference paper at ICLR 2026
C
AIE SCORES
Llama 8B
Llama 70B
10a
Head Count
10a
10a
10
10
10
10
0.0
0.5
1.0
0.1
0.0
0.1
0.2
0.3
AIE
Top 10 Heads
Top Heads
Top Heads
Qwen 7B
Qwen 72B
Head Count
10a
10a
102
10
10
0.2
0.0
0.2
0.4
0.6
-0.1
0.0
0.1
0.2
AIE
AIE
Figure 12: Histogram of AIE scores for Llama 3.1 8B, 70B, Qwen 2.5 7B, and Qwen 2.5 72B, Note, the y-axis is log-scale. Takeaway: AIE scores are highly sparse.
D DATA GENERATION PROCESS
Concept sourcing: For most concepts (antonym, synonym, translation, present-past, singular-plural), sourced word pairs from the datasets used by Todd (2024). categorical and causal concepts, we generated word pairs using OpenAl's ' GPT-4o model (OpenAl 2024..
Translation generation: French and Spanish translations were created using DeepL's translation service (DeepL)
SE 2025) to ensure high-quality, contextually appropriate translations.
Generated concepts (categorical and causal): We prompted GPT-4o generate exemplar.category pairs (c. "apple:fruit", "bluc:colour") and causcreffect pairs (c.g., "stumble:fall", "storm:flood"), The model was given examples of the desired format and asked to produce 100 pairs per batch. We generated pairs batches of 100 until reaching approximately 1000 examples per concept, with retry mechanisms to ensure sufficient coverage. The final datasets were saved as JSON files containing input-output pairs.
Generated pairs underwent several filtering steps: (1) removal of duplicates based input words, (2) exclusion pairs containing underscores or numbers, (3) restriction to single words or two-word phrases (maximum one space per input/output), and (4) conversion to lowercase for consistency.
Quality filtering:
Multiple choice format: For multiple choice prompts, we generated four options per question by randomly sampling three additional outputs from the same concept dataset, ensuring all four options were unique. The correct answer randomly positioned among the four options.
16

---

Published conference paper at ICLR 2026
E SIGNIFICANCE TEST FOR RSA-AIE HEAD OVERLAP
We assess whether the observed overlap hetween the top-K heads selected by Concept-RSA and by ATE is larger than expected by chance under simple null model. Let N denote the total number of attention heads in the model (layers x heads per layer). For fixed K, each method selects size-K K subset of heads. Under the null hypothesis that these two subsets independent, uniformly random size-K K subsets of {I......V J. J. the overlap size
X = SRSA, & O SAFF,
follows hypergeometric distribution X Hypergeom (N, K, K). For observed intersection x, we report the one-sided tail probability
= Pr X
R
Entries with P2* < 0.05 are typeset in bold in Table !
F STEERING HYPERPARAMETERS
To optimize the intervention performance, conduct hyperparameter search for parameters:
a: steering weight that controls the strength of the intervention
K: the number of attention heads to extract for concept vector computation
We evaluate the following parameter ranges:
K [1,3,5, 10, 20, 50) for the number of heads
a (1,3,5, 10, 15) for the steering weight
The hyperparameter optimization is performed separately for each model using antonym prompts. We select the parameter combination that maximizes the average steering effect across all input formats. This ensures that chosen hyperparameters generalize well across different prompt structures. We report the best hyperparameters for each model Table
Model
Best K
Best
Llama 3. 8B
10
Llama 3.1 70B
10
Qwen 2.5 7B
10
Qwen 2.5 72B
15
Table 3: Optimal hyperparameters for steering interventions across different models. K represents the number of attention heads used for FV/CVextraction, while controls the intervention strength.
17

---

Published conference paper at ICLR 2026
G
INPUT FORMAT MIXING IN FUNCTION VECTORS
Llama 8B
Llama 70B
Qwen 7B
Qwen 72B
0.2
PV
Antonym
Categorical
0.0
Causal
Synonym
0.2
Present-Past
CV
Singular-Plural
0.0
20
50
20
0
50
Layer
Figure 13: AP for French translations of all the concepts. FVs and CVss are extracted from open-
ended French prompts.
Llama 8B
Llama 70B
Qwen 7B
Qwen 72B
0.2
PV
Antonym
Categorical
0.0
Causal
Synonym
0.2
Present-Past
CV
Singular-Plural
0.0
0
20
50
Layer
20
0
50
Figure 14: AP for French translations of all the concepts. FVs and CV's are extracted from open-
ended Spanish prompts.
Liama 8B
Liama 708
Qwen 7B
Qwen 728
0.50
0.02
7V0.02
Antonym
Categorical
0.00
1.00
0,00
0.00
Causal
CV9.02
0.50
0.02
0.01
Present-Past
Synonym
Singular-Plural
0.00
20
Layer
Figure 15: AP for the opening bracket token (. FV's and CV's are extracted from multiple-choce prompts.
18

---

Published conference paper at ICLR 2026
H STEERING RESULTS
antonym
categorical
causal
synonym
presentPast
singularPlural
0.06
0.20
rv
0.05-
0.00
0.00
0.00
0.00
0.00
0.00
AP
20
20
20
20
20
20
0.10-
0.01
0.08
cV
0.02
0.00
0.00
0.00
20
20
20
20
20
20
Layer
English
French
Multiple Choice
(a) Llama 3.1 8B
antonym
categorical
causal
synonym
presentPast
singularPlural
0.15
0.30
0.05
0.50
0.00
0.00
0.00
0.00
0.00
AP
50
50
50
0.20
00.08
0.03
0.10
0.20
CV
0,00
0.00
0.00
0.00
50
50
50
Layer
English
French
Multiple Choice
(b) Llama 3.1 70B
antonym
categorical
causal
synonym
presentPast
singularPlural
FV
0.25
0.15
0.08
0.30
0.40
0.00
0.00
0.00
0.00
0.00
AP
0.01
0.03
0.04
0.10
0.00
0.00
0.00
0.00
Layer
English
French
Multiple Choice
(c) Qwen 2.5 7B
antonym
categorical
causal
synonym
presentPast
singularPlural
0.15
0.25
0.40
0.30
0.00
0.00
0.00
0.00
0.02
0.02
0.06
0.06
CV
0.00
0.00
0.00
0.00
0.00
0.00
-0.06
50
50
50
50
50
Layer
English
French
Multiple Choice
(d) Qwen 72B
Figure 16: Steering effect across layers and all concepts for different models.
19

---

Published conference paper at ICLR 2026
I
O-SHOT STEERING RESULTS
antonym
causal
categorical
synonym
presentPast
sinqularFlure ural
7V0.3
0.025
0.0
0.00
0.0
0.05
0.02
070.2
0.025
0.0
0.00
0.0
20
20
Layer
FR
MC
(a) Llama 3.1
antonym
causal
categorical
synonym
presentPast
singularPlural
0.025
0.4
0.50
0.2-
0.2
0.0
0.0
0.00
CP
0.2
0.025
0.1
0.1
0.4
0.000
0.2
0.25
0.0
0.0
50
Layer
(b) Llama 3.1 70B
antonyym
causal
calegorical
presentFast
singularflural
0.05
02
0,2
0.0
0.00
0.0
0,05
0.4
0.0.2
0.05
0.2
0.00
20
Layer
(c) Qwen 2.5 7B
antonyy
categorical
presentPast
singularPtural
0.10
02
0.05
0.1
0.00
0.2
0.0
0.4
0.02
0.1
0.4
0.2
0.00
0.0
0.0
0.0
Layer
MC
(d) Qwen 2.5 72B
Figure 17: 0-shot steering effect across layers and all concepts for different models.
20

---

Published conference paper at ICLR 2026
J QWEN 2.5 72B OUTLIER ANALYSIS
We identified anomalous CTE values for Qwen 2.5 72B in the Categorical concept across French open-ended and multiple-choice formats. As shown in Figure T8 these conditions exhibit unusually high CIE values with a bimodal distribution that deviates from the expected pattern. We excluded these two datasets from the final AIE caleulations. This exclusion has minimal impact overlap), confirming that our main findings are robust this methodological decision.
results: the top-5 head rankings remain identical (100%
Antonym
Categorical
Causal
02-
0.
0.1
-02
6.8
-04
Prempta
Prompla Singulärplural
Presentpast
Synonym
0.8
08
1,4
0.4
02
10
Englah
French
Multble Cho
Eng kh
French Prompts
Inglin
fratct Prompts
Chsloe
Figure 18: Violin plots of CIE for different concepts and prompts.
21

---

Published conference paper at ICLR 2026
K DISENTANGLING FORMAT-SPECHRT AND ABSTRACT REPRESENTATIONS IN FUNCTION VECTORS
Multiple-choice (MC) format involves distinct computational steps beyond open-ended generation: evaluating options, comparing candidates, and selecting among labeled alternatives (Tulchinskii et al. 2024 Wiegreffe al. 2025). When FV heads are extracted from prompts, they must therefore capture both (a) task representation (c.g., antonym, causation) and (b) the MC-specific formatting demands. We test whether partitioning out of the MC information, makes the task representation abstract, i.c., is shared across all formats?
We partition FV heads in Llama 3.1 70B into three subsets:
All Heads: Top-5 heads identified by AIE computed over all input formats (standard FV selection, Eq.
Common Heads: Hcads that appear in the top-10 heads for all three input formats independently. (3
heads). Unique MC Heads: Heads that appear the top-10 heads for format only. (6 heads)
abstract task representations exist in FVs independent of format, we would expect them to reside primarily in the common heads that are causally important across all formats. We then computed similarity matrices and RSA scores for each head subset.
We that within the MC format, common heads cluster by concept (Figure 19 but the representations nearly orthogonal between open-ended and multiple-choice prompts for the same concept (Figure 20)..
is still possible that these heads could encode both abstract task in open-ended and format features in MC prompts in superposition. However, under the Linear Representation Hypothesis (Park et al. (2024), shared abstract concept should occupy consistent linear subspace detectable via cosine similarity. The observed orthogonality across formats implics that any abstract representation invariant way. This suggests that FVs encode tasks at lower level abstraction (i.c., 'antonym in MC format') rather than a shared, formattindependent concept.
lincarly accessible format-
Unique
Antonym
Antonym
Antonym
Categorical
Categorical
Categorical
Causal
Causal
Causal
Translation
Translation
Translation
PresentPast
PresentPast
PresentPast
SingularPlural
SingularPlural
SingularPlural
Synonym
Synanym
Synonym
Concopt:RSA: 0.47
Concopt:RSA: 0.45
Concept 0.09
Figure 19: Similarity matrices for MC prompts only. Each panel shows the similarity matrix for a different subset of AlE-selected heads computed over MC prompts only concepts x 50 prompts each). Common heads show stronger concept clustering than unique MC heads (albeit with large similarity between concepts due to the shared MC structure). Full FVs show a very similar representational structure the common heads. Model: Llama 3.1 70B.
FP
FT: Shared Heads
FV: MC Unique Heads
1.0
Antonym
Antonym
Antonym
0.8
Categorical
Categorical
Categorical
Causal
0.6 g
Translation
Translation
Translation
PresentPast
PresentPast
PresentPast
0.4
SingularPlural
SingularPlural
SingularPlural
0.2
Synonym
Synonym
Synonym
Concept
Concept
0.0
Qllype-RS RSA:
Q-lype RSA:
Qllype
Figure Similarity matrices for different FV head subsets across all formats. Same Figure 19 but computed over all input formats (7 concepts x 3 formats 50 prompts). Within the same concept the representations are nearly orthogonal between open-ended and multiple-choice prompts. Model: Llama 3.1 70B.
22

---

Published conference paper at ICLR 2026
L STEERING RESULTS (ACCURACY)
Antonym
Categorical
Causal
Singular-Plural
0V0.40
0.30
0.10
0.50
Accuracy
0.00
0.00
0.00
0.00
CV 0V4.40
0.30
0.10
0.50
0.00
0.00
0.00
0.00
50
50
50
50
Layer
English (ID)
French (OOD)
Multiple Choice (OOD)
Figure 21: Steering effect across layers (Accuracy). We inject CV's and FVs into Llama-3 3.B-70B and plot the Top-1 accuracy for four representative concepts (columns). The grey horizontal line indicates the accuracy of the unsteered model. See Figure 7 for A results.
M MULTIPLE-CHOICE FORMAT WITH WORDS AS OUTPUT
Example prompt:
Instructicn: Q: spherical A: ?
unconstitutionas flat demand healthy
Response: flat
Instruction: unveil A:
optional
mild
sturdy
conceal
Response:
Antonym
Antonym
1.0
Antonym OE-FR
Antonym OE-FR
Antonym MC-Letter
Antonym MC-Letter
0.8
Antonym MC-Word
Antonym MC-Word
Categorical OE-EN
Categorical DE-EN
0.6
5mm
Categorical OE-FR
Categorical OE-FR
Categorical MC-Letter
Categorical MC-Letter
Categorical MC-Word
Categorical MC-Word
0.4
Causal OE-EN
Causal OE-EN
Causal OE-FR
Causal OE-FR
0.2
Causal MC-Letter
Causal MC-Letter
Causal MC-Word
Causal MC-Word
0.0
Figure 22: Similarity matrices for FVs and CVs with the inclusion of MC prompts where the model is expected to produce word instead of letter. Takeaway: Unlike CVs, FV MC representations still cluster due the input format, therefore MC cluster effect is not due to the model producing words/letters. Model: Llama 3.1 70B.
23

---

Published conference paper at ICLR 2026
N AMBIGUOUSICL IN DIFFERENT LANGUAGES
original AmbiguousICL implementation, the concepts are presented English the translations are presented French.
Here, we test the effect of presenting the concepts in difforent language to determine CVs are tied to specific surface form (e.g., "English Antonym"). Specifically, we present the concepts in Spanish and interleave them with Spanish-to-Engle Englissh translations.
Example antonym prompt:
Q:
final
A: inicial
Q: inmaduro
A: madura
Q: norte
A: sur
Q: descendiente
A: descendant
Q: probablemes emente
A: probable
Q: vivo
A:
The expected response the Spanish antonym "vivo" - 'muerto'. Therefore, the vectors are extracted from open-ended Spanish antonym prompts and the OOD vectors are extracted from open-ended English and multiple-choice prompts.
In Figure 23 we see that the steering effect trends are similar to the original implementation (although the absolute performance is lower for both CV's and FVs). In Figure [4] also that the KL divergence lower for the CV's compared the FVV.. Crucially, the CV's (extracted from English) steer the model produce the Spanish antonym (the contextually appropriate response), rather than the English antonym or the English translation. This demonstrates that CVss encode the abstract 'Antonym' concept rather than 'English Antonym' Mechanistically, this suggests that the CV amplifies the task probability ("do antonym") while relying the prompt's existing context determine the surface form ('in Spanish"), rather than injecting language-specific content.
Antonym
Categorical
Causal
Singular-Plural
0.20
0.06
PV
0.10
0.00
0.00
0.00
0.00
-0.40
AP
0.20
0.06
CV
0.10
0.00
0.00
0.00
0.00
-0.40
50
50
50
Layer
Spanish (ID)
English (OOD)
Multiple Choice (OOD)
Figure 23: Steering effect across layers and all concepts for different languages.
24

---

Published conference paper at ICLR 2026
Divergence from ID
Multi-Ci Choice
English
2.0
1.0
0.0
0.0
Figure 24: KL divergence between ID and OOD for different languages. Model: Llama 3.1 70B.
O CROSS-FORMAT ACTIVATION PATCHING
determine whether the dissociation between causal (FV) and invariant (CV) heads stems from activation patching setup-specifically. whether patching within the same format biases results toward format-specific heads- we conducted foss-format activation patching across all 6 combinations of input formats. In this experiment, extracted activations from clean prompts in format (e.g.. Open-Ended English) and patched them into a corrupted run of different format (e.g.. Multiple-Chooll Choice).
We found cross-forme patching consistently identified a subset the original Function Vector heads (e.g.. Llama-3. 3,1-0BB head L31HI8 always appears in the top-5 for both within-foo format and cross-format patching). It did not identify Concept Vector heads. This result shows that FVs are the primary causal mechanism for the task across all formats, despite their representations being format-specifie. CVs, while representationally invariant, not appear causally drive the model's behavior these tasks.
Source Format
Target Format
Max AIE
Overlap w/ FV (top-5) Overlap (top-5)
OE-ENG
OE-FR
0.22
OE-ENG
MC
0.02
0
OE-FR
MC
0.01
0
MC
OE-ENG
0.23
0
OE-FR
OE-ENG
0.41
MC
OE-FR
0.12
Table 4: Cross-format Activation Patching Results (Llama 3.1 70B). We show the max Average Indirect Effect (AIE) and the number of overlapping heads with the standard Function Vectors (FV) and Concept Vectors (CV) (top-5 heads). The patching source refers the format from which activations were extracted, and the target refers to the corrupted prompt format into which they were patched. FV heads consistently identified across formats (specifically L3 L3IH18 and L35H57), while CV heads are not.
25


Reviews:
[{"review_id": "ZxPkFNp1LF", "reviewer": "Reviewer_mXJu", "summary": "This paper distinguishes two types of interpretable vector representations in LLMs: Function Vectors (FVs) derived via activation patching and Concept Vectors (CVs) derived via representational similarity analysis (RSA). Through systematic experiments across seven conceptual relations and three input formats (open-ended English, multilingual, and multiple-choice), the authors show that FVs capture causal features that drive in-distribution performance but are sensitive to surface format, while CVs encode more abstract, format-invariant concept representations. The two sets of attention heads occupy similar layers but are largely disjoint, suggesting distinct mechanisms for causal effectiveness and conceptual abstraction.", "questions": "What suggestions would you provide to deep learning theoretical literature to model the function-vector and concept vector? Especially, for Bu et al., what can you suggest for their theoretical modeling to consider tasks beyond factual recall? Would there be any better algebratic manner to handle theoretical modeling of in-context learning with those vectors based on your findings, in the consideration of broader task types?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Well-designed methodology combining activation-based and representational analyses; broad and carefully controlled experimental setup (languages, formats, concepts); clear empirical evidence supporting the causal vs. invariant distinction; insightful visualization and analyses of head overlap and cross-format transfer; overall a solid and novel contribution to mechanistic interpretability of ICL.", "weaknesses": "- It is not surprising that the function vector is different from concept vector. Function vector is responsible in \"doing this task\", where concept vector is tailored for \"summarizing the topic of this task\". One is execution (based on low-level semantics such as current formats) and one is abstraction.\n- A detailed study of the mechanism - how the two sets of heads cooperate with each other semantically - is crucial but currently absent.\n- Several related work might be of insteast and warrent a detailed discussion:\n\nYang et al. Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning\n\nBu et al. Provable In-Context Vector Arithmetic via Retrieving Task Concepts.\n\nHan et al. Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective\n\n- 1050 prompts might be insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HPz5Z9pRlR", "reviewer": "Reviewer_Y7v1", "summary": "This paper investigates whether LLMs contain representations of concepts that are robust to prompt formatting. It revisits function vectors [Todd et al.], which are implicit representations of functions constructed from attention heads. Using RSA, they find that function vector representations in MCQ settings differ from those in open-ended settings. They propose Concept Vectors, which can steer the model’s output to be invariant to the source prompt format (producing the same output even under language or structure changes).\n\nThe main contributions seem to be twofold: (1) a critique that function vectors are not invariant to prompt format, and (2) that there exist attention heads that produce concept representations that are invariant to prompt format which are separate from function vector heads.", "questions": "- A listed contribution is that concept vector heads “encode concepts at a higher level of abstraction than FV heads” - can you elaborate on what is meant by this? (Line 86) \n- In Figure 3, left, it appears that the MCQ function vectors are all similar to each other. Do you think this is because they are steering the model to output a letter? Perhaps they are picking up on MCQ “task/concept” rather than the “concept” being specifically tested (e.g. antonyms). If you have the model output the answer instead of a letter in the MCQ setting and recompute the similarity matrices for function vectors, do they become concept-specific again? \n- If function vectors encode both concept and format, have you tried disentangling function vectors further to extract either the concept or the format from them separately?\n- The claim that \"invariance and causality\" are implemented separately by the model is a bit strange to me. I understand the part of the argument: attention heads for the concept vectors and attention heads for function vectors have little to no overlap. Is there more you're trying to say here beyond that? If so, I'm not sure what the evidence is that supports further claims\n\nOther Notes:\n- Related to your ambiguousICL setting, you may be interested in [Xiong et al]’s setup of task superposition, which multi-task behavior seems like a natural consequence of cross-entropy loss.\n- You may also be interested in [Davidson et al], who study instruction prompt version of function vectors. They find mostly disjoint attention heads are responsible for the same task when specified via few-shot or instruction prompts.\n- From your results it seems like function vectors are more “equivariant” than “invariant”, i.e., the output when steering tends to change along with the input content compared to the concept vectors. The idea that equivariance to prompting style (function vectors) and invariance to prompting style (concept vectors) are implemented separately is interesting (extrapolating from your finding that the heads are mostly disjoint). I wonder if this can tell us something about the nature of in-context learning and adaptive computation in LLMs? Perhaps it is best viewed as a composition of several mechanisms, some of which are equivariant to the context and others which are invariant for certain concepts. Curious if you have thoughts here if you have any. \n\n\n- Minor Typos:\n    - Figure 6, Line 296, Line 412: “Ambigous” -> Ambiguous\n    - Line 852: “mulitple” -> multiple", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The experiments were clearly explained, and figures communicate key results nicely.\n- The idea that the prompt form and underlying concept being tested can be separated is interesting and deserves further study. \n- Robustness of methods to prompting styles/rephrasings is an important problem that others would be interested in learning about\n- The paper acknowledges existing limitations of the study (e.g., concept vectors don’t steer as strongly in-distribution)", "weaknesses": "- The motivation of the paper (do LLMs represent concepts abstractly across surface forms?) currently feels disconnected from the experiments and results of the paper. I think it comes from the fact that this paper implicitly takes the view that a “concept” is a relation or function between two entities. Because “concept” is an overloaded term, it may be worthwhile to include an explanation/discussion of why this definition of “concept” was chosen, since it seems central to the motivation of the paper. For example, another common framing of “concept” is that of multi-token entities [Meng et al, Nanda et al], but this is very different from your view.\n\n- The tasks used for evaluation in the paper are very simple, and because of this, the contributions appear to be mostly conceptual. This is fine by itself, though choosing stronger baselines (e.g. prompting, fine-tuning) and more complex tasks to evaluate the proposed concept vectors method would certainly strengthen the paper in terms of methodological contributions. \n\n- The steering results in section 3.1 are somewhat mixed. In-distribution, concept vectors steer behavior somewhat poorly compared to function vectors (there are no other baselines presented). And while concept vectors do indeed seem to be invariant to the prompt format and steer towards the same answer in English under OOD formats, it’s unclear to me whether this is actually desirable behavior. Can you explain your intuition here as to why we'd want the English answer? Isn’t the idea behind in-context learning that the model should adapt based on the input? In this case, the equivariant behavior of the function vector (i.e. varies with changes in the source prompt) seems maybe more reasonable/desirable in some cases (e.g. if our goal is trying to understand ICL). Or even a composition of the two tasks (e.g. the French antonym). It seems like there is adequate evidence presented to claim that function vectors (FVs) are not exactly \"invariant\" to prompt format, but a discussion of “invariance” vs “equivariance” might be helpful to include because the behavior of FVs in the OOD setting seems somewhat akin to “equivariance” based on the description/examples in the paper you've provided. \n\n___\n- Todd et al. [Function Vectors in Large Language Models](https://openreview.net/forum?id=AwyxtyMwaG)\n\n- Meng et al. [Locating and Editing Factual Associations in GPT](https://arxiv.org/pdf/2202.05262)\n\n- Nanda et al. [Fact Finding: Attempting to Reverse Engineer Factual Recall](https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall)\n\n- Xiong et al. [Everything Everywhere all at once: LLMs can In-Context Learn Multiple Tasks in Superposition](https://arxiv.org/pdf/2410.05603)\n\n- Davidson et al. [Do different prompting methods yield a common task representation in language models?](https://arxiv.org/pdf/2505.12075)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ejO9DjoUPQ", "reviewer": "Reviewer_eth4", "summary": "The paper investigates whether Function Vectors (FVs) are input-invariant: whether the performance with FVs vary across different input formats. They find that across different inputs, the representations are unstable. They instead introduce Concept Vectors (CVs) which are optimized via Representational Similarity Analysis (RSA). Their results show that CVs perform better out-of-domain than FVs do and that CVs and FVs are distinctly different mechanisms. Though the general performance of CVs are lower, the analysis is done well and the motivation for CVs make them a compelling source of future investigation in interpretability.", "questions": "1. Fig4, do you have some intuition about why the performance of CVs decrease as the number of K heads increases, in the \"Concept\" category?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The figures in the paper are well made and looks very good.\n2. The takeaways are clear and marked in each figure, which makes it very easy to read.\n3. Overall, the paper from start to end is also simple to read.\n4. The question is answered with a simple approach of RSA, and the authors do extensive analysis over in-domain and out-of-domain comparisons with both FVs and CVs.", "weaknesses": "1. I'd like to see the experiments with more prompts than just open-ended types and MC -- for instance, adding errors to the input prompt or small punctuation differences, if the claim is that CVs are input invariant. It's already good that there are experiments over multiple choice format too, but it would also be helpful to extend this towards formats that are \"incorrect\".\n2. The analysis is done well to contrast in what situations FVs or CVs might be preferable. But the results also do leave the reader wondering why the heads chosen for CVs are better OOD than FVs and why CV performance may be lower than FV performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DUJpssq4nG", "reviewer": "Reviewer_yZrA", "summary": "This paper introduces an RSA-based approach to identify *Concept Vector Heads*, attention heads in LMs that capture an abstract representation of the task. The authors compare Concept Heads with Function Vector heads identified in [Todd et al, 2023](https://arxiv.org/pdf/2310.15213) and show that Concept Heads better capture the abstract task representation whereas Function vector heads tend to be more susceptible to different input formats.", "questions": "1. The authors treat the MCQ format as trivial and an artifact of the input format. However, I think that MCQ or more generally selecting from a list of options meeting some criteria itself is a non-trivial task that may trigger different reasoning strategies in the LM, which are investigated in [Wiegreffe et al, 2024](https://arxiv.org/abs/2407.15018) and [Tulchinskii et al, 2024](https://arxiv.org/abs/2410.02343). Could the authors comment on this?\n\n   1.1. When Function Vector heads are being used to capture the task representation in the MCQ format you are actually asking them to capture two things: **(a)** the abstract task representation (e.g., antonym, singular-plural, etc) and **(b)** the MCQ-formatting task (select a/b/c/d). The results presented in the paper suggest that FV heads are capturning more of (b) than (a). However, I wouldn't assume this to be a failure of FV heads as they are still capturing *a* task representation, just not the one you want them to capture. And the CV heads were identified *intentionally* to capture (a) better than (b), unlike FV heads which do not get any such signals. So, of course the CV heads will perform better when evaluated on (a). This is not a fair comparison in my opinion.\n\n2. The authors reported probability difference as a performance metric for steerability. In my opinion, it makes more sense to report the accuracy (i.e., can CV or FV *cause* the LM to output the target answer). Although this is a harder metric, I think accuracy captures more the causal role of such abstract representations in LMs computation and was also reported in the FV paper [Todd et al, 2023](https://arxiv.org/pdf/2310.15213).\n   \n    2.1. In Figure 7, the scales in the y-axis are not aligned, neither vertically nor horizontally. The authors should at least consider aligning the y-axis scales vertically to make it easier to compare the trends across CV and FV.\n\n\n### Typos and Minor Suggestions\n* Replace \"Open-ended\" with \"Open-ended ICL\". Or, at least clarify when you first introduce the term.\n* Line 068: *\"... FVs extracted from different input formats (open-ended vs. multiple-choice) are nearly orthogonal (cosine similarity = **0.9**)\"*. I think there is a typo here. Cosine similarity is a measure of alignment, so 0.9 indicates that they are very aligned, not orthogonal. Your Figure 4 also support that this is a typo.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 1, "strengths": "A simple idea presented nicely.", "weaknesses": "This paper treats MCQ, more generally selecting from a list of options meeting some criteria, as trivial and treats it as a artifact of the input format. I am not sure if that is a valid assumption, See more in Question 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)