Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions
Authors: Sarah Wiegreffe
Oyvind Tafjord
Yonatan Belinkov
Hannaneh Hajishirzi
Ashish Sabharwal
Venue: ICLR 2025 Spotlight (2025)
Keywords: interpretability; multiple-choice question answering
Primary area: interpretability and explainable AI

Abstract: Multiple-choice question answering (MCQA) is a key competence of performant transformer language models that is tested by mainstream benchmarks. However, recent evidence shows that models can have quite a range of performance, particularly when the task format is diversified slightly (such as by shuffling answer choice order). In this work we ask: how do successful models perform formatted MCQA? We employ vocabulary projection and activation patching methods to localize key hidden states that encode relevant information for predicting the correct answer. We find that prediction of a specific answer symbol is causally attributed to a few middle layers, and specifically their multi-head self-attention mechanisms. We show that subsequent layers increase the probability of the predicted answer symbol in vocabulary space, and that this probability increase is associated with a sparse set of attention heads with unique roles. We additionally uncover differences in how different models adjust to alternative symbols. Finally, we demonstrate that a synthetic task can disentangle sources of model error to pinpoint when a model has learned formatted MCQA, and show that logit differences between answer choice tokens continue to grow over the course of training.

Published as a conference paper at ICLR 2025
ANSWER, ASSEMBLE, ACE: UNDERSTANDING How LMS ANSWER MULTIPLE CHOICE QUESTIONS
Sarah Wiegreffe Hannaneh Hajishirzi
Oyvind Tafjord
Yonatan Belinkov
Ashish Sabharwal
Allen Institute for AI, University of Washington, Techhnin wi @maaiag@@@@aaa. com
ABSTRACT
Multiple-choice question answering (MCQA) is key competence of performant transformer language models that is tested by mainstream benchmarks. However, recent evidence shows that models can have quite range of performance, par- ticularly when the task format is diversified slightly (such as by shuffling answer choice order). In this work we ask: how do successful models perform format- ted MCQA? We employ vocabulary projection and activation patching methods localize key hidden states that encode relevant information for predicting the correct answer. We find that the prediction of specific answer symbol is causally attributed few middle layers, and specifically their mechanisms. We show that subsequent layers increase the probability of the pre- dicted answer symbol in vocabulary space, and that this probability increase associated with sparse set of attention heads with unique roles. We additionally uncover differences in how different models adjust to alternative symbols. Finally, we demonstrate that synthetic task can disentangle sources of model error to pin- point when a model has learned formatted MCQA, and show that logit differences between answer choice tokens continue to grow over the course of training
self-attention
INTRODUCTION
We investigate the ability transformer LMs to answer formatted multiple-choice ques- tions, which involves producing an answer choice symbol (here, A B). We discover 1-3 middle layers at the last token position, and particularly their multi-head self-attention functions, respon- sible for answer selection. Later layers assign in- creasing probability to the symbol of interest in the model's vocabulary space, for which sparse of attention heads are responsible. Finally, when the prompt contains unusual answer choice symbols such as Q/z 2/R/x, some models initially assign high values to common answer symbols like A/B BBCCDD before aligning to the symbols in the prompt a late layer.
Figure 1:
Symbol-prompt alignment
Symbol promotion
Answer selection
Question
Ar..
Bi...
Answer:
Multiple-choice question answering (MCQA) is a mainstay of language model (LM) evaluation (Liang et al. 2022 Gao et al., 2023 Beeching et al. 2023), not in the least because it avoids the chal-
lenges of open-ended text evaluation. A prominent example is the MMLU benchmark (Hendryeks)
ct al. 2021, which is considered meaningful signal for new model releases (Lambert 2024a b).
Coode available at httpa///////ttttbboc//aaaaaaaaaa http:///hhhhhhacccmmmaaaaaaaeee/

---

Published as a conference paper at ICLR 2025
While early LMs were evaluated on MCQA questions by treating the task next-token next-token-preedition and scoring each answer choice appended to the input independently, many benchmarks and datasets now evaluate models on their ability produce the correct answer choice symbol in single infer- ence run when given the answer choices in the prompt (formatted MCQA; Figure and (3.1). In- deed, it has been shown that capable models, and particularly those that have been instruction-tune perform well on formatted MCQA tasks (Liang et al., 2022) Robinson & Wingate 2023; WWegreffee al., 2023). At the same time, despite their strong end-task performance, some models are not robust to seemingly inconsequential format changes, such as shuffling the position of the correct an- swer choice, changing letters that answers are mapped to, or using different symbols (Pezeshkpour
& Hruschka 2024 Alzahrani et al. 2024 Khatun & Brown 2024).
Understanding how LMs form their predictions, particularly on real-world benchmarks and task formats such as MCQA, is important for reliability reasons. Motivated by this, we ask: How do successful models perform formatted MCQA?
In this paper, we interpret internal components to uncover how models promote the prediction of specific answer choice symbols. We perform vocabulary projection and activation patching on three model families -alasa--aaaa 3.1 (Dubey et al. 2024), Olmo 0724 (Groeneveld et al. 2024) and Qwen
2.5 Yang et al. 2024) and three datasets MMLU (Hendrycks et al. 2021). HellaSwag (Zellers)
ct al. 2019), and copying task we create. Our findings, summarized in Figure are:
When models are correct, they both encode information needed to predict the correct answer symbol and promote answer symbols in the vocabulary space in very similar fashion across tasks, even when their overall task performance varies.
2. Answer symbol production is driven by a sparse portion of the network, namely, attention heads.
3. The process for correctly answering more unnatural prompt formats in some cases more complex: Olmo 7B Instruct and Qwen 2.5 1.5B models sometimes only begin producing the correct symbols for these prompts in later layers. We discover that the models' hidden states initially assign high logit values to expected answer symbols (here, A/B/A//////// before switching to the symbols given in the prompt (here, random letters like Q/2/R/X or O O/E/B/P).
4. simple synthetic task can disentangle formatted MCQA performance from dataset-specifts performance, and allows us to narrow down point during training at which Olmo 7B learns formatted MCQA.
RELATED WORK
Behavioral analyses of LM abilitics to answer formatted MCQA questions prompt models in a black-
box manner by constructing different input prompts and observing how they affect models' predic-
tions (Wiegrefte et al. 2023 PPeeeskpoorr & Hruschka 2024 Sun et al. 20224 ZZeng et al. 2024]
Alzahrani al. 2024, Khatun & Brown 2024]] Balepur et al. 2024
Wang et al. 2024). These
methods and findings are complementary to ours.
P for MCQA are limited. Most similar ours is Lieberum et al. 2023). who study what attention heads at the final token position are doing, and isolate a subset they coin "correct letter heads". They demonstrate how these heads attend to answer symbol representations at earlier token positions and promote the correct symbol based on its position in the symbol order, though they show this behavior does not hold entirely for answer choices other than A/B/C/D. Their experiments are limited to one task (MMLU) and one closed- source model (Chinchilla-70B). Apart from broadening the scope of analysis substantially, we also analyze model behavior over the course of training and when it is poor, and create a synthetic task to disentangle task-relevan knowledge from the ability to answer formatted MCQA questions. We additionally put forward hypothesis for how models adapt when other answer choice symbols are used.
Efforts interpret transformer LMs mechanistically?
Li & Gao 2024 use vocabulary projection and study the norms of weighted value vectors in multi- layer perceptrons of GPT-2 models in order to localize the production of A, which GPT-2 models are biased towards producing when prompted O-shot. They find that updating a single value vector can substantially reduce this label bias. Unlike that work, we investigate how models do symbol binding
II the narrow technical sense of the term SSahhaa & Wiegreffe 2024
2

---

Published as a conference paper at ICLR 2025
when they are performant formatted MCQA; label bias is reason why model may not be correctly performing symbol binding.
Outside of MCQA, many works investigate how models internally build up representations that lead
to their predictions, such as on factual recall (Geva et al. [222]] Dai et al. 2022 Geva et al. 2022b
Meng et al. 2022 Geva et al. 2023; Yu al. 2023] Merullo et al. (024a)
basic linguistic (Vig
et al.) 2020; Wang et al. [023]] Merullo et al. [024bb] Prakash et al. 2024 Todd et al.) (2024 or
arithmetic (Stolfo al. 2023; Hanna et al. [023] Wu al. (202)) tasks. MCQA differs because
is both directly represented in real benchmarks and involves multiple reasoning steps.
3
FORMATTED MCQA
3.1 TASK NOTATION
Formatted MCQA is prompt format in which possible answer choices are presented the model as an enumerated list, each associated with an answer choice symbol. All of our dataset instances are either O-shot or 3-shot with 4 answer choices, formatted as follows:
For each of the tollowing phreses, seloct the
zest complction.
Soptional in-oonLext exanples of the sane Cosnal>
Corn yellow, oolor corn?
Choides:
&- yellow
2- geay
ping
The correct anOVGE 1s:
where y² A, B, C. or D (A this example) Models parameterized by are evaluated on their ability assign higher probability to the symbol associated with the correct answer choice (here, A) as opposed to the alternative symbols (here, B, C. and D). We compute ý argmax,.a,c. (p((ar)) po (C)))
5
To perform well at this task, models must
only predict the correct answer phrase (yellog low in the above example). but then map it to its corre- sponding symbol (A). Robinson Wingate (2023) refers to this capability as symbol binding.
3.2 DATASETS
We experiment two challenging real-world multiple-choice datasets from LLM benchmarks: HellaSwag (Zellers et al., 2019) and MMLU (Hendrycks et al. (202)). Details are given in Ap- pendix A.1 We additionally
prototypical colors dataset (Norlund et al. 2021) create synthetic 4-aay task disentangling -ptaset- knowledge from the ability to perform sym- bol binding: Copying Colors from Context ('Colors'). The dataset consists of instances such as & =Corn is, y 1l=eelllom low.. We include y in the context, so the model's only task is to produce the symbol associated with that answer choice given three distractors randomly selected from the ten colors present in the dataset (i.e., perform symbol binding). We use 3 instances as in-context examples and the remaining 105 as our test set. An example formatted instance is given above.
We validate that all models achieve 100% 3-shot accuracy on generative version of the Colors task: we provide instances in the format .* Corn is yellow. What color is corn?' and take the first greedy-decoded token as the prediction (i.e., we expect the model to produce 'yellew'' ").
3.3 MODELS
Olmo family. We experiment on the base (Olmo 0724 7B) and instruction-tuned (Olmo 0724 7B Instruct) versions of the most recent (0724) release of the Olmo model (Groeneveld et al. 2024). These have 32 layers with 32 attention heads per layer. We additionally benchmark performance on the smaller Olmo 0724 IB and supervised finetuned Olmo 0724 7B SET models.
Llama family. We experiment with the smallest Llama 3.1 models: Llama 3.1 8B base model and Llama 3.1 8B Instruct (Dubey et al. 2024). These have 32 layers with 32 attention heads per layer.
The actual answer tokens include preceding spaces some of the tokenizers of the models study.
3

---

Published as a conference paper at ICLR 2025
Hellaneng Crg
VALU Rige
Colrs Darsst
32% randors
0.99
1.30
Aug. Acouracy prompts
576
377
077
2.44
3.53
66
1:31
0.51
737
3.54
06so-1007724 0b10750722
Ohsa-T5-FTT
Ornc-B-iarrd
LiTTa 1.60
ONS125050
Ow8125150
Figure 2: Results by model on Colors, Hellaswag and MMLU. Plotted is the minimum accuracy across A/B/C/D, Q/2/R/ //// x, and 1/2/3/22//////// prompts, where the accuracy for each prompt taken as the average over all four correct answer positions. O-shot results for select models in Fig.
Qwen family. To study smaller performant models, we include the base and instruct versions of the 0.5B and 1.5B Qwen 2.5 models (Yang et al., 2024). The 1.5B model has 28 layers with 12 attention heads per layer.
3.4 IDENTIFYING CONSISTENT MCQA MODELS
We next isolate models capable of performing formatted MCQA by testing whether models answer formatted MCQA prompts consistently, i.e., they robust to the position and symbol of the correct answer choice. For each dataset instance, we construct four versions where we vary the location of the correct answer string, and thus y". We denote these A/B/C/D, A/B/C/D A/B/C/D, and A/B/C/D, where bold indicates the location of the correct answer. We average over these four prompts when reporting accuracy, and refer to this collective as A/B/C C/B/C/D prompts Lastly, to understand what behaviors we localize are specific to the letters used, we additionally include prompts Q 2/2/R. //X and 1 1/2/3/4 We evaluate each model per dataset instance on this set of 12 total prompts.
of instances of size * |dataset]
Results are Fig. where the minimum performance across A/B/C/D, Q/2/R/X, and 1/2/3/4 is plotted. Models exhibit differing performance the 3 datasets, indicating the datasets capture meaningful range difficulty. We additionally observe that many models achieve perfect or near- perfect performance error can be attributed to dataset difficulty as opposed to symbol binding. Finally, most (but not all) models are capable of performing symbol binding the Colors dataset with all 3 sets of answer choice symbols, despite the Q prompt being far less likely to occur in the training data 6
Colors while being far from it the other two datasets. indicating that
We select the best- pest-performin model from each model family for subsequent analysis: Olmo 0724 7B Instruct, Llama 3.1 8B Instruct, and Qwen 2.5 1.5B. They perform all tasks sufficiently above random accuracy (>20% above random, when averaged across prompts and answer positions).
LOCALIZING ANSWER SYMBOL PRODUCTION
4.1 ACTIVATION PATCHING
Activation patching, sometimes referred to as causal tracing (Meng et al. 2022 causal mediation analysis /Vig et al., 2020), or interchange intervention (Geiger al. 2020), is a method for perform- ing mediation on neural network hidden states to localize which network components have a causal effect on a model's prediction. The method involves performing the following steps:
4 A lways predicting single letter results in 25% accuracy when averaged across correct answer positions. This ensures that models with strong bias towards particular symbol are penalized.
When this prompt format used, we format
in-context examples with the same symbols.
For models that do not perform well on the Colors task, this is due to the MCQA format, since all models capy and produce the color string from the context with 100% accuracy 3.2
4

---

Published as a conference paper at ICLR 2025
1. Run inference dataset instance that the model predicts correctly, say, formatted as A/B/C/D (IA)). This will produce scores sc(A[zA) (higher) and sc(B|= sc(C/2a), sc(D] (D (A) (lower).
2. Run inference on the same dataset instance, but in a different format, such as A/B/C/D, that the model still predicts correctly (Ig)) This will produce second set of scores sc sc(3|23)) (higher) and sc(A|z sc(C/æ)), sc(D xΦ) (lower). We store hidden state activations of interest.
3. While running inference on A, replace the output hidden state at the final token position T and layer E, h with the hidden state from the same layer and token position, but from the infer- ence run of TB. ng. We measure and plot sc(A| (B)) sc(B|IA, b se(C|2 b (B)), )
and sc(D ( A. (
4. Repeat step 3 for each layer (L E L).
We use the notation øB - IA to indicate patching a hidden state from zn into I Meng et al. (222)) proposed performing activation patching not only layerwise outputs, but also the out- puts of multi-layer perceptron (MLP) and multi-head self-attention (MHSA) functions; Todd et al. 2202)) extend the latter by patching the outputs of cach weighted attention head / We do the same. Note two preliminary conditions for activation patching: the model predicts the correct answer for both FA and rB, and Z and 2g do not have the same answer. The score can be either logit or probit value; we discuss this in
4.2 VOCABULARY PROJECTION
Residual connections have been shown allow for the iterative refinement of features in neural networks (Jastrzebski et al. 2018 Simoulin & Crabbé] 2021). In addition to understanding which hidden states have the largest causal effect on predictions, is useful to understand how predictions form in the vocabulary space defined by the dot product between the hidden state output by the LM, hr,r, and the unembedding matrix, Wy. Recall that for an L-layer autoregressive LM,, model predictions are derived from logit values assigned to each item in the vocabulary V, sc E IRI given by: sc Wi- LN(h,rr) where LN is layer normalization, and h is the hidden state output by the final layer at the last token position. The scores are then further normalized into probits that sum to over the vocabulary: p Softmax(sc).
Prior work (nostalgebraist, 2020; Geva et al. 2021 2022a b Dar et al. 2023 Katz & Belinkov 2023
Yu et 2003) Merullo et al.) 2024a i.a.) has proposed projecting any hidden state in the Transformer block of dimensionality d to the vocabulary space, in order to inspect when and how models build up se and/or p via their internal representations. This technique, "vocabulary pro- jection", also sometimes called "logit lens" when logits are used, is equivalent to early exiting on the Transformer block at inference time (Dehghani al. 2019 Elbayad et al.) 20200] SShuuster al.) 2021 [222 i.a.). We plot the values assigned to the answer symbol tokens, as well as the maximum values se and p assigned to any other token, model states from each layer at the final token po- sition T. The "max other token" line delineates when scores assigned to the tokens of interest reach the top of the vocabulary ranking. The "logit difference" line plots the gap between the predicted token and the next-largest of the answer choices (akin to the scoring rule described in $311
4.3 LOGITS Vs. PROBITS
Logits are useful choice of metric for activation patching and vocabulary projection, in part because they capable of detecting model components which demote particular answer choices by assign- ing tower-than-avee from average logit values (i.e., both the direct additive contribution that each model component makes to the residual stream, and thus have a loosely causal interpretation, assuming no layer normalization on model outputs (Lieberum et al. 2023 Zhu et al. 2024). On the other hand, it is valuable to measure scores assigned to the tokens of interest (such answer symbol tokens) with respect to scores assigned to other tokens in the vocabulary and on a normalized 0-1 scale, as probits do. This 1) gives us insight into the rank
logit values (Zhang & Nanda 2024]; these can become indistinguishable near 0) after Softmax normalization. They also measure
MMHA output is weighted sum of individual attention heads. See Appendix A.2 for background on the Transformer architecture.
5

---

Published as a conference paper at ICLR 2025
Log
RRRZARR
Probabiity Poo LANeE
822888ANR88
intevertton Layar
Intorvnnioo Layer
Marverton Layer
(a) TTB/C/D-A/B/CL (HellaSwag)
(d)
U/B/C/D-A/B/OL/
02//R/X -A/B/C/D 1/2/3/4 +A/B/C/D (HellaSwag)
(Colors)
(HellaSwag)
Figure 3: Average effect (top: logits; bottom: probits) of patching individual output hidden states for
Olmo 7B 0724 Instruct (rg + AA predictions correct under both prompts. Patterns are largely similar regardless which position is used for replacement and the direction replacement. See Fig. T6 for additional results.
predicted
choice token 4th choice tokan logi: diff
Projected Ing
other
NARRR
1.2
Projected Probosblit
0.2
Projection Layer
(a) Colors (100% acc.)
(b) HellaSwag (62.4% acc.)
(c) MMLU (53.5% acc.)
Figure 4: Average projected logits (top) and probits (bottom) of answer tokens each layer for Olmo 0724 7B Instruct, for correct 3-shot predictions with the prompt A/B/C/D. See Fig. [3] for Llama 3.1 8B Instruct and Fig. for Qwen 2.5 1.5B Instruct. See Fig. for 0-shot results.
of the tokens of interest in vocabulary space, which affects greedy generation; 2) allows us to com- pare processing patterns and 3) allows us to disentangle logit changes that are specific to the tokens of interest as opposed general trends the entire model vocabulary. We thus use both metries because they provide complementary information.
different models, which often have different absolute logit scales,
4.4 INITIAL OBSERVATIONS
Activation patching and vocabulary projection complementary. As elucidated in Fig. 3b activation patching highlights the key causal role that layer 24 plays in encoding the answer choice that Olmo 7B Instruct will predict on Colors. While layers prior to 24 may promote or demote

---

Published as a conference paper at ICLR 2025
Projectdd Logit Valee
ligit diff
other
Projection Layar
Projecion Layer
Projection Layer
trojectdd Probebtts
RASEERSARS RRR Projctioo Layer
S Projction Layer
2A ARRA Proectioo Layer
2A R Nejectionn Layer
A/B//DD
(b) A/B//DDD
A/B//D
///////D
Figure 5: Average projected logits (top) and probits (bottom) of answer tokens at each layer for correct 3-shot predictions by Olmo 0724 7B Instruct on HellaSwag, with the prompt A/B/C/DB See Fig. 17 for Llama 3.1 8B Instruct, Fig. for Qwen 2.5 1.5B Instruct, and Fig. for 0-shot results.
Projectedd Prosibil
0.0
////R/X
2/2///RRXX
O/2////XX
(d)////R/X
Prjjectd Prpcebli
max other toten
22 1238R2ARR2 Projettion Layer
22R Proection Layer
33 RRR
20 R Projection Layer
Pajectionaye
1/2/3/4
O 1/2//3
3 1/2/3/4
1/2/3/4
Figure 6: Average projected probits of answer tokens at each layer for correct 3-shot predictions by Olmo 0724 7B Instruct on HellaSwag for the Q/2/R/2 (top) and 1 1/2/3/4 (bottom) prompts with various correct answers (indicated in bold). Results for another random set of letters
are in 2T] logit values are in Fig. 20 See Fig. 22 for Llama 3. 8B Instruct and Fig. 23 for Qwen 2.5 1.5B Instruct.

---

Published as a conference paper at ICLR 2025
specific token choice, the predicted token does not change, indicating that these layers' contributions to the residual stream are either nonexistent, or are overridden by layer 24's. Similarly, later layers appear to be carrying forward information contributed to the residual stream by layer 24 (while also slightly reducing the logits of the other answer choices), since they do not have any effect at undoing the change in predicted token caused by intervening on layer 24's output representation.
Compared with Fig. 3b Fig. 4a illustrates the complementary nature of the two methods: activation patching reveals important mechanisms before hidden states are projectable to the vocabulary space, while vocabulary projection shows how relevant information in the residual stream appears in the vocabulary space over remaining layers. Indeed, one hypothesis that arises from these results is that it takes a couple of layers of processing for information encoded in a hidden state to become mapped to higher scores on those tokens.
ROLES OF SPECIFIC LAYERS
Key Layers differ across models and # shots, but are consistent across datasets and predicted tokens. Fig. 4 presents projected token scores for Olmo 7B Instruct across all three datasets, and Figs.Ba||nd] 3hjativtttiiiiaattioon patches for two datasets. Despite Olmo 7B Instruct's varying performance on the datasets, key layers are largely consistent. Notably, layer 24 also plays the key causal role in the promotion of answer symbol tokens for HellaSwag, though layers 22-23 also have a small effect. In Fig. A the outputs of layers 26 and 29 lead to the largest logit and probit increases for the predicted answer token, with layer 31 serving to boost the logits of all tokens (this has no effect in probability space). Similar results are found for the Llama and Qwen models (Figs. 13 and 14), corroborating the hypothesis that the same mechanisms are responsible for promoting answer symbol letters across tasks, regardless of task complexity. However, comparing these graphs with Fig. 4 reveals that logit magnitudes and key layers differ across models, even for those of comparable size.
Comparing Fig. 4 (3-shot) to O-shot results (Fig. [5], the role of in-context examples primarily manifests in not only increasing the logit values of all answer choices in the last layers of the model (29-31), but also leading to a substantial increase in the predicted token's value such that it far surpasses other tokens in the vocabulary and accumulates nearly all the probability mass. This does not happen in the 0-shot case despite overall accuracy being similar, and also coincides with a noticeable (temporary) demotion of valid answer symbols at layer 28.
We break down Fig. 4bjby predicted answer letter (Fig. 5p. observing that trends are generally similar across A, B, C, and D for Olmo 7B Instruct and Llama 3.1 8B Instruct (Fig. 17 Qwen 2.5 1 1.5B exhibits fairly strong early preference for A at layer 20 which may be due to the token's 's corpus frequency, but is otherwise similar across symbols (Fig. (8), particularly from layer 22 onward.
Some answer symbols are produced in a two-stage process. Processing trends are noticeably different for some models when using different symbol space (Q/2/R/X, 1/2/3/ 4, another set of random letter symbols, O/E/B/P) For random letters, Olmo and Qwen models first assign non-negligible probability labels that are more likely expected (A, B, C, D) even though they are not included in the prompt, before making an abrupt switch to the correct symbols (0/2/RA C/R/X or O/E/B/P) at layers 29 and 23, respectively (Fig 6 top, Fig. 21 and Fig. 23 Activation patching reveals the same trend (Figs. 3c and 16). For example, for the Olmo model, patching in from prompts containing Q///R/X 29, noticeably later than layer 24 in the A/B/C/D experiments (Figs. [a| and (b). This provides evidence that Olmo 7B Instruct solidifies final label predictions for "OOD" prompts in later layers, and could explain why some models struggle with OOD formats. In the case of a more standard prompt, 1/2/3/4 3// 4, scores assigned A/B/C/D remain negligible (Fig. 6 bottom) 0
O/ O/E/B/P only decisively promotes the new symbol from layer
'We observe similar results in activation patching experiments to Figs, correct answer letters, further corroborating this trend.
3a and
3b when using different
'It is of note that activation patching from A/B/C/D to 1/2/3/4 (Fgg. 3d also identifies layer 29 as a key layer. This differs from Fig. Bal where position but not symbols changed. Since correct answer position does not change in this experiment, positional information, while layer 29 is attributed encoding the relevant answer token at that position.
possible explanation is that layer 24 is causally attributed to encoding
8

---

Published as a conference paper at ICLR 2025
predicted toker 2nd chokc toban
Iojected Loge Ve00
LOgHt Prul 1
legit
A2928R2 MXRA Layer
R
Insovvenion Layer
(a) MHSA Projection
(b) MLP Projection
(c) MHSA Patching
MLP Patching
Figure 7: Avg. projected logits with the A/B/C/D and avg. logit values at the final layer (patching
A/B/C/D - -A/B/C/D) hidden states output by different functions of the Olmo 7B 0724 Instruct
model on HellaSwag. These are function-wise breakdowns of Figs. 3a and 4b breakdown of Figure 7c by attention heads.
See
Figure 12 for
ROLES OF LAYER COMPONENTS
We next investigate the role that MHSA and MLP functions play in the observed behavior 10 weighted by their respective rows of the MHSA output matrix, since the MHSA output vector is weighted sum of the heads' output vectors. See Eqs. to (3) in Appendix A.2 for the derivation.
Self-attention mechanisms dominate the production of answer choice symbols (over MLPs). In Figs. 7 7b directly project the hidden states output by these functions (positions A and B in Fig. (0). Both functions map to large increases in logit values at layer 29, However, MLP outputs serve to increase the logit value of all answer choices, not to be discriminative (as noted by the near- 0 logit difference line). Increases in the logit difference are attributed to the MHSA functions at layers 26, 27, and 29, This is despite the fact that MHSA functions only contain "half the learnable parameters of MLPs ("67M vs. "135M per layer). However, attention heads have access to global context which an MLP does not have, making them especially useful for resolving back-references to portions of the input sequence or copying information from one position to another.
Answer symbol production is driven by a sparse portion of the network. The corresponding breakdown in the logit and probit differences from projecting individual attention head output vec- tors to the vocabulary space (Fig. Sa) shows that this effect quite sparse, with 1-4 attention heads layer (out of 32) projecting to non-negligible values any answer choice symbol. We addi- tionally observe high absolute magnitude overlap between Fig. Sa and Fig. 8b &c indicating that the same components promoting probability criminative in preferring one choice over the other (Fig. &b]Sc). However, the differences between Figs. [8b] and 8c reveal that some heads play letter-specific roles. Similar trends hold for the Qwen model (Fig. 25 Activation patching results (Figs. [7c and (7 further elucidate that the MHSA mechanism is driving the encoding of relevant information to predict the answer primarily at layer 24, with Fig. demonstrating that a single attention head is responsible for this effect.
any answer choices (Fig Sal are also generally dis-
7 WHERE ARE POORLY-PERFORMIIG MODELS GOING WRONG?
Our synthetic task separates formatted MCQA performance from dataset-specific perfor- mance. In Fig. g our synthetic task confirms that there is a crucial point at which formatted MCQA skill is learned for Olmo 0724 7B base between 80k and 100k training steps. Perfor- mance goes from near-random to near- 100%, even for checkpoints with poor performance on more challenging datasets. This result highlights the value of synthetic task: it helps to disentangle to what extent poor MCQA performance on a dataset is due to a lack of dataset-specific knowledge vs. an inability to perform formatted MCQA.
Vocabuuary projections in logit space of layer's 's ML. and MHSA functions direct additive decom- position of the projections on the layer's final hidden state. This is also for individual attention heads

---

Published as a conference paper at ICLR 2025
15
10
Layer
25-
20
25
200
15
20
20
(a) of A, when correct
max(Is C, D))
(c) 8- max(A, C, D)) when correct
when correct
Figure 8: Average (left) vs. difference (right two; separated by predicted letter) of logits when individual attention heads are projected to vocabulary space (Footnote [0). Plotted are results for correctly-predictee HellaSwag instances by Olmo 0724 7B Instruct. See Fig. 24 for probits. See for Qwen 2.5 1.5B Instruct.
Fig. 25
Figure 9: 3-shot accuracy of various Olmo 0724 7B Base checkpoints the Colors task. Avg. performance across all four correct answer posi- tions is plotted, with the dotted line indicating ran- dom performance. MCQA format is learned at specific (early) point training. Black dots indi- cate checkpoints used in subsequent analysis, rep- resenting 3 distinct points on the learning curve. See Fig. 26 for O-shot results.
1.0
08
03
03
0.
0.0
200000
Training Step
Poorly performing models cannot separate answer choice symbols in vocabulary space. We vocabulary projection to inspect differences between these checkpoints the Colors task. Fig. 27 (Appendix) illustrates the separability between the answer choice symbols, which grows function of training for more steps. While at 100k and 200k steps the model represents meaningful ranking of the answer tokens that results in high accuracy using the multiple-choice prediction rule, only with sustained training does the model widen the logit difference substantially, leading to the model assigning high probability to the predicted answer alone. The small differences in logit scores observed earlier checkpoints may explain why small edits network parameters are effective at decreasing label bias (Li & Gao 2024).
Finally, we observe in Fig. (Appendix) that the ability do symbol binding at the final training checkpoint, as demonstrated on Colors, does not alleviate small logit differences between answer symbols, even for correct predictions, on datasets on which the model less performant: scores assigned to correct answers are both higher variance and tempered in layers 30-31 in these cases.
CONCLUSION
This work provides novel insights into the mechanisms language models employ when answering formatted multiple-choice questions. Our analysis leverages two complementary techniques: vocab- ulary projection and activation patching. Using carefully designed prompts and a synthetic task, we uncover series of stages, attributable to sparse set of specialized network components, in which models select, promote, and align (for OOD prompts) the answer choice symbol they predict. With MCQA datasets such MMLU routinely used performance signal for model development, it important to disentangle and test formatted MCQA ability in order to build models that operate in more robust and reliable ways. Our work makes important first step towards this goal.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To the best of our ability, we have included all details necessary to reproduce our experi- ments in the text of this paper and accompanying Appendix. We have also open-sourced our code http://p./...ttt.ooooooaaoaaaaaa (https: http///huggggnccceeeeeee wtt.aaac..ooaaaataaaaaaaaa http:/assttteaaaaaaaeeeeeeeeeeee cww.aaawweecoyycooorrs..oo _mcaa.)_____.a..m.mom We fixed
mcga) and Colors dataset
co
random seeds to ensure full reproducibility of our experiments.
ACKNOWLEDGMENTS
We thank the anonymous reviewers, members of the Aristo team at A12, and members of the H2Lab at the University of Washington for valuable feedback. YB was supported by the Israel Science Foundation (grant No. 448/20), Azrieli Foundation Early Career Faculty Fellowship, and AI Alignment grant from Open Philanthropy. This research was partially funded by the European Union (ERC, Control-LM,, 1011654)2). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
REFERENCES
Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. When benchmarks model leaderboards. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13787-13805, Bangkok, Thailand, August 2024, Association for Computational Lin- guistics. doi: 10. www.18653/1122224.oo acl-long. 7447
targets: Revealing the sensitivity of large language
744. UK//tttpss///allaaaaooo.../////
Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts abduction: How do LLMs answer multiple-choice questions without the question? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers). pp. 10308-10330, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18663///20024.allloooggg https://cclatthoooge hwtp://www.oy..../2222.o. 5555
URL
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Huggingface open Ilm leader- board, 2023. URL https: httts::/hhugingffee htt.s///c.........e./////aaaaaaa /opee.
co/ spaces/ /Hwgggggggagggiiaa
11m leaderboard
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493-8502, Dublin, Ireland, May 2022. Association for Computational org/
Linguistics. doi: 10. 12111166/112222.ac-aaaaa
URL
581
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 16124-16170, Toronto, Canada, July 2023. Association for Computational Linggss- tics. doi: acl- acl-long. 893
URL https: waw.aaaathlloogo wrgpg00..../0..3..
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transforeers. In International Conference on Learning Representations, 2019. URL hetps:
//openrevien ew. net. www.:/ttttt.tetteem d=HyzdR: R9Y7

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
hatpgg/ab//24400 21783
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In In- lernational Conference on Learning Representations, 2020. URL htt.:ps:://pppreeiieeeee
net feet/ooru So KKVVKPH
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Nicholas Joseph, Nova DasSarma, Ben Mann, Amanda Askell, et al. A mathematical frame- work for transformer circuits. Transformer Circuits Thread Blogpost, December 2021. URL https://transformer cireritssrruutt....... pttp:///02////aaeeooo////eeeee html
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, et al. A framework for few-shot language model evaluation, 12 2023. URL https: /tt::///eedooooo org http://rooodd1050000
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupala, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.), Proceedings the Third BlackboxNLPP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 163- 173, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2000 http://000..bcccooooooooo
Backbbxnnlp1..66 URL httthhts:://// https////aaaaathooocooooo
16
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November Associatinn for Computational Linguistics. doi: 10. 221555/112221..mlmmmaaal https://acll antho ogy. org/ 2021 emnlp-ma ma i n. 446
URL
Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, and Yoav Goldberg. LM-debugger: An interactive tool for inspection and intervention in transformer- based language models. In Wanxiang Che and Ekaterina Shutova (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pp. 11, Abu Dhabi, UAE, December 2022a. Association for Computational Linguis- tics. doi: 10. emn p-pppodeo.....
URL httphtttp///// https///alaaahhhhhcc antho conthhlogy.co wwwphhhlogy...r//20/
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. doi: 00.1663//110022emmpppp. wwwpo////222111111122222 mom-aain.3
URL
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 12216-12235 Singapore. December 2023. Association for Computational Linguis- URL //acl UKL.httpps//eee.labbboooo..... wntthhlooy..rggg.....
tics. doi: 10. 101553/112222..mmmppmaaall lemn 1mmnppmaa ma 1 n.. 751
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. OLMo: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 15789- Bangkok, Thailand, August 2024. Association for Computational Lin- URL ht https: httpaa/see/eeelatttooo..ooo/////
guistics. doi: 1/224:33////222aallloooooo aal-iong 841/
12

---

Published as a conference paper at ICLR 2025
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater- than?: Interpreting mathematical abilities in a pre-trained language model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 76033-76060. Curran Associates, Inc., ine.ri cc httppppprrleseppppee_eeeee www.eeConneeeecceeeeeeeeee pdf
2023. URL https:://pooeeeeeee igs nwwrips
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL https::/
d7KB jmI 3GmQ.
Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018. URL
net/ wtt////rrummoooeeeeeee 91
Shahar Katz and Yonatan Belinkov. VISIT: Visualizing and interpreting the semantic information flow of transformers. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the As- sociation for Computational Linguistics: EMNLP 2023, pp. 14094-14113, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10111866/1122233 findings-emnlp.955 URL https: wwwanthooogy.oooor.oooooo 2005/2023..c0 - 939
Aisha Khatun and Daniel G Brown. study on large language models' limitations in multiple- choice question answering. arXiv preprint arXiv:2401.0798 2024. URL
org/abs/ 2401. 07955
János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp*: An efficient and scalable
method for localizing Ilm behaviour to components. arXiv preprint arXiv:2403.00745, 2024.
URL https: /tap:s://aaxvv.ooo
abs/2403. 00745)
Nathan Lambert. The end of the "best open Ilm". 2024a.
URL https: /www. Blogpost.
interconnects. http:/aaaa.//cooeeeeeeeiiiie
11ms
Nathan Lambert. Llama 3: Scaling open 1lms to agi. 2024b. URL https:// //WW..
3-ndd-------- swwlsagaaaaaoaaeaaaaaaa open-ppen----e open-1lms Blogpost
Ruizhe Li and Yanjun Gao. Anchored answers: Unravelling positional bias in gpt-2's multiple-
choice questions. arXiv preprint arXiv:2405.03205. 2024. URL https::/aaxiv..rrg/g/...o.//
2405. 03205
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint atXiv:2211.091 2022. URL htttpss:/aaxxvv.org//aassss 09110
Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chin- chilla. arXiv preprint arXiv:2307.09458, 2023. URL htttpp://axxiv.oo/ abs/2307.
09458
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Advances in Neural Information Process- ing Systems, volume 35, pp. 17359-17372. Curran Associates. Inc., 2022. URL
6f htpp:://proceeiingg...nee neurips. http://prrrp.eeeee//p///////////
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple Word2Vec- style vector arithmetic. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5030-50000 5047, Mexico City, Mexico, June 2024a. Association for Computational Linguistics. doi: 10.18653/v//0024, naaci-long.288 281. URL htRtsttps:: //acl o wwttholog...rg//224....mm naacl long. htw.:/44aaaa1lloaaaa 1
13

---

Published as a conference paper at ICLR 2025
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in trans- former language models. In The Tweifth International Conference on Learning Representations, 2024b. URL
net wtt.:/oonnmnndad/aa/ee.e
Tobias Norlund, Lovisa Hagström, and Richard Johansson. Transferring knowledge from vision to language: How to achieve and how to measure it? In Jasmijn Bastings, Yonatan Be- linkov, Emmanuel Dupoux, Mario Giulianelli, Dieuwke Hupkes, Yuval Pinter, and Hassan Saj- jad (eds.), Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 149-162, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 http:///wo//2..blbccoooo..pppppp www.bbbonll-......... URL
https://aclantne www.aaallannoooooooooooooooo ogy. wtt.y.orgg//2/0 bl ackboxmlp-1.11
nostalgebraist. Interpreting gpt: the logit lens. 2020. URL https: http:://wwwwweeswweee lesswrong. com/
-ens Blogpost.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity the order of op- tions in multiple-choice questions. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2006-2017, Mex- ico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/0024 Endings-naacl.. 130.
URL httpe://aclartthooooo.oooo
findiigs naacl.
1307
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: case study on entity tracking. In The Twelfth International Conference on Leurning Representations, 2024. TRLLhttps://openreeett
net forum?
id=8sXcAn0fF2D
Joshua Robinson and David Wingate. Leveraging large language models for multiple choice ques- tion answering. In The Eleventh International Conference on Learning Representations, 2023.
URL htttps:://ppereeeeeeet wew.:/umtt/ooumeeemmmm weweeeoooriddyeeeeeeeee
Naomi Saphra and Sarah Wiegreffe. Mechanistic? In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th Black- boxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 480-498, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/v/2024 blackboxnlp- 1.30. URL
307
Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelerated inference via confident adaptive transformers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 4962-4979, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics. doi: 406. httppss://aclaatthttttta
1ttp://112221emmm/.. org/ 202100011211 emn p- m. in. 406.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 17472. Curran Associates, Inc., neurips ec paperrappp...///rer httpppppppepepeeppaapeaeaeeeepae files/paper/2022/ Carrrr-Coneeeenccc.mmm pdf
2022. URL www.aaeelllaeeeeeeeaeeeeeeeeeaa1
File/ 6fwcce316a4eee
Antoine Simoulin and Benoit Crabbé. How many layers and why? An analysis of the model depth in transformers. Jad Kabbara, Haitao Lin, Amandalynne Paullada, and Jannis Vamvas (eds.), Proceedings the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pp. 221-228, Online, August 2021. Association for Computational Linguistics. doi: 1ttp:/5//w122221..lsssoo.... http://////////////aaaanooooooo/ wttp://ooo..rgg2//2.............
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Houda Bouamor,
14

---

Published as a conference paper at ICLR 2025
Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Meth-
ods in Natural Language Processing, pp. -033- Singapore, December 2023. Associa-
tion for Computational Linguistics. doi: 10. 10.18653/11/0233mmmmlt www.lmmmii.....e URL https:
//ac 1/aclanthologys .repy...rr/// prg/2023,3.000 emn p- ma inaain.... 435
Jiuding Sun, Chantal Shaib, and Byron C Wallace. Evaluating the zero-shot robustness of instruction-tuned language models. In The Twelfth International Conference on Learning Repre- sentations, 2024, URL
Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated cir- cuit discovery. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet. Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 407-416 Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10.18653/1/2024.000 htwp///////...kakkooooo.... 1.25. URL
fitts://acclattinn
wog..orgo. 2024 b. h14444..acccbooo www.axalb-pppppp. 1.21.
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL htthttps: htts: http::///oppeeeeeeeeeeeee net worummiciiooooooo Eorum?ic-wwyxtyyyyra Awwxttywwa
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
Sataaper_flesspeeeeee wwwess/aper//00/1 Paper.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 12388-12401. Curran Associates, Inc., 2020. URL
neur wttpcppcpppppeeeeeeeeeeeeeeeeeee saraaerr_rll/////e___err_rr____r files/ 2app://2271122220022222271222222
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In
The Eleventh International Conference on Learning Representations, 2023. URL https:
/openzeview wwwp/eeetemiiite NpsVSN6o4u1
Xinpeng Wang, Bolei Ma. Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. "my answer C": First-token probabilities do not match text answers in instruction citereettoo-tueedd language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACI. 2024, pp. 7407-7416, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653v1/2024, findings-acl.441 URL
hwwphhoooogyyrrg/2222444.. wtw.:/24....nniinn--e-e-eeee 4417]
Sarah Wiegreffe, Matthew Finlayson, Oyvind Tafjord, Peter Clark, and Ashish Sabharwal. In- creasing probability mass on answer choices does not always improve accuracy. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing, pp. 3392-8417, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 105553////222.emnlp....o www.3..mlllmmmmm iimaain.55 522
URL.
Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. In- terpretability scale: Identifying causal mechanisms in alpaca. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36. pp. 78205-78226. Curran Associates, Inc., 2023. URL /tttp::/pproeetiingg nwwrips cc pappr files//aapr/20023/
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv.2407.10671 2024. ARI.bhtts://arxtttt http://///r.a..or//////220..2.. 10671
15

---

Published as a conference paper at ICLR 2025
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Confer- on Empirical Methods in Natural Language Processing, pp. 9924-9959, Singapore, Decem- ber 2023, Association for Computational I Linguistics. doi: 10. 1215566/1/2222.emmlg 102533//11222.mmmmmmmmm...... URL https:
loatthooog...o. loylloly..roo.o... org 2023 emnlp-maill....mmmmm n.. 615
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Märquez (eds.). Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational l Linguistics. doi: 10. 18653/v1/P19-144 1472. T
/ ant ogy. wgg..a////991111..//
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth Interaational Conference on Learning Representations, 2024. RRLLstttps///peeeeeeee
net/ forum?id-:f11yyyuu
Chujie Zheng, Hao Zhou, Fandong Meng. Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors, The Twelfth International Conference on Learning Representations, 2024. URL UtLhttps:: htttps://ppenreeeee//
net/ Settttoouuunni--hh2PPXeTe
Zining Zhu, Hanjie Chen, Xi Ye, Qing Lyu, Chenhao Tan, Ana Marasovic, and Sarah Wiegreffe. Explanation in the era of large language models. In Rui Zhang, Nathan Schneider, and Snigdha Chaturvedi (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Tutorial Abstracts), pp. 19-25, Mexico City, Mexico, June 2024, Association for Computational Linguis- tics. doi: 16.3 htwp://2222.aactutorrriiiii tutciatt URL htthttts::// http::::s//allaathlllggeoo org/2224, htt.oon/SSS////tttt///aaaaaooooo
maacl-tutorials.3
A APPENDIX
A.I DATASET DETAILS
HellaSwag (Zellers et al. 2019) is a way multiple-choice commonsense natural language infer- ence dataset. The goal is select the best completion for given sentence prompt. We sample fixed set of 1000 instances from the test set used in our experiments. We sample 3 random training set instances to serve as in-context examples.
MMLU (Hendrycks et al. 2021). or the "Massive Multitask Language Understanding" benchmark, spans 57 different topical areas. The questions are 4-way multiple-choice spanning subjects in social sciences, STEM, and humanities that were manually scraped from practice materials available online for exams such as the GRE and the U.S. Medical Licensing Exam. We sample a fixed set of 1000 instances from the test set used in our experiments. We sample a fixed set of 3 in-context example instances from the provided for each topical area.
In-context example selection. We randomly select the correct answer position for each in-context example (which comes to positions 012; or A B C. Q R X, or 3, depending on the prompt format. We find that models have more consistent performance when in-context examples (particularly, the models we benchmark which are not instruction--uuedd) but we do also include O-shot results for subset of strong models, where the inclusion of in-context examples does not have a large effect on performance (Fig.
used
A.2 TRANSFORMER BACKGROUND
The output hidden states of transformer models (Vaswani al. 2017) are linear combination of the outputs of non- non-linear functions (i.e., multi-head self-attention and multi-layer perceptrons) at each layer. For most models, which sequentially apply multi-head self-attention (MHSA) and multi-layer perceptron (MLP), for input hidden state x,-1 E IRd, the output hidden state E Rd of layer is defined by the following equation:
X₁ = X + MHSAA, (LN(x + MLP X + MHSAa, (LN(x
(1)
16

---

Published as a conference paper at ICLR 2025
Where MHSAj, and MLP3, represent the multi-head self-attention operation and multi-layer per- ceptron, respectively, defined by their unique parameters at layer L, and LN represents layer nor- malization Note that + represents simple vector addition, which serves establish residual nections. This is possible because the output of each operation (MHSA, MIP, and LN) is a vector
RR
We visualize single layer's block structure in Fig. 10
Elhage et al. (021) hypothesize and provide evidence for the "residual stream" view Transformer inference, in which MHSA and MLP functions "read from" and "write to" the residual stream, which carries information through the layers. Additional evidence that MHSA and MLP functions promote specific tokens by writing the residual stream is given by Geva al. (2022b).
Given ipput token embedding X E F and applying Eq. recursively, the final hidden state of Transformer with L layers resolves
MHSA (LN(x))) MLP8 (x MHSA (LN(x)))
Xr. = X0 +
The output a MHSA function can be further broken down into sum of the output of each attention head. For the input vector to layer f, X the output of MHSA is computed as the concatenation of each head's vector output, Att. (LN(x(-1)) Rd// times an output weight matrix w Rddd which can be simplified into sum as follows (originally elucidated in (Elhage et al. 2021
H
MHSAa, (LN(xc-1)) = Σ w Att. (LN(xe-1))
(3)
0,h E (4/M)/ are the specific columns W corresponding to head When we perform experiments on individual attention heads, we are referring to the individual components of this sum (i.e., weighted attention head outputs).
where W:(
B STRENGTHS AND WEAKNESSES VOCABULARY PROJECTION VS. ACTIVATION PATCHING
Vocabulary projection provides a meaningful notion of how hidden states assign probabilities to tokens in the vocabulary space defined by the unembedding matrix, but it is not causal intervention, and it cannot uncover ways in which hidden states could be working to promote tokens in other linear non-linearly decodable) subspaces. For these reasons, negative results in earlier layers are uninformative. We supplement our findings by using activation patching to localize effects at earlier layers.
Activation patching makes some assumptions about the independence of model components to make computation tractable (namely, that one can patch in individual hidden states isolation measure their effect on the network opposed to patching all possible combinations of states). A nascent line of work focuses on making activation patching more efficient so that a larger number of states and/or state combinations can be intervened on Kramár et al.) (2024): Syed et al. (2024); this is an interesting direction for future work.
"MHSAg, also receives input the hidden states at layer é - from other token positions in the rollout (as dictated by the attention mask), which omit here for brevity.
17

---

Published as a conference paper at ICLR 2025
Output hidden state
Nonker
MLP
Min Avg. Accu uay over piommtt
Post-attention layernorm
self-attention
input layernrrm
Figure 11: 0-shot performance results on Colors, Hellaswag and MMLU datasets. Plotted is the minimum accuracy across A/B/C/D, Q/2/R/X, and 1/2/3/4 4 prompts, where the accuracy for each prompt is taken as the average over the correct answer choice being at each position.
Input hidden state
Figure 10: Structure of single layer in the Transformer architecture for the models we study. Green lines indicate residual con- nections. We perform vocabulary projection and activation patching on different represen- tations both pre aad-aaddsssss residual combina- tion, indicated by the letters.
18

---

Published as a conference paper at ICLR 2025
Lagit at Final Layer
Attention Head
Head
Attention Head
(a) Layer 22
(b) Layer 23
(c) Layer 24
Logit at Final Layer
Attention Head
Attention Head
Attention Head
(d) Layer 25
(c) Layer 26
(f) Layer 27
Logit Rinal Layer
Head
(g) Layer 28
(h) Layer
(i) Layer 30
Final Layer
R
Attention Head
(i) Layer 31
Figure 12: Logit values the final layer (patching A/B/C/D -A/B/C/D) for each attention head in the last 10 layers of the Olmo 7B 0724 Instruct model on a subset of HellaSwag. We do not patch attention heads before layer 22 because no effect observed in Figure 7c The results demonstrate that the promotion of specific answer choices (i.e., the spikes in Figure 7c are attributed to specific heads per layer, demonstrating the unique causal roles of individual heads.
19

---

Published as a conference paper at ICLR 2025
predicted tocen
2nd choice token
choic
Projected log
logit
max other token
Projection
Projectio
Proerten Probscstry
64-
Projection Layer
Projection Layer
Projection Layer
(a) Colors
(b) HellaSwag
(c) MMLU
Figure 13: Fig. results on Llama 3.1 8B Instruct.
predicted toicen
Projected Loge value
25-
3rd choice towen
2nd choice taken
20
15.
logit
max other token
RASERRAR Projection Layer
Prsection
Projection Layer
Probweiing
644
Projection ayer
Projection Rayer
Projection Ayer
(a) Colors
(b) HellaSwag
(c) MMLU
Figure 14: Fig. rrssuts on Qwen 2.5 1.5B Instruct.
20

---

Published as a conference paper at ICLR 2025
predicted
2nd choice token
3rd choice taben
Projected legh
max other towen
Propectoo
Project Donciiin
Irqeeted Prosbiity
0.-
04
Projection Layer
Projection Layer
Projection Layer
(a) Colors
(b) HellaSwag
(c) MMLU
Figure 15: O-shot version of Fig.
Logit at Rinal Layer
10-
10
12
18
22
24 26
10
32
24 16
20
20 52
30 30
20
Probabiity at Final Layer
64-
82-
P R Intervention Layer
Intervention Layer
Intervention
(a) Q/2/R/X -A/B//DD (Llama 3.1 Instruct)
(b) Q02//// A/B///D (Qwen 2.5 1.5B Instruct)
(Olmo 7B 0724 Instruct)
Figure 16: Average effect (top: logits; bottom: probits) of patching individual output hidden states for various models on predictions correct under both prompts on Hellaswag dataset. See Fig. for more details.
21

---

Published as a conference paper at ICLR 2025
Projected Logit Velue
3422382ARR
mrgjetto Procecity
Projection
Projection
(a) A/B///D
(b) A/B/C/D
A/B///D
ad)//B/CDD
Figure 17: Fig. for Llama 8B Instruct.
10
Projeced Logit valle
12
20
ciff
15
Projection
Projection Layer
Projected Protablity
ARRR Proaction Layer
BERRA Projection Layer
HARRAR
Polecction Laver
Projecsion Layer
(a) Aa)A///CDD
(b) A/B/C/D
A/B/C/D
(d) A/B/C/D
Figure 18: Fig, 5 for Qwen 2.5 1.5B Instruct.
Projected Logt Volue
logt dif
max other token
SAR2SRNARRR Loye
NARRRRNRRR Loyor
BERARRR
Layer
Proected Protabitty
Layer
Layer
(a) A/B/C/D
b/)///CDD
L/B///D
A/B/C/D
Figure 19: 0-shot version of Fig.
22

---

Published as a conference paper at ICLR 2025
POoeEEMA Looit ValUe
dif
RR
(a) 2/R/X
(b) Q/2///X
0/2///XX
Q/2///R/X
Projected Logit VilUe
20
token
2AAB5RNARR ARRRR Projection
2 NARR
Mnjecciin
Projection
(e) 1/2/3/4
(01/2/3/4
(g) 1/2/3/4
(h) 1/2/3/4
Figure 20: Logit plots for Fig. See Fig. 22 (top) for Llama 3.1 8B Instruct and Fig. 23 (top)
Qwen 2.5 1.5B Instruct.
Prgected Probeaiitty
2 Projection Layer
Projection Layer
Layer
Projection Layer
(a) O/E/B/E
(b) O/BBB/
(c) C/E/B/P
(d)
O/E/B/P
Figure 21: Average projected probits of answer tokens each layer for correct 3-shot predictions by
Olmo 7B 0724 Instruct for the O/E/B/P prompt with various correct answers (indicated in bold).
Proeectdd Loght Value
1.5
dift
ingeecte Procosity
(a) Q/2/R/X
0/2R//X
(c) 0/2/R/XX
O/2/////X
Figure 22: Average projected logits (top) and probits (bottom) of answer tokens at each layer correct 3-shot predictions by Llama 3.1 8B Instruct for the Q/2/B/X prompt with various correct answers (indicated bold).
23

---

Published as a conference paper at ICLR 2025
Projectod Logh Value
dit
SAREBRARR
mnjecreo Prosotiity
(a) Q/2/RXX
(b) 2/R//
(c) 0/2/R/X
(d) 0/2/B/X
Figure 23: Average projected logits (top) and probits (bottom) of answer tokens at each layer for correct 3-shot predictions by Qwen 2.5 1.5B Instruct for the 3/2/R/X prompt with various correct answers (indicated bold).
311
of A, C, when (b) max(B, C, when max(A, C, when
correct
rect
rect
Figure 24: Probit plot version of Fig.
See Fig.
25 for Qwen 2.5 1.5B Instruct.
10
Head
Head
Head
Head
Head
10
Head
(a) sum of A, B. C, D when A (b) A max(B, D) when & cor- (c) B max(A, C, D) when cor- correct
rect
Figure 25: Fig for Qwen 2.5 1.5B Instruct.
24

---

Published as a conference paper at ICLR 2025
1.0
0.7
Figure 26: O-shot accuracy of various Olmo 0724 7B Base checkpoints on the Colors task. See Fig. 9 more info.
0.
04
02
100020
200000
902000
400000
500000
600000
Trahiing Stop
precicted
Projcted Lagt Vetre
choice tokers
logi: ciff
max other tnow
1.0
Proseilty
0.-
Prodction Laver
Layer
(a) 50K steps (25% acc.)
(b) 100K steps
(c) 200 steps
(d) Final checkpoint (100% acc.)
(97% acc.)
(81 % acc.)
Figure 27: Average projected logits (top) and probits (bottom) of answer tokens at the final token position across each layer for preedictd instances from the Prototypical Colors dataset with the 3-shot ///C/D prompt, across various Olmo 0724 7B Base checkpoints. The final checkpoint approx. 652K steps.
25

---

Published as a conference paper at ICLR 2025
predicted token
Projected Logit value
2nd choice taken
25
choice
20-
4th choice token
logit diff
10
10
12
22
24
20
20
28
20
1.0
Projected Probability
0.4
30
Projection Layer
Projection Layer
(a) Colors (100% acc.)
(b) MMLU (53% acc.)
(c) HellaSwag (39% acc.)
Figure 28: Average projected logits (top) and probits (bottom) answer tokens the final token position across each layer for perrectly-predietee instances for Olmo 0724 7B Base with the 3-shot A/B/C/D prompt, across all datasets.
26


Reviews:
[{"review_id": "NxNQMAEpco", "reviewer": "Reviewer_E4U5", "summary": "The paper is devoted to the analysis of the internal processing of Transformer LLMs when answering Multiple Choice Question Answering task. First, it is shown that the ability to answer such question can be attributed to a few layers in the model. Second, this ability is implemented by a sparse set of attention heads. Third, this ability emerges at some point during model's training. Finally, an interesting observation is that in case of on-standard label symbols (e.g. QXYZ instead of ABCD) the model first tries to predict typical characters corresponding the answer position, which are replaced with the correct character in higher layers. \n\nThe analysis is performed on three MCQA dataset: MMLU, HellaSwag, and synthetic Colors dataset. As for models, three families are considered: LlaMa, Qwen and Olmo. For deep analysys, the models with consistent results (i.e. robust to answer order and labels perturbation) on all the three datasets are chosen", "questions": "1. In sec.3.4, why the threshold on 20%+ random (i.e. 45%) is chosen? Is there any reason behind this choice?\n\n2. Evalusting 3-shot accuracy with alternative symbols (e.g. QXYZ), which symbols are used for in-context examples?\n\n3. Do you have any ideas which of the observed phenomena can be extended to other tasks, beyond MCQA?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The paper is clearly written. All the claims of the paper are supported by experiments, and the results are sound\n+ The claimed properties are consisstently observed over datasets and model's families\n+ The observed results are interesting and helps to understand the internal mechanics of LLMs\n+ Althougt the methods used in the paper are not novel, the quality of the presented analysis is quite high. I especially appreciate the models's robustness analysis and the difference between probits and logits", "weaknesses": "- The range of considered model sizes is limited (0.5B-7B)\n- Although the observations presented in the paper are clear and consistent, they don't provide any real understanding how the model works. Besides, it is limited to Multiple Choic eQuestion Answering tasks, and there is no attempts to extend it to more general understanding of the inner Transformer's mechanics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B5pLYIhLp0", "reviewer": "Reviewer_FfQL", "summary": "The study investigates how LLMs solve multiple-choice question-answering (MCQA) tasks, focusing on intrinsic interpretability. The authors explore how the specific layers and attention heads contribute to selecting the correct answer from a set of choices. By applying methods like vocabulary projection and activation patching, the study reveals that particular middle transformer layers play a critical role in aligning the model’s predictions with the correct answer symbol. These components act selectively to bind the answer symbol (e.g., A, B, C, D) to the correct answer phrase. The paper provides a comprehensive pre-analysis of multiple models, including Olmo, Llama, and Qwen, across various prompt formats and datasets to select the models that understand the MCQA task. Additionally, it introduces a synthetic MCQA task to isolate MCQA capabilities from dataset-specific challenges. Key findings indicate that robust MCQA performance relies on both specific layers and sparse attention heads that drive answer selection, with some layers adapting to unusual symbol choices in later stages. This work offers a deeper understanding of model-specific MCQA mechanisms.", "questions": "- Could you please specify what values are presented by Fig. 8? Is taken from logit lens applied to MSHA components by heads? \n- What conclusion do you make from the fact that in Fig. 7c (MHSA) change in answers are visible only in 24 layer, while in Fig.3 it maintains further. It will be interesting to describe in detail that mechanism.\n- It is relatively small issue, but do you consider using another variant of logit lens? The reason is that logit lens by itself could be potentially worse for earlier layers [2] \n\n\nSmall comments:\n- Fig.8 is not aligned \n\n[2] Belrose, Nora, et al. \"Eliciting latent predictions from transformers with the tuned lens.\" (2023).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Evaluating across various prompt formats and permutations is really convincing, it really highlights this paper\n- Interesting analysis that includes both MHSA and MLP patching\n- Good visualization of when the MCQA understanding forms during training", "weaknesses": "- The motivation for selecting the patching format is not fully substantiated. By \"patching format,\" I refer to the reordering of the correct answer to a new position. Although this approach is intuitively clear, I believe the motivation could be strengthened by exploring different patching formats (for example, removing correct answer from sample).\n- The circuits are shown to be dependent on the nodes (parts of model) for patching [1]. Patching the layers is interesting, but for full picture it would be good to examine the attention maps by patching. \n\n[1] Miller, Joseph, Bilal Chughtai, and William Saunders. \"Transformer Circuit Evaluation Metrics Are Not Robust.\" First Conference on Language Modeling. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AuAd62fhBm", "reviewer": "Reviewer_xW8X", "summary": "This research work studies how LLMs solve MCQA task. Authors show which layers and layer components play the most crucial role in this process and at which point the OLMO model starts to learn to solve this task. They also find out how different is this process when the options are named by other symbols (not A,B,C,D).", "questions": "- Fig.9: Do you have such curve for any other model, aside of Olmo 0724 7B base?\n- Fig.4: I don't understand the purpose of making this figure. What should it show to us?\n- Fig.8: Can you, please, elaborate the captions a), b), c)?\n- In the most of your experiments, you only consider the models from the range of 1.5-8B (smaller models turned out to be too weak to make good MCQA predictions). Do you have a physical possibility to do at least some experiments with some bigger models (at least, 13B)? It would be especially interesting to see the effect of switching from A/B/C/D to Q/Z/R/X (and other non-standard letters/symbols sets) at such model.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper analyses an important topic, since MCQA tasks are very common in LLM benchmarks.\n- Authors came up with a good idea for their synthetic dataset. That dataset allows them to distinguish a model's understanding of MCQA format from it's knowledge in particular area, that is important for proper analysis.\n- A number of experiments are done on several models: Olmo, LLaMA 3.1, Qwen: -chat and -base versions. These models look like an appropriate objects for this study.\n- At the second half of their paper, authors especially concentrate on those models that are good in solving three MCQA tasks (synthetic, HellaSwag and MMLU). It is also a good move, because it allows to ignore \"noisy\" properties of weak models that could mislead researcher and reader otherwise.\n- Authors use a good practices in MCQA research, i.e. they permute A/B/C/D symbols and consider alternative symbols for answer options.", "weaknesses": "- The claim in the lines 518-519 (_\"these results demonstrate that an inability to disentangle answer symbol tokens in vocabulary space is a property of poorly-performing models\"_) is not sufficiently supported. Such claim definitely cannot be deduced from the analysis of the checkpoints of only one particular Olmo 0724 7B base model (however, at lines 79-81 you make much more humble version of this claim, with which I agree).\n- In the lines 76-78 you say: _\"We discover that the model’s hidden states initially assign high logit values to expected answer symbols (here, A/B/C/D) before switching to the symbols given in the prompt (here, the random letters Q/Z/R/X).\"_, but this claim is also not sufficiently supported. You only showed this effect for one non-standard letter set, and, again, I see only Olmo 7B experiments here (please, correct me if I'm wrong).\n- Some parts of the paper are poorly-written and unclear. E.g. I don't understand the caption for the Figure 1, especially the sentence _\"Finally, when we switch to more unusual answer choice symbols, one behavior by which models adjust is to initially operate in the space of more standard tokens, such as A/B/C/D, before aligning probability to the correct symbols at a late layer\"_. See also \"Question\" sections for the questions about other Figures.\n- There are following minor problems:\n-- Text in the diagrams is extremely small. Please, make it larger to make it readable.\n-- The paper contains typos such as word \"projectioan\" on line 066. Please, check the grammar of your text.\n-- Citations in the line 264 should be put inside the braces.\n\n**UPDATE**: Authors addressed the second major weakness by adding the results for other letter sets and other models. Due to this, **I raised my main score from 5 to 6** and \"Soundness\" score from 2 to 3. They also answered the questions, made their points more clear and addressed minor problems in the text of the paper. Due to this I raised \"Presentation\" score from 2 to 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gJuKzL33Dg", "reviewer": "Reviewer_WzgP", "summary": "The authors study the mechanisms that exist in LLMs that enable them to successfully answer multiple choice questions. They primarily focus on three models from different model families, chosen for their well-above-average accuracy across answer orderings and datasets. They use in their study three datasets: MMLU, Hellaswag, and a synthetic dataset (\"Colors\"). Mechanism study is done with activation patching and vocabulary projection. Among the authors' key findings are:\n* There is a specific layer or layers that are responsible for promoting the letter to be chosen.\n* Letter selection is handled by the multi-head self-attention part of the model, and in particular by a small proportion of total heads.\n* When unusual symbol groups (e.g., Q/Z/R/X) are used, the model first promotes A/B/C/D and then abruptly changes to promoting the unusual symbol group.\n* Models that don't perform well on multiple choice cannot effectively separate the answer letters in vocabulary space.", "questions": "* As mentioned in \"Weaknesses\" I'd be interested to hear the reason for using 0-shot accuracy in Figure 9.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper seems quite original. It is well contextualized with respect to prior work, and has novel contributions. The authors' claims are clear and are backed by broad and consistent experimental results. The interpretability findings may be of interest to researchers working in the space of MCQA and LLMs, or in the space of model interpretability in general.", "weaknesses": "* I'm surprised that the OLMo 0724 7B Instruct (Figure 9) study used 0-shot accuracy. In general, the authors use 3-shot accuracy in the main body of the paper, which seems prudent as the 0-shot case is somewhat ambiguous (it's unclear whether the correct response is answer label or answer text). It would be interesting to see 3-shot results for the study or to hear a justification for use of 0-shot accuracy.\n* The findings in this paper rely on some assumptions inherent in activation patching and vocabulary projection. For example, that it's reasonable to project hidden layers from early in the network to the vocabulary space. I didn't take this potential weakness into account in my rating of the paper, as I'm not intimately acquainted with very recent mechanistic interpretability work. But it is something the authors could address in the paper if desired.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)