Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions

## Abstract

Abstract Multiple-choice question answering (MCQA) is a key competence of performant transformer language models that is tested by mainstream benchmarks. However, recent evidence shows that models can have quite a range of performance, particularly when the task format is diversified slightly (such as by shuffling answer choice order).
In this work we ask: how do successful models perform formatted MCQA? We employ vocabulary projection and activation patching methods to localize key hidden states that encode relevant information for predicting the correct answer. We find that prediction of a specific answer symbol
is causally attributed to a single middle layer, and specifically its multi-head self-attention mechanism.
We show that subsequent layers increase the probability of the predicted answer symbol in vocabulary space, and that this probability increase is associated with a sparse set of attention heads with unique roles. We additionally uncover differences in how different models adjust to alternative symbols. Finally, we demonstrate that a synthetic task can disentangle sources of model error to pinpoint when a model has learned formatted MCQA, and show that an inability to separate answer symbol tokens in vocabulary space is a property of models unable to perform formatted MCQA tasks. 1 1 1 Code is available at https://github.com/allenai/understanding_mcqa .

## 1 Introduction

Multiple-choice question answering (MCQA) is a mainstay of language model (LM) evaluation (Liang et al., ; Gao et al., ; Beeching et al., ), not in the least because it avoids the challenges of open-ended text evaluation. One prominent example is the MMLU benchmark Hendrycks et al. (), which is considered a meaningful signal for new model releases Lambert (, ).

Figure: Figure 1: We investigate the ability of transformer language models to answer formatted multiple-choice questions, which involves producing an answer choice symbol (here, ‘‘A’’ or ‘‘B’’). We discover a single middle layer at the last token position, and particularly, its multi-head self-attention function, responsible for answer selection. Hidden states output by later layers continue to assign higher and higher probability to the symbol of interest in the model’s vocabulary space, for which a sparse set of attention heads are responsible. Finally, when we switch to more unusual answer choice symbols, one behavior by which models adjust is to continue operating in the space of ‘‘A’’ and ‘‘B’’, before aligning probability to the correct symbols at a late layer.
Refer to caption: extracted/5743777/figures_png/fig1.png

While early LMs were evaluated on MCQA questions by treating the task as next-token-prediction and scoring each answer choice appended to the input independently,
many benchmarks and datasets now evaluate models on their ability to produce the correct answer choice symbol in a single inference run when given the answer choices in the prompt (*formatted MCQA*; Figs. 1 and 3). Indeed, it has been shown that capable models, and particularly those that have been instruction-tuned, perform well on formatted MCQA tasks Liang et al. (); Robinson and Wingate (); Wiegreffe et al. ().
At the same time, despite their strong end-task performance, some models
are not robust to seemingly inconsequential format changes, such as shuffling the position of the correct answer choice, changing letters that answers are mapped to, or using different symbols (Pezeshkpour and Hruschka, ; Alzahrani et al., ; Khatun and Brown, ).

Understanding how LMs form their predictions, particularly on real-world benchmarks and task formats such as MCQA, is important for reliability reasons. Motivated by this, we ask: *How do successful models perform formatted MCQA?*

In this paper, we set out to interpret internal components to uncover how models promote the prediction of specific specific answer choice symbols.
We perform vocabulary projection and activation patching analyses on various model families—Llama 2 (Touvron et al., ) and Olmo v1 (Groeneveld et al., ) and v1.7 AI2 ())—and on three datasets (MMLU Hendrycks et al. (), HellaSwag Zellers et al. (), and a copying task we create). We find:

- 1.
Models both encode information needed to predict the correct answer symbol and promote answer symbols in the vocabulary space in a very similar fashion across tasks, even when their overall task performance varies.
- 2.
Answer symbol production is driven by a sparse portion of the network, dominated by a few attention heads, which specialize in promoting specific answer symbols.
- 3.
The process for correctly answering more unnatural prompt formats is in some cases more complex – a finetuned Olmo 7B model only begins producing the correct symbols for these prompts in later layers. We observe evidence the model may be making predictions first in the space of *expected* answer symbols (‘‘A’’, ‘‘B’’) before switching to the symbols provided in the prompt (‘‘C’’, ‘‘D’’).
- 4.
A simple synthetic task is able to disentangle formatted MCQA performance from performance on a specific dataset, and allows us to narrow down a point during training at which Olmo v1.7 learns formatted MCQA.
- 5.
Poorly performing models cannot separate answer choice symbols in vocabulary space.

## 2 Related Work

Behavioral analyses of LM abilities to answer formatted MCQA questions prompt models in a black-box manner by constructing different input prompts and observing how they affect models’ predictions (Wiegreffe et al., ; Pezeshkpour and Hruschka, ; Sun et al., ; Zheng et al., ; Alzahrani et al., ; Khatun and Brown, ; Balepur et al., ; Wang et al., ). These methods and findings are complementary to ours.

Efforts to interpret transformer LMs for MCQA are limited. Most similar to ours is Lieberum et al. (), who
study what attention heads at the final token position are doing, and isolate a subset they coin “correct letter heads”. They demonstrate how these heads attend to answer symbol representations at earlier token positions and promote the correct symbol based on its position in the symbol order, though they show this behavior does not hold entirely for answer choices other than {‘‘A’’, ‘‘B’’, ‘‘C’’, ‘‘D’’.} Their experiments are limited to one task (MMLU) and one closed-source model (Chinchilla-70B). Unlike us, they
do not inspect model behavior when it is poor, nor do they put forth a hypothesis for model behavior when other answer choice symbols are used. We additionally find diverse roles of important attention heads, with some always promoting specific symbols.

Li and Gao () use vocabulary projection and study the norms of weighted value vectors in multi-layer perceptrons of GPT-2 models in order to localize the production of ‘‘A’’, which GPT-2 models are biased towards producing when prompted 0-shot. They find that updating a single value vector can substantially reduce this label bias. Unlike that work, we investigate how models do symbol binding when they are performant at formatted MCQA; label bias is *one* reason why a model may not be correctly performing symbol binding.

Outside of MCQA, many
works investigate
how models internally build up representations that lead to their predictions, such as on factual recall
Geva et al. (); Dai et al. (); Geva et al. (); Meng et al. (); Geva et al. (); Yu et al. (); Merullo et al. () or basic linguistic Vig et al. (); Wang et al. (); Merullo et al. (); Prakash et al. (); Todd et al. () or arithmetic Stolfo et al. (); Hanna et al. (); Wu et al. () tasks. MCQA differs because it is both directly represented in real benchmarks and a more challenging task.

## 3 Formatted MCQA

We first introduce task notation, datasets, and our protocol for selecting strong models to analyze.

Formatted MCQA is a prompt format in which possible answer choices are presented to the model as an enumerated list, each associated with an answer choice symbol. All of our dataset instances are formatted as follows, unless otherwise noted:

For each of the following phrases, select the  best completion (A or B).    <in-context examples>    Phrase: <question>  Choices:  A: <correct answer>  B: <incorrect answer>  The correct answer is:
where $y=$ ‘‘A’’.(^2^22The actual token is ‘‘ A’’ in the tokenizers of the models we study, but we leave out preceding spaces for simplicity.) We include 3 in-context examples for each instance with labels ‘‘A’’, ‘‘B’’, ‘‘B’’.

Models are evaluated on their ability to assign higher probability to the symbol associated with the correct answer choice (here, ‘‘A’’) as opposed to the alternative symbol (here, ‘‘B’’). We compute $\hat{y}$ as ‘‘A’’ if $p(\text{{``A''}}|x)>p(\text{{``B''}}|x)$, else ‘‘B’’.

We convert datasets with $>2$ answer choices per instance to binary multiple-choice by pairing a randomly-selected
incorrect answer choice with the correct answer choice. We restrict instances to two options because the quantity we will plot when measuring discriminativity of model components (Section 6) only takes $1$ value for $2$ choices, but $n$ values for $n>2$ (see App. Section A.1 for an elaboration).

### 3.1 Datasets

We experiment
on two challenging real-world multiple-choice datasets from LLM benchmarks: HellaSwag (Zellers et al., ) and MMLU Hendrycks et al. (). More details are given in Section A.2. We additionally use a prototypical colors dataset (Norlund et al., ) to create a synthetic MCQA task that disentangles dataset-specific knowledge from the ability to perform symbol binding: Copying Colors from Context (‘Colors’). The dataset consists of instances such as $x=$‘‘Corn is’’, $y=$‘‘yellow’’. We include $y$ in the context, so the model’s only task is to produce the symbol associated with that answer choice, given one distractor answer randomly selected from the ten colors present in the dataset. We use 3 instances as in-context examples and the remaining 105 as our test set. An example formatted instance is as follows: ‘‘Phrase: Corn is yellow. What color is corn?\nChoices:\nA: yellow\nB: grey\nThe correct answer is:’’

### 3.2 Models

#### Olmo.

We experiment on both the base (Olmo-7B-Base) and instruction-tuned (Olmo-7B-SFT) versions of the Olmo model Groeneveld et al. ().
We also experiment on the Olmo-v1.7-Base model of the same size, which achieves better performance on a 4-way 0-shot formatted MCQA version of MMLU by making 0.01% of the pretraining data multiple-choice AI2 (). This performance gain is reported to occur at ~500B tokens of training Gu et al. ()— to investigate how model representations change once the formatted MCQA task is learned, we investigate one checkpoint before this point (350B training tokens), one checkpoint after (1T tokens), and the final checkpoint (~2T tokens).

#### Llama2.

We experiment with two sizes of base models from the Llama2 family Touvron et al. (): 7B and 13B, as well as the instruction-tuned $+$ RLHF-aligned 7B-Chat and 13B-Chat models.

All 7B (13B) models investigated have 32 (40) layers with 32 (40) attention heads per layer.

### 3.3 Identifying Consistent MCQA Models

We next isolate models capable of performing formatted MCQA.
We design a set of 4 prompts to test whether models answer formatted MCQA prompts *consistently*, i.e., they are robust to the position and symbol of the correct answer choice.

We define consistency as always selecting one semantic answer for a dataset instance regardless of how the prompt for that instance is constructed. For example, for the two prompts ‘‘A:_B:_’’ and ‘‘A:_B:_’’, where bold denotes the location of the correct answer,
the model is consistent if its two predictions are ‘‘A’’ followed by ‘‘B’’ (consistently correct)
or ‘‘B’’ followed by ‘‘A’’ (consistently incorrect).
Conversely, if the model always predicts the same token for both prompts, this indicates a preference towards that specific token or that specific position. We refer to this as being inconsistent (always ‘‘A’’ or always ‘‘B’’, depending on the model’s preference).
Henceforth, since we are interested in consistent models, we will always average over ‘‘A:_B:_’’ and ‘‘A:_B:_’’ prompts when reporting accuracy, and refer to these as ‘‘A:_B:_’’ prompts.(^3^33Always predicting ‘‘A’’ or ‘‘B’’ will result in 50% accuracy.) Lastly, to understand what behaviors we localize are specific to the letters used, we additionally include prompts ‘‘C:_D:_’’ and ‘‘C:_D:_’’.(^4^44When this prompt format is used, we format our context examples with labels ‘‘C’’, ‘‘D’’, ‘‘D’’.)

Figure: Figure 2: Results for each model on Hellaswag and MMLU datasets. Plotted is the minimum accuracy across ‘‘A:_B:_’’ and ‘‘C:_D:_’’ prompts. Only Olmo-7B-SFT and Llama2-13B-Chat achieve $\geq 20$% above random performance ($50$%), passing our consistency test. Full results available in Appendix (Tables 1 and 2).
Refer to caption: x2.png

We are interested in models that demonstrate learned ability to do the task of formatted MCQA with consistent symbol binding. We define such models as those that achieve
at least 20% above random accuracy, i.e., $\geq 70\%$ average accuracy, on both ‘‘A:_B:_’’ and ‘‘C:_D:_’’.
As Fig. 2 shows, only Olmo-7B-SFT and Llama2-13B-Chat achieve this criterion on all considered datasets. We focus on them in subsequent analysis.

Full results tables and statistics on consistency are included in Tables 1, 2, 3, 4 and 5 (Appendix), from which we make two observations:
(I) Models that are consistently correct on ‘‘A:_B:_’’ also maintain this level of performance on ‘‘C:_D:_’’, despite the latter prompt being less likely to occur in the training data.
(II)
Models exhibit differing performance on the 3 datasets, indicating the datasets capture a meaningful range of difficulty.

## 4 Localizing Answer Symbol Production

### 4.1 Activation Patching

Activation patching, sometimes referred to as causal tracing Meng et al. (), causal mediation analysis Vig et al. (), or interchange intervention Geiger et al. (),
is a method for performing mediation on neural network hidden states to localize which network components have a causal effect on a model’s prediction. The method involves performing the following steps:

- 1.
Run inference on a dataset instance that the model predicts correctly, say, one formatted as ‘‘A:_B:_’’ ($x_{A}$). This will produce probabilities $p(\text{{``A''}}|x_{A})$ (high) and $p(\text{{``B''}}|x_{A})$ (low).
- 2.
Run inference on the same dataset instance, but in a different format, such as ‘‘A:_B:_’’, that the model still predicts correctly ($x_{B}$). This will produce a second set of probabilities, $p(\text{{``A''}}|x_{B})$ (low) and $p(\text{{``B''}}|x_{B})$ (high). We store hidden state activations of interest.
- 3.
While running inference on $x_{A}$,
replace the output hidden state at the final token position $T$ and layer $\ell$, $\mathbf{h}_{\ell,T}^{(A)}$, with the hidden state from the same layer and token position, but from the inference run of $x_{B}$, $\mathbf{h}_{\ell,T}^{(B)}$. We measure $p(\text{{``A''}}|x_{A},\mathbf{h}_{\ell,T}^{(B)})$ and $p(\text{{``B''}}|x_{A},\mathbf{h}_{\ell,T}^{(B)})$.
- 4.
Repeat step 3 for each layer ($\ell\in[1,L]$).

We will use the notation $x_{B}\rightarrow x_{A}$ to indicate patching a hidden state from $x_{B}$ into $x_{A}$.
Meng et al. () proposed performing activation patching on the outputs of multi-layer perceptron (MLP) and multi-head self-attention (MHSA) functions; Todd et al. () on the outputs of each weighted attention head.(^5^55MHSA output is a weighted sum of individual attention heads. See Section A.3 for background on Transformers.) We also perform activation patching on these granularities (positions A and B in Fig. 10). Note two preliminary conditions for activation patching: the model predicts the correct answer for both $x_{A}$ and $x_{B}$,
and $x_{A}$ and $x_{B}$ do not have the same answer.

### 4.2 Vocabulary Projection

Figure: (a) Olmo; generative
Refer to caption: extracted/5743777/figures_png/colors/olmo-7b-sft_prototypical_colors_coarse_next_token_3ICE_ABBlabels_Nonecorrect_all_prob_new.png

Residual connections have been shown to allow for the iterative refinement of features in neural networks Jastrzebski et al. (); Simoulin and Crabbé ().
In addition to understanding which hidden states have the largest causal effect on predictions, it is useful to understand how predictions form in the vocabulary space defined by
the dot product between the hidden state output by the LM, $\mathbf{h}_{L,T}$, and
the unembedding matrix, $W_{U}$. Recall that for an $L$-layer autoregressive LM, model predictions are a probability distribution over each item in the vocabulary $\mathcal{V}$, $\mathbf{p}\in\mathbb{R}^{|\mathcal{V}|}$, given by:
$\mathbf{p}=\text{Softmax}(W_{U}\cdot\text{LN}(\mathbf{h}_{L,T}))$
where
LN is layer normalization, and $\mathbf{h}_{L,T}$
is the hidden state output by the final layer at the last token position.

Prior work (nostalgebraist, ; Geva et al., , , ; Dar et al., ; Katz and Belinkov, ; Yu et al., ; Merullo et al., , *i.a.*) has proposed to project *any* hidden state in the Transformer block of dimensionality $d$ to the vocabulary space, in order to inspect when and how models build up to $\mathbf{p}$ via their internal representations.
This technique, “vocabulary projection”, is equivalent to early exiting on the Transformer block at inference time (Dehghani et al., ; Elbayad et al., ; Schuster et al., , , *i.a.*). We plot the probability assigned to the two answer symbol tokens (‘‘A’’ and ‘‘B’’ or ‘‘C’’ and ‘‘D’’) as well as the maximum probability value in $\mathbf{p}$ at various model states at final token position $T$.

### 4.3 Initial Observations

#### Vocabulary projection reveals patterns distinct to MCQA.

Since vocabulary projection can produce small values in earlier layers of a network simply because hidden states have not yet been transformed into a space
mappable
to high probabilities on individual tokens, we first compare results on output hidden states of individual layers between the formatted MCQA task on Colors, and a greedy generation version, where we provide instances in the format ‘‘Corn is yellow. What color is corn?’’ and take the first greedy-decoded token as the prediction (i.e., we expect the model to produce‘‘ yellow’’).(^6^66All models investigated achieve 100% accuracy on this task given 3 in-context examples.) Results are shown in Fig. 3.

Figure: (a) Colors
Refer to caption: extracted/5743777/figures_png/colors/olmo-7b-sft_prototypical_colors_coarse_3ICE_ABBlabels_Acorrect_all_correct_prob_new.png

Figure: (a) ‘‘A:_B:_’’
Refer to caption: extracted/5743777/figures_png/hellaswag/olmo-7b-sft_hellaswag_coarse_3ICE_ABBlabels_Acorrect_correct_only_prob_new.png

We observe, notably, that while the greedy version of the task and the “top token” line demonstrate that hidden states *can* be transformed by the unembedding matrix into non-negligible probabilities as early as the output of layer 22 (for Olmo; layer 18 for Llama2), non-negligible probabilities on answer symbols emerge a couple of layers later (layer 23 for both models). Additionally, from these noted layers onward, the relative difference between the predicted token and the second choice is much smaller than in the greedy version. We conclude that it takes some extra layers of processing before output hidden states decisively predict an answer choice symbol (~layer 30 for Olmo and layer 32 for Llama2), even though updates to hidden state representations after layer 22/18 (Olmo/Llama2) are generally not changing models’ greedy-generated color predictions.

#### Correctness does not dictate predictive patterns.

We next investigate a dataset where a model performing consistent MCQA (Olmo-7B-SFT) does not achieve perfect performance: HellaSwag. We separate predictions by two attributes: *correctness of the prediction*, and *which letter is predicted* (‘‘A’’ or ‘‘B’’). Fig. 11 (Appendix) reveals that vocabulary projection does not reveal any meta-notions of correctness, as trends in vocabulary space align more with *what token* will be predicted (diagonal pairs) as opposed to *whether* that token is correct (vertical pairs). The same trend holds for activation patching (Fig. 12, Appendix). Thus, as we are most interested in correct predictions, we will only include correctly-predicted instances in subsequent plots (i.e., Figs. 5(a) and 5(b) for vocabulary projection and Fig. 6(a) for activation patching).

#### Activation patching and vocabulary projection are complementary.

As elucidated in Fig. 6(a), activation patching highlights the key causal role that layer 21 plays in encoding the answer choice that Olmo-7B-SFT will predict. While layers prior to 21 may promote or demote a specific token choice,
their contributions to the residual stream are overridden by layer 21’s, and similarly, later layers
appear to be
carrying forward information contributed to the residual stream by layer 21. In Llama2-13B-Chat (Fig. 16(a)), we observe a more diffuse window of ~6 layers (15-21) at which activation patching is increasingly able to change the model’s prediction.

Compared with Fig. 6(a), Figs. 5(a) and 5(b) illustrate the complementary nature of the two methods: activation patching reveals important mechanisms *before* hidden states are projectable to the vocabulary space,
while vocabulary projection shows how relevant information in the residual stream appears in the vocabulary space over remaining layers. Indeed, one hypothesis that arises from these results is that it takes a couple of layers of processing for information encoded in a hidden state to become mapped to higher probabilities on those tokens.

## 5 Roles of Specific Layers

Figure: (a) ‘‘A:_B:_’’ $\rightarrow$‘‘A:_B:_’’ (HellaSwag)
Refer to caption: extracted/5743777/figures_png/causal_tracing/olmo-7b-sft_hellaswag_layer_standard_to_inverse_3ICE_ABBlabels_1000insts_consistent_correct_new.png

#### Key Layers are consistent across datasets.

We observe projected token probabilities for Olmo-7B-SFT across all three datasets in Fig. 4, and activation patches for two datasets in Figs. 6(a) and 6(b). Despite Olmo-7B-SFT’s varying performance on the datasets, key layers are highly consistent.
Notably, layer 21 also plays a key causal role in the promotion of answer symbol tokens for Colors. Similar results are found for Llama2-13B-Chat (Figs. 14, 16(a) and 16(b)). These findings corroborate the hypothesis that the same mechanisms are responsible for promoting answer symbol letters, regardless of task complexity.

#### Some answer symbols are produced in a two-stage process.

When using a different symbol space (i.e., ‘‘C:_D:_’’), processing trends are noticeably different for the Olmo model (Figs. 5(c) and 5(d)).
The model first assigns probability to the labels that are *more likely or expected* (‘‘A’’ and ‘‘B’’), before making an abrupt switch to the correct symbol (‘‘C’’ or ‘‘D’’) at layer 30. Activation patching reveals the same trend (Figs. 6(c) and 6(d)):
patching in prompts that promote the production of ‘‘C’’ and ‘‘D’’ causes the relative difference between ‘‘A’’ and ‘‘B’’ to close at layers 21-29, before an abrupt change to predicting ‘‘C’’ driven by layer 30. This provides evidence that Olmo-7B-SFT solidifies final label predictions for “OOD” prompts in later layers, and could explain why some models struggle with OOD formats.(^7^77Note that this is specific to Olmo-7B-SFT. Llama2-13B-Chat immediately promotes ‘‘C’’ and ‘‘D’’ (Figs. 16(c), 16(d), 17(c) and 17(d)).)

## 6 Roles of Layer Components

Figure: (a) MHSA Projection
Refer to caption: extracted/5743777/figures_png/hellaswag/olmo-7b-sft_hellaswag_finegrained_attnOutput_3ICE_ABBlabels_Acorrect_all_correct_prob_new.png

We next investigate the role that MHSA and MLP functions play in the observed behavior.

#### Answer symbol production is driven by a sparse portion of the network.

In Figs. 7(a) and 7(b), we directly project the hidden states output by these functions (positions A and B in Fig. 10). Large spikes in probabilities come from *both* MHSA (layers 23, 30, 31) and MLP outputs (layers 28, 30). The breakdown in spiking behavior by attention heads in Fig. 8(a) (bottom) shows just how sparse the number of attention heads projecting to non-negligible probability values is (~8 total), even in layers where the MHSA output hidden state assigns large probabilities to ‘‘A’’ and ‘‘B’’ (recall that the MHSA output vector is a sum of these weighted attention head vectors). Activation patching results (Figs. 7(c) and 7(d)) further elucidate that the MHSA mechanism is driving the encoding of relevant information to predict the answer at layer 21, and at layer 29 for ‘‘C:_D:_’’ (Fig. 18, Appendix). The importance of the MHSA function as the source of the causal effect observed at layers 15-21 is also evident for Llama2-13B-Chat (Fig. 20, Appendix), though more attention heads in the model project to the vocabulary tokens (Fig. 21(a)).(^8^88It is unclear why vocabulary projections of the two functions are so different from activation patches for the Llama2 model.)

#### Individual attention heads dominate the production of answer choice symbols (over MLPs).

Plotting all attention heads and MLP layers that assign high probability to the sum of answer choices ($p(\text{{``A''}})+p(\text{{``B''}})>0.01$) in Fig. 8(a) (top; Fig. 21(a) for Llama2-13B-Chat), we see that attention heads generally dominate the distribution, despite the fact that they only contain ~half the learnable parameters of MLPs (~67M vs. ~135M per layer). However, attention heads have access to global context which an MLP does not have, making them especially useful for resolving back-references to portions of the input sequence or copying information from one position to another.

#### The same network components that promote the production of any answer choice promote specific answer choices as well.

Increasing probability on both answer choices is not the same as being *discriminative*, i.e., promoting one choice at the expense of another. We plot all heads and MLPs that are discriminative ($|p(\text{{``A''}})-p(\text{{``B''}})|>0.01$) in Figs. 8(b) and 8(c) (top; Figs. 21(b) and 21(c) for Llama2-13B-Chat).
There is 100% overlap: every plotted component also appears in Fig. 8(a).

Figure: (a) $p(\text{{``A''}})+p(\text{{``B''}})$ for ‘‘A:_B:_’’
Refer to caption: extracted/5743777/figures_png/hellaswag/olmo-7b-sft_hellaswag_finegrained_attnHeads_3ICE_ABBlabels_Acorrect_all_correct_deepmind_prob_barGraph_new_layer0_sum.png

#### Model components specialize.

We observe in Figs. 8(b) and 8(c) (bottom) attention heads that take on different roles depending on the predicted symbol. Of the 13 total model components in the two plots, we observe:

- 1.
3 that consistently assign high probability to ‘‘A’’. While they are consistently assigning higher probability to ‘‘A’’ than to ‘‘B’’, their average magnitudes drop when ‘‘B’’ is predicted.
- 2.
4 that consistently assign high probability to ‘‘B’’. While they consistently assign higher probability to ‘‘B’’ than to ‘‘A’’, their average magnitudes drop when ‘‘A’’ is predicted.
- 3.
3 that assign high probability to *whichever* token that is ultimately predicted, analogous to “correct letter heads” Lieberum et al. ().
- 4.
3 that only activate when ‘‘B’’ is predicted, though all assigning probabilities $<0.1$.

## 7 Where are poorly-performing models going wrong?

#### Our synthetic task separates formatted MCQA performance from dataset-specific performance.

In Fig. 9 (Table 3 for other models), our synthetic task confirms that there is a crucial point at which formatted MCQA skill is learned for Olmo-v1.7-7B: between 350B and 1T tokens. Performance goes from near-random to near-100%, even for checkpoints with poor performance on more challenging datasets.(^9^99As a reminder, *all* models achieve 100% accuracy on the color string generation version of the task (Section 4.3), meaning that low performance can be attributed to formatted MCQA.) This result highlights the value of a synthetic task: it helps to disentangle to what extent poor MCQA performance on a dataset is due to a lack of dataset-specific knowledge vs. an inability to perform formatted MCQA.

#### Poorly performing models cannot separate answer choice symbols in vocabulary space.

Figure: Figure 9: Accuracy of various Olmo v1.7 checkpoints. Plotted is the minimum performance between ‘‘A:_B:_’’ and ‘‘C:_D:_’’. Performance on Colors demonstrates that MCQA format is learned between 350B and 1T tokens. See Table 3 for results on all models.
Refer to caption: x3.png

We use vocabulary projection to inspect differences between these checkpoints on the Colors task. Fig. 22 (Appendix) illustrates the separability between the top and second-top answer choices, which grows as a function of training for more steps. Indeed, the primary distinction observed between the strong checkpoints (Figs. 22(b) and 22(c)) and the weak checkpoint (Fig. 22(a)) is the separability of ‘‘A’’ and ‘‘B’’, particularly on and after layer 29. This is in spite of the fact that the 350B checkpoint model predicts ‘‘B’’ as the final answer on 99.5% of instances (Table 3), and
may explain why small edits to network parameters can decrease label bias Li and Gao ().
Finally, we observe in Fig. 23 (Appendix) that Olmo v1.7 7B’s ability to do symbol binding as demonstrated on Colors does not alleviate the token separability problem on other datasets on which the model is less performant. Overall, these results demonstrate that an inability to disentangle answer symbol tokens in vocabulary space, *not* an inability to assign high probability to these tokens, is a property of poorly-performing models.

## 8 Conclusion

This work provides novel insights into the mechanisms language models employ when answering formatted multiple-choice questions. Our analysis leverages two complementary techniques: vocabulary projection and activation patching. Using carefully-designed prompts and a synthetic task, we uncover a series of stages, attibutable to a very sparse set of specialized network components, in which transformers select, promote, and align (in the case of OOD prompts) the answer choice symbol they predict. With MCQA datasets such as MMLU routinely used as a performance signal for model development, it is important to disentangle and test formatted MCQA ability in order to build models that operate in more robust and reliable ways. Our work makes an important first step towards this goal.

## Acknowledgments

We thank members of the Aristo team at AI2 and the H2Lab at the University of Washington for feedback and discussions.

## Limitations

Vocabulary projection provides a meaningful notion of how hidden states assign probabilities to tokens in the vocabulary space defined by the unembedding matrix, but it is not a causal intervention, and it cannot uncover ways in which hidden states could be working to promote tokens in other linear (or non-linearly decodable) subspaces. For these reasons, negative results in earlier layers are uninformative. We supplement our findings by using activation patching to localize effects at earlier layers.

Activation patching makes some assumptions about the independence of model components to make computation tractable (namely, that one can patch in individual hidden states in isolation to measure their effect on the network as opposed to patching all possible combinations of states). A nascent line of work focuses on making activation patching more efficient so that a larger number of states and/or state combinations can be intervened on Kramár et al. (); Syed et al. (); this is an interesting direction for future work. We additionally only patch on pairs of prompts where the model predicted both labels correctly; it could be interesting to compare behavior on consistently incorrect instances instead.

Finally, we primarily focus on a couple of performant small-to-medium-size models in this work, but expanding to other model families (and sizes) capable of formatted MCQA is an interesting direction for future work in order to understand the generality of our findings.

## Appendix A Appendix

### A.1 Plot Complexity w.r.t Number of Answer Choices

The extent to which a model component prefers ‘‘A’’ over other answer choices in a binary setup is equivalent to $p(\text{{``A''}})-p(\text{{``B''}})$; this quantity simultaneously illustrates (in the negative direction) the extent to which the component prefers ‘‘B’’. However, for 3 answer choices, say ‘‘A’’, ‘‘B’’, ‘‘C’’, 3 quantities are needed to measure preference for each choice: $p(\text{{``A''}})-\text{max}\{p(\text{{``B''}}),p(\text{{``C''}})\}$, $p(\text{{``B''}})-\text{max}\{p(\text{{``A''}}),p(\text{{``C''}})\}$, and $p(\text{{``C''}})-\text{max}\{p(\text{{``A''}}),p(\text{{``B''}})\}$.

### A.2 Dataset Details

HellaSwag (Zellers et al., ) is a 4-way multiple-choice commonsense natural language inference dataset. The goal is to select the best completion for a given sentence prompt.
The test set is not publicly available; we sample a fixed set of 5000 instances from the validation set used in our experiments (apart from activation patching, for which we sample 1000 instances due to compute restrictions). We use the first 3 training set instances as in-context examples for all instances.

MMLU Hendrycks et al. (), or the “Massive Multitask Language Understanding” benchmark, spans 57 different topical areas. The questions are 4-way multiple-choice spanning subjects in social sciences, STEM, and humanities that were manually scraped from practice materials available online for exams such as the GRE and the U.S. Medical Licensing Exam. We sample a fixed set of 5000 instances from the test set used in our experiments (apart from activation patching, for which we sample 1000 instances due to compute restrictions). We sample a fixed set of 3 in-context example instances from the 5 provided for each topical area.

We convert both tasks to binary multiple-choice following the procedure in Section 3.

### A.3 Transformer Background

The output hidden states of transformer models Vaswani et al. () are a linear combination of the outputs of non-linear functions (i.e., multi-head self-attention and multi-layer perceptrons) at each layer.
For the Olmo and Llama2 models, which sequentially apply multi-head self-attention (MHSA) and a multi-layer perceptron (MLP), for input hidden state $\mathbf{x}_{\ell-1}\in\mathbb{R}^{d}$,
the output hidden state $\mathbf{x}_{\ell}\in\mathbb{R}^{d}$ of layer $\ell$ is defined by the following equation:

$$ $\displaystyle\begin{split}\mathbf{x}_{\ell}&=\mathbf{x}_{\ell-1}+\text{MHSA}_{ \theta_{\ell}}\big{(}\text{LN}(\mathbf{x}_{\ell-1})\big{)}+\\ &\text{MLP}_{\theta_{\ell}}\Big{(}\mathbf{x}_{\ell-1}+\text{MHSA}_{\theta_{ \ell}}\big{(}\text{LN}(\mathbf{x}_{\ell-1})\big{)}\Big{)}\end{split}$ (1) $$

Where $\text{MHSA}_{\theta_{\ell}}$ and $\text{MLP}_{\theta_{\ell}}$ represent the multi-head self-attention operation and multi-layer perceptron, respectively, defined by their unique parameters at layer $\ell$, and LN represents layer normalization.(^10^1010$\text{MHSA}_{\theta_{\ell}}$ also receives as input the hidden states at layer $\ell-1$ from other token positions in the rollout (as dictated by the attention mask), which we omit here for brevity.)
Note that $+$ represents simple vector addition, which serves to establish residual connections. This is possible because the output of each operation (MHSA, MLP, and LN) is a vector $\in\mathbb{R}^{d}$. We visualize a single layer’s block structure in Fig. 10.

Elhage et al. () hypothesize and provide evidence for the “residual stream” view of Transformer inference, in which MHSA and MLP functions “read from” and “write to” the residual stream, which carries information through the layers. Additional evidence that MHSA and MLP functions promote specific tokens by writing to the residual stream is given by Geva et al. ().

Figure: Figure 10: Structure of a single layer in the Transformer architecture for the models we study (Olmo and Llama2). Green lines indicate residual connections. We perform vocabulary projection and activation patching on different representations both pre-and-post residual combination, indicated by the letters.
Refer to caption: x4.png

Given an input token embedding $\mathbf{x}_{0}\in\mathbb{R}^{d}$ and applying Eq. 1 recursively, the final hidden state of a Transformer with $L$ layers resolves to:

$$ $\displaystyle\begin{split}\mathbf{x}_{L}&=\mathbf{x}_{0}+\sum_{\ell=0}^{L-1} \bigg{[}\text{MHSA}_{\theta_{\ell+1}}\big{(}\text{LN}(\mathbf{x}_{\ell})\big{) }+\\ &\text{MLP}_{\theta_{\ell+1}}\Big{(}\mathbf{x}_{\ell}+\text{MHSA}_{\theta_{ \ell+1}}\big{(}\text{LN}(\mathbf{x}_{\ell})\big{)}\Big{)}\bigg{]}\end{split}$ (2) $$

The output of a MHSA function can be further broken down into a sum of the output of each attention head. For the input vector to layer $\ell$, $\mathbf{x}_{\ell-1}$, the output of MHSA is computed as the concatenation of each head’s vector output, $\text{Att}_{h}^{(\ell)}\big{(}\text{LN}(\mathbf{x}_{\ell-1})\big{)}\in\mathbb{
R}^{d/H}$, times an output weight matrix $W_{O}^{(\ell)}\in\mathbb{R}^{d\times d}$ which can be simplified into a sum as follows (originally elucidated in Elhage et al. ()):

$$ $\displaystyle\begin{split}\text{MHSA}_{\theta_{\ell}}\big{(}\text{LN}(\mathbf{ x}_{\ell-1})\big{)}&=\\ &\sum_{h=1}^{H}W_{O,h}^{(\ell)}\cdot\text{Att}_{h}^{(\ell)}\big{(}\text{LN}( \mathbf{x}_{\ell-1})\big{)}\\ \end{split}$ (3) $$

where $W_{O,h}^{(\ell)}\in\mathbb{R}^{d\times(d/H)}$ are the specific columns of $W_{O}^{(\ell)}$ corresponding to head $h$. When we perform experiments on individual attention heads, we are referring to the individual components of this sum (i.e., weighted attention head outputs).

**Table 1: HellaSwag (binary) accuracy by prompt and model. In the “Prompt” column, the correct answer for all instances in each setting is bolded. All averages >20% over random performance are bolded, and >15% italicized. Models that perform significantly better than random on the in-distribution (‘‘A:_B:_’’) prompt also do so on the more unnatural ‘‘C:_D:_’’ prompt variant.**
|  | Model |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Olmo-7b | Olmo-v1.7-7b | Llama2-7b | Llama2-13b |  |  |  |  |  |
| Prompt | Base | SFT | Base@350B | Base@1T | Base | Base | Chat | Base | Chat |
| ‘‘A:_B:_’’ | $0.8464$ | $0.8278$ | $0.851$ | $0.7$ | $0.9952$ | $0.9194$ | $0.5174$ | $0.7812$ | $0.7422$ |
| ‘‘A:_B:_’’ | $0.2192$ | $0.8396$ | $0.2034$ | $0.406$ | $0.059$ | $0.2686$ | $0.9284$ | $0.7592$ | $0.8698$ |
| Avg (‘‘A:_B:_’’) | $0.5328$ | \B$0.8337$ | $0.5272$ | $0.553$ | $0.5271$ | $0.594$ | \B$0.7229$ | \B$0.7702$ | \B$0.806$ |
| ‘‘C:_D:_’’ | $0.5184$ | $0.864$ | $0.7496$ | $0.9174$ | $0.992$ | $0.618$ | $0.721$ | $0.8092$ | $0.8212$ |
| ‘‘C:_D:_’’ | $0.5906$ | $0.8024$ | $0.2454$ | $0.1612$ | $0.1142$ | $0.637$ | $0.7978$ | $0.665$ | $0.7904$ |
| Avg (‘‘C:_D:_’’) | $0.5545$ | \B$0.8332$ | $0.4975$ | $0.5393$ | $0.5531$ | $0.6275$ | \B$0.7594$ | \B$0.7371$ | \B$0.8058$ |

**Table 2: MMLU (binary) accuracy by prompt and model. See caption of Table 1 for more details.**
|  | Model |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Olmo-7b | Olmo-v1.7-7b | Llama2-7b | Llama2-13b |  |  |  |  |  |
| Prompt | Base | SFT | Base@350B | Base@1T | Base | Base | Chat | Base | Chat |
| ‘‘A:_B:_’’ | $0.1872$ | $0.7246$ | $0.7128$ | $0.5994$ | $0.8796$ | $0.7834$ | $0.5842$ | $0.732$ | $0.7254$ |
| ‘‘A:_B:_’’ | $0.8754$ | $0.6882$ | $0.3178$ | $0.644$ | $0.4742$ | $0.5408$ | $0.7844$ | $0.7084$ | $0.7066$ |
| Avg (‘‘A:_B:_’’) | $0.5313$ | \B$0.7064$ | $0.5153$ | $0.6217$ | \I$0.6769$ | \I$0.6621$ | \I$0.6843$ | \B$0.7202$ | \B$0.716$ |
| ‘‘C:_D:_’’ | $0.2424$ | $0.7652$ | $0.8874$ | $0.8202$ | $0.8802$ | $0.6812$ | $0.7464$ | $0.8372$ | $0.7972$ |
| ‘‘C:_D:_’’ | $0.8278$ | $0.64$ | $0.1494$ | $0.3464$ | $0.4818$ | $0.6358$ | $0.6268$ | $0.5672$ | $0.6062$ |
| Avg (‘‘C:_D:_’’) | $0.5351$ | \B$0.7026$ | $0.5184$ | $0.5833$ | \I$0.681$ | \I$0.6585$ | \I$0.6866$ | \B$0.7022$ | \B$0.7017$ |

**Table 3: Colors accuracy by prompt and model. See caption of Table 1 for more details.**
|  | Model |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Olmo-7b | Olmo-v1.7-7b | Llama2-7b | Llama2-13b |  |  |  |  |  |
| Prompt | Base | SFT | Base@350B | Base@1T | Base | Base | Chat | Base | Chat |
| ‘‘A:_B:_’’ | $0.97$ | $1.0$ | $0.01$ | $1.0$ | $0.99$ | $1.0$ | $0.98$ | $1.0$ | $1.0$ |
| ‘‘A:_B:_’’ | $0.52$ | $0.99$ | $1.0$ | $0.98$ | $0.99$ | $0.97$ | $0.92$ | $1.0$ | $1.0$ |
| Avg (‘‘A:_B:_’’) | \B$0.75$ | \B$1.0$ | $0.51$ | \B$0.99$ | \B$0.99$ | \B$0.99$ | \B$0.95$ | \B$1.0$ | \B$1.0$ |
| ‘‘C:_D:_’’ | $0.93$ | $1.0$ | $0.91$ | $0.8$ | $0.94$ | $0.98$ | $0.98$ | $1.0$ | $1.0$ |
| ‘‘C:_D:_’’ | $0.54$ | $1.0$ | $0.24$ | $1.0$ | $1.0$ | $1.0$ | $0.97$ | $1.0$ | $0.97$ |
| Avg (‘‘C:_D:_’’) | \B$0.74$ | \B$1.0$ | $0.58$ | \B$0.9$ | \B$0.97$ | \B$0.99$ | \B$0.98$ | \B$1.0$ | \B$0.99$ |

**Table 4: % pairs of predictions falling into each consistency category for HellaSwag. We bold the largest category for each model/prompt combination. Recall that a high percentage of consistent predictions indicates models are performing multiple-choice symbol binding rather that using label or positional bias to select labels.**
|  |  | Model |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Olmo-7b | Olmo-v1.7-7b | Llama2-7b | Llama2-13b |  |  |  |  |  |
| Prompt | Statistic | Base | SFT | Base@350B | Base@1T | Base | Base | Chat | Base | Chat |
| ‘‘A:_B:_’’ | Consistently Correct | $0.157$ | \B$0.7576$ | $0.129$ | $0.197$ | $0.0586$ | $0.2564$ | \B$0.5108$ | \B$0.6564$ | \B$0.7032$ |
| vs. ‘‘A:_B:_’’ | Consistently Incorrect | $0.0914$ | $0.0902$ | $0.0746$ | $0.091$ | $0.0044$ | $0.0684$ | $0.065$ | $0.116$ | $0.0912$ |
|  | Incons. Correct (Both ‘‘A’’) | \B$0.6894$ | $0.0702$ | \B$0.722$ | \B$0.503$ | \B$0.9366$ | \B$0.663$ | $0.0066$ | $0.1248$ | $0.039$ |
|  | Incons. Correct (Both ‘‘B’’) | $0.0622$ | $0.082$ | $0.0744$ | $0.209$ | $0.0004$ | $0.0122$ | $0.4176$ | $0.1028$ | $0.1666$ |
| ‘‘C:_D:_’’ | Consistently Correct | $0.2434$ | \B$0.755$ | $0.1068$ | $0.133$ | $0.114$ | \B$0.4124$ | \B$0.645$ | \B$0.5608$ | \B$0.7064$ |
| vs. ‘‘C:_D:_’’ | Consistently Incorrect | $0.1344$ | $0.0886$ | $0.1118$ | $0.0544$ | $0.0078$ | $0.1574$ | $0.1262$ | $0.0866$ | $0.0948$ |
|  | Incons. Correct (Both ‘‘C’’) | $0.275$ | $0.109$ | \B$0.6428$ | \B$0.7844$ | \B$0.878$ | $0.2056$ | $0.076$ | $0.2484$ | $0.1148$ |
|  | Incons. Correct (Both ‘‘D’’) | \B$0.3472$ | $0.0474$ | $0.1386$ | $0.0282$ | $0.0002$ | $0.2246$ | $0.1528$ | $0.1042$ | $0.084$ |

**Table 5: % pairs of predictions falling into each consistency category for MMLU. See caption of Table 4 for more info.**
|  |  | Model |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Olmo-7b | Olmo-v1.7-7b | Llama2-7b | Llama2-13b |  |  |  |  |  |
| Prompt | Statistic | Base | SFT | Base@350B | Base@1T | Base | Base | Chat | Base | Chat |
| ‘‘A:_B:_’’ | Consistently Correct | $0.1146$ | \B$0.4484$ | $0.0856$ | \B$0.3476$ | \B$0.576$ | \B$0.4486$ | \B$0.5044$ | \B$0.5644$ | \B$0.5778$ |
| vs. ‘‘A:_B:_’’ | Consistently Incorrect | $0.052$ | $0.0946$ | $0.055$ | $0.1042$ | $0.1632$ | $0.1244$ | $0.1358$ | $0.124$ | $0.1458$ |
|  | Incons. Correct (Both ‘‘A’’) | $0.0726$ | $0.4312$ | \B$0.6272$ | $0.2518$ | $0.1486$ | $0.3348$ | $0.0798$ | $0.1676$ | $0.1476$ |
|  | Incons. Correct (Both ‘‘B’’) | \B$0.7608$ | $0.0258$ | $0.2322$ | $0.2964$ | $0.1122$ | $0.0922$ | $0.28$ | $0.144$ | $0.1288$ |
| ‘‘C:_D:_’’ | Consistently Correct | $0.12$ | \B$0.4522$ | $0.0598$ | $0.2362$ | \B$0.5642$ | \B$0.4334$ | \B$0.5178$ | \B$0.51$ | \B$0.5282$ |
| vs. ‘‘C:_D:_’’ | Consistently Incorrect | $0.0498$ | $0.0902$ | $0.023$ | $0.0696$ | $0.159$ | $0.1164$ | $0.1446$ | $0.1056$ | $0.1248$ |
|  | Incons. Correct (Both ‘‘C’’) | $0.1224$ | $0.428$ | \B$0.8276$ | \B$0.584$ | $0.201$ | $0.2478$ | $0.2286$ | $0.3272$ | $0.269$ |
|  | Incons. Correct (Both ‘‘D’’) | \B$0.7078$ | $0.0296$ | $0.0896$ | $0.1102$ | $0.0758$ | $0.2024$ | $0.109$ | $0.0572$ | $0.078$ |

Figure: (a) ‘‘A:_B:_’’ & model correct ($\hat{y}=\text{{``A''}}$)
Refer to caption: extracted/5743777/figures_png/hellaswag/olmo-7b-sft_hellaswag_coarse_3ICE_ABBlabels_Acorrect_correct_only_prob.png

Figure: (a) ‘‘A:_B:_’’ $\rightarrow$‘‘A:_B:_’’; Model correct on both.
Refer to caption: extracted/5743777/figures_png/causal_tracing/olmo-7b-sft_hellaswag_layer_standard_to_inverse_3ICE_ABBlabels_1000insts_consistent_correct.png

Figure: (a) ‘‘A:_B:_’’ & model correct ($\hat{y}=\text{{``A''}}$)
Refer to caption: extracted/5743777/figures_png/hellaswag/llama2-13b-chat_hellaswag_coarse_3ICE_ABBlabels_Acorrect_correct_only_prob.png

Figure: (a) Colors
Refer to caption: extracted/5743777/figures_png/colors/llama2-13b-chat_prototypical_colors_coarse_3ICE_ABBlabels_Acorrect_all_correct_prob_new.png

Figure: (a) ‘‘A:_B:_’’ $\rightarrow$‘‘A:_B:_’’ (HellaSwag)
Refer to caption: extracted/5743777/figures_png/causal_tracing/olmo-7b-sft_hellaswag_layer_inverse_to_standard_3ICE_ABBlabels_1000insts_consistent_correct_new.png

Figure: (a) ‘‘A:_B:_’’ $\rightarrow$‘‘B:_A:_’’ (HellaSwag)
Refer to caption: extracted/5743777/figures_png/causal_tracing/llama2-13b-chat_hellaswag_layer_standard_to_inverse_3ICE_ABBlabels_1000insts_consistent_correct_new.png

Figure: (a) ‘‘A:_B:_’’
Refer to caption: extracted/5743777/figures_png/hellaswag/llama2-13b-chat_hellaswag_coarse_3ICE_ABBlabels_Acorrect_correct_only_prob_new.png

Figure: (a) MHSA: ‘‘A:_B:_’’ $\rightarrow$‘‘C:_D:_’’
Refer to caption: extracted/5743777/figures_png/causal_tracing/olmo-7b-sft_hellaswag_attn_cd_label_mapping_to_standard_3ICE_ABBlabels_1000insts_consistent_correct.png

Figure: (a) MHSA: ‘‘A:_B:_’’ $\rightarrow$‘‘C:_D:_’’
Refer to caption: extracted/5743777/figures_png/causal_tracing/llama2-13b-chat_hellaswag_attn_cd_label_mapping_to_standard_3ICE_ABBlabels_1000insts_consistent_correct.png

Figure: (a) MHSA
Refer to caption: extracted/5743777/figures_png/hellaswag/llama2-13b-chat_hellaswag_finegrained_attnOutput_3ICE_ABBlabels_Acorrect_all_correct_prob.png

Figure: (a) $p(\text{{``A''}})+p(\text{{``B''}})$ for ‘‘A:_B:_’’
Refer to caption: extracted/5743777/figures_png/hellaswag/llama2-13b-chat_hellaswag_finegrained_attnHeads_3ICE_ABBlabels_Acorrect_all_correct_deepmind_prob_barGraph_new_layer0_sum.png

Figure: (a) $350B$ tokens ($51$% acc.)
Refer to caption: extracted/5743777/figures_png/colors/olmo-v1.7-7b-350B_prototypical_colors_coarse_3ICE_ABBlabels_Acorrect_all_10k_prob_new.png

Figure: (a) Colors ($99$% acc.)
Refer to caption: extracted/5743777/figures_png/colors/olmo-v1.7-7b_prototypical_colors_coarse_3ICE_ABBlabels_Acorrect_all_10k_prob_new.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. In sec.3.4, why the threshold on 20%+ random (i.e. 45%) is chosen? Is there any reason behind this choice?

2. Evalusting 3-shot accuracy with alternative symbols (e.g. QXYZ), which symbols are used for in-context examples?

3. Do you have any ideas which of the observed phenomena can be extended to other tasks, beyond MCQA?
soundness: 4
strengths: + The paper is clearly written. All the claims of the paper are supported by experiments, and the results are sound
+ The claimed properties are consisstently observed over datasets and model's families
+ The observed results are interesting and helps to understand the internal mechanics of LLMs
+ Althougt the methods used in the paper are not novel, the quality of the presented analysis is quite high. I especially appreciate the models's robustness analysis and the difference between probits and logits
summary: The paper is devoted to the analysis of the internal processing of Transformer LLMs when answering Multiple Choice Question Answering task. First, it is shown that the ability to answer such question can be attributed to a few layers in the model. Second, this ability is implemented by a sparse set of attention heads. Third, this ability emerges at some point during model's training. Finally, an interesting observation is that in case of on-standard label symbols (e.g. QXYZ instead of ABCD) the model first tries to predict typical characters corresponding the answer position, which are replaced with the correct character in higher layers. 

The analysis is performed on three MCQA dataset: MMLU, HellaSwag, and synthetic Colors dataset. As for models, three families are considered: LlaMa, Qwen and Olmo. For deep analysys, the models with consistent results (i.e. robust to answer order and labels perturbation) on all the three datasets are chosen
weaknesses: - The range of considered model sizes is limited (0.5B-7B)
- Although the observations presented in the paper are clear and consistent, they don't provide any real understanding how the model works. Besides, it is limited to Multiple Choic eQuestion Answering tasks, and there is no attempts to extend it to more general understanding of the inner Transformer's mechanics.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - Could you please specify what values are presented by Fig. 8? Is taken from logit lens applied to MSHA components by heads? 
- What conclusion do you make from the fact that in Fig. 7c (MHSA) change in answers are visible only in 24 layer, while in Fig.3 it maintains further. It will be interesting to describe in detail that mechanism.
- It is relatively small issue, but do you consider using another variant of logit lens? The reason is that logit lens by itself could be potentially worse for earlier layers [2] 


Small comments:
- Fig.8 is not aligned 

[2] Belrose, Nora, et al. "Eliciting latent predictions from transformers with the tuned lens." (2023).
soundness: 3
strengths: - Evaluating across various prompt formats and permutations is really convincing, it really highlights this paper
- Interesting analysis that includes both MHSA and MLP patching
- Good visualization of when the MCQA understanding forms during training
summary: The study investigates how LLMs solve multiple-choice question-answering (MCQA) tasks, focusing on intrinsic interpretability. The authors explore how the specific layers and attention heads contribute to selecting the correct answer from a set of choices. By applying methods like vocabulary projection and activation patching, the study reveals that particular middle transformer layers play a critical role in aligning the model’s predictions with the correct answer symbol. These components act selectively to bind the answer symbol (e.g., A, B, C, D) to the correct answer phrase. The paper provides a comprehensive pre-analysis of multiple models, including Olmo, Llama, and Qwen, across various prompt formats and datasets to select the models that understand the MCQA task. Additionally, it introduces a synthetic MCQA task to isolate MCQA capabilities from dataset-specific challenges. Key findings indicate that robust MCQA performance relies on both specific layers and sparse attention heads that drive answer selection, with some layers adapting to unusual symbol choices in later stages. This work offers a deeper understanding of model-specific MCQA mechanisms.
weaknesses: - The motivation for selecting the patching format is not fully substantiated. By "patching format," I refer to the reordering of the correct answer to a new position. Although this approach is intuitively clear, I believe the motivation could be strengthened by exploring different patching formats (for example, removing correct answer from sample).
- The circuits are shown to be dependent on the nodes (parts of model) for patching [1]. Patching the layers is interesting, but for full picture it would be good to examine the attention maps by patching. 

[1] Miller, Joseph, Bilal Chughtai, and William Saunders. "Transformer Circuit Evaluation Metrics Are Not Robust." First Conference on Language Modeling. 2024.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Fig.9: Do you have such curve for any other model, aside of Olmo 0724 7B base?
- Fig.4: I don't understand the purpose of making this figure. What should it show to us?
- Fig.8: Can you, please, elaborate the captions a), b), c)?
- In the most of your experiments, you only consider the models from the range of 1.5-8B (smaller models turned out to be too weak to make good MCQA predictions). Do you have a physical possibility to do at least some experiments with some bigger models (at least, 13B)? It would be especially interesting to see the effect of switching from A/B/C/D to Q/Z/R/X (and other non-standard letters/symbols sets) at such model.
soundness: 3
strengths: - The paper analyses an important topic, since MCQA tasks are very common in LLM benchmarks.
- Authors came up with a good idea for their synthetic dataset. That dataset allows them to distinguish a model's understanding of MCQA format from it's knowledge in particular area, that is important for proper analysis.
- A number of experiments are done on several models: Olmo, LLaMA 3.1, Qwen: -chat and -base versions. These models look like an appropriate objects for this study.
- At the second half of their paper, authors especially concentrate on those models that are good in solving three MCQA tasks (synthetic, HellaSwag and MMLU). It is also a good move, because it allows to ignore "noisy" properties of weak models that could mislead researcher and reader otherwise.
- Authors use a good practices in MCQA research, i.e. they permute A/B/C/D symbols and consider alternative symbols for answer options.
summary: This research work studies how LLMs solve MCQA task. Authors show which layers and layer components play the most crucial role in this process and at which point the OLMO model starts to learn to solve this task. They also find out how different is this process when the options are named by other symbols (not A,B,C,D).
weaknesses: - The claim in the lines 518-519 (_"these results demonstrate that an inability to disentangle answer symbol tokens in vocabulary space is a property of poorly-performing models"_) is not sufficiently supported. Such claim definitely cannot be deduced from the analysis of the checkpoints of only one particular Olmo 0724 7B base model (however, at lines 79-81 you make much more humble version of this claim, with which I agree).
- In the lines 76-78 you say: _"We discover that the model’s hidden states initially assign high logit values to expected answer symbols (here, A/B/C/D) before switching to the symbols given in the prompt (here, the random letters Q/Z/R/X)."_, but this claim is also not sufficiently supported. You only showed this effect for one non-standard letter set, and, again, I see only Olmo 7B experiments here (please, correct me if I'm wrong).
- Some parts of the paper are poorly-written and unclear. E.g. I don't understand the caption for the Figure 1, especially the sentence _"Finally, when we switch to more unusual answer choice symbols, one behavior by which models adjust is to initially operate in the space of more standard tokens, such as A/B/C/D, before aligning probability to the correct symbols at a late layer"_. See also "Question" sections for the questions about other Figures.
- There are following minor problems:
-- Text in the diagrams is extremely small. Please, make it larger to make it readable.
-- The paper contains typos such as word "projectioan" on line 066. Please, check the grammar of your text.
-- Citations in the line 264 should be put inside the braces.

**UPDATE**: Authors addressed the second major weakness by adding the results for other letter sets and other models. Due to this, **I raised my main score from 5 to 6** and "Soundness" score from 2 to 3. They also answered the questions, made their points more clear and addressed minor problems in the text of the paper. Due to this I raised "Presentation" score from 2 to 3.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: * As mentioned in "Weaknesses" I'd be interested to hear the reason for using 0-shot accuracy in Figure 9.
soundness: 4
strengths: This paper seems quite original. It is well contextualized with respect to prior work, and has novel contributions. The authors' claims are clear and are backed by broad and consistent experimental results. The interpretability findings may be of interest to researchers working in the space of MCQA and LLMs, or in the space of model interpretability in general.
summary: The authors study the mechanisms that exist in LLMs that enable them to successfully answer multiple choice questions. They primarily focus on three models from different model families, chosen for their well-above-average accuracy across answer orderings and datasets. They use in their study three datasets: MMLU, Hellaswag, and a synthetic dataset ("Colors"). Mechanism study is done with activation patching and vocabulary projection. Among the authors' key findings are:
* There is a specific layer or layers that are responsible for promoting the letter to be chosen.
* Letter selection is handled by the multi-head self-attention part of the model, and in particular by a small proportion of total heads.
* When unusual symbol groups (e.g., Q/Z/R/X) are used, the model first promotes A/B/C/D and then abruptly changes to promoting the unusual symbol group.
* Models that don't perform well on multiple choice cannot effectively separate the answer letters in vocabulary space.
weaknesses: * I'm surprised that the OLMo 0724 7B Instruct (Figure 9) study used 0-shot accuracy. In general, the authors use 3-shot accuracy in the main body of the paper, which seems prudent as the 0-shot case is somewhat ambiguous (it's unclear whether the correct response is answer label or answer text). It would be interesting to see 3-shot results for the study or to hear a justification for use of 0-shot accuracy.
* The findings in this paper rely on some assumptions inherent in activation patching and vocabulary projection. For example, that it's reasonable to project hidden layers from early in the network to the vocabulary space. I didn't take this potential weakness into account in my rating of the paper, as I'm not intimately acquainted with very recent mechanistic interpretability work. But it is something the authors could address in the paper if desired.

Meta-review: This paper investigates how LLMs solve Multiple Choice Question Answering (MCQA) tasks through mechanistic interpretability analysis. The key findings show that specific middle transformer layers and sparse attention heads play critical roles in selecting correct answers, with some models using a two-stage process when handling non-standard answer symbols (e.g., Q/Z/R/X instead of A/B/C/D). The paper's main strengths lie in its comprehensive evaluation across multiple models (Olmo, Llama, Qwen) and datasets (MMLU, HellaSwag, Colors), robust experimental design including permutation tests, and clear visualization of results. While the work is limited to relatively small models (0.5B-7B parameters) and focused specifically on MCQA rather than broader model mechanics, the thorough analysis and consistent findings across different models make this a valuable contribution to understanding LLM behavior. The paper merits acceptance due to its sound methodology, clear presentation, and important insights into how transformer models process structured multiple choice tasks.

Decision: Accept (Spotlight)