SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use the logit lens throughout this paper for consistency, both internally and with prior work. However, the logit lens has flaws: in reality, the residual stream is normalized using layer normalization prior to being transformed into the logits. This nonlinearity could in theory cause the scaling of logit lens to be misleading, perhaps in such a way that affects our conclusions. In this section, we will show that the logit lens results can be recreated with another technique, direct effects via path patching, that avoids this flaw. The two techniques fortunately yield the same insights.
Using the logit lens, we would normally multiply our component of interest's output by the unembedding matrix. To measure direct effects, we take the unpatched, original model's year logits, over each YY, as a baseline. We then patch the direct path between our component and the logits with the 01-dataset, and again record the logits. The difference between the unpatched logits, and the logits when we patch (ablate) our component of interest, reveals the direct effect that said component had. This approach eliminates one major concern of the logit lens: it yields the difference of two sets of logits, which were both produced with layer normalization, and which has interpretable units.
We can perform our neuron-level logit lens experiments by instead patching the direct paths to the top-3 neurons of MLP 10 (Figure 22). The results are essentially identical to the logit lens results, though they differ in magnitude. We can also sum these direct effects, as we summed the logit lens outputs; again, results differ from those of the logit lens only in magnitude (Figure 23). Finally, we can also patch all top-10 neurons as a group, and view their direct effects; these results are identical to those of the summed direct effects (Figure 23). This suggests that our summed logit lens approach genuinely reflected the direct effects that these neurons have on the logits.
We conclude that the results given by the logit lens and those given by direct effects are largely similar, so concerns about the logit lens are not dire. However, we note that all of our logit lens results (including those for entire MLPs and attention heads) are reproducible using direct effects.
Figure 24: Probability heatmaps for (left to right) "1799, 1753, 1733, 1701, 16YY, 16", "1695, 1697, 1699, 1701, 1703, 17", and "17YY is smaller than 17".
Figure 25: Probability heatmaps for (left to right) "The <noun> started in the year 17YY and ended in the year 17", "The <noun> happened in 17YY. Some years later, it is now the year 17", and "1599, 1607, 1633, 1679, 17YY, 17".
F Year-Span Circuit Generalization
In this section we provide evidence for the generalization of the year-span circuit to some (but not all) tasks. For tasks where the model failed entirely, we provide the probability heatmaps (to show its failure). For tasks where the model succeeded, or incorrectly generalized the greater-than circuit, we additionally provide path patching results and logit lens heatmaps to show how circuit structure and semantics are preserved.
Tasks Failed Figure 24 displays probability heatmaps for "1799, 1753, 1733, 1701, 16YY, 16", "1695, 1697, 1699, 1701, 1703, 17", and "17YY is smaller than 17". In the first case, GPT-2 predicts a roughly uniform distribution around YY. In the second case, the right answer varies: though the penultimate number in the provided sequence is  $\mathrm{YY} =$  "03", and the sequence increases by 2 step, depending on YY, we must vary the increase per step, in order to avoid the single-token number "1700". In any case, the model fails to predict the correct answer, often predicting  $\mathrm{YY} + 1$ , although the step is always  $> 1$ . In the final case, the model always outputs YY.
Tasks Completed Correctly Figure 25 displays probability heatmaps for “The <noun> started in the year 17YY and ended in the year 17”, “The price of that <luxury good> ranges from $ 17YY to $ 17”, and “1599, 1607, 1633, 1679, 17YY, 17”. All tasks are completed successfully, just like year-span prediction, though note that the second task is completed less well, as is visible in its heatmap. Its probability difference is only  $75\%$ , as opposed to  $90\%$ .
Given this, we proceed using iterative path patching direct logits connections as before; Figure 26 shows the results. All of these plots look almost identical to our original plots, so we evaluate the circuit on each of tasks using the methodology from Section 3.2. This works for the first two tasks, with performance recoveries  $>90\%$ , but not the last.
For the last task, we observe that MLP 8 relies also on MLP 7, which in turn relies on two extra attention heads not observed in our original circuit: a7.h11 and a6.h1 (Figure 27). Accounting for this in our circuit leads us back to  $>90\%$  loss recovery.
Figure 26: Iterative path patching plots (C, logits) for (left to right) "The <noun> started in the year 17YY and ended in the year 17", "The price of that <luxury good> ranges from $ 17YY to $ 17", and "1599, 1607, 1633, 1679, 17YY, 17".
Figure 27: Iterative path patching plots for "1599, 1607, 1633, 1679, 17YY, 17", searching for components that influence the circuit via MLPs 8 (left) and 7 (right).
Tasks Completed Incorrectly Finally, we address the tasks "The <noun> ended in the year 17YY and started in the year 17" and "The <noun> lasted from the year 7YY BC to the year 7", which do use our circuit, but should not do so.
Figure 28: Probability heatmaps for "The <noun> ended in the year 17YY and started in the year 17" (left) and "The <noun> lasted from the year 7YY BC to the year 7" (right).
Figure 29: Iterative path patching plots  $(C, \logit)$  for "The <noun> ended in the year 17YY and started in the year 17" (left) and "The <noun> lasted from the year 7YY BC to the year 7" (right).
Figure 28 displays probability heatmaps for "The <noun> ended in the year 17YY and started in the year 17" and "The <noun> lasted from the year 7YY BC to the year 7". Both tasks are completed successfully, though note that the latter task is completed less well.
As with the other tasks using this circuit, the iterative path patching plots (Figure 29) look similar to those of year-span prediction. Note that since the goal for these tasks is to produce "less-than", patching and impeding circuit components improves task performance (indicated in blue rather than red), since the circuit performs "greater-than". We evaluate the circuit on each of tasks using the methodology from Section 3.2. This works for both tasks, with performance recoveries  $>90\%$ .
G Noun Pool for Templated Sentences
We use the following nouns in our main template: abduction, accord, affair, agreement, appraisal, assaults, assessment, attack, attempts, campaign, captivity, case, challenge, chaos, clash, collaboration, coma, competition, confrontation, consequence, conspiracy, construction, consultation, contact, contract, convention, cooperation, custody, deal, decline, decrease, demonstrations, development, disagreement, disorder, dispute, domination, dynasty, effect, effort, employment, endeavor, engagement, epidemic, evaluation, exchange, existence, expansion, expedition, experiments, fall, fame, flights, friendship, growth, hardship, hostility, illness, impact, imprisonment, improvement, incarceration, increase, insurgency, invasion, investigation, journey, kingdom, marriage, modernization, negotiation, notoriety, obstruction, operation, order, outbreak, outcome, overhaul, patrols, pilgrimage, plague, plan, practice, process, program, progress, project, pursuit, quest, raids, reforms, reign, relationship, retaliation, riot, rise, rivalry, romance, rule, sanctions, shift, siege, slump, stature, stint, strikes, study, test, testing, tests, therapy, tour, tradition, treaty, trial, trip, unemployment, voyage, warfare, work.
For the <luxury good> noun considered in the generalization section, we use a smaller pool of nouns: gem, necklace, watch, ring, suitcase, scarf, suit, shirt, sweater, dress, fridge, TV, bed, bike, lamp, table, chair, painting, sculpture, plant.
H Applying the Circuits Approach to Other Problems
In order to find a circuit for a given model ability, one must define a task, dataset, and metric; one can then apply the path patching approaches we use. However, task, dataset, and metric design are important; not just any task, dataset, or metric will be compatible. In this section, we discuss the qualities that a task, dataset, and metric should have, in order to be used with our approach.
Task The path-patching approach is compatible with a variety of tasks. The chosen task should:
- Have a clearly delimited set of correct and incorrect answers for each example.
- Require only one forward pass of the model (as opposed to e.g. generation tasks which require multiple passes).
- Be solvable by your model: if your model cannot solve the task, there may be no circuit. At minimum, the model should exhibit consistent behavior (even if it's not exactly correct)
The granularity of insights will depend on the granularity of the task chosen. Complex tasks like natural language inference could require different (sub-)circuits depending on the specific question; it might be hard to find one precise circuit responsible for the task. Smaller, simpler tasks will likely yield easier to interpret results.
For the purpose of this example, we consider the task of fact retrieval, much akin to the task of Meng et al. [28]. Each input will have an (ideally single-token) correct answer, which can be predicted with one forward pass. Moreover, it seems possible that facts are mostly stored and retrieved using the same circuit.
Dataset Path-patching requires two datasets: a normal and corrupted dataset. The normal dataset is just a collection of examples/Input for the task; its examples should:
- Clearly indicate the task at hand. LMs perform language modeling, and do not natively perform other tasks; they may leak probability to answers that are not correct or incorrect but simply task-irrelevant. The inputs to the LM should push as much probability as possible onto the task's output space.
- Allow evaluation based on only the distribution over possible next tokens (generated via one forward pass)
- Be representative of your task. Your choice of datasets effectively defines the scope of the phenomenon you study, so it is essential that the scopes of the dataset and the intended task match!
Each example from the normal dataset should have a corresponding corrupted example / input. The corrupted input should:
- form a minimal pair with the normal input: they should differ minimally from each other (being the same length, and differing by only one or two tokens)
- elicit a different model response, with a distinct correct answer, compared to the normal input
- belong to the same sort of task. We locate the circuit by activating the same circuit with two different inputs.
For fact retrieval, a normal input could be "Paris is the capital of"; the corrupted counterpart could be "Rome is the capital of". Both of these examples are reasonable input for fact retrieval, but the two will elicit very different responses. Note that an input like "Paris is in" would be less appropriate, because it doesn't clearly indicate that the task is fact retrieval, or what fact should be retrieved.
Metric The metric is a function that takes in model logits and labels. It should:
Output a real number measuring model behavior/performance on the task.
- Detect small changes in whether the model is behaving according to the normal or corrupted input, or somewhere in between. A continuous loss is thus preferable to metrics like 0-1 loss/accuracy.
One family of metrics used in previous work is the probability assigned to the correct answer(s), minus the probability of the incorrect answer(s) induced by the corrupted input. In the greater-than case, this is  $p(y > \mathrm{YY}) - p(y \leq \mathrm{YY})$ . For fact retrieval, we would compute  $p(\mathrm{France}) - p(\mathrm{Italy})$ . This family of metrics the model is implicitly sensitive to the model generating off-task output as this generally takes away from the probability of correct answers. It is explicitly sensitive to the probability of the corrupted input's answer.
Other metrics are possible. For example, if it is difficult to quantify task performance, but still possible to create minimal pairs, one could simply measure the KL-divergence between the original
and (partially) patched / corrupted distributions. However, this is harder to interpret, and less targeted at your actual task of interest.
I Computational Resources
All experiments were performed on an Nvidia A100 GPU. The path patching experiments and circuit semantics experiments take no longer than an hour to complete. The generalization experiments take a similar amount of time, being very similar to the original experiments. The neuron-level experiments (in particular finding top neurons) can take multiple hours to run. Overall, the final experiments can be run in less than 24 hours.