The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks
Authors: Ziqian Zhong
Ziming Liu
Max Tegmark
Jacob Andreas
Venue: NeurIPS 2023 oral (2023)
Keywords: mechanistic interpretability
algorithmic phase transitions
arithmetic learning
neural network
transformer
ensemble
TL;DR: We find that neural networks do not always rediscover known algorithms (Clock), but also discover new ones (Pizza), with modular addition as a prototypical example.

Abstract: Do neural networks, trained on well-understood algorithmic tasks, reliably rediscover known algorithms? Several recent studies, on tasks ranging from group operations to in-context linear regression, have suggested that the answer is yes. Using modular addition as a prototypical problem, we show that algorithm discovery in neural networks is sometimes more complex: small changes to model hyperparameters and initializations can induce discovery of qualitatively different algorithms from a fixed training set, and even learning of multiple different solutions in parallel. In modular addition, we specifically show that models learn a known *Clock* algorithm, a previously undescribed, less intuitive, but comprehensible procedure we term the *Pizza* algorithm, and a variety of even more complex procedures. Our results show that even simple learning problems can admit a surprising diversity of solutions, motivating the development of new tools for mechanistically characterizing the behavior of neural networks across the algorithmic phase space.

The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks
Ziqian Zhong*, Ziming Liu", Max Tegmark, Jacob Andreas Massachusetts Institute of Technology {ziqianz, zmliu, tegmark, jda)@mit. edu
Abstract
Do neural networks, trained on well-understood algorithmic tasks, reliably re- discover known algorithms for solving those tasks? Several recent studies, on tasks ranging from group arithmetic to in-context linear regression, have suggested that the answer is yes. Using modular addition as prototypical problem, we show that algorithm discovery in neural networks is sometimes more complex. Small changes to model hyperparameters and initializations can induce discovery of qualitatively different algorithms from a fixed training set, and even parallel implementations of multiple such algorithms. Some networks trained perform modular addition implement familiar Clock algorithm (previously described by Nanda al. [T); others implement previously undescribed. less intuitive, but comprehensible procedure we term the Pizza algorithm, or a variety of even more complex procedures. Our results show that even simple learning problems can admit a surprising diversity of solutions, motivating the development of new tools for characterizing the behavior of neural networks across their algorithmic phase space.
Introduction
Mechanistically understanding deep network models-rees reverse-nngineering their learned algorithms a major challenge across problem domains. Several recent and representation have exhibited specific examples of models apparently re-discovering inter- studies [2] 3 [ I
pretable (and some cases familiar) solutions to tasks like curve detection, sequence copying and modular arithmetic. Are these models the exception or the rule? Under what conditions do neural network models discover familiar algorithmic solutions to algorithmic tasks?
In this paper, we focus specifically on the problem of learning modular addition. training networks to compute sums like 8 8 geometric solution, familiar to anyone who has learned to read clock: every integer is represented as an angle, input angles are added together, and the resulting angle evaluated to obtain a modular sum (Figure I] left). Nanda et al. [ show that specific neural network architectures, when trained to perform modular addition, implement this Clock algorithm. In this work, we show that the Clock algorithm is only one part of more complicated picture of algorithm learning in deep networks. In particular, networks structurally similar to the ones trained by Nanda et preferentially implement qualitatively different approach to modular arithmetic, which we term the Pizzu algorithm (Figure right), and sometimes even more complex solutions. Models exhibit sharp algorithmic phase transitions (6] between the Clock and Pizza algorithms as their width and attention strength very, and often implement multiple, imperfect copies of the Pizza algorithm in parallel.
(mod 12). Modular arithmetic can be implemented with a simple
Equall contribution.
Code is available at https//giitbb.com//jjzz222221/11
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

circle where wy Z0ktp 11.2-pp11 p11 Step (ooseyya iitoyaliid (E.,. JonninyAssiiieeee
Embed
E,
Clock Algorithm
Pizza Algorithm
MLP or
Ba
compute using Iuttonlcation
Step 2.1 the vector
11
pjo 89) Lakateele --1)
(E, K,Y2 - (ontrrul e sole,,i.sittmul isteo00/2
R.A..
Stop 2.2: using Eu and nominearties compute R,
Unembwd
C
coooggi jontenn
Logt
Step 3: scare possible outputs using a dot product. 2x U, Ha.. U, m (K,,, R. (oosos.)).s shilecl)
Qu ondtage -877) coctleo +3-+j)
10
10
bane petoriaa
collupe to namber
100% 5 snco 10)
Figure 1: Illustration of the Clock and the Pizza Algorithm.
Our results highlight the complexity of mechanistic description in even models trained to perform simple tasks. They point to characterization of algorithmic phase spaces, not just single algorithmic solutions, as an important goal in algorithm- leveel interpretability.
Organization In Section we review the Clock algorithm [1] and show empirical evidence of deviation from deviations can be explained by an alternative Pizza algorithm. In Section 4
models trained to perform modular addition. Section 3 we show that these we define additional
metrics to distinguish between these algorithms, and detect phase transitions between these algorithms (and others Non-circular algorithms) when architectures and hyperparameters are varied. We discuss the relationship between these findings and other work on model interpretation in Section 5 and conclude in Section 6
2 Modular Arithmetic and the Clock Algorithm
Setup We train neural networks to perform modular addition a + = c (mod p), where a, b, 0, .P- 1. We use p 59 throughout the paper. these networks, every integer has an associated embedding vector E, Rdd. Networks take as input embeddings [E., Ey] Rdd and predict a categorical output c. Both embeddings and network parameters are learned. preliminary experiments, we train two different network architectures on the modular arithmetic task, which we refer to as: Model A and Model B. Model A a one-layer ReLU transformer 77] with constant attention, while Model B is standard one-layer ReLU transformer (see Appendix F. for details). As attention is not involved in Model A, it can also be understood as a ReLU MLP (Appendix
2.1 Review of the Clock Algorithm
As in past work, find that after training both Model A and Model B, embeddings (E., E, in Figure [ usually describe a circle (8] in the plane spanned by the first two principal components of the embedding matrix. Formally, E, (cos(waa), sinwwaa) where we 2.rkpp. 1 is an integer in [1, p - 1). Nanda et al. I discovered a circuit that uses these circular embeddings to implement an interpretable algorithm for modular arithmetic, which we call the Clock algorithm.
Algorithm
Learned Embeddings
Gradient Symmetry Required Non- Nin-line Multiplication Absolute value
Clock
Cirele
No
Pizza
Circle
Non-circular
Line, Lissajous-like curves, etc.
N/A
N/A
Table 1: Different neural algorithms for modular addition

---

"If a meeting starts at 10, and lasts for 3 hours, then it will end at 1." This familiar fact is a description of modular sum, 10+-3 (mod 12). and the movement of clock describes simple algorithm for modular arithmetic: the numbers through 12 are arranged on a circle in 360" /12 30 increments, angles of 10 30 and 3 x 30 are added together, then this angle evaluated to determine that corresponds to 1 x 30".
Remarkably, Nanda et al. find that neural networks like our Model implement this Clock algorithm, visualized in Figure (left): they represent tokens and as 2D vectors, and adding their polar angles using trigonometric identities. Concretely, the Clock algorithm consists of three steps: In step 1, tokens and b are embedded as E, = [cos( (w)a)) sin( [xaa] and E, = [cos(wy,b) sin by)]], respectively, where Wa 2 p (an everyday clock has p = 12 and 1). Then the polar angles of E, and E, are added (in step 2) and extracted (in step 3) via trigonometric identities. For each candidate output e, we denote the logit Qabe: the predicted output c* argmax, Qahe-
Crucial to this algorithm is the fact that the attention mechanism can be leveraged to perform multiplication. What happens in model variants when the attention mechanism is absent, as in Model A? We find two pieces of evidence of deviation from the Clock algorithm in Model A.
2.2 First Evidence for Clock Violation: Gradient Symmetricity
Since the Clock algorithm has logits:
Clookk (Eu,. Es,. B.g.E.. Eb,y)),E, + (Ea,. Eb. + Ea.y Eb,.) Beyy:
(1)
(see Figure 1b the gradients of Qate generically lack permutation symmetry in argument order: F, Qate F VE, Qabe- Thus, learned models exhibit permutation symmetry (Vr, Qabe
VE. E. Qabe), they must be implementing some other algorithm.
We compute the largest principal components of the input embedding vectors. We then compute the gradients of output logits (unnormalized log- probaabiiiiiss from the model) with respect to the input embeddings. We then project them onto these principal components (since the angles relevant the Clock and Pizza algorithms are encoded in the first few principal components). These projections are shown in Figure 2 symmetry.
While Model B demonstrates asymmetry in general, Model exhibits gradient
(a.b.c)-(7.49.33)
(9.b.c)=137.28.66)
2nd ombediings
40
Groctlents 2nd taken embeddinng mgs
40
20
20
10-
0-
0-
-10-
-10
-20
-20
-30-
Model A PCA Gradients PCA Gradients
-30
Model A PCA Gradients
-40
-00
Gradients
-00
-20
20
40
-40
-20
20
40
Gradients on 1st token embeddings
Gradients on 1st token embeddings E,
Figure 2: Gradients on first six principal components of input embeddings. (a, c) in the title stands for taking gradients the output logit for input (a,b). and y axes represent the gradients for embeddings of the first and the second token. The dashed line y = x signals a symmetric gradient.
2.3 Second Evidence for Clock Violation: Logit Patterns
Inspecting models' outputs, in addition inputs, reveals further differences. For each input pair (a, b), we compute the output logit assigned to the correct label + b. We visualize these correct logits from Models A and
Figure Notice that the rows are indexed by a b and the columns by + b. From Figure 3 we can see that the correct logits of Model A have clear dependency on - b in that within each row, the correct logits are roughly the same, while this pattern is not observed in Model B. This suggests that Models A and B are implementing different algorithms.
3

---

Correct Logits Model A
Correct Logits Model
3-
16
74
76
30
30
31
39
45
56
12:15 1821822223 2427
30 36-3633 424444444 58555557
12:55 12:5519:222
45
(a-3) mod 59
1a+5) mad 59
Figure 3: Correct Logits Model A Model B. The correct logits of Model A (left) have clear dependence on b, while those of Model B (right) do not.
3 An Alternative Solution: the Pizza Algorithm
How does Model A perform modular arithmetic? Whatever solution implements must exhibit gradient symmetricity in Figure the output patterns in Figure 3 In this section, describe new algorithm for modular arithmetic, which we call the Pizza algorithm, and then provide evidence that this is the procedure implemented by Model A.
3.1 The Pizza Algorithm
Unlike the Clock algorithm, the Pizza algorithm operates inside the circle formed by embeddings (just as pepperoni spread all over pizza), instead of operating The basic idea is illustrated in Figure given a fixed label c, for all (a. b) with a + = c (mod p). the points Eas (E,, E lie a line though the origin of a 2D plane, and the points closer to this line than to the lines corresponding to any other form two out of 2p mirrored "pizza slices", shown the right of the figure. Thus, perform modular arithmetic, network can determine which slice pair the average of the two embedding vectors lies in. Concretely, the Pizza algorithm also consists of three steps. Step 1 the same as in the Clock algorithm: the tokens and b embedded at E, = (cos( wxaa), sin (wya)) and E, = (cos(wbb), sin(w,b)), respectively. Step 2 and Step 3 are different from the Clock algorithm. In Step 2.1, E, and Eg are averaged to produce an embedding E.b- In Step 2.2 and Step 3, the polar angle of E.s is (implicitly) computed by computing the logit Qute for any possible outputs c. While one possibility of doing value of the dot product of Egb with (cos(w)c/2), sin (wxc/)2)), it is not commonly observed in neural networks (and will result different logit pattern). Instead, Step 2.2 transforms Eab into vector encoding cos(wu(a - (cos(( (a (a b)), sin(wx (a 10++ b))). which is then dotted with the output embedding U, = (cos( ey sin(wac)). Finally, the prediction is e* argmax, argmax,Quua Qabe- See Appendix A and Appendix real network.
the circumference of the circle.
is to take the absolute
[ for more detailed analysis of neural circuit that computes Ha in
The key difference between the two algorithms lies in what non-linear operations are required: Clock requires multiplication of inputs in Step 2, while Pizza requires only absolute value computation, which is easily implemented by the ReLU layers. If neural networks lack inductive biases toward implementing multiplication, they may be more likely implement Pizza rather than Clock, as we will verify in Section
3.2 First Evidence for Pizza: Logit Patterns
Both the Clock and Pizza algorithms compute logits Qabe in Step 3, but they have different forms. shown in Figure Specifically, Qube has an extra multiplicative factor |cos(w,, b)/2) compared to Qahe Clock). As a result, given e = a + b, Qate (Pizza) dependent on a b. but Qabel Clock) is not. The intuition for the dependence is that sample is more likely to be classified correctly if Egp longer. The norm of this vector depends on a b. As we observe in Figure the logits in Model A indeed exhibit strong dependence a b.

---

3.3 Second Evidence for Pizza: Clearer Logit Patterns via Circle Isolation
To better understand the behavior of this algorithm, we replace the embedding matrix E with series of rank-2 approximations: using only the first and second principal components, or only the third and fourth, etc. For each such matrix, embeddings lie in two-dimensional subspace. For both Model A and Model B, we find that embeddings form a circle in this subspace (Figure aland 5 bottom). We call this procedure circle isolation. Even after this drastic modification the trained models' parameters, both Model A and Model B continue to behave in interpretable ways: a subset of predictions remain highly accurate, with this subset determined by the periodicity of the of the isolated circle. As predicted by the Pizza and Clock algorithms described in Figure Model A's accuracy drops to zero at specific values of a - b. while Model B's accuracy is invariant in a - Applying circle isolation to Model A on the two principal components (one circle) yields model with 32.8% overall accuracy, while retaining the first six prineipal components (three circles) yields an overall accuracy of 91 .4%. See Appendix D for more discussion. By contrast, Model B achieves 100% when embeddings are truncated to the first six principal components. Circle isolation thus reveals error correction mechanism achieved via ensembling: when an algorithm (clock pizza) exhibits systematic errors on subset of inputs, models can implement multiple algorithm variants in parallel to obtain more robust predictions.
Correct Logits of Model
Circle Isolation
to
12% 35a
144
bntoeta inniiiitii (b
Ma BL MAw.EIaiii 13101
13
Crce from Princical Component 1.
Cirde from Princical 30
Circe from Princps Componert5.6
Circe from Priniipal 10
06
of
we
02
50
Dinle 8-00
60
Crde 83.6-4
66
Orde 5--5-2
-30
42
4
-1
Figure 4: Correct logits of Model A (Pizza) after circle isolation. The rightmost pizza is accompanying the third pizza (discussed in Section 3.4tionn344mm 3.4 Appendix D) Top: The logit pattern depends on b. Bottom: Embeddings for each circle.
Using these isolated embeddings, we may additionally calculate the isolated logits directly with formulas in Figure and compare with the actual logits from Model A. Results are displayed in Table We find that Qase( Qase(Pizza) explains substantially more variance than Qate Daabe(Clon (Clock).
2
Why do only analyze correct logits? The logits from the Pizza algorithm
given by Qutel (Pizza) = |cos(w><(a b)/2) cos (a + - c)). By contrast, the Clock algorithm has logits Qatel Clock) cos(wa(a + b c)). In word, Qate( (Pizza) has extra multiplica- tive factor coss(uu w.. - b)/2)l compared to Qanel Clock). By constraining c = a + b (thus cos(wa, cos(wu(a + c)) 1), the factor (cos(wx(e) b)/2)l can be identified.
(Unexpected) dependence of logits Qatel (Clock) on a + b: Although our analysis above expects logits Qate Clock) not to depend on a - b. they do not predict its dependence a + 6. In Figure 5 we surprisingly find that Qate Clock) is sensitive to this sum. Our conjecture is that Step and Step 2 of the Clock are implemented (almost) noiselessly, such that same-label samples collapse to the same point after Step 2. However, Step 3 (classification) is imperfect after circle isolation, resulting in fluctuations of logits. Inputs with common sums b produce the same logits.
5

---

Correct Logits Model after Circle Isolation
-18
54
-2
-0
-40
(81b)
Crcle from Priiipal Component 1.2
Crde from Frincipo Cumporent
Ciick from Prinipal St
Crèe from B iidpl Comzenent
08
Nad # Ph
00
Crde 91,6
Orde 12.6-11
Clicke 6wll
Chco M4,8
35
-875 0000 1423
026
675
-277 -038 4.2 on
02
466
-04
44
-02
00
di
Figure 5: Correct logits of Model B (Clock) after circle isolation. Top: The logit pattern depends on + b. Bottom: Embeddings for each circle.
Circle
Wk
Qabe(clock) FVE
Qabe(pizza) FVE
#1
2T/550
75.41% 1
99.18%
#2
2T
/59 3
75.62%
99.18%
#3
2T 259 44
75.38%
99.28%
Table 2: After isolating circles in the input embedding, fraction of variance explained (FVE) of all Model A's output logits (59 x x 59 them) by various formulas. Both model output logits and formula results' are normalized to mean 0 variance before taking FVE. we. are calculated according the visualization. For example, distance between 0 and 1 in Circle 17, so w/k = 2n 2n/59. 17.
3.4 Third Evidence for Pizza: Accompanied Accompanying Pizza
The Achilles' heel of the Pizza algorithm is antipodal pairs. If two inputs (a, happen lie antipodally, then their middle point will lie at the origin, where the correct "pizza slice" difficult to identify. For example right, antipodal pairs are (1,7), (2,8), (3,9) etc., whose middle points all collapse to the origin, but their class labels are different. Models cannot distinguish between, and thus correctly classify, these pairs. Even for odd p's where there are strict antipodal pairs, approximately antipodal pairs are also more likely to be classified incorrectly than non-antipodal pairs.
Intriguingly, neural networks find a clever way to compensate for this failure mode. we find that pizzas usually come with "accompanying pizzas". An accompanied pizza and its accompanying pizza complement each other the sense that near-antipodal pairs in the accompanied pizza become adjacent close (i.e, very non-antipodal) in the accompanying pizza. If we denote the difference between adjacent numbers on the circle as 6 and 6,, 82 for accompanied and accompanying pizzas, respectively, then 8, 28, (mod p). In the experiment, we found that pizzas #1/#2/#3 in Figure all have accompanying pizzas, which we call pizzas #4/#5/#6 (see Appendix D for details). However, these accompanying pizzas do not play a significant role final model predictions We conjecture that training dynamics are as follows: (1) At initialization, pizzas 1/#2/##3 correspond to three different "lottery tickets" 9. (2) In early stages of training, to compensate the weaknesses (antipodal pairs) of pizzas # #1/#2/#3, pizzas #4/#5/#6 are formed. (3) As training goes on (in the presence of weight decay), the neural network gets pruned. As result, pizzas #4/#5/#6 are not significantly involved in prediction, although they continue to be visible in the embedding space.
2 Accompanied pizzas #1/#2/#3 can achieve 99.7% accuracy, but accompanying pizzas #4/#5/#6 can only achieve 16.7% accuracy.

---

The Algorithmic Phase Space
In Section ]] have demonstrated typical Clock (Model A) and typical Pizza (Model B). In this section, we study how architectures and hyperparametes govern the selection of these two algorithmic "phases". In Section4.11
propose quantitative metrics that distinguish between Pizza and Clock. In Section 4.2 we observe how these metrics behave with different architectures and hyperparameters, demonstrating sharp phase transitions. The results in this section focus Clock and Pizza models, but other algorithmic solutions to modular addition are also discovered, and explored in more detail in Appendix B
4.1 Metrics
We wish study the distribution of Pizza and Clock algorithms statistically, which will require us to distinguish between two algorithms automatically. In order to do so, we formalize our observations in Section 2.2
and 2.3 arriving at two metrics: gradient symmetricity and distance irrelevance.
4.1.1 Gradient Symmetricity
To measure the symmetricity of the gradients, we select some input-output group (a, b, c). compute the gradient vectors for the output logit at position with respect to the input embeddings, and then compute the cosine similarity. Taking the average over many pairs yields the gradient symmetricity. Definition 4.1 (Gradient Symmetricity). For fixed set S S Z of input-outpul pair define gradient-symmetriity of a network M with embedding layer as
(8Qahe aQ EE EE
Σ sim
g
(a.b.c)
where sim(a, o. b) a-b clear that $g 111
is the cosine-similarity, Qate is the logit for class c given input a and is |
As discussed in Section 2.2 the Pizza algorithm has symmetric gradients while the Clock algorithm has asymmetric ones. Model A and Model B in Section have gradient symmetricity 99.37% and 33.36%, respectively (Figure 2
4.1.2 Distance Irrelevance
To measure the dependence of correct logits on differences between two inputs, which reflect the distances of the inputs on the circles, we measure how much of the variance in the correct logit matrix depends on We so by comparing the average standard deviation of correct logits from inputs with the same differences and the standard deviation from all inputs.
Definition 4.2 (Distance Irrelevance). For some network M with correct logit matrix L (Lig Qigátth define its distance irrelevance as
g Enez, stdl (Li,idd E Ep)
sld [Lis i E 23)
where std computes the standard deviation of a set. is clear that E [0, 1].
Model A and Model B in Section 3 give distance irrelevance 0.17 and 0.85, respectively (Figure B). A typical distance irrelevance from the Pizza algorithm ranges from 0 to 0.4 while a typical distance irrelevance from Clock algorithm ranges from 0.4 to 1.
4.1.3 Which Metric is More Decisive?
When the two metries have conflicting results, which one is more decisive? We consider distance irrelevance as the decisive factor of the Pizza algorithm, as the output logits being dependent on the distance is highly suggestive of Pizza. On the other hand, gradient symmetricity can be used rule out the Clock algorithm, as requires multiplying (transformed) inputs which will result in asymmetric gradients. Figure O confirmed that at low distance irrelevance (suggesting pizza) the gradient symmetricity almost always close to (suggesting non-clock).
To speed-up the calculations, in experiments S is taken random subset of 23 of size 100.

---

Gradient Symmetricity
1.0
0.5
-0.5
0.2
0.6
0.8
1.0
Distance Irrelevance
Figure 6: Distance irrelevance vs gradient symmetricity over all the main experiments.
4.2 Identifying algorithmic phase transitions
How do models "choose" whether to implement the Clock Pizza algorithm? We investigate this question by interpolating between Model A (transformer without attention) and Model B (transformer with attention). To do so, introduce new hyperparameter we call the atfention rate.
For model with attention rate a, we modify the attention matrix M for each attention head to be M' = Ma + (1 a). In other words, we modify this matrix to consist of linear interpolation between the all-one matrix and the original attention (post-softmax), with the rate a controlling how much of the attention is kept. The transformer with and without attention corresponds to the where a = (attention kept) and a O (constant attention matrix). With this parameter, we can control the balance of attention versus linear layers in transformers.
performed the following set of experiments on transformers (see Appendix F. for architecture and training details). (1) One-layer transformers with width 128 and attention rate uniformly sampled in [0, 1] (Figure 7.. (2) One-layer transformers with width log-uniformly sampled [32, 512] and attention rate uniformly sampled in [0, 1] (Figure 7). (3) Transformers with 2 to layers, width 128 and attention rate uniformly sampled in jo, 1 (Figure
The Pizza and the Clock algorithms are the dominating algorithms with circular embeddings. For circular models, most observed models either have low gradient symmetricity (corresponding to the Clock algorithm) or low distance irrelevance (corresponding the Pizza algorithm).
Two-dimensional phase change observed for attention rate and layer width. For the fixed- width experiment, we observed clear phase transition from the Pizza algorithm to the Clock algorithm (characterized by gradient symmetricity and distance irrelevance). We also observe an almost linear phase boundary with regards to both attention rate and layer width. other words, the attention rate transition point increases the model gets wider.
Dominance of linear layers determines whether the Pizza or the Clock algorithm is preferred. For lne-layer transformers, study the transition point against the attention rate and the width:
The Clock algorithm dominates when the attention rate is higher than the phase change point, and the Pizza algorithm dominates when the attention rate lower than the point. Our explanation is: At high attention rate, the attention mechanism more prominent in the network, giving rise to the clock algorithm. At a low attention rate, the linear layers are prominent, giving rise to the pizza algorithm.
The phase change point gets higher when the model width increases. Our explanation is: When the model gets wider, the linear layers become more capable while the attention mechanism receive less benefit (attentions remain scalars while outputs from linear layers become wider vectors). The linear layer therefore gets more prominence with a wider model.
Possibly hybrid algorithms between the Clock and the Pizza algorithms. The continuous phase change suggests the existence of networks that lie between the Clock and the Pizza algorithms. This is achievable by having some principal components acting as the Clock and some principal components acting the Pizza.

---

Gradient Symmetricity 0.4
Distance Irrelevance 0.4
0.2
0.6
0.8
0.2
0.6
0.8
1.0
1.0
Distance Irrelevance
Gradient Symmetricity
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Attention Rate
Attention Rate
Gradient Symmetricity
Distance Irrelevance 0.4
0.2
0.4
0.6
0.2
512
512
256
256
Model Width
Model Width
128
128
64
64
32
32
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Attention Rate
Attention Rate
Figure 7: Training results from 1-layer transformers. Each point in the plots represents a training reaching circular embeddings and 100% validation accuracy. See Appendix [ for additional plots. Top: Model width fixed to be 128. Bottom: Model width varies. The phase transition lines calculated by logistic regression (classify the runs by whether gradient symmetricity > 98% and whether distance irrelevance < 0.6).
Existence of non-circular algorithms. Although our presentation focuses circular algorithms (i.e., whose embeddings are circular), find non-cireular algorithms (i.e.. whose embeddings do form a circle when projected onto any plane) to be present in neural networks. See Appendix B for preliminary findings. We find that deeper networks are more likely form non-circular algortthms. We also observe the appearance of non-circular networks at low attention rates. Nevertheless, the Pizza algorithm continues to be observed (low distance irrelevance, high gradient symmetricity).
5 Related Work
Mechanistic interpretability aims to mechanically understand neural networks by reverse engi- neering them 2
12
13 [4] One either look for patterns in weights and
activations by studying single-neuron behavior (superposition monosemantic neurons (15)), or study meaningful modules or circuits grouped by neurons
Mechanistic interpretability is
closely related to training dynamics [8 1]
Learning mathematical tasks: Mathematical tasks provide useful benchmarks for neural network interpretabilityy. since the tasks themselves are well understood. The setup could be learning from images [16 [7]. with trainable embeddings [18), with number inputs [59 Beyond arithmetic relations, machine learning has been applied to learn other mathematical structures, including geometry (20), knot theory (1) and group theory [22].
Algorithmic phase transitions: Phase transitions are present classical algorithms 23 and deep
learning 66 24 25). Usually the phase transition means that the algorithmic performance sharply

---

changes when a parameter is varied (e.g., amount of data, network capacity etc). However, the phase transition studied in this paper is representational: both clock and pizza give perfect accuracy, but arrive at answers via different interal computations. These model-internal phase transitions are harder to study, but closer to corresponding phenomena in physical systems [24].
Emergent abilities in deep neural networks, especially large language models, have recently attracted significant attention [26]. An ability is "emergent" if the performance on a subtask suddenly increases with growing model sizes, though such claims depend on the choice of metric 27 has been hypothesized that the emergence of specific capability in model corresponds to the emergence of a modular circuit responsible for that capability, and that emergence of some model behaviors thus results from sequence of quantized circuit discovery steps [S].
Algorithm learning in neural networks:
6 Conclusions
We have offered a closer look at recent findings that familiar algorithms arise in neural networks trained on specific algorithmic tasks. In modular arithmetic, we have shown that such algorithmic discoveries are not inevitable: in addition to the Clock algorithm reverse-engineered by [. we find other algorithms (including Pizza algorithm, and more complicated procedures) be prevalent in trained models. These different algorithmic phases can be distinguished using a variety of new and existing interpretability techniques, including logit visualization, isolation of principle components in embedding space, and gradient-based measures of model symmetry. These techniques make it possible automatically classify trained networks according to the algorithms they implement, and reveal algorithmic phase transitions in the space of model hyperparameters. Here we found specifically that the emergence of Pizza or Clock algorithm depends on the relative strength of linear layers and attention outputs. We additionally showed that these algorithms are not implemented in isolation; instead, networks sometimes ensemble multiple copies of an algorithm in parallel. These results offer exciting new challenges for mechanistic interpretability:
(1) How to find, classify, and interpret unfamiliar algorithms in a systematic way? (2) How to disentangle multiple, parallel algorithm implementations in the presence of ensembling?
Limitations We have focused on a single learning problem: modular addition. Even in this restricted domain, qualitatively different model behaviors emerge across architectures and seeds. Significant additional work is needed to scale these techniques to the even more complex models used in real-world tasks.
We believe interpretability techniques can play a crucial role in creating and improving safe Al systems. However, they may also be used to build more accurate systems, with the attendant risks inherent in all dual-use technologies. It is therefore necessary to exercise caution and responsible caaisioo-makinn when deploying such techniques.
Broader Impact
Acknowledgement
We would like to thank Mingyang Deng and anonymous reviewers for valuable and fruitful discussions and MIT SuperCloud for providing computation resources. ZLI and MT are supported by the Foundational Questions Institute, the Rothberg Family Fund for Cognitive Science and LAIFI through NSF grant PHY-2019786. JA is supported by a gift from the OpenPhilanthropy Foundation.
References
[1] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023.
[2] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020.
[3] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain,
10

---

Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kermion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan. Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. http://waaaooomneecc..u////////a inndincci -indiinuut heads/index.htm..
[4] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023.
[5] Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. arXiv preprint arXiv:2303. 13506, 2023.
[6] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learn- ing algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023.
[7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[8] Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651-34663, 2022.
[9] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
[10]
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds. Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. htpps//transoorme--iicuuuss puu//uit /frammwwrr/indxx..iiee
[11] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
[12] Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. In The Fortieth International Conference on Machine Learning, 2023.
[13] Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data.
In The Eleventh International Conference Learning Representations, 2023.
[14] Arthur Conmy, Augustine N Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. arXiv preprint arXiv:2304. 2023.
[15] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.
[16] Yedid Hoshen and Shmuel Peleg. Visual learning of arithmetic operation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
[17] Samuel Kim, Peter Y Lu, Srijon Mukherjee, Michael Gilbert, Li Jing, Vladimir Čeperić, and Marin Soljačié. Integration of neural network-basee symbolic regression in deep learning for scientific discovery. IEEE Transactions Neural Networks and Learning Systems, 32(9):4166 1666 4177, 2020.
[18] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 2022.
[19] Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress deep learning: SGD learns parities near the computational limit, Advances in Neural Information Processing Systems, 35:21750-21764 2022.

---

[20] Yang-Hur Hui He. Machine-learinng mathematical structures. International Journal of Data Science in the Mathematical Sciences. 1(01):23 47, 2023.
[21] Sergei Gukov, James Halverson, Fabian Ruehle, and Piotr Sufkowski. Learning unknot. Machine Learning: Science and Technology, 2(2):025035, 2021.
[22] Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanbum, Peter Battaglia, Charles Blundell, András Juhász, al. Advancing mathemat-
ics by guiding human intuition with ai. Nature, 600(7887):70-74 2021.
[23]
Alexander K Hartmann and Martin Weigt. Phase transitions in combinatorial optimization problems: basics, algorithms and statistical mechanics. John Wiley & Sons, 2006.
[24] Lorenza Saitta, Attílio Giordana, and Antoine Cornuejols. Phase transitions in machine learning. Cambridge University Press, 2011.
[25] Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In International Conference on Machine Learning, pages 22965-23004. PMLR, 2023.
[26] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. Survey Certification.
[27] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? arXiv preprint 2300 2023.
[28] Michael Hassid, Hao Peng, Daniel Rotem, Jungo Kasai, Ivan Montero, Noah A. Smith, and Roy Schwartz. How much does attention actually attend? questioning the importance of attention in pretrained transformers. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1403-1416, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019. 2019.
12

---

Supplementary material
Mathematical Analysis and An Example of Pizza Algorithm
In the pizza algorithm, we have E.s cos(wu (a 6) (cos(wy (a + b)/2), sin(wy + bb)/2)) as cos + cos = cos((x - 2) (x +1) and sin r + sin cos( ( - y) ()) (2 sin y)/2)).
To get [cos(ux<(a b)/2) (a + b)). sin(wx (a + b))), we generalize this to |cos( u (a b)/ cos( (a b u)) (the two given cases correspond to 0 and u m//2/wk)
[ccos u u/2, (in u22) Eabl cos(wu (a b)/2) cos (w + u) [(-sin (w // 2)),, cos( 22) Eaol = Icos(ux 2)/2) sin (a + u)/2)
thus their difference will equal to
(cos(wx (os u)/2) |in (an(wua + u)/
Now notice |cos (t) [tin(t) 2 c2s(2t) for any (Figure 8 so the difference approximately ccos{ WC (a - b)/ cos (a + bu))..
1.00
07
0.50
25
-050
707
cos(2t) [cas(t))-|sintt)
-100
Figure [cos(t)] - (sin))))))))))) is approximately cos(2t) for any &
Plugging in and 1/2/un uy mentioned, get the following particular implementation of the pizza algorithm.
Algorithm: Pizza, Example
Step On given input a and b, circularly embed them to two vectors on the circumference (oos(wa), and (ios( (i).b))).
Step Compute:
(cos(w).a) + cos( (si((waa)) + sin(wab))/2
[ass(w b)/
+ b))
8 [cos(w).a) cos b) sin(waa) + sin(ww,)))/222 (
(cos(waa) + cos( (wxb) - sin(wx)))
ccos(wıa - π /4) + cos(w,b m/)) isin wtma T sin(wrb x/4)1/2 [cos(wx((a b) /2) cos(ww
/ /2) Icos(wx b) 22) sin(ws b))
Step 3 Output of this pizza is computed as a dot product.
D'te = a cos( c)) + sin zs |cos(wr (a b)/2)l cos(cccss + b-bba bcc)
Similar circuits are observed in the wild, but instead the above two-term approximation, a more complicated one observed. See Appendix for for details.
The extra coos(wx(e (a b)/2)l term not coincidence. We can generalize our derivation as the following.
13

---

Lemma A.I. A symmetric function far 3) that is a linear combination of cos z, sin 2, cos y, sin can always be written as cos () + y) for some function 9.
Proof. Notice cos + cosy = cos((r - (2 cos((2 (I + v)/2)) and sin r + sin = cos( (2 + f)/2), so a<cos cos y) + B(sin x sin y) cos((I (x - 2)/2) cos y)/2) 28 sin((2 u)/2)).
This is why we consider the output pattern with the (ww (a b)/2)] terms rather than the actual computation circuits as the determinant feature of the pizza algorithm.
B Non-Circular algorithms
One thing that further complicates our experiment is the existence non-circular embeddings. While only circular algorithms reported in the previous works [8] [. many non-circular embeddings are found in our experiments, e.g., ID lines or 3D Lissajous-like curves, as shown in Figure 9 We leave the detailed analysis of these non-circular algorithms for future study. Since circular algorithms are our primary focus of study, we propose the following metric circularity to filter out non-circular algorithms. The metric reaches maximum 1 when the principal components aligns with cosine waves.
0.5
Principal Comporent
-0.5
16
50
25
01/54
9
43
18
32
11
45
20
54
22
50
31
40
15
49
24
58
Principal Componert
Phincppa Componert
1.0
Princ ponccpalll Component
0.5
0.5
0.0
0.0
0.0
11.0
11.0
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
Principal Component
Principal Component
Principal Component
Figure Visualization of the principal components of input embeddings for two trained non-circular models. Top: A line-like first principal component. Notice the re-arranged x axis (token id). Bottom: First three principal components forming a three-dimensional non-circular pattern. Each point represents the embedding of token.
Definition B.1 (Circularity). For some network, suppose the I-th principal component of its input embeddings is 11 U1,1, U1,--1, define its circularity based on first four components as
Σ
2nd e jk/p
max kE[1,2,-
vr
p-11
p P>
of
where is the imaginary unit. E [0, 1] by Fourier analysis. Fourier waves.
means first four components are
Both Model A and Model B in Section have a circularity around 99.8% and we consider models with circularity > 99.5% circular.
C More Results from the Main Experiments
Here we provide Figure 7 with non-circular networks unfiltered (Figure 1O). We can see more noise emerging the plot. We also provide the training results from multi-layer transformers (Figure [T].
*The actual neural networks could be more complicated even if our neural network is locally linear and symmetric, locally they could be asymmetric (e.g. [æ yy could locally be æ y). Nevertheless, the pattern is observed in trained networks.
14

---

Gradient Symmetricity 0.4
Distance Irrelevance 0.4
0.2
0.6
0.0
0.2
0.6
0.8
1.0
1.0
Gradient Symmetricity
Distance Irrelvvance
0.8
0.6
0.4
0.2
circular
0.2
circular non-circular
non-circular
0.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.6
0.8
1.0
Attention Rate
Attention Rate
Gradient Symmetricity 0.4
Distance Irrelevance
0.8
0.2
0.4
0.6
512
512
256
256
Mocel Width
Mocel Width
128
128
64
non-circular circular
64
non-circular
circular
32
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Attention Rate
Attention Rate
Figure 10: Training results from 1-layer transformers. Each point in the plots represents training run reaching 100% validation accuracy. Among all the trained l-layer transformers, 34.3 % are circular. Top: Model width fixed be 128. Bottom: Model width varies.
D Pizzas Come in Pairs
Cautious readers might notice that the pizza algorithm is imperfect for near antipodal points, the sum vector will have very small norm and the result will be noise-sensitive. While the problem is partially elevated by the of multiple circles instead of one, also noticed another pattern emerged: accompanying pizzas.
The idea is the following: suppose the difference between adjacent points is 2k mod p, then the antipodal points have difference tk. Therefore, if we arrange a new circle with a difference & for adjacent points, we will get pizza that works best for formerly antipodal points.
Algorithm: Accompanying Pizza
Step Take wi of the accompanied pizza. On given input and circularly embed them two vectors on the circumference (cos(2wra), sin(2waa)) and (cos(2wxb), sin(2wx))).
Step 2 Compute the midpoint:
2 (cos(2wpa) + cos(2wgk)) sin (2w:a) + sin(2w,,))
Step Output of this pizza is computed as dot product.
A (cos(wac).... sin sin (in(w))))
This is exactly what we observed Model A (Table 3 Figure 13). With the six circles (pizzas and accompanying pizzas) included in the embedding, Model A also gets 100% accuracy.
15

---

Gradient Symmetriclty
Distance Inelevance 0.4
Gradient Symmetricity 0.6
0.2
0.4
0.6
0.8
0.2
0.6
0.3
0.4
0.8
Gradient Symimetricnt
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
Distance
0.4
0.2
02
non-Liiculer circular
0.2
circular
circulor
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Attention
Attention
Gradient Symmetricity
Dishance Imelevance
Gradiiett Symmatricity 0.6
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.4
3.0
Gredient Symmetricity
0.6
0.6
0.6
Distance
0.4
0.4
Distance
0.4
0.2
circulr
0.2
non-circan circular
0.2
circular
circulor
0.0
0.0
0.0
0.0
0.2
U.4
0.6
0.8
1.0
0.0
0.2
0.4
U.D
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Rate
Gradient Symmatricity
Distance Irreevannce
Gradient Symmetricity
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.6
0.8
1.0
1.0
Distance Irrelevance
Gredient Symmertee
Ireeevance
non-cirçulor circular
non circular
circular
0.2
0.4
0.6
0.8
1.0
0.0
0.6 Attention Rate
0.4
0.8
1.0
0.2
0.6
0.8
1.0
Attention Rate
Attention Rate
Training results from transformers with 2, 3 and 4 layers. Among all the trained transformers with 2, 3 and 4 layers, 9.95%, 11 11.5% and 6.08% are circular, respectively.
Figure
Accompanied and Accompanying Pizza (p 11)
0
10
5
6
9
0+1
2
10
0+5
0-1
direction
8
0+5
3
direction
4
2
0+1 larger norm likely correct
0+1 sum points point negative result
0+5 - smaller norm more likely to be wrong
0+5 points between points - positive result
Figure 12: An Illustration the Accompanying Pizza Algorithm
16

---

Circle
Wk
A. FVE
#4 (accompanying #1)
27/59
-199-11 17
97.56%
#5 (accompanying #2)
2x/59-
97.23%
#6 (accompanying #3)
2n/559 44
97.69%
Table After isolating accompanying circles the input embedding, fraction of variance explained (FVE) of all Model A's output logits by various formulas. Both model output logits and formula results' are normalized mean 0 variance before taking FVE. Accompanying and accompanied pizza have the same w/k-
Correct Logits of Model A after Accompanying Circle Isolation
-20
-10
-0
10
(e-b17 mod 59
(a-b):20 med 59
Co-b144 mos 50
121510212473302
(a+5)
(a+b)
Circle from Principal Compunert
Circle from Principal Componert
Citcle from Principel Cun 10, 11
de
2a0 203 Ae 2p.
2065
53
0.2
0.1
211
24
0.0
2,00
Circle 44, 26
3.0
Circle #5, 10
0.-
Circle 46,6
-31
2
-22
-02
-03
-0.4
0.4
-0.4
0.2
0.4
-03
-02
-01
01
02
Figure 13: Correct logits of Model A (Pizza) after circle isolation. Only accompanying pizzas displayed. Notice the complementing logit pattern (Figure
E Results in Other Linear Architectures
While this is not the primary focus of our paper, we also ran experiments on the following four different linear model setups (see Section F22 for setup details).
For all the models, first encode input tokens (a, 6) with trainable embedding layer WE: = WF,a I2 W We. (positional embedding removed for simplicity). L1, L2, La are trainable linear layers. The outmost layers (commonly referred unembed layers) have no biases and the other layers have biases included for generality.
Model a: calculate output logits as L2 (z) + +
Model 3: calculate output logits as L3 [RLeeLLL/LLLRRLLLUL/ +z)))))
Model ":: calculate output logits as La [RLLU[L + (z2))))).
Model d: calculate output logits Lec
(11; stands for the concatenation of #1 and 2z2)
The results are shown Figure 14 Rather surprisingly, Model a, Model ,3 and Model gave radically different results. Model 3 and Model y are very similar, and in general they more pizza-like
17

---

1.0
1.0
0.9
Gradient Symmetricity
0.8
Ciculantty
0.6
0.4
0.7
Type
Type
0.2
0.6
B
0.0
0.5
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.2
0.3
0.4
0.5
0.6
0.7
0.3
0.9
1.0
Distance Inrelevancy
Distance Irrelevancy
35
Model
40
Model
40
Model
35
Model
30
30
30
Runs
20
15
15
10
10
10
10
0
0.4
0.5
0.6
0.7
0.4
0.6
0.8
0.92
0.94
0.96
0.98
0.4
0.5
0.6
0.7
Distance Irrelevancy
Distance Irrelevancy
Distance Irrelevnnc
Distance Irrelevancy
Figure 14: Training results from linear models. Each point in the first-row plots represents a training run. The second are histograms for distance irrelevancy of each model type.
than Model a, with lower distance irrelevancy and higher circularity. This could be explained by the addition of an extra linear layer.
However, Model gave very different results from Model although they are both one-layer linear models. It is more likely to non-circular and have very high distance irrelevancy in general. In other words, concatenating instead of adding embeddings yields radically different behaviors in one-layer linear model. This result, again, alarmed us the significance of induction biases in neural networks.
We also note that using different embeddings on two tokens of Model doesn't resolve the discrepancy. The following model
Model a': calculate output logits as L2 (L + x2))) where x
W. E.a: 32
WBB E.b input (a, b) and W.A. W are diffferent embedding layers.
gives roughly the same result as of Model a (Figure 14 lower right corner).
Figure 15 shows the correct logits after circle isolation (Section 3.3 of circular model from Model implementing the pizza algorithm. Figure [6 shows the correct logits after circle isolation (Section the pattern is similar but different from the one of clock algorithm (Figure 5) We leave the study of such models to future work.
3.3) of circular model from Model 8. We
F
Architecture and Training Details
F1
Transformers
Here deseribe setup for the main experiments. See Appendix(Fland Appendix experiments on different setups.
We train bidirectional transformers (attention unmasked) to perform modular addition mod where
Architecture
59. To calculate (a + b) mod the input is provided to the model as sequence
18

---

Figure 15: Correct logits from Model B3 after circle isolation.
10
14
10
12
14
12
(3 6)2 mod 59
12b122 moo 99
[a-bi/27
4591244
Circ from Principal Comporent 1,
Circle from Principal Component 3,
Circle from Principal Component 5,
1.00
s0d0
1.60
102
d4
2012
aty Hor
6.75
0.75
0.50
0.0
$85
408
6.00
Circle 01,6-2
c.00
A07
Circle #2, 2
+
0.00
Circle 43,0
-6.5
-0.55
0.5
2303
30
23 2500
-1.00
-1.00
-05
0.0
-05
0.5
-65
0.5
Figure 16: Correct logits from Model ó after circle isolation.
5.0
00-01/2 Fod 59
222222 mod 59
0 3 1116001222700000000 45455551117
3 210113672110222445000
(8+3) mnd 59
(a+b) mad 59
(a+b) mad 59
Circle from Principal Component
Circle from Principal Component
Circie from Principal Component
o8 37 29.09
24
1.0
10-
26
0.5
0.5
0.
Circle +1,6=23
Circle #2
Circle #3, o=24
40
405
-65
-2.
69
22
40
-1.0
-19
4
-1.0
-05
0.
055
1.0
-10
-35
0.0
0.5
1.8
-1.0
-0.
0.6
05
1.
of two tokens [a, b). The output logit at the last token is considered as the output of the model. For a transformer with "width" d, the input embedding and the residue stream will be d-dimensional, 4 attention heads of [d/4] dimensions will be employed, and the MLP will be of 4d hidden units. By default 128 chosen. ReLU used as the activation function and layer normalization isn't applied. The post-softmax attention matrix is interpolated between all-one matrix and original as
19

---

specified by the attention rate (Section 4.2 We want to point out that the setup of constant-attentioo transformers also considered the previous work [28].
Among all possible data points (p² = 3481 of them), we randomly select 80% as training samples and 20% as validation samples. This choice (small p and high training data fraction) helps accelerating the training.
Data
Training We used A optimizer [29] with learning rate 7 = 0.001 and weight decay factor B 2. We do not use minibatches and the shuffled training data is provided as a whole batch in every epoch. For each run, we start the training from scratch and train for 20, 000 epoches. We removed the runs that did not reach 100% validation accuracy at the end of the training (majority of the runs reached 100%).
F.2 Linear Models
Here we describe our setup for the linear model experiments (Appendix Eb.
Architecture We train several types of linear models to perform modular addition mod p where p 59. The input embedding, residue stream and hidden layer are all d = 256 dimensional. ReLU
used as the activation function. The actual structures of network types are specified in Appendix E
Data & Training
Same as in the previous section (Section
F.1
F3 Computing Resources
total of 226 GPU days of NVidia V100 is spent on this project, although we expect a replication would take significantly fewer resources.
G Mathematical Description of Constant-A transformer
In this section, we examine the structure of constant-attention transformers loosely following the notation of [TO]
Denote the weight of embedding layer as WE, the weight of positional embedding as Wpss: the weight of the value and output matrix of the j-th head of the (-th layer as W and W the weights and biases of the input linear map MLP in the t-th layer as W and by. the corresponding weights and biases of the output linear map as W out and blmm and the weight of the unembedding layer as Wo. Notice that the query and the key matrices are irrelevant the attention matrix is replaced with an all-one matrix. Denote x as the value of residue stream vector after the first j layers and denote as the character the i-th position. We use subscripts like 2, to denote taking a specific element of vector.
We formalize the logit calculation the following:
Embedding: z! WE,C, + W poss..
For each layer / from 1 to Mayer.
E, - -Constant Attention: wf = x -f E, Wo' Σn xf-1
- MLLP: x² = w² + bbm + W out ReLU(b + W.ww))
Output: O Woze
In the particular case where the input length is 2. the number of layer is 1, and we focus on the logit of the last position, we may restate as the following (denote z as x" and as w
Embedding: x WF.a, + W 1, I2 = WF.cz + W Wpos.2-
Constant Attention: y = 22 E, W (Z + 22).
MLP: = y blut + W Why))
20

---

Output: Woz.
we remove the skip connections, the network after embedding could seen as
Σnb Lb (4 (21 + x2))
Lu
Louit
ReLU
Lin
where Liv: Ligs Lins Lout, Lu are series of linear layers corresponding the matrices.
H Pizza with Attention
we trained transformers with width 1024 and attention rate (normal attention). After several tries, we able observe trained circular model with distance irrelevance 0.156 and gradient symmetricity 0.995, which fits our definition of Pizza (Figure 17
Extrapolating from Figure 7
20- TOUE
To BUCY mot
(a+b) mod 59
(b+bl mod
101b mod 59
G- mad
Princioal Component 1.2
from Componert
from Component
from Component
62
Grdt 6-
-24
-35
Figure 17: Correct logits of the trained model in Section after circle isolation (Section 3.3
I Results on Slightly Different Setups
We considered the following variations of our setups (Appendix FT Section 4.. for which the existence of pizzas and clocks as well as the phase changes are still observed.
GeLU instead of ReLU We conducted the same l-layer transformer experiment with activation function GeLU instead of ReLU. Very similar results are observed (Figure 18).
Encode Two Tokens Differently We conducted the 1-layer transformer experiments but with different embedding for the two tokens. Again very similar results are observed (Figure [9). We also discovered that the two tokens' embeddings are often aligned to implement the Pizza and Clock algorithm (Figure 20).
Adding Equal Sign We conducted the 1-layer transformer experiment with an equal sign added. Very similar results are observed (Figure 2T).
Pizza Occurs Early in the Clock Training
We plotted intermediate states during the training of model with attention (attention rate 1). Pizza- like pattern was observed early the training, but the pattern gradually disappeared during the run (Figure 22)
21

---

Figure 18: Training results from 1-layer transformers with GeLU instead of ReLU as the activation function. Each point in the plots represents a training run that reached 100% validation accuracy.
Gradient Symmetricity
Gradient Symmetricity
0.2
0.0
0.2
0.4
0.6
0.8
0.4
0.6
0.8
1.0
Distance Irrelevance
Distance Iirelevance
0.8
0.8
0.6
0.6
0.4
0.4
0.2
non-circular
0.2
circular
circular
0.0
0.0
0.2
0.6 Attention Rate
0.4
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Attention Rate
Distance Irrelevance 0.6
Circularity 0.6
0.2
0.4
0.8
0.4
0.8
1.0
1.0
Gradient Symmetricity
Distance Irreevance
0.8
0.8
0.6
0.4
0.4
0.2
non-circular
0.2
non-circular circular
circular
0.0
0.0
0.6
0.0
0.2
0.B
Attention Rate
Attention Rate
Figure 19: Training results from I-layer transformers where the two tokens use different embeddings (feed (a, + p] to the model on input (a, b): 2p tokens are handled the embedding layer). Each point in the plots represents training run that reached 100% validation accuracy. We did not use circularity to filter the result because is no longer well-defined.
Gradient Symmetricity
Distance Irrelevance
0.25
0.00 0.25
0.50
0.75
0.2
0.4
0.6
0.8
Gradient Symmetricity
Distance Irrelevance
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Attention Rate
Attention Rate
22

---

Figure 20: Correct logits after circle isolation from trained model where two tokens different embeddings. The blue points represent the embeddings for the first token and the green points represent the embeddings for the second token. The model is implementing the Pizza algorithm. The correct logit pattern is shifted comparing to the previous patterns because the embeddings of two tokens do not line up exactly. For example, the third circle has near-maximum correct logit for = 6,b = 3 (the two points lining up on the top) and (a b)/18 = 10 (mod 59). This is the reason that the correct logit pattern appears to shifted 10 units down.
23
10.
12.5 15.
(e-M nod 59
S5 mo C77 1ece
(a-0/18 mod 59
1215199244733033 55955599999
12155993224339333 (a+b)
121538212437003330 220.2222222333322240000000
(a+ mod
(a+b)
Circle from Principal Component 1,
Circle from Principal Companent
Circle from Principal Component
0.-
0.2
02-
0.0
10
3.0
1=
0.-
-32-
-04
-02
0.4
-02
0.2
Figure 21: Training results from 1-layer transformers where an equal sign is added (feed [a, b, =] the model on input (a, b) where a special token; p + tokens are handled in the embedding layer; context length of the model becomes 3). Each point in the plots represents a training run that reached 100% validation accuracy. We did not use circularity filter the result because it no longer well-defined.
Gradient Symmetricity
Gradient Symmetricity
0.92
0.90
0.92
0.94
0.96
0.98
1.0
1.0
Distance Irrelevance
Distance Inelevance
0.8
0.8
0.6
0.6
0.4
0.4
0.2
non-circular
circular
circular
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Attention Rate
Attention Rate
K
Accompanying Pizza Occurs Early in the Pizza Training
We plotted intermediate states during the training model without attention (attention rate 0). We observed the early emergence of pattern similar to accompanying pizza in training runs (Figure
23

---

90 Steps
1C
1,
2C
2,
PC 3,4
0.5
0.0
0.0
-0.5
0.5
-0.5
0.5
-0.5%
-0.5
-0.5
0.0
0.5
-0.
0.0
6.5
-0.
0.0
6.5
-0.5
06
0.5
-0.5
0.0
0.5
-05
06
0.5
Iso. Correct Logits
Iso. Conrect Loghts
Iso. Connect Logits
Iso. Correct Logits
1so. Correct Looits
Iso. Conrect Logis
10
-10
-10
19 b
1000
210 Steps
PC
1,
PC 1,4
2.
FC 4
PC 3,
0.5
0.5
0.5
0.5
0.5
0.0-
0.0
0.0
0.0
0055
-0.5
0.0
0.5
-0.5
-0.5
0.0
0.5
-05
-0.5
0.0
0.5
-0.
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
Iso. Correct Logits
Iso. Correct Log ts
Iso. Cornect Logits
Iso. Correct Logits
Iso. Correct Logits
Iso. Correct Logits
0-2044 #00
-20
10
06-0044
10
(a+b)
18-11 med
(a+b)
Steps
2,
1,
2,
0.5
0.5
0.0
0.0
0.0
0.0
0.0
-0.5
-0.
-0.5
-0.3
0.0
0.
5.0
0.0
Correct Logits
Logits
Iso. Correct Loghs
Logits
Logit
Isc.
Logits
102014 moo3
2000 JO0L
000166 mod
20
10
(1+ mod 59
(2+01 mod 59
(b+t) mod
(110) moc 59
(2-b] mod 59
(5+10) mo0 59
510 Steps
1,1
1,
PC 1,
2.
2C 4
PC 3,
0.5
0.5
0.0
-05
-0.5
-0.5
-0.5
0.5
0.0
-05
0.6
-0.5
0.0
0.5
-03
-0.5
0.0
Iso. Correct Logits
Iso. Correct Logis
Iso. Conrect Logits
Isc. Correct Logits
Iso. Correct Logits
Isc. Correct Logits
-40
-40
0 0004 Fo0d
20
0402/ mod
-20
20002 mede
20
F0022 1o0d
-0
[-+8)
(5+0)
(h+10)
840 Steps
PC 1,2
3,
1,
PC 2,
PC 2.
3,
0.0
0.0
0.0
0,0
-0.5
-05
-0.5
-0.3
-05
-0.5
0.5
-03
0.0
0.5
-05
0.0
0.3
-0.5
0.0
0.5
-03
0.
-0.3
0.5
Iso. Correct Logits
Iso. Correct Logits
Iso. Correct Logis
Iso. Cornect Logits
Iso. Connec Logits
1s0. Correct Logits
-00
Gooll moze
20
(a+b)
Figure 22: For a I-layer transformer with attention, correct logits after principal component (possibly non-circle) isolations various states during the training. The pizza-like pattern gradually desolved.
23 and removing that circle brings accuracy down from 99.7% to 97.9%. They are less helpful later in the network (removing accompanying pizzas in trained Model only brings accuracy down to 99).%
24

---

(a-b)/20 59
be-b1.20 rrod 59
121518212427 353666 124544521155 (a+b) mod 59
921551822
121518212422
36 48515 5457 (a+b) mod 59
Circle from Principal Component 1,
Circle from Principal Component 13, 14
25
0.2
68
207
43
26
29
23
0.1
20
$9
0.1
25
0.0
22
2122
50
J43
56
200
40
-0.1
68
-0.2
26
58
27
30
-02
0.3
Figure 23: Immediate state after 600 epochs of training for a 1-layer transformer with constant attention.
L A Closer Look a Linear Pizza Model
In this section, we provide full picture of the linear model shown in Figure 15 by investigating the actual weights in the model.
L.1 Model Structure
As described in Appendix on input (a, b), the output logits of the model is computed as Ly [ReLLU(L (Embed[a) + Embed(b))))).
Denote the weight of embedding layer as WE, the weight of the unembedding layer (Lg) as Wu. and the weights and biases of Lı and L2 Wi,b, by and W2, d2, respectively, then the output logits input (a, b) can be written as
We.ReLU(b + W.ReLU(L, + Wi + We())))
L.2 General Picture
We first perform principal component visualizations the embedding and unembedding matrices. From Figure 24 we can see that the embedding and unembedding matrices formed matching circles (circles with the same gap between adjacent entries).
We now give the general overview of the circuit. Each pair of matching circles forms an instance of Pizza and they operate independently (with rather limited interference). Specifically for each pair,
The embedding matrix first places the inputs o, b on the circumference: [ Rs (cos(waa), (in(:ra)) and W [b] as (oos(wxb),sinff (in(w))) (wx- 2rkkp for some inte- ger & E [1.p as Section 2.1 W stands for the two currently considered principal components of W rotation and scaling omitted for brevity).
25

---

The embeddings are added to get
(cos(wxa) + cos(w)b), sin(wra) + sin(wxb)) (wx win(ww((e + b)/2))
=cos(w;(e (a - b)/2) (cos(wx(e (a + 6)/2), sin
It is then passed through the first linear layer L1. Each result entry Pre- will thus be linear combination of the two dimensions of the aforementioned vectors, i.c. cos(w,,e b)/22 (ar cos( (wx b)/2) + 3 sin(wn (a + 2)) for some a, 8, which will become cos(wu (a b)/2)|| cos(ux(( + b)/2) + sin wx + b)/2)> after ReLU.
These values then passed through the second linear layer L2. Empirically the ReLU is not observed to be effective as the majority of values is positive. The output entries are then simply linear combinations of aforementioned outputs of L1.
The unembedding matrix is finally applied. In the principal components we are considering, Wolcc] = (cos(w;c), sin (wge)). (Wy stands for the two currently considered principal components of Wi; rotation and scaling omitted for brevity) and these two principal components correspond to linear combination of the output entries of T.2, which then correspond to a linear combination of the outputs of L1 (thanks to the non-functional Rel.U).
Similar the formula sin(t)) cos(()) cos(2f) diseussed in Appendix A these linear combinations provide good approximations for Icos(w; (a b)/2) cos Wn (a + b)) and cossuu (a b)/2)| sin(wx (a + b)). Finally arrive
ccos( (u - -) (cos) cos( (os(wa(a + b)) + sin(w,e) sin(wx(( b))) |cos(wu b) / cos ooswwx(o (a + b c))
Embeddiing: Clicle from FC 2,2
Embeddiig: g: Crtie from PC 3,
Unembedding: Circle from FC 1,2
Unembedding Cirde from PC 3,
1.0
B5
do
Circle #-
Circle 6
Circle
Circee #2,
8.5
6.5
-10
-0.5
0.0
0.5
1.0
-05
0.2
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.5
Figure 24: Visualization of the principal components of the embeddings and unembedding matrices.
L.3 Aligning Weight Matrices
We first verify that the ReLU from the second layer is not functional. After removing it, the accuracy of the model remains 100% and the cross-entropy loss actually decreased from 6.20 x 10-7 to 5.89 x 10-?
Therefore, the model output can be approximately written as
Wo WWyleWeRRReenbnn +W; (W We (b)))
We now "align" the weight matrices Wi and W2 by mapping through the directions of the principal components of the embeddings and unembeddings. That is, we calculate how these matrices act on and onto the principal directions (consider Wyv for every principal direction in Wa and v W2 for every principal direction v in Wt). We call the other dimension of aligned W1 and W2 output and source dimensions, respectively (Figure 25).
In the aligned weight matrices, we can see a clear domino-like pattern: in most output or source dimensions, only two principal components have significant non-zero values, and they correspond to a pair of matching circle, or a pizza. In this way, every immediate dimension serves for exactly one pizza, so the pizzas do not interfere with each other.
26

---

Aligned acting principal components
0.0
-05
-10
212
230
224
77
Allgnee Wg: Ws acting onta the first princ pel components Wu
-05
-10
-15
22
22
Figure 25: Visualization of the aligned W1 and W2.
L.4 Approximation
Everything becomes much clearer after realigning the matrices. For a pizza and its two corresponding principal embedding/ / unembedding dimensions, W/ + [/] ss cos( (a-swaaa-b/2) - (cos(wy, (a 6)/2), sin(wx (a + b)/2)) will be mapped by realigned W1 into its corresponding columns (which are different for every pizza), added with b1 and apply ReLU. The result will then be mapped by the realigned W2, added with realigned b2, and finally multipled by (cos(wge), sin(w,c)),
For the first two principal dimensions, realigned W1 has 44 corresponding columns (with coefficients of absolute value > 0. Let the embedded input be (ær, y) W W = cos(w; /) (c 6) sin 2))) the intermediate columns are
R=LU((0.530x= 1e 1.10 1.210 0.3 -0448
1.072y + 0.215, 0.799y 0.249, 0.048y + 0.085, 1.149r 0.598y
1.038
0.905y + 0.190, -|
1.052y
+ 0.209, 1.293y
1.252y
0223110000-222200000000000000010 1,145
0..273, 0.435 -.000-088....33..-00444444401--4
0.285y
0.218,
102p:0044444444444444.4444400010 -0.244
0.880y 0.106, -0.956. 0.581y -0006--0.00:--00044---00000 0.064 0.152, - 1.24g 0.199,
0.120, 0.506x
-0.522
For the first principal unembedding dimension, will be taken dot product with
[1.326, 0.179, 0.142, -0.458, 1.101, 1.101,-0.083 -0.083, 0.621, 0.621, 0.709, 0. 0.123, -1.366, 10.51,,,1116 1.337. 0.732, 0.839, 0.129. 0.804, 0.377, 0.078. 1.322. -1021, 0.799, -0.339, -1.12,, -1.423, -1.157, 1.37,,,,3,,,,556 0.156, -0.165, -0.451, -1111,,,-006111111100-00171 111722-11.0 11.10, - 1.386, -1.346, -.346---.,, -0.226, 1.091, 1.159, -0.949, -1.2448)
Call this function f((:, y). When we plug in x cos(l), = sin(l), we get a function that well- approximated 8 coss(2t + 2) (Figure 26). Therefore, let wa + b)/2. the dot product will be approximately (Jcos(wy( (a b)/2) cos(wy b) + 2), or ccos (ww (a /2) cos cos(wyk] (a + b)) ignore the phase and scaling. This completes the picture we described above.
27

---

2cc00204-20
200 wewetttooosseeeee aisvettt)
22
Figure 26: (cos(tt), sin(t)) well-approximates 8 cos(2t 2).
28


Reviews:
[{"review_id": "JHrAzGjjPO", "reviewer": "Reviewer_wANB", "summary": "In this work, the authors focus on the problem of learning modular addition in NNs. Using clock and pizza algorithms, they show that model exhibits sharp algorithm transitions, which are affected by layer width and attention strengths, often resulting in the parallel occurrence of these phases.  A series of experiments are performed on the single-layer network to support the hypothesis proposed in this work. ", "questions": "In terms of interpretability there are models inspired by formal language theory that insert rules [1,3] and extract rules [2-7], termed as interpretable by extractions and such models are even tested on mathematical reasoning[4] task and are known to be turing complete even with finite precision and time [8]. Authors should discuss this line of work, as they are relevant.\n\nHow to determine the threshold for the circularity, does value >= 99.5% works for all model/architectures in terms of thresholding for circularity. It would be ideal if authors can provide empirical bound for this and how to determine such threshold. \n\nCan this framework be extended to convolutions with tensor weights or stateful models such as RNNs?\n\nAuthors do point out that deeper models leads to non-circular algorithms, but what is the bound for that? After how many layers non-circular behavior is shown by various models? At minimum providing empirical results will further strengthen this work. I would also like to see some empirical bounds on attention rate, what quantifies as high attention rate and what is low attention rate. Can author provide ablation study on various value of attention rate and also the width of layers?\n\nFew additional comments that are not clear from the manuscript.\n\nHow does the model effectively interpolate between the memorizing and generalizing solutions? \n\nDoes this also work with sinusoidal embeddings, masked embedding? Does the choice of embedding have an issue in generalization?\n\nThe authors do mention pruning the weights, so what effect does sparsity have in the model performance? Can authors comment on this? Like how the two-phase switch? \n\n\n\n\n\nFinally I would like to see total computational time required by the model including FLOPS and also standard error for various trials on proposed experiments.\n\nMinor comments\nThe figure 4 should be improved, its difficult to read values on y-axis and also values overlap in the circular diagram.  Same goes for other figures too.\n\t\n1.\tOmlin, C.W. and Giles, C.L., 1996. Rule revision with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering, 8(1), pp.183-188.\n2.\tTiňo, P. and Šajda, J., 1995. Learning and extracting initial mealy automata with a modular neural network model. Neural Computation, 7(4), pp.822-844.\n3.\tMali, A.A., Ororbia II, A.G. and Giles, C.L., 2020. A neural state pushdown automata. IEEE Transactions on Artificial Intelligence, 1(3), pp.193-205.\n4.\tMali, A., Ororbia, A.G., Kifer, D. and Giles, C.L., 2021, May. Recognizing and verifying mathematical equations using multiplicative differential neural units. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 6, pp. 5006-5015).\n5.\tWeiss, G., Goldberg, Y. and Yahav, E., 2018, July. Extracting automata from recurrent neural networks using queries and counterexamples. In International Conference on Machine Learning (pp. 5247-5256). PMLR.\n6.\tWang, C., Lawrence, C. and Niepert, M., 2022. State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6), pp.7739-7750.\n7.\tOkudono, T., Waga, M., Sekiyama, T. and Hasuo, I., 2020, April. Weighted automata extraction from recurrent neural networks via regression on state spaces. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 5306-5314).\n8.\tStogin, J., Mali, A. and Giles, C.L., 2020. A provably stable neural network Turing Machine. arXiv preprint arXiv:2006.03651.", "limitations": "As highlighted above, the main point is an ablation study to support the hypothesis and computational overhead.\n\n**********\nScore increased after Author rebuttal Responses\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Well-written paper.\n2, an useful contribution in terms of interpretability of NNs.\n3. Novel contribution in terms of analyzing the algorithmic transitions.", "weaknesses": "1. Ablation study is missing\n2. Did authors perform a grid search to select best hyper-parameters? Then that should be mentioned with ranges in the appendix.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your helpful and constructive questions/suggestions! Below is our reply to your questions:\n\n> Q1: Ablation study is missing. Can authors provide ablation studies on various values of attention rate and also the width of layers?\n\nA1: Thank you for the suggestion! If we understand correctly, the ablation study you are requesting is already provided in Figure 6 of the submission, which shows behavior at  various values of attention rate and model width.\n\n> Q2: Did authors perform a grid search to select the best hyper-parameters? Then that should be mentioned with ranges in the appendix.\n\nA2: No. We largely followed [Nanda2023] for hyperparameter setups and we chose p=59 following [Liu2023] to simplify the investigation.\n\n\n> Q3: Can you discuss the literature and cite relevant papers?\n\nA3: Thanks for pointing us to these references, which are indeed relevant to our work. We would like to include the citations in the next updated version.\n\n\n> Q4: How to determine the threshold for circularity?\n\nA4: This is necessarily a bit subjective. Circularity is fundamentally a qualitative phenomenon, and this threshold indicates how tolerant we are in terms of considering some shapes as being approximately circular (which the authors all agreed was true of shapes with circularity >= 99.5%). \n\n\n> Q5: Can this analysis apply to convolutional neural networks or recurrent neural networks?\n\nA5: Yes, our analysis can apply to other architectures, e.gCNN and RNN. This analysis does not require inspecting latent representations (which are specific to architectures), only involving output logits and gradients wrt to input embeddings (which are universal to all architectures).\n\n> Q6: For the transition from circular to non-circular algorithms, what is the bound (phase transition point) along depth, attention rate, width?\n\nA6: There is no clear phase transition from circular to non-circular algorithms against attention rate and width, but depth 1 networks are clearly more likely to be circular solutions than deeper (2-4 layers) networks. See figure b and c in the attached pdf.\n\n\n\n> Q7: How does the model effectively interpolate between the memorizing and generalization solutions?\n\nA7: Our work focuses mostly on analyzing the final (generalization) solution. The training dynamics (how the model interpolates from a memorizing to a generalization solution) is interesting but might be out of the scope of this work.\n\n\n> Q8: Does this also work with sinusoidal embeddings, masked embedding? Does the choice of embedding have an issue in generalization?\n\nA8: Yes, we believe other types of embeddings can also lead to generalization. We’re currently using learnable positional embedding, but our proposed pizza algorithms do not depend on them, so we believe the pizza algorithms also exist under sinusoidal positional embeddings. As for masked embeddings, in the 1-layer case masked embedding is equivalent to bidirectional embedding (since attention to token #2 to token #1 doesn’t count) so our conclusions should remain valid.\n\n> Q9: How does pruning weights (hence sparsity) affect performance?\n\nA9: It is unclear to us whether our analysis has implications for sparsity. Norm-wise strong concentration is observed and its relationship with attention rate, distance irrelevance is observed but weak (figure g in the attached pdf). We observed some difference in parameter distribution for Model A and Model B (figure d the attached pdf) and we believe it is primarily a result of different model configuration (for example, query and key matrices are ignored by constant-attention Transformers).\n\n\n> Q10: Can you provide computation time (including FLOPs) and error bars?\n\nQ10: We spent roughly 226 GPU days on a V100 cluster with ~30% utilize rate, so the total computation is around 4e19 FLOPs.\nIt is hard to provide an error estimation since we are sampling with respect to multiple parameters, but we have made the full distribution available.\n\n\n> Q11: Figure 4 (and so as other figures) should be improved to maximize readability. \n\nA11: Thanks for the suggestion! We will increase fonts and do other optimization if needed.\n\n\n**References**\n\n[Nanda2023] “Progress measures for grokking via mechanistic interpretability”, Nanda et al.\n\n[Liu2023] “Towards Understanding Grokking: An Effective Theory of Representation Learning”, Liu et al.\n"}, {"review_id": "cwBJT1ZshW", "reviewer": "Reviewer_ZmSF", "summary": "The authors present a novel algorithm as a mechanistic explanation of neural networks for modular addition. It is noted that the model without attention fails to implement the ‘Clock’ algorithm. This assertion is substantiated with evidence related to gradient symmetricity and logit patterns. The authors then propose an alternative solution, named the ‘Pizza’ algorithm, supported by evidence concerning logit patterns via circle isolation and accompanying ‘pizza’. Ultimately, they demonstrate the presence of an algorithmic phase transition along the attention rate and model width, employing metrics that indicate gradient symmetricity and distance irrelevance.\n", "questions": "What prevents us from directly ascertaining the algorithm employed by the model? Couldn't it be possible to determine the intermediate vector $E_{ab}$ to gain insights into the algorithmic process?", "limitations": "As the authors have noted, their focus lies on a simple learning problem. Significant further work is required to adapt their techniques for use with the more complex models typically employed in real-world tasks.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-structured and supports its arguments with solid experiments. The authors demonstrate that a neural network is capable of learning diverse algorithms for the same task. They introduce an impressive procedure for interpreting the neural network via embedding vectors. This methodology has the potential for extension to more complex models and tasks.", "weaknesses": "The authors employ the term logit $Q_{abc}$ as well as the term output logit, which refers to the un-normalized log probability. The choice of terminology, however, proves to be confusing. Given that $Q_{abc}$ is not used in the model and is a concept introduced by the authors themselves, it would be beneficial to rename $Q_{abc}$ to a more intuitive term like \"value\" or \"rank\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your helpful and constructive questions/suggestions! Below is our reply to your questions:\n\n> Q1: Calling $Q_{abc}$ output logit is confusing.\n\nA1: This terminology has been used in many previous interpretability studies [Nanda2023] [Wang2023], so we are using standard nomenclature in this research area.\n\n\n\n> Q2: Can we determine the intermediate vector $E_{ab}$ to gain insights into the algorithmic process?\n\nA2: It is certainly possible, and we can prove that mechanistically in constant-attention Transformers, the computation starts by adding two embeddings (Appendix G). \n\n\n**Reference**\n\n[Nanda2023] “Progress measures for grokking via mechanistic interpretability”, Nanda et al.\n\n[Wang2023] “Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small”, Wang et al."}, {"review_id": "7v099To47j", "reviewer": "Reviewer_HMiA", "summary": "*Background*: Previous work in mechanistic interpretability has identified a particular algorithm that NNs implement (the clock algorithm) to solve modular addition. However, under certain architecture changes, the authors notice that NNs implement a different algorithm.\n\nThe main goal of this paper is to present inconsistencies in the clock algorithm for neural networks without attention (ie: an inductive bias that allows the model to implement multiplication) and motivate a different algorithm (the pizza) that explains a different algorithm through which such networks learn modular attention. The observations are backed by experiments on linearly interpolating between a NN with attention and an NN without it. The experiments also demonstrate that a single algorithm doesn’t always win: different models can and do ensemble multiple copies of both algorithms in parallel.", "questions": "* L78: Why the choice of 6 vectors specifically? For exposition? Will choosing less significant components introduce unnecessary noise?\n* L113: What would happen with an odd number of ReLU units?\n", "limitations": "The authors have addressed limitations in the manuscript. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is very well written and the proofs and arguments presented seem airtight.  Most of the questions I’ve had while reading the paper are either addressed in subsequent sections or in the appendix. The experiments are simple yet, I believe, comprehensive in evaluating the arguments presented. This is also an exciting emerging area of research and should lead to interesting discussions in the mechanistic interpretability community.", "weaknesses": "This work raises a lot of interesting questions, but I really can’t find any egregious logical inconsistencies with this work.  \nmeta-(non)concern: This work largely relies on a problem from a paper that hasn’t been peer-reviewed. However, I do not think this is a reason to reject this work.\n\nOverall, I recommend _acceptance_.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your helpful and constructive questions/suggestions! Below is our reply to your questions:\n\n> Q1: L78: Why the choice of 6 vectors specifically? For exposition? Will choosing less significant components introduce unnecessary noise?\n\nA1: The main motivation is that the gradient *a*symmetricity is more prominent for the first principal components, as these are more important for the function, and being symmetric is likely easier for the network. The choice also helps to be more consistent with the later discussions on 3 circles (Fig 4, Fig 5), which correspond to the first 6 principal components. In fact, in the later calculation of gradient symmetricity (Def 4.1) no translation to the principal component space is performed. We’ve attached the same figure with more (20) principal components and without the principal component projection (figure e in attached pdf).\n\n> Q2: L113: What would happen with an odd number of ReLU units?\n\nA2: We can implement absolute value $|x|$ by $\\text{ReLU}(x)-\\text{ReLU}(-x)$. If there are an odd number of ReLU units, some could be dead neurons (in the sense that the activation is near-zero for all inputs). There are also multiple possible variants of the pizza algorithm (Appendix A).\n"}, {"review_id": "rIP1KAFghC", "reviewer": "Reviewer_7M7r", "summary": "In this paper the authors study neural networks doing modular addition of integers, i.e. output $= mod_P( a + b)$ for fixed integer P and input integers a and b. Previous work on these networks has found that small transformers implement a simple _clock_ algorithm. This work verifies this, but shows that if you simplify the transformer’s attention and make your network more like a simple feedforward ReLU network, then you find the networks implement a completely different algorithm they call the _pizza_ algorithm. Their evidence for this includes (i) strange patterns in the correct logit outputs, (ii) gradients that do not fit the clock model but can be understood in their pizza model, (iii) patterns in the logits when the inputs are restricted to particular 2D planes, and (iv) the need for error correcting that leads to particular patterns in a pizza embedding. They then study an algorithmic phase change from pizza to clock as you vary how much attention is included.", "questions": "\n1.\tWhy do you compute the gradients after having projected to first 6 principal components of the embedding space? What happens if you don’t do this?\n2.\tFurther, why on earth did you think to make this gradient symmetry plot?! Did you already have the pizza algorithm in mind and knew this would distinguish them???\n3.\tIn table 1 it says non-circular algorithms show gradient symmetry, but in general (figure 9) it appears like they don’t. Why does it say this?\n4.\tThe phase transition appears to happen at slightly different points when measured by distance irrelevance or by gradient symmetry, is this true? This definitely seems true in the 2D phase plots. How do you interpret this?\n5.\tThis study restricts itself to circular algorithms, what proportion of models were circular? \n6.\tThe phase change is not discrete, i.e. it does not just jump from the most pizza-ey to the most clock-ey. This could be outside the scope, but what do you think happens in between? An algorithm that is a mixture of the two somehow? Or an output that is a mixture of both algorithms running concurrently? Any evidence in any direction?\n", "limitations": "The authors did a good job discussing limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "I thought the large-scale motivation for this work was justified, previous people have claimed that this specific task leads many networks to solve it in the same way. Turns out that isn’t true. That seems important.\nI thought the main discovery of this work was very interesting.\nI thought the evidence to back up the claim was convincing.\nI thought the experiments performed were pretty thorough.\nFor the most part the claims were not overblown (for example I really appreciated the discussion of non-circular algorithms for solving this task)\nFinally the appendix was a bit of a treasure trove.\n", "weaknesses": "I think the paper’s clarity could definitely improve. At times, mainly in the appendix where a lot of the explanation gets relegated, the paper felt rushed and the explanations were terse, expecting a lot from the reader. \n\nI found this especially true of the equations in the appendix, for example:\n\n1.\tIn appendix A you introduce $s$ on the first line, which is the same as $E_{ab}$ I think. Why do you introduce a new symbol? Why do you not make it clear it is the same as $E_{ab}$ in the main paper? \n2.\tYou then do the same, again in Appendix A, when introducing the symbol $P_c$, which is I believe $Q_{abc}$ from the main paper? You also say thus, which in maths makes me expect to understand that you have derived a result, instead at the moment it reads like a definition of $P_c$ so the use of thus is confusing. \n3.\tLine 392, in step 1 you talk about an accompanied pizza in a section that readers are expected to reach long before they read about accompanied pizzas where that adds no value as far as I can tell, but just makes the whole thing confusing.\n4.\tAppendix G was a minefield of strange notational choices in my mind. The layer index was initially denoted with i, but then changes to t, and i is reused as the token index.  I found this needlessly confusing.\n5.\tYou define $x^j$ as the value of the residual stream after j layers, but then talk about $x_{i}^j$ for a couple of lines of the algorithm, before dropping the lower index. I think this is because there’s only one output logit so after the attention you can drop which token the input is coming from, but I still found it confusing at first (because you never say what I just wrote). It felt important to specify that the lower index of x is tokens for understanding the sum over k in the constant attention equation.\n6.\tFor the next description you switch notation but still use x, but now without the top index rather than the bottom one, and use y and z. Could you highlight what is changing?\n\nAnd beyond the equations I thought that occasionally the appendix was a tough read. For example in appendix H you talked about adding an equal sign. I eventually looked at the caption for figure 19 and understood what that meant, but I thought I’d missed some previously discussed equals sign (I now realise this is a hangover from the original clocks paper). Making sure the writing doesn’t have these kind of moments when the reader hasn’t been told about something and isn’t completely sure where to find out about it seems good. Perhaps you could edit the writing and captions when you try and re-read the paper with fresh eyes to see what is confusing - or get some fresh eyes on it.\n\nA few of small things I am confused by: \n\n1.\tI think the sine and cosine addition formulae you are using in step 2 of the algorithm in appendix A, during the development of alpha and beta, are missing factors of 2. [since you use them so often maybe it would be good to state the sine and cosine addition formulae somewhere]\n2.\tIn section 3.4 it says the condition for there to be no antipodal pairs is for p to be prime, isn’t it for p to be odd? What am I missing?\n3.\tThe plots in figure 6 (and all figures like it in the appendix) are titled wrong.\n4.\tIn the formula for distance irrelevance i on the top row should i be a member of $\\mathbb{Z}_P$ not $\\mathbb{Z}_P^2$?\n\nIt seems there are a class of pizza like algorithms (e.g. the two in appendix A), and the evidence listed does not distinguish them. Is this true? Do you know which is happening? If not, perhaps figure 1 is misleading. Instead your claim is that step 3 is one example of a class of pizza type algorithms that are being implemented, and perhaps figure 1 could say that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your helpful and constructive questions/suggestions! Below is our reply to your questions:\n\n> Q1: Are $E_{ab}$ and $s$ the same?\n\n> Q2: Are $P_{c}$ and $Q_{abc}$ the same?\n\nA2: Thanks for pointing them out! Yes to both - Appendix A was following an older set of notations. We will make sure to correct these mistakes in the final version.\n\n> Q3: Line 392, an accompanied pizza is mentioned but is only defined long after.\n\nA3: Agreed. We will remove the first sentence in the final version.\n\n> Q4: The notations in Appendix G are confusing.\n\nA4: Thanks for the feedback. We have changed all i’s in the first paragraph to t’s.\n\n> Q5: Are $x^j$ and $x^j_{i}$ the same?\n\nA5: Here $x^j$ stands for the whole residual stream vector and $_{i}$ stands for taking its i’th element / dimension. When we drop the lower index, we are performing vector operations on the whole vector $x^j$. We agree it is very confusing and we will make sure to explain the notation choice.\n\n> Q6: Why the change from x to y,z ?\n\nA6: In our particular case there is only one layer, so we feel using x^0 and x^1 will be more confusing so we switched to x and z instead. Except for the notation choice and the expanded loop, nothing is changed.\n\n> Q7: The sine and cosine formula is missing a factor 2. Also specify the formula somewhere.\n\nA7: Great catch! We have added the missing factor and added the two relevant trigonometry formulas at the beginning of Appendix A.\n\n\n> Q8: No antipodal pairs only require p to be odd, not necessarily prime. \n\nA8: Our intention was to stress the case where p is an odd prime which is the most typical setup, but it is indeed confusing. We will change it to be p odd.\n\n> Q9: Plots in Figure 6 (and similar figures in the appendix) are titled wrong.\n\nA9: If we understood your concerns correctly, the plots are not titled (and are labeled) and the text on top are descriptions for the color bar. We agree it is a bit confusing and we will left-align the text to make it clearer. We will update it in the next version.\n\n> Q10: $\\mathbf{Z}_p$ or $\\mathbf{Z}_p^3$?\n\nA10: It should be $\\mathbf{Z}_p$, thanks for pointing that out! We will surely correct it.\n\n> Q11: It seems there are a class of pizza-like algorithms (e.g. the two in appendix A), and the evidence listed does not distinguish them.\n\nA11: Indeed there exists a class of pizza algorithms. The pizza algorithms can differ in how the terms $\\cos(w_k(a+b))$ and $\\sin(w_k(a+b))$ are approximated by ReLU neurons (Figure 7 and Step 2 in Appendix A). More active neurons could lead to better approximation, but different random seeds and/or hyperparameters may lead to different numbers of active neurons. We will follow your suggestion to update Figure 1 such that it can encompass the whole pizza family and use the current algorithm as a possible special case.\n\n> Q12: Why compute gradients after projecting onto the first six principal components?\n\nA12: The gradient *a*symmetricity is more prominent for the first principal components, as these are more important for the function, and being symmetric is likely easier for the network, and we choose six to be consistent with the later discussion on the three circles. In fact, in the later calculation of gradient symmetricity (Def 4.1) no translation to the principal component space is performed. We’ve attached the same figure with more principal components and without the projection (figure e in the attached pdf). We can see that the gradients are most symmetric for the later principal components as they are not very useful for the algorithm, and without the projection step, the gradient asymmetricity is, in fact, more pronounced for Model B, as the asymmetric gradients on the few principal components are now pronounced across multiple dimensions.\n\n> Q13: What's the rationale behind the gradient symmetry plot?\n\nA13: Past work has shown that neural networks (especially without attention) struggle to learn how to multiply inputs. In this respect, the Clock algorithm felt *unnatural* and we suspected there might be an alternative Pizza-like solution based on linear combination instead.\n\n> Q14: In table 1 it says non-circular algorithms show gradient symmetry, but in general (figure 9) it appears like they don’t.\n\nA14: Thanks for pointing that out! There indeed exist both gradient symmetric and gradient asymmetric non-circular algorithms. We will modify Table 1.\n\n> Q15: The phase transition points seem different when measured by distance irrelevance or gradient symmetricity.\n\nA15: Yes. In short, we believe this is caused by algorithms that are neither clock nor pizza. We consider distance irrelevance to be the *defining* feature of Pizza. Gradient symmetricity is mainly presented as supplementary evidence against the Clock algorithm, which requires multiplying (transformed) inputs, which will result in asymmetric gradients. From figure a in the attached pdf we can see that having low distance irrelevance is indeed a stronger condition than having high gradient symmetricity, suggesting the existence of algorithms that are not clock (high gradient symmetricity) and not pizza (high distance irrelevance).\n\n> Q16: What proportion of models are non-circular?\n\nA16: For our trained 1,2,3,4-layer 128-width models, the circular (circularity >= 99.5%) ones are 34.31%, 9.95%, 11.55% and 6.08%, respectively. We also attached the circular rate at each attention rate decile and at each width range (figure b & c in the attached pdf).\n\n> Q17: The phase change is continuous. What happens in between?\n\nA17: We conjecture that it is a hybrid of clock and pizza: the Algorithm in Appendix 1 takes the dot product of $(\\alpha,\\beta)$ with $(\\cos(w_k c),\\sin(w_k c))$ - same as in the clock algorithm. Therefore, it is possible to have some PCA circles operating as the clock algorithm and some operating as the pizza algorithm, and their results are added together before the final dot product with $(\\cos(w_k c),\\sin(w_k c))$."}, {"review_id": "tRNegnkdmk", "reviewer": "Reviewer_DtSC", "summary": "This paper takes a closer look at the mechanistic explanation of neural networks learning to perform modular addition expanding on recent work that argued that such networks discover a simple \"clock\" algorithm. The authors demonstrate that changes in initialization and hyperparameters can lead to the discovery of qualitatively different algorithms - most notably what is referred to here as the \"pizza\" algorithm. This provides evidence that even the simple learning problem of modular addition leads to the discovery of diverse solutions in neural networks and mechanistic explanation requires a more complex analysis.", "questions": "1. Focusing on one of the main claims that \"some networks very similar to the ones trained by [1] preferentially implement a qualitatively different approach\": What is the fraction of neural networks with attention rate 1 that according to your metrics implement the \"pizza\" algorithm? Judging from Figure 6 it seems to me that this claim might be too strong if the originally investigated model with full attention almost always discovers \"clock\" solutions.\n2. You state that you also find \"non-circular algorithms\" in your trained networks. What fraction of trained models is non-circular and removed from the main analysis? Again this would be especially insightful to understand in dependence of the attention rate.\n3. How are Distance Irrelevance and Gradient Symmetricity related to each other? From my understanding they both intend to measure the same property (pizza vs clock). A scatter plot showing one vs. the other might give some insight on their relationship. \n4. In cases where the metrics contradict each other (judging from Figure 6 this sometimes happens), can you still make a confident statement on what algorithm such solutions implement?\n6. Could you elaborate how you come to the conclusion that some solutions \"implement multiple, imperfect copies of either the Clock or Pizza algorithm in parallel.\"? How do such solutions work?\n7. Could you elaborate why accompanied pizzas achieve almost perfect accuracy (footnote of page 6) despite the failure mode of antipodal pairs?\n8. In section 3.4 you conjecture that \"accompanying pizzas\" are primarily used early in training. Would it be possible to test this hypothesis by comparing the accuracy of \"accompanied pizzas\" early in training with and without the hypothesised \"accompanying pizzas\"?\n9. Minor point: The word \"symmetricity\" is unfamiliar to me. Is there a reason to deviate from the more common term symmetry, i.e. calling your metric \"Gradient Symmetry\"?", "limitations": "Limitations have been addressed appropriately.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and succeeds in clearly communicating the findings on an active topic in the field of mechanistic interpretability. The analysis is carefully conducted, empirical results are mostly convincing and the conclusion is of importance for the broader field.", "weaknesses": "One of the main claims of the paper is that \"some networks very similar to the ones trained by [1] preferentially implement a qualitatively different approach\" but most of the evidence presented for such different solutions only apply to models that (transiently) remove the attention mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your helpful and constructive questions/suggestions! Below is our reply to your questions:\n\n> Q1: The argument “some networks very similar to the ones trained by [1] preferentially implement a qualitatively different approach\" seems too strong. What is the fraction of neural networks with attention rate 1 that according to your metrics implement the \"pizza\" algorithm?\n\nA1: By “similar” we mean structurally similar: our setup is almost identical to [1] except for the introduction of attention rate. From the data we have it seems quite unlikely for a network with attention rate near 1 to implement the pizza algorithm, although we did observe many non-circular (and thus unlikely clock) ones (Fig 6, Appendix C Fig 9). We agree it is somewhat misleading and we will tone down the claim in the updated version.\n\n\n> Q2: What fraction of trained models is non-circular?\n\nA2: For our trained 1,2,3,4-layer 128-width models, the circular (circularity >= 99.5%) ones are 34.31%, 9.95%, 11.55% and 6.08%, respectively. We also attached the circular rate at each attention rate decile and at each width range (figure b & c in the attached pdf).\n\n\n> Q3: What’s the relation between distance irrelevance vs gradient symmetricity?\n\nA3: We consider distance irrelevance as the deciding factor of pizza, as there seem to be limited other reasons for the output logits to depend on the distance. Gradient symmetricity is mostly used to rule out the clock algorithm as the clock algorithm requires multiplying (transformed) inputs, which will result in asymmetric gradients. Following your suggestion, we compiled the scatterplot of distance irrelevance vs gradient symmetricity over all the standard structure experiments we’ve done, and we can indeed see at low distance irrelevance (suggesting pizza) the gradient symmetricity is always close to 1 (suggesting non-clock) except for a few outliers (figure a in the attached pdf).\n\n> Q4: When two metrics contradict, how to make a confident statement?\n\nA4: Following the answer of Q3, we consider distance irrelevance as the defining signature of the Pizza algorithm,  while gradient symmetricity is additional evidence against Clock.\n\n> Q5: What’s the evidence for some solutions \"implement multiple, imperfect copies of either the Clock or Pizza algorithm in parallel.\"?\n\nA5: We agree that it is a bit confusing, but here we refer to the algorithms operating on a single circle as the clock or pizza algorithm, and they are imperfect (the pizza algorithm suffers from antipodal pairs; keeping only the first circle in Model A gives only 32.8% accuracy (L135)). We will clarify this in the revision.\n\n> Q6: Why do accompanied pizzas achieve almost perfect accuracy despite the failure mode of antipodal pairs?\n\nA6: Mechanically, the numbers are arranged differently in each circle so they have different antipodal pairs. In circle #2 of Fig 4, 0 and 10 are roughly antipodal, so circle #2 alone might not be able to get the input (0,10) correct, but we can see that they are relatively close in circle #1, and circle #1 is likely to provide the correct answer. In other words, the multiple copies of the pizza algorithm error correct each other. The accompanying pizzas can also be helpful (Appendix D) although the three circles alone are enough to get close to 100% accuracy.\n\n> Q7: Test the hypothesis that “accompanying pizzas are primarily used early in training”.\n\nA7: We observed the early emergence of a pattern similar to accompanying pizza in training runs (figure f in attached pdf) and removing that circle brings accuracy down from 99.7% to 97.9%. They are less helpful later in the network (removing accompanying pizzas in trained Model A only brings accuracy down to 99.7%).\n\n> Q8: Why use the terminology “symmetricity” rather than “symmetry”?\n\nA8: We think “symmetry” is mostly used as a binary adjective (something either poses symmetry or not), so we used the word “symmetricity” to emphasize the continuous aspect of our metric.\n"}]

Author rebuttal:
We would like to thank all reviewers for their helpful and constructive suggestions, which will greatly improve the final version of the paper. Besides individual responses, we want to summarize our responses/updates to reviewers’ common questions here. Reviews prompted us to try several additional experiments, which have led to fruitful discoveries:

### Distance irrelevance vs gradient symmetricity

Distance irrelevance is a rather surprising and defining feature of the pizza algorithm while the gradient symmetricity is mainly presented as supplementary evidence mostly used to rule out the Clock algorithm, which requires multiplying (transformed) inputs and hence has asymmetric gradients. We plotted the relationship between gradient symmetricity and gradient irrelevance for all the 1~4 layer 128-width models we trained, and we confirmed that low distance irrelevance (suggesting Pizza) almost always implies close to 1 gradient symmetricity (suggesting non-Clock) (figure a in the attached pdf).

### Accompanying pizzas are employed early in training

We observed the early emergence of accompanying pizza in training runs (figure f in the attached pdf; irrelevant principal components not displayed for space concerns). The model was trained 600 epochs at the time and reached 99.7% accuracy on the validation set (for reference, all the models we reported are trained for 20000 epochs). From the logit pattern, the first two principal components of the input embedding resemble the pizza algorithm, and the 13th and 14th principal components resemble the accompanying pizza. It is surprising that although the two components do not exactly resemble a circle due to the lack of training, the logit pattern is still clear and corresponds to the first circle. Removing this “accompanying pizza” brings the accuracy down to 97.9%.

### Projection for gradient symmetricity
We projected the gradients of the models to the principal components so as to match our description of algorithms on principal components. The less significant principal components account less for the correct functioning of the model and we observed their corresponding gradients concentrating near 0 (fig e left in the attached pdf). If we consider the raw unprojected gradients, the asymmetricity of a few model B’s principal components’ gradients is more pronounced as it now affects multiple raw gradient dimensions (fig e right in the attached pdf).

### Circularity with respect to layer, attention rate, and width
We computed the circular rate (circularity >= 99.5%) of models with respect to the number of layers, attention rate, and width (fig b and fig c in the attached pdf). We found out that the circular rate is higher for 1-layer models than multiple-layer models, and among 1-layer models circular rate is higher when the attention rate is closer to 0 or 1. Our explanation is that Pizza and Clock are two circular phases that are easiest to obtain at 1 layer and attention rate 0 or 1, correspondingly, so setups closer to these two phases are more likely to be drawn to them, resulting in similarly circular states.

### Sparsity and norm distribution
We plotted the relationship between attention rate, distance irrelevance, gradient symmetricity, and parameter L2 norm (figure g in the attached pdf). Here the parameter stands for all the trainable coefficients in the trained model. Besides clear concentration, we can see a slight increase in the mean L2 norm as attention rate and distance irrelevance increase. We also observed a slight increase in L2 norm from Model A (22.9) to Model B (24.8). Their parameter distributions are also different (figure d in the attached pdf). We believe this is a result of different attention rate setups. Namely, for Model A with constant attention (attention rate 0), the query and key matrices are ignored so they are optimized to near-0 values.



Besides empirical experiments, we also incorporate other suggestions from reviewers, including clarifying the family of pizza algorithms (Reviewer 7M7r), related work discussions (Reviewer wANB) and writing (Reviewer 7M7r, ZmSF, DtSC).

Meta-review: All reviewers agree that this paper is clearly presented and soundly supported, both mathematically and empirically. There is generally a high level of enthusiasm for accepting it. The particular contribution---showing that one of the main demonstrations of mechanistic interpretability is not robust---is timely, and should influence the thinking of people working in this area.
Decision: Accept (oral)