SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We performed the following set of experiments on transformers (see Appendix F.1 for architecture and training details). (1) One-layer transformers with width 128 and attention rate uniformly sampled in [0, 1] (Figure 7). (2) One-layer transformers with width log-uniformly sampled in [32, 512] and attention rate uniformly sampled in [0, 1] (Figure 7). (3) Transformers with 2 to 4 layers, width 128 and attention rate uniformly sampled in [0, 1] (Figure 11).
The Pizza and the Clock algorithms are the dominating algorithms with circular embeddings. For circular models, most observed models either have low gradient symmetry (corresponding to the Clock algorithm) or low distance irrelevance (corresponding to the Pizza algorithm).
Two-dimensional phase change observed for attention rate and layer width. For the fixed-width experiment, we observed a clear phase transition from the Pizza algorithm to the Clock algorithm (characterized by gradient symmetry and distance irrelevance). We also observe an almost linear phase boundary with regards to both attention rate and layer width. In other words, the attention rate transition point increases as the model gets wider.
Dominance of linear layers determines whether the Pizza or the Clock algorithm is preferred. For one-layer transformers, we study the transition point against the attention rate and the width:
- The Clock algorithm dominates when the attention rate is higher than the phase change point, and the Pizza algorithm dominates when the attention rate is lower than the point. Our explanation is: At a high attention rate, the attention mechanism is more prominent in the network, giving rise to the clock algorithm. At a low attention rate, the linear layers are more prominent, giving rise to the pizza algorithm.
- The phase change point gets higher when the model width increases. Our explanation is: When the model gets wider, the linear layers become more capable while the attention mechanism receives less benefit ( attentions remain scalars while outputs from linear layers become wider vectors). The linear layer therefore gets more prominence with a wider model.
Possibly hybrid algorithms between the Clock and the Pizza algorithms. The continuous phase change suggests the existence of networks that lie between the Clock and the Pizza algorithms. This is achievable by having some principal components acting as the Clock and some principal components acting as the Pizza.
Figure 7: Training results from 1-layer transformers. Each point in the plots represents a training run reaching circular embeddings and  $100\%$  validation accuracy. See Appendix C for additional plots. Top: Model width fixed to be 128. Bottom: Model width varies. The phase transition lines are calculated by logistic regression (classify the runs by whether gradient symmetricity  $>98\%$  and whether distance irrelevance  $< 0.6$ ).
Existence of non-circular algorithms. Although our presentation focuses on circular algorithms (i.e., whose embeddings are circular), we find non-circular algorithms (i.e., whose embeddings do not form a circle when projected onto any plane) to be present in neural networks. See Appendix B for preliminary findings. We find that deeper networks are more likely to form non-circular algorithms. We also observe the appearance of non-circular networks at low attention rates. Nevertheless, the Pizza algorithm continues to be observed (low distance irrelevance, high gradient symmetricity).
Mechanistic interpretability aims to mechanically understand neural networks by reverse engineering them [2, 3, 5, 4, 10, 11, 1, 12, 13, 14]. One can either look for patterns in weights and activations by studying single-neuron behavior (superposition [11], monosemantic neurons [15]), or study meaningful modules or circuits grouped by neurons [4, 14]. Mechanistic interpretability is closely related to training dynamics [8, 13, 1].
Learning mathematical tasks: Mathematical tasks provide useful benchmarks for neural network interpretability, since the tasks themselves are well understood. The setup could be learning from images [16, 17], with trainable embeddings [18], or with number as inputs [19, 5]. Beyond arithmetic relations, machine learning has been applied to learn other mathematical structures, including geometry [20], knot theory [21] and group theory [22].
Algorithmic phase transitions: Phase transitions are present in classical algorithms [23] and in deep learning [6, 24, 25]. Usually the phase transition means that the algorithmic performance sharply
changes when a parameter is varied (e.g., amount of data, network capacity etc). However, the phase transition studied in this paper is representational: both clock and pizza give perfect accuracy, but arrive at answers via different interal computations. These model-internal phase transitions are harder to study, but closer to corresponding phenomena in physical systems [24].
Algorithm learning in neural networks: Emergent abilities in deep neural networks, especially large language models, have recently attracted significant attention [26]. An ability is "emergent" if the performance on a subtask suddenly increases with growing model sizes, though such claims depend on the choice of metric [27]. It has been hypothesized that the emergence of specific capability in a model corresponds to the emergence of a modular circuit responsible for that capability, and that emergence of some model behaviors thus results from a sequence of quantized circuit discovery steps [5].
We have offered a closer look at recent findings that familiar algorithms arise in neural networks trained on specific algorithmic tasks. In modular arithmetic, we have shown that such algorithmic discoveries are not inevitable: in addition to the Clock algorithm reverse-engineered by [1], we find other algorithms (including a Pizza algorithm, and more complicated procedures) to be prevalent in trained models. These different algorithmic phases can be distinguished using a variety of new and existing interpretability techniques, including logit visualization, isolation of principle components in embedding space, and gradient-based measures of model symmetry. These techniques make it possible to automatically classify trained networks according to the algorithms they implement, and reveal algorithmic phase transitions in the space of model hyperparameters. Here we found specifically that the emergence of a Pizza or Clock algorithm depends on the relative strength of linear layers and attention outputs. We additionally showed that these algorithms are not implemented in isolation; instead, networks sometimes ensemble multiple copies of an algorithm in parallel. These results offer exciting new challenges for mechanistic interpretability: (1) How to find, classify, and interpret unfamiliar algorithms in a systematic way? (2) How to disentangle multiple, parallel algorithm implementations in the presence of ensembling?
Limitations We have focused on a single learning problem: modular addition. Even in this restricted domain, qualitatively different model behaviors emerge across architectures and seeds. Significant additional work is needed to scale these techniques to the even more complex models used in real-world tasks.
Broader Impact We believe interpretability techniques can play a crucial role in creating and improving safe AI systems. However, they may also be used to build more accurate systems, with the attendant risks inherent in all dual-use technologies. It is therefore necessary to exercise caution and responsible decision-making when deploying such techniques.
We would like to thank Mingyang Deng and anonymous reviewers for valuable and fruitful discussions and MIT SuperCloud for providing computation resources. ZL and MT are supported by the Foundational Questions Institute, the Rothberg Family Fund for Cognitive Science and IAIFI through NSF grant PHY-2019786. JA is supported by a gift from the OpenPhilanthropy Foundation.
[1] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023.
[2] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. https://distill.pub/2020/circuits/zoom-in.
[3] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain,
[20] Yang-Hui He. Machine-learning mathematical structures. International Journal of Data Science in the Mathematical Sciences, 1(01):23-47, 2023.
[21] Sergei Gukov, James Halverson, Fabian Ruehle, and Piotr Sulkowski. Learning to unknot. Machine Learning: Science and Technology, 2(2):025035, 2021.
[22] Alex Davies, Petar Velicković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomasev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, et al. Advancing mathematics by guiding human intuition with ai. Nature, 600(7887):70-74, 2021.
[23] Alexander K Hartmann and Martin Weigt. Phase transitions in combinatorial optimization problems: basics, algorithms and statistical mechanics. John Wiley & Sons, 2006.
[24] Lorenza Saitta, Attilio Giordana, and Antoine Cornuejols. Phase transitions in machine learning. Cambridge University Press, 2011.
[25] Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In International Conference on Machine Learning, pages 22965-23004. PMLR, 2023.
[26] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. Survey Certification.
[27] Ryan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304.15004, 2023.
[28] Michael Hassid, Hao Peng, Daniel Rotem, Jungo Kasai, Ivan Montero, Noah A. Smith, and Roy Schwartz. How much does attention actually attend? questioning the importance of attention in pretrained transformers. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1403–1416, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
A Mathematical Analysis and An Example of Pizza Algorithm
In the pizza algorithm, we have  $E_{ab} = \cos(w_k(a - b)/2) \cdot (\cos(w_k(a + b)/2), \sin(w_k(a + b)/2))$ , as  $\cos x + \cos y = \cos((x - y)/2)(2\cos((x + y)/2))$  and  $\sin x + \sin y = \cos((x - y)/2)(2\sin((x + y)/2))$ .
To get  $|\cos(w_k(a - b)/2)| (\cos(w_k(a + b)), \sin(w_k(a + b)))$ , we generalize this to  $|\cos(w_k(a - b)/2)| \cos(w_k(a + b - u))$  (the two given cases correspond to  $u = 0$  and  $u = \pi/2/w_k$ ).
$$
\begin{array}{l} \left| \left(\cos \left(w _ {k} u / 2\right), \sin \left(w _ {k} u / 2\right)\right) \cdot E _ {a b} \right| = \left| \cos \left(w _ {k} (a - b) / 2\right) \cos \left(w _ {k} (a + b - u) / 2\right) \right| \\ | (- \sin (w _ {k} u / 2), \cos (w _ {k} u / 2)) \cdot E _ {a b} | = | \cos (w _ {k} (a - b) / 2) \sin (w _ {k} (a + b - u) / 2) | \\ \end{array}
$$
thus their difference will be equal to
$$
| \cos (w _ {k} (a - b) / 2) | (| \cos (w _ {k} (a + b - u) / 2) | - | \sin (w _ {k} (a + b - u) / 2) |).
$$
Now notice  $|\cos(t)| - |\sin(t)| \approx \cos(2t)$  for any  $t \in \mathbb{R}$  (Figure 8), so the difference is approximately  $|\cos(w_k(a - b)/2)|\cos(w_k(a + b - u))$ .
Figure 8:  $|\cos(t)| - |\sin(t)|$  is approximately  $\cos(2t)$  for any  $t \in \mathbb{R}$
Plugging in  $u = 0$  and  $u = \pi /2 / w_{k}$  as mentioned, we get the following particular implementation of the pizza algorithm.
Algorithm: Pizza, Example
Step 1 On given input  $a$  and  $b$ , circularly embed them to two vectors on the circumference  $(\cos(w_k a), \sin(w_k a))$  and  $(\cos(w_k b), \sin(w_k b))$ .
Step 2 Compute:
$$
\begin{array}{l} \alpha = | \cos (w _ {k} a) + \cos (w _ {k} b) | / 2 - | \sin (w _ {k} a) + \sin (w _ {k} b) | / 2 \\ \approx | \cos (w _ {k} (a - b) / 2) | \cos (w _ {k} (a + b)) \\ \end{array}
$$
$$
\begin{array}{l} \beta = | \cos (w _ {k} a) + \cos (w _ {k} b) + \sin (w _ {k} a) + \sin (w _ {k} b) | / (2 \sqrt {2}) \\ - \left| \cos \left(w _ {k} a\right) + \cos \left(w _ {k} b\right) - \sin \left(w _ {k} a\right) - \sin \left(w _ {k} b\right) \right| / (2 \sqrt {2}) \\ = | \cos (w _ {k} a - \pi / 4) + \cos (w _ {k} b - \pi / 4) | / 2 - | \sin (w _ {k} a - \pi / 4) + \sin (w _ {k} b - \pi / 4) | / 2 \\ \approx | \cos (w _ {k} (a - b) / 2) | \cos (w _ {k} (a + b) - \pi / 2) = | \cos (w _ {k} (a - b) / 2) | \sin (w _ {k} (a + b)) \\ \end{array}
$$
Step 3 Output of this pizza is computed as a dot product.
$$
Q _ {a b c} ^ {\prime} = \alpha \cos \left(w _ {k} c\right) + \beta \sin \left(w _ {k} c\right) \approx | \cos \left(w _ {k} (a - b) / 2\right) | \cos \left(w _ {k} (a + b - c)\right)
$$
Similar circuits are observed in the wild, but instead of the above two-term approximation, a more complicated one is observed. See Appendix L for details.
The extra  $|\cos(w_k(a - b)/2)|$  term is not a coincidence. We can generalize our derivation as the following.
Lemma A.1. A symmetric function  $f(x,y)$  that is a linear combination of  $\cos x, \sin x, \cos y, \sin y^4$  can always be written as  $\cos((x - y)/2)g(x + y)$  for some function  $g$ .
Proof. Notice  $\cos x + \cos y = \cos ((x - y) / 2)(2\cos ((x + y) / 2))$  and  $\sin x + \sin y = \cos ((x - y) / 2)(2\sin ((x + y) / 2))$ , so  $\alpha (\cos x + \cos y) + \beta (\sin x + \sin y) = \cos ((x - y) / 2)(2\alpha \cos ((x + y) / 2) + 2\beta \sin ((x + y) / 2))$ .
This is why we consider the output pattern with the  $|\cos(w_k(a - b)/2)|$  terms rather than the actual computation circuits as the determinant feature of the pizza algorithm.
B Non-Circular algorithms
One thing that further complicates our experiment is the existence of non-circular embeddings. While only circular algorithms are reported in the previous works [8, 1], many non-circular embeddings are found in our experiments, e.g., 1D lines or 3D Lissajous-like curves, as shown in Figure 9. We leave the detailed analysis of these non-circular algorithms for future study. Since circular algorithms are our primary focus of study, we propose the following metric circularity to filter out non-circular algorithms. The metric reaches maximum 1 when the principal components aligns with cosine waves.
Figure 9: Visualization of the principal components of input embeddings for two trained non-circular models. Top: A line-like first principal component. Notice the re-arranged x axis (token id). Bottom: First three principal components forming a three-dimensional non-circular pattern. Each point represents the embedding of a token.
Definition B.1 (Circularity). For some network, suppose the  $l$ -th principal component of its input embeddings is  $v_{l,0}, v_{l,1}, \dots, v_{l,p-1}$ , define its circularity based on first four components as
$$
c = \frac {1}{4} \sum_ {l = 1} ^ {4} \left(\max  _ {k \in [ 1, 2, \dots , p - 1 ]} \left(\frac {2}{p \sum_ {j = 0} ^ {p - 1} v _ {l , j} ^ {2}} \left| \sum_ {j = 0} ^ {p - 1} v _ {l, j} e ^ {2 \pi i \cdot j k / p} \right| ^ {2}\right)\right)
$$
where  $\pmb{i}$  is the imaginary unit.  $c\in [0,1]$  by Fourier analysis.  $c = 1$  means first four components are Fourier waves.
Both Model A and Model B in Section 3 have a circularity around  $99.8\%$  and we consider models with circularity  $\geq 99.5\%$  circular.
C More Results from the Main Experiments
Here we provide Figure 7 with non-circular networks unfiltered (Figure 10). We can see more noise emerging in the plot. We also provide the training results from multi-layer transformers (Figure 11).
4The actual neural networks could be more complicated - even if our neural network is locally linear and symmetric, locally they could be asymmetric (e.g.  $|x| + |y|$  could locally be  $x - y$ ). Nevertheless, the pattern is observed in our trained networks.
Figure 10: Training results from 1-layer transformers. Each point in the plots represents a training run reaching  $100\%$  validation accuracy. Among all the trained 1-layer transformers,  $34.31\%$  are circular. Top: Model width fixed to be 128. Bottom: Model width varies.
D Pizzas Come in Pairs
Cautious readers might notice that the pizza algorithm is imperfect - for near antipodal points, the sum vector will have a very small norm and the result will be noise-sensitive. While the problem is partially elevated by the use of multiple circles instead of one, we also noticed another pattern emerged: accompanying pizzas.
The idea is the following: suppose the difference between adjacent points is  $2k \mod p$ , then the antipodal points have difference  $\pm k$ . Therefore, if we arrange a new circle with a difference  $k$  for adjacent points, we will get a pizza that works best for formerly antipodal points.
Algorithm: Accompanying Pizza
Step 1 Take  $w_{k}$  as of the accompanied pizza. On given input  $a$  and  $b$ , circularly embed them to two vectors on the circumference  $(\cos(2w_{k}a), \sin(2w_{k}a))$  and  $(\cos(2w_{k}b), \sin(2w_{k}b))$ .
Step 2 Compute the midpoint:
$$
s = \frac {1}{2} (\cos (2 w _ {k} a) + \cos (2 w _ {k} b), \sin (2 w _ {k} a) + \sin (2 w _ {k} b))
$$
$$
A _ {c} = - \left(\cos \left(w _ {k} c\right), \sin \left(w _ {k} c\right)\right) \cdot s
$$
This is exactly what we observed in Model A (Table 3, Figure 13). With the six circles (pizzas and accompanying pizzas) included in the embedding, Model A also gets  $100\%$  accuracy.
Figure 11: Training results from transformers with 2, 3 and 4 layers. Among all the trained transformers with 2, 3 and 4 layers,  $9.95\%$ ,  $11.55\%$  and  $6.08\%$  are circular, respectively.
Accompanied and Accompanying Pizza  $(p = 11)$
Figure 12: An Illustration on the Accompanying Pizza Algorithm
Circle	wk	AcFVE
#4 (accompanying #1)	2π/59·17	97.56%
#5 (accompanying #2)	2π/59·3	97.23%
#6 (accompanying #3)	2π/59·44	97.69%
Table 3: After isolating accompanying circles in the input embedding, fraction of variance explained (FVE) of all Model A's output logits by various formulas. Both model output logits and formula results' are normalized to mean 0 variance 1 before taking FVE. Accompanying and accompanied pizza have the same  $w_{k}$ .
Correct Logits of Model A after Accompanying Circle Isolation
Figure 13: Correct logits of Model A (Pizza) after circle isolation. Only accompanying pizzas are displayed. Notice the complementing logit pattern (Figure 4).
E Results in Other Linear Architectures
While this is not the primary focus of our paper, we also ran experiments on the following four different linear model setups (see Section F.2 for setup details).
- For all the models, we first encode input tokens  $(a,b)$  with a trainable embedding layer  $W_{E}$ :  $x_{1} = W_{E,a}$ ,  $x_{2} = W_{E,b}$  (positional embedding removed for simplicity).  $L_{1}, L_{2}, L_{3}$  are trainable linear layers. The outmost layers (commonly referred as unembed layers) have no biases and the other layers have biases included for generality.
- Model  $\alpha$ : calculate output logits as  $L_{2}(\mathrm{ReLU}(L_{1}(x_{1} + x_{2})))$
- Model  $\beta$ : calculate output logits as  $L_{3}(\mathrm{ReLU}(L_{2}(\mathrm{ReLU}(L_{1}(x_{1} + x_{2}))))$
- Model  $\gamma$ : calculate output logits as  $L_{3}(\mathrm{ReLU}(L_{2}(\mathrm{ReLU}(L_{1}(x_{1}) + L_{1}(x_{2}))))$
- Model  $\delta$ : calculate output logits as  $L_{2}(\mathrm{ReLU}(L_{1}([x_{1};x_{2}]))$ $([x_{1};x_{2}]$  stands for the concatenation of  $x_{1}$  and  $x_{2}$ )
The results are shown in Figure 14. Rather surprisingly, Model  $\alpha$ , Model  $\beta$  and Model  $\delta$  gave radically different results. Model  $\beta$  and Model  $\gamma$  are very similar, and in general they are more pizza-like
Figure 14: Training results from linear models. Each point in the first-row plots represents a training run. The second row are histograms for distance irrelevancy of each model type.
than Model  $\alpha$ , with lower distance irrelevancy and higher circularity. This could be explained by the addition of an extra linear layer.
However, Model  $\delta$  gave very different results from Model  $\alpha$  although they are both one-layer linear models. It is more likely to be non-circular and have very high distance irrelevancy in general. In other words, concatenating instead of adding embeddings yields radically different behaviors in one-layer linear model. This result, again, alarmed us the significance of induction biases in neural networks.
We also want to note that using different embeddings on two tokens of Model  $\alpha$  doesn't resolve the discrepancy. The following model
- Model  $\alpha'$ : calculate output logits as  $L_{2}(\mathrm{ReLU}(L_{1}(x_{1} + x_{2})))$  where  $x_{1} = W_{E,a}^{A}$ ,  $x_{2} = W_{E,b}^{B}$  on input  $(a, b)$  and  $W_{E}^{A}, W_{E}^{B}$  are different embedding layers.
gives roughly the same result as of Model  $\alpha$  (Figure 14, lower right corner).
Figure 15 shows the correct logits after circle isolation (Section 3.3) of a circular model from Model  $\beta$  implementing the pizza algorithm. Figure 16 shows the correct logits after circle isolation (Section 3.3) of a circular model from Model  $\delta$ . We can see the pattern is similar but different from the one of clock algorithm (Figure 5). We leave the study of such models to future work.
F Architecture and Training Details
F.1 Transformers
Here we describe our setup for the main experiments. See Appendix E and Appendix I for experiments on different setups.
Architecture We train bidirectional transformers (attention unmasked) to perform modular addition mod  $p$  where  $p = 59$ . To calculate  $(a + b) \mod p$ , the input is provided to the model as a sequence
Figure 15: Correct logits from Model  $\beta$  after circle isolation.
Figure 16: Correct logits from Model  $\delta$  after circle isolation.
of two tokens  $[a, b]$ . The output logit at the last token is considered as the output of the model. For a transformer with "width"  $d$ , the input embedding and the residue stream will be  $d$ -dimensional, 4 attention heads of  $\lfloor d / 4 \rfloor$  dimensions will be employed, and the MLP will be of  $4d$  hidden units. By default  $d = 128$  is chosen. ReLU is used as the activation function and layer normalization isn't applied. The post-softmax attention matrix is interpolated between an all-one matrix and original as
specified by the attention rate (Section 4.2). We want to point out that the setup of constant-attention transformers is also considered in the previous work [28].
Data Among all possible data points  $(p^2 = 3481$  of them), we randomly select  $80\%$  as training samples and  $20\%$  as validation samples. This choice (small  $p$  and high training data fraction) helps accelerating the training.
Training We used AdamW optimizer [29] with learning rate  $\gamma = 0.001$  and weight decay factor  $\beta = 2$ . We do not use minibatches and the shuffled training data is provided as a whole batch in every epoch. For each run, we start the training from scratch and train for 20,000 epochs. We removed the runs that did not reach  $100\%$  validation accuracy at the end of the training (majority of the runs reached  $100\%$ ).
F.2 Linear Models
Here we describe our setup for the linear model experiments (Appendix E).
Architecture We train several types of linear models to perform modular addition mod  $p$  where  $p = 59$ . The input embedding, residue stream and hidden layer are all  $d = 256$  dimensional. ReLU is used as the activation function. The actual structures of network types are specified in Appendix E.
Data & Training Same as in the previous section (Section F.1).
F.3 Computing Resources
A total of 226 GPU days of NVidia V100 is spent on this project, although we expect a replication would take significantly fewer resources.