Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks
Authors: Gavin McCracken
Gabriela Moisescu-Pareja
Vincent Létourneau
Doina Precup
Jonathan Love
Venue: NeurIPS 2025 poster ()
Keywords: mechanistic interpretability
interpretability
universality
modular addition
group theory
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We show that deep neural networks across architectures and training conditions all instantiate the same abstract algorithm for modular addition.

Abstract: We propose a testable universality hypothesis, asserting that seemingly disparate neural network solutions observed in the simple task of modular addition actually reflect a common abstract algorithm. While prior work interpreted variations in neuron-level representations as evidence for distinct algorithms, we demonstrate---through multi-level analyses spanning neurons, neuron clusters, and entire networks---that multilayer perceptrons and transformers universally implement the abstract algorithm we call the approximate Chinese Remainder Theorem. Crucially, we introduce approximate cosets and show that neurons activate exclusively on them. Furthermore, our theory works for deep neural networks (DNNs). It predicts that universally learned solutions in DNNs with trainable embeddings or more than one hidden layer require only $\mathcal{O}(\log n)$ features, a result we empirically confirm. This work thus provides the first theory‑backed interpretation of \textit{multilayer} networks solving modular addition. It advances generalizable interpretability and opens a testable universality hypothesis for group multiplication beyond modular addition.

Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks
Gavin McCracken Gabriela Moisescu-Pareja 12,, Vincent Létourneau Doina P2 Jonathan Love
Abstract
We propose a testable universality hypothesis, asserting that seemingly disparate neural network solutions observed in the simple task of modular addition unified under a common abstract algorithm. While prior work interpreted variations neuron-level representations as evidence for distinct algorithms, demonstrate, through multi-level analyses spanning neurons, neuron clusters, and entire net- works, that multilayer perceptrons and transformers universally implement the abstract algorithm we call the approximate Chinese Remainder Theorem. Crucially, introduce approximate cosets and show that neurons activate exclusively them. Furthermore, our theory works for deep neural networks (DNNs). It predicts that universally learned solutions in DNNs with trainable embeddings more than one hidden layer require only (llo((n) features, a result we empirically con- firm. This work thus provides the first theory-backer interpretation of multilayer networks solving modular addition. It advances generalizable interpretability and opens a testable universality hypothesis for group multiplication beyond modular addition.
Introduction
The universality hypothesis posits that neural networks learning related tasks converge to similar internal solutions and that shared principles will underlie their representations regardless of archi- tecture or initialization [+3] If true, it could provide a theoretical foundation for generalizing interpretability across diverse neural systems. Yet recent studies related tasks (modular addition 4 and permutations 8 have cast doubt on this hypothesis by presenting disjoint interpretations of what networks learn between the two tasks, and even within the sole task of modular addition.
We unify prior interpretations on modular addition. By presenting generalization of cosets-soe of elements with strict modular cquivalence- to approximate cosets containing elements that are "behaviorally similar," instead of equivalent, our results abstract away the low-level details of how weights compute activations. This lets show all previous interpretations | [0 are consistent under one common abstract algorithm we call the approximate Chinese Remainder Theorem (aCRT) (section 4.2). This abstraction reconciles the diversity in previously discovered mechanisms by interpreting them as different realizations of one algorithmic template. The main empirical results validate the breadth of our abstraction's accuracy across hyperparameters, architectures, and depth.
We open the universality hypothesis as a testable conjecture across all group-theoretic datasets. On modular addition (cyclic groups) we prove that all ReLU neurons learning sinusoidal functions activate only on approximate cosets or linear combinations of them (Theorem 4.4). giving direct construction that instantiations of the abstract aCRT are learned. As our approximate cosets generalize cosets, work finding coset circuits in networks trained on permuting lists (permutation groups) aligns with our results. This gives universality between datasets involving incredibly different groups
"qqual contribution. fgavin. mccracken, gabriela. moisescu- parescu-paeejaaa parepaareeaaaaaaeaa amail. 0tt.mmmmmmmm.m.......... wwm.mmlll....................m.m Mila MMeGill University Universsité de Montréal *Google DeepMind 'Leiden University
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We further the community's understanding of interpretations on modular addition. Assuming neurons each learn single frequency, we prove that networks exponentially reduce incorrect logit mass as more distinct frequencies are learned, concentrating the output near a Dirac on the correct answer. This recovers the theoretical result of [7], that 1-layer networks with neurons corresponding to each of the [ total frequencies that exist modulo have learned the maximum margin solution. Furthermore, a corollary predicts that deep neural networks (DNNs) have small margins between correct and incorrect logits unless (I)) features are learned. These predictions are empirically validated across architectures, training regimes and both prime and composite moduli, whereas past works focused prime moduli and representative networks from few seeds.
2
Related work
The first interpretability work in this domain aimed to understand the phenomenon of grokking [[]]. Nanda et al. [4] analyzed I-hidden layer transformers trained on modular addition, finding sinusoidal patterns of three to eight different frequencies in the weights, activations and attention. They showed that for each frequency, the embeddings placed inputs on circle, and the network performed angle addition on this circle. Since adding angles corresponds to multiplying complex numbers, this nonlinear operation was attributed to the attention mechanism. This was termed the Fourier Multiplication Algorithm (later Clock (5), and validated through ablation experiments.
Follow-up work proposed generalization of the Fourier Multiplication Algorithm called Group Composition via Representations (GCR) algorithm (T), which extended the idea of angle addition by treating group elements as linear operators and composing them to simulate group multiplication, aiming to unify mechanisms across group tasks. They applied GCR both modular addition and permutations (S,), as representative group settings. However, later work by Stander et al. [9] reverse- engineered models trained on S, under identical conditions and found that networks instead learn coset-based circuits. refuting the GCR universality claim. Furthermore, Zhong et al. S showed that in modular addition, training hyperparameters could induce learning qualitatively different mechanisms. They introduced the Pizza circuit, which contrasted with 4 ss Clock. They even showed that both clock and pizza circuits could coexist within the same network simultaneously, suggesting that even with fixed data, networks could converge to non-unique mechanisms.
Meanwhile, theoreticians explored idealized settings for modular addition: Gromov 6. constructed a solution for l-layer multilayer perceptrons (MLPs) with quadratic activations, showing local minimum exists where each neuron specializes a sinusoid of single frequency, and consequently each of the [ frequencies is represented by some neuron in the network. Later, it was proven that this solution maximizes the margin 77, while independently and simultaneously. work connected margin maximization to grokking in similar networks [12]
By this point, the universality hypothesis appeared untenable. No similarities were found between
groups and even on just modular addition, 1-layer MLPs found [ frequencies, 1-layer transformers learned substantially fewer, and changing hyperparameters resulted in learning different circuits (5)
Background
Modular addition, written as c = (a + b) mod n, gives the remainder e when the sum a + is divided by n. For example, 5 5 7 = 12, and 12 mod 12 = 0. We can think of this as wrapping numbers around a circle of size n, once we pass n, we start over at O. This arithmetic defines a structure known the cyclic group C, 1, .I, 1}. In Cp. every number is equivalent to its remainder class modulo n, denoted (mod n). e.g. 8 2 (mod 6). since 8 = 6 1 + 2. Modular arithmetic also supports multiplication: for instance, y 1 mod n when y is the modular inverse of x, which we denote x= These inverses exist when x and n are coprime.
Next, consider remainders mod m, where m divides n. This coarser division groups elements of Cn into cosets- of values that differ by multiples of m. For example, in Co, the elements can be grouped into three cosets mod 3: 30.3). {1, 4). and {2, 5). Each coset marks out equally spaced points on the circle-it is a cycle. They will play a key role in our work, as neurons (and neuron clusters) will perform coset-like computations. See Appendix A for more discussion on group theory.

---

The Chinese Remainder Theorem (CRT) provides way to simplify computations modulo by breaking them into smaller, independent computations. If factors into coprime integers (meaning they share no factors)
92 (j. then computing (a + b) mod n is equivalent to computing (a + b) mod qi for each qt. then reconstructing the original result. The CRT guarantees that the system of congruences (a - b) = m2 (mod q,) (for i = 1,.... k) has a unique solution modulo n. Each equation defines a coset, and the intersection of these cosets gives the final result. For example, suppose we want to find c (a + 6) mod 91 that satisfies: == 3 (mod 7), c 10 (mod 13). Each congruence defines a coset {3, 10, 17....) } and {10, 23, 36, ... } respectively. Their intersection is 10 and thus the unique solution is = 10. We hypothesize that networks learn structure similar the CRT decomposition to solve a + b (mod n))
1}; take 8 E Cn, 8 * O. We generate the following Cayley graph, call it T, by starting at g E Cn and making an edge to (g + s) mod n, then an edge to (g + 2s) mod n...... until cycle is made. For example, arrange 6 vertices in a circle. Using s = 1, step around the circle, connecting neighbors with an edge, generating 6-cycle. Using& = 2 connects every second vertex, generating two disconnected 3-cycles based on where you start. These 3-cycles are the cosets: {0, 2, 4]. {1,3, 5). Using # = gives three 2-cycles. See 11, 66 in panel 1 of Fig. 2 For an exposition including helpful visualizations involving how sine functions fit through cosets and approximate cosets, refer to A.2.I these visualizations may help readers understand cosets geometrically.
Cayley graphs are critical for understanding section 4.1
Recall C, {0,
Clock and Pizza Interpretations. Both circuits embed inputs a and b on a circle as E,, (cos(2 pp. sin (2 p.) E, = (cos(2wb/pp) sin sin(2rb/p)) Post-attention, clocks compute the angle sum: h(a, b) = [cos(2n 2m (a+bb//at))) (p), sin (b+))//p)) and pizzas compute vector mean of the embed- dings on the circle: h(a, b) = (E, + E,)
(2osa/aa))) cos p), sin(2ra/p)) + (2nbb///p))
Problem setting and setup. The task is modular addition: given inputs (a, predict c = a a b mod n. The dataset includes all n² input pairs. Inputs are either one-hot encoded or embedded via learned matrix with n rows and 128-dimensional vectors, resulting in concatenated input pairs E.). We train 1-4 layer MLPs and 1-4 layer transformers. We use the exact transformer architectures from [5], where attention is modulated by a coefficient a. Pizza (a 0.0) uses constant, uniform attention (all-ones matrix), while clock (a = 1.0) has learnable sigmoidal attention. Both models share the same structure: an embedding layer, one transformer block, and a 1-hidden layer MLP. We follow prior work in applying 1.2 regularization, shown to encourage generalization [1]] Also, we apply discrete Fourier transforms (DFT) to analyze the frequencies learned by each neuron.
4 Theoretical and empirical results
An intuitive overview of the mathematical details in section is: neurons learn sinusoidal functions and we explain how to identify which Cayley graph a neuron understands; the Cayley graphs for the eyclic group circle graphs; these Cayley graphs generated by connecting every step size dh vertex on the circle; approximate cosets are sets of vertices that are close on the graph generated by d.
4.1 Simple neurons as (approximate) coset detectors
The simple neuron model. The mathematics in this paper assumes the simple newron model: neurons approximate or specialize to sinusoidal functions of frequency f. The primary empirical results (section 5) validate this assumption across architectures, hyperparameters, random seeds, moduli and depth (Figs. [4-9]). This model is derived by generalizing the sinusoidal model for neurons from 6 7 13 as they assume one-hot encoded inputs to the network, to a model fitting both one-hot encoded architectures and those used in practice (having trainable embeddings (1))) For a + mod n, the inputs a and b are encoded as embeddings A
and B = [Bo.... B, Dn-1] E RD. Let wt, V) be the dot
where A,, B, E Rd. The output logits are D = [Do.... Bin....D_il product of all values from U with edge weights going to V. Then simple neuron N has frequency and phase shifts for A and B: f, SAS SB E Cn,, and positive real number a such that for each k E Cn we have
w(A,, cos 2-f(i-sa), n w w(B,, N) = cOS 2т5()-8B) w(N,Da D&) = cos 2r sa 88-)
n
3

---

If training makes neurons converge to simple neurons, then their frequencies can be normalized to be 1 by applying an isomorphism (Def. 4.2). Consequently, qualitative comparisons between neurons of different frequencies are now possible (Fig. D
Definition 4.1 (Step size). Let the step size be geddffm) inverse used.
f ) (mod gealf.n)) where the modular
Definition 4.2 (Remapping: frequency normalization). Consider the function h(x)) = f with frequency f. We define new function g, allowing us to perform something analogous
change of variables using the step size d: g(d x) = h(a) - g(x) = h(d-1.... x).
preactivation value
MLP (Original) MLP (Remapped)
Pizza (Original) Pizza (Remapped) Clock (Original) Clock (Remapped)
-2
25
50
50
input a
input a
input a
input a
input a
input a
Figure 1: Preactivation values over fixing b= 5 on c (a+b) b) mod 59 of a neuron from an MLP, pizza and clock show qualitative equivalence after remapping (Def. (Def. they all have frequency 1. To see a neuron that is not qualitatively equivalent (corresponding to poor local minima), please see Appendix G.4, particularly Fig. 28
Approximate cosets. The CRT relies cosety. A neuron with frequency f will only take values on a coset when scd(f,n) > 1. Since our experiments suggest neural networks approximate the CRT's decomposition when the learned frequencies don't 't factor the modulus, we instead generalize cosets from requiring a strict equivalence among elements to approximate cosets. These require elements to be similar, which means the shortest path distance their Cayley graph is small. Later, theorem 4.4 gives that all neuron activations (ReLU > 0) in all layers occur on approximate cosets.
Let f E C.. Recall Def. 4.1 d determines how we step around the circle Cn. There are n' = ged(f.n) distinct positions reachable in this way. These positions form cycle C.', which is smaller (or equal) copy of C.. d E C, is the step size in Cz' Generate r, the Cayley graph of C,2' using d. We now introduce approximate cosets (approximate equivalence classes) using the minimum path distance between vertices in T. For distances < k I n and 1 I 12 S n, the approximate coset is the set of vertices on the path from c (dk:) ) to c + (dk2), stepping by d. If god(S,n) > 1: elements in the same coset as are distance 0 from each other as they are the same vertex on the Cayley graph, adjacent vertices are distance 1, etc (see panel Fig 2). If ged(J, n) 1: I has one element with distance 0: e (see panel 4 Fig 2 Thus, approximate cosets are more general than cosets.
Definition 4.3 (Approximate cosets). Let 1 < k1 < and 1 < h2 S n. We call the set fc 2d,... .d.....c.. k2d} an approximate coset.
kjd..... C 2d, c d, c, d,
Theorem 4.4. Simple neurons in layer I activate (ReLU > 0) on an approximate coset containing the correct answer e, by concentrating their preactivations approximate cosets that contain a and by all neurons in later hidden layers activate on linear combinations of approximate cosets.
See Appendix A.2.I for examples further illustrating the definitions introduced in this section.
4.2 The abstract approximate Chinese Remainder Theorem
By defining approximate cosets, and proving Theorem 4.4 a straightforward construction for the abstract algorithm being instantiated by neural networks can now be given. An abstract algorithm is general template or high-level strategy for problem. It outlines the steps be followed but leaves room for different low-level implementations. For example, the classic breadth- first search algorithm traverses graph by visiting all neighbors of a node, then all neighbors of those nodes, and so on. While the abstract idea is the same, the implementation details can vary: one version may use a linked list, while another may use a queue. Thus, the compiled machine code can differ significantly, like how the networks features can be computed by very different pizza or clock circuits.

---

covna 2
00
sarapped conl2 2011
griegrghh. mc
namaaped castE Stia EE1
2.5
00000000
activstion
36 activsatin
sctivatton
Sive
400
60
Figure 2: Visualizing how neurons learn approximate coset structure. Panel shows the circle graph on 66 elements generated by starting
0) and taking 6 steps of +11, creating the 66 TI = 6 cosets , {a (mod 6) = 5). The graph distance to each of points {a (mod 6) = 0}, {a (mod 6) = .,,...,
coset from coset {a (mod 6) 0} (in yellow) is given. 2: the newron learned cos( [1(2)))) ); the distances annotated on points follow from 1. This neuron only activates (ReLU > 0) on distances 0 and 1. 3: remapping shows all members of each coset collapse into an equivalence class. Panels 4-6 show the circle graph on 67 elements generated by +11; since ged(11, = 1, the neuron can't 't activate the same strength proportional to distances on the Cayley graph. All elements the neuron takes positive values on are an approximate coset, shown in bright viridis colors decaying with distance. Note: each neuron "divides" the Cayley graph it activates on approximately in half.
equivalent points (cosets) and instead activates with strengths
Remark 4.5. The sinusoidal neuron based CRT. The CRT decomposes the modular system (a + b) mod n into (n)g(()) modular subsystems, which follows from a number having at most (llog(n) prime factors. The CRT solves the original modular system by intersecting the cosets that the solution of each subsystem belongs to. Suppose the CRT can be used to decompose (a + b) mod n. Then, a sinusoidal neuron based CRT is constructed with (log(n)) unique frequencies with ged(f, n) = f. Make y sinusoids, one for each coset (O(n) neurons), that are only positive on one of the cosets {a b (mod }}} using the y-intercept (neuron bias) the neuron only activates if the answer is in the coset. The argmax of the linear combination of these neurons to the output logits selects the correct answer deterministically.
instantiates is simple. Remark 4.5 assumes "the CRT can decompose (a + b) mod n² into coset structure, but cosets are a subset of approximate cosets, making cosets a specific implementation under an abstract template. Furthermore, Theorem 4.7 addresses the approximate cosets case (f does not divide n.) in section 4.3 gives that (log((n) (n)) randomly selected frequencies are enough to get reasonable margins between correct and incorrect logits, matching the number of frequencies required by the CRT.
Armed with Theorem 4.4 deriving an abstract algorithm that Remark 4.5
Abstract algorithm 4.6. The minimal template: the abstract aCRT. Take (log(n)) random fre- quencies and for each frequency generate sinusoidal neurons with that frequency and set their phases such that they pick out different approximate cosets that the answer (a + b) mod n is in.
Note, Alg. i.6 more general than the CRT and Remark 4.5 by handling cases where neurons learn frequencies with greatest common divisor (GCD) with the modulus (approximate cosets). Two things are left to show: the (log(n)) frequency bound in section 4.3 and that DNNs learn solutions that are realizations of Algorithm 4.6 The latter is exhaustively validated in section 5 finding all architectures are well abstracted by approximate cosets.
4.3 How many frequencies are needed to instantiate the aCRT with simple neurons?
We now present Theorem 4.7 which assumes that sinusoids are learned by the neurons, matching the simple neuron model. It predicts that DNNs push incorrect logit values down exponentially as the number of frequencies leamned in the network increases. Analogously to Morwani et al. (7]. it gives that the maximum margin solution requires all frequencies to be learned, but also yields additional information about the size of margins network can acquire with fewer frequencies. A corollary gives that (log(n)) frequencies are sufficient to get margins larger than ((log(n)..
Let be the number of output logits (matching the modulus), let be the number of distinct frequencies learned by the network and m be the maximum output logit value across the dataset. Fix two parameters: 0 < 8 < 1, controlling the required margin between the correct and incorrect
5

---

outputs: 0 < < 1, the target probability of success. We model the neural network's 's output at logit
2 (k - --))) where each frequency fe is drawn uniformly at random .. " 1 We seek conditions on m so that, with probability least p. the value h(k)) is well-separated from the maximum m for all incorrect outputs k + i + mod
k by ((kk) = Σx=1 cos from (1......,
Under the simple neuron model, the following result holds.
Theorem 4.7. Suppose the integer number of distinct frequencies m and reals < p. satisfy the inequality
m > 2log. n - -2loge 2p)
(=/6 -
Then, with probability at least for all ls + iij mod n, we have m' - l((k) > 8ml' See Appendix Cfor proof. Note, we want large 6, i.e. large margins.
Corollary 4.8. Learning (log(n)) distinct frequencies gives logit margin (l(log(n)): after softmax, incorrect classes receive most -2() probability mass.
Note: it is still possible for networks to learn solutions utilizing single frequency! Indeed, these solutions have poor margins, making them poor local minima. Unsurprisingly, they are rarely learned and only show up at the edge of the grid search returned by hyperparameter tuning. We empirically validate corollary 4.8 in Fig. by varying the moduli over multiple orders of magnitude including moduli that are prime, composite numbers, highly composite numbers and powers of only 2 (e.g. 64, 256, etc.). The samples are such that the R of fitting them with logarithmic functions is very high, empirically verifying the prediction of Corollary 4.8) that indeed O (log(n)) frequencies are reasonable. As the data contains both prime and composite moduli, suggests moduli have little effect on what the network ultimately learns, though if frequency divides the modulus it can be the case that fewer neurons of that frequency exist (see Appendix, G.7
# frequencies found moduli and architectures
Aug frequencies
10.0
0.600
5.0
cm0.728, b-0.12
2.5-
10a
10a
10a
1000
2000
3000
4000
5000
mod (log scale)
Figure 3: The number of frequencies found in clocks, pizzas, and MLPs as the modulus increases. We plot the data on logarithmic and linear axes, showing logarithmic fits have very high R² scores.
5 Empirical results supporting the simple neuron model and approximate coset abstraction
The details for the experimental results of this section be found in Appendix
One-hot encoded MIPs. Previous work shows n
types of neurons are found, each specializing to sinu-
2
Average trequencies trained model
35
soid with one frequency with hidden layer [6] 77. We
G
30
ove hot tawes
show that adding either depth, or a trainable embedding matrix for inputs, causes the network to transition from learning 2 types of neurons to much fewer types in Fig. The presence of the trainable embeddings why the models trained in 4 g [ were observed to learn handfuls of frequencies (3-7) instead of ! frequencies, despite being one- me-hhidden-laye layer models.
@ 25
one-hot layer
ans-hot layer
20
15
#00
avg
5
Figure 4: Average number of frequencies found in various MLPs over moduli 59-66.
The neural pre-activations l-layer networks. In the vast majority of cases, neuron pre-activations in all l-layer architectures can be approximated well by degree 1 sine functions with frequency f. This is because the preactivations of most neurons

---

"simple", meaning that they have frequency equal to once remapped (Definition 4.2 (Fig D.. This is despite the presence of secondary frequencies in smaller width architectures. These occur less often as the width of the layers is increased 5 which shows neurons with secondary frequencies on the left, and the R² of fitting a single sine, or a sum of two sines with different frequencies, through the preactivations on the right. Thus, as the width is scaled, approximating neurons as simple neurons (1 sine is fit through their preactivations) becomes better. Note: at widths excessive for this task (=2048 neurons) it rarely occurs, but it's the case that few neurons can learn sinusoids with frequency / (App. G.4 Fig. 28 While these still satisfy our definition of approximate cosets and Theorem 4.4
covers their existence, they break previous theoretical models assuming integer frequency 6 7
Clock cluster 1-2 preactivations
Clock cluster F-2 DFT
4p
preacivatoon voliiCocce prescivation
0.5
Average model's RP vs. width fitting a sum of 1 or sines (mod 59)
8
0.98
-mlp sine
reu-on
0.96
- piiza sine
20
40
10
20
4 0.94
-lock
Input
frequency
0.92
pizza sines
SClock cluster f-10 preatthations
cluster f-10 DFT
0 0.9
clock 2 sines
raures 5
0.88
0.5
0.86
2k
4k
6k
8k
10k
12K
14k
10
20
30
40
19 15 20
25
Tearon 78
width
frequency
Figure 5: Left: cluster preactivations from a clock with small but present secondary spikes in the DFT. Right: as the width of the models is increased, the presence of the secondary spikes fades, 2 sines is fitting a sum of 2 different f sines, allowing the inclusion of a secondary peak in the fit.
Depth's effect neural preactivations. The first layer is fit with very high R in models of all depths, but adding layers introduces a caveat in the transformer architectures. In MLPs, it is possible fit every neuron in every layer and maintain 100% test accuracy by assuming the neural preactivations are of the form (a, b) = sin(fa + Φ) + sin( fbb + Φ). where & is the phase shift (Fig. 6). In transformers however, this only works with high R² for the first layer. The reason is that the form of the logits f(a, b) = cos(f(a + b - c)), described by [4] is second-order (quadratic) sinusoid and starts appear layers after the 1st layer, but before the logits. Indeed, we find that in deeper networks, neurons after the first layer can be either simple, co + c)), or linear combination in superposition of these two forms. Thus, fitting just cos(a + b) or just cos(a) + cos(l) is not sufficient to maintain 100% accuracy. To see this, see Fig. 8 which shows the percentage of activations that have their best R² achieved by fitting just order one sinusoids in and in 2-hidden layer transformer. Thus, we fit linear combinations of (cos( f (a + +))) + (cos(fa) + cos( through neurons in layers after in Fig. 7 Note, MIPs can fit well using only first-order sinusoids.
Furthermore, we could see a preference for learning precise cosets (should they exist) over approxi- mate cosets as this could reduce approximation error in DNNs. We explore this in Fig. 9 showing that for n = 66, all architectures present preference for learning precise cosets. This is strong evidence supporting the abstract aCRT algorithm as it implies DNNs try to learn CRT-like behaviour.
Our results show that in all architectures, layer I uses only simple neurons, with other layers still utilizing them, implying Algorithm 4.6 instantiated. Furthermore, it follows from this that we' 'ee shown that all neurons downstream of layer 1 activate on linear combinations of approximate cosets. Combined with the results of [9] that the GCR algorithm circuits are learned in networks learning group multiplication in the permutation group, we open the universality hypothesis group multiplication datasets Conjecture 5.1
[ is not universal and instead coset
Conjecture 5.1.
The universality in structures learned by DNNs trained fu group maltiplication will be found as coset circuits, and more generally as approximate coset circuits computing features. DNNs will make use of these circuits in serriid-and-conuur---eee manner to achieve logarithmic efficiency.
At this point, we have shown that our definition of approximate cosets functions as sufficient abstraction for simplifying the representations learned by networks of various architectures.
7

---

of
of
14-4
Hit test accuracy 100% C/)
12
14-4
1e-4
1e:5
Te-5
100% ttt socuracy 100% [+]
Te-d
Netther accurary (K0
1e-7
Layer ouu of3
Layer of R
Layer R2
Learning FO0e
1e-3
1e-4
12 17
1er5
1e-6
Layer Luyer1 4
Layer 2 oF
Layer 3 out 4
Layer ou 4
1e-3
Te-3
1e-8
5-91
S-11
Figure 6: R of fitting each neuron in layer simple neuron and fitting sum of sines of each frequency in layer through layers 2-4 for 1,2,3 and 4 layer MLPs. The large volume of green checkmarks implies replacing neurons with simple neurons does not decrease the network's 's accuracy implying that our abstraction is robust to changes in training conditions and architectures.
6 Discussion and Conclusion
We argue that approximate cosets are critical in all architectures because they instantiate the aCRT. We support this both with empirical evidence across a large range of hyperparameters, seeds and different moduli, and theorems. Approximate cosets provide DNNs with structures analogous to the cosets the CRT operates on. The CRT uses (log(n)) modular subsystems, and Corollary 4.8 gives that DNNs need (log(n)) unique frequencies to analogously induce modular subsystems. Furthermore, approximate cosets shed light on how the network learns second order sine functions with ReLU activations (activating on the coset of e requires understanding coset membership of + b). a previously unexplained result in Nanda et al. [4] and Chughtai et al. [10] As the proof for theorem 4.4 shows, a neuron learns to fire strongly on the coset that e in, by understanding which cosets a and 6 are in. The conclusion is that in abstracting away the small details in how weights in different architectures explicitly compute modular addition, we unify previous interpretations under the abstract aCRT (algorithm 4.6).
Our initial hypothesis was that neural networks trained on modular addition with composite moduli would learn the Chinese Remainder Theorem (CRT), leveraging coset structure where it naturally applies. Early empirical results supported this view, revealing preference for frequencies that cleanly divide the nuggels--sugeesting alignment with exact coset structure. However, we observed that networks often learned only a single such frequency, alongside others that did not correspond to precise cosets. This prompted further investigation. Upon examining both qualitative and quantita- tive behavior, we found no meaningful distinction between neurons associated with coset-aligned frequencies and those that were not. This observation led to critical insight: networks may be implementing an algorithmic template resembling the CRT even when its mathematical prerequisites are not strictly satisfied. This realization motivated the formulation of the abstract approximate CRT (aCRT), generalizing the role of cosets to approximate cosets as a unifying structure.
Due to modular addition being multiplication in the cyclic group and our approximate cosets general- izing cosets, our interpretation establishes universality between cyclic and permutation groups This follows from the result that coset circuits are learned in networks trained on the permutation group [9]. Thus, we establish universality between very different tasks, related only by the fact they are both groups. This allows us open the testable universality hypothesis group multiplication datasets.
8

---

Pizza and Clock R across training hyperparameters for 1, 2, 3, 4 layer models
Pizza: 1"11
Pizza: 12
Pizza: 2
Pizza: 13/3
Pizza: 73
Pizza: 3
Pizza: 14
10
regularizaaion
10
R2
10
10*-
102.
19
104
Pizza: 2d
Pizza: 3
Pizza: /4
Clock: 19/1
Clock: 14/2
Clock: 2
04
12 regultrriatoon
10
10
10.
both fi & test 100le
10*
test 100% fir e 100%
107.
6%
14
100%
10*-
05
Clock: 1*/3
Clack: 2
Clock: 3
Clock: 14/4
Clock: 2
Clock:
Clock: 4'
L2 regulataatio
10
10:1.
10-.
16
03
10
FM
10?.
64
10
Learning rate
Learning rate
Learnnng rate
Learning rate
Learning rate
Learning rate
Learning rate
Figure 7: R of fitting order 1 sines through neurons in layer 1, then fitting a sum of length equal to the number of unique frequencies in layer of order one or order two sines for layers 2, 3, 4. The large volume of green checkmarks tells us that our abstraction doesn't affect the model 's accuracy and is robust to varying training conditions and architectures.
% Neurons with their best Rs order sinusoids (2048 neurons pizza vs clock)
Layer (pizza)
Layer 1 (clock)
Leyer (pizza)
Layer 2 (clock)
10-
16"
10
101
10*
10
10"
10"
10
La
XXX
J/4
XXX
XXXXX
9.
501
Learning
Learning
Figure 8: The percent of neurons with their best fit involving only first-order sines. The result can be interpreted as neurons existing that are operating on first-order sinusoids in deep layers. With ideal hyperparameters, almost 100% of neurons in layer can have their best fits coming from order one sinusoids, though this occupies very little volume in the hyperparameter grid. This plot shows that the second-order sinusoidal fits of 4 are more experessiveness than necessary in the first layer, and not optimal.
6.1 Toward uncovering the nature of universality
It's important to note that our work doesn't fully illuminate how deep neural networks are universally learning modular addition. We propose the following definition, being abstract universality, which attempts to capture what is meant by the universality hypothesis which asserts that "models learn similar features and circuits across different models when trained on similar tasks." [8] Our issue with the quoted statement, is "what does similar" mean? The work of Zhong et al. [5] claimed to find two disjoint and disparate circuits, being a clock, or a pizza, but our work shows these two circuits can be unified under one abstraction. It is therefore the case that these two circuits could be different implementations of one abstract wwwiid-anddconquuu algorithm. By abstract, we mean an object that can't be directly instantiated (e.g. abstract class in computer programming).
With that said, we propose the idea of abstract universality. DNNs trained on similar data learn different implementations of one abstract algorithmic class i.e. one algorithmic strategy.
9

---

Frequency counts in 1 & 4 layer models mod 66 = (2x3x11)
Pizza: 1/1
Pizza: 1/4
Pizza: 2/4
Pizza: 3/4
Pizza: 4/4
count count
2000
1000
Clock: 1/1
Clock: 1/4
Clock: 2/4
Clock: 3/4
Clock: 4/4
2000
1000
0
10
20
30
10 20
30
10 20
30
10
20
30
10 20
30
Frequency
Frequency
Frequency
Frequency
Frequency
Figure 9: Histograms of the number of times each frequency was learned while training on (a + 6)
mod 66. Note: attention in the clock models results in learning frequency 6 cosets in layer 1/4.
6.2 Limitations and future work
Finding cosets key to learning modular addition aligns with Stander et al. 99's results on permutation groups. A group with the least structure cyclic group of prime order, but Cayley's s theorem gives all groups are subgroups of the permutation group the king of group structure. Despite these groups occupying upper and lower extremes of structure, our findings yield universality in the structure DNNs learn on these two disjoint tasks. This is why we believe it's likely that networks learning every group multiplication will utilize structures that can be viewed under a more general refinition-aproxiinie
cosets, involving distances Cayley graphs. A core limitation of our work is that we do not explore the groups between these two extremes. We believe that this is a promising avenue for future work to explore, with successful testing offering the potential to demonstrate that the universality hypothesis is true in the very diverse space of all possible group multiplications.
While we are the first to point out how to cause a phase transition from (()) to (log(n)) learned frequencies via depth or a trainable embedding (Fig. 4). we don't 't know why it occurs. Theorem 4.7 says this solution should have great margins, and it does 77] This gives two directions for future work. Why this happens: are the training dynamics wildly different? What causes it: does the network learn something significantly different that both we, and prior work, fail to see? The fact these are open questions-on a math task that's 's become very well understood over 1500 years since inception-suggeat. answer how many neurons are needed per learned frequency; giving a trivial lower bound of (1) neurons. Empirical results are in Appendix G.7] but don't 't give an obvious direction for proving bounds. We believe answering this is necessary for the interpretability community to gain a full understanding of task. This may be entire paper in itself. It looks non-trivial and requires careful arguments with non-linear ReLU activations.
urgent need for new interpretability tools. Finally, Theorem 4.7 doesn't 't
6.2.1 Roadmap to stress testing the universality hypothesis
There's opportunity to learn a great deal more about the nature of the solutions DNNs learn via reverse engineering models trained on group multiplications (Conjecture In particular, we believe the logarithmic efficiency to be a potentially remarkable result and hope for its generality. Due to our study being scoped to modular addition (cyelic group multiplication), and the other sample point finding cosets being permutation groups (2] we propose studies on the following tasks to fill in the blanks to compose a more general theory and understanding.
Approximate cosets were not observed in Stander et al. I9'ss study on the non-commutative permutation group multiplication. Are approximate cosets unique to commutative tasks or just cyclic groups? This can be resolved by studying the non-commutative dihedral group. 2. Abelian group multiplication is commutative; will networks always learn to divide and conquer via approximate cosets on all Abelian groups? 3. As cyclic and dihedral groups are both constructed from cyclic groups as the base structure, can approximate cosets be found on DNNs trained on other groups with their underlying simple group being not the cyclic group? We suspect that once these three questions are answered, it will look as though on all tasks DNNs learn logarithmic efficiency (of the order of the group). If this is true (Conjecture 5.T), we believe the result will serve to help explain the efficiency of DNNs from a new perspective.
strategy with
10

---

Acknowledgements
Sihui Wei and Harley Wiltzer were of great help during the preparation of this manuseript, serving as pseudo-reviewers. We thank Arthur Ayestas-Hilgert, Colin Daniels, Darshil Doshi, and Étienne -errey for helpful discussions. This work was supported in part by compute resources from Mila.
References
[1] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3))e00024-000 0011, 2020.
[2] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning:
Do different neural networks learn the same representations? arXiv preprint arXiv:1511007543, 2015.
[3] Minyoung Huh. Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint wxXi:22405.09998 2024.
[4] Neel Nanda, Lawrence Chan, Tom Lieherum, Jess Smith, and Jacob Steinhardt. Progress mea- sures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023. URL httptt:/oopeeereeieeee wwtp::/wweeeooooooooeeeeeeeeaaae wtt.immddddxxPPPppexomm
[5] Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. In Thirrt--eventh Conference on Neural Information Processing Systems, 2023. URLittps///ppeneeeieer.re www.:/rrwwww.e.eeeeormmmm.aae
S5wmbQc1We
[6] Andrey Gromov. Grokking modular arithmetic. arXiv preprint arXiv:2301.02677 2023.
[7] Depen Morwani, Benjamin I.. Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham M. Kakade. Feature emergence via margin maximization: case studies in algebraic tasks. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview http://rrrmmi.1110000000005050
[8] Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. In International Conference on Machine Learning, pages 6243-6267. PMLR, 2023.
[9] Dashiell Stander, Qinan Yu, Honglu Fan, and Stella Biderman. Grokking group multiplication with cosets. In Forty-first International Conference Machine Learning, 2024.
[10] Bilal Chughtai, Lawrence Chan, and Neel Nanda. Neural networks learn representation theory: Reverse engineering how networks perform group operations. In ICLR 2023 Workshop on Physics Machine Learning. 2023.
[11] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting small algorithmic datasets, 2022. URL https://arxiv
htggabb/2201..222
[12] Mohamad Amin Mohamadi, Zhiyuan Li, Lei Wu, and Danica Sutherland. Grokking modular arithmetic can be explained by margin maximization. In NewrIPS 2023 Workshop on Math- ematics of Modern Machine Learning, 2023. URL https:://penneewww..etttt wwt.:/iwwwwww...eeeeeeeeeeee id= QPMECLnIqE)
[13] Darshil Doshi, Aritra Das, Tianyu He, and Andrey Gromov. To grok or not to grok: Disen- tangling generalization and memorization corrupted algorithmic datasets. arXiv preprint arXiv:2310. 13061, 2023.
[14] Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi
11

---

Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper. Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger. Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath. Open problems in mechanistic interpretability. 2025. htl.httss://arrt org/ hts/2501.16496
[15] Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020, doi: 10.23915/ distill.00024. http://diisillpuuu/20/iiiiiioiii
[16] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://ranssfrmmer ciretttsrrrrrrreeeeeeeeeee putpits..ppp2211111 httaa///////aameoo////eeeee
[17] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac
Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. http::/tansfoommrrciiiiussspppp pwwp://///mm.ooettttaaaaiinnnnna heads/index.html
[18] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zae Hatfield- -Doddd, Robert Lasenby, Dawn Drain, Carol Chen, al. Toy models of superposition. arXiv preprint arXiv:2209. (00):22299.06,,, 2022.
[19] Wilson Wu, Louis Jaburi, Jacob Drori, and Jason Gross. Towards a unified and verified understanding of group-operation networks. arXiv preprint
2024.
[20] Marr and T Poggio. From understanding computation to understanding neural circuitry. Neuroscience Reseurch Program Bulletin, 178848470-488 1979.
[21] Jessica Hamrick and Shakir Mohamed. Levels of analysis for machine learning. arXiv preprint arXiv:2004.05107 2020.
[22] Zhonghao He, Jascha Achterberg, Katie Collins, Kevin Nejad, Danyal Akarca, Yinzhu Yang, Wes Gurnee, Ilia Sucholutsky, Yuhan Tang, Rebeca lanov, et al. Multilevel interpretability of artificial neural networks: Leveraging framework and methods from neuroscience. arXiv preprint arXiv:2408. 12664, 2024.
[23] Martina G. Vilas, Federico Adolfi, David Poeppel, and Gemma Roig. Position: An inner interpretability framework for Al inspired by lessons from cognitive neuroscience. In Forty-firtt International Conference on Machine Learning, 2024. URL https://openreviiew net/ forum?1d=666mm/ii
[24] Emily Wenger, Mingjie Chen, François Charton, and Kristin Lauter. Salsa: Attacking lattice cryptography with transformers, 2023. URL https://axxxii.ccoo org/ abs//207 wttp:/i/...rrg////220////// 04785
[25] Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651-346633 2022.
[26] Neil Mallinar, Daniel Beaglehole, Libin Zhu, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Emergence in non nen-ne models: grokking modular arithmetic via average gradient outer product, 2024. http:ttt/////aii..ooo org/ abs/2407 20199
[27] Hiroki Furuta, Gouki Minegishi, Yusuke Iwasawa, and Yutaka Matsuo. Towards empirical
interpretation of internal circuits and properties in grokked transformers on modular polynomials, 2024. URL httpss::.... //trxivvvvv////////////a///aaaal org/
16726.
12

---

[28] Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon S. Du, Jason D. Lee, and Wei Hu. Dichotomy of early and late phase implicit biases can provably induce grokking, 2024. URL https 7/arxiv. org/ porg/abs. abs//311....././//////.///11111/ 2311. 18817
[29] Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics, 2024. URL htttp::/aaxxv.org/bbbb...
06110
[30] Darshil Doshi, Tianyu He, Aritra Das, and Andrey Gromov. Grokking modular polynomials. arXiv preprini arXiv:2406.03495, 2024.
[31] Gavin McCracken, Arthur Ayestas Hilgert, Sihui Wei, Gabriela Moisescu-Pareja. Zhaoyue Wang, and Jonathan Love. Interpreting deep neural networks trained on elementary p groups reveals algorithmic structure. In Topology, Algebra, and Geometry in Data Science.
[32] Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. arXiv preprint arXiv:2406.022 220.00200. 2024.
[33] Ezra Edelman, Nikolaos Tsilivis, Benjamin Edelman, Eran Malach, and Surbhi Goel. The evolution statistical induction heads: In-context learning markov chains. Advances in Neural Information Processing Systems, 37:64273-664 64311,, 2024.
[34] François Charton and Julia Kempe. Emergent properties with repeated examples. arXiv preprint arXiv:2410.07041, 2024.
[35] Nayoung Lee, Ziyang Cai, Avi Schwarzschild, Kangwook Lee, and Dimitris Papailiopoulos. Self-improving transformers overcome easy-to-har and length generalization challenges. arXiv preprint arXiv:2502.01612, 2025.
[36] Samy Jelassi, Stéphane d' A'cooli Carles Domingo-Enrich, Yuhuai Wu, Yuanzhi Li, and François Charton. Length generalization arithmetic transformers. arXiv preprint arXiv:2306. 15400, 2023.
[37] François Charton. Learning the greatest common divisor: explaining transformer predictions, 2024. URL httpss/ts:///at http::::////xvvo//// www.ii.wo...a////////222 15594
[38] Baran Hashemi, Roderic G Corominas, and Alessandro Giacchetto. Can transformers do enumerative geometry? arXiv preprint arXiv:2408. 11008.14911 14915, 2024.
[39] Alhussein Fawzi, Matej Balog. Aja Huang. Thomas Hubert, Bernardino Comera-Paredess Mohammadamin Barekatain, Alexander Novikov, Francisco R. Ruiz, Julian Schrittwieser, Grzegorz Swirszez, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610((930) 2022.
[40] Daniel J Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean- Srun-Baptiste Lespiau, Alex Ahern, et al. Faster sorting algorithms discovered using deep reinforcement learning. Nature, 618(7964):277-233, 2023.
[41] Gavin McCracken. Using Exact Models to Analyze Policy Gradient Algorithms. McGill University (Canada), 2021.
[42] Maithra Raghu, Alex Irpan, Jacob Andreas, Bobby Kleinberg, Quoe Le, and Jon Kleinberg. Can deep reinforcement learning solve erdos-selfridge-spenne seeeide games? In International Conference on Machine Learning, pages 2238-4246.. PMLR, 2018.
[43] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, 2017, URL https://arxias org/ aba//aa//11.. 6980
13

---

Table of Contents for the Appendix
A. Additional Background
A.1.
Conceptual Background and Additional Related Work
14
A.2.
Additional Mathematical Background
16
1.2.1.
Examples: cosets, Cayley graphs, step size d
19
B. Proof of Theorem
Proofs and details for Theorem 4.7 and Corollary 4.8
Embeddings contain projections of representations, not representations
E.
Main-body Experimental Details
25
E.1.
Figure
simple neurons and remapped simple neurons
26
E.2.
Figure
approximate cosets in cayley graphs, cosines and remapped cosines
26
E.3.
Figure
scaling moduli gives (n))((n) frequency growth rate
26
E.4.
Figure
showing average frequencies found in MLPs
27
E.5.
Figure
clusters of simple neurons & R for scaling widths in architectures
28
E.6.
Figure
R vs hyperparameters vs depth in MLPs
28
E.7.
Figure
R vs hyperparameters vs depth in pizzas & clocks
28
E.8.
Figure
the % neurons that are simple in deeper layers varies
29
E.9.
Figure
networks are blased to learn cosets over approximate cosets
29
F.
Further Experimental Results
29
F.1.
Goodness Fit of the Single Sinusoidal Approximation
Additional Experiments We Didn't Put in the Main Text
29
G.1.
Principal Component Analyses (PCA) of embeddings
29
G.2.
More examples of simple neurons
G.3.
Studying phase shifts in simple neurons
G.4.
Studying fine-tuning neurons)
G.5.
Histograms frequencies associated with fine- tiee-tning neurons
G.6.
Fine-tuning neurons like additive and subtractive relations
G.7.
Histograms of counts of frequencies being learned in mod59 and mod66 across varying
depths
G.8.
Noise and ablation studies
G.9.
Number of frequencies
48
G.10. Qualitative: equivariance of the cluster contributions to logits
51
G. Pizza model]
53
G.10.2.
Clock model
54
A Additional Background
A.1 Conceptual Background and Additional Related Work
This appendix expands on the conceptual foundations and related work that motivated our approach. around universality and
We first review key debates in mechanistic secepretablltty-pariiuu Abstraction-belle turning to mathematical tasks as ideal testbeds for studying learned structure.
Mechanistic Interpretability. Universality and Levels of Abstraction.
Mechanistic interpretability seeks everse-engineer trained neural networks by identifying the roles of individual components-sucle as neurons, attention heads, or MLP weights-in in the model's learned function [14]. aims to explain how models arrive at their outputs by analyzing the specific components and pathways involved in their internal computations. A central focus of this paradigm is on circuits [S] 16]: small groups of components (essentially subnetwork) that together perform recognizable subtask such copying, induction composition [17]. Over time, researchers

---

have identified recurring patterns in these sreuitssknowwn as motifs such as superposition [8 (where multiple features share the same subspace), equivariance (where computations respect certain transformations) or unioning over cases (where unit activates for multiple distinct patterns without distinguishing between them). These recurring motifs offer generalizable insight into how networks compute (--
central idea in mechanistic interpretability is universality-thh informal hypothesis that indepen- dently trained neural networks tend to develop similar internal structures M.. The appeal is clear: if models trained in different conditions all learn the same solution, then interpreting one model could offer insight into many. However, what "similar" means in this context has often gone unstated. Universality might refer alignment in learned features, to similarity in neuron roles or circuits, or to shared algorithmic structure- but these possibilities are rarely distinguished. At the same time, empirical findings are mixed: while some studies report that representations in vision and language models become increasingly aligned as model scale increases [3 others show divergences in learned mechanisms even on simple algorihhmicmmathmaaiiee
tasks [ [9]. Together, these findings highlight deeper issue: the field lacks clear definition of what kind of structure should be expected to be universal-and at what level of abstraction such universality should be evaluated. In essence, key goal of interpretability is to find the correct level of abstraction for humans to be able to intuitively understand the functionality of what was learned.
A major challenge in this area is the ambiguity of the term circuit, which can refer to anything from small neuron clusters to nearly full-network subnetworks. This flexibilityy enables compelling case studies, but hinders comparisons across scales. In early interpretability work, A introduced the Fourier Multiplication Algorithm (later called the Clock), showing that transformers trained on modular addition learned to represent inputs on circles and perform angle addition via attention. Chughtai et al. [8] proposed a broader generalization-tia Group Composition via Representations (GCR) Seorithm-Suggesting a universal algorithm for group tasks based on multiplying group representations. In both cases, the term "algorithm" referred to the local computation implemented by a circuit defined over specific Fourier frequency or irreducible representation (irrep). However, later work challenged the universality of these circuit-level "algorithms". Zhong et al. [5] found that different training settings could induce qualitatively different frequency-speciffe circuits (c.g.. the Pizza circuit), and showed that multiple distinct frequency-based circuits could cocxist within the same model. Stander al. (9] analyzed models trained S,, and found boset-based circuits-rather than irrep-based implementations of further undermining its algorithmic universality claim. These results suggest that even when models solve the same task, they may not implement the same algorithm at the circuit level. Our work shifts perspective. We define a model's algorithm as a global computational strategy realized across the full network. To uncover this, we take a multiscale approach-analyzzin the behavior of individual neurons, how frequency-aligned clusters of neurons work together, and how these clusters interact to form a coherent global solution. Our simple neuron model helps reconcile prior findings by showing that seemingly different circuit behaviors can be well- well-approximated within a unified functional form. At the cluster level, we have coset computations. At the full-network level, we identify a consistent solution that emerges across architectures and training runs: a universal abstract algorithm, formalized as an approximate version of the Chinese Remainder Theorem (aCRT). This perspective explains how models can converge to the same high- level structure even when their lower-level mechanisms diverge. Unlike prior analyses focused on isolated subcircuits, we sought to understand how computation emerges across scales-from individual neurons to full-model solutions-and show that models can share the same network-level algorithmic structure despite mechanistic variability.
Much of the confusion around universality stems from comparing models without distinguishing between different levels of abstraction. Recent work has proposed frameworks from cognitive especiall Marr's levels of analysis 20 as useful tools interpretability. helping clarify what kind of explanations are being offered. Marr distinguishes between (1) the computational level (what problem is solved), (2) the algorithmic level (how is solved), and (3) the implemen- tational level (how it is physically realized). Although not developed for studying universality and not formally used in our main analysis, we find these ideas helpful as a retrospective lens-both for understanding why prior analyses disagreed-and and for identifying common structure where others saw divergence. For example, Vilas et al. (33 propose looking for invariances across levels and enforcing mutual constraints between levels as guiding principles. Our approach reflects both: uncover a consistent computational-lleed (in the sense of Marr's 's levels) solution, the aCRT, that
15

---

unifies divergent circuit-level behaviors and reveals form of universality that holds at a more abstract level than previously recognized- what we refer to the universal abstract algorithm.
Mathematical and Algorithmic Tasks as Interpretable Testbeds for Machine Learning.
Mathematical and algorithmic tasks-such modular addition, group operations, sorting, and Markov chains have become valuable testbeds for studying machine learning systems. Their appeal lies in their formal structure: these problems have been studied for centuries, with well- understood properties. Because the task structure is fully known, optimal solutions are analyzable and generalization behavior can be sharply characterized--unlee makes them ideal environments for probing what neural networks learn and how. While primarily used to study model internals, these tasks also have practical relevance; for example, transformers trained on modular arithmetic have been applied to attack lattice-based cryptographic schemes (24).
in typical natural data settings. This
These tasks have been central to studying grokking, the phenomenon where models generalize abruptly after a period of overfitting [II]. Using modular addition as a testbed, researchers have linked grokking to structured internal representations [25 6 margin maximization [12 7 label
corruption [13], and non-neural architectures (26). or how distinct circuits emerge during grokking for different modular arithmetic tasks [27]. Other work has connected grokking to training phase transitions, such as shifts between lazy and rich regimes in modular addition (28] and polynomial
regression [29]. Several papers also provide exact analytical constructions of specific network weights: one-layer, one-hot encoded networks with quadratic activations solving modular addition 777), and ReLU networks solving modular multiplication for modeling grokked solutions in modular polynomials (3) Stepping beyond modular addition to a more general Abelian group, the activations of DNNs trained elementary p group multiplication have been studied well [31]
Beyond grokking, mathematical tasks have helped probe generalization, learning dynamics, and in-context learning. Modular addition and Markov chains have served as controlled environments for studying how transformers acquire in-context learning capabilities 3 3 Other work has shown that repeating training examples affects generalization in tasks like greatest common divisor (GCD), modular multiplication, and matrix eigenvalue prediction [3], and that arithmetic tasks shed light on how transformers handle length generalization 335 36). In the GCD setting specifically, models appear to select from a small, learned set of candidate divisors (37. Transformers trained in the context of enumerative geometry have also been studied (38]
While most research in this area focuses on supervised learning, some work investigates how reinforcement learning (RL) agents operate in mathematical environments. Agents trained tasks like matrix multiplication or sorting have been observed to discover novel, interpretable algorithms Other work leverages group-theored theoretic structure to enable exact analysis: environments built using the temporal symmetries of affine Weyl groups allow analytical characterization of the policy gradient landscape, yielding closed-fome gradient dynamics and local optima and providing insight into how exploration difficulty affects learning [4T] Similarly, interpretable RL environments based on Erdos-Selfridge- Selfridge--Seeneer games have been developed, with exact optimal strategies and tunable difficulty controlled by human-interpretable environment parameters 42
33 40)
Together, this body of work establishes mathematical tasks as invaluable tools for interpretability. Their known structure enables precise analysis of learned behavior, supports abstraction-driven explanations, and provides testbeds where claims about generalization and universality can be rigorously evaluated.
A.2 Additional Mathematical Background
This section provides formal definitions and examples of the group-theoretic structures that underlic analysis of modular addition networks: groups, cosets, Cayley graphs, group representations, and the Chinese Remainder Theorem (CRT). These definitions support the structures described in the main text: simple neurons, approximate cosets and the approximate CRT algorithm we identify in trained networks.
Groups, Subgroups and Cosets.
Definition A.1 (Group). A group (G,o) consists of set G equipped with binary operation o: ::G x G + G satisfying:
1. Associativity: gf (g h) for all f.g. h e G.
16

---

2. Identity: There exists an element e G such that e e goc g all g E G.
3. Inverses: For each g E G, there exists g E G such that g
Definition A.2 (Subgroup). subset И S is subgroup it is itself a group under the same operation That is, H must contain the identity, be closed under the operation, and contain inverses of its elements.
Subgroups induce a natural partitioning of the group via cosets, which are key to understanding modular structure and factorization.
Definition A.3 (Cosets). Let H be subgroup of G, and let g E G. The left coset of H with representative g is:
g = og o h: :h:hhe E H.. Right cosets are defined similarly: Hg (hog h E H}. Cosets partition G into disjoint, equally sized subsets.
Example A.4 (Integers and Even/Odd Cosets). The set (Z.+) is a group. The identity is O; the inverse of n is - The even integers 2Z form subgroup. This gives two cosets:
27 {..., 2,,
1 =2 (..., 1,3,,....
These correspond to the even and odd integers familiar example of partitioning via cosets.
Definition A.5 (Homomorphism), A map v + H between groups a homomorphism preserves the group operation:
(g1 9G 92) = (01) ои (02)), for all 91.92 G.
If is also bijective, it is called group isomorphism.
Homomorphisms are the natural notion of "struccuure-ressrvinn maps between groups. Cyclic Groups and Modular Arithmetic.
In this paper, we focus modular addition, which forms the cyclic group.
Definition A.6 (Cyelic Group C.)). The cyclic group of order n, denoted C, or Zn, the set (0, 1...., 1} equipped with addition modulo The group operation is defined by
o b (a + b) mod n, with identity element and inverses given by
a) mod
Subgroups of Cn correspond to evenly spaced subsets, and their cosets partition C, into congruence classes modulo divisor of
Example A.7 (Cosets mod in Ez). Let n 8, and consider the subgroup H = {0, 4}. The left cosets are:
0 H (0,4), 1 H (1,5),
2+ H = (2,6). 3 H (3, 7}.
This partitions Zg into four disjoint cosets of size 2. Definition A.8 (Modular Inverse). Let E Z,, The modular inverse of a an element E Z., such that
a a- = mod n. A modular inverse exists if and only if ged(a, n) ie., and coprime.
The Chinese Remainder Theorem.
The Chinese Remainder Theorem (CRT) gives powerful way to decompose modular arithmetic large modulus into multiple, independent modular systems smaller, coprime moduli. This decomposition mirrors the modular structure learned by networks trained on addition tasks, and is central our concept of the approximate CRT.
Theorem A.9 (Chinese Remainder Theorem). Let n = 9192 ... ql be product of pairwise coprime integers. Then the map
d Z., > Zug X .... Reg.. x > (xr mod q....., mod qk)
group isomorphism. That is, each element Zn corresponds uniquely to a tuple of residues modulo the qu and vice versa.
17

---

Example A.10 (Coset Intersection View). Let n = 91 = 7 13. Suppose we want to solve:
r r=3 mod 7,
mod 13.
Each congruence defines a coset:
17, 24, x = 10 mod 13 > (10, 23, 36, 49,
mod 7 > (3, 10, 10,
The unique solution mod 91 is the number common to both cosets: 10
Cayley Graphs.
Definition A.11 (Generating Set). Let G be a group. A subset S C G called a generating set of Gf every element E G can be written as a finite product of elements from S and their inverses. That is, for all g E G, there exist -1.-... 8k and signs e, {-1,1) such that:
81 ......88 Fk
Example A.12 (Generating Sets in Zo). The group Ze {0, 1,,,23,4,55 under addition mod 6 can
be:
Generated by {1}, since repeated addition gives all elements: 1,2,...,
Generated by {5), since 5 - 5 = 10 4 mod 6, and so on.
Not generated by (2), since 2 +
+ 0, and the subgroup (0, 2, 4) is too small.
Definition A.13 (Cayley Graph). Let G be a group and let S G be any subset (not necessarily generating set). The Cayley graph [GG S) is directed graph with one vertex for each element of G, and directed edge from g to g for every S.
symmetric (i.e., $ E S s E S), then the graph is often treated as undirected. The graph
is:
Connected if and only if S generates G.
Disconnected if S generates proper subgroup of G.
Example A.14 (Cayley Graphs in Zg). Let Zg = 20,1,2,3,4,5 under addition mod
Using $ = {1), the graph connects:
0 0+1-22-3 -
forming a connected 6-cycle. Here, 5 {1) generating set.
Using S = (2), we only get:
- - 1-3+5-11
which are two disconnected 3-cycles. This reflects the subgroup {0, 2, 4) and its coset 5). The S {2] does not generate Zg. and the graph is disconnected.
Group representattons and the Discrete Fourier Transform (DFT)
Definition A.15 (Group representation). A representation of group G on vector space V is homomorphism p: GL(V), where GL(V) is the group of invertible linear maps on V.
Example A.16 (Discrete Fourier Transform). The discrete Fourier transform (DFT) comes from the complex representations of the cyclic group C.,. Each element j E C, is mapped to the complex number exp 2mr kj for integer l: E {0, 1, .... 1}, encoding modular structure rotations in the complex plane. These complex representations correspond the DFT. They also induce real-valued representations as 2D rotation matrices acting (cos, sin) components.
18

---

A.2.1 Examples: cosets, Cayley graphs, step size d
In the background (Section 3 we gave examples of cosets on Cg. In this section we will show these examples visually to help the reader better understand approximate cosets, generating Cayley graphs, and labeling the vertices them with the step size (Definition 4.1p. Recall, the step size: d ged(f.n) (mod ged)..))) where the modular inverse is used. There are n' ged(f.n) distinct positions reachable in this way, thus we could write: d= (gea((m))) ) (modn'). Note: d is the step size in the graph C.'.
consider the eyclic group on elements, Cs- this example, we will show how cosines of different frequencies encode coset information and how we can construct the corresponding Cayley graphs from this. Since n = 6, then [ = 3, and we have possible frequencies = 1, 2,
for cos depending on f.
f.r across a E Co- These three cosines will each step around Cs in different ways,
f 1, our cosine is cos (2m11.a) We compute n' = 6, thus there are 6 elements we can reach in Cg. This gives 6 cosets of size 1:
(0), (1), (2), (3), (4), (5).
If = 2, our cosine is cos (=222) ). We compute n' 3, thus we're in C3, which has 3 elements. This gives 3 cosets, each of size 2, corresponding to three different 2-cycles in the original Cs graph, but where each coset is now a point in new graph for C, C3:
(0, (2,5).
If f 3, our cosine is cos (mm3mm) ). We compute n' = 2. thus we're in C2, which has 2 elements. This gives cosets, each of size corresponding to two different 3-cycles the original Cs graph, but where each coset is now a point in our new graph for C,- = C2:
(0,
See these cosines in Figure 10
1,2,2,
0.5
0.5
0.5
COS value
cos value
COS value
0
-0.5
-0.5
-0.5
0 1
01121
0
4
5
6
Figure 10: Cosine functions centered at O, with f = 1, f = 2, f = 3. The points are colored based on their coset membership, i.e. equivalence class.
We can calculate the step size, Definition 4.1 and all three cases we get these sets in figure coming shortly.
1. We will visualize
Row 1 shows the original Cayley graph and row shows the new graph after collapsing cosets. Row I will be the cyclic group on 6 elements, Cg: in other words, we will not collapse the cosets yet into their equivalence classes. This means we aren't making the graphs C., and resultantly, we plot Ce with distances from the vertices the first vertex = 0. We also show the cosets, the 3-cycles
19

---

Co generatons f+1, f+3. step size di=1 in three cases cosCI22ptiiNNn
(wi: (2NEA221) (abb-rccles
Fwd:t costat2li Gajhii (ubb-crclee)
[wz: cost2tatsiiaatii (sub-tircle)
(2.5)
(1,3)}
(1.4)
Figure 11: In row show the Cayley graphs and the distances, which is bit confusing in panel 3 because we don't collapse the cosets into their equivalence classes. To make easier to see the distances and where they come from, in row 2 collapse equivalent points (all points in coset) into a subcirele. Doing this gives us less points on the graph, giving us Cw', which is the graph d is the step size on. now easier see why the distances approximate cosets are what they are. In row 2 panel 3, we chose to put the two darkly colored cosets on the other side of the circle to emphasize that ReLU is 0 on them.
and 2-cycles. In row 2, we present equivalent picture that makes things clearer: we plot Car- Furthermore, the step size d actually the step size this cyclic group, Cn', making the distances easier to see compared to looking at Cg. By collapsing vertices into their cosets (equivalence classes), becomes clear that everything is distance I from the yellow coset. We will do this in row 2, giving Cs corresponding Figure 11
1, C3 corresponding to f 2 and C2 corresponding to = 3. See this in
In the case of approximate cosets, nothing changes. Calculate nt', calculate d, and step around the cyclic group C,2' using d. labeling the distances of elements. To build an approximate coset, for some multiples k1, k2. take k1 steps backward and k steps forward. The path is the approximate coset-all elements that are "close" on the Cayley graph.
A.2.2 An example for how the network gets the correct answer
Simplified Divide and Conquer: Consider (a + 6) mod 6 =
There are two coset types:
mod? 2:
I mod 2: (0, 2, 4} mod? 2: (1,3,5)
mod? 3:
20

---

0 mod 3: {0,3)
mod 3: (1,4)
2 mod 3 (2,5)
Let neuron-l have frequency 3 and activate when the answer c E {0, (0, i.e. it activates on this coset of learns:
(a b) = cos( a + cos( 6mb
This neuron activates maximally when both a, b E 20,2,4). guaranteeing (0,
Let neuron-2 have frequency 2 and activate when E {0, It learns:
fa(a, b) cos( 4 ) cos( 4mb
This function activates maximally when a, {0,3). ensuring E {0,3).
Summing the outputs of these two neurons gives the following logits:
== 0 -
c 1 +
c = 1 + 0 c 2
c 3 - 2
c = 4 = 5 +
The argmax operation now correctly selects c 0.
Suppose alternatively, that neuron-3 activates for E (1,4) (e.g., by firing when E {0, 3) and E [1,4]). Then neuron-1 (firing on c E {0, 2, 4)) and neuron-3 activate simultancously. the argmax would select = 4 due the higher logit value. This logic continues you add neurons corresponding to every coset.
This demonstrates divide-and-conques strategy: each neuron rules out a large fraction of incorrect outputs, analogously how binary search eliminates half the search space at each step. This example with mod 6 and exact cosets. Approximate cosets generalize exact cosets by capturing cases where neurons learn frequencies that are not divisors of the modulus. Theorem 4.7 shows approximate cosets are sufficient for networks to attain strong margins, matching the cosets required by the classical Chinese Remainder Theorem.
As grows, the number of coset types is logarithmic, e.g.
mod6 + 2 coset types
mod3628800 > 10 coset types
B Proof of Theorem 4.4
For empirical evidence supporting this theorem, see Figure 2 in the main paper. Every point is in the approximate coset colored by viridis colors, with strength of viridis decaying the point gets farther from the center element of the approximate coset (an element getting closer to where ReLU won't activate, means it less bright).
In reality, all sinusoidal functions, iee. our simple neuron assumption, will satisfy this theorem. The simplicity of this proof therefore results from the fact that we came up with very powerful definition for approximate cosets that actually reflects what neurons in the network are learning. This proof requires the assumption that neurons learn periodic functions, and this is why the majority of the paper was dedicated to empirically proving that simple neurons are indeed learned by networks.
Proof. Simple neurons learn approximate cosets. neuron satisfying the simple neuron model computes trigonometric function that has its maxima on the elements of a coset "approximate coset". If g == god(f, n) > 1, the neuron has learned the coset order g containing 8 + 8g. More precisely: writing = n'g and f f'g for ged(f, n1), we can rewrite 2xf
fx So if the
21

---

input neurons are at positions and b where SA (mod n²) and = a (mod n²), then the activation of the neuron has maximum: cos 2rf(a-sA) cos n most strongly to every logit satisfying Sa + s (mod n'), because for all such output logits cos P=f(c=sA-SH 1. We see that the neuron strongly associates elements of Cn that are congruent
2r/(6-sa] 1. The neuron points
n
modulo n/.
Whether f is a divisor of the order n or not, the neuron will activate on what we defined as an approximate coset. More precisely, we can ask the following: for a # 84 (mod n'), which values of a have the largest activation? We have cos [a-aa= very close to if and only if s'(a 8A) very close to an integer multiple of n'; that is, say, f'(a 100-5.0 SA) = m (mod n²) for some integer with small absolute value. Letting d denote modular inverse of f' mod n, this is equivalent to a A dm (mod n'). In other words, by taking a = SA + dm for small integers m, the neuron will be activated very strongly. Likewise if b 8B dm' for some other small integer m'. Now this neuron will point most strongly to c = SA + SB (mod n') as discussed above, but if e is small number of steps of size d away from 8 SB, it will still have large activation. To summarize: if you can reach each of o, b, e via a small number of steps of size d from 8,, $ B, & + 8B, respectively, then N fires strongly on inputs a, b, and points strongly at
After ReLU, it follows that since all neurons output (activate) only on approximate cosets, all neurons in the following layers activate linear combinations of approximate cosets (follows from networks being fully connected, and ReLU only having the potential to make the cosets smaller, i.c. elements below 0 are cut off).
C Proofs and details for Theorem 4.7 and Corollary 4.8
Empirical avg margins on (a+b) mod over 500 networks; freq count avg_aaaaeeee margin
3/4 avg margin
5
R² 0.988
2.2012-login) + 2.9748
empirical avg_margin
2
3
# unique frequencies
Figure 12: Empirically, hidden layer, trainable embedding matrix, networks have margins grow like (n)) as more frequencies are learned. In Corollary 4.8 we prove it. Note: std dev error bars on y=2.82); they are just small, ranging from 2.8-2.84. This plot is made by computing the average margin of each network across the full dataset, given the network had learned x =# unique frequencies. The plot shows, for networks that learned the same number of frequencies, the average average margin and the std dev error bars of this data.
Assume that training results neurons learning the simple neuron model.
Theorem C.1. Suppose the integer number of distinct frequencies m and reals 0 p. 8 satisfy the inequality
2 log, n 22oog 2p)
m
(
Then, with probability at least p. for all l fitj mod n, we have m hm (k) > 8m'.
Proof. Each simple neuron maximally activates single output, namely 8A 8B, (or possibly maximally activates on coset of outputs containing SA 3g). However, if we combine the
22

---

contributions from all simple neurons in single cluster (i.e. all with the same frequency f), we observe that the activation level can be maximized at any desired output; more precisely, the activation level at output l given inputs j,j will be of the form A cos (2Tr (f(k - - - j)/n)). Note this has been observed experimentally e-g. see Fig F (6(a) and Fig. [44] and has also been previously noticed in the literature: see e.g. the last equation of Section 3 in [10]. In fact, the analysis below still works even if + j is only somewhat close to the maximal activation of the cluster (see Figure though we assume the maximum is at + j for simplicity. However, even in this case there will be many output logits that all activate nearly as strongly on the correct answer. To isolate a single answer, we use superposition of sine waves of multiple frequencies; observe experimentally in Figure 44 that this process makes the correct answer stand out from the rest.
In light of the above and Section 4.1 if we fix inputs i,j E C₂ then combining the contributions from all clusters (and assuming for the heuristic that the contributions from all clusters have the same amplitude), the sum
2 (
1m (k) =
Σ COS
(k - i-
gives a model for the activation energy output logit k. If h: = i j. then (k)) takes the maximum value m. If we want to guarantee the neural net will consistently select k. we need to show that (k)() is significantly less than m for all other values of k. We'll assume random model where m frequencies fi,...,fw fm are chosen uniformly at random from 1, 1,2....... - 1. Fix a parameter 0 < we will compute an approximation for the probability that m Im (k) dm for all k +i+ (mod m).
Let {x} == x + 1 be the signed distance to the nearest integer and set d := ks i j. Then using a Taylor expansion,
fed }) ~ m
fed
2
fed
2
m - 1mmmmm()))))))))))))))) = m - Σ cos
2n
Σ (=1
2T
2m
2
n
E=1
(Note that the Taylor approximation is quite bad when fed/n is far from integer, and if k is close to an integer then {k} is close to O. is reasonable to expect that m - hmm (mm(k) will be minimized when the values fed/n n are all close to integers, in which case the approximation is more accurate. (Note, getting a rigorous proof instead of this toy model will require more precise bounds.)
Thus the condition m Imm 1k)(k)== > 8m is related to the following condition: defining the vector V v=== v=== -J Sm) E [0, 1jmh we need that for all 1 S d S n - 1, the point du has distance at least /ōm/2x² away from any point in Tm. Note that no is an integer point, so (n d)v v is always the same distance from an integer point as du is. Thus it suffices require to be least 1/2 // 0m//27 2
away from a point in 1 for d 1...., [n/2].
We compute an upper bound on the volume of the region to be avoided: that is, the set of all points in [0, 11 within 2 //mm222 of a point of 1 for some d = 1,..., n1/2. For each d, there are dm points in this region, and each has ball of radius /ömm/nx² around it; the total volume of the m/2 Thus the probability that region to be avoided is therefore bounded above by 2mmm2++1 2
m fm (k) > 6m is approximately equal to minus this value.
For given n, let's compute the value of m that makes this probability greater than, say, p.
n
5m
m/2
2πT
m/2
1 21 (m/2 +
>p
T(m/2+
2T
dm
2 2p
Taking a natural logarithm, applying Stirling's approximation log, I(x + 1) 2 x log, (x) I, and solving for m,
log, l(m/2 +
m log,
2T dm
log, n log, (2 2p)
2 log 2 2og. (2 - -pp)
log, (7
23

---

Thus if the number of neuron clusters m is greater than this expression, then with probability at least P. the separation m (k) will be least óm. We see that the number grows linearly in log, n.
Choosing the parameters and can significantly change the precise value of m. needed, and it's not clear which values most accurately model the true behavior of the neural net.
As an example, note that we take TT T//²* s 0.1364, and = 2.2 then this whole expression log, neuron clusters, then this
simplifies to just m log, Thus, if the neural net uses heuristic predicts that will guarantee separation m (k) > 0.15m for k + i + with 50% certainty. For 89, 91 have loge = 4.5, which agrees with the number of clusters found in Figures 40 and This process can be interpreted as the approximate CRT; see Remark 4.3 for the analogy.
C.1 Proof of Corollary 4.8
Recall is the model network evaluated at input i, hn (k:) is the value of the output distribution at k with maximum value hm + j) m and m is the number of distinct frequencies simple neurons of the network.
Corollary C.2 (Logarithmic number of frequencies suffices for non-trivial margin). Let 0 < 8 <1 and 0 < p < and define
log( 166) (1 1/2 loge (2- 2p))
C(ô.p)
(> 0).
the number m of simple neurons of the network satisfies
m 2 C(ó, p) log n
then with probability at least
Nm (i 5) - hm (k) > 5 m,
i.e. the logit margin is M(logn.).
Proof. Theorem
4.7 states that the inequality
hm (i j) 1m (k) 8m
(f)
holds for all k + zit j with probability at least p provided
loga n 2 log (2 2p)
m
($)
log. (T/0)
Choose the constant so m C(ă, p) log and inequality ($) both hold.
Finally, the guaranteed margin 5m 2 5C(8, p) log is R)logn), which is strictly larger than the O(1) margin attained with only a constant number of frequencies ("minimal margin").
D Embeddings contain projections of representations, not representations
Chughtai et al. [ discover representation values in the embedding matrix. The first step in their GCR algorithm is not true in general. They state, "Translates one-hot a, to representation matrices". We provide evidence against this by training with mini-batch size equal the modulus n and training with full batch size. See the difference in the distribution of the resulting embedding matrices in Fig. 13 Furthermore, neurons in cluster of frequency have different phase shifts, and 2 x rotation matrices in the embeddings doesn't 't suffice to explain this behaviour.
Instead, the values found in the embedding matrix may encode scaled projections of a 2 x 2 rotation matrix onto a one dimensional subspace. Note that such structure is implied by the hypothesis that neural networks trained group tasks learn representations, but is more general because of the existence of both amplitude and phase shifts. To get an exact equivalence, we note that this neuron structure can be obtained by arbitrary scaled projection of representations. Suppose
p(k) = (cos(2m fk/n) [inn(2
(2mff///) cos (kt/k//)))
24

---

is 2 X 2 matrix representation of Cn. If we apply p(k) to the vector (1, 0) and then take the dot product with (a cos(2Tfs./n) -a fin f/)n)) (which is the same as projecting onto the subspace spanned by this vector and scaling by a)) we obtain exactly
cos 2m/k cos 2mfsa sin 2mʃk sin 2m/SA
cos
2n 2
N).
Thus we have explained the phase shifts of different neurons in a cluster, and shown that it's not just the components of p(k) that appear in the embeddings, but rather scaled projections of the representations onto arbitrary 1-d subspaces. In our model of simple neurons we ignore the amplitude to make the analysis simpler, but in general it does need to be included. See Fig 28 for example where the amplitudes are greater than 2.
Inspecting the distribution of embedding matrix weights. Contrary to findings by [4 g we did observe the 2x2 representation matrix values (used to encode rotations) our embedding matrices outside their reported training conditions. As shown in Fig. [3 the distribution of embedding weights varies significantly between small and full batch size and the tails of the distributions are quite different. In the case of small batch size, numbers can be found in the range (-2, 2), whereas large batch size contains numbers between (-1.5, 1.5). Note that we choose to remove weights that are between (-0.025, 0.025) to make it casier to see the tails of the distribution; this was done due to 2.4million weights occurring within this range when training with the small batch size. Specifically, in the small batch size regime, around 5% of the weights fell outside the interval [-1, 1J. including some weights larger than 2. These values are not consistent with rotation matrix entries. Other than this, could not identify any significant differences in the structures of what the neural net learns between the batch sizes.
Combining these experimental findings (Fig. 13 with this model (see explains that the embedding matrices may contain scaled projections representations. This explains the different shifts in the periodic functions that can be seen in Figs 26(a) 26(b) and 24 which GCR [8] fails to explain.
batch size train set size
batch size 91
300k
300k
Count of occurrences
200k
200k
100k
100k
C 2.5
C 2.5
-2
-1.5
-0.5
0
0.5
1.5
2
-2
-155
-1
-0.5
0
0.5
1
1.5
2
Magnitude
Magnitude
Figure 13: The histograms of embedding weight magnitudes found across 10k random seeds for mod
91 provide evidence against rotation matrices. With batch-size about 5% of the weights are > or
< -1, whereas when the batch size is the training set size fewer than 0.5% of the weights are > 1 or <
-1. The bin with was removed for batch size 91 due to so many dead weights obfuscating the plot. The value was 2.4 mil, implying that small batches find sparse embeddings with larger magnitude weights.
E
Main body experimental details
The train and test splits are 90% and 10% respectively, unless stated otherwise. The optimizer used to update the neural networks weights is always Adam 43
25

---

The only plot with error bars in the main text is Figure 3 which uses I standard deviation (std dev) error bars. If plot has error bars, they are std dev.
E.1
Figure
The point of this figure is to show the reader that qualitatively, neurons are learning sinusoidal functions that are identical after normalization, even when secondary frequencies exist in the Discrete Fourier Transforms. Furthermore, it serves to immediately show the reader that the "remapping: normalizing frequeney 1" definition makes sinusoidal function have frequency 1. The figure is very easy to generate. just grab arbitrary neurons, plot them, and plot their remapped version. The neuron from the MI.P model comes from an MI.P with frequency 14. The MI.P is one of the MLPs trained on mod 59 for Figure The pizza transformer is model A prodel_p99zdpzes checkpoint) and the clock transformer model (model_ 18k www.iii.x................ii.iiii pt checkpoint) from Zhong et al. (5'ss Github repository.
E.2
Figure
This plot shows the simple neuron model, approximate cosets, and Cayley graphs that neurons understand distances on. The point of this figure is to familiarize the reader with our definitions as they are essential to understanding how we derive the abstract aCRT. The code to generate this plot included in the supplementary materials mmake_figure coo__appro__coat cosetsspy"..
E.3 Scaling experiment in Figure
We report std dev error bars here, as we do on all our plots. though since our arguments are probabilistic in nature, our growth rate is supposed to be in expectation. We just added the 1 std dev to show the std dev if a normal distribution was with the same average. Indeed, the standard deviations are low, For the transformer models (clock and pizza), we take the exact model classes from (5)s Github repository. and translate them into Jax. For clocks, the attention coefficient is set to 1.0 and for pizzas, the attention coefficient is set to 0.0. The d model is always taken to be the smallest power heads is always 4, and the d head is such that 4 times that number is equal to the d model. For the hyperparameters used when training the transformers, see Tables
that larger than n, because the architecture requires it. The number of
Number
Learning Rate
Batch Size
Weight Decay
Training Set Size
0.001
0.0001
49
17
0.001
34
0.0001
289
27
0.001
100
0.0001
729
59
0.001
200
0.0001
1770
97
0.001
200
0.0001
4850
113
0.001
500
0.0001
6780
303
0.0002
909
0.00002
45450
499
0.0002
1497
0.00002
124750
977
0.0001
4885
0.00001
488500
1977
0.000035
39540
0.0000075
2965500
4013
0.00004
16052
0.000006
3691960
Table 1: Experimental results with Adam optimizer across varying parameters for both pizza and clock.
Number
Learning Rate
Batch Size
Weight Decay
Training Set Size
64
0.001
64
0.0001
49
128
0.001
128
0.0001
289
256
0.001
256
0.0001
729
310
0.001
310
0.0001
1770
720
0.0001
720
0.00001
309600
Table 2: Experimental results with Adam optimizer across varying parameters in pizzas.
26

---

Number
Learning Rate
Batch Size
Weight Decay
Training Set Size
64
0.001
64
0.0001
49
128
0.001
128
0.0001
289
256
0.001
256
729 0,0001
0.0001
310
0.001
310
1770
720
0,0001
720
0.00001
309600
Table 3: Experimental results with Adam optimizer across varying parameters in clocks.
For the hyperparameters used when training the MLP, see Table 4 the moduli
The number of neurons is 8 times
Number
Learning Rate
Batch Size
Weight Decay
Training Set Size
0.01
0.005
5
0.01
0.005
25
7
0.01
0.005
49
13
0.009
13
0.004
169
17
0.009
0.004
169
59
0.008
59
0.001
1770
64
0.005
64
0,0005
2048
113
0.004
113
0.0003
6780
128
0.002
128
0.0002
13568
193
0.003
193
0.0001
18914
256
0.001
256
0.0001
34560
310
0.0009
310
0.00007
51150
433
0.0006
433
0.00005
86600
499
0.0005
499
0.00003
124750
720
0.0004
720
0.000015
259200
757
0.0003
757
0.0000085
280090
997
0.0003
997
0.0000015
498500
1409
0.00028
1409
0.0000009
986300
1999
0.00024
1999
0.0000008
2398800
2999
0,00018
2999
0.0000007
4798400
4999
0.0001
4999
0.0000005
14997000
Table 4: Experimental results with Adam optimizer across varying parameters for the MLP in Figure
E.4 Figure
We trained one hot encoded 1,2,3,4 hidden layer MLPs and also trained trainable embed- ding matrix I hidden layer MLP over moduli 59-66. The classes for these models are in the Smlp_modelss ponddelsmmutillyeeppp file.
Table 5: Hyperparameter configurations for one-hot models with varying hidden layers and embed-
ding.
architecture
hidden layers
# neurons
L2 regularization
learning
train size
One-hot
1024
10-5
0.00075
90%
One-hot
2
1024
x 10-5
0.00075
90%
One-hot
3
1024
1 x 10-5
0.00075
90%
One-hot
4
1024
1 10-5
0.00075
90%
Embedding
1
1024
1 10-5
0.00075
90%
27

---

E,5 Figure
trained 500 models of each architecture with 1-hidden layer, and each combination of number of neurons in [512, 2048, 8196, 16392] mod 59. The models are only saved if final accuracy 2 99.9999. The neurons with maximum preactivation below 0.01 across deleted and considered "dead" neurons. We find that as the model width is increased, a single sine wave better and better approximates the preactivations the neurons.
the data were
The classes for the architectures are in the @nlp_models goo__mddel__muiilaeeeeee multilayerpp"" "transformer-trail trrii_gee__attt Beaen_geeddtta__t__teeeee 12 caatmap_att-al attn=0_top-k Laekk-aye__aaa layer_all.py" "transformer train _eettngte____ get_data________ r2 heatmap_ampaa attn= top-k layer_alllpp files. See Table E.5 for precise experimental details.
and
Table Hyperparameter configurations for Figure --embed, 1-hidden-layer models with varying hidden unit sizes and architectures.
architecture
hidden layers
neurons
L2 regularization
learning rate
train size
MLP (baseline)
Embed=128, ML.
512
1 10-5
0.00075
90%
Embed=128, MLP
1
2048
1 1 10
0.00075
90%
Embed=128, MLP
8192
1 x 10
0,00075
90%
Embed=128, MLP
1
16384
1 10-5
0.00075
90%
"Pizza"
Embed=128, Pizza
512
X 10-1
0.00050
90%
Embed=128, Pizza
2048
1 x 10-4
0.00050
90%
Embed=128, Pizza
8192
1 1X 10
0.00050
90%
Embed=128, Pizza
16384
1 10-4
0.00050
90%
"Clock"
Embed=128, Clock
512
1 10
0.00050
90%
Embed=128. Clock
2048
10-4
0.00050
90%
Embed=128. Clock
8192
1 10-4
0.00050
90%
Embed=128, Clock
16384
1 1 10-4
0.00050
90%
E.6 Figure 6
There are 10 models trained with 10 different random seeds (different random init and different random train test splits) for every (learning rate, weight decay) combination. This is the most pes- simistic case for our green checkmark vs purple dot scenario because if single model doesn't 't have 100% accuracy after ablating the neurons for our fits, then it would receive purple dot (assuming accuracy of the trained model with no ablations was 100%). The learning rates and weight decays are:
learning_ratee rates= [0.0025, 0.001, 0.00075, 0.0005, 0.00025, 0.0001, 0.000075, 0.00005, 0.000025, 0.00001]
weight _ecctttdeeyssseeee = 0.001, 0.00075, 0.0005, 0.00025, 0.0001, 0.000075, 0.00005, 0.000025, 0.00001, 0.0000075, 0.000005, 0,0000025, 0.000001, 0.00000075, 0.0000005, 0.00000025, 0.0000001, 0.000000075, 0.00000005, 0. 0.00000001
There are 1024 neurons in every layer of the models. The embedding matrix has 128 features.
E.7 Figure 7
There are 10 models trained with 10 different random seeds (different random init and different random train test splits) for every (learning rate, weight decay) combination. This is the most pes- simistic case for our green checkmark vs purple dot scenario because if single model doesn' 't have 100% accuracy after ablating the neurons for our fits, then it would receive purple dot (assuming accuracy of the trained model with no ablations was 100%). The learning rates and weight decays are:
learning rates [0.0025, 0.001, 0.00075, 0.0005, 0.00025, 0.0001, 0.000075, 0.00005, 0.000025.
28

---

0.00001]
weight decays = 0.001, 0.00075, 0.0005, 0.00025, 0.0001, 0.000075, 0.00005, 0.000025, 0.00001. 0.0000075, 0.000005, 0.0000025, 0.000001, 0.00000075, 0.0000005, 0.00000025, 0.0000001, 0.000000075, 0.00000005, 0.00000001 J.
The embedding matrix has 128 features. There are 1024 neurons in every layer.
E.8 Figure 8
There are 10 models trained with 10 different random seeds (different random init and different random train test splits) for every (learning rate, weight decay) combination. This is the most pes- simistic case for our green checkmark vs purple dot scenario because if single model doesn't 't have 100% accuracy after ablating the neurons for our fits, then it would receive purple dot (assuming accuracy of the trained model with no ablations was 100%). The learning rates and weight decays are: learning_rates = [0.0025, 0.001, 0.00075, 0.0005, 0.00025, 0.0001, 0.000075, 0.00005, 0.000025,
0.00001]
weight decays = 0.001, 0.00075, 0.0005, 0.00025, 0.0001, 0.000075, 0.00005, 0.000025, 0.00001, 0.0000075, 0.000005. 0,0000025, 0.000001, 0.00000075, 0.0000005, 0.00000025, 0.0000001, 0.000000075, 0.00000005, 0.00000001 2048 neurons in each layer, layers.
E.9 Figure
5000 models were trained to make these figures. and 4 hidden layers mod 66. The learning rate = 0.00075. The weight decay penalty was L2 regularization = 0.0001.
F Further experimental results
F.1 Goodness of fit of the single sinusoidal approximation
These plots ended up below as Figure 36 and Figure 37 This section will be removed in future versions.
G
Additional experiments we didn't put in the main text
The point of this section is to hopefully provide interested readers with empirical results that weren't t necessary for the main text. These experiments omitted due either space constraints low priority.
Note: call the neurons learning non-integer frequencies over 2, e.g. f 1 after remapping "fine-tuning" neurons.
G.1 Principal component analyses (PCA) of embeddings
Below, we answer all the open problems of Zhong al. (5] involving non-cireular embeddings. The idea of doing the Discrete Fourier Transform (DFT) on the embedding PCA's, gives that they are caused by two different frequency sines: one coming from each principle component (PC).
We replicate the results of [5] and add an additional Fourier transform plot next to their PCA plots, which makes it obvious that the principal components come from clusters with the same frequency. can be seen that all non-circular embeddings and Lissajous embeddings are caused by the two principal components coming from different cosets. This means that answer all of the open problems [ involving non-circular embeddings.
To make this easy to understand, please see Fig. 14 showing this random seed has four clusters, with key frequencies 35, 25, 8, 42. We now know what frequencies to look for the DFT plots and thus, can figure out which PCs come from which frequency clusters. Doing reveals that all PCs where
29

---

both PCs come from the same frequency cluster are circular. Conversely, all PCs where the PCs come from different clusters are non-circular Lissajous curves.
Neurons [2] Cluster Combined DFT Plot
Embeddingg (DFTT)
Embedding (DFT)
Meg 1o/d
Map Tbude
Frequenryy
Fiequeney
Embedding A (DFT)
Embedding 5 (DFT)
Mag Bhud
[9]- Combined
Embeddinng (DF))
Embedding (DFT)
100
Magnttuee
100
100
Frequency
Frequency
Nourons [14] Cluster Combined DF1 Plot
Embbediding (DFT)
Embesdding B (DFT)
100
Napitb33
Magntadde
30
Frequeeccy
Rrequency
Figure 14: DFT of neurons in each of the four clusters in this random seed. Cluster 0 has frequency 35, cluster has frequency 25, cluster has frequency 8, and cluster 3 is fine-tuning cluster with frequencies at multiples of 7, 14, 21, 28, 35, and 42.
30

---

Embedding PC1 vs PC2
Embedding B: PC1 vs PC2
120
1.0
0.5
Magnitude
80
0.0
40-
PC1
Frequency (0 to 90)
(a) (a) PCI vs PC2 Scatter Plot
(b) (b) PCI vs PC2 DFT
Figure 15: PCA and DFT for PC1 vs PC2 showing circular embedding clustered into cosets. The and y axis of the left plot are the PC1 and PC2 values for the concatenated embedding matrix for each point (a, b) mod 91 E (0,0), (1, ..... (90, 90). Note that this covers all output classes of the ..., neural network exactly once. Also note that the embedding here is showing 13 cosets with 7 points in them each, i.e. all 13 cosets (a + b) mod 13 i, {0, ..... 12) are in the plot. Both PCI and PC2 have = 35 and since (3d 91) = 7, a prime factor, it's possible to learn the exact cosets.
Embedding DFT of vs
Embedding B: PC1 vs PC3
1.0
50
2
Magnitude
30-
68
-1.0
-0.5
0.0
0.5
1.0
1.5
0
20
40
60
60
Frequency to 90)
(a) PC1 vs PC3 Scatter Plot
(b) PC1 vs PC3 DFT
Figure 16: PCA and DFT for PCI (ʃ -f= 35) vs PC3 (f = 8), non-circular embedding.
Embedding B: DFT of PC1 vs
Embedding B: PC1 vs PC4
60
1.
28
Magnitude
PC4
0.0
30
12
-0.5
10
192
16
-1.0
-0.5
0.0
0.5
1.0
2.5
0
20
40
60
80
Frequency to 90)
(a) PC1 vs PC4 Scatter Plot
(b) PC1 vs PC4 DFT
Figure 17: PCA and DFT for PCI (ʃ -f= 35) vs PC4 (ʃ = -8), non-circular embedding.
31

---

Embedding B: DFT of PC2 vs PC3
Embedding PC2 vs
60
1.0
hagnitude
40
PC3
-0.5
-1.0
10.
-1.5
-1.0
-0.5
0.0
0.5
PC2
Frequency (0 to 90)
(a) PC2 vs PC3 Scatter Plot
(b) PC2 vs PC3
Figure 18: PCA and DFT for PC2 (f 35) PC3 (f 8), non-cireular embedding.
Embedding PC3 PCA
Embedding DFT of PC3 vs PC4
100
0.5
Magnitude
PC4
-0.5
PC3
Frequency (0 to 90)
(a) PC3 PC4 Scatter Plot
(b) PC3 PC4 DFT
Figure 19: PCA and DFT for PC3 (f 8) PC4 (f 8), which is circular embedding because both PC's come from the same frequency cluster.
Embedding B: DFT of PC3 vs PC5
Embedding B: PC3 vs PCS
1.0
Magnitude
-0.5
11.5
-1.0
-0.5
0.0
0.5
1.0
20 Frequency (0 to 90)
PC3
(a) PC3 vs PC5 Scatter Plot
(b) PC3 vs DFT
Figure 20: PCA and DFT for PC3 (f = vs PC5 (f 25), non-cireular embedding.
32

---

Embedding B: DFT of PCB vs PC6
Embedding
PC6
8
0.5
Magnitude
PC6
0.0
-0.5
12
1.0
20
PC3
Frequency (0 to 90)
(a) PC3 PC6 Scatter Plot
(b) PC3 PC6 DFT
Figure 21: PCA and DFT for PC3 (f 8) vs PC6 (f 25), non-cireular embedding.
Embedding B: DFT of PC4 PCS
Embedding B: PCA vs PCS
V
0.5
Magnitude
PCS
2
W
-0.5
-1.0
5
-1.0
-0.5
0.0
0.5
1.0
0
20
40
PC4
Frequency (0 to 90)
(a) PC4 vs PC5 Scatter Plot
(b) PC4 vs PCS DFT
Figure 22: PCA and DFT for PC4 (f = 8) vs PC5 (f = 25), non-circular embedding.
Embedding B: DFT of PCS vs PC6
Embedding B: PCS PC6
0.5
Magnitude
PC6
0.0
-0.5
20
0.5
1.0
20
40
60
PC5
Frequency (0 to 90)
(a) PC5 vs PC6 Scatter Plot
(b) PCS vs PC6 DFT
Figure 23: PCA and DFT for PC4 (f = 25) vs PC5 (f 25), circular embedding as both PCs come from the same frequency cluster.
33

---

G.2
More examples of simple neurons
See Fig. for cluster of simple neurons. The neuron remapping (via group isomorphism) can be seen in Fig. 25
Pre-ictiation Villue Neurtne Oluster
I'beedding.- Neurer
Figure 24: an example cluster of 14 simple neurons of frequency 21.
Prededivtion Malano for Reuararo Cksslrr
Paa mebedinng
Enbeading. Nameer 5-10
Enbeoding: Nomses 11-34
Figure 25: A cluster of simple neurons (from Fig. 24 transformed by group isomorphism so that all neurons have period!
G.3
Studying phase shifts in simple neurons
Here we show how the phases of different neurons in cluster overlap to give some more information about how clusters neurons function. See Fig 26(a) for the histograms of the phases of the preactivations of the neurons in cluster.
34

---

Claster 1: 14 seurniy, treganng =11
Blngra (Tasn Clickr avamus,
Phose +ipoggamms for Cudter1
(a) Histogram of preactivations for two neurons in fine-tuning cluster. The x-axis is the input value into the network for (left) and b (right).
Cfftrent cre Phassea Phane,h
15
533
64
(b) 2D scatter plot created by grouping the phases for each neurons and preactivations into a pair (phase-a, phase-b) plotting the points for all neurons cluster 2D plane as black point. In this case, the cluster has 14 neurons of frequency 21.
Histogram phases (top) and 2D scatter plot of phases (bottom) for a simple neuron cluster with frequency 21.
Figure 26:
For higher resolution view of what's going on, see scatter plot created by grouping the phases for each neuron's s and b preactivations into pair (phase-a, phase-b) and plotting the points for all neurons the cluster the 2d plane as a black point, see (6b) It's worth noting that the phases nice and spread out uniformly like in Fig Tig266b))n)) about half the time. In the other cases
We aren't 't sure what causes the phases to sometimes align in a nice grid, vs a much more "random" looking configuration seen in Figure 27 Understanding this is likely essential for proving realistic bound for the number of neurons needed with the same frequency. We leave this for future work.
35

---

Breapped Pare Suate Pu for bote Hhaee
Prach
AUOLA
Figure 27: Here's an example where the phases aren't in a nice grid like they were in Figure 26
G.4 Studying fine-tuning neurons
Fine-tuning neurons are composed of linear combinations of group representations, contrast with simple neurons which are composed by one group representation. Additionally, fine-tuning neurons are composed of group representations for what are called harmonic frequencies of the main frequency, for f 7 these would be the multiples of 7, egg. {14, 21, 28, 35, J..
36

---

Inspection of simple neuron mod 89 Embed Preactivations
Discrcte Fouricr Transform
100
-2
0
20
40
60
80
Transformed Embed A Preactivations
Transformed DFT
100
50
-2
0
P
0
20
40
60
80
0
20
40
60
80
Inspection fine-tuning neuron Embed Preactivations
Discrete Fourier Transform
60
40
-1
20
-2
00
0
20
40
60
80
20
40
60
80
TransFormed Embed Preactivations
Transformed
80
60
40
-1
20
-2
0
20
40
60
80
20
40
60
80
Figure 28: Comparing a simple neuron and fine-tuning neuron before and after transformation by group isomorphism. The fine-tuning neuron has its DFT concentrating strongest on (27, 35, 19).
We train a neural network with random seed 133 and discover a cluster of fine-tuning neurons. The preactivations for of these neurons shown in Fig. 29 and the DFT's for these neurons shown in Fig. 30
We show that these neurons can be generated by linear combinations of representations in Fig. 31
37

---

3lndte3
Nouran 14. Ch-Cha 3- S-Cominnd Predutidtion Plut
land 20000000 Neaaron
Embeecdinnggg
152.0
Figure 29: This shows a cluster of fine-tuning neurons and shows the preactivations of the first two neurons in the cluster. The x-axis is the input value into the network for a on the left, and the input value for the right.
Cluster
Veuum 1d- www.ireen Pet
Noxsn B2
No http://4-bbbbbbrg000
Neuron34 34- Cluter 3- Combined PPT
Eara MiLu
Figure 30: This shows the DFT's of the preactivations of the fine-tuning neurons seen in Fig. 24
The x-axis is the frequency (from 0-90 because this is (a bb) mod 91. The y-axis shows that the representations contributing 42, 35, 28, 21, 14, in descending order. Note the DFT is symmetric about its midpoint so values after 45 contain the same information as the values up to 45.
38

---

lmpp- 91;
For 1, 10, 1,+,,
phase RandonReal [
valu [Re] (8 (2 91) + (2 x/91) / (2 9/)1) /10) phase)) (s. O, 90]];
Print(LiatFiot Joined Trae, FlotRange A11, Mesh Al1]]
1.5
-10
Figure 31: Constructing fine-tuning neuron. This diagram illustrates the step-by-sten process of linear combination of representations.
constructing a fine-tuning neuron, highlighting that it
G.S Histograms of frequency learned counts for simple and fine-tuning neurons
Note that the next two histograms are created by recording frequencies with weights in the DFT the range of (7.5, 30). This is not a sufficient way to always detect fine tuning neurons, and sometimes it will include simple neurons in its counts, however this much more rare. If you consider the ability for neurons with preactivations of specific frequencies to contaminate other neurons frequencies slightly (because they may modify values in the embedding matrix by a small amount), you will see where this counting method can go awry. It is however the case that usually, the contamination coming from different cluster of simple neurons is below 7.5. Thus, these plots should not be considered "accurate" and just approximations.
These plots still useful show the relative frequency of simple neurons fine tuning neurons. The histogram of Frequencies found Fig. 32(a) found a uniform distribution with each frequency showing up about 10k times. Removing the vast majority of contamination by filtering with 7.5 (usually the DFT magnitudes on other frequencies are 0 and if they aren't 't near 0 then they are less than and there simple neuron making use of that frequency in a different cluster (i.e. simple neuron has one big spike with magnitude over 60 on that frequency). This gives us about 2200 fine-tuning neurons found with each frequency, including overcounting because fine-tuning neurons make use of linear combinations of representations and thus their DFT usually has three or more values the range (7.5, 30). Thus the histograms of frequencies associated with fine-tuning neurons are upper bounds on the number of clusters that are identified across 100k random seeds to be fine-tuning neurons. Around 25 percent of runs have fine-tuning neurons in them, but we aren't sure how hyperparameter settings affect this.
39

---

mod 91
mod 89
10k
10k
Counts
5k
5k
0 10 20 30 40
10 20 30 40
Frequency
Frequency
(a)
Counts of occurrences
7 is learned
if 2 is learned
1000
1000
500
500
0
0
O 10 20 30 40
10 20 30 40 Frequency
Frequency
(b)
Figure 32: Histograms of frequencies found across 100k random seeds in MLPs mod 91 (factors 7 and 13) and mod 89 (prime) are both uniform. The fact both prime and composite numbers give uniform distributions is strong evidence that networks are learning the same thing in both problem settings. In fact, this observation that networks do not prefer prime factor frequencies (exact cosets) iss what led which is learned with both prime and composite moduli. (b) Conditional histograms of frequencies over 100k seeds, both mod 91. Left: frequency is found then neurons with
define approximate cosets and identify the abstract approximate CRT algorithm,
or / 43 are less likely (note that 2 43 = -7 (mod 91)). Right: if frequency is present then frequencies 1 and 4 are less likely. The conditional histograms show that networks try to avoid learning frequencies with additive and subtractive relations. This is for similar reason to why the CRT does not work unless all factors of are coprime- they would intersect and boost the value of incorrect logits, substantially increasing the loss.
40

---

Frequencies involved with fine-tuning neurons mod
1500
Frequency
Figure 33: Histogram of frequencies (0-45) associated with fine-tuning neurons over 100k random seeds for modulus91. Note that frequencies 22 and 44 least common, and 13, 26, also appear less frequently, giving case where the neural net is less likely to find some prime factors. It makes sense that neural networks won't always discover the prime factors or they be solving prime factorization.
41

---

Frequencies involved with line tun-tnnnngg neurons mod 89
2000
Frequency
Figure 34: Histogram of frequencies (0-44) associated with fine-tuning neurons over 100k random seeds for modulus 89, Note that frequencies 23 and 43 are the least common.
42

---

G.6 Fine-tuning neurons like additive and subtractive relations
Cond
(a) Fine-tuning Neuron Additive Relations given 15 is a frequency; (a + b) mod 91.
(b) Fine-tuning Neuron Additive Relations given 7 is frequeney; (a mod 91.
Fine-tuning neuron additive relations for two different cases. neuron with frequency 15 is learned, frequencies that are multiples of are more likely to be found. The same applies to 7, which is prime factor of 91, the moduli.
Figure 35:
G.7 Histograms of counts of frequencies being learned in mod 59 and mod 66 across varying depths in the pizza and clock transformers as well as MLPs (with embedding layer)
Here we show histograms counting the frequencies learned and the lengths of the average number of neurons involved frequency cluster given frequency f was learned over 500 seeds for each architecture.
43

---

par-rreuuecy aog
512
Wett 2543
Weth-8105
1632
08
ca
MLP:
Aug
Avg
Avg.
Avg
Prequancy
Frequency
Frequency
Frequency
0.2
0.8
0.8
Pizza:
Aug
04
06
0.6
0.4
0.8
0.4
Frequency
Frequuency
Frequency
Frequency
0.
0.4
0.8
Aug R
22
06
04
Clock:
AWy
0.2
20
Frequency
Prequency
Frequency
Frequency
Proguency Ouster LusterL Liggh
2008
WABE
Wath 1682
MLP:
Pizza:
Ag
Clock:
Figure 36: (a + b mod 59): The first three rows show the histograms of learned frequencies and the bottom three rows show histograms of the average length of cluster, i.e. number of neurons in the cluster.
We check the goodness of fit as a function of frequency, and find that on mod 66 with prime factors 2, 3 and 11, the R² value is much closer to 1.0 for MLPs when frequencies 22 33 are learned (2 11) and (3 x 11). Indeed, the network learns 22 or 33, we see that the length (the number of neurons with that frequency) is substantially lower than if it learns other frequencies. The R² is also higher for MLPs if it learns (2 x 6), however it's not as high in the previous cases, and doesn't 't have the change in the number of neurons that the other two cases have. Overall, this makes sense as in this situation, it can learn the exact cosets instead of needing to learn approximate cosets. Investigation into which frequencies require less neurons is an interesting subject for future study; why is not all cosets?
Some cosets are more likely to be learned than others.
44

---

One nin per frequency mod 55 ang Hf (MUR Mizza clodo
w 20 03
P931 12:4
14277 6796
AiDh THINS
E 8
Avg R
Nig R
MLP:
Avg
P
10
20
30
20
90
Prequency
Frequency
Frequancy
Frequency
R.R. B
Aug R2
Avg 82
Avg R2
Pizze:
20
30
Frequency
Frequency
Prequency
Frequency
B
0.0
Clock:
Aug
R R.
Avg
Avg
AVg
02
Frequency
Frequency
Frequency
Frequency
(he- cun--reeaanyy Ang Sluate Length (HAL Phssa Dook
MUP:
Pizzar
Figure 37: (a + b mod 66): The first three rows show the histograms of learned frequencies and the bottom three rows show histograms of the average length of cluster, i.e. number of neurons in the cluster; note that learning precise cosets results in less neurons being required.
G.8 Noise and ablation studies
In this section, we take the clusters from random seed 133 and we randomly inject multiplicative scaling noise into every weight attached to neurons in the cluster. We do this by multiplying the weight by ex, s ~ NO, o), for various a on the x-axis in Fig. 38(b)
45

---

Vean
(a) Multiplicative Noise injected into every weight of every neuron cluster from a normal distribution dev σ.
Meen Linders
Clubte
(b) Effect of Multiplicative Noise on the function.
Figure 38: Neural network robustness to injected multiplicative noise. The loss remains stable even with std dev of 0.225. Note cluster 3 and 4 are composed of four fine tuning neurons each. Every other cluster composed simple neurons.
46

---

acrou Allasin
Clarar
dadee Lss
Duter Juse
(a) Ablation study showing the impact on the loss function when removing neurons from specific clusters.
Meon Acuuracy LooGs Cusstrr with Abkutoon
Impact of neuron removal accuracy.
Figure 39: Ablation study results. Loss and accuracy metrics highlight the impact of randomly removing neurons "number of ablations" number neurons from a cluster. Note cluster 3 and 4 are composed of four fine tuning neurons each and deletion of every neuron in the cluster doesn't 't affect the accuracy by much. Every other cluster is composed of simple neurons.
47

---

G.9 Number of frequencies
G99.1 More overparameterized means less frequencies
Scaling the number of neurons in the layer achieves experimental results within (logg(n))
Experiments in scaling the number of neurons show that the average number of frequencies found can be shifted based on hyperparameters, but this something we don't fully understand this point in time, but as our results in the main body show-it is still logarithmic. See Fig. 40
Histgrra Number of Oustars Found. num. 2nn.ncuronn1122 sper91 (hwerage 4457)
Mistagram of Numbbrr Clastara Found, nummreurrnee 128, baichhateemddee (huerage 1.3)
198
Haltggram Num be of Cuntara found, NoTu//urrru1111 baich (aiaen) (Aow
Number Clusters Found www.nnoooee-12222 50be
Histtggam Nummbrr of Clastar round www.nuuunnnne batch (óverage
Hittogram am of Number of Ousters Found num, www.onee--S-c barch sitee 9096 (hutrage -5
Hillogram Number Clssars found manunnurrr--8106 batch_ (Avarage 4400))
Hettgramm Nftuuber of Cussters Found teuronshé barch sizergt (Average
300
Cunte
Figure 40: This figure shows that the scaling is always (log(n)), even as the number of neurons is increased from 128, to 512, to 2048. to 8196. The first column is batch size 91, and the second column batch_ size 5096, i.e., the entire training set size. All results are upper bounded by (log(n)).
G.9.2 Adding depth also means less frequencies are learned
In Figure we that adding layers results in fewer frequencies being learned.
48

---

(a+b) mod59: average freqs 1c-03, WD vs. layers
Averagge number frequencles
layers
Figure 41: Adding depth causes the network to learn slightly less frequencies.
Indeed, instead of just looking at this for one hyperparameter combination, check it out for many like we did in figures the main paper (see Figure 42
G.9.3 Deep networks learn error correcting codes: empirical results
Deep networks learn error correcting codes. This discussion was omitted from the main paper's Discussion due to space constraints, but we believe it's interesting.
Our result finding that in deep networks, layers after keep around % of first-order sinusoids (Figure 8) can be interpreted as the network constructing an error correcting code. We see in Figure 42 that deeper networks learn less frequencies. While this is true, they simultaneously achieve lower cross entropy loss (and better margins) with less frequencies than shallower networks [43] This because the first order (simple) neurons in layer 2, compute the exact same coset computation simple neurons performed layer Resultantly, the second order cosine neurons that store the correct answer as cos( (x-f(a+b-o) ), which is maximized at the correct answer (a (b+b) mod = c. receive an additional linear combination Theorem 4.7 This boosts the height of the correct answer (linearly in the number of layers) as function of number of distinct frequencies. Furthermore, it boosts the height of incorrect logits at most (Iog(n)). After softmax, the exponential difference between the correct logit and incorrect logits is thus amplified, and thus the softmax (cross entropy loss) is much lower. With hidden layer it was On (nn)))) In general, with L the number of layers, we conjecture that after softmax, using (log(n)) distinct frequencies will give closer to (n-((() as the height of incorrect logits. This results from the network redundantly doing the same computation "error correct" and thus reduce errors.
49

---

Layer-wise average frequency heat-maps (1 layers)
byer averoge frequencies
Ie-03
Se 04
be-05
30.00 05
Se 00
1e-00
bw-07
10.00
50me
50
70-0
20
LU
Learning
layer average Inqqeencles
Inyer 2 average frequencies
Se 04
5e
Weight doc-v
Sc 05
Sc
1e-05
1e
Se-08
3e-00
20.05
96-9
07
1e 07
te
50 24
10-0
10-35
50-94
E0
Average number
Learning rate
Learning
uyer average lnequencies
layer 2 average frecuanciss
layer averraee Iraquenciss
Se-06
te-0
Sec4
Weight cecay
Weight docay
Se 03
Se
1e
te-00
te-t0
Se-0d
le 05
le
1e 06
1e407
tw-07
teor
50 3u
poos
po-e
E9-9
50 90
ES-
Po
Learning rate
Learning rase
Learning rate
byer avergg frquuedees
laver average freeuende
laye average frequencies
Jayer average froquencics
56.04
Se
$e.04
Se-44
decay
Weight decay
04
Weigh decay
1c
Se-05
09-09
Se-00
Se-05
be-05
Tw-20
Weight
Tw00
1w-0
5e:05
2e-06
Sec0
be-00
l6 05
le
1e 06
1e
07
10:00
10:07
17:07
05
ce
50
00-06
10-95
10-50
50-90
10-11
10-6
10
E0-R
50
10-26
1009
10-96
50-96
20-e1
10
10
in-00
Learning rate
Learning rate
Learning rate
Leaming rate
Figure 42: The number of frequencies found in the network decays as we add layers across almost all
hyperparameter combinations.
50

---

Test CE-loSS vs Total Avg Margin (epoch 5000): MLPs with one embedding
0.014
0.012
-8
avg margin over all data
0.01
test ce loss
-7.5
0.008
test ce loss
total avg margin
0.006
0.004
-655
2
layer count
Figure 43: The loss and margins improve as layers are added, yet figure 42 shows that less frequencies are learned. This can be explained by the presence of first order sinusoidal neurons in layers after layer 1.
G.10 Qualitative: equivariance of the cluster contributions to logits
Here you that the clusters of neurons approximately equivariant to shifts in the inputs, Le.. the cosets shift with the inputs. We show that if you shift (a.b) both by 2, the clusters shift by 4.
51

---

Breaking Down How Clusters Contribute to the Output Logits a=10, b=10- Cluster
a= 12, b= Cluster 0
Contribution
-5
a=10, b=10 Cluster
a=12, b= Cluster
Contribution
2
-2
a=10, b=10- Cluster
a=12, b=12 - Cluster 2
Contribution
1=10, b=10- - Cluster
1=12, b= - Cluster 3
2
Contribution
-0.55
a=10, b=10- Total
a= 12, b= 12 Total
10
10
Contribution
10
55
20
45
50
55
60
65
70
75
80
g0
10
15
20
40
45
50
55
00
0L
08
58
06
Logits
Logits
Figure 44: Clusters of neurons are approximately equivariant to shifts in the inputs, meaning coset clusters shift with the inputs. This suggests the network has learned cosets that it uses to intersect, via linear combinations, to perform the approximate CRT. This example demonstrates that the network did not learn a global minimum-e. e.g., Cluster has only four neurons, limiting its expressivity and equivariance. Cluster frequencies: Cluster 0 (35, coset), Cluster (25, approximate coset), Cluster 2 (8, approximate coset), and Cluster 3 (42, coset). This example uses the same random seed as the ablation study (Fig. (a)),, where Cluster 0 is the most active. This data is from an MLP.
52

---

G.10.1 Pizza model
We take model A, specifically mode1_p99zzpze55/ pt, from S and make Figure 45
which shows that pizzas also output on approximate cosets and perform an approximate CRT. Note for example, that the output logits for the cluster with max freq 15: has maximum activation along approximate coset 3.93, and the neuron activates strongly at a then also activates strongly at
Output logits for cluster with max freq 13 amp, b-10
Neuron O Contributions from Imax freg 15) Nouron OFF Magninude (Coetributions from
Logit value
Regnizude
B
-10
Outgeut takam inddex
20
Output logits for cluster froq
Neuron Contributions
Magnitude
Pte Te-aClvvt
Nogntude
10
30
Frequanicy
30
43
-20
Output token Index
.oontrbbbtios
Neurgn Nagnitude (Contr
Ourput logit cluster with max fron, Input a-?, bw10
20
30
10
20
30
50
00
Frequency Index
Outout token index
Figure 45: This figure shows three neurons and their DFT's, each from one of three clusters in model (a uizza-transformet from [5] for these experiments. Note that the pizza neurons (and clusters) are also implementing the abstract approximate Chinese Remainder Theorem algorithm, despite their low level differences with clocks.
Furthermore, consider that remapping the pizza neurons makes their behavior look almost identical simple neurons when they are remapped, see Figure
53

---

Neuron Normalized Frequency (Cluster freq
Pre-activation
20
30
40
50
60
Remapped
Neuron 1 Normalized Frequency (Cluster max freq 17)
Pre-activation
50
Remapped
Neuron 6
Normalized Frequency (Cluster max freq 3)
Pre-activation
0-
-1
Remapped
Figure 46: Remapping the pizza neurons shown in Figure 45 shows that they look identical to simple neurons.
G.10.2 Clock model
Here we show that the approximate CRT in clock model (Fig. looks just like it does in a pizza model (Fig. 45
54

---

Approximate CRT in a clock-transformer
a=10, b=10 Cluster (Freq: 6)
-2
a=10, b=10 Cluster 1 (Freq: 7)
2
1
-1
a=10, b=10 Cluster (Freq: 9)
0
-2
a=10, b=10 Contributions to Logits
5
0
-5
12
Logits
Figure 47: A view of the approximate CRT within dloc--transformer. We cluster all neurons together with the same frequencies, then inspect the cluster's 's contribution to each logit by summing the contributions of each neuron the cluster each logit. Note that the three clusters each contribute to the logits.
55

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are backed by proof' in Appendix B and Appendix [ and/or backed by figures over many random seeds located in
sections 4.3 anddd
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The limitations discussed in section 6.2 the limitations of this work closer to large open questions rather than limitations. We list them primarily because we want attention drawn to them as we believe them valuable for future discussions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
56

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The entire paper is built around justifying the assumption that neural networks learn sinusoidal functions, required by all theoretical work in the paper. All prior works this same assumption, but we are the first work to display this quantitatively over a gigantic range of hyperparameters, architectures, depths, widths, seeds and moduli.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes. The paper outlines the architectures tested, gives the precise experimental details for making all plots and provides open source code that obsessively GPU optimized: giving anyone with just one GPU the ability to reproduce every plot in the paper over few days worth of compute. We give all versions, even the cuda version in the codehase, in case the order of operations of Jax varies across versions.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
57

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: See section E the code requires all version information in requirements. including information about cuda versions in the readme.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ wttppssccpppbllc/////coc
stipoimiiiiioooooooo.ooc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All of this is in the main paper, but is thoroughly detailed in Appendix 5
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The paper goes to extreme lengths to quantitatively analyze the correctness of our claims, providing R² of our claims over large ranges of hyperparameters, architectures, depths, widths and varied training data (moduli). The data we present actually the worst case for our arguments, being that if model, after replacing neurons by simple neuron
Justification:
58

---

model, classifies a single point wrong, while the model had 100% accuracy on the test and train data, we mark with purple dot. These purple dots are rare and only show up on the edges of where networks can learn i.e. regions with suboptimal hyperparameters: networks are likely fitting more noise in such regions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments in the paper can be reproduced with our code with a single RTX8000 GPU in less than two weeks, we went to great lengths to GPU optimize the code much as possible. The training we did used RT'X8000 GPU's, ancient by today's standards. Each GPU added would divide the training time linearly. The dataset is tiny until large n and is easy to train, with networks learning within 100 epochs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps:/neuuriss. http:///wwurrr..cccaaaaccccccaca
Answer: [Yes]
Justification:
We follow the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
59

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We highlight the positive impacts of our work: better understanding what deep networks learn in the discussion and limitations section 6.2 We can not conceive any negative aspects; this is not capabilities research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
60

---

Answer: [Yes]
Justification: We wrote all the code ourselves. Part of the code a translation, the clock and pizza code from [5] was read, and translated into GPU optimized Jax, this is stated in the main paper "We use the exact transformer architectures from
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcoeee thcode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pane se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
61

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM for what should or should not be described.
62


Reviews:
[{"review_id": "plxn5yngkj", "reviewer": "Reviewer_PXn7", "summary": "The paper presents a novel theoretical framework for understanding how neural networks solve modular addition tasks through a method called aCRT. The authors propose that various neural network architectures utilize the same underlying algorithm to tackle these tasks, relying on a mechanism based on approximate cosets.  The paper also extends the framework to group multiplication tasks, offering empirical results to support the validity of the proposed theory.", "questions": "How does the proposed framework extend beyond modular addition? Can you provide experiments showing its applicability to more complex tasks? How does the framework apply to tasks beyond modular arithmetic or group multiplication? Are there limitations in generalizing the aCRT framework to other operations? I would appreciate some clarification on these points, particularly in the context of LLM applications. How could aCRT be applied to LLMs? Can it improve efficiency or interpretability, especially for tasks like reasoning or generation?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- This paper presents a novel approach by unifying previous interpretations of modular addition and extending this to group multiplication, which could have broader implications for understanding neural network mechanis\n- The concept of aCRT and the use of approximate cosets to explain modular addition provides a new perspective on neural network behavior and offers potential computational benefits, improving efficiency.\n- The theoretical framework is backed by solid empirical results that demonstrate its applicability to neural network architectures.\n \nWeaknesses:\n- The experiments in the paper primarily focus on modular addition tasks, with limited validation in more complex or real-world tasks. While the theoretical framework is strong, there is less discussion on its practical applications. Could you provide more experiments on the framework's application to more complex tasks, especially those beyond modular addition and group multiplication? This would help better understand the framework's generalizability and practical feasibility.\n- The paper introduces the aCRT framework but does not directly compare it with existing similar methods. A comparison with existing approaches would offer clearer insights into the advantages and limitations of the framework in practical applications.", "was_revised": true, "final_justification": "This paper explores the idea that neural networks solving modular addition may share a common underlying algorithmic structure. I initially raised questions about its generalizability beyond modular addition and the lack of comparison with similar methods. The rebuttal clarified these points and better positioned the contribution. While experiments remain focused on modular addition, the theoretical insight is strong and may inspire future work.", "rebuttal": "Thank you for your review. We’re glad that you highlighted the unification of prior interpretations of modular addition (and its extension to group multiplication) as a key strength, as this was a central goal. We also appreciate your recognition of the novelty of the approximate Chinese Remainder (aCRT) framework, its potential to shed light on broader neural network mechanisms, and the potential computational benefits suggested by this perspective. Finally, we're glad you found our theoretical contributions well-supported by broad empirical results across architectures, which we see as essential to making robust and testable interpretive claims.\n\nThe concerns mostly relate to the framework's applicability to more complex tasks. We address all concerns below.\n\n# Weaknesses\n\n> W1. The paper introduces the aCRT framework but does not directly compare it with existing similar methods. A comparison with existing approaches would offer clearer insights into the advantages and limitations of the framework in practical applications.\n\nWe utilize the methods of prior work and in fact build on them. \n- **Nanda et al.**: Fit second order (degree 2) sines and cosines through neuron activations, then replaced the neurons by the fit to ensure accuracy remained 100%. We augment this with a more rigorous evaluation framework, testing order 1 (degree 1) sine and cosine fits as well as many more hyperparameters. This gave two novelties:\n  - Figure 8 shows that order 2 sinusoids are more expressive than necessary. Note: we will update the caption to state clearly that the purpose of this figure is to show that Nanda et al.’s second order sinusoid framework is too expressive for layer 1.\n  - The discovery of fine-tuning neurons that learn sawtooth functions, studied in Appendices G.4, G.5 and G.6. Notably, sawtooth functions violate the interpretations of all prior work, but they still fire on approximate cosets as remapping shows in Figure 28 in the appendix. Remark: they are only found with bad hyperparameters that cause overfitting of the training set so they may not be of relevance to deep learning practitioners, but serve to show how exhaustively we pushed prior methods. \n- **Zhong et al.** Primarily utilize Principal Component Analyses (PCA) to conclude two algorithms exist: pizza and clocks. \n  - Please see Appendix G.1 where we answer the open questions presented by Zhong et al., about their principal component analyses (PCA). By augmenting their PCA plots with the 2d-Discrete Fourier Transform (DFT), we show their circularity metric isn’t needed and that non-circular embeddings (Lissajous) are simply an artifact of PCA, i.e. Lissajous embeddings do not correspond to networks learning differences. We show a direct comparison between Zhong et al.’s PCA plots and our DFT augmentation in Figures 14-23.\n  - See Appendix E.1. where we state we download checkpoints for networks on their Github (clock and pizza).\n- **Morwani et al.** Showed empirically that 1 layer networks learn O(n) features (frequencies) and rigorously proved it. Morwani et al. only study the margin in one layer shallow networks. \n  - We generalize to depth (see Fig. 4 showing 1-layer networks learn O(n) whereas deep networks learn O(log(n))); the caption will be updated to indicate Morwani et al.’s case is in blue learning O(n) features. \n  - See Appendix C “Proofs and details for Theorem 4.7 and Corollary 4.8” where Figure 12 studies the empirical average margin over 500 networks. This supports the predicted margin of Corollary 4.8 for deeper networks.\n  - Appendix G.9.3, Fig. 43 shows how margin and cross entropy loss correlate as the number of layers are increased.\n\n> W2. The experiments in the paper primarily focus on modular addition tasks, with limited validation in more complex or real-world tasks. While the theoretical framework is strong, there is less discussion on its practical applications. Could you provide more experiments on the framework's application to more complex tasks, especially those beyond modular addition and group multiplication? This would help better understand the framework's generalizability and practical feasibility.\n\nWe believe this is outside the scope of the paper. We worry such additions could distract from our core message and the importance of confirming Conjecture 4.9. We think our result that different architectures do not learn disparate algorithms like prior work suggested, and simply learn different implementations of one abstract algorithm is fundamental. **We’d like to note that reviewer EPiJ stated under significance** “Thus, the work stands to be highly significant.\" and later “I think this work will likely inform a lot of subsequent work and provide a useful perspective on the emergent behaviour of neural networks on algorithmic tasks.”. \n\nThat said, you may find interesting that training language models to output a sorted list when given an unsorted list as input can be rewritten formally as a group multiplication. Thus, Conjecture 4.9 predicts that such models will utilize approximate cosets so we give a direct path for future work to immediately study more complex questions with language models using our framework.\n\n# Questions\n\n> Q1. How does the proposed framework extend beyond modular addition? \n\nRecall: approximate cosets are literally subproblems of the original problem—they divide the Cayley graph into smaller sets. Thus, the confirmation of Conjecture 4.8, that approximate cosets will be learned for all group multiplication tasks, would provide the foundation for a falsifiable theory of deep learning: the weights of neural networks converge to implementations of abstract divide and conquer algorithms. \n\n> Q2. Can you provide experiments showing its applicability to more complex tasks? \n\nSee weakness 1 above Questions.\n\n> Q3.  How does the framework apply to tasks beyond modular arithmetic or group multiplication?\n\nSee Weakness 2).\n\n> Q4. Are there limitations in generalizing the aCRT framework to other operations? \n\nThere are no limitations in generalizing it to other groups, though the work to do so is constructive and will build on our framework. To generalize the aCRT to all tasks, first, conjecture 4.9 should be resolved. It’s formal resolution will yield hints for how to generalize the framework toward proving neural networks may implement abstract divide and conquer algorithms on all datasets. We could make this conjecture now because it does make sense—it explains the unreasonable efficiency of deep learning—but we believe it’s too early because it’s not yet resolved on group multiplications.\n\n> Q5. I would appreciate some clarification on these points, particularly in the context of LLM applications. How could aCRT be applied to LLMs? Can it improve efficiency or interpretability, especially for tasks like reasoning or generation?\n\nPrimarily, this work serves to help interpretability by resolving the disparate interpretations of prior work and re-opening the universality hypothesis as a now testable conjecture. Secondarily, the ideas in this work have the potential to scale to address what LLMs learn on non-group tasks. Thus, they could, if they workout, provide the groundwork for a falsifiable theory of deep learning.\n\n## A recent position paper gives context: our work is timely\n\nThe ICML 2025 position paper, “We Need an Algorithmic Understanding of Generative AI” [1], argues that the ML community must move beyond fragmented, bottom-up interpretability, ​​which often focuses on individual neurons, components or circuits without a guiding hypothesis. Instead, they propose the prioritization of top-down hypothesis/theory-driven research that explains how models implement full algorithms. The authors write in reference to prior approaches: \"current findings are still largely fragmented, and we lack a solid theoretical foundation for understanding how these various components come together to implement algorithms.\" Furthermore, they argue that such an approach would close the theory-interpretability gap, since both communities are operating rather disjointly. And this would lead to better understood, safer, and more efficient models.\n\nOur core message is in service of the research agenda outlined in [1], and we think this will help our work be extended to more complex tasks and relates to questions about LLMs.\n\n# Final thoughts\n\nWe hope that we’ve successfully clarified our contributions and how our research fits into the literature and provides ample opportunities for future work. If this addresses your concerns clearly, we would greatly appreciate if you could consider increasing your score.\n\n## References\n\n[1] Eberle et al., \"Position: We Need An Algorithmic Understanding of Generative AI\", ICML 2025."}, {"review_id": "zzlI1JyFv0", "reviewer": "Reviewer_EPiJ", "summary": "This work presents a theory of how ReLU neural networks (MLPs and Transformers) learn to perform modular addition. The algorithm implemented by the network is termed the approximate Chinese Remainder Theorem (aCRT). This is achieved by forming structures called approximate cosets - a generalisation of cosets which define equivalence classes within the cyclic group defined by the modulo operator. The generalisation to approximate cosets then lets sets of \"close\" cosets be treated equivalently. Empirical experiments support that neurons in networks trained to perform modular addition can be replaced by \"simple neurons\" - neurons specialised to specific frequencies - without a significant drop in performance, support the notion that the networks are learning the aCRT.", "questions": "1. What qualitative differences should I be noting in Figure 1.\n2. Why is an $R^2$ score of $0.3$ acceptable in Figure 7 as long as the green tick is there?\n3. How does the presence of a green tick in Figure 6 and 7 corresponding to the actual behaviour or representations of the network.\n4. Why is the maximum output logit over the entire dataset used in Theorem 4.7 and is there no tighter bound for this?\n\nI would definitely be inclined to raise my score if some of the questions were answered and it turns out I have missed something important.", "limitations": "Overall the assumptions of the work are clearly stated and the Discussion section is fair and stated the necessary limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n## Quality\nThe work does a good job of covering the theoretical background on group theory. Particularly sections 1, 2 and 3 are particularly effective at contextualising the work and also introducing the technical details of modular arithmetic. The assumptions necessary for the theory are stated clearly and the design of the experiments is well suited to establishing the validity of the assumptions. Precise definitions are given for the necessary concepts. Overall, the goals of the work are clear and the theory presented is precise with the necessary empirical support to justify the framework.\n\n## Clarity\nI reiterate that I think the first few sections (up to Section 4) are very well done and support clarity immensely. Figures, particularly Figure 2, are helpful with sufficiently detailed captions to aid understanding. I also appreciated the helpful analogies which help broaden the applicable audience of the work by given more readers something to related to - for example the analogy to the breadth-first search on Lines 161 to 168. The notation used is intuitive and consistent which aids clarity.\n\n## Significance\nThe claimed findings of this work unify a number of different perspectives on a topic which has received quite a bit of recent interest. Thus, the work stands to be highly significant. I do have some questions about the precision of the framework, but taking the claims on their own merit then I think this work will likely inform a lot of subsequent work and provide a useful perspective on the emergent behaviour of neural networks on algorithmic tasks. I also find the comparison between MLPs and Transformers here useful and the general lack of a very clear behavioural difference is interesting as I usually only associate these sorts of tasks (and mechanistic interpretability broadly) with Transformers.\n\n## Originality\nSimilar to the point on significance, I think that the combination of recent ideas from the literature being combined is original and results in a potentially powerful framework for understanding the sub circuits responsible for modular addition. The approximation to the CRT is also new and so there is a decent amount of originality overall. The originality is also supported by the detailed background in Section 2 which helps make it clear what is new to this work.\n\n# Weaknesses\n## Clarity\nSection 4 and the following sections do lose some of the clarity which was fantastic up to that point. For example, the equations on the bottom of page 3 are foundational to the rest of the work (introducing the very concept of a simple neutron) but introduced quickly in comparison to the previous concepts. I also just found the notation of $\\omega(A,B)$ to be a bit unintuitive as it is the dot product of an embedding $U$ with a set of weights not depicted as parameters to the function. Secondly, Theorem 4.7 and Corollary 4.8 are the main theoretical results and yet they are stated quite bluntly, with little attempt to provide intuition or any sort of proof sketch. Some intuition on why is the key equation of interest here $m' -h(k) >\\delta m'$ would be very helpful. As a minor point I also think writing this as $m'(1-\\delta) > h(k)$ would be more intuitive and in line with the point of the theorem.\n\n## Quality\nSome of the results of the theory appear to be true by construction or at least the conclusions drawn seem to not be wholly justified. I am hoping these concerns are more a consequence of my own misunderstanding. Firstly, Figure 1 shows the preactivations of a neuron where the frequency has been normalised. It is claimed that this figure shows the qualitative equivalence of the neurons, but to me everything about it seems different except for the frequency. The range of values and the phase are different, so it is unclear what qualitative property I should be focusing on here. Secondly, the general approach of Section 4.4 is a bit of a concern, but also the statement on Lines 188 and 189 which says that all architectures trained to perform modular addition are abstracted by approximate cosets. I am not certain what the alternative would be - is this not true by virtue of the network being trained to perform modular addition itself? This alone does not seem to be a tight enough correspondence to draw conclusions about the actual circuitry of the network itself. Relatedly, in Figure 7 for example there are points where the $R^2$ value is as low as $0.2$ to $0.3$ between the network and simple neutron replacement but then there is still a green tick (this occurs in Clock: $1^{st}/4)$ for example. The green ticks presenting no loss in accuracy is used as the key point demonstrating that simple neurons are a good abstraction but this just makes me thing the green arrows are not a precise enough measure of success when it can have such as high range of $R^2$ values. Finally, in Theorem 4.7, it appears strange to me that the bound is given in terms of the maximum output logic across the dataset ($m'$) when it is compared to individual output logics $(h(k))$. I appreciate that the point is to show that sufficiently large bounds exists between the maximum logit and all possible incorrect logins, but the fact that this is not done per data point seems to make this quite imprecise. I suspect this question has more to do with my difficulty with the clarity of this portion of the paper.", "was_revised": true, "final_justification": "The authors provided a very detailed response to my review and answered my questions. With these points made clear I am happy to advocate for acceptance and thing the proposed changes will benefit the clarity of the work.", "rebuttal": "Thank you for your thoughtful and detailed review. We're excited that you recognized the originality and significance of our framework. We appreciate your compliments on the clarity and precision of the early sections. Your summary reflects a strong understanding of our goals and we’re glad that you thought our work stands to be highly significant. We think your concerns stem from areas we can readily clarify and are encouraged that you indicated you could raise your score if we succeed.\n\n# Weaknesses\n\nThanks again for praising the clarity in sections 1, 2, 3 of our paper. We think that by addressing your points and incorporating your suggestions here, we can clarify the difficult sections of the paper.\n\n## W1. Clarity and Presentation\n\n**Drop in clarity in section 4.** Due to lack of space, we agree that some things were introduced tersely.\n- We will revise the simple neuron equations to be more prominently presented and clarified, and revise the notation to reflect their dependence on network weights.\n- We will add intuition for Theorem 4.7 and Corollary 4.8, though we may relegate the proof sketch to the appendix as we don't think it's strictly necessary in the main text.\n- We agree with you to state Theorem 4.7 as $m’(1-\\delta) > h(k)$. Thank you for the suggestion.\n\n## W2. Quality\n\nThe concerns and questions you raise here appear to be in direct correspondence with Q1, Q2, Q3, and Q4, so we address them in the Questions section.\n\n# Questions\n\n> Q1. What qualitative differences should I be noting in Figure 1\n\n- **Fig. 1 should qualitatively show the effect of remapping.** Neurons with different frequencies in MLPs, clocks and pizza transformers look like sine functions, but appear to have slight qualitative differences, e.g. MLP looks less structured compared to pizza, but after remapping, they’re qualitatively both sine functions with frequency 1.\n- Due to our exhaustive search over training settings, we found neurons missed by prior work that qualitatively looked like sine functions, yet $R^2$ scores implied they weren’t quantitatively.\n- After remapping, we saw these neurons had learned sawtooth functions, not sine functions. \n- Sawtooth functions are a sum of sine functions with different frequencies. Note: the first sine function in the sum explains some of the variance, so a sine function of best fit captures the key quantitative properties.\n- These neurons emerge at the boundary of generalization: when hyperparameters are set before the network fails to generalize to 100% test accuracy in training.\n- Replacing all neurons (including sawtooth neurons) with sine functions of best fit doesn’t change the networks test accuracy because the extra sine functions they learned were helping them overfit the training set—they were not relevant for generalization. \n- We study various properties of these neurons in Appendices G.4, G.5 and G.6.\n- Of note is Figure 28 in G.4 that shows how remapping can qualitatively show if a different function is learned. Figure 30 shows that the sawtooth shown in Figure 29 is a sum of sine functions with frequencies {42, 35, 28, 21, 14, 7}. \n- We aren’t sure if these are of practical relevance—they would never be found by a deep learning practitioner hypertuning for good parameters. Indeed, this is why prior works missed them. \n\nThus, we will make the caption of Fig. 1 point to appendix G.4, Figure 28, so that interested readers can see how remapping can qualitatively discern different functions. \n\n> Q2. Why is an $R^2$ score of 0.3 acceptable in Figure 7 as long as the green tick is there?\n\nBecause the green tick indicates 100% test accuracy was preserved after replacing all neurons with their sine fits. Even if the $R^2$ of individual neurons is low (e.g., for sawtooths), the network’s behavior remains unchanged, showing sawtooths aren’t critical to generalization out of the training data. \n\n> Q3. How does the presence of a green tick in Figure 6 and 7 corresponding to the actual behaviour or representations of the network.\n\nThe green tick shows whether the network gets the correct answer after we replace all neurons with their best fits. It serves to elucidate the boundary cases, where hyperparameters are bad, but not so bad as to prevent generalization.\n\nElaboration on weakness. The paper states: all architectures trained to perform modular addition are abstracted well by approximate cosets. You ask: I am not certain what the alternative would be - is this not true by virtue of the network being trained to perform modular addition itself? This alone does not seem to be a tight enough correspondence to draw conclusions about the actual circuitry of the network itself.\n\nThere are two questions here and the first one is truly profound. The answer is no: it’s not true simply by virtue of the task. The second question: “This alone does not seem to be a tight enough correspondence to draw conclusions about the actual circuitry of the network itself.” is actually our goal. We are not making claims about the circuitry. We are abstracting the specific details of the circuitry away with a data structure—paths on Cayley graphs called approximate cosets. We use this abstraction to rigorously prove that disparities in prior interpretations occurred due to weights of different architectures converging to different implementations of one divide and conquer algorithm that we give: abstract algorithm 4.6.\n\nOur idea to abstract away circuits is very timely. A recent ICML 2025 position paper, “We Need an Algorithmic Understanding of Generative AI” [1], argues that the ML community must move beyond fragmented, bottom-up interpretability, ​​which often focuses on individual neurons, components or circuits—without a guiding hypothesis. Their position is to prioritize top-down hypothesis/theory-driven research that explains how models implement full algorithms. The authors write in reference to prior approaches and even cite the works we unify: \"current findings are still largely fragmented, and we lack a solid theoretical foundation for understanding how these various components come together to implement algorithms.\" Furthermore, they argue that such an approach would close the theory-interpretability gap, since both communities are operating rather disjointly.\n\nOur idea to abstract away circuit level details gave two contributions that we believe are substantial. 1) we address the call to close the theory-interpretability gap and provide a theoretical framework and show how to utilize abstraction to form an algorithm out of learned circuits. 2) To our knowledge, we are the first to prove that the solutions learned by stochastic gradient methods can converge to implementations of divide and conquer algorithms by providing an example of it happening. Our Conjecture 4.9 gives a route to proving that neural networks learn implementations of divide and conquer algorithms on many tasks.\n\nBack to the profound question: the community has no theoretical reason to believe the network should prefer to learn cosets or approximate cosets. Recall our approximate coset definition is a path on a Cayley graph. Since neurons only activate on approximate cosets, each neuron activates if the answer to the question: “is the answer in my part of the Cayley graph?” is yes. But why learn approximate cosets? Ultimately, approximate cosets are functions on the Cayley graph. Any function can pick elements on the Cayley graph to serve as the foundation for these questions. Indeed, approximate cosets tend to divide a Cayley graph in half—this is in fact why we conjectured a O(log(n)) bound and later proved it. But many functions could be learned that divide the Cayley graph in half!\n\nThe goal of Morwani et al., was to propose a theory to explain why networks learn sinusoids. For their theory, they rigorously proved it was because learning at least 4 sine functions of each frequency was the max margin solution. They tested this in 1-layer MLPs and found that empirical reality matched. Our tests on multi-layer networks tell us this isn’t the full story: >=2-layer networks no longer learn the max margin solution.\n\nIndeed, we prove deep networks utilize O(log(n)) types of approximate cosets (features) and not the brute force O(n) max-margin, thus achieving the algorithmic efficiency of a divide and conquer strategy. We form a connection to the famous Chinese Remainder Theorem divide and conquer algorithm, showing it utilizes O(log(n)) cosets, which we point out are modular subsystems of the original problem (subproblems). \n\nThus, neither the community or we have an answer. Lines 312-313 in future work propose further investigation into this.\n\n> Q4. Why is the maximum output logit over the entire dataset used in Theorem 4.7 and is there no tighter bound for this?\n\nThe maximum logit for each piece of data is used. For any individual datum (a,b) input to the network, the maximum output logit is the correct logit. Thus, the bound is tight for all data. This will be clarified in the text.\n\nWe think it’s unlikely a better bound exists because the bound is tight to experimental results. Recall: it predicts O(log(n)) features will be learned on average and the >= 0.989 $R^2$ values for logarithmic fits in Figure 3 support this. It also predicts the margin will grow logarithmically (before applying the softmax exponential), which is verified experimentally in Figure 12 in appendix C with $R^2 \\geq 0.988$.\n\n# Final thoughts\n\nWe hope that in light of these clarifications, and our presentation of a new position paper you see added value in our contributions at this time. We look forward to our continued discussion. \n\n## References\n[1] Eberle et al., \"Position: We Need An Algorithmic Understanding of Generative AI\", ICML 2025."}, {"review_id": "nyWoQ4ZqVj", "reviewer": "Reviewer_2YER", "summary": "This paper examines the mechanism by which neural networks solve the modular addition task and advances a testable universality hypothesis. While prior studies have attributed diverse internal representations—such as the block and pizza algorithms—to fundamentally different learned solutions, the authors argue that these can be unified under a single abstract algorithm, namely the approximate Chinese Remainder Theorem (aCRT). A key theoretical contribution is the derivation that deep neural networks (DNNs) require only $O(\\log n)$ features to solve modular addition, matching the empirical results.\n\n\n**Overall Recommendation**\nThe paper presents an interesting attempt to unify multiple mechanisms by which DNNs solve modular addition tasks. However, it lacks a clear articulation of how the proposed aCRT framework differs from and subsumes prior interpretations. Providing a clearer intuition explanation  would substantially enhance the contribution and may warrant a higher score.", "questions": "- Could the proposed explanation be extended to modular addition tasks with longer length, i.e., $a_1 + \\cdots+a_k (\\text{mod}   p)$? \n-  Could the authors clarify, perhaps through a simple example, how the approximate Chinese Remainder Theorem is applied to modular addition? Traditionally, the Chinese Remainder Theorem addresses systems of congruences rather than modular addition itself.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Clear and straightforward setup\n\n- The paper offers a conceptually compelling attempt to unify disparate neural network mechanisms for modular addition within a single theoretical framework, called the approximate Chinese Remainder Theorem. The motivation and ides are interesting.\n\n- In illustrating how many frequencies are needed to instantiate the aCRT, Figure 3 provides strong empirical support for the theoretical result established in Theorem 4.7.\n\n**Weakness**\n-  The formal definitions (Definition 4.1, 4.2 and 4.3 ) are difficult to follow and lack sufficient intuitive grounding. To enhance clarity and accessibility, it is recommended that the authors illustrate key concepts—such as step size, frequency normalization, and approximate sets—through simplified, concrete examples. In addition, the manuscript would benefit from a clearer articulation of the motivation and underlying intuition behind these definitions.\n- From the reader’s perspective, the proposed explanation based on the approximate Chinese Remainder Theorem offers an alternative perspective on how neural networks solve the modular addition task. However, it remains unclear whether this explanation genuinely subsumes prior interpretations—such as the pizza and clock algorithms—or merely stands as another competing account. The manuscript lacks a clear theoretical or intuitive justification for why these previously observed mechanisms can be interpreted as specific instances of the proposed abstract algorithm. To strengthen the unifying claim, the authors are encouraged to provide deeper theoretical insights or illustrative examples demonstrating how and why these distinct algorithmic patterns emerge as manifestations of their framework.\n- The authors draw conclusions based on a narrow set of experimental conditions, while the phenomena they aim to unify were originally observed in diverse settings.  The paper would be more convincing if it provided experimental evidence from the original settings used by others and demonstrated that the coset explanation holds consistently across different settings for the same task.", "was_revised": true, "final_justification": "The authors have addressed some of my concerns in their revisions. As a result, I am raising my score to 4.", "rebuttal": "Thanks for reviewing our paper! We appreciate the time you took to engage with our ideas. We're glad you found our unification via the approximate Chinese Remainder Theorem (aCRT) conceptually compelling, and that you highlighted the clarity of our setup and the strength of the empirical support for our theoretical claims (e.g., Figure 3, Theorem 4.7). We're encouraged by your openness to increasing the score with further clarification.\n\n# Weaknesses\n\n> The authors draw conclusions based on a narrow set of experimental conditions, while the phenomena they aim to unify were originally observed in diverse settings. The paper would be more convincing if it provided experimental evidence from the original settings\n\nWe respectfully disagree that our conclusions rely on narrow experimental conditions and believe there may be a misunderstanding. Our analyses both replicate prior setups and substantially expand their scopes through a far more rigorous evaluation framework. Specifically, we vary key hyperparameters: depth, width, learning rate, L2 regularization, batch size, and training seed, training over 1 million networks in total. To our knowledge, no prior work has conducted such large-scale empirical verification. These experiments show that our theory holds robustly across diverse training settings.\n\nTo demonstrate the breadth of our experiments, and how our theory subsumes prior work, we highlight key empirical findings that go beyond all existing models:\n- Counterexamples to prior theories: We show that small batch sizes prevent the formation of representation matrices (Figure 13), producing clear counterexamples to the interpretation proposed by Chughtai et al. These are detailed in Appendix D.\n- Previously undocumented behavior (sawtooth functions): We identify a new class of representations, sawtooth functions, which arise at the boundary between successful and failed training. These functions are not captured by prior theories but are accounted for by ours, as they activate on approximate cosets (Appendices G.4-G.6).\n- Reanalysis of prior models: We directly evaluate publicly released checkpoints from Zhong et al. (“pizza” and “clock” transformers; see Appendix E.1). Using their methodology, we extend the analysis and resolve their open questions about circular embeddings. Specifically, we show that models do not learn non-circular embeddings, and that their proposed circularity metric is unnecessary (Appendix G.1).\n- In Appendix G we run 26 pages of additional results\n\nThese results reveal where previous interpretations fail and demonstrate that our framework is robust. For additional comparisons to prior work, please refer to our response to reviewer 1.\n\n> The formal definitions (Definition 4.1, 4.2 and 4.3 ) are difficult to follow and lack sufficient intuitive grounding. To enhance clarity and accessibility, it is recommended that the authors illustrate key concepts—such as step size, frequency normalization, and approximate sets—through simplified, concrete examples. In addition, the manuscript would benefit from a clearer articulation of the motivation and underlying intuition behind these definitions.\n\nIn the main text we provide examples of these key definitions; please see Section 3 and for illustrations of the key definitions please see Figure 2. For “simplified, concrete examples” please see Appendix A.2 titled “Additional Mathematical Background”. For more examples and “a clearer articulation of the motivation and underlying intuition behind these definitions.” see A.2.1 titled “Examples: cosets, Cayley graphs, step size d”. \n\n> It remains unclear whether the aCRT explanation genuinely subsumes prior interpretations (e.g., pizza and clock algorithms), or merely stands as another competing account.\n\nWe believe the evidence provided demonstrates that the aCRT framework does not merely offer an alternative explanation; it subsumes all prior interpretations. Unlike previous work, which presents disjoint, architecture-specific theories (e.g., Nanda et al., Zhong et al., Morwani et al.), our theory offers a single, mathematically grounded explanation that holds across all tested architectures and training regimes.\n\nThis work represents a substantial contribution because it unifies previously disjoint theories, and introduces a new line of thinking. Prior to our study, the field had begun to accept that different architectures learn fundamentally different circuits, in part due to conflicting interpretations in the literature. For example, Zhong et al. has received significant attention for highlighting such divergence, leading to broader concerns that mechanistic interpretability may not generalize across architectures.\n\nThis concern is echoed in papers citing Zhong et al., including MoSSAIC: AI Safety After Mechanism (Farr et al.), which writes: \"Researchers have successfully reverse-engineered these models and discovered that they learn a specific, intricate algorithm based on trigonometric identities and Fourier transforms, implemented via clock-like representations in the attention heads. The specific implementation is fundamentally tied to the architectural properties of the transformer substrate. A different architecture would almost certainly learn a different algorithm, rendering this detailed explanation obsolete.\"\n\nOur findings directly challenge this conclusion. We show that different architectures do not learn fundamentally different algorithms; rather, they learn distinct implementations of a shared abstract strategy, formalized by the aCRT. In this sense, our framework reveals a common algorithmic structure underlying previously conflicting interpretations.\n\nThis insight also aligns with the argument in “Position: We Need an Algorithmic Understanding of Generative AI” (Eberle et al., ICML 2025), which emphasizes the need for theoretical foundations that explain how neural components work together to implement algorithms. We believe our paper is the first to rigorously demonstrate that neural networks can instantiate abstract algorithms in their learned weights. In doing so, we contribute two key insights:\n1. While it is widely hypothesized that neural networks may learn algorithms, we offer the first formal demonstration that they instantiate an abstract algorithm.\n2. Proving such behavior requires a new theoretical framework, one that characterizes the learned solution at an abstract level, as our notion of approximate cosets enables.\n\n\n> The manuscript lacks theoretical or intuitive justification for why these previously observed mechanisms can be interpreted as specific instances of the proposed abstract algorithm. To strengthen the unifying claim, the authors are encouraged to provide deeper theoretical insights or illustrative examples demonstrating how and why these distinct algorithmic patterns emerge as manifestations of their framework.\n\nIn section 4.1 we introduce approximate cosets. These are more general than cosets, as every coset is an approximate coset, but approximate cosets also include modular subsystems that do not perfectly divide the modulus. Then we provide Fig. 2 as a visual example to help the reader geometrically understand the approximate coset definition. Then, we introduce Theorem 4.4, which states: a ReLU neuron only activates if it is activating on an approximate coset. Later, section 4.2 provides a step-by-step guide on how to build the abstract approximate Chinese Remainder Theorem (CRT) algorithm utilizing Theorem 4.4. \n\nThis is done by first guiding the reader through a construction (Remark 4.5) that creates an abstract classical CRT algorithm that neural nets utilize only if they learn frequencies that are divisors of the modulus. The classical CRT (as you ask later in your review) only works for non-prime moduli because it utilizes cosets which only exist if divisors of the moduli exists, but the approximate CRT will work for frequencies that don’t divide the modulus. Thus, to build the more general approximate CRT from the classical CRT (Remark 4.5), we only need to relax one thing: we replace cosets by approximate cosets. This gives Abstract Algorithm 4.6: the approximate CRT. \n\nThe theoretical justification: By Theorem 4.4, we know for a fact that all neurons activate only on approximate cosets. We know from number theory that a number can have at most O(log(n)) prime factors (e.g. consider 2^n, which has the factor 2 n times). So we must prove the aCRT can use O(log(n)) frequencies. Indeed, this is Theorem 4.7, and we empirically verify that networks have logarithmic growth rates with R^2 scores >= 0.989 in Figure 3 on moduli ranging from 3 to 4999.\n\nThe empirical justification: As explained above, the results in the paper show beyond any reasonable doubt in Figures 3-9 that across very diverse training settings our theory always explains the weights networks converge to. No previous paper tested their interpretation under as broad of a variety of circumstances or reported this. \n\n# Questions\n\n> Could the proposed explanation be extended to modular addition of longer length i.e., a_1+a_2+...+a_k (mod p)?\n\nYes. Simple neurons become a sum of k sinusoids, one for each a_i.\n\n> Perhaps through a simple example, how is the approximate Chinese Remainder Theorem is applied to modular addition? Traditionally, the Chinese Remainder Theorem addresses systems of congruences rather than modular addition itself.\n\nPlease see Appendix A.2 dedicated to examples, particularly A.2.1, which visually illustrates how the Chinese Remainder Theorem (CRT) could be computed by a network that learned sine functions in its neurons. After seeing this, we trust it will be clear how approximate cosets can replace cosets, giving our core result: the approximate CRT.\n\n# Final thoughts\n\nWe appreciate your suggestions to improve clarity and your openness to increasing the score. We hope this rebuttal addresses your concerns and are happy to provide further clarifications."}, {"review_id": "cS8b0p3KIW", "reviewer": "Reviewer_RppL", "summary": "The paper introduces a unifying theory to show how the model learn to sum two numbers mod $n$. \n\nThey introduce the aCRT idea and show that NN model requires $O(\\log n)$ features, which is the upper bound of the number of prime factors which divides $n$. aCRT theory in conjunction with the experimental section shows also why NN networks solve modular addition.", "questions": "Q1. Do you have an explanation of why 1 layer model cannot learn this? Is it related to the famous paper from Antrophic about induction head and non-learnability of the copy task from 1 layer model?\n\nQ2. Going from $O(\\log n)$ to $O(1)$ features is extremely important. Can we find some particular versions of NN with some form of induction heads to effectively use less features?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "S1. The most important contribution is how authors unified the theory behind modular arithmetic disseminated across multiple works into a single theory. Also, although I didn't check all the details of the theorems, it sounds pretty right.\n\nS2. Authors opened a new route on the mechanical interpretability, without looking only at the model weights only or PCA style analysis, but more holistically at the entire model itself. \n\nS3. The experimental section is just enough to show results that are coming from the theory.\n\nW1. I suggest authors to change the \"spectacular\" wording used in the abstract: I am not sure this understanding cannot be easily extended to other modular tasks, especially when other operations are introduced. \n\nObservation. I probably need more explanations on how model goes from coset to the final output representation, it's not super clear from the writing. It's written in Remark 4.5 that \"the argmax of ... selects the correct answer\", but I am interested to clearly understand the mechanism to select the correct answer.", "was_revised": true, "final_justification": "Authors justified and explained the minor weakness and the questions I had.", "rebuttal": "Thank you for your review. We're especially glad you identified our unification of prior work across theory, interpretation, and architectures as the central contribution. That was precisely our goal. Prior interpretations were often cited as evidence *against* universality, presenting a fragmented picture; we show that they are simply different implementations by different architectures of the same algorithmic strategy, what we call an **abstract algorithm**. We also appreciate your recognition of our holistic, network-level analysis, and that our theoretical predictions were validated empirically.\n\n# Weaknesses \n\n> W1. I suggest authors to change the \"spectacular\" wording used in the abstract: I am not sure this understanding cannot be easily extended to other modular tasks, especially when other operations are introduced.\n\nWe thank the reviewer for this feedback. While the phrasing is somewhat ambiguous, we interpret it as questioning the generalizability of our findings to other modular or non-group tasks. If there are specific phrases in the abstract that appear overstated or unclear, we would welcome concrete suggestions. That said, we’d like to clarify that the abstract is intended to reflect both the core contributions of our work and the scientific philosophy behind them.\n\nYou identified the unification of prior work as the most important contribution, and we agree. Unifying previously incompatible interpretations was not only novel but necessary: without understanding the algorithmic structure in modular addition, a task once seen as fragmented, there is little hope of generalizing to more complex settings.\n\nThis resolution reframes modular addition: what were thought to be incompatible solutions now reveal a single underlying algorithm implemented differently across architectures. This provides a solid footing for future work on algorithmic understanding.\n\nThe abstract was designed to communicate how this unification was achieved through hypothesis-driven research. We began with a testable hypothesis: that neural networks learn an approximate Chinese Remainder Theorem (aCRT) on modular addition (lines 95–96). That hypothesis drove the rest of our investigation, which delivered:\n- A unifying algorithmic structure (aCRT) across three architectures and all hyperparameter settings, including depth\n- Multi-level analysis (neurons, clusters, networks) that verified this structure\n- A theoretical prediction (an O(log n) feature bound) validated empirically\n- A testable universality hypothesis for group tasks, supported by our insights and theory derived from this process\nThis work demonstrates that interpretability can follow the scientific method: make hypotheses, derive consequences, and validate them. That process enables interpretability to yield generalizable, falsifiable insights. On this foundation, we can hopefully move towards more complex tasks. But before this unification, it was certainly not obvious.\n\nWe do *not* claim that our findings trivially extend to non-group tasks. But having a fitting interpretation of how networks solve modular addition is an important stepping stone in this search.\n\nFinally, we note that **reviewer EPiJ, acknowledged the potential of the work’s broader impact**:\n\"the work stands to be highly significant\" \nand \"this work will likely inform a lot of subsequent work and provide a useful perspective on the emergent behaviour of neural networks on algorithmic tasks.\"\n\nWe hope this clarifies the tone of the abstract. We are happy to revise its wording for clarity, but we stand by its core message as a faithful reflection of our contributions and approach.\n\n## A recent position paper gives context: our work is timely\n\nThe ICML 2025 position paper, “We Need an Algorithmic Understanding of Generative AI” [1], argues that the ML community must move beyond fragmented, bottom-up interpretability, ​​which often focuses on individual neurons, components or circuits without a guiding hypothesis. Instead, they propose the prioritization of top-down hypothesis/theory-driven research that explains how models implement full algorithms. The authors write in reference to prior approaches: \"current findings are still largely fragmented, and we lack a solid theoretical foundation for understanding how these various components come together to implement algorithms.\" Furthermore, they argue that such an approach would close the theory-interpretability gap, since both communities are operating rather disjointly. And this would lead to better understood, safer, and more efficient models.\n\nOur work is timely and directly answers this call by providing such a theoretical foundation in the well-studied setting of modular addition, which up to this point had fragmented findings. We offer a clear instantiation of several core elements that [1] proposes as future goals:\n\n| [1] proposes    | Our work |\n| -------- | ------- |\n| Identify algorithmic primitives  | approximate cosets    |\n| Compose primitives into algorithms | approximate CRT    |\n| Testable computational hypotheses grounded in theory    | 1. Theorem 4.7: predicts O(log(n)) features are required by deep networks on modular addition 2. Conjecture 4.9: containing many diverse tasks of interest, e.g. sorting lists   |\n\n# Questions\n\n> I probably need more explanations on how model goes from coset to the final output representation, it's not super clear from the writing. It's written in Remark 4.5 that \"the argmax of ... selects the correct answer\", but I am interested to clearly understand the mechanism to select the correct answer.\n\nThe network implements a divide and conquer algorithm that is distributed across the neurons. Figure 2 in the main paper demonstrates that each neuron that learned a coset or approximate coset only activates on half of the Cayley graph (first and fourth panels). This means that each neuron acts as a question “Which half of the Cayley graph is the answer in?”. Since different cosets and approximate cosets have different frequencies, they correspond to different graphs. Thus, Theorem 4.7 presents that only log(n) graphs are needed in order to divide and conquer in this way to get the correct answer while minimizing the loss sufficiently by Corollary 4.8, which gives that each additional frequency pushes the value on incorrect logits down exponentially. Therefore, the algorithm the network implements requires log(n) different graphs (different approximate cosets) and acquires the answer by asking log(n) questions: which half of the graph is the correct answer in.\n\nEssentially, we’ve so far explained the post-activations, as the ReLU ensures the negative values are 0 on all vertices in the Cayley graph that are farthest away (in graph distance) from the element at the center of the approximate coset. After this non-linearity, the problem has become linearly separable and achieves the most efficient solution possible due to the divide and conquer strategy. Each neuron can now have its activation projected to the half of the logits that are closest to the coset element the neuron best understands. \n\n> Q1. Do you have an explanation of why 1 layer model cannot learn this? Is it related to the famous paper from Antrophic about induction head and non-learnability of the copy task from 1 layer model?\n\nYes. The reason 1-layer models learn O(n) features instead of O(log(n)) features is because 1 layer models do not implement the divide and conquer algorithm explained in the previous answer. Instead they learn something much more in the flavour of a dynamic program: they learn every frequency and use the frequencies as a look up table in order to get the right answer. Yes, this directly relates to Anthropic's work on the induction head and builds on it by presenting a novel idea: deep transformers learn implementations of divide and conquer algorithms (we showed this in MLPs also). \n\n> Q2. Going from O(logn) to O(1) features is extremely important. Can we find some particular versions of NN with some form of induction heads to effectively use less features?\n\nCan networks be more efficient, i.e. can they do better than O(log(n)) features and learn O(1) features? Theorem 4.7 tells us that it’s possible for a network to learn this task with O(1) features, but the network will have very small margins (e.g. 1e-3) between the correct logit and second largest, but incorrect logit. This means the cross entropy loss will not be minimized, and thus the network will try to learn more features. This is why we performed the scaling experiment in Figure 3, which shows that for all moduli from 3 to 4999, neural networks of all architectures learned on average O(log(n)) features. \n\nIndeed, a satisfying aspect of our work shows how feature efficiency is exponentially increased from O(n) to O(log(n)) by depth. \n\n# Final thoughts\n\nWe thank you again for your review. We hope this addresses your concerns clearly. If so, we would greatly appreciate a reconsideration of your score.\n\n## References\n[1] Eberle et al., \"Position: We Need An Algorithmic Understanding of Generative AI\", ICML 2025."}, {"review_id": "YhkWsnc3KT", "reviewer": "Reviewer_hSU2", "summary": "The paper presents a universal algorithm -  through the notion of approximate cosets - to explain learned feature representations in models solving modular arithmetic.", "questions": "- Definition 4.2 doesn't clarify what $c, d$ are in the particular context. In particular, how does $d$ which is a function of $f, n$ relate to $c$? \n\n- The authors show that the simple neuron model activates on approximate cosets with ReLU activation. How does this extend to smooth activations like tanh and quadratic?\n\n- The assumption that training converges to simple neuron models, may not necessarily be true, as Ref. 6 in the paper (Fig 5) shows that the final trained model differs from the \"simple neuron model\". This is however not in conflict with the accuracy remaining unchanged in Fig. 6 of the paper, since the simple neuron model can give high accuracy but high MSE loss.\n\n- In Section 4.4 \"depth's effect on neural preactivations\", what exactly do the authors mean by fitting cos(a+b), cos a + cos b ? What are the fitting parameters in each case?", "limitations": "- While the paper addresses applications to discrete group operations, insights on how to generalize this notion for arbitrary tasks, including continuous groups, is missing. For example, what is the equivalent of the \"simple neuron model\" for a continuous group like $U(1)$? \n\n- Apart from that, I think the authors acknowledge other important limitations.\n\n[1] Doshi, D., Das, A., He, T., \\& Gromov, A. (2024). To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets. In Proceedings of the International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=UHjE5v5MB7\n\nOverall, I feel the paper does a good job of tackling an important question, but could do with some improvements.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The abstraction is useful since it unifies previous interpretations of learned features in modular arithmetic datasets, as well as provides a universal testable hypothesis.\n- The authors perform extensive sets of experiments, across various architectures and range of moduli, which shows that their claims are universally true.\n\n\nWeaknesses: \n- It would be useful if Section 4.1, Approximate cosets, can be rewritten to distinguish between precise cosets and approximate cosets, maybe with an example.\n- It wasn't immediately obvious why activation on approximate cosets can help the model solve modular arithmetic.\n- Missing reference to previous work : the order-1 sinusoids in layer 1 was analyzed in [1], Section 2.2.", "was_revised": true, "final_justification": "The paper is significant and proposes a \"top-down\" approach to unify previous interpretability research in modular arithmetic.", "rebuttal": "Thank you for your thoughtful review. Your questions and comments reflected a clear understanding of our work’s originality, and we’re grateful for the opportunity to clarify several key points. We also appreciate your recognition of the value of our extensive experimentation and our unified theory reconciling competing interpretations in the literature.\n\n# Weaknesses\n\nConfident that we made a significant contribution, we included Appendix A to rigorously and intuitively explain the mathematics underpinning our framework. We believe Conjecture 4.9 is of substantial importance, and our goal is to make it accessible to a broad range of researchers. Notably, Appendix A directly addresses all three weaknesses raised in your review.\n\n> W1. It would be useful if Section 4.1, Approximate cosets, can be rewritten to distinguish between precise cosets and approximate cosets, maybe with an example.\n\n**We already provide worked examples in the main paper and in the appendix.**\nIn the main paper:\n- Figure 2 provides 3 visual examples to distinguish neurons that learn precise cosets vs those that learn approximate cosets.\nIn the appendix:\n- Appendix A.2 \"Additional mathematical background\" walks the reader through group theory and worked examples.\n- Appendix A.2.1 \"Examples: cosets, Cayley graphs, step size d\" builds on the intuitions developed in A.2 to ensure readers have the background to understand the main text.\n\nThis should provide the relevant background to understand the visual examples in Figure 2. We will add a link to Appendix A.2.1 in the caption of Figure 2 to make this clearer to readers.\n\n**Elaboration to ensure your understanding.** Note in Figure 2, both coset and approximate coset neurons activate on half of the Cayley graph (panels 1, 4). \nThe coset neuron (panels 1,2,3) activates strongest on the coset a mod 66 = 0, and second strongest on cosets a mod 66 = 1 and a mod 66 = 5 and 0 otherwise. Panels (4-6) show a neuron that learned an approximate coset centered on a = 0. The approximate coset neuron doesn’t have discrete level sets like the coset neuron, so it activates on the half of the Cayley graph centered on the group element it best understands. Thus, the approximate coset definition generalizes cosets to group elements that are close on the Cayley graph. \nFigure 2: This shows approximate cosets have no equivalence classes, i.e. the neuron doesn’t activate on nice level sets like in the coset case (panel 2 shows 4 level sets). We show coset neurons don’t have invertable remappings, so all the points collapse onto 6 equivalence classes (giving 4 level sets in panel 3), whereas approximate coset neurons have invertible remappings (bijective; panel 6). Also note, the 6 equivalence classes of the coset neuron are shown as circles overlaid on the circle graph in panel 1, and everything is color coded to make it clear where the points in each coset (or approximate coset) come from.\n\n> W2. It wasn't immediately obvious why activation on approximate cosets can help the model solve modular arithmetic.\n\nPlease see Figure 10 in Appendix A.2.1. Now, suppose we have a 1 layer MLP with an embedding matrix. The preactivations of a neuron are the sum of two sinusoids, one for a and one for b. These two sinusoids always have the same frequency (as our R^2 fits support). Suppose that a neuron has the leftmost sinusoid for both a and b (on the x axis). Then the preactivations of this neuron receive cos(a) + cos(b). Since cos(0) = 1 and is the maximum, when a = b = 0, this neuron will output +2. Now, the weight connecting this neuron to the output logit at 0 will be large and positive. Thus this neuron used the coset information of a and b to activate strongly on the coset of the answer (0 + 0 mod 6 = 0). \n\n> W3. Missing reference to previous work : the order-1 sinusoids in layer 1 was analyzed in [1], Section 2.2.\n\nWe actually do cite [1] in Appendix A.1. That said, we will add a citation to [1] in the main text where we introduce the simple neuron model.\n\n# Questions\n\n> Q1. Definition 4.2 doesn't clarify what c, d are in the particular context. In particular, how does d which is a function of f, n relate to c?\n\nDid you mean definition 4.3 since c is only introduced after definition 4.2? c can be thought of as the \"center\" or starting point, and where a neuron activates most strongly. d is the step size, a function of f, n as you point out and traces out (approximate) cosets that are close. These are independent of each other. d relates to distances and c just relates to a starting point. We can make this clearer in the manuscript.\n\n\n> Q2. The authors show that the simple neuron model activates on approximate cosets with ReLU activation. How does this extend to smooth activations like tanh and quadratic?\n\nIf the neuron preactivations are still sinusoidal based periodic functions, our theory will transfer, e.g. the theoretical results of Gromov and Morwani et al., are covered by our theory. \n\nAn example of our theory working a bit “out of distribution” can be seen in Figure 28 in Appendix G.4. Notably, G.4, G.5, G.6 study fine-tuning neurons that learn sawtooth functions. When neurons learn these functions, they break the interpretations of all prior work. We do not think their existence is practically relevant. This is because they only show up with bad hyperparameters: they would never be found in practice due to corresponding to poor local minima (hyperparameter tuning for good settings avoids them). \n\n> Q3. The assumption that training converges to simple neuron models, may not necessarily be true, as Ref. 6 in the paper (Fig 5) shows that the final trained model differs from the \"simple neuron model\". This is however not in conflict with the accuracy remaining unchanged in Fig. 6 of the paper, since the simple neuron model can give high accuracy but high MSE loss.\n\nPlease see the answer to question Q2, particularly appendices G.4, G.5, G.6. The regions corresponding to low R^2 values are on the edge of where it’s possible to train a network. This is because this is the region that hyperparameters cause overfitting of the training set. Thus, neurons can learn sawtooth functions and become what we call “fine-tuning\" neurons because they are “overfitting the training set”. Thus, when you replace them with a single sinusoid, the network still gets the correct answer because the extra “fine-tuning” they were doing was not critical to the network selecting the correct answer. We put this section in the appendix due to our comments above: we do not believe these are practically relevant since proper training setups will never find them. They can only be found by exhaustively searching with hyperparameters that are bad, but just good enough to learn.\n\n> Q4. In Section 4.4 \"depth's effect on neural preactivations\", what exactly do the authors mean by fitting cos(a+b), cos a + cos b ? What are the fitting parameters in each case?\n\nWe are fitting the amplitudes (A_a, A_b) and phases (phi_a, phi_b) of two order-1 sinusoids, e.g. g(a,b) = A_a * cos((f * 2 * pi * a )/ n + phi_a) + A_b * cos((f * 2 * pi * b) / n + phi_b). We assume the frequency is the same for both sinusoids (which it is, or else our R^2 would be low and the prediction would not remain correct after replacing the neurons with the fit). We get the frequency f deterministically from the discrete fourier transform (DFT), i.e. it is not fit.\n\nSimilarly, for order-2 sinusoids we fit the amplitude and phase, and take the frequency f from the DFT: g(a,b) = Acos(f*2*pi(a+b)/ n + phi).\n\n\n\n## Limitations\n\n> While the paper addresses applications to discrete group operations, insights on how to generalize this notion for arbitrary tasks, including continuous groups, is missing. For example, what is the equivalent of the \"simple neuron model\" for a continuous group like U(1)?\n\n​​We believe the generalization for a continuous group is projections of group representations. See Appendix A.2 for where we introduce group representations and Appendix D where we discuss projections of representations as a more general theory. \n\n# Final thoughts\n\nWe once again thank you for your review. We hope in light of our clarifications you’ll consider increasing your score and we look forward to our continued discussion."}]

Author rebuttal:

Meta-review: **Scientific claims and findings**:
This paper proposes a unifying theoretical framework, the approximate Chinese Remainder Theorem (aCRT), to explain how neural networks solve modular addition. The authors claim that seemingly different solutions learned by various architectures are actually implementations of this single, universal abstract algorithm. A key finding is the theoretical and empirically verified result that deep networks require only $O(\log n)$ features for this task, a significant efficiency gain over shallow networks.

**Strengths**:
* **Unifying Theory (aCRT):** The primary strength is its success in unifying several disparate and conflicting prior interpretations of how neural networks perform modular addition under a single, coherent framework, the approximate Chinese Remainder Theorem (aCRT).
* **Novel Concepts:** It introduces novel concepts like "approximate cosets" to provide a more general explanation that holds across different training conditions and architectures (MLPs, Transformers).
* **Strong Empirical Support:** The theoretical claims, particularly the $O(\log n)$ feature scaling for deep networks, are backed by extensive experiments across over a million trained models, demonstrating the robustness of the findings. 

**Weaknesses**:
* **Clarity in Technical Sections:** Some reviewers found the initial presentation of the core theoretical concepts in Section 4 to be dense and lacking intuition, though the authors committed to revisions to improve clarity based on feedback.

**Reason for Accept**:
The paper's main contribution is conceptual and empirical: it resolves a key puzzle in the interpretability community by showing that different learned "circuits" can be manifestations of the same abstract algorithm. While the paper presents some theoretical analysis regarding margins, it does not culminate in a complete, end-to-end argument. In particular, a more relevant margin notion should be normalized margin, i.e., margin divided product of norm of parameters from each layer. (Otherwise large margin could be trivial to attain by just scaling up the last layer.) Nevertheless, this paper provides interesting and insightful new perspective on an important problem to the community, which will likely to inspire follow-up works.
Decision: Accept (poster)