Towards a Unified and Verified Understanding of Group-Operation Networks
Authors: Wilson Wu
Louis Jaburi
jacob drori
Jason Gross
Venue: ICLR 2025 Spotlight (2025)
Keywords: mechanistic interpretability
verification
proof
guarantees
interpretability
equivariance
group theory
representation theory
Primary area: interpretability and explainable AI
TL;DR: We investigate how neural networks compute group operations, finding an explanation that unifies those of previous works, then verify this explanation by translating it into a compact proof of model performance.

Abstract: A recent line of work in mechanistic interpretability has focused on reverse-engineering the computation performed by neural networks trained on the binary operation of finite groups. We investigate the internals of one-hidden-layer neural networks trained on this task, revealing previously unidentified structure and producing a more complete description of such models in a step towards unifying the explanations of previous works (Chughtai et al., 2023; Stander et al., 2024). Notably, these models approximate equivariance in each input argument. We verify that our explanation applies to a large fraction of networks trained on this task by translating it into a compact proof of model performance, a quantitative evaluation of the extent to which we faithfully and concisely explain model internals. In the main text, we focus on the symmetric group S5. For models trained on this group, our explanation yields a guarantee of model accuracy that runs 3x faster than brute force and gives a >=95% accuracy bound for 45% of the models we trained. We were unable to obtain nontrivial non-vacuous accuracy bounds using only explanations from previous works.

Published as a conference paper at ICLR 2025
TOWARDS A UNIFIED AND VERIFIED UNDERSTAND- ING OF GROUP-OPERATION NETWORKS
Wilson Wu!
Louis Jaburi Jacob Drori Jason Gross?
University of Colorado Boulder 2 Independent
wi wiwu23900coloradde edu [Touis you Jacobcd52, 9 com
ABSTRACT
A recent line of work in mechanistic interpretability has focused on reverse- engincering the computation performed by neural networks trained the binary operation finite groups. We investigate the internals of one-hidden-layer neu- ral networks trained on this task, revealing previously unidentified structure and producing a more complete description of such models in step towards unifying the explanations of previous works (Chughtai et al.) 2023 Stander al., 2024). Notably, these models approximate equivariance in each input argument. We ver- ify that our explanation applies to large fraction of networks trained on this task by translating it into a compact proof of model performance, a quantitative eval- uation of the extent which we faithfully and concisely explain model internals. In the main text, we focus on the symmetric group Sg. For models trained on this group, our explanation yields a guarantee of model accuracy that runs 3x faster than brute force and gives 295% accuracy bound for 45% of the models we trained. We were unable to obtain nontrivial non-vacuous accuracy bounds using only explanations from previous works.
INTRODUCTION
Modern neural network models, despite their widespread deployment and success, remain largely inscrutable in their inner workings, limiting their use in safety-critical settings. The emerging field of mechanistic interpretability seeks to address this issue by reverse engineering the behavior of trained neural networks. One major criticism of this field is the lack of rigorous evaluations of interpretability results; indeed, many works rely on human intuition to determine the quality of an interpretation (Miller 2019 Casper 2023; Räuker et al., 2023). This insufficiency of evaluations has proved detrimental to interpretability research: recent work finds many commonly used model interpretations to be imprecise or incomplete (Miller et al., 2024 Friedman et al.) (024).
A simplified research program has focused on toy algorithmic settings, which are made more tractable by the presence of complete mathematical descriptions of the task and dataset (Nanda) et al. 2023a by Zhong et al.) (024)). However, even in these settings, the lack of rigorous evaluations for interpretations is consequential, leading different researchers to come up with divergent expla- nations for the same empirical phenomena: recently, Chughtai al. (2023) claimed that models trained on finite groups implement group composition via representations algorithm, while subse- quent work (Stander al. 2024) studies the same model and task and instead argues that the model implements a coset concentration algorithm.
In this work, take the challenge of reconciling their interpretations. We investigate the same setting and find internal model structure that was overlooked by both previous works: the irreducible representations noticed by CCughtai et al. (023) act by permutation on discrete set of vectors learned by the model. Based on our observations, we propose a model explanation that unifies those found by both previous works. In particular find that the model approximates a function that preserves the group symmetry in each of its input arguments, i.e. a bi-equivariant function.
Thhse authors contributed equally. See Author Contributions

---

Published as a conference paper at ICLR 2025
Following Gross al. 2024), then evaluate interpretation and that of previous work by converting them into compact proofs of lower bounds on model accuracy.
Figure 1: Examples of p-sets extracted directly from the weights of models trained on the symmetric group S1 (left, a tetrahedron) and the alternating group As (right, an icosahedron). Both lie in RR The vectors of the p-sets are depicted as points-the the connecting edges are merely for illustration. See Section 3 for the definition of p-sets and Section 4 for how they are in by models to compute the group operation. See Figure TQ and Figure [T] for compact proof bound results for S4 and As, respectively. The standard irrep of St. the focus of the main text, is cour-dimensional and hence more difficult to visualize.
Our philosophy is that any rigorous mechanistic knowledge model's inner workings should yield guarantee the model's performance. In more detail, given a mechanistic explanation, the real model will somewhat differ from due to noise or imperfections in our analysis. To rigorously validate the explanation, validation is done using a program that takes in a model as input and guarantees that the model obeys focus on properties of the form "the model will give the correct answer for at least some X %% of the input space", i.e. lower bounds on accuracy. The simplest such guarantee is to brute force try every possible input to the model, which does not require mechanistic explanation and is a perfectly tight bound. However, we believe that nontrivial mechanistic understanding of model should yield more efficient programs, e.g. by exploiting symmetries in the proposed explanation.
thus need bound the effect of this deviation
model behavior. This
some property;
If the program guarantees a property such as model accuracy, then we can uniformly turn an execu- tion trace of the program into an formal proof of this property. This an (automatically generated) proof in the standard mathematical sense: proof that mathematical object (the model parameters) satisfies the desired property. Proofs corresponding to more efficient execution traces are shorter, i.e. more compact The efficiency of the program, and closeness of the accuracy bound to the true accuracy, can be taken as metrics of quality of our explanation: More complete explanations should yield either tighter performance bounds measure of explanation completeness. We find that our interpretation is indeed an improvement over previous ones because it yields more compact proofs of tighter bounds.
more compact proof, providing quantitative
Our contributions are as follows:
We provide a mechanistic explanation of models trained on the group composition task (Section
We verify this explanation by translating it into guarantees model accuracy (Section For a substantial fraction of models, these guarantees are near the true accuracy, providing strong positive evidence for explanation in these cases (Section 5.2
We clarify previous mechanistic interpretability results this same task and argue that they do not fully explain model behavior (Section 6 We show that our more complete interpretation is step towards unifying the findings of previous works (Section
'The length of the proof is linear in the running time of the program.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Groups, mechanistic interpretability, and grokking Our work can be seen as a direct follow-up to Chughtai al. (2023) and Stander al. (2024), which both perform mechanistic interpretability on one-hidden-layer neural networks trained on the binary operation of finite groups. These papers in turn build on work that studies models trained on modular arithmetic (Nanda et al. 2023a Zhong
et al. 2024), i.e. the binary operation of the cyclic group. Models trained on group composition exhibit the grokking phenomenon (Power et al. 2022). in which a model trained on an algorithmic task generalizes to the test set many epochs after attaining perfect accuracy on the training set. Morwani et al. (224)) study the group composition task from the viewpoint of inductive biases, showing that, for one-hidden-layer models with quadratic activations, the max-margin solution must match the observations of Chughtai et al. (2023)
Evaluation of explanations Several techniques evaluate interpretations of models have been suggested, such as causal interventions (Wang et al. 2023) and causal scrubbing (Chan et al. 2022). We discuss merits and limitations of causal interventions in our setting, which were first explored in
Stander et al. (2024). More recently, Gross et al. (2024) use mechanistic interpretability to obtain compact formal proofs of model properties for the max-of-four task. Yip et al.] (0224 study the modular arithmetic setting, finding that the ReLU nonlinearities can be thought of as performing numerical integration, and use this insight compute bounds on model error in linear time.
Equivariance We find that neural networks trained on group composition learn to be equivariant in both input arguments, i.c. bi-equivariance, despite this condition not being enforced the ar- chitecture. Learned equivariance has been noticed and measured in other settings (Lenc & Vedaldi 2019 Olah et al., 2020; |rrveer et al.] (023).. This is distinct from the area of equivariant networks, which equivariance enforced by model architecture (Bronstein et al. 2021).
PRELIMINARIES
3.1
MATHEMATICAL BACKGROUND: GROUPS, ACTIONS, REPRESENTATIONS, P-SETS
This paper uses ideas from finite group theory and representation theory. We provide rapid and informal introduction to the most important definitions and refer the reader Section 3 and Appen- dices D, E. F of Stander et al. (2024) and/or relevant textbooks (Dummit & Foote 2004 Fulton & Harris 1991) for more details.
Groups and permutations A group G' is set with associative binary operation and an identity element e such that every element has an inverse. We write S, for the group of permutations on clements. Maps of G on sets of size n. Recall that each permutation a E Sa can be represented as a permutation matrix in Raxn that applies the permutation to the basis vectors of R Thus, any permutation representation of G is linear representation, i.c. a mapping from G to the group of invertible n x n matrices (n,,, RR). The group operation translates to matrix multiplication.
S,, are called permutation representations and are equivalent to actions
Irreps Linear representations that cannot be decomposed into direct sum of representations of strictly smaller dimension are called irreducible representations or irreps for short. A representation is an irrep if and only if there is no nontrivial subspace that is closed under p(g) for all G. Every finite group G has a finite set of irreps (up to isomorphism), which we denote by Irrep(G') Any linear representation can be decomposed uniquely into irreps.
p-sets By the preceding discussion, given a permutation representation p: G - S., we can con- sider its decomposition into irreps. Let p E Irrep(G)) be one irrep present in this decomposition, acting on some subspace W C I". Since, by definition, P acts on standard basis vectors e en by permutation, the constituent irrep p acts on the projection of the basis vectors onto W by the same
"By "maps" we mean group homomorphisms.
In this paper, consider irreps over TR. In particular, all irreps of Sn are real. For discussion of preliminary results for groups with complex irreps, see Appendix K.2
3

---

Published as a conference paper at ICLR 2025
permutation. We refer to any subset of W that p acts by permutation p-ser; in particular, projections of the basis vectors onto W fit this criterion
Example: Ss Our primary example throughout the main text the group of permutations Sa- The identity map Sş- + Ss is permutation representation. As linear representation, it is not irreducible, as fixes the all ones vector E RRS Projecting out this vector results what is called the standard four-dimensional irrep of S5; call it p. This irrep acts by permutation on the projections of the standard basis vectors, so these five vectors in F+ form a p-set. In this example, the p-set consists of five evenly spaced vectors on the surface of the sphere in R-
3.2 TASK DESCRIPTION AND MODEL ARCHITECTURE
Our task and architecture are identical to that of previous works (Chughtai al. 2023 Stander et al. 2024). Fix a finite group G. We train a model on the supervised task : G x + G; i.c. given G.
y G the task is to predict the product
We train a one-hidden-layer two-input neural network on this task. The input to the model two elements z. G. embedded as vectors (z). (v) E R", which we refer to as the left and right embeddings, respectively. These are multiplied by the left and right linearities Wi, W, E Rmxm, summed, applied with elementwise ReLU nonlinearity, and finally multiplied by the unembedding matrix U E R | and summed with the bias wt.
We can simplify the model's description by noting that the left embedding E, and left linearity Wi only occur product E,E, and likewise for the right embedding and linearity. Also, the product between the unembedding U and the embedding vectors can be decomposed into a sum over the hidden dimensionality [m]] Hence, letting w/ (x), wo((y)) w denote the ith entries of Wx), E, (v), (z) respectively, the forward pass is
m
fo x, = w2)2) + Σ (x) ReLU(((()))) + w.
(I)
paramaterized by 0 (ws, (w), w w ) Each vector wi, w 1 for i [m] can be thought of as a function G + I,, and we refer to them as the left, right, and unembedding neurons, respec- tively. The refer to i [m]] as the neuron index.
GROUP COMPOSITION BY p-SETS
4.1
THE P-SET CIRCUIT
Our central finding is that trained models implement circuits of the form
f..(2 x,
Σ bJ pl x ReLU(a [b= B')] b.
(2)
where B C Rd is p-set and Rd. This fo.a(2 | x. depends only on x call such functions bi-equivariant Furthermore, we show in Lemma G.5 that for certain irreps, fo.B(2 x, y) is guaranteed to be maximized at the correct logit 2 = x*y y We find that each trained model implements several such circuits and that the logits approximately linear combinations of terms of the form Equation
We now explain how the model weights implement the circuit in Eq 2 Recall from Eq that fo can be written as the sum of the contributions of each neuron plus the bias. As seen in Section 6.2 each neuron is in the span of single irrep p E GGr) We find that, furthermore, these neurons
"The term "p-set" is our own. All other definitions and notations introduced in this section are standard. "To see the equivariance, note that if fo.s is of this form, then, for g, h G. have fp.s(2 (2 g*az, y+h) fp.5(g * 2 * x. y).
'Notice that implication goes both ways if p is faithful, which is the for all irreps of Sy with dimension 1.
if and only if -1 y which implies p(x-1 2 1 I. This
4

---

Published as a conference paper at ICLR 2025
actions of finite p-sets B C R projected onto dimension. Moreover, the unembedding weights of each neuron are related to the left and right embedding weights, so that, for some b, b' from p-set B and some projection vector a, up scaling,
(z) = p(2)b',
rof = b p(z)a,
(v) = -a p(y))'
(3)
Additionally, there is one neuron of this form for each of the [B]² pairs (b, 6') and a is constant across all such pairs. See Observation B.1 for a full enumeration of our findings.
Based on these observations, we partition neurons into independent p-ser circuits; each circuit is associated with a p E Irrep(G) of dimension d, a finite p-ser B C IR" that p acts on transitively by permutation, and a constant vector E IRd. Call this circuit fp.8: then,
f. x.
Σ 6
b p(2)b' ReLUib p(z)a - a p(y)6b)
(4)
Using the p-set structure of B, we can change variables via b p(z) = p(x and arrive at the aforementioned circuit Eq (with b, b' exchanged for b, b'). Since fe is a sum of such bi-equivariant circuits, it also bi-equivariant.
f(x=1)p b)b and d' = p(y)
Further, terms of the summation in Eq 2 are zero whenever b = 6', which is precisely when b" p(e)b' (b, b') is largest. Heuristically, these properties explain why fp.5(z x, max- imized when 2 = 2 * y. For certain
Irrep(G). we can rigorously show that the function is indeed maximized at z = z * for any value of a; see Lemma[G.3] for details. Eq also be as a separating hyperplane in the ambient space inhabited by irreps p. See Sec B.2 for details.
4.2 THE SIGN CIRCUIT
For irreps of dimension strictly greater than one, we observe that the model learns circuits closely approximating what we describe in Section 4.1 However, for the sign irrep the model is able to use one-dimensionality to avoid the expense of the double summation in Eq
Explicitly, up to scaling, the sign circuit is
fagu(2 (z xx, xx,)) p(z) - p(z) ReLU[p(2)) - p - p(z) ReLU -())) | p(x * x ),
where p is the sign irrep. The circuit comprises two neurons -p(=) ReLU(p(x) p(y)] and -p()) ReLU(p(y) p(z)], and uses the unembedding bias to strip out an extraneous p(z) term See Appendix [B. for further discussion.
EXPLANATIONS AS COMPACT PROOFS OF MODEL PERFORMANCE
Following Gross et al. 2024), we evaluate the completeness of mechanistic explanation by trans- lating it into a compact proof of model performance. Intuitively, given the model weights and an interpretation of the model, we aim to leverage the interpretation to efficiently compute guarantee on the model's 's global accuracy. More precisely, we construct a verifier program 3 V that takes as input the model parameters the group G, and an encoding of the interpretation into a string n, and returns a real number. We require that V always provides valid lower bounds on accuracy; that is, V@0, G. m) S (0) regardless of the given interpretation æ. [0 Our measure of r's faithfulness the tightness of the output guarantee V(0, G, IT), i.e. how close it is true to the true accuracy ag (O))
Two simple examples of verifiers are:
TTo be more precise, we find that there are often multiple neurons per (b, b'). However, they are scaled such that the sum of neuron contributions corresponding to each pair is uniform.
'The sign irrep maps permutations #1 depending whether they decompose into odd the only nontrivial one-dimensional irrep of S.. In general, any real-valued one- dimensional irrep of any group must take values 11, and the same circuit as deseribed here works.
ber transpositions.
Formally, a Turing machine. We elide any implementation details related to encoding finite group theoretic from finite floating-point precision, etc.
objects strings,
10Thi soundness requirement prevents from (for example) simply providing the true accuracy ag (0) to the verifier. If V takes I' 's veracity for granted, it is no longer sound-r v could falsely claim the accuracy to be higher than the truth, causing V(0, G. x) > (0).
5

---

Published as a conference paper at ICLR 2025
1. The vacuous verifier Vvac (O, 0. This is valid verifier because 0 is lower bound on any model's ' accuracy.
2. The brute force verifier Virute G. = (o) which takes the model's weights and runs its forward pass on every input to compute the global accuracy.
While the brute force approach attains the optimal bound by recovering the true accuracy, it is computationally expensive. (Indeed, it is intractable in any real-world setting, where the input space too large to enumerate.) Notice also that neither example is provided an interpretation æ, without any information about 0, we cannot expect the verifier to do better than these trivial examples. We aim to construct verifiers that, when π is a meaningful interpretation, (1) give non-vacuous guarantees model accuracy and (2) are compact, i.c more time-efficient than brute force.
A good understanding of the model's internals should allow us to compress its description and therefore make reasonable estimates on its error. A more complete explanation should yield tighter bound, while also reducing the computation cost. Therefore if we think of bounding the accuracy as trade-off between accuracy and computational cost, a good explanation of the model should push the Pareto frontier outward. See Gross et al. (2024) for more thorough discussion.
5.1 CONSTRUCTING COMPACT PROOFS
The verifier lower-bounds the model's accuracy by trying to prove for each x. y E G that the model's output is maximized at x * i.e. that so(x y v) >> max:/zzy fa(z xr v). More precisely, the verifier strategy is:
(I) Given model parameters 0, use the interpretation to construct an idealized model
(2) For v, y E G, lower bound the margin fā(x y xr v) max Sō(z x.y)).
(3) For x; y E G. upper bound the maximum logit distance
( x, v) - fo x,,)1 + fo( ky x. v. fo( * (x,)). =
max
(4) The accuracy lower bound the proportion of inputs æ, y E G such that the margin lower bound exceeds the distance upper bound. For such input pairs, the margin by which the idealized model's logit value on the correct answer exceeds the logit value of any incorrect answer is larger than the error between the original and idcalized model, so the original model's logit output must be maximized at the correct answer as well. See Figure
Recall our model architecture Eq I The brute-force verifier runs a forward pass with time complex- ity Om]]G]) over |G|2 input pairs, and so takes time (m total. [T] Our verifiers need to be asymptotically faster than this in order to be performing meaningful compression. Naïvely, though, both steps (2) and (3) take time Omm]G|)) no better than brute force. However, we can reduce the time complexity of each to ((²) for (2) by exploiting the internal structure of the idealized model, and for (3) by using Lemma G.
Compact proofs via coset concentration Intuitively, coset concentration (ttander al.) 2024) gives a way to perform nontrivial compression-if the interpretation says that a neuron is constant on the cosets of a specific group, then we need only to check one element per coset, instead of a full iteration over G. However, the shortcomings listed in Section 6. are obstacle to formalizing this intuition into a compact proof. The verifier Veoset we construct pessimizes over the degrees of freedom not explained by the coset concentration explanation, resulting in accuracy bounds that are vacuous (Section 5.2). See Appendix D for details of Vroset 's construction.
Compact proofs via p-sets We are able to turn our p-set circuit interpretation into proof of model accuracy that gives non-vacuous results on a majority of trained models; see Appendix for details and Section 5.2 for empirical results.
11 For simplicity of presentation, assume all matrix multiplication is performed with the naive algorithm; that is, the time complexity of multiplying two matrices of size and l O(mnk).

---

Published as a conference paper at ICLR 2025
6
Virrep
The interpretation string π labels each neuron with its corresponding irrep p and its p-set. The verifier Vireep is then able to this interpreta- tion string to construct an idealized version of the input model that implements p-set circuits (Eq 4 exactly. By bi-equivariance, this ideal- ized model's 's accuracy can then be checked with a single forward pass. Finally, Virrep bounds the distance between the original and the ideal- ized models using Lemma G.
Veoset
4
Margin
2
0
Logit distance
Figure 2: Margin lower bound vs. logit distance upper bound over z, y E S5 for Virrep and Veaset on single example model. The accuracy lower bound is precisely the number of points for which the margin lower bound larger than the logit upper bound (shaded region); in this example, the bound from Vitrep is 100% while that from Veoset is 0%. The margin lower bound of Virrep is con- stant due to bi-equivariance.
5.2 EMPIRICAL.
RESULTS FOR COMPACT PROOFS
We train 100 one-hidden-laye -laer neural network models from random initialization on the group S5- We then compute lower bounds racy obtained by brute force, the cosets expla- nation, and the p-sets explation, which refer to as Voruter Vooset: Vireep respectively. We eval- uate these lower bounds on both their runtime D2 (compactness) and by the tightness of the bound.
accu-
See Appendix for full experiment details.
As expected, Virute obtains the best accuracy bounds (indeed, they are exact), but has a slower run- time than Virrep; see Figure 3 On the other hand, across all experiments, Vouset failed to yield non- vacuous accuracy bounds; while the margin lower bounds of Veoset 's idcalized models are nonzero, they are swamped by the upper bound on logit distance to the original model; see Figure
Looking again at Figure 3 we see that the accuracy bound due to p-sets is bimodal: the verifier Vireep obtains a bound of near 100% for roughly half the models, and vacuous bound of 0% for another half. Investigating the models for which Virrep does not obtain a good bound, we are able to discover for many of them aspects in which they deviate from our p-sets explanation:
(a-bad) The projection vector (Eq 4 fails to be constant across terms of the double sum over b, b' e B, so the change of variables showing bi-cquivariance (Eq 2 is invalid [3 We find that such models have poorer cross-entropy loss and larger weight norm than those with constant a, suggesting they have converged to suboptimal local minimum (Figure in Appendix [.6).
(p-bad) The double sum over B (Eq 4 misses some b, b' pairs. Again, we are unable to prove bi-equivariance when this happens. We speculate that in this case the model is approximating the discrete summation by a numerical integral à la Yip et al. (2024).
Although these cases are failures our p-sets interpretation to explain the model, their presence can be seen as success for compact proofs measure of interpretation completeness. For models we do not genuinely understand, we are unable to achieve non-vacuous guarantees on accuracy.
REVISITING PREVIOUS EXPLANATIONS: COSETS AND IRREPS
In this section, we recall the coset algorithm described in Stander et al. (024), and the notion of irrep sparsity observed in Chughtai et al. (2023). We find that although these works correctly identify properties of individual neuron weights, they lack precise picture of how these neurons combine to compute the group operation. We conclude the section by clarifying the logical relationship between these observations and our present work.
We use verifier's time elapsed a proxy for FLOPs, which a non-asymptotic measure runtime. If depends on (b, b'), then there is remaining dependence on (p(x)b, -(y))))) inside the ReL.U after changing variahles.
7

---

Published as a conference paper at ICLR 2025
1.0
V Virute: exact accuracy but slow runtime
0.8
Vinep mean
Accuracy bound
excluding unexplained
brute
V Virrep expl
0.6
Virrep unexpl
0.4
Vineep faster but inexact accuracy bound
brute mean
0.2
Virepp all mean
unexplained by p-sets
V 'irrep expl mean
0.0
0.8
1.0
1.2
1.4
1.6
1.8
2.0
2.2
2.4
Time (s)
Figure 3: Accuracy bound vs. computation time for Virrep and Veute on 100 models trained on S3- Points in green (Vinep unexpl) models for which we find by inspection that p-sets explanation does not hold, i.c. either ( (a-bbad)ee or (orange), 0% for Veoset (not shown). 50.4% for Vinep (union of blue and green), and 91.7% for occur (blue, 55% of total). Mean time elapsed is 2.20s for Vbrute and 0. 75s for V Virrep. The asymptotic time complexity of Virute
Mean accuracy bound is 100% for Vorute
Vireep when only including models for which neither (artherraabda)nno((obbba
Om GI$) while that of Virrep is (m)))))
6.1 COSET CONCENTRATION
Recall the model architecture Eq T In Stander al. (2024), they make the following observation: For each neuron i, the left embeddings are approximately constant on the right cosets of a certain subgroup K₁ of G, while the right embeddings are approximately constant on the left cosets of conjugate subgroup K2
Kg.. That is, they are coset concentrated; see Definition 6.2 They then observe that there is a subset X K1 h h' K2. On inputs x, y E G. the left and right embeddings of the neuron i sum to near zero precisely when x E X. Meanwhile, the unembedding takes smaller values on elements of X; thus, when æ * y É X, the model's confidence in G' X is increased In the example of S5, these X typically take the form of Xas := (o E Ss (a) = b}. See Stander et al. (2024) Section 5) and Appendix A for more details.
This explanation leaves several things unclear:
Even given coset concentration, there are many choices of left/right embeddings that sum to zero whenever y E X. The choice matters: for example, if there many input pairs where x & but the sum of embeddings is near zero, then the model cannot be expected perform well. How do we know whether the model has made good choice?
The unembedding is similarly underdetermined: there are many degrees of freedom in choosing weights satisfying the sole constraint of being smaller on X than on G' \ X.
The bias term mentioned, despite being present in the models under study.
In Proposition A.2
we provide a more precise version of this explanation, assuming the model weights satisfy stronger constraints. However, the actual models we investigate not match these assumptions closely, which is reflected by our failure to convert this explanation into non-vacuous bounds in Appendix
6.2 IRREP SPARSITY
Chughtai al. 2023 notice that each neuron in the trained model is in the linear span of matrix elements of some irrep E Trrep(G); we refer to this condition as irrep sparsity (see Definition 6.1 for formal statement). Based on this observation, they propose that the model:
"4ehee subgroups might vary from
8

---

Published as a conference paper at ICLR 2025
1. Embeds the input pair z, y E G as d d irrep matrices p(x) and p(y).
2. Uses ReLU nonlinearities to compute the matrix multiplication p(z) = p(x v).
3. Uses ReLU nonlinearities and p(x+y) from the previous step to compute tr(p(x*y*==1)) which is maximized at x
+ x y.
However, because they leave the ReLU computations as a black box, they are unable to fully ex- plain the model's implemented algorithm. We were able deduce more complete description by carefully investigating which linear combinations of p each neuron uses.
6.3 RELATING IRREP SPARSITY, COSET CONCENTRATION, AND p-SETS
This paper and prior works observe multiple properties of neurons viewed as functions G + R: Definition 6.1 (Chughtai et al. 2023). A function f: G + R is irrep sparse it is linear combi- nation of the matrix entries of an irrep of G. That is, there exists E Irrep(G)) of dimension d and a matrix E Raxd such that (g) = tr(p(g)AA).
Definition 6.2 (Stander et al. 2024
function f: G + R is coset concentrated there exists a nontrivial subgroup H I G such that f is constant on the cosets (either left or right) of G.
Definition 6.3. A function f: > IR is projected p-set for E Irrep(G) of dimension if there exist a, b Rd such that (g) p(g)b and is in a p-set with nontrivial stabilizer.
In this section, we clarify the logical relationships between these three properties.
Projected p-sets are irrep-sparse This fact, while immediate from definitions, resolves a mystery from Chughtai et al. 2023 Figure 7): why is the standard irrep Pstd of Ss learned significantly more frequently than the sign-standard irrep Pagustd, when both have the same dimensionality dim Pstd = dim Pagnstd 4? The answer is that the smallest Pet has size 5, while the smallest PSe has size 10 (Appendix H). Thus minimum complete Peta-Set circuit needs 52 25 neurons, while a minimum complete Pegustd -Sct circuit needs 102 = 100. The order of frequencies with which p E (G) is learned (Chughtai et al. 2023 Figure 7) is the same the ordering of Trrep(G) by minimum p-set size (Table not by dimensionality.
Projected p-sets are coset concentrated Our p-set interpretation immediately explains coset con- centration: the map + p(g) constant on precisely the cosets of the stabilizer Sta := {g E G p(g)b = b}. Further, since the action of G is transitive, Stabg for any other element b' conjugate of Stabo((): as consequence left and right preactivations of each neuron concentrate on cosets of conjugate subgroups.
Coset concentration fails to explain irrep sparsity Stander al. (2024) partially explain irrep sparsity via coset concentration. We paraphrase their key lemma here:
Lemma 6.4 (Stander et al., 2024). Let H be subgroup of G. The Fourier transform of a sunc- tion constant on the cosets of H is nonzero only at the irreducible components of the permutation representation corresponding to the action of G on G/H.
This lemma fails fully explain irrep sparsity, since the permutation representation of G on G/H is never itself an irrep: it always contains the trivial irrep, possibly among others. Thus, it does explain why the models' neurons are supported purely on single irreps and not, say, on linear combinations of each irrep with the trivial irrep.
Notice also that, since there are many more subgroups than irreps (Stander al. 2024 Appendix G.3), most subgroups H < G have corresponding actions of G on G/H that decompose into more than two irreps; otherwise, since the decomposition is unique, there would be at most [Irrep(G) 2 total possibilities. Explicit examples of subgroups whose indicators are supported on more than two irreps are given in Appendix H
Projected p-sets are equivalent to the conjunction of irrep sparsity and coset concentration Note that neither irrep sparsity nor coset concentration alone is equivalent to the condition of be- ing projected p-set. For an example of an irrep-sparse function that is not p-set, consider the

---

Published as a conference paper at ICLR 2025
function X(g) = tr(p(g)). If p is faithful and has nontrivial stabilizer (for example, the standard 4-dimensional irrep of St), then X cannot be a projected p-set, because is maximized uniquely at the identity (Chughtai et al.) 2023 Theorem D.7) and thus is not conenccomee...atee. To see that coset concentration does not imply being a projected p-set, recall from above that there are coset- concentrated functions that are not irrep-sparse, but all projected p-sets irrep sparse.
On the other hand, if f: G + is both irrep-sparse and coset-concentrated, then it must be a projected p-set; for proof see Lemma G.6 Hence, if we consider by itself single embedding of a single neuron on fe. then our Observation B.2 2 is logically equívalent to the combination of observations from previous works. However, our perspective gives us more insight into the relation- ship between left/right embedding and unembedding neurons (Observation relationship between different neurons (Ohservation B.2
as well as the
DISCUSSION
Limitations of the p-sets interpretation The p-set explanation we provide has a rather limited scope: we only claim to understand roughly half of the models we examine, all of which are trained on Sy. On the other hand, all of the models we examine satisfy both irrep sparsity and coset concen- tration. It was by trying and failing to obtain non-vacuous compact guarantees on model accuracy that discovered that our understanding of some models is still incomplete. Hence, compact proof' are a quantitative means of detecting gaps in proposed model explanations.
Causal interventions In an attempt to verify the validity of the coset concentration interpretation, Stander et al. (2024) perform a series of causal interventions. While the results do not yield evidence that their interpretation incorrect, they also do not provide strong evidence that it is correct. Indeed, we perform the same interventions on a model for which we know the cosets explanation cannot hold, and find results the same direction; see Appendix [F] Thus, in this case, causal interventions might yield strong negative evidence against an explanation, but provide weaker positive evidence. Furthermore, causal interventions lack a notion of an explanation's simplicity independent from its faithfulness-they do not provide a quantitative measure of how much an explanation compresses the model.
Compact proofs For the models we do understand, we obtain tight accuracy bounds in signifi- cantly less time than brute force, providing strong positive evidence for our understanding. Hence, view the compact proof approach complementary bound is strong positive evidence for an explanation, whereas a poor or vacuous bound does not give strong negative evidence. Indeed, the coset interpretation does make nontrivial obser- vations and yield partial explanations of model performance, but this is not reflected in the vacuous bounds obtain. It may be the case that some of the pessimizations Veoset were unnecessarily strong; the translation from an informal explanation into rigorous bound itself informal, by necessity.
the causal intervention one: A tight
in our construction of
CONCLUSION
Multiple previous works (Chughtai et al.) 2023 Stander et al. (0244) have examined the group com- position setting and claimed mechanistic understanding of model internals. However, as we have demonstrated here, these works left much internal structure unrevealed. Our own interpretation incorporates this structure, resulting in a more complete explanation that unifies previous attempts.
We verify our explanation with compact proofs of model performance, and obtain strong positive evidence that it holds for a large fraction of the models we investigate. For models where we fail to obtain bounds, we find that many indeed do not fit our proposed interpretation. Compact proofs thus provide rigorous and quantitative positive evidence for an interpretation' 's completeness. We see this work as a preliminary step towards a more rigorous science of interpretability.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
See Appendix I for experiment details. Code for reproducing our experiments can be found attps://anonymouss 4open. wwwtncc////groeeeeeeeeeesee E024
ACKNOWLEDGEMENTS
We thank Jesse Hoogland, Daniel Filan, Ronak Mehta, Mathijs Henquet, and Alice Rigg for helpful discussions and feedback on drafts of this paper. WW was supported by grant from the Lerm Future Fund. JD was supported by a grant from Open Philanthropy. This research was conducted in part during the ML Alignment & Theory Scholars Program. Computing resources for some experiments were provided by Timacus. We thank the anonymous reviewers for providing many suggestions that improved the clarity of this paper.
AUTHOR CONTRIBUTIONS
Wilson Wu Led the engineering and ran the bulk of all experiments. Designed, implemented, and ran compact proofs. Proved majority the theoretical results. Contributed to writing of paper and rebuttals.
Louis Jaburi Initiated the project and suggested experiments to run. Contributed to writing of paper and rebuttals.
Jacob Drori Made contributions leading to discovery of the p-set circuit: pushed study the predicted {b;] is
structure of A, B,, C; (defined in Observation B.I), predicted Observation [ p-set; noticed b, and e; vary independently, leading to the double-sum in Equation
Jason Gross Advised the project, including developing proof strategies and providing feedback on experimental results and presentation.
REFERENCES
Amir Beck. First-Order Methods in Optimization. Society for Industrial and Applied Mathematics, 1974997. URL/attps://epubs. siam. org/ doi//bs//0.. 1137/1 97111777999111979999999
Philadelphia, PA, 2017. doi: 10.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge University Press, 2004,
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. CoRR, abs/2104.13478, 2021. URL https:
13478
Stephen Casper. FIS III: Broad critiques of interpretability reserach. AI Alignment Fo-
rum, 2023. URL https://wee al gnment forum www.o//gg//aooo. http://.........ee/eee/etttrr//
gwG9ugw255gaf jYN4
Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas.
Causal scrubbing: method for rigorously testing interpretabiliity hypotheses. 2022. 7RN/
URL org.e
www http::/ww.aLiimmentttBt.. wowgn http:///www...tr./pp////////////
causal causal-scrubble scrubbing- a for
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: reverse engineering how networks learn group operations. In Proceedings of the 40th International Conference on Machine Learning. ICML'23, JMLR.org, 2023.
David S. Dummit and Richard M. Foote. Abstract algebra. Wiley, New York, 3rd ed edition, 2004.
Dan Friedman, Andrew Kyle Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioun. In- terpretability illusions in the generalization of simplified models. In Forty-first International Con- ference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
ew net www.ooowrud..........o. dJW UMW6YP

---

Published as a conference paper at ICLR 2025
William Fulton and Joe Harris. Representation theory. A first course., volume 129 of Graduate Texts in Mathematics. Springer-Verlag, Verlag, 1991.
GAP Groups, Algorithms, and Programming, Version 4.13.1. The GAP Group, 2024. URL
https:/ wttp:/wwwwppppp...ppp system. org
Jason Gross, Rajashree Agrawal, Thomas Kwa, Euan Ong, Chun Hei Yip, Alex Gibson, Soufiane Noubir, and Lawrence Chan. Compact proofs of model performance via mechanistic interpretabil- ity. In ICML. 2024 Workshop on Mechanistic Interpretability, 2024. doi: 10.4 11779. hhttps:://axiiii
abs 11779
Nate Gruver, Mare Anton Finzi, Micah Goldblum, and Andrew Gordon Wilson. The Lie deriva- tive for measuring learned equivariance. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
https://epenreviiee net/ www.:/eeee/ooommeeeee Torum?1d==117aavy/
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. Int. J. Comput. Vision, 127(5):456-77, May 2019. ISSN 0920-5691. doi: 10.1007/s11263-011 121---1-------.. URL
5 611263-018-109--
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL URL htttps://opeereeiiieee
wwt//forooie d= CqY?
Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit faithfulness metrics are not robust, 2024. URL https:://////. wttpaaaiwi...aa//aaaaaaaaaaaeeee
Tim Miller.
Explanation in artificial intelligence: Insights from the social sciences. Arti- ficial Intelligence, 267:1-38, 2019. ISSN 0004-3702. doi: hutps://oo.org//00000//jj.j
4018.07.007, URL https: wttp://wwwwwceiirreetttttc http:///c/cceece//aaatllee/e///a
SDC04370218305982
Depen Morwani, Benjamin L. Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham M. Kakade. Feature emergence via margin maximization: case studies algebraic tasks. The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL URL/https:/ wwt.//wenneee.eee////pp/.. /ttpr/////ttt///eeeeeeeeeerrrrr net www.nneen//ooiinooooooo 11W
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In International Conference on Learning Represen- tutions (ICLR), 2023a.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world mod- els of self-supervised sequence models. Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Na- joung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Newral Networks for NL.P, pp. 16-30, Singapore, Decem- blackhoxnlp-11.2 ber 2023b. Association for Computational Linguistics. doi: 10. 1 URL
org. bwwckboxn.cpccpppcooooooc.e 1.2
Chris Olah, Nick Cammarata, Chelsea Voss, Ludwig Schubert, and Gabriel Goh. Naturally oc- curring equivariance in neural networks. Distill, 2020. doi: 10.2391 2/iiil//isslll URL
httts::/diisill: http://isiii..pp/222////e/// wttp:///00//.rrrrtt////uuu//////
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach Vito. Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, Pirfo-Performnce Deep Learning Library, pp. 8026-8037. Curran Associates Inc., Red Hook, NY, USA, 2019.
12

---

Published as a conference paper at ICLR 2025
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gener- alization beyond overfitting on small algorithmic datasets. CoRR, abs/2201.02177,, 2022. URL
htttps://axxa wwt./rgg////ooooooootoo/ot..aere abs 2201 02177
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent Al: A survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2023, Raleigh, NC, USA, February 8- URL https:
10, 2023, pp. 464 483. IEEE, 2023. doi: 10.1 10090AATML555555222...000 //doi. org/1000 Ttt.09/////aM/ Sa TMM.55555 2023. 00039
Dashiell Stander, Qinan Yu, Honglu Fan, and Stella Biderman. Grokking group multiplication with cosets. In Proceedings of the s1 International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 46441-46467. PMLR, 21-27 Jul 2024.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. In- terpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:: httpss///ppeeevieveeee net / forum?id- NpsVSN604u1.
Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Gen- eralization and optimization of neural nets v.s. their induced kernel.
Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL File/ 1e//7745 (1w.4444422204433300000000000000
files/paper/2019/
Chun Hei Yip. Rajashree Agrawal, and Jason Gross. ReLU MLPs can compute numerical integra- tion: Mechanistic interpretation of non-linear activation. In ICMI. 2024 Workshop on Mecha- nistic Interpretabilityy 2024.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: two stories in mechanistic explanation of neural networks. In Proceedings of the 37th International Con- ference on Neural Information Processing Systems, NeurlPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
A COSET CONCENTRATION IN DETAIL
The main observation of Stander et al. 2024) is that neurons are concentrated the cosets of subgroups. More precisely,
Observation A.1 (tanner et al. 2024). For each E [m], there exists a subgroup H; S G and element E G such that the left and right neurons are approximately constant on the right cosets of H, and the left cosets of the conjugate subgroup K; = 9, H,,, respectively:
xy E H; = wi(x) = wi)
''y E K; wx((x) z (.)
(5)
This forces the summed embeddings wi (x) + w)) to be approximately constant on the double cosets H; H,G,, Further, is observed that
w (x) + 1 (v) = ry E H,zyg,K, -,
If we include several additional assumptions, this observation is sufficient for fe to attain perfect ac- curacy. Note was not explicitly investigated in Stander et al.) (024) to what extent these additional conditions met.
Proposition A.2. Suppose the model parameters 0 are such that Observation A. holds, the unem- beddings satisfy
max "
wx(
min 2gH1.."
wx(2) max w((()
2EH.g."
Throughhut the appendix, we denote the group multiplication of v, G by my instead of by x
13

---

Published as a conference paper at ICLR 2025
and the bias wa is zero. Further, defining
s(H,9)
min x.yEC H
ReLU ReLUUof(z)) + (w),,
(8)
sElm (H..)))(H.g)
suppose every singleton {z) for 2 G be written as an intersection of sets from the family
{G H.g, i
[m]] sHH, g:) > 0}
Then, ac(s) =
Proof. 2 E with 2 + xy. Then,
f(ay ((ay12,)) - f(z x, v) Σ (xy)) (x))
(2)) (c) + w (v)]
Σ
8 {H, & Hig
(w((xy) w (z))
(H,g)
(HH))))(Ha))
0,
To see that the inequality is strict, choose such that s(His gi) > 0 with E H4g," and ry $ H
B P-SET CIRCUITS IN DETAIL
B.1 LIST OF NOVEL OBSERVATIONS
Chughtai et al. (2023 observe that neurons are irre--sparse. That is,
Observation B.1 (Chughtai et al. 2023). For each E [m], there exists real-valued irrep p.: G - GL(d, (d, IR) of degree d, as well as A,, B,, C. Rddiddi such that
wf) tr(p(z)AA),
(x) = tr(p()))
(x) tr(p(z)C,).
However, this is not sufficient fully describe how the model computes the group operation; in particular the ReLU nonlinearities are left as black boxes.
We observe further structure:
Observation B.2 (Ours). Let (pi, A,, Bi, C.) be in Observation B.1 Suppose that all irreps of G over C are real-valued; in particular, this is the case for S, T6
1. R r,,,,BAA some r; > O.
2. Each of A,, B., C₁ are rank one, with |A = Bille. Hence, we may write
A s, B B s.N. C. as ro a.). where 8s E IR and a,, b,, C, d Rd are unit vectors.
3. a d,
Further, fix irrep p of G and consider I, [m] | p}. the subset of neurons supported on Then,
4. a; is approximately constant on I That is, there exists a unit vector ap E RRd such that
Vi Ip a, ss d, ap.
"Fquivalentyy the Frohenius-Schur indicator of each Trrep(G)) is positive. We briefly consider complex and quaternionic irreps in Appendix K.2
14

---

Published as a conference paper at ICLR 2025
5. (bihee, (-cihi,,.
6. Each (bi can be partitioned into (Bpatgg such that, for each neuron the correspond- ing vectors b; and -c, must belong the same partition. Further, p acis on each partition by permulation; that is, p induces a left G-sel structure on every Bpg We say that such Bp.q p-set.
7. If dim >> 1, then, for each partition Bp.q. there exists Cp.q. E R such that, for every pair b, b' B.q
Σ s, b. -b' Cp.4- E
8. If dim p 1, then, since we assume p real-valued, it must be either trivial or the sign irrep. We observe that the former never occurs. In the latter case, B, = {=1}, and there exist C c- E
such that
C4 ~
Σ
=1,b 13 ~
28
1,b;
Σ
2 stbrr:lbb -1,b; =-1
EELp
iELp
e. ~ Σ
= - -
Σ
2 rbb - =
iEIp
9. The bias wa satisfies
wr(2) ~ (c_ c+) sgn(2).
B.2 SEPERATING HYPERPLANE INTERPRETATION
Another way to express Eq
2 is as
fp.((z r, = plo *
ReLU (b - b')]b'T b
Σ
(10)
where (: is the Frobenius inner product of matrices That is, each p-set circuit learns (a, B)- parameterized separating hyperplane { Z(a, B)) between p(e) I and (p(=) = * e). Since the are all unitary and thus of equal Frobenius norm, such a hyperplane always exists (e.g. = I), though we do not show in general that it can be expressed in the form of Eq
B.3 BI-EQUIVARIANCE
The observations Section B.I force the network to be bi-equivariant:
Proposition B.3. If So satisfies Observations B. and B.2 exactly, then for all x, 2, 92 E G,
fo(z y1)) = fo 2g2"" x, xz
We say that So is bi-equivariant. In particudar,
fo(z z. = - ee,e).
so such fo depends only on x 2y Observe that r 2y e iff 2 = xy..
Proof. In the notation of Observation B.2 for each p Irrep(G) and partition Bp. let fo
Eier, 1(b; E Bp H. Then, if so satisifes Observation B.2 exactly,
((A, B) = A B Ecs A Biy.
15

---

Published as a conference paper at ICLR 2025
If dim p 1, for every p... there exists E Rdim p, cc R (all depending p) such that, by re-indexing neurons,
&
(z x, v) = c Σ
b p(2)b, ReLU p())a - ap(y)b,)
i,j=1
Σ
bi p( 2y 1), ReLUla (b, b,)l
1,j=1
plz p(z-122 22 c
ReLU[a (b. b,]]6,b
If dim p 1, then p must be the sign irrep. In this case,
fg x. v) + ws(2)
+ + c+p(=) +c( -e))
c+p(z)
ReL - p(v) -c
-c) ReLU (()) + + cc=((z)
c+(((=))p(z) + (())l-((2)) - -(y) p(z))
c+(p(=)(1 +
-c))) ((b)))))(rr)))))))))))))))))))>) p(=))
(c+ +c-)p(zoy)
+c+ c- c-)p(x - 1)
B.4 STEPS TO DISCOVER THE P-SET CIRCUIT
In the main text, we presented the p-set circuit, and validated by using our new understanding to efficiently lower-boune bound model performance. However, we did not describe the process by which discovered the circuit the first place. Here, we lead the reader through the steps of this process. This section can be viewed
annotated walkthrough of the list of observations in Section B.
0. Train network G-multiplication. Recall that, given inputs x, y G, the trained network assigns the following logit value to output 2:
fe x, wr(2) + Σ w (x) (c) (y)l-
1. Confirm neurons are irrep-sparse. Given d-dimensional irrep of G, construct tensor T, of shape (IG), d, d), interpreted as d x representation matrix for each element of G. We may also think Tp d' many vectors of size [G]. We compute the span of these vectors Sp span(T,) S RIGI, By the Schur orthogonality relations, the subspaces (Sppee))))()) are mutually orthogonal.
Now, if we fix a neuron i, then w// wi, wt, are each functions G -> R. i.e. vectors of dimensionality GG. these vectors are all approximately contained in single Sp (say each with > 90% variance explained), we say the neuron is supported on p. We then use least-squares linear regression to find A,, B; C, E Rdxd satisfying:
wx(z) R= tr(p(z)
wx ~ (.)) wx(x) = tr(p(x)C.).
Note: the number of features in each of the three least-squares problems is d2 and the number of data points is |G| d,, We can now write the network as
fa(z x. R Σ tr(p(=)C,) ReLUItr(p(s)), + B.))
and task hereafter is to look for structure in A,, B,, C;.
We use GAP 2024 for this. See arc/ atc//rrups...oo./. groups. py:gett i irreps in the provided repository.
16

---

Published as a conference paper at ICLR 2025
2. Observe C R r,B,A, for some r; > 0.
We guess this relation by analogy with the modular addition circuit in Yip et al. (2024). We then verify it by defining Frobenius- normalized matrices
A, A./ B. = B./llBlllll C, C./ICI
and noting that the unexplained variance |Cł; /1I.;1//II;:1 is small.
3. Observe that for real irreps, A,, B,, C, are approximately rank 1, with ||A.]| Rs |B:
For each matrix, check that the variance explained by its top principal component is
= 1. We also check A.11/1 Bell
= So we have
A. ~ s,a,b, B B,,,,
C. r (.)...)
where 8; is the top singular value of A, or Bj, (a,, b.) are the top left and right singular vectors of A,, and (cs, d,) are the top left and right singular values B,.
4. Observe a, d,. We check the unexplained variance ||a: d / /l.// small.
Now we restrict to neurons E pp i.e those supported on a given irrep p. Let us take stock, and use observations thus far to write out the contribution to the network due to these neurons:
(Ø 2.
Σ b, p(z)c, ReLU p(z)a, + a ply)c.).
EE
5. Observe a, a, constant. We simply check (a,, aj} for all pairs i.j. Now the only remaining degrees of freedom to understand are the b, and C..
6. Cluster [b.} and {c.). Using k-means clustering,
19 we find that {b.} and {c:) each consist of tight clusters. Let B, and C, be the of means of these respective clusters. the simplest (and most illustrative) case, {bb, c.)leE,, = B, x Cp- In other words, and "vary independently". Moreover, for any pair (b, c) B, x Cp. we observe
Σ
ri87 Cp
{iEI, (b..c.)=((be))
where Cp is a constant independent of i. In general, though, find {(b,, ci)lies, = Uy x Cp. for some partitions pBp.gg and } of Bp and C,- In other words, b and vary independently within each partition. Moreover, we observe that for any (b, c) E Bpn x Cps
Σ
r
Cp.q
(EE if (b.,ci)ss(b,e))
This split into partitions is technical detail (the partitions are easy to find in practice) and the reader may wish to ignore it.
7. Observe Bp.g -Cp.g- We check that for each b E B, there exists C with (b, c) xe -1 (and vice versa with B and C swapped).
8. Observe that Bp.q is approximately a p-set. We check that for all x E G and e B, there exists D' B such that p(x))b zs b' (that is, (p(z))b, b') Rs 1).
Putting these observations together, we arrive our final expression for the contribution to the network due to single Bpg (whose indices we drop to simply call B):
fp.((z x,,) = Σ b p(z))b ReLU p(r)a -aa p(v))b
This is precisely the Equation that we stated when introducing the p-set circuit in Section
For exploratory work, we use standard k-means. For the interpretation string of the compact proof. we also try a modification of i-means that takes into account the symmetries due to the corresponding irrep p. We find that this modified k--means algorithm does not yield substantially different results from the original.
17

---

Published as a conference paper at ICLR 2025
B.5 SIGN IRREP AND MODULAR ARITHMETIC
Let us briefly extend attention groups with complex-valued irreps and consider cyclic groups 7.p2Z. i.e. arithmetic modulo p. All irreps of eyclic groups are one-dimensional, looking like p(z) = 22rkk//p = c2sk2 p) + sin(2nka /p). The modular arithmetic setting is studied by Yip al. (2024), where it is found that models use approximation trick involving the sum-of-cosines formula and integration over a single variable:
cos(2 + 2Φ) CeLU + +))== cos(y + o)]dó
cos
y
COs - 2Φ) cos
x +
- 6) dō
2
2
2
2 c08
[
z)
Cos
2
3
Here, analogously to the sign circuit of our setting, the sum of the group elements' embeddings expressed the embedding of their sum in order compute the desired inequality with only single sum/integral, instead of double sum. (In this case, the extraneous |cos((z: v)/2) is not removed; indeed, this additional the model use multiple irreps, even though each one is faithful (Zhong et al. (024).)
the "Achilles' heel" of this strategy, and necessitates that
B.6 CONSTANT PROJECTION VECTORS
0.065
110
0.060
108
0.055
Weight norm
106
Loss
0.050
104
0.045
102
0.040
0.035
100
0.030
98
10 6
10
10
100
10 6
10
10
variance
variance
vs. model loss and weight norm, where a; the projection vector for neuron i, and expectation is taken across neurons within the 4d standard irrep of Sg. Each point is one model out of 100 trained on St. Notice that constant a; across neurons correlated with better model performance and lower weight norm.
Figure 4: Plots normalized variance E, [||aa 1o,-E,,,118/mm
Why does the network learn to set the a vector constant across neurons? A heuristic explanation is that such constant vector is one way to enforce bi-equivariance, which then leads the margin attained be uniform across inputs x, y E G. MMowani et al. (2024) show that this uniform margin must necessarily be the case at maximum margin solution; further, models trained cross-entropy loss the zero weight decay limit indeed attain the maximum margin (Wei et al. 2019).
However, we do see that some fraction of trained models do not have constant we do not have a full understanding of these models, and thus are unable to non- voc-ouuuy bound accuracy. We notice that these models tend to have inferior performance and higher weight norm, suggesting that they have converged to poor local minimum by chance; see Figure Further, even in these cases, find that the a; all lie in two-dimensional subspace.
18

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EVIDENCE FOR P-SET CIRCUITS
In this section we provide additional evidence that models implement p-set circuits (Eq. 2) For each trained model, constructed idealized version of the model that implements p-set circuits exactly; each irrep p. corresponding p-set B, and constant vector are found automatically using the steps described in Section B.4
Figure [ plots the distance between original model parameters and parameters of idealized mod- els. Figure 7arlusstttttsse the effect of replacing each parameter type (w), Wyy wn) in the original model with is idcalized version. Figure is the same but instead aggregated by neurons correspond- ing each irrep. Figure deppics the bi-equivariance of idealized models, original trained models, and randomly initialized models. We measure bi-equivariance by
Var f0 2, Euy=: 2, v)
equiv (0) E
(11)
If fe is exactly bi-equivariant, then equiv(0) O.
100
10-1
10
10
8
8
10
embedding left
embedding right
unembedding
unembed bias
Figure 5: Normalized distance between original and idealized model parameters ||w ~ / || || (i.e. 1 R22) for each of left embedding wi, right embedding w,, unembedding w., and unembed bias wy of 100 models trained on Ss. Green boxes include all models while blue boxes exclude models for which we find that the p-set explanation does not hold (i.e. cither((a-bad) badddooor (p-bad))
ideal
original
random
10 10
10-2
100
102
Bi-oquivariance
Figure 6: Bi-equivariance of idealized models, original trained models, and randomly initialized models for 100 models on S5. See Eq. I1 for the definition of our bi-equivariance metric equiv(0). Lower means more bi-equivariant and value of zero means exactly bi-equivariant. Theoretically. the idealized model is exactly bi-equivariant when parameters are considered over R; however, some non-bi-equivarian is introduced by floating point imprecision.
D ACCURACY BOUNDS VIA COSET CONCENTRATION
We construct a verifier Veuset that takes as input 0 and an interpretation string
((Hi, 9t))
19

---

Published as a conference paper at ICLR 2025
and returns lower bound ag (0) in time (m/]G]))
For each z, E G, the verifier Vooset computes lower bound on the margin
M (z) S min (zy æ, v) fo(z' x,
2. zy=
by doing the following:
1. Check that the (H:) are each subgroups of and construct the conjugate subgroups K; = gi H. in time (H,
2. Construct idealized parameters θ by averaging over the cosets given in x. Explicitly.
u |1,1-
Σ (x')
N'EHar
क (y) |K -1 Σ (v')
y y'EyK;
|G H H1 "x -- (x
2 z H,g.
t (2)
min w (z), min 1 (2')
E Hig.
w 0.
This takes time m))
3. Compute sHH, 9i) for each i according to Eq
This takes time 0 G())).
4.
For each
G. compute
(2) min Σ E'EG (H. s(H,g)1{z Hg
Σ
(w w (z'))
ie(m) (He.))) H.
This takes time (m (
5. Lemma G.2 gives upper bound the for norm between the output logits of original and idealized models for each E G in time (m))))²) Sum the difference in logit values of the original and idealized models on the correct answer, which again can be computed in time O G)²). This gives
L(x,y) 2 [lfe( (x,))
fa(. 2. 1l fa(øy x. folay x.,).
(12)
The final accuracy bound
Pr Uniff((
[M(2)
max L(x,
zyyEG
zy=s
The total time complexity is dominated by the last two steps. Soundness, i.c. VT Venset (0, (0, G)) ogG (0), follows from Proposition A.2
The bias term The description of the coset circuit in Stander et al. (2024) makes mention of the unembedding bias term, although it is present in the models they train for their experiments. In practice, we find that the bias term large. The maximum minus minimum value of the bias would then be added directly to L(x, y) for every I, E G if we were to run Veoset as written. Thus, in addition, we train models without an explicit bias term for our Veuset experiments. We find that such models are qualitatively similar to models with an explicit bias; the missing bias term is simply added uniformly to each unembedding weight wt Perhaps because of this, we are still unable to obtain nonvacuous bounds from Veoset even for these bias-less models.
E ACCURACY BOUNDS VIA P-SETS
We construct a verifier Virrep that takes as input and an interpretation string π and returns a lower bound on ar. (0) in time O(m G).). The interpretation . comprises ((p;, qe, a; b, ());),) where z. is the index of the corresponding p-set Bpg (b. P: = P. qi = =q). The verifier then does
20

---

Published as a conference paper at ICLR 2025
(i)))) 1. Check that each Bp.g q is indeed a p-set. This takes time (C ( Since
m 2 Ep.l and Epelreep(G) dim(p)2 = |G). this upper-bounded by O(G)
2. Within neurons corresponding to each (p. q). check that a is constant. This again takes time no more than (m).
3. For each (p. q) where p the sign irrep, check that the coefficients c; across all neurons corresponding to (p, q) is constant. This takes time (m)).
4. For neurons corresponding to the sign irrep. there is only one p-set B = {=1}. Check that the constraint in Observation B.2 8) holds; that is, that the neurons corresponding to
(+1,+1) have coefficients summing to the same value as those corresponding to (-1, -1), and likewise for (+1,-1) and (-1,+1).
5. Construct the idcalized parameters consisting
wi (ax) b7. p(x));;
w (v) a, p(a)cs
i (z) -.,b, p(=)b;
(x) = (c_ c4) sgn(z).
This takes time o dim (im(p:) (p:) O()
6. Compute the idealized margin
M min fā fzy x, a.vEG
max fō(2 xy).
z'æey
By Proposition B.3 this can be done with a single forward pass of fa. in time O(G
7. Use Lemma G.2 to compute L as in Eq 12 in time (mm(G(3). The final accuracy bound is
Pr [M > L(x,y)). UnifUUiff(C)
The total time complexity is dominated by the last step. Soundness, i.e. VT Virrep (0, G, T) s og (0), is obvious from construction and Proposition B.3
F INSUFFICIENCY OF CAUSAL INTERVENTIONS
Stander et al. (2024) perform a series of causal interventions on a model trained on S5, and find results consistent with their description of the cosets algorithm. However, we expect that these interventions would have the same result for models implementing different algorithms. To verify this, we replicate their outcomes with model trained on the cyclic group = Z 5355 such a model cannot be using the coset algorithm, as G as non-trivial subgroups. Models trained cyclic groups were studied in Yip et al. (2024) and found to be using a distinct algorithm; see also discussion in Section B.5
In detail, the interventions that Stander et al. (024) perform on S₁ are:
1. Embedding exchange: Swapping the model's left and right embeddings destroys model performance. Since S₁ is non-commutative, we expect this to be the case regardless of what algorithm the model implementing. Even with / 2/532, which is commutative, we get this result, since (x) + (,),(y) + W,E, + Wx).
2. Switch permutation sign Multiplying either the left or right embeddings individually by
-1 destroys model performance, while multiplying both preserves model performance. We find this to be the case with 2 2/532 as well.
3. Absolute value non-linearity Replacing the ReLU nonlinearity with an absolute value improves model performance. Again, this is the case with Z/53ž as well. We explain this by decomposing ReLU(x) = (ær + |=1)//2 By inspecting the summation in Eq 2 we see that the z/2 component sums zero, only the x21/2 term contributes. Thus, replacing the ReLU with an absolute value is approximately equivalent to doubling the activations, which reduces loss assuming the model already has near-perfect accuracy.
21

---

Published as a conference paper at ICLR 2025
4. Distribution change Perturbing model activations by (1, reduces performance greater extent than perturbing with (1,, Again, we observe this with 2 2/53Z.
See Table for results.
Table I: Causal interventions aggregated over 128 runs Z/53Z (ours) juxtaposed with the same interventions aggregated over 128 runs on Sy (Stander et al., (024). We train our models with fewer iterations than Stander et al. (2024), resulting in higher base loss. However, the directional effect of each intervention is the same, even though the coset concentration explanation does not hold for Z 2/53Z.
Z/53Z (ours)
S3 Stander et al.
2024
Intervention
Mean accuracy
Mean loss
Mean accuracy
Mean loss
Base model
99.55%
0.0711
99.99%
1.97e-6
Embedding swap
03.85%
4.15
1%
4.76
Switch left and right sign
99.69%
0.0663
100%
1.97e-6
Switch left sign
00.00%
17.2
0%
22.39
Switch right sign
00.00%
17.2
0%
22.36
Absolute value nonlinearity
99.88%
0.0045
100%
3.69e-13
Perturb (0 1)
76.90%
0.829
97.8%
0.0017
Perturb (0.11)
99.51 %
0.0752
99.99%
2.96e-6
Perturb (11
49.80%
1.79
88%
0.029
Perturb (1,
83.17%
0.780
98%
0.0021
G ADDITIONAL PROOFS
Lemma G.1. Let
0 = (U, E,, E,, wib = (ww wi, w wT),
0 (Ü, E,, E,,, wbb ((wi), uof, ut whs
22

---

Published as a conference paper at ICLR 2025
Then, for any x. G. max]So(z x. v) [or(= x,,)
zEG
max #EG
(x) ReLU(w)(() + w/(r + wx(z) -x) (eLUU(r((z) - -0:
206 (z))
(w))) ReLU)u(() wi
max
xEG
t=1
+ max
(=) (ReLU + w/ (w) - (eLU/o((x)) + wi)
zEG
==1
+ max (b(2) - (bb())
sEG
-I - -) ReLU([((z) + (y)]loo
+ ūT (ReLU[E(()) E,)) ReLU[Ei(z) E,(0)) (E)))) I.
+ 106 2is lo
|I(U - Ū) 1|,,00 (x) Ev)ll
+ | |2,00|
ReLU (x) (v)) (eLU(E((x)) (w)l) ll
+ wwb w. 1oc
II(U Ú) 2 (1 [| ( 1.2 + (E 1.2)
+ U 12,00 (1) Es Ei + |, E, (12
+ um Us lxc
Lemma G.2. Let and 1 be as in Lemma G.1 Further, suppose we have a margin lower bound function for 0
M(=) < min
folay (x,)) - fe(2' x.
k.y.='EG xy=z4='
Then,
aG(0) 2
Pr u.ig-Unif(G)
M(ay) > I1(U [B)
((()) E,
(13)
+ | |2,00 (ReLU[E(()) + Ex)) (cLU + B.(i))
+ wwb w.s loc
Pr (n)
M(2) II Ú) 2| (x)) |1 1,2 + (1)(2)))
(14)
+ | 2,, (1) E Eill.. + E, E, 1122)
+ |m - ubo |loc
The bound in Equation 13 can be computed in time O(m]G(2), while that in Equation 14 can be computed in time (m GT).
Proof. This follows immediately from Lemma G.I
For the remainder of this section, let denote the cross-entropy loss:
((x
log Σ, zz
23

---

Published as a conference paper at ICLR 2025
Lemma G.3. The cross- cross-entropy loss is 2--Lsscchitz
Proof. Cross-entropy loss is differentiable with
(e(((2,, i)); 6 dij Ex c
Ex 1.0k
Hence,
(2) EEnges exx)2
322 Ekeph 2202
2.
Exeenn
Lemma G.4. For 2, E R" and i E [n], the cross-entropy loss {(., satisfies
((y.1) f(x, + vi(ar) (y - x) + y1
1 11 y112.
f(x, + [((=))]2|1> 2l12 +
Proof. It suffices to show
(mx 1 2 I, whence {(.,)) is convex and 1 1/2-smooth; the
desired inequality is then well-known consequence (Beck 2017).
Write p: = Σ c
Then Boyd & Vandenberghe 2004),
vil(o,i) log Σ a
diag(p) - pp"
80, for any vector R
V (((())= = (ddag(pp) pp Jp Varp(v) 2 0,
confirming that 22((x, i) > 0. Furthermore, applying the Gershgorin circle theorem to the Hessian,
1 max 2p, - P3) S 2' jEln
Amaxx (2(((r, i)) < max
Pj
p)
P,Pı
jE(n]
k
so V2E(x, i) 1 I.
Lemma G.5. Let p: G GL(n, R) be a permutation representation G that decomposes into two subspaces V @ W such that acts trivially W. (For example, the standard (n -)ddimensianal irrep of Sn is of this form.) Let V and W be orthonormal bases of V, W E R", respectively, and WWT be the corresponding orthogonal projections, that P Q I. Denote b; V e, where (e,) are the standard basis of R.. Then, for any a E V,
let P = VV and
((2) = (pl(E))) Σ ReLU a (b. -b,1]bbb
2,5
maximized
24

---

Published as a conference paper at ICLR 2025
Proof.
(0(x) = Plv(2), Σ
ReI.Ula T (b: 1..-bo bb,}}b,b]
(l)( Σ
ReLUIa (b; - b,)]V e
(Vllv(z)VT, Σ ReLUUaa (b. bylle,e,
Pp(z) P. Σ
ReLU/a (b, b,)lee,,
p(z), Σ
ReLUla (b; - by)le,e?
QT(=))T
ReLUIa (b: bylle,e,
p(z), Σ
ReLUIa [(; -b,- bb)e,e
QQQT
Σ ReLUIa ((a (b,
b))le,e,)
where the last step uses that plw is trivial. The first term of the last line is the negation of sum over off-diagonal entries of the permutation matrix p(z), and thus maximized z e. The second term does not depend on z.
An irrep V of G admits such a p with W being the trivial representation, for the corresponding subgroup H C G the double coset has two elements H G/H. This applies in G = S5 and = S4.
where
Lemma G.6. Suppose f: G is coset concentrated and irrep sparse; that is, f is constant on the cosets of some H < G and there exists p Irrep(G) mapping p: - GL(d, C) such that I is a linear combination of the entries of p. Then, there must exist an embedding t:: G/H + C such that the action of p on ( H) isomorphic to the action of G on G/H. H. Further, there exist a E (CAy" and E Cd such that f(g) = (a, b).
The converse also holds and is immediate.
Proof. Let (gi,..., O be representatives for the cosets G/H H with e, and let 5: G - Sk S GL( k C) V be the permutation representation corresponding to the action of G on G/ H.H. (That p is induced by the trivial representation on H.) Let (eg, 1t denote the basis vectors of V. Define a E V* by (a, e,.) = f(g) for each i E [k]. Hence f(g:) (a, p(g))e,)) Since f is coset
concentrated, this same relation holds for all G. Now, let V = = V, be the decomposition
of into irreps, with corresponding projections my: have
V, and irreps Pj: G + GL(V)). We
f(g) = (a, plg)en)
Eialy, pgg))) = Σ (a p 1=1
1=1
By the irrep sparsity assumption, at most term of this sum is nonzero, and that corresponding term must have Pj- = p.
f(g) (aly, p(g)TT
Then P acts on e 1e isomorphically to the action of on G/H H and (a (alv,, e, is the desired covector and vector pair from the theorem statement.
25

---

Published as a conference paper at ICLR 2025
H PERMUTATION REPRESENTATIONS AND p-SETS OF S
In this section we enumerate all irreps of S5 and their corresponding minimum p-sets.
Irrep
Minimum p-set size
Stabilizer
trivial (1d-0)
S₃
sign (1d-1)
2
As
standard (4d-0)
5
S4
sign-standard (4d-1)
10
A4
5d-0
6
F3
5d-1
12
D12
6d-0
20
Z/6Z or Sy
Table 2: Irreps E Irrep(S;) by the size the minimum p-set, and corresponding stabilizers. We name each irrep by its dimension and an arbitrary disambiguating integer; e.g. 5d-0 is a five- dimensional irrep. A projected p-set (Definition6.. 6.3) is constant on the cosets of its stabilizer. Notice that the ordering of Irrep(S;) by minimum p-set size matches the ordering by frequencies with which irreps are learned (Chughtai et al. 2023 Figure 7).
Stabilizer
G-set size
Irreps present
Ss
1d-0
As
2
1d-0,1d-1
S4
5
1d-0,4d-0
F'
6
1d-0,5d-0
A4
10
1d-0,1d-1,44-0,440
D12
10
1d-0,4d-0,5d-1
D10
12
1d-0,1d-1,5d-0,,53 5d-1
D&
15
1d-0,44d-0,5d-0,,50 6d-0
Z/6Z
20
S
20
S
20
5d-1
Z/5Z
24
Ve
30
1ef6010e11(66000011110000.660 5d-1
VÍ
30
16-0,AGd-0,58-0555-1-6666 4d
Z/4Z
30
6d-0
Table 3: All transitive permutation representations of S with size more than 30 along with decomposition into irreps. By the orbit-stabilizer theorem, transitive permutation representations correspond directly left actions of Sş cosets of its subgroups; the subgroup acted upon is then the stabilizer of the action. Upper indices disambiguate subgroups of S3 that are isomorphic but not conjugate. Fs is the Frobenius group of order 20 and Va is the Klein four-group.
I EXPERIMENT DETAILS
For the main text, we train 100 one-hidden-layer models with hidden dimensionality m = 128 on the group S5. The training set comprises iid samples from the test set and has size 40% of the test set. Note that we use the same training set for each of the 100 training runs. Each model was trained over 25000 epochs. Learning rate was set to le-2. We use the Adam optimizer (Kingma & Ba 2015) with weight decay 2e-4 and (B1,32) = (0.9, 0.98) 20 All models were trained on one Nvidia A6000 GPU. Compact
is all pairs of two inputs from S,. with |S, 14400 points total. The
Note that previous work uses AdamW [Loshchilov Hutter 2019 instead, with weight decay However, we found that models trained with Adam grok the group composition task in order of magnitude fewer epochs. We did not notice significant differences in the end result post-grokking hetween models trained with Adam A
26

---

Published as a conference paper at ICLR 2025
proof verifiers were run an Intel Core i5- 135PP CPU. Neural networks implemented in PyTorch (Paszke et al., 2019). Their group-theoretic properties were analyzed with GAP (GAP) 2024).
In Section K. models are trained with the same hyperparameters as described above, except 1)
For As, the hidden dimensionality is bedding bias is omitted. Recall that, in our explanation, the role of the unembedding is to deal with the sign irrep, which is not present for alternating groups. The larger hidden dimensionality and smaller weight decay were used an attempt to reduce occurrences of (p-bad) though we did not observe these changes to have significant effect
256, the weight decay is 10 and the unem-
For S4, the hidden dimensionality m 64 and the training set is 80% of the test set in order to account for the smaller total number of data points.
As input to the verifier Vesset recall that the interpretation string looks like ((H,, ai) where the left embedding at neuron E [m] should be constant on the right cosets of H₁ and the right embedding neuron should be constant on the left cosets of g; H,g," When constructing m, we set each H; to be the largest subgroup of G such that
E Var (w) HAG)
< 0.01,
Var(w)) (wf)
and K2 to the largest subgroup such that
E Var(w) G/Ki) Var(w;)
< 0.01,
and check for the existence of gi such that K; = gi H.g. The quotient on the LHS is bounded in [0, 1] by the law of total variance.
As input to the verifier Virees the interpretation string is found using automated version of the steps discussed in Section B.4 The automated process labels each neuron with an p: E Irrep(G) and corresponding p:-set B. The irrep p; is chosen to have the largest R²2 against wf. none, if the RD if no irrep exceeds 95%. The p-set is recovered using singular value decomposition of the coefficient matrices A, B. C₁ defined in Section B.4 and variant of k-means clustering. We find that the clustering step is the most fragile part of the interpretation creation process in practice, for each model, highest accuracy bounds from the verifier. Note also that the construction of the interpretation string does not count towards the runtime of the verifier (see Section 5).
the process several times and choose the interpretation string that yields the
J BOUNDING THE LOSS
In this paper we focus on lower-bounding the test-set accuracy. Another natural choice is the test- set cross-entropy loss, which contains information about the model's confidence in its answers that accuracy obscures. Note that, in principle, the compact proofs framework can be applied just as well to this metric, or indeed any well-defined quantity associated to the model. We view this flexibility as strength of the framework. However, those using compact proofs to evaluate model interpretations must take care that the choice of quantity being bounded is relevant to the task.
We consider two simple techniques for reusing our accuracy bound work for loss bound:
Recall from Section 5.T that we bound the accuracy by lower- bounding for each input pair v, y G the margin Ma, with which the correct logit value exceeds any incorrect logit value in the original model. (This margin Mr,y is computed as the difference hetween the idcalized model's margin and the maximum logit difference between the original and idealized models.) For each input pair æ, E G, we can guarantee that the original gets the correct answer on x, y if Mz,y > O; this results in a lower bound on accuracy. When considering loss, we can instead use tribution to the loss due r. y
of softmax to find that the con-
M
Loo (0: x, y) S - log
GG| 1 M
27

---

Published as a conference paper at ICLR 2025
The average of these terms over all y E G is then an upper bound for the total cross- entropy loss.
Another approach is to first use bi-equivariance to compute the true cross-entropy loss of the idealized model with single forward pass and then to bound the & norm between the idealized and original models' output logits using a variation of Lemma G.1 Combined
with either the fact that cross-entropy loss is V2-Lipschitz (Lemma G.3 or with a inequal- ity that takes into account second-degree information about cross-entropy (Lemma G.4
this gives a bound on the original model's loss.
In our experiments, we find that neither of these techniques suffices to give meaningful bounds on cross-entropy loss. This lack of success is somewhat to be expected-wy start with approaches designed to bound accuracy, and then attempt to crudely adapt them to loss. Better bounds on loss would likely require new techniques which are out of scope for this paper. See Figure 9 for an example of loss bounds through the margin for models trained on Sr.
K
BEYOND SYMMETRIC GROUPS
K.1 OTHER GROUPS WITH REAL IRREPS
In the main text, we focus on the symmetric group Ss. For our purposes, this group is especially nice for several reasons:
Symmetric groups S,, have only real irreps, the sense that every irrep over C isomor- phic to an irrep with only real matrix entries. See Section K.2 for preliminary discussion of groups that do not have all real irreps.
The minimum faithful p-sets of Sn are small relative to the order of the group. In other words, S, has small faithful permutation representations because it can be embedded into itself S,, + Sn. (In general, an arbitrary finite group G can be embedded into a symmetric group G + Sn by Cayley's 's theorem, but unless G itself is symmetric group must have S.. > (G,l-)
For groups of significantly smaller order, the training dataset is too small and we do not observe the grokking phenomenon. (Recall that training size is proportional [G|².) For groups of significantly larger
Related to the second point above, we empirically observe that groups with larger p-sets relative to group order more prone to failure mode (o-bad) i.e. they typically miss substantial portion of pairs in the double summation of Eq. 4 In this situation, we do not have a complete understanding of how the model attains high accuracy, and thus our bounds are correspondingly poor. Note that, although we cannot fully explain how neurons interact in this case, our per-neuron observations (B.2 1-3) hold for all finite groups we examine.
Despite these points, we are able to obtain nonvacuous bounds for models trained on the symmetric group S4 (see Figure TOp and for models trained on the alternating group As (see Figure I1
K.2 COMPLEX AND QUATERNIONIC TRREPS
An irrep being real is equivalent to it having positive Frobenius-Schie indicator ((p) " tr(p(g)). In general, irreps have Frobenius-Schur indicator in 0, -11}. correspond- ing to the irrep being real, complex, and quaternionic respectively. These three cases correspond to the ring of G-linear endomorphisms of the irrep being isomorphic to either IR, C, H... By Schur's lemma, the endomorphism ring is a real associative division algebra, these are the only three cases.
In preliminary investigations of more general irreps p. convert irreps over C with nonpositive Frobenius-Seee Schuur indicator to irreps over It of twice the dimensionality:
Re p(g) Im p(g)
Im p(g)]
((g) =
(15)
Re p(g)-
28

---

Published as a conference paper at ICLR 2025
approximately rank when p real, rank two when p is complex, and rank four when p is quaternionic. In the complex case, we find also that when IR is given the complex structure induced by Eq [5| the two singular vectors are conjugate, and correspond to equal singular values. Thus, we speculate that the neural network uses the same p-sets algorithm in the real case, but over C, and then takes the real part: letting E GL(n, () and B S Cd a finite p-set,
We then find that the A, B, C' matrices of Observation B.2
fp,(( [x,y) -- Σ
Reb" p(=)b' ReLU p(z)a a ((y)b')]
b
29

---

Published as a conference paper at ICLR 2025
all
0.036
embedding left embedding right unembedding unembed.bit
0.034
Ideal model loss
0.032
0.030
0.028
0.028
0.030
0.032
0.034
0.036
Original model loss
Figure 7: Cross-entropy loss of original model cross-entropy loss of model with parameters partially exchanged for idealized version. 100 models trained on St, restricted to those where neither (a-bad) nor (p-bad) Legend indicates which parameters exchanged; for instance, red points have unembedding weights w, swapped for idealized version. Blue points are the full idealized model. Note they have loss uniformly lower than original model. Points corresponding left embedding are obstructed by those corresponding to right embedding.
all
0.036
all iireers
5d-0
0.034
4d-0
Ideal model loss
Id-1
0.032
0.030
0.028
0.028
0.030
0.032
0.034
0.036
Original model loss
Figure 8: Cross-entropy loss of original model vs. cross-entropy loss of model with neurons (em- bedding and unembedding) corresponding specified irreps exchanged for idealized version. 100 models trained on Sc, restricted to those where neither (a-bad) (p-rd)) Legend indicates which parameters exchanged. Blue points are the full idealized model while for orange points only neurons corresponding to some irrep are swapped (that is, dead neurons are preserved from the orig- inal). Red points correspond to the standard irrep 4d-0 and purple points correspond to the sign irrep d-1. See Section H for a full enumeration of irreps of Ss-
30

---

Published as a conference paper at ICLR 2025
brute
Virrep expl
Cross entropy bound
10
Virrep unexpl
brute mean
100
Virrep all mean
V irrpp expl mean
10-1
0.75
1.00
1.25
1,50
1.75
2.00
2.25
2.50
Time (s)
Figure 9: Cross-entropy bound vs. computation time for Vireep and Vrute on 100 models trained on S3- Points green (Virep unexpl) are models for which find by inspection that our p-sets explanation does not hold, i.c. either (a-bad) or (p-bad); they make up 45% of the total. Points in blue are Virrep for explained models and points in orange are Virute. Black dashed line is log|Gi R 4.79, the loss attained by model that outputs uniform logit values. A priori, there is no guarantee that given model does least as well the uniform baseline. Thus, in sense, any finite upper bound cross-entropy is nonvacuous.
1.0
Vr Virrep expl
0.8
Accuracy bound
Virrep unexpl
0.6
Vrutee mean
Virrep all mean
0.4
Virrep expl mean
0.2
0.0
0.04
0.06
0.08
0.10
0.12
0.14
Time (s)
Figure 10: Accuracy bound computation time for Vireep and Varute 100 models trained on S₁. Points in green (Virep unexpl) are models for which we find by inspection that our p-sets explanation does not hold, i.c. either (a-bad) (p-bad), they make up 48% of the total. Note that the latter condition occurs much more frequently than for Ss. Points in blue are Virep for explained models and points in orange are Vorute-
31

---

Published as a conference paper at ICLR 2025
1.0
0.8
Accuracy bound
brute
0.6
Yirrep expl
Vitrep unexpl
0.4
brute mean
0.2
Virrep all mean
Virep expl mean
0.0
000
0.2
0.3
0.4
0.5
0.6
Time (s)
Figure 11: Accuracy bound vs. computation time for Virrep and Virute 100 models trained on Aş. Points in green (Vireep unexpl) are models for which we find by inspection that our p-sets explanation does not hold, iec. either (a-bd)J [p-bad)] they make up 78% of the total. Note that the latter condition occurs much more frequently than for Sg. Points in blue are Virep for explained models and points in orange are Vorute-
32


Reviews:
[{"review_id": "ZshMrAy4Fm", "reviewer": "Reviewer_QyuD", "summary": "This paper contributes to a recent line of work aiming to mechanistically understand the computations performed by neural networks trained on the symmetric group. It takes a step towards this goal by developing an interpretation of the model's computation that can be formally translated into a compact proof of model performance. This compact proof of performance can be measured against the actual performance of the network, serving as a quantitative measure of the quality of a proposed interpretation. The interpretation proposed by the authors is based on their notion of rho-sets, which corresponds to an interpretation of the network learning to become approximately equivariant in each of its inputs. The rho-set interpretation gives rise to a compact proof that can account for the behaviour of approximately half of the models they train. Previous work on the symmetric group came to differing conclusions based on \"irrep sparsity\" and cosets, but for the approximately 50% of models the rho-set interpretation can account for, it is able to unify the differing interpretations of previous works and show that they are not at odds in these cases. For the other half of models which they are unable to account for with their interpretation, the compact proofs fail to attain non-vacuous bounds. Thus, the authors argue that compact proofs are a concrete way to measure the validity of one's interpretation of neural network computations.", "questions": "1. Can compact proofs be used on datasets without \"closed-form\" solutions?\n2. The way that V_coset is being computed could be fully responsible for the results that cosets have loose vacuous bounds. Reverse engineering the problem qualitatively shows cosets (Stander et al.). I'm wondering if you tried other ways of modeling V_coset? If so, how many other ways did  you try?\n3. In section 3.2 why do you take the number of neurons to be equal to the embedding dimension (m)? Is this by chance or necessary for your proofs and interpretation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Compact proofs are a new way of supporting model interpretations and it seems like they could be interesting, since as the paper states, valid compact proofs can be generated from interpretations one is certain of.\n2. For the half of the models that the rho-set interpretation works for, explaining how to reconcile the irrep sparsity and cosets interpretation is helpful.", "weaknesses": "1. While compact proofs are an interesting way to approach interpretability, it's unclear whether they could be used to help interpret neural network solutions for datasets where no or limited explicit information is known about the distribution it was sampled from (e.g. any language task, CIFAR-10, etc.).\n2. The fact that the compact proofs derived in this work only get approximately a 50% success rate is concerning, as it implies that the framework using rho-sets is possibly not general enough.\n3. Unifying is too strong a word to use in the title when rho-set compact proofs only work approximately 50% of the time.\n4. As someone who has familiarity with representation theory, it's still quite hard to understand the rho-set construction and specifically how it can be identified within the network. I must believe that it works since you can write a compact proof (verifier) that empirically matches the network's performance around half the time. However, since it's unclear how you arrived at this rho-set interpretation by mechanistically inspecting the network, it's not clear how other people can use this to come up with compact proofs for other datasets. If compact proofs are to be useful in the field of interpretability, you should be more clear about how you went about figuring this out. E.g. what are the concrete steps you thought of and experiments you ran to define everything in section 5.1 as well as the observations in Appendix B. Being explicit about these things could greatly help the community understand how to integrate and improve interpretations and contribute to compact proofs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dVIQ24Baym", "reviewer": "Reviewer_ZpTx", "summary": "- The paper presents an algorithm by which a one hidden layer MLP network (with embeddings) does composition in $S_5$\n    - The algorithm is somewhat involved, but the key step is constructing an expression for $f(z | x,y)$ (eqn 4) that is purely a function of $x^{-1}zy^{-1}$ and maximised at $e$. \n    - This is constructed in an interpretable way using $\\rho$-sets, an introduced concept of a set of vectors permuted by irreps of the group G. For a set of $k$ vectors, it needs $k^2$ neurons (one corresponding to each pair of $\\rho$-set vectors). \n    - The full network consists of several such constructions, in parallel, using disjoint sets of neurons and possibly different irreps and $\\rho$-sets\n- This is the same setting as studied by previous mechanistic interpretability works Chughtai et al & Stander et al, but presents a much more detailed algorithm, with a clear story for each parameter and the role of each neuron. \n    - The story provided here helps unify and clarify observations in each paper such as coset concentration and irrep sparsity. \n    - Some aspects of prior work are criticised, such as the specific causal interventions used as evidence in Stander et al\n- This explanation is used to provide a lower bound on the model's accuracy, by analysing the margin (correct logit - max incorrect logit) in the idealised model, and comparing that to the worst case logit deviation in the real model.\n    - This accuracy bound is proven via a scheme that takes asymptotically less time than brute force trying every input (about 3x faster in practice). This proof fails on some networks, but this seems to correspond to ones where the explanation is imperfect.\n    - This is referred to as a compact (i.e. short) proof, inspired by Gross et al\n    - The existence of a compact proof seems to be taken as evidence that the explanation is correct, as it gives a faster yet provably valid verification method", "questions": "# Major Comments\n*The following are things that, if adequately addressed, would increase my score*\n1. The key thing that would improve this paper, in my opinion, is making it clearer, especially key technical details.\n    - I found sections 4 and 5 difficult to follow - the concepts of coset concentration and irrep sparsity are introduced, without much motivation, and are then not necessary to explain the algorithm. I would personally reverse the flow of 4 & 5.1 and work backwards:\n        1. Begin with Eqn 4, and observe that this is bi-equivariant, and sometimes maximised at e (citing lemma F.3)\n        2. Show that this is equivalent to Eqn 3\n        3. Observe that each term in the sum can be a neuron, and what the relevant w_r, w_l, and w_u need to be for that - we've now constructed a valid algorithm!\n        4. Irrep sparsity and coset concentration can then be explained as prior observations, and shown to follow from this algorithm. \n    - I find the term \"compact proof\" quite cryptic, and it took me until about page 7 to figure out what was going on. A decent part of the confusion is that you use proof to refer to what I'd normally call a program. I would have benefitted a lot from an intuitive explanation in the intro or start of section 6 (ideally both, plus something in the abstract). Something like:\n        - A guarantee of model accuracy is a program that can be run to guarantee that models will always give the correct answer at least X% of the time, for some X (i.e. lower bound its accuracy). This can always be done for the brute force \"try every possible input\" program, but it seems that mechanistic understanding of a model should enable more efficient programs. These are referred to as compact proofs. The efficiency of the program, and closeness of the accuracy bound to the true accuracy, can be taken as metrics of the quality of our explanation.\n2. I'm somewhat skeptical of compact proofs as an interpretability metric, for several reasons - I would love to be convinced otherwise though:\n    - They bound accuracy, not loss. But a model's performance can be over-determined, with several parallel components each being sufficient to ensure perfect accuracy but all needed to recover the loss, as is common in toy systems like this (you do need to be able to bound the effect of other components, but not necessarily to understand them). IMO an explanation that doesn't understand all such components is incomplete, but it may get fantastic accuracy.\n    - It's not clear to me that a mechanistic explanation, even if extremely accurate, should always enable faster proofs. Or even be robust to worst case guarantees at all.\n        - While it did in this work, this was *extremely* specific to the setting and explanation, and I don't feel confident there would be other approaches for less mathematically elegant algorithms.\n    - The framing in the paper was that being asymptotically faster than brute force was the key thing. But in practice, the coefficient on the compact proof was much worse, and it was 3x faster not 120x faster. IMO 3x is the relevant number here.\n    - That said, I find the fact that they seem to identify networks where your explanation is incomplete to be quite compelling.\n3. Similarly, I would be excited to see other evidence that your explanation is correct - it makes a lot of predictions about the form of the parameters and activations!\n    - Can you learn a and a $\\rho$-set that explains a cluster of neurons? Does it have size $k^2$ exactly?\n    - How well does your prediction for a neuron's activations match it in practice? What's the MSE and correlation? If you replace the neuron with the prediction (either one at a time, or on the full group for one $\\rho$-set) what happens to model performance.\n    - How close are the model's parameters (or at least, $w^i_l$ etc) to the predicted form? What if you substitute part of those?\n    - I found the claim in Lines 502-503 that causal interventions can only yield weaker positive evidence to be overstated - this applies to the interventions used in Stander et al, but IMO those are pretty weak interventions and not that compelling by the standards of current mechanistic interpretability. This doesn't mean there don't exist more compelling causal interventions.\n4. Separately, I'm extremely skeptical that compact proofs will scale beyond very toy networks (let alone to frontier systems). Being robust to worst case scenarios on the entire space of inputs seems highly unrealistic to me for eg a language model or image model. Methods don't have to scale to be interesting, of course, but this limits my excitement about the method. I'd value arguments for why the method might extend to eg imperfect coverage of the input space.\n5. I'd be very curious to see how well \"the square of the size of the smallest $\\rho$-set$\" correlates with the probability that $\\rho$ is learned, eg using the numbers in Chughtai et al. This would significantly clarify the results in Chughtai et al re universality if true.\n\n# Minor Comments\n*The following are unlikely to change my score, but are comments and suggestions that I hope will improve the paper, and I leave it up to the authors whether to implement them. No need to reply to all of them in the rebuttal*\n1. Line 243: Explicitly note that we can set w^i_l(x) to whatever we want, it's a lookup table. It's a construction, not a conclusion. This confused me at first\n2. I find the term \"compact proof\" somewhat confusing. To me, proof connotes showing something rigorously about abstract mathematical objects . But perhaps this is just taste, and this notion of proof is common in e.g. the field of formal verification?\n3. I don't understand what Figure 1 is trying to show, a shame as you clearly put in effort there! How is S3 mapped to points on a hexagon? What are the terms in the top row with 4 vertices circled? What does adding them mean? What is X_12? Etc I recommend significantly clarifying or changing the figure\n4. Obviously, it would be great to replicate the paper's results on other groups! A5, A6, S4 seem natural to try. I predict the results to hold up though.\n5. I find the definition of compact proofs somewhat odd - what exactly does it mean to be a valid lower bound for *any* explanation string? It strikes me as odd that your compact proof must first begin by eg verifying if a subset of G is a subgroup, when that seems independent of the model.\n6. I personally find Eqns 2 and 3 clearer by replacing $b^T \\rho(x)a$ with $a^T \\rho(x^{-1}) b$ (which is equal since the transpose of a scalar is the identity), as this motivates the subtraction and substitution more clearly.\n7. I liked the point in the appendix that $ReLU(x) = (x + |x|)/2$, though it was not clear to me why the $x$ part cancelled out.\n8. Line 109: Causal Scrubbing was introduced in [Chan et al](https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing), not Geiger et al\n9. Line 116: It would be good to define the word equivariance here\n10. Line 194: I found the claims about embeddings being constant on a coset for a neuron very confusing. It felt like it was asserting this about *all* networks, not explaining a property of a specific constructed network\n11. Line 196: What does it mean for a subset of G to be common to a family of cosets? Cosets are subsets of G, so surely the intersection of a family of cosets is a set of subsets, not a subset?\n12. Line 436: Why do you refer to neurons as functions $G\\to \\mathbb{R}$ rather than $G^2\\to\\mathbb{R}$? They have two inputs, $x$ and $y$, right?\n13. Lemma F.3: Are you arguing that all $\\rho$ have the decomposition described here? If not, which ones do, and does this correspond to the irreps learned by the model?\n14. In table 2 in Appendix G, why does the minimal $\\rho$-set size go above 5? Naively, it feels like an irrep of S5 should always be able to permute a set of 5 vectors.\n15. Line 452: The claim that the frequency $\\rho$ is learned correlates with its minimal $\\rho$-set size seems questionable to me. The less frequent 4D one and more frequent 5D one are learned about the same amount of the time in Chughtai et al (Figure 7), despite having $\\rho$-set size 10 and 6 respectively, and both less than half as often as the more frequent 4D one (minimal size 5). Your finding may help explain which of the representations at a given dimension is chosen, but it's clearly incomplete", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The authors have found an elegant yet highly non-obvious algorithm for group composition in a single ReLU layer, that multiple prior papers missed. This is a valuable contribution to the literature.\n    - Further, the provided explanation clarifies and adds useful context to observations made in prior work\n2. The presented compact proofs are highly detailed and rigorous, and actually explicitly go through every key detail, rather than hand-waving annoying points.\n3. It demonstrates compact proofs as an interpretability metric in a much more complex setting than prior work (Gross et al)\n4. The authors do a good job of highlighting weaknesses, e.g. the times the proof does not work, the failed V_coset proof, and clearly reporting the actual runtimes of the proofs\n5. Though highly technical and complex, to the best of my ability to tell, the maths largely checks out. I did not follow the fine detail of all of the appendices.", "weaknesses": "1. This is only studied on $S_5$\n2. The compact proof is only 3x faster than brute force\n3. The paper is highly technical and at times quite difficult to follow, especially as it builds deeply on 3 prior papers! Though the authors have clearly made an effort to be clear, and this is an inherently complex work. This took me significantly longer than other reviews.\n4. The link between finding a compact proof of a bound on accuracy, and verifying a mechanistic explanation, seems somewhat unclear", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7KLFtJDTro", "reviewer": "Reviewer_t8QQ", "summary": "This paper unifies two previously proposed explanations for a small neural network trained in a controlled setting. Specifically, it studies the internals of a one-layer neural network trained to perform group composition on finite groups $G$. The input to the model is an ordered pair $(x, y)$ with $x, y \\in G$, which are embedded as vectors, and the output is the group element. Prior work studied the same setting and proposed different explanations for the model behaviour: Stander et al. (2024) suggested that individual neurons develop a specialised coset behaviour where their left embeddings remain constant on right cosets and their right embeddings remain constant on left cosets, creating specific subsets $X_i$ where neuron activations sum to zero. Chughtai et al. (2023) found that each neuron operates in the linear span of matrix elements of some irreducible representation, implementing the matrix multiplication through ReLU nonlinearities and then maximising a trace computation to predict the group composition. First, for each of these explanations, the authors point out parts of the behaviour that is left unexplained. They then unify these explanations by showing that neurons are not just using irreducible representations randomly, but are organised in specific circuits. They also show exactly how the ReLU computation works through equation 4, which suggests that the function is bi-equivariant and is maximised when $z = x \\star y“. Together these findings provide a more complete picture of the models computations. \n\nTo evaluate the quality of their explanation, they convert their understanding into a verifier program that aims to provide a compact proofs of model performance. Specifically, this verifier aims to use mechanistic insights to reduce the description length of the program (i.e. its runtime). They compare brute-force, the coset explanation (Stander et al.), and their own explanation in terms of accuracy bound and runtime over 100 models trained on the same task. They find that brute-force provides exact accuracy bounds but takes the longest to run (2.2 seconds), the coset explanation does not provide any non-vacuous accuracy bounds, and their explanation obtains a bound of 80 - 100 % for half the models and a non-vacuous bound of 0 % for the other half. Upon inspection, they find that the models where their explanation was not able to obtain non-vacuous bounds, the model converged to other solutions.", "questions": "- You currently cite the survey of Geiger et al. (2024) for causal scrubbing (line 109). However, I believe it was first introduced in Chan et al. (2022; https://www.lesswrong.com/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The authors successfully reverse-engineer a neural network trained to perform group operations and provide a more complete explanation than prior work. \n- They rigorously evaluate the quality of their explanation, highlighting that it only explains a subset of solutions a model with this architecture might learn in practice. \n- Their evaluation exposes limitations of causal interventions as positive evidence of explanations.", "weaknesses": "Overall, I think this is a solid contribution without significant weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)