On the Symmetries of Deep Learning Models and their Internal Representations
Authors: Charles Godfrey
Davis Brown
Tegan Emerson
Henry Kvinge
Venue: NeurIPS 2022 Accept (2022)
Keywords: symmetry in deep learning
representation learning
representation similarity

Abstract: Symmetry has been a fundamental tool in the exploration of a broad range of complex systems. In machine learning, symmetry has been explored in both models and data. In this paper we seek to connect the symmetries arising from the architecture of a family of models with the symmetries of that family’s internal representation of data. We do this by calculating a set of fundamental symmetry groups, which we call the intertwiner groups of the model. Each of these arises from a particular nonlinear layer of the model and different nonlinearities result in different symmetry groups. These groups change the weights of a model in such a way that the underlying function that the model represents remains constant but the internal representations of data inside the model may change. We connect intertwiner groups to a model’s internal representations of data through a range of experiments that probe similarities between hidden states across models with the same architecture. Our work suggests that the symmetries of a network are propagated into the symmetries in that network’s representation of data, providing us with a better understanding of how architecture affects the learning and prediction process. Finally, we speculate that for ReLU networks, the intertwiner groups may provide a justification for the common practice of concentrating model interpretability exploration on the activation basis in hidden layers rather than arbitrary linear combinations thereof.

On the Symmetries of Deep Learning Models and their Internal Representations
Charles Godfrey! 1, Davis Brown'.', Tegan Emerson 1,3,4 Henry Kvinge 12,33 Pacific Northwest National Laboratory, 2 Department of Mathematics, University of Washington, Department of Mathematics, Colorado State University,
DDpaartment of Mathematical Sciences, University of Texas, El Paso "qqual contribution f first. last@pnnl.t9gg.
Abstract
Symmetry has been fundamental tool in the exploration of broad range of complex systems. In machine learning, symmetry has been explored in both mod- els and data. In this paper we seek to connect the symmetries arising from the architecture of a family of models with the symmetries of that family's internal representation of data. We do this by calculating a groups, which we call the intertwiner groups of the model. Each of these arises from a particular nonlinear layer of the model and different nonlinearities result in different symmetry groups. These groups change the weights of model in such way that the underlying function that the model represents remains constant but the internal representations of data inside the model may change. We connect intertwiner groups model's internal representations of data through a range of experiments that probe similarities between hidden states across models with the same architecture. Our work suggests that the symmetries of network are propagated into the symmetries in that network's representation of data, provid- ing us with a better understanding of how architecture affects the learning and prediction process. Finally, we speculate that for ReLU networks, the intertwiner groups may provide a justification for the common practice of concentrating model interpretability exploration the activation basis in hidden layers rather than arbitrary linear combinations thereof.
fundamental symmetry
Introduction
Symmetry provides important path understanding across range of disciplines. This principle well-established in mathematics and physics, where it has been a fundamental tool (e.g., Noether's Theorem [Noe18]). Symmetry has also been brought to bear deep learning problems from number of directions. There is, for example, a rich research thread that studies symmetries in data types that can be used inform model architectures. The most famous examples of this are standard convolutional neural networks which encode the translation invariance of many types of semantic content in natural images into network's 's architecture. In this paper, we focus on connections between two other types of symmetry associated with deep learning models: the symmetries in the learnable parameters of the model and the symmetries across different models' internal representation of the same data.
The first of these directions of research starts with the observation that in modern neural networks there exist models with different weights that behave identically on all possible input. We show in section 3 that at least some of these equivalent models arise because of symmetries intrinsie the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

nonlinearities of the network. We call these groups of symmetries. each of which is attached to particular type of nonlinear layer a of dimension n. the intertwiner groups G., of the model. These intertwiner groups come with natural action on network weights for which the realization map to builds on a significant recent advance in the field.
function space of [JGH18] 8] is invariant (proposition 3.4). As such they provide unifying framework in which to discuss well-known weight space symmetries such as permutation of neurons [Bre+19] and scale invariance properties of ReLU and batch-norm [IS NH10J.
Next, we tie our intertwiner groups to the symmetries between different model's internal represen- tations of the same data. We do this through a range of experiments that we describe below; each
The work of [BNB21; Csi+21; LV15] demonstrated that one can take two trained neural networks, say A and B, with the same architecture but trained from different randomly initialized weights, and connect the early layers of network A to the later layers of network B with a simple "stitching" layer and achieve negligible loss in prediction accuracy. This was taken as evidence of the similarity of strong model's representations of data. Though the original experiments use fully connected linear layer to stitch, we provide theoretical evidence in theorem 4.2 that much less is needed. Indeed, we show that the intertwiner group (which has far fewer parameters in general) is the minimal viable stitching layer to preserve accuracy. We conduct experiments stitching networks ReLU activation layers with the stitching layer restricted to elements of the group GReLU showing in fig. that one can stitch CNNs on CIFAR-10 10 [Kri09] with only elements of G incurring less than 10% accuracy penalty at most activation layers. This is surprisingly close to the losses found when one allows for a much more expressive linear layer to be used stitch two networks together. However, we see that there remains a significant gap between the stitching accuracies obtained using GReLU and fully connected linear layers; this provides independent confirmation of earlier findings that neurons of networks trained with different random seeds (i.e. with independent initializations and different random batches) are not simply permutations of each other [Li+15; Wan+18J, also consistent with observed phenomena such as distributed representations in hidden features [GBC16, $ $15.4] and perhaps also polysemantic neurons [Ola+20].
Neural stitching with intertwiner groups:
stitching penalties in myrtie CNN
full 1x1 canv
60
strching penalty (5% acc)
rank 1x1
50
rank 2 1x1 canv
rank: 1x1 conv
Grelu 1x1 conv
30
20
10
stitching layer
Figure 1: Full, reduced rank and GRELU 1-by-1 convolution stitching penalties (4.3) for Myrtle CNNs [Pag on CIFAR-10. Confidence intervals were obtained by evaluating stitching penalties for 32 pairs models trained with different random seeds. The accuracy of the models was 91.3 + 0.2%.
Representation dissimilarity measures for GReLU: In section 5 we present two statistical dissimi- larity measures, GRLLU Procrusrss and FRRLU for ReLUvactivaeed hidden features in different networks, say A and B. Our measures are counterparts of orthogonal Procrustes distance (see e.g. [DDS21b]) and Centered Kernel Alignment (CKA)' [Kor+19] respectively, which are invariant to orthogonal transformations, and are maximized when the hidden features of networks A and agree up to orthogonal transformations. In contrast Gn Proocu-rosssssss and GReLU-CK. are invariant to GRELU transformations, and are maximized when hidden features agree up to GRELU transformations. We compare and contrast our measures with their orthogonal counterparts, as well as with stitching experiment results. Figure 2 shows a comparison of GRLLU and orthogonal CKA measures.
with linear RBF kernl..
2

---

Grale CKA narwi20
orttopornl 1eane220
00
04
60
95
68
-04
65
68
54
12"
727 01
344
45
10
model kaper
mosel 2 layer
Figure 2: GR-LU-CKA and orthogonal CKA between layers of ResNet20s trained CIFAR-10. Results averaged over 16 pairs of models trained with different random seeds Layers marked with *** occur inside residual blocks (remark 3.6). For further details see section 5.
frandion interprelable units
06
64
Resliet 90,
modiled ed Roshi: 50, Rhal reskd. GoroNee, trai cencus
0.2
6.1
0.0
02
04
0.88
0.5 1.0
fraction rotation
Figure 3: Fraction network dissection interpretable units under rotations of the representation basis for ResNet-55,, well as a modified ResNet-50 and a ConvNeXt model (both without an activation function on the residual output). Section 6 contains details and further discussion.
Impact of activation functions interpretahility: An intriguing finding from [Bau+11] 17, $3.2] was that individual neurons are more interpretable than random linear combinations of neurons. Our results intertwiner groups (theorem 3.3) predict that this is a particular feature of ReLU networks. Indeed, fig. 3 shows that in the absence of ReLU activations interpretability does not decrease when one moves from individual neurons to linear combinations of neurons section describes this experiment in further detail. This result suggests that intertwiner groups provide theoretical justification for the explainable AI community's 's focus individual neuron activations, rather than linear combinations thereof [Erh+09; Na+19; Yos+15; ZF14; Zho+15], but that this justification only valid for layers with certain activation functions.
Taken together, our experiments provide evidence that a network's 's symmetries (realized through intertwiner groups) propagate down to symmetries of model's internal representation of data. Since understanding how different models process the same data a fundamental goal in fields such as explainable AI and the safety of deep learning systems, hope that results will provide an additional lens under which to examine these problems.
Related work
The research on symmetries of neural networks is extensive, hence we aim to provide a representative
sample knowing it will be incomplete. (Bre+19; FB 17; GBC16; Yi+19] study the effect of weight space symmetries on the loss landscape. On the other hand, [BMC15; GBC16; Kun+21; Men+19 19] study the effect of weight space symmetries training dynamics, while [GBC16; RK20] show that weight space symmetries pose an obstruction to model identifiability.
Neural stitching was introduced [LV15]. In [BNB21] was shown to have intriguing connections with the "Anna Karenina" " (high performance models share similar internal representations of data) and "more is better" (stitching
means of comparing learned representations between networks

---

later layers of weak model to early layers of a model trained with more data/parameters/epoohh can improve performance) phenomena. [Csi+21] considered constrained stitching layers by restricting the rank of the stitching matrix or by introducing an & sparsity penalty. Our methods are distinct in that we explicitly optimize over the intertwiner group for ReLU nonlinearities (permutations and scalings). Both [BNB21; Csi+21] compare their stitching results with statistical dissimilarity measures such as CKA. Our CReLU-PrOOS Procrustes measure is a close relative the permutation Procrustes distance introduced in [Wil+21], II, and our GReLU-CKA is instance of CKA [Kor+19] in which the kernel is taken to be max 31,..., Za Ma).
[Li+15] developed algorithms for obtaining a permutation to align neurons, and [Wan+ 18] introduced neuron activation subspace matching and used it to study similarity of hidden feature representations. [AHS22; Ent+22; Tat+20] all aligned neurons with permutations with the goal of obtaining low-loss paths between the weights of networks. The objectives for neuron alignment used in these works include maximizing correlation ([AHS22; Li+15; Tat+20J), maximizing a "match" (as defined in [Wan+18], simulated annealing search algorithms ([Ent+22]), direct alignment of weights via a bilinear assignment problem and "straight- estimator" of back- pack-propagated training loss ([AHS22]). Each of these is distinct from our method, which explicitly seeks a permutation mini- mizing training loss and searches for one using standard convex relaxation methods for permutation optimization.
Approaches to deep learning interpretability sometimes assume that the activation basis is special [Erb+09; Na+19; Yos+15; ZF14; Zho+15]. Studying individual neurons rather than linear combina- tions of neurons significantly reduces the complexity of low-level approaches to the understanding of neural networks [E]h+21]. In tension with this, many different projections of hidden layer activations appear to be semantically coherent [Sze+14] However, [Bau+17] 17] found evidence that the hidden feature vectors closer to the coordinate basis align more with human concepts than vectors sampled uniformly from the unit sphere.
3 The symmetries of nonlinearities
Let Mat (IR) be the algebra of all no x ni real matrices and GL,, (IK) be the group of all invertible X matrices. Let a: IR + IR be continuous function. For any E N. we can build nonlinearity on from 8" to IR" by applying a coordinatewise, i.e., on (Zi,..... In) = (o (z) ( (xx))) Fix
some k > 1 and for each 1 < i k let l R + R. be the composition of affine layer and a nonlinear layer, so that (z(2) == on, (W,z + b,), and let (x) = WAI + by. Here We E Mata, ni-1 (R) and b, E R" are the weights and bias of layer i respectively. We define f R" + I2* to be the neural network f la o ..... l, For each 1 S i < k 1 we can then decompose f as f = Szi Isi where
fer = l ... and fxt = 0 .... 1.111
We define
W = (W,
k) and W:= II (Mat (Mat (R) x R":)
where the former is the collection of all weights of ff and the latter is the space of all possible weights for given architecture. When we want to emphasize the dependence of f on weights W, we write s(-,W) (and similarly
One of the topics this work will consider is vector space bases for f's hidden spaces RT, for We will investigate the legitimacy of analyzing features (D) for dataset C R with respect to the activation basis for R which is simply the usual coordinate basis, ci...., ex, where e, = [ is naturally parameterized by individual neuron activations. Note that R has an infinite number of other possible bases that could be chosen.
1 I i S
3.1 Intertwiner Groups
For any 0 < i < ks elements of GLn,, (R) can be applied to the hidden activation space R% both before and after the nonlinear layer an,- We define
Gon, == {A GL,,, (RR there exists B GLi,,, (R) such that one A = Bo B.

---

Activation
Go,
(A)
(x)) (identity)
GL, (R)
A
(2) 14
Σn
A
(x) RcLU(a)
Matrices PD, where D has positive entries
A
(xx) LeakyReLU(A
Same as ReLU as long negative slope +
A abs(A)
(2) V2* e- + (RBF)
Matrices PD, where D has entries in {+1}
(x)) = x² (polynomial)
Matrices PD, where D has non-zero entries
A
Table 1: Explicit descriptions of Go,, and Oe for six different activations. Here P E Σn is permutation matrix, D is diagonal matrix, abs denotes the entrywise absolute value, and Add denotes the entrywise dth power.
Informally, we can understand Gon, to be the set of all invertible linear transformations whose action on IR prior to the nonlinear layer on; has equivalent invertible transformation after o. This is an instance of the common procedure of understanding a function by understanding those operators that commute with it. For any A E GLn,, (R), we can write o (AA) for the x n, matrix formed by applying to all entries in A.
Lemma 3.1. Suppose o (In) is invertible and for each A E GL,, Then Go.. is a group, &a Gon - +L,,, (R) is homomorphism and on A = Φo (A) o dr.
(R) define do (A) = (A))
We defer all proofs to appendix E. We include concrete examples of for small n, there as well. Definition 3.2. When the hypotheses of lemma 3.1 are satisfied (namely, (4,, is invertible) we call Go, the intertwiner group of the activation on. We denote the image of the homomorphism &a as & (G..).
The intertwiner group Go,, and 6 are concretely described for a range of activations in table 1 the last two examples motivate the generality of definition 3.2. Note also that in both of those cases A + 6b (A) is not a homomorphism on all of L,R)., but is a homomorphism when restricted to the appropriate subgroup G.,. While substantial part of table can be found scattered in prior work, our calculations in appendix E.2 deal with the different cases of table in a uniform way, by what amounts to an algorithm that compute Go,, and de given in terms of elementary properties of any (reasonable) activation function a." As design of activation functions remains an active industry (for example [Elh+22]), our techniques for computing Go,, could be useful in future studies of network symmetries.
The following theorem shows that the activation basis is intimately related to the intertwiner group of RcLU: GRLLU admits a natural group-theoreti characterization in terms of the rays spanned by the activation basis, and dually the rays spanned by the activation basis be recovered from GReLU While both its statement and proof are elementary, our interest in this theorem lies in the question of whether it could potentially provide theoretical justification for focusing model interpretation studies on individual activations. We investigate this question further in section 6.
Theorem 3.3. The group GReLU,, is precisely the stabilizer of the sei of rays (R>ge, C RRIi I...., n). Moreover if R>ov1,.... "rri,.,..,,,,,,, RzouN C E is finite set of rays stabilized by GReLU, then for each [UI..... Uin]", J", it must be that Viş = 0 for all but
E nI... Equivalently up to multiplication by a positive scalar every vi is of the form Ie for some j.
3.2 Weight space symmetries
The intertwiner group is also a natural way to describe the weight space symmetries of neural network. We denote by F C C(R" 1x*x) ) the space of continuous functions that can be described by network with the same architecture as f. As described in [JGH18] there is realization map $ W - F mapping weights W E W to the associated function f E F. t arises because there are generally multiple sets of weights that yield the same function. We will show that tb is invariant
We defer further discussion of and references to this prior work to appendix E.2.
5

---

Grelu 1xt convolution
otthogonal
rotat ootaton penelly
rotation layer
Figure 4: Rotation penalties for Myrtle CNNs on the CIFAR-10 dataset. Confidence intervals were obtained by performing 10 independent trials of the experiment with different random seeds, and baseline accuracy was Rs 87%..
with respect action of the intertwiner groups W so that intertwiner groups form set of "built-in" weight space symmetries of f. This result, which encompasses phenomena including permutation symmetries of hidden neurons, is well known in many particular cases (c.g.. [GBC16, $8.2.2). [Bre+19, $31, [FB17, $2], [Men+19, $3J, [RK20, $3, AJ). From Proposition 3.4 we can also derive corollaries regarding symmetries of the loss landscape Proposition 3.4. Suppose A; E Go., for S i k - 1, and ler
these are included in appendix E.
W' (, A1b,, A2W20 (A71) A2b2.
(2 bn)
Then, as functions, for each m
JSm (Ir, W")= = ( Ssm(1, W) and Jum W') - ->m (2,, W) do (Am) (3.5) In particular, f2x (') f(x, W) for all E R Equivalently, we have ((W) +(W E Remark 3.6. We will show in appendix E that the statement of this theorem must be modified if the architecture of f contains residual connections. By placing suitable restrictions the matrices A, we can recover a form of eq. (3.5) provided occurs at the end of residual block. However, there doesn't It seem be way obtain such identity when y occurs inside a residual block; empirical evidence consistent with this point in figs. 2, 6 and 9 below.
3.3 A "sanity test" for intertwiners
To test proposition 3.4 with simple experiment, we begin with a Myrtle CNN [Pag18] 8] network' trained for 50 epochs on the CIFAR-10 dataset, fix a pre-activation layer I, and apply transformation A to the weights Wi and biases by to obtain AW; and Ab; (we only act on channels, hence in practice this is implemented by an auxiliary 1-by-1 convolution layer). We consider 2 choices of A: (i) random element of GRELU. where P is random permutation and the diagonal entries of D are sampled from a lognormal distribution, and (ii) a random orthogonal matrix, obtained as the "Q" in RR-decomposition of random matrix X with independent standard normal entries.
Next, we freeze layers up and including and finetune the later layers for another 50 epochs. We refer to the difference between the validation accuracy before and after applying the transformation A and finetuning as a rotation penalty. Based on proposition 3.4, when E GneLU the network should be able to recover reasonable performance even with the transformed features for example, On the other hand, with probability there no matrix such
by updating W4+1 to (A) that updating W4+1 to W4+ B counteracts the effect of an orthogonal rotation A on all possible input. We see that this is indeed the case fig. 4: transforming by E GRLLU rather a random orthogonal matrix results in significantly smaller rotation penalties.
4 Intertwining group symmetries and model stitching
In this section we provide evidence that some of the differences between distinct model's internal representations can be explained in terms of symmetries encoded by intertwiner groups. We do this
e Namely, that A; = Am layers and m are joined by a sequence of residual connections. "Thi is simple 5-layer CNN, with residual connections described further in appendix D.

---

using the stitching framework from [BNB21; Csi+21], and begin by reviewing the concept of network stitching.
Suppose f. f are networks as in section 3, with weights W, W respectively. For any 1 S 1 k- we may form stitched network S(S, f, I =R defined in the notation of section by f(, far p fzi here y _ R is a stitching layer. In typical stitching experiment one trains networks f and from different initializations and freezes their weights, constrains yp to some simple function class S (e.g., affine maps in [BNB21], D. and trains S f. 4) by optimizing ş alone. The final validation accuracy Acc S(, 1, sp) of S f,J, sp) is then considered measure of similarity (or lack therof) of the internal representations of f and f in R in this framework the situation
Ace S(f, = Acc f
(4.1)
corresponds to high similarity since the hidden representations model f and could be related by a transformation S.
Recall that even though the networks f(W) and f(W") may be equal as functions, their hidden representations need not be the same (an example of this is given in appendix C). Our next result shows that in the case where f and f do only differ up to an element of Gan, eq. (4.1) is achievable even when the stitching function class S is restricted down to elements of do (Gom.) ) (see definition 3.2).
Theorem 4.2. Suppose W (A, A1b1, A2 (AT"), A2b2,.... (A (A where A; E Go., for all i. Then eq. (4.1) is achievable with equality if the stitching function class S containing ip contains @o (Gam)
Motivated by theorem 4.2, we attempt to stitch various networks at ReLU activation layers using the group GReLU described in Figure 1. Every matrix A GReLU can be written PD, where P is a permutation matrix and D is diagonal with positive diagonal entries - hence optimization over GReLU requires optimizing over permutation matrices. We use the well-known convex relaxation of permutation matrices to doubly stochastic matrices and describe our optimization procedure in greater detail in D.2.
Figure gives the difference between the average test error of Myrtle CNN networks f and f and the network
which we call the stitching penalty:
Acc(f) + Acc(f) - -cc((SU
(4.3)
2
In experiments S(f. F. f, was stitched together at layer via stitching transformation ip that was either optimized over all affine transformations, reduced rank affine transformations as in [Csi+21] or transformations restricted to GReLU. We consider only the ReLU activation layers, as these are the only layers where the theory of section applies, and we only act on the channel tensor dimension in practice, this is accomplished by means of 1-by-1 convolution operations. In particular, with GReLU we are only permuting and scaling channels. Lower values indicate that the stitching layer sufficient to translate between the internal representation of f at layer and the internal representation of f.
We find that when we learn a stitching layer over arbitrary affine transformations of channels, we can nearly achieve the accuracy of the original models. When we only optimize over GRLLU there an appreciable increase in test error difference. This is consistent with findings [Csi+21; Li+15; Wan+ discussed in section 2, and also consistent with observations that hidden features of neural networks exhibit distributed representations and polysemanticism [Ola+20]. Nonetheless, that S(f f. (, :p) is able get within less than 10% of the accuracy of f and f in all but one layer suggests that elements of GReLU account for substantial amount of the variation in the internal representations of independently trained networks. We include the reduced rank transformations as the dimension of their parameter spaces is greater than that of GReLU,, and yet they incur significantly higher stitching penalties. If n, is the number of channels, we have dim GReLU", = my whereas the dimension of rank transformations is 2n 2 (hence greater than dim GReLU even for
7

---

r r= 1).² Finally, in the specific case of the Myrtle CNNs the stitching penalties incurred when using any layer other than 1-by-1 convolution with rank matrix all follow similar trends: they increase up to the third activation layer, then decrease at the final activation layer.
Further stitching results on the ResNer20 architecture can be found in appendix D.3, including an experiment where we modify the architecture to have LeakyReLU activation functions, vary the negative slope of the LeakyReLU, and find similar stitching penalties up to but not including slope of 1. This result is consistent with our calculations in table 1, where we find that for any LeakyRell negative slope # 1 the intertwiner the same as GReLU (when the negative slope 1, LeakyReL RuixyyeLLU(())) = r and so the intertwiner is all of GL
5 Dissimilarity measures for the intertwiner group of ReLU
Stitching penalties can be viewed as task oriented measures of hidden feature dissimilarity. From a different perspective, we can consider raw statistical measures of hidden feature dissimilarity. In the design of measures of dissimilarity, a crucial choice is the group of transformations under which the dissimilarity measure is invariant. For example, Centered Kernel Alignment (CKA) [Kor+19] with the dot product kernel is invariant with respect to orthogonal transformations and isotropic scaling. We ask for statistical dissimilarity metric μa on datasets X, Y E R xd with the properties that (0) 0 S μ(X, < 1, (i) (GReLU-Inee IIvariance) If A, B E GROLU, and v, w E Rd then ((XA + 1oT,, YB lw") = (xX, Y), and (ii) (Alignment Property) μ(X, Y)
if (*) Y = XAA 1x for some A E GReLUU, v Rdd To motivate this question, we note that given such a metric Jt, one can detect if X and Y do not differ by an element of GReLU by checking if (Y < 1. Our basic tool for ensuring (i) is the next lemma.
Lemma 5.1. Suppose u(XA,YD) = ,(X, Y) if B, B are either positive diagonal matrices or permutation matrices. Then, (i) holds.
In effect, this allows to divide the columns of X and Y by their norms to achieve invariance to the action of positive diagonal matrices and then apply dissimilarity measures for the permutation group such as those presented in [Wil+21]. Ensuring (ii) seems to require case-by-case analysis to determine an appropriate normalization constant.
(.4])) and Dy diag( diag((Y (.4l)) Assuming these are invertible, let X = XDx and = YDy Let ö be the permutation Procrustes distance between X, Y, defined by 8 = minpex,, XX Y Pl (as pointed out in [Wil+21] this can be computed via the linear sum assignment problem). Then the GReLU-Procruse
Definition 5.2 (GReL.--Prose Procrustes). Let Dx diag((X(...))
measure is
PProcrustes (X,Y) =
2vd
The factor of 2/d ensures this lies in 00, 1]. and equals 1 if (and only if) the condition of (ii) holds.
layer 3
layer 6
layer 10
layer 14
GRELU 0.6208 + 0.008
0.5106 + 0.005
0.4432 + 0.004 0.4899 + 0.002
Orthogonal
0.7724 + 0.028
0.5743 = 0.040
0.5087 = 0.016 0.5825 + 0.019
Table 2: GReLU and orthogonal Procrustes similarities for Myrtle CNNs trained CIFAR-10. Confidence intervals were obtained by evaluating similarities for 32 pairs models trained with different random seeds.
We apply CReL.U-rrocruse and orthogonal Procrustes similarities to 4 different hidden represen- tations from Myrtle CNNs in table and many more layers of ResNet20s in fig. 9, all trained on CIFAR-10 [Kri09]. In keeping with the discussion of section 4, we only consider permuta- tions or orthogonal transformations of channels (for details on how this is implemented we refer
'A valid concern that the preceding analysis underestimates size GReLU by ignoring a large discrete factor: GROLU., has ny! connected components. In appendix E.7 we carry out comparison of the sizes of the parameter spaces of GRaLU,, and reduced rank transformations inspired by the machinery of e-nets, obtaining the same conclusion that even the space of rank I transformations is larger than GReLU.
8

---

Grelu CKA for myrtle CNN
orthogonal CKA myrtle CNN
0.8
0.84
0.6
0.31
0.21
3
0.98
0.91
0.53
0.37
model 1 layer
model layer
0.8
0.61
0.31
0.21
0.6
0.9
0.97
0.66
10
0.31
0.31
0.45
0.34
1.0
0.52
0.66
0.92
0.83
0.6
0.4
0.2
0.21
0.35
0.43
0.37
0.49
0.83
0.93
0.4
10
14
3
6
10
14
model layer
model 2 layer
Figure 5:
GReLU-CK. AKAA and orthogonal CKA for two Myrtle CNNs with different random seeds trained on CIFAR-10. Results averaged over 16 such pairs of models.
to appendix D.7). We see that distinct representations register less similarity in terms of GReLU- Procrustes than they do in terms of orthogonal Procrustes. This makes sense as similarity up to tteuu-transformatio requires greater degree of absolute similarity between representations than is required of similarity up to orthogonal transformation (the latter being a higher-dimensional group containing all of the permutations in GReLU). Otherwise patterns in GReLU-PrOcHt Procrustes similaritiy largely follow those of orthogonal Procrustes, with similarity between representations decreasing as one progresses through the network, only to increase again in the last layer. This correlates with the stitching penalties of fig. 1, which increase with depth only to decrease in the last layer.
Definition 5.3 (GReLU- -KK)) Assume that X and Y are data matrices that have been centered by subtracting means of rows: X + X 1111 and + Y 111"Y. Let X = XD and Ý DDT Let 1,,...... .,,..., N be the rows of X, and similarly for Y. Form the matrices K,L E 12 N defined by KG max(ã; 2 @ 25) and Lyj max(ū; @ Ut) where @ is the Hadamard product. Then the GReLU-CKE -KA for X and Y is defined as:
HSTC (K, L)
(X)
(5.4)
V HSIC, K) (L,
where HSTC; is the unbiased form of the Selbert-Schmidt independence criterion of [NRK21, eq. 3].
Symmetry of the max function ensures (i). the Cauchy-Schwarz inequality ensures μACKA (X, Y) E if the condition of (ii) is met. We do not claim 'ff and only if", however we point out the following in lemma E.44: if A is matrix such that max(Az) Ar²) max(#1 3 x2) for all I1, Fd then A is of the form PD where is permutation matrix and D is diagonal with diagonal entries in { {+1] In fact, PCKA is simply an instance of CKA for the "max kernel."
[0, 1], and we claim that (CKA(X, Y)
Lemma 5.5. The function Rd x Rdd - R defined by k(x, max(# O v) is positive semi-definite kernel.
As with CKA [Kor+19], this metric makes sense even if X, Y are datasets in Rd _2d respectively with d * d'. Results for pair of Myrtle CNNs trained on CIFAR- 10 with different random seeds, as well as standard orthogonal CKA for comparison, are shown in fig. 5. Analogous results for ResNet20s are shown in fig. 2. We find that GRLLU-CKA respects basic trends found in their orthogonal counterparts: model layers at the same depth are more similar, early layers are highly similar, and the metric surfaces the block structure of the ResNet in fig. (layers inside residual blocks are less similar than those at residual connections). One notable difference for GRA.U-CKA in figs. 2 and 5 is that the similarity difference between early and later layers in the orthogonal CKA (discussed for ResNets in [Rag+21]] shown in (b) is less pronounced in (a), and in fact later layers are found to be less similar between runs. We found similar results for stitching in figs. I and 6.
Interpretability of the coordinate basis
In this section we explore the confluence of model interpretability and intertwiner symmetries using network dissection from [Bau+17]. Network dissection measures alignment between the individual

---

neurons of a hidden layer and single, pre-defined concepts (see appendix F.1 for the methodology). We adapt an experiment from [Bau+17] to compare the axis-aligned interpretability of hidden activation layers with and without an activation function. Bau et al. compares the interpretability of individual neurons, measured via network dissection, with that of random orthogonal rotations of neurons. We likewise rotate the hidden layer representations and then measure their interpretability. Using the methodology from [Dia05], we define a random orthogonal transform Q drawn uniformly from (O(n) by using Gram-Schmidt to orthonormalize the normally-distributee QR A E R Like in [Bau+17], 17], also consider smaller rotations Qa (O(n) where 0 I a S 1, where is chosen to form a minimal geodesic rotating from / to Q- [Bau+17] found that the number of interpretable units decreased away from the activation basis as a increased for layer5 of an AlexNet.
We compare three models trained on ImageNet: a ResNet-50,, a modified ResNet-50 where we remove the ReLU on the residual outputs (training details in appendix F.3), and a ConvNeXt [Liu+22] analog of the ResNet-50, which also does not have an activation function before the final residual output. We give results in fig. 3. and provide sample unit detection outputs and full concept labels for the figures in appendix F.2. As was shown in [Bau+17], interpretability decreases as we rotate away from the axis for the normal ResNet-50 in appendix F33 On the other hand, with no activation function, neuron interpretability does not drop with rotation for the modified ResNet-50 and the ConvNeXt. We note that the models without residual activation functions also have far fewer concept covering units for given basis. Interestingly, while the number of interpretable units remains constant for the residual output of the modified ResNet-50,, for the ConvNeXt model it actually increases. We find similar results, where the number of interpretable units increase with rotation, for the convolutional layer inside the residual block for the modified ResNet-50 in fig. 19.
7 Limitations
Our theoretical analysis in section 3 does not account for standard regularization techniques that are known to have symmetry-breaking effects (for example weight decay reduces scaling symmetry). More generally, we do not account for any implicit regularization of our training algorithms. As illustrated in figs. 1 and 6, stitching with intertwiner groups appears to have significantly more cetaitectuur-eeenddeer behaviour than stitching with arbitrary affine transformations (however, since different architectures have different symmetries this is to be expected). Our empirical tests of the dissimilarity measures in section 5 are limited to what [Kor+19] terms "sanity tests"; in particular did not perform the specificity, sensitivity and quality tests of [DDS21a].
8 Conclusion
In this paper we describe groups of symmetries that arise from the nonlinear layers of neural network, calculate these symmetry groups for number of different types of nonlinearities, and explore their fundamental properties and connection to weight space symmetries. Next, we provide evidence that these symmetries induce symmetries in network's 's internal representation of the data that it processes, showing that previous work on the internal representations of neural networks can be naturally adapted to incorporate awareness of the intertwiner groups that we identify. Finally, in the special case where the network in question has ReLU nonlinearities, we find experimental evidence that intertwiner groups justify the special place of the activation basis within interpretable Al research.
Acknowledgements
This research was supported by the Mathematics for Artificial Reasoning in Science (MARS) initiative at Pacific Northwest National Laboratory. It was conducted under the Laboratory Directed Research and Development (LDRD) Program at at Pacific Northwest National Laboratory (PNNL), multiprogram National Laboratory operated by Battelle Memorial Institute for the U.S. Department of Energy under Contract DE-ACOS-76RLO1830
The authors would also like to thank Nikhil Vyas for useful discussions related to this work.
10

---

References
[AHS22] Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. "Git Re-Basin: Merging Models modulo Permutation Symmetries". In: (2022). DOL: 10. 48550/850/AA0 /AXIVV. 2209. 04836. URL: https: /
//tpxa/xvvv////////////ae///aaa/ .rg/ abb./209... abbs/2209008366.
[Bau+17]
David Bau et al. "Network Dissection: Quantifying Interpretability of Deep Visual Rep- resentations". In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), pp. 3319-3327.
[BMC15]
Vijay Badrinarayanan, Bamdev Mishra, and R. Cipolla. "Understanding Symmetries in Deep Networks". In: ArXiv (2015).
[BNB21]
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. "Revisiting Model Stitching to Compare Neural Representations". In: NewrIPS. 2021.
[Bre+19]
Johanni Brea et al. "Weight-Space Symmetry in Deep Networks Gives Rise to Per- mutation Saddles, Connected by Equal-Loss Valleys across the Loss Landscape". In: (July 5, 2019). arXiv: 1907. 02911 [cs, stat]. URL: http /ttx/aaxivv/////////////////aall org/ abs/ 1907. 02911 (visited on 01/12/2022).
[Csi+21]
Adrián Csiszárik et al. "Similarity and Matching of Neural Network Representations" In: Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer et al. 2021. URL:
[DDS21a] Frances Ding, Jean-Stanislas Denain, and J. Steinhardt. "Grounding Representation Similarity with Statistical Testing". In: ArXiv (2021).
[DSS21b] Frances Ding, Stan-Stanislas Denain, and Jacob Steinhardt. "Grounding Representation Similarity Through Statistical Testing". In: Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer et al. 2021. URL: htttts:///peeeeeee http///ppeeeeeeeeeeeeeeeeeeeeeee wtwps//weeeeoooooeeeeeeeeeeeeeee horrf?
kd=_kvj6V5324B.
[Dia05]
Persi Diaconis. "What a random matrix". In: Notices of the AMS 52.11 (2005), pp. 3348-1349
[Eth+21]
N Elhage et A mathematical frammaaticaalammewoor. for transformer circuits. 2021.
[Eth+22]
Nelson Elhage et al. "Softmax Linear Units". In: Transformer Circuits Thread (2022).
[Ent+22]
Rahim Entezari et al. "The Role of Permutation Invariance in Linear Mode Connectivity
of Neural Networks". In: International Conference Learning Representations, 2022.
URL: https: http://pppereeeieeeeeeeee net/
[Erh+09]
Dumitru Erhan et al. "Visualizing higher-layer features a deep network". In: University
of Montreal 1341.3 (2009), p. 1.
[FB17]
Daniel Freeman and Joan Bruna. "Topology and Geometry of Half-Rectified Network Optimization". In: ArXiv abs/1611 111611001544 (2017).
[GBC16]
Ian Goodfellow. Yoshua Bengio, and Aaron Courville. Deep Learning. attp://wer deeplearningbooe org. MIT Press, 2016.
[1S15]
S. loffe and Christian Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift". In: ICML (2015).
[JGH18]
Arthur Jacot, Franck Gabriel, and Clément Hongler. "Neural tangent kernel: convergence and generalization in neural networks (invited paper)". In: Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing (2018).
[Ko++19]
Simon Kornblith et al. "Similarity of Neural Network Representations Revisited". In: ICML (2019).
[Kri09]
Alex Krizhevsky. Learning multiple layers of features from tiny images. Tech. rep. 2009.
[Kun+21]
Daniel Kunin et al. Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics. Mar. 29, 2021. arXiv: 2012. 04728 [cond-mat, q-bio, stat]. URL: attp::/a http::://arvvv..vo abs //1220012222222 (visited on 01/12/2022).
[Li+15]
Yixuan Li et al. "Convergent Learning: Do Different Neural Networks Learn the Same Representations? " In: NIPS. 2015.
[Liu+22]
Zhuang Liu et "A ConvNet for the 2020s". In: arXiv preprint arXiv:2201.03545 (2022).
[LV15]
Karel Lenc and A. Vedaldi. "Understanding Image Representations by Measuring Their Equivariance and Equivalence". In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015). DOI: 10. 1109/CVVR, 2015. 7298701.

---

[Men+19]
Qi Meng et al. "G-SGD: Optimizing ReLU Neural Networks in Its Positively Scale- Invariant Space". In: ICLR. 2019.
[Na+19]
Seil Na et al. "Discovery natural language concepts individual units of cnns". In: arXiv preprint http:::/02.074999 (2019).
[NH10]
Vinod Nair and Geoffrey E. Hinton. "Rectified Linear Units Improve Restricted Boltz- mann Machines". In: JCML. 2010.
[Noe18]
E Noether. "Invariante Variationsspoolemme", In: 1918, pp. 235-257.
[NRK21]
Thao Nguyen, Maithra Raghu, and Simon Kornblith. "Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth". In: International Conference Learning Representations. 2021. URL: https:://... // http//ppereveeeeeeeee rewnreeeiew..... wwt/forummaaii
[Ola+20]
Chris Olah et al. "Zoom in: An introduction to circuits". : In: Distill 5.3 (2020), c00024- 001.
[Pag18]
David Page. How to Train Your ResNet. Myrtle. Sept. 24, 2018. URL: httpt::t/t/tttteeeee
ai ha//earn/how--ee http://oo-ttttri--ourrere trrrinaoooourr - tore-----rrii-y----reeeetet////
(visited on 05/09/2022).
[Rag+21]
Maithra Raghu al. "Do vision transformers like convolutional neural networks?"
In: Advances in Neural Information Processing Systems 34 (2021).
[RK20]
D. Rolnick and Konrad Paul Kording, "Reverse-Engineering Deep ReLU Networks", In: ICML. 2020.
[Sze+14]
Christian Szegedy et al. "Intriguing properties of neural networks". In: CoRR abs/1312.6199 (2014).
[Tat+20]
N. Joseph Tatro "Optimizing Mode Connectivity via Neuron Alignment". In: (Nov. 2, 2020), arXiv: 2009. 02439 [cs, math, stat]. URL: http://arxiv. org/ abs/2009. 02439 (visited on 05/13/2022).
[Wan+18]
Liwei Wang et al. "Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation". In: NeurIPS. 2018.
[Wil+21]
Alex H. Williams al. "Generalized Shape Metries Neural Representations" In: NeurIPS. 2021.
[Yi+19]
Mingyang Yi et al. "Positively Scale-Invariant Flatness of ReLU Neural Networks". In: ArXiv (2019).
[Yos+15]
Jason Yosinski et al. "Understanding Neural Networks Through Deep Visualization". In: Deep Learning Workshop, International Conference Machine Learning (ICML). 2015.
[ZF14]
Matthew D Zeiler and Rob Fergus. "Visualizing and understanding convolutional net- works". In: European conference on computer vision. Springer. 2014, pp. 818-833 Bolei Zhou et al. "Object Detectors Emerge Deep Scene CNNs." In: ICLR. 2015.
[Zho+15]
URL: http:// /ttt:///axivv www.aiiiw..aanaaa...a..a.o httgg/abs/4412/ 6856.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This paper is largely focused on the mathematical aspects of deep learning so we do not think there are any immediate negative societal impact to the methods described. From a broader perspective though, we see this work helping to create a more principled groundwork for many interpretable Al techniques. We explain why this could have positive societal impacts in Section A.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs can be found in appendix E.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URI.)? [TODO] We are in the process of making code publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See appendix D.
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See appendix D- while did not keep precise track of CPU/GPU hours, we do specify the hardware used.
4. If you are using existing assets (e.g., code, data, models) or curating/relcusing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Section G.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "bliQtuEWd3Q", "reviewer": "Reviewer_kCuW", "summary": "The paper contributes to understanding the phenomenon that different networks produce similar performance, which the symmetries can cause. \n\nThe paper mainly focuses on ReLU-activated layers, and contributes in the following way: 1) the paper first provides empirical evidence of the similarity of strong model’s representations, then there is a theoretical analysis of the minimal layer for neural stitching that can preserve accuracy—the intertwiner group; 2) the paper proposes a measure of similarity between representations learned with different neural networks, which considers representations have the highest similarity if they differ only up to orthogonal rotation and isotropic scaling; and 3) the paper further analyzes the interpretability of intertwiner groups, though the theory is limited to the certain activation function.\n", "questions": "Please see **Strengths And Weaknesses**.", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper shows the following **strengths**:\n\n- The topic studied in this paper helps with understanding the neural network better. For now, most of the time, how the neural network works and what representation it extracts remain unclear. This paper contributes to providing a representation similarity measure that would not be affected by orthogonal rotation, which also emphasizes the difference between representations than the baseline does, especially between the lower layer representation and the higher layer representation.  \n\n- The paper focuses on ReLU activation, which is a commonly used activation function in deep learning. Thus the result could be helpful in further research in the community.\n\n- The proposed method's efficiency is supported by theoretical analysis and empirical results. Conditions that when theorem (3.3) does not hold are also reported.\n\n- The submission is well organized.\n\nIn the meanwhile, I still have the following **questions and concerns**.\n\n- In line 218, the paper mentions the stitching penalty increases up to the third activation layer. The number here (the third layer) is very specific. Would it be too specific to mention this number in conclusion? According to the evidence provided in the paper, I believe there exists such a pattern that the penalties increase first and then decrease, but I am not sure three is a threshold. There are always different network settings, such as the total number of layers and the type of neural network. Different settings might give different thresholds. (To be more clear, I am not saying the experiment is not enough; I am only concerned that it might be too specific to say three is the threshold.)\n\n- The experiment has 5 independent runs, which is acceptable as the confidence interval seems small, though more runs could make the result more convincing.\n\n**Small Things**:\n\n- In line 59, it might be better to add the full name of CKA. I understand that the full name is mentioned in Section 5, but it might be more clear if the full name gets explained as soon as CKA is mentioned.\n\n- In line 197, is it $S(f,\\tilde{f}, l, \\phi)$?\n\n- In line 259, is it $X-\\frac{1}{n}\\mathbf{1}\\mathbf{1}^T X$ and $Y-\\frac{1}{n}\\mathbf{1}\\mathbf{1}^T Y$?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MNG8LPeRiiJ", "reviewer": "Reviewer_Eay8", "summary": "This paper introduce an intertwiner group $G_\\sigma$ of an activation $\\sigma$, which is a subgroup of $GL(n)$ that a set of invertible matrices that commute regarding the activation function, i.e., $A \\circ \\sigma = \\sigma \\circ A$ (for, e.g., ReLU). This group structure can naturally explain the weight space symmetry of neural networks, i.e., a realization map $\\phi: W \\mapsto f$ is $G_\\sigma$-invariant s.t. $\\phi(W') = \\phi(W)$ where $W'$ is a transformed weight by acting a set of elements in $G_\\sigma$. The authors use the intertwiner group to investigate the internal representations of ReLU networks from the perspective of the model stitching and representation dissimilarity. The $G_{ReLU}$-based stitching and dissimilarity metric provide a consistent result with previous works. In addition, they provide an application of the intertwiner group for explainable AI.", "questions": "For the major comments, please see Strengths and Weaknesses. In short, I believe the presentation of key results and their significance needs improvement.\n\nA minor comment: in line 173 of page 6, it seems that $f_{\\leq i}$ should be  $f_{\\leq l}$.", "limitations": "The authors discuss the limitation of the proposed method in the Limitations section, and I agree with such limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Investigating the learned representation based on symmetry is an interesting problem. The intertwiner group might be a principled group to explain the underlying symmetry behavior of neural networks. \n\nThe mathematical concept of the intertwiner group and its property are well-presented and discussed. \n\nThe paper is generally well-written, while it is a bit hard to read mainly due to the confusing structure of the paper. For example, Fig. 2 and Fig. 3 appear in the early stage of the paper; however, they are hard to understand before reading the experiments (Section 4 - 6).\n\nRegarding the weight space symmetry, I am not sure whether the introduced intertwiner group is particularly novel or significant compared to previous studies based on other groups, e.g., permutations and scaling. At first glance, the intertwiner group seems to be a comprehensive version of such studies (from the perspective of various activations beyond ReLU, e.g., LeakyReLU, RBF, and polynomial?), but it is not clearly presented in the current version of this paper.\n\nFor the model stitching experiment, I agree that the internal representations of separately trained ReLU networks might be invariant up to the intertwiner group. It might be a piece of good evidence or tool when discussing the “Anna Karenina” scenario of deep neural networks. One concern is that the stitching penalty of full 1 x 1 convolution is much lower than that of the intertwiner group, as shown in Fig. 1. It will be nice if the authors clarify it.\n\nFrom the perspective of the metric, it seems that the intertwiner group provides a tighter bound for hidden layer dissimilarity compared to the previously proposed one (that is based on the orthogonal group). However, I am not sure what is the main use-case of this metric. It would be nice if the authors mention regarding it more precisely.\n\nOverall, to me, the significance of the intertwiner groups compared to the previous work is not entirely clear, although the introduced mathematical concept of them seems to be solid.  Because I am not familiar with the topic of this paper, there might be mistakes in my current evaluation. Please correct me if I am wrong.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1CvlJfYIN9j", "reviewer": "Reviewer_cBdu", "summary": "This paper studies symmetries of the space of neural network parameters, i.e., invertible transformations of the parameters which leave the forward function invariant.  They compute this set of symmetries, called the intertwiner group, for networks with different activation functions.  As those authors note, these symmetries have been defined and studied before by other authors.  The primary focus of this work is on the impact of these symmetries on learned hidden representations in the network.  The author pursue a line of research which asks to what extent networks trained from different random initializations learn effectively the same hidden features?  In particular, what amount of variation is due to parameter space symmetry?  Through splicing experiments (Sec. 4), they shown that representations from one network can spliced into another using the intertwiner group for only a small drop in performance.  In Section 5, the authors compute directly using two novel metrics how close learned hidden representations are to being related by a single intertwiner for the whole dataset.  Section 6 shows, using network dissection, that the significance neuron activations (versus linear combinations of activations) to interpretability depends on the choice of activation function. ", "questions": "\n## Questions \n- Line 165, skip connections: I think this can be addressed by choosing a different group action than the one in Prop 3.4 which acts by the same A_i on layer which are tied by a skip connection.  \n- Why call G_{relu}-Procrustes distance a distance when it is larger for more similar inputs?  Perhaps \"similarity measure\"? \n- Figure 5: What is meant by 5 runs of the experiment? Does this mean you trained 10 independently initialized networks and reported distances for 5 of the pairs?  What happens to the value is we compare all 5*4/2 pairs? The similarity scores don't seem that high, implying a high amount of variability, so it seems a bit surprising the numbers are so consistent between pairs.  \n- Figure 6: Grelu CKA seems to fall off faster with depth than CKA, why do you think this is?\n- The argument for needing Grelus version of Procrustes and CKA (it seemed to me) was that the similarity measure should be adapted to the symmetry group of the network.  However, empirically they are very well correlated (esp. -Procrustes.)  If so, then doesn't that undermine the need for a new metric?  Also, while the gap between the rows is explained and makes sense, why are the rows in Figure 5 so well-correlated? \n- Line 310-311: Since the rotation is random and the activations should be independent of basis, how do you explain the surprising increase in performance? \n\n\n## Minor Points\n- Figure 4 and Figure 5 are actually tables and should read ``Table'' \n- Figure 3 the y-axis is written as percent.  Is it percentage or fraction?\n- Figure 3 the x-axis is again a fraction of the sampled rotation.  I wouldn't use the word degree which can be confused with rotations out of 360 degrees. \n- Line 109 You should clarify $k$ is the last layer before giving a different definition for $l_k$.\n- Line 173 should $i$ be $l$?\n- In the example in Appendix C (second part), A is defined, but then the 2 and 1 switch places in the equation.  Is it correct? \n- Line 206: \"we are only allow\" \n- Line 254  \" Grelu-orthogonal\" should be \"orthogonal-procrustes\"?\n- Line 260 $\\tilde{X}$\n- Line 306 in Figure 3\n\n## Post-Response Edit\nI appreciate the author's response to my questions.  I particularly appreciate the explanation on \"Empirical correlation between G_{relu} and orthogonal dissimilarity measures\" and alternate analysis of capacity or stitching maps.   The additional experiment examining \"decay of G_{relu},\" rules out one possible explanation, but I suppose it remains for future work to examine more closely. \n\n I looked at the MLP experiment in Appendix D.4 as an attempt to answer whether this hypothesis holds for MLPs as well as CNNs.  It seems the experiment is negative result in that symmetry does not well explain the difference between hidden reps here.  That's too bad, but I don't think it detracts overly from the current paper.  It does raise the question for future study of when parameter-space symmetry does or does not account for variation in latent representation. \n\nI have read the other reviews and author responses, and I looked at the revised submission.   I will maintain my positive assessment. ", "limitations": "Yes, the author include a clear limitations section. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \n\n- This work presents a nice mixture of theoretical and empirical results connecting symmetries of parameter space to learned hidden features.  Although the authors are not the first to point out these symmetries, their treatment is rigorous, more general, and easy to follow.  Unlike earlier work, they connect parameter space symmetries to the question of variability of learned hidden representations.  The intertwiner symmetries may be considered change-of-basis symmetries for hidden layers.  If one is going to ask how much learned hidden features vary for different initializations, it is natural to ask how much of this variation is due to symmetry alone.  Network splicing where the splices are given by intertwiners is a clever way to get at this question.  The empirical splicing results in Section 4 and the computed similarity measures in Section 5 show that much (but not all) of the variability is reducible to symmetries of parameter space.  \n- The definition of intertwiners given (Defn 3.2) is more general than that given by other authors, allowing for the group action to be transformed after applying non-linearity.  This expands the set of a potential symmetries as in the examples in Figure 4.\n- Splicing with G_{relu} requires optimizing over the (large and discreet) permutation group.  This would appear to be a difficult problem but the authors achieve good results with a convex relaxation to doubly stochastic matrices.   \n- I read parts of the appendix, which is quite thorough, provides good proofs for the statements in the paper, additional results and examples which are helpful for understanding.  I want to point out for the other reviewers there are some small mistake corrections and improvements to the main pointed out in the appendix (e.g. A --> A_m in (5)).  \n\n## Weaknesses\n- The splicing comparison in Sec. 4 could be more exhaustive.  a) The comparison is done only for convolutional networks across the channels using 1x1 convolutions.  Does a similar result hold for MLPs?  b) Moreover, the justification that the rank 1, rank 2, and rank 4 1x1 have greater capacity to achieve lower error doesn't seem that convincing to me.  The advantage of Grelu is that it has access to a very large space of (full rank) permutation symmetries.  Since these are discreet, they don't count towards dimension, but they do give access to many useful transforms in GL_n.  In other words, dimension may not be the fairest way to compare.\n- I personally find the interpretability argument put forward in Section 6 to be not that strong and somewhat aside from the main point of the paper.  The design of artificial neural networks using component-wise activations in order to model distinct neurons means that the distinct choice of basis in the activation vector space is baked into the network design.  The fact that linear combinations of neuron activations are less interpretable neuron activations is thus reasonable on its own without reference to the (also interesting) observation regarding the permutation symmetry of component-wise activations.  \n- I find it a bit hard to interpret the results in Section 5.  In particular, I lack a scale/baseline for, e.g., the numbers in Figure 6.  What does a Grelu-CKA of 0.63 for layer 6 of two models mean in terms of understanding how much of the variability of trained representations comes from symmetry?  Or are there other lessons I should draw from the results? \n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept