Investigating how ReLU-networks encode symmetries
Authors: Georg Bökman
Fredrik Kahl
Venue: NeurIPS 2023 poster (2023)
Keywords: loss landscape
network merging
linear mode connectivity
equivariance
group convolutional neural network
permutation
group
symmetry
invariance
weight space ensembling
TL;DR: If a neural network is group equivariant, does it have to be layerwise equivariant? We investigate the question theoretically and empirically.

Abstract: Many data symmetries can be described in terms of group equivariance and
 the most common way of encoding group equivariances in neural networks is by building linear layers that are group equivariant.
In this work we investigate whether equivariance of a network implies that all layers are equivariant.
On the theoretical side we find cases where equivariance implies layerwise equivariance, but also
demonstrate that this is not the case generally.
Nevertheless, we conjecture that CNNs that are trained to be equivariant will exhibit layerwise equivariance and explain how this conjecture is a weaker version of the recent permutation conjecture by Entezari et al.\ [2022].
We perform quantitative experiments with VGG-nets on CIFAR10 and qualitative experiments with ResNets on ImageNet to illustrate and support our theoretical findings. 
These experiments are not only of interest for understanding how group equivariance is encoded in ReLU-networks, but they also give a new perspective on Entezari et al.'s permutation conjecture as we find that it
is typically easier to merge a network with a group-transformed version of itself than merging two different networks.

Investigating how ReLU-networks encode symmetries
Georg Bökman Fredrik Kahl
Chalmers University of Technology {bokman, fredrik. Jahl
Abstract
Many data symmetries can be described in terms of group equivariance and the most common way of encoding group equivariances in neural networks is by building linear layers that are group equivariant. In this work we investigate whether equivariance of network implies that all layers are equivariant. On the theoretical side we find cases where equivariance implies layerwise equivariance, but also demonstrate that this is the case generally. Nevertheless, we conjecture that CNNs that are trained to be equivariant will exhibit layerwise equivariance and explain how this conjecture is a weaker version of the recent permutation conjecture by Entezari et al. [2022]. We perform quantitative experiments with VGG-nets on CIFAR10 and qualitative experiments with ResNets ImageNet illustrate and support our theoretical findings. These experiments are not only of interest for understanding how group equivariance is encoded Entezari et al.'s permutation conjecture
Relttwntss
but they also give a new perspective as we find that typically easier to merge a network with a group-transformed version of itself than merging two different networks.
Introduction
Understanding the inner workings of deep neural networks is a key problem in machine learning and it has been investigated from many different perspectives, ranging from studying the loss landscape and its connection to generalization properties to understanding the learned representations of the networks. Such understanding may lead improved optimization techniques, better inductive biases the network architecture and to more explainable and predictable results. this paper, we focus ReLU-networks- nettookk with activation function ReLU(x:) max(0, x) and how they encode and learn data symmetries via equivariance.
Equivariances can be built into neural network by design. The most classical example is of course the CNN where translational symmetry is obtained via convolutional layers. Stacking such equivariant layers in combination with pointwise ReLU activations results in equivariant network. We will pursue different research path and instead start with a neural network which has been trained to be cquivariant and ask how the equivariance is encoded in the network. This approach in itself is not new and has been, for instance, experimentally explored in (24] where computational methods for quantifying layerwise equivariance new theoretical results when network equivariance implies layerwise equivariance. We will also give counterexamples for when this is not the case. Another insight is obtained via the recent conjecture by Entezari et al. [13 which states that networks with the same architecture trained on the same data often close to each other in weight space modulo permutation symmetries of the network weights. Our new conjecture 3.2 which we derive from the conjecture Entezari et al, states that most SGD CNN solutions will be close to group equivariant CNNs (GCNNs).
developed. We will shed new light the problem by deriving
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Net A
Net B
Figure 1: Illustration of how VGGI1 encodes the horizontal flipping symmetry. The 64 filters in the first convolutional layer of two VGGI I-nets trained on CIFAR1O are shown, where each filter is next filter in the same net which after horizontally flipping the filter results in similar convolution output. The order of the filters in the right columns is a permutation of the original order in the left columns. This permutation obtained with the method Figure 2 i.e., the columns here correspond to 1. and 4. there. Net A is trained with an invariance loss to output the same logits for horizontally flipped images. It has learnt very close to a GCNN structure where each filter the first layer is either horizontally symmetric or has mirrored twin. Net B is trained with horizontal flipping data augmentation. This net quite close to a GCNN structure, but the mirrored filters are less close to each other than in Net A. More details are given in Section
1. 2.
3.
4.
Figure 2: Permutation aligning horizontally flipped filters.
1. The three original filters in convolutional layer.
2. The three filters flipped horizontally.
3. Permutation of the flipped filters to align them with the originals.
This permutation found using activation matching following [25].
4. Flipping the filters back to their original form for illustration.
Our theoretical results and new conjecture are also validated with experiments. We focus exclusively on the horizontal flipping symmetry image classification, for the following three reasons: 1. Hori- zontal flipping is ubiquitously assumed to not change the class of an image and horizontal flipping data augmentation is practice always used. 2. The group of horizontal flips (S2) is small and has simple representation theory. 3. As we study ReLU as activation function, if a network is layerwise equivariant, then the representations acting on the feature spaces must be permutation representations as we shall prove later (cf. Section 2.1
The experiments show that networks trained on CIFARIO and ImageNet with horizontal flipping data augmentation are close to GCNNs and in particular when we train networks with an invariance loss, they become very close GCNNs. This illustrated in Figure T and also discussed in Section As further support of our main conjecture, we find that the interpolation barrier is lower for merging ResNet50 with a flipped version of itself than the barrier for merging two separately trained ResNet50's. s. In summary, our main contribution is a new conjecture on how ReLU-networks encode symmetries which is supported by both theory and experiments.
1.1 Related work
Group equivariant neural networks. The most common approach to encoding group symmetries into neural networks is by designing the network so that each layer is group equivariant [4] 14]. This
has been done for many groups, e.g., the permutation group 43 27 and the 3D rotation group
39
34
[6 4 37 Another possibility is symmetry regularization during training 332 Group

---

equivariant nets have many possible applications, for instance estimating molecular properties [3 or image feature matching [3]. For this paper, the most relevant group equivariant nets are so-called GCNNs acting on images, which are equivariant to Fuclidean transformations [10 42 40 38 In
the experiments in Section we will investigate how close to GCNNs ordinary CNNs are when they are trained to be equivariant-either by using an invariance loss or by using data augmentation.
measure layerwise equivariance by fitting linear group representations in intermediate feature spaces of networks. Our approach is similar but we restrict ourselves to permutation representations and consider fitting group representations in all layers simultaneously rather than looking at a single layer at a time. We explain in Section 2.1 why it is enough to search for permutation representations, given that we have network with ReLU activation functions, [5] measure layerwise equivariance by counting how many filters in each layer have trau-rrranforree copies of themselves in the same layer. They find that often more layerwise equivariance means better performance. Our approach explicitly looks for permutations of the filters in each layer that align the filters with group-transformed versions of themselves. We are thus able to capture how close the whole net is to being GCNN. [29] find evidence for layerwise equivariance using a qualitative approach of visualizing filters in each layer and finding which look like group-transformet versions of each other. 120 measure local layerwise equivariance, i.e., layerwise robustness to small group transformations, by computing derivatives of the output of a network w.r.t group transformations of the input. They interestingly find that transformers can be more translation equivariant than CNNs, due to aliasing effects in CNN downsampling layers.
Measuring layerwise equivariance. 24
Networks modulo permutation symmetries. [25] demonstrated that two CNNs with the same architecture trained on the same data often learn similar features. They did this by permuting the filters in one network to align with the filters in another network. [13] conjectured that it should be possible to permute the weights of one network to put in the same loss-basin as the other network and that the networks after permutation should be linearly mode connected. Le., it should be possible to average ("merge") the weights the two networks to obtain a new network with close to the same performance. This conjecture has recently gained empirical support in particular through M. where several good methods for finding permutations were proposed and [22] where the performance of a merged network was improved by resetting batch norm statistics and batch statistics of individual neurons to alleviate what the authors call variance collapse. In Section 3 we give a new version of [13]ss conjecture by conjecturing that CNNs trained on group invariant data should be close to GCNNs.
1.2 Limitations
While able show several relevant theoretical results, we have not been able to give conclusive answer in terms of necessary and sufficient conditions to when equivariance of a network implies layerwise equivariance that the network be rewritten to be layerwise equivariant. An answer to this question would be valuable from a theoretical point of view.
In the experiments, flipping. As in most prior work on finding weight space symmetries in trained nets, we only search for permutations and not scaled permutations which would also be compatible with the ReLU-nonlin eenunnooninc.artt. Our experimental results depend on a method of finding permutations between networks that is not perfect (25). Future improvements to permutation finding methods may increase the level of certainty which we can have about Conjectures 3.1 and and 3.2
limit ourselves to looking
single symmetry of images horizontal
and
2 Layerwise equivariance
We will assume that the reader has some familiarity with group theory and here only briefly review a couple of important concepts. Given group G, representation of G' is group homomorphism p: G GL( from G to the general linear group of some vector space V. E.g., if V = RTM then GV) consists of all invertible m m-matrices and p assigns matrix to each group element, so that the group multiplication of G is encoded as matrix multiplication in p(G) c GL(V).
A function f Vo + Vi is called equivariant with respect to group G with representations Po on Vo and # on Vi if [po(g)x)) = pr (x) for g E G and x E Vn- An important representation that exists for any group on any vector space is the trivial representation where p(g) I for all g. If
3

---

function f is equivariant w.r.t. Pu and Pi as above, and P1 is the trivial representation, we call f invariant. A permutaation representation a representation p for which p(g) is permutation matrix for all g E G. We will also have use for the concept of group invariant data distribution. We say that a distribution μ on a vector space G-invariant w.r.t. a representation p on V, if whenever X is a random variable distributed according to j4, then p(g) X is also distributed according to p..
Let's for now! consider neural network f 1 + RT as composition of linear layers W R 7 " and activation functions a::
f(z) = WW (2 ..)))
(1)
Assume that f is equivariant with respect a group G with representations po G - (LR on the input and PL : G + GL(RTT.) on the output, i.c.,
f(polg))) = (L())f()), for all x E RR.. .g e G.
(2)
natural question ask is what the equivariance of means for the layers it is composed of. Do they all have to be equivariant? For a layer W3 to be G-equivariant we require that there is a representation Py-1 on the input and a representation Pj on the output of W, such that P3 (g))); = ()))-1(9) for all E G. Note again that the only representations that are specified for f to be equivariant are Po and PL. so that all the other Pj + GL()")) can be arbitrarily chosen. If there exists a choice of py"s that makes each layer in J equivariant (including the nonlinearities r), we call f layerwise equivariant. In general we could have that different representations act on the input and output of the nonlinearities a, but as we explain in Section 2.1 this cannot be the case for the nel--nonlin,,rity which will be our main focus. Hence we assume that the the same group representation acts on the input and output of as above.
The following simple example shows that equivariance of f does not imply layerwise equivariance. Example 2.1. Let G = S2 = hi be the permutation group on two indices, where is the identity permutation and h the transposition of two indices. Consider a two-layer network f IR22 - R.
fx) = W2 1e),
where W₁ = (1 0) and = = 0. I' is invariant to permutation of the two coordinates of (indeed,
is constant 0). Thus, if we select (h) = (1)) and (h) 1, then I is equivariant (note that a representation of S₂ is specified by giving an involutory p(h) as we always have p(i) = I). However, there is no choice of a representation P1 that makes W1 equivariant since that would require (h()) = Wi (h) = (0 1), which is impossible (note that P1 (h) is a scalar). The reader will however notice that we can define network f, with W (0 0), W₂ = 0 for which we have fx) f(x) and then f is layerwise equivariant when choosing P1 (h) 1.
In the example just given was easy to, given an equivariant f. find an equivalent net f which is layerwise equivariant. For very small 2-layer networks can prove that this will always be the case, see Proposition D.I
Example 2.1 might seem somewhat unnatural, but we note that the existence of "dead neurons" (also known as "dying ReLUs") with constant zero output is well known (26). Hence, such degeneracies could come into play and make the search for linear representations in trained nets more difficult. We will however ignore this complication in the experiments in the present work.
From an intuitive point of view, equivariance of neural network should mean that some sort of group action is present on the intermediate feature spaces as the network should not be able to "forget" about the equivariance in the middle of the net only to recover it at the end. In order to make this intuition more precise we switch to a more abstract formulation in Appendix D.I The main takeaway will be that it is indeed possible to define group actions modified forms of the intermediate feature spaces whenever the network is equivariant, but this will not be very practically useful as changes the feature spaces from vector spaces to arbitrary sets, making it impossible to define linear layers to/from these feature spaces.
An interesting question that was posed by Elesedy and Zaidi 12 is whether non-layerwise equivari- ant nets can ever perform better at equivariant tasks than layerwise equivariant nets. In Appendix C we give positive answer by demonstrating that when the size of the net is low, equivariance can hurt
'The main results in this section are generalized to networks with affine layers in Appendix E

---

performance. In particular we give the example of cooccurence of equivariant features in C.1 This is scenario in image classification, where important features always occur multiple orientations in every image. It is intuitive that in such a case, it suffices for the network to recognize feature in single orientation for it to be invariant on the given data, but network recognizing features only one orientation will not be layerwise equivariant.
2.1 From general representations to permutation representations
We will now first sidestep the issue of an equivariant network perhaps not being layerwise equivariant, by simply assuming that the network is layerwise equivariant with group representation acting on every feature space (this will to some degree be empirically justified in Section 4 Then we will present results on 2-layer networks, where we can in fact show that layerwise equivariance is implied by equivariance.
The choice of activation function determines which representations are at all possible. In this section lay out the details for the ReLU- beLU-nonlinearity. We will have use for the following lemma, which is essentially the well known property of ReLU being "positive homogeneous".
Lemma 2.2. |Godfrey et al. [18 Lemma 3. Table 1]] Let A and B be invertible matrices such that PD where P is a permutation matrix and D is
ReLU(Az) = ReLU(x) (x) for all x. Then A diagonal matrix with positive entries on the diagonal.
We provide an elementary proof in Appendix B. It immediately follows from Lemma 2.2 that if ReLU is Geeuivvariant with respect to representations Po and P1 on the input and output respectively, then (o(g) = (1(g) = P(g) (gg) for all g E G. Le.. the representations acting on the intermediate layerwise equivariant ReLU-network are scaled permutation representations. This holds also if we add bias terms to the layers in (I.
feature spaces
We note that Godfrey et al. 188 consider more nonlinearities than ReLU, and that their results other nonlinearities could similarly be used to infer what input and output group representations admissible for these other nonlincarities. Also, Wood and Shawe-Taylor [41] derive large generality what nonlinearities commute with which finite group representations, which is very related our discussion. However, apply the results from [T] would have first prove that the representations acting on the inputs and outputs of the nonlinearity are the same.
For two-layer networks with invertible weight matrices layerwise equivariance with a scaled permutation representation acting on the feature space on which ReLU is applied. The reader should note that the invertibility assumption is strong and rules out cases such as Example(211
show that equivariance implies
Proposition 2.3. Consider the case of two-layer network f IR" + R"
f(x) W2 Re
where ReLU applied point-wise. Assume that the matrices W1 E mm W2 E mmxm are non-singular. Then is G-equivariant with Pj - GL(R"") for j 0, the input and the output respectively if and only if
polg) WT P(g)D()WW, and pz(g) W2PP
where P(y) is a permutation and D(g) diagonal matrix with positive entries. Furthermore, the network I is layerwise equivariant with pa pı(g) = D(g).
Proof. Invertibility of W, and Wa means that f being equivariant w.r.t. Po. P2 equivalent to ReLU being equivariant w.r.t P1 (g) W and (g( W P2(g)),. The discussion after
Lemma 2.2 now shows that Pi (g) = (g) = D(g) and the proposition follows.
n
The set of group representations for which a two-layer -eett-nk can be G-equivariant is hence quite restricted. It is only representations that are similar (or conjugate) to scaled permutations that are feasible. In the appendix, we also discuss the case of two-layer networks that are G-invariant and show that they have to be layerwise equivariant with permutation representations in Proposition B.]
5

---

and P1 has no 2.2 Permutation representations in -aENs-group convolutional neural networks
As explained in Section 2.1 for a ReLU-network be layerwise equivariant, the representations must be scaled permutation representations. We will now review how such representations can be used to encode the horizontal flipping symmetry in CNNs. This is special case of group equivariant convolutional networks ECCNNs-whie were introduced by [1O]. An extensive reference is [38 The reader familiar with TO 3 can skip this section, here we will try to lay out in a condensed manner what the theory of horizontal flipping equívariant GCNNs looks like.
Note first that horizontally flipping an image corresponds to a certain permutation of the pixels. Let's denote the action of horizontally flipping by т. This is permutation representation of the abstract group S2. If a convolutional layer V only contains horizontally symmetric filters, then it is immediate that V (T))) = r( (z)). It is however possible to construct more general equivariant layers.
Let's assume that the representations acting the input and output of v split into the spatial permutation T and a permutation P of the channels. Po on the input and P1 on the output. Po and P need to have order maximum
define representations of S2. One kernel constraint [39 38), what form the convolution kernel a of Vi needs to have to be equivariant. The kernel constraint says that the convolution kernel & E R xoo needs satisfy
show, using the so-called
(vb) = PIOPT (= P1bPP))
where T acts on the spatial k x part of v/ P1 on the e₁ output channels and Po on the co input channels. The last equality holds since Po is of order 2. In short, permuting the channels of s with Po and Pi should be the same as horizontally flipping all filters. We see this in action in Figure] even for layers that are not explicitly constrained to satisfy (3) An intuitive explanation of why should hold for equivariant layers is that it guarantees that the same information will be captured by & on input z and flipped input 7 (Ir), since all horizontally flipped filters in (()) do exist in vb.
Channels that are fixed under a channel permutation are called invariant and channels that are permuted are called regular, as they are part of the regular representation of Sz. We point out three special cases-if Po = Pi- = 7, then 1/ has to be horizontally symmetric. If Po diagonal entries, then we get a lifting convolution and if Po and P1 both have no diagonal entries then we get a regular group convolution [1O]. In the following when referring to a regular GCNN, we mean the "most generally equivariant" case where P has no diagonal entries.
3 The permutation conjecture by Entezari et al. and its connection to GCNNs
Neural networks contain permutation symmetries in their weights, mcaning that given, c.g., a neural network of the form I with pointwise applied nonlinearity a, can arbitrarily permute the inputs and outputs of each layer
Wi + PiWi, W, P,W,Pp
WI W.PE
and obtain a functionally equivalent net. The reader should note the similarity to the kernel constraint (3). was conjectured by [ that given two nets of the same type trained on the same data, should be possible to permute the weights of one net to put it in the same boss-basin as the other net. Recently, this conjecture has gained quite strong empirical support 22].
When two nets are in the same loss-basin, they exhibit close to linear mode connectivity, meaning that the loss/accuracy barrier on the linear interpolation between the weights of the two nets will be close to zero. Let the weights of two nets be given by 0₁ and 62 respectively. In this paper we define the barrier of a performance metric Ç on the linear interpolation between the two nets by
(S(O.) + ((02)) - < (5 (0) + +2))
[(((0)) + ((02))
Here Ç will most commonly be the test accuracy. Previous works 15
B I 22 have defined the
barrier in slightly different ways. In particular they have not included the denominator which makes difficult to compare scores for models with varying performance. We will refer to 4 without the denominator the absolute barrier. Furthermore we evaluate the barrier only at single interpolation point-halfway between 0 and 02-as as compared to earlier work taking the maximum of barrier values when interpolating between 0, and oz. We justify this by the fact that the largest barrier value

---

Figure 3: For every CNN that is trained with hor- izontal flipping data augmentation, there is cor- responding equally likely CNN that was initial- ized with filters horizontally flipped and trained horizontally flipped images. This CNN is the same as the original but with flipped filters, also after training. According the permutation con- jecture [13. there should be a permutation of the channels of the flipped CNN that aligns it close to the original CNN. This implies that the CNN is close to a GCNN, Lighter blue means spot in the parameter landscape with higher accuracy,
Network
Flipped
initialization
mitialization
SGD
path
Flipped SGD path
Trained
filters
Trained filters from flipped initialization flipped trained filters
Filter
permutation
is practically almost always halfway between 01 and 02 (in fact 13 also only evaluate the halfway point in their experiments). The permutation conjecture can now be informally stated follows.
Conjecture 3.1 (Entezari et al. [3] Sec. 3.21). Most SGD solutions belong to a set S whose elements can be permuted in such way that there is no barrier on the linear interpolation between any two permuted elements in S.
Importantly, we note that when applied to CNNs, the conjecture should be interpreted in the sense that we only permute the channel dimensions of the convolution weights, not the spatial dimensions. We will now explain why applying Conjecture [3. to CNNs that are trained on invariant data distributions leads to a conjecture stating that most SGD solutions are close to being GCNNs. For simplicity, we
discuss the case where G = horizontal flips of the input image, but the argument works equally well for vertical flips or 90 degree rotations. The argument summarised in Figure 3
We will consider an image classification task and the common scenario where images do not change class when they are horizontally flipped, and where flipped version of an image equally likely as the original (in practice this is very frequently enforced using data augmentation). Let T be the horizontal flipping function on images. We can also apply 7 to feature maps in the CNN, whereby we mean flipping the spatial horizontal dimension of the feature maps.
Initialize one CNN + and copy the initialization to a second CNN Φ but flip all filters of & horizontally. If we let Fo, #2, be the samples drawn during SGD training of 6 then an equally likely drawing of samples for SGD training of à is r(2)), T Tx11) T (x22),... .... After training &b using SGD and & using the horizontally flipped version of the same SGD path, & will still be a copy of + where all filters are flipped horizontally f This means according to Conjecture 3.1 that & can likely be converted close to & by only permuting the channels in the network.
Let's consider the j' convolution kernel s IR 1xkxk of 4. There should exist permutations P3 and Pa-1 such that T( = P, where the permutations act on the channels of si and on the spatial & x part. But if we assume that the Pj 's are of order 2 so that they define representations of the horizontal flipping group, this is precisely the kernel constraint (3). which makes the CNN close to a GCNN! seems intuitive that the P 's are of order 2, i.e. that if a certain channel should be permuted to another to make the filters as close to each other possible then it should hold the other way around as well. However, degeneracies such as multiple filters being close to equal can in practice hinder this intuition. Nevertheless, we conclude with the following conjecture which in some sense is a weaker version of Conjecture 3.1 as we deduced it from that one.
Conjecture 3.2. Most SGD CNN-solutions image data with a distribution that is invariant to horizontal flips of the images will be close to GCNNs.
"Tiis not strictly true networks that contain stride-2 convolutions or pooling layers with padding, as they typically start at the left edge of the image and if the image has an even width won't 't reach to the right edge of the image, meaning that the operation is nonequivalent to performing it on horizontally flipped input. Still it will be approximately equivalent. We will discuss the implications of this defect in Section A.2.
7

---

Table 1: Types of VGG11 nets considered. All except "wo aug" are trained with horizontal flipping augmentation.
Name
Description
CNN
Ordinary VGGII trained using cross-entropy loss (C.-E.). 9.23M parameters.
CNN w/o aug
Ordinary VGG11 trained without horizontal flipping augmentation.
CNN + inv-loss
Ordinary VGG11 trained using C.-E. and invariance loss (inv-loss).
CNN + late inv-loss
Ordinary VGG1I trained using C.-E. and inv-loss after 20% of the epochs. A regular horizontal flipping GCNN trained using C.-E. 4.61M parameters. partial horizontal flipping GCNN trained using C.-E. The first two conv-layers are G-conv-layers and the rest are ordinary conv-layers. 9.19M parameters.
GCNN
PGCNN
PGCNN + late inv-loss
The PGCNN trained using C. E. and lnv-loss after 20% of the epochs.
A measure for closeness to being a GCNN. To measure how close a CNN 4 is to being a GCNN we can calculate the barrier, as defined in 4. of the linear interpolation between & and permutation of the flipped version of o. We call this barrier for the test accuracy the GCNN barrier.
4 Experiments
The aim of the experimental section to investigate two related questions.
(QI) If a CNN is trained to be invariant to horizontal flips of input images, is it GCNN? This question is related the theoretical investigation of layerwise equivariance in Section 2
(Q2) If CNN is trained on horizontal flipping-ivvariin invariant data distribution, will it be close to GCNN? This is Conjecture 3.2
To answer these two questions evaluate the GCNN barrier for CNNs trained CIFARIO and ImageNet. We look at CNNs trained with horizontal flipping data augmentation to answer (Q2) and
CNNs trained with invariance loss on the logits to answer [11] In fact, for all training recipes horizontal flipping data augmentation is used. The invariance loss applied during training of a CNN
given by (x) & (r())) where horizontally flips r. It is added the standard cross-entropy classification loss. To evaluate the invariance of a CNN & we will calculate the relative invariance error (xx ((((x))) (1)))) (x)) |I + 0 (7(())))), averaging over a subset of the training data. Another way to obtain invariance to horizontal flips is to use some sort of self-supervised learning approach. We will investigate selervupe learning of ResNets in Section 4.2
To align networks we will use activation matching 25 I.. In activation matching, the similarity between channels in feature maps of the same layer in two different networks is measured over the training data and a permutation that aligns the channels as well as possible between the networks according to this similarity is found. Furthermore, consists of-after averaging the weights of two networks-reeeighhiing each channel in the obtained network to have the average batch statistics of the original networks. This reweighting can be merged into the weights of the network so that the original network structure is preserved. REPAIR is a method compensate for the fact that when two filters are not perfect copies of each other, the variance of the output of their average will be lower than the variance of the output of the original filters. We will also report results without REPAIR.
will use the REPAIR method by (22,, which
Experimental details can be found in Appendix A. We provide code for merging networks with their flipped selfs at /attts://githute com georg- bn, www.///ewwseeeeeeeeeeeeeeeee wew.sssiieeeviiiaaaneaee
4.1 VGG11 on CIFAR10
We train VGG11 nets 3K on CIFAR10
23
We will consider couple of different versions of
trained VGG11 nets, they are listed in Table
We train 24 VGG11 nets for each model type and discard crashed runs and degenerate runs to obtain 18-24 good quality nets of each model type. In Figure I we visualize the filters of the first layer in two
By degenerate run we mean net with accuracy 10%. There was one GCNN and one CNN + late inv-loss for which this happened.
8

---

Table 2: Statistics for VGG1 nets trained on CIFARIO.
Name
Accuracy
Invariance Error
GCNN Barrier
CNN
0.901 + 2.1 100
0.282 + 1.8 10
1.00-10 -.00-10- 4.9-10
CNN w/ aug
0.879 + 1.8 10
0.410 = 4.0 10
4.98-10-46.1
CNN inv-loss
2.3-10
0.0628 = 6.7 10-3
1.90-10"28.9-
CNN + late inv-loss
0.902 1: 20
0.126
1.6. 10-2
1.33-10-2
GCNN
0.899 2.5
103--0
1.5. 10
1.
PGCNN
0.902 3.7-
0.274
10-2
3.09-10-2
PGCNN + late inv-loss
0 10
0.124 10-
7.56-10 -
Table 3: Results for ResNet50's trained using various methods on ImageNet. The model with an
asterisk is trained by us. The last five are self-supervised methods. Flip-Aceuracy is the accuracy of net with all conv-filters horizontally flipped. Halfway Accuracy is the accuracy of net that has merged the weights of the original net and the net of the net with flipped filters-afe after permuting the second net to align it to the first. For the accuracy and harrier values w/o REPAIR we still reset batch norm statistics after merging by running over a subset of the training data.
Hallway
GCNN
Invariance Error
Flip- Accuracy
Accuracy,
Hallway
Barrier,
GCNN
Truining Method
Accuracy
REPAIR
Accuracy
no REPAIR
Barrier
Torchvision new
0.284
0.803
0.803
0.731
0.759
0.0893 0.0467
0.0545
Torchvision old
0.228
0.761
0.746
0.718
0.725
0.0384
"Torchvision old
inv-loss
0.0695
0.754
0.745
0.745
0.745
0.00636
0.0054
BYOL
0.292
0.704
0.712
0.573
0.624
0.19
0.118
DINO
0.169
0.753
0.744
0.611
0.624
0.184
0.166
Moco-v3
0.16
0.746
0.735
0.64
0.681
0.136
0.0805
Simsiam
0.174
0.683
0.667
0.505
0.593
0.252
0.121
VGG11 nets. More such visualizations are presented in Appendix A We summarise the statistics of the trained nets in @ The most interesting findings are that the models with low invariance error also have low GCNN barrier, indicating that invariant models are layerwise equivariant. Note that the invariance error and GCNN barrier should both be zero for a GCNN. In order to have something compare the numbers in Table to. provide barrier levels for merging two different nets in Table in the appendix. Of note is that merging a model with a flipped version of itself is consistently easier than merging two separate models. In Figure 4 in the appendix, we show the distribution of permutation order for channels in different layers. The matching method sometimes fails to correctly match GCNN channels (for which we know ground truth matches), but overall it does a good job. In general, the unconstrained nets seem to learn a mixture between invariant and regular GCNN channels. Preliminary attempts with training GCNN-VGG1 1s with such a mixture of channels did however not outperform regular GCNNs.
4.2 ResNet50 on ImageNet
We consider a couple of different training recipes. First of all two supervised methods-the the latest Torchvision recipe (35 and the old Torchvision recipe [36] which is computationally cheaper. Second, four self-supervised methods: BYOL [9. DINO 77. Moco-v3 and Simsiam [8]. The results are summarised in Table 3 There are a couple of very interesting takeaways. First, the GCNN barrier for the supervised methods is unexpectedly low. When merging two separate ResNet50's trained on ImageNet, (2 report an absolute barrier of 20 percentage points, whereas we here see barriers of 4.4 percentage points for the new recipe and 2.9 percentage points for the old recipe. This indicates that it easier to merge net with flipped version of itself than with different net. However, we also observe that for the self-supervised methods the barrier is quite high (although still lower than 20 percentage points). This is curious since they are in some sense trained to be invariant to aggressive data augmenation-iinchuee
Next we look at the GCNN barrier for ResNet50 (1 trained on ImageNet 11
horizontal flips. Finally we note that when training with an invariance loss. the GCNN barrier vanishes, meaning that the obtained invariant net is close to being a GCNN.

---

The reader may have noticed that the "Flip-Accuracy" i.e. accuracy of nets with flipped filters, is markedly different than the accuracy for most nets in Table This is a defect stemming from using image size 224. We explain this and present few results with image size 225 in Appendix A.2.1
5 Conclusions and future work
We studied layerwise equivariance of ReL.Unnetworss theoretically and experimentally. Theoretically we found both positive and negative results in Section 2 showing that layerwise equivariance is in some cases guaranteed given equivariance but in general not. In Section [] explained how Entezari et al. permutation conjecture 3.1] be applied to a single CNN and the same CNN with horizontally flipped filters. From this we extrapolated Conjecture stating that SGD CNN solutions are likely to be close to being GCNNs, also leading us to propose new measure for how close a CNN is to being a GCNN for the fact that ReLU-network that has been trained to be equivariant will be layerwise equivariant. We also found that is easier to merge a ResNet with flipped version of itself, compared to merging it with another net. Thus, Conjecture 3.2 might be worthwhile stepping stone for the community investigating Conjecture 3.1 addition to being interesting in and of itself. If a negative GCNN barrier is achievable, it would imply that we can do "weight space test time data augmentation" analogously to how merging two separate nets can enable weight space ensembling.
GCNN barrier. In Section we saw quite strong empirical evidence
Acknowledgements
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at the Chalmers Centre for Computational Science and Engineering (C3SE) partially funded by the Swedish Research Council through grant agreement no. 2022-06725 and by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
References
[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023. URL
http://penrriee.eee//oooooo met/forumm wodoooowmmiiooooi -id= Cdd=C@mmMM1155
[2] Erik J. Bekkers, Maxime W. Lafarge, Mitko Veta, Koen A. J. Eppenhof, Josien W. Pluim, and Remco Duits. Roto- Translation Covariant Convolutional Networks for Medical Image Analysis. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-Lóper and Gabor Fichtinger, editors, Medical Image Computing and Computer Assisted Intervention MICCAI 2018, Lecture Notes in Computer Science, pages 440-448, Cham, 2018. Springer International Publishing. ISBN 978-3-030-00928-11 1. doi: 10.
1_500.
[3] Georg Bökman and Fredrik Kahl. A case for using rotation invariant features in state of the art feature matchers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2022.
[4] Georg Bökman, Fredrik Kahl, and Axel Flinth. 77-net: A universal rotation equivariant architecture for 2d point clouds. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10976-10985, 2022.
[5] Robert-Jan Bruintjes. Tomasz Motyka, and Jan van Gemert. What affects learned equivariance in deep image recognition models? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pages
June 2023.
[6] Lucas Brynte, Georg Bökman, Axel Flinth, and Fredrik Kahl. Rigidity preserving image transformations and equivariance in perspective. In Scandinavian Conference on Image Analysis, 2023.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference Computer Vision (ICCV), 2021.
10

---

[8] Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In 2027 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 15745-
15753, June 2021. doi: 10.1 10.1109/CVPR44437222211 01554..
[9] Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Suterviperi Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[10] Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2990-2999, New York, New York, USA. 20-22 Jun 2016. PMLR. URL BtLLhhtps://ppooee https://prroceedings.. mmr cTess/v48//ohenc16. html
[11] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
[12] Bryn Elesedy and Sheheryar Zaidi. Provably strict generalisation benefit for equivariant models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2959 2969. PMLR, 2021. URL httrps://proceedinggs mlr wwtppppwwse/eeeeeeeeeeee elesedy21a html
[13] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022. URL http::///oppeeeeeeeeeetetttrrrro
gytemkL.
[14] Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International Conference on Machine Learning, 2021.
[15] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 3259-3269, PMLR, 13-18 Jul 2020. URL
https: h/tppe////roceeingg. mlr pptss//119/frankleert.. html
[16] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Transfrraosss::e::: 3D Roto- Translation Equivariant Attention Networks. In Advances in Neural Information Processing Systems, volume 33, pages 1970-1981. Curran Associates, Inc., 2020.
[17] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. URL wtRt.httts://// /tttap:///perreviiee net
[18] Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symme- tries of deep learning models and their internal representations. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 1 11893-1 11905. Curran Associates, Inc., 2022. URL httphttts//prcceedingg neurips wtwp/pape_fiiiie ffl_fiieesss www.ffeeesppppee wttpee///aap////22//1122 4dd33:10a022aaliid333333aaa org.aaa0eee..0o----aaeooooooeeee pdf
[19] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap Your Own Latent- A New Approach to Self- Self-Supervieed Learning. In Advances in Neural Information Processing Systems, volume 33, pages 21271-21222 1212844. Curran Associates, Inc., 2020.
[20] Nate Gruver, Marc Anton Finzi, Micah Goldblum, and Andrew Gordon Wilson. The lie deriva- tive for measuring learned equivariance. The Eleventh International Conference on Learning Representations, 2023. URL htttps://openrevier:
11

---

[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), doi: 10.1 N..109//VVP.2011..09
pages 770-778, 2016.
[22] Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. The Eleventh International Conference on Learning Represenlations, 2023. URL htt.t:ttpp///.. http:///peeeeeee.eeeeeeee net/forum?
id=gU6sJ62egex
[23] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images, 2009.
[24] Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991-999, 2015.
[25] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning:
Do different neural networks learn the same representations? In Dmitry Storcheus, Afshin Rostamizadch, and Sanjiv Kumar, editors, Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges NIPS 2015, volume 44 of Proceedings of Machine Learning Research, pages 196-212, Montreal, Canada, 11 Dec 2015. PMLR. URL .ml.gggsm...............g..mrrrr mlr prwssp http://44411111ooerrrrree html
https://prooeeedings
[26] Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis. Dying relu and initialization: Theory and numerical examples. Communications in Computational Physics, 28(5):1671-1700 2020. ISSN 1991-7120. doi: https://doi.org/1...223 12200/cicp.OA-20000
[27] Haggai Maron, Heli Ben- Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference on Learning Representations, 2019.
[28] Mirgahney Mohamed, Gabriele Cesa, Taco S. Cohen, and Max Welling. data and compute efficient design for limited- deep learning, 2020.
[29] Chris Olah, Nick Cammarata, Chelsea Voss, Ludwig Schubert, and Gabriel Goh. Naturally occurring equivariance in neural networks. Distill, 2020. doi: 10.23915/distill.000 .004. hitp://lill..ll222222eeeceeeeeee
[30] David W. Romero, Erik J. Bekkers, Jakub M. Tomezak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, ICML'20 20. JMLR. org, 2020.
[31] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robed Müller. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, Fer- gus, S. Vishwanathan, and R. Gamett, editors, Advances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc., 2017. URL URL/tttss//ppro/eeeineeee newriiinnnnnnn...s....... nwwrine..i.... cc/ /tt///01///////
pdf [32] Mehran Shakerinava, Arnb Kumar Mondal, and Siamak Ravanbakhsh. Structuring representa- tions using group invariants. In Advances in Neural Information Processing Systems, volume 35, pages 34162-34174, Curran Associates, Inc., 2022.
paper_files files/ee///e
[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[34] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-eeuiivrrian neural networks for 3D point clouds. arXiv: 0821 [cs], May 2018. URL org/ absaaaas//11 08219
[35] Torchvision. Improve the accuracy of Classification models by using SOTA recipes and primitives Issue #3995 pytorch/vision. 3995 2023. (accessed 2023-05-16).
[36] Torchvision, Vicconieferrnns/clasiiffattt at main pytorch/vision. httpps://gite com/
pytorch/vision, 2023. (accessed 2023-05-16).
12

---

[37] Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith Scalars are universal: Equivariant machine learning, structured like classical physics. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https: httpp:://ppenreveer net//forum2id-ba27-Raaa
[38] Maurice Weiler and Gabriele Cesa. General equivvaura steerable cnns. In H. Wallach, H. Larochelle, A. Beygelzimer, F. T'lché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https:://proceedinngs http://.....iia...app/////////// 44666777766072222276661111116622
pdf
[39] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[40] Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning Steerable Filters for Rotation Equivariant CNNs. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 149-858, Salt Lake City, UT, June 2018. IEFE ISBN 978-1-5386-6420-9 doi: 10.1 10.1109/CPP..011 8.00095.
[41] Jeffrey Wood and John Shawe- Shawe-Taylor, Representation theory and invariant neural networks. Discrete Applied Mathematics, 69(1-2):33 60, August 1996. ISSN 0166218X. doi: 10/c3qmr6. URL
elsevier http:///e........aaaaa/////aaa00
[42] Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic Networks: Deep Translation and Rotation Equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5028-5037, 2017.
[43] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep Sets. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
13

---

Figure 4: Number of channels of different permutation order that are found by activation matching in different VGG11 versions. In the ideal case, for GCNNs all channels should have order < 2. The matching method fails this, but not extremely badly. The main reason for failure that we have observed is that layers contain many zero-filters.
A Experimental details and more results
A.1 VGG11 CIFARIO
We present the distribution of various channels types in Figure We present statistics for merging two different nets in Table
In Figures 5 6 we show the convolution kernels of the second convolution layers of the same nets as
in Figure | Figures 7 8 show the filters of a GCNN-VGGII, which has hardcoded equivariance in each layer.
There are some interesting takeaways from these filter visualizations. First of all, the number of zero-filters in Figure 6 second layer of an ordinary VGG11 and Figure 8 second layer of GCNN-VGG11 - is quite high and it is lower for the VGGII trained using invariacce loss - Figure 5 Also, the GCNN has why these zero-filters appear and if there is some training method that doesn't 't lead to such apparent under-utilization of the network parameters.
zero-filters than the ordinary net. It would be valuable to investigate
Second, naturally when enforcing maximally equivariant layers using a lifting convolution in the first layer (recall Section 2.2). the network is forced to learn duplicated horizontal filters see Figure 7 e.g. the all green filter. This could explain the worse performance of GCNN:s relative to ordinary VGGII: iss. The choice of representations to use in each layer of GCNN remains a difficult problem, and as we mentioned in the main text, in our initial experiments it did not seem to work to set the representations similar to those observed in Figure 4 Further research representations would be very valuable. However, as discussed in Section C.T. it may be the case that no layerwise equivariant net outperforms nets which not restricted to be equivariant in late layers.
the optimal choice of
A.1.1 Training details
The nets were trained on NVIDIA T4 GPUs on computing cluster. We train 4 nets in parallel on each GPU, each run taking between 1 and hours, depending on net type. This yields less than (24/4) 6 2 = 72 compute hours in total.
14

---

Figure 5: Illustration of how a VGGII encodes the horizontal flipping symmetry. The 128 filters in the second convolutional layer of VGGIl-net trained CIFARIO are shown, where each filter is next to a filter in permuted version of the horizontally flipeed convolution kernel. Both the input and output channels of the flipped convolution kernel permuted, as described in Section The 64 input channels are projected by random projection matrix to 3 dimensions for visualization as RGB. The net is trained with an invariance loss to output the same logits for horizontally flipped images. It has learnt very close to a GCNN structure where the horizontally flipped filters can be permuted close to the original.
Figure 6: Illustaation of how VGGI1 encodes the horizontal flipping symmetry. The 128 filters in the second convolutional layer of a VGG11-nee trained on CIFAR10 are shown, where each filter is next to a filter permuted version of the horizontally flipped convolution kernel. Both the input
and output channels of the flipped convolution kernel are permuted, as described in Section 3 The 64 input channels are projected by random projection matrix to 3 dimensions for visualization RGB. The net is trained with horizontal flipping data augmentation. This net is quite close to GCNN structure, but the permuted filters are less close to each other than in Figure 5
15

---

Table 4: Statistics for merging VGG11 nets that where trained in different manners.
Model 1
Model 2
Barrier
CNN
CNN
5.08 10 2 5.7 10 3
CNN
CNN inv-loss
4.87 10 = 4.5- 10
CNN
GCNN
6.50-10 1.9
CNN inv-loss
CNN inv-loss
CNN + inv-loss
GCNN
7.53 10.10022 9.6
GCNN
GCNN
6.59 10- 7.7 10
Figure 7: Illustration of how a GCNN-VGGII encodes the horizontal flipping symmetry. The 64 filters in the first convolutional layer of GCNN-VGG1 1-net trained CIFAR 10 are shown, where each filter is next filter in permuted version of the horizontally flipped convolution kernel.
Figure 8: Illustration how a GCNN-VGGI encodes the horizontal flipping symmetry. The 128 filters in the second convolutional layer of a GCNN-VGG1 1-net trained on CIFAR10 10 are shown, where each filter is next to a filter in a permuted version the horizontally flipped convolution kernel. Both the input and output channels of the flipped convolution kernel are permuted, described in Section 3 The 64 input channels are projected by a random projection matrix to 3 dimensions for visualization as RGB.
16

---

Original weights
Fllpped weights pormuted original weights
Figure 9: Left: Visualization of the filters in the first layer of ResNet50. Right: The same filters but horizontally flipped and permuted align with the original filters to the right.
A.2 ResNet50 on ImageNet
We show the first layer filters of the torchvision new recipe ResNet50 in Figure 9
A.2.1 The peculiarities of image size 224
As mentioned in footnote 2 a convolution layer with stride 2 can not be equivariant to horizontal flipping if the imput has even width. This problem has been discussed previously by [2] 30) All the methods in Table excepp for the new Torchvision recipe use image size 224 for training and all use it for testing. It so happens that 224 be halved 5 times without remainder meaning that all feature spaces in ResNet50 will have even width. As luck would have it, if we use image size 225, then all feature spaces have odd width instead. Therefore we report results for few methods with image size 225 in T this table contains all numbers from Table 3 as well for convenience. We see that the simple change of image size lowers the invariance score and GCNN barrier. However does not seem to influence the accuracy much, meaning that the networks seem to learn to compensate for the non nonequuivariance with image size 224.
A.2.2 Training details
We train on NVIDIA A100 GPUs on a computing cluster. Each run was on either 4 or 8 parallel GPUs. For the supervised net, training took around between 9 and 13 hours depending on image size and whether invariance loss was applied. For the Simsiam training, the self-supervised pre-training took 23 hours and training the supervised linear head took 15 hours. total this gives an upper bound of 13 8 3 + (15 23) 8 = 616 GPU hours.
17

---

Table 5: Results for ResNet50's trained using various methods on ImageNet. The models with an asterisk are trained by us. The last five are self-supervised methods. Flip-Aceuracy is the accuracy of net with all conv-filters horizontally flipped. For the accuracy and barrier values w/o REPAIR we still reset batch norm statistics after merging by running over subset of the training data.
Halfway
Accu-
GCNN
racy
Halfway
Barrier
Training Method
Invariance Error
Flip-
w/o RE-
Accu-
w/o RE-
GCNN
Accuracy
Accuracy
PAIR
racy
PAIR
Barrier
Torchvision new
0.284
0.803
0.803
0.731
0.759
0.0893
0.0545
Torchvision old
0.228
0.761
0.746
0.718
0.725
0.0467
0.0384
TTrchvision old
+ inv-loss
0.0695
0.754
0.745
0.745
0.745
0.00636
0.0054
"Tochhvision old
+ img size 225
0.162
0.764
0.764
0.714
0.722
0.0657
0.0558
"Tochviision old
+ inv-loss img
9.4
6.72
size 225
0.00235
0.744
0.745
0.744
0.744
10-5
10
BYOL
0.292
0.704
0.712
0.573
0.624
0.19
0.118
DINO
0.169
0.753
0.744
0.611
0.624
0.184
0.166
Moco-v3
0.16
0.746
0.735
0.64
0.681
0.136
0.0805
Simsiam
0.174
0.683
0.667
0.505
0.593
0,252
0,121
"Simsiam img
size 225
0.113
0.68
0.68
0.535
0.611
0.213
0.103
B Layerwise equivariance of ReLU-networks proofs and further results
B.1 Proof of Lemma 2.2
Lemma 2.2. /Godfrey et al. 188 Lemma 3.1, Table 1]] Let A and B be invertible matrices such that ReLU(Ax) B ReLU(x)) for all Then A B PD where P is permutation matrix and D is diagonal matrix with positive entries on the diagonal.
We will use the following notation. For scalar a, can uniquely write it as a a where a 2 and a 2 0, and either a or a = 0. Similarly for matrices (elementwise), A = A - A
Proof. As the equation ReLU(Ax) Ax) = B ReLU(r) should hold for all x, we can start by setting x -Eş where Eş is the canonical basis. Then, as ReLU(-ex) = 0, we get that ReLU(-Aez, A and consequently A+ Inserting æ = eg into the equation, we can conclude that A = B A+
Hence, A
Now suppose that elements the some k * k'' Let's see what happens when we set x = A - Aigres . On the left hand side, the itth element simply becomes ReLU(A,,
non-zero, say Au 0 and A >> 0 on row for
A.:) 0 whereas on the right hand side we get Asre)) )== = A ReLU(A,
A ReLU( which is contradiction. Hence, we can conclude that each row has at most one positive element and since the matrices A and B are non-singular, they will have exactly one positive element in each row and we can conclude that they must be scaled permutation matrices.
Ag, that is, the i:th element is Au & O,
B.2 A result on layerwise equivariance given an invariant network
Next we are going to look at two-layer networks f Imm + IR which are invariant, that is, G-equivariant with P2(g) = Now, if (a) W2 ReLU(W,ar] one can without loss of general- ity assume that it is only the signs of the elements in W2 that matter, and move the magnitudes +1] ReLU(W Also, one can sort the indices such that -1 -1). This leads a convenient characterization of invariant two-layer
into W, and write (xx [1 W2 [[1 networks.
18

---

Proposition B.1. Consider the case of a two-layer network f R - R.
f(x)= = W2 ReL)(
where ReLU is applied point-wise. Assume that the matrix W1 E IR is non-singular and that W2 E Rmxi has the form [1 negative elements (m my + m. Further, assume p2(g) for the output, that is, the trivial representation. Then, f G-equivariant with po G -G..(T"")) if and only if
... -1] with m4 positive elements and m-
po(g) = WT P+(g) 0 |
P-(g) WT.
where P+(9) is m4 m4 permutation matrix and P- (g) Furthermore, the network f is layerwise equivariant with p1 (g)
m_ permutation matrix.
P+ 0
P-(g)
Proof. The condition for invariance is that the following equation should hold for all g G and all
x E Rmm
W2 ReLu(WWaa)) 1)) W2 ReLu (W)P))))x)
Now, let y Wir and A (i WT+ get that the following equivalent condition
W₂ ReLU(y) W2 ReLU(Ay)
should hold for all A A(g) Wi (WP In particular, it should hold for y = cx - Xeg for k 1 and À O. For this choice, the left hand side is equal to +1 and hence the right hand side should also be independent of X. The right hand side has m4 positive terms and m. negative terms,
A AA
Σ xmvi
(As )AAr)
One possibility is that one of the positive terms cancels out negative one. However, this would imply that A would contain two identical rows, making A singular, which is not feasible. Now, if Age < 0. then for sufficiently large X, would follow that (Ai ) = As XAa which makes the right hand side dependent X, contradiction. Hence, all the elements of A must be positive. Now, both Ask > and A > 0, then for sufficiently small X, we get (Ai >Aax)++ = Ai AAi again contradiction. Therefore, most element be positive each row. By also checking y = Cg. one can draw the conclusion that A must be permutation matrix of the required form and the result follows.
C Answer to an open question by Elesedy and Zaidi
question which was proposed by Elesedy and Zaidi 12 Sec. 7.4] is whether non-layerwise equivariant nets can ever be better at equivariant tasks than layerwise equivariant nets. We saw in Example [2. that they can be equally good, but not whether they can be better.
We formulate the question as follows.
(C.Q) Given a G-invariant data distribution μr, and ground truth equivariant function s m m can we ever have for neural networks of the form (T) that
inf W, E5jER-11800
Exe (f (x) -(x)]l <
inf
Exsp Be s(>)]l?
w, ER 1xmm,, W, equivariant
Here we assume that the layer dimensions my fixed and equal on both hand sides. We write f(W,) to make the parameterisation f terms of the linear layers explicit.
We will now give simple example showing that the answer to the question yes. Example C.I. Let y be a distribution on that symmetric about the origin and let R. - be given by s(x) [z). is invariant to changing the sign of the input, i.c., equivariant to S₂ {i,
19

---

Figure 10: Illustration of what we term cooccurence of equiv- ariant features. We imagine training a network to classify the two depicted classes. Both classes consist of images containing a shape and horizontally flipped copies of the same shape - in particular, every image contains the shape in both orientations. On this data, a network only recognising the existence of the shape < (in this orientation) will be invariant to horizontal flips of the input. However, such a network is not invariant to horizontal flips of general data and can hence not be written layerwise equivariant network.
Class
Class 2
with representation choice Pa (h) -1 on the input and (2(h) 1 on the output. Now, we consider approximating s with neural network ft:
f(x) W2 Re)u(
where W1 and W2 are both scalars. Recall (from Section 2.1 that for ReLU to be equivariant, the representation acting the output of Wi has to be a scaled permutation representation. The only one-dimensional permutation representation is the trivial representation, but this means that W1 = and so f is constant 0. Also consider the non-equivariant net f with W1 = 1, W2 = 1. Since f(x) = (xx for æ S and f(x) = a(x) for x > 0, it is clear that
E E f(z) s(x) - E f(())) s(x))..
2
showing that in this example, non-equivariant nets outperform the layerwise equivariant f.
The reason that non-equivariant nets outperform equivariant ones in Example C. is that the ca- pacity of the net so low that requiring equivariance leads to a degenerate net. we increase the intermediate feature dimension from to 2, then equivariant nets can perfectly fit s(x) | (by
taking W1 = (1 -1)². W₂ = (1 1)). In the next section we take this idea of insufficient capacity for equivariance step further and give less rigorous but more practically relevant example of scenario where non-equivariant nets outperform equivariant nets on an equivariant task. The idea is that when features cooccur with group transformed versions of themselves, it can suffice for network to recognize feature in a single orientation. The network can then still be equivariant on data which has such cooccurring features, but it can be smaller than layerwise equivariant network which would be forced to be able to recognize features in all group transformed orientations.
C.1 Cooccurence of equivariant features
Here give a more practical example to answer the question (C.Q) by Elesedy and Zaidi 12
Example C.2. This example is illustrated in Figure IO We consider image classification using a CNN. For simplicity we will consider invariance horizontal flipping of the images, but the argument works equally well for other transformations. For the sake of argument, assume that our CNN has very limited capacity and can only pick up the existence of a single local shape in the input image. The CNN might consist of single convolution layer with one filter, followed by ReLU, and spatial average pooling. As shown in Figure 10 if the identifiable features in the images always cooccur in an image with horizontally flipped versions of themselves, then it suffices to recognise feature in single orientation for the feature detector be invariant to horizontal flips on the data. We refer to this type of data as having cooccurence of equivariant features. Since a layerwise equivariant CNN with a single filter has have horizontally symmetric filter, it can not be as good as the non-equivariant CNN on this task.
The reader might consider Example to be somewhat artificial. However, we note that it has been demonstrated [77 that CNN image classifiers often look at mostly textures to identify classes. Textures such as the skin of an elephant will often occur in multiple orientations in any given image of an elephant. A network might hence make better use of its limited capacity by being able to identify multiple textures than by being able to identify single texture in multiple orientations.
20

---

D Rewriting networks to be layerwise equivariant
The aim of this section is to investigate when we can, given an equivariant net, rewrite the net to be layerwise equivariant. Le., given an equivariant find a functionally equivalent layerwise equivariant net with the same layer dimensions as the original. We first give a result for small two-layer networks showing that for them in Section D.I Unfortunately the conclusion there will not be as clear. While we can view an equivariant network whether we can rewrite it as a layerwise network with the same layer structure as the original.
always do this. Then we look at the problem more generally
composition of equivariant functions in certain sense, it remains unclear
Proposition D.1. Consider the case of two-layer networks
f(z) = W₂ 1e),
where ReLU applied point-wise, the input dimension is 2, the output dimension 1 and the intermedi- ate dimension that W1 IR2x2, W2 E g Any such f that is invariant permutations of the two input dimensions must be equivalent to a layerwise equivariant network with the same layer dimensions as f.
Proof. Note that non- oon-negatvve constants factor through ReLU so that
W2 (
( W1x
3 ReLU(W11)) W2 ReLU
for any a 2 0, 3 2 0. Hence we can assume that W22 only contains elements 1, -1 by factoring the non-negative constants into Wi. So (WLOG) will only have consider W2 = =1).
Let
(1 0
!
J=
be the permutation matrix that acts the input. It will also be convenient to introduce the notation f2(x) W2 ReLU(x) for the last part of the network so that we can write the invariance condition f2(1x))
Var E R f(Jx) f(z) as Vx E IR? fz(WiIz)
We first consider the case when W, has full rank. In this case Vo E RR f2(WiJa) f2(1) is equivalent to
Vx E fe(WaJWT f2(z).
(5)
For convenience set
WIJW A
which satisfies A = I (A is involution
f
It is easy show that this means that either
= a
-a)
where
a² bc 1,
(6)
or A +.. However, the = + case easy to rule out since implies
# WiJ,
meaning that the columns of W, are linearly dependent, but we assumed full rank. Explicitly writing out (5) with WWJW-1 A, we get that for all I22 we must have
We ReRUee W2 ReLU
Now we divide into two cases depending on the form of Wz. The strategy will be to plug in the columns of A as z and use the fact that A I.
'erre we note that since (2) = 1, the map p(J) wiwww1 actually defines a representation of S2 on R² We are hence trying find for which representations p S2. f2 is invariant.
21

---

Case I: W2 (1 -1). By plugging = (a c) into we see that
= Re)U(a)) ReLU c),
meaning that 1. Similarly, plugging in (b -a) see that
-1 = ReLU(() (eeU((a))
implying that a < -1. We have reached a contradiction and can conclude Wa * (1 -1).
Case II: W2 = (1 1). This time we will need four plug-and-playy with 7
x (-a -c)² yields
ReLU(-a) + ReLU(-())
so 2 0 and c 2 O.
x (-b a) T
yields
0 == ReLU(-b) + ReLU(a),
so a < 0 and b 2 0. From this and the previous we conclude a = 0.
x = (a c) yields
1 ReLU(a) + ReLU(c),
which implies e & 1.
4. (b -a) T yields
1= ReLU(b) ReLU(-a))
which implies b < 1.
From 0 and (b, c) 1] we conclude from (6) that b 1. Thus, if W22 = (1 1). then = J. which means that
WiJW =
But this is precisely the equivariance condition on W,, and since W2 is invariant we have layer-wise equivariant network.
Next, we consider the case when W1 has rank 1. We can then write Wi uv for some vectors in 2 v contains two equal values, then W1 is row-wise constant, so permutation invariant, and thus the network is layer-wise equivariant. If contains zero, then row of W1 is zero and the statement to prove reduces to the case of a network with W1 x2, W2 {-1,1} } in which case is easy to show that W2 has to be permutation invariant for the network to be invariant.
Finally if contains two different values and does not contain zero, let WLOG 81 # 0 and consider the following two cases for the invariance equation below
W2 ReLU(uv z) W2 ReLU(ux Jr).
Case I: u contains and find that uofxx x = (#) 12) 2u, while ur Jr - (vi 12 Pu. One of those mapped to O by ReLU and the other one isn't, implying that W₂ has map it to 0, W2 (1 - 1) and the two values in are the same. This leads to a zero network which can be equivalently written layer-wise equivariant.
values with the same sign. We plug in x 01 U2, 22 v/2 01
Case II: wn and u2 have different signs. WLOG let w1 0. Plugging in the same values as above, we find that W, = (1 1) and u2 = -1 Next, we plug in x1 sign(r1), ). D2 = 0 yielding uuTr- x = |o lu and ub T Ja = (i Since 1/2 * 0, we must now have that U2 Ui However, u2 u and U2 - in fact yields an equivariant
W1
V1
M1 V1
01 U1
W1
and since W2 is invariant, we have a layer-wise equivariant network.
22

---

D.1 Equivariant composition of functions
In this section we switch to a more abstract formulation of layerwise equivariance see what it means for individual functions & and vú if their composition vá & is equivariant. For related discussion of equivariance under general group actions we refer the reader [32].
Definition D.2. Given group G and set X, group action of G on X is function
G X - X
such that for the identity element of G and any x E X we have
ali, x) = æ.
(8)
and for any two g, and any X we have
a)hg,x) ((.
(9)
First of all is practical to recall that invertible functions transfer group actions from domain to codomain and vice versa.
Lemma D.3. Given group G and an invertible function 6 X YY
1. If there is a group action a on X, then we can define a group action ay on Y by oy(9. = ( (v)))
and is equivariant with respect this group action. Furthermore, this is the unique group action on Y that makes & equivariant.
2. Similarly, if there is a group action ay on Y, then we can define a group action ox on X by
a xx( = x (av ((x))
and is equivariant with respect this group action. Furthermore, this is the unique group action on X that makes & equivariant.
Proof. We prove 1., the proof of 2. is analogous. there is a group action ay on Y making equivariant, then we must have
ar (g. = ((ax(1,x)))
for x E X. So by change of variable ~ (1),
(y = fax (g. ( (t)))
for all X. This proves uniqueness. Finally, ay a group action as
(r = (x)) (v)
and
ay (hg. 3) 6
olox((,.xx (
cl((
ar
We can now state a proposition telling us that if a composite function is equivariant, then must be composite of equivariant functions.
Proposition D.4. Assume that are given a group G and two sets X and Z with group actions ox and azz Assume also that we are given a further set Y and two functions & X - Y and Y Z such that their composition to X + Z is G-equivariant w.r.t ax and az-
23

---

1. If ib is invertible we can define group action ay on Y such that & and ab are G-equivariant with respect to this action.
2. If vb is not invertible, introduce Y' /(X)/ where is the equivalence relation identifying elements of Y that map to the same element of Z, i.e,
y1 y2 > (yy11) (1y)).
Then we can define a group action ay on Y' such that & and èb are equivariant when seen as functions toffrom Y'.
Proof. Assume first that & is invertible. From Lemma D.3 we get a group action ay on Y defined by
aY (g. = x 1 (az(g, (b))))
Lemma D.3 also states that d is equivariant with this choice of ay Next we show that e is equivariant, which follows from the fact that s o @ equivariant:
or (g, ((x)) & (azla, 1b (()))
o(ax v).
If not invertible, then is anyway invertible as map v Y' e ((X). Note that the action is well defined even when restricting from Z to 1 o(X), because by the equivariance of th Φ oz can't move an element from o ((X) to Z \ a o (x)):
(2 sb ((x)) vb Φ(ax (). E
Hence the earlier argument for constructing ay works for constructing group action ay on Y such that the maps @ and sp (appropriately restricted) become equivariant.
IT
Looking back at Example 2.1 with é W,, s/ = W2 ReLU(.), we see that Y' that case would be single point space since W2 maps everything to O. The action ay would hence be trivial.
As mentioned before, the main problem with changing Y to Y' above is that if X, Y, Z are feature spaces in a neural network & o, where é and v themselves might be decomposable into multiple linear layers and activation functions, there no guarantee that Y' will be a vector space or that and vi can still be expressed as decompositions into linear layers and activation functions when changing from Y Y". Still, the intuition that the equivariance should be preserved layerwise holds in the interpretation provided by Proposition D.4
E Network layers with bias
In this section we generalize Proposition 2.3 to work for layers with bias, i.e. x H> Wx + for some bias vector f. The discussion up to Section 2.1 works with biases as well, does Lemma 2.2 (as it regards ReLU and not the layers themselves). For Proposition 2.3 it gets more complicated. What makes the proof of easy is that given an invertible linear layer {(x) = Wx, group representation p(g) on x is transferred to a group representation a(g, x) = {(p(g) (x)) = W W on E(x) w.r.t. which W is equivariant. If we consider affine layer {(x) Wz + with bias vector t, then we can play the same game, but p(g) is not transferred to a group representation (linear action) but instead to the affine action al x) W W r (I - W W
f(z). This means that we can not apply Lemma 2.2 We can however find a generalization as follows:
Lemma E.1. If A, B are
invertible matrices and a and b are n-vectors, such that ReLU Ar a) = (ReLU(x) + b for all x e IR, then a = =bb bb 0.
Proof. Inserting x
yields ReLU(a)
b. Inserting x A-la yields ReLU(2a)
BReLU a) b so that b BReLU(A" -1daaa)).. a). Inserting x = -- yields 0 = BReLU - A b so b = - TReLUU-AA110 that B-1 must be zero and hence so must b. Finally, inserting x. -2A--1 yielss ReLU (aa)
Combined we have that B-1 ReLU(A--a) = ReLU a
BReLU(-2A. -11a- a) + so that ReLU(-a) - 0 which combined with ReLU(a) = b = 0 gives
0.
24

---

This lemma shows that if ReLU commutes with affine actions, then the affine actions must in fact be linear and so Lemma 2.2 applies. This shows that Proposition h.33 with affine layers as well. We thank the anonymous reviewer who prompted this generalization.
25


Reviews:
[{"review_id": "rM4Kst6yzC", "reviewer": "Reviewer_k5hy", "summary": "This paper shows that CNNs will be close to G-CNNs if they are trained to be equivariant. In addition, they also provide some theoretical analysis and conjectures regarding the layerwise equivariant of ReLU-networks.", "questions": "1. What is 'ppt' defined in section 4.2? I did not find explanations regarding it.\n2. Could you give more explanations regarding the last sentence of the conclusion 'If a negative GCNN barrier is achievable ...... enable weight space ensembling.'? How 'negative GCNN barrier' is related to 'weight space test time data augmentation', etc?", "limitations": "Limitations are given.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. They show that equivariance implies layerwise equivariance with a scaled permutation representation acting on the feature maps (Proposition 2.3.).\n2. They propose a new conjecture 'Most SGD CNN-solutions on image data with a distribution that is invariant to horizontal flips of the images will be close to GCNNs.' (Conjecture 3.2.) They proposed a new measure for closeness to being a GCNN.\n3. Experiments on CIFAR-10 and ImageNet support Conjecture 3.2.", "weaknesses": "1. In general, the paper may have limited or vague benefits for applications. First, the work focuses on ReLu-networks in the theoretical analysis, ignoring normalization layers. Modern neural network designs heavily rely on different normalization layers. In addition, in the ImageNet experiments, ResNet-50 uses batch normalization, which is not aligned with the theoretical analysis. \n2. The paper did not explain what are the insights for network designs if ReLu CNNs are layerwise equivariance. In practice, it even hurts the performance of the model if the invariance loss is applied. Then, the generalization of the model might contradict the actual layerwise equivariance of the model. And the invariance loss adds an extra restriction for learning. \n3. The evidence of Q1 is much weaker than Q2 in section 4. The  G-CNN Barrier of 'CNN' in table 2 is 20$\\times$ larger than 'CNN + inv-loss.' I wonder if it is still valid to say it is close to G-CNN. Maybe a baseline without horizontal data augmentation can better justify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful review. We aim to clarify the writing in the paper in accordance with our replies to the reviewers comments below.\n\n> In general, the paper may have limited or vague benefits for applications.\n\nIt is true that there might not be immediate benefits, but this is the case for much theoretical work on neural networks. See also the clarification of \"weight space test time data augmentation\" further down.\n\n> First, the work focuses on ReLu-networks in the theoretical analysis, ignoring normalization layers. Modern neural network designs heavily rely on different normalization layers. In addition, in the ImageNet experiments, ResNet-50 uses batch normalization, which is not aligned with the theoretical analysis.\n\nBatch normalization is not included in the theory section, however as the reviewer notes the experiments suggest that layerwise equivariance is still compatible with batch normalization. Note that our result says that using ReLU implies that if we have layerwise equivariance then the representations acting on the features are permutation representations. Batch normalization is equivariant w.r.t. such representations if the batch statistics are computed jointly for channels in the same cycle in the cyclic decomposition of the representation [1]. The fact that we have positive experimental results indicates that statistics for such channels are approximately the same.\n\n> In practice, it even hurts the performance of the model if the invariance loss is applied. Then, the generalization of the model might contradict the actual layerwise equivariance of the model. And the invariance loss adds an extra restriction for learning. \n\nThe invariance loss is not added to boost performance, but rather to test the theory of whether an invariant CNN is layerwise equivariant. Surprisingly, the setting with invariance loss added after 20% of the training epochs did improve performance of simple VGG nets. This is a result that we have not seen in prior literature.\n\n> The evidence of Q1 is much weaker than Q2 in section 4. The G-CNN Barrier of 'CNN' in table 2 is 20 larger than 'CNN + inv-loss.' I wonder if it is still valid to say it is close to G-CNN.  Maybe a baseline without horizontal data augmentation can better justify this.\n\nThe closeness to a G-CNN should be interpreted in relation to Entezari's conjecture (3.1). Our experiments suggest that a CNN trained with data augmentation is closer to being a G-CNN than it is to another independently trained CNN.\nIt is indeed the case that it is not super close to being a G-CNN, but it also is not invariant to horizontal flips so this does not contradict our theory.\nIn any case, a baseline w/o horizontal flipping augmentation is a great suggestion and we have carried it out. Please refer to the global rebuttal.\n\n> What is 'ppt' defined in section 4.2? I did not find explanations regarding it.\n\nIt is an abbreviation for percentage point. We will write it out in full to avoid confusion.\n\n> Could you give more explanations regarding the last sentence of the conclusion 'If a negative GCNN barrier is achievable ...... enable weight space ensembling.'? How 'negative GCNN barrier' is related to 'weight space test time data augmentation', etc?\n\nOne of the motivations for studying the Entezari conjecture has been that if we can find permutations relating two networks, then we can average their weights so that we obtain a so called weight space ensemble of them. The goal being to obtain the benefits of ensembling while still working with a single network. If we can find permutations relating a network A to its horizontally flipped self B, then we can perform such weight space ensembling between A and B. Note that the output of B will be the same (up to border/stride effects) as what we would get if we insert a horizontally flipped image into A. Thus ensembling A and B is the same as doing test time augmentation with horizontal flipping, and weight space ensembling of A and B could be called weight space test time augmentation. If the GCNN barrier is negative it means that we get improved performance by doing this. Note that a negative barrier has not been obtained for the Entezari setting of two different networks yet and our results suggest that it should be easier to obtain for our setting of a network and its flipped self.\n\n\n\n[1] Weiler & Cesa, General E(2)-Equivariant Steerable CNNs, NeurIPS, 2019\n"}, {"review_id": "f52RddMA3q", "reviewer": "Reviewer_A92L", "summary": "Exploring the symmetries of representations and parameters in neural networks is crucial. This paper provides several valuable contributions. First, the authors theoretically proved that for Relu-Networks equivariance implies layer-wise equivariance, but not vice versa. Second, inspired from the conjecture by [22], the authors discussed a weaker version that connects CNNs with GCNNs. Finally, quantitative experiments were performed and showed that a ReLU-network that has been trained to be equivariant will be layer-wise equivariant.", "questions": "See the weakness part above. ", "limitations": "The authors have adequately discussed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper is well written. I enjoy the writing. While the concepts particularly those related to group representation are complicated and non-trivial, the authors did a great job in introducing the relevant definitions, examples and propositions in an elegant and compact way. The authors also provided necessary intuitive understanding, which greatly help readers to digest the motivation of how the investigation is performed. \n\n2.\tThe conclusion by Proposition 2.3 is interesting and valuable, even the proof is a direct result of Lemma 2.2 by [14]. I have checked [14] and find no similar conclusion. The authors also provided abundant theoretical understandings and derivations in the supplementary materials including the discussion of the invariant case. \n\n3.\tThe discussion of how GCNNs are related with the notion of Conjecture 3.1 [14] is insightful and inspiring. \n\n4.\tThe experimental evaluations are scrupulously carried out and are able to support the claims by the authors to a certain extent. \n", "weaknesses": "I have no major concern. There are still some questions below:\n\n1.\tProposition 2.3 shows that layer-wise equivariance is NOT a necessary condition of equivariance. But the experiments did show that CNN with equivariant training boils down to be layer-wise equivariant. So, why we have this observation and does it mean that the examples derived from Proposition 2.3 are just corner cases that rarely happen in practice? Moreover, how does Proposition 2.3 influence the design of GCNNs? Or particularly, what is the relationship between the equivariant constraint in Proposition 2.3 and the derived kernel constraint (Eq. 3) by [31,30]? I would expect the authors to explain more on these points?\n\n2.\tWould the authors explain why they have the claims in Lines 151-155 that equivariance can hurt the performance? Why in Table 2, the unconstrained models obtained better performance than those constrained or equivariant ones?\n\n3.\tTable 1 only defined the models trained with invariant losses, which indicates that the performance in Table 2 can only answer Q1? How about the experiments of models trained with horizontal flipping data for Q2? Is Table 3 mainly for Q2?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their good comments and questions which will help us improve the paper.\n\n> Proposition 2.3 shows that layer-wise equivariance is NOT a necessary condition of equivariance. But the experiments did show that CNN with equivariant training boils down to be layer-wise equivariant. So, why we have this observation and does it mean that the examples derived from Proposition 2.3 are just corner cases that rarely happen in practice?\n\nLet us clarify that Proposition 2.3 shows for simple networks that equivariance and layer-wise equivariance are equivalent. Example 2.1 however indeed shows that when the conditions in Proposition 2.3 (specifically, invertibility) do not hold, then a network may be equivariant without being layerwise equivariant. The fact that we don't observe the degeneracy from 2.1 in the experiments could be explained as follows. Note that if several input channels are set to zero by a weight matrix + ReLU, then their values do not matter for the network output. Thus even if they are not equivariant, we can permute them without changing the output of the network. So the permutation found by the weight matching procedure in the experiments might not be actually permuting these, later killed off, channels in an equivariant matter, meaning that the network is not strictly layerwise equivariant. But it is layerwise equivariant for the channels that matter.\n\n> Moreover, how does Proposition 2.3 influence the design of GCNNs? Or particularly, what is the relationship between the equivariant constraint in Proposition 2.3 and the derived kernel constraint (Eq. 3) by [31,30]? I would expect the authors to explain more on these points?\n\nThis is an excellent question. While the kernel constraint says given certain group representations what the possible linear layers are, our results instead concern given a certain activation function which group representations are possible. So the results are complementary.\n\n> Would the authors explain why they have the claims in Lines 151-155 that equivariance can hurt the performance? \n\nThis refers to the discussion in Appendix C, where we give specific examples. The main point is that an invariant/equivariant network must be equally good at detecting every pattern in all orientations. In practice given a limited network capacity, it may give better performance to be able to recognize more patterns in fewer orientations than fewer patterns in more orientations.\n\n> Why in Table 2, the unconstrained models obtained better performance than those constrained or equivariant ones?\n\nWe do not know. However a possibility is that it relates to the mentioned discussion in Appendix C, i.e. that the invariant/equivariant networks do not have enough capacity to learn all patterns necessary to classify the data well.\n\n> Table 1 only defined the models trained with invariant losses, which indicates that the performance in Table 2 can only answer Q1? How about the experiments of models trained with horizontal flipping data for Q2? Is Table 3 mainly for Q2?\n\nAll the networks in Table 1 in fact are trained with horizontal flipping data augmentation (see line 282). We will make this more explicit. We have now additionally performed experiments without flip augmentation, please refer to the global rebuttal."}, {"review_id": "vucSV1yXEC", "reviewer": "Reviewer_k3h9", "summary": "This paper provides an investigation on whether equivariance of a trained deep neural network with ReLU activations implies that each of its learned layers are equivariant. The authors show that this should be true in some sense, i.e., some kind of group action must be present in the intermediate feature spaces (Line 141-143 and Appendix D.1), as the network should be able to always encode the group transformation of the input in some way to achieve overall equivariance. However, this applies to intermediate feature spaces where the clamping behavior of (ReLU) activations are involved; the authors argue that, when considering the learned linear layers, this is not always true (Example 2.1). Regarding ReLU activation, the authors show that if ReLU is G equivariant, the representations on the input and output must be scaled permutation representations (Lemma 2.2). From that, for two-layer networks with a strong assumption of invertible weight matrices, the authors show that overall equivariance implies layerwise equivariance where representations on intermediate features are scaled permutation representations (Proposition 2.3). Based on that, the authors investigate how the (requirement for) scaled permutation representations provides an implication of how horizontal flipping symmetry (related to the representation of S2 group) can be encoded in CNNs. The key intuition here is that permutation representation exactly appears in the characterization of GCNNs (Cohen et al., 2016), where the kernel constraint gives that an element of S2 group acts on the group convolution kernel through the joint action of spatial permutation representation (horizontal flipping here) and channel permutation representation (allocation). The authors further make a connection to the invariance of neural networks under permutation of neurons, or channels in case of CNNs due to spatial structure of convolutions, and the permutation conjecture (Entezari et al., 2022) that networks of the same type trained on the same data would lie in the same loss basin up to some permutation of neurons (channels). Then, the authors logically combine the requirement for permutation representations on ReLU for G equivariance, the spatial and channel permutation representation on group convolution filters, and the permutation conjecture, leading to the following conjecture: CNNs trained on a data distribution invariant to horizontal flips would be close to being GCNNs. The authors empirically test their conjecture using VGG and ResNet architectures on CIFAR10 and ImageNet, which supports the proposed conjecture.\n\nCohen et al., Group Equivariant Convolutional Networks (2016)\n\nEntezari et al., The role of permutation invariance in linear mode connectivity of neural networks (2022)", "questions": "Q1. Reading Line 260-268, a question I had was to which extent the proposed conjecture depends on the particular nature of S2, i.e., how specifically it may generalize to other discrete groups such as 90-degree rotations. May I ask for an explanation on this?", "limitations": "The authors have clarified the limitation of the work in Section 1.2.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "S1. I think this is a solid work that establishes a creative combination of ideas and findings from multiple sub-areas of deep learning theory, and from that proposes some original theoretical findings that combines into a very interesting conjecture that is potentially impactful in the field of equivariant deep learning.\n\nS2. In addition to the above, the proposed conjecture is equipped with a proper empirical support involving multiple deep architectures and datasets, which is an important strength of this work.", "weaknesses": "W1. While reading the paper, I was quite confused about the implication of formulation in Appendix D.1. It seems the results, in particular Proposition D.4, explicitly proves that equivariance of a network implies layerwise equivariance, even under non-invertibility of individual layers. Because of this, I was confused when reading the discussion on Example 2.1 as well as Line 158 that overall equivariance might not lead to layerwise equivariance, as these seem like conflicting arguments. Am I missing something?\n\nW2. The discussion on identifying an equivalent equivariant network with layerwise equivariance (Line 132-136), while understandably leading to discussion in Appendix C, seems not critical in describing the main conjecture of the paper (please correct me if I am wrong). It might be better in terms of readability to contain the relevant discussion in some separated section.\n\nW3. In describing Eq. (3), it might be beneficial to explain how the transformation of filter relates to transformation of input and output (in context of equivariance) for readers not familiar with GCNN.\n\nW4. The description of why and how the REPAIR algorithm is used in Line 289-296 was hard to understand, I think it has room for improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and helpful comments.\n\n> While reading the paper, I was quite confused about the implication of formulation in Appendix D.1. It seems the results, in particular Proposition D.4, explicitly proves that equivariance of a network implies layerwise equivariance, even under non-invertibility of individual layers. Because of this, I was confused when reading the discussion on Example 2.1 as well as Line 158 that overall equivariance might not lead to layerwise equivariance, as these seem like conflicting arguments. Am I missing something?\n\nThis is an excellent questions and something we have struggled with ourselves. The way to interpret it is that as linear functions between vector spaces, the layers need not be equivariant. But if we redefine the domains/codomains of the layers then they become equivariant functions with respect to specific nonlinear group actions. In particular, the framing in Appendix D does not rule out group actions in the middle of the network which are nonlinear. In the main part of the paper we discuss the case of linear group actions (=representations) acting on all feature spaces.\n\n> W2, W3, W4\n\nWe thank the reviewer for these helpful suggestions.\n\n> Reading Line 260-268, a question I had was to which extent the proposed conjecture depends on the particular nature of S2, i.e., how specifically it may generalize to other discrete groups such as 90-degree rotations. May I ask for an explanation on this?\n\nIn the 90-degree rotation case, Entezari's conjecture plus the assumption that the data distribution is rotation invariant gives that when we rotate all filters of the initial CNN 90 degrees, there should be a permutation that aligns these rotated filters with the original ones. If this permutation is cyclic of order 4, then it is a permutation representation of the group of 90 degree rotations. Thus the CNN is a G-CNN w.r.t. this group.\n"}, {"review_id": "NhV4V0RU7P", "reviewer": "Reviewer_79cV", "summary": "This work investigates the relationship between end-to-end equivariance of a network and layerwise equivariance. It theoretically investigates when we can guarantee that an equivariant network is layerwise equivariant, and also cases where layerwise equivariant is not guaranteed or is harmful. In the case of CNNs, the authors draw a connection between horizontal flip invariance and the permutation conjecture for linear mode connectivity.", "questions": "What do you mean on Page 5 when you say that the results for Godfrey et al. 2022 for other nonlinearities \"also straightforwardly translate to the group equivariance case\"?\n\nMinor:\n* Page 4: \"and then $\\tilde f$ is equivariant\" should be \"... layerwise equivariant\"\n* Can you give a citation for the existence of dead neurons on page 4?\n* Page 6, several times you refer to the numerator of (4) when you mean the denominator.\n", "limitations": "Good discussion of limitations in Sectino 1.2.\n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Good exposition in Sections 1 and 2\n2. Appendices C and D have nice observations and are illustrative. I quite like the examples in C and C.1., which concretely show ways in which layerwise equivariance is not good enough when network capacity is low.\n3. Neat simpler proof of the lemma from Godfrey et al. 2022\n4. Nice connection drawn to the permutation conjecture that I would not have expected.\n5. The trained VGGs are remarkably similar to GCNNs.", "weaknesses": "1. The point about layerwise equivariance is covered in some prior works that are not cited. Much of appendix D.1. in particular is discussed in depth in [1]. Limitations of linear layerwise equivariance is discussed in [Finzi et al. 2021].\n2. As noted by the authors, the restriction to horizontal flips is restrictive for empirical evaluation, though to be fair horizontal flips consistently improve many vision systems.\n\n[Shakerinava et al. 2022] Structuring Representations Using Group Invariants. NeurIPS 2022.  \n[Finzi et al. 2021] A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups. ICML 2021\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their useful suggestions and helpful review.\n\n> The point about layerwise equivariance is covered in some prior works that are not cited. Much of appendix D.1. in particular is discussed in depth in [1]. Limitations of linear layerwise equivariance is discussed in [Finzi et al. 2021].\n\nThank you for pointing out these overlooked references. We will of course incorporate them.\n\n> What do you mean on Page 5 when you say that the results for Godfrey et al. 2022 for other nonlinearities \"also straightforwardly translate to the group equivariance case\"?\n\nWe apologize for the sloppy formulation which we will improve. What is meant is that Section 3.1/Table 1 of Godfrey et al. contains several results of the type (paraphrased) \"For nonlinearity $\\sigma$ if $\\sigma(Ax)=B\\sigma(x)$ for invertible matrices $A, B$ then $A$ and $B$ have the following specific forms...\". This implies that the representations acting on the feature spaces in a layerwise equivariant network with activation function $\\sigma$ must have matrices of the respective forms of $A$ and $B$.\n\n> Minor [...]\n\nThanks!"}, {"review_id": "MF097CmhdC", "reviewer": "Reviewer_ZqAK", "summary": "The paper considers networks with ReLU activations and investigates in which specific way their internal activations learn to be equivariant given an invariant data distribution.\nSection 2 presents a theoretical result, applying specifically to two-layer networks with non-singular weight matrices. It is shown that the network's equivariance requires 1) that its input and output activations are acted on by scaled permutation representations, and 2) that the network is layerwise equivariant with a related scaled permutation representation acting on the hidden features. Example 2.1 clarifies that this result does not necessarily hold in the case of non-singular weights.\nThe third section proposes the conjecture that ReLU-CNNs that are trained on a reflection-invariant data distribution are close to regular GCNNs, i.e. GCNNs whose group representations are permutation representations. This conjecture is based on Entezari et al.'s conjecture that the activations of neural networks can always be permuted such that the networks are closely connected in the loss landscape. The authors argue that there always exists a CNN with exactly flipped kernels at initialization, which is due to the assumed invariant distribution preserved throughout training. Since these nets can be Entezari's conjecture be aligned via channel permutations, it follows that the CNN has either invariant kernels or kernels which are related by reflections - they are hence layerwise equivariant regular GCNNs.\nThe experimental section investigates whether these two results hold in practice, using an activation matching technique to find reflected kernels. The results suggest that layerwise equivariance holds indeed.", "questions": "Why is section 2 considering linear instead of affine layers? Do the results not hold when biases are summed?\nLemma 2.2 claims that the diagonal scaling matrix would be required to have (strictly) positive entries, however, non-negative entries, including zeros, seem sufficient. The resulting set of matrices does then no longer form a group since invertibility is lost, but the original group is contained as subset. The \"intertwiner group\" should furthermore be renamed to \"equivariance group\" since intertwiners are by definition linear, which ReLU is not.\nThe activation matching technique should be briefly explained in the current submission instead of only pointing to related work.\nThe paper is currently only mentioning trivial and regular representations, however, everything should apply to more general quotient representations as well, which should be briefly clarified.", "limitations": "Limitations are clearly addressed in section 1.2. and societal impacts are not to be expected.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and gives insight into the nature of networks which are trained to be equivariant. This is an original idea since most publications on equivariant networks are rather investigating models which are by design constrained to be equivariant. The actual necessity of regular group convolutions is clarified by the submission.\nWhere possible, the networks layerwise equivariance is undoubtably proven with an analytical approach. While this approach does not generalize to networks with more than two layers or singular weight matrices, the authors use a different approach to tackle these cases. All results and conjectures are supported empirically.\nThe extent of the paper is comprehensive, containing multiple additions in the supplementary material.", "weaknesses": "The strong assumptions in the analytical result seem initially like a weakness, but are addressed by the alternative approach based on Entezari's conjecture. The argument that this conjecture implies layerwise equivariance seems reasonable, but could be explained better. I am also not sure about the conjecture applies in the first place, however, this is not the scope of the current paper. \nTechnically, I do not see the necessity for the definition of the barrier in Eq. (4) exact at the parameter's average, instead of at the maximizing parameter value.\nAnother weakness is that the authors are only considering reflection-equivariant GCNNs, but not more general symmetry groups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and helpful review.\n\n> Why is section 2 considering linear instead of affine layers? Do the results not hold when biases are summed?\n\nThis is an excellent question. We wrote it this way (1) for simplicity, (2) because Elesedy & Zaidi [9] use the same setting and (3) for the fact that Prop 2.3 deals with bias free layers. The discussion up to Section 2.1 works with biases as well, as does the Lemma 2.2 (as it regards ReLU and not the layers themselves). The paragraph after Lemma 2.2 also works fine. For Proposition 2.3 it gets more complicated. What makes the proof of 2.3 easy is that given an invertible linear layer $\\ell:X\\to Y$, $\\ell(x)= Wx$, a group representation $\\rho(g)$ on $X$ is transferred to a group representation $\\alpha(g, y) =\\ell(\\rho(g)\\ell^{-1}(y))=W\\rho(g)W^{-1}y$ on $Y$ w.r.t. which $\\ell$ is equivariant. If we consider an affine layer $\\ell(x) = Wx + t$ with bias vector $t$, then we can play the same game, but $\\rho(g)$ is not transferred to a group representation (linear action) but instead to the affine action $\\alpha(g, y) = \\ell(\\rho(g)\\ell^{-1}(y))= W\\rho(g)W^{-1}y  + (I - W\\rho(g)W^{-1})t$ on $Y$. This means that we can not apply Lemma 2.2. \nWe can however find a generalization as follows:\n\n**Lemma** If $A$, $B$ are $n\\times n$ invertible matrices and $a$ and $b$ are $n$-vectors, such that $\\mathtt{ReLU}(Ax + a) = B\\mathtt{ReLU}(x) + b$ for all $x\\in\\mathbb{R}^n$, then $a=b=0$.\n\n*Proof:* Inserting $x=0$ yields $\\mathtt{ReLU}(a)=b$. Inserting $x=A^{-1}a$ yields $\\mathtt{ReLU}(2a)=B\\mathtt{ReLU}(A^{-1}a)+b$ so that $b=B\\mathtt{ReLU}(A^{-1}a)$. Inserting $x=-A^{-1}a$ yields $0=B\\mathtt{ReLU}(-A^{-1} a) + b$ so $b=-B\\mathtt{ReLU}(-A^{-1}a)$. Combined we have that $B^{-1}b=\\mathtt{ReLU}(A^{-1}a)=-\\mathtt{ReLU}(-A^{-1}a)$ so that $B^{-1}b$ must be zero and hence so must $b$. Finally, inserting $x=-2A^{-1}a$ yields $\\mathtt{ReLU}(-a)=B\\mathtt{ReLU}(-2A^{-1}a) + b$ so that $\\mathtt{ReLU}(-a)=-b=0$ which combined with $\\mathtt{ReLU}(a)=b=0$ gives $a=0$.\n\nThis lemma shows that if ReLU commutes with affine actions, then the affine actions must in fact be linear and so Lemma 2.2 applies. This shows that Proposition 2.3 holds with affine layers as well. We would like to greatly thank the reviewer for prompting this generalization which we will include in the appendix.\n\n> Lemma 2.2 claims that the diagonal scaling matrix would be required to have (strictly) positive entries, however, non-negative entries, including zeros, seem sufficient. The resulting set of matrices does then no longer form a group since invertibility is lost, but the original group is contained as subset.\n\nIt is correct that non-negative diagonal matrices commute with ReLU. The reason for excluding them in the lemma is that otherwise we would have to exclude them later when talking about group representations, for the reason the reviewer states - non-invertibility.\n\n> The \"intertwiner group\" should furthermore be renamed to \"equivariance group\" since intertwiners are by definition linear, which ReLU is not. \n\nThis terminology comes from Godfrey et al. 2022. We agree with the reviewer that the naming is a bit unfortunate. We are unsure if the best course of action would be to add a footnote or to simply remove the sentence, and are happy to receive further feedback.\n\n>  The activation matching technique should be briefly explained in the current submission instead of only pointing to related work. The paper is currently only mentioning trivial and regular representations, however, everything should apply to more general quotient representations as well, which should be briefly clarified.\n\nWe thank the reviewer for these helpful suggestions. We would like to clarify that trivial and regular representations are the only ones discussed for the horizontal flipping case as any permutation representation of the 2 element cyclic group must consist of fixed elements and transpositions only. I.e. the permutation representations are direct sums of trivial and regular representations."}]

Author rebuttal:
We thank all reviewers for their well-written and very helpful reviews.

Reviewer k5hy suggested an extra experiment which we have carried out and believe will be of interest to all reviewers.
We train VGG-11 nets on CIFAR-10 *without* horizontal flipping data augmentation. These models have lower accuracy and lower invariance than the ones trained with flip augmentation. Most importantly, they have higher GCNN barrier. The GCNN barrier for the nets trained without flip augmentation is approximately as large as the barrier between two separate nets a la Entezari et al. Since the data still contains horizontal flipping symmetries we find it non-surprising that the obtained nets are still quite close to being GCNNs in this sense. Please find the relevant numbers in the table below:

| Name                                   | Accuracy                     | Invariance Error             | Barrier                                  |
|----------------------------------------|------------------------------|------------------------------|------------------------------------------|
| CNN (Table 2)                          | $0.901 \pm 2.1\cdot 10^{-3}$ | $0.282 \pm 1.8\cdot 10^{-2}$ | $4.00\cdot 10^{-2}\pm 4.9\cdot 10^{-3}$  |
| CNN w/o horizontal flip augmentation (NEW)                | $0.879 \pm 1.8\cdot 10^{-3}$ | $0.410 \pm 4.0\cdot 10^{-2}$ | $4.98\cdot 10^{-2}\pm 6.1\cdot 10^{-3}$  |
| CNN merged with separate net (Table 4, appendix) | $0.901 \pm 2.1\cdot 10^{-3}$ | $0.282 \pm 1.8\cdot 10^{-2}$ | $5.08\cdot 10^{-2}\pm 5.7\cdot 10^{-3}$  |

Also of general interest could be that reviewer ZqAK prompted a generalization of Lemma 2.2 and Proposition 2.3 to affine layers. We refer to the individual rebuttal for this - it is the first question answered there.

Meta-review: A paper investigating whether equivariance of a network implies layerwise equivariance. For two-layer networks, a simple characterization is provided. For multi-layer CNNs, a range of numerical experiments supporting the considered thesis under horizontal flipping symmetry is given.

The reviewers found the research problem put forward in the paper novel and intriguing, and coincided that sufficient empirical evidence has been given to make the numerical results sufficiently solid. Please make sure to address critical comments raised by the reviewers; in particular, applicability beyond the simple settings studied in the work.
Decision: Accept (poster)