Equivariant neural networks and equivarification
Authors: Erkao Bao
Linqi Song
Venue:  (2020)
Keywords: equivariant
invariant
neural network
equivarification

Abstract: A key difference from existing works is that our equivarification method can be applied without knowledge of the detailed functions of a layer in a neural network, and hence, can be generalized to any feedforward neural networks. Although the network size scales up, the constructed equivariant neural network does not increase the complexity of the network compared with the original one, in terms of the number of parameters. As an illustration, we build an equivariant neural network for image classification by equivarifying a convolutional neural network. Results show that our proposed method significantly reduces the design and training complexity, yet preserving the learning performance in terms of accuracy.

Under review as a conference paper at ICLR 2020
EQUIVARIANT NEURAL NETWORKS AND EQUIVARIFI- CATION
Anonymous authors Paper under double-blind review
ABSTRACT
Equivariant neural networks are special types of neural networks that preserve some symmetries on the data set. In this paper, we provide a method to modify a neural network to an equivariant one, which we call equivarification.
INTRODUCTION
One key issue in deep neural network training is the difficulty of tuning parameters, especially when the network size grows larger and larger Han et al. (2055). In order to reduce the complexity of the network, many techniques have been proposed by analyzing the structural characteristies of data, for example, sparsity Wen et al. (2016), invariance in movement Goodfellow et al. (2009).
One of the most important structural characteristics of data is symmetry. By utilizing the translation symmetry of an object in the photo, convolutional neural network (CiNN(Krizhessky et all (2012) uses shared filters reduce the number of parameters compared with fully connected networks. However, to handle the case of rotation and reflection, people usually use the data augmentation approach to generate additional input data that has a bunch of training images with different rotation angles of the original images.
In contrast to data augmentation approach, another idea to design more sophisticated neural net- works, such that the input data with certain symmetries can be trained together and applied to reduce the training complexity. Recent attempts have been made in the equivariant CNN Cohen & Welling (016;);; Cohen et all 20 Weiler al. (2018). These existing works target at special cases and cannot be easily generalized to arbitrary networks and arbitrary symmetries.
In this paper, take advantage of the symmetry of the data set and provide general method to modify an arbitrary neural network so that it preserves the symmetry. The process is called equivarification. A key feature is that knowledge of layer in neural network, and hence, can be generalized to any feedforward neural networks. Another feature that the number of parameters in the new neural network that we need to train is the same as the original one, if we equivarify the original network globally (see the second paragraph in Section 4 for details). In addition, we can also output how each data instance is changed from the canonical form (for example, in an image classification problem, additionally we can also output how many degrees an image is rotated from original upside-up image) using the same network.
equivarification method can be applied without detailed
We rigorously prove that our equivarification method produces truely equivariant neural networks. Practically, equivarify CNN as an illustration. We conduct experiments using the MNIST data set where the resulting equivariant neural network predicts the number and also the angle. we forget the angle and keep only the number, we get invariant neural network.
1.1 MOTIVATION
By the symmetry of the data set, we mean a group action (see Definition 2.2 on the data set. Let us consider a simple cat image classification example. Then the data set can be the set of all images of a fixed size. Rotation symmetry means that we can rotate an image and the resulting image still lies in the data set. In other words, the group of rotations
the data set.
One can build a cat classifier that assigns a number between O and to each image indicating the probability that it is image of cat. If one image is rotation (say by 90 degree counterclockwise)

---

the ini- Under review as a conference paper at ICLR 2020
of another one, then it makes sense require a classifier to assign the same probability to these two images. A classifier that satisfies this property is said to be invariant under 90-degree rotation, which special case of being equivariant under 90-degree rotation. To give an example of an (non- invariant) equivariant neural network, we furthermore require our classifier not only produces the probability of being cat image, but also outputs angle, say in {0, 90, 180, 270) (more precisely, the probability of each angle), Suppose that the equivariant classifier predicts that an image is rotated by 90 degrees, then it should predict a 270-degree rotation for the same image but rotated by 180 degrees.
Not only does make sense to require the cat classifier to be equivariant, but also it is more "eco- nomical" to be equivariant if implemented correctly. Roughly speaking, a regular classifier "treats" an image and its rotated ones as separated things, but ideally an equivariant neural network "sees" their connections and "treats" them together. From another point of view, given regular neural network, if we apply the data augmentation method carefully, since the training data is symmetric, it is possible that after training, the network is (approximately) equivariant (depending tialization). The fact that it is equivariant implies that there is now symmetry among the parameters. For example, some parameters our equivariant neural network, the equivariance of our neural network is built into the structure of the neural network by sharing parameters, and in particular, it is independent of the loss function, initialization, and the data set. For instance, for the training data, it does not make any difference in results whether prepare it by randomly rotating and recording the angles, not rotating and labeling everything as degree 0.
the same as other parameters, so there is redundancy. While in
In our approach, to make neural network equivariant, at each layer other than the input layer we add a bunch of neurons (multiplying by the order of the group), but we don't 't introduce new parameters. Instead the added neurons share parameters with the original ones.
1.2 RELATED WORK
Invariance in neural networks has attracted attention for decades, aiming at designing neural networks that capture the invariant features of data, for example, face detection system at any degree of rotation in the image plane Rowley et al. (1998), invariance for pattern recognition Barnard & Casasent 1991),, translation, rotation, and scale invariance Perantonis
invariance|Perantoniss & I Lisboa (992).
Recently, several works have started to look into equivariant CNN, by studying the exact map- ping functions of each laver Cohen & Welling b2)16a Cohen et al. (2018): Marcos et al. (2017); :018):Cohenn
Lenssen et al. (2018); Cohen et al. (201)), and some symmetries are studied such as translation sym- metry, rotation symmetry. These methods have been used in different application domains, such as in remote sensing Marcos et all (2018). digital pathology Veeling et al. (2018), galaxy morphology prediction Dieleman et al. (2015).
There are also works that aiming to construct an equivariant neural network without having to spec- ify the symmetry (ahour et al. (2017).
As far as we know, our construction provides the first truly equivariant neural network (See Sec- tion 5.
2 PRELIMINARIES
In this section, we talk about some basics in group theory, like group actions, equivariance, etc. For those who would like to get a more close look at the group theory and various mathematical tools behind it, please refer to any abstract algebra books, for example, Lang 2002).
Here, we first give a couple of definitions about groups in order to help readers quickly grasp the concepts.
Definition 2.1. A group (G.-) consists of a set G together with a binary operation "." (which we usually call multiplication without causing confusing with the traditional sense of multiplication for real numbers) that needs to satisfy the four following axioms.
1) Closure: for all a, b E G. the multiplication b E G.
2

---

Under review as a conference paper at ICLR 2020
2) Associativity: for all b, G, the multiplication satisfies (a - b) . (b.c).
3) Identity element: there exists a unique identity element e E G, such that, for every element
E G, we have e
4) Inverse element: for each element a E G, there exists an element b G, denoted a -1 such that 6 = 6 e, where is the identity element.
Note that in general, commutativity does not apply here, namely, for a, b G, a = a does not always hold true.
For example, integers with the operation addition (2, +). One can easily check that the four axioms satisfied and 0 is the identity element.
As another example, a group consists a set {0, together with the operation + (mod 2) where 0+1= 1+0 and 1 1+1 = 0+0 0. The identity element is O. When applying this to the image processing tasks, this can be interpreted as follows. represents the action rotating an image by 180" and 0 represents the action of not rotating image. Then + represents the combination of operations that we first rotate an image by 180" and then keep it as is, so that the final effect is to rotate an image 180%: while + represents that first rotate image by 180 and then rotate again by 180%. which is equivalent to that we do not rotate the original image.
Similarly, we can define the element 1 the operation of flipping an image vertically or horizontally, which we can give similar explanations for the group.
Therefore, instead of using translations, rotations, flippings, etc., we can use abstract groups to represent operations on images, and hence, we are able to design corresponding equivariant neural networks disregarding the operations of images (symmetries of data) and just following the group representation.
In the following, we give a definition about group actions. Let X be set, and G group. Definition 2.2. We define G-action on X to be map
T:GxX-X
(on the left) such that for any X
T(e, x) = x, where e G is the identity element,
for any g1, 92 G, we have
T(g,, T(g2, x)) = (9112,22).
Frequently, when there is no confusion, instead of saying that T is G-action on X or equivalently that G acts X via T, we simply say that G acts X; and is also understood from the notation, iee, instead of T(g, æ) we simply write gr, and the above formula becomes (g2x) = (g1g2)..
We say G acts trivially on X if gz for all g E G and x X.
Let X, Y be two sets, and G be group that acts on both X and Y.
Definition 2.3. A map F X Y is said to be G-equivariant, if F(gx) gF(x) for all x X and E G. Moreover, if G acts trivially on Y then we say is G-invariant.
Example 2.4. Let X be the space of all images of 28 28 pixels, which contains the MNIST data set. Let G be the cyclic group of order 4. Pick a generator of G, and we define the action of on X by setting gø to the image obtained from rotating counterclockwise by 90 degrees. Let Y be the set {0, 1, 2, .... 9} x {0, 90, 180, 270}. For any y (num, 0) E Y we define
gy (num, (0 + 90) m0)
An equivariant neural network that classifies the number and rotation angle can be viewed map F from to Y. Equivariance means if Fxx) = (num, 0) then F(gz) (num, (0 + m0)moo)0)) for all x E X.
Thus,
that model each layer in the neural network as group G' that acts set X, where we can interpret the set X as input to this layer and the group action as the function map- ping operation of this layer. By abstracting the behaviors on the original input data using groups
3

---

Under review as a conference paper at ICLR 2020
(for example, using same group to represent either rotation by 180 and flipping, more abstract operations during intermediate layers), we are able to apply group actions on different sets X1, X2, X3.... (where each represents input to different layers) and design similar equivariant network structures based on an original one.
3
EQUIVARIFICATION
In this section, we talk about the detailed method of performing equivarification and its theoretical foundation. This is the key part of the paper understand our proposed equivarification method. Those who would like to avoid mathematical proofs can directly go to the examples we provide to get intuitive ideas of how we construct the equivariant neural networks.
In this section we fix and to be two sets, and G to be a group that acts on X.
Definition 3.1. Define the G-product of to be
Z z = fs: G - Z),
the set of all maps from G to Z. We define G-action zxG by
G zxG Z*G
(0,8) 98,
where as a map from G to Z is defined as
(g))(gg) = s(g 'g),
(3.1)
for any g' E G.
We have the projection map p: Z + Z defined by
p(s) = s(e),
(3.2)
for any E Z where G is the identity element. Then
Lemma 3.2. For any map F X - Z. there exists unique G-equivariant map F X , 2 such that p( (x)))) = (())for all X.
Proof. For any E X, we define F(x) as map from G to Z by (f())))) = F(g
for any g G. To see that is G-equivariant, we need to check for any x X and g G, F(ox) g (x)))) For any h G, (F(g)))(h) F(h-lgx) by the definition of F, while (g(F())))(h) (F(z))(g h)= F(h gr). We leave the proof of uniqueness to the readers. O
Remark 3.3. In the Definition 3.1 and Lemma 3.2 G is an arbitrary group and Z is an arbitrary set. easy see that we can adjust them. if we consider other categories. For example, when G is compact Lie group and Z is differentiable manifold, we can re- re-define x to be the space of differentiable maps from to Z; when is a non-compact Lie group and is differentiable manifold, we can consider the space of compact supported smooth maps. When we implemented the neural network for infinite G. we have to approximate it by finite subset of G. Then this case, we need to work in the realm of approximately equivariant. For this implementation reason, we restrict our group G to be a finite group for the rest of the paper.
This lemma can be summarized as the commutative diagram Figure 1 It motivates the following general definition.
Definition 3.4. We say a tuple (2, T.p)) a G-equivarification of 2 if
- is set with G-action T:
4

---

Under review as a conference paper at ICLR 2020
Z
3.
P
7
F
Figure 1: For any X and F, there exists a unique lift F such that the diagram commutes.
is a map from to Z;
For any set X with G-action, and map F X + Z, there exists a G-equivariant map F: X + such that F = F.
Here denotes the composition of maps. As usual, T will be omitted from notation.
In Section 4 will applications of G-equivarification to neural networks. From Lemma 3. we know that the triple of the G-product 7 7xG, the G-action defined the Formula 3. and the projection p defined in Formula is G-equivarífication There are other G-equivarifications. See Appendix for more discussion.
Example 3.5. Let G be the cyclic group of order 4. More concretely, we order elements of G by (e.g. g. gd). The set Z G can be identified with Z x4 Z Z Z via the map
- (s(e), (g), +()))
(3.3)
Then G acts on x by g00 21, 22, 23) (Z3, 20, 22), and the projection map p: 2 + Z is given by (Zo; 21, 22, 23)
2g. Let F X - Z be arbitrary map, then after the identification F
becomes a map from to x4 and
F(x) = (F(x), F(g 1x)
One can check that F is G-equivariant. The map is given by
p(=0, 21, 23)
It is easy to see that == F.
APPLICATION TO NEURAL NETWORKS
In this section, we show through an example how our proposed equivarification method works.
Let (T, X; > N be an n-layer neural network (which can be CNN, multi-layer percep- trons, etc.). In particular, Xa the input data set, and X,+1 is the output data set. Let be finite group that acts on Xo. Let be the composition of all layers
L Ln Ln-
Lo Xo X
Then equivarify T and get maps Ī. Xo 8n and p 8a41 X Then F. is cquivariant neural network.
Alternatively, one can construct an equivariant neural network layer by layer. More precisely, the equivariant neural network is given by (I4 o op: : & + 8,+1)% )oo where I, p; is the equivari- fication of L, p; for i E {0, nI.. X, Xo and po id is the identity map (See Figure 2 More precisely, by the commutativity of Figure
2 we know that
Pat1 Ln o Pr. o Ln Pn-ppn1 o Lp = po
Then both I,, Pn - Pn-1 ....o Lo Po and 7. are equivarífications of L. Suppose that for both equivarifications we have chosen Xn be X x Then by the statement in Theorem 3.2 we have
Ii Pr I Pn-1 La o = T.

---

Under review as a conference paper at ICLR 2020
X
14 1
Pi+1
X
X
L op
Figure 2: Equivarification layer by layer.
Sometimes, other than equivarifying the map L; o Ps X, 7 X it makes sense to construct some other map LA from X; to some other set X and then we can equivarify LA This makes the equivariant neural network more interesting (see the example below).
Example 4.1. Let the O-th layer Lo Xo + X1 of a neural network that is defined on the MNIST data set be convolutional layer, and X1 R where of channels (strides = (1, padding = 'same"). Let G = {e,g.9 g..g...0 ggy) be the cyclic group of order 4 such that g acts on Xo as the 90-degree counterelockwise rotation. Then construct Lg Xo 2. by
28 x 28 C1, and ei is the number
(Lo(zo), Lolg Lo(g"'0)) 20), Lo(g->o)) Lolg Lo(g))o))) xz))).
For the next layer, instead of equivarifying L1 p1 : R44, + R we can construct another con- volutional layer directly from R46 by concatenating the four copies of R along the channel axis to obtain 228x28x4ee, and build a standard convolution layer on it. This new construction of course changes the number of variables compared that of the original network.
From the above analysis and Lemma 3.2 it is not hard derive the following summary.
Main result Let X = (L, X; Xarlosignnt be an original neural network that can process imput data (2 C Xo and labelling data { Xn+1. Let G be a finite group that acts on
Xo- The proposed G-equivarification method is able to generate G-equivarian neural network X =ff. X; - *it1JosiSntt that can process input data {xots No Xo and enhanced labeling data (2at1l S. Xa41. Furthermore, the number of parameters of X is the same as that X
5 EXPERIMENTS
In this section, we show our experiments on the MNIST dataset, which achieve promising perfor- mancd]
Our designed equivariant neural network using the proposed equivarification method is shown in Figure Note that equivarification process does increase the number of variables. In our case, in order to illustrate flexibility we choose not to simply equivarify the original neural network, so the layer 2onv2 and conv3 have four times the number variables compared to the corresponding original layers.
Next, we discuss the labeling of the input data. Since the neural network is G-equivariant, makes sense to encode the labels G-equivariantly. Suppose (ro, xn-1)) E X, x X,+1 is one labelled data point. Then in the ideal case, one hopes to achieve L(xa) = In+1. Assuming this, to give new label for the data point To for our equivariant neural network we need to define Tnt1 = [(xx). For this, it is sufficient to define (g) for all g E G. By equivariance, [()o) = g e then ((x) not have label, so we do not know ideally what L(g-1aa 'xo) should be. In the naive data augmentation approach, L(g 2x) ) is labeled the same L(xo) hoping to get L close an invariant map as possible. In our case, we do not have such restriction, since we do not need L to be invariant. In
If L(xo) Zn+1 g + e, is very likely that the data g 1 ar originally does
'The code in tensorflow is uploaded the supplementary material. In the code, also include version that allows 90 degree rotation and horizontal and vertical flips, just make the group non-commutative.

---

Under review as a conference paper at ICLR 2020
8444
8
X2
dew c pid com og
X2
dene pool wot
Com by"
X2
X
200r1
Figure 3: In this figure, convl I is standard convolution layer with input = Xo and output Xi- After equivarification of convl, we get four copies of X,. Then stack the four copies along the channel direction, and take this whole thing as an input of a standard convolution layer conv2. We equivarify conv2, stack the four copies of X2, and feed it another convolution layer conv3, Now instead of equivarifying conv3, we add layer pool and layer dense (logistic layer), and then we
equivarify their composition dense pool conv3 og and get X 4o To get the predicted classes, we can take an argmax afterwards.

---

Under review as a conference paper at ICLR 2020
(7, 278.0)
5.15240384e--07
9.97985136-12
3.34337755-05
1.27654692--07
1.38555313e-49 3.10:30470-09
3.88220069-044
1.41850875e-08
4.376 1.3767561e--07 2.571:5262--55
1.85485029e-03
3.34237237-04
2000
4.249175:3e-48
5.35921688e-01
L.1269225e--06
L.621:0627--01
4.22584851--12
7-36331956-16
1.2586661e-10
3.96176119--13
L.23941522c-12 L.070237450-13
L.206382786-12
3.46224758p-11
1.78956502e-12
4.06354692e-16
9.83018693e-12
1.37454061e-14
4.25693346e-12
4.94812546e-18
3.94265287--15
1.48675974e-15)
(7, 188-0)
KSM
[S. 79642399-99-1
1.745518866-17
1-78956302e-12
L-0638699e-15
$500
8.22018922e-13
1.27484041e-14
6.25492346--12
4.23832846--18
1.9425527e-15
1.48675974e-16
6.395:7124e-11
5.35246394e-41
1.3497795--99
1.27654632--07
1.38545313e-09
1.18:304786-09
1.61859815--06
4.3767567le-07
2.57115262e-05
8300
5.C5485029e-41
3.34337237e-44
6.848:7519--08
8.15991488--01
L-22611233--04
1.621186227-01
6.72524531--12
7.35331976-16
5.25985621e-10
3.34725115e-13
1.13841572--12
1.07023745b-113
1.5743319e--22
[3. 00..885119--04
1.61350115e-08
4.97675471e-63
2.57:15263e-05
L.85485029-933
3.343372176-04
5.84817519e-06
9:35991(48b-01
..82691263--46
1.821:3627e-41
4.7255483:e-12
7-151319676-11
1.25886631--10
3.94726135e-13
1.13841592--12
1-87623745--13
1.186892786-12
3.46224758e-11
1.19656125e-13
L-57653189-112
3.03018893e-13
1.37454066e-14
4.26433346e-12
6.965528466-11
1.34265277=-15
1.48675975e-16
2.85646876e-11
6.195:71244-11
5-01169980s-07
6.22865715--09
5.35240344-07
9.57538194--12
6.34397755--09
1.27654692e-07
1.18545313e-09
3:18130478p-09)
(7, $0.31
71-7254891e-12
7.3533190Te-15
1-259066010-18
3:96726135e-13
1.79656135e-13
1.57643158e-12
5.79642349e-15
1.74651886--17
1.28954512e-12
4.86354692e-16
3.830:8891e-13
1.3745405lo-14
4.26493366e-12
4. 4.9411286e-18
3.94265287e-15
1-8867597--16
2.85644875e-11
6.895:7124e-11
5.81189990e-07
6.22665715--05
5.35240534e-07
5.975191746-12
3.342971956-02
1.276346226-07
4.3767561le-07
2.57115252e-05
1.85435029e-03
3.341372376-04
4.84017519e-01
8.35981488e-01
1.82691263e-04
1.621:0227=-01)
On the left, we have the rotated images; on the right, we have the predicted numbers, angles, and the probability vectors in R40, each component of which corresponds the probability of a (number, angle) combination. The equivariance in this case means that the four vectors are related by shifts.
Figure 4:
practice, Xn+1 is a vector space, and we choose to label L(g-1xo) by the origin of X,+1- In our MNIST example, this choice the same as the following.
For m {0, 1,2, .). 9} denote
Em = (0, ... 1,0, ................................ .., 10
m-th spot
For an unrotated image E Xo that represents the number m, we assign the label @ #0@000 4: Then based on the equivariance, we assign
grn + $ em
+
0,
-²2 + 0 + 0
+ 0,
-0 - 0 + 0 4 Em
For each testing image in the MNIST data set, we randomly rotate it by an angle of degree in (0, 90, 180, 270), and we prepare the label as above. For the training images, we can do the same, but just for convenience, we actually do not rotate them, since it won't affect the training result at all.
5.1 EQUIVARIANCE VERIFICATION
To spot check the equivariance after implementation, print out probability vectors in IR:0 of an image the number and its rotations. We see that the probability vectors are identical after a shift by 10 slots. See Figure
8

---

Under review as a conference paper at ICLR 2020
5.2 ACCURACY
Here we count the prediction as correct if both the number and the angle are predicted correctly. The accuracy of neural network the data is 96.8%. This promising when considering the fact that some numbers are quite hard to determine its angles, such as 0, 1, and 8.
6 CONCLUSION
In this paper, we proposed an equivarification method to design equivariant neural networks that are able to efficiently process input data with symmetries. Our proposed method can be generalized to arbitrary networks or functions by leveraging group action, which enables our design to be uniform across layers of feedforward neural networks, such multi-layer perceptrons, CNNs, without being aware of the knowledge of detailed functions a layer a neural network. As an illustration example, show how to equivarifying CNN for image classification. Results show that proposed method performs as expected, yet with significantly reduction in the design and training complexity.
A APPENDIX MORE ABOUT EQUIVARIFICATION
In Section 3 we define (zxG,) ,p) example of G-equivarification. In this section, we show that it is "minimal" in the sense of its universal property.
Lemma A.1 (universal property). For any G-equivarification (Z',') of Z, there exists G- equivariant map Z' + 2 such that p' pen. Moreover, for any set X and map F X + Z, the lifts F X z and F' z - Z' of F satisfy F' =.
FF (See Figure 5
22), where E E Z' and g E G. To show p' P n, for any 2' E Z, we check po o (3²) = p3m( = (2')((e) = p(3'). To show π is G-equivariant, for any E' E 2, and h E G. we compare π (1h3') and hT'()): for any
Proof. We define the map ग Ż' zx by (())) p''g-'''''
g E G, (=()))(()) = p' 112") and (h)())) = (m()) (19(h-110 p' h2²). Lastly, we show a F = F. Note that π F'r is a G-equivariant map from X to xxG and
( Fr) =p'oF'= F=
so by the uniqueness of Lemma we get क F = F.
Now discuss about finding "smaller" equivarification in another direction, shrinking the group by bring in the information about X. Let = [g E G gz = x for all E Gj. the set of elements in G that acts trivially on X. It is easy to check that N is normal subgroup of G. We say G acts on effectively if N = {e]. In the case when G does not act effectively, it makes sense to consider the G/N- C/N-product of Z, where G/N is the quotient group. More precisely, consider zxG/N
{s G/N + Z}. which is smaller in size than xxG For any map F X + Z, we can get a G/N eequvaariat lift F of F following the same construction as Lemma 3.2 (with G replaced by G/N). Since G maps to the quotient G/N, we have that G acts on Z/N and F also G-equivariant.
REFERENCES
Etienne Barnard and David Casasent. Invariance and neural nets. IEEE Transactions on neural networks, 6(5):498-508 1991.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999, 2016a. 6a.
Taco S Cohen and Max Welling, Steerable cnns. arXiv preprint arXiv: 2016b.
Taco Cohen, Mario Geiger, Jonas Kohler, and Max Welling. Spherical CNNs. arXiv preprint arXiv: 10/30, 2018.

---

Under review as a conference paper at ICLR 2020
Z'
F
7
F
X
Z
Figure 5: Factors through.
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolu- tional networks and the icosahedral cnn. arXiv preprint arXiv: (.0iv:/902.0465.. 2019.
Sander Dieleman, Kyle W Willett, and Joni Dambre. Rotation- works for galaxy morphology prediction. Monthly notices of the royal astronomical society, 450
convolutional neural net-
(2):1441-1459, 2015.
Ian Goodfellow, Honglak Lee, Quoc V Le, Andrew Saxe, and Andrew Y Ng. Measuring invariances in deep networks. In Advances in neural information processing systems, pp. 646-654, 2009.
Song Han, Huizi Mao, and William Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Serge Lang, Algebra, GTM211. Springer, 2002.
Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. In Advances in Neural Information Processing Systems, pp. 2018.
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5048- 5057, 2017.
Diego Marcos, Michele Volpi, Benjamin Kellenberger, and Devis Tuia. Land cover mapping at very high resolution with rotation equivariant cnns: Towards small yet accurate models. ISPRS ISPRS journal of photogrammetry and remote sensing, 145:96-107, 2018.
Stavros Perantonis and Paulo JG Lisboa. Translation, rotation, and scale invariant pattern recog- nition by high-order neural networks and moment classifiers. IEEE Transactions on Neural Net- works, 3(2):241-25 1992.
Henry A Rowley, Shumeet Baluja, and Takeo Kanade. Rotation invariant neural network- face detection. In Proceedings. 1998 IEEE Computer Sociery Conference on Computer Vision and Patlern Recognition (Cal. No. 98CB36231), pp. 38 44. IEEE, 1998.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. tween capsules. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and Garnett (eds.), Advances Neural Information Processing Systems 30, pp. 3856-3866. Curran Associates, Inc., 2017. URL wwwttiinngbetteee-eee
Dynamic routing be-
http://seepppsccppppeeeeeeeepeee http://97755ymmmmmmmm. dynamic
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation equiv- ariant cnns for digital pathology. In Intermational Conference on Medical image computing and computer-ass assistee intervention, pp. 210-218. Springer, 2018.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equiv- ariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849-854 858, 2018.
10

---

Under review as a conference paper at ICLR 2020
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.


Reviews:
[{"review_id": "r1xeNwATKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a method for making a neural network equivariant. Their method also can be applied to make each layer equivariant too. \n\nStrengths:\n-- The paper is very well written and easy to follow with clear notation. \n\n-- The derivations seem to be correct.\n\n\nWeaknesses:\n-- The experiment is nice but very limited and does not demonstrate the benefits of having an equivariant network. For example, the authors do not report the accuracy of recovering the original (0) rotation.\n\n-- The novelty of the work is questionable. While the development is different, the final example for equivarification of a neural network is very similar to the existing works by Cohen and Welling.\n\n-- There are other works on equivarification that are missed by this paper. For example, consider the following paper:\nLenssen, J. E., Fey, M., & Libuschewski, P. (2018). Group equivariant capsule networks. In NeurIPS.\n\n-- The layer-wise equivariant method does have extra computational overheads.\n\n-- The fact that we have to specify the groups that we want to make the network equivariant with respect to is a limitation. The promise of capsule networks, in contrast, is to \"ideally\" learn the pose (variation) vectors in a data-driven way.\nSabour, S., Frosst, N., & Hinton, G. E. (2017). Dynamic routing between capsules. In NeurIPS.\n\n-- The following statements need more explanation:\n  * \"However, these may require extra training overhead as the augmented data increase.\"", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "H1lsO14PtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors employ concepts from group theory to turn an arbitrary feed forward neural network into an equivariant one, i.e. a network whose output transforms in a way that is consistent with the transformation of the input. To this end, the authors first introduce the basic concepts of group theory required to follow their work and provide a comprehensive definition of equivariance. They then explain how to equivarify (w.r.t. a finite group G) a given neural network, and present experimental results on rotated MNIST digits to support their approach.\n\nI find that the proposed approach is very elegant and addresses a highly important question, namely how to devise or modify an architecture such that it preserves symmetries. In my opinion, the current paper makes an interesting theoretical contribution towards achieving this goal, but it also has several shortcomings that are detailed below. Based on these shortcomings, I recommend rejecting the paper but I would be willing to increase the score if these points were addressed in sufficient detail.\n\nMajor comments:\n\n1) Scaling\nThe authors mention in the abstract that ‘although the network size scales up, the constructed equivariant neural network does not increase the complexity of the network compared with the original one in terms of the number of the parameters.’ Based on Eq. (3.0.2) and Fig. 3, it is my understanding that n evaluations of each data point (input to a layer) are required for a cyclic group of order n. If the outputs of a layer are then concatenated, the input dimension of the subsequent layer grows by a factor of n. I would therefore argue that out-the-box application of the proposed approach does increase the number of variables dramatically and that the abstract is misleading in this respect. The authors briefly comment on this point with one sentence in the third paragraph of the introduction. However, I would appreciate if this point was addressed in more detail, for example in a dedicated paragraph after the theory is introduced. Please also address the question of whether variable sharing is essential from an equivariance point of view, or whether it’s simply a necessity to prevent an explosion of the number of parameters. Furthermore, convolutions encode translational symmetry which may be beneficial for the current application but may not be desirable for other datasets. A few comments for clarification would be very helpful. Finally, the equivarified network seems to increase the required number of computations significantly compared to the original one, which I find worth a comment .\n\n2) Experiment\nThe authors only consider a convolutional architecture and say that ‘in order to illustrate flexibility we choose not to simply equivarify the original network’. However, to me one of the main advantages of the paper seems to be that you can take this approach to equivarify any FFN. It would therefore be interesting to see this approach be applied to different networks starting with a simpler one, e.g. a 2-layer MLP. The authors could then compare the original network to the equivarified one with and without variable sharing. That would not only help the reader understand the approach better but also be much more in line with the main motivation of the paper. Then adding a second experiment, e.g. a convolutional architecture, to demonstrate flexibility would be very interesting. With regard to Fig. 4, I think there may be better ways of summarising the results than dumping 160 numbers of which only 4 seem to be of interest. The message seems to be that the network yields identical probabilities irrespective of the degree of rotation. What I find surprising is that all numbers are actually identical (shifted by 10). Is this by construction?\n\n3) Limitations\nAs indicated in the second paragraph of Sec. 4, this approach is limited to finite groups and the authors only consider image rotations w.r.t. the cyclic group of degree 4. Although I appreciate that this is meant to serve as a toy problem to illustrate that the approach works, I do not think that rotations by a constant angle are very interesting. What would be really interesting is equivariance w.r.t continuous rotations (Lie Groups), e.g. the SO(2) in this particular case. I doubt that an extension to the SO(2) is straightforward within the current theoretical framework. However, even if that is the case, I would appreciate if the authors could comment on this in a paragraph.\n\nMinor comments:\n\ni) There are many typos and grammar mistakes in the paper:\n‘any feedforward networks’ -> ‘any feedforward network’.\n‘enables to design’ -> ‘enables us to design’\n‘our proposed equivarification method use’ -> ‘our proposed equivarification method uses’\n‘traditional sense multiplication’ -> ‘traditional sense of multiplication’\n‘a group G acts’ -> ‘a group G that acts’\n‘neural network that defined on the’ -> ‘neural network that is defined on the’\n‘which achieves promising performance’ -> ‘which achieve promising performance’\n‘supplymentary material’ -> ‘supplementary material’ \nEtc.\n\nii) I think there may be a mistake in the 3rd point of Definition 3.0.3: For consistency with the previous definitions and with Fig. 1, shouldn’t F map from X to Z and \\hat F from X to \\hat Z? \n\niii) Last paragraph of Sec 3: ‘then after the identification \\hat F becomes a map from Z to...'.  Should it be ‘a map from X to ..’?\n\niv) In Definition 3.0.3 you define the tuple (\\hat Z, T, p) to be a G-equivarification, but in the paragraph below you call the G-product itself a G-equivarification (without including T and p). \n\nv) Footnote 2: You could correct for that and present the theory shifting by g instead of g^-1 to make it easier for the reader to follow. Or, at least, give a reference to the footnote earlier on in Example 4.0.1 to avoid confusion.\n\nvi) Unless there is a special reason for this, I would suggest changing the enumeration of definitions, lemmas, examples and equations, i.e. (3.0.1) -> (3.1), etc...\n\n\n*********************************************************\nI increased my rating based on the authors addressing many of my comments. \n*********************************************************\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkgdz79Mor", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper adds an interesting new perspective to equivariant neural nets. However, the actual construction looks equivalent to steerable neural nets to me (see the papers by Cohen and Welling). The generalization of steerable nets has been published under the name \"gauge equivariant neural nets\", it would be very interesting to chart out the exact connections between these concepts. \n\nThe authors mention that Z^{\\times G} is not the only possible lifting space. I believe that the general case would be Z^V where V is a representation of G. \n\nMany of the earlier papers on equivariant nets were written in the language of representation theory. It is interesting that similar nets can be constructed by purely group theoretic methods, but I really think that ultimately they are same thing. Consequently, I would expect all the experimental results to be identical.\n\nWhat would make this paper really valuable for didactic purposes is if these connections were carefully mapped out and presented with intuitive diagrams and examples.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xIzwVAcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by group action theory, this paper proposes a method to obtain ‘equivariant’ neural nets given trained one, where ‘equivariant’ refers to a network that gives identical output if certain symmetry of the dataset is performed on the input (for example, if we rotate a sample the predicted class should not change). \n\nAfter reading the paper, I don’t understand the experiments section. In particular, it is not clear to me how the proposed method differs from regular data augmentation, as to my understanding, the input to conv1 is copied 4 times and performed rotation for 0, 90, 180 and 270 degrees and the 4 times increased number of parameters (in-depth) of conv1 are shared. Furthermore, the same rotations are performed to the input to the second layer-conv 2: as the augmentation is cyclic I don’t understand why the authors perform this operation second time. Could the authors elaborate on this? After reading this section, I don’t understand the proposed fine-tuning procedure (pre-train, finetune and test): (1) what is the accuracy of the pre-trained network that was started from? (2) how is the initial network fine-tuned and modified? (as the authors mention that during training the samples are not rotated). Also, I am confused with the first sentence on page 8: ‘the complexity of the constructed network does not grow in terms of the number of parameters’. It would be useful if the results in Fig. 4 are more clearly illustrated.\n\nIs the order or increased computation 4x4x4? It would be useful to compare the method (computation & performance) with a baseline where the dataset is enlarged with data augmentation. The authors mention in the introduction that this increases training overhead, whereas the proposed practical method increases the computation at inference as well as the memory footprint of the model and the forward pass. It would be useful if the authors compared empirically with baselines with (1) data augmentation (2) network with an increased number of parameters (same as the proposed one).\n\nIn summary, the idea of using group action theory seems interesting. However after reading the paper, it is not clear to me how the idea is carried out, and although the authors provide theoretical justification, it is not clear how this connects with the practical proposed method and whether it outperforms standard data augmentation (see above). Moreover, I find the writing of the paper quite unclear (see discussion above and examples below).\n\n- If digits 6 or 9 are rotated the label changes, how does the proposed method handle this?\n- page 8, conclusion: The authors claim that the proposed approach yields a ‘significant reduction in the design and training complexity’. I don’t understand relative to what this comparison refers to, as the regular data augmentation approach is more straightforward in my opinion. Also, given that this is pointed as an important contribution, in my opinion, an empirical comparison must be done with such baselines (see above).\n- Page 1: it is mentioned that ‘the number of parameters can be the same as the original network’ but the experiments do not include such architecture. After reading the paper I don’t understand how such a network can be implemented and whether it works.\n\n— Minor —\n- Page 1 & 1par-Pg2: I don’t understand what the authors mean by ‘uniformly *across layers* of NN? \n- Page 2: In these existing works, … I don’t understand this sentence\n- Page 2: our .. method use -> uses\n- Page 2: map over the orbits. I don’t understand this\n- Page 2: the first truly equivariant NN. After reading Sec 5 I don’t understand this point.\n- Sec. 4: how to equivarifying -> equivarify\n- Page 4: ‘pick a generator’, would recommend elaborating this term or only mentioning g as an element of G for clarity for readers unfamiliar with group theory\n- What is the testing accuracy if rotated for different angles than trained (e.g. 45 degrees)?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a way to construct group equivariant neural networks from pre-trained non-equivariant networks. The equivarification is done with respect to known finite groups, and  can be done globally or layer-wise. The authors discuss their approach in the context of the image data domain. The paper is theoretically sound and proposes a novel perspective on equivarification, however, the reviewers agree that the experimental section should be strengthened and connections with other approaches (e.g. the work by Cohen and Welling) should be made clearer. The reviewers also had concerns about the computational cost of the equivarification method proposed in this paper. While the authors’ revision addressed some of the reviewers’ concerns, it was not enough to accept the paper this time round. Hence, unfortunately I recommend a rejection.
Decision: Reject