Equivariant neural networks and equivarification

1 INTRODUCTION . One key issue in deep neural network training is the difficulty of tuning parameters , especially when the network size grows larger and larger Han et al . ( 2015 ) . In order to reduce the complexity of the network , many techniques have been proposed by analyzing the structural characteristics of data , for example , sparsity Wen et al . ( 2016 ) , invariance in movement Goodfellow et al . ( 2009 ) . One of the most important structural characteristics of data is symmetry . By utilizing the translation symmetry of an object in the photo , convolutional neural network ( CNN ) ( Krizhevsky et al . ( 2012 ) ) uses shared filters to reduce the number of parameters compared with fully connected networks . However , to handle the case of rotation and reflection , people usually use the data augmentation approach to generate additional input data that has a bunch of training images with different rotation angles of the original images . In contrast to data augmentation approach , another idea is to design more sophisticated neural networks , such that the input data with certain symmetries can be trained together and applied to reduce the training complexity . Recent attempts have been made in the equivariant CNN Cohen & Welling ( 2016a ) ; Cohen et al . ( 2018 ) ; Weiler et al . ( 2018 ) . These existing works target at special cases and can not be easily generalized to arbitrary networks and arbitrary symmetries . In this paper , we take advantage of the symmetry of the data set and provide a general method to modify an arbitrary neural network so that it preserves the symmetry . The process is called equivarification . A key feature is that our equivarification method can be applied without detailed knowledge of a layer in a neural network , and hence , can be generalized to any feedforward neural networks . Another feature is that the number of parameters in the new neural network that we need to train is the same as the original one , if we equivarify the original network globally ( see the second paragraph in Section 4 for details ) . In addition , we can also output how each data instance is changed from the canonical form ( for example , in an image classification problem , additionally we can also output how many degrees an image is rotated from original upside-up image ) using the same network . We rigorously prove that our equivarification method produces truely equivariant neural networks . Practically , we equivarify a CNN as an illustration . We conduct experiments using the MNIST data set where the resulting equivariant neural network predicts the number and also the angle . If we forget the angle and keep only the number , we get an invariant neural network . 1.1 MOTIVATION . By the symmetry of the data set , we mean a group action ( see Definition 2.2 ) on the data set . Let us consider a simple cat image classification example . Then the data set can be the set of all images of a fixed size . Rotation symmetry means that we can rotate an image and the resulting image still lies in the data set . In other words , the group of rotations acts on the data set . One can build a cat classifier that assigns a number between 0 and 1 to each image indicating the probability that it is an image of a cat . If one image is a rotation ( say by 90 degree counterclockwise ) of another one , then it makes sense to require a classifier to assign the same probability to these two images . A classifier that satisfies this property is said to be invariant under 90-degree rotation , which is a special case of being equivariant under 90-degree rotation . To give an example of an ( noninvariant ) equivariant neural network , we furthermore require our classifier not only produces the probability of being a cat image , but also outputs an angle , say in { 0 , 90 , 180 , 270 } ( more precisely , the probability of each angle ) . Suppose that the equivariant classifier predicts that an image is rotated by 90 degrees , then it should predict a 270-degree rotation for the same image but rotated by 180 degrees . Not only does it make sense to require the cat classifier to be equivariant , but also it is more “ economical ” to be equivariant if implemented correctly . Roughly speaking , a regular classifier “ treats ” an image and its rotated ones as separated things , but ideally an equivariant neural network “ sees ” their connections and “ treats ” them together . From another point of view , given a regular neural network , if we apply the data augmentation method carefully , since the training data is symmetric , it is possible that after training , the network is ( approximately ) equivariant ( depending on the initialization ) . The fact that it is equivariant implies that there is now symmetry among the parameters . For example , some parameters are the same as other parameters , so there is redundancy . While in our equivariant neural network , the equivariance of our neural network is built into the structure of the neural network by sharing parameters , and in particular , it is independent of the loss function , initialization , and the data set . For instance , for the training data , it does not make any difference in results whether we prepare it by randomly rotating and recording the angles , or not rotating and labeling everything as degree 0 . In our approach , to make a neural network equivariant , at each layer other than the input layer we add a bunch of neurons ( multiplying by the order of the group ) , but we don ’ t introduce new parameters . Instead the added neurons share parameters with the original ones . 1.2 RELATED WORK . Invariance in neural networks has attracted attention for decades , aiming at designing neural networks that capture the invariant features of data , for example , face detection system at any degree of rotation in the image plane Rowley et al . ( 1998 ) , invariance for pattern recognition Barnard & Casasent ( 1991 ) , translation , rotation , and scale invariance Perantonis & Lisboa ( 1992 ) . Recently , several works have started to look into equivariant CNN , by studying the exact mapping functions of each layer Cohen & Welling ( 2016a ; b ) ; Cohen et al . ( 2018 ) ; Marcos et al . ( 2017 ) ; Lenssen et al . ( 2018 ) ; Cohen et al . ( 2019 ) , and some symmetries are studied such as translation symmetry , rotation symmetry . These methods have been used in different application domains , such as in remote sensing Marcos et al . ( 2018 ) , digital pathology Veeling et al . ( 2018 ) , galaxy morphology prediction Dieleman et al . ( 2015 ) . There are also works that aiming to construct an equivariant neural network without having to specify the symmetry Sabour et al . ( 2017 ) . As far as we know , our construction provides the first truly equivariant neural network ( See Section 5.1 ) . 2 PRELIMINARIES . In this section , we talk about some basics in group theory , like group actions , equivariance , etc . For those who would like to get a more close look at the group theory and various mathematical tools behind it , please refer to any abstract algebra books , for example , Lang ( 2002 ) . Here , we first give a couple of definitions about groups in order to help readers quickly grasp the concepts . Definition 2.1 . A group ( G , · ) consists of a set G together with a binary operation “ · ” ( which we usually call multiplication without causing confusing with the traditional sense of multiplication for real numbers ) that needs to satisfy the four following axioms . 1 ) Closure : for all a , b ∈ G , the multiplication a · b ∈ G. 2 ) Associativity : for all a , b , c ∈ G , the multiplication satisfies ( a · b ) · c = a · ( b · c ) . 3 ) Identity element : there exists a unique identity element e ∈ G , such that , for every element a ∈ G , we have e · a = a · e = a . 4 ) Inverse element : for each element a ∈ G , there exists an element b ∈ G , denoted a−1 , such that a · b = b · a = e , where e is the identity element . Note that in general , commutativity does not apply here , namely , for a , b ∈ G , a · b = b · a does not always hold true . For example , all integers with the operation addition ( Z , + ) . One can easily check that the four axioms are satisfied and 0 is the identity element . As another example , a group consists of a set { 0 , 1 } together with the operation + ( mod 2 ) where 0+1 = 1+0 = 1 and 1+1 = 0+0 = 0 . The identity element is 0 . When applying this to the image processing tasks , this can be interpreted as follows . 1 represents the action of rotating an image by 180o and 0 represents the action of not rotating an image . Then 0 + 1 represents the combination of operations that we first rotate an image by 180o and then keep it as it is , so that the final effect is to rotate an image by 180o ; while 1+1 represents that we first rotate an image by 180o and then rotate again by 180o , which is equivalent to that we do not rotate the original image . Similarly , we can define the element 1 as the operation of flipping an image vertically or horizontally , which we can give similar explanations for the group . Therefore , instead of using translations , rotations , flippings , etc. , we can use abstract groups to represent operations on images , and hence , we are able to design corresponding equivariant neural networks disregarding the operations of images ( symmetries of data ) and just following the group representation . In the following , we give a definition about group actions . Let X be a set , and G be a group . Definition 2.2 . We define a G-action on X to be a map T : G×X → X ( on the left ) such that for any x ∈ X • T ( e , x ) = x , where e ∈ G is the identity element , • for any g1 , g2 ∈ G , we have T ( g1 , T ( g2 , x ) ) = T ( g1g2 , x ) . Frequently , when there is no confusion , instead of saying that T is a G-action on X or equivalently that G acts on X via T , we simply say that G acts on X ; and T is also understood from the notation , i.e. , instead of T ( g , x ) we simply write gx , and the above formula becomes g1 ( g2x ) = ( g1g2 ) x . We say G acts trivially on X if gx = x for all g ∈ G and x ∈ X . Let X , Y be two sets , and G be a group that acts on both X and Y . Definition 2.3 . A map F : X → Y is said to be G-equivariant , if F ( gx ) = gF ( x ) for all x ∈ X and g ∈ G. Moreover , if G acts trivially on Y then we say F is G-invariant . Example 2.4 . Let X be the space of all images of 28 × 28 pixels , which contains the MNIST data set . Let G be the cyclic group of order 4 . Pick a generator g of G , and we define the action of g on X by setting gx to be the image obtained from rotating x counterclockwise by 90 degrees . Let Y be the set { 0 , 1 , 2 , ... , 9 } × { 0 , 90 , 180 , 270 } . For any y = ( num , θ ) ∈ Y we define gy : = ( num , ( θ + 90 ) mod360 ) . An equivariant neural network that classifies the number and rotation angle can be viewed as a map F from X to Y . Equivariance means if F ( x ) = ( num , θ ) then F ( gx ) = ( num , ( θ + 90 ) mod360 ) , for all x ∈ X . Thus , we can see that we can model each layer in the neural network as a group G that acts on a set X , where we can interpret the set X as input to this layer and the group action as the function mapping or operation of this layer . By abstracting the behaviors on the original input data using groups ( for example , using a same group to represent either rotation by 180o and flipping , or even more abstract operations during intermediate layers ) , we are able to apply group actions on different sets X1 , X2 , X3 , . . . ( where each one represents input to different layers ) and design similar equivariant network structures based on an original one .

Summary: The paper adds an interesting new perspective to equivariant neural nets. However, the actual construction looks equivalent to steerable neural nets to me (see the papers by Cohen and Welling). The generalization of steerable nets has been published under the name "gauge equivariant neural nets", it would be very interesting to chart out the exact connections between these concepts.