Equivariant neural networks and equivarification

1 INTRODUCTION . One key issue in deep neural network training is the difficulty of tuning parameters , especially when the network size grows larger and larger Han et al . ( 2015 ) . In order to reduce the complexity of the network , many techniques have been proposed by analyzing the structural characteristics of data , for example , sparsity Wen et al . ( 2016 ) , invariance in movement Goodfellow et al . ( 2009 ) . One of the most important structural characteristics of data is symmetry . By utilizing the translation symmetry of an object in the photo , convolutional neural network ( CNN ) ( Krizhevsky et al . ( 2012 ) ) uses shared filters to reduce the number of parameters compared with fully connected networks . However , to handle the case of rotation and reflection , people usually use the data augmentation approach to generate additional input data that has a bunch of training images with different rotation angles of the original images . In contrast to data augmentation approach , another idea is to design more sophisticated neural networks , such that the input data with certain symmetries can be trained together and applied to reduce the training complexity . Recent attempts have been made in the equivariant CNN Cohen & Welling ( 2016a ) ; Cohen et al . ( 2018 ) ; Weiler et al . ( 2018 ) . These existing works target at special cases and can not be easily generalized to arbitrary networks and arbitrary symmetries . In this paper , we take advantage of the symmetry of the data set and provide a general method to modify an arbitrary neural network so that it preserves the symmetry . The process is called equivarification . A key feature is that our equivarification method can be applied without detailed knowledge of a layer in a neural network , and hence , can be generalized to any feedforward neural networks . Another feature is that the number of parameters in the new neural network that we need to train is the same as the original one , if we equivarify the original network globally ( see the second paragraph in Section 4 for details ) . In addition , we can also output how each data instance is changed from the canonical form ( for example , in an image classification problem , additionally we can also output how many degrees an image is rotated from original upside-up image ) using the same network . We rigorously prove that our equivarification method produces truely equivariant neural networks . Practically , we equivarify a CNN as an illustration . We conduct experiments using the MNIST data set where the resulting equivariant neural network predicts the number and also the angle . If we forget the angle and keep only the number , we get an invariant neural network . 1.1 MOTIVATION . By the symmetry of the data set , we mean a group action ( see Definition 2.2 ) on the data set . Let us consider a simple cat image classification example . Then the data set can be the set of all images of a fixed size . Rotation symmetry means that we can rotate an image and the resulting image still lies in the data set . In other words , the group of rotations acts on the data set . One can build a cat classifier that assigns a number between 0 and 1 to each image indicating the probability that it is an image of a cat . If one image is a rotation ( say by 90 degree counterclockwise ) of another one , then it makes sense to require a classifier to assign the same probability to these two images . A classifier that satisfies this property is said to be invariant under 90-degree rotation , which is a special case of being equivariant under 90-degree rotation . To give an example of an ( noninvariant ) equivariant neural network , we furthermore require our classifier not only produces the probability of being a cat image , but also outputs an angle , say in { 0 , 90 , 180 , 270 } ( more precisely , the probability of each angle ) . Suppose that the equivariant classifier predicts that an image is rotated by 90 degrees , then it should predict a 270-degree rotation for the same image but rotated by 180 degrees . Not only does it make sense to require the cat classifier to be equivariant , but also it is more “ economical ” to be equivariant if implemented correctly . Roughly speaking , a regular classifier “ treats ” an image and its rotated ones as separated things , but ideally an equivariant neural network “ sees ” their connections and “ treats ” them together . From another point of view , given a regular neural network , if we apply the data augmentation method carefully , since the training data is symmetric , it is possible that after training , the network is ( approximately ) equivariant ( depending on the initialization ) . The fact that it is equivariant implies that there is now symmetry among the parameters . For example , some parameters are the same as other parameters , so there is redundancy . While in our equivariant neural network , the equivariance of our neural network is built into the structure of the neural network by sharing parameters , and in particular , it is independent of the loss function , initialization , and the data set . For instance , for the training data , it does not make any difference in results whether we prepare it by randomly rotating and recording the angles , or not rotating and labeling everything as degree 0 . In our approach , to make a neural network equivariant , at each layer other than the input layer we add a bunch of neurons ( multiplying by the order of the group ) , but we don ’ t introduce new parameters . Instead the added neurons share parameters with the original ones . 1.2 RELATED WORK . Invariance in neural networks has attracted attention for decades , aiming at designing neural networks that capture the invariant features of data , for example , face detection system at any degree of rotation in the image plane Rowley et al . ( 1998 ) , invariance for pattern recognition Barnard & Casasent ( 1991 ) , translation , rotation , and scale invariance Perantonis & Lisboa ( 1992 ) . Recently , several works have started to look into equivariant CNN , by studying the exact mapping functions of each layer Cohen & Welling ( 2016a ; b ) ; Cohen et al . ( 2018 ) ; Marcos et al . ( 2017 ) ; Lenssen et al . ( 2018 ) ; Cohen et al . ( 2019 ) , and some symmetries are studied such as translation symmetry , rotation symmetry . These methods have been used in different application domains , such as in remote sensing Marcos et al . ( 2018 ) , digital pathology Veeling et al . ( 2018 ) , galaxy morphology prediction Dieleman et al . ( 2015 ) . There are also works that aiming to construct an equivariant neural network without having to specify the symmetry Sabour et al . ( 2017 ) . As far as we know , our construction provides the first truly equivariant neural network ( See Section 5.1 ) . 2 PRELIMINARIES . In this section , we talk about some basics in group theory , like group actions , equivariance , etc . For those who would like to get a more close look at the group theory and various mathematical tools behind it , please refer to any abstract algebra books , for example , Lang ( 2002 ) . Here , we first give a couple of definitions about groups in order to help readers quickly grasp the concepts . Definition 2.1 . A group ( G , · ) consists of a set G together with a binary operation “ · ” ( which we usually call multiplication without causing confusing with the traditional sense of multiplication for real numbers ) that needs to satisfy the four following axioms . 1 ) Closure : for all a , b ∈ G , the multiplication a · b ∈ G. 2 ) Associativity : for all a , b , c ∈ G , the multiplication satisfies ( a · b ) · c = a · ( b · c ) . 3 ) Identity element : there exists a unique identity element e ∈ G , such that , for every element a ∈ G , we have e · a = a · e = a . 4 ) Inverse element : for each element a ∈ G , there exists an element b ∈ G , denoted a−1 , such that a · b = b · a = e , where e is the identity element . Note that in general , commutativity does not apply here , namely , for a , b ∈ G , a · b = b · a does not always hold true . For example , all integers with the operation addition ( Z , + ) . One can easily check that the four axioms are satisfied and 0 is the identity element . As another example , a group consists of a set { 0 , 1 } together with the operation + ( mod 2 ) where 0+1 = 1+0 = 1 and 1+1 = 0+0 = 0 . The identity element is 0 . When applying this to the image processing tasks , this can be interpreted as follows . 1 represents the action of rotating an image by 180o and 0 represents the action of not rotating an image . Then 0 + 1 represents the combination of operations that we first rotate an image by 180o and then keep it as it is , so that the final effect is to rotate an image by 180o ; while 1+1 represents that we first rotate an image by 180o and then rotate again by 180o , which is equivalent to that we do not rotate the original image . Similarly , we can define the element 1 as the operation of flipping an image vertically or horizontally , which we can give similar explanations for the group . Therefore , instead of using translations , rotations , flippings , etc. , we can use abstract groups to represent operations on images , and hence , we are able to design corresponding equivariant neural networks disregarding the operations of images ( symmetries of data ) and just following the group representation . In the following , we give a definition about group actions . Let X be a set , and G be a group . Definition 2.2 . We define a G-action on X to be a map T : G×X → X ( on the left ) such that for any x ∈ X • T ( e , x ) = x , where e ∈ G is the identity element , • for any g1 , g2 ∈ G , we have T ( g1 , T ( g2 , x ) ) = T ( g1g2 , x ) . Frequently , when there is no confusion , instead of saying that T is a G-action on X or equivalently that G acts on X via T , we simply say that G acts on X ; and T is also understood from the notation , i.e. , instead of T ( g , x ) we simply write gx , and the above formula becomes g1 ( g2x ) = ( g1g2 ) x . We say G acts trivially on X if gx = x for all g ∈ G and x ∈ X . Let X , Y be two sets , and G be a group that acts on both X and Y . Definition 2.3 . A map F : X → Y is said to be G-equivariant , if F ( gx ) = gF ( x ) for all x ∈ X and g ∈ G. Moreover , if G acts trivially on Y then we say F is G-invariant . Example 2.4 . Let X be the space of all images of 28 × 28 pixels , which contains the MNIST data set . Let G be the cyclic group of order 4 . Pick a generator g of G , and we define the action of g on X by setting gx to be the image obtained from rotating x counterclockwise by 90 degrees . Let Y be the set { 0 , 1 , 2 , ... , 9 } × { 0 , 90 , 180 , 270 } . For any y = ( num , θ ) ∈ Y we define gy : = ( num , ( θ + 90 ) mod360 ) . An equivariant neural network that classifies the number and rotation angle can be viewed as a map F from X to Y . Equivariance means if F ( x ) = ( num , θ ) then F ( gx ) = ( num , ( θ + 90 ) mod360 ) , for all x ∈ X . Thus , we can see that we can model each layer in the neural network as a group G that acts on a set X , where we can interpret the set X as input to this layer and the group action as the function mapping or operation of this layer . By abstracting the behaviors on the original input data using groups ( for example , using a same group to represent either rotation by 180o and flipping , or even more abstract operations during intermediate layers ) , we are able to apply group actions on different sets X1 , X2 , X3 , . . . ( where each one represents input to different layers ) and design similar equivariant network structures based on an original one .

Summary: In this work, the authors employ concepts from group theory to turn an arbitrary feed forward neural network into an equivariant one, i.e. a network whose output transforms in a way that is consistent with the transformation of the input. To this end, the authors first introduce the basic concepts of group theory required to follow their work and provide a comprehensive definition of equivariance. They then explain how to equivarify (w.r.t. a finite group G) a given neural network, and present experimental results on rotated MNIST digits to support their approach.