Cohen Welling bases & SO(2)-Equivariant classifiers using Tensor nonlinearity.

In this paper we propose autoencoder architectures for learning a Cohen-Welling ( CW ) -basis for images and their rotations . We use the learned CW-basis to build a rotation equivariant classifier to classify images . The autoencoder and classifier architectures use only tensor product nonlinearity . The model proposed by Cohen & Welling ( 2014 ) uses ideas from group representation theory , and extracts a basis exposing irreducible representations for images and their rotations . We give several architectures to learn CW-bases including a novel coupling AE architecture to learn a coupled CW-bases for images in different scales simultaneously . Our use of tensor product nonlinearity is inspired from recent work of Kondor ( 2018a ) . Our classifier has very good accuracy and we use fewer parameters . Even when the sample complexity to learn a good CW-basis is low we learn classifiers which perform impressively . We show that a coupled CW-bases in one scale can be deployed to classify images in a classifier trained and tested on images in a different scale with only a marginal dip in performance . 1 INTRODUCTION . A fundamental problem in vision is understanding how the human eye sees objects and images as being the same even when they undergo transformations . To obtain such a behaviour in a machine learning set-up , a natural idea is to construct representations of the object which remain the same even when the object undergoes transformations . This ’ handcrafting ’ of representations of the object invariant to transformations was the preferred method of the vision community for a long time . Since the pathbreaking achievements of modern convolutional neural networks beginning with the seminal work of Krizhevsky et al . ( 2012 ) the focus has changed - the interest is more now on developing algorithms that learn to construct these invariant representations from transformed examples . Goodfellow et al . ( 2009 ) were among the first to address the question of invariance in deep networks . Cohen & Welling ( 2014 ) were among the first to use representation theory of compact Lie groups to give it a sound mathematical framework . Developing on on ideas from earlier works of Rao & Ruderman ( 1999 ) and Sohl-Dickstein et al . ( 2010 ) on the Lie group model , Cohen and Welling build a model for the abelian group of transformations SO ( 2 ) , the group of rotations of the plane . Using their model they learn a nice basis for the underlying vector space in which the images sit , which allows them to read off the types and multiplicities of the irreducible representations of SO ( 2 ) which occur in images under rotations . In this basis exposing irreducible representations1 , the action of an element of SO ( 2 ) on an image is very easy to describe . From projections of images onto this basis,2 Cohen and Welling obtain features of images which they use for classification . A number of experts attribute the impressive performance of modern deep convolutional networks ( CNN ’ s ) to the fact that more complex and abstract features are learned as one moves down the network . Nonlinearity seems to be essential to learn such features . Deep networks achieve their nonlinearity by the use of activation functions such as RELU , Krizhevsky et al . ( 2012 ) . Convolution networks are by design invariant to translations . Given the incredible success of these deep networks the quest is now to build networks that have invariance to a larger group of symmetries . 1a phrase from Cohen & Welling ( 2014 ) 2hereafter called a CW-basis In Bruna & Mallat ( 2013 ) the authors show that the first two layers of the scattering network yield a powerful representation of images which is invariant to geometric transformations . Gens & Domingos ( 2014 ) proposed deep symmetry networks , symnets , that form feature maps over large symmetry groups . They showed that Symnets over the affine group have smaller sample complexity . More recently , Group Convolutional Networks ( GCNN ’ s ) were introduced in Cohen & Welling ( 2016a ) and Harmonic nets were introduced by Worrall et al . ( 2017 ) . Both GCNN ’ s and Harmonic nets are designed to learn representations of images invariant to a larger set of symmetries than translations . The resulting networks perform impressively - the error rate reported by GCNN ’ s on the MNIST-rot ( MNIST Variations ) data set is only 2.28 % . Harmonic nets are now the state of the art on MNIST-rot with a reported 1.62 % error . More recently Cohen & Welling ( 2016b ) introduced Steerable CNN ’ s based on more sophisticated ideas from group representation theory . This puts constraints on the network weights and architecture , and results in a reduction in the number of parameters to be learned . Steerable CNN ’ s were shown to outperform ResNets , He et al . ( 2016 ) , and achieve state of the art results on CIFAR 10,100 . In both GCNN ’ s as well in in steerable CNN ’ s equivariance to the symmetries of a group G is achieved by a generalized form of convolution . The standard convolution in CNN ’ s is replaced with the convolution of functions from G to C. Kondor & Trivedi ( 2018b ) proved the converse - they showed that any neural network which achieves equivariance with respect to the action of a group G , necessarily implements such a generalized convolution of functions on G. If one were to factor out depth , it appears that it is the nonlinearity of the activation functions used in deep networks that is responsible for their impressive performance . But nonlinear functions like RELU are mathematically hard to analyze . It would be nice to design networks which construct features , just as deep networks seem to do in their various layers , with more mathematically amenable activation functions . This is the first motivation of this study . Our other motivation comes from what is done in Pattern theory , Mumford ( 1997 ) , where to learn images a stochastic model of the images to be learned is first built . To build a stochastic model of natural images it is important to view images under different scales simultaneously ( Mumford ( 1997 , Chapter 6 ) ) . Scattering networks address this issue because they use a wavelet basis for images . This leads us to the following motivating questions . Can one use a mathematically amenable nonlinearity like tensor-product nonlinearity to design neural nets for image classification , leveraging ideas from group representation theory ? How does one obtain complex abstract features from simple features using such a nonlinearity ? Does it help to view images under different scales simultaneously to build such neural nets ? In this paper we make a modest attempt to answer such questions . We combine ideas from Cohen & Welling ( 2014 ) and Kondor ( 2018a ) , and design neural networks leveraging notions of tensor products of group representations . We decouple the classification process in two steps - first discover Cohen-Welling bases and then deploy them to train classifiers using ideas from Kondor . 2 PRELIMINARIES . We start with some definitions which set the stage for the work of Cohen & Welling ( 2014 ) . Let G be a group and let V be a finite-dimensional vector space over a field - we will always assume that the underlying field is the field of complex numbers C , or the field of real numbers R. Let GL ( V ) denote the group of invertible linear transformations of V . Definition 1 We say G acts on V if there is a group homomorphism ρ from G to GL ( V ) . So g ∈ G acts on a vector v ∈ V by sending v to ρ ( g ) v. If V has dimension d , then with respect to a basis of V , ρ ( g ) is given by a d × d matrix . The action is then given by multiplying ρ ( g ) with v. We denote this by g · v. Definition 2 We say a subspace W of V is G-invariant under the action of G if for all g ∈ G and all w ∈W , ρ ( g ) w ∈W . If V has no proper subspace which is G-invariant we say V is irreducible . The restriction of the action of G to a G-invariant subspace W is called a subrepresentation . Suppose V has an invariant subspace W , and we can also find a G-invariant subspace U such that W ⊕ U = V . Then one can choose a basis of W and of U giving us a basis of V . In this basis each ρ ( g ) will be a block diagonal matrix consisting of two blocks of size dim ( W ) and dim ( U ) . For certain groups ( called reductive groups ) there is always such a U , and the group of rotations of the plane , SO ( 2 ) , is an example of such a group . For reductive groups we can continue this process till we have a decomposition of V into irreducibles , V = U1 ⊕ U2 ⊕ . . .⊕ Uk . Definition 3 If V and W are representations of a group G , a G-morphism ( aka a G-equivariant map ) from V to W is a linear map φ from V to W such that φ ( g · v ) = g · φ ( v ) . If V andW are vector spaces over complex numbers Schur ’ s lemma , Serre ( 1977 ) , places restrictions on the dimension of the space of G-morphisms . First , if V and W are irreducible representations of G , Schur ’ s lemma states the space ofG-morphisms between them is either one dimensional ( i.e every G-morphism between them is given by a scalar matrix ) or zero dimensional ( i.e the zero morphism is the only G-morphism between them ) . This allows us to group irreducible representations into types , with two of them being of the same type when there is a nonzero G-morphism between them . Given a decomposition of a vector space with a G-action into a direct sum of irreducibles as above , we collect all irreducibles of the same type together and write V = m1St1 ⊕ m2St2 · · · ⊕ ckStk . Here tk is an indexing set for types , Sti is an irreducible G-representation of type ti , and Sti and Stj are of different types when i 6= j . We then say Sti occurs in V with multiplicity mi in the decomposition of V . Now assuming that V = ⊕i=ki=1miSti and W = ⊕ i=k i=1niSti , it follows that the dimension of the space of G-morphisms between V and W is ∑ imini . Cohen & Welling ( 2014 ) considered the problem of supervised learning of images under the group SO ( 2 ) . Viewing an N × N image as an N2 dimensional vector , an element of SO ( 2 ) acts on an N2 dimensional vector by taking the vector representing the image to the vector representing the rotated image . SO ( 2 ) is an abelian group ( rotating an image by θ1 and then by θ2 or rotating it first by θ2 and then θ1 yields the same image ) . Hence , over complex numbers , the irreducible representations of SO ( 2 ) are one-dimensional ( see , Kanatani ( 1990 ) ) . When the underlying space is a real vector space ( as was considered in Cohen & Welling ( 2014 ) ) , the types are parametrized by nonnegative integers n ( over complex numbers the irreducible representations are parametrized by all integers ) . The irreducible representation corresponding to type n = 0 , S0 is one-dimensional and is an invariant under SO ( 2 ) . For n > 0 , the irreducible representation Sn is two dimensional ( over complex numbers this splits into two one-dimensional representations parametrized by ±n ) . It follows , say , from Kanatani ( 1990 , 2.3.7 , 2.4.7 ) that there is a change of basis of the underlying vector space V with respect to which the action of the rotation group is given by a block diagonal matrixD with blocks of size 1 and 2 . It is easy to describe these blocks . Blocks of size 1 correspond to invariants , of type n = 0 , and have entry 1 . The number of such blocks is called the multiplicity of the space of invariants in V . The block corresponding to nonzero n is the elementary rotation matrix [ cos ( nθ ) − sin ( nθ ) sin ( nθ ) cos ( nθ ) ] . It follows that the number of such blocks is the multiplicity of irreducible type n in V . So there exists an N × N orthogonal matrix W and a diagonal matrix D , as above , with the property that when an image x is rotated by θ the transformed image is given by WDW tx . In Cohen & Welling ( 2014 ) , the authors address the question of determining the matrix W and the multiplicity with which a given n appears . The input is collection of pairs ( x , y ) , of an image x and it ’ s transformation y when it is rotated by an unknown angle θ . The authors find W using expectation-maximization over an interesting prior . The W their algorithm outputs is almost always a small subspace of the image space , spanning only some of the irreducible subspaces ( similar to an SVD procedure which just outputs only the top few singular vectors ) . So strictly speaking we should be calling this a CW-subbasis since it may not span the image space , but we ignore this technicality . An image x is then projected onto theW constructed and the vector of projected values , WTx , were used by Cohen and Welling as elementary features of the image for classification . Recently Kondor ( 2018a ) proposed a neural network for learning the behaviour and properties of complex many-body physical systems.The neurons in this system operate entirely in the Fourier space.The neurons compose activations following Schur ’ s lemma , thereby ensuring that the activations are covariant to rotations . We pose our motivating questions more precisely using the language of Cohen-Welling . 1 . Can we obtain a CW-basis of images and their rotations using tensor product nonlinearity ? 2 . How can elementary features obtained using a CW-basis be combined to get more complex abstract features for building good classifiers ? 3 . Can one discover CW-bases of images in two scales in tandem , with one influencing the discovery of the other ? Do such coupled bases perform better in classification ? Can one use these coupled CW-bases interchangeably ? Our contributions We give an affirmative answer to all the questions . We give several architectures to construct such bases . We use Kondor ’ s framework to construct a classifier which takes elementary features and combines them using simple ideas from the representation theory of SO ( 2 ) .3 We give a simple coupled autoencoder architecture which answers question 3 above - we use tensor product nonlinearity crucially . While such bases are not better at classification we show that they can be interchangeably used with a marginal drop in performance .

Summary: Recently there has been a spate of work on generalized CNNs that are equivariant to various symmetry groups, such a 2D and 3D rotations, the corresponding Euclidean groups (comprising not just rotations but also translations) and so on. The approach taken in most of the recent papers is to explicitly build in these equivariances by using the appropriate generalization of convolution. In the case of nontrivial groups this effectively means working in Fourier space, i.e., transforming to a basis that is adapted to the group action. This requires some considerations from represntation theory.