Year: 2018

Abstract:
In this paper we propose autoencoder architectures for learning a Cohen-Welling (CW)-basis for images and their rotations. We use the learned CW-basis to build a rotation equivariant classifier to classify images. The autoencoder and classifier architectures use only tensor product nonlinearity. The model proposed by Cohen & Welling (2014) uses ideas from group representation theory, and extracts a basis exposing irreducible representations for images and their rotations. We give several architectures to learn CW-bases including a novel coupling AE architecture to learn a coupled CW-bases for images in different scales simultaneously. Our use of tensor product nonlinearity is inspired from recent work of Kondor (2018a). Our classifier has very good accuracy and we use fewer parameters. Even when the sample complexity to learn a good CW-basis is low we learn classifiers which perform impressively. We show that a coupled CW-bases in one scale can be deployed to classify images in a classifier trained and tested on images in a different scale with only a marginal dip in performance.

text: In this paper we propose autoencoder architectures for learning a Cohen-Welling (CW)-basis for images and their rotations. We use the learned CW-basis to build a rotation equivariant classifier to classify images. The autoencoder and classifier architectures use only tensor product nonlinearity. The model proposed by Cohen & Welling (2014) uses ideas from group representation theory, and extracts a basis exposing irreducible representations for images and their rotations. We give several architectures to learn CW-bases including a novel coupling AE architecture to learn a coupled CW-bases for images in different scales simultaneously. Our use of tensor product nonlinearity is inspired from recent work of Kondor (2018a). Our classifier has very good accuracy and we use fewer parameters. Even when the sample complexity to learn a good CW-basis is low we learn classifiers which perform impressively. We show that a coupled CW-bases in one scale can be deployed to classify images in a classifier trained and tested on images in a different scale with only a marginal dip in performance.

heading: 1 INTRODUCTION
text: A fundamental problem in vision is understanding how the human eye sees objects and images as being the same even when they undergo transformations. To obtain such a behaviour in a machine learning set-up, a natural idea is to construct representations of the object which remain the same even when the object undergoes transformations. This ’handcrafting’ of representations of the object invariant to transformations was the preferred method of the vision community for a long time. Since the pathbreaking achievements of modern convolutional neural networks beginning with the seminal work of Krizhevsky et al. (2012) the focus has changed - the interest is more now on developing algorithms that learn to construct these invariant representations from transformed examples.
Goodfellow et al. (2009) were among the first to address the question of invariance in deep networks. Cohen & Welling (2014) were among the first to use representation theory of compact Lie groups to give it a sound mathematical framework. Developing on on ideas from earlier works of Rao & Ruderman (1999) and Sohl-Dickstein et al. (2010) on the Lie group model, Cohen and Welling build a model for the abelian group of transformations SO(2), the group of rotations of the plane. Using their model they learn a nice basis for the underlying vector space in which the images sit, which allows them to read off the types and multiplicities of the irreducible representations of SO(2) which occur in images under rotations. In this basis exposing irreducible representations1, the action of an element of SO(2) on an image is very easy to describe. From projections of images onto this basis,2 Cohen and Welling obtain features of images which they use for classification.
A number of experts attribute the impressive performance of modern deep convolutional networks (CNN’s) to the fact that more complex and abstract features are learned as one moves down the network. Nonlinearity seems to be essential to learn such features. Deep networks achieve their nonlinearity by the use of activation functions such as RELU, Krizhevsky et al. (2012). Convolution networks are by design invariant to translations. Given the incredible success of these deep networks the quest is now to build networks that have invariance to a larger group of symmetries.
1a phrase from Cohen & Welling (2014) 2hereafter called a CW-basis
In Bruna & Mallat (2013) the authors show that the first two layers of the scattering network yield a powerful representation of images which is invariant to geometric transformations. Gens & Domingos (2014) proposed deep symmetry networks, symnets, that form feature maps over large symmetry groups. They showed that Symnets over the affine group have smaller sample complexity. More recently, Group Convolutional Networks (GCNN’s) were introduced in Cohen & Welling (2016a) and Harmonic nets were introduced by Worrall et al. (2017). Both GCNN’s and Harmonic nets are designed to learn representations of images invariant to a larger set of symmetries than translations. The resulting networks perform impressively - the error rate reported by GCNN’s on the MNIST-rot(MNIST Variations) data set is only 2.28%. Harmonic nets are now the state of the art on MNIST-rot with a reported 1.62% error.
More recently Cohen & Welling (2016b) introduced Steerable CNN’s based on more sophisticated ideas from group representation theory. This puts constraints on the network weights and architecture, and results in a reduction in the number of parameters to be learned. Steerable CNN’s were shown to outperform ResNets, He et al. (2016), and achieve state of the art results on CIFAR 10,100.
In both GCNN’s as well in in steerable CNN’s equivariance to the symmetries of a group G is achieved by a generalized form of convolution. The standard convolution in CNN’s is replaced with the convolution of functions from G to C. Kondor & Trivedi (2018b) proved the converse - they showed that any neural network which achieves equivariance with respect to the action of a group G, necessarily implements such a generalized convolution of functions on G.
If one were to factor out depth, it appears that it is the nonlinearity of the activation functions used in deep networks that is responsible for their impressive performance. But nonlinear functions like RELU are mathematically hard to analyze. It would be nice to design networks which construct features, just as deep networks seem to do in their various layers, with more mathematically amenable activation functions. This is the first motivation of this study.
Our other motivation comes from what is done in Pattern theory, Mumford (1997), where to learn images a stochastic model of the images to be learned is first built. To build a stochastic model of natural images it is important to view images under different scales simultaneously (Mumford (1997, Chapter 6)). Scattering networks address this issue because they use a wavelet basis for images.
This leads us to the following motivating questions. Can one use a mathematically amenable nonlinearity like tensor-product nonlinearity to design neural nets for image classification, leveraging ideas from group representation theory? How does one obtain complex abstract features from simple features using such a nonlinearity? Does it help to view images under different scales simultaneously to build such neural nets?
In this paper we make a modest attempt to answer such questions. We combine ideas from Cohen & Welling (2014) and Kondor (2018a), and design neural networks leveraging notions of tensor products of group representations. We decouple the classification process in two steps - first discover Cohen-Welling bases and then deploy them to train classifiers using ideas from Kondor.

heading: 2 PRELIMINARIES
text: We start with some definitions which set the stage for the work of Cohen & Welling (2014). Let G be a group and let V be a finite-dimensional vector space over a field - we will always assume that the underlying field is the field of complex numbers C, or the field of real numbers R. Let GL(V ) denote the group of invertible linear transformations of V .
Definition 1 We say G acts on V if there is a group homomorphism ρ from G to GL(V ).
So g ∈ G acts on a vector v ∈ V by sending v to ρ(g)v. If V has dimension d, then with respect to a basis of V , ρ(g) is given by a d × d matrix. The action is then given by multiplying ρ(g) with v. We denote this by g · v.
Definition 2 We say a subspace W of V is G-invariant under the action of G if for all g ∈ G and all w ∈W , ρ(g)w ∈W . If V has no proper subspace which is G-invariant we say V is irreducible. The restriction of the action of G to a G-invariant subspace W is called a subrepresentation.
Suppose V has an invariant subspace W , and we can also find a G-invariant subspace U such that W ⊕ U = V . Then one can choose a basis of W and of U giving us a basis of V . In this basis each ρ(g) will be a block diagonal matrix consisting of two blocks of size dim(W ) and dim(U). For certain groups (called reductive groups) there is always such a U , and the group of rotations of the plane, SO(2), is an example of such a group. For reductive groups we can continue this process till we have a decomposition of V into irreducibles, V = U1 ⊕ U2 ⊕ . . .⊕ Uk.
Definition 3 If V and W are representations of a group G, a G-morphism (aka a G-equivariant map) from V to W is a linear map φ from V to W such that φ(g · v) = g · φ(v).
If V andW are vector spaces over complex numbers Schur’s lemma, Serre (1977), places restrictions on the dimension of the space of G-morphisms. First, if V and W are irreducible representations of G, Schur’s lemma states the space ofG-morphisms between them is either one dimensional (i.e every G-morphism between them is given by a scalar matrix) or zero dimensional (i.e the zero morphism is the only G-morphism between them). This allows us to group irreducible representations into types, with two of them being of the same type when there is a nonzero G-morphism between them.
Given a decomposition of a vector space with a G-action into a direct sum of irreducibles as above, we collect all irreducibles of the same type together and write V = m1St1 ⊕ m2St2 · · · ⊕ ckStk. Here tk is an indexing set for types, Sti is an irreducible G-representation of type ti, and Sti and Stj are of different types when i 6= j. We then say Sti occurs in V with multiplicity mi in the decomposition of V . Now assuming that V = ⊕i=ki=1miSti and W = ⊕ i=k i=1niSti, it follows that the dimension of the space of G-morphisms between V and W is ∑
imini.
Cohen & Welling (2014) considered the problem of supervised learning of images under the group SO(2). Viewing an N × N image as an N2 dimensional vector, an element of SO(2) acts on an N2 dimensional vector by taking the vector representing the image to the vector representing the rotated image. SO(2) is an abelian group (rotating an image by θ1 and then by θ2 or rotating it first by θ2 and then θ1 yields the same image). Hence, over complex numbers, the irreducible representations of SO(2) are one-dimensional (see, Kanatani (1990)). When the underlying space is a real vector space (as was considered in Cohen & Welling (2014)), the types are parametrized by nonnegative integers n (over complex numbers the irreducible representations are parametrized by all integers). The irreducible representation corresponding to type n = 0, S0 is one-dimensional and is an invariant under SO(2). For n > 0, the irreducible representation Sn is two dimensional (over complex numbers this splits into two one-dimensional representations parametrized by ±n).
It follows, say, from Kanatani (1990, 2.3.7, 2.4.7) that there is a change of basis of the underlying vector space V with respect to which the action of the rotation group is given by a block diagonal matrixD with blocks of size 1 and 2. It is easy to describe these blocks. Blocks of size 1 correspond to invariants, of type n = 0, and have entry 1. The number of such blocks is called the multiplicity of the space of invariants in V . The block corresponding to nonzero n is the elementary rotation matrix [
cos(nθ) − sin(nθ) sin(nθ) cos(nθ)
]
. It follows that the number of such blocks is the multiplicity of irreducible
type n in V .
So there exists an N × N orthogonal matrix W and a diagonal matrix D, as above, with the property that when an image x is rotated by θ the transformed image is given by WDW tx. In Cohen & Welling (2014), the authors address the question of determining the matrix W and the multiplicity with which a given n appears. The input is collection of pairs (x, y), of an image x and it’s transformation y when it is rotated by an unknown angle θ. The authors find W using expectation-maximization over an interesting prior. The W their algorithm outputs is almost always a small subspace of the image space, spanning only some of the irreducible subspaces (similar to an SVD procedure which just outputs only the top few singular vectors). So strictly speaking we should be calling this a CW-subbasis since it may not span the image space, but we ignore this technicality.
An image x is then projected onto theW constructed and the vector of projected values, WTx, were used by Cohen and Welling as elementary features of the image for classification.
Recently Kondor (2018a) proposed a neural network for learning the behaviour and properties of complex many-body physical systems.The neurons in this system operate entirely in the Fourier space.The neurons compose activations following Schur’s lemma, thereby ensuring that the activations are covariant to rotations.
We pose our motivating questions more precisely using the language of Cohen-Welling.
1. Can we obtain a CW-basis of images and their rotations using tensor product nonlinearity?
2. How can elementary features obtained using a CW-basis be combined to get more complex abstract features for building good classifiers?
3. Can one discover CW-bases of images in two scales in tandem, with one influencing the discovery of the other? Do such coupled bases perform better in classification? Can one use these coupled CW-bases interchangeably?
Our contributions We give an affirmative answer to all the questions. We give several architectures to construct such bases. We use Kondor’s framework to construct a classifier which takes elementary features and combines them using simple ideas from the representation theory of SO(2).3 We give a simple coupled autoencoder architecture which answers question 3 above - we use tensor product nonlinearity crucially. While such bases are not better at classification we show that they can be interchangeably used with a marginal drop in performance.

heading: 2.1 NOTIONS FROM REPRESENTATION THEORY
text: To describe our set up and explain our experiments we will need a few more notions from representation theory. Throughout we will assume that the underlying group G is a reductive group, in fact the reader may think of G as SO(2).
If V is a representation of a group G then the dual vector space V ∗ acquires a natural G action given by (g · f)(v) = f(g−1v), for f ∈ V ∗, v ∈ V . If V,U are representations of a reductive group then V ⊗U is a representation of the product group G×G. Furthermore, since G is a subgroup of G×G under the diagonal embedding (g mapping to (g, g)), V ⊗ U is a representation of G. And by what we described above V ⊗ U splits into a direct sum of irreducible representations of G identified by their types and multiplicities. It also follows that the tensor algebra, T (V ) = ⊕iV
⊗i of V , is an (infinite dimensional, algebraic) representation of G.
Remark 4 When G is SO(2) and 0 6= i < j then Si ⊗ Sj splits into a direct sum Si+j ⊕ Sj−i. When i = 0, Si ⊗ Sj is isomorphic to Sj , (Kanatani (1990)).
To make precise the question about coupled bases and features from a coupled bases we need one more definition.
Definition 5 We say G-representations V and W are coupled if W is the image of a G-morphism of a finite dimensional subrepresentation of the tensor algebra of V , and V is the image of a Gmorphism of a finite dimensional subrepresentation of the tensor algebra of W .
The above definition is equivalent to requiring that W can be realized as a subrepresentation (or a quotient) of the tensor algebra of V and V can realized as a subrepresentation (or a quotient) of the tensor algebra of W .
Definition 6 Let V be the vector space of 14× 14 images and let W be the vector space of 28× 28 images. Assume the rotation group SO(2) acts on both. We say a Cohen-Welling basis X of V is coupled to a Cohen-Welling basis Y of W if the vector space dual of the subspace spanned by X (with its SO(2)-action) is coupled to the vector space dual of the subspace spanned by Y .
Recall that by projecting an image onto a CW-basis we get elementary features of the image which are used for classification. These elementary features are elements of the vector space dual of the space spanned by the CW-basis, hence the need for the word dual in the above definition. Motivated by the above definition we say features obtained from coupled CW-bases are coupled features.
The definition above is motivated by our question of whether one can build CW-bases in two different scales with one influencing the discovery of the other and vice-versa. Our definition suggests
3As we were getting this write up ready we were pointed to some recent work of Kondor et al. (2018c) - the authors do similar work, but use the rotation group SO(3) - for completeness we have included their results in Table 1.
that in order to generate coupled CW-bases we will need to view the images at different scales simultaneously, and use tensor product nonlinearity to generate them, thereby forcing the influence we are looking for.
Remark 7 In the appendix we describe our problem and solution in a language that is more familiar to the ML community - as that of seeking filters in the Fourier space of images.
Remark 8 One last remark is in order before we describe our setup and experiments. We are working over the field of real numbers, so Schur’s lemma does not apply in our situation. Nevertheless when we look for SO(2)-morphisms between two irreducible vector spaces with an SO(2)-action we will assume that the morphisms are of the kind Schur’s lemma dictates - either we have no morphism or the morphism is given by a scalar matrix rId, r a real number and Id being the identity matrix.

heading: 3 EXPERIMENTS AND RESULTS
text: We use the notation Wℓ for a CW-basis of images of size ℓ × ℓ. So Wℓ is a ℓ 2 × d matrix for some d (usually smaller than l2). Then, rotation of an ℓ × ℓ image x by an angle θ is the ℓ2-size vector WℓDℓ(θ)W T ℓ x, where Dℓ is the block diagonal matrix described in section 2, of size d × d. If a column of Wℓ represents a basis vector which is invariant, the corresponding block is 1, and if a pair of columns in Wℓ is a basis for an irreducible representation of type n, the block in Dℓ for this pair is the elementary rotation matrix described in section 2. If there is no confusion we ignore the subscripts. We give three neural net architectures to discover CW-bases of images.

heading: 3.1 LEARNING CW-BASES
text: 1. An autoencoder(AE) architecture. The autoencoder(AE) architecture is given as Alogrithm 1. An equivalent block diagram is given in the Appendix (see figure 6). Details of computing Zi below are given in Appendix, Section E.
Algorithm 1: AutoEncoder
Input: Xi ∈ R 784, i ∈ {1, 2, . . . } Hyper-parameters :aj , bj , β Let da = a0 + ∑ j=1 2× aj ; db = b0 + ∑
j=1 2× bj Output: W28 ∈ R 784×da Output: φ ∈ Rdb×da // Equivariant linear maps (Block diagonal Output: ψ ∈ Rda×(db+db 2) // matrices) as dictated by Remark 8
for i = 1, 2, . . . do θi ∈ U [0, 360) // Choosen uniformly at random Yi = Rotate (Xi, θi) // Rotate is the standard image rotation
Ŷi =W T 28Yi Zi = ψ(φ(Ŷi)⊗ φ(Ŷi)⊕ φ(Ŷi))
X̂i =W28D(−θi)Zi // Unrotate and reconstruct the image end L = ∑
i |Xi − X̂i| 2 2 // Loss function
W28, φ, ψ = min W28,φ,ψ
(L+ β|WT28W28 − Idda×da |1)
return W28, φ, ψ
2. Coupled Autoencoder(CAE) architecture. In this setup, Figure 1, we learn W14 and W28 in tandem. We feed both, an image X and a scaled down version of the image x to the network.
The network on top takes x and produces X̂ a 28x28 image. The bottom network takes X and produces x̂ a 14x14 image. The two networks are connected and we use the same W28, and W14 in the top and bottom layer. Hyperparameters are multiplicities of SO(2)-irreducible representations in W28 and W14. We learn the W14, W28 and two SO(2)-equivariant maps φ and ψ. The bottom network takes X , rotates it by θ and projects the resulting Y on the current
W28, to get Ŷ = W T 28Y . Applying φ we get φ(Ŷ ). We unrotate φ(Ŷ ) using D14(−θ) to get z. We generate a 14x14 image using the current W14 to get x̂ = W14z. The top network takes x and rotates it by the same θ. This is projected onto the current W14 to get ŷ. We apply ψ to ŷ ⊕ (ŷ ⊗ ŷ) and unrotate the resulting vector using D28(−θ) to get Z. We use the current W28 to get X̂ =W28Z. We minimize the reconstruction error |X − X̂| 2 + |x− x̂|2.
3 Bootstrapping W14 to W28, W28 to W14. The appendix describes an architecture in which we bootstrap, starting with a W14 learned in experiment 1 or 2 and learn a W28.

heading: 3.2 CLASSIFICATION USING THE LEARNED CW-BASIS
text: Figure 2 shows a 2 layer classification network for 28 × 28 images - this can be extended to any number of layers, and other image sizes. This is a logical extension of the auto encoder network. It is a CNN in Fourier domain using tensor product nonlinearity as in Thomas et al. (2018). Assume the dimension of W28 is dW . We project the image x onto W28 to get W T 28x. Let L1 be a vector space with an SO(2)-action - hyperparameters are multiplicities l1j of irreducibles SO(2)-representations Sj in L1. We learn an SO(2)-equivariant map φ1 from the image space of W T 28 to L1. The SO(2)decomposition of l1 = φ1(W T 28x)⊕ (φ1(W T 28x)⊗ φ1(W T 28x)) is computed. This is the input to the next layer. There we choose a vector space L2 with a SO(2)-action which splits into irreducibles Sj with multiplicities l2j (hyperparameters). We learn a second SO(2)-equivariant map φ2 from L1 ⊕ (L1 ⊗L1) to L2 and compute φ2(l1). We compute the tensor product of φ2(l1) with itself and produce l2 = φ2(l1) ⊕ (φ2(l1) ⊗ φ2(l1)). This is the input to the next layer. So we learn as many SO(2)-equivariant maps as there are layers. In the final layer, lf is projected onto the invariant space (the multiplicity of S0 in Lf ⊕ (Lf ⊗ Lf ) and fed to a soft max classifier.

heading: 3.3 RESULTS

heading: 1. Construction of CW-basis
text: We construct CW-basis W14 for 14 × 14 images and CW-basis W28 for 28 × 28 in the AE and CAE architectures. We evaluate these bases in terms of image reconstruction error (MSE). We also used rotation reconstruction error by comparing with scikits-image
rotation. The data set for which these errors are reported is MNIST-rot. We choose [10, 5, 5, 5, 5, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1] as our multiplicities for the ai’s (hyperparameters) and [8, 4, 4, 4, 4, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1] for the bi’s (hyperparameters). Figure 3 and Figure 4 shows graphs of the error functions for W28 constructed in AE, CAE architectures as a function of the number of samples given to learn the W28. Even when the number of samples is as small as 50, we discover CW-bases which are good at rotation and reconstruction in both architectures. In Figure 7, in the Appendix we visualize a few basis elements learned.
2. Results on Classification, MNIST In Figure 5 we plot the accuracy of these various W28 obtained above when deployed for classification. We plot the accuracy of the classifier as a function of the number of samples used to construct the CW-bases. When the number of samples is as low as 50 a CAE-W28 performs better than an AE-W28. Beyond 100 samples the difference is insignificant. Plots using a bootstrapping CW-basis are similar to that of the AE and omitted.
In Table 1 we give the mean accuracy(and stdev) of our classifier when trained and tested on combinations using MNIST(NR) and MNIST-rot(R). The R/NR column for example denotes the accuracy when trained on MNIST-rot and tested on MNIST (LeCun et al.). For comparison we use Planar, Spherical CNN accuracies given in Cohen et al. (2018). We also compare with FFS2CNN, (see Kondor et al. (2018c)).
The fourth row shows the accuracy of our classifier that used a CAE-W28. This was obtained using CAE architecture which was given 12000 training samples of MNIST-rot to discover the W28. Our classifier performs better in all the scenarios.
Coupling interchangeability To test how coupled the CAE-W28 and CAE-W14 are, we did the following experiment - the classifier trained in row 4 with CAE-W28 was presented with down sized 14x14 images for classification. No additional training was done. Instead we use the top half of the coupling network from Figure 1 - given a test image y we compute ŷ = WT14y using the CAE-W14. Then ψ((ŷ ⊗ ŷ) ⊕ ŷ) is fed to the trained classifier of row 4. These results are reported in row 5 as [28/14 Tensor].
For comparison we took the 14x14 images and scaled them up to 28x28 and then fed them to the trained classifier of row 4. These results are reported in Row 6 as [28/14 Scale]. Using CAE-W14 is better, indicating that a coupled bases retains scale information well.
We repeated the same experiment when the coupled network is given only 500 samples to learn the CAE-W28 and CAE-W14. As row 8 shows, performance is still very good.
3. Results on Classification, Fashion-MNIST. In Table 2 we report mean classification accuracy(and stdev) on the Fashion-MNIST data set(Xiao et al. (2017)). We rotate each data point around the origin by an angle chosen uniformly between 0 and 2π to create F-MNIST-rot. The CW basis was learned in the AE architecture with Fashion-MNIST as input. The classifier is a four layer network (96K parameters). We compared with a Depth 5 CNN (102K parameters).

heading: 3.4 IMPLEMENTATION
text: Our autoencoders and classifiers were implemented as neural nets in Python (TensorFlow). We used the Adam optimizer.

heading: 4 CONCLUSION AND FUTURE WORK
text: We answer in the affirmative all our motivating questions - we design architectures which learn CW-bases and classify using only tensor product nonlinearity. We obtain some state of the art results. It appears that coupling bases do retain scale information and can be interchangeably used for classification albeit with a small drop in performance. There is no appreciable advantage in using coupled bases for classification excepting when the bases are learned using small samples of inputs.
Tensor product nonlinearity allows us to decouple the classification problem in two steps. One advantage of decoupling the process is that the basis can be obtained offline. Our ideas can be extended to other groups. We have begun experiments with the action of the Sn on images.
We are testing our architectures on CIFAR-10. One issue we run into when using architectures with a lot of depth, is that accuracy increases and then drops. A similar issue is resolved in RES-nets He et al. (2016) by computing residual functions. In ongoing work we proceed similarly since the residual function is SO(2)-equivariant.
Robustness of classifiers to noise is an active area of research, (see Goodfellow et al. (2018)). We believe our classifiers built from bases learnt in a CAE architecture should be robust to noise - our belief comes from the fact that coupling bases can be used interchangeably. We plan to investigate this.

heading: ACKNOWLEDGEMENTS
text: The authors would like to thank Infosys foundation for a generous grant.

heading: A CONVOLUTION IN THE FOURIER DOMAIN
text: A CNN is equivariant to translations - i.e, whether we translate the image and convolve it with a filter or we convolve the image first and then translate it we get the same result. Thus the filters are naturally equivariant to the group of translations. We would like obtain such an equivariance with respect to the group of rotations.
To do so it is easier to work in the Fourier space of the group SO(2). However a vector space basis in the Fourier space is not readily available - (this is what we call CW basis - Cohen and Welling compute this in their paper). Now every CW basis vector comes indexed by a non-negative integer (Paragraph 5 on Page 3). There could be multiple basis vectors indexed with the same integer which is called the multiplicity (Paragraph 6 of page 3). So in the Fourier space an image is a linear combination of CW basis vectors with coefficients (which of course depend upon the image). Lets call them CW coefficients. Denoting the CW basis by W , x̂ = WTx are the CW-coefficents of x. We first compute the CW basis given a reasonable number of samples of images x and their rotations. The CW basis does not necessarily span the entire Fourier space, we need to find enough basis vectors which give a good approximation (Paragraph 7, page 3).
Convolution of an image with a SO(2)-equivariant filter means that whether we rotate an image and then convolve, or first convolve and then rotate we should end up transforming the image in the same way. This translates to the following in the Fourier domain - taking linear combinations of CW coefficients of CW-basis vectors of the same type m, thereby getting an equivariant linear map. The entries of this linear map L is what we seek to find. So if we have a space V with a CW basis having mi basis vectors indexed by integer i and another space W with ni basis vectors indexed by integers i, the search space for SO(2)-equivariant filters in the Fourier domain has dimension ∑
imini, corresponding to block diagonal matrices, the i-th block being of size nimi (see paragraph 4 on page 3).
The natural nonlinearity in the Fourier space is multiplication of CW coefficients. When we multiply the coefficient of a basis vector of type m and the coefficient of a basis vector of type n, we get two coefficients, for basis vectors of type m+n and m-n (content of Remark 4). These are quadratic functions, of the starting CW coefficients - since this nonlinearity is obtained by taking tensor products of irreducible representations, we call it tensor product nonlinearity.
Our learning and classification takes place in the Fourier world.

heading: B PROOFS
text: Here we prove that the bases obtained in the coupling architecture satisfy the definition of coupling. The proof is straightforward assuming that we have zero loss.
Let W28 and W14 be the final CW-bases learned. Following the bottom half of the network describing the coupling architecture, we see that under the morphism φ, the image space of WT28 surjects onto the image space of WT14. Surjection follows since the dotted arrow going from W14 on the top to W14 below shows that we can start with any linear combination of the basis elements of W14 at the top and obtain it as the image of φ applied to an appropriate linear combination of basis elements of W28. So the image space of W T 14 is isomorphic to a quotient of the image space of W T 28. On the other hand following the top half of the network it is clear that ψ maps the image space of WT14 ⊕ (W T 14 ⊗W T 14) surjectively onto the image space of W T 28. Surjectivity follows by reasoning as above. Hence the bases obtained in the coupling architecture satisfy definition 6.

heading: C LEARNING CW-BASIS
text: 1. AutoEncoder(AE) architecture We describe this using figure Figure 6 (this architecture works for 14x14 images, 56x56 images). The hyperparameters are multiplicities with which SO(2) irreducible representations occur in the images viewed as vectors in a 784 dimensional vector space with an action of SO(2). We choose nonzero multiplicities aj for representations Sj with
0 ≤ j ≤ 24 giving us a subspace of the image space of dimension dW = a0 + ∑
j 2 ∗ aj . We denote this subspace by W28 and learn a CW-basis also denoted W28, a 784 × dW matrix. We initialize W28 randomly. We rotate a 28 x 28 image x by a random angle θ, to get y and project y onto W28 to get ŷ = W T 28y. We learn an SO(2)-equivariant map φ from the image space of WT28 to a vector space L with an SO(2)-action, with multiplicities (hyperparameters) bj . Consider φ(ŷ) ∈ L. Now L⊗L gets a natural SO(2) action. Recall that the tensor product of an SO(2) irreducible representation of typem and an SO(2)-irreducible representation of type type n yields irreducible representations of types m+ n and m− n, (w.l.o.g m > n). We use this to compute the decomposition of φ(ŷ)⊗φ(ŷ). The last set of parameters to be learned are an SO(2)equivariant map ψ from L⊕(L⊗L) to the image space ofWT28. Let z = ψ(φ(ŷ)⊕(φ(ŷ)⊗φ(ŷ))). We unrotate z using D(−θ). Then we reconstruct the image x̂ =W28D(−θ)z We expect this to be close to x so we minimize the reconstruction error |x̂ − x|2 with a regularizer to ensure that WT28W28 is identity.
2. Bootstrapping architecture
In this experiment we start with a known W14 obtained using experiment 1 for 14x14 images, to learn aW28. The setup to go fromW14 toW28 is exactly the part of the network in Figure 1 going from X to X̂ along the top. The hyperparameters are the multiplicities of SO(2)-irreducible representations in W28. We learn the SO(2)-equivariant ψ and W28 by minimizing |X − X̂| 2. Similarly starting with a known W28 we can learn W14 by following the path in Figure 1 from x to x̂ along the bottom.
D VISUALIZATION OF CW BASIS
Figure 7 shows a few basis elements learned using AE architucture on MNIST-rot.

heading: E EXAMPLE
text: Let us see an example of how Zi is calculated in Algorithm 1. Let da and db be as given in the Algorithm. φ ∈ Rdb×da is a block diagonal matrix with blocks of sizes b0 × a0, 2b1 × 2a1, 2b2 × 2a2, . . . The first block has (b0∗a0) entries ,the second block has (b1∗a1) entries with the following structure.

         α11 0 α12 0 . . . αa11 0 0 α11 0 α12 . . . 0 αa11 α21 0 α22 0 . . . αa12 0 0 α21 0 α22 . . . 0 αa12 ... ... ... ... . . . ... ...
α1b1 0 α2b1 0 . . . αa1b1 0 0 α1b1 0 α2b1 . . . 0 αa1b1

        
The multiplicities in Ŷ are a = [2, 1, 2] and that of φ(Ŷ ) are b = [0, 1, 1]. Hence da = 8, db = 4 and φ with arbitrary chosen values is given below.
φ =

  0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 2 0 3 0 0 0 0 0 0 2 0 3

 
Boxes indicate the block for type 1 and type 2.
Let φ(Ŷ ) = (p, q, r, s)T
When we tensor (p, q)T of type n with (r, s)T of type m (m > n), we get (pr − qs, ps + qr)T of type (m+n) and (pr + qs, ps − qr) of type (m-n) (see Remark 4). In this example, when we tensor
φ(Ŷ ) with itself, we get
φ(Ŷ )⊗ φ(Ŷ ) =

                       
p2 + q2
0 r2 + s2
0 pr + qs ps− qr pr + qs ps− qr p2 − q2
2pq pr − qs ps+ qr pr − qs ps+ qr r2 − s2
2rs

                       
with the following types : multiplicities - type 0 : 4, type 1 : 2, type 2 : 1, type 3 : 2, type 4 : 1.
During implementation we avoid duplicates and zeros and we use the following:
φ(Ŷ )⊗ φ(Ŷ ) =

            
p2 + q2 r2 + s2 pr + qs ps− qr p2 − q2
2pq pr − qs ps+ qr r2 − s2
2rs

            
Also, φ(Ŷ )⊗ φ(Ŷ )⊕ φ(Ŷ ) after grouping the types, is given below:
φ(Ŷ )⊗ φ(Ŷ )⊕ φ(Ŷ ) =

                    
p2 + q2 r2 + s2 pr + qs ps− qr
p q
p2 − q2
2pq r s
pr − qs ps+ qr r2 − s2
2rs

                    
The linear map ψ ∈ Rdc×da is similar to φ. Only the input and output multiplicities are different. In the example we have c = [2, 2, 2, 1, 1] and a = [2, 1, 2]. A particular ψ with some fixed values is given below,
ψ =

         1 2 0 0 0 0 0 0 0 0 0 0 0 0 3 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 6 0 0 0 0 0 0 0 0 0 0 0 0 5 0 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 0 8 0 0 0 0 0 0 0 0 0 0 0 0 7 0 8 0 0 0 0 0 0 0 0 0 0 9 0 10 0 0 0 0 0 0 0 0 0 0 0 0 9 0 10 0 0 0 0

        
And Zi = ψ(φ(Ŷ )⊗ φ(Ŷ )⊕ φ(Ŷ ))

Review summaries:
Recently there has been a spate of work on generalized CNNs that are equivariant to various symmetry groups, such a 2D and 3D rotations, the corresponding Euclidean groups (comprising not just rotations but also translations) and so on. The approach taken in most of the recent papers is to explicitly build in these equivariances by using the appropriate generalization of convolution. In the case of nontrivial groups this effectively means working in Fourier space, i.e., transforming to a basis that is adapted to the group action. This requires some considerations from represntation theory.