Learning the Irreducible Representations of Commutative Lie Groups

Taco Cohen
Max Welling
Machine Learning Group, University of Amsterdam

Abstract
We present a new probabilistic model of compact
commutative Lie groups that produces invariantequivariant and disentangled representations ofdata. To define the notion of disentangling, weborrow a fundamental principle from physics thatis used to derive the elementary particles of a system from its symmetries. Our model employs a
newfound Bayesian conjugacy relation that enables fully tractable probabilistic inference overcompact commutative Lie groups – a class that
includes the groups that describe the rotation andcyclic translation of images. We train the modelon pairs of transformed image patches, and showthat the learned invariant representation is highly
effective for classification.

1. Introduction
Recently, the field of deep learning has produced some remarkable breakthroughs. The hallmark of the deep learning approach is to learn multiple layers of representation ofdata, and much work has gone into the development of representation learning modules such as RBMs and their generalizations (Welling et al., 2005), and autoencoders (Vincent et al., 2008). However, at this point it is not quite clear
what characterizes a good representation. In this paper, wetake a fresh look at the basic principles behind unsupervised representation learning from the perspective of Liegroup theory1 .
Various desiderata for learned representations have been
expressed: representations should be meaningful (Bengio& Lecun, 2014), invariant (Goodfellow et al., 2009), abstract and disentangled (Bengio et al., 2013), but so far
most of these notions have not been defined in a mathematically precise way. Here we focus on the notions of invariance and disentangling, leaving the search for meaning
for future work.
Proceedings of the 31 st International Conference on Machine
Learning, Beijing, China, 2014. JMLR: W&CP volume 32. Copyright 2014 by the author(s).

T.S.C OHEN @ UVA . NL
M.W ELLING @ UVA . NL

What do we mean, intuitively, when we speak of invariance and disentangling? A disentangled representation is
one that explicitly represents the distinct factors of variation in the data. For example, visual data (i.e. pixels) canbe thought of as a composition of object identity, positionand pose, lighting conditions, etc. Once disentangling isachieved, invariance follows easily: to build a representation that is invariant to the transformation of a factor of variation (e.g. object position) that is considered a nuisance fora particular task (e.g. object classification), one can simply ignore the units in the representation that encode thenuisance factor.
To get a mathematical handle on the concept of disentangling, we borrow a fundamental principle from physics,
which we refer to as Weyl’s principle, following Kanatani(1990). In physics, this idea is used to tease apart (i.e.
disentangle) the elementary particles of a physical systemfrom mere measurement values that have no inherent physical significance. We apply this principle to the area of vision, for after all, pixels are nothing but physical measurements.
Weyl’s principle presupposes a symmetry group that actson the data. By this we mean a set of transformations
that does not change the “essence” of the measured phenomenon, although it may change the “superficial appearance”, i.e. the measurement values. As a concrete example
that we will use throughout this paper, consider the groupknown as SO(2), acting on images by 2D rotation about theorigin. A transformation from this group (a rotation) may
change the value of every pixel in the image, but leaves invariant the identity of the imaged object. Weyl’s principle
states that the elementary components of this system aregiven by the irreducible representations of the symmetrygroup – a concept that will be explained in this paper.
Although this theoretical principle is widely applicable,
we demonstrate it for real-valued compact commutative
groups only. We introduce a probabilistic model that de1. We will at times assume a passing familiarity with Liegroups, but the main ideas of this paper should be accessible toa broad audience.

Learning the Irreducible Representations of Commutative Lie Groups

scribes a representation of such a group, and show how it
can be learned from pairs of images related by arbitrary andunobserved transformations in the group. Compact commutative groups are also known as toroidal groups, so werefer to this model as Toroidal Subgroup Analysis (TSA).
Using a novel conjugate prior, the model integrates probability theory and Lie group theory in a very elegant way.
All the relevant probabilistic quantities such as normalization constants, moments, KL-divergences, the posterior
density over the transformation group, the marginal density
in data space, and their gradients can be obtained in closed
form.
1.1. Related work
The first to propose a model and algorithm for learning
Lie group representations from data were Rao & Ruderman (1999). This model deals only with one-parametergroups, a limitation that was later lifted by Miao and Rao(2007). Both works rely on MAP-inference procedures thatcan only deal with infinitesimally small transformations.
This problem was solved by Sohl-Dickstein et al. (2010)
using an elegant adaptive smoothing technique, making it
possible to learn from large transformations. This model
uses a general linear transformation to diagonalize a oneparameter group, and combines multiple one-parametergroups multiplicatively.
Other, non-group-theoretical approaches to learning transformations and invariant representations exist (Memisevic& Hinton, 2010). These gating models were found to
perform a kind of joint eigenspace analysis (Memisevic,2012), which is somewhat similar to the irreducible reduction of a toroidal group.
Motivated by a number of statistical phenomena observedin natural images, Cadieu & Olshausen (2012) describe amodel that decomposes a signal into invariant amplitudes
and covariant phase variables.
None of the mentioned methods take into account the fulluncertainty over transformation parameters, as does TSA.
Due to exact or approximate symmetries in the data, there
is in general no unique transformation relating two images,
so that only a multimodal posterior distribution over thegroup gives a complete description of the geometric situation. Furthermore, posterior inference in our model is performed by a very fast feed-forward procedure, whereas the
MAP inference algorithm by Sohl-Dicksteint et al. requiresa more expensive iterative optimization.

2. Preliminaries
2.1. Equivalence, Invariance and Reducibility
In this section, we discuss three fundamental concepts on
which the analysis in the rest of this paper is based: equivalence, invariance and reducibility.
Consider a function Φ : RD → X that assigns to each possible data point x ∈ RD a class-label (X = {1, . . . , L})or some distributed representation (e.g. X = RL ). Such afunction induces an equivalence relation on the input space
RD : we say that two vectors x, y ∈ RD are Φ-equivalent if
they are mapped onto the same representation by Φ. Symbolically, x ≡Φ y ⇔ Φ(x) = Φ(y).
Every equivalence relation on the input space fully determines a symmetry group acting on the space. This
group, call it G, contains all invertible transformations
ρ : RD → RD that leave Φ invariant: G = {ρ | ∀x ∈RD : Φ(ρ(x)) = Φ(x)}. G describes the symmetries ofΦ, or, stated differently, the label function/representation Φ
is invariant to transformations in G. Hence, we can speakof G-equivalence: x ≡G y ⇔ ∃ρ ∈ G : ρ(x) = y. For
example, if some elements of G act by rotating the image,two images are G-equivalent if they are rotations of eachother.
Before we can introduce Weyl’s principle, we need one
more concept: the reduction of a group representation(Kanatani, 1990). Let us restrict our attention to linear
representations of Lie groups: ρ becomes a matrix-valued
function ρg of an abstract group element g ∈ G, such that∀g, h ∈ G : ρg◦h = ρg ρh . In general, every coordinate yi
of y = ρg x can depend on every coordinate xj of x. Now,
since x is G-equivalent to y, it makes no sense to consider
the coordinates xi as separate quantities; we can only consider the vector x as a single unit because the symmetrytransformations ρg tangle all coordinates. In other words,
we cannot say that coordinate xi is an independent part of
the aggregate x, because a mapping x → x0 = ρg x that is
supposed to leave the intrinsic properties of x unchanged,
will in fact induce induce a functional dependence between
all supposed parts x0i and xj .
However, we are free to change the basis of the measurement space. It may be possible to use a change of basis to
expose an invariant subspace, i.e. a subspace V ⊂ RD thatis mapped onto itself by every transformation in the group:∀g ∈ G : x ∈ V ⇒ ρg x ∈ V . If such a subspace exists and its orthogonal complement V ⊥ ⊂ RD is also aninvariant subspace, then it makes sense to consider the twoparts of x that lie in V and V ⊥ to be distinct, because theyremain distinct under symmetry transformations.
Let W be a change of basis matrix that exposes the invari-Learning the Irreducible Representations of Commutative Lie Groups

ant subspaces, that is,ρg = W

 1
ρg


ρ2g

W−1 ,

the total energy (norm) of the image, they can be represented by orthogonal matrices acting on vectorized images.
(1)

for all g ∈ G. Both ρ1g and ρ2g form a representation of
the same abstract group as represented by ρ. The grouprepresentations ρ1g and ρ2g describe how the individual parts
x1 ∈ V and x2 ∈ V ⊥ are transformed by the elements ofthe group. As is common in group representation theory,
we refer to both the group representations ρ1g , ρ2g and thesubspaces V and V ⊥ corresponding to these group representations as “representations”.
The process of reduction can be applied recursively to ρ1g
and ρ2g . If at some point there is no more (non-trivial) invariant subspace, the representation is called irreducible.
Weyl’s principle states that the elementary components ofa system are the irreducible representations of the symmetry group of the system. Properly understood, it is not aphysics principle at all, but generally applicable to any situation where there is a well-defined notion of equivalence2 .
It is completely abstract and therefore agnostic about thetype of data (images, optical flows, sound, etc.), making it
eminently useful for representation learning.
In the rest of this paper, we will demonstrate Weyl’s principle in the simple case of a compact commutative subgroupof the special orthogonal group in D dimensions. We want
to stress though, that there is no reason the basic ideas cannot be applied to non-commutative groups acting on nonlinear latent representation spaces.
2.2. Maximal Tori in the Orthogonal GroupIn order to facilitate analysis, we will from here on consider only compact commutative subgroups of the special
orthogonal group SO(D). For reasons that will become
clear shortly, such groups are called toroidal subgroups ofSO(D). Intuitively, the toroidal subgroups of general compact Lie groups can be thought of as the “commutativepart” of these groups. This fact, combined with their analytic tractability (evidenced by the results in this paper)
makes them suitable as the starting point of a theory ofprobabilistic Lie-group representation learning.
Imposing the constraint of orthogonality will make thecomputation of matrix inverses very cheap, because for orthogonal Q, Q−1 = QT . Orthogonal matrices also avoid
numerical problems, because their condition number is always equal to 1. Another important property of orthogonal transformations is that they leave the Euclidean metricinvariant: kQxk = kxk. Therefore, orthogonal matrices
cannot express transformations such as contrast scaling, butthey can still model the interesting structural changes in images (Bethge et al., 2007). For example, since 2D image rotation and (cyclic) translation are linear and do not change

As is well known, commuting matrices can be simultaneously diagonalized, so one could represent a toroidal groupin terms of a basis of eigenvectors shared by every elementin the group, and one diagonal matrix of eigenvalues foreach element of the group, as was done in (Sohl-Dickstein
et al., 2010) for 1-parameter Lie groups. However, orthogonal matrices do not generally have a complete set ofreal eigenvectors. One could use a complex basis instead,but this introduces redundancies because the eigenvalues
and eigenvectors of an orthogonal matrix come in complex conjugate pairs. For machine learning applications,this is clearly an undesirable feature, so we opt for a joint
block-diagonalization of the elements of the toroidal group:ρϕ = WR(ϕ)WT , where W is orthogonal and R(ϕ) is a
block-diagonal rotation matrix3 :R(ϕ1 )

..
(2)
R(ϕ) = .
.
R(ϕJ )The diagonal of R(ϕ) contains 2 × 2 rotation matrices
cos(ϕj ) − sin(ϕj )R(ϕj ) =.
(3)
sin(ϕj ) cos(ϕj )In this parameterization, the real, orthogonal basis W identifies the group representation, while the vector of rotation
angles ϕ identifies a particular element of the group. It
is now clear why such groups are called “toroidal”: theparameter space ϕ is periodic in each element ϕj andhence is a topological torus. For a J-parameter toroidal
group, all the ϕj can be chosen freely. Such a group is
known as a maximal torus in SO(D), for which we write
TJ = {ϕ | ϕj ∈ [0, 2π], j = 1, . . . J}.
To gain insight into the structure of toroidal groups withfewer parameters, we rewrite eq. 2 using the matrix exponential:JX
R(ϕ) = exp ϕj Aj .
(4)
j=1

The anti-symmetric matrices Aj = dϕd j R(ϕ) 0 are known
as the Lie algebra generators, and the ϕj are Lie-algebra
coordinates.
The Lie algebra is a structure that largely determines the
structure of the corresponding Lie group, while having theimportant advantage of forming a linear space. That is, all
2. The picture becomes a lot more complicated, though, when
the group does not act linearly or is not completely reducible
3. For ease of exposition, we assume an even dimensional space
D = 2J, but the equations are easily generalized.

Learning the Irreducible Representations of Commutative Lie Groups

one of the elementary parts uj is functionally independentof the others under symmetry transformations.

Figure 1. Parameter space ϕ = (ω1 s, ω2 s) of two toroidal subgroups, for ω1 = 2, ω2 = 3 (left) and ω1 = 1, ω2 = 2π (right).
The point ϕ moves over the dark green line as s is changed. Wrapping around is indicated in light gray. In the incommensurable
case (right), coupling does not add structure to the model, because
all transformations (points in the plane) can still be constructed by
an appropriate choice of s.

linear combinations of the generators belong to the Lie algebra, and each element of the Lie algebra corresponds toan element of the Lie group, which itself is a non-linear
manifold. Furthermore, every subgroup of the Lie groupcorresponds to a subalgebra (not defined here) of the Liealgebra. All toroidal groups are the subgroup of some maximal torus, so we can learn a general toroidal group by firstlearning a maximal torus and then learning a subalgebra of
its Lie algebra. Due to commutativity, the structure of the
Lie algebra of toroidal groups is such that any subspace of
the Lie algebra is in fact a subalgebra. The relevance of
this observation to our machine learning problem is that tolearn a toroidal group with I parameters (I < J), we cansimply learn a maximal toroidal group and then learn an
I-dimensional linear subspace in the space of ϕ.
4

In this work, we are interested in compact subgroups only ,which is to say that the parameter space should be closed
and bounded. To see that not all subgroups of a maximal
torus are compact, consider a 4D space and a maximal torus
with 2 generators A1 and A2 . Let us define a subalgebrawith one generator A = ω1 A1 + ω2 A2 , for real numbers
ω1 and ω2 . The group elements generated by this algebra
through the exponential map takes the form
R(ω1 s)R(s) = exp (sA) =. (5)R(ω2 s)Each block R(ωj s) is periodic with period 2π/ωj , but theirdirect sum R(s) need not be. When ω1 and ω2 are notcommensurate, all values of s ∈ R will produce different R(s), and hence the parameter space is not bounded.
To obtain a compact one-parameter group with parameter
space s ∈ [0, 2π], we restrict the frequencies ωj to be integers, so that R(s) = R(s + 2π) (see figure 1).
It is easy to see that each block of R(s) forms a real-valued
irreducible representation of the toroidal group, making
R(s) a direct sum of irreducible representations. From thepoint of view expounded in section 2.1, we should view thevector x as a tangle of elementary components uj = WjT x,
where Wj = (W(:, 2j−1) , W(:, 2j) ) denotes the D×2 submatrix of W corresponding to the j-th block in R(s). Each

The variable ωj is known as the weight of the representation (Kanatani, 1990). When the representations are equivalent (i.e. they have the same weight), the parts are “of thesame kind” and are transformed identically. Elementarycomponents with different weights transform differently.
In the following section, we show how a maximal toroidal
group and a 1-parameter subgroup can be learned from correspondence pairs, and how these can be used to generate
invariant representations.

3. Toroidal Subgroup Analysis
We will start by modelling a maximal torus. A data pair(x, y) is related by a transformation ρϕ = WR(ϕ)WT :y = WR(ϕ)WT x + ,

(6)

where  ∼ N (0, σ 2 ) represents isotropic Gaussian noise.
In other symbols, p(y|x, ϕ) = N (y|WR(ϕ)WT x, σ 2 ).
We use the following notation for indexing invariant subspaces. As before, Wj = (W(:, 2j−1) , W(:, 2j) ). Let
uj = WjT x and vj = WjT y. If we want to access one
Tof the coordinates of u or v, we write uj1 = W(:,2j−1) xTor uj2 = W(:,x.
2j)
We assume the ϕj to be marginally independent and vonMises distributed. The von-Mises distribution is an exponential family that assigns equal density to the endpoints ofany length-2π interval of the real line, making it a suitable choice for periodic variables such as ϕj . We willfind it convenient to move back and forth between the conventional and natural parameterizations of this distribution.
The conventional parameterization of the von-Mises distribution M(ϕj |µj , κj ) uses a mean µj and precision κj :p(ϕj ) =1
exp (κj cos(ϕj − µj )).
2πI0 (κj )(7)

The function I0 that appears in the normalizing constant is
known as the modified Bessel function of order 0.
Since the von-Mises distribution is an exponential family, we can write it in terms of natural parameters ηj =(ηj1 , ηj2 )T as follows:p(ϕj ) =1
exp (ηjT T (ϕj )),2πI0 (kηj k)(8)where T (ϕj ) = (cos(ϕj ), sin(ϕj ))T are the sufficient
statistics. The natural parameters can be computed from
4. The main reason for this restriction is that compact groups
are simpler and better understood than non-compact groups. In
practice, many non-compact groups can be compactified, so notmuch is lost.

Learning the Irreducible Representations of Commutative Lie Groups

conventional parameters using,ηj = κj [cos(µj ), sin(µj )]Tsensible result follows directly from the consistent application of the rules of probability.
(9)and vice versa,
κj = kηj k,µj = tan−1 (ηj2 /ηj1 )(10)

Using the natural parameterization, it is easy to see thatthe prior is conjugate to the likelihood, so that the posterior p(ϕ|x, y) is again a product of von-Mises distributions.
Such conjugacy relations are of great utility in Bayesianstatistics, because they simplify sequential inference. To
our knowledge, this conjugacy relation has not been described before. To derive this result, first observe that thelikelihood term splits into a sum over the invariant subspaces indexed by j:p(ϕ|x, y) ∝ p(y|x, ϕ)p(ϕ)
1∝ exp − 2 ky − WR(ϕ)WT xk2 p(ϕ)
2σ


J
X
vjT R(ϕj )uj∝ exp + ηjT T (ϕj )
2
σ
j=1

Both the bilinear forms vjT R(ϕj )uj and the prior terms
ηjT T (ϕ) are linear functions of cos(ϕj ) and sin(ϕj ), so thatthey can be combined into a single dot product:JXp(ϕ|x, y) ∝ exp η̂jT T (ϕj ),
(11)
j=1

which we recognize as a product of von-Mises in naturalform.

Observe that when using a uniform prior (i.e. ηj = 0), theposterior mean µ̂j (computed from η̂j by eq. 10) will be
exactly equal to the angle θj between uj and vj . We will
use this fact in section 3.1 when we derive the formula forthe orbit distance in a toroidal group.
Previous approaches to Lie group learning only provide
point estimates of the transformation parameters, whichhave to be obtained using an iterative optimization procedure (Sohl-Dickstein et al., 2010). In contrast, TSA provides a full posterior distribution which is obtained using a simple feed-forward computation. Compared to thework of Cadieu & Olshausen (2012), our model deals wellwith low-energy subspaces, by simply describing the uncertainty in the estimate instead of providing inaccurate estimates that have to be discarded.
3.1. Invariant Representation and Metric
One way of doing invariant classification is by using an invariant metric known as the manifold distance. This metricd(x, y) is defined as the minimum distance between the orbits Ox = {ρϕ x |ϕ ∈ G} and Oy = {ρϕ y |ϕ ∈ G}.
Observe that this is only a true metric that satisfies the coincidence axiom d(x, y) = 0 ⇔ x = y if we take thecondition x = y to mean “equivalence up to symmetrytransformations” or x ≡G y, as discussed in section 2.1.
In practice, it has proven difficult to compute this distanceexactly, so approximations such as tangent distance havebeen invented (Simard et al., 2000). But for a maximal
torus, we can easily compute the exact manifold distance:d2 (x, y) = min ky − WR(ϕ)WT xk2
ϕX=
min kvj − R(ϕj )uj k2

The parameters η̂j of the posterior are given by:1η̂j = ηj + 2 [uj1 vj1 + uj2 vj2 , uj1 vj2 − uj2 vj1 ]Tσ(12)
kuj kkvj kT= ηj +[cos(θ
),
sin(θ
)]
,
j
j
σ2
where θj is the angle between uj and vj . Geometrically,
we can interpret the Bayesian updating procedure in eq. 12
as follows. The orientation of the natural parameter vector
ηj determines the mean of the von-Mises, while its magnitude determines the precision. To update this parameter with new information obtained from data uj , vj , one
should add the vector (cos(θj ), sin(θj ))T to the prior, using a scaling factor that grows with the magnitude of uj
and vj and declines with the square of the noise level σ.
The longer uj and vj and the smaller the noise level, thegreater the precision of the observation. This geometrically

j

=

X

ϕj

(13)

kvj − R(µ̂j )uj k2 ,jwhere µ̂j is the mean of the posterior p(ϕj |x, y), obtainedusing a uniform prior (κj = 0). The last step of eq. 13
follows, because as we saw in the previous section, µ̂j is
simply the angle between uj and vj when using a uniform
prior. Therefore, R(µ̂j ) aligns uj and vj , minimizing the
distance between them.
Another approach to invariant classification is through aninvariant representation. Although the model presented
above aims to describe the transformation between observations x and y, an invariant-equivariant representation appears automatically in terms of the parameters of the posterior over the group. To see this, consider all the transformations in the learned toroidal group G that take an image

Learning the Irreducible Representations of Commutative Lie Groups

x to itself. This set is known as the stabilizer stabG (x)of x. It is a subgroup of G and describes the symmetriesof x with respect to G. When a transformation ϕ ∈ G isapplied to x, the stabilizer subgroup is left invariant, for ifθ ∈ stabG (x) then ρθ ρϕ x = ρϕ ρθ x = ρϕ x and henceρθ ∈ stabG (ρϕ x).
The posteriorof x transformed into itself, p(ϕ|x, x, µ, κ =Q0) = j M(ϕj |µ̂j , κ̂j ) gives a probabilistic descriptionof the stabilizer of x, and hence must be invariant. Clearly,
the angle between x and x is zero, so µ̂ = 0. On the other
hand, κ̂ contains information about x and is invariant. To
see this, recall that κ̂j = kη̂j k. Using eq. 12 we obtainκ̂j = kuj k2 σ −2 = kWjT xk2 σ −2 . Since every transformation ϕ in the toroidal group acts on the 2D vector uj byrotation, the norm of uj is left invariant.
We recognize the computation of κ̂ as the square pooling operation often applied in convolutional networks togain invariance: project an image onto filters W:,2j−1 andW:,2j and sum the squares. This computation is a directconsequence of our model setup. In section 3.3, we willfind that the model for non-maximal tori is even more informative about the proper pooling scheme.
Since we want to use κ̂ as an invariant representation, we should try to find an appropriate metric on κ̂space. Let κ̂(x) be defined by p(ϕ|x, x, κ = 0) =Q
j M(ϕ|µ̂j , κ̂j (x)). We suggest using the Hellinger distance:

1X
H (κ̂(x), κ̂(y)) =
2 j
2

=

q

2
q
κ̂j (x) − κ̂j (y)1 X
kuj k2 + kvj k2 − 2kuj kkvj k2σ 2 j1 X
kvj − R(µ̂j )uj k2 ,=2σ 2 jWe show that the DFT is a special case of TSA. The DFTof a discrete 1D signal x = (x1 , . . . , xD )T is defined:

n=0

xn ρ−jn= (cos(2πj/D), . . . , cos(2πj(D − 1)/D))T
W(:, 2j) = I(ρ−j , . . . , ρ−j(D−1) )T= (sin(−2πj/D), . . . , sin(−2πj(D − 1)/D))T ,then the change of basis performed by W is a DFT. Specifically, R(Xj ) = uj1 and I(Xj ) = uj2 .
Now suppose we are interested in the transformation taking some arbitrary fixed vector e = W(1, 0, . . . , 1, 0)T
to x. The posterior over ϕj is p(ϕj |e, x, ηj = 0, σ =1) = M(ϕj |η̂j ), where (by eq. 12) we have η̂j =kuj k[cos(θj ), sin(θj )]T , θj being the angle between uj
and the “real axis” ej = (1, 0)T . In conventional coordinates, the precision of the posterior is equal to the modulus of the DFT, κ̂j = kuj k = |Xj |, and the mean of
the posterior is equal to the phase of the Fourier transform,µ̂ = θj = arg(Xj ). Therefore, TSA provides a probabilistic interpretation of the DFT coefficients, and makes it
possible to learn an appropriate generalized transform fromdata.
3.3. Modeling a Lie subalgebra
Typically, one is interested in learning groups with fewer
than J degrees of freedom. As we have seen, for one parameter compact subgroups of a maximal torus, the weightsof the irreducible representations must be integers. We
model this using a coupled rotation matrix, as follows:
R(ω1 s) T..
ρs = W (15)
W.
R(ωJ s)We have found that the conjugate prior for this likelihood
is the generalized von-Mises (Gatto & Jammalamadaka,2007):
p(s) = M+ (s|η + ) = exp η + · T + (s) / Z +


K
X
+ 
= exp κ+/ Z+
j cos(js − µj )j=13.2. Relation to the Discrete Fourier Transform

D−1X

W(:, 2j−1) = R(ρ−j , . . . , ρ−j(D−1) )T

Where s ∈ [0, 2π] is the scalar parameter of this subgroup.
The likelihood then becomes y ∼ N (y|ρs x, σ 2 ).

which is equal to the exact manifold distance (eq. 13) up
to a factor of 2σ1 2 . The first step of this derivation uses eq.
12 under a uniform prior (ηj = 0), while the second stepagain makes use of the fact that µ̂j is the angle between uj
and vj so that kuj kkvj k = uTj R(µ̂j )vj .

Xj =where ρ = e2πi/D is the D-th primitive root of unity. If wechoose a basis of sinusoids for the filters in W,(14)

where T + (s) = [cos(s), sin(s), . . . , cos(Ks), sin(Ks)]T .
This conjugacy relation p(s|x, y) ∝ exp (η̂ + · T + (s)) is
obtained using similar reasoning as before, yielding the update equation,Xη̂j+ = ηj+ +η̂k(16)
k:ωk =jLearning the Irreducible Representations of Commutative Lie Groups

where η̂k is obtained from eq. 12 using a uniform priorηk = 0. The sum in this update equation performs a pooling operation over a weight space, which is defined as thespan of those invariant subspaces k whose weight ωk = j.
As it turns out, this is exactly the right thing to do in orderto summarize the data while maintaining invariance. As
explained by Kanatani (1990), the norm kη̂j+ k of a linear
combination of same-weight representations is always invariant (as is the maximum of any two kη̂j k or kη̂j+ k). The
similarity to sum-pooling and max-pooling in convnets isquite striking.
In the maximal torus model, there are J = D/2 degrees
of freedom in the group and the invariant representationis D − J = J-dimensional (κ̂1 , . . . , κ̂J ). This representation of a datum x identifies a toroidal orbit, and hence
the vector κ̂ is a maximal invariant with respect to thisgroup (Soatto, 2009). For the coupled model, there is only
one degree of freedom in the group, so the invariant representation should be D − 1 dimensional. When all ωk are+
distinct, we have J variables κ+
1 , . . . , κJ that are invariant. Furthermore, from eq. 15 we see that as x is transformed, the angle between x and an arbitrary fixed reference vector in subspace j transforms as θj (s) = δj + ωj s
for some data-dependent initial phase δj . It follows thatωj θk (s) − ωk θj (s) = ωj (δk + ωk s) − ωk (δj + ωj s) =ωj δk − ωk δj is invariant. In this way, we can easily construct another J − 1 invariants, but unfortunately these are
not stable because the angle estimates can be inaccurate forlow-energy subspaces. Finding a stable maximal invariant,and doing so in a way that will generalize to other groups
is an interesting problem for future work.
+

The normalization constant Z for the GvM has so far
only been described for the case of K = 2 harmonics, but we have found a closed form solution in termsof the so-called modified Generalized Bessel Functions+(GBF) of K-variables κ+ = κ+
1 , . . . , κK and parame+5+
ters exp (−iµ ) = exp (−iµ1 ), . . . , exp (−iµ+K ) (Dattoli et al., 1991):+

Z + (κ+ , µ+ ) = 2πI0 (κ+ ; e−iµ ).

(17)

We have developed a novel, highly scalable algorithm forthe computation of GBF of many variables, which is described in the supplementary material.
Figure 2 shows the posterior over s for three image pairs
related by different rotations and containing different symmetries. The weights W and ω were learned by the procedure described in the next section. It is quite clear fromthis figure that MAP inference does not give a complete
description of the possible transformations relating the images when the images have a degree of rotational symmetry. The posterior distribution of our model provides asensible way to deal with this kind of uncertainty, which

.35
0

0

π

2π 0

π

2π 0

π

2π

Figure 2. Posterior distribution over s for three image pairs.

(in the case of 2D translations) is at the heart of the wellknown aperture problem in vision. Having a tractable posterior is particularly important if the model is to be used
to estimate longer sequences (akin to HMM/LDS models,
but non-linear), where one may encounter multiple highdensity trajectories.
If required, accurate MAP inference can be performed using the algorithm of Sohl-Dickstein et al. (2010), as described in the supplementary material. This allows us tocompute the exact manifold distance for the coupled model.
3.4. Maximum Marginal Likelihood Learning
We train the model by gradient descent on the marginal
likelihood. Perhaps surprisingly given the non-linearities
in the model, the integrations required for the evaluation of
the marginal likelihood can be obtained in closed form forboth the coupled and decoupled models. For the decoupledmodel we obtain:ZYp(y|x) =N (y|WR(ϕ)WT x)
M(ϕj |ηj )dϕ
ϕ∈TJ

j


exp − 2σ1 2 (kxk2 + kyk2 ) Y I0 (κ̂j )p.
=
I0 (κj )(2πσ)Dj(18)
Observing that I0 (κ̂j )/I0 (κj ) is the ratio of normalization constants of regular von-Mises distributions, the analogous expression for the coupled model
Q is easily seen tobe equal to eq. 18, only replacing j I0 (κ̂j ) / I0 (κj )by Z + (κ̂+ , µ̂+ )/Z + (κ+ , µ+ ). The derivation of this result
can be found in the supplementary material.
The gradient of the log marginal likelihood of the uncoupled model w.r.t. a batch X, Y (both storing N vectors in
the columns) is:dln p(Y|X) = X(RT (µ)WT Y  A + WT X  By )TdW+ Y(R(µ)WT X  A + WT Y  Bx )T .
where we have used (P  Q)2j,n = Qj,n P2j,n and (P Q)2j−1,n = Qj,n P2j−1,n as a “subspace weighting” oper5. As described in the supplementary material, we use a slightly
different parameterization of the GBF.

Learning the Irreducible Representations of Commutative Lie Groups

ation. A, B(x) and B(y) are D × N matrices with elements
ajn =I1 (κ̂jn )κj,
I0 (κ̂jn )κ̂jn σ 2

bjn =kWj y(n) k2,
κj σ 2where the κ̂jn is the posterior precision in subspace j forimage pair x(n) , y(n) (the n-th column of X, resp. Y).
The gradient of the coupled model is easily computed using
the differential recurrence relations that hold for the GBF(Dattoli et al., 1991).
Figure 3. Filters learned by TSA, sorted by absolute frequency|ωj |. The learned ωj -values range from from −11 to 12.
1
Accuracy

We use minibatch Stochastic Gradient Descent (SGD) on
the log-likelihood of the uncoupled model. After every parameter update, we orthogonalize W by setting all singularvalues to 1: Let U, S, V = svd(W), then set W := UV.
This procedure and all previous derivations still work when
the basis is undercomplete, i.e. has fewer columns (filters)
than rows (dimensions in data space). To learn ωj , we estimate the relative angular velocity ωj = θj /δ from a batchof image patches rotated by a sub-pixel amount δ = 0.1◦ .

.8
.4
.2
0

4. Experiments
We trained a TSA model with 100 filters on a stream of
250.000 16 × 16 image patches x(t) , y(t) . The patches
x(t) were drawn from a standard normal distribution, andy(t) was obtained by rotating x(t) by an angle s drawn uniformly at random from [0, 2π]. The learning rate√α was initialized at α0 = 0.25 and decayed as α = α0 / T , where
T was incremented by one with each pass through the data.
Each minibatch consisted of 100 data pairs. After learning W, we estimate the weights ωj and sort the filter pairs
by increasing absolute value for visualization. As can beseen in fig. 3, the filters are very clean and the weights are
estimated correctly except for a few filters on row 1 and 2
that are assigned weight 0 when in fact they have a higher
frequency.
We tested the utility of the model for invariant classification on a rotated version of the MNIST dataset, using a
1-Nearest Neighbor classifier. Each digit was rotated by arandom angle and rescaled to 16 × 16 pixels, resulting in
60k training examples and 10k testing examples, with norotated duplicates. We compared the Euclidean distance(ED) in pixel space, tangent distance (TD)√(Simard et al.,2000), Euclidean distance on the space of κ̂ (equivalent
to the exact manifold distance for the maximal torus, seesection 3.1), the true manifold distance for the 1-parameter2D rotation group (MD), and the Euclidean distance on the
non-rotated version of the MNIST dataset (ED-NR). The
results in fig. 4√show that TD outperforms ED, but is outperformed by κ̂ and MD by a large margin. In fact, theMD-classifier is about as accurate as ED on a much sim-

ED
TD
√
κ̂
MD
ED-NR

.6

100

1k
10k
Trainset size

60k

Figure 4. Results of classification experiment. See text for details.

pler dataset, demonstrating that it has almost completely
modded out the variation caused by rotation.

5. Conclusions and outlook
We have presented a novel principle for learning disentangled representations, and worked out its consequences fora simple type of symmetry group. This leads to a completely tractable model with potential applications to invariant classification and Bayesian estimation of motion. The
model reproduces the pooling operations used in convolutional networks from probabilistic and Lie-group theoretic
principles, and provides a probabilistic interpretation of the
DFT and its generalizations.
The type of disentangling obtained in this paper is contingent upon the rather minimalist assumption that all that canbe said about images is that they are equivalent (rotatedcopies) or inequivalent. However, the universal nature ofWeyl’s principle bodes well for future applications to deep,
non-linear and non-commutative forms of disentangling.

Acknowledgments
We would like to thank prof. Giuseppe Dattoli for help with
derivations related to the generalized Bessel functions.

Learning the Irreducible Representations of Commutative Lie Groups

References
Bengio, Y. and Lecun, Y.
International Conference on Learning Representations,
2014.
URL
https://sites.google.com/site/
representationlearning2014/.
Bengio, Y., Courville, A., and Vincent, P. Representation
Learning: A Review and New Perspectives. IEEE transactions on pattern analysis and machine intelligence, pp.
1–30, February 2013.
Bethge, M., Gerwinn, S., and Macke, J. H. Unsupervised
learning of a steerable basis for invariant image representations. Proceedings of SPIE Human Vision and Electronic Imaging XII (EI105), February 2007.
Cadieu, C. F. and Olshausen, B. A. Learning intermediatelevel representations of form and motion from natural
movies. Neural computation, 24(4):827–66, April 2012.
Dattoli, G., Chiccoli, C., Lorenzutta, S., Maino, G.,
Richetta, M., and Torre, A. A Note on the Theory ofn-Variable Generalized Bessel Functions. Il Nuovo Cimento B, 106(10):1159–1166, October 1991.
Gatto, R. and Jammalamadaka, S. R. The generalized vonMises distribution. Statistical Methodology, 4(3):341–
353, 2007.
Goodfellow, I. J., Le, Q. V., Saxe, A. M., Lee, H., and Ng,
A. Y. Measuring invariances in deep networks. Advancesin Neural Information Processing Systems, 2009.
Kanatani, K. Group Theoretical Methods in Image Understanding. Springer-Verlag New York, Inc., Secaucus, NJ,
USA, 1990. ISBN 0387512535.
Memisevic, R. On multi-view feature learning. International Conference on Machine Learning, 2012.
Memisevic, R. and Hinton, G. E. Learning to Represent Spatial Transformations with Factored HigherOrder Boltzmann Machines. Neural Computation, 22(6):1473–1492, June 2010. ISSN 1530-888X. doi:
10.1162/neco.2010.01-09-953.
Miao, X. and Rao, R. P. N. Learning the Lie groups ofvisual invariance. Neural computation, 19(10):2665–93,
October 2007.
Rao, R. P. N. and Ruderman, D. L. Learning Lie groups forinvariant visual perception. Advances in neural information processing systems, 816:810–816, 1999.
Simard, P. Y., Le Cun, Y. A., Denker, J. S., and Victorri, B.
Transformation invariance in pattern recognition: Tangent distance and propagation. International Journal of
Imaging Systems and Technology, 11(3):181–197, 2000.

Soatto, S. Actionable information in vision. In 2009 IEEE
12th International Conference on Computer Vision, pp.
2138–2145. IEEE, September 2009.
Sohl-Dickstein, J., Wang, J. C., and Olshausen, B. A. An
unsupervised algorithm for learning lie group transformations. arXiv preprint, 2010.
Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.
Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international
conference on Machine learning, pp. 1096–1103, 2008.
Welling, M., Rosen-zvi, M., and Hinton, G. Exponential
Family Harmoniums with an Application to Information
Retrieval. Advances in neural information processing
systems, 17, 2005.