Learning Stable Group Invariant Representations with Convolutional
    Networks
Authors: Joan Bruna, Arthur Szlam, Yann LeCun
ICLR 2013

Abstract: Transformation groups, such as translations or rotations, effectively express part of the variability observed in many recognition problems. The group structure enables the construction of invariant signal representations with appealing mathematical properties, where convolutions, together with pooling operators, bring stability to additive and geometric perturbations of the input. Whereas physical transformation groups are ubiquitous in image and audio applications, they do not account for all the variability of complex signal classes. 
We show that the invariance properties built by deep convolutional networks can be cast as a form of stable group invariance. The network wiring architecture determines the invariance group, while the trainable filter coefficients characterize the group action. We give explanatory examples which illustrate how the network architecture controls the resulting invariance group. We also explore the principle by which additional convolutional layers induce a group factorization enabling more abstract, powerful invariant representations.

arXiv:1301.3537v1  [cs.AI]  16 Jan 2013
Learning Stable Group Invariant Representations
with Convolutional Networks
Joan Bruna, Arthur Szlam and Y ann LeCun
Courant Institute
New York University
New Nork, NY , 10013
{bruna,lecun}@cims.nyu.edu
1 Introduction
Many signal categories in vision and auditory problems are invariant to the action of transformation
groups, such as translations, rotations or frequency transpositions. This property motivates the study
of signal representations which are also invariant to the ac tion of these transformation groups. For
instance, translation invariance can be achieved with a registration or with auto-correlation measures.
Transformation groups are in fact low-dimensional manifolds, and therefore mere group invariance
is in general not enough to efﬁciently describe signal classes. Indeed, signals may be perturbed with
additive noise and also with geometrical deformations, so one can then ask for invariant representa-
tions which are stable to these perturbations. Scattering convolutional networks [1] construct locally
translation invariant signal representations, with addit ive and geometrical stability, by cascading
complex wavelet modulus operators with a lowpass smoothing kernel. By deﬁning wavelet decom-
positions on any locally compact Lie Group, scattering operators can be generalized and cascaded to
provide local invariance with respect to more general trans formation groups [2, 3]. Although such
transformation groups are present across many recognition problems, they require prior information
which sometimes cannot be assumed.
Convolutional networks [4] cascade ﬁlter banks with point- wise nonlinearities and local pooling
operators. By remapping the output of each layer with the inp ut of the following one, the trainable
ﬁlters implement convolution operators. We show that the invariance properties built by deep convo-
lutional networks can be cast as a form of stable group invariance. The network wiring architecture
determines the invariance group, while the trainable ﬁlter coefﬁcients characterize the group action.
Deep convolutional architectures cascade several layers of convolutions, non-linearities and pooling.
These architectures have the capacity to generate local inv ariance to the action of more general
groups. Under appropriate conditions, these groups can be factorized as products of smaller groups.
Each of these factors can then be associated with a subset of consecutive layers of the convolutional
network. In these conditions, the invariance properties of the ﬁnal representation can be studied
from the group structure generated by each layer.
2 Problem statement
2.1 Stable Group Invariance
A transformation group Gacts on the input space X (assumed to be a Hilbert space) with a linear
group action (g,x) ↦→g.x∈ X , which is compatible with the group operation.
A signal representation Φ : X − → Z is invariant to the action of Gif ∀ g ∈ G,x ∈ X , Φ( g.x) =
Φ( x) . However, mere group invariance is in general too weak, due to the presence of a much
larger, high dimensional variability which does not belong to the low-dimensional group. It is then
necessary to incorporate the notion of outer “deformations ” with another group action ϕ : H ×
1
X − → X ,where H is a larger group containing G. The geometric stability can be stated with a
Lipschitz continuity property
∥Φ( ϕ(h,x)) − Φ( x)∥ ≤ C∥x∥ k(h,G) , (1)
where k(h,G) measures the “distance” from hto the invariance group G. For instance, when Gis
the translation group of Rd and H ⊃ Gis the group of C2 diffeomorphisms of Rd, then ϕ(h,x) =
x◦ hand one can select as distance the elastic deformation metri c k(h,G) := |∇τ|∞ + |Hτ|∞,
where τ(u) = h(u) − u[2].
Even though the group invariance formalism describes global invariance properties of the represen-
tation, it also provides a valid and useful framework to stud y local invariance properties. Indeed, if
one replaces (1) by
∥Φ( ϕ(h,x)) − Φ( x)∥ ≤ C∥x∥ (∥hG∥G + k(h,G)) , (2)
where hG is a projection of hto Gand ∥g∥G is a metric on Gmeasuring the amount of transforma-
tion being applied, then the local invariance is expressed b y adjusting the proportionality between
the two metrics.
2.2 Convolutional Networks
A generic convolutional network deﬁned on a spaceX = L2(Ω 0) of square-integrable signals starts
with a ﬁlter bank {ψλ }λ ∈Λ 1 , ψλ ∈ L1(Ω 0) ∀λ, which for each input x(u) ∈ X produces the
collection
z(1)(u,λ) = x⋆ψ λ (u) =
∫
x(u− v)ψλ (v)dv, u ∈ Ω 0 ,λ ∈ Λ 1 .
If the ﬁlter bank deﬁnes a stable, invertible frame, then there exist two constants a,A> 0 such that
∀x,a ∥x∥ ≤ ∥ z(1)∥ ≤ A∥x∥ ,
where ∥z(1)∥2 = ∑
λ ∈Λ 1 ∥z(1)(·,λ)∥2. By deﬁning Ω 1 = Ω 0 × Λ 1, the ﬁrst layer of the network
can be written as the linear mapping
F1 : L2(Ω 0) − → L2(Ω 1)
x(u) ↦− → z(1)(u,λ) .
z(1) is then transformed with a point-wise nonlinear operatorM : L2(Ω) → L2(Ω) which is usually
non-expansive, meaning that ∥Mz∥ ≤ ∥ z∥. Finally, a local pooling operator P can be deﬁned as
any linear or nonlinear operator
P : L2(Ω) − →L2(˜Ω)
which reduces the resolution of the signal along one or more c oordinates and which avoids “alias-
ing”. If Ω = Ω 0 × Λ 1,, ×Λ k and (2J0 ,, 2Jk ) denote the loss of resolution along each coordinate, it
results that ˜Ω = ˜Ω 0 × ˜Λ 1,, ×˜Λ k, with |˜Ω 0| = 2 −αJ 0 |Ω 0|, |˜Λ i| = 2 −αJ i |Λ i|, where αis an oversam-
pling factor. Linear pooling operators are implemented as l owpass ﬁlters φJ (u,λ1,,λ m) followed
by a downsampling.
Then, a k-layer convolutional network is a cascade
L2(Ω 0)
M◦F1
− → L2(Ω 1)
P1
− →L2(˜Ω 1)
M◦F2
− → L2(Ω 2) · · ·
Pk
− →L2(˜Ω k) , (3)
which produces successively z(1),z(2),...,z (k).
The ﬁlter banks (Fi)i≤k, together with the pooling operators (Pi)i≤k, progressively transform the
signal domain; ﬁlter bank steps lift the domain of deﬁnition by adding new coordinates, whereas
pooling steps reduce the resolution along certain coordinates.
3 Invariance Properties of Convolutional Networks
3.1 The case of one-parameter transformation groups
Let us start by assuming the simplest form of variability pro duced by a transformation group. A
one-parameter transformation group is a family {Ut}t∈R of unitary linear operators of L2(Ω) such
2
that (i) t ↦→Ut it is strongly continuous: limt→t0 Utz = Ut0 z for every z ∈ L2(Ω) , and (ii)
Ut+s = UtUs. One parameter transformation groups are thus homeomorphic to R (with the addition
as group operation), and deﬁne an action which is continuousin the group variable. Uni-dimensional
translations Utx(u) = x(u− tv0), frequency transpositions Utx= F−1(Fx(ω− tω0)) (where F,
F−1 are respectively the forward and inverse Fourier transform ) or unitary dilations Utx(u) =
2−t/ 2x(2−tu) are examples of one-parameter transformation groups.
One-parameter transformation groups are particularly simple to study thanks to Stone’s theorem [5],
which states that unitary one-parameter transformation groups are uniquely generated by a complex
exponential of a self-adjoint operator:
Ut = eitA , t∈ R .
Here, the complex exponential of a self-adjoint operator sh ould be interpreted in terms of its spec-
tra. In the ﬁnite dimensional case (when Ω is discrete), this means that there exists an orthogonal
transform Osuch that if ˆz(ω) = Oz, then
∀ z, Utz= O−1diag(eitω .ˆz(ω)) . (4)
In other words, the group action can be expressed as a linear p hase change in the basis which
diagonalizes the unique self-adjoint operator Agiven by Stone’s theorem. In the particular case of
translations, the change of basis O is given by the Fourier transform. As a result, one can obtain
a representation which is invariant to the action of {Ut}t with a single layer of a neural network:
a linear decomposition which expresses the data in the basis given by Ofollowed by a point-wise
complex modulus. In the case of the translation group, this corresponds to taking the modulus of the
Fourier transform.
3.2 Presence of deformations
Stone’s theorem provides a recipe for global group invariance for strongly continuous group actions.
Without noise nor deformations, an invariant representati on can be obtained by taking complex
moduli in a basis which diagonalizes the group action, which can be implemented in a shallow 1-
layer architecture. However, the underlying low-dimensio nal assumption is rarely satisﬁed, due to
the presence of more complex forms of variability.
This complex variability can be modeled as follows. IfOis the basis which diagonalizes a given one-
parameter group, then the group action is expressed in the ba sis F−1O as the translation operator
Tsz(u) = z(u− s). Whereas the group action consists in rigid translations onthis basis, by analogy
a deformation is deﬁned as a non-rigid warping in this domain: Lτ z(u) = z(u− τ(u)), where τ is
a displacement ﬁeld along the indexes of the decomposition.
The amount of deformation can be measured with the regularit y of τ(u), which controls how dis-
tant the warping is from being a rigid translation and hence a n element of the group. This suggests
that, in order to obtain stability to deformations, rather t han looking for eigenvectors of the in-
ﬁnitesimal group action, one should look for linear measure ments which are well localized in the
domain where deformations occur, and which nearly diagonal ize the group action. In particular,
these measurements can be implemented with convolutions using compactly supported ﬁlters, such
as in convolutional networks.
Let z(n)(u,λ1,...,λ n) be an intermediate representation in a convolutional netwo rk, and whose
ﬁrst layer is fully connected. Suppose that Gis a group acting on zvia
g.z(n)(u,λ1,...,λ n) = z(n)(u,λ1 + η(g),λ2,...,λ n) , (5)
where η : G→ Λ 1. This corresponds to the idealized case where the transform ation only modiﬁes
one component of the representation. A local pooling operat or along the variable λ1, at a certain
scale 2J , attenuates the transformation bygas soon as|η(g)| ≪ 2J . It thus produces local invariance
with respect to the action of G.
3.3 Group Factorization with Deep Networks
Deep convolutional networks have the capacity to learn comp lex relationships of the data and to
build invariance with respect to a large family of transform ations. These properties can be partly
explained in terms of a factorization of the invariance groups performed successively.
3
Whereas pooling operators efﬁciently produce stable local invariance, convolution operators pre-
serve the invariance generated by previous layers. Indeed, suppose z(n)(u,λ1) is an interme-
diate representation in a convolutional network, and that G acts on z(n) via g.z(n)(u,λ1) =
z(n)(f(g,u),λ1 + η(g)). It follows that if the next layer is constructed as
z(n+1)(u,λ1,λ2) := z(n)(u,·) ⋆ψλ 2 (λ1) ,
then Gacts on z(n+1) via g.z(n+1)(u,λ1,λ2) = z(n)(f(g,u),λ1 + η(g),λ2), since convolutions
commute with the group action, which by construction is expressed as a translation in the coefﬁcients
λ1. The new coordinates λ2 are thus unaffected by the action of G.
As a consequence, this property enables a systematic proced ure to generate invariance to groups
of the form G = G1 ⋊ G2 ⋊ ... ⋊ Gs, where H1 ⋊ H2 is the semidirect product of groups. In
this decomposition, each factor Gi is associated with a range of convolutional layers, along th e
coordinates where the action of Gi is perceived.
4 Perspectives
The connections between group invariance and deep convolutional networks offer an interpretation
of their efﬁciency on several recognition tasks. In particu lar, they might explain why the weight
sharing induced by convolutions is a valid regularization method in presence of group variability.
More concretely, we shall also concentrate on the following aspects:
• Group Discovery. One might ask for the group of transformations which best ex plains the
variability observed in a given dataset {xi}. In the case where no geometric deformations
are present, one can start by learning the (complex) eigenvectors of the group action:
U∗ = arg min
UT U=1
var(|Uxi|) .
When the data corresponds to a uniform measure on the group, t hen this decomposition
can be obtained from the diagonalization of the covariance operator Σ = E(xT x). In that
case, the real eigenvectors of Σ are grouped into pairs of vectors with identical eigenvalue,
which then deﬁne the complex decomposition diagonalizing the group action.
In presence of deformations, the global invariance is replaced by a measure of local invari-
ance. This problem is closely related to the sparse coding with slowness from [6].
• Structured Convolutional Networks . Groups offer a powerful framework to incorporate
structure into the families of ﬁlters, similarly is in [7]. O n the one hand, one can enforce
global properties of the group by deﬁning the convolutions a ccordingly. For instance, by
wrapping the domain of the convolution, one is enforcing a pe riodic group to emerge. On
the other hand, one could further regularize the learning by enforcing a group structure
within a ﬁlter bank. For instance, one could ask a certain ﬁlt er bank F = {h1,...,h n} to
have the form F = {Rθ h0}θ , where Rθ is a rotation with an angle θ.
References
[1] J. Bruna, S. Mallat, “Invariant Scattering Convolution al Networks”, IEEE TPAMI, 2012.
[2] S. Mallat, “Group Invariant Scattering”, CPAM, 2012.
[3] L.Sifre, S.Mallat, “Combined Scattering for Rotation I nvariant Texture Analysis”, ESANN,
2012.
[4] Y .LeCun, L.Bottou, Y .Bengio, P.Haffner, “Gradient-Ba sed Learning Applied to Document
Recognition”, IEEE, 1998
[5] M.H. Stone, “On one-parameter unitary Groups in Hilbert Space”, Ann. of Mathematics, 1932.
[6] C.Cadieu, B.Olshausen, “Learning Transformational In variants from Natural Movies”, NIPS
2009.
[7] K.Gregor, A. Szlam, Y .Lecun, “Structured Sparse Codingvia lateral inhibition”, NIPS, 2011.
4


Peer review:
[Review]:
This short paper presents a discussion on the nature and the type of invariances that are represented and learned by convolutional neural networks.  It claims that the invariance a layer in a convolutional neural network can be expressed with a Lie group, and that the invariance of a deep convolutional neural network can be expressed with a product of groups.

This is a discussion paper that is difficult to understand without being familiar with group theory.  It would be easier to read if there were even toy examples that illustrate the concepts presented in this work.  In its current form the paper is incomplete; to be useful, it needs to use these ideas to somehow improve the training or generalization of convolutional neural networks.  On a related note, it is hard to understand the significance of the results.  So the invariance of a deep convolutional neural network can be expressed with a semi-direct product of some groups; this is nice, but what does it lead to, how can it be used?  

To summarize, paper has intriguing ideas, but they are not sufficiently developed, and their significance is not clearly explained.