Feature grouping from spatially constrained multiplicative interaction
Authors: Felix Bauer, Roland Memisevic
ICLR 2013

Abstract: We present a feature learning model that learns to encode relationships between images. The model is defined as a Gated Boltzmann Machine, which is constrained such that hidden units that are nearby in space can gate each other's connections. We show how frequency/orientation 'columns' as well as topographic filter maps follow naturally from training the model on image pairs. The model also helps explain why square-pooling models yield feature groups with similar grouping properties. Experimental results on synthetic image transformations show that spatially constrained gating is an effective way to reduce the number of parameters and thereby to regularize a transformation-learning model.

Feature grouping from spatially constrained
multiplicative interaction
Felix Bauer
Frankfurt Institute for Advanced Studies
Frankfurt, Germany
fbauer@fias.uni-frankfurt.de
Roland Memisevic
University of Montreal
Montreal, Canada
memisevr@iro.umontreal.ca
Abstract
We present a feature learning model that learns to encode relationships between
images. The model is deﬁned as a Gated Boltzmann Machine, which is con-
strained such that hidden units that are nearby in space can gate each other’s con-
nections. We show how frequency/orientation “columns” as well as topographic
ﬁlter maps follow naturally from training the model on image pairs. The model
also offers a simple explanation why group sparse coding and topographic feature
learning yields features that tend to by grouped according to frequency, orienta-
tion and position but not according to phase. Experimental results on synthetic
image transformations show that spatially constrained gating is an effective way
to reduce the number of parameters and thereby to regularize a transformation-
learning model.
1 Introduction
Feature-learning methods have started to become a standard component in many computer-vision
pipelines, because they can generate representations which are better at encoding the content of
images than raw images themselves. Feature learning works by projecting local image patches onto
a set of feature vectors (aka. “ﬁlters”), and using the vector of ﬁlter responses as the representation of
the patch. This representation gets passed on to further processing modules like spatial pooling and
classiﬁcation. Filters can be learned using a variety of criteria, including maximization of sparseness
across ﬁlter responses [18], minimizing reconstruction error [24], maximizing likelihood [5], and
many others. Under any of these learning criteria, Gabor features typically emerge when training on
natural image patches.
There has been an increasing interest recently in imposing group structureon learned ﬁlters. For
learning, ﬁlters are encouraged to come in small groups, such that all members of a group share
certain properties. The motivation for this is that group structure can explain several biological phe-
nomena such as the presence of complex cells [6], it provides a simple way to model dependencies
between features (eg., [9] and references therein), and it can make learned representations more
robust to small transformations which is useful for recognition [10]. Filter grouping is referred to
also as “structured sparse coding” or “group sparse coding”. Feature grouping can also be used as a
way to obtain topographic feature maps[8]. To this end, features are layed out in a 2-dimensional
grid, and groups are deﬁned on this grid such that each ﬁlter group shares ﬁlters with its neighboring
groups. In other words, groups overlap with each other. Training feature grouping and topographic
models on natural image patches typically yields Gabor ﬁlters whose frequency, orientation and po-
sition is similar for all the ﬁlters within a group. Phase, in contrast, tends to vary randomly across
the ﬁlters within a group (eg. [7, 10, 8]).
Various approaches to performing group sparse coding and topographic feature learning have been
proposed. Practically all of these are based on the same recipe: The set of ﬁlters is pre-partitioned
1
arXiv:1301.3391v3  [cs.LG]  11 Mar 2013
into groups before training. Then during learning, a second layer computes a weighted sum over
the squares of all ﬁlter responses within a group (eg., [6, 9, 10]). The motivation for using square-
pooling architectures to learn group structure is based in part on empirical evidence (it seems to work
across many models and learning objectives). There have also been various attempts to explain it,
based on a variety of heuristics: One explanation, for example, follows from the fact thatsome non-
linearity must be used before pooling features in a group, because in the absence of a nonlinearity,
we would wind up with a standard (linear) feature learning model. The square is a non-linearity that
is simple and canonical. It can also be shown that even-symmetric functions, like the square, applied
to ﬁlter responses, show strong dependencies. So they seem to capture a lot of what we are missing
after performing a single layer of feature learning [8]. Another motivation is that Gabor features
are local Fourier components, so computing squares is like computing spectral energies. Finally, it
can be shown that in the presence of a upstream square-root non-linearity, using squared features
generalizes standard feature learning, since it degenerates to a standard feature-learning model when
using group size 1 [10]. For a summary of these various heuristics, see [8] (page 215). Topographic
grouping of frequency, orientation and position is also a well-known feature of mammalian brains
(eg., [8]). Thus, another motivation for using square-pooling in feature learning in general has been
that, by means of replicating this effect, it may yield models that are biologically consistent.
In this work, we show that a natural motivation for the use of squared ﬁlter responses can be de-
rived from the perspective of encoding relationships between images. In particular, we show that
the emergence of group structure and topography follows automatically from the computation of
binocular disparity or motion, if we assume that neurons that are nearby in space exert multiplica-
tive inﬂuences on each other. Our work is based on the close relationship between the well-known
“energy models” of motion and binocularity [1, 17] and the equivalent “cross-correlation” models
[3, 14]. It may help shed light onto the close relationship between topographic organization and
motion processing as well as binocular vision. That way, it may also help shed light onto the phe-
nomenon that topographic ﬁlter maps do not seem to be present in rodents [20].
2 Factored Gated Boltzmann Machine
While feature learning has been applied predominantly to single, static images in the past, there has
been an increasing interest recently in learning features to encode relationships between multiple
images, for example, to encode motion (eg., [11, 22]). We focus in this work on the Gated Boltzmann
Machine (GBM) [15, 22] which models the relationship between two binary images x and y using
the three-way energy function
E(x,y,h) =
∑
ijk
wijkxiyjhk (1)
The energy gets exponentiated and normalized to deﬁne the probability over image pairs (we drop
any bias-terms here to avoid clutter):
p(x,y) = 1
Z
∑
h
exp
(
E(x,y,h)
)
, Z=
∑
x,y,h
exp
(
E(x,y,h)
)
(2)
By adding appropriate penalty terms to the log-likelihood, one can extend the model to learn real-
valued images [15].
Since there is a product involving every triplet of an input pixel, an output pixel and a mapping
unit, the number of parameters is roughly cubic in the number of pixels. To reduce that number,
[16] suggested factorizing the three-way parameter tensor W with entries wijk in Equation 1 into a
three-way inner product:
wijk =
∑
f
wx
if wy
jf wh
kf (3)
Here, f is a latent dimension that has to be chosen by hand or by cross-validation. This form of
tensor factorization is also known as PARAFAC or “canonical decomposition” in the literature, and
it can be viewed as a three-way generalization of the SVD [2]. It is illustrated in Figure 1 (left). The
factorization makes use of a diagonal core tensor, that contains ones along its diagonal and that is
zero elsewhere. Plugging in the factorized representation forW and using the distributive law yields
2
x y
h
x y
h
· · ·
Figure 1: Factorizing the parameter tensor of a Gated Boltzmann Machine ( left) is equivalent to
gating ﬁlter responses instead of raw pixels ( middle). In a group-gating model ( right), ﬁlters are
grouped, and products of all ﬁlters within a group are passed on to the mapping units. In the example,
there are two groups, resulting in four unique products per group.
the factorized energy function:
E =
∑
f
(∑
i
wx
if xi
)(∑
j
wy
jf yj
)(∑
k
wh
kf hk
)
(4)
Inferring the transformation h, given two images, x and y, is efﬁcient, because the hidden variables
are independent, given the data [16]. More speciﬁcally we have p(h|x,y) =∏
k p(hk|x,y) with
p(hk|x,y) =σ
(∑
f
wh
kf
(∑
i
wx
if xi
)(∑
j
wy
jf yj
))
, (5)
where σis the logistic sigmoid σ(z) = 1
1+exp(−z) . Thus, to perform inference, images are projected
onto F basis functions (“ﬁlters”) and those basis functions that are in correspondence (i.e., that have
the same index f) are multiplied. Finally, each hidden unit, hk, receives a weighted sum over all
products as input. An illustration is shown in Figure 1 (middle, right).
It is important to note that projections onto ﬁlters typically donot serve to reduce the dimensionality
of inputs. Rather, it is the restricted connectivity in the projected space that leads to the reduction in
the number of parameters. In fact, it is not uncommon to use a number of factors that is larger than
the dimensionality of the input data (for example, [16, 19]).
To train the model, one can use contrastive divergence [16], score-matching or a variety of other
approximations. Recently, [21, 13] showed that one may equivalently add a decoder network, effec-
tively turning the model into a “gated” version of a de-noising auto-encoder [24] and train it using
back-prop. Inference is then still the same as in a Boltzmann machine. We use this approach in
most of our experiments, after verifying that the performance is similar to contrastive divergence.
More speciﬁcally, for a training image pair (x,y), let ph(x,y) denote the vector of inferred hidden
probabilities (Eq. 5) and let Wx,Wy,Wh denote the matrices containing the input ﬁlters, output
ﬁlters, and hidden ﬁlters, respectively (stacked column-wise). Training now amounts to minimizing
the average reconstruction error
(
y −ˆy(x)
)2
+
(
x −ˆx(y)
)2
, with
ˆy(x) =Wy
((
WT
h ph(x,y)
)
∗
(
WT
x x
))
, (6)
where ∗denotes element-wise multiplication. ˆx(y) is deﬁned analogously, with all occurrences of
x and y exchanged. One may add noise to the data during training (but reconstruct the original, not
noisy, data to compute the cost) [24]. We observed that this helps localize ﬁlters on natural video,
but on synthetic data (shifts and rotations) it is possible to train without noise.
2.1 Phase usage in modeling transformations
Figure 2 (left, top) shows ﬁlter-pairs that were learned from translated random-dot images using
a Factored Gated Boltzmann Machine (for details on training, see Section 4 below). Training on
translations turns ﬁlters into Fourier components, as was initially observed by [16]. The left-bottom
plot shows histograms over the occurrence of frequencies and orientations for input- and output-
image ﬁlters, respectively. These were generated by ﬁrst performing a 2D-DFT on each ﬁlter and
3
Figure 2: Left, top: Filter pairs learned from translated random-dot images. (Input ﬁlters on the
left, output ﬁlters on the right). Left, bottom: Corresponding histograms showing the number
of ﬁlters per frequency/orientation bin. The size of each blob is proportional to the number of
ﬁlters in each bin. Right: Phase differences between learned ﬁlters. Each row corresponds to one
frequency/orientation bin. Colors are used to simplify visibility.
then picking the frequency and orientation of the strongest component for the ﬁlter. The histograms
show that the learned ﬁlters evenly cover the space of frequencies and orientations. This is to be
expected, as all frequencies and orientations contribute equally to the set of random translations
(e.g., [4]).
It is also well-known, however, that multiple different translations will affect each frequency and
orientation differently. More speciﬁcally, any given translation induces a set of phase-shifts for every
component of a given frequency and orientation. The phase-shift depends linearly on frequency
[4]. Likewise, two different image translations of the same orientation will induce two different
phase-shifts for each frequency/orientation. In order to represent translations, it is necessary to
specify the phase at every frequency and orientation. This shows that mapping units in a GBM
must have access to multiple different phase-shifts at every frequency and orientation to be able to
represent translations. Also, there is no need to multiply ﬁlter responses of different frequencies
and/or orientations, only of different phases.
As a repercussion, for each input ﬁlter in Figure 2 (left, top) there need to be multiple phase-shifted
copies present in the output-ﬁlter bank, so that ﬁlters with varying phase differences can be matched
(which means the ﬁlters’ responses will be multiplied with each other). Likewise for each output
ﬁlter. When the number of factors is small, the model has to ﬁnd a compromise, for example, by
connecting hidden variables, hk, to multiple phase-shifted versions of ﬁlters with the correct phase-
shift but with only a similar, but not the same frequency and orientation. Figure 2 (right) depicts the
occurrences of phase differences between input- and output-image ﬁlters. It shows that the model
learned to use a variety of phase differences at each frequency/orientation to represent the training
transformations. In the model, each phase difference corresponds to exactly one ﬁlter pair.
The analysis generalizes to other transformations, such as rotations or natural videos (e.g., [14]).
In particular, it also generalizes to Gabor features which are localized Fourier features that emerge
when training the GBM on natural video (e.g., [22]).
3 Group gating
The analysis in the previous section strongly suggests using a richer connectivity that supports the
re-use of ﬁlters. In this case the model can learn to match any ﬁlter wx
f with multiple phase-shifted
copies wy
f of itself rather than with a single one. All phase differences in Figure 2 (right) can then
in principle be obtained using a much smaller number of ﬁlters.
4
One way to support the re-use of ﬁlters to represent multiple phase-shifts is by relaxing the diagonal
factorization (Eq. 3) with a factorization that allows for a richer connectivity:
wijk =
∑
def
Cdef wx
idwy
jewz
kf (7)
where we Cdef are the components of a (non-diagonal) core tensor C. Note that, if C is diagonal
so that Cdef = 1iff d= e= f, we would recover the PARAFAC factorization (Eq. 3). The energy
function now turns into (cf. Eq. 10):
E =
∑
def
Cdef
(∑
i
wx
idxi
)(∑
j
wy
jeyj
)(∑
k
wh
kf hk
)
(8)
and inference into
p(hk|x,y) =σ
(∑
def
Cdef wh
kf
(∑
i
wx
idxi
)(∑
j
wy
jeyj
))
(9)
As the number of factors is typically large, a full matrixCwould be computationally too expensive.
In fact, as we discuss in Section 2 there is no other reason to project onto ﬁlters than reducing
connectivity in the projected representation. Also, by the discussion in the previous section, there is
very a strong inductive bias towards allowing groups of factorsto interact.
This suggests using a core-tensor that allows features to come in groups of a ﬁxed, small size, such
that all pairs of ﬁlters within a group can provide products to mapping units. By the analysis in
Section 2, training on translations or natural video is then likely to yield groups of roughly constant
frequency and orientation and to differ with respect to phase. We shall refer to this model as “group-
gating” model in the following. As the values Cdef may be absorbed into the factor matrices and
are learned from data, it is sufﬁcient to distinguish only between non-zero and zero entries inC, and
we set all non-zero entries to one in what follows.
By deﬁning the ﬁlter groups Gg,g = 1,...,G , we can write inference in the model consequently as
p(hk|x,y) =σ
(∑
g
∑
d∈Gg
∑
e∈Gg
wh
k,d·|Gg|+e
(∑
i
wx
idxi
)(∑
j
wy
jeyj
))
(10)
which is illustrated in Figure 1 (right). Note that each hidden unit can still pool over all pair-wise
products between features. The overall number of feature products is equal to the number of groups
times the group size. In practice, it makes sense to set the number of factors to be a multiple of the
group size, so that all groups can have the same size.
A convenient way to implement the group-gating model is by computing all required products by
creating Gcopies of the input-factor matrix and Gcopies of the output-factor matrix and permuting
the columns (factors) of one of the two matrices appropriately. (Note in particular that when using
a large number of factors, masking, i.e. forcing a subset of entries of C to be zero, would not
be feasible.) It is possible to reduce the number of ﬁlters further by allowing for multiplicative
interactions between only one ﬁlter per group from the input image and all ﬁlters from the output
image (or vice versa). This leads to an asymmetric model, where the number of ﬁlters is not equal
for both images.
3.1 Signiﬁcance for square-pooling models
It is interesting to note that the same analysis applies to the responses of energy-model complex cells
(e.g., [7, 19]), too, if images and ﬁlters are contrast normalized [14]. In this case, the response of the
energy model is the same as a factored GBM mapping unit applied to a single image, i.e.x = y (see,
for example, [19, 14]). This shows that the gating interpretation of frequency/orientation groups and
topographic structure applies to these models, too.
In the same way, we can interpret a square-pooling model applied to a single imageas an encod-
ing of the relationship between, say, the rows or the columns within the patch. In natural images,
the predominant transformation that takes a row to the next row is a local translation. The emer-
gence of oriented Gabor features can therefore also be viewed as the result of modeling these local
translations.
5
Figure 3: Classiﬁcation accuracy of a Gated Boltzmann Machine vs. a cRBM on shifts (left) and
rotations (right). A total of 25 different sets of parameters were tested, with the numbers of ﬁlters
and mapping units being varied between parameter sets.
4 Experiments
Cross-correlation models vs. energy models: To get some quantitative insights into the rela-
tionship between cross-correlation and energy models, we compared the standard Gated Boltzmann
Machine with a square-pooling GBM [e.g. 19, 13] trained on the concatenation of the images. We
used the task of classifying transformations from the mapping units, using transformations for which
we can control the ground truth. The models are trained on image patches of size 13 ×13 pixels,
which are cropped from larger images to ensure that no border artifacts are introduced. Training-,
validation- and test-data contain 100.000 cases each. We use logistic regression to classify transfor-
mations from mapping units, where we determine the optimal number of mapping units and factors,
as well as the learning rates for the transformation models on the validation data. Using random
images ensures that single images contain no information about the transformation; in other words,
a single image cannot be used to predict the transformation.
We compare the models on translations and rotations. Shifts are drawn uniform-randomly in the
range [−3,3] pixels. We used four different labels corresponding to the four quadrants of motion
direction. Rotations are drawn from a von Misesdistribution (range [−π,π] rad), which we scaled
down to a maximal angle of 36◦. We used 10 labels by equally dividing the set of rotation angles.
The results are shown in Figure 3 and they demonstrate that both types of model do a reasonably
good job at prediction the transformations from the image pairs. The experiment veriﬁes the approx-
imate equivalence of the two types of model derived, for example, in [3, 14]. The cross-correlation
model does show slightly better performance than the energy model for large data-set sizes, and the
difference gets more pronounced as the training dataset size is decreased, which is also in line with
the theory. Energy models have been the standard approach to feature grouping in the past [8].
Learning simple transformations: We trained the group-gating model with group size 3, 128
mapping units and 392 ﬁlters on translating random-dot images of size 13 ×13 pixels. Figure 4
shows three pairs of plots, where the left, center and right pair depict the dominant frequencies,
orientations and phases of the ﬁlters, respectively. We extracted these from the ﬁlters using FFT.
Each pair shows properties of input (top) and output ﬁlters (bottom).
Within an image, each ﬁlter is represented by a single uni-colored square. For frequency plots,
the squares are in gray-scale with the brightness corresponding to the frequency of the ﬁlter (black
represents frequency zero, white is the highest frequency across all ﬁlters); in the orientation and
phase plots, the squares differ in color, where the angle determines the color according to the HSV
representation of RGB color space. The ﬁgures conﬁrm that ﬁlter-groups tend to be homogeneous
with respect to frequency and orientation 1. Contrary to that, phases differ within each group, as
expected. The same can be seen in Figure 5 (left and middle), which shows subsets of ﬁlter groups
learned from translations and rotations of random images, using patchsize25 ×25 and group size 5.
1In some groups, there appear to be some outliers, whose frequency or orientation does not match the other
ﬁlters in the group. These are typically the result of spurious maxima in the FFT amplitude due to small
irregularities in the ﬁlters.
6
Figure 4: Frequency (left), orientation (middle) and phase (right) of ﬁlters trained on translated
image pairs, with ﬁlters applied to the input images in the top and ﬁlters for the output image on the
bottom. A group size of 3 is used here, which means each three consecutive input ﬁlters and their
corresponding output ﬁlters form a group. Notice that frequency and orientation are often identical
for all ﬁlters within a group, whereas the phase varies. Best viewed in color.
Figure 5: Filter-groups learned from translations (left) and rotations (middle) using a group-gating
model with group size 5. Each group corresponds to one column in the plot. Right: Filter-groups
learned from natural videos, using a group-gating model with group size 4. Only output ﬁlters are
shown. Frequency, orientation and position are constant, while phase varies, within each group.
Learning transformation features from natural videos: Figure 5 (right) shows a subset of ﬁlter
groups learned from about one million patches of size 12 ×12 pixels that we crop from the van
Hateren broadcast television database [23]. We trained the asymmetric model to predict each frame
from its predecessor. All patches were PCA-whitened, retaining 95% of the variance. We trained
a model with 128 mapping units and 256 ﬁlters, and we used a group size of 4. The ﬁgure shows
that the model learns Gabor features, where frequency, orientation and position are nearly constant
within groups, and the set of possible phases is covered relatively evenly per group.
Quantitative results for group-gating: Classiﬁcation accuracies for the same model and data as
described in Section 4 are reported in Table 1. The equivalent number of group-gating ﬁlters is
shown in parentheses, where equivalence means that the group-gating model has the same num-
ber of parameters in total as the factored GBM. This makes it possible to obtain a comparison
that is fair in terms of pure parameter count by comparing performance across the columns of the
tables. However, the table also shows that even along columns, the group-gating model robustly
outperforms the factored GBM, except when using a very small number of factors. In this case, the
parameter-equivalent number of 44 factors is probably too small to represent the transformations
with a sufﬁciently high resolution.
4.1 Topographic feature maps from local gating
We also experimented with overlapping group structures by letting Gg share ﬁlters. This makes it
necessary to deﬁne which ﬁlters are shared among which groups. A convenient approach is to lay
out all features (“simple cells”) in a low-dimensional space and to deﬁne groups over all those units
which reside within some pre-deﬁned neighborhoods, such as in a grid of sizen×nunits (e.g., [8]).
Figure 6 shows the features learned from thevan Haterendata (cf., Section 4) with patchsize16×16
and 99.9% variance retained after whitening. We used 400 ﬁlters that we arranged in a 2-D grid
(with wrap-around) using a group size of 5 ×5. We found that learning is simpliﬁed when we also
7
TRANSLATION ROTATION
NUMBER OF FILTERS GATING GROUP GATING GATING GROUP GATING
81 (44) 89.69% 81.68% 90.08% 89.06%
225 (121) 92.40% 92.59% 91.50% 94.65%
441 (237) 92.20% 93.36% 90.76% 95.65%
729 (392) 91.44% 93.46% 91.71% 95.51%
900 (483) 91.29% 93.20% 89.95% 95.00%
Table 1: Classiﬁcation accuracy of the standard and group-gating models on image pairs showing
translations and rotations.
Figure 6: Topographic ﬁlter maps learned by laying out features in two dimensions and allowing for
multiplicative interactions among nearby units. Left: Input ﬁlters, Right: Output ﬁlters.
initialize the factor-to-mapping parameters so that each mapping unit has access to the ﬁlters of only
one group.
The ﬁgure shows how learned ﬁlters are arranged in two topographic feature maps with slowly
varying frequency and orientation within each map. It also shows how low-frequency ﬁlters have
the tendency of being grouped together [8]. From the view of phase analysis (see Section 2), the
topographic organization is a natural result of imposing an additional constraint: Filter sets that
come in similar frequency, orientation and position are now forced to share coefﬁcients with those
of neighboring groups. The simplest way to satisfy this constraint is by having nearby groups be
similar with respect to the properties they share. The apparent randomness of phase simply follows
from the requirement that multiple phases be present within each neighborhood. It is interesting to
note that using shared group structure is equivalent to letting units that are nearby in space affect
each other multiplicatively. Localized gating may provide a somewhat more plausible explanation
for the emergence of pinwheel-structures than squaring non-linearities, which are used, for example,
in subspace models or topographic ICA (see [6, 7]).
5 Conclusions
Energy mechanisms and “square-pooling” are common approaches to modeling feature dependen-
cies in sparse coding, and to learn group-structured or invariant dictionaries. In this work we re-
visited group-structured sparse coding from the perspective of learning image motion and disparity
from local multiplicative interactions. Our work shows that the commonly observed constancy of
frequency and orientation of ﬁlter-groups in energy models can be explained as a result of rep-
resenting transformations with local, multiplicative feature gating. Furthermore, topographically
8
structured representations (“pinwheels”) can emerge naturally as the result of binocular or spatio-
temporal learning that utilizes spatially constrained multiplicative interactions. Our work may pro-
vide some support for the claim that localized multiplicative interactions are a biologically plausible
alternative to square pooling for implementing stereopsis and motion analysis [12]. It may also help
explain why the development of pinwheels in V1 may be tied the presence of binocular vision and
why topographic organization of features does not appear to occur, for example, in rodents [20].
References
[1] E.H. Adelson and J.R. Bergen. Spatiotemporal energy models for the perception of motion. J. Opt. Soc.
Am. A, 2(2):284–299, 1985.
[2] J. Douglas Carroll and Jih-Jie Chang. Analysis of individual differences in multidimensional scaling via
an N-way generalization of Eckart-Young decomposition. Psychometrika, 35(3), 1970.
[3] D. Fleet, H. Wagner, and D. Heeger. Neural encoding of binocular disparity: Energy models, position
shifts and phase shifts. Vision Research, 36(12):1839–1857, June 1996.
[4] Rafael C. Gonzalez and Richard E. Woods. Digital Image Processing (3rd Edition). Prentice Hall, 3
edition, August 2007.
[5] Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Compu-
tation, 14(8):1771–1800, 2002.
[6] Aapo Hyv ¨arinen and Patrik Hoyer. Emergence of phase- and shift-invariant features by decomposition of
natural images into independent feature subspaces. Neural Computation, 12:1705–1720, July 2000.
[7] Aapo Hyv ¨arinen, Patrik O. Hoyer, and Mika Inki. Topographic ICA as a model of natural image statistics.
In Biologically Motivated Computer Vision. Springer, 2000.
[8] Aapo Hyv ¨arinen, J. Hurri, and Patrik O. Hoyer. Natural Image Statistics: A Probabilistic Approach to
Early Computational Vision. Springer Verlag, 2009.
[9] Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, and Francis Bach. Proximal methods for sparse
hierarchical dictionary learning. In ICML, 2010.
[10] Koray Kavukcuoglu, Marc’Aurelio Ranzato, Rob Fergus, and Yann LeCun. Learning invariant features
through topographic ﬁlter maps. In CVPR 2009. IEEE, 2009.
[11] Q.V . Le, W.Y . Zou, S.Y . Yeung, and A.Y . Ng. Learning hierarchical spatio-temporal features for action
recognition with independent subspace analysis. In CVPR 2011, 2011.
[12] Bartlett W. Mel, Daniel L. Ruderman, and Kevin A. Archie. Toward a single-cell account for binocular
disparity tuning: an energy model may be hiding in your dendrites. In NIPS 1997.
[13] Roland Memisevic. Gradient-based learning of higher-order image features. In ICCV 2011, 2011.
[14] Roland Memisevic. On multi-view feature learning. In ICML 2012, 2012.
[15] Roland Memisevic and Geoffrey Hinton. Unsupervised learning of image transformations. In CVPR
2007, 2007.
[16] Roland Memisevic and Geoffrey E Hinton. Learning to represent spatial transformations with factored
higher-order Boltzmann machines. Neural Computation, 22(6):1473–92, 2010.
[17] Izumi Ohzawa, Gregory C. Deangelis, and Ralph D. Freeman. Stereoscopic Depth Discrimination in the
Visual Cortex: Neurons Ideally Suited as Disparity Detectors. Science, 249(4972):1037–1041, 1990.
[18] B. Olshausen and D. Field. Emergence of simple-cell receptive ﬁeld properties by learning a sparse code
for natural images. Nature, 381(6583):607–609, June 1996.
[19] Marc’Aurelio Ranzato and Geoffrey E. Hinton. Modeling Pixel Means and Covariances Using Factorized
Third-Order Boltzmann Machines. In CVPR 2010, 2010.
[20] CF Stevens. A universal design principle for visual system pinwheels. Brain, behavior and evolution,
77(3):132, 2011.
[21] Kevin Swersky, Marc’Aurelio Ranzato, David Buchman, Benjamin Marlin, and Nando Freitas. On au-
toencoders and score matching for energy based models. In ICML 2011, 2011.
[22] Graham, W. Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler. Convolutional learning of spatio-
temporal features. In ECCV 2010, 2010.
[23] L. van Hateren and J. Ruderman. Independent component analysis of natural image sequences yields
spatio-temporal ﬁlters similar to simple cells in primary visual cortex. Proc. Biological Sciences,
265(1412):2315–2320, 1998.
[24] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and compos-
ing robust features with denoising autoencoders. In ICML 2008, 2008.
9


Peer review:
[Review]:
Points raised by reviewers:
reviewer 43a2:
(1) Good classification of rotations and scale are reported in Table 1, unfortunately these appear to be on toy, not natural, images. Impressive grouping of complex transformations such as translations and rotations are shown in Figure 3.
(2) While the gabors learned on natural image patches are interesting it is hard to judge them without knowing how large they are. These details seem to be omitted from the paper.
(3) It is not immediately obvious what applications would benefit from this type of model. It also seems like it could be relatively expensive computationally, and there was no mention of timing versus the standard gated Boltzmann machine model.

reviewer ea89:
(4) Please write the formulas for the full model that you train, not just the encoding. Even though they exist in other papers, they are not so complicated to write them down here.
(5) You say that the pinwheel patterns don't appear in rodents because they don't have a binocular vision. However you haven't actually obtained the pinwheels from binocularity but from video.
(6) The formula (9) is unclear and should be fixed. For example, how come the f index appears only once?
(6a) Figure 3: What is index of parameter set? In text you talk about different datasizes - where are the results for these?

reviewer cce5:
(7a) * Targets somewhat of a 'niche audience'; may be less accessible to the general representation learning community
(7b) * Presents a lot of qualitative but not quantitative results
(8) Fig 2: It's difficult to read/understand the frequency scale (left, bottom); it seems that frequency has been discretized; what do these bins represent, and how are they constructed?
(9) In section 2.1, could you be more explicit about what you mean by 'matching' the input filters (and the output filters). I assume the matching is referring to the connectivity by connecting filters to mapping units? Matching comes up again in Section 3, so it would help to clarify this early on.
(10) Check equation 8: what happened to C - should it not show up there?

Our response:
We thank the reviewers for their comments and suggestions. We submitted an updated version of the paper in which we address these points:

(1, 7b) We agree. While the toy results do suggest that the reduction in the number of parameters caused by grouping helps generalize, real world applications like activity recognition work much better with local receptive fields and pooling, which we feel is much too complicated for a first paper in this direction.
(2) We updated the paper to include a more detailed description of the datasets and experiments.
X (3) We included a discussion of computational complexity. The complexity scales with the square of the group size (which is typically small, like 5). This is not a huge increase in complexity because without grouping the model has to account for the equivalent number of products by replicating filters.
(4) We included the equations as suggested.
(5) This is a good point, and we now clarify this in the updated version of the paper. Binocular stimuli, like video, are dominated by local translation, and the exact same biological mechanisms have traditionally been assumed to model both (multiview complex cells). A simple GBM has in fact been applied to binocular 3D inference tasks in the past (eg. 'Stereopsis via Deep Learning', Memisevic, Conrad, 2011).
(6) We fixed this in the updated version.
(6a): We now describe the figure in more detail in the updated version. (Along the x-axis we show models with varying numbers of factors and mapping units)
(6a): We now describe the figure in more detail in the updated version. (Along the x-axis we show models with varying numbers of factors and mapping units)
(7a): While the model and experiments we discuss in the paper are very specific and technical, we see the main contribution of the paper in explaining concisely why square-pooling, group sparse coding and topographic feature models learn to group frequency, orientation and position and not phase. While it is well-known that they show this behaviour we show that thinking of squares as representing transformations can explain why. We rewrote the text to make this point clearer.
(8) They are DFT bins: We generate the plots by performing a 2d FFT on the learned Fourier filters (note that, since we train on translations in this experiment, we get Fourier components rather than Gabor features). We clarified this in the updated version.
(9) We use 'matching' synonymous with 'multiplying'. Indeed, this means that 'matched' filters are those whose product gets fed into a mapping unit (along with other products). We clarified this in the updated version.
(10) Yes. We fixed this in the updated version.