Learnable Pooling Regions for Image Classification
Authors: Mateusz Malinowski, Mario Fritz
ICLR 2013

Abstract: From the early HMAX model to Spatial Pyramid Matching, pooling has played an important role in visual recognition pipelines. Spatial pooling, by grouping of local codes, equips these methods with a certain degree of robustness to translation and deformation yet preserving important spatial information. Despite the predominance of this approach in current recognition systems, we have seen little progress to fully adapt the pooling strategy to the task at hand. This paper proposes a model for learning task dependent pooling scheme -- including previously proposed hand-crafted pooling schemes as a particular instantiation. In our work, we investigate the role of different regularization terms used in the proposed model together with an efficient method to train them. Our experiments show improved performance over hand-crafted pooling schemes on the CIFAR-10 and CIFAR-100 datasets -- in particular improving the state-of-the-art to 56.29% on the latter.

Learnable Pooling Regions for Image Classiﬁcation
Mateusz Malinowski
Computer Vision and Multimodal Computing
Max Planck Institute for Informatics
Campus E1 4, 66123 Saarbr¨ucken, Germany
mmalinow at mpi-inf.mpg.de
Mario Fritz
Computer Vision and Multimodal Computing
Max Planck Institute for Informatics
Campus E1 4, 66123 Saarbr¨ucken, Germany
mfritz at mpi-inf.mpg.de
Abstract
Biologically inspired, from the early HMAX model to Spatial Pyramid Match-
ing, pooling has played an important role in visual recognition pipelines. Spatial
pooling, by grouping of local codes, equips these methods with a certain degree
of robustness to translation and deformation yet preserving important spatial in-
formation. Despite the predominance of this approach in current recognition sys-
tems, we have seen little progress to fully adapt the pooling strategy to the task at
hand. This paper proposes a model for learning task dependent pooling scheme
– including previously proposed hand-crafted pooling schemes as a particular in-
stantiation. In our work, we investigate the role of different regularization terms
showing that the smooth regularization term is crucial to achieve strong perfor-
mance using the presented architecture. Finally, we propose an efﬁcient and par-
allel method to train the model. Our experiments show improved performance
over hand-crafted pooling schemes on the CIFAR-10 and CIFAR-100 datasets –
in particular improving the state-of-the-art to 56.29% on the latter.
1 Introduction
Spatial pooling plays a crucial role in modern object recognition and detection systems. Motivated
from biology [Riesenhuber and Poggio, 2009] and statistics of locally orderless images [Koenderink
and Van Doorn, 1999], the spatial pooling approach has been found useful as an intermediate step
of many today’s computer vision methods. For instance, the most popular visual descriptors such as
SIFT [Lowe, 2004] and HOG [Dalal and Triggs, 2005], which compute local histograms of gradi-
ents, can be in fact seen as a special version of the spatial pooling strategy. In order to form more
robust features under translation or small object deformations, activations of the codes are pooled
over larger areas in a spatial pyramid scheme [Lazebnik et al., 2006, Yang et al., 2009]. Unfortu-
nately, this critical decision, namely the spatial division, is most prominently based on hand-crafted
algorithms and therefore data independent.
Related Work As large amounts of training data is available to us today,, there is an increasing
interest to push the boundary of learning based approaches towards fully optimized and adaptive
architectures where design choices, that would potentially constrain or bias a model, are kept to a
minimum. Neural networks have a great tradition of approaching hierarchical learning problems
and training intermediate representations [Ranzato et al., 2007, Le et al., 2012a]. Along this line, we
propose a learnable spatial pooling strategy that can shape the pooling regions in a discriminative
manner. Our architecture has a direct interpretation as a pooling strategy and therefore subsumes
popular spatial pyramids as a special case. Yet we have the freedom to investigate different regular-
ization terms that lead to new pooling strategies when optimized jointly with the classiﬁer.
Recent progress has been made in learning pooling regions in the context of image classiﬁcation
using the Spatial Pyramid Matching (SPM) pipeline [Lazebnik et al., 2006, Yang et al., 2009]. Jia
1
arXiv:1301.3516v3  [cs.CV]  5 May 2015
and Huang [2011], Jia et al. [2012] and Feng et al. [2011] have investigated how to further liber-
ate the recognition from preconceptions of the hand crafted recognition pipelines, and include the
pooling strategy into the optimization framework jointly with the classiﬁer. However, these methods
still make strong assumptions on the solutions that can be achieved. For instance Jia and Huang
[2011] optimizes binary pooling strategies that are given by the superposition of rectangular basis
functions, and Feng et al. [2011] ﬁnds pooling regions by applying a linear discriminant analysis
for individual pooling strategies and training a classiﬁer afterwards. Also as opposed to Ranzato
and Hinton [2010], we aim for discriminative pooling over large neighborhoods in the SPM fashion
where the information about the image class membership is available during training.
Outline We question restrictions imposed by the above methods and suggest to learn pooling
strategies under weaker assumptions. Indeed, our method discovers new pooling shapes that were
not found previously as they were suppressed by the more restrictive settings.
The generality that we are aiming for comes at the price of a high dimensional parameters space.
This manifests in a complex optimization problem that is more demanding on memory requirements
as well as computations needs, not to mention a possibility of over-ﬁtting. Therefore, we also discuss
two approximations to our method. First approximation introduces a pre-pooling step and therefore
reduces the spatial dimension of the codes. The second approximation divides the codes into a set
of smaller batches (subset of codes) that can be optimized independently and therefore in parallel.
Finally, we evaluate our method on the CIFAR-10 and show strong improvements over hand-crafted
pooling schemes in the regime of small dictionaries where our more ﬂexible model shows its capa-
bility to make best use of the representation by exploring spatial pooling strategies speciﬁc to each
coordinate of the code. Despite the diminishing return, the performance improvements persist up to
largest codes we have investigated. We also show strong classiﬁcation performance on the CIFAR-
100 dataset where our method outperforms, to the best of our knowledge, the state-of-the-art.
2 Method
As opposed to the methods that use ﬁxed spatial pooling regions in the object classiﬁcation task
[Lazebnik et al., 2006, Yang et al., 2009] our method jointly optimizes both the classiﬁer and the
pooling regions. In this way, the learning signal available in the classiﬁer can help shaping the
pooling regions in order to arrive at better pooled features.
2.1 Parameterized pooling operator
The simplest form of the spatial pooling is computing histogram over the whole image. This can
be expressed as Σ(U) := ∑M
j=1 uj, where uj ∈RK is a code (out of M such codes) and an
index j refers to the spatial location that the code originates from 1. A code is an encoded patch
extracted from the image. The proposed method is agnostic to the patch extraction method and
encoding scheme. Since the pooling approach looses spatial information of the codes, Lazebnik
et al. [2006] proposed to ﬁrst divide the image into subregions, and afterwards to create pooled
features by concatenating histograms computed over each subregion. There are two problems with
such an approach: ﬁrst, the division is largely arbitrary and in particular independent of the data;
second, discretization artifacts occur as spatially nearby codes can belong to two different regions
as the ’hard’ division is made.
In this paper we address both problems by using a parameterized version of the pooling operator
Θw(U) :=
M∑
j=1
wj ◦uj (1)
where a ◦b is the element-wise multiplication. Standard spatial division of the image can be re-
covered from Formula 1 by setting the vectors wj either to a vector of zeros 0, or ones 1. For
instance, features obtained from dividing the image into 2 subregions can be recovered from Θ by
1That is j = (x, y) where x and y refer to the spatial location of the center of the extracted patch.
2
concatenating two vectors: ∑M
2
j=1 1 ◦uj + ∑M
j= M
2 +1 0 ◦uj, and ∑M
2
j=1 0 ◦uj + ∑M
j= M
2 +1 1 ◦uj,
where
{
1,..., M
2
}
and
{M
2 + 1,...,M
}
refer to the ﬁrst and second half of the image respectively.
In general, let F := {Θw}w be a family of the pooling functions given by Eq. 1, parameterized
by the vector w, and let w∗,l be the ’best’ parameter chosen from the familyF based on the initial
conﬁguration land a given set of images.2 First row of Figure 2 shows four initial conﬁgurations that
mimic the standard 2-by-2 spatial image division. Every initial conﬁguration can lead to different
w∗,l as it is shown in Figure 2. Clearly, the family F contains all possible ’soft’ and ’hard’ spatial
divisions of the image, and therefore can be considered as their generalization.
2.2 Learnable pooling regions
In SPM architectures the pooling weights w are designed by hand, here we aim for joint learning
w together with the parameters of the classiﬁer. Intuitively, the classiﬁer during training has access
to the classes that the images belong to, and therefore can shape the pooling regions. On the other
hand, the method aggregates statistics of the codes over such learnt regions and pass them to the
classiﬁer allowing to achieve higher accuracy. Such joint training of the classiﬁer and the pooling
regions can be done by adapting the backpropagation algorithm [Bishop, 1999, LeCun et al., 1998],
and so can be interpreted as a densely connected multilayer perceptron [Collobert and Bengio, 2004,
Bishop, 1999].
Consider a sampling scheme and an encoding method producing M codes each K dimensional.
Every coordinate of the code is an input layer for the multilayer perceptron. Then we connect every
j-th input unit at the layer kto the l-th pooling unit ak
l via the relation wk
ljuk
j . Since the receptive
ﬁeld of the pooling unit ak
l consists of all codes at the layer k, we have ak
l := ∑M
j=1 wk
ljuk
j , and so
in the vector notation
al :=
M∑
j=1
wl
j ◦uj = Θwl(U) (2)
Next, we connect all pooling units with the classiﬁer allowing the information to circulate between
the pooling layers and the classiﬁer.
Although our method is independent of the choice of a dictionary and an encoding scheme, in this
work we use K-means with triangle coding fk(x) := max {0,µ(z) −zk}[Coates et al., 2011].
Similarly, every multi-class classiﬁer that can be interpreted in terms of an artiﬁcial neural network
can be used. In our work we employ logistic regression. This classiﬁer is connected to the pooling
units via the formula
J(Θ) := −1
D
D∑
i=1
C∑
j=1
1{y(i) = j}log p(y(i) = j|a(i); Θ) (3)
where Ddenotes the number of all images, Cis the number of all classes, y(i) is a label assigned to
the i-th input image, anda(i) are responses from the ’stacked’ pooling units[al]l for the i-th image3.
We use the logistic function to represent the probabilities: p(y = j|x; Θ) :=
exp(θT
j x)∑C
l=1 exp(θT
l x) . Since
the classiﬁer is connected to the pooling units, our task is to learn jointly the pooling parameters W
together with the classiﬁer parameters Θ, where W is the matrix containing all pooling weights.
Finally, we use standard gradient descent algorithm that updates the parameters using the following
ﬁxed point iteration
Xt+1 := Xt −γ∇J(Xt) (4)
where in our caseX is a vector consisting of the pooling parametersW and the classiﬁer parameters
Θ. In practice, however, we employ a quasi-Newton algorithm LBFGS4.
2 We will show the learning procedure that can select such parameter vectors in the following subsection.
3Providing the codes U(i) are collected from the i-th image and a(i)
l := Θwl(U(i)) then a(i) := [a(i)
l ]l.
4The algorithm, developed by Mark Schmidt, can be downloaded from the following webpage:
http://www.di.ens.fr/ mschmidt/Software/minFunc.html
3
2.3 Regularization terms
In order to improve the generalization, we introduce regularization of our network as we deal with
a large number of the parameters. For the classiﬁcation Θ and pooling parameters W, we employ
a simple L2 regularization terms: ||Θ||2
l2 and ∑
k ||Wk||2
l2 . We improve the interpretability of the
pooling weights as well as to facilitate a transfer among models by adding a projection onto a unit
cube. To reduce quantization artifacts of the pooling strategy as well as to ensure smoothness of
the output w.r.t. small translations of the image, the model penalizes weights whenever the pooling
surface is non-smooth. This can be done by measuring the spatial variation, that is ||∇xWk||2
l2 +
||∇yWk||2
l2 for every layer k. This regularization enforces soft transition between the pooling
subregions.
Every regularization term comes with its own hyper-parameter set by cross-validation. The overall
objective that we want to optimize is
minimize
W,Θ
JR(Θ,W) := (5)
− 1
D
D∑
i=1
C∑
j=1
1{y(i) = j}log p(y(i) = j|a(i); Θ)
+ α1
2 ||Θ||2
l2 + α2
2 ||W||2
l2
+ α3
2
(
||∇xW||2
l2 + ||∇yW||2
l2
)
subject to W ∈[0,1]K×M×L
where al is the l-th pooling unit described by Formula 2, and ||W||l2 is the Frobenius norm.
2.4 Approximation of the model
The presented approach is demanding to train in the means of the CPU time and memory storage
when using high dimensional representations. That is, the number of the pooling parameters to
learn grows as K×M×L, where Kis dimensionality of codes, M is the number of patches taken
from the image and Lis the number of pooling units. Therefore, we propose two approximations
to our method making the whole approach more scalable towards bigger dictionaries. However, we
emphasize that learnt pooling regions have very little if any overhead compared to standard spatial
division approaches at test time.
First approximation does a ﬁne-grained spatial partition of the image, and then pools the codes over
such subregions. This operation, we call it a pre-pooling step, reduces the number of considered
spatial locations by the factor of the pre-pooling size. For instance, if we collect M codes and the
pre-pooling size is Sper dimension, then we reduce the number of codes to a number M
S2 . The pre-
pooling operation ﬁts well into our generalization of the SPM architectures as by choosing S := M
2
we obtain a weighted quadrants scheme. Moreover, the modeler has the option to start with the
larger S when little data is available and gradually decreases S as more parameters can be learnt
using more data.
The second approximation divides a K dimensional code into K
D batches, each D dimensional
(where D≤Kand Kis divisible by D). Then we train our model on all such batches in parallel to
obtain the pooling weights. Later, we train the classiﬁer on top of the concatenation of the trained,
partial models. As opposed to Le et al. [2012b] our training is fully independent and doesn’t need
communication between different machines.
Since the ordering of the codes is arbitrary, we also considerDdimensional batches formed from the
permuted version of the original codes, and combine them together with the concatenated batches to
boost the classiﬁcation accuracy (we call this approximation redundant batches). Given a ﬁxed sized
dictionary, this approximation performs slightly better, although it comes at the cost of increased
number of features due to the redundant batches.
4
Finally, our approximations not only lead to a highly parallel training procedure with reduced mem-
ory requirements and computational demands, but also have shown to greatly reduce the number
of required iterations as they tend to converge roughly 5 times faster than the full model on large
dictionaries.
3 Experimental Results
We evaluate our method on the CIFAR-10 and CIFAR-100 datasets [Krizhevsky and Hinton, 2010].
Furthermore, we provide insights into the learnt pooling strategies as well as investigate transfer
between datasets. In this section we describe our experimental setup, and present our results on both
datasets.
3.1 CIFAR-10 and CIFAR-100 datasets
The CIFAR-10 and CIFAR-100 datasets contain 50000 training color images and 10000 test color
images from respectively 10 and 100 categories, with 6000 and 600 images per class respectively.
All images have the same size: 32 ×32 pixels, and were sampled from the 80 million tiny images
dataset [Torralba et al., 2008].
3.2 Evaluation pipeline
In this work, we follow the Coates and Ng [2011] pipeline. We extract normalized and whitened
6 ×6 patches from images using a dense, equispaced grid with a unit sample spacing. As the next
step, we employ the K-means assignment and triangle encoding [Coates and Ng, 2011, Coates et al.,
2011] to compute codes – a K-dimensional representation of the patch. We classify images using
either a logistic regression, or a linear SVM in the case of transferred pooling regions. Optionally we
use two approximations described in subsection 2.4. As we want to be comparable to Coates et al.
[2011], who use a spatial division into 2-by-2 subregions which results in 4 ·Kpooled features, we
use 4 pooling units. Furthermore, we use standard division (ﬁrst row of Figure 2) as an initialization
of our model.
To learn parameters of the model we use the limited-memory BFGS algorithm (details are described
in subsection 2.2), and limit the number iterations to 3000. After the training, we can also concate-
nate the results of the parameterized pooling operator[Θwl(U)]4
l=1. This yields a 4 ·Kdimensional
feature vector that can be again fed into the classiﬁer, and trained independently with the already
trained pooling regions. We call this procedure transfer of pooling regions.
The reason behind the transfer is threefold. Firstly, we can combine partial models trained with our
approximation in batches to a full, originally intractable, model 5. Secondly, the transfer process
allows to combine both the codes and the learnt model from the dictionaries of different sizes.
Lastly, it enables training of the pooling regions together with the classiﬁer on one dataset, and then
re-train the classiﬁer alone on a target dataset. To transfer the pooling regions, we tried logistic
regression classiﬁer and linear SVM showing that both classifying procedures can beneﬁt from the
learnt pooling regions. However, since we achieve slightly better results for the linear SVM (about
0.5% for bigger dictionaries), only those results are reported. Similarly, we don’t notice signiﬁcant
difference in the classiﬁcation accuracy for smaller dictionaries when the pre-pooling is used (with
the pre-pooling size S := 3), and therefore all experiments refer only to this case. Finally, we select
hyper-parameters of our model based on the 5-fold cross-validation.
3.3 Evaluation of our method on small dictionaries
Figure 1(a) shows the classiﬁcation accuracy of our full method against the baseline [Coates and Ng,
2011]. Since we train the pooling regions without any approximations in this set of experiments the
results are limited to dictionary sizes up to 800. Our method outperforms the approach of Coates by
10% for dictionary size 16 (our method achieves the accuracy 57.07%, whereas the baseline only
46.93%). This improvement is consistent up to the bigger dictionaries although the margin is getting
5The reader can ﬁnd details of such approximation in subsection 2.4.
5
0 50 100 150 200 250 300 350 40035
40
45
50
55
60
65
70
75
80
Dictionary size
Accuracy
 
 
Our
Coates
Random Pooling
Bag of Features
(a)
200 400 600 800 1000 1200 1400 160055
60
65
70
75
80
85
Dictionary size
Accuracy
 
 
Our (redundant batches)
Our (batches)
Our
Coates
Random Pooling
Bag of Features (b)
Figure 1: Figure 1(a) shows accuracy of the classiﬁcation with respect to the number of dictionary elements on
smaller dictionaries. Figure 1(b) shows the accuracy of the classiﬁcation for bigger dictionaries when batches,
and the redundant batches were used. Experiments are done on CIFAR-10.
smaller. Our method is about 2.5% and 1.88% better than the baseline for 400 and 800 dictionary
elements respectively.
3.4 Scaling up to sizable dictionaries
In subsection 2.4 we have discussed the possibility of dividing the codes into low dimensional
batches and learning the pooling regions on those. In the following experiments we use batches
with 40 coordinates extracted from the original code, as those ﬁt conveniently into the memory of a
single, standard machine (about 5 Gbytes for the main data) and can all be trained in parallel.
Besides a reduction in the memory requirements, the batches have shown multiple beneﬁts in prac-
tice due to smaller number of parameters. We need less computations per iterations as well as
observe faster convergence. Figure 1(b) shows the classiﬁcation performance for larger dictionar-
ies where we examined the full model [Our], the baseline [Coates], random pooling regions (de-
scribed in subsection 3.5), bag of features, and two possible approximation - the batched model
[Our (batches)], and the redundantly batched model [Our (redundant batches)].
Our test results are presented in Table 1. When comparing our full model to the approximated
versions with batches for dictionaries of size 200, 400 and 800, we observe that there is almost no
drop in performance and we even slightly improve for the bigger dictionaries. We attribute this to the
better conditioned learning problem of the smaller codes within one batch. With an accuracy for the
batched model of 79.6% we outperform the Coates baseline by 1.7%. Interestingly, we gain another
small improvement to 80.02% by adding redundant batches which amounts to a total improvement
of 2.12% compared to the baseline. Our method performs comparable to the pooling strategy of
Jia and Huang [2011] which uses more restrictive assumptions on the pooling regions and employs
feature selection algorithm.
Method Dict. size Features Acc.
Jia 1600 6400 80.17%
Coates 1600 6400 77.9%
Our (batches) 1600 6400 79.6%
Our (redundant) 1600 12800 80.02%
Table 1: Comparison of our methods against the baseline [Coates and Ng, 2011] and Jia and Huang [2011] with
respect to the dictionary size, number of features and the test accuracy on CIFAR-10.
To the best of our knowledge Ciresan et al. [2012] achieves the best results on the CIFAR-10 dataset
with an accuracy 88.79% with a method based on a deep architecture – different type of architecture
to the one that we investigate in our study. More recently Goodfellow et al. [2013] has achieved
accuracy 90.62% with new maxout model that takes an advantage of dropout.
6
regularization pooling weights
dataset: CIFAR-10 ; dictionary size: 200
Coates (no learn.)
l2
smooth
smooth & l2
dataset: CIFAR-10 ; dictionary size: 1600
smooth & batches
dataset: CIFAR-100 ; dictionary size: 1600
smooth & batches
Table 2: Visualization of different pooling strategies obtained for different regularizations, datasets and dic-
tionary size. Every column shows the regions from two different coordinates of the codes. First row presents
the initial conﬁguration also used in standard hand-crafted pooling methods. Brighter regions denote larger
weights.
3.5 Random pooling regions
Our investigation also includes results using random pooling regions where the weights for the
parameterized operator (Eq. 2) were sampled from normal distribution with mean 0.5 and standard
deviation 0.1, that is wl
j ∼N (0.5,0.1) for all l. This notion of the random pooling differs from
the Jia et al. [2012] where random selection of rectangles is used. The experiments show that the
random pooling regions can compete with the standard spatial pooling (Figure 1(a) and 1(b)) on the
CIFAR-10 dataset, and suggest that random projection can still preserve some spatial information.
This is especially visible in the regime of bigger dictionaries where the difference is only 1.09%.
The obtained results indicate that hand-crafted division of the image into subregions is questionable,
and call for a learning-based approach.
3.6 Investigation of the regularization terms
Our model (Eq. 5) comes with two regularization terms associated with the pooling weights, each
imposing different assumptions on the pooling regions. Hence, it is interesting to investigate their
role in the classiﬁcation task by considering all possible subsets of {l2,smooth}, where “l2” and
“smooth” refer to ||W||2
l2 and
(
||∇xW||2
l2 + ||∇yW||2
l2
)
respectively.
Table 3 shows our results on CIFAR-10. We choose a dictionary size of 200 for these experiments,
so that we can evaluate different regularization terms without any approximations. We conclude that
the spatial smoothness regularization term is crucial to achieve a good predictive performance of
our method whereas the l2-norm term can be left out, and thus also reducing the number of hyper-
parameters. Based on the cross-validation results (second column of Table 3), we select this setting
for further experiments.
Regularization CV Acc. Test Acc.
free 68.48% 69.59%
l2 67.86% 68.39%
smooth 73.36% 73.96%
l2 + smooth 70.42% 70.32%
Table 3: We investigate the impact of the regularization terms on the CIFAR-10 dataset with dictionary size
equals to 200. Term “free” denotes the objective function without the l2-norm and smoothness regularization
terms. The cross-validation accuracy and test accuracy are shown.
7
3.7 Experiments on the CIFAR-100 dataset
Although the main body of work is conducted on the CIFAR-10 dataset, we also investigate how
the model performs on the much more demanding CIFAR-100 dataset with 100 classes. Our model
with the spatial smoothness regularization term on the 40 dimensional batches achieves 56.29%
accuracy. To our best knowledge, this result consitutes the state-of-the-art performance on this
dataset, outperforming Jia and Huang [2011] by 1.41%, and the baseline by 4.63%. Using different
architecture Goodfellow et al. [2013] has achieved accuracy 61.43%.
Method Dict. size Features Acc.
Jia 1600 6400 54.88%
Coates 1600 6400 51.66%
Our (batches) 1600 6400 56.29%
Table 4: The classiﬁcation accuracy on CIFAR-100, where our method is compared against the Coates and
Ng [2011] (we downloaded the framework from https://sites.google.com/site/kmeanslearning, we also use 5-
fold cross-validation to choose hyper-parameter C) and Jia and Huang [2011] (here we refer to the NIPS 2011
workshop paper).
3.8 Transfer of the pooling regions between datasets
Beyond the standard classiﬁcation task, we also examine if the learnt pooling regions are trans-
ferrable between datasets. In this scenario the pooling regions are ﬁrst trained on the source dataset
and then used on the target dataset to train a new classiﬁer. We use dictionary of 1600 with 40-
dimensional batches. Our results (Table 5) suggest that the learnt pooling regions are indeed trans-
ferable between both datasets. While we observe a decrease in performance when learning the
pooling strategy on the less diverse CIFAR-10 dataset, we do see improvements for learning on the
richer CIFAR-100 dataset. We arrive at a test accuracy of 80.35% which is an additional improve-
ment of 0.75% and 0.18% over our best results (batch-based approximation) and Jia and Huang
[2011] respectively.
Source Target Accuracy
CIFAR-10 CIFAR-100 52.86%
CIFAR-100 CIFAR-10 80.35%
Table 5: We train the pooling regions on the ’Source’ dataset. Next, we use such regions to train the classiﬁer
on the ’Target’ dataset where the test accuracy is reported.
3.9 Visualization and analysis of pooling strategies
Table 2 visualizes different pooling strategies investigated in this paper. The ﬁrst row shows the
widely used rectangular spatial division of the image. The other visualizations correspond to pooling
weights discovered by our model using different regularization terms, datasets and dictionary size.
The second row shows the results on CIFAR-10 with the “l2” regularization term. The pooling is
most distinct from the other results, as it learns highly localized weights. This pooling strategy has
also performed the worst in our investigation (Table 3).
The ”smooth” pooling performs the best. Visualization shows that weights are localized but vary
smoothly over the image. The weights expose a bias towards initialization shown in the ﬁrst row.
All methods with the spatial smoothness regularization tend to focus on similar parts of the image,
however “l2 & smooth” is more conservative in spreading out the weights.
The last two rows show weights trained using our approximation by batches. From visual inspection,
they show a similar level of localization and smoothness to the regions obtained without approxima-
tion. This further supports the use of our approximation into independent batches.
8
4 Conclusion
In this paper we propose a ﬂexible parameterization of the pooling operator which can be trained
jointly with the classiﬁer. In this manner, we study the effect of different regularizers on the pooling
regions as well as the overall system. To be able to train the large set of parameters we propose
approximations to our model allowing efﬁcient and parallel training without loss of accuracy.
Our experiments show there is a room to improve the classiﬁcation accuracy by advancing the spatial
pooling stage. The presented method outperforms a popular hand-crafted pooling based method and
previous approaches to learn pooling strategies. While our improvements are consistent over the
whole range of dictionary sizes that we have investigated, the margin is most impressive for small
codes where we observe improvements up to 10% compared to the baseline of Coates. Finally, our
method achieves an accuracy of 56.29% on CIFAR-100, which is to the best of our knowledge the
new state-of-the-art on this dataset.
As we believe that our method is a good framework for further investigations of different pooling
strategies and in order to speed-up progress on the pooling stage we will make our code publicly
available at time of publication.
References
David H Hubel and Torsten N Wiesel. Receptive ﬁelds, binocular interaction and functional architecture in the
cat’s visual cortex.The Journal of physiology, 160(1):106, 1962.
K. Fukushima and S. Miyake. Neocognitron: A new algorithm for pattern recognition tolerant of deformations
and shifts in position. Pattern recognition, 15(6):455–469, 1982.
Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Handwritten
digit recognition with a back-propagation network. In NIPS, 1990.
M. Riesenhuber and T. Poggio. Hierarchical models of object recognition in cortex. Nature Neuroscience,
2009.
J. J. Koenderink and A. J. Van Doorn. The structure of locally orderless images. International Journal of
Computer Vision, 31(2):159–168, 1999.
D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60(2):91–110, 2004.
N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005.
S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing
natural scene categories. In CVPR, 2006.
J. Yang, K. Yu, Y . Gong, and T. Huang. Linear spatial pyramid matching using sparse coding for image
classiﬁcation. In CVPR, 2009.
M. A. Ranzato, F. J. Huang, Y . Boureau, and Y . LeCun. Unsupervised learning of invariant feature hierarchies
with applications to object recognition. In CVPR, 2007.
Q. V . Le, M. A. Ranzato, R. Monga, M. Devin, K. Chen, G. S. Corrado, J. Dean, and A. Y . Ng. Building
high-level features using large scale unsupervised learning. In ICML, 2012a.
Y . Jia and C. Huang. Beyond spatial pyramids: Receptive ﬁeld learning for pooled image features. In NIPS
Workshop on Deep Learning, 2011.
Y . Jia, C. Huang, and T. Darrell. Beyond spatial pyramids: Receptive ﬁeld learning for pooled image features.
In CVPR, 2012.
J. Feng, B. Ni, Q. Tian, and S. Yan. Geometric lp-norm feature pooling for image classiﬁcation. In CVPR,
2011.
M. A. Ranzato and G. E. Hinton. Modeling pixel means and covariances using factorized third-order boltzmann
machines. In CVPR, 2010.
C. M. Bishop. Neural Network for Pattern Recognition. Oxford University Press, 1999.
Y . LeCun, L. Bottou, G. Orr, and K. M¨uller. Efﬁcient backprop. Neural networks: Tricks of the trade, pages
546–546, 1998.
R. Collobert and S. Bengio. Links between perceptrons, mlps and svms. In ICML, 2004.
A. Coates, H. Lee, and A. Y . Ng. An analysis of single-layer networks in unsupervised feature learning. In
AISTATS, 2011.
Q. V . Le, R. Monga, M. Devin, G. Corrado, K. Chen, M. A. Ranzato, J. Dean, and A. Y . Ng. Building high-level
features using large scale unsupervised learning. 2012b.
9
A. Krizhevsky and G. Hinton. Convolutional deep belief networks on cifar-10. Technical report, 2010.
A. Torralba, R. Fergus, and W. T. Freeman. 80 million tiny images: A large data set for nonparametric object
and scene recognition. PAMI, 2008.
A. Coates and A. Y . Ng. The importance of encoding versus training with sparse coding and vector quantization.
In ICML, 2011.
D. Ciresan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classiﬁcation. In
CVPR, 2012.
I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y . Bengio. Maxout networks. InICML, 2013.
10


Peer review:
[Review]:
We thank all the reviewers for their comments.
We will include suggested papers on related work and origins of pooling architectures as well as improvement on the state of the art that occurred in the meanwhile.
The reviewers acknowledge our analysis of regularization schemes to learn weighted pooling units together with a regularizer that promotes spatial smoothness.

Our work aims at replacing hand-crafted pooling stage in computer vision architectures ([1], [2], [3] and [4]) where the pooling is a way to reduce dimensionality of the features while preserving spatial information. Handcrafted spatial pooling schemes that operate on an image-level are still part of many state of the art architectures. In particular, recent approaches that aim at higher-level semantic representations (e.g. [3], [6]) follow this paradigm and are within the scope of our method. We therefore believe that our method will find wide applicability in those scenarios.

Anonymous 45d8:
We don't agree that CIFAR-100 is less-competitive as the state-of-the-art results are lower than CIFAR-10, moreover CIFAR-100 contains fewer examples per class for training and 10x more classes.
We are not restricted to sum pooling as back-propagation over the max operator is possible. 
We use non-negativity constraint for the weights as Formula 5 shows.
Sparsity constraint on the weights has no computational benefits at test time as the weighted sum ranges over the whole image. 
Concerning the remarks about increased computation time, we would like to point out that computational costs are dominated by the coding procedure. The pooling stage - hand-crafted or learnt - is on the order of milliseconds per image.
The connection between the matrix factorization of the weights of the softmax classifier and pooling stage is an interesting additional observation, however, the paper analyzes the regularization terms of the pooling operator and therefore regularization of the factorized weight matrix.
In our work we want to make our architecture consistent with other computer vision architectures that use image-level pooling stage ([1], [2], [3], [4] and [5]) exploiting the shared representation among classes and computational benefits of this method.

Anonymous 2426:
The method produces state-of-the-art results, at the time of submission, on CIFAR-100 and state-of-the-art on both CIFAR-10 and CIFAR-100 given SPM architecture ([1], [2], [4]).
As our results show, the smoothness constrain/regularization is the most crucial (Table 3), non-negativity constraint though increases the interpretability of the results. We use lbfgs with projection onto a unit box after every weights update. 
Although some of our speed-ups to make the system more scalable are heuristic, they are appreciated e.g. by 'Anonymous c1a0' and share similarities with recently proposed approaches for scalable learning as we reference in the paper.

Anonymous c1a0:
Increasing number of classification parameters in the SPM architecture ([1], [2], [4]) requires a bigger codebooks which increases the complexity of encoding step as every image patch has to be assigned to a cluster via triangle coding [4]. This would lead to a significant increase at test time. On the other hand, our architecture adds little overhead compared to SPM architectures at the test time.

Anonymous 45d8 & Anonymous 2426:
The pre-pooling step is pooling over a small neighborhood (over a 3x3 neighboring pixels), and therefore can be seen as form of weight sharing. This is a technical detail in order to reduce memory consumption and training time. This doesn't defy the main argument given in the paper as pooling is learnt over larger areas.