Gradient Driven Learning for Pooling in Visual Pipeline Feature Extraction Models
Authors: Derek Rose, Itamar Arel
ICLR 2013

Abstract: Hyper-parameter selection remains a daunting task when building a pattern recognition architecture which performs well, particularly in recently constructed visual pipeline models for feature extraction. We re-formulate pooling in an existing pipeline as a function of adjustable pooling map weight parameters and propose the use of supervised error signals from gradient descent to tune the established maps within the model. This technique allows us to learn what would otherwise be a design choice within the model and specialize the maps to aggregate areas of invariance for the task presented. Preliminary results show moderate potential gains in classification accuracy and highlight areas of importance within the intermediate feature representation space.

Gradient Driven Learning for Pooling in Visual
Pipeline Feature Extraction Models
Derek Rose and Itamar Arel
Deptartment of Electrical Engineering and Computer Science
University of Tennessee
derek@utk.edu, itamar@ieee.org
Abstract
Hyper-parameter selection remains a daunting task when building a pattern recog-
nition architecture which performs well, particularly in recently constructed visual
pipeline models for feature extraction. We re-formulate pooling in an existing
pipeline as a function of adjustable pooling map weight parameters and propose
the use of supervised error signals from gradient descent to tune the established
maps within the model. This technique allows us to learn what would otherwise
be a design choice within the model and specialize the maps to aggregate areas
of invariance for the task presented. Preliminary results show moderate poten-
tial gains in classiﬁcation accuracy and highlight areas of importance within the
intermediate feature representation space.
1 Introduction
Multi-stage visual pipelines for learning feature representations of images have recently proven
valuable for classifying small objects from a variety of lighting conditions, scales, and poses. An
effective variant of these was explored in experiments by [1] that compared features learned and
encoded by stages that originate in encoder-decoder networks, deep learning [2], and bags of fea-
tures models [3]. This architecture partitions images into patches to perform local learning of an
over-complete codebook and uses this codebook to form global representations of the images for
classiﬁcation. Intermediate or mid-level representations are of high-dimensionality and retain the
spatial structure of the image. Pooling is an integral late stage that performs the same role as the
sub-sampling layer of a convolutional neural network (CNN): reduction in the ﬁnal number of fea-
tures (passed to the classiﬁer or next layer) by an aggregation operation meant to improve invariance
to small changes [4].
Unfortunately, each additional stage of the architecture often adds hyper-parameters for model selec-
tion that must be explored. For the pooling layer the number of pools, their structure or spatial lay-
out, weights within this region, and operator (often max, average, or p-norm) are hyper-parameters
that are frequently chosen by rules of thumb. Recently, [5] explored pool selection by optimization
over a full training set with an over-complete number of pools and achieved excellent improvements
over standard pyramid models, although their method uses a feature selection stage with retraining
to tractably create a classiﬁer. In this paper we present a method for learning pooling maps with
weight parameters that may optimize or tune the feature space representation for better discrimina-
tion through stochastic gradient descent. This converts two of the model choices above to parameters
which may be learned from a limited stream of labeled training data.
Back-propogation through the architecture in [1] is used to obtain the appropriate weight updates.
This technique stems back at least to the inception of convolutional neural networks and graph
transformer networks [4], where each module or layer of the network may be utilized in a forward
pass output calculation and backward pass parameter update.
1
arXiv:1301.3755v1  [cs.CV]  16 Jan 2013
2 Architecture Description and Design
The multi-stage image recognition architecture considered from [1] consists of patch extraction, nor-
malization and whitening, codebook learning, feature encoding, spatial pooling, and classiﬁcation.
As guided by [1] we use hyper-parameters such as the number of codewords, k = 400, patch size
w = 6 pixels, stride= 1 for dense extraction, triangle encoding, and use of patch whitening and
pre-processing with a focus on the CIFAR-10 dataset. Quadrant based average pooling was used to
sub-sample the intermediate feature space representation down to 4k. This choice stemmed from
spatial pyramids and was not a heavy focus for [1], although motivating work in [5] has shown there
are both a large number of options for pooling regions and much performance to be gained.
Although building pools for invariance can be partially based on intuition when patch extraction is
performed spatially relevant to the original image (as a single layer in the architecture considered),
pooling region choice for layers of features that build upon those generated in the lowest layer is not
as straightforward (as noted by [6]). Special care must be taken to reorder features and assemble
pools that maintain the structure inherent in the image. It would be beneﬁcial if the region could
change relative to problem demands as well as layer context, and we propose the use of pooling
weight maps that may be adjusted as learning proceeds with gradient descent.
3 Stochastic Gradient Descent Based Method for Weighted Pooling
To obtain continuous updates as well as a gradient signal we replace the support vector machine
originally used for classiﬁcation in [1] with a feed-forward neural network with single hidden layer
and mean-square error cost function. The outputs are one-hot binary vectors for the t = 10 target
classes and the inputs are pooled features h. If we have a n× npixel square image and let P =
(n(w− 1) + 1), with dense patch extraction we obtain a P × P × kmid-level representation (this
may be modiﬁed for non-square images). Pooling reduces this representation to p× k that can be
ﬂattened to form h. We then extend the network back to include an additional input layer that holds
our pweight maps with size P × P that we denote as Wi. The inputs to this new layer are encoded
features gcomputed from the P × P × kpatches.
This method shares many similarities with the sub-sampling layer in a CNN. In the sub-sampling
layer each neuron receives the average of the features from the prior layer in its receptive ﬁeld or
pool (receptive ﬁelds of units do not overlap). Each unit has a coefﬁcient and bias that are trained
with gradient information and feed into a sigmoid activation function that controls the response of
the sub-sampling unit [4]. Aside from its application in an alternate context, our approach differs in
that the features which are averaged are no longer restrained to being equally weighted and multiple
pools may utilize the same encoded patches. Learned weights combine encoded features g from
the mid-level representation and pass these through a linear activation with unity weight to a neural
network classiﬁer, i.e. pooled inputs h over encoded patches g in pool iare computed1 by
hi = fpool(g) =
P∑
m=1
P∑
n=1
Wi
m,n ∗ gm,n. (1)
We currently have not explored using non-linear activation units with adjustable biases and coef-
ﬁcients and although we have no restriction that pools must be non-overlapping as in CNNs, we
are currently examining modiﬁcation to the loss function to maximize diversity among the pools.
The weight sharing scheme employed by [4] for learning feature maps is similarly used here to
reduce the parameter search space by sharing weights in codeword dimension k of the mid-level
representation. Figure 1(a) illustrates the positioning of the learned parameters within the architec-
ture. To replicate the quadrant based pooling we employ four maps that connect to every patch and
are initially set to Wm,n = 1/(P/2)2 inside the hot zone of the corresponding quadrant map and
zero elsewhere. After learning the codebook we train the original network with 128 hidden neurons
and stochastic average mini-batch gradient descent (random batches of 10 images) while the pool
weights are ﬁxed. Network inputs are normalized to ¯hpost pooling with a mean and variance that
is not modiﬁed. After a learning period we ﬁx the classiﬁer weights and learn the pooling weights.
We have found this alternate training to perform better than simultaneous learning where the neural
network can compensate for pooling weight changes.
1Codeword index has been omitted for clarity. h and g have a k-size dimension which Wis shared over.
2
P=(n-w)/stride + 1 
(fully connected) 
1 
2 
k 
… 
* starting maps 
* 
FFNN 
h1 
h2 
h3 
h4 
target 
(one hot) 
g11 g12 
g21 
gPP 
… 
… 
… 
(k-dim) 
Mid-Level 
Representation 
W1 (shared) 
(a)
1 2
3 4 (b)
Figure 1: (a) Pool weight map connections and feed-forward net extension, (b) Learned weight maps
3.1 Weight Update Rule and Preliminary Results
To update each pool i’s weights with loss function J note we want ∆Wi
m,n = −η∂J/∂Wi
m,n =
−η∂J
∂¯hi
∂¯hi
∂Wim,n
. From the original backpropagation pass we may create a new set of sensitivities
δ0 = −∂J/∂¯h = v1T
δ1 (v representing the original input to hidden weights) for the prepended
network layer. Noting ¯hi = (hi−µi)/σi and ∂¯hi/∂Wi
m,n = gm,n/σi we obtain the update rule
∆Wi
m,n = ηδ0
i
gm,n
σi
. (2)
The pool weight gradient updates for the images in each mini-batch are averaged; in each, every
codeword dimension k contributes to the update as a sum. In preliminary tests we freeze network
learning after presenting 250k examples from 80% of the CIFAR-10 training set. The last validation
accuracy before pool learning on the remaining 20% averaged over ﬁve trials was 67.56%. After
training with an additional 15k training images and checking the validation set every 500 images,
the average best post pool learning accuracy was 68.03% for an improvement of 0.57%. Figure
1(b) contains an example of the weights learned that highlights the inﬁnite number of possible pools
gradient descent searches over.
This method may also be used to tune the choice of p within the weighted p-norm. Open issues
remain, particularly in the learning rate choice (for which we have traded the pooling structure for
hyper-parameter η= 5e−5) and the number of maps needed to cover separate areas of invariance. It
would be preferred to select more than enough maps than necessary and later trim down redundant
features, although we need to be careful to avoid overﬁtting the training set here (and in general
for this approach). Unfortunately, each additional map adds kinputs to the classiﬁer. This problem
relates closely to feature map count or hidden neuron count hyper-parameters in CNNs.
References
[1] A. Coates, H. Lee, and A. Y . Ng, “An analysis of single-layer networks in unsupervised feature learning,”
in AISTATS 14, 2011.
[2] Y . Bengio, “Learning deep architectures for AI,” Foundations and Trends in Machine Learning, vol. 2,
no. 1, pp. 1–127, 2009.
[3] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial pyramid matching for recogniz-
ing natural scene categories,” in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society
Conference on, vol. 2, 2006, pp. 2169–2178.
[4] Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recogni-
tion,”Proceedings of the IEEE, vol. 86, no. 11, p. 22782324, 1998.
[5] Y . Jia, H. Chang, and T. Darrel, “Beyond spatial pyramids: Receptive ﬁeld learning for pooled image
features,” inComputer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, 2012.
[6] A. Coates and A. Y . Ng, “Learning feature representations with k-means,” in Neural Networks: Tricks of
the Trade, Reloaded. Springer LNCS, 2012.
3


Peer review:
[Review]:
The paper by Rose & Arel entitled 'Gradient Driven Learning for Pooling in Visual  Pipeline Feature Extraction Models' describes a new approach for optimizing hyper parameters in spatial pyramid-like architectures.

Specifically, an architecture is presented which corresponds to a spatial pyramid where a  two-layer neural net replaces the SVM in the final classification stage. The key contribution is to formulate the pooling operation in the spatial pyramid as a weighted sum over inputs which enables learning of the pooling receptive fields via back-propagation.

Pros: The paper addresses an important problem in the field of computer vision. Spatial pyramids are currently quite popular in the computer vision community and optimizing the many free parameters which are normally tuned by hand is a key problem.

Cons: The contribution of the present work remains very limited both in terms of the actual problem formulation and the empirical evaluation (no comparison to alternative approaches such as the recent work by Jia et al [ref 5] is shown). The overall 0.5% improvement in accuracy over non-optimized hyper-parameters is quite disappointing. In future work, the authors should compare their approach with alternative approaches in addition to suggesting significant improvement over non-optimized/standard parameters.

Additional comments: Additional references that could be added, discussed and/or used for benchmark. Y Boureau and J Ponce. A theoretical analysis of feature pooling in visual recognition. In ICML, 2010 and Pinto N, Doukhan D, DiCarlo JJ, Cox DD (2009). A High-Throughput Screening Approach to Discovering Good Forms of Biologically-Inspired Visual Representation. PLoS Computational Biology 5(11): e1000579. doi:10.1371/journal.pcbi.1000579.