Summary:
We propose a method for human pose estimation based on Deep Neural Networks
(DNNs). The pose estimation is formulated as a DNN-based regression problem
towards body joints. We present a cascade of such DNN regressors which results
in high precision pose estimates. The approach has the advantage of reasoning
about pose in a holistic fashion and has a simple but yet powerful formulation
which capitalizes on recent advances in Deep Learning. We present a detailed
empirical analysis with state-of-art or better performance on four academic
benchmarks of diverse real-world images.

IEEE Conference on Computer Vision and Pattern Recognition, 2014

DeepPose: Human Pose Estimation via Deep Neural Networks
Alexander Toshev Christian Szegedy
Google
1600 Amphitheatre Pkwy
Mountain View, CA 94043
toshev,szegedy@google.com
Figure 1. Besides extreme variability in articulations, many of the
joints are barely visible. We can guess the location of the right
arm in the left image only because we see the rest of the pose and
anticipate the motion or activity of the person. Similarly, the left
body half of the person on the right is not visible at all. These
are examples of the need for holistic reasoning . We believe that
DNNs can naturally provide such type of reasoning.
Abstract
We propose a method for human pose estimation based
on Deep Neural Networks (DNNs). The pose estimation
is formulated as a DNN-based regression problem towards
body joints. We present a cascade of such DNN regres-
sors which results in high precision pose estimates. The
approach has the advantage of reasoning about pose in a
holistic fashion and has a simple but yet powerful formula-
tion which capitalizes on recent advances in Deep Learn-
ing. We present a detailed empirical analysis with state-of-
art or better performance on four academic benchmarks of
diverse real-world images.
1. Introduction
The problem of human pose estimation, deﬁned as the
problem of localization of human joints, has enjoyed sub-
stantial attention in the computer vision community. In
Fig. 1, one can see some of the challenges of this prob-
lem – strong articulations, small and barely visible joints,
occlusions and the need to capture the context.
The main stream of work in this ﬁeld has been motivatedmainly by the ﬁrst challenge, the need to search in the large
space of all possible articulated poses. Part-based models
lend themselves naturally to model articulations ([16, 8])
and in the recent years a variety of models with efﬁcient
inference have been proposed ([6, 19]).
The above efﬁciency, however, is achieved at the cost of
limited expressiveness – the use of local detectors, which
reason in many cases about a single part, and most impor-
tantly by modeling only a small subset of all interactions
between body parts. These limitations, as exempliﬁed in
Fig. 1, have been recognized and methods reasoning about
pose in a holistic manner have been proposed [15, 21] but
with limited success in real-world problems.
In this work we ascribe to this holistic view of human
pose estimation. We capitalize on recent developments of
deep learning and propose a novel algorithm based on a
Deep Neural Network (DNN). DNNs have shown outstand-
ing performance on visual classiﬁcation tasks [14] and more
recently on object localization [23, 9]. However, the ques-
tion of applying DNNs for precise localization of articulated
objects has largely remained unanswered. In this paper we
attempt to cast a light on this question and present a simple
and yet powerful formulation of holistic human pose esti-
mation as a DNN.
We formulate the pose estimation as a joint regression
problem and show how to successfully cast it in DNN set-
tings. The location of each body joint is regressed to using
as an input the full image and a 7-layered generic convolu-
tional DNN. There are two advantages of this formulation.
First, the DNN is capable of capturing the full context of
each body joint – each joint regressor uses the full image
as a signal. Second, the approach is substantially simpler
to formulate than methods based on graphical models – no
need to explicitly design feature representations and detec-
tors for parts; no need to explicitly design a model topology
and interactions between joints. Instead, we show that a
generic convolutional DNN can be learned for this problem.
Further, we propose a cascade of DNN-based pose pre-
dictors. Such a cascade allows for increased precision of
1arXiv:1312.4659v3  [cs.CV]  20 Aug 2014
joint localization. Starting with an initial pose estimation,
based on the full image, we learn DNN-based regressors
which reﬁnes the joint predictions by using higher resolu-
tion sub-images.
We show state-of-art results or better than state-of-art on
four widely used benchmarks against all reported results.
We show that our approach performs well on images of peo-
ple which exhibit strong variation in appearance as well as
articulations. Finally, we show generalization performance
by cross-dataset evaluation.
2. Related Work
The idea of representing articulated objects in general,
and human pose in particular, as a graph of parts has been
advocated from the early days of computer vision [16]. The
so called Pictorial Strictures (PSs), introduced by Fishler
and Elschlager [8], were made tractable and practical by
Felzenszwalb and Huttenlocher [6] using the distance trans-
form trick. As a result, a wide variety of PS-based models
with practical signiﬁcance were subsequently developed.
The above tractability, however, comes with the limita-
tion of having a tree-based pose models with simple binary
potential not depending on image data. As a result, research
has focused on enriching the representational power of the
models while maintaining tractability. Earlier attempts to
achieve this were based on richer part detectors [19, 1, 4].
More recently, a wide variety of models expressing complex
joint relationships were proposed. Yang and Ramanan [27]
use a mixture model of parts. Mixture models on the full
model scale, by having mixture of PSs, have been studied
by Johnson and Everingham [13]. Richer higher-order spa-
tial relationships were captured in a hierarchical model by
Tian et al. [25]. A different approach to capture higher-
order relationship is through image-dependent PS models,
which can be estimated via a global classiﬁer [26, 20, 18].
Approaches which ascribe to our philosophy of reason-
ing about pose in a holistic manner have shown limited
practicality. Mori and Malik [15] try to ﬁnd for each test
image the closest exemplar from a set of labeled images
and transfer the joint locations. A similar nearest neighbor
setup is employed by Shakhnarovich et al. [21], who how-
ever use locality sensitive hashing. More recently, Gkioxari
et al. [10] propose a semi-global classiﬁer for part conﬁg-
uration. This formulation has shown very good results on
real-world data, however, it is based on linear classiﬁers
with less expressive representation than ours and is tested
on arms only. Finally, the idea of pose regression has been
employed by Ionescu et al. [11], however they reason about
3D pose.
The closest work to ours uses convolution NNs together
with Neighborhood Component Analysis to regress toward
a point in an embedding representing pose [24]. However,
this work does not employ a cascade of networks. Cascadesof DNN regressors have been used for localization, however
of facial points [22]. On the related problem of face pose
estimation, Osadchy et al. [17] employ a NN-based pose
embedding trained with a contrastive loss.
3. Deep Learning Model for Pose Estimation
We use the following notation. To express a pose, we en-
code the locations of all kbody joints in pose vector deﬁned
asy= (:::;yT
i;:::)T;i2f1;:::;kg, where yicontains
thexandycoordinates of the ithjoint. A labeled image is
denoted by (x;y)wherexstands for the image data and y
is the ground truth pose vector.
Further, since the joint coordinates are in absolute image
coordinates, it proves beneﬁcial to normalize them w. r. t. a
boxbbounding the human body or parts of it. In a trivial
case, the box can denote the full image. Such a box is de-
ﬁned by its center bc2R2as well as width bwand height
bh:b= (bc;bw;bh). Then the joint yican be translated by
the box center and scaled by the box size which we refer to
as normalization by b:
N(yi;b) =1=bw 0
0 1=bh
(yibc) (1)
Further, we can apply the same normalization to the ele-
ments of pose vector N(y;b) = (:::;N (yi;b)T;:::)Tre-
sulting in a normalized pose vector . Finally, with a slight
abuse of notation, we use N(x;b)to denote a crop of the
imagexby the bounding box b, which de facto normalizes
the image by the box. For brevity we denote by N()nor-
malization with bbeing the full image box.
3.1. Pose Estimation as DNN-based Regression
In this work, we treat the problem of pose estimation as
regression, where the we train and use a function  (x;)2
R2kwhich for an image xregresses to a normalized pose
vector, where denotes the parameters of the model. Thus,
using the normalization transformation from Eq. (1) the
pose prediction yin absolute image coordinates reads
y=N1( (N(x);)) (2)
Despite its simple formulation, the power and complex-
ity of the method is in  , which is based on a convolutional
Deep Neural Network (DNN). Such a convolutional net-
work consists of several layers – each being a linear trans-
formation followed by a non-linear one. The ﬁrst layer takes
as input an image of predeﬁned size and has a size equal to
the number of pixels times three color channels. The last
layer outputs the target values of the regression, in our case
2kjoint coordinates.
We base the architecture of the  on the work by
Krizhevsky et al. [14] for image classiﬁcation since it has
shown outstanding results on object localization as well
(xi, yi)
(x(s-1)i, y (s-1) i)xsi - x(s-1)iysi - y(s-1)iInitial stageStage s
send reﬁned values to next stage220 x 220
DNN-based regressor
27 x 27 x 128
13 x 13 x 192
13 x 13 x192
13 x 13 x192
4096
4096
55 x 55 x 48xiyi...DNN-based reﬁner
27 x 27 x 128
13 x 13 x 192
13 x 13 x192
13 x 13 x192
4096
4096
55 x 55 x 48Figure 2. Left: schematic view of the DNN-based pose regression. We visualize the network layers with their corresponding dimensions,
where convolutional layers are in blue, while fully connected ones are in green. We do not show the parameter free layers. Right: at stage
s, a reﬁning regressor is applied on a sub image to reﬁne a prediction from the previous stage.
[23]. In a nutshell, the network consists of 7layers (see
Fig. 2 left). Denote by Ca convolutional layer, by LRN
a local response normalization layer, Pa pooling layer
and byFa fully connected layer. Only CandFlayers
contain learnable parameters, while the rest are parame-
ter free. Both CandFlayers consist of a linear trans-
formation followed by a nonlinear one, which in our case
is a rectiﬁed linear unit. For Clayers, the size is de-
ﬁned as widthheightdepth, where the ﬁrst two di-
mensions have a spatial meaning while the depth deﬁnes
the number of ﬁlters. If we write the size of each layer in
parentheses, then the network can be described concisely
asC(555596)LRNPC(2727256)
LRNPC(1313384)C(1313384)
C(1313256)PF(4096)F(4096) . The ﬁlter
size for the ﬁrst two Clayers is 1111and55and for
the remaining three is 33. Pooling is applied after three
layers and contributes to increased performance despite the
reduction of resolution. The input to the net is an image
of220220which via stride of 4is fed into the network.
The total number of parameters in the above model is about
40M. For further details, we refer the reader to [14].
The use of a generic DNN architecture is motivated by
its outstanding results on both classiﬁcation and localization
problems. In the experimental section we show that such a
generic architecture can be used to learn a model resulting
in state-of-art or better performance on pose estimation as
well. Further, such a model is a truly holistic one — the
ﬁnal joint location estimate is based on a complex nonlinear
transformation of the full image.
Additionally, the use of a DNN obviates the need to de-
sign a domain speciﬁc pose model. Instead such a model
and the features are learned from the data. Although the re-
gression loss does not model explicit interactions between
joints, such are implicitly captured by all of the 7hidden
layers – all the internal features are shared by all joint re-
gressors.
Training The difference to [14] is the loss. Instead of a
classiﬁcation loss, we train a linear regression on top of thelast network layer to predict a pose vector by minimizing
L2distance between the prediction and the true pose vec-
tor. Since the ground truth pose vector is deﬁned in abso-
lute image coordinates and poses vary in size from image to
image, we normalize our training set Dusing the normal-
ization from Eq. (1):
DN=f(N(x);N(y))j(x;y)2Dg (3)
Then theL2loss for obtaining optimal network parameters
reads:
arg min
X
(x;y)2DNkX
i=1jjyi i(x;)jj2
2 (4)
For clarity we write out the optimization over individual
joints. It should be noted, that the above objective can
be used even if for some images not all joints are labeled.
In this case, the corresponding terms in the sum would be
omitted.
The above parameters are optimized for using Back-
propagation in a distributed online implementation. For
each mini-batch of size 128, adaptive gradient updates are
computed [3]. The learning rate, as the most important pa-
rameter, is set to 0:0005 . Since the model has large number
of parameters and the used datasets are of relatively small
size, we augment the data using large number of randomly
translated image crops (see Sec. 3.2), left/right ﬂips as well
as DropOut regularization for the Flayers set to 0:6.
3.2. Cascade of Pose Regressors
The pose formulation from the previous section has the
advantage that the joint estimation is based on the full im-
age and thus relies on context. However, due to its ﬁxed
input size of 220220, the network has limited capacity
to look at detail – it learns ﬁlters capturing pose properties
at coarse scale. These are necessary to estimate rough pose
but insufﬁcient to always precisely localize the body joints.
Note that we cannot easily increase the input size since this
will increase the already large number of parameters. In or-
der to achieve better precision, we propose to train a cascade
of pose regressors. At the ﬁrst stage, the cascade starts off
by estimating an initial pose as outlined in the previous sec-
tion. At subsequent stages, additional DNN regressors are
trained to predict a displacement of the joint locations from
previous stage to the true location. Thus, each subsequent
stage can be thought of as a reﬁnement of the currently pre-
dicted pose, as shown in Fig. 2.
Further, each subsequent stage uses the predicted joint
locations to focus on the relevant parts of the image – sub-
images are cropped around the predicted joint location from
previous stage and the pose displacement regressor for this
joint is applied on this sub-image. In this way, subsequent
pose regressors see higher resolution images and thus learn
features for ﬁner scales which ultimately leads to higher
precision.
We use the same network architecture for all stages of
the cascade but learn different network parameters. For
stages2 f1;:::;Sgof totalScascade stages, we de-
note bysthe learned network parameters. Thus, the
pose displacement regressor reads  (x;s). To reﬁne a
given joint location yiwe will consider a joint bounding
boxbicapturing the sub-image around yi:bi(y;) =
(yi;diam(y);diam(y))having as center the i-th joint
and as dimension the pose diameter scaled by . The diam-
eter diam (y)of the pose is deﬁned as the distance between
opposing joints on the human torso, such as left shoulder
and right hip, and depends on the concrete pose deﬁnition
and dataset.
Using the above notation, at the stage s= 1we start with
a bounding box b0which either encloses the full image or
is obtained by a person detector. We obtain an initial pose:
Stage 1 : y1 N1( (N(x;b0);1);b0)(5)
At each subsequent stage s2, for all joints i2f1;:::;kg
we regress ﬁrst towards a reﬁnement displacement ys
i
y(s1)
i by applying a regressor on the sub image deﬁned
byb(s1)
i from previous stage (s1). Then, we estimate
new joint boxes bs
i:
Stages:ys
i y(s1)
i +N1( i(N(x;b);s);b)(6)
forb=b(s1)
i
bs
i (ys
i;diam(ys);diam(ys)) (7)
We apply the cascade for a ﬁxed number of stages S,
which is determined as explained in Sec. 4.1.
Training The network parameters 1are trained as
outlined in Sec. 3.1, Eq. (4). At subsequent stages
s2, the training is done identically with one im-
portant difference. Each joint ifrom a training exam-
ple(x;y)is normalized using a different bounding box(y(s1)
i;diam(y(s1));diam(y(s1)))– the one cen-
tered at the prediction for the same joint obtained from pre-
vious stage – so that we condition the training of the stage
based on the model from previous stage.
Since deep learning methods have large capacity, we
augment the training data by using multiple normalizations
for each image and joint. Instead of using the prediction
from previous stage only, we generate simulated predic-
tions. This is done by randomly displacing the ground truth
location for joint iby a vector sampled at random from a
2-dimensional Normal distribution N(s1)
i with mean and
variance equal to the mean and variance of the observed dis-
placements (y(s1)
iyi)across all examples in the train-
ing data. The full augmented training data can be deﬁned
by ﬁrst sampling an example and a joint from the original
data at uniform and then generating a simulated prediction
based on a sampled displacement fromN(s1)
i :
Ds
A=f(N(x;b);N(yi;b))j
(x;yi)D;N(s1)
i;
b= (yi+;diam(y))g
The training objective for cascade stage sis done as in
Eq. (4) by taking extra care to use the correct normalization
for each joint:
s= arg min
X
(x;yi)2Ds
Ajjyi i(x;)jj2
2 (8)
4. Empirical Evaluation
4.1. Setup
Datasets There is a wide variety of benchmarks for hu-
man pose estimation. In this work we use datasets, which
have large number of training examples sufﬁcient to train a
large model such as the proposed DNN, as well as are real-
istic and challenging.
The ﬁrst dataset we use is Frames Labeled In Cinema
(FLIC), introduced by [20], which consists of 4000 train-
ing and 1000 test images obtained from popular Hollywood
movies. The images contain people in diverse poses and es-
pecially diverse clothing. For each labeled human, 10upper
body joints are labeled.
The second dataset we use is Leeds Sports Dataset [12]
and its extension [13], which we will jointly denote by LSP.
Combined they contain 11000 training and 1000 testing im-
ages. These are images from sports activities and as such
are quite challenging in terms of appearance and especially
articulations. In addition, the majority of people have 150
pixel height which makes the pose estimation even more
challenging. In this dataset, for each person the full body is
labeled with total 14joints.
For all of the above datasets, we deﬁne the diameter of a
poseyto be the distance between a shoulder and hip from
opposing sides and denote it by diam (y). It should be noted,
that the joints in all datasets are arranged in a tree kinemat-
ically mimicking the human body. This allows for a deﬁni-
tion of a limb being a pair of neighboring joints in the pose
tree.
Metrics In order to be able to compare with published re-
sults we will use two widely accepted evaluation metrics.
Percentage of Correct Parts (PCP) measures detection rate
of limbs, where a limb is considered detected if the distance
between the two predicted joint locations and the true limb
joint locations is at most half of the limb length [5]. PCP
was the initially preferred metric for evaluation, however it
has the drawback of penalizing shorter limbs, such as lower
arms, which are usually harder to detect.
To address this drawback, recently detection rates of
joints are being reported using a different detection crite-
rion – a joint is considered detected if the distance between
the predicted and the true joint is within a certain fraction of
the torso diameter. By varying this fraction, detection rates
are obtained for varying degrees of localization precision.
This metric alleviates the drawback of PCP since the de-
tection criteria for all joints are based on the same distance
threshold. We refer to this metric as Percent of Detected
Joints (PDJ).
Experimental Details For all the experiments we use the
same network architecture. Inspired by [7], we use a body
detector on FLIC to obtain initially a rough estimate of the
human body bounding box. It is based on a face detector –
the detected face rectangle is enlarged by a ﬁxed scaler. This
scaler is determined on the training data such that it contains
all labeled joints. This face-based body detector results in
a rough estimate, which however presents a good starting
point for our approach. For LSP we use the full image as
initial bounding box since the humans are relatively tightly
cropped by design.
Using a small held-out set of 50 images for both datasets
to determine the algorithm hyperparameters. To measure
optimality of the parameters we used average over PDJ at
0:2across all joints. The scaler , which deﬁnes the size of
the reﬁnement joint bounding box as a fraction of the pose
size, is determined as follows: for FLIC we chose = 1:0
after exploring values f0:8;1:0;1:2g, for LSP we use =
2:0after tryingf1:5;1:7;2:0;2:3g. The number of cascade
stagesSis determined by training stages until the algorithm
stopped improving on the held-out set. For both FLIC and
LSP we arrived at S= 3.
To improve generalization, for each cascade stage start-
ing ats= 2 we augment the training data by sampling 40
randomly translated crop boxes for each joint as explained
in Sec. 3.2. Thus, for LSP with 14joints and after mirror-
ing the images and sampling the number training examples
is1100040214 = 12M, which is essential for
training a large network as ours.The presented algorithm allows for an efﬁcient imple-
mentation. The running time is approx. 0:1sper image,
as measured on a 12core CPU. This compares favorably
to other approaches, as some of the current state-of-art ap-
proaches have higher complexity: [20] runs in approx. 4s,
while [27] runs in 1:5s. The training complexity, however,
is higher. The initial stage was trained within 3days on
approx. 100 workers, most of the ﬁnal performance was
achieved after 12hours though. Each reﬁnement stage was
trained for 7days since the amount of data was 40larger
than the one for the initial stage due to the data augmenta-
tion in Sec. 3.2. Note that using more data led to increased
performance.
4.2. Results and Discussion
Comparisons We present comparative results to other ap-
proaches. We compare on LSP using PCP metric in Fig. 1.
We show results for the four most challenging limbs – lower
and upper arms and legs – as well as the average value
across these limbs for all compared algorithms. We clearly
outperform all other approaches, especially achieving bet-
ter estimation for legs. For example, for upper legs we ob-
tain0:78up from 0:74for the next best performing method.
It is worth noting that while the other approaches exhibit
strengths for particular limbs, none of the other dataset con-
sistently dominates across all limbs. In contrary, DeepPose
shows strong results for all challenging limbs.
Using the PDJ metric allows us to vary the threshold for
the distance between prediction and ground truth, which de-
ﬁnes a detection. This threshold can be thought of as a
localization precision at which detection rates are plotted.
Thus one could compare approaches across different de-
sired precisions. We present results on FLIC in Fig. 3 com-
paring against additional four methods as well is on LSP in
Fig. 4. For each dataset we train and test according the pro-
tocol for each dataset. Similarly to previous experiment we
outperform all ﬁve algorithms. Our gains are bigger in the
low precision domain, in the cases where we detect rough
pose without precisely localizing the joints. On FLIC, at
normalized distance 0:2we obtain a an increase of detection
rates by 0:15and0:2for elbow and wrists against the next
best performing method. On LSP, at normalized distance
0:5we get an absolute increase of 0:1. At low precision
regime of normalized distance of 0:2for LSP we show com-
parable performance for legs and slightly worse arms. This
can be attributed to the fact that the DNN-based approach
computes joint coordinates using 7 layers of transformation,
some of which contain max pooling.
Another observation is that our approach works well for
both appearance heavy movie data as well as string articu-
lation such as the sports images in LSP.
Effects of cascade-based reﬁnement A single DNN-
based joint regressor gives rough joint location. However,
0 0.05 0.1 0.15 0.20.10.30.50.70.9Elbows
Normalized distance to true jointDetection rate

DeepPoseMODECEichner et al.Yang et al.Sapp et al.
0 0.05 0.1 0.15 0.20.10.30.50.70.9Wrists
Normalized distance to true jointDetection rateFigure 3. Percentage of detected joints (PDJ) on FLIC for two joints: elbow and wrist. We compare DeepPose, after two cascade stages,
with four other approaches.
MethodArm LegAve.Upper Lower Upper Lower
DeepPose-st1 0.5 0.27 0.74 0.65 0.54
DeepPose-st2 0.56 0.36 0.78 0.70 0.60
DeepPose-st3 0.56 0.38 0.77 0.71 0.61
Dantone et al. [2] 0.45 0.25 0.65 0.61 0.49
Tian et al. [25]0.52 0.33 0.70 0.60 0.56
Johnson et al. [13] 0.54 0.38 0.75 0.66 0.58
Wang et al. [26]0.565 0.37 0.76 0.68 0.59
Pishchulin [18]0.49 0.32 0.74 0.70 0.56
Table 1. Percentage of Correct Parts (PCP) at 0.5 on LSP for Deep-
Pose as well as ﬁve state-of-art approaches.The authors use a
slightly looser version of PCP in which for each limb the average
distance from predicted limb joints belonging to the correspond-
ing true joints is used to determine whether the limb is correctly
detected.The authors use person-centric joint annotations.
0 0.1 0.2 0.3 0.4 0.50.10.30.50.70.9Arms
Normalized distance to true jointDetection rate

DeepPose − wristsDeepPose − elbowsDantone et al. − wristsDantone et al. − elbows
0 0.1 0.2 0.3 0.4 0.50.10.30.50.70.9Legs
Normalized distance to true jointDetection rate

DeepPose − ankleDeepPose − kneeDantone et al. − ankleDantone et al. − knee
Figure 4. Percentage of detected joints (PDJ) on LSP for four
limbs for DeepPose and Dantone et al. [2] over an extended range
of distances to true joint: [0;0:5]of the torso diameter. Results of
DeepPose are plotted with solid lines while all the results by [2]
are plotted in dashed lines. Results for the same joint from both
algorithms are colored with same color.
to obtain higher precision the subsequent stages of the cas-
cade, which serve as a reﬁnement of the initial prediction,
are of paramount importance. To see this, in Fig. 5 we
present the joint detections at different precisions for the ini-
tial prediction as well as two subsequent cascade stages. As
expected, we can see that the major gains of the reﬁnement
00.05 0.10.15 0.20.25 0.30.35 0.40.10.30.50.70.9Wrists
Normalized distance to true jointDetection rate

DeepPose − initial stage 1DeepPose − stage 2DeepPose − stage 3
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.40.10.30.50.70.9Elbows
Normalized distance to true jointDetection rateFigure 5. Percent of detected joints (PDJ) on FLIC or the ﬁrst three
stages of the DNN cascade. We present results over larger spec-
trum of normalized distances between prediction and ground truth.
procedure are at high-precision regime of at normalized dis-
tances of [0:15;0:2]. Further, the major gains are achieved
after one stage of reﬁnement. The reason being that subse-
quent stages end up using smaller sub-images around each
joint. And although the subsequent stages look at higher
resolution inputs, they have more limited context.
Examples of cases, where reﬁnement helps, are visual-
ized in Fig. 6. The initial stage is usually successful at es-
timating a roughly correct pose, however, this pose is not
”snapped” to the correct one. For example, in row three the
pose has the right shape but incorrect scale. In the second
row, the predicted pose is translated north from the ideal
one. In most cases, the second stage of the cascade resolves
this snapping problem and better aligns the joints. In more
rare cases, such as in ﬁrst row, further facade stages improve
on individual joints.
Cross-dataset Generalization To evaluate the general-
ization properties of our algorithm, we used the trained
models on LSP and FLIC on two related datasets. The full-
body model trained on LSP is tested on the test portion of
the Image Parse dataset [19] with results presented in Ta-
ble 2. The ImageParse dataset is similar to LSP as it con-
tains people doing sports, however it contains a lot of peo-
ple from personal photo collections involved in other activ-
ities. Further, the upper-body model trained on FLIC was
applied on the whole Buffy dataset [7]. We can see that our
Initial stage 1stage 2stage 3Figure 6. Predicted poses in red and ground truth poses in green
for the ﬁrst three stages of a cascade for three examples.
0 0.05 0.1 0.15 0.20.10.30.50.70.9Elbows
Normalized distance to true jointDetection rate

Eichner et al.Yang et al.Sapp et al.MODECDeepPose
0 0.05 0.1 0.15 0.20.10.30.50.70.9Wrists
Normalized distance to true jointDetection rate
Figure 7. Percentage of detected joints (PDJ) on Buffy dataset
for two joints: elbow and wrist. The models have been trained on
FLIC. We compare DeepPose, after two cascade stages, with four
other approaches.
MethodArm LegAve.Upper Lower Upper Lower
DeepPose 0.8 0.75 0.71 0.5 0.69
Pishchulin [18] 0.80 0.70 0.59 037 0.62
Johnson et al. [13] 0.75 0.67 0.67 0.46 0.64
Yang et al. [27] 0.69 0.64 0.55 0.35 0.56
Table 2. Percentage of Correct Parts (PCP) at 0.5 on Image Parse
dataset for DeepPose as well as two state-of-art approaches on Im-
age Parse dataset. Results obtained from [18].
approach can retain state-of-art performance compared to
other approaches. This shows good generalization abilities.
Example poses To get a better idea of the performance of
our algorithm, we visualize a sample of estimated poses on
images from LSP in Fig. 8. We can see that our algorithm is
able to get correct pose for most of the joints under variety
of conditions: upside-down people (row 1, column 1), se-
vere foreshortening (row1, column 3), unusual poses (row
3, column 5), occluded limbs as the occluded arms in row
3, columns 2 and 6, unusual illumination conditions (row 3,
column 3). In most of the cases, when the estimated pose isnot precise, it still has a correct shape. For example, in the
last row some of the predicted limbs are not aligned with
the true locations, however the overall shape of the pose is
correct. A common failure mode is confusing left with right
side when the person was photographed from the back (row
6, column 6). Results on FLIC (see Fig. 9) are usually better
with occasional visible mistakes on lower arms.
5. Conclusion
We present, to our knowledge, the ﬁrst application of
Deep Neural Networks (DNNs) to human pose estimation.
Our formulation of the problem as DNN-based regression to
joint coordinates and the presented cascade of such regres-
sors has the advantage of capturing context and reasoning
about pose in a holistic manner. As a result, we are able to
achieve state-of-art or better results on several challenging
academic datasets.
Further, we show that using a generic convolutional neu-
ral network, which was originally designed for classiﬁca-
tion tasks, can be applied to the different task of localiza-
tion. In future, we plan to investigate novel architectures
which could be potentially better tailored towards localiza-
tion problems in general, and in pose estimation in particu-
lar.
Acknowledgements I would like to thank Luca Bertelli,
Ben Sapp and Tianli Yu for assistance with data and fruitful
discussions.
References
[1] M. Andriluka, S. Roth, and B. Schiele. Pictorial structures
revisited: People detection and articulated pose estimation.
InCVPR , 2009.
[2] M. Dantone, J. Gall, C. Leistner, and L. Van Gool. Human
pose estimation using body parts dependent joint regressors.
InCVPR , 2013.
[3] J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient
methods for online learning and stochastic optimization. In
COLT . ACL, 2010.
[4] M. Eichner and V . Ferrari. Better appearance models for
pictorial structures. 2009.
[5] M. Eichner, M. Marin-Jimenez, A. Zisserman, and V . Ferrari.
Articulated human pose estimation and search in (almost)
unconstrained still images. ETH Zurich, D-ITET, BIWI,
Technical Report No , 272, 2010.
[6] P. F. Felzenszwalb and D. P. Huttenlocher. Pictorial struc-
tures for object recognition. International Journal of Com-
puter Vision , 61(1):55–79, 2005.
[7] V . Ferrari, M. Marin-Jimenez, and A. Zisserman. Progressive
search space reduction for human pose estimation. In CVPR ,
2008.
[8] M. A. Fischler and R. A. Elschlager. The representation and
matching of pictorial structures. Computers, IEEE Transac-
tions on , 100(1):67–92, 1973.
Figure 8. Visualization of pose results on images from LSP. Each pose is represented as a stick ﬁgure, inferred from predicted joints.
Different limbs in the same image are colored differently, same limb across different images has the same color.
Figure 9. Visualization of pose results on images from FLIC. Meaning of stick ﬁgures is the same as in Fig. 8 above.
[9] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich fea-
ture hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014.
[10] G. Gkioxari, P. Arbel ´aez, L. Bourdev, and J. Malik. Articu-
lated pose estimation using discriminative armlet classiﬁers.
InCVPR , 2013.
[11] C. Ionescu, F. Li, and C. Sminchisescu. Latent structured
models for human pose estimation. In ICCV , 2011.
[12] S. Johnson and M. Everingham. Clustered pose and nonlin-
ear appearance models for human pose estimation. In BMVC ,
2010.
[13] S. Johnson and M. Everingham. Learning effective human
pose estimation from inaccurate annotation. In CVPR , 2011.
[14] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet clas-
siﬁcation with deep convolutional neural networks. In NIPS ,
2012.
[15] G. Mori and J. Malik. Estimating human body conﬁgurations
using shape context matching. In ECCV , 2002.[16] R. Nevatia and T. O. Binford. Description and recognition of
curved objects. Artiﬁcial Intelligence , 8(1):77–98, 1977.
[17] M. Osadchy, Y . LeCun, and M. L. Miller. Synergistic face
detection and pose estimation with energy-based models.
The Journal of Machine Learning Research , 8:1197–1215,
2007.
[18] L. Pishchulin, M. Andriluka, P. Gehler, and B. Schiele. Pose-
let conditioned pictorial structures. In CVPR , 2013.
[19] D. Ramanan. Learning to parse images of articulated bodies.
InNIPS , 2006.
[20] B. Sapp and B. Taskar. Modec: Multimodal decomposable
models for human pose estimation. In CVPR , 2013.
[21] G. Shakhnarovich, P. Viola, and T. Darrell. Fast pose estima-
tion with parameter-sensitive hashing. In CVPR , 2003.
[22] Y . Sun, X. Wang, and X. Tang. Deep convolutional net-
work cascade for facial point detection. In Computer Vision
and Pattern Recognition (CVPR), 2013 IEEE Conference on ,
pages 3476–3483. IEEE, 2013.
[23] C. Szegedy, A. Toshev, and D. Erhan. Object detection via
deep neural networks. In NIPS 26 , 2013.
[24] G. W. Taylor, R. Fergus, G. Williams, I. Spiro, and C. Bre-
gler. Pose-sensitive embedding by nonlinear nca regression.
InNIPS , 2010.
[25] Y . Tian, C. L. Zitnick, and S. G. Narasimhan. Exploring the
spatial hierarchy of mixture models for human pose estima-
tion. In ECCV , 2012.
[26] F. Wang and Y . Li. Beyond physical connections: Tree mod-
els in human pose estimation. In CVPR , 2013.
[27] Y . Yang and D. Ramanan. Articulated pose estimation with
ﬂexible mixtures-of-parts. In CVPR , 2011.


Summary:
The latest generation of Convolutional Neural Networks (CNN) have achieved
impressive results in challenging benchmarks on image recognition and object
detection, significantly raising the interest of the community in these
methods. Nevertheless, it is still unclear how different CNN methods compare
with each other and with previous state-of-the-art shallow representations such
as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper conducts
a rigorous evaluation of these new techniques, exploring different deep
architectures and comparing them on a common ground, identifying and disclosing
important implementation details. We identify several useful properties of
CNN-based representations, including the fact that the dimensionality of the
CNN output layer can be reduced significantly without having an adverse effect
on performance. We also identify aspects of deep and shallow methods that can
be successfully shared. In particular, we show that the data augmentation
techniques commonly applied to CNN-based methods can also be applied to shallow
methods, and result in an analogous performance boost. Source code and models
to reproduce the experiments in the paper is made publicly available.

Ken Chatfield
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman

Published in proceedings of BMVC 2014

1
Return of the Devil in the Details:
Delving Deep into Convolutional Nets
Ken Chatﬁeld, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman
Visual Geometry Group, Department of Engineering Science, University of Oxford
fken,karen,vedaldi,az g@robots.ox.ac.uk
Abstract —The latest generation of Convolutional Neural Networks (CNN) have achieved impressive results in chal-
lenging benchmarks on image recognition and object detection, signiﬁcantly raising the interest of the community in
these methods. Nevertheless, it is still unclear how different CNN methods compare with each other and with previous
state-of-the-art shallow representations such as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper
conducts a rigorous evaluation of these new techniques, exploring different deep architectures and comparing them on
a common ground, identifying and disclosing important implementation details. We identify several useful properties
of CNN-based representations, including the fact that the dimensionality of the CNN output layer can be reduced
signiﬁcantly without having an adverse effect on performance. We also identify aspects of deep and shallow methods
that can be successfully shared. In particular, we show that the data augmentation techniques commonly applied to
CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost. Source
code and models to reproduce the experiments in the paper is made publicly available.
F
1 I NTRODUCTION
PERHAPS the single most important design
choice in current state-of-the-art image classiﬁ-
cation and object recognition systems is the choice
of visual features, or image representation. In fact,
most of the quantitative improvements to image
understanding obtained in the past dozen years can
be ascribed to the introduction of improved repre-
sentations, from the Bag-of-Visual-Words (BoVW) [1],
[2] to the (Improved) Fisher Vector (IFV) [3]. A
common characteristic of these methods is that
they are largely handcrafted . They are also relatively
simple, comprising dense sampling of local image
patches, describing them by means of visual de-
scriptors such as SIFT, encoding them into a high-
dimensional representation, and then pooling over
the image. Recently, these handcrafted approaches
have been substantially outperformed by the in-
troduction of the latest generation of Convolutional
Neural Networks (CNNs) [4] to the computer vision
ﬁeld. These networks have a substantially more
sophisticated structure than standard representa-
tions, comprising several layers of non-linear fea-
ture extractors, and are therefore said to be deep
(in contrast, classical representation will be referred
to as shallow ). Furthermore, while their structureis handcrafted, they contain a very large number
of parameters learnt from data. When applied to
standard image classiﬁcation and object detection
benchmark datasets such as ImageNet ILSVRC [5]
and PASCAL VOC [6] such networks have demon-
strated excellent performance [7], [8], [9], [10], [11],
signiﬁcantly better than standard image encod-
ings [12].
Despite these impressive results, it remains un-
clear how different deep architectures compare to
each other and to shallow computer vision meth-
ods such as IFV . Most papers did not test these
representations extensively on a common ground,
so a systematic evaluation of the effect of differ-
ent design and implementation choices remains
largely missing. As noted in our previous work [12],
which compared the performance of various shal-
low visual encodings, the performance of computer
vision systems depends signiﬁcantly on implementation
details . For example, state-of-the-art methods such
as [13] not only involve the use of a CNN, but
also include other improvements such as the use
of very large scale datasets, GPU computation, and
data augmentation (also known as data jittering or
virtual sampling). These improvements could also
transfer to shallow representations such as the IFV ,
potentially explaining a part of the performancearXiv:1405.3531v4  [cs.CV]  5 Nov 2014
2
gap [14].
In this study we analyse and empirically clarify
these issues, conducting a large set of rigorous
experiments (Sect. 4), in many ways picking up
the story where it last ended in [12] with the
comparison of shallow encoders. We focus on meth-
ods to construct image representations ,i.e. encod-
ing functions mapping an image Ito a vector
(I)2Rdsuitable for analysis with a linear classi-
ﬁer, such as an SVM. We consider three scenarios
(Sect. 2, Sect. 3): shallow image representations,
deep representations pre-trained on outside data,
and deep representation pre-trained and then ﬁne-
tuned on the target dataset. As part of our tests,
we explore generally-applicable best practices that
are nevertheless more often found in combina-
tion with CNNs [13] or, alternatively, with shallow
encoders [12], porting them with mutual beneﬁt.
These are (Sect. 2): the use of colour information ,
feature normalisation , and, most importantly, the
use of substantial data augmentation . We also de-
termine scenario-speciﬁc best-practices , improving
the ones in [12], [15] and others, including dimen-
sionality reduction for deep features. Finally, we
achieve performance competitive with the state
of the art [16], [17] on PASCAL VOC classiﬁcation
using less additional training data and signiﬁcantly
simpler techniques. As in [12], the source code and
models to reproduce all experiments in this paper
is available on the project website1.
2 S CENARIOS
This section introduces the three types of image
representation (I)considered in this paper, de-
scribing them within the context of three different
scenarios. Having outlined details speciﬁc to each,
general methodologies which apply to all three
scenarios are reviewed, such as data augmentation
and feature normalisation, together with the linear
classiﬁer (trained with a standard hinge loss). We
also specify here the benchmark datasets used in
the evaluation.
2.1 Scenario 1: Shallow representation (IFV)
Our reference shallow image representation is the
IFV [3]. Our choice is motivated by the fact that
IFV usually outperforms related encoding methods
such as BoVW, LLC [12], and VLAD [18]. Given an
imageI, the IFVFV(I)is obtained by extracting
1. http://www.robots.ox.ac.uk/vgg/research/deep eval/a dense collection of patches and corresponding
local descriptors xi2RD(e.g. SIFT) from the image
at multiple scales. Each descriptor xiis then soft-
quantized using a Gaussian Mixture Model with
Kcomponents. First and second order differences
between each descriptor xiand its Gaussian cluster
meankare accumulated in corresponding blocks
uk,vkin the vector FV(I)2R2KD, appropriately
weighed by the Gaussian soft-assignments and
covariance, leading to a 2KD-dimensional image
representation FV(I) = [u>
1;v>
1;:::u>
K;v>
K]>:The
improved version of the Fisher vector involves post-
processing FVby computing the signed square-
root of its scalar components and normalising the
result to a unit `2norm. The details of this con-
struction can be found in [3]; here we follow the
notation of [12].
2.2 Scenario 2: Deep representation (CNN) with
pre-training
Our deep representations are inspired by the suc-
cess of the CNN of Krizhevsky et al. [13]. As shown
in [7], [19], the vector of activities CNN(I)of the
penultimate layer of a deep CNN, learnt on a large
dataset such as ImageNet [5], can be used as a pow-
erful image descriptor applicable to other datasets.
Numerous CNN architectures that improve the
previous state of the art obtained using shallow
representations have been proposed, but choosing
the best one remains an open question. Many are in-
spired by [13]: DeCAF [7], [11], Caffe [20], Oquab et
al. [8]. Others use larger networks with a smaller
stride of the ﬁrst convolutional layer: Zeiler and
Fergus [19] and OverFeat [10], [9]. Other differences
include the CNN pre-training protocols. Here we
adopt a single learning framework and experiment
with architectures of different complexity exploring
their performance-speed trade-off.
2.3 Scenario 3: Deep representation (CNN) with
pre-training and ﬁne-tuning
In Scenario 2 features are trained on one (large)
dataset and applied to another (usually smaller).
However, it was demonstrated [11] that ﬁne-tuning
a pre-trained CNN on the target data can sig-
niﬁcantly improve the performance. We consider
this scenario separately from that of Scenario 2, as
the image features become dataset-speciﬁc after the
ﬁne-tuning.
3
2.4 Commonalities
We now turn to what is in common across the
scenarios.
2.4.1 Data augmentation
Data augmentation is a method applicable to shal-
low and deep representations, but that has been
so far mostly applied to the latter [13], [19]. By
augmentation we mean perturbing an image Iby
transformations that leave the underlying class un-
changed ( e.g. cropping and ﬂipping) in order to
generate additional examples of the class. Augmen-
tation can be applied at training time, at test time,
or both. The augmented samples can either be taken
as-is or combined to form a single feature, e.g. using
sum/max-pooling or stacking.
2.4.2 Linear predictors
All the representations (I)in the three scenarios
are used to construct linear predictorshw;(I)ifor
each class to be recognized. These predictors are
learnt using Support Vector Machines (SVM) by
ﬁtting wto the available training data by mini-
mizing an objective function balancing a quadratic
regularizer and the hinge-loss. The parameter C
in the SVM, trading-off regularizer and loss, is
determined using an held-off validation subset of
the data. Here we use the same learning framework
with all representations. It is common experience
that linear classiﬁers are particularly sensitive to
thenormalisation of the data and that, in particular,
SVMs tend to beneﬁt from `2normalisation [3] (an
interpretation is that after normalisation the inner
product corresponds to the cosine similarly).
2.5 Benchmark data
As reference benchmark we use the PASCAL
VOC [6] data as already done in [12]. The VOC-
2007 edition contains about 10,000 images split into
train, validation, and test sets, and labelled with
twenty object classes. A one-vs-rest SVM classiﬁer
for each class is learnt and evaluated independently
and the performance is measured as mean Average
Precision (mAP) across all classes. The VOC-2012
edition contains roughly twice as many images
and does not include test labels; instead, evaluation
uses the ofﬁcial PASCAL Evaluation Server. To train
deep representations we use the ILSVRC-2012 chal-
lenge dataset. This contains 1,000 object categories
from ImageNet [5] with roughly 1.2M training im-
ages, 50,000 validation images, and 100,000 testimages. Performance is evaluated using the top-5
classiﬁcation error. Finally, we also evaluate over
theCaltech-101 and Caltech-256 image classiﬁca-
tion benchmarks [21], [22]. For Caltech-101, we
followed the protocol of [12], and considered three
random splits into training and testing data, each
of which comprises 30 training and up to 30 testing
images per class. For Caltech-256, two random
splits were generated, each of which contains 60
training images per class, and the rest are used for
testing. On both Caltech datasets, performance is
measured using mean class accuracy.
3 D ETAILS
This section gives the implementation details of the
methods introduced in Sect. 2.
3.1 Improved Fisher Vector details
Our IFV representation uses a slightly improved
setting compared to the best result of [12].
Computation starts by upscaling the image Iby a
factor of 2 [23], followed by SIFT features extraction
with a stride of 3 pixels at 7 different scales withp
2
scale increments. These features are square-rooted
as suggested by [24], and decorrelated and reduced
in dimension from 128Dto80Dusing PCA. A
GMM with K= 256 components is learnt from fea-
tures sampled from the training images. Hence the
Fisher Vector FV(I)has dimension 2KD= 40;960.
Before use in classiﬁcation, the vector is signed-
square-rooted and l2-normalised (square rooting
correspond to the Hellinger’s kernel map [25]). As
in [12], square-rooting is applied twice, once to the
raw encodings, and once again after sum pooling
and normalisation. In order to capture weak geo-
metrical information, the IFV representation is used
in a spatial pyramid [26]. As in [12], the image is
divided into 11,31, and 22spatial subdivisions
and corresponding IFVs are computed and stacked
with an overall dimension of 82KD= 327;680
elements.
In addition to this standard formulation, we ex-
periment with a few modiﬁcations. The ﬁrst one
is the use of intra-normalisation of the descriptor
blocks, an idea recently proposed for the VLAD
descriptor [27]. In this case, the `2normalisation
is applied to the individual sub-blocks (uk;vk)of
the vector FV(I), which helps to alleviate the
local feature burstiness [28]. In the case of the
improved intra-normalised features, it was found
4
that applying the square-rooting only once to the
ﬁnal encoding produced the best results.
The second modiﬁcation is the use of spatially-
extended local descriptors [23] instead of a spatial
pyramid. Here descriptors xiare appended with
their image location (xi;yi)before quantization
with the GMM. Formally, xiis extended, after PCA
projection, with its normalised spatial coordinates:
[x>
i;xi=W0:5;yi=H0:5]>, whereWHare the
dimensions of the image. Since the GMM quantizes
both appearance and location, this allows for spatial
information to be captured directly by the soft-
quantization process. This method is signiﬁcantly
more memory-efﬁcient than using a spatial pyra-
mid. Speciﬁcally, the PCA-reduced SIFT features
are spatially augmented by appending (x;y)yield-
ingD= 82 dimensional descriptors pooled in a
2KD= 41;984dimensional IFV .
The third modiﬁcation is the use of colour fea-
tures in addition to SIFT descriptors. While colour
information is used in CNNs [13] and by the orig-
inal FV paper [3], it was not explored in our pre-
vious comparison [12]. We do so here by adopting
the same Local Colour Statistics (LCS) features as
used by [3]. LCS is computed by dividing an input
patch into a 44spatial grid (akin to SIFT), and
computing the mean and variance of each of the
Lab colour channels for each cell of the grid. The
LCS dimensionality is thus 4423 = 96 . This
is then encoded in a similar manner to SIFT.
3.2 Convolutional neural networks details
The CNN-based features are based on three CNN
architectures representative of the state of the art
(shown in Table 1) each exploring a different accu-
racy/speed trade-off. To ensure a fair comparison
between them, these networks are trained using the
same training protocol and the same implemen-
tation, which we developed based on the open-
source Caffe framework [20]. `2-normalising the
CNN features CNN(I)before use in the SVM was
found to be important for performance.
Our Fast (CNN-F) architecture is similar to the
one used by Krizhevsky et al . [13]. It comprises
8 learnable layers, 5 of which are convolutional,
and the last 3 are fully-connected. The input image
size is 224224. Fast processing is ensured by the
4 pixel stride in the ﬁrst convolutional layer. The
main differences between our architecture and that
of [13] are the reduced number of convolutional
layers and the dense connectivity between convolu-tional layers ([13] used sparse connections to enable
training on two GPUs).
Our Medium (CNN-M) architecture is similar
to the one used by Zeiler and Fergus [19]. It is
characterised by the decreased stride and smaller
receptive ﬁeld of the ﬁrst convolutional layer, which
was shown to be beneﬁcial on the ILSVRC dataset.
At the same time, conv2 uses larger stride (2 instead
of 1) to keep the computation time reasonable. The
main difference between our net and that of [19] is
we use less ﬁlters in the conv4 layer (512 vs. 1024).
Our Slow (CNN-S) architecture is related to the
‘accurate’ network from the OverFeat package [10].
It also uses 77ﬁlters with stride 2in conv1.
Unlike CNN-M and [19], the stride in conv2 is
smaller (1 pixel), but the max-pooling window in
conv1 and conv5 is larger ( 33) to compensate for
the increased spatial resolution. Compared to [10],
we use 5 convolutional layers as in the previous
architectures ([10] used 6), and less ﬁlters in conv5
(512 instead of 1024); we also incorporate an LRN
layer after conv1 ([10] did not use contrast normal-
isation).
3.2.1 CNN training
In general, our CNN training procedure fol-
lows that of [13], learning on ILSVRC-2012 us-
ing gradient descent with momentum. The hyper-
parameters are the same as used by [13]: momen-
tum0:9; weight decay 5104; initial learning rate
102, which is decreased by a factor of 10, when
the validation error stop decreasing. The layers are
initialised from a Gaussian distribution with a zero
mean and variance equal to 102. We also employ
similar data augmentation in the form of random
crops, horizontal ﬂips, and RGB colour jittering.
Test time crop sampling is discussed in Sect. 3.3;
at training time, 224224 crops are sampled ran-
domly, rather than deterministically. Thus, the only
notable difference to [13] is that the crops are taken
from the whole training image P256;P256,
rather than its 256256 centre. Training was per-
formed on a single NVIDIA GTX Titan GPU and
the training time varied from 5 days for CNN-F to
3 weeks for CNN-S.
3.2.2 CNN ﬁne-tuning on the target dataset
In our experiments, we ﬁne-tuned CNN-S using
VOC-2007, VOC-2012, or Caltech-101 as the target
data. Fine-tuning was carried out using the same
framework (and the same data augmentation), as
we used for CNN training on ILSVRC. The last
5
Arch. conv1 conv2 conv3 conv4 conv5 full6 full7 full8
CNN-F64x11x11 256x5x5 256x3x3 256x3x3 256x3x3 4096 4096 1000
st. 4, pad 0 st. 1, pad 2 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- soft-
LRN, x2 pool LRN, x2 pool - - x2 pool out out max
CNN-M96x7x7 256x5x5 512x3x3 512x3x3 512x3x3 4096 4096 1000
st. 2, pad 0 st. 2, pad 1 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- soft-
LRN, x2 pool LRN, x2 pool - - x2 pool out out max
CNN-S96x7x7 256x5x5 512x3x3 512x3x3 512x3x3 4096 4096 1000
st. 2, pad 0 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- soft-
LRN, x3 pool x2 pool - - x3 pool out out max
TABLE 1
CNN architectures. Each architecture contains 5 convolutional layers (conv 1–5) and three
fully-connected layers (full 1–3). The details of each of the convolutional layers are given in three sub-rows:
the ﬁrst speciﬁes the number of convolution ﬁlters and their receptive ﬁeld size as “num x size x size”; the
second indicates the convolution stride (“st.”) and spatial padding (“pad”); the third indicates if Local
Response Normalisation (LRN) [13] is applied, and the max-pooling downsampling factor. For full 1–3, we
specify their dimensionality, which is the same for all three architectures. Full6 and full7 are regularised
using dropout [13], while the last layer acts as a multi-way soft-max classiﬁer. The activation function for all
weight layers (except for full8) is the REctiﬁcation Linear Unit (RELU) [13].
fully-connected layer (conv8) has output dimen-
sionality equal to the number of classes, which
differs between datasets, so we initialised it from
a Gaussian distribution (as used for CNN training
above). Now we turn to dataset-speciﬁc ﬁne-tuning
details.
VOC-2007 and VOC-2012. Considering that PAS-
CAL VOC is a multi-label dataset ( i.e. a single
image might have multiple labels), we replaced the
softmax regression loss with a more appropriate
loss function, for which we considered two options:
one-vs-rest classiﬁcation hinge loss (the same loss
as used in the SVM experiments) and ranking hinge
loss. Both losses deﬁne constraints on the scores of
positive (Ipos) and negative ( Ineg) images for each
class:wc(Ipos)>1;wc(Ineg)<1 +for the
classiﬁcation loss, wc(Ipos)> w c(Ineg) + 1
for the ranking loss ( wcis thec-th row of the
last fully-connected layer, which can be seen as a
linear classiﬁer on deep features (I);is a slack
variable). Our ﬁne-tuned networks are denoted as
“CNN S TUNE-CLS” (for the classiﬁcation loss)
and “CNN S TUNE-RNK” (for the ranking loss).
In the case of both VOC datasets, the training
and validation subsets were combined to form a
single training set. Given the smaller size of the
training data when compared to ILSVRC-2012, we
controlled for over-ﬁtting by using lower initial
learning rates for the ﬁne-tuned hidden layers.The learning rate schedule for the last layer /
hidden layers was: 102=104!103=104!
104=104!105=105.
Caltech-101 dataset contains a single class label
per image, so ﬁne-tuning was performed using the
softmax regression loss. Other settings (including
the learning rate schedule) were the same as used
for the VOC ﬁne-tuning experiments.
3.2.3 Low-dimensional CNN feature training
Our baseline networks (Table 1) have the same
dimensionality of the last hidden layer (full7): 4096.
This design choice is in accordance with the state-
of-the-art architectures [13], [19], [10], and leads to
a 4096-D dimensional image representation, which
is already rather compact compared to IFV . We
further trained three modiﬁcations of the CNN-M
network, with lower dimensional full7 layers of:
2048, 1024, and 128 dimensions respectively. The
networks were learnt on ILSVRC-2012. To speed-
up training, all layers aside from full7/full8 were
set to those of the CNN-M net and a lower initial
learning rate of 103was used. The initial learning
rate of full7/full8 was set to 102.
3.3 Data augmentation details
We explore three data augmentation strategies. The
ﬁrst strategy is to use no augmentation . In con-
trast to IFV , however, CNNs require images to
6
be transformed to a ﬁxed size ( 224224) even
when no augmentation is used. Hence the image is
downsized so that the smallest dimension is equal
to224 pixels and a 224224 crop is extracted
from the centre.2The second strategy is to use
ﬂip augmentation , mirroring images about the y-
axis producing two samples from each image. The
third strategy, termed C+F augmentation , combines
cropping and ﬂipping. For CNN-based representa-
tions, the image is downsized so that the smallest
dimension is equal to 256 pixels. Then 224224
crops are extracted from the four corners and the
centre of the image. Note that the crops are sampled
from the whole image, rather than its 256256
centre, as done by [13]. These crops are then ﬂipped
about they-axis, producing 10perturbed samples
per input image. In the case of the IFV encoding, the
same crops are extracted, but at the original image
resolution.
4 A NALYSIS
This section describes the experimental results,
comparing different features and data augmenta-
tion schemes. The results are given in Table 2 for
VOC-2007 and analysed next, starting from gener-
ally applicable methods such as augmentation and
then discussing the speciﬁcs of each scenario. We
then move onto other datasets and the state of the
art in Sect. 4.7.
4.1 Data augmentation
We experiment with no data augmentation (de-
noted Image Aug=– in Tab. 2), ﬂip augmenta-
tion ( Image Aug=F ), and C+F augmentation ( Image
Aug=C ). Augmented images are used as stand-
alone samples ( f), or by fusing the corresponding
descriptors using sum ( s) or max ( m) pooling or
stacking ( t). So for example Image Aug=(C) f s in
row [f] of Tab. 2 means that C+F augmentation is
used to generate additional samples in training ( f),
and is combined with sum-pooling in testing ( s).
Augmentation consistently improves perfor-
mance by3%for both IFV ( e.g. [d] vs. [f]) and
CNN ( e.g. [o] vs. [p]). Using additional samples for
training and sum-pooling for testing works best
([p]) followed by sum-pooling [r], max pooling [q],
and stacking [s]. In terms of the choice of transfor-
mations, ﬂipping improves only marginally ([o] vs.
2. Extracting a 224224 centre crop from a 256256
image [13] resulted in worse performance.[u]), but using the more expensive C+F sampling
improves, as seen, by about 23%([o]vs. [p]). We
experimented with sampling more transformations,
taking a higher density of crops from the centre of
the image, but observed no beneﬁt.
4.2 Colour
Colour information can be added and subtracted in
CNN and IFV . In IFV replacing SIFT with the colour
descriptors of [3] (denoted COL inMethod ) yields
signiﬁcantly worse performance ([j] vs. [h]). How-
ever, when SIFT and colour descriptors are com-
bined by stacking the corresponding IFVs ( COL+ )
there is a small but signiﬁcant improvement of
around1%in the non-augmented case ( e.g. [h]
vs. [k]) but little impact in the augmented case
(e.g. [i] vs. [l]). For CNNs, retraining the network
after converting all the input images to grayscale
(denoted GSinMethods ) has a more signiﬁcant
impact, resulting in a performance drop of 3%
([w] vs. [p], [v] vs. [o]).
4.3 Scenario 1: Shallow representation (IFV)
The baseline IFV encoding using a spatial pyramid
[a] performs slightly better than the results [I] taken
from Chatﬁeld et al. [12], primarily due to a larger
number of spatial scales being used during SIFT
feature extraction, and the resultant SIFT features
being square-rooted. Intra-normalisation , denoted as
INin the Method column of the table, improves the
performance by 1%(e.g. [c] vs. [d]). More inter-
estingly, switching from spatial pooling (denoted
spm in the SPool column) to feature spatial aug-
mentation ( SPool=(x,y) ) has either little effect on the
performance or results in a marginal increase ([a] vs.
[c], [b] vs. [d]), whilst resulting in a representation
which is over 10 smaller. We also experimented
with augmenting with scale in addition to position
as in [23] but observed no improvement. Finally,
we investigate pushing the parameters of the repre-
sentation setting K= 512 (rows [h]-[l]). Increasing
the number of GMM centres in the model from
K= 256 to512 results in a further performance
increase (e.g. [h] vs. [d]), but at the expense of
higher-dimensional codes (125K dimensional).
4.4 Scenario 2: Deep representation (CNN) with
pre-training
CNN-based methods consistently outperform the
shallow encodings, even after the improvements
7
Method SPool Image Aug. Dim mAP
(I) FK BL spm – 327K 61.69 79.0 67.4 51.9 70.9 30.8 72.2
(II) DECAF – (C) t t 327K 73.41 87.4 79.3 84.1 78.4 42.3 73.7
(a) FK spm – 327K 63.66 83.4 68.8 59.6 74.1 35.7 71.2
(b) FK IN spm – 327K 64.18 82.1 69.7 59.7 75.2 35.7 71.3
(c) FK (x,y) – 42K 63.51 83.2 69.4 60.6 73.9 36.3 68.6
(d) FK IN (x,y) – 42K 64.36 83.1 70.4 62.4 75.2 37.1 69.1
(e) FK IN (x,y) (F) f - 42K 64.35 83.1 70.5 62.3 75.4 37.1 69.1
(f) FK IN (x,y) (C) f s 42K 67.17 85.5 71.6 64.6 77.2 39.0 70.8
(g) FK IN (x,y) (C) s s 42K 66.68 84.9 70.1 64.7 76.3 39.2 69.8
(h) FK IN 512 (x,y) – 84K 65.36 84.1 70.4 65.0 76.7 37.2 71.3
(i) FK IN 512 (x,y) (C) f s 84K 68.02 85.9 71.8 67.1 77.1 38.8 72.3
(j) FK IN COL 512 – – 82K 52.18 69.5 52.1 47.5 64.0 24.6 49.8
(k) FK IN 512 COL+ (x,y) – 166K 66.37 82.9 70.1 67.0 77.0 36.1 70.0
(l) FK IN 512 COL+ (x,y) (C) f s 166K 67.93 85.1 70.5 67.5 77.4 35.7 71.2
(m) CNN F – (C) f s 4K 77.38 88.7 83.9 87.0 84.7 46.9 77.5
(n) CNN S – (C) f s 4K 79.74 90.7 85.7 88.9 86.6 50.5 80.1
(o) CNN M – – 4K 76.97 89.5 84.3 88.8 83.2 48.4 77.0
(p) CNN M – (C) f s 4K 79.89 91.7 85.4 89.5 86.6 51.6 79.3
(q) CNN M – (C) f m 4K 79.50 90.9 84.6 89.4 85.8 50.3 78.4
(r) CNN M – (C) s s 4K 79.44 91.4 85.2 89.1 86.1 52.1 78.0
(s) CNN M – (C) t t 41K 78.77 90.7 85.0 89.2 85.8 51.0 77.8
(t) CNN M – (C) f - 4K 77.78 90.5 84.3 88.8 84.5 47.9 78.0
(u) CNN M – (F) f - 4K 76.99 90.1 84.2 89.0 83.5 48.1 77.2
(v) CNN M GS – – 4K 73.59 87.4 80.8 82.4 82.1 44.5 73.5
(w) CNN M GS – (C) f s 4K 77.00 89.4 83.8 85.1 84.4 49.4 77.6
(x) CNN M 2048 – (C) f s 2K 80.10 91.3 85.8 89.9 86.7 52.4 79.7
(y) CNN M 1024 – (C) f s 1K 79.91 91.4 86.9 89.3 85.8 53.3 79.8
(z) CNN M 128 – (C) f s 128 78.60 91.3 83.9 89.2 86.9 52.1 81.0
(a) FK+CNN F (x,y) (C) f s 88K 77.95 89.6 83.1 87.1 84.5 48.0 79.4
(b) FK+CNN M 2048 (x,y) (C) f s 86K 80.14 90.9 85.9 88.8 85.5 52.3 81.4
(g) CNN S TUNE-RNK – (C) f s 4K 82.42 95.3 90.4 92.5 89.6 54.4 81.9
TABLE 2
VOC 2007 results (continued overleaf) . See Sect. 4 for details.
discussed above, by a large 10% mAP margin
([i]vs. [p]). Our small architecture CNN-F, which is
similar to DeCAF [7], performs signiﬁcantly better
than the latter ([II] vs. [s]), validating our imple-
mentation. Both medium CNN-M [m] and slow
CNN-S [p] outperform the fast CNN-F [m] by a
signiﬁcant 23% margin. Since the accuracy
of CNN-S and CNN-M is nearly the same, we
focus on the latter as it is simpler and marginally
(25%) faster. Remarkably, these good networks
work very well even with no augmentation [o].
Another advantage of CNNs compared to IFV is
the small dimensionality of the output features,although IFV can be compressed to an extent. We
explored retraining the CNNs such that the ﬁnal
layer was of a lower dimensionality, and reducing
from 4096 to 2048 actually resulted in a marginal
performance boost ([x] vs. [p]). What is surprising
is that we can reduce the output dimensionality
further to 1024D [y] and even 128D [z] with only
a drop of2%for codes that are 32smaller
(650smaller than our best performing IFV [i]).
Note,`2-normalising the features accounted for up
to5%of their performance over VOC 2007; it
should be applied before input to the SVM and
after pooling the augmented descriptors (where
8
(I) 79.9 61.4 56.0 49.6 58.4 44.8 78.8 70.8 85.0 31.7 51.0 56.4 80.2 57.5
(II) 83.7 83.7 54.3 61.9 70.2 79.5 85.3 77.2 90.9 51.1 73.8 57.0 86.4 68.0
(a) 80.7 64.4 53.8 53.8 60.2 47.8 79.9 68.9 86.1 37.3 51.1 55.8 83.7 56.9
(b) 80.6 64.8 53.9 54.9 60.7 50.5 80.4 69.5 86.2 38.3 54.4 56.3 82.7 56.7
(c) 81.1 64.2 51.1 53.4 61.9 50.0 80.0 67.5 85.3 35.7 51.9 53.8 83.5 58.9
(d) 80.5 66.9 50.9 53.9 62.1 51.5 80.5 68.5 85.9 37.2 55.2 54.3 83.3 59.2
(e) 80.5 66.8 51.0 54.1 62.2 51.5 80.4 68.2 86.0 37.3 55.1 54.2 83.3 59.2
(f) 82.4 71.6 52.8 62.4 63.4 57.1 81.6 70.9 86.9 41.2 61.2 56.9 85.2 61.5
(g) 81.9 71.0 52.8 61.6 62.2 56.8 81.8 70.0 86.5 41.5 61.0 56.5 84.3 60.9
(h) 81.1 67.9 52.6 55.4 61.4 51.2 80.5 69.1 86.4 41.2 56.0 56.2 83.7 59.9
(i) 82.5 73.2 54.7 62.7 64.5 56.6 82.2 71.3 87.5 43.0 62.0 59.3 85.7 62.4
(j) 66.1 46.6 42.5 35.8 41.1 45.5 75.4 58.3 83.9 39.8 47.3 35.6 69.2 49.0
(k) 80.0 65.9 52.8 56.1 61.0 56.9 81.4 69.6 88.4 49.0 59.2 56.4 84.7 62.8
(l) 81.6 70.8 52.9 59.6 63.1 59.9 82.1 70.5 88.9 50.6 63.7 57.5 86.1 64.1
(m) 86.3 85.4 58.6 71.0 72.6 82.0 87.9 80.7 91.8 58.5 77.4 66.3 89.1 71.3
(n) 87.8 88.3 61.3 74.8 74.7 87.2 89.0 83.7 92.3 58.8 80.5 69.4 90.5 74.0
(o) 85.1 87.4 58.1 70.4 73.1 83.5 85.5 80.9 90.8 54.1 78.9 61.1 89.0 70.4
(p) 87.7 88.6 60.3 80.1 74.4 85.9 88.2 84.6 92.1 60.3 80.5 66.2 91.3 73.5
(q) 87.6 88.6 60.7 78.2 73.6 86.0 87.4 83.8 92.3 59.3 81.0 66.8 91.3 74.0
(r) 87.5 88.1 60.4 76.9 74.8 85.8 88.1 84.3 92.2 59.5 79.3 65.8 90.8 73.5
(s) 87.3 87.6 60.1 72.3 75.3 85.2 86.9 82.6 91.9 58.5 77.9 66.5 90.5 73.4
(t) 85.7 87.9 58.3 74.2 73.9 84.7 86.6 82.0 91.0 55.8 79.2 62.1 89.3 71.0
(u) 85.3 87.3 58.1 70.0 73.4 83.5 86.0 80.8 90.9 53.9 78.1 61.2 88.8 70.6
(v) 85.0 84.9 57.8 65.9 69.8 79.5 82.9 77.4 89.2 42.8 71.7 60.2 86.3 67.8
(w) 87.2 86.5 59.5 72.4 74.1 81.7 86.0 82.3 90.8 48.9 73.7 66.8 89.6 71.0
(x) 87.6 88.4 60.2 76.9 75.4 85.5 88.0 83.4 92.1 61.1 83.1 68.5 91.9 74.2
(y) 87.8 88.6 59.0 77.2 73.1 85.9 88.3 83.5 91.8 59.9 81.4 68.3 93.0 74.1
(z) 86.6 87.5 59.1 70.0 72.9 84.6 86.7 83.6 89.4 57.0 81.5 64.8 90.4 73.4
(a) 86.8 85.6 59.9 72.0 73.4 81.4 88.6 80.5 92.1 60.6 77.3 66.4 89.3 73.3
(b) 87.7 88.4 61.2 76.9 76.6 84.9 89.1 82.9 92.4 61.9 80.9 68.7 91.5 75.1
(g) 91.5 91.9 64.1 76.3 74.9 89.7 92.2 86.9 95.2 60.7 82.9 68.0 95.5 74.4
TABLE 2
VOC 2007 results (continued from previous page)
applicable).
4.5 Scenario 3: Deep representation (CNN) with
pre-training and ﬁne-tuning
We ﬁne-tuned our CNN-S architecture on VOC-
2007 using the ranking hinge loss, and achieved
a signiﬁcant improvement: 2:7%([g]vs. [n]). This
demonstrates that in spite of the small amount
of VOC training data (5,011 images), ﬁne-tuning
is able to adjust the learnt deep representation to
better suit the dataset in question.4.6 Combinations
For the CNN-M 2048 representation [x], stacking
deep and shallow representations to form a higher-
dimensional descriptor makes little difference ([x]
vs. [b]). For the weaker CNN-F it results in a small
boost of0:8%([m] vs. [a]).
4.7 Comparison with the state of the art
In Table 3 we report our results on ILSVRC-2012,
VOC-2007, VOC-2012, Caltech-101, and Caltech-256
datasets, and compare them to the state of the art.
First, we note that the ILSVRC error rates of our
CNN-F, CNN-M, and CNN-S networks are better
9
ILSVRC-2012 VOC-2007 VOC-2012 Caltech-101 Caltech-256
(top-5 error) (mAP) (mAP) (accuracy) (accuracy)
(a) FK IN 512 - 68.0 – – –
(b) CNN F 16.7 77.4 79.9 – –
(c) CNN M 13.7 79.9 82.5 87.15 0.80 77.030.46
(d) CNN M 2048 13.5 80.1 82.4 86.64 0.53 76.880.35
(e) CNN S 13.1 79.7 82.9 87.76 0.66 77.610.12
(f) CNN S TUNE-CLS 13.1 - 83.0 88.350.56 77.330.56
(g) CNN S TUNE-RNK 13.1 82.4 83.2 – –
(h) Zeiler & Fergus [19] 16.1 - 79.0 86.5 0.5 74.20.3
(i) Razavian et al. [9], [10] 14.7 77.2 – – –
(j) Oquab et al. [8] 18 77.7 78.7 (82.8*) – –
(k) Oquab et al. [16] - - 86.3*– –
(l) Wei et al. [17] - 81.5 ( 85.2*) 81.7 ( 90.3*) – –
(m) He et al. [29] 13.6 80.1 - 91.40.7 –
TABLE 3
Comparison with the state of the art on ILSVRC2012, VOC2007, VOC2012, Caltech-101, and
Caltech-256. Results marked with * were achieved using models pre-trained on the extended ILSVRC
datasets (1512 classes in [8], [16], 2000 classes in [17]). All other results were achieved using CNNs
pre-trained on ILSVRC-2012 (1000 classes).
than those reported by [13], [19], and [10] for the
related conﬁgurations. This validates our imple-
mentation, and the difference is likely to be due to
the sampling of image crops from the uncropped
image plane (instead of the centre). When using
our CNN features on other datasets, the relative
performance generally follows the same pattern
as on ILSVRC, where the nets are trained – the
CNN-F architecture exhibits the worst performance,
with CNN-M and CNN-S performing considerably
better.
Further ﬁne-tuning of CNN-S on the VOC
datasets turns out to be beneﬁcial; on VOC-2012,
using the ranking loss is marginally better than
the classiﬁcation loss ([g] vs. [f]), which can be
explained by the ranking-based VOC evaluation
criterion. Fine-tuning on Caltech-101 also yields a
small improvement, but no gain is observed over
Caltech-256.
Our CNN-S net is competitive with recent CNN-
based approaches [19], [9], [8], [16], [17], [29] and
on a number of datasets (VOC-2007, VOC-2012,
Caltech-101, Caltech-256) and sets the state of the
art on VOC-2007 and VOC-2012 across methods
pre-trained solely on ILSVRC-2012 dataset. While
the CNN-based methods of [16], [17] achieve better
performance on VOC (86.3% and 90.3% respec-
tively), they were trained using extended ILSVRCdatasets, enriched with additional categories se-
mantically close to the ones in VOC. Addition-
ally, [17] used a signiﬁcantly more complex clas-
siﬁcation pipeline, driven by bounding box pro-
posals [30], pre-trained on ILSVRC-2013 detection
dataset. Their best reported result on VOC-2012
(90.3%) was achieved by the late fusion with a
complex hand-crafted method of [31]; without fu-
sion, they get 84.2%. On Caltech-101, [29] achieves
the state of the art using spatial pyramid pooling
of conv5 layer features, while we used full7 layer
features consistently across all datasets (for full7
features, they report 87:08%).
In addition to achieving performance comparable
to the state of the art with a very simple approach
(but powerful CNN-based features), with the mod-
iﬁcations outlined in the paper (primarily the use
of data augmentation similar to the CNN-based
methods) we are able to improve the performance
of shallow IFV to 68.02% (Table 2, [i]).
4.8 Performance Evolution on VOC-2007
A comparative plot of the evolution in the per-
formance of the methods evaluated in this paper,
along with a selection from our earlier review of
shallow methods [12] is presented in Fig. 1. Clas-
siﬁcation accuracy over PASCAL VOC was 54.48%
mAP for the BoVW model in 2008, 61.7% for the
10
      BOW
32K
–FK-BL
327K
–
[I]FK
327K
–
[a]FK-IN
84K
f s
[i]DeCAF
4K
t t
[II]CNN-F
4K
f s
[m]CNN-M 2K
2K
f s
[y]CNN-S
4K (TN)
f s
[γ]

545658606264666870727476788082mAP 68.02
54.4861.6963.6673.4177.1580.132008 2010 2013 2014 ...
82.42
Method
Dim.
Aug.
Ref.
Fig. 1. Evolution of Performance on PASCAL VOC-2007 over the recent years. Please refer to Table 2
for details and references.
IFV in 2010 [12], and 73.41% for DeCAF [7] and
similar [8], [9] CNN-based methods introduced in
late 2013. Our best performing CNN-based method
(CNN-S with ﬁne-tuning) achieves 82.42%, compa-
rable to the most recent state-of-the-art.
4.9 Timings and dimensionality
One of our best-performing CNN representations
CNN-M-2048 [x] is 42more compact than the
best performing IFV [i] (84K vs. 2K) and CNN-M
features are also 50faster to compute ( 120s
vs.2:4sper image with augmentation enabled,
over a single CPU core). Non-augmented CNN-M
features [o] take around 0:3sper image, compared
to0:4sfor CNN-S features and 0:13sfor CNN-
F features.
5 C ONCLUSION
In this paper we presented a rigorous empiri-
cal evaluation of CNN-based methods for image
classiﬁcation, along with a comparison with more
traditional shallow feature encoding methods. We
have demonstrated that the performance of shal-
low representations can be signiﬁcantly improved
by adopting data augmentation, typically used indeep learning. In spite of this improvement, deep
architectures still outperform the shallow methods
by a large margin. We have shown that the per-
formance of deep representations on the ILSVRC
dataset is a good indicator of their performance
on other datasets, and that ﬁne-tuning can further
improve on already very strong results achieved
using the combination of deep representations and
a linear SVM. Source code and CNN models to
reproduce the experiments presented in the paper
are available on the project website [32] in the
hope that it would provide common ground for
future comparisons, and good baselines for image
representation research.
ACKNOWLEDGEMENTS
This work was supported by the EPSRC and ERC
grant VisRec no. 228180. We gratefully acknowl-
edge the support of NVIDIA Corporation with the
donation of the GPUs used for this research.
REFERENCES
[1] G. Csurka, C. Bray, C. Dance, and L. Fan, “Visual catego-
rization with bags of keypoints,” in Workshop on Statistical
Learning in Computer Vision, ECCV , 2004, pp. 1–22.
11
[2] J. Sivic and A. Zisserman, “Video Google: A text retrieval
approach to object matching in videos,” in Proc. ICCV ,
vol. 2, 2003, pp. 1470–1477.
[3] F. Perronnin, J. S ´anchez, and T. Mensink, “Improving the
Fisher kernel for large-scale image classiﬁcation,” in Proc.
ECCV , 2010.
[4] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel, “Backpropaga-
tion applied to handwritten zip code recognition,” Neural
Computation , vol. 1, no. 4, pp. 541–551, 1989.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei,
“Imagenet: A large-scale hierarchical image database,” in
Proc. CVPR , 2009.
[6] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman, “The PASCAL Visual Object Classes
(VOC) challenge,” IJCV , vol. 88, no. 2, pp. 303–338, 2010.
[7] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell, “Decaf: A deep convolutional
activation feature for generic visual recognition,” CoRR ,
vol. abs/1310.1531, 2013.
[8] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning
and Transferring Mid-Level Image Representations using
Convolutional Neural Networks,” in Proc. CVPR , 2014.
[9] A. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson,
“CNN Features off-the-shelf: an Astounding Baseline for
Recognition,” CoRR , vol. abs/1403.6382, 2014.
[10] P . Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y. LeCun, “OverFeat: Integrated Recognition, Local-
ization and Detection using Convolutional Networks,” in
Proc. ICLR , 2014.
[11] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich
feature hierarchies for accurate object detection and se-
mantic segmentation,” in Proc. CVPR , 2014.
[12] K. Chatﬁeld, V . Lempitsky, A. Vedaldi, and A. Zisserman,
“The devil is in the details: an evaluation of recent feature
encoding methods,” in Proc. BMVC. , 2011.
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet
classiﬁcation with deep convolutional neural networks,”
inNIPS , 2012, pp. 1106–1114.
[14] M. Paulin, J. Revaud, Z. Harchaoui, F. Perronnin, and
C. Schmid, “Transformation Pursuit for Image Classiﬁca-
tion,” in Proc. CVPR , 2014.
[15] F. Perronnin, Z. Akata, Z. Harchaoui, and C. Schmid,
“Towards good practice in large-scale learning for image
classiﬁcation,” in Proc. CVPR , 2012, pp. 3482–3489.
[16] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Weakly
supervised object recognition with convolutional neural
networks,” INRIA, Tech. Rep. HAL-01015140, 2014.
[17] Y. Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y. Zhao, and
S. Yan, “CNN: Single-label to multi-label,” CoRR , vol.
abs/1406.5726, 2014.
[18] H. J ´egou, F. Perronnin, M. Douze, J. S ´anchez, P . P ´erez, and
C. Schmid, “Aggregating local images descriptors into
compact codes,” IEEE P AMI , 2012.
[19] M. D. Zeiler and R. Fergus, “Visualizing and understand-
ing convolutional networks,” CoRR , vol. abs/1311.2901,
2013.
[20] Y. Jia, “Caffe: An open source convolutional architecture
for fast feature embedding,” http://caffe.berkeleyvision.
org/, 2013.
[21] L. Fei-Fei, R. Fergus, and P . Perona, “Learning generative
visual models from few training examples: An incremen-
tal bayesian approach tested on 101 object categories,”
inIEEE CVPR Workshop of Generative Model Based Vision ,
2004.[22] G. Grifﬁn, A. Holub, and P . Perona, “Caltech-256 object
category dataset,” California Institute of Technology,
Tech. Rep. 7694, 2007. [Online]. Available: http://
authors.library.caltech.edu/7694
[23] J. S ´anchez, F. Perronnin, and T. Em ´ıdio de Campos,
“Modeling the spatial layout of images beyond spatial
pyramids,” Pattern Recognition Letters , vol. 33, no. 16, pp.
2216–2223, 2012.
[24] R. Arandjelovi ´c and A. Zisserman, “Three things every-
one should know to improve object retrieval,” in Proc.
CVPR , 2012.
[25] A. Vedaldi and A. Zisserman, “Efﬁcient additive kernels
via explicit feature maps,” IEEE P AMI , 2011.
[26] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond Bags
of Features: Spatial Pyramid Matching for Recognizing
Natural Scene Categories,” in Proc. CVPR , 2006.
[27] R. Arandjelovi ´c and A. Zisserman, “All about VLAD,” in
Proc. CVPR , 2013.
[28] H. J ´egou, M. Douze, and C. Schmid, “On the burstiness
of visual elements,” in Proc. CVPR , Jun 2009.
[29] K. He, A. Zhang, S. Ren, and J. Sun, “Spatial pyramid
pooling in deep convolutional networks for visual recog-
nition,” in Proc. ECCV , 2014.
[30] M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P . H. S. Torr,
“BING: Binarized normed gradients for objectness esti-
mation at 300fps,” in Proc. CVPR , 2014.
[31] S. Yan, J. Dong, Q. Chen, Z. Song, Y. Pan, W. Xia,
H. Zhongyang, Y. Hua, and S. Shen, “Generalized hi-
erarchical matching for subcategory aware object classi-
ﬁcation,” in The P ASCAL Visual Object Classes Challenge
Workshop , 2012.
[32] K. Chatﬁeld, K. Simonyan, A. Vedaldi, and A. Zisserman,
“Return of the devil in the details: delving deep into
convolutional nets webpage,” 2014. [Online]. Available:
http://www.robots.ox.ac.uk/vgg/research/deep eval