2013 IEEE Conference on Computer Vision and Pattern Recognition

MODEC: Multimodal Decomposable Models for Human Pose Estimation
Ben Sapp
Google, Inc

Ben Taskar
University of Washingtonbensapp@google.com

taskar@cs.washington.edu

Abstract

foreshortening, self-occlusion and physically different body
contours).

We propose a multimodal, decomposable model for articulated human pose estimation in monocular images. A
typical approach to this problem is to use a linear structured model, which struggles to capture the wide range ofappearance present in realistic, unconstrained images. In
this paper, we instead propose a model of human pose that
explicitly captures a variety of pose modes. Unlike othermultimodal models, our approach includes both global andlocal pose cues and uses a convex objective and joint training for mode selection and pose estimation. We also employ
a cascaded mode selection step which controls the trade-off
between speed and accuracy, yielding a 5x speedup in inference and learning. Our model outperforms state-of-theart approaches across the accuracy-speed trade-off curvefor several pose datasets. This includes our newly-collected
dataset of people in movies, FLIC, which contains an order of magnitude more labeled data for training and testing
than existing datasets. The new dataset and code are available online. 1

Most models developed to estimate human pose in thesevaried settings extend the basic linear pictorial structuresmodel (PS) [9, 14, 4, 1, 19, 15]. In such models, part detectors are learned invariant to pose and appearance—e.g., one
forearm detector for all body types, clothing types, poses
and foreshortenings. The focus has instead been directed
towards improving features in hopes to better discriminate
correct poses from incorrect ones. However, this comes ata price of considerable feature computation cost—[15], forexample, requires computation of Pb contour detection andNormalized Cuts each of which takes minutes.
Recently there has been an explosion of successful work
focused on increasing the number of modes in human posemodels. The models in this line of work in general can be
described as instantiations of a family of compositional, hierarchical pose models. Part modes at any level of granularity can capture different poses (e.g., elbow crooked, lower
arm sideways) and appearance (e.g., thin arm, baggy pants).
Also of crucial importance are details such as how models
are trained, the computational demands of inference, andhow modes are deﬁned or discovered. Importantly, increasing the number of modes leads to a computational complexity at least linear and at worst exponential in the numberof modes and parts. A key omission in recent multimodal
models is efﬁcient and joint inference and training.

1. Introduction
Human pose estimation from 2D images holds great potential to assist in a wide range of applications—for example, semantic indexing of images and videos, action recognition, activity analysis, and human computer interaction.
However, human pose estimation “in the wild” is an extremely challenging problem. It shares all of the difﬁculties
of object detection, such as confounding background clutter, lighting, viewpoint, and scale, in addition to signiﬁcantdifﬁculties unique to human poses.
In this work, we focus explicitly on the multimodal nature of the 2D pose estimation problem. There are enormous appearance variations in images of humans, due toforeground and background color, texture, viewpoint, andbody pose. The shape of body parts is further varied byclothing, relative scale variations, and articulation (causing

In this paper, we present MODEC, a multimodal decomposable model with a focus on simplicity, speed and accuracy. We capture multimodality at the large granularityof half- and full-bodies as shown in Figure 1. We deﬁnemodes via clustering human body joint conﬁgurations ina normalized image-coordinate space, but mode deﬁnitions
could easily be extended to be a function of image appearance as well. Each mode is corresponds to a discriminative
structured linear model. Thanks to the rich, multimodal nature of the model, we see performance improvements evenwith only computationally-cheap image gradient features.
As a testament to the richness of our set of modes, learning a ﬂat SVM classiﬁer on HOG features and predicting
the mean pose of the predicted mode at test time performs

1 This research was conducted at the University of Pennsylvania.

1063-6919/13 $26.00 © 2013 IEEE
DOI 10.1109/CVPR.2013.471

3672
3674


 

s(x, z)

 
 

      







       






 
 


 


 
sc (x, yc , zc ) 

c 





 

   

Figure 1. Left: Most pictorial structures researchers have put effort into better and larger feature spaces, in which they ﬁt one linear
model. The feature computation is expensive, and still fails at capturing the many appearance modes in real data. Right: We take adifferent approach. Rather than introduce an increasing number of features in hopes of high-dimensional linear separability, we model thenon-linearities in simpler, lower dimensional feature spaces, using a collection of locally linear models.

competitively to state-of-the-art methods on public datasets.
Our MODEC model features explicit mode selection
variables which are jointly inferred along with the best layout of body parts in the image. Unlike some previous work,
our method is also trained jointly (thus avoiding difﬁculties calibrating different submodel outputs) and includes
both large-scope and local part-level cues (thus allowing
it to effectively predict which mode to use). Finally, we
employ an initial structured cascade mode selection stepwhich cheaply discards unlikely modes up front, yielding
a 5× speedup in inference and learning over considering allmodes for every example. This makes our model slightly
faster than state-of-the-art approaches (on average, 1.31
seconds vs. 1.76 seconds for [24]), while being signiﬁcantly
more accurate. It also suggests a way to scale up to evenmore modes as larger datasets become available.

Model
Basic PS
Wang & Mori [20]Johns. & Evering. [11]Wang et al. [21]Zhu & Ramanan [25]Duan et al. [3]Tian et al. [18]Sun & Savarese [17]Yang & Ramanan [24]
MODEC (ours)# partlevels
1
1
1
4
1
4
3
4
1
3

# global
modes
1
3
16
1
18
9
5
1
1
32

# partmodes
1
1
4*
5 to 20
1
4 to 65 to 15
44 to 6
1

training
obj.
n/a
greedyindep.
approx.
indep.†
approx.
indep.
joint
joint
joint

* Part modes are not explicitly part of the state, but instead are maxed over
to form a single detection.
† A variety of parameter sharing across global models is explored, thus
there is some cross-mode sharing and learning.

Table 1. In the past few years, there have been many instantiationsof the family of multimodal models. The models listed here andtheir attributes are described in the text.

2. Related work

As can be seen in Table 1, compositional, hierarchicalmodeling of pose has enjoyed a lot of attention recently.
In general, works either consider only global modes, local
modes, or several multimodal levels of parts.
Global modes but only local cues. Some approaches use
tens of disjoint pose-mode models [11, 25, 20], which they
enumerate at test time and take the highest scoring as a predictor. One characteristic of all such models in this category is that they only employ local part cues (e.g. wrist
patch or lower arm patch), making it difﬁcult to adequately
represent and predict the best global mode. A second issue is that some sort of model calibration is required, sothat scores of the different models are comparable. Methods such as [11, 13] calibrate the models post-hoc using
cross-validation data.
Local modes. A second approach is to focus on modelingmodes only at the part level, e.g. [24]. If n parts each use
k modes, this effectively gives up to k n different instantiations of modes for the complete model through mixing-

and-matching part modes. Although combinatorially rich,
this approach lacks the ability to reason about pose structure larger than a pair of parts at a time. This is due to thelack of global image cues and the inability of the representation to reason about larger structures. A second issue is thatinference must consider a quadratic number of local mode
combinations—e.g. for each of k wrist types, k elbow types
must be considered, resulting in inference message passingthat is k 2 larger than unimodal inference.
Additional part levels. A third category of models consider both global, local and intermediate part-granularity
level modes [21, 17, 3, 18]. All levels use image cues, allowing models to effectively represent mode appearance at
different granularities. The biggest downside to these richer
models are their computational demands: First, quadraticmode inference is necessary, as with any local mode modeling. Second, inference grows linearly with the numberof additional parts, and becomes intractable when part rela-

3675
3673

   

δ(z) = max
y

c∈C

y4

z5

zc 1

z3
z1

z6



z = arg max s(x, z) + δ(z)y  s(x, y, z) = s(x, z) +

y5

zc 3

z4
z2

c2

zc 2

s(x, z)a sum of two terms:

sc2 (x, yc2 , zc2 )sc (x, yc , zc )sc3 (x, yc3 , zc3 )c1

sc (x, yc , zc ).

(1)

c∈C

c3

y2

y1



y3

y6

y7

y8

y9

This scores a choice of output variables y and mode variables z in example x. The yc denote subsets of y that are indexed by a single clique of z, denoted zc . Each sc (x, yc , zc )can be thought of as a typical unimodal model over yc ,one model for each value of zc . Hence, we refer to theseterms as mode-speciﬁc submodels. The beneﬁts of such amodel over a non-multimodal one s(x, y) is that different
modeling behaviors can be captured by the different mode
submodels. This introduces beneﬁcial ﬂexibility, especiallywhen the underlying submodels are linear and the problem
is inherently multimodal. The ﬁrst term in Equation 1 cancapture structured relationships between the mode variables
and the observed data. We refer to this term as the mode
scoring term.
Given such a scoring function, the goal is to determine
the highest scoring value to output variables y and mode
variables z given a test example x:

sc1 (x, yc1 , zc1 )y  = arg max s(x, y, z  )yFigure 2. A general example of a MODEC model in factor graphform. Cliques zc in s(x, z) are associated with groups yc . Eachsubmodel sc (x, yc , zc ) can be a typical graphical model over yc
for a ﬁxed instantiation of zc .

tions are cyclic, as in [21, 3].
Contrast with our model. In contrast to the above, ourmodel supports multimodal reasoning at the global level, asin [11, 25, 20]. Unlike those, we explicitly reason about,represent cues for, and jointly learn to predict the correct
global mode as well as location of parts. Unlike local mode
models such as [24], we do not require quadratic part-mode
inference and can reason about larger structures. Finally,
unlike models with additional part levels, MODEC supports
efﬁcient, tractable, exact inference, detailed in Section 3.
Furthermore, we can learn and apply a mode ﬁltering step toreduce the number of modes considered for each test image,
speeding up learning and inference by a factor of 5.
Other local modeling methods: In the machine learning
literature, there is a vast array of multimodal methods forprediction. Unlike ours, some require parameter estimation
at test time, e.g. local regression. Locally-learned distancefunction methods are more similar in spirit to our work:[10, 13] both learn distance functions per example. [12]proposes learning a blend of linear classiﬁers at each exemplar.

z  , y  = arg max s(x, y, z)
z,y

(2)

In order to make MODEC inference tractable, we need a
few assumptions
 about our model: (1) It is efﬁcient to compute maxy c∈C sc (x, yc , zc ). This is the case in common
structured scoring functions in which variable interactions
form a tree, notably pictorial structures models for humanparsing, and star or tree models for object detection, e.g.
[8]. (2) It is efﬁcient to compute maxz s(x, z), the ﬁrst
term of Equation 1. This is possible when the network ofinteractions in s(x, z) has low treewidth. (3) There is a oneto-many relationship from cliques zc to each variable in y:zc can be used to index multiple yi in different subsets yc ,but each yi can only participate in factors with one zc . This
ensures during inference that the messages passed from thesubmodel terms to the mode-scoring term will maintain thedecomposable structure of s(x, z). A general, small example of a MODEC model can be seen in Figure 2.
When these conditions hold, we can solve Equation 2
efﬁciently, and even in parallel over possibilities of zc , although the overall graph structure may be cyclic (as in
Figure1). The full inference involves computing δ(z) =maxy c∈C sc (x, yc , zc ) independently (i.e., in parallel)
for each zc possibility, then computing the optimal z  =arg maxz s(x, z) + δ(z), then backtracking to retrieve themaximizing y  .

3. MODEC: Multimodal decomposable model
We ﬁrst describe our general multimodal decomposable(MODEC) structured model, and then an effective special
case for 2D human pose estimation. We consider problems modeling both input data x (e.g., image pixels), output variables y = [y1 , . . . , yP ] (e.g., the placement of P
body parts in image coordinates), and special mode variables z = [z1 , . . . , zK ], zi ∈ [1, M ] which capture different
modes of the input and output (e.g., z corresponds to human joint conﬁgurations which might semantically be interpreted as modes such as arm-folded, arm-raised, arm-down
as in Figure 1). The general MODEC model is expressed as

3.1. MODEC model for human pose estimation
We tailor MODEC for human pose estimation as follows(model structure is shown in Figure 1). We employ twomode variables, one for the left side of the body, one for

3676
3674



 

   
$""

)

%!$

$$ ""
$%

z θ#('

#%!

#&!

#$&

Figure 3. An illustration of the inference process. For simplicity, only a left-sided model is shown. First, modes are cheaply ﬁltered via acascaded prediction step. Then each remaining local submodel can be run in parallel on a test image, and the argmax prediction is taken as
a guess. Thanks to joint inference and training objectives, all submodels are well calibrated with each other.

the right: z and zr . Each takes on one of M = 32 possible modes, which are deﬁned in a data-driven way aroundclusters of human joint conﬁgurations (see Section 4). The
left and right side models are standard linear pairwise CRFs
indexed by mode:term s(x, z):s(x, z) = w,r · f (z , zr ) + w · f (x, z ) + wr · f (x, zr ) (4)The ﬁrst term represents a (z , zr ) mode compatibility
score that encodes how likely each of the M modes on one
side of the body are to co-occur with each of the M modeson the other side—expressing an afﬁnity for common posessuch as arms folded, arms down together, and dislike of uncommon left-right pose combinations. The other two terms
can be viewed as mode classiﬁers: each attempts to predictthe correct left/right mode based on image features.
Putting together Equation 3 and Equation 4, the full
MODEC model iss (x, y , z ) = zzwi  · fi (x, yi , z ) +wij· fij (yi , yj , z ). (3)i∈V(i,j)∈E

The right side model sr (·, zr ) is analogous. Here each variable yi denotes the pixel coordinates (row, column) of part iin image x. For “parts” we choose to model joints and their
midpoints (e.g., left wrist, left forearm, left elbow) whichallows us ﬁne-grained encoding of foreshortening and rotation, as is done in [24, 16]. The ﬁrst terms in Equation 3
depend only on each part and the mode, and can be viewed
as mode-speciﬁc part detectors—a separate set of parameters wiz is learned for each mode for each part. The second terms measure geometric compatibility between a pairof parts connected in the graph. Again, this is indexed by
the mode and is thus mode-speciﬁc, imposed because different pose modes have different geometric characteristics.
Details of the features are in Section 5. The graph overvariable interactions (V , E ) forms a tree, making exact inference possible via max-sum message passing.
We employ the following form for our mode scorings(x, y, z) = s (x, y , z ) + sr (x, yr , zr )+ w,r · f (z , zr ) + w · f (x, z ) + wr · f (x, zr )(5)

The inference procedure is linear in M . In the next
section we show a speedup using cascaded prediction toachieve inference sublinear in M .

3.2. Cascaded mode ﬁltering
The use of structured prediction cascades has been a successful tool for drastically reducing state spaces in structured problems [15, 23]. Here we employ a simple multiclass cascade step to reduce the number of modes considered in MODEC. Quickly rejecting modes has very appeal-

3677
3675

ing properties: (1) it gives us an easy way to tradeoff accuracy versus speed, allowing us to achieve very fast stateof-the-art parsing. (2) It also makes training much cheaper,
allowing us to develop and cross-validate our joint learning
objective (Equation 10) effectively.
We use an unstructured cascade model where we ﬁlter
each mode variable z and zr independently. We employ a
linear cascade model of the form

Learning formulation. We seek to learn to correctly identify the correct mode and location of parts in each example.
Intuitively, for each example this gives us hard constraintsof the form

κ(x, z) = θz · φ(x, z)

In words, Equation 8 states that the score of the true joint
conﬁguration for submodel z t must be higher than z t ’sscore for any other (wrong) joint conﬁguration in example
t—the standard max-margin parsing constraint for a single
structured model. Equation 9 states that the score of the true
conﬁguration for z t must also be higher than all scores anincorrect submodel z  has on example t.
We consider all constraints from Equation 8 and Equation 9, and add slack variables to handle non-separability.
Combined with regularization on the parameters, we get aconvex, large-margin structured learning objective jointly
over all M local modelss(xt , y t , z t ) − s(xt , y  , z t ) ≥ 1, ∀y  = y ttz∈[1,M ]

1−α
M

(6)



κ(x, z)}z∈[1,M ]The metaparameter α ∈ [0, 1) is set via cross-validationand dictates how aggressively to prune—between pruning
everything but the max-scoring mode to pruning everything
below the mean score. For full details of structured prediction cascades, see [22].
Applying this cascade before running MODEC resultsin the inference task z  , y  = arg maxz∈M̄ , y∈Y s(x, y, z)
where |M̄ | is considerably smaller than M . In practice it ison average 5 times smaller at no appreciable loss in accuracy, giving us a 5× speedup.

S||y t − μi ||2





t

(8)
(9)

(10)

s(xt , y t , z t ) − s(xt , y  , z t ) ≥ 1 − ξt∀y  = y ts(xt , y t , z t ) − s(xt , y, z  ) ≥ 1 − ξt∀z  = z t ∈ M̄ t , ∀y{w },{ξt }subject to:Note the use of M̄ t , the subset of modes unﬁltered by ourmode prediction cascade for each example. This is considerably faster than considering all M modes in each training
example.
The number of constraints listed here is prohibitively
large: in even a single image, the number of possible outputs is exponential in the number of parts. We use a cutting
plane technique where we ﬁnd the most violated constraint
in every training example via structured inference (which
can be done in one parallel step over all training examples). We then solve Equation 10 under the active set of constraints using the fast off-the-shelf QP solver liblinear [7].
Finally, we share all parameters between the left and rightside, and at test time simply ﬂip the image horizontally tocompute local part and mode scores for the other side.

During training, we have access to a training set of images with labeled poses D = {(xt , y t )}Tt=1 . From this, weﬁrst derive mode labels z t and then learn parameters of ourmodel s(x, y, z).
Mode deﬁnitions. Modes are obtained from the data by
ﬁnding centers {μi }Mi=1 and example-mode membershipsets S = {Si }M
i=1 in pose space that minimize reconstruction error under squared Euclidean distance:
M 


t

M
T
1
C
||wz ||22 +
ξt
2 z=1
T t=1

min
z

4. Learning

S  = arg mints(x , y , z ) − s(x , y, z ) ≥ 1, ∀z = z , ∀ywhose purpose is to score the mode z in image x, in order toﬁlter unlikely mode candidates. The features of the modelare φ(x, z) which capture the pose mode as a whole insteadof individual local parts, and the parameters of the model
are a linear set of weights for each mode, θz . Following thecascade framework, we retain a set of mode possibilitiesM̄ ⊆ [1, M ] after applying the cascade model:M̄ = {z | κ(x, z) ≥ α max κ(x, z) +t

(7)

i=1 t∈Si

where μi is the Euclidean mean joint locations of the examples in mode cluster Si . We approximately minimize
this objective via k-means with 100 random restarts. We
take the cluster membership as our supervised deﬁnition ofmode membership in each training example, so that we augment the training set to be D = {(xt , y t , z t )}.
The mode memberships are shown as average images in
Figure 1. Note that some of the modes are extremely difﬁcult to describe at a local part level, such as arms severely
foreshortened or crossed.

5. Features
Due to the ﬂexibility of MODEC, we can get rich modeling power even from simple features and linear scoringterms.
Appearance. We employ only histogram of gradients(HOG) descriptors, using the implementation from [8]. For
local part cues fi (x, yi , z) we use a 5 × 5 grid of HOG cells,with a cell size of 8×8 pixels. For left/right-side mode cues

3678
3676

30 movies. People detected with high conﬁdence (roughly
20K candidates) were then sent to the crowdsourcing marketplace Amazon Mechanical Turk to obtain groundtruth labeling. Each image was annotated by ﬁve Turkers for $0.01
each to label 10 upperbody joints. The median-of-ﬁve labeling was taken in each image to be robust to outlier annotation. Finally, images were rejected manually by us if theperson was occluded or severely non-frontal. We set aside
20% (1016 images) of the data for testing.

f (x, z) we capture larger structure with a 9 × 9 grid and acell size of 16 × 16 pixels. The cascade mode predictor
uses the same features for φ(x, z) but with an aspect ratio
dictated by the extent of detected upper bodies: a 17 × 15
grid of 8 × 8 cells. The linearity of our model allows us toevaluate all appearance terms densely in an efﬁcient manner
via convolution.
Pairwise part geometry. We use quadratic deformation
cost features similar to those in [9], allowing us to use distance transforms for message passing:fij (yi , yj , z) = [(yi (r) − yj (r) − μzij (r))2 ;(yi (c) − yj (c) − μzij (c))2 ]6.2. Evaluation measure(11)

There has been discrepancy regarding the widely reported Percentage of Correct Parts (PCP) test evaluation
measure; see [5] for details. We use a measure of accuracy that looks at a whole range of matching criteria, similar to [24]: for any particular joint localization precision
radius (measured in Euclidean pixel distance scaled so thatthe groundtruth torso is 100 pixels tall), we report the percentage of joints in the test set correct within the radius. Fora test set of size N , radius r and particular joint i this is:where (yi (r), yi (c)) denote the pixel row and column represented by state yi , and μzij is the mean displacement between parts i and j in mode z, estimated on training data.
In order to make the deformation cue a convex, unimodal
penalty (and thus computable with distance transforms),
we need to ensure that the corresponding parameters onz
are positive. We enforce this by adding
these features wij
additional positivity constraints in our learning objective:z≥ , for small  strictly positive2 .
wij

100 
acci (r) =
1
N t=1
N



100 · ||yit − yit ||2≤rtt||ylhip− yrsho||2where yit is our model’s predicted ith joint location on
test example t. We report acci (r) for a range of r resulting
in a curve that spans both the very precise and very loose
regimes of part localization.
We compare against several state-of-the-art models
which provide publicly available code. The model of
Yang & Ramanan [24] is multimodal at the level of local
parts, and has no larger mode structure. We retrained their
method on our larger FLIC training set which improved
their model’s performance across all three datasets. The
model of Eichner et al. [4] is a basic unimodal PS model
which iteratively reparses using color information. It has no
training protocol, and was not retrained on FLIC. Finally,
Sapp et al.’s CPS model [15] is also unimodal but terms are
non-linear functions of a powerful set of features, some ofwhich requiring signiﬁcant computation time (Ncuts, gPb,
color). This method is too costly (in terms of both memory
and time) to retrain on the 10× larger FLIC training dataset.

6. Experiments
We report results on standard upper body datasets Buffy
and Pascal Stickmen [4], as well as a new dataset FLIC
which is an order of magnitude larger, which we collected
ourselves. Our code and the FLIC dataset are available at
http://www.vision.grasp.upenn.edu/video.

6.1. Frames Labeled in Cinema (FLIC) dataset
Large datasets are crucial when we want to learn richmodels of realistic pose3 . The Buffy and Pascal Stickmen datasets contain only hundreds of examples for training pose estimation models. Other datasets exist with a
few thousand images, but are lacking in certain ways. The
H3D [2] and PASCAL VOC [6] datasets have thousands ofimages of people, but most are of insufﬁcient resolution,
signiﬁcantly non-frontal or occluded. The UIUC Sports
dataset [21] has 1299 images but consists of a skewed distribution of canonical sports poses, e.g. croquet, bike riding,badminton.
Due to these shortcomings, we collected a 5003 image dataset automatically from popular Hollywood movies,
which we dub FLIC. The images were obtained by runninga state-of-the-art person detector [2] on every tenth frame of

7. Results
The performance of all models are shown on FLIC,
Buffy and Pascal datasets in Figure 4. MODEC outperforms the rest across the three datasets. We ascribe its success over [24] to (1) the ﬂexibility of 32 global modes (2)large-granularity mode appearance terms and (3) the ability
to train all mode models jointly. [5] and [15] are uniformly
worse than the other models, most likely due to the lack of
discriminative training and/or unimodal modeling.
We also compare to two simple prior pose baselines that
perform surprisingly well. The “mean pose” baseline sim-

2 It may still be the case that the constraints are not respected (dueto slack variables), but this is rare. In the unlikely event that this ocz ←curs, we project the deformation parameters onto the feasible set: wijzmax(, wij ).
3 Increasing training set size from 500 to 4000 examples improves test
accuracy from 32% to 42% wrist and elbow localization accuracy.

3679
3677

 

$&"

elbowswrists5% 

100

elbows "
wristselbowswrists

100

100

100

100

100

80

80

80

80

80

80

60

60

60

60

60

60

40

40

40

40

40

40

20

20

20

20

20

20

5

10

15

20

5

10

15

20

5

10

15

20

5

10

15

20

% +,6 +/(1 ),!(0./.

* +2 ),- 0.//

5

10

15

!(0./.

20

5

10

15

20

" Figure 4. Test results. We show results for the most challenging parts: elbows and wrists. See text for discussion. Best viewed in color.

ply guesses the average training pose, and can be thoughtof as measuring how unvaried a dataset is. The “mean cluster prediction” involves predicting the mean pose deﬁned
by the most likely pose, where the most likely pose is determined directly from a 32-way SVM classiﬁer using thesame HOG features as our complete model. This baseline
actually outperforms or is close to CPS and [5] on the threedatasets, at very low computational cost—0.0145 seconds
per image. This surprising result indicates the importanceof multimodal modeling in even the simplest form.
Speed vs. Accuracy. In Figure 5 we examine the tradeoff
between speed and accuracy. On the left, we compare different methods with a log time scale. The upper left corneris the most desirable operating point. MODEC is strictly
better than other methods in the speed-accuracy space. On
the right, we zoom in to investigate our cascaded MODEC
approach. By tuning the aggressiveness (α) of the cascade,we can get a curve of test time speed-accuracy points. Notethat “full training”—considering all modes in every training
example—rather than “cascaded training”—just the ones
selected by the cascade step—leads to roughly a 1.5% performance increase (at the cost of 5× slower training, butequal test-time speed).


8. Conclusion
We have presented the MODEC model, which provides
a way to maintain the efﬁciency of simple, tractable models while gaining the rich modeling power of many global
modes. This allows us to perform joint training and inference to manage the competition between modes in a principled way. The results are compelling: we dominate acrossthe accuracy-speed curve on public and new datasets, anddemonstrate the importance of multimodality and efﬁcientmodels that exploit it.

Acknowledgments
The authors were partially supported by ONR MURI
N000141010934, NSF CAREER 1054215 and by STARnet, a Semiconductor Research Corporation program sponsored by MARCO and DARPA.

References[1] M. Andriluka, S. Roth, and B. Schiele. Pictorial structures
revisited: People detection and articulated pose estimation.
In Proc. CVPR, 2009.
[2] L. Bourdev and J. Malik. Poselets: Body part detectors
trained using 3d human pose annotations. In Proc. ICCV,2009.
[3] K. Duan, D. Batra, and D. Crandall. A multi-layer composite
model for human pose estimation. In Proc. BMVC, 2012.
[4] M. Eichner and V. Ferrari. Better appearance models forpictorial structures. In Proc. BMVC, 2009.
[5] M. Eichner, M. Marin-Jimenez, A. Zisserman, and V. Ferrari.
Articulated human pose estimation and search in (almost)
unconstrained still images. Technical report, ETH Zurich,
D-ITET, BIWI, 2010.
[6] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, andA. Zisserman. The PASCAL Visual Object Classes Challenge 2009 (VOC2009) Results, 2009.
[7] R. Fan, K. Chang, C. Hsieh, X. Wang, and C. Lin. A library
for large linear classiﬁcation. JMLR, 2008.

   46
 !%&"&) (
 

45
44

    
%& '
' $



42& &

  !'"

43

AUC

  

Qualitative results. Example output of our system on the
FLIC test set is shown in Figure 6.



41



  

40
39






   

37
0.5

  

%



38

    
1

1.5

2

2.5
seconds

3

3.5

4

4.5

  

Figure 5. Test time speed versus accuracy. Accuracy is measured
as area under the pixel error threshold curve (AUC), evaluated on
the FLIC testset. Speed is in seconds on an AMD Opteron 4284
CPU @ 3.00 GHz with 16 cores. See text for details.

3680
3678

Figure 6. Qualitative results. Shown are input test images from FLIC. The top 4 modes unpruned by the cascade step are shown in order of
their mode scores z , zr on the left and right side of each image. The mode chosen by MODEC is highlighted in green. The best parse y 
is overlaid on the image for the right (blue) and left (green) sides. In the last row we show common failures: ﬁring on foreground clutter,background clutter, and wrong scale estimation.

[8] Felzenszwalb, Girshick, and McAllester. Discriminatively
trained deformable part models, release 4, 2011.
[9] P. Felzenszwalb and D. Huttenlocher. Pictorial structures forobject recognition. IJCV, 2005.
[10] A. Frome, Y. Singer, F. Sha, and J. Malik. Learning globallyconsistent local distance functions for shape-based image retrieval and classiﬁcation. In Proc. ICCV, 2007.
[11] S. Johnson and M. Everingham. Learning effective humanpose estimation from inaccurate annotation. In Proc. CVPR,
2011.
[12] L. Ladicky and P. H. Torr. Locally linear support vector machines. In ICML, 2011.
[13] T. Malisiewicz, A. Gupta, and A. Efros. Ensemble ofexemplar-svms for object detection and beyond. In Proc.
ICCV, 2011.
[14] D. Ramanan and C. Sminchisescu. Training deformablemodels for localization. In Proc. CVPR, 2006.
[15] B. Sapp, A. Toshev, and B. Taskar. Cascaded models forarticulated pose estimation. In Proc. ECCV, 2010.
[16] B. Sapp, D. Weiss, and B. Taskar. Parsing human motion
with stretchable models. In Proc. CVPR, 2011.

[17] M. Sun and S. Savarese. Articulated part-based model forjoint object detection and pose estimation. In Proc. ICCV,2011.
[18] Y. Tian, C. Zitnick, and S. Narasimhan. Exploring the spatial
hierarchy of mixture models for human pose estimation. In
Proc. ECCV, 2012.
[19] D. Tran and D. Forsyth. Improved Human Parsing with a
Full Relational Model. In Proc. ECCV, 2010.
[20] Y. Wang and G. Mori. Multiple tree models for occlusionand spatial constraints in human pose estimation. In Proc.
ECCV, 2008.
[21] Y. Wang, D. Tran, and Z. Liao. Learning hierarchical poselets for human parsing. In Proc. CVPR, 2011.
[22] D. Weiss, B. Sapp, and B. Taskar. Structured prediction cascades (under review). In JMLR, 2012.
[23] D. Weiss and B. Taskar. Structured prediction cascades. In
Proc. AISTATS, 2010.
[24] Y. Yang and D. Ramanan. Articulated pose estimation using
ﬂexible mixtures of parts. In Proc. CVPR, 2011.
[25] X. Zhu and D. Ramanan. Face detection, pose estimationand landmark localization in the wild. In Proc. CVPR, 2012.

3681
3679