2D Human Pose Estimation: New Benchmark and State of the Art Analysis
Mykhaylo Andriluka1,3 , Leonid Pishchulin1 , Peter Gehler2 , and Bernt Schiele1
1

2

Max Planck Institute for Informatics, Germany
Max Planck Institute for Intelligent Systems, Germany
3
Stanford University, USA

cally include highly articulated poses, but are limited withrespect to variability of appearance since people are typically wearing tight sports outfits. In turn, datasets such as“FashionPose” [2] and “Armlets” [9] aim to collect imagesof people wearing a variety of different clothing types, andinclude occlusions and truncation but are dominated by people in simple upright standing poses.
To the best of our knowledge no attempt has been madeto establish a more representative benchmark aiming tocover a wide pallet of challenges for human pose estimation. We believe that this hinders further development on
this topic and propose a new benchmark “MPII HumanPose”. Our benchmark significantly advances state of theart in terms of appearance variability and complexity, andincludes more than 40,000 images of people. We used
YouTube as a data source and collected images and image
sequences using queries based on the descriptions of morethan 800 activities. This results in a diverse set of images
covering not only different activities, but indoor and outdoor scenes, a variety of imaging conditions, as well as bothamateur and professional recordings (c.f . Fig. 1). This allows us to study existing body pose estimation techniques
and identify their individual failure modes.

Abstract
Human pose estimation has made significant progress
during the last years. However current datasets are limitedin their coverage of the overall pose estimation challenges.
Still these serve as the common sources to evaluate, train
and compare different models on. In this paper we introduce a novel benchmark “MPII Human Pose”1 that makes
a significant advance in terms of diversity and difficulty,
a contribution that we feel is required for future developments in human body models. This comprehensive dataset
was collected using an established taxonomy of over 800
human activities [1]. The collected images cover a wider
variety of human activities than previous datasets including
various recreational, occupational and householding activities, and capture people from a wider range of viewpoints.
We provide a rich set of labels including positions of body
joints, full 3D torso and head orientation, occlusion labels
for joints and body parts, and activity labels. For each image we provide adjacent video frames to facilitate the use of
motion information. Given these rich annotations we perform a detailed analysis of the leading human pose estimation approaches gaining insights for the success and failures of these methods.

Related work The commonly used publicly available
datasets for evaluation of 2D human pose estimation aresummarized in Tab. 1 according to the year of the corresponding publication. Both full body and upper body
datasets are included.
Existing benchmarks cover aspects of the human pose
estimation task such as sport scenes [12, 21], frontal-facing
people [8, 3, 17], people interacting with objects [23], pose
estimation in group photos [5] and pose estimation of people performing synchronized activities [4].
Earlier datasets such as “Parse” [16] and “Buffy” [8] arestill commonly found in evaluations [22, 15]. However thesmall training sets included in these datasets make them unsuitable for training models with complex appearance representations and multiple components [13, 17, 2], which havebeen shown to perform best.

1. Introduction
Recent pose estimation methods employ complex appearance models [2, 9, 15] and rely on learning algorithms
to estimate model parameters from the training data. The
performance of these approaches crucially depends on theavailability of annotated training images that are representative for the appearance of people clothing, strong articulation, partial (self-)occlusions and truncation at image borders. Although there exists training sets for special scenarios such as sport scenes [12, 13] and upright people [17, 2],
these benchmarks are still limited in their scope and variability of represented activities. Sport scene datasets typi1 Available at human-pose.mpi-inf.mpg.de.

1bicycling
bicycling, BMX

conditioning exercise
ski machinedancingballroomfishing and hunting
fish. from river bankhome activities
tanning hides

home repair
carpentry

inactivity quiet
sitting quietly

lawn and garden
driving tractor

miscellaneous
standingmusic playing
violin, sitting

occupation
horse grooming

religious activities
sit., playing instrum.

runningrunning, stairs, up

self caretaking medication

sportssoccertransportation
riding in a bus

volunteer activities
playing with children

walkingbird watchingwater activities
snorkelingwinter activities
skating, ice dancing

Figure 1. Randomly chosen images from each of 20 activity categories of the proposed “MPII Human Pose” dataset. Image captions indicate activity category (1st row) and activity (2nd row). To
view the full dataset visit human-pose.mpi-inf.mpg.de.

Some efforts have been made to collect larger sets ofimages. For example [13] extends the LSP dataset to10, 000 images of people performing gymnastics, athletics
and parkour. [2] proposes a large “FashionPose” dataset
collected from fashion blogs. This dataset aims to covera wide variety in people clothing. The LSP and FashionPose datasets are complementary and focus on two different
challenges for human pose estimation: pose variability andvariability of people appearance. However since they arecollected with a specific focus in mind, these datasets do
not cover real-life challenges such as truncation, occlusions
by scene objects and variability of imaging conditions.
The works of [6] and [9] propose a challenging dataset
building on the PASCAL VOC image collection. Results
reported in [9] indicate that the best performing approaches
for pose estimation of people in the presence of occlusionand complex appearance are under-performing on sportoriented datasets such as LSP [12] and vice versa. There are
qualitative differences between methods that work well for
LSP and “Armlets” datasets. On LSP the best performing
methods are typically based on flexible part-based models
that are well suited for capturing pose variability. In contrary on the “Armlets” dataset the best performing approach[9] uses a set of rigid detectors for groups of parts, that are
more robust to the variability in appearance.
Our dataset is complementary to the J-HMDB dataset[11] and provides more images and a wider coverage of ac-

Dataset#training#test

img. type

Full body pose datasets
Parse [16]LSP [12]
PASCAL Person Layout [6]Sport [21]UIUC people [21]LSP extended [13]FashionPose [2]J-HMDB [11]

100
1,000
850
649
346
10,0006,530
31,838

205
1,000
849
650
247
775
-

diversesports (8 types)
everydaysportssports (2 types)
sports (3 types)
fashion blogs
diverse (21 act.)

Upper body pose datasets
Buffy Stickmen [8]ETHZ PASCAL Stickmen [3]Human Obj. Int. (HOI) [23]
We Are Family [5]Video Pose 2 [18]
FLIC [17]Sync. Activities [4]Armlets [9]472180
350 imgs.
766
6,543
9,593

276
549
120
175 imgs.
519
1,016
357 imgs.
2,996

TV show (Buffy)
PASCAL VOC
sports (6 types)
group photos
TV show (Friends)
feature movies
dance / aerobics
PASCAL VOC/Flickr

MPII Human Pose (this paper)

28,821

11,701

diverse (491 act.)

Table 1. Overview of the publicly available datasets for articulated
human pose estimation. For each dataset we report the number ofannotated people in training and test sets and the type of images theset include. The numbers indicate the number of unique annotatedpeople without mirroring.

tivities (491 in our dataset vs. 21 in J-HMDB), whereas
J-HMDB provides densely annotated image sequences andlarger number of videos for each activity. Our dataset
also addresses a different set of challenges compared to thedatasets such as “HumanEva” [19] and “Human3.6M” [10]that include images and 3D poses of people but are captured
in the controlled indoor environments, whereas our dataset
includes real-world images but provides 2D poses only.

2. Dataset
In this paper we introduce a large dataset of images thatcovers a wide variety of human poses and clothing types
and includes people interacting with various objects and environments. The key rationale behind our data collection
strategy is that we want to represent both common and rarehuman poses that might be missed when simply collecting
more images without aiming for good coverage. To this end,
we use a two-level hierarchy of human activities proposedin [1] to guide the collection process. This hierarchy wasdeveloped for the assignment of standardized energy levels
during physical activity surveys and includes 823 activities
in total of 21 different activity categories. The activities at
the first level of the hierarchy correspond to thematically related groups of activities such as “Home Activities”, “Lawn
and Garden” or “Sports”. The activities at the second level
then correspond to individual activities such as “Washingwindows”, “Picking fruit” or “Rock climbing”. Note thatusing the activity hierarchy for collection has an additional
advantage that all images have an associated activity label.
As a result one can assess and analyze any performancemeasure also on subsets of activities or activity categories.

(a)

(b)

(c)

Figure 2. Visualization of upper body pose variability. From left
to right we show, (a) color coding of the body parts (b) annotationsof the “Armlets” dataset [9], and (c) annotations of this dataset.

Due to the coverage of the hierarchy the images in ourdataset are representative of the diversity of human poses,
overcoming one of the main limitations of previous collections. In Fig. 2 we visualize this diversity by comparing
upper body annotations of the “Armlets” dataset Fig. 2(b)and our proposed dataset (c). Note that although “Armlets”
contain about 13,500 images, the annotations resemble aperson with arms down along the torso (distribution of red,cyan, green, and blue sticks).
We collect images from YouTube using queries basedon the activity descriptions. Using YouTube allows us toaccess a rich collection of videos originating from various
sources, including amateur and professional recordings andcapturing a variety of public events and performances. In
Fig. 2 (c) we show the distribution of upper body poses on
our dataset. Note the variability in the location of hands
and the absence of distinctive peaks for the upper and lower
arms that are present in the case of the “Armlets” dataset.
Data collection. As a first step of the data collection wemanually query YouTube using descriptions of activities
from [1]. We select up to 10 videos for each activity filtering out videos of low quality and those that do not include
people. This resulted in 3, 913 videos spanning 491 different activities. Note that we merged a number of the original
823 activities due to high similarity between them, such as
cycling at different speeds. In the second step we manually
pick several frames with people from each video. As the focus of our benchmark is pose estimation we do not include
video frames in which people are severely truncated or inwhich pose is not recognizable due to poor image quality
or small scale. We aim to select frames that either depict
different people present in the video or the same person ina substantially different pose. In addition we restrict the
selected frames to be at least 5 seconds apart. This step
resulted to a total of 24, 920 extracted frames from all collected videos. Next, we annotate all people present in thecollected images, but ignore dense people crowds in whichsignificant number of people are almost fully occluded. Following this procedure we collect images of 40, 522 people.
We allocate roughly tree quaters of the collected images fortraining and use the rest for testing. Images from the same
video are either all in the training or all in the test set. This

(a)

(b)

(c)

Figure 3. Example of the provided annotations. Annotated are(a) positions and visibility of the main body joints, locations of
the eyes and nose and the head bounding box (occluded joints areshown in red), (b) occlusion of the main body parts (occluded parts
are shown with filled rectangles), and (c) 3D viewpoints of thehead and torso. On the illustration the viewpoint is shown using asimplified body model, the front face of the model is shown in red.

results in a training/test set split of 28, 821 to 11, 701.
Data annotation. We provide rich annotations for the collected images, an example can be seen in Fig. 3. Annotated
are the body joints, 3D viewpoint of the head and torso,
and position of the eyes and nose. Additionally for all body
joints and parts visibility is annotated. Following [13, 9]
we annotate joints in a “person centric” way, meaning thatthe left/right joints refer to the left/right limbs of the person. At test time this requires pose estimation with both acorrect localization of the limbs of a person along with thecorrect match to the left/right limb. The annotations are performed by in-house workers and via Amazon Mechanical
Turk (AMT). In our annotation process we build and extend
the annotation tools described in [14]. Similarly to [13, 20]
we found that effective use of AMT requires careful selection of qualified workforce. We pre-select AMT workers
based on a qualification task, and then maintain data quality
by manually inspecting the annotated data.
Experimental protocol and evaluation metrics. We define the baseline evaluation protocol on our dataset following the current practices in the literature [13, 9, 15]. We
assume that at test time the rough location and scale of aperson are known, and we exclude the cases with multiple
people in close proximity to each other from the evaluation.
We feel that these simplifications are necessary for the rapid
adoption of the dataset as the majority of the current approaches does not address multiple people pose estimationand does not search over people positions and scales.
We consider three metrics as indicators for the pose estimation performance. The widely adopted “PCP” metric [8]that considers a body part to be localized correctly if theestimated body segment endpoints are within 50% of theground-truth segment length from their true locations. The“PCP” metric has a drawback that foreshortened body parts
should be localized with higher precision to be consideredcorrect. We define a new metric denoted as “PCPm” thatuses 50% of the mean ground-truth segment length over theSetting

Torso Upper Lower Upper Fore- Head Upper Fulllegleg
arm arm
body body

Gkioxari et al. [9]Sapp&Taskar [17]Yang&Ramanan [22]Pishchulin et al. [15]

51.3
51.3
61.0
63.8

36.6
39.6

36.5
37.3

28.0
27.4
34.8
39.0

12.4
26.4
16.3
27.8
17.4 70.2 33.1 38.3
26.8 70.7 39.1 42.3

Gkioxari et al. [9] + loc
65.1
Sapp&Taskar [17] + loc
65.1
Yang&Ramanan [22] + loc 67.2
Pishchulin et al. [15] + loc 66.6

39.7
40.5

39.4
38.2

33.7
32.6
37.4
40.4

14.9
32.4
19.2
33.7
18.6 75.7 35.8 41.4
27.7 74.5 40.6 43.9

Table 2. Pose estimation results (PCPm) on the proposed dataset
without and with using rough body location (“+ loc” in the table).

entire test set as a matching threshold, but otherwise follows
the definition of “PCP”. Finally, we consider the “PCK”
metric from [22] that measures accuracy of the localizationof the body joints. In [22] the threshold for matching of thejoint position to the ground-truth is defined as a fraction of
the person bounding box size. We use a slight modificationof the “PCK” and define the matching threshold as 50% of
the head segment length. We denote this metric as “PCKh”.
We choose to use head size because we would like to makethe metric articulation independent.

3. Analysis of the state of the art
In this section we analyse the performance of leading
human pose estimation approaches on our benchmark. We
take advantage of our rich annotations and conduct a detailed analysis of various factors influencing the results,
such as foreshortening, activity and viewpoint, previously
not possible in this detail. The goal of this analysis is toevaluate the robustness of the current approaches in various
challenges for articulated pose estimation, identify the existing limitations and stimulate further research advances.
In our analysis we consider two full body and two upper body pose estimation approaches. The full body approaches are the version 1.3 of the flexible mixture of parts(FMP) approach of Yang and Ramanan [22] and the pictorial structures (PS) approach of Pishchulin et al. [15]. The
upper body pose estimation approaches are the multimodal
decomposable models (MODEC) approach of Sapp et al.
[17] and the Armlets approach of Gkioxari et al. [9]. In
case of FMP and MODEC we use publicly available codeand pre-trained models. The PS model used here corresponds to our best model published in [15]. In case of theArmlets model, the code and pre-trained model provided by
the authors correspond to the version from [9] that includes
the HOG features only. The performance of our version ofArmlets on the “Armlets” dataset is 3.3 PCP lower than theversion based on combination of all features.2
Note that the approaches considered in this evaluation
are the best performing ones in their respective categories.
2 See Tab.1 in [9] for the comparison.

The PS approach achieves the best results to date on LSP
that is focused on the strongly articulated people [15]. The
Armlets approach is best on the “Armlets” dataset [9] that
includes large number of truncation and occlusions, and
MODEC is the best on the recent upper body pose estimation dataset “FLIC” [17]. We include the FMP approachthat is widely used in the literature and typically shows competitive performance for a variety of settings. In the following experiments we use “PCPm” as our working metric, while also providing results for “PCP” and “PCKh” inthe supplementary material. While we observe little performance differences when using each metric, all conclusions obtained during “PCPm”-based evaluation are valid
for “PCP” and “PCKh”-based evaluations as well.
Overall performance evaluation. We begin our analysis by reporting the overall pose estimation performance ofeach approach and summarize the results in Tab. 2. We include both upper- and full body results to enable comparison across different models. The PS approach achieves thebest result of 42.3% PCPm, followed by the FMP approachwith 38.3% PCPm. On the upper body evaluation, PS performs best with 39.1%, while both MODEC (27.8% PCPm)
and Armlets (26.4% PCPm) perform significantly worse.
The interesting outcome of this comparison is that both
upper body approaches MODEC and Armlets are outperformed by the full body approaches PS and FMP evaluatedon upper body only. This is interesting because significant
portion of the dataset (15 %) includes people that have only
upper body visible. It appears that the PS and FMP approaches are sufficiently robust to missing parts to produce
reliable estimates even in the case of lower body occlusion.
Lower part of Tab. 2 shows the results when using provided rough location of person during test time inference.
We observe, that while the performance increases for all
methods, upper body approaches profit at most, as theyheavily depend on correct torso localization. For the sakeof fair comparison among the methods, we do not use therough location in the following experiments. Another interesting outcome is that the achieved performance is substantially lower than current best results on the sports-centric
LSP dataset, but comparable to results on the “Armlets”
dataset (42.2 PCP on our benchmark (see supplemental) vs.
69.2 on LSP [15] vs. 36.2 PCP on “Armlets”). This suggests that sport activities are not necessary the most difficult cases for pose estimation; challenges such as appearance variability, occlusion and truncation apparently deserve more attention in the future.

3.1. Analysis of pose estimation challenges
We now analyse the performance of each approach withrespect to the following five factors: part occlusion, foreshortening, body pose, viewpoint, and activity of the person. For the purpose of this analysis we define quantitative

80
80

70

70

60

60
108

109
110
111
112
113
114

115
116
117
118
119

120
121
122
123
124
125

126
127
128
129
130
131

132
133
134
135
136
137

138
139
140
141
142

143
144
145
146
147
148

50

149
150
151
152
153
154

50

Pishchulin et al.
Yang&Ramanan
Sapp&Taskar
Gkioxari et al.

70

155
156
157
158
159
160

PCPm, %

60

Occlusion
Part length
Pose
VP torso
Truncation

161

PCPm, %

80

Occlusion
Part length
Pose
VP torso
Truncation

50
40

30

30
1000

2000

3000 4000 5000
Number of images

6000

7000

0

Pishchulin et al. [15]803000 4000 5000
Number of images

6000

40

7000

30

60

50

20

80

Occlusion
Part length
Pose
VP torso
Truncation

70

PCPm, %

PCPm, %

2000

Yang&Ramanan [22]Occlusion
Part length
Pose
VP torso
Truncation

70

1000

10

60

0

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

0

PCPm, %

40

50

−20

−50
100

0

−100 50−50 100−100
50−100 50

−50

100

2

−20
0

−20

−20

−20

0

0

0

Sapp&Taskar [17]

−20
0

6000

7000

Gkioxari et al. [9]Figure 4. Performance (PCPm) as a function of the five complexity
measures.

−20

0

−20

−20

50

−50

100

0

150

−80

100
−60

−40

150−200

20 −100

40

60−50

−20

0

80

0

−100
40

60

−50 80

100

0

−40

100
−20

40
−40

−20
60

80
0

20

40

−1006080−50 100120

050 −60

−40
100

−20

0

40

−5060

0

−100

50

−50

100

0

−40

100
−20

50
−80

−60

100
−40

−20

0

40
−80

60−60

80−40

100−20

160
0

40

60−50

80

0 −150

20

0

20

40

0

−80

−6050

−40

−20 100 0

20 −100 40

80
−50

100

120

0

120
−80

100

100
−20

0

−80 40 −60

60
−40

−20
80

0

120

20

−80 40

−60
60

−40
80

100
−20

0

40

60

0

80

0

20

−100 40

60−50

80

0

−150

50

−100

−100
50

−50

100

0

−100

−80
50

−60

100−40−20−800−60 20 −40

40−20

600

−120
20

60

−80 80

−60
100

−40

−80
−20

−60 0

−40 20

−20

40

0

−50

60

80

−150 0−100 50

0

80

0

−40

100 −20

0

20

40

60

40

60
60
80
80
100
120

100
−50

0

140

160

160

180
100

50
−100

120

140
60

80

−50
60

−6050

20

40

100
40
60
80

100
160
40

100
0−80

0

20
40

60

140

−100
20

−50 100

−20

0
20

0

20

40
80

120

100

40−100

100

−20
−20

60

100

120

140

80

140

160
−5020

20

40

80

100

80

120

160

100
−100
20

100
−20

60

60

100

80

140

140
−40

−40

−20

60
40

80

60

80
120

−60
50

−6050

20
40

20

40

40

100

60
80

100

60

40

180

160

0 −80

20
60

40
60

80
80

100

120

100−5080 −50100

0

20

0

40
20

60

60

60

80

100

−100
50

40 −100 60

−20

0
0

20

40
40

60

80

0

0

20

160

160

20

0

120

−50100

0

0

140

160
−100
50

−2060

−20

−20

0

0

20
40

40

60

120

140

180

−100
20

−60 40 −40

−20
−20

0

−20

140

140

−100 20−80

−20

−20
−20

20

20

40

80

140

−100
20

0

120

120
140

100

180
50

100

80
100
120
80

160

−60

80

60
80
100

60

140

140

70
20−100

60

40

60
40

120

60

−80

40

20

40

80
100

80

100

120

20−60

20

0

20

20
60

60

40

140
150

0

0

−20

0
0

40
40

30

80

50

−60
50

−20

−40

−20
−20

0
20
20

20

60

120
80

160

−80

−20

0

140
−100

−20

0

0

10

40

100

100

60

140

100
120

−100
50

−20

−10

0

20

0

50

120

100

160

0 −100

100

120

140
120

140
20

−80

50−60

−40

100 −20

−100
0

80
20
−80

40−60

60−40

80 −20

−60
0

20−40

40 −2060

020 −10040−50 60050−100100 −50

150

0

50

100

Figure 5. Performance (PCPm) on images clustered by full bodypose. Clusters are ordered by increasing mean pose complexity
and representatives are shown beneath. Results using upper bodyand lower body clusters can be found in supplementary material.

162
163
164
165
166

Body pose performance. As stated above the complexityof the pose is a dominating factor for the performance of
all considered approaches. For example the PS approachachieves 72.8% PCPm on the 1000 images with lowest posecomplexity, compared to 42.3% for the entire dataset. The
same is true for the FMP model, 63.4% PCPm on 1000 leastpose complex images vs. 38.3% overall.
To highlight variations in performance across different
body configurations we cluster the test images according
to the body pose and measure performance for each cluster. We repeat this three times, clustering all body joints,
only the upper body joints, and finally the lower body joints.

167
168
169
170
171
172

173
174
175
176
177

178
179
180
181
182
183

184
185
186
187
188
189

190
191
192
193
194
195

196
197
198
199
200

201
202
203
204
205
206

sures as the complexity increases. There are interesting differences however. Body pose complexity clearly influences
the performance of all approaches the most. The second
most influential factor is the viewpoint of the torso. For upper body pose estimation approaches this factor is equally
influential as body pose. The third most influential factors
is occlusion while for the full body estimation approaches
this is equally influential as the torso orientation. Contrary
to our expectation we found that the part length is less influential. Part length and in particular foreshortening effects
are considered to be the key difficulties for both pose estimation. Based on this analysis the above mentioned factors
have a higher influence on the performance. The least influential factor is truncation having the smallest effect. In
the case of upper body estimation the performance evenslightly increases as the amount of truncation increases dueto two factors. As truncation if more likely for the lower
body these approaches suffer less from truncation and also
truncated poses are biased towards frontal views for which
the methods are more suited. We now discuss and analyzeeach factor in more detail.
207
208
209
210
211
212

Performance as a function of the complexity measures
To visualize the influence of the various factors on pose estimation performance we plot PCPm scores for the images
sorted in the order of increasing complexity (see Fig. 4). In
general and as expected, the performance drops for all mea-

−40

213
214
215

complexity measures that map body image annotations to areal value that relates to the complexity of the image withrespect to each factor.
Let us denote the annotation of the person by L ={Lpose , Lview , Lvis }, where Lpose = {li , i = 1, . . . , N }corresponds to the positions of body parts, Lview ={α1 , α2 , α3 } are the Euler angles representation of the torso
rotation, and Lvis = {(ρi , θi ), i = 1, . . . , N } encodes bodypart visibility via a set of occlusion labels ρi ∈ {0, 1} andtruncation labels θi ∈ {0, 1}.
We define the following complexity measures. Pose
complexity is measured as the deviation from the meanpose
Q on the entire dataset. We define mpose (L) =(i,j)∈E pps (li |lj ), where E is a set of body joints andpps (li |lj ) is a Gaussian distribution measuring relative position of the two adjacent body parts using the transformed state-space representation introduced in [7]. Notethat mpose (L) corresponds to the likelihood of the pose
under the tree structured pictorial structures model [7].
The amount of foreshortening is measured by mf (L) =PN
i=1 |d(li )−mi |/mi , where d(li ) is the length of the bodypart i, and mi is the mean length over the entire dataset. The
viewpoint complexity is measured
P3 by the deviation from thefrontal viewpoint: mv (L) = i=1 αi . Finally, the amountof occlusion and truncation correspond to thenumber ofPN
occluded and truncated body parts: mocc = i=1 ρi , andPN
mt = i=1 τi .

100

20
20

80
100

120
100

160

−60

0

100

80
80

100

120

180

3000 4000 5000
Number of images

−50

−20

60
80

60

80

120

160

−80

2000

−50

0

40
100

80
80

140

180
−100 50

0

40

60

100
140

120

1000

−150 0

−20

20

60

40
60

0

100
−50

20

80
60

60

120

120

200
50−100

−20

0

60
40
40

80
100

80

100

160
180
0 −150

100
−50

40

20

60

60

120

160

−100
50

40
40

80

120

7000

0

20
20

40

60

100
80

6000

100

20
20
40

60

3000 4000 5000
Number of images

−50

−20

20

40

2000

80
100

140
140

50

0

0

20

1000

40

60

100

−100

20

20

40

40

80

120

140

0

0

0

20
20

50

60

100

120

160

180
0

100
−50

−20

0

0

20

40

80

150

0

−20

100

0

−20

−20
−20

0

20

60

100

140

160
160

180
−100
50

0

120

140
140

160

−50

100

120
120

140

160
160
−100

80

100
100

120

140
140

30

−20

−20
0

0

0

20

60

80

80

100

120

30

−20

−20

0

40

60

60

80
80
100

120

40

20

40

40
60

60

80

100

40

−20

0
20

40

40

40

60

20

20

20

40

0

0

0

0

20

−20

−20

−20

−20

In the latter two cases we measure performance on the upper/lower body parts only. These three clusterings correspond both to different types of challenges as well as applications. Furthermore, this allows to directly compare fullvs. upper body techniques. We show the average PCPm for
all full body clusters with more than 25 examples in Fig. 5ordering the results from left to right by increasing meanpose complexity. Note the significant variations in performance across different clusters. For example, results onfull body clusters vary between 77% and 2% PCPm. The
best performance is achieved on clusters with poses similar to the mean pose e.g. clusters 1 and 5 (see Figure 5).
Examining clusters with poor performance we immediately
discover several failure modes of PS and FMP approaches.
Consider the clusters 42 and 43 that correspond to people
with slightly foreshortened torso. FMP improves over PS
by 14% PCPm on cluster 25 (54% PCPm for PS vs. 68%
PCPm for FMP) and by 16% PCMm on cluster 42 (44%
PCPm for PS vs. 60% PCPm for FMP), as it can bettermodel torso foreshortening by representing torso as configuration of multiple flexible parts, whereas PS models torso
as a single rigid part. Also, the flexibility of FMP modelaccounts for its better performance on frontal sitting people (cluster 43) where FMP improves over PS by 7% PCPm
(46% PCPm for FMP vs. 39% PCPm for PS), mainly due
to better modeling of the foreshortened upper legs. However, performance on the sideview sitting people (e.g. clusters 26, 30, 34, 44) is poor for all methods. Another prominent failure mode for all approaches are people facing awayfrom the camera, e.g. cluster 50. Such part configurations
are commonly mistaken for the frontal view which leads toa mismatch between left and right body parts resulting inincorrect estimation. These findings demonstrate inabilityof current methods to reliably discriminate between frontal
and backward views of people. Interestingly, upper body
approaches outperform full body methods on the full bodycluster 31. This is an easy case for the former group of
methods due to frontal upright upper body, but is a challenging task for the full body upproaches as legs are hard to
estimate in this case. However, both MODEC und Armlets
fail on examples when torso start deviating from canonicalorientation (e.g. clusters 20, 27, 37). At the same time bothfull body methods perform better, as they are more robust
to the viewpoint changes. Surprisingly, full body methods
outperform upper body approaches on “easy” examples (c.f.
cluster 1, 3 and 5). We attribute this effect to the correct integration of signals from the legs into a more reliable upper
body estimate.
Occlusion and truncation performance. In Fig. 4 weclearly see difference in how occlusion and truncation influences the results. As expected we observe that the performance is best for fully visible people, but full visibility
does not result in success rate similar to the one we observed

for the images with simple poses, e.g. PS approach achieves
72.8% PCPm for 1000 most simple poses vs. 60% PCPmfor same amount of people with least occlusion. We observe that occlusion results in significant performance dropon the order of 10% PCPm, e.g. in the case of PS approach
19.3% vs. 31.2% PCPm for the forearm with and without
occlusion.
As mentioned above, truncation showed the least influence overall among the discussed factors. There are at least
two reasons. First, the number of images with truncation
is limited in our dataset (about 30% of the test data contain
truncated people). Second, and more importantly, for truncation one cannot annotate positions of body parts outsideof the image. Therefore the standard procedure is to exclude
truncated body parts from the evaluation. In that sense approaches that wrongly estimate the position of a truncatedbody part are not punished for that. This limitation could
be addressed by requiring that models have to also reportwhich parts of the body are truncated.
Viewpoint performance. We evaluate the pose estimation for various torso viewpoints in two ways. In Fig. 4
we show results using our standard analysis method basedon images ordered by deviation from the frontal viewpoint.
For a more detailed analysis we quantize the space of viewpoints by clustering training examples according to their 3D
torso orientations. We show results for the viewpoint clusters in Fig. 6 ordering them by the number of examples corresponding to each cluster. The number of examples per
cluster ranges between 1453 examples for the largest cluster corresponding to the frontal viewpoint, and 53 examples
for the viewpoint with extreme torso tilt.
We observe that in contrast to the full body approaches,
viewpoint has profound influence on the performance of
the upper body approaches considered in our evaluation.
The performance of both Armlets and MODEC approaches
drops significantly for non-frontal views.
A per viewpoint evaluation reveals significant performance differences across viewpoints. In Fig. 6 we show
the results for the “person centric” annotations that we use
throughout experiments in this paper and in addition for the“observer centric” (OC) annotations, in which body limbs
are labeled as left/right based on their image location withrespect to the torso. Frontal and near-frontal viewpoints are
performing best. We observe a large drop in performancefor backward facing people when performance is measured
in “person centric” manner, which suggests that large portion of incorrect pose estimates for backward views is due
to incorrect matching of left/right limbs.
We observe that all approaches handle extreme viewpoints poorly. PS approach is the only one in our evaluation that gracefully handles in-plane rotations (cluster 12),
whereas performance of other approaches significantly degrades in that case. Also, PS outperforms other methods

60

50

PCPm, %

40

60
PCPm, %

Pishchulin et al.
Pishchulin et al. OC
Yang&Ramanan
Yang&Ramanan OC
Sapp&Taskar
Sapp&Taskar OC
Gkioxari et al.
Gkioxari et al. OC

40
20
0

1

2

3

4

5

6

7
8
9 10
Viewpoint clusters

11

12

13

14

15

70

30
60

Pishchulin et al.
Pishchulin et al. retrained
Yang&Ramanan
Yang&Ramanan retrained

PCPm, %

20

10

50

40

30

0

1

2

3

4

5

6

7
8
9 10
Viewpoint clusters

11

12

13

14

15

co

nd

s
iti oc po
on cu rtin pa sg
ex tionercw
at da ise
er nc
ac in
h ti gho om vitieemla e re spw
fis n act air
hi an ivi
ng d tie
an gar s
m d h denu u
w sic ntininte pla gr a yi
ct ng
iv
iti
m
is wa esce lk
lla in
ne g
bi ou
in
cy s
ac
cl
tiv
i
ity run ngnq
tra uie ing
ns t/lpo igh
re
rta t
lig
t
vo iou self ion
lu s
nt ac caree ti er a viti
ct es
iv
iti
es

20

Figure 6. Pose estimation results (PCPm) grouped by viewpoint.
Viewpoint clusters ordered decreasingly w.r.t. number of images.
Each cluster is visualized in bottom row using 3D model of thetorso corresponding to the cluster medoid.

Setting65
Pishchulin et al.
Yang&Ramanan
Sapp&Taskar
Gkioxari et al.

60
55
50
PCPm, %

Figure 8. Comparison of performance (PCPm) on viewpoint (top)categories
and activity category clusters activity(bottom)
before and after retraining.
See Fig. 6 for visualization of the viewpoint clusters.

45
40

Torso Upper Lower Upper Fore- Head Upper Fulllegleg
arm arm
body bodyYang&Ramanan [22]61.0
Yang&Ramanan [22] retrained 69.3
Pishchulin et al. [15]63.8
Pishchulin et al. [15] retrained 68.4

36.6
39.5
39.6
42.7

36.5
38.8
37.3
42.8

34.8
43.4
39.0
42.0

17.4
27.7
26.8
29.2

70.2
74.6
70.7
76.3

33.1
42.3
39.1
42.1

38.3
44.7
42.3
46.1

Table 3. Comparison of performance (PCPm) before and after retraining. For PCKh results see supplementary material.

35
30
25

co

nd

iti

o sp
on ccu ort
in pa sg
ex tionercw
at da ise
er nc
ac in
h ti gho om vitieemla e re spw
fis n act air
hi an ivi
ng d tie
an gar s
m d h denu u
w sic ntininte pla gr a yi
ct ng
iv
iti
m
is wa esce lk
lla in
ne g
bi ou
in
cy s
ac
cl
tiv
i
ity run ngnq
tra uie ing
ns t/lpo igh
re
rta t
lig
t
vo iou self ion
lu s
nt ac caree ti er a viti
ct es
iv
iti
es

20

Figure 7. Pose estimation results (PCPm) grouped by activity catactivity categories
egories shown in decreasing order w.r.t. number of images.

in case of extreme torso tilts (e.g. cluster 14). The performance for clusters with extreme torso rotation is on thelevel of 20 - 30% PCPm for the best method, corresponding
to only 2 - 3 out of 10 body parts being localized correctlyfor such viewpoints.
Part length performance. Fig. 4 also shows the influence of part length on the performance of each approach.
In this context, foreshortening is the most influential aspect
and considered an important challenge for articulated pose
estimation. The key observation is that the presence or absence of foreshortening has relatively little influence on the
result compared to the other factors such as pose and occlusion. The best performing PS model is the most robust

to foreshortening compared to other three approaches. For
example the performance for the first 4000 images orderedby increasing foreshortening remains nearly constant.
Activity performance. Finally, we evaluate pose estimation performance as a function of the person activity. To
that end we group test images by the activity categories in
the hierarchy used for the image collection [1] and compute
PCPm for each category. The results are shown in Fig. 7,
where we order categories from left to right according to thenumber of test examples.
We observe strong variation of performance for different
activity types. Best results are obtained on the sports- anddancing-centric activities (e.g. “Sports”, “Running”, “Winter Activities” and “Dancing”). Most difficult turn out to be
activities that are performed in bulky clothing and involveuse of tools (e.g. “Home Repair”) and activities performedin cluttered scenes (e.g. “Fishing and Hunting”). MODEC
outperforms all other approaches on the “Self care” activities (examples of activities from this category are “Eating,sitting”, “Hairstyling”, “Grooming” etc. with “Eating, sitting” containing by far the largest number of images.)Retrained models. To showcase the usefulness of thebenchmark as an analysis tool we retrain the PS and FMP
models on the training set from our benchmark. To speed
up training we consider a subset of 4000 images, which is 4
times as many images as in the LSP and 40 times as manyas in the PARSE datasets used by the publicly available PS
and FMP models. The results are shown in Tab. 3. FMP significantly benefits from retraining (44.7 PCPm for retrainedvs. 38.3 for original). PS achieves slightly better result,although overall improvement due to retraining is smaller(46.1 PCPm for retrained vs. 42.3 PCPm the original).
Although performances for FMP and PS are close overall, we observe interesting differences when examining performance at the level of individual activities and viewpoints(thereby exploiting the rich annotations of our benchmark).
Results are shown in Fig. 8. We observe that our publicly
available PS model is winning by a large margin on thehighly articulated categories, such as “Dancing” and “Running”. Retraining the model boosts performance on activities with less articulation but more complex appearance(e.g. “Home Activities”, “Lawn and Garden”, “Bicycling”,
and “Occupation”). Our results show that training on thelarger amount of more variable data significantly improved
robustness of FMP to viewpoint changes. Performance of
FMP improves on the difficult viewpoints by a large margin (e.g. for viewpoint cluster 10 improvement is from 17
to 31% PCPm). Retraining improves the performance ofPS model on difficult viewpoints as well, although not as
dramatically as for FMP, likely because PS already models
in-plane rotations explicitly.

4. Conclusion
In this work we advance the state of the art in humanpose estimation by establishing new qualitatively higher
standards for evaluation and analysis of pose estimation
methods and demonstrate the most promising research directions for the next years. To that end we propose a novel“MPII Human Pose” benchmark that we collected by leveraging a taxonomy of activities established in the literature.
Compared to current datasets our benchmark covers significantly wider range of human poses spanning from householding to recreational activities and sports. Rich labelingof the collected data and a set of developed evaluation tools
enable comprehesive analysis which we perform to demonstrate the strengths and weaknesses of the current methods.
Our findings indicate that current methods are significantly challenged by cases outside their comfort zone, suchas large torso rotation and loose clothing. From all otherfactors, pose complexity has the most profound effect on
the pose estimation performance. Current methods perform
best on activities with simple tight clothing (e.g. in sportscenes), and are challenged by images with complex clothing and background clutter that are typical for many occu-

pational and outdoor activities.
We will make the data, rich annotations for training images and evaluation tools publicly available in order to enable detailed analysis of future pose estimation methods. To
prevent accidentally tuning on the test set, the annotations
for the test images will be withheld and made accessible
through an online evaluation tool. In the future we plan toextend our benchmark to joint pose estimation of multiple
people and pose estimation in image sequences.
Acknowledgements. This work has been supported by
the Max Planck Center for Visual Computing & Communication. The authors are thankful to Steve Hillyer and numerous anonymous Mechanical Turk workers for the help
with preparation of the dataset.

References[1] B. Ainsworth, W. Haskell, S. Herrmann, N. Meckes, D. Bassett, C. TudorLocke, J. Greer, J. Vezina, M. Whitt-Glover, and A. Leon. 2011 compendiumof physical activities: a second update of codes and MET values. MSSE’11.
[2] M. Dantone, J. Gall, C. Leistner, and L. V. Gool. Human pose estimation using
body parts dependent joint regressors. In CVPR’13.
[3] M. Eichner and V. Ferrari. Better appearance models for pictorial structures. In
BMVC’09.
[4] M. Eichner and V. Ferrari. Human pose co-estimation and applications.
PAMI’12.
[5] M. Eichner and V. Ferrari. We are family: Joint pose estimation of multiple
persons. In ECCV’10.
[6] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.
The PASCAL visual object classes (VOC) challenge. IJCV’10.
[7] P. F. Felzenszwalb and D. P. Huttenlocher. Pictorial structures for object recognition. IJCV’05.
[8] V. Ferrari, M. Marin, and A. Zisserman. Progressive search space reduction forhuman pose estimation. In CVPR’08.
[9] G. Gkioxari, P. Arbelaez, L. Bourdev, and J. Malik. Articulated pose estimation
using discriminative armlet classifiers. In CVPR’13.
[10] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3.6M: Large
Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. PAMI’13.
[11] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black. Towards understanding
action recognition. In ICCV’13.
[12] S. Johnson and M. Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC’10.
[13] S. Johnson and M. Everingham. Learning Effective Human Pose Estimationfrom Inaccurate Annotation. In CVPR’11.
[14] S. Maji. Large scale image annotations on amazon mechanical turk. Technicalreport, EECS UC Berkeley, 2011.
[15] L. Pishchulin, M. Andriluka, P. Gehler, and B. Schiele. Strong appearance andexpressive spatial models for human pose estimation. In ICCV’13.
[16] D. Ramanan. Learning to parse images of articulated objects. In NIPS’06.
[17] B. Sapp and B. Taskar. Multimodal decomposable models for human pose
estimation. In CVPR’13.
[18] B. Sapp, D. Weiss, and B. Taskar. Parsing human motion with stretchable
models. In CVPR’11.
[19] L. Sigal, A. Balan, and M. J. Black. Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated humanmotion. International Journal of Computer Vision, 87, 2010.
[20] C. Vondrick, D. Patterson, and D. Ramanan. Efficiently scaling up crowdsourced video annotation. IJCV’12.
[21] Y. Wang, D. Tran, and Z. Liao. Learning hierarchical poselets for human parsing. In CVPR’11.
[22] Y. Yang and D. Ramanan. Articulated human detection with flexible mixturesof parts. PAMI’13.
[23] B. Yao and L. Fei-Fei. Modeling mutual context of object and human pose in
human-object interaction activities. In CVPR, 2010.