Fine-grained Activity Recognition with Holistic and Pose based Features
11,2
1
Leonid Pishchulin , Mykhaylo Andriluka , and Bernt Schiele
1sports

occupationwater activities

home activities

hockey, icebakeryskindiving, scuba

child carehome exercise

rope skipping

typing, electricswimming, synchr.

polishing floors

trampoline

locksmithswimming, breaststr.

rock climbing

masonry, concretetubing, floating

Goal
Analysis of the state of the art in the fine-grained humanactivity recognition on a large scale dataset

Contributions• Analysis of holistic and pose based approaches
for human activity recognition• Large scale comparison on “MPII Human Pose” dataset

2

Max Planck Institute for Informatics,
Germanycondition. exerc. fishing and hunt.

Stanford University,USAreligious activ.

winter activ.

walking

dancinglawn and garden

self carebicycling

miscellaneous

home repair

runningmusic playing

transportation

trapping gamesitting in churchskiing, downhill

descending stairs

Anishinaabe Jingle

planting trees

grooming, washing

unicycling

chess game, sitting

home repair

runningflute, sitting

pushing planeski machine

hunting, birds

serving foodskiing, cross-countr.

backpacking

general dancing

raking roof with snow

taking medication

bicycling, racingboard game playing

put on and remov.

jogging

accordion, sitting

riding in a car

kitchen activity

stair-treadmill

huntingkneeling in churchskiing, climbing up

walking, for exerc.

aerobic, stepwatering lawn or gard. showering, towelingbicycling, BMX

copying documents

caulking, chinking

running, training

conducting orchestra

motor scooter, motor

playing with child.

elliptical trainer

fishing, ice

sitting, playing

skating, ice dancing

pushing a wheelchairballet, modern

bicycling

standing, talking

repairing appliances

running, stairs, up

piano, sitting

riding in a bus

driving tractor

hairstyling

Randomly chosen activities and images from 18 top level categories of our “MPII Human Pose” dataset. One image per activity is shown. The full dataset is available at human-pose.mpi-inf.mpg.de.

• Analysis of factors responsible for success and failureof holistic and pose based methods“MPII Human Pose” dataset [2]Overall Activity Recognition Performance

Analysis of Activity Recognition Challenges• Order activities based on training set size

Motion specific challenges

yoga, bicycl., skiing, cooking skate- rope softball,forestry
power mount.downh. or food board. skip. general• # DT: number of dense trajectories• Systematically collected from YouTube videos using
established taxonomy [1] of everyday human activities

30• Covers 410 human activities

Dense trajectories (DT)GT single pose (GT)GT single pose + track (GT−T)
PS single pose + track (PS−T)
PS multi−pose (PS−M)
PS−M + DT (features)
PS−M filter DT
PS−M + DT (classifiers)

25• Contains around 25K images, 40K annotated poses• Rich annotations on test set: 3D torso and headorientation, body part occlusions

mAP20• Video snippet for each image, over 1M frames

15• MS: motion speed of dense trajectories• # DT body, MS body: trajectories on body mask only
• # people: number of people
Human pose specific challenges• Pose: deviation from the mean pose
• Occlusion: number of occluded body parts
25DT01• Dense Trajectories (DT) [5]

50

100

Pose based methods

150200 250# activities

300350151001p⇒ Performance quickly drops for large number of classes• GT single pose + track (GT-T)⇒ Holistic DT outperforms all pose based methods• Pictorial Structures (PS) single pose + track (PS-T) [4]⇒ PS-M performs best among pose based approaches

25⇒ Combination PS-M + DT (features) outperforms both
PS-M and DT

20

Successful and failure cases
cooking orfood prep.

# examples/activity

carpentry,
generalsanding
floorsballet,modernaerobic
stepmAP

1510p01

50

01

army type ballet,
training modernropeskipping

PS-M50• One-vs-all SVMs using SGD
0
1 50 100 150 200 250 300 350 400
2activityand χ kernel• Evaluation using mean Average Precision (mAP)

5075
100
# activities

25

125

150

Pose
Occlusion
Viewpoint

20

5.5
7.1
8.7
0.5
8.6
12.1
15.5

33.0
36.1
25.3
14.7
47.9
51.9
15.9

rock ballet, aerobic resist.
climb. modern step train.

total

41.5
2.3
8.9
1.2
4.7
14.4
38.6

19.0
11.2
13.9
8.5
20.2
23.5
19.5

12.7
1.0
1.7
2.8
22.9
23.7
7.1

36.0
51.2
69.2
63.1
90.5
88.7
55.724.5
1.1
3.3
11.1
10.4
17.1
25.8

16.5
1.4
1.3
1.6
7.2
14.4
9.6

⇒ Poor performance on “cooking” and “forestry”:
high variability in motion and poses⇒ Combination PS-M + DT (features) is best on average⇒ Holistic and pose based methods complementary

15Conclusion

100
1150

yoga,
power

circuittraining

pdrumming
bongo

canoeing,
kayaking

carpentry,
furniturechildrens
games

ballet,modernaerobicstep# dense trajectories (DT)Motion speed (MS)# DT body
MS body# people

20mAP

carpentry,
generalplaying drums, canoeing,
carrying
sanding
sitting
kayaking stacking wood floors

10025• Striking performance differences across activities
2550

75
100
# activities

125150• Holistic method influenced by high degree of motion• Pose methods affected by human pose and viewpoint

Close to mean poses and frontal views - easy poses

PS-M + DT

mowing lawn, canoeing,
walking
kayaking

200
150

125

5.5
2.7
2.8
5.3
3.4
5.0
6.1

8.4
1.4
1.2
1.8
5.2
7.2
10.6

11.4
3.4
4.6
1.7
17.2
19.5
13.5

⇒ Good performance on “golf” and “rope skipping”:
simple poses and motions

High MS is indicative - “easy” sport related poses

25train
test

75
100
# activities

DTGT
GT-T
PS-T
PS-M
PS-M + DT (feat.)
PS-M filter DT

12.7
2.2
3.6
1.6
3.0
3.9
4.2

0.5
1.8
2.6
1.3
2.6
2.2
0.8

5

5

25

51.9
7.5
10.9
6.0
27.3
32.9
52.2

⇒ Each method performs best on few activities only150

# dense trajectories (DT)Motion speed (MS)# DT body
MS body# people

14.5
26.5
28.0
6.6
34.0
40.7
20.4

• Combination holistic + pose method performs best

× High # DT: water related activities - hard poses

PS-M + DT• Bag-of-Words representation# examples/activity

Training and evaluation

canoeing,kayakingpDT

Experimental Setup• Sufficiently separated people

125

15
10
50
1

25

PoseOcclusion
Viewpoint

20mAP• PS-M filter DT: filter using body part masks

250

75
100
# activities

5• PS-M + DT (classifiers): decision level fusion

Data50

10.6
22.3
37.0
8.8
18.3
19.6
16.1

10

× Close to mean poses produce non-discriminative DT⇒ Holistic and pose based methods complementary• PS-M + DT (features): feature level fusion

15

Much motion indicative for good performance

PS-M

Holistic + pose based methods

25PoseOcclusion
Viewpoint

20

5400

• Ground truth (GT) single pose

• PS multi-pose (PS-M) [3]

25mAP

5mAP

20

Holistic method• 15,2K videos train / 5,7K test

# dense trajectories (DT)Motion speed (MS)# DT body
MS body# peoplemAP10MethodsDTGT
GT-T
PS-T
PS-M
PS-M + DT (feat.)
PS-M filter DT

carpentry, bicycl., golf
general racing• Viewpoint: deviation of 3D torso rotation from frontal• Available at human-pose.mpi-inf.mpg.deDetailed Analysis on a Subset of Activities

15105255075100# activities

1251500
1255075100# activities

125150Combination inherit positive qualities of both methods⇒ Holistic and pose based methods complementaryReferences[1] B. Ainsworth, W. Haskell, S. Herrmann, N. Meckes, D. Bassett, C. Tudor-Locke, J. Greer,J. Vezina, M. Whitt-Glover, and A. Leon. 2011 compendium of physical activities: a second
update of codes and MET values. MSSE’11.
[2] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2d human pose estimation: Newbenchmark and state of the art analysis. In CVPR’14.
[3] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black. Towards understanding action
recognition. In ICCV’13.
[4] M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele. A database for fine grained activity
detection of cooking activities. In CVPR’12.
[5] H. Wang, A. Kläser, C. Schmid, and C.-L. Liu. Dense trajectories and motion boundary
descriptors for action recognition. IJCV’13.

Acknowledgements. The authors thank Marcus Rohrbach and Sikandar Amin for helpful discussions. This work was supported by the Max Planck Center for Visual Computing & Communication.