HAND DETECTION AND
TRACKING FOR FINE-GRAINEDACTION RECOGNITION

Nga H. Do and Keiji Yanai
The University of Electro-Communications, Tokyo

Hand Detection and Tracking for fine-grained action recognition

Introduction and Related Work
• General recognition vs. fine-grained recognition

Large intra-classvariability
umbrella

open

fridgewinebottle

UCF Sports dataset

Hand Detection and Tracking for fine-grained action recognition

Introduction and Related Work• Fine-grained activity recognition• Database: Cooking fine-grained activities(Rohrbach et al. CVPR2012)• Activity = composite of multiple actions• Eg.: take X from fridge = open fridge + take X• Recognition accuracies by state-of-the-art:• Rohrbach et al. ECCV2012: 21.3%• Ni et al. CVPR2014: 28.6%
tough problem!

Should we consider a simpler problem?

Hand Detection and Tracking for fine-grained action recognition

Introduction and Related Work• Fine-grained action recognition• Sub-problem of fine-grained activity recognition• Target: single actions

Can we apply fine-grained activity recognition
approaches to fine-grained action recognition?

Hand Detection and Tracking for fine-grained action recognition

Introduction and Related Work• Most popular approaches for fine-grained

activity recognition:• Object detection + Human-object interactions

B. Ni, V. R. Paramathayalan, and P. Moulin. Multiple granularity analysis forfine-grained action detection. CVPR 2014.

Hand Detection and Tracking for fine-grained action recognition

Introduction and Related Work• Most popular approaches for fine-grained

activity recognition:• Object detection + Human-object interactions• Disadvantages:• requirement of training data for object detectors• inefficiency in case of similar objects

guitarDifferent hand movements

playviolin

Hand Detection and Tracking for fine-grained action recognition

Introduction and Related Work• This work:• fine-grained action classification based on how

human use their hands to operate the actions• Hand motion features for fine-grained action recognition• A system of hand detection and tracking in

uncontrolled videos

Hand Detection and Tracking for fine-grained action recognition

Proposed Method
Fine-grained action recognition based on hand movements

Hand detection
and trackingFeatureExtraction

VideoEncoding

HandMotion
Features

Gaussian
Mixture Model
Fisher Vectors

ActionClassification

Earlyfusion

Multiclasslinear SVMs

Hand Detection and Tracking for fine-grained action recognition

Hand Detection
http://groups.inf.ed.ac.uk/calvin/calvin_upperbody_detector/(1a) Upper bodydetection

currentframe(1b) Upper bodyselection

flow field

nextframe

P. Weinzaepfel, J. Revaud, Z. Harchaoui, andC. Schmid. DeepFlow: Large displacement
optical flow with deep matching. ICCV2013.

Hand Detection and Tracking for fine-grained action recognitionHand detection(2) Staticcue based
handdetection
A. Mittal, A. Zisserman, and P. H. S. Torr.
Hand detection using multiple proposals.
BMVC 2011.

HOG based
Skin based
hand detection hand detection

Image••••

PASCAL VOC
Buffy Stickman
INRIA Pedestrianetc

Super-pixel
based NMS

Postprocessing

Hand Detection and Tracking for fine-grained action recognition

Hand detection
Flowinformation

Upper bodyposition(3) Refinement

candidates

𝒇𝒊𝒏𝒂𝒍 𝒔𝒄𝒐𝒓𝒆 = 𝑴𝒊𝒕𝒕𝒂𝒍′ 𝒔 𝒅𝒆𝒕𝒆𝒄𝒕𝒐𝒓 𝒔𝒄𝒐𝒓𝒆 + 𝒘𝒇 𝜶𝒇 + 𝒘𝒖 𝜶𝒖
1𝑓𝑙𝑜𝑤 𝑠𝑐𝑜𝑟𝑒: 𝛼𝑓 =𝑁𝑁𝑢𝑖 2 + 𝑣𝑖 2
𝑖=1

𝑎𝑟𝑒𝑎(𝐵𝑑 ∩ 𝐵𝑢 )𝑢𝑝𝑝𝑒𝑟𝑏𝑜𝑑𝑦 𝑠𝑐𝑜𝑟𝑒: 𝛼𝑢 =𝑎𝑟𝑒𝑎 𝐵𝑑
𝑤𝑓 = 0.7, 𝑤𝑢 = 0.2

Hand Detection and Tracking for fine-grained action recognition

Proposed Method
Hand tracking
Select detections which overlap with many others

・・・・・・・・・・・・・

propagateHand Detection and Tracking for fine-grained action recognition

Extraction of Hand Motion Features
Wang et al. Action recognition with improved dense trajectories. CVPR 2013.

Only
hand regions

Handtracks
𝑺′ =∆𝑷𝒕 , … , ∆𝑷𝒕+𝑳−𝟏
𝒕+𝑳−𝟏
𝒋=𝒕∆𝑷𝒋Hand Detection and Tracking for fine-grained action recognition

5.4 Experiments and Results
Data and Evaluation Methods
Exp.
Handdetection

DatasetVideo Pose 2.0• 14 video shots• 2453 frames• 3814 hands

Fine-grained Playing Instruments
action(UCF101)
recognition • 10 types of instruments

Evaluation method• Average precision• Overlap score > 0.5
𝑂=••𝑎𝑟𝑒𝑎 𝐵𝑔 ∩ 𝐵𝑑

𝑎𝑟𝑒𝑎 𝐵𝑔 ∪ 𝐵𝑑

Average precision
3 training/test splits 1• 1428 videos

1 ICCV2013 Workshop on Action Recognition with a Large Number of Classes

Hand Detection and Tracking for fine-grained action recognition

Data Thumbnails

play celloplay dafplay tablaHand Detection and Tracking for fine-grained action recognition

Experiments and Results
Hand Detection Performance
MethodPrecision

Mittal et al.

41.7%

still features

Sapp et al.*18.6%

motion features

Our (upper body)

42.6%

Our (flow)

45.5%

Our (flow+body)

46.3%

still features +motion features* Sapp et al. Parsing human motion with stretchable models. CVPR2011.
(flow based hand detector only)Chapter 5. Hand Detection and Tracking for fine-grained action recognition

Examples of Detection Results
Mittal et al.

Sapp et al.

Our

UCF101
Playinginstruments

Hand Detection and Tracking for fine-grained action recognition

Action recognition performance
MethodPrecision

𝐷𝑇

66.7%

𝐻𝑂𝐹𝑑𝑡

83.8%

𝑀𝐵𝐻𝑑𝑡

86.6%

𝑀𝐵𝐻𝑑𝑡 + 𝐻𝑂𝐹𝑑𝑡 + 𝐷𝑇

87.3%

𝐻𝐷𝑇

66.1%

𝐻𝑂𝐹ℎ𝑑𝑡81.4%

𝑀𝐵𝐻ℎ𝑑𝑡85.7%

𝑀𝐵𝐻ℎ𝑑𝑡 + 𝐻𝑂𝐹ℎ𝑑𝑡 + 𝐻𝐷𝑇

86.2%

𝐻𝑇

36.0%

𝑀𝐵𝐻𝑑𝑡 + 𝐻𝑂𝐹𝑑𝑡 + 𝐷𝑇 + 𝐻𝑇

87.6%

𝑴𝑩𝑯𝒉𝒅𝒕 + 𝑯𝑶𝑭𝒉𝒅𝒕 + 𝑯𝑫𝑻 + 𝑯𝑻

88.5%

DT: Dense Trajectory
HDT: Dense Trajectoryon Hands
HT: Hand Tracks

whole frame(more features)

only handregions

Hand Detection and Tracking for fine-grained action recognition

Conclusions• Proposed a system of hand detection

and tracking in videos• Applied the system to fine-grained

action recognition• Action recognition based on only handmotion features