Greater Tokyo Area
Multimedia/Vision Workshop

Automatic Collection of Web Video ShotsCorresponding to Specific Actions

Do Hang NgaKeiji Yanai

The University of Electro-Communications
Tokyo, Japan

Greater Tokyo Area
Multimedia/Vision WSReferences• Do Hang Nga and Keiji Yanai: AutomaticConstruction of an Action Video Shot Databaseusing Web Videos. ICCV. Nov. 2011. pp 527-534• Do Hang Nga and Keiji Yanai: Automatic Collection ofWeb Video Shots Corresponding to Specific Actions
using Web Images. CVPR Workshop on Large-Scale
Video Search and Mining (LSVSM). Jun. 2012.
• Demo page:
http://mm.cs.uec.ac.jp/webvideo/Greater Tokyo Area
Multimedia/Vision WS

Outline• Motivation• Objective• Related work
• Proposed Method• Experiments & Results• Conclusion & Future works

Greater Tokyo Area
Multimedia/Vision WSOutline• Motivation• Objective• Related Work
• Proposed Method• Experiments & Results• Conclusion & Future works

Greater Tokyo Area
Multimedia/Vision WSMotivation• Web data source: huge + free, but noisy
ActionKeywordBunch ofVideos

Many are irrelevant/ partly relevant

Surfing wave

• Web videos based action database
construction: extremely time-consuming work

Greater Tokyo Area
Multimedia/Vision WS

Objective
Unsupervised construction of an action video database

Surfwave

Automatic extraction ofrelevant video shotsfrom the Web videos

Greater Tokyo Area
Multimedia/Vision WS

Examples of Results
Dancing+flamenco

Surfing+wave

Related work

Greater Tokyo Area
Multimedia/Vision WS

N. I. Cinbis, R. G. Cinbis and S. Sclaroff:“Learning actions from the web”, ICCV2009
Cinbis et.al.
Web images
Static features

Proposed method

Web videos＋images
Spatio-temporal features
Metadata (Tags)

Greater Tokyo Area
Multimedia/Vision WSOutline• Motivation• Objective• Related work• Proposed Method• Experiments & Results• Conclusion• Future works

Greater Tokyo Area
Multimedia/Vision WS

Proposed Method
Step 1: Tag-based
video selection
ActionKeyword

WebVideos

Step 2: Feature-based
shot ranking

Relevant
VideosRelevant
Video Shots

2 steps Unsupervised Method

 Experimented 100 kinds of human actions

e.g. Grill fish
Ride bicycle

Greater Tokyo Area
Multimedia/Vision WS

Tag-Based Video Selection
WEBWebAPI

Tags’ frequencies& co-frequencies

TAGSQuery：running+marathon

Tag Relevance
ScoresTag
Relevance Score
Run0.18248175
Training 0.13321168…….……………………

Tag-based
Video Selection

Videos with manyhighly scored tags
Relevant
VideosSelect Top 200
to download

Greater Tokyo Area Multimedia/Vision WSFeature-based shot ranking
Downloaded VideosVideo
Segmentation
Shot 1
Bag of Features
Representation

Colorhistogram
…..

……...

Shot nSpatio-Temporal
Feature Extraction

Histogram Intersection

Similarity Matrix
Calculation

VisualRank
Calculation

Relevant Shots (Top Ranked Shots)

A. Noguchi and K. Yanai.
A SURF-based spatiotemporal feature forfeature-fusion-based action
recognition. ECCV2010
WS on Human Motion.
𝐻𝑠 𝐻1 , 𝐻2 =𝑚𝑖𝑛 𝐻1𝑖 , 𝐻2𝑖𝑖=1

Greater Tokyo Area
Multimedia/Vision WS

Shot ranking by VisualRank• VisualRank(*):• How to calculate∗𝑟 = 𝑑𝑆 𝑟 + 1 − 𝑑 𝑝

Damping vector(*) Y. Jing and S. Baluja. Visualrank: Applying pagerank to large-scale

image search. PAMI, 30(11):1870–1890, 2008.

Greater Tokyo Area
Multimedia/Vision WS

Shot ranking by VisualRank• Image ranking by VisualRank:

1𝑟 = 𝑑𝑆 𝑟 + 1 − 𝑑 𝑝, 𝑤ℎ𝑒𝑟𝑒 𝑝 =𝑛 𝑛×1∗• Our method: damping value is high if shot is from videowith high tag scores𝑝𝑖 =10,𝑘𝑖≤𝑘𝑖>𝑘𝒌 𝒕𝒐𝒑 𝒔𝒄𝒐𝒓𝒆 𝒔𝒉𝒐𝒕𝒔
𝑘~1000Greater Tokyo Area
Multimedia/Vision WS

Experiments & Results
Experimented 100 kinds of human actions
36.6% prec@100 for 100 actions
79.8% for the best 10

e.g. Painting picture
Playing pianoCounting money
Riding bicycle• How to evaluate results
prec@100 = number of relevant shots in top 100 shots

Greater Tokyo Area
Multimedia/Vision WSResults
See relevant shot extraction resultsof 100 actions at

http://mm.cs.uec.ac.jp/webvideo/

Greater Tokyo Area
Multimedia/Vision WS

Improve Shot Ranking Step
by Exploiting Web Images• Idea : Bias shots with many scenes similar torelated images𝑟 = 𝑑𝑆 ∗ 𝑟 + 1 − 𝑑 𝑝Playtrumpet

Video Shots

Action Images

Greater Tokyo Area
Multimedia/Vision WS

Introducing Web Images
Relevant
VideosWebVideos

ActionKeywordWebImages

Images based
shot ranking

ActionImagesHuman detection
based image selection

Relevant
Video Shots

Shot-Images
similarity calculation

Extended part

Greater Tokyo Area
Multimedia/Vision WSAction Image Collection
ActionKeywordWebImages

Poselets(*) based
Human Detection

ActionImages(*) Lubomir Bourdev, Jitendra Malik, Poselets: Body Parts Detectors

Trained using 3D Human Pose Annotations, ICCV 2009

Greater Tokyo Area
Multimedia/Vision WS

Shot-Images similarity based
Shot Ranking
ActionImages

SURF extracted
from images

SURF extracted
from shots𝑁

𝑆𝐼 𝑆𝑖 =

max 𝑆𝐼 𝐹𝑗 |𝐼𝑘

𝑗=1:𝑀

Feature matching𝑘=1
2∗𝑀𝑎𝑡𝑐ℎ𝑃𝑜𝑖𝑛𝑡 𝐹𝑗 ,𝐼𝑘

𝑆𝑖 𝐹𝑗 |𝐼𝑘 =𝑃𝑜𝑖𝑛𝑡 𝐹𝑗 +𝑃𝑜𝑖𝑛𝑡 𝐼𝑘

Shot-Images Similarity Scores

Bias shots with highsimilarity scores𝐞𝐱𝐩(𝜸 𝑺 𝒊 )𝒑𝒊 = 𝒏𝛾 = 𝑙𝑜𝑔2
𝒋=𝟏 𝒆𝒙𝒑(𝜸 𝑺 𝒋 )Shot Ranking

Greater Tokyo Area
Multimedia/Vision WS

Efficiency of introducing Web Images• Dataset: failed categories in the previous work
- 28 human action categories (Prec@100 < 20%)
- 8 non-human action categories (Prec@100 < 15%)

• Evaluation: percentage of relevant shots over 100
top ranked shots (Precision@100)• Results:- human actions:

10.1% → 16.3% ( 6.2% )- non-human actions: 2% → 18.6% (16.6% )Improved categories

Greater Tokyo Area
Multimedia/Vision WS

Top 5 actions in terms of improvement

Swim+butterfly(1): previouswork (2): this workActions
swim+butterfly(1)7(2) gain
31 +24serve+volleyball731+24

grill+fish

526+21squat

1932+13

bake+bread

618+12Previous work

This work

Greater Tokyo Area
Multimedia/Vision WS

Degraded categories
Worst 5 actions in terms of improvement(1): previous work (2): this work
Actions(1)(2) gainslap+face

20

13-7

wash+clothes

15

10-5

drink+coffee

14

9-5

boil+egg

96-3

slice+apple

52-3

Greater Tokyo Area
Multimedia/Vision WS

Why worsen some categories?
(1) human-detection-based image selection
selects very few relevant images

Top selected Web Images for ‘slap+face’Greater Tokyo Area
Multimedia/Vision WS

Why worsen some categories?
(2) shots-images similarity calculation method is
not effective- gaps between selected images and downloaded videos

Selected Web Images (washing+clothes) Downloaded Videos

Greater Tokyo Area
Multimedia/Vision WSConclusion & Future works• Proposed a framework of automaticallyextracting action video shots from the Web• Applied Web action images to improve theframework• Future works:- improve video selection step- try more features

serve+tennis

shoot+arrowsnow+falling

airplane+flying