CVPR Workshop 2012:
Large-Scale Video Search and Mining

Automatic Collection of Web Video Shots
Corresponding to Specific Actions
using Web Images
Do Hang Nga

Keiji Yanai

The University of Electro-Communications
Tokyo, Japan

Outline• Motivation & Objective• Contributions• Related work
• Previous work
• This work
• Experiments & Results• Conclusion & Future works

Outline• Motivation & Objective• Contributions• Related work
• Previous work
• This work
• Experiments & Results• Conclusion & Future works

Motivation & Objective• Web data source: huge + free, but noisy• Web videos based action database
construction: extremely time-consuming work

Surfwave

Automatic extraction ofrelevant video shots
fromf the Web videos

Unsupervised
construction of an action
video database
Play trumpet
Tie shoelaceContributions• Refine our previous approach [ICCV11]
by introducing the use of Web action images- propose to select relevant shots based on theirsimilarities with action Web images• Improve results for the failed action categories- 28 human actions: 6%- 8 non-human actions: 16%[ICCV11] Do Hang Nga and Keiji Yanai: Automatic Construction of anAction Video Shot Database using Web Videos. ICCV2011.527-534.

Related work
N. I. Cinbis, R. G. Cinbis and S. Sclaroff:“Learning actions from the web”, ICCV2009
Cinbis et.al.

Web images
Static features

Proposed method

Web videos＋images
Spatio-temporal features

Outline• Objective & Motivation• Contributions• Related work
• Previous Work & its Problems• This work
• Experiments & Results• Conclusion & Future works

Previous work
Step 1: Tag-based
video selection
ActionKeywordWebVideos

Step 2: Feature-based
shot ranking

Relevant
VideosRelevantVideo Shots2 steps Unsupervised Method

 Experimented 100 kinds of human actions

e.g. Grill fish
Ride bicycleTag-Based Video Selection
WEB

Web
API

Tags’ frequencies& co-frequencies

TAGS

Query：
running+marathon

Tag Relevance
Scores Table

Tag
Relevance Score
Run
0.18248175
Training 0.13321168…….……………………

Tag-based
Video Selection

Videos with manyhighly scored tags
Relevant
VideosSelect Top 200
to download

Feature-based shot ranking
Downloaded Videos
Video
Segmentation
Shot 1
Bag of Features
Representation

Colorhistogram
…..

……...

Shot nSpatio-Temporal
Feature Extraction

Histogram Intersection

Similarity Matrix
Calculation

VisualRank
Calculation

Relevant Shots (Top Ranked Shots)

A. Noguchi and K. Yanai.
A SURF-based spatiotemporal feature forfeature-fusion-based action
recognition. ECCV2010
WS on Human Motion.
𝐻𝑠 𝐻1 , 𝐻2 =𝑚𝑖𝑛 𝐻1𝑖 , 𝐻2𝑖
𝑖=1Feature-based shot ranking• Shot ranking by VisualRank(*):
1𝑟 = 𝑑𝑆 𝑟 + 1 − 𝑑 𝑝, 𝑤ℎ𝑒𝑟𝑒 𝑝 =𝑛 𝑛×1∗• Our previous work: Bias shots from highly scored videos
𝑝𝑖 =10,𝑘𝑖≤𝑘𝑖>𝑘(*) Y. Jing and S. Baluja. Visualrank: Applying pagerank to large-scale

image search. PAMI, 30(11):1870–1890, 2008.

Problems
Exploit only Web Videos and their metadata (tags)→ Failed in some categories due to noisy tags
[Comedy] <Take This Pill>
health; reminder; reform; cartoon;
comedy; flash; animation; funny;satire; salad; fingers; hospital; nhs;
doctor; nurse; healthcare;treatment; medicine; medication;
pills; tablets; cure; remedies;humour; humor; tumor; private;public; care

The top selected video for ‘take+medicine’ and its tagsOutline• Motivation & Objective• Contributions• Related work
• Previous work
• This work
- Overview of this work
- The introduction of Web Images- Image collection- Shot-Images similarity based shot ranking• Experiments & Results• Conclusion & Future Works

Overview of this work
Relevant
VideosWebVideos

ActionKeywordWebImages

Images based
shot ranking

ActionImagesHuman detection
based image selection

Relevant
Video Shots

Shot-Images
similarity calculation

Extended partAction Image Collection
ActionKeywordWebImages

Poselets(*) based
Human Detection

ActionImages(*) Lubomir Bourdev, Jitendra Malik, Poselets: Body Parts Detectors

Trained using 3D Human Pose Annotations, ICCV 2009

Shot-Images similarity based
Shot Ranking
ActionImages

SURF extracted
from images

SURF extracted
from shots𝑁

𝑆𝐼 𝑆𝑖 =

max 𝑆𝐼 𝐹𝑗 |𝐼𝑘

𝑗=1:𝑀

Feature matching𝑘=1
2∗𝑀𝑎𝑡𝑐ℎ𝑃𝑜𝑖𝑛𝑡 𝐹𝑗 ,𝐼𝑘

𝑆𝑖 𝐹𝑗 |𝐼𝑘 =𝑃𝑜𝑖𝑛𝑡 𝐹𝑗 +𝑃𝑜𝑖𝑛𝑡 𝐼𝑘

Shot-Images Similarity Scores

Bias shots with highsimilarity scores
𝑝𝑖 =exp(𝛾 S i )𝛾 = 𝑙𝑜𝑔2𝑛
𝑗=1 exp(𝛾 S 𝑗 )Shot Ranking

Outline• Objective & Motivation• Contributions• Related work
• Previous Work & its Problems• This work
• Experiments and Results• Conclusion & Future works

Experiments & Results• Dataset: failed categories in the previous work
- 28 human action categories (Prec@100 < 20%)
- 8 non-human action categories (Prec@100 < 15%)

• Evaluation: percentage of relevant shots over 100
top ranked shots (Precision@100)• Results:- human actions:

10.1% → 16.3% ( 6.2% )- non-human actions: 2% → 18.6% (16.6% )Improved categories
Top 5 actions in terms of improvement

Swim+butterfly(1): previouswork (2): this work

Actions
swim+butterfly(1)7(2) gain
31 +24serve+volleyball731+24

grill+fish

526+21squat

19

32+13

bake+bread

618+12Previous work

This work

Degraded categories
Worst 5 actions in terms of improvement(1): previous work (2): this work
Actions(1)(2) gainslap+face

20

13-7

wash+clothes

15

10-5

drink+coffee

14

9-5

boil+egg

96-3

slice+apple

52-3

Why worsen some categories?
(1) human-detection-based image selection
selects very few relevant images

Top selected Web Images for ‘slap+face’Why worsen some categories?
(2) shots-images similarity calculation method is
not effective- gaps between selected images and downloaded videos

Selected Web Images (washing+clothes) Downloaded Videos

Conclusion & Future works• Apply Web action images to the problem ofautomatically extracting action video shots• Promising results show effectiveness of ourmodifications• Future works:- improve video selection step- try more features- apply cross-domain learning

serve+tennis

shoot+arrowsnow+falling

airplane+flying