Ａｕｔｏｍａｔｉｃ Construction of an Action Video Shot Database using Web Videos
DO HANG NGA and KEIJI YANAIThe University of Electro-Communications, Tokyo, Japan
Summary

Tag-based Video Selection• A novel method to collect “Action” video shots from the Web• Fully-automatic and unsupervised• Only providing “Action keywords” such as “paint+picture” at first• Tag-based video selection and ST-feature-based shot selectionWebAPI• Large-scale experiments for 100 kinds of human actions• 36.6% prec@100 for 100 actions, 79.8% for the top 10 actions

Introduction & Objective
Many are irrelevant/ partly relevant
ActionKeywordBunch ofVideos

Counting
tag frequencies &co-frequencies

WE
B

Query：
running+marathon

Tag Relevance
Score Table [1]Tag listsof 1000videos

Tag
Relevance Score
Run0.18248175
Training 0.13321168…….……………………

Select the top
200 videos

Video relevance
scoresactionkeywordregarding thevideo scoresfor 1000 videos

Download thetop 200 videos

batting

TagBiased
VisualMean
Exp No. based
damp.
Feature prec@100
Ranking
vec.
RND Randomly-selected 100 shots 14.2%

jumping trampoline

shooting football

How to Calculate Video Relevance Scores

Objective

𝑐𝑜−𝑓𝑟𝑒𝑞 𝑎, 𝑏
𝑃 𝑏𝑎 =𝑓𝑟𝑒𝑞 𝑏

Unsupervised
MethodRelevant
Video Shots

Calculate tag-based “video relevance scores”
for all the 2000 videos regarding one keyword

1𝑆 𝑉𝑡 =𝑛Overview of Proposed Method

Downloaded Videos
VideoSegmentation

𝑙𝑜𝑔𝑃 𝑡𝑖 |𝑡𝑡𝑖 ∈𝑇𝑣

𝑡: given keyword
𝑇𝑣 ={all the tags except 𝑡in Video 𝑉}

Tag-based
Video Selection

Feature-based
Shot Selection
Relevant
Shots

Relevant
VideosShot 1…..

……...

Bag of Features
Representation

Bias the computation① Spatio-Temporal Feature(ST)[2]② Motion Feature (MO)③ Appearance Feature (AP)
④ Fusion of the above 3 features

𝟏(1)

(2)

Similarity Matrix
Calculation

Shotsegmentation

Shot ranking𝒓 = 𝒅𝑺 𝒓 + 𝟏 − 𝒅 𝒑

Feature Extraction

Histogram Intersection

Video ranking∗Shot n2 steps Unsupervised Method

surf+wave

VisualRank [3]Colorhistogram“surf wave” shots

VisualRank
Calculation

Relevant Shots (Top Ranked Shots)

𝒑𝒊 =𝟏
≤𝒊
≤
𝒎𝒎

𝟎 𝒎<𝒊≤𝒏𝑛 ≈ 2000, 𝑚 = 1000

𝒑𝒊 =𝑺𝒄 𝒋𝑪walking street

TAG

✔

－－

23.5%

123(1)
3(2)
567－✔
✔
✔
✔
✔
✔

－－✔(1)
✔(2)
✔(1)
✔(1)
✔(1)

STSTSTSTMotion
Appear.
Fusion

33.7%
41.0%
47.3%
44.8%
31.8%
39.7%

49.5%

(2) Large-scale 100-action experiments
Table2:2:Prec@100
Prec@100ofofTop
Top60
60inin100
100human
humanactions
actionsResults
Results(%)
(%)
Table

Feature-based Shot Selection“surf wave” videos on

Table 1: The conditions of the experiments andthe results (Mean Precision@100 of 6 actions)running marathon

Construct “a tag relevance score table” bycounting tag frequencies and co-frequencies

WebVideos(1) Six-action experiments under various conditions

eating ramen

Gathering no video files but
only tag lists for 1000 videos

Surfingwave

Web videos
of given keyword

Experiments & Results, 𝑪=𝒏
𝒋=𝟏 𝑺𝒄 𝒋

𝑆𝑐 𝑗 ∶ tag relevance score of videofrom which shot j was extracted

Combine the 3 features
𝑺∗𝐀𝐋𝐋 = 𝒘𝑺𝑻 𝑺∗𝑺𝑻 + 𝒘𝑴𝑶 𝑺∗𝑴𝑶 + 𝒘𝑨𝑷 𝑺∗𝑨𝑷
11𝑤𝑆𝑇 = , 𝑤𝑀𝑂 = 𝑤𝐴𝑃 =24

Actionsoccer+dribble
fold+origami
crochet+hat
arrange+flower
paint+picture
boxingcomb+hairparachute+jump
do+exercise
do+aerobics
do+yogaserve+volleyballsurf+wave
shoot+arrowfix+tirebasketball+dribble
blow-dry+hair
ride+bicycle
bowl+ballcurl+bicep

Prec
Action100
shoot+football
96
tie+shoelace
95
laugh94
dive+sea88
harvest+rice
86
ski
83
iron+clothes
82
twist+crunch
79 dance+flamenco
78
dance+hiphop
77
eat+ramen
75
dance+tango
75
play+trumpet
73
play+drum
67
skate
64
swim+crawl
64
cut+hair62
run+marathon
58
count+money
58
paint+wall

Prec
58
57
50
49
49
49
47
47
45
43
42
41
41
40
37
36
35
35
33
33

Actionshoot+football
draw+eyebrowsfieldhockey+dribble
hit+golfballlunge
play+pianorow+boatsing
chat+friend
clean+floor
cut+onion
shave+mustache
pick+lockplaster+wall
blow+candle
wash+facewalking+street
brush+teeth
catch+fish
drive+car

Prec3332323232323232313131313030292929282828

See 100 action shot results at

http://mm.cs.uec.ac.jp/webvideo/[1]Q.Yang, X.Chen and G.Wang: “We2.0 Dictionary”, Proc. of ACM International Conference on Imageand Video Retrieval (CIVR), p.591-600, 2008.
[2]A.Noguchi and K.Yanai: “A SURF-based Spatio-Temporal Feature for feature-fusion-based
action recognition”, Proc. Of ECCV WS on Human Motion: Understanding, Modeling, Capture andAnimation, 2010.
[3]Y. Jing and S. Baluja: “Visualrank: Applying pagerank to large-scale image search”, IEEE Trans.
on Pattern Analysis and Machine Intelligence, Vol. 30, No. 11, pp. 1870–1890, 2008.