Automatic Construction of Action Datasets using Web videos withDensity-based Cluster Analysis and Outlier Detection
Nga Do and Keiji Yanai (The University of Electro-Communications, Tokyo)

IntroductionPreviouswork:requireadditionaldata(e.g.:tags[3]),Web videos
ignoreconcept
diversity
problem
Relevant
Action This work: can exploit Web videos without tags,
Shots
Concept Automatic Approach
Action Dataset
copes with concept diversity

Proposed Approachsoccerjuggling

Actionconcept

Video search,
download &segmentation

Shot Clustering

ShotCollection

ii. Video search “verb” & “verb-ing” (dive &diving)iii. Video filtering No videos of “Entertainment”

iv. Video downloading Web API (e.g. Youtube API)v. Shot segmentation
 Color histogram….

with OPTICS[1]i. Word preparation
 “verb” (dive), “verb+non-verb”(throw hammer), “non-verb”(vault)ShotRanking
by LOF

𝒌=𝟒op o𝒓𝒆𝒂𝒄𝒉‐ 𝒅𝒊𝒔𝒕𝒌 𝒑, 𝒐= 𝒌‐ 𝒅𝒊𝒔𝒕(𝒐)𝑵𝒌‐𝒅𝒊𝒔𝒕 𝒐 (𝒐)qA low reachability distanceindicates an object withina cluster.
A high reach-dist indicatesa noise or a jump fromone cluster to another.

ShotSelection......

LOF (Local Outlier Factor) [5]
𝑳𝑶𝑭𝑴𝒊𝒏𝑷𝒕𝒔 𝒑
𝑴𝒊𝒏𝑷𝒕𝒔 − 𝒅𝒊𝒔𝒕 𝒑𝒐𝝐𝑵𝑴𝒊𝒏𝑷𝒕𝒔−𝒅𝒊𝒔𝒕 𝒑 𝒑 𝑴𝒊𝒏𝑷𝒕𝒔 − 𝒅𝒊𝒔𝒕 𝒐=𝑵𝑴𝒊𝒏𝑷𝒕𝒔−𝒅𝒊𝒔𝒕 𝒑 𝒑

Small 𝑴𝒊𝒏𝑷𝒕𝒔 − 𝒅𝒊𝒔𝒕 corresponds to
𝟏. 𝒂𝒕 𝒍𝒆𝒂𝒔𝒕 𝒌 𝒐𝒃𝒋𝒆𝒄𝒕𝒔 𝒒: 𝒅 𝒐, 𝒒 ≤ 𝒅(𝒐, 𝒑)a region with high density.
𝒌‐ 𝒅𝒊𝒔𝒕 𝒐 = 𝒅 𝒐, 𝒑 :2. 𝒂𝒕 𝒎𝒐𝒔𝒕 𝒌 − 𝟏 𝒐𝒃𝒋𝒆𝒄𝒕𝒔 𝒒: 𝒅 𝒐, 𝒒 < 𝒅(𝒐, 𝒑) Shots with low LOF are considered as
relevant
shots
andranked
tothetop.
𝒓𝒆𝒂𝒄𝒉 − 𝒅𝒊𝒔𝒕 𝒑, 𝒐 = 𝒎𝒂𝒙 𝒌‐ 𝒅𝒊𝒔𝒕 𝒐 , 𝒅 𝒑, 𝒐As visual features, we extract motion features using
ConvNet models trained on UCF-101 dataset (split 1)with multi-frame stacking optical flows[4].

Shots are selected from allclusters to guarantee diversity ofselection results.

Experiments and Results
Experiment 1: Dataset Construction• Data: Web videos (YouTube)• Actions: 11 actions in UCF11[2]• Precision rate = percentage of relevant
shots among top 100 shots [3]• Baseline[3]: VisualRank based method

Experiment 2: Action Classification

Action
Proposedbasketball50
biking23
diving35
golf_swing
52
horse_riding
50soccer_juggling68

golf_swing

Baseline
351728544263Actionswing
tennis_swing
trampoline_jumping
volleyball_spiking
walking
Average

Proposed
36
47
54
58
1444.3Baseline
31515469
941.1

horse_riding

Classification Rate (%)• Dataset: UCF11[2]• Precision = average of 25-fold validation• Training data: standard data[2] & shots
automatically obtained in Experiment 145

With standard training data [2]: 81.5%

4035
Proposed
Baseline

30
2520
Top30

Top50
Top100
Selected Shots
Proposed
Baseline
Proposed
Baseline[1] Mihael et al. OPTICS: Ordering Points To Identify the Clustering Structure. ACM SIGMOD International Conference on Management of Data, 1999, pp. 49-60.
[2] Jingen et al. Recognizing realistic actions from videos. IEEE Computer Vision and Pattern Recognition, 2009, pp. 1996-2003.
[3] Nga et al. Automatic Construction of an Action Video Shot Database using Web Videos. IEEE International Conference on Computer Vision, 2011, pp. 527-534.
[4] Karen et al. Two-Stream Convolutional Networks for Action Recognition in Videos. Advances in Neural Information Processing Systems 27, 2014, pp. 568-576.
[5] Chiu et al. Enhancements on local outlier detection. IEEE Database Engineering and Applications Symposium, 2003, pp. 298 – 307.