Automatic Construction of an Action Video Shot Database using Web Videos
Do Hang Nga

Keiji Yanai

Department of Informatics, The University of Electro-Communications, Tokyo
1-5-1 Chogugaoka, Chofu-shi Tokyo 182-8585 Japandohang@mm.cs.uec.ac.jp, yanai@cs.uec.ac.jp

Abstract

most relevant video shots to given keywords from a large
number of tagged Web videos, which requires no supervision and just only providing keywords at the beginning. As
keywords, we mainly focus on the words related to humanaction such as “eating” and “running”. To this end, we use
a state-of-the-art spatial-temporal feature [24] to represent
each video shot. Note that video shots mean small fragments of a video obtained by dividing the given video at the
points of scene change or camera change.
If video shots corresponding to any “action” verbs canbe obtained automatically, we can build training data on
human action recognition for unconstrained videos easily.
So far, constructing of action training data is very expensive, since collecting only video sequences corresponding
to a speciﬁc action is very time-consuming, which is totally different from the situation of ﬁnding still images corresponding to a speciﬁc object. In fact, the largest action
dataset used commonly so far includes only 14 kinds of
categories [27]. On the other hand, we can gather video
shots associated with unlimited kinds of actions by using
the proposed method, although brief cleaning by hand is
still needed to use them as training data for action recognition. Our ﬁnal objective is automatic construction of anaction video shot database which is helpful for the researchcommunity on action recognition.
In the proposed method, ﬁrstly, we rank the 1000 videos
which have the given tags based on tag co-occurrence evaluation after obtaining video IDs and their tag lists via Web
API. Secondly, we download the top 200 video in terms oftag co-occurrence scores and segment all the downloaded
videos into video shots, and thirdly apply a graph-basedranking method, VisualRank [13], to rank video shots by
taking account of visual features of video shots and tag cooccurrence scores so that shots corresponding to the given
keywords are ranked higher.
To summarize our contribution in this paper, it consistsof three-fold: (1) fully-unsupervised construction of an action video shot database, (2) two-step video shot selection
consisting of tag-based video selection from large numberof tagged videos, and visual-feature-based shot selection
with the state-of-the-art spatio-temporal feature, and (3) alarge-scale experiments on 100 kinds of actions with video
metadata analysis on 100,000 YouTube videos and spatiotemporal feature analysis on 20,000 YouTube videos.
In the rest of this paper, we describe related work in
Section 2. Then in Section 3, we explain the overview ofthe proposed method, and in Section 4 we explain the de-There are a huge number of videos with text tags on the
Web nowadays. In this paper, we propose a method of automatically extracting from Web videos video shots corresponding to speciﬁc actions with just only providing action
keywords such as “walking” and “eating”.
The proposed method consists of three steps: (1) tagbased video selection, (2) segmenting videos into shots andextracting features from the shots, and (3) visual-featurebased video shot selection with tag-based scores taken into
account. Firstly, we gather video IDs and tag lists for 1000
Web videos corresponding to given keywords via Web API,and we calculate tag relevance scores for each video using
a tag-co-occurrence dictionary which is constructed in advance. Secondly, we fetch the top 200 videos from the Web
in the descending order of the tag relevance scores, and segment each downloaded video into several shots. From eachshot we extract spatio-temporal features, global motion features and appearance features, and convert them into thebag-of-features representation. Finally, we apply the VisualRank method to select the video shots which describe
the actions corresponding to the given keywords best after
calculating a similarity matrix between video shots. In theexperiments, we achieved the 49.5% precision at 100 shots
over six kinds of human actions by just providing keywords
without any supervision. In addition, we made large-scale
experiments on 100 kinds of action keywords.

1. Introduction
A huge number of videos have been stored at video sharing sites on the Web such as YouTube and DailyMotion,and many videos are being uploaded to them every second. When people uploads their videos, they usually attach
to the videos text keywords called as “tags” which enable
other people to search for the uploaded videos by keywordbased search. However, in general, tags are attached to awhole video sequence of each video. Therefore, if attachedtags correspond to only a speciﬁc part of the video, it is unknown which part of the video corresponds to the tags. For
example, some videos which has “eating” as a tag might
include the scenes of entering restaurants, ordering meals,
and drinking coffee after eating. People who want to watch
only “eating something” scenes have to skip the scenes of
no interest, and search for eating scenes manually. This is avery troublesome and time-consuming task.
Then, in this paper, we propose a new method to detect1tail of each processing including tag-based ranking, visualfeature-based ranking and the visual feature representation.
Section 5 describes the experimental results. Finally weconclude this paper in Section 6.

2. Related WorkIn this section, we refer to some related works on action
recognition, Web image mining, and tag ranking methods.
Action recognition: For these ﬁve years, spatiotemporal (ST) features and their bag-of-features (BoF) representation have drawn attention for human action recognition and content-based video analysis, since by using themaction recognition problem can be regarded as being almost
the same problem as object recognition except feature extraction methods. Until two or three years ago, most of theworks on action recognition focused on controlled video
data such as KTH dataset [26] and Weizmann dataset [2]the videos of which are taken by a ﬁxed camera with uniform backgrounds. Recently some works dealt with uncontrolled video such as “in-the-wild” YouTube dataset [19]and Hollywood action dataset [21], since classiﬁcation
rates on KTH and Weizmann has reached nearly perfectrecognition rate, 95.5% and 100%, respectively [17]. In
2010, more works focused uncontrolled video categorization for YouTube videos [29, 28] and Kodak consumer
video dataset [7]. Most of these works aimed to categorize
whole videos into one of the pre-deﬁned categories, while
our objective is to search a large number of Web videos forpart of videos associated with the given keywords.
In a few works, unsupervised methods were attempted
for action recognition. Niebles et al. [23] categorized action videos in KTH datasets and their original ice-skating
video data using the PLSA model. Niebles et al. [22] also
proposed a method to extract human action sequences fromunconstrained Web videos. Cinbis et al. [4] proposed amethod to learn action models automatically from Web images gathered via Web image search engines, and recognize
action for the same video dataset as [22]. Although Cinbis et al.’s work is the most similar to our work, they used
Web images as a training source and only static features as
an action descriptor, while we use Web videos and spatiotemporal features. In addition, in both works by Niebles
et al. and Cinbis et al. a people detector based on HOG(Histogram of Oriented Gradient) [6] were used to extracta region of a human body, which restricts kinds of actions,while our method does not limit applicable actions to only
human actions and it might be able to collect non-humanactions such as “airplane-ﬂying” and “tornado”. As anothersimilar work, Ballan et al. [1] proposed a method to add tags
to video shots by using Web images obtained from Flickr as
training samples. Meanwhile, Laptev et al. [14, 21, 8] proposed methods to associate movies and movie scripts automatically. These methods enable us to build an action shot
database in a unsupervised manner, although target videos
are limited to only the movies the scripts of which are available.
Web image mining: Regarding still images, many works
on image gathering from the Web to build an image

database automatically has been proposed so far [30, 10,
9, 31, 16, 25] Most of these works employed object recognition methods to select relevant images to given keywords
from “raw” images collected from the Web using Web image search engines. Now we can import this idea to action
video recognition domain by using the BoF representationof video shots. This is our initial motivation of this work,and our work can be regarded as being video shot versionof these automatic Web image gathering.
Tag ranking: In this paper, we perform tag analysis tocompute tag-based relevance scores. Having tags is common characteristic of consumer generated media (CGM)
data on the Web. To have uploaded contents searched for bymany people, adding tags to the uploaded contents is essential. Therefore, in general, images and videos on the Web
have more than two tags, or sometimes more than ten tags.
Yang et al. [32] proposed a method to evaluate a tag relevance score on each tag based on tag co-occurrence statistics which is called as “Web 2.0 Dictionary” in the paper.
Since this method requires only tag analysis and no visualfeature analysis, we use this to select Web videos to download. As a similar method which does not require visualfeatures, Dong et al. [18] proposed a method to evaluate tag
relevance score by combining the probabilistic relevance
score estimation and random walk-based reﬁnement. Although this two-step method is similar to ours, they use only
tag information, while you use tag and visual features.

3. Overview of the Methods
In this paper, we propose a new method of automatically extracting from tagged Web videos video shots corresponding to speciﬁc actions with just only inputting action
keywords such as “walking” and “eating”. The proposed
method consists of three processing steps (Figure 1): (1)tag-based video selection, (2) segmenting videos into shotsand extracting features from all the shots, and (3) visualfeature-based video shot selection with tag-based scores
taken into account.
In the ﬁrst step, we evaluate a relevance score of each
video to the given keywords before downloading videos
from the Web, and select more relevant videos to be downloaded from a large number of videos which have the given
keywords as the tags, since we evaluate the score by using only tag co-occurrence statistics without visual features.
Note that a list of tags and video IDs corresponding to thegiven query words can be obtained via video search Web
API ofﬁcially provided by Web video sharing sites.
After downloading the videos, we divide each of theminto video shots and extract visual features from each of all
the shots. In this paper, as visual features extracted fromeach video shot, we use the spatio-temporal (ST) features
proposed by Noguchi et al. [24], global motion features,
Gabor appearance features, and their fusion.
In the third step, we rank video shots by applying graphbased ranking method, VisualRank [13], with a visualfeature-based similarity matrix and a bias damping vector
based on tag-based video relevance scores. Finally we canobtain video shots corresponding to the given keywords infor V and w(t, ti ) for P (ti |t) as follows:
P (V |t) ∝ P (TV |t)∏=
P (ti |t)ti ∈TV=∏w(t, ti )(2)

ti ∈TV

Figure 1. Overview of the proposed method.

the upper rank of the video shot ranking. Note that video
shots are ranked in the third step, while whole videos areranked in the ﬁrst step.

4. Methods
In this section, we describe the detail on tag-based video
ranking, feature extraction from shots and visual-featurebased video shot ranking.

4.1. Tag-based Video Ranking
We can easily obtain Web videos associated with thegiven keywords by using Web API. In case of YouTube,
they provide Web API to search their video database for thevideos tagged with the given query words. However, tags
are sometimes only weakly related or unrelated to the corresponding videos, since tags are assigned subjectively by
the uploaders. The objective of this step is to select the morequery-related videos to download.
Firstly, we send the given keywords to the Web API of
Web video sharing sites, and we obtain sets of video IDs
and tags. Using co-occurrence of tags, we evaluate relevance of the video to the given keyword. To this end, we
use the “Web 2.0 Dictionary” method proposed by Yang etal. [32]. “Web 2.0 Dictionary” corresponds to statistics on
tag co-occurrence, which we need to construct in advanceby gathering a large number of video tags from the Web.
Assume that N (t) is the number of the videos taggedwith word t among all the Web videos, and T is a set of
all the words other than t over all the the Web videos. The
correlation of parent word t and its child word ti ∈ T isdeﬁned as
F (t, ti )(1)w(t, ti ) =N (t)where F (t, ti ) is the number of videos tagged with bothword t and word ti at the same time. When TV represents a
set of all the words other than t in video V , we can estimate
relevance of video V for word t, P (V |t), by substituting TV

This is the original method to calculate relevance of an image/video to the give keyword as described in [32]. Thisis based on an idea that other tags than the query tag aresupporters of the query tag, and the query tag can be regarded as being more relevant to the video when the query
tag is supported by many supporter tags strongly related tothe query word.
However, due to multiplying of all the correlation values
between the query tag and the rest of tags within one video,
the value of Eq.(2) becomes smaller as the number of tags
increases. To prevent this, we modify it so that the numberof co-occurrence words to be used for calculation is limited
to m at most, and deﬁne the relevance score Sct (V ) using
average log likelihood as follows:
1 ∑S(V |t) =log2 w(t, ti )nti ∈T 01 ∑=(log2 F (t, ti ) − log2 N (t))
nti ∈T 01 ∑log2 F (t, ti ) − log2 N (t)=(3)
nti ∈T 01 ∑Sct (V ) =log2 F (t, ti )
(4)
n
0
ti ∈T

where T 0 contains at most the top m word ti in terms ofthe descending order of w(t, ti ), and n (n ≤ m) represents|T 0 |. Since the second term of Eq.(3) is always the same in
the video set over the same action keyword, we omit it anddeﬁne the relevance score Sct (V ) as shown in Eq.(4). In theexperiment, we set m as 10, and select more relevant 200
videos to the given keyword from the 1000 videos returnedby the Web API. This tag-based selection in the ﬁrst step isimportant to feed promising videos to the second step whichcontains much more expensive feature extraction process.
Note that in case of compound keywords such as “drink
coffee”, we regard compound keywords as just one word
and N (t) as the number of the videos including all of theelement word of the compound keyword in their tag sets.
w(t, ti ) is regarded as the number of videos having all thewords of t and ti even if ti is also a compound word. For avideo having no co-occurrence tags, we ignore such video
in the experiments since we cannot calculate the relevance
score.
In the experiments, as seed words, we prepared 150 setsof verbs and nouns which are related to actions such as “ridebicycle” and “launch shuttle”. We gathered 1000 video tagsfor each seed word, and extracted all the tags. As a result,
we obtained 12,471 tags which appear more than ﬁve times
among all the collected tags. For each of 12,471 words,

we gathered 1000 video tags again, and constructed “Web
2.0 Dictionary” by counting tag co-frequencies according
to Eq.(1).

4.2. Visual-feature-based Shot Ranking
After downloading the top 200 videos from the Web
in the descending order of the relevance score estimatedby the tag-based ranking method, we segment the downloaded videos into shots by a simple shot boundary detection method by thresholding color-histogram distances between adjacent frames.
Before extracting visual features, we select video shotsfrom all the shots extracted from 200 videos, since thenumber of all the shots sometimes exceeds 10,000 andthe total time of all the shots exceeds ﬁfteen hours. To
make computational cost feasible, in the experiment, we
set the upper limit number of shots extracted from one
video, and selected only the 2000 shots according to thefollowing heuristic which intends to balance selecting more
shots from the higher-ranked videos against selecting various shots from as many videos as possible.
Nupper (Vi ) = c × Sc(Vi ) + f (N (Vi ))(5){20(x ≤ 20)
20 + (x − 20)/4 (20 < x < 100)
where f (x) =40(x ≤ 100)
and, Nupper (Vi ) and N (Vi ) represents the limit number ofshots and the number shots extracted from the i-th video,
respectively. Sc(Vi ) represents a tag-based relevance scoreof the i-th video. c is a constant which depends on the sizeof the “Web2.0 dictionary”. In the experiment, we set c as
10. Basically we took into account both the number of shots
detected by shot boundary detection and the tag relevance
score of the video. We select Nupper (Vi ) shots at most fromthe i-th video at even intervals, and aggregate 2000 shots inthe descending order of the tag relevance score Sc(V ).
After selecting shots to feed into visual-feature-basedranking, we extract visual features including spatiotemporal (ST) features, global motion features and appearance features from all the shots. The detail on the visualfeatures will be explained in the next subsection.
As a method on visual-feature-based shot ranking, we
employ the VisualRank method [13], which is an image
ranking method based on the widely known Web page raking method, PageRank [3]. PageRank calculates ranking
of Web pages using hyper-link structure of the Web. The
rank values are estimated as the steady state distribution of
the random-walk Markov-chain probabilistic model. In theiterative processing, each page gives out ranking points to
its hyperlink destinations. Therefore, the ranking point of
the page linked by more pages having much ranking points
becomes higher. VisualRank uses a similarity matrix of images instead of hyper-link structure. Eq.(6) represents anequation to compute VisualRank.
r = αSr + (1 − α)p (0 ≤ α ≤ 1)
(6)
where S is the column-normalized similarity matrix of images, p is a damping vector, and r is the ranking vector
each element of which represents a ranking score of each

image. α plays a role to control the extent of effect of p.
Commonly, α is set as 0.85. The ﬁnal value of r is estimated by updating r iteratively with Eq.(6). Because S iscolumn-normalized and the sum of elements of p is 1, thesum of elements of ranking vector r also stays 1. Note that
we assume that the elements of S and r correspond to thevideo shots in the descending order of the tag-based scores.
Although p is set as a uniform vector in VisualRank aswell as normal PageRank, it is known that p can plays a bias
vector which affects the ﬁnal value of r. Haveliwala [11]proposed to let topic-preferences reﬂect PageRank scoresby giving larger values on the elements corresponding to
the Web page related to the given topic. Basically, a bias
vector can adjust ranking scores of images so that the rank
scores of the biased images become higher.
In the similar way, in this paper, we propose to use a
non-uniform damping vector according to the tag-based relevance score estimated in the ﬁrst step in place of a uniform
damping vector. We deﬁne two kinds of bias vectors as follows:{
1/k(i ≤ k)
(1)
pi
=
(7)
0
(i > k){Sc(Vi )/C (i ≤ k)
(2)
pi
=
(8)
0
(i > k)where C =k∑Sc(Vj )

j=1

Sc(j) represents the tag relevance score of the video fromwhich shot j was extracted.
The bias vector (1) represented in Eq.(7) is deﬁned bygiving uniform bias values to the elements corresponding to
the top k shots regarding tag-based video scores, while thebias vector (2) represented in Eq.(8) is deﬁned by setting
the bias vector as the normalized values proportional to thetag-based scores within the top k shots. In fact, the bias
vector (1) is similar to that Jing et al. [13] assigned bias
values to only the top k images in terms of image rankingof commercial image search engine outputs.

4.3. Feature Extraction from Video Shots
To calculate the ranking scores of video shots using
Eq.(6) via iterative computation, we need to prepare a visual similarity matrix S between video shots in addition toa bias vector p. In this subsection, we describe the features
to be extracted from video shots and how to compute a similarity matrix.
In this paper, as spatio-temporal (ST) features, we use
the method we proposed in the previous paper [24], which
has achieved the best 80.4% classiﬁcation rate regarding“in-the-wild” YouTube dataset [19] by integrating themwith global motion features and Gabor appearance features
by Multiple Kernel Learning (MKL), while Liu et al., Cinbis et al. and Le et al. achieved 71.2% [17], 75.2% [5] and75.8% [15], respectively. In this paper, we use this ST feature primarily, and global motion features and Gabor appearance features as optional features for feature fusion. We
chose these three features so that their characteristics areaverage ﬁlter responses within the block, and obtain a 24dim Gabor feature vector for each block. Totally, we extract
400 24-dim Gabor vectors from each frame image.
Figure 2. Steps to extract the ST feature. (1) detected SURF points,(2) detected SURF points with motion, and (3) obtained Delaunay
triangles. (Cited from [24])different from each other. All these features are not used asthey are, but are vector-quantized and converted to bag-offeatures (BoF) vectors regarding each shot.
4.3.1

Spatio-Temporal Feature

Following the method described in our previous paper [24],ﬁrstly, we detect interest points and extract feature vectors employing the SURF method [12], and then we select moving interest points employing the Lucas-Kanade
method [20]. In this method, only moving interest points
are considered as ST interest points and static interest points
are discarded, because it is expected that a ST feature represents how objects in a video are moving. After detecting
moving interest points, we apply Delaunay triangulation toform triples of interest points where both local appearance
and motion features are extracted. In addition, we track
each interest point for consecutive ﬁve frames, and describe
ﬂow directions of interest points and change of the size ofthe triangles. This enables us to extract ST features not fromone point but from a triangle surface patch, which makes
the feature more robust and informative. The ST features
are extracted every ﬁve frames. This method is relatively
faster than the other ST features such as cuboid-based features, since it employs SURF [12] and the Lucas-Kanade
method [20], both of which are known as very fast detectors. Totally the dimension of the ST feature vector is 256.
Figure 2 shows an example of the processing steps to extract
the ST features.
4.3.2

Motion Feature

Although the proposed ST feature contains motion information, it represents only local motion. As a holistic motion feature, we build motion histograms over a frame image. This feature is expected to have different discriminative power from the ST feature. We extract motion featuresat grid points with every 8 pixels using the Lucas-Kanade
method [20]. Extracted motion features from each grid are
voted to histogram of 7 direction and 8 motion magnitude.
4.3.3

Appearance Feature

We use Gabor texture histograms as an appearance feature.
A Gabor texture feature represents texture patterns of local
regions with several scales and orientations. In this paper,
we use 24 Gabor ﬁlters with four kinds of scales and six
kinds of orientations. Before applying the Gabor ﬁlters, we
divide a frame image extracted from video shots into 20×20
blocks. We apply the 24 Gabor ﬁlters to each block, then

4.3.4

Vector Quantization of Features for a Shot

We extract global motion and appearance features from every 4 frames within each video shot, while we extract ST
features from every 5 frames, and we vector-quantize all ofthem and convert them into the bag-of-features (BoF) representation within each shot. While the standard BoF represents the distribution of local features within one image,the BoF employed in this paper represents the distributionof features within one shot which consists of several frame
images. We call this BoF regarding one video shot as bagof-frames (BoFr).
In the experiment, we set the size of the codebook of
the 256-dim ST features, the 24-dim appearance features
and the 56-dim motion features as 5000, 5000 and 3000,
respectively.
4.3.5

Computation of Similarity Matrix

To obtain a similarity matrix S between video shots for calculation of the ranking scores of video shots using Eq.(6)via iterative computation, we use histogram intersection as
follows:|H|∑
s(Hi , Hj ) =min(hi,l , hi,l )(9)l=1

where Hi , hi,l and |H| represents the BoF vector of the i-th
shots, its l-th element and the dimension number of the BoF
vector, respectively.
In addition to single features, we made experiments on
fusion of the three kinds of features by linear combination.
A combined similarity is obtained as follows:
Scombined = wST × SST + wmot × Smot + wapp × Sapp
(10)
where SST , Smot and Sapp represents the similarity matrix
of ST features, motion features and appearance features, respectively.
In case of supervised action recognition, optimal fusionweights can be estimated by machine learning methods suchas Multiple Kernel Learning and AdaBoost. However, in
our case, we have no training data, since this work aims unsupervised action database construction. Therefore, it difﬁcult to use any optimization methods for estimating weights.
Instead, in the experiment, we set three weights heuristically as follows:
1
1
1
(11)
wST = , wmot = , wapp =244
To estimate optimal weights even under unsupervised way
is one of our future works.
4.3.6

Treatment for Camera Motion

Most existing works on action recognition do not consider
camera motions, since most of them assume a ﬁxed camera.

However, it is important to cope with camera motion in caseof Web videos, and then some recent works on action recognition for unconstrained Web videos coped with it. Cinbis etal. [5] employed a homography-based camera motion compensation approach at the task of action recognition on Web
videos. Unfortunately, since they did not compare their results with the results without camera motion compensation,
the effectiveness of camera motion compensation for action
recognition on unconstrained Web videos is unclear.
Then, in this paper, we examine and compare two cases:
with camera motion removal, and without it. To compensate
camera motion, we make use of a homography-based camera motion compensation approach, which is basically thesame as [5]. Although compensation of camera motion ispossible, accurate compensation is difﬁcult for Web videos.
This is because Web videos contain various kinds of intentional and unintentional camera motions and their resolution is usually low. Especially, for our objective which is automatic building of an action shot database, the shots stored
in the database should have no camera motion. The situation is different from the case of action recognition where
they must classify all the given videos which might include
camera motion. Therefore, in this paper, we adopt a simple
strategy that we discard the shots where camera motion isdetected as a method with no camera motion compensation,
which is the same as [19]. In the actual implementation,
we detect camera motion before extracting features as preprocessing.
To detect camera motion, we calculate motion features
based on the Lucas-Kanade method at every 8-pixel grid. If
the region where motion is detected is larger than a predeﬁned threshold, we consider camera motion is detected.

5. Experimental Results
To examine effectiveness of the proposed method, we
made experiments with various conditions using six kindsof human action keywords: “batting”, “eating ramen”,“jumping trampoline”, “running marathon”, “shooting football” and “walking street”. Some examples of the frame
images of six actions are shown in Figure 3. All the verbs
included in these six keywords are the same as ones used in[24], since we will compare our results with Noguchi et al.’s
results brought by the state-of-the-art supervised method
employing MKL-based feature fusion.
In addition, we made two additional experiments: collecting video shots of non-human actions, and building 100
kinds of action database.
In the experiments, we obtained rankings of 2000 shots
for each actions. For evaluation of ranking results, in general, average precision is widely used. However, we mainly
use the precision rate at rank 100 rather than average precision, since commonly used dataset on human action such as
KTH dataset [26] and “in-the-wild” YouTube dataset [19]has about the 100 video shots per action 1 and the objectiveof our work is automatic construction of an action video
1 KTH dataset has 599 shots for 6 actions, and “in-the-wild” dataset has
1168 shots for 11 actions.

Figure 3. Examples of six kinds of the actions.

shot database which is helpful for the research community
on action recognition.
In all the experiments, we used YouTube.com as data
source. We collected video metadata including video IDs
and tags using YouTube Data API.

5.1. Experiments with various settings
As shown in Table 1, we made 10 kinds of experiments
by changing the conditions. The condition of each experiment means as follows:
RND Download 200 videos for the given keywords, andselect 100 shots randomly.
TAG Download the top 200 videos among 1000 videos
in terms of tag-based ranking, and select 100 shots
randomly.
Exp.1 Download 200 videos for the given keywords
without tag-based ranking, select 2000 shots randomly, and re-ranked them by visual-featurebased ranking using only ST features with a uniform damping vector.
Exp.2 Add tag-based video shot selection to Exp.1.
Exp.3 Use a bias damping vector obtained by (1) Eq.(7)or (2) Eq.(8) instead of a uniform damping vector
in Exp.2.
Exp.4 Add camera motion compensation to Exp.3(1).
Exp.5 Use motion features instead of ST features inExp.3(1).
Exp.6 Use appearance features instead of ST features inExp.3(1).
Exp.7 Use combined features of three kinds of featuresinstead of ST features in Exp.3(1).
Exp.8 Add camera motion compensation to Exp.7.
The results of the experiments explained above areshown in the last columns of Table 1. From these results,
RND brought the worst result, which selected just randomly
100 shots. TAG improved RND by 9.3% by introducing
tag-based ranking to select videos to download. Exp.1 outperformed TAG, which used visual-feature-based ranking.
Exp.2 improved Exp.1 by 7.3% by combining tag-based

Table 1. The conditions of the experiments and their results.
Exp.
no
RND
TAG
1
2
3(1)
3(2)
4
5
6
7
8

tag-based biased camera motion
ranking damp. vec. compensation feature√ randomly-selected 100 shots
ST
√
ST
√
√ST
√
√(1)
(2)
ST
√
√
√
ST
√
√
motion√√appear.
√√fusion√√√fusionmean
prec@10014.2 %
23.5 %
33.7 %
41.0 %
47.3 %
44.8 %
39.8 %
31.8 %
39.7 %
49.5 %
41.2 %

Figure 4. Mean precision at the n-th shot over six actions.

Table 2. Precision@100 of six actions (%).
Exp. no
RND
TAG
1
2
3(1)
3(2)
4
5
6
7
8
9 (0.75)
9 (0.80)
9 (0.85), 7
9 (0.90)9’(0.95)
MKL [24]

batting eating jumping running shoot walking

12
32
56
77
66
73
38
69
35
69
61
68
82
69
77
74
83

13
14
37
30
42
28
31
10
57
39
37
32
34
39
31
31
78

17
45
69
75
82
90
78
67
64
87
74
80
83
87
79
81
98

23
23
20
23
35
38
45
12
46
30
38
29
29
30
26
28
87

4
20
5
17
33
27
12
32
7
36
16
26
30
36
28
25
82

16
7
23
24
29
30
36
20
29
36
21
29
33
36
35
32
52

AVG.
14.2 %
23.5 %
33.7 %
41.0 %
47.3 %
44.8 %
39.8 %
31.8 %
39.7 %
49.5 %
41.2 %
44.0 %
48.5 %
49.5 %
46.0 %
45.2 %
80.0 %

ranking and visual-feature-based ranking.
In Exp.2, we used a uniform damping vector for VisualRank computation. In Exp.3, we tried using a non-uniform
bias damping vector in the two ways: Exp.3(1) gave uniform weights to the top 1000 shots, and Exp.3(2) assigned
the weights in proportion to the tag-based relevance scores.
From the results of Exp.3(1) and Exp.3(2), it turned out thatthe method (1) was superior to the method (2) as well as
Exp.2, although the difference is not so large. Therefore, inall the the experiments after Exp.3, we used the method (1)to set a bias damping vector.
In Exp.4, we introduced camera motion compensation.
However, Exp.4 was degraded from Exp.3. This nevermeans that camera motion removal is useless. This just indicates that the default strategy that shots with camera motion
are discarded is superior to the alternative strategy that shotswith camera motions are used after camera motion removal.
Since many good-conditioned video shots with no camera
motion can be extracted from the downloaded videos, we
do not need to use less-conditioned video shots in whichcamera motion were removed artiﬁcially.
In Exp.5 and Exp.6, we used global motion features andappearance features instead of ST features, which achieved
31.9% and 39.7%, respectively. This shows that ST features, which achieved 47.3%, were much superior to the twoother features in this task.
Exp.7 achieved the best result, 49.5%, which used all
the proposed method including feature fusion except cam-era motion compensation. Compared with Exp.3(1), featurefusion improved the result slightly from 47.3% to 49.5%. In
Exp.8, we added camera motion compensation to Exp.7. As
a result, Exp.8 was degraded in the similar way as Exp.4.
We show all the results of six actions in Table 2. In
Exp.4, the results for “walking” and “running” were improved compared to Exp.3(1). Since the videos related
to “walking” and “running” are usually recorded by moving cameras, the number of good-conditioned shots without
camera motion is limited. Therefore, in case of discarding the shots with camera motion, too many shots were discarded. This is the reason why using even less-conditioned
shots after motion removal is better strategy for “walking”
and “running”.
In Table 2, we add Exp.9 which means the experiments
in case of changing the value of α in Eq.(6) from 0.75 to
0.95 with the step of 0.05. The conditions except α is thesame as Exp.7 which achieved the best results, 49.5%. The
value in the parenthesis in the table represents α. Note that
Exp.9 (0.85) is equivalent to Exp.7. The results on Exp.9
indicate that 0.85 is the best value for α. In addition, weshow the result by the state-of-the-art supervised learning
methods [24] in the bottom row in Table 2, which showsthat the difference is still larger in terms of precision except
for “batting”.
In Figure 4, we show the mean precision at the n-th shotover six actions. From this graph, Exp.7 achieved the best
precision even at the lower rank.

5.2. Additional Experiments on Non-human Actions and 100 Actions
In addition, we made two additional experiments: collecting video shots with non-human action keywords suchas “airplane ﬂying” and building a large-scale action video
shot database with 100 kinds of action keywords. In all
the additional experiments, we used only ST features in thesame way as Exp.3(1), since extraction of three kinds offeatures for fusion requires heavy computation.
The results of 6 kinds of non-human actions including
artifact actions and natural phenomena are shown in Table
3. The results were not as good as the six human actions,since we used only ST features which was suitable for representing human actions. Seeking suitable features for nonhuman action is one of our future works.
We made large-scale experiments on 100 kinds of actions

Table 3. Precision@100 of non-human actions (%)
aircraftblooming airplane
shuttle+landing tornado +ﬂower +ﬂying earthquake +launching
30
39
44
14
7
18

AVG.
25.3

Table 4. Precision@100 of 100 human actions (%)
climb+tree
24soccer+dribble
100
play+drum40
ride+horse
24
fold+origami
96skate
37 roll+makizushi
24crochet+hat
95
swim+crawl
36
sew+button
24
arrange+ﬂower
94
cut+hair
35
fry+tempura
23
paint+picture88
run+marathon
35
slap+face20
boxing86
count+money
33
read+book19
jump+parachute82
paint+wall
33
squat
19
jump+trampoline
82
shoot+football33
row+dumbell
16
do+exercise
79
draw+eyebrows32
wash+clothes
15
do+aerobics
78
ﬁeldhockey+dribble 32
wash+dishes
15
do+yoga77
hit+golfball32
comb+hair
14
surf+wave
75
lunge
32
drink+coffee
14
shoot+arrow
73
play+piano
32 swim+breaststroke 13
massage+leg
72
row+boat32
cry
12
ﬁx+tire67
sing
32
eat+sushi
12batting66
chat+friend
31
serve+teniss
11basketball+dribble 64
clean+ﬂoor
31
tying+tie
11
blow-dry+hair
64
cut+onion
31
boil+egg
9knit+sweater
64
shave+mustache
31
head+ball9
ride+bicycle
62
pick+lock30 swim+backstroke
9
curl+bicep
58
plaster+wall
30 take+medicine
8shoot+ball
58
blow+candle
29 serve+volleyball
7
tie+shoelace
57
wash+face
29 swim+butterﬂy
7
laugh
50
walking+street
29
bake+bread
6
dive+sea
49
brush+teeth28
cook+rice6
harvest+rice
49
catch+ﬁsh
28
grill+ﬁsh5
ski
49
drive+car
28
jog
5iron+clothes
47
plant+ﬂower
28
slice+apple
5
twist+crunch
47
play+guitar
28
peel+apple
5
dance+ﬂamenco
45
lift+weight27
bowl+ball4
dance+hiphop
43
raise+leg
27
smile
4
eat+ramen
42
hang+wallpaper
26
kiss
2
dance+tango
41
jump+rope
26
play+trumpet
41
AVG. (68-100) 12.2
AVG. (35-67)
31.0
AVG. (1-34)
65.9
AVG. (ALL)36.6

with video metadata analysis on 100,000 YouTube videos
and spatio-temporal feature analysis on 20,000 YouTube
videos. We show the results for 100 kinds of actions inTable 4 including 6 kinds of actions used in the previous
experiments. The mean of the precision at 100 shots over
100 actions was 36.6%, and the precision of each action
varies from 2 to 100. This shows that the result depends on
the kinds of actions and selection of action keywords sent to
Web API as query words greatly. Especially, single action
keywords such as “smile” and “cry” were too ambiguous toobtain good candidate videos. Although the current performance is not enough to built action shot database for mostof actions in the full automatic manner, for some actions the
proposed method worked very well.
All the results including a video summary of eachof 100 actions and direct links to the original videos
on the YouTube can be seen at our project Web page:
http://mm.cs.uec.ac.jp/webvideo/ .

6. Conclusions
In this paper, we presented a method of automaticallyextracting from Web videos video shots corresponding to
speciﬁc actions with just only providing action keywords.

In the experiments, we achieved the 49.5% precision at 100
top-ranked shots over six kinds of human actions and the
36.6% precision for 100 kinds of human actions without
any supervision. Although the obtained results were notenough, we believe that the direction we proposed in thepaper is promising.
As future works, we plan to improve the way of setting
bias damping vectors in VisualRank calculation to achieve
better results, and to seek new kinds of visual features which
work even for non-human actions.

References[1] L. Ballan, M. Bertini, A. D. Bimbo, M. Meoni, and G. Serra. Tag suggestion
and localization in user-generated videos based on social knowledge. In Proc.
ACM MM WS on Social Media, pp. 3–7, 2010. 2[2] M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri. Actions as spacetime shapes. In ICCV, 2005. 2[3] S. Brin and L. Page. The anatomy of a large-scale hypertextual web searchengine. In WWW, 1998. 4[4] N. I. Cinbins, R. G. Cinbins, and S. Sclaroff. Learning action from the web. In
ICCV, pp. 995–1002, 2009. 2[5] N. I. Cinbis and S. Sclaroff. Object, scene and actions: Combining multiple
features for human action recognition. In ECCV, pp. 494–507, 2010. 4, 6
[6] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection.
In CVPR, vol. 1, pp. 886–893, 2005. 2[7] L. Duan, D. Xu, I. W. Tsang, and J. Luo. Visual event recognition in videos bylearning from web data. In CVPR, 2010. 2[8] O. Duchenne, I. Laptev, J. Sivic, F. Bach, and J. Ponce. Automatic annotationof human actions in video. In CVPR, pp. 1491–1498, 2009. 2[9] R. Fergus, L. Fei-Fei, P. Perona, and A. Zisserman. Learning object categories
from google’s image search. In ICCV, pp. 1816–1823, 2005. 2[10] R. Fergus, P. Perona, and A. Zisserman. A visual category ﬁlter for google
images. In ECCV, pp. 242–255, 2004. 2[11] T. Haveliwala. Topic-sensitive PageRank: A context-sensitive ranking algorithm for web search. IEEE trans. on KDE, 15(4):784–796, 2003. 4[12] B. Herbert, E. Andreas, T. Tinne, and G. Luc. Surf: Speeded up robust features.
CVIU, pp. 346–359, 2008. 5[13] Y. Jing and S. Baluja. Visualrank: Applying pagerank to large-scale image
search. PAMI, 30(11):1870–1890, 2008. 1, 2, 4
[14] I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld. Learning realistic human actions from movies. In CVPR, 2008. 2[15] Q. Le, W. Zou, S. Yeung, and A. Ng. Learning hierarchical invariant spatiotemporal features for action recognition with independent subspace analysis. In
CVPR, 2011. 4[16] L. Li and L. Fei-Fei. OPTIMOL: automatic Online Picture collecTion via Incremental MOdel Learning. In CVPR, 2007. 2[17] C. Liu, J. Yuen, and A. Torralba. Nonparametric scene parsing: Label transfer
via dense scene alignment. In CVPR, 2009. 2, 4[18] D. Liu, X. Hua, L. Yang, M. Wang, and H. Zhang. Tag ranking. In WWW, pp.
351–360, 2009. 2[19] J. Liu, J. Luo, and M. Shah. Recognizing realistic action from videos. In CVPR,
2009. 2, 4, 6
[20] B. Lucas and T. Kanade. An iterative image registration technique with anapplication to stereo vision. In IJCAI, pp. 674–679, 1981. 5[21] M. Marszalek, I. Laptev, and C. Schmid. Actions in context. In ICCV, 2009. 2[22] J. Niebles, B. Han, A. Ferencz, and L. Fei-Fei. Extracting moving people frominternet videos. In ECCV, pp. 527–540, 2008. 2[23] J. Niebles, H. Wang, and L. Fei-Fei. Unsupervised learning of human action
categories using spatial-temporal words. In BMVC, 2006. 2[24] A. Noguchi and K. Yanai. A surf-based spatio-temporal feature for featurefusion-based action recognition. In ECCV WS on Human Motion: Understanding, Modeling, Capture and Animation, 2010. 1, 2, 4, 5, 6, 7
[25] F. Schroff, A. Criminisi, and A. Zisserman. Harvesting image databases fromthe web. In ICCV, 2007. 2[26] C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: A local
SVM approach. In ICPR, pp. 32–36, 2004. 2, 6
[27] J. Sun, X. Wu, S. Yan, L. F. Cheong, T. S. Chua, and J. Li. Hierarchical spatiotemporal context modeling for action recognition. In CVPR, 2009. 1[28] G. Toderici, H. Aradhye, M. Pasca, L. Sbaiz, and J. Yagnik. Finding meaning
on YouTube: Tag recommendation and category discovery. In CVPR, pp. 3447–
3454, 2010. 2[29] Z. Wang, M. Zhao, Y. Song, S. Kumar, and B. Li. YouTubeCat: Learning tocategorize wild web videos. In CVPR, 2010. 2[30] K. Yanai. Generic image classiﬁcation using visual knowledge on the web. In
ACM MM, pp. 67–76, 2003. 2[31] K. Yanai and K. Barnard. Probabilistic Web image gathering. In ACM MIR, pp.
57–64, 2005. 2[32] Q. Yang, X. Chen, and G. Wang. Web 2.0 dictionary. In CIVR, pp. 591–600,
2008. 2, 3