Automatic Collection of Web Video Shots
Corresponding to Speciﬁc Actions using Web Images
Do Hang Nga

Keiji Yanai

Department of Informatics, The University of Electro-Communications, Tokyo
1-5-1 Chogugaoka, Chofu-shi Tokyo 182-8585 Japandohang@mm.cs.uec.ac.jp, yanai@cs.uec.ac.jp

Abstractvant videos of a speciﬁc action the retrieved results generally contain noise, some of them may actually hold relevant
scenes that correspond to the action. Here we consider avideo shot as a set of consecutive frames which represent
a scene. If video shots corresponding to any action can beobtained automatically from Web source, so that building
action database will become easier than before.

In this paper, we apply Web images to the problem of
automatically extracting video shots corresponding to speciﬁc actions from Web videos. Our framework modiﬁes theunsupervised method on automatic collecting of Web video
shots corresponding to the given actions which we proposedlast year [9]. For each action, following that work, we ﬁrst
exploit tag relevance to gather 200 most relevant videos of
the given action and segment each video into several video
shots. Shots are then converted into bags of spatio-temporalfeatures and ranked by the VisualRank method. We reﬁnethe approach by introducing the use of Web action images
into shot ranking step. We select images by applying Poselets [2] to detect human in the case of human actions. We
test our framework on 28 human action categories whose
precision values were 20% or below and 8 non-human action categories whose precision values were less than 15%
in [9]. The results show that our model can improve the precision approximately 6% over 28 human action categories
and 16% over 8 non-human action categories.

The problem of automatically extracting video shots related to speciﬁc human actions from Web videos in theunsupervised manner was ﬁrst proposed in our paper lastyear [9]. We aimed to collect most relevant video shots
to given action keywords from a large number of tagged
YouTube videos by an unsupervised ranking method. The
top ranked shots are supposed to be shots corresponding to
the actions. We tried as much as 100 human action categories including sport activities like “jog” or “swim butterﬂy” and activities of daily living such as “drink coffee”
and “wash dishes”. Our previous approach achieved 50%
or over in precision over 24 categories. Note that the precision here is deﬁned as the proportion of relevant shots over
100 top ranked shots (Precision@100). However, in case ofsome other actions, due to the extremely noisy tags whichhave been tagged subjectively by YouTube users, we did
not succeed in selecting relevant videos, and our previous
approach did not obtain as much relevant shots as desired.

1. Introduction
In the research community of action recognition, asclassiﬁcation rates on small and fairly controlled datasets
like KTH [12] or Weizmann [1] has reached nearly perfect rates [8], there is a need for a large and unconstrained
dataset. The largest and latest proposed action dataset isquite large with 51 action categories [6]. However, so
far constructing action training database is known as avery time-consuming process, since video sequences corresponding to a speciﬁc action are usually recorded or collected from video sources like movies manually. On theother hand, more and more videos are being uploaded to
the Web through video sharing web sites such as YouTubeand DailyMotion. Even though when we search for rele-

Then, in this paper, we modify our previous method bytaking Web still images corresponding to given actions into
account, with an intuition that the shots which are more similar to Web action images may be more likely relevant shots
so thus they should be ranked higher. In fact, recent works[11, 17, 13, 16] show that action recognition for still images
is possible. Then, we collect images related to the given actions automatically via Web image search engines by only
provided keywords, and evaluate the similarities between
video shots and selected images by feature matching. Thatmeans our modiﬁcation also does not require any supervision, so that the automaticity of the whole framework can
1be preserved.
In our framework, with each action keyword, we ﬁrst
download 200 most action related videos from YouTube;segment each downloaded video into several shots andrepresent each shot as a bag of spatio-temporal features following the method described in [9]. At the same time, we
download hundreds of images using Bing API and apply
Poselets [2] to detect human in those images. Next, we extract SURF features [5] from video shots and human detected images, then calculate similarity between each shot
and those images by counting matched local features. Notethat we apply human-detection-based image selection onhuman actions only. In case of non-human actions, we simply use images directly retrieved by Bing API. Finally, weapply VisualRank ranking method [7], to rank video shotsby taking account of spatio-temporal features of video shotsand shots-images similarities so that we can obtain relevant
shots to the given keywords as highly ranked shots.
We test our framework on 28 human action categories
whose precision values are less than 20% in our previous
work [9]. The results show that our modiﬁcation enhances
the performance on tested categories by approximately 6%
in average. Especially, for 8 categories whose precision is
below 10%, the performance is improved signiﬁcantly from
5.7% to 21.6%. We also verify the efﬁciency of the proposed framework to non-human actions whose precision
values is less than 15% in [9]. These results demonstratethat by introducing Web images into shot ranking, we canboost the precision from 2% to 16% in average. That means
even in the case where tags are too noisy so that they canlead to the selection of mostly irrelevant videos, our proposed method still can extract from those videos quite anumber of action related video shots.

2. Related WorkIn this section, we refer to some related works on action
recognition with uncontrolled video datasets.
Recently, works which are dealing with video categorization for YouTube videos [15, 14] and Kodak consumer
video dataset [4] has been increasing. All of them employedsupervised learning which requires training samples, andtheir objective is categorizing videos into one of the predeﬁned categories. On the other hand, our work do not require any training samples and detecting video shots associated with the given keywords for a large number of Web
videos.
As the most similar work to ours, Cinbis et al. [3]’smethod learn action models automatically from Web images
gathered via Web image search engines. While we use both
Web videos and Web images as training source and spatiotemporal features as action descriptors, they use only Web
images and static features. In addition, they concentrate on
only human actions, while we aim to all kinds of action in-cluding non-human actions such as “airplane+ﬂying” and“ﬂower+blooming”.

3. Proposed Framework
The framework presented in this paper is built on theframework of automatic construction of an action video shot
database using Web videos [9], which aims to extract most
relevant video shots to given keywords from a large numberof tagged YouTube videos in an unsupervised manner. The
introduction of Web images into video shot ranking process make it more possible to obtain relevant shots in thecase that tag noisy causes the failure on collecting relevant
videos.

3.1. Previous Framework
Our approach bases on our previous work [9], whichﬁrst introduced the problem of constructing automatically alarge scale action video shot database using Web videos in
the unsupervised way. Given the action keywords such as“walking” or “surﬁng+wave”, we proposed to extract fromtagged YouTube videos video shots corresponding to thoseactions by following three steps: (1) tag-based video selection, (2) segmenting videos into shots and extracting features from all the shots, (3) visual feature-based video shot
selection with tag-based scores taken into account.
In the ﬁrst step, for each given action keyword, we rank
1000 YouTube videos which have been tagged with the keyword regarding the relevance of each video to the action byusing tag co-occurrence statistics and select 200 most relevant videos to download. Note that a list of tags and video
IDs corresponding to the given query word can be obtained
via YouTube API, so it is not necessary to download all1000 videos for tag-based video ranking and just 200 top
ranked ones will be downloaded using their IDs. In the second step, each of downloaded video is divided into video
shots and visual features are extracted from each of all the
shots. As visual features, we use the spatio-temporal (ST)features proposed by Noguchi and Yanai [10]. In the third
step, they rank video shots by applying graph-based ranking
method, VisualRank [7], with a visual-feature-based similarity matrix and a bias damping vector which is calculated
based on tag relevance scores of videos.
This framework works quite well in many action categories. However, in the case where the tags are too noisy,
most of results of tag based video selection are irrelevant
videos and shot selection step obtains just a very few relevant video shots.

3.2. Improvements
In this paper, we enhance our previous work [9] by introducing Web images into video shot selection so that evenwhen noisy tags caused failures on video selection, our ap-

YouTube

Bing

Bing
API

Web
API

Select videos based on
tag co-occurrence

Download images
Humanactions

Detect humanusing poselets

Non‐
Human
actions

RGB histogram

Segment
videos to shots

Extract visual featuresfrom shots & images
Feature matching

Most relevant
200 videos

Extract spatio-temporalfeatures from shots
BoF representation

Calculate similarities
between shots & images

Histogramintersection

Calculate similarity matrix

Rank shots
Figure 1. Overview of the proposed framework. The half left partwith a dotted red box shows our modiﬁcations.

Figure 2. The top six Web images after Poselets-based ﬁltering.

ages to select will be discussed in the experiment section.
In the next step, we extract visual features from selectedshots and images and evaluate the similarities between shots
and images based on local image feature matching. Here
we use SURF [5] as visual feature. Finally, with the similarity matrix between shots obtained in the ﬁrst step and adamping vector calculated by using the similarities between
shots and images, we rank video shots applying VisualRank
method [7].

4. Methods
proach still can detect quite a number of video shots corresponding to the given actions.
The proposed method consists of four processing steps(Figure 1):(1) tag-based video selection; video shot segmentation;
spatio-temporal feature extraction from shots; bagof features representation; similarity matrix between
shots calculation,(2) for only human actions: Web images selection using
Poselets-based human detection [2],(3) visual feature extraction from shots and images; feature matching based similarity calculation, and(4) spatio-temporal-feature-based video shot selection
with shots-images similarities taken into account.
For each action, following [9], we ﬁrst evaluate the relevance of each video in 1000 videos which have action
keyword as one of their tags to the action using tag cooccurrence; download 200 most relevant videos; segmenteach video into shots based on its RGB histogram; extract
spatio-temporal features from all the shots and represent
them as bags of features; calculate similarity matrix for shotranking using histogram intersection. The next steps are our
modiﬁcation to the previous work and represented as theleft part of Figure 1 with a dotted red box. This is how we
automatically select action related images and use selected
images to improve relevant shots selection. We download
hundreds of images from the Web via a Web image search
API, Bing API, which is ofﬁcially provided by Microsoft.
We then apply Poselets proposed by [2] to detect human insearched images. The human detected images will be selected to go to the next step. The appropriate number of im-

In this section, we describe in detail Web action image selection using Poselets-based human detection; feature matching based shots-images similarities calculationand shot ranking with shots-images similarities taken into
account.

4.1. Web action image selection
When one queries an action keyword on Web image
search engine, thousands of images might be returned.
However, in general, even the top returned images may be
not relevant images of the queried action because of a wide
variety of meaning of the keyword as well as the action itself, especially in the case of human action. On the other
hand, we also want to preserve the automaticity of the original framework, since manual selection is not preferred here.
We postulate two assumptions: (1) the set of retrieved images contains relevant images of the queried action and (2)human or body part should be seen in human action images.
It is reasonable to consider that images which contain human poses are likely related to that human action. Based onthese assumptions, we select a collection of action images
by applying Poselets-based human detection [2] on Web images. Poselets are demonstrated as effective body part detectors trained by 3D human pose annotations. We apply
Poselets detector tools which are ofﬁcially offered by theauthors 1 on the set of retrieved Web images using default
parameters. Figure 2 illustrates some examples of selected
Web images using Poselets-based human detection.
The appropriate number of images to select then become
a question. Let N be the number of action images that will
1 http://www.cs.berkeley.edu/%7Elbourdev/poselets/be used to improve relevant shot detection, is it true that thelarger N is, the more relevant shots will be detected? Weactually try several values for N : 10, 20, 30, and 50, anddiscuss more about this issue in the experiment section.

4.2. Shots-images similarity calculation
To evaluate the similarity between a video shot and given
set of action images, we ﬁrst extract SURF local featuresfrom all action images of selected set and each one frame
per ﬁve consecutive frames of all the shots. For each shot,
we count matching points between SURF local features extracted from each frame and each Web image by thresholding Euclidean distances between SURF feature vectors.
The similarity SI(Si ) between a shot Si which has M frame
images (Fj (j = 1..M )) and an image set I which has N
images (Ik (k = 1..N )) is calculated by the following equations:
SI(Si )

=

N
∑
k=1

where Si (F |Ik )=max SI(Fj |Ik ),

j=1:M

(1)

2 ∗ MatchPoint(Fj , Ik ), (2)(Point(Fj ) + Point(Ik ))MatchPoint(Fj , Ik ), Point(Fj ) and Point(Ik ) represent
the number of matched points between a frame image Fj
and a Web image Ik , the number of extracted SURF features from Fj and the number of extracted SURF featuresfrom Ik , respectively.

images would become higher than unbiased ones. In ourprevious work [9], damping vector is proposed to be calculated based on tag relevance scores of videos. We deﬁned
two kinds of bias vectors as follows:{
1/k(i ≤ k)
(1)
pi
=
(4)
0
(i > k){Sc(Vi )/C (i ≤ k)
(2)
pi
=
(5)
0
(i > k)where C =r = αSr + (1 − α)p(0 ≤ α ≤ 1)

(3)

where S is the column-normalized similarity matrix of images, p is a damping vector, and r is the ranking vector
with each element which represents a ranking score of acorresponding image. α plays a role to control the extent ofeffect of p. Commonly, α is set as 0.85. The ﬁnal value ofr is estimated by updating r iteratively with Eq.(3). Notethat S is calculated as similarity matrix of shots using their
histograms of spatio-temporal features [10].
Although p is set as a uniform vector in original VisualRank equation [7], p is known as a bias vector which affects the ﬁnal value of r since the rank scores of the biased

Sc(Vj )

j=1

where Sc(j) represents the tag relevance score of the videofrom which shot j was extracted.
The bias vector (1) represented in Eq.(4) is deﬁned bygiving uniform bias values to the elements corresponding to
the top k shots regarding tag-based video scores, while thebias vector (2) represented in Eq.(5) is deﬁned by setting
the bias vector as the normalized values proportional to thetag-based scores within the top k shots. These method of
calculating damping vector may work effectively in the casewhere tags information is reliable. In reverse, when tags are
too noisy, tag relevance score will be noisy as well.
In this paper, to bias the shots we propose to give larger
values on the elements more similar to the selected action
images with the following equation:(3)pi

4.3. Relevant shot selection
As a method on visual-feature-based shot ranking, we
also employ VisualRank method [7] following [9]. Whilein [9] shots from videos which have larger tag relevance
scores were assigned higher probability of being ranked tothe top, in our approach we assign higher probability to theshots that are most similar to the selected set of action Web
images. We believe that when tags are too noisy, tag relevance based shot ranking will not help to rank relevant shots
to the top.
Eq.(3) represents an equation to compute VisualRank.

k∑exp(γSI(Si ))= ∑n
j=1 exp(γSI(Sj ))(6)

In the experiments, we set a constant value γ as log 3.

5. Experiments
We examine the effectiveness of our approach on 28 human action categories and 8 non-human action categories
whose precision is quite low in [9]. We use the same video
dataset and the same evaluation method as [9]. So that precision here means the precision rate over 100 top ranked
shots (Precision@100). We try several values for numberof selected images N = 10, 20, 30 and 50. N = 0 refers
to the precision obtained by [9]. The results are shown in
Figure 3 and 4 for human actions and non-human actions,
respectively. Each result is the average precision over tested
action categories. The results of all categories are shown in
Table 1 (human actions) and Table 2 (non-human actions).
The results show that for both human actions and nonhuman actions, using N = 20 or N = 30 images brings best
performance by enhancing the precision approximately 6%
and 16% in average in the case of human actions and nonhuman actions, respectively. Especially, by referring Table
1 and 2, we can see that our approach could improve signiﬁcantly the results of 10 human action categories including “bake+bread”, “jog”, “squat”, “swim+breaststroke”,

“serve+volleyball”, “smile”, “cook+rice”, “grill+ﬁsh”,“swim+butterﬂy”, “tie+tie” and 6 non-human action categories including “falling+leaves”, “snow+falling”, “typhoon”, “airplane+ﬂying”, “earthquake”, “waterfall”.
However, our proposed framework did not achieve
good results on some categories such as “slap+face” and“wash+clothes”. To explain for this, we think of the following two cases:(1) human-detection-based image selection selects very
few relevant images(2) shots-images similarity calculation method is not effective
The ﬁrst case corresponds to categories like “slap+face”.
Figure 5 shows some ﬁrst results of image selection step for“slap+face”. We can see that most of selected images areirrelevant to the action. That can explain why introducing
these images causes a decrease in precision on “slap+face”
category. The second case corresponds to categories like
“wash+clothes”. In this case, even image selection actually
succeeds in selecting relevant images of the action (Figure6), applying these action images cannot help to select more
shots corresponding to the action. The reason is that, notthe selected action images, but the variety of the action itself causes failure to shots-images similarity calculation sothat similarity-with-images-based shot ranking cannot detect relevant shots correctly. While selected images containmainly “outdoor washing clothes using hands” (Figure 6),
most of the downloaded videos are about “indoor washingclothes using washing machine” (Figure 7).
Discussing about the appropriate number of images touse, as results shown in Figure 3 and 4, the precision increases while number of images is going up from 10 to 20
but decreases while number of images is rising from 30 to
50. That means it is not always true that the larger numberof images is, the more relevant shots will be detected. Our
intuition is that, here we use Web images as training source,
and it is well known that in general, top retrieved Web images are more related to the query keyword, so using justthe top images should obtain better results. To verify thisintuition, we evaluate the precision rates of image selection
by counting number of relevant images over all selected images and show the results in Figure 8 (human actions) and
Figure 9 (non-human actions). The results show that the
proportion of relevant images over N = 10, 20 and 30 selected images are quite high while it is comparatively low
when N = 50.

6. Conclusions
In this paper, we apply Web images to the problem of
automatically extracting from Web video shots corresponding to speciﬁc actions. In case of human actions, we apply Poselets [2] to detect human, and then use human detected images to improve that relevant video shots extrac-

Figure 3. The mean precision of the top 100 ranked video shots
over 28 human action categories.

Figure 4. The mean precision of the top 100 ranked video shots
over 8 non-human action categories.
Table 1. Results of 28 human action categories depending on number of selected images
Actionslap+face
read+booksquat
row+dumbbell
wash+clothes
wash+dishes
comb+hairdrink+coffee
swim+breaststrokecry
eat+sushi
serve+tennis
tie+tieboil+egg
head+ballswim+backstroke
take+medicine
serve+volleyballswim+butterﬂybake+bread
cook+rice
grill+ﬁshjog
pick+apple
slice+apple
bowl+ball
smile
kiss
Average

[9]
20
19
19
16
15
15
14
14
13
12
12
11
11
9
9
9
8
7
7
6
6
5
5
5
5
4
4
2
10.1

N=10
16
21
34
23
9
23
12
8
23
4
13
14
18
4
5
10
5
20
29
18
15
21
15
8
2
18
16
2
14.5

N=20
14
22
38
23
10
21
15
10
27
6
13
18
17
8
9
12
8
24
33
19
16
23
19
10
4
17
17
4
16.3

N=30
13
23
32
24
10
25
12
9
31
5
11
15
23
6
7
14
7
31
31
18
15
26
21
9
2
15
18
3
16.3

N=50
17
24
35
22
9
24
23
16
24
4
10
14
30
6
6
12
6
23
36
14
13
17
20
10
6
5
15
2
15.8

Table 2. Results of 8 non-human action categories depending on
number of images
Actionexplosion
falling+leaves
snow+falling
typhoonairplane+ﬂying
earthquake
heavy+rain
waterfall
Average

[9]
0
3
0
4
2
7
0
0
2

N=10
4
12
18
21
29
26
4
15
16.1

N=20
5
14
21
25
30
24
3
15
17.1

N=30
5
16
22
29
32
25
3
17
18.6

N=50
1
9
24
24
27
23
4
15
15.9

Figure 8. The mean precision of the top N relevant Web images
over all the 28 human actions

Figure 5. The top 18 Web images selected by Poselets for“slap+face” action. We can see that most of them are irrelevantto the action.

Figure 9. The mean precision of the top N relevant Web images
over all the 10 non-human actions

Figure 6. The top 18 Web images selected by Poselets for“wash+clothes” action. We can see that most of them represent“outdoor washing clothes using hands”

Figure 7. Some video shots of “wash+clothes” automatically extracted from the Web video dataset. We can see that most of thempresent “indoor washing clothes using washing machine”

tion. For non-human actions, we simply use top retrieved
images. The results demonstrated the effectiveness of our
proposed framework on categories whose precision is quite
low in [9].

References[1] M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri. Actions
as space-time shapes. In ICCV, 2005. 1[2] L. Bourdev and J. Malik. Poselets: Body part detectors trained using
3d human pose annotations. In ICCV, 2009. 1, 2, 3, 5

[3] N. I. Cinbins, R. G. Cinbins, and S. Sclaroff. Learning actions fromthe web. In ICCV, pp. 995–1002, 2009. 2[4] L. Duan, D. Xu, I. W. Tsang, and J. Luo. Visual event recognition invideos by learning from web data. In CVPR, 2010. 2[5] B. Herbert, E. Andreas, T. Tinne, and G. Luc. Surf: Speeded up
robust features. CVIU, pp. 346–359, 2008. 2, 3
[6] H. A. Jhuang, H. A. Garrote, E. A. Poggio, T. A. Serre, and T. .
Hmdb: A large video database for human motion recognition. In
ICCV, 2011. 1[7] Y. Jing and S. Baluja. Visualrank: Applying pagerank to large-scale
image search. PAMI, 30(11):1870–1890, 2008. 2, 3, 4
[8] J. Liu, J. Luo, and M. Shah. Recognizing realistic action from videos“in the wild”. In CVPR, 2009. 1[9] D. H. Nga and K. Yanai. Automatic construction of an action video
shot database using web videos. In ICCV, 2011. 1, 2, 3, 4, 5, 6
[10] A. Noguchi and K. Yanai. A surf-based spatio-temporal feature forfeature-fusion-based action recognition. In ECCV WS on Human
Motion: Understanding, Modeling, Capture and Animation, 2010.
2, 4[11] K. Schindler and L. van Gool. Action snippets: How many frames
does human action recognition require? In CVPR, 2008. 1[12] C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: A
local SVM approach. In ICPR, pp. 32–36, 2004. 1[13] C. Thurau and V. Hlavac. Pose primitive based human action recognition in videos or still images. In CVPR, 2008. 1[14] G. Toderici, H. Aradhye, M. Pasca, L. Sbaiz, and J. Yagnik. Finding
meaning on YouTube: Tag recommendation and category discovery.
In CVPR, pp. 3447–3454, 2010. 2[15] Z. Wang, M. Zhao, Y. Song, S. Kumar, and B. Li. YouTubeCat:
Learning to categorize wild web videos. In CVPR, 2010. 2[16] D. Weinland and E. Boyer. Action recognition using exemplar-based
embedding. In CVPR, 2008. 1[17] W. Yang, Y. Wang, and G. Mori. Recognizing human actions fromstill images with latent poses. In CVPR, 2010. 1