Objects2action: Classifying and localizing actions without any video example
Mihir Jain?
?

Jan C. van Gemert?‡

University of Amsterdam‡

Thomas Mensink?

Delft University of Technology†

Cees G. M. Snoek?†

Qualcomm Research Netherlands

Abstract
The goal of this paper is to recognize actions in video
without the need for examples. Different from traditional
zero-shot approaches we do not demand the design andspecification of attribute classifiers and class-to-attributemappings to allow for transfer from seen classes to unseenclasses. Our key contribution is objects2action, a semantic
word embedding that is spanned by a skip-gram model of
thousands of object categories. Action labels are assigned
to an object encoding of unseen video based on a convex
combination of action and object affinities. Our semantic
embedding has three main characteristics to accommodate
for the specifics of actions. First, we propose a mechanism to exploit multiple-word descriptions of actions andobjects. Second, we incorporate the automated selectionof the most responsive objects per action. And finally, wedemonstrate how to extend our zero-shot approach to thespatio-temporal localization of actions in video. Experiments on four action datasets demonstrate the potential of
our approach.

1. Introduction
We aim for the recognition of actions such as blow dryhair and swing baseball in video without the need for examples. The common computer vision tactic in such a challenging setting is to predict the zero-shot test classes fromdisjunct train classes based on a (predefined) mutual relationship using class-to-attribute mappings [1, 8, 20, 32, 36].
Drawbacks of such approaches in the context of action
recognition [24] are that attributes like ‘torso twist’ and‘look-down’ are difficult to define and cumbersome to annotate. Moreover, current zero-shot approaches, be it for image categories or actions, assume that a large, and labeled,
set of (action) train classes is available a priori to guide theknowledge transfer, but today’s action recognition practiceis limited to at most hundred classes [16, 19, 35, 42]. Different from existing work, we propose zero-shot learning
for action classification that does not require tailored definitions and annotation of action attributes, and not a single
video or action annotation as prior knowledge.

Figure 1. We propose objects2action, a semantic embedding toclassify actions, such as playing football, playing volleyball, andhorse-riding, in videos without using any video data or action annotations as prior knowledge. Instead it relies on commonly available textual descriptions, images and annotations of objects.

We are inspired by recent progress in supervised video
recognition, where several works successfully demonstrated
the benefit of representations derived from deep convolutional neural networks for recognition of actions [14,17,39]
and events [40, 49]. As these nets are typically pre-trainedon images and object annotations from ImageNet [5], andconsequently their final layer represent object category
scores, these works reveal that object scores are wellsuited for video recognition. Moreover, since these objects have a lingual correspondence derived from nouns in
WordNet, they are a natural fit for semantic word embeddings [6,9,26,29,41]. As prior knowledge for our zero-shot
action recognition we consider a semantic word embedding
spanned by a large number of object class labels and theirimages from ImageNet, see Figure 1.
Our key contribution is objects2action, a semantic embedding to classify actions in videos without using any
video data or action annotations as prior knowledge. Instead
it relies on commonly available object annotations, images
and textual descriptions. Our semantic embedding has threemain characteristics to accommodate for the specifics of actions. First, we propose a mechanism to exploit multipleword descriptions of actions and ImageNet objects. Sec-

ond, we incorporate the automated selection of the most responsive objects per action. And finally, we demonstrate
our zero-shot approach to action classification and spatiotemporal localization of actions.
Before going into detail, we will first connect our approach to related work on action recognition and zero-shotrecognition.

2. Related work
2.1. Action Recognition
The action classification literature offers a mature repertoire of elegant and reliable methods with good accuracy.
Many methods include sampling spatio-temporal descriptors [7, 47], aggregating the descriptors in a global video
representation, such as versions of VLAD [12, 33] or Fisher
Vectors [34] followed by supervised classification with an
SVM. Inspired by the success of deep convolutional neuralnetworks in image classification [18], several recent works
have demonstrated the potential of learned video representations for action and event recognition [14, 17, 39, 40, 49].
All these deep representations are learned from thousandsof object annotations [5], and consequently, their final output layer corresponds to object category responses indicating the promise of objects for action classification. We also
use a deep convolutional neural network to represent ourimages and video as object category responses, but we do
not use any action annotations nor any training videos.
Action classification techniques have recently been extended to action localization [2, 13, 21, 30, 46] where in addition to the class, the location of the action in the videois detected. To handle the huge search space that comeswith such precise localization, methods to efficiently sample action proposals [13, 30, 46] are combined with the encodings and labeled examples used in action classification.
In contrast, we focus on the zero-shot case where there isno labeled video data available for classification nor for localization. We are not aware of any other work on zero-shot
action localization.

2.2. Zero-Shot Recognition
The paradigm of zero-shot recognition became popularwith the seminal paper of Lampert et al. [20]. The idea isthat images can be represented by a vector of classification
scores from a set of known classes, and a semantic link can
be created from the known class to a novel class. Existingzero-shot learning methods can be grouped based on the different ways of building these semantic links.
A semantic link is commonly obtained by a human provided class-to-attribute mapping [1, 8, 20, 32], where foreach unseen class a description is given in terms of a setof attributes. Attributes should allow to tell classes apart,but should not be class specific, which makes finding good

attributes and designing class-to-attribute mappings a nontrivial task. To overcome the need for human selection, atleast partially, Rohrbach et al. evaluate external sources fordefining the class-to-attribute mappings [36]. Typically, attributes are domain specific, e.g. class and scene properties [20] or general visual concepts [23] learned from images, or action classes [37] and action attributes [24] learned
from videos. In our paper we exploit a diverse vocabularyof object classes for grounding unseen classes. Such a setup
has successfully been used for action classification when
action labels are available [14]. In contrast, we have a zeroshot setting and do not use any action nor video annotations.
Zero-shot video event recognition as evaluated in
TRECVID [31] offers meta-data in the form of a an event
kit containing the event name, a definition, and a precise
description in terms of salient concepts. Such meta-data
can cleverly be used for a class-to-attribute mapping basedon multi-modal concepts [11, 48], seed a sequence of multimodal pseudo relevance feedback [15], or select relevanttags from Flickr [3]. In contrast to these works, we do notassume any availability of additional meta-data and only
rely on the action name.
To generalize zero-shot classification beyond attributeto-class mappings, Mensink et al. [25] explored various
metrics to measure co-occurrence of visual concepts for establishing a semantic link between labels, and Froome etal. [9] and Norouzi et al. [29] exploit semantic word embeddings for this link. We opt for the latter direction, andalso use a semantic word embedding [22, 26, 27] since thisis the most flexible solution, and allows for exploiting object and action descriptions containing multiple words, suchas the WordNet synonyms and the subject, verb and object
triplets to describe actions [10, 43] used in this paper.

3. Objects2action
In zero-shot classification the train classes Y are different from the set of zero-shot test classes Z, such thatY ∩ Z = ∅. For training samples X , a labeled dataset
D ≡ {X , Y} is available, and the objective is to classify
a test sample as belonging to one of the test classes Z. Usually, test samples v are represented in terms of classification
scores for all train classes pvy ∀y ∈ Y, and an affinity score
gyz is defined to relate these train classes to the test classes.
Then the zero-shot prediction could be understood as a convex combination of known classifiers [1, 25, 29]:
X
C(v) = argmax
pvy gyz .
(1)zyOften there is a clear relation between training classes
Y and test classes Z, for example based on class-toattribute relations [1, 20] or all being nouns from the ImageNet/Wordnet hierarchy [9, 29]. It is unclear, however,3.1. Semantic embedding via Gaussian mixtures

Figure 2. Dataflow in objects2action. Intermediate processes, dataand corresponding symbols are specified in Section 3. Sparse
tranlsation is only shown for action to object affinity. Note that
we do not require any action class labeled visual examples nor any
video examples as prior knowledge.

how to proceed when train classes and test classes are semantically disjoint.
Our setup, see Figure 2, differs in two aspects to the standard zero-shot classification pipeline: i) our zero-shot test
examples are videos V to be classified in actions Z, while
we have a train set D with images X labeled with objects
Y derived from ImageNet [5]. Therefore, we aim to transfer from the domain of images X to the domain of videos
V, and ii) we aim to translate objects semantics Y to thesemantics of actions Z.
Object encoding We encode a test video v by the classification scores to the m = |Y | objects classes from thetrain set:
pv = [p(y1 |v), . . . , p(ym |v)]T(2)

where the probability of an object class is given by a deep
convolutional neural network trained from ImageNet [18],as recently became popular in the video recognition literature [14, 17, 39, 40, 49]. For a video v the probabilityp(y|v) is computed by averaging over the frame probabilities, where every 10th frame is sampled. We exploit thesemantics of in total 15,293 ImageNet object categories forwhich more than 100 examples are available.
We define the affinity between an object class y and action class z as:
gyz = s(y)T s(z),(3)

where s(·) is a semantic embedding of any class Z ∪ Y,
and we use gz = [s(y1 ) . . . s(ym )]T s(z) to represent thetranslation of action z in terms of objects Y. The semantic
embedding function s is further detailed below.

The objective for a semantic embedding is to find a ddimensional space, in which the distance between an object
s(y) and an action s(z) is small, if and only if their classes
y and z are found in similar (textual) context. For this we
employ the skip-gram model of word2vec [26,27] as semantic embedding function, which results in a look-up table foreach word, corresponding to a d-dimensional vector.
Semantic word embeddings have been used for zero-shot
object classification [29], but in our setting the key differences are i) that train and test classes come from different
domains: objects in the train set and actions in the test set;
and ii) both the objects and actions are described with asmall description instead of a single word. In this section we
describe two embedding techniques to exploit these multiword descriptions to bridge the semantic gap between objects and actions.
Average Word Vectors (AWV) The first method to
exploit multiple words is take the average vector of the embedded words [28]. The embedding s(c) of a multi-words
description c is given by:
1 X
s(w).
(4)
sA (c) =|w| w∈c
This model combines words to form a single average word,
as represented with a vector inside the word embedding.
While effective, this cannot model any semantic relations
that may exist between words. For example, the relations
for the word stroke, in the sequence stroke, swimming, water is completely different than the word relations in the sequence stroke, brain, ambulance.
Fisher Word Vectors (FWV) To describe the precise meaning of distinct words we propose to aggregate theword embeddings using Fisher Vectors [38]. While thesewere originally designed for aggregating local image descriptors [38], they can be used for aggregating words aslong as the discrete words are transformed into a continuous space [4]. In contrast to [4], where LSI is used toembed words into a continuous space, we employ the wordembedding vectors of the skip-gram model. These vectors
for each word are then analogous to local image descriptors
and a class description is analogous to an image.
The advantage of the FWV model is that it uses an underlying generative model over the words. This generativemodel is modeling semantic topics within the word embedding. Where AWV models a single word, the FWV models
a distribution over words. The stroke example could for example be assigned to two clear, distinct topics infarct andswimming. This word sense disambiguation leads to a more
precise semantic grounding at the topic-level, as opposed tosingle word-level.
In the Fisher Vector, a document (i.e. a set of words)
is described as the gradient of the log-likelihood of theseobservations on an underlying probabilistic model. Following [38] we use a diagonal Gaussian Mixture Model withk components as probabilistic model, which we learn on
approximately 45K word embedding vectors from the 15K
object classes in ImageNet.
The Fisher Vectors with respect to the mean µk and variance σk of mixture component k are given by:
s(w) − µk
1 Xcγw (k),(5)Gµk = √
πk w∈cσk(s(w) − µk )2
1 X
c
−1 ,
(6)
Gσk = √
γw (k)
σk2
2πk w∈cwhere πk is the mixing weight, and γw (k) denotes the responsibility of component k and we use the closed-form approximation of the Fisher information matrix of [38]. The
final Fisher Word Vector is the concatenation of the Fisher
Vectors (Eq. (5) and Eq. (6)) for all components:sF (c) = [Gµc 1 , Gσc 1 , . . . , Gµc k , Gσc k ]T .

(7)

3.2. Sparse translations
The action representation gz represents the translation of
the action to all objects in Y. However, not all train objects
are likely to contribute to a clear description of a specificaction class. For example, consider the action class kayaking, it makes sense to translate this action to object classes
such as kayak, water, and sea, with some related additional
objects like surf-board, raft, and peddle. Likewise a similarity value with, e.g., dog or cat is unlikely to be beneficial
for a clear detection, since it introduces clutter. We consider
two sparsity metrics that operate on the action classes or thetest video.
Action sparsity We propose to sparsify the representation gz by selecting the Tz most responsive object classes
to a given action class z. Formally, we redefine the action
to object affinity as:ĝz = [gzy1 δ(y1 , Tz ), . . . , gzym δ(ym , Tz )]T(8)

where δ(yi , Tz ) is an indicator function, returning 1 if class
yi is among the top Tz classes. In the same spirit, the objects could also have been selected based on their distance,
considering only objects within an z distance from s(z).
We opt for the selection of the top Tz documents, since it iseasier to define an a priori estimate of the value. Selecting
Tz objects for an action class z means that we focus only on
the object classes that are closer to the action classes in thesemantic space.
Video sparsity Similarly, the video representation pv
is, by design, a dense vector, where each entry contains the(small) probability p(y|v), of the presence of train class yin the video v. We follow [29] and use only the top Tv mostprominent objects present in the video:p̂v = [p(y1 |v)δ(y1 , Tv ), . . . , p(ym |v)δ(ym , Tv )]T(9)

where δ(yi , Tv ) is an indicator function, returning 1 if class
yi is among the top Tv classes. Increasing the sparsity, by
considering only the top Tv class predictions will reduce theeffect of adding random noise by summing over a lot of unrelated classes with a low probability mass and is therefore
likely to be beneficial for zero-shot classification.
The optimal values for both Tz and Tv are likely to depend on the datasets, the semantic representation and thespecific action description. Therefore they are consideredas hyper-parameters of the model. Typically, we would expect that T  m, e.g., the 50 most responsive object classes
will suffice for representing the video and finding the best
action to object description.

3.3. Zero-shot action localization
Objects2action is easily extendable to zero-shot localization of actions by exploiting recent advances in sampling
spatio-temporal tube-proposals from videos [13, 30, 46].
Such proposals have shown to give a high localization recall with a modest set of proposals.
From a test video, a set U of spatio-temporal tubes are
sampled [13]. For each test video we simply select the maximum scoring tube proposal:Xpuy gyz ,(10)C(v) = argmaxz∈Z,u∈Uvywhere u denotes a spatio-temporal tube proposal, and puy
is the probability of the presence of object y in region u.
For the spatio-temporal localization, a tube proposal
contains a series of frames, each with a bounding-box indicating the spatial localization of the action. We feed justthe pixels inside the bounding-box to the convolutional neural network to obtain the visual representation embedded in
object labels. We will demonstrate the localization ability
in the experiments.

4. Experiments
In this section, we employ the proposed object2action
model on four recent action classification datasets. We firstdescribe these datasets and the text corpus used. Second,
we analyze the impact of applying the Fisher Word Vector
over the baseline of the Average Word Vector for computing
the affinity between objects and actions, and we evaluate theaction and video sparsity parameters. Third, we report zeroshot classification results on the four datasets, and we compare against the traditional zero-shot setting where actions
are used during training. Finally, we report performance ofzero-shot spatio-temporal action localization.

0.3

0.25

0.25

Average accuracy

Average accuracy

0.3

0.2

0.15

FWV: Tz
FWV: Tv
AWV: Tz
AWV: Tv

0.1

0.2

Tz =5
Tz =10Tz =50Tz =100
Tz =10k
Tz =15k

0.15

0.1

0.05

0.05
1

10

100

1k

10k

Number of objects selected (Tz or Tv)

1

10

100

1k

10k

Number of objects selected for test video (Tv)

Figure 3. Impact of video Tv and action Tz sparsity parameters, individually (left) and when combined (right) on UCF101 dataset.

4.1. Prior knowledge and Datasets
Our method is based on freely available resources which
we use as prior knowledge for zero-shot action recognition.
For the four action classification datasets datasets used weonly use the test set.
Prior knowledge We use two types of prior knowledge. First, we use deep convolutional neural network
trained from ImageNet images with objects [18] as visual
representation. Second, for the semantic embedding wetrain the skip-gram model of word2vec on the metadata (title, descriptions, and tags) of the YFCC100M dataset [44],this dataset contains about 100M Flickr images. Preliminary experiments showed that using visual metadata results
in better performance than training on Wikipedia or GoogleNews data. We attribute this to the more visual descriptions
used in the YFC100M dataset, yielding a semantic embedding representing visual language and relations.
UCF101 [42] This dataset contains 13,320 videos of
101 action classes. It has realistic action videos collectedfrom YouTube and has large variations in camera motion,
object appearance/scale, viewpoint, cluttered background,illumination conditions, etc. Evaluation is measured using
average class accuracy, over the three provided test-splits
with around 3,500 videos each.
THUMOS14 [16] This dataset has the same 101 action classes as in UCF101, but the videos are have a longer
duration and are temporally unconstrained. We evaluate on
the testset containing 1,574 videos, using mean average precision (mAP) as evaluation measure.
HMDB51 [19] This dataset contains 51 action classes
and 6,766 video clips extracted from various sources, ranging from YouTube to movies, and hence this dataset contains realistic actions. Evaluation is measured using averageclass accuracy, over the three provided test-splits with each
30 videos per class (1,530 videos per split).
UCF Sports [35] This dataset contains 150 videos ofEmbeddingSparsityBest
Accuracy at
accuracy Tz =10, Tv =100
Video
18.0%
17.5%
AWV
Action22.7%
21.9%
Combine
22.7%
21.6%
Video
29.1%
29.0%
FWV
Action30.8%
30.3%
Combine30.8%
30.3%
Table 1. Evaluating AWV and FWV for object to class affinity, andcomparing action and video sparsity on UCF101 dataset.

10 action classes. The videos are from sports broadcasts
capturing sport actions in dynamic and cluttered environments. Bounding box annotations are provided and thisdataset is often used for spatio-temporal action localization.
For evaluation we use the test split provided by [21] andperformance is measured by average class accuracy.

4.2. Properties of Objects2action
Semantic embedding We compare the AWV with the
FWV as semantic embedding. For the FWV, we did a run ofpreliminary experiments to find suitable parameters for thenumber of components (varying k = {1, 2, 4, 8, 16, 32}),
the partial derivatives used (weight, mean, and/or variance)
and whether to use PCA or not. We found them all to perform rather similar in terms of classification accuracy. Considering a label has only a few words (1 to 4), we therefore
fix k = 2, apply PCA to reduce dimensionality by a factorof 2, and to use only the partial derivatives w.r.t. the mean(conforming the results in [4]). Hence, the total dimensionality of FWV is d, equivalent to the dimensionality of
AWV, which allows for a fair comparison. The two embeddings are compared in Table 1 and Figure 3 (left), and FWV
clearly outperforms AWV in all the cases.
Sparsity parameters In Figure 3, we evaluate the action sparsity and video sparsity parameters. The left plotshows average accuracy versus Tz and Tv . It is evident that

Embedding Sparsity UCF101 HMDB51 THUMOS14 UCF Sports
None
16.7%
8.0%
4.4%
13.9%
AWV
Video
17.5%
7.7%
10.7%
13.9%
Action
21.9%
9.9%
19.9%
25.6%
None
28.7%
14.2%
25.9%
23.1%
FWV
Video
29.0%
14.5%
27.8%
23.1%
Action
30.3%
15.6%
33.4%
26.4%
Supervised
63.9%
35.1%
56.3%
60.7%
Table 2. Evaluating semantic embeddings, action and video spartsity: Average accuracies (mAP for THUMOS14) for the four datasets.
Action sparsity and FWV both boost the performance consistently. Supervised upper-bound using object scores as representation.
THUMOS14 test set

0.4

0.3

mAP

action sparsity, i.e., selecting most responsive object classes
for a given action class leads to a better performance thanvideo sparsity. The video sparsity (green lines) is more stable throughout and achieves best results in the range of 10
to 100 objects. Action sparsity is a bit sinuous, nevertheless it always performs better, independent of the type of
embedding. Action sparsity is at its best in the range of
selecting the 5 to 30 most related object classes. For the remaining experiments, we fix these parameters as Tz = 10
and Tv = 100.
We also consider the case when we apply sparsity on
both video and actions (see the right plot). Applying sparsity on both sides does not improve performance, it is equivalent to the best action sparsity setting, showing that selecting the most prominent objects per action suffice for zeroshot action classification. Table 1 summarise the accuracies
for the best and fixed choices of Tz and Tv .

0.2

0.1

FWV, Tz =10
Objects
MBH+FV

0
0

1

2

3

4

5

6

7

8

9

10

11

Number of training examples per class

4.3. Zero-shot action classification

Figure 4. Our approach compared with the supervised classification with few examples per class: State-of-the-art object and motion representations respectively require 4 and 10 examples perclass to catch up with our approach, which uses no example.

In this section we employ the obtained parameters of Object2action, from the previous section, for zero-shot action
classification on the test splits of all four datasets. We evaluate the benefit of using the FWV over AWV, and the effect of using sparsity (video sparsity, action sparsity or nosparsity at all). The results are provided in Table 2. We
observe that the FWV always outperforms AWV, and that
it is always beneficial to apply sparsity, and action sparsity
with FWV performs the best. We also provide the supervised upper-bounds using the same video representation of
object classification scores in Table 2. Here and for all theexperiments, we power normalize (α = 0.5) the video representations before applying `2 normalization.
Comparison to few-shot supervised learning In thisexperiment we compare the zero-shot classifier against fewshot supervised learning, on the THUMOS14 dataset. For
this we consider two types of video representation. The first
representations, uses the state-of-the-art motion representation of [47], by encoding robust MBH descriptors along theimproved trajectories [47] using Fisher Vectors. We follow the standard parameter cycle, by applying PCA, using
a GMM with K = 256 Gaussians, employing power and`2 normalization. The second representation uses the object

scores pv of a video, here also we apply power and `2 normalization. For both representations, we train one-vs-rest
linear SVM classifiers and we average performance over 20
runs for every given number of train examples.
The results in mAP are shown in Figure 4. Interestingly,
to perform better than our zero-shot classification, fully supervised classification setup requires 4 and 10 samples perclass for object and motion representations respectively.
Object transfer versus action transfer We now experiment with the more conventional setup for zero-shotlearning, where we have training data for some action
classes, disjoint from the set of test action classes. We keep
half of the classes of a given dataset as train labels and theother half as our zero-shot classes. The action classifiers
are learned from odd (or even) numbered classes and videos
from the even (or odd) numbered classes are tested.
We evaluate two types of approaches for action transfer,i.e., when training classes are also actions. The first method
uses the provided action attributes for zero-shot classification with direct attribute prediction [20]. Since attributes areavailable only for UCF101, we experiment on this dataset.

Method

Train
Even
Odd
Even
Odd

Test UCF101 HMDB51
Odd
16.2%
—
Action attributes
Even
14.6%
—
Odd
16.1%
12.4%
Action labels
Even
14.4%
13.4%
Odd
37.3%
15.0%
Objects2action ImageNet
Even
38.9%
24.5%
Table 3. Object transfer versus action transfer in a conventional
zero-shot set-up. Direct attribute prediction [20] is used with action attributes, FWV is used to embed action labels, and in our
objects2action.

MethodmAP
Wu et al. [48] (Google images)
1.21%
Chen et al. [3] (Flickr images)
2.40%
Wu et al. [48] (YouTube thumbnails) 3.48%
AWV
3.49%
Objects2action
FWV
4.21%
Table 4. Zero-shot event retrieval on TRECVID13 MED testset:
Comparison with the state-of-the-art methods having similar zeroshot setup as ours. Inspite of using only event labels and images, we outperform methods that use event description and video
thumbnails.
0.6
0.5
0.4

AUC

The train videos of the training classes are used to learn
linear SVMs for the provided 115 attributes. The second
method uses action labels embedded by FWV to computeaffinity between train and test action labels. We use thesame GMM with k = 2 components learned on ImageNet
object labels. Here linear SVMs are learned for the training action classes. The results are reported for UCF101 andHMDB51 datasets. For both the above approaches for action transfer, we use MBH descriptors encoded by Fisher
vectors for video representation. The results are reported inTable 3.
For comparison with our approach, the same setup of
testing on odd or even numbered classes is repeated withthe object labels. The training set is ImageNet objects, so
no video example is used for training. Table 3 compares object transfer and action transfer for zero-shot classification.
Object transfer leads to much better learning compared toboth the methods for action transfer. The main reason for
the inferior performance using actions is that there are notenough action labels or action attributes to describe the testclasses, whereas from 15k objects there is a good chance tofind a few related object classes.
Zero-shot event retrieval We further demonstrate ourmethod on the related problem of zero-shot event retrieval.
We evaluate on the TRECVID13 MED [31] testset for EK0
task. There are 20 event classes and about 27,000 videos
in the testset. Instead of using the manually specified event
kit containing the event name, a definition, and a precise
description in terms of salient attributes, we only rely on
the class label. In Table 4, we report mAP using event labels embedded by AWV and FWV. We also compare withthe state-of-the-art approaches of Chen et al. [3] and Wu etal. [48] reporting their settings that are most similar to ours.
They learn concept classifiers from images (from Flickr,
Google) or YouTube video thumbnails, be it that they also
use the complete event kit description. Using only the event
labels, both of our semantic embeddings outperform thesemethods.
Free-text action search As a final illustration weshow in Figure 6 qualitative results from free-text querying of action videos from the THUMOS14 testset. We used

FWV, Tz =10
Objects
MBH+FVLan et al.

0.3
0.2
0.1
0
0.1

0.2

0.3

0.4

0.5

0.6

Overlap threshold

Figure 5. Action localization without video example on UCF
Sports: AUCs for different overlap thresholds are shown for Tz =10 and also for the fully supervised setting with motion and object
representations. The performance is promising considering no example videos are used.

the whole dataset for querying, and searched for actions
that are not contained in the 101 classes of THUMOS14.
Results show that free-text querying offers a tool to explore a large collection of videos. Results are best whenthe query is close to one or a few existing action classes,
for example “Dancing” retrieves results from “salsa-spin”
and other dancing clips. Our method fails for the query “hitwicket”, although it does find cricket matches. Zero shot action recognition through an object embedding unlocks freetext querying without using any kind of expensive video annotations.

4.4. Zero-shot action localization
In our final experiment, we aim to localize actions invideos, i.e., detect when and where an action of interest occurs. We evaluate on the UCF Sports dataset, following thelatest convention to localize an action spatio-temporally asa sequence of bounding boxes [13,21,45]. For sampling theaction proposal, we use the tubelets from [13] and computeobject responses for each tubelet of a given video. We compare with the fully supervised localization using the object
and motion representations described in Section 4.3. The
top five detections are considered for each video after nonmaximum suppression.

Fight in ring

Dancing

Martial arts

Smelling food

Hit wicket

Figure 6. Illustration of never seen actions on THUMOS14 testset: For a given textual action query the top five retrieved videos are shown.
The 101 classes of THUMOS14 do not contain these five action label queries. The first two queries are somewhat close to classes ‘Sumo
wrestling’ and ‘Salsa spin’. All retrieved videos for the query: ‘Fight in ring’ include sumo wrestling. The videos retrieved for the second
query: ‘Dancing’ also includes two instances of dancing other than salsa. All results for the these two queries are technically correct. The
third query ‘Martial arts’ finds mostly gymnasts, and a karate match. The fourth query is: ‘Smelling food’, where we still obtain cakes,
food items and dining table in the background. For the fifth query: ‘hit wicket’ (in cricket) we do not succeed but retrieve some cricket
videos. This illustration shows the potential for free keyword querying of action classes without using any examples.

The three are compared in Figure 5, which plots area
under the ROC (AUC) for varying overlap thresholds. We
also show the results of another supervised method of Lan etal. [21]. It is interesting to see that for higher thresholds ourapproach performs better. Considering that we do not use
any training example it is an encouraging result. There areother state-of-the-art methods [13, 14, 45] not shown in the
figure to avoid clutter. These methods achieve performancecomparable to or lesser than our supervised case.
For certain action classes many objects and scene fromthe context might not be present in the groundtruth tubelets.
Still our approach finds enough object classes for recognizing the zero-shot classes in the tubelets, as we have large
number of train classes. In contrast, finding atomic partsof actions such as ‘look-up’, ‘sit-down’, ‘lift-leg’ etc are
difficult to collect or annotate. This is one of the most critical advantages we have with objects, that it is easier to findmany object or scene categories.

tions are completely avoided by using abundantly available
object images and labels and a freely available text corpus
to relate actions into an object embedding. In addition, we
showed that modeling a distribution over embedded words
with the Fisher Vector is beneficial to obtain a more precise sense of the unseen action class topic, as compared toa word embedding based on simple averaging. We exploredsparsity both in the object embedding, as well as in the unseen action class, showing that sparsity is beneficial overmere feature-dimensionality.
We validate our approach on four action datasets andachieve promising results for action classification and localization. We also demonstrate our approach for action andevent retrieval on THUMOS14 and TRECVID13 MED respectively. The most surprising aspect of our objects2actionis that it can potentially find any action in video, without
ever having seen the action before.

5. Conclusion
We presented a method for zero shot action recognition
without using any video examples. Expensive video annota-

Acknowledgments This research is supported by the STW
STORY project and the Dutch national program COMMIT.

References[1] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid. Labelembedding for attribute-based classification. In CVPR, 2013. 1, 2
[2] L. Cao, Z. Liu, and T. S. Huang. Cross-dataset action detection. In
CVPR, 2010. 2[3] J. Chen, Y. Cui, G. Ye, D. Liu, and S.-F. Chang. Event-driven semantic concept discovery by exploiting weakly tagged internet images. In
ICMR, 2014. 2, 7
[4] S. Clinchant and F. Perronnin. Textual similarity with a bag-ofembedded-words model. In ICTIR, 2013. 3, 5
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
1, 2, 3[6] M. Elhoseiny, B. Saleh, and A. Elgammal. Write a classifier: Zeroshot learning using purely textual descriptions. In ICCV, 2013. 1[7] I. Everts, J. C. van Gemert, and T. Gevers. Evaluation of color
spatio-temporal interest points for human action recognition. TIP,
23(4):1569–1580, 2014. 2[8] A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. Describing objects
by their attributes. In CVPR, 2009. 1, 2
[9] A. Frome, G. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato,
and T. Mikolov. Devise: A deep visual-semantic embedding model.
In NIPS, 2013. 1, 2
[10] S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan, R. Mooney, T. Darrell, and K. Saenko. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In ICCV, 2013. 2[11] A. Habibian, T. Mensink, and C. Snoek. Composite concept discovery for zero-shot video event detection. In ICMR, 2014. 2[12] M. Jain, H. Jégou, and P. Bouthemy. Better exploiting motion for
better action recognition. In CVPR, 2013. 2[13] M. Jain, J. van Gemert, H. Jégou, P. Bouthemy, and C. Snoek. Action
localization with tubelets from motion. In CVPR, 2014. 2, 4, 7, 8
[14] M. Jain, J. van Gemert, and C. Snoek. What do 15,000 object categories tell us about classifying and localizing actions? In CVPR,
2015. 1, 2, 3, 8
[15] L. Jiang, T. Mitamura, S. Yu, and A. Hauptmann. Zero-example
event search using multimodal pseudo relevance feedback. In ICMR,
2014. 2[16] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah,and R. Sukthankar. THUMOS challenge: Action recognition with alarge number of classes. http://crcv.ucf.edu/THUMOS14/,
2014. 1, 5
[17] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and
L. Fei-Fei. Large-scale video classification with convolutional neuralnetworks. In CVPR, 2014. 1, 2, 3
[18] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 2, 3,
5[19] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB:
A large video database for human motion recognition. In ICCV,2011. 1, 5[20] C. Lampert, H. Nickisch, and S. Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In CVPR,
2009. 1, 2, 6, 7
[21] T. Lan, Y. Wang, and G. Mori. Discriminative figure-centric models
for joint action localization and recognition. In ICCV, 2011. 2, 5, 7,
8[22] Q. Le and T. Mikolov. Distributed representations of sentences anddocuments. In ICML, 2014. 2[23] L.-J. Li, H. Su, Y. Lim, and L. Fei-Fei. Object bank: An objectlevel image representation for high-level visual recognition. IJCV,
107(1):20–39, 2014. 2[24] J. Liu, B. Kuipers, and S. Savarese. Recognizing human actions byattributes. In CVPR, 2011. 1, 2

[25] T. Mensink, E. Gavves, and C. Snoek. Costa: Co-occurrence statistics for zero-shot classification. In CVPR, 2014. 2[26] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimationof word representations in vector space. In ICLR, 2013. 1, 2, 3[27] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013. 2, 3
[28] D. Milajevs, D. Kartsaklis, M. Sadrzadeh, and M. Purver. Evaluatingneural word representations in tensor-based compositional settings.
In EMNLP, 2014. 3[29] M. Norouzi, T. Mikolov, S. Bengio, Y. Singer, J. Shlens, A. Frome,
G. Corrado, and J. Dean. Zero-shot learning by convex combinationof semantic embeddings. In ICLR, 2014. 1, 2, 3, 4[30] D. Oneata, J. Revaud, J. Verbeek, and C. Schmid. Spatio-temporal
object detection proposals. In ECCV, 2014. 2, 4[31] P. Over, G. Awad, J. Fiscus, and G. Sanders. Trecvid 2013–an introduction to the goals, tasks, data, evaluation mechanisms, and metrics.
In TRECVID Workshop, 2013. 2, 7
[32] D. Parikh and K. Grauman. Relative attributes. In ICCV, 2011. 1, 2
[33] X. Peng, L. Wang, Y. Qiao, and Q. Peng. Boosting vlad with supervised dictionary learning and high-order statistics. In ECCV, 2014.
2[34] X. Peng, C. Zou, Y. Qiao, and Q. Peng. Action recognition withstacked fisher vectors. In ECCV, 2014. 2[35] M. D. Rodriguez, J. Ahmed, and M. Shah. Action MACH: a spatiotemporal maximum average correlation height filter for action recognition. In CVPR, 2008. 1, 5[36] M. Rohrbach, M. Stark, and B. Schiele. Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In CVPR, 2011. 1,
2[37] S. Sadanand and J. J. Corso. Action bank: A high-level representation of activity in video. In CVPR, 2012. 2[38] J. Sánchez, F. Perronnin, T. Mensink, and J. Verbeek. Image classification with the fisher vector: Theory and practice. IJCV, 2013. 3,
4[39] K. Simonyan and A. Zisserman. Two-stream convolutional networks
for action recognition in videos. In NIPS, 2014. 1, 2, 3[40] C. Snoek et al. MediaMill at TRECVID 2013: Searching concepts,
objects, instances and events in video. In TRECVID, 2013. 1, 2, 3[41] R. Socher, M. Ganjoo, C. D. Manning, and A. Ng. Zero-shot learning
through cross-modal transfer. In NIPS, 2013. 1[42] K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101
human actions classes from videos in the wild. CoRR, 2012. 1, 5[43] C. Sun and R. Nevatia. Semantic aware video transcription using
random forest classifiers. In ECCV, 2014. 2[44] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni,
D. Poland, D. Borth, and L.-J. Li. The new data and new challenges
in multimedia research. arXiv preprint arXiv:1503.01817, 2015. 5[45] Y. Tian, R. Sukthankar, and M. Shah. Spatiotemporal deformablepart models for action detection. In CVPR, 2013. 7, 8
[46] J. van Gemert, M. Jain, E. Gati, and C. Snoek. APT: Action localization proposals from dense trajectories. In BMVC, 2015. 2, 4[47] H. Wang and C. Schmid. Action recognition with improved trajectories. In ICCV, 2013. 2, 6
[48] S. Wu, S. Bondugula, F. Luisier, X. Zhuang, and P. Natarajan. Zeroshot event detection using multi-modal fusion of weakly supervised
concepts. In CVPR, 2014. 2, 7
[49] Z. Xu, Y. Yang, and A. G. Hauptmann. A discriminative CNN video
representation for event detection. In CVPR, 2015. 1, 2, 3