EXMOVES: Classifier-based Features for Scalable Action Recognition
Authors: Du Tran, Lorenzo Torresani
ICLR 2014

Abstract: This paper introduces EXMOVES, learned exemplar-based features for efficient recognition of actions in videos. The entries in our descriptor are produced by evaluating a set of movement classifiers over spatial-temporal volumes of the input sequence. Each movement classifier is a simple exemplar-SVM trained on low-level features, i.e., an SVM learned using a single annotated positive space-time volume and a large number of unannotated videos. Our representation offers two main advantages. First, since our mid-level features are learned from individual video exemplars, they require minimal amount of supervision. Second, we show that simple linear classification models trained on our global video descriptor yield action recognition accuracy approaching the state-of-the-art but at orders of magnitude lower cost, since at test-time no sliding window is necessary and linear models are efficient to train and test. This enables scalable action recognition, i.e., efficient classification of a large number of different actions even in large video databases. We show the generality of our approach by building our mid-level descriptors from two different low-level feature representations. The accuracy and efficiency of the approach are demonstrated on several large-scale action recognition benchmarks.


Peer review:
[Review]:
We thank the reviewer for the comments and for the useful suggestions to improve the paper. In the final version of our article we will address these issues as follows:
- We will clarify further that having more than one example per basis-activity yields improved accuracy, as shown in the experiments.
- We will stress that tags and similar metadata information can be exploited to select the negative videos. We already hint to it in the first paragraph of section 3.2.
- We will specify the dimensionality of all descriptors in our experiments. In any case these are indeed comparable to the dimensionality of our feature vector: EXMOVES contain 41,172 features, Action Bank has 44,895 entries, while BOWs based on Dense Trajectories have dimensionality equal to 25,000.

As for the large-scale experiments involving Action Bank, for UCF50 we used the accuracy number reported by the authors in their paper. We have estimated that extracting Action Bank features for UCF101 (part 2) would take 132 days by using 10 nodes of our cluster exclusively for this computation. For this reason we are unable to carry out such evaluation. Extraction of Action Bank features from Hollywood-2 would take even longer.

As mentioned in the conclusions, we will release the software implementing our features and all the data used in our experiments.