EXMOVES: Classifier-based Features for Scalable Action Recognition
Authors: Du Tran, Lorenzo Torresani
ICLR 2014

Abstract: This paper introduces EXMOVES, learned exemplar-based features for efficient recognition of actions in videos. The entries in our descriptor are produced by evaluating a set of movement classifiers over spatial-temporal volumes of the input sequence. Each movement classifier is a simple exemplar-SVM trained on low-level features, i.e., an SVM learned using a single annotated positive space-time volume and a large number of unannotated videos. Our representation offers two main advantages. First, since our mid-level features are learned from individual video exemplars, they require minimal amount of supervision. Second, we show that simple linear classification models trained on our global video descriptor yield action recognition accuracy approaching the state-of-the-art but at orders of magnitude lower cost, since at test-time no sliding window is necessary and linear models are efficient to train and test. This enables scalable action recognition, i.e., efficient classification of a large number of different actions even in large video databases. We show the generality of our approach by building our mid-level descriptors from two different low-level feature representations. The accuracy and efficiency of the approach are demonstrated on several large-scale action recognition benchmarks.


Peer review:
[Review]:
The paper describes a mid-level representation for videos that can be faster than existing representations and yields similar performance. The idea is to train many SVMs to detect predefined action instances on sub-blocks from the video, and then to aggregate the SVM responses into a representation for the whole video.

This work seems like a fairly straightforward extension of previous similar work that was done on images (Malisiewicz et al.), but there are some technical differences like the use of an integral video trick to compute SVM responses fast, which seems nice.

I don't really understand the mining of negative examples for training the exemplar SVMs. Why is it not possible to train the SVM, say with stochastic gradient descent, on all or many negative examples?

The method relies on extra, intermediate labels for training which are not used by bag-of-words. This makes it hard to judge the performance differences between the two and seems to be unfair towards bag-of-words. 

3.3: Rather than learning to scale svm confidences within a sigmoid, why not train a regularized logistic regression classifier in the first place, instead of the svm?

The feature extraction time is reported as the total on the whole UT-I dataset. It would be much better to report it in frames per second to make it comparable with other datasets without having to dig up the description of this dataset. If I am not mistaken it amounts to approximately 5 frames (with more or less standard resolution) per second? If correct, this means that despite the improvement over action bank, the bottleneck is really feature exctraction not classification. So the speed up due to the linear classifier will be swamped by the feature extraction and is not really that relevant, unless I'm missing something. 

pro:
- Well-written, uses some nice engineering tricks like the integral video for computing SVM responses.


neg:
- The paper seems like a slightly strange fit for this conference as it describes a vision system rather than investigating the learning of representations. That intermediate labels are useful on this data is well-known (and unsurprising). The paper does propose a faster way to use them, which is probably worthwhile.