EXMOVES: Classifier-based Features for Scalable Action Recognition
Authors: Du Tran, Lorenzo Torresani
ICLR 2014

Abstract: This paper introduces EXMOVES, learned exemplar-based features for efficient recognition of actions in videos. The entries in our descriptor are produced by evaluating a set of movement classifiers over spatial-temporal volumes of the input sequence. Each movement classifier is a simple exemplar-SVM trained on low-level features, i.e., an SVM learned using a single annotated positive space-time volume and a large number of unannotated videos. Our representation offers two main advantages. First, since our mid-level features are learned from individual video exemplars, they require minimal amount of supervision. Second, we show that simple linear classification models trained on our global video descriptor yield action recognition accuracy approaching the state-of-the-art but at orders of magnitude lower cost, since at test-time no sliding window is necessary and linear models are efficient to train and test. This enables scalable action recognition, i.e., efficient classification of a large number of different actions even in large video databases. We show the generality of our approach by building our mid-level descriptors from two different low-level feature representations. The accuracy and efficiency of the approach are demonstrated on several large-scale action recognition benchmarks.


Peer review:
[Review]:
This paper proposes a novel method for human activity recognition in video. The main properties it seeks are: 1) developing invariance to the various types of intra-class variation; 2) efficiency in training - both in terms of complexity and minimizing human involvement, 3) efficiency at test time (e.g. able to handle YouTube-scale collections). The main idea is to train, from typical low-level bag-of-visual-words features, a series of exemplar-SVMs which learn on a single positive example and many negative examples, and use the collective output of the SVMs at multiple scales and positions as a mid-level descriptor. The final stage of the method is a linear SVM classifier. The method performs comparably, or better than other recent methods on three modern datasets: HMDB51, Hollywood2, UCF50/101, moreover it is up to two orders of magnitude more efficient than the other methods.

The main contribution is a simple yet effective method that can perform activity recognition on large video databases. It would be easy to re-implement, though I hope the authors release a baseline implementation. As the idea of developing invariance to intra-class variability is a key consideration in the motivation of the work, I'd like to see more discussion on this. To me, the fact that the SVMs are trained on single exemplars is an interesting feature and certainly important when human labeling is expensive. However, this seems at odds with learning invariance. For example, if we only ever see one example of someone performing a tennis swing, will this not make it much more difficult to recognize the tennis swing under different conditions: clothing, body type, scale, etc.?

Positives
* Paper is well written and the description of the method is clear. In particular, the presentation of Algorithm 1 and its associated description in the text is good.
* As stated earlier, it's a simple and effective method, and I see others using this as a baseline
* Datasets considered are modern and challenging
* Speedup over other methods is impressive: key to this is the use of the integral video, a nice idea

Negatives
* Paper is very specific to an application and type of dataset, may not be as of wide of interest as some other papers
* Related to above, the representation extracted likely would not be useful to other AI problems (e.g. those with structured output)
* Method still requires first stage of engineered feature extraction pipeline

Overall, while it may not have the broad appeal across ICLR, I think this is a good paper which clearly presents an effective method for activity recognition.

Questions & Comments
====================
Emphasize at the beginning that 'exemplars' do not necessary have to be 1 per activity type. This was the impression I got at the beginning of the paper, but the description of the experiments (e.g. 188 exemplars and 50 actions with multiple exemplars per action) made this clear. 

As mentioned, there is still some human effort in finding the 'negative' examples. For example, one needs to watch a video to determine for sure that it does not contain any tennis. Perhaps you could use tags or some other means to propose negative videos that you are very confident don't contain a particular activity. In that case, how sensitive would the exemplar-SVM be to label noise (if the negative videos contained volumes semantically very similar to that of the exemplar)?

The final dimensionality of the EXMOVE descriptor is 41,172 which is going to benefit the linear SVM. Is the dimensionality of the other methods comparable? You may want to indicate this somewhere, say in Table 1.

It would be nice to see Table 1 contain more low-level feature/mid-level descriptor pairs (e.g. the outer product of low-level features and mid-level descriptors). 

With respect to the comment 'we were unable to test [Action Bank] on the large-scale datasets of Hollywood-2 and UCF101. For Hollywood2, I can understand that it's the case. But would UCF101 not just be roughly 2x the time to train on UCF50? Can you clarify this? I realize it may have just not been a situation of not enough time before the deadline to run the experiment, so could this be reported in the final version of the paper?