Evaluation of Local Spatio-temporal Features for Action Recognition

Local space-time features have recently become a popular video representation for action recognition. Several methods for feature localization and description have been proposed in the literature, and promising recognition results were demonstrated for different action datasets. The comparison of those methods, however, is limited given the different experimental settings and various recognition methods used. The purpose of this paper is first to define a common evaluation setup to compare local space-time detectors and descriptors. All experiments are reported for the same bag-of-features SVM recognition framework. Second, we provide a systematic evaluation of different spatio-temporal features. We evaluate the performance of several space-time interest point detectors and descriptors along with their combinations on datasets with varying degree of difficulty. We also include a comparison with dense features obtained by regular sampling of local space-time patches. Feature detectors. In our experimental evaluation, we consider the following feature detectors. (1) The Harris3D detector  extends the Harris detector for images to image sequences. At each video point, a spatio-temporal second-moment matrix μ is computed using a separable Gaussian smoothing function and space-time gradients. Interest points are located at local maxima of H = det(μ)− k trace3(μ). (2) The Cuboid detector  is based on temporal Gabor filters. The response function has the form: R = (I ∗ g ∗ hev) + (I ∗ g ∗ hod), where g(x,y;σ) is the 2D Gaussian smoothing kernel, and hev and hod are 1D Gabor filters. Interest points are detected at local maxima of R. (3) The Hessian detector [6] is a spatio-temporal extension of the Hessian saliency measure. The determinant of the 3D Hessian matrix is used to measure saliency. The determinant of the Hessian is computed over several spatial and temporal scales. A non-maximum suppression algorithm selects extrema as interest points. (4) Dense sampling extracts multi-scale video blocks at regular positions in space and time and for varying scales. In our experiments, we sample cuboids with 50% spatial and temporal overlap. Feature descriptors. The following feature descriptors are investigated. (1) For the Cuboid descriptor , gradients computed for each pixel in a cuboid region are concatenated into a single vector. PCA projects vectors to a lower dimensional space. (2) The HOG/HOF descriptors  divide a cuboid region into a grid of cells. For each cell, 4-bin histograms of gradient orientations (HOG) and 5-bin histograms of optic flow (HOF) are computed. Normalized histograms are concatenated into HOG, HOF as well as HOG/HOF descriptor vectors. (3) The HOG3D descriptor  is based on histograms of 3D gradient orientations. Gradients are computed via an integral video representations. Regular polyhedrons are used to uniformly quantize the orientation of spatio-temporal gradients. A given 3D volume is divided into a grid of cells. The corresponding descriptor concatenates gradient histograms of all cells. (4) The extended SURF (ESURF) descriptor [6] extends the image SURF descriptor to videos. Again 3D cuboids are divided into a grid of cells. Each cell is represented by a weighted sum of uniformly sampled responses of Haar-wavelets aligned with the three axes. Experimental Setup. We represent video sequences as a bag of local spatio-temporal features . Spatio-temporal features are first quantized HOG3D HOG/HOF HOG HOF Cuboids ESURF Harris3D 89.0% 91.8% 80.9% 92.1% – – Cuboids 90.0% 88.7% 82.3% 88.2% 89.1% – Hessian 84.6% 88.7% 7767% 88.6% – 81.4% Dense 85.3% 86.1% 79.0% 88.0% – – Table 1: Average accuracy on the KTH actions dataset.