# Agglomerative Clustering and Residual-VLAD Encoding for Human Action Recognition
###### Abstract
The BoW framework [17; 18; 19] has been widely used for action recognition, as it provides a simplified representation for classification tasks. It consists of the following steps: (i) feature extraction, (ii) vocabulary generation, (iii) feature encoding and classification. First, the feature vectors are extracted using a feature descriptor, for example, 3D-SIFT [6] or a ConvNet. The obtained features are quantized using a clustering algorithm [20] such as k-means or gaussian mixture models (GMMs) to form a visual vocabulary. The entries in the vocabulary are known as visual words. These visual words form the basis for feature encoding and play an important role in recognition performance. Therefore, an optimal codebook generation algorithm that can provide discriminative visual words is highly desirable. The third step involves the encoding of features for each individual video. Many researchers have worked in feature encodings [21; 22; 23] to improve the final representation. This encoded vector is classified in a final stage, which is most commonly performed by support vector machines. In this paper, the focus is to develop a discriminative codebook and a features encoding method, which provide an all-encompassing representation for action videos. The codebook generation and encoding are the two most crucial steps for a BoW pipeline. The overall framework of the proposed method can be seen in Figure 1. In the first step, spatio-temporal features are extracted from ConvNets. The saliency of these features greatly impacts the overall recognition performance. Although hand-crafted features have had some success, such features do not generalize well to large scale video datasets [1]. Hence, we use the spatio-temporal two-stream ResNet [24] and a single stream 3D ResNet [25] architecture to extract feature descriptors. In the second step, these features are used for learning an agglomerative codebook. In the third step, the proposed agglomerative codebookis used for feature encoding. Finally, a support vector machine classifier is used for classification. The contribution of this work is three-fold:  
* First, we propose an agglomerative codebook. The aim of this codebook is to agglomerate the benefits of global codebook representation with class-specific codebook representations. The final agglomerative codebook provides discriminative codewords for the feature encoding step. * Second, we have proposed a modified Vector of Locally Aggregated Descriptors (VLAD) vector, known as Residual-VLAD (R-VLAD), which computes the higher order statistics by finding a difference vector between the feature descriptor and mean of the nearest codeword. It requires less computational resources than VLAD due to a reduced size of the encoded vector. * Third, to further enhance the capability of the R-VLAD vector, a hybrid feature vector is formed by fusing the locality-based descriptor with R-VLAD. The final encoded vectors are L2 normalized, which serve as the input to the classifier. The rest of the paper is organized as follows: In Section 2, we explain the proposed methodology; Section 3 describes the detailed experimentation and comparisons with the state-of-the-art. Finally, the paper is concluded in Section 4.