# Agglomerative Clustering and Residual-VLAD Encoding for Human Action Recognition
###### Abstract  
Human action recognition has gathered significant attention in recent years due to its high demand in various application domains. In this work, we propose a novel codebook generation and hybrid encoding scheme for classification of action videos. The proposed scheme develops a discriminative codebook and a hybrid feature vector by encoding the features extracted from CNNs (convolutional neural networks). We explore different CNN architectures for extracting spatio-temporal features. We employ an agglomerative clustering approach for codebook generation, which intends to combine the advantages of global and class-specific codebooks. We propose a Residual Vector of Locally Aggregated Descriptors (R-VLAD) and fuse it with locality-based coding to form a hybrid feature vector. It provides a compact representation along with high order statistics. We evaluated our work on two publicly available standard benchmark datasets HMDB-51 and UCF-101. The proposed method achieves 72.6% and 96.2% on HMDB51 and UCF101, respectively. We conclude that the proposed scheme is able to boost recognition accuracy for human action recognition. action recognition; bag-of-words; deep residual networks; clustering; feature encoding; classification +
Footnote †: journal:These local spatio-temporal features have shown promising results and gave direction to future research. Moreover, spatio-temporal feature detector 3D Harris [5] and 3-D scale invariant feature transform [6] developed from their 2D counterparts provide interest point based video descriptors. These techniques usually do not require human detection algorithms. However, [PERSON] et al. [7] have stated that dense sampling methods outperform the interest point based methods based on their evaluation of several interest point based descriptors and detectors. These spatio-temporal points extract information from short temporal durations, which can result in distortion of long-term temporal actions. Hence, techniques such as feature trajectories [8; 9] are used to track these interest points in order to get a robust and accurate representation. These methods have made significant progress and have shown the ability of handling some recognition challenges such as noise, lighting changes and background clutter. Recently, deep learning algorithms [10; 11] have been widely used for feature learning and classification tasks. These deep architectures build the high-level features on top of low-level features to provide an optimal representation and have proved to be quite successful in image classifications. Inspired from their success on images, researchers have proposed a variety of architectures for video classifications. Some of these architectures include spatio-temporal convolutions [10], recurrent neural networks [12] and two-stream networks [13], which are used for training two individual streams of RGB data and motion-based optical flow data. These architectures, however, often fail to register or model the motion patterns and temporal information concurrently because both streams are trained independently with no link inbetween the streams. To overcome this limitation, 3D convolutional neural network (CNN) architectures [14; 15] have been proposed. These networks perform 3D convolutions and pooling, which enables the flow of temporal information across all layers. These architectures provide simultaneous feature learning for motion and appearance data, but have not proven to be effective due to their large number of parameters. More recently, [PERSON] et al. [16] have presented state-of-the-art results using 3D CNNs and encoding of human pose motion but requires the extraction of joint heatmaps for key points localization. The action recognition approaches discussed earlier, provide action representations in the form of embedding or feature vectors that are either classified directly using a SoftMax layer or rely on a Bag-of-Words (BoW) pipeline.