# Agglomerative Clustering and Residual-VLAD Encoding for Human Action Recognition
## 2 Proposed Methodology
### Agglomerative Codebook Generation  
The codebook formation is an essential step in any BoW pipeline as discussed in Section 1. The codebook is a compact representation of the dominant features from the training data. This codebook is used for encoding of both training and testing data. Figure 2 depicts the proposed agglomerative clustering approach (see Algorithm 1).  
Figure 2: Agglomerative clustering for codebook generation.  
A feature set \(F\) is extracted using a CNN architecture [24; 25] for \(N\) number of action classes, i.e., \(F=[F_{1}+F_{2}+,\ldots,F_{N}]\). A global codebook \(C_{G}\) is generated by clustering all these features. Features having similar patterns will be grouped into the same clusters. The k-means++ algorithm [27] is used as a cluster initialization method. Cluster optimization is performed by Lloyd's method [28]. These methods provide good results as compared to the simpler k-means. The same setting is used in class-specific and final codebook generation. The global codebook with M number of clusters is represented as: \(C_{G}=c_{m}|m=1,\ldots,M\). The global codebook can cause discriminative features to be grouped into the same cluster as unlabelled data is used. This codebook provides a global simplified representation of the entire training data. The second phase of codebook generation is learning a class-specific codebook. In a class-specific codebook, the features from each class are clustered separately. For a given class \(i\), features \(F_{i}=[f_{1}+f_{2}+,\ldots,f_{n}]\) having \(n\) number of features are extracted, the class-specific codebook is generated given by \(c_{i}=c_{k}|k=1,\ldots,K\) where \(K\) represents the total number of clusters for class \(i\). The final class-specific codebook is formed by appending the class-specific codebooks for all classes given as \(C_{cs}=c_{j}|j=1,\ldots,(K_{i}\times N)\), where \(N\) represents the total number of classes. The codewords in this codebook are class-specific, meaning that equal contributions are made in the codebook from each individual class of features. However, this codebook lacks the global representation, which is present in the global codebook. The final phase of clustering involves the clustering of codewords. To combine the advantage of global representation along with class-specific representation, we introduce the idea of clustering these global and class-specific codewords. k-means clustering is applied on centroids of the global \(C_{G}\) and the class-specific \(C_{cs}\) codebooks. This step helps in finding linkages between the codewords having similar information in the global and the class-specific codebooks. The nearest codewords are combined, which results in a better representation containing information of global as well as class-specific visual words. The combination of two different codebooks proves to be beneficial for encoding of features. The size of the codebooks is chosen very cautiously, as it affects the overall performance of the BoW approach. Large sizes can cause processing overheads, redundant codewords and less generalizability. A small size can cause distinct features to be clustered under the same codeword, which can reduce the discriminative power of the codebook. Different codebook sizes have been tested experimentally with details given in Section 3.3.1.