# Fine-Grained Action Recognition by Motion Saliency and Mid-Level Patches
## 3 Mid-Level Patch Mining Based on Motion Saliency
### Object Proposal Generation by the Idea of the Huffman Algorithm  
In Figure 3c, the human body is divided into several small motion saliency regions. The granularity of these regions is single, and these regions contain limited information and are not suitable as the elements for extracting mid-level patches. Therefore, we use a region merging algorithm to obtain multi-granularity motion regions [48; 49]. The Huffman algorithm is a simple and efficient lightweight algorithm. Inspired by the Huffman algorithm, we propose a region merging algorithm to merge the motion saliency regions with the image appearance features and the motion saliency features. All node regions in the Huffman tree are regarded as object proposals, which can obtain multi-granularity region partition results [50]. The initial region partition result of the foreground is \(P^{0}=\{p_{1}^{0},p_{2}^{0},\cdots,p_{n}^{0}\}\). For each region merging iteration \(i\), two adjacent regions with the highest similarity are merged to obtain a new non-terminal node. The two merged regions are deleted from \(P^{i}\) and the new node is added to \(P^{i}\). The algorithm continues to iterate until the Huffman tree is built or there are no adjacent regions that have not been merged. In order to describe the characteristics of different aspects of these regions more detailed, we choose a variety of features, including color similarity, motion saliency similarity, and region size factor. The feature values are in the range [0, 1] to facilitate merging. (1) Color Similarity  
In order to account for the different scenes and lighting conditions, we describe the motion saliency regions in a variety of color spaces, including RGB, Lab, HSV, normalized RGB (nRGB), and rgi [22]. Experimental results show that the nRGB color space has the best discrimination, where it is chosen as the color space. The color feature of a motion region is represented by a color histogram, and each color channel in the histogram has \(k\) bins. In order to extract image features from different scales, \(k\) can take multiple values similar to image pyramids. The color histogram of the \(i\)th region \(p_{i}\) is \(C_{i}=\left\{c_{1}^{1},\cdots,c_{i}^{m}\right\}\), and \(m=k*3\) for color images. The color histograms are normalized with the \(L1\) norm. The color similarity between region \(p_{i}\) and \(p_{j}\) is measured by the histogram intersection as  
Figure 4: Histograms of motion saliency maps. \[P_{color}\left(i,j\right)=\sum_{l=1}^{m}\min\left(c_{i}^{l},c_{j}^{l}\right). \tag{1}\]  
(2) Motion Saliency Similarity  
The motion saliency similarity between two adjacent motion regions is also measured by histogram intersection. The motion saliency map only has one channel, and the motion saliency histogram of the \(i\)th region \(p_{i}\) is \(MS_{i}=\left\{ms_{i}^{1},\cdots,\ \ ms_{i}^{m}\right\}\). The motion saliency similarity between region \(p_{i}\) and \(p_{j}\) is shown as  
\[P_{motion}\left(i,j\right)=\sum_{l=1}^{m}\min\left(ms_{i}^{l},ms_{j}^{l}\right). \tag{2}\]  
(3) Region Size Factor  
The reason for introducing region size factor is to ensure that the small regions could be merged early. The calculation method is as  
\[P_{size}\left(i,j\right)=1-\frac{size\left(p_{i}\right)+size\left(p_{j}\right) }{size\left(im\right)}, \tag{3}\]  
where \(size(im)\) is the size of the image \(im\), and \(size(p_{i})\) is the size of the region \(p_{i}\). Our final similarity \(P_{i,j}\) between region \(p_{i}\) and \(p_{j}\) is the combination of these three factors as  
\[P_{i,j}=\alpha_{1}P_{colour}\left(i,j\right)+\alpha_{2}P_{motion}\left(i,j \right)+\alpha_{3}P_{size}\left(i,j\right), \tag{4}\]  
where \(\alpha_{i}\in[0,1]\), \(\sum\alpha_{i}=1\), and \(\alpha_{i}\) denotes the weight of the factor \(i\). In the experiment, all factors have equal weights. The number of motion saliency regions in each frame is uncertain. If there are \(n\) regions, then the similarity measure calculation between neighbor regions could be \(\frac{n(n-1)}{2}\) times.