# Fine-Grained Action Recognition by Motion Saliency and Mid-Level Patches
## 4 Action Recognition with Graph Structure
### Motion Cooperation Relationship of Mid-Level Patches  
The key problem is how the relationship between mid-level patches can be calculated. We adopt the appearance and motion features to describe the patches. Furthermore, the relationship between any two related patches is described by the binary motion cooperation relationship. The feature term of a patch is \(u_{i}=\omega_{i}\cdot x_{i}\), which uses SVM to calculate the score of patch \(i\). The DT feature \(x_{i}\) is used to describe the patch \(i\), and the parameter \(\omega_{i}\) is obtained by learning. The binary relation term \(u_{i,j}\) describes the spatial motion cooperation relationship between different patches \(i\) and \(j\). This paper calculates the relationship between mid-level patches from two aspects, which are trajectory consistency and motion consistency. (1) The Trajectory Consistency  
For each patch \(i\), the mean patch trajectory is used to describe its motion trajectory, and it is defined as  
\[ \begin{split} pt_{i}\left[t\right]=\frac{1}{\left|\left\{tc_{i} \right\}\right|}\sum_{\forall m,\overline{t}\in\left[ts_{m},td_{m}\right]}tc_{ i}^{m}\left[t\right],t\in T_{i}.\end{split}  \tag{6}\]  
In order to describe the change of trajectories of two patches effectively, the \(pt_{i}\left[t\right]\) and \(pt_{j}\left[t\right]\) are further processed [21]. First, the canonical relative position is estimated, \(d_{f_{k}^{i},f_{k}^{i}}\left[t\right]=\left|pt_{i}\left[t\right]-pt_{j}\left[ t\right]\right|/\sigma\), where \(\sigma\) is a scale variable. \(d_{f_{k}^{i},f_{k}^{j}}\left[t\right]\) takes a value only in the coexisting time interval. Subsequently, a feature capturing the rate of the convergence and divergence of the trajectory pairs is calculated, where \(v^{x,y}\left[t\right]=d_{f_{k}^{i},f_{k}^{i}}\left[t+1\right]-d_{f_{k}^{i},f_{ k}^{i}}\left[t\right]\)[21]. The relation between the two patches is symmetric. For convenience, we quantize two elements \(x\) and \(y\) in the vector \(v^{x,y}\left[t\right]\) of any two related patches \(i\) and \(j\) at time \(t\), respectively. The quantized vector is as follows:  
\[ \begin{split}\varphi^{i,j}\left(x\right)=\left[v_{-1}\left(x \right),\rho_{-\eta}\left(x\right),...,\rho_{0}\left(x\right),...,\rho_{\eta} \left(x\right),v_{1}\left(x\right)\right],\\ v_{\pm 1}\left(x\right)=\left(1+e^{\frac{\pm x+y_{0}}{\eta_{0}}} \right)^{-1},\rho_{d}\left(x\right)=e^{-\frac{\left(x-\mu t\right)^{2}}{z_{d}^ {2}}},d\in[-\eta,...,0,...,\eta].\end{split}  \tag{7}\]  
Equation (7) is performed on the average trajectories \(pt_{i}\) and \(pt_{j}\) in the time interval \(T_{i}\cap T_{j}\), and the feature vector \(\phi_{i,j}=\left(\varphi^{i,j}\left(x\right),\varphi^{i,j}\left(y\right)\right)\) is obtained, where \(\phi_{i,j}\in\mathbb{R}^{\left(\left(2\eta+3\right)\times\left\{T_{i}\cap T_{ j}\right\}\right)2}\). The parameters in Equation (7) are set as Raptis et al. [21], where \(\eta=3\). \(s_{0}\) and \(\mu_{0}\) are set so that the sigmoidal functions \(v\) can cover the extremes of the domain. In Gaussian functions \(\rho\), \(\mu_{d}\) is set to cover the relative velocity domain spanned by the database, and \(s_{d}\) changes along with \(\mu_{d}\) according to the variance formula. The feature vector \(\phi_{i,j}\) can better describe the trajectory consistency between the two mid-level patches \(i\) and \(j\)[21]. (2) Motion Consistency  
Motion consistency is calculated by feature histograms of HOG (Histograms of Oriented Gradients), HOF (Histograms of Optical Flow), and MBH (Motion Boundary Histograms, MBHx and MBHy). For the \(i\)th patch \(f_{k}^{i}\) in the frame \(F_{k}\), the feature histogram is \(H_{k}^{i}=\left\{h_{i}^{c}\right\}_{c=1}^{4}\), where \(c\) represents the four different features and \(h_{i}^{c}\) is the Bow of the \(cth\) feature of patch \(i\).