## 4 A Framework for Feature Curves on SPD Manifold
### Recognition between Families of Features Curves  
In our proposed method, time series of ROIs are extracted from short video. The comparison of similarity between short videos is transformed into the comparison between families of time series. The length of two time series may not be equal. Moreover, different frame rate, variable durations, and arbitrary starting/ending intensities of video also bring about obstacles. To tackle this problem, we adopted dynamic time warping (DTW) to find an alignment between the two videos. DTW needs to define appropriate metrics for recognition. As we introduced in Section 4.3, the time series of ROIs are transformed as feature curves in SPD manifolds, which introduces discriminative Riemannian metrics and divergence. Given two feature curves \(\Gamma_{1}(t)\) and \(\Gamma_{2}(t)\) concatenated by \(L_{1}\) and \(L_{2}\) SPD matrices, respectively, and a pair of warping paths \(\beta=(\beta_{1},\beta_{2})\) between \(\Gamma_{1}(t)\) and \(\Gamma_{2}(t)\), similar to the optimal path between time series introduced in Section 3.2, the optimal path in the set of all possible paths \(\Gamma\) between two feature curves \(\Gamma_{1}(t)\) and \(\Gamma_{2}(t)\) is  
\[\beta^{*}=\textit{arg}\min_{\beta\in\Gamma_{1}}\sum_{j=1}^{q}\delta(\Gamma_{1} (\beta_{1}(i)),\Gamma_{2}(\beta_{2}(i)))\, \tag{29}\]  
where \(\  
where \(\delta(\cdot,\cdot)\) can be defined as Riemannian metrics/divergence in SPD manifold. Similarity measure between two feature curves \(\Gamma_{1}(t)\) and \(\Gamma_{2}(t)\) under the optimal path can be defines as  
\[\delta_{\text{DTW}}(\Gamma_{1},\Gamma_{2})=\frac{1}{q}\sum\limits_{i=1}^{q}\delta (\Gamma_{1}(\beta_{1}^{*}(i)),\Gamma_{2}(\beta_{2}^{*}(i)))\;. \tag{30}\]  
Then, the similarity measure between two video needs to fuse information provided by each curve of its curve family. The classification strategy can also be divided into two types. One is an overall classification strategy that is suitable for situations where the number of ROIs \(m\) in a pattern is smaller. Given a family of feature curves \(\left\{\Gamma_{R_{i}}^{query}(t)\right\}_{i=1}^{m}\) extracted from query video \(V_{query}\) and sample families of feature curves \(\left\{\left\{\Gamma_{R_{i}}^{j}(t)\right\}_{i=1}^{m}\right\}_{j=1}^{m}\) from sample videos \(\left\{V_{1},\cdots,V_{n}\right\}\), the \(i\)-th feature curve \(\Gamma_{R_{i}}^{query}(t)(1\leq i\leq m)\) in \(V_{query}\) is independently compared with all the \(i\)-th feature curves \(\left\{\left\{\Gamma_{R_{i}}^{j}(t)\right\}\right\}_{j=1}^{n}(1\leq i\leq m)\) in sample videos. This approach extends to each feature curve of \(V_{query}\) to obtain the \(m\times n\) similarity measure matrix \(\Psi\) using DTW distance. The average similarity measure \(\frac{1}{m}\sum\limits_{i=1}^{m}\Psi_{i,j}(1\leq j\leq n)\) of all feature curves from \(V_{query}\) represents the similarity measure between query video and sample videos. Taking the KNN classifier as an example, the steps of overall classification method are shown in Algorithm 2. ```
Input:\(n\) sample families of feature curves \(\left\{\left\{\Gamma_{R_{i}}^{j}(t)\right\}_{i=1}^{m},\Gamma_{j}^{j}\right\}_{j =1}^{n}\) with their corresponding labels of each family, a family of feature curves \(\left\{\Gamma_{R_{i}}^{query}(t)\right\}_{i=1}^{m}\) of query video \(V_{query}\).