# HiTIM: Hierarchical Task Information Mining for Few-Shot Action Recognition
###### Abstract
For example, a second-order similarity network (SoSN) [17] uses second-order statistics for feature strengthening; alternatively, optical flow [18; 19; 20] or skeletal motion [21; 22] are added. To tackle the problem of insufficient mutual information, temporal alignment strategies [20; 23; 24] are typically employed. [PERSON] [23] proposed a frame tuple matching method that involves extracting various combinations of frame tuples from video segments to find the key frame tuples in the videos. Although these methods improve recognition accuracy, they increase computational complexity and reduce efficiency. Attention mechanism [25; 26] has been widely used for mining key feature information, exhibiting excellent performance and computational efficiency. However, its optimization effects are limited and only focus on either the temporal or spatial dimensions. Therefore, more efficient and flexible matching modules capable of effectively mining key information have to be designed. Inspired by the above observations, we propose a hierarchical task information mining (HiTIM) approach, as shown in Figure 1, consisting of inter-task learner (K\({}^{\text{inter}}\)) and a matching module with intra-task learner (K\({}^{\text{intra}}\)). The proposed K\({}^{\text{inter}}\) can adaptively construct the feature space according to the task through episodic training, thereby extracting task-specific features. To enhance the modeling capability of K\({}^{\text{inter}}\), we employ deformable three-dimensional convolution (D3D), which provides a more flexible receptive field than the standard convolution. K\({}^{\text{inter}}\) can mine the critical discriminative information for the task. The matching module with K\({}^{\text{intra}}\) consists of two branches: the spatiotemporal self-attention matching (STM) and correlated cross-attention matching (CM). We consider that the temporal and spatial information of videos are related. Thus, we concatenate the temporal and spatial dimensions of features into a new spatiotemporal dimension. The STM emphasizes the key regions in the spatiotemporal dimension of features using the self-attention, and the CM employs the cross-attention mechanism to emphasize the strongly correlated parts between support-set and query-set features. The shared K\({}^{\text{intra}}\) further optimizes the attention mechanism. Through these two components, HiTIM can integrate the critical discriminative knowledge between tasks and the key information of features within tasks. We evaluated the performance of our method using the UCF101 [27], HMDB51 [28], and Kinetics datasets [3]. The contributions of the study are as follows:We designed \(\text{K}^{\text{inter}}\) to mine the key discriminative information of tasks, generating task-specific features. * We designed a matching module with \(\text{K}^{\text{intra}}\), consisting of two branches for STM and CM. These branches mine the key spatiotemporal regions of features and the strongly correlated parts between features, with the shared \(\text{K}^{\text{intra}}\) further optimizing both branches. * We propose an end-to-end few-shot action recognition method named HiTIM, which can use either a 2D convolutional neural network (CNN) or 3D CNN as embedding. In experiments involving the UCF101, HMDB51, and Kinetics datasets, HiTIM achieved recognition accuracy that outperformed or was comparable to other SOTA few-shot action recognition methods on five-way one-shot and five-way five-shot tasks.