# HiTIM: Hierarchical Task Information Mining for Few-Shot Action Recognition
###### Abstract  
Although the existing few-shot action recognition methods have achieved impressive results, they suffer from two major shortcomings. (a) During feature extraction, few-shot tasks are not distinguished and task-irrelevant features are obtained, resulting in the loss of task-specific critical discriminative information. (b) During feature matching, information critical to the features within the task, i.e., self-information and mutual information, is ignored, resulting in the accuracy being affected by redundant or irrelevant information. To overcome these two limitations, we propose a hierarchical task information mining (HiTIM) approach for few-shot action recognition that incorporates two key components: an inter-task learner (K\({}^{\text{inter}}\)) and an attention-matching module with an intra-task learner (K\({}^{\text{intra}}\)). The purpose of the K\({}^{\text{inter}}\) is to learn the knowledge of different tasks and build a task-related feature space for obtaining task-specific features. The proposed matching module with K\({}^{\text{intra}}\) consists of two branches: the spatiotemporal self-attention matching (STM) and correlated cross-attention matching (CM), which reinforce key spatiotemporal information in features and mine regions with strong correlations between features, respectively. The shared K\({}^{\text{intra}}\) can further optimize STM and CM. In our method, we can use either a 2D convolutional neural network (CNN) or 3D CNN as embedding. In comparable experiments using two different embeddings in the five-way one-shot and five-way five-shot task, the proposed method achieved recognition accuracy that outperformed other state-of-the-art (SOTA) few-shot action recognition methods on the HMDB51 dataset and was comparable to SOTA few-shot action recognition methods on the UCF101 and Kinetics datasets. few-shot learning; action recognition; dynamic network; attention mechanism +
Footnote †: journal: Medical  
information such as self-information and mutual information, is overlooked. Research indicates [7] that accurate predictions may only require processing a small portion of key temporal segments or spatial regions. Videos contain redundant information, such as unrelated frames and backgrounds. Furthermore, videos are not temporally or spatially aligned, with variations in action start times, scene locations, and angles leading to nonuniform key information distribution across videos. Therefore, neglecting the self-information and mutual information of features within tasks when conducting matching can severely impact the recognition accuracy. To overcome the generalization limitations caused by task-agnostic features, research involving domain generalization [8] aims to enable models trained in the source domain to better adapt to tasks in other target domains. Existing methods for improving model generalization include techniques such as self-supervised learning [9] and meta-learning [10]. Self-supervised learning leverages freely available data labels and introduces auxiliary tasks to predict augmented labels. For instance, ARN [11] engages in self-supervision via jigsaw [12] and rotation to help obtain a more robust encoder through training. Meta-learning [13] exposes the model to a domain shift during the training process and continuously improves the algorithm across multiple learning stages, with the core idea of learning prior knowledge. Although both methods improve model generalization, they do not address the issue of features being task-agnostic. Other methods [14; 15] employ dynamic networks [16] to mine task information, but these approaches have redundant model structures and weak modeling capabilities, limiting their performance. To address the issue of weak self-information and mutual information in feature matching, existing methods generally employ feature enhancement techniques for insufficient self-information.