# Relational Action Bank with Semantic-Visual Attention for Few-Shot Action Recognition
## 1 Introduction  
In recent years, video action recognition has been well studied. However, conventional action recognition requires a large amount of labelled data for training, such as Sports-1M [1] and Kinetics-700 [2] datasets. Labeling these video data for training is a highly expensive process. Identifying methods that use a smaller amount of labelled data to train a better available model remains an open issue that needs to be solved urgently. For image classification, there are many classical methods to solve few example issues. The Siamese Network [3] enabled the model to generalize from few examples, through limiting assumptions on the input structure and acquiring features automatically. To eliminate the process of fine-tuning to adapt to new class types, ref. [4] designed Match Network that maps a small labelled support set and an unlabelled example to its label. The work in [5] introduced Prototype Network, a few example image-classification method that computes distances to prototype representations of each class in the learned metric space. However, for video action recognition, there remains a lack of effective works. Ref. [6] attempted to reproduce the methods from image to video and proposed CMN method to solve the few example video action recognition issue, which improved the performance of action recognition compared against reproduced methods. Another method presented by [7] can simultaneously capture local and long-term spatial temporal information employing the proposed dilated dense network, whose blocks consist of densely connected dilated convolutions layers. Subsequently, [PERSON] et al. [8] introduced the temporal attentive relation network (TARN) to perform temporal alignment and learn a deep-distance measure on the aligned representations at the video segment level. Although some researchers have studied few-example-based video action recognition, they all omit the existing prior knowledge. Consequently, they have suffered large losses in performance. As shown in Figure 1, we can see that the \"shoot ball\" action is related to the existing actions, especially the action \"turn\" and \"shoot bow\". As we all know, when humans memorize and recognize novel few example actions, we will use other related and familiar action categories to assist this process. How can we make CNNs perform few-shot action recognition like a human? How can CNNs be enabled as to exploit past knowledge? The naive method is that we train the CNNs on the past knowledge and fine-tune the network on novel few shot categories; however, the subsequent performance of the action recognition is poor. To solve this problem, we propose a novel relational action bank with semantic-visual attention for few example action recognition. Our proposed RABSA module consists of two submodules, relational action bank and semantic-visual attention. The action bank provides prior knowledge for the network and the semantic-visual attention mechanism generates attention weights to take advantage of the action bank. Thus, the RABSA can utilize both the semantic attention and visual similarity attention simultaneously to acquire useful action information from the action bank. At the same time, our proposed action bank can learn the clip relations in each video and then use graph neural network (abbreviated as GNN) to cluster the clips of each class to obtain the representation of each category. Overall, our contributions can be summarized as follows:  
1. To the best of our knowledge, we are the first to propose a relational action bank to assist the few-shot action recognition. 2. The proposed relational action bank can both enhance the representation ability of clips in each video and learn the feature vector representations for each class adaptively. 3. We propose a semantic-visual attention mechanism, which can utilize the relational action bank both in semantic categories and visual similarities. 4.