# Relational Action Bank with Semantic-Visual Attention for Few-Shot Action Recognition
## 3 Method
### Relational Action Bank  
The Action Bank is the bank that stores the feature vectors of existing actions. As we all know, we can employ the existing knowledge (the action representations in action bank) to assist the recognition of novel categories. Specifically, we can use the exclusive method as one of the straightforward methods. In fact, we use the features of base category video clips to enhance the modeling ability of the network for novel categories. We cluster video clips of each category into one feature vector as the general representation of corresponding categories. Then, we use these feature vectors to enhance the modeling ability of the network for novel categories. Here, we first give the definition of relational action bank (RAB), which enables a more robust representation of learned class actions by modeling the relationships between different videos of the same class action. We extract the features of the base category video clips via backbone network and form the feature action bank \(\mathbf{Z}=\{\mathbf{z}_{i,j,k}\in\mathbb{R}^{d}|i\in\{1,2,\cdots,N\},j\in\{1,2,\cdots,v\},k\in\{1,2,\cdots,s\}\}\), where \(N\), \(v\) and \(s\) denote the number of base categories, the number of videos per category, the number of clips in each videos, respectively. \(\mathbf{z}_{i,j,k}\) is the output of one video clip passing through the backbone subnet. In fact, the number of videos \(v\) and the number of clips \(s\) are not the same for different categories and different videos, but we assume \(v\) and \(s\) are the same for different categories and different videos for simplicity (in practice, we can use the max value as the corresponding \(v\) and \(s\), and use zeros to fill corresponding positions when the video or the clip does not exist). The output of RAB is \(\widetilde{\mathbf{Z}}=\{\widetilde{\mathbf{z}}_{i}\in\mathbb{R}^{d}|i\in\{1, \ldots,N\}\}\). Thus, the RAB submodule only needs to cluster the feature vectors in each category into one feature vector. We propose to use graph neural network (GNN) to model relations of the action clips which are stored in the action bank and to aggregate video clips of the same category into one feature vector. Our relational action bank has \(N\) base categories, each category has \(v\) videos and each video has \(s\) clip vectors which are extracted via pretrained neural network from the base categories. As the clips in each video have strong relations, inspired by Non-local [54], SENet [60] and bottleneck structure [81], we design our learnable relational action bank as Figure 3 shows, which consists of aggregate operations (\(f_{1}\) and \(f_{2}\)) and expand operation (\(g\)). To learn the strong relations among clips in each video, we use graph neural network (GNN) to realize an aggregate-expand operation (\(f_{1}\) and \(g\)). We aggregate the clips in each video and then resume the dimensions via expand operation. During this aggregate-expand process, our RAB module can learn the relations of clips in one video. At last, we use  
Figure 2: The architecture of our relational action bank with semantic–visual attention module. This module includes relational action bank and semantic–visual attention submodules. \(\odot\) denotes the element-wise multiplication. \(\otimes\) denotes matrix multiplication. GNN to aggregate the relationally enhanced clips of each category into one feature vector. As we have action clip feature vectors \(\mathbf{Z}\), we consider every clip feature vector \(\mathbf{z}_{i,j,m}\) as one node, and build the edges between nodes in the same video (we only build undirected fully connected graph for simplicity) to build the relations among the clips. We use GNN to accomplish aggregate operations, so here we first give the definition of GNN.