Video‐based action recognition using spurious‐3D residual attention networks

IET Image ProcessingVolume 16, Issue 11 p. 3097-3111 ORIGINAL RESEARCHOpen Access Video-based action recognition using spurious-3D residual attention networks Bo Chen, Bo Chen orcid.org/0000-0003-1030-5660 Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, China School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing, ChinaSearch for more papers by this authorHongying Tang, Hongying Tang Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, ChinaSearch for more papers by this authorZebin Zhang, Zebin Zhang Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, ChinaSearch for more papers by this authorGuanjun Tong, Corresponding Author Guanjun Tong [email protected] Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, China Correspondence Guanjun Tong, Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 201800, China. Email: [email protected]Search for more papers by this authorBaoqing Li, Baoqing Li Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, ChinaSearch for more papers by this author Bo Chen, Bo Chen orcid.org/0000-0003-1030-5660 Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, China School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing, ChinaSearch for more papers by this authorHongying Tang, Hongying Tang Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, ChinaSearch for more papers by this authorZebin Zhang, Zebin Zhang Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, ChinaSearch for more papers by this authorGuanjun Tong, Corresponding Author Guanjun Tong [email protected] Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, China Correspondence Guanjun Tong, Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 201800, China. Email: [email protected]Search for more papers by this authorBaoqing Li, Baoqing Li Science and Technology on Microsystem Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, ChinaSearch for more papers by this author First published: 26 May 2022 https://doi.org/10.1049/ipr2.12541 Funding information: National Key Research and Development Program of China under Grant No. (255) AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Recently, 3D Convolutional Neural Networks (3D CNNs) have attracted extensive attention in extracting spatial and temporal features in videos for their efficient feature extraction ability. However, it also brings enormous model parameters by training very deep 3D CNNs. Here, a novel network named spurious-3D Residual Attention Networks (S3D RANs) is proposed for video-based action recognition, which has the powerful capacity to learn collaborative spatiotemporal features. In particular, by leveraging the merits from 2D Convolutional Neural Networks (2D CNNs) and 3D CNNs, 2D CNNs are applied rather than 3D CNNs on frames of the single view of volumetric videos data to learn temporal motion features directly. Furthermore, view and channel-wise attention mechanism submodules are employed in the residual unit to learn the importance of each view for action recognition and guide the network to pay more attention to the more useful information for action recognition. Experimental results on UCF-101, HMDB-51 datasets demonstrate that our S3D RANs have higher accuracy and lower model complexity than existing works. 1 INTRODUCTION With the continuous progress of Internet technology, video acquisition hardware devices have been widely used in the whole society. These devices are mainly used in intelligent transportation, intelligent medical care, military monitoring, and life and entertainment. They also play a significant role in security prevention in crowded places like campuses, banks, airports etc. As video has clearly become a widely used medium in video-based human action recognition, it has drawn considerable attention in the field of research in the past few years [1-3]. The task of action recognition is to recognize different human actions from all kinds of video resources. The actions may be performed or not performed during the entire duration of the video. The main research content of action recognition is feature representation and feature extraction efficiently and accurately. Videos can be easily edited into video frames so that video-based action recognition appears to be a fusing task of the image recognition on multiple frames. Although there are many similarities between image recognition and video recognition, they are different since the consecutive video frames have a correlated background with slightly different motions by learning the spatiotemporal features information, which makes better achievements. In other words, besides the spatial characteristics such as the appearance of the target and scene information employed in image recognition, video action recognition also needs to take into account the temporal motion information. The key of the action recognition task is how to extract these spatiotemporal features better. Although researchers have done many studies, action recognition in video is still challenging due to extracting complicated spatiotemporal features. Therefore, it is critical to design the network's architecture. Due to the success of Convolutional Neural Networks (CNNs) in image recognition [4-7], in recent years, many CNN-based works studying video-based action recognition have achieved remarkable results [8, 9]. The human action recognition approaches based on deep networks can be classified into three categories: Two-streams: In [2, 10], Simonyan et al. proposed a two-stream CNN architecture that used 2D CNNs to extract spatial and temporal features from RGB images and optical flow images, respectively. The final action classification score is obtained by fusing the scores of the two streams. This method is expensive to compute because of calculating optical flows and training independently two networks. Zhang et al. [11] proposed an action recognition algorithm that combines optical flow and RGB to realize feature representation through channel fusion. Then, using boundary compensation, three keyframe selection methods are introduced, and the two-stream network with space-time structure is used for action recognition. 3D-CNNs: As 3D convolution is suitable for handling the extra dimension in videos data, 3D-CNNs have been deemed a promising technique for video-based action recognition tasks. The authors in [8] proposed 3D CNNs to extract the spatial and temporal features from the video frames to capture the moving images without using optical flow images. Although it saves the substantial computational cost of calculating optical flow images, its model complexity is still high with the introduction of the extra time dimension. CNN-LSTM: CNN in this model generates the spatiotemporal feature, and the final frame representation is fed into the long short-term memory (LSTM) network for further classifying the actions in the video. In [12], the authors propose a spatial-temporal attention module combined with a hierarchical convolutional LSTM network to model the structures of actions in the spatial and temporal domains. In [13, 14], the LSTM models combine with CNNs, which realizes the long-term memory of relevant information and thus extracts the temporal characteristics. However, when LSTM is applied in a short-term frame, it is difficult to capture the finer temporal relation and does not perform well. Since most of the actions occur in a very short time, the models based on 2D and 3D CNNs are still the most widely used. It is generally believed that, compared to 2D CNNs, 3D CNNs [3, 8] could achieve superior performance by simultaneously learning the spatial and temporal information and directly using video frames as input rather than optical flow images at the expense of higher complexity and longer training time. Different from the above methods, in [15], Chao Li et al. put forward a new viewpoint. Figure 1 denotes the 2D images from three views of an entail video clip. Looking at the left view and the vertical view alone, we can clearly see that a group of people is doing some actions. Therefore, we can extract features from the view of W–T and H–T. Video data can be easily viewed as a 3D volumetric tensor. We can get three views from multi-view of videos data, which we denote the front view, left view, and vertical view as W–H, T–H, and T–W, respectively. By sampling the video frame with a certain sampling interval along the dimension of time, we can get the snapshots from the view of W–H, which is the standard view of still images. This view is also the 2D image data that we are most familiar with. We can intuitively analyse and understand the video content. However, it is not enough to only extract the spatial features without extracting the temporal features. FIGURE 1Open in figure viewerPowerPoint Three views from the entire videos clip Notably, the convolution kernels’ parameters of different views are shared for the following reasons. The visual appearance of frames of different views is compatible with the visualization of frames of different views. As everyone knows, the features of images can be divided into low-level features and high-level features. The underlying features of the image are independent of the position of the features in the image. For example, we can use the edge detection operator to filter out whether the edge is in the centre or around the image. Common spatial patterns like edges and colour blobs also exist in temporal views (T–H and T–W). Therefore, the same set of convolution kernels can be used on frames of different views. Besides, model parameters are drastically reduced, so the network is easier to train and less prone to overfit, improving the performance. Here, we propose a novel network model named spurious-3D Residual Attention Networks (S3D RANs), which use spurious 3D CNNs to execute convolution operations to extract collaborative spatiotemporal features from video frames. The “spurious” here refers to the use of 2D convolution rather than expensive 3D convolution. Taking the PyTorch framework as an example, when building the network model of action recognition algorithm, the 3D model uses nn.Conv3d while spurious-3D uses nn.Conv2d, the former deals with 5D tensors, while the latter deals with 4D tensors. The basic architectures are illustrated in Figure 2. Our S3D RANs consist of extract spatiotemporal features submodule and view and channel-wise attention mechanism submodule. The same as getting the W–H view, we can get the snapshots from the view of T–H and T–W. Although compared with the view of W–H, these are difficult for human beings to explain intuitively, they still contain specific features as the standard W–H view. The action usually continues for a period in videos data, and abundant motion information is embedded with the interframe. Therefore, it is very important to process video frames along the time dimension. Note that the time dimension is different from other dimensions because it contains rich motion information, which needs to be handled carefully. We perform 2D convolutions instead of 3D convolutions on the view of T–H and T–W, aiming at learning the features of the time dimension in a single W dimension and a single H dimension. Meanwhile, we also perform 2D convolutions instead of 3D convolutions on the view of H–W, to learn the spatial features of the static image. Finally, fusing complementary features of the three views, we can extract collaborative spatiotemporal features. FIGURE 2Open in figure viewerPowerPoint Structure of our S3D RANs The rest of this paper is organized as follows. In Section 2, we briefly review the related works. In Section 3, we describe the S3D RANs in detail. In Section 4, after introducing the benchmark datasets, we present the training pipeline and the inference of the experiments and then show and analyse the experiment results. Finally, we give a brief conclusion in Section 5. Our contributions are summarized as follows: We propose a novel network named spurious-3D Residual Attention Networks (S3D RANs) for video-based action recognition, which has the powerful capacity to learn collaborative spatiotemporal features. We introduce the attention module including view and channel-wise attention into our network, which selectively focuses on the significant information. Experiment results on UCF101 and HMDB51 datasets verify the effectiveness of the proposed method and achieve comparable performance with the state-of-the-art methods. 2 RELATED WORK Video action recognition Action recognition is a challenging task that has attracted a lot of attention from academic researchers. Early work on action recognition in videos focused on hand-crafted features. Laptev [16] proposed space-time interest points (STIPs) to extract the discriminative local or global features from the videos. Some local features, such as histograms of oriented gradients (HOG) [17] and histograms of optical flow (HOF) [18], are used to describe STIPs by Laptev et al. Wang et al. [19] proposed the improving dense trajectories (IDT) algorithm, which introduced the elimination method of background optical flow to make the features more focused on the description of human motion. However, these methods take too much time to extract hand-crafted features, which become difficult to deal with on large-scale datasets. Different from these methods, our method uses deep convolution networks to extract features automatically. With the improvement of modern GPUs computing power in recent years, researchers pay more attention to 3D CNNs [20-22]. In [8], the authors developed a novel model based on the 3D convolution feature extractors, which perform convolution and subsampling separately in a 3D volumetric tensor formed through stacking consecutive frames. In this structure, each feature map in the convolutional layer is connected to multiple adjacent contiguous frames in the upper layer, thus capturing motion information. In [23], Qiu and Yao et al. proposed a novel network named Pseudo-3D (P3D) which decomposes a 3 × 3 × 3 convolution (3D) into a 1 × 3 × 3 convolution (2D) followed by a 3 × 1 × 1 convolution (1D). Another work [24] proposed a similar network named R(2 + 1)D, which replaces the 3D convolution with a 2D spatial convolution and a 1D temporal convolution. What's more, the model increases the number of layers of nonlinearity and makes network optimization easier. In [4], Carreira et al. proposed a novel network named I3D by inflating the 2D convolutional filters into 3D so that the 3D convolutional network can benefit from developing a 2D convolutional network. Inspired by P3D [23], from the perspective of modelling spatiotemporal features, we also modify the size of the convolution kernel to better learn spatiotemporal features. An obvious difference is that the bottleneck block of P3D is different from our basic composition. We used different model architectures. Specifically, we design a bottleneck composed of 1 × 3 × 3 convolutions, 3 × 1 × 3 convolutions, and 3 × 3 × 1 convolution to learn collaborative spatiotemporal features, while P3D is composed of 1 × 1 × 1 convolution, 1 × 3 × 3 convolutions and one 3 × 1 × 1 convolution. The basic convolution architecture of P3D (three P3D ResNet variants, P3D-A, P3D-B and P3D-C) and our S3D RANs are illustrated in Figure 3. Secondly, some variants of P3D are also different from ours. Our experimental results show that S3D compared with most models not only reduces the complexity of the model but also improves the accuracy of recognition, which proves the effectiveness of our model. FIGURE 3Open in figure viewerPowerPoint Bottleneck building blocks of residual unit and our S3D RANs Attention module Visual attention is the result of the human visual system. When human beings are observing external things, the visual processing system selectively focuses on a specific target area rather than the whole thing. Instead, they will think of what they are observing. For example, when we see a person walking towards us, we generally pay attention to the person's face first, then to some of his specific behaviours, and finally integrate the information of all areas in the visual system to get the overall impression of the observed object to observe external things. Human vision can quickly scan the object's overall image, determine the focus position of attention, and then devote more attention to the focus area while suppressing other unimportant information. The attention mechanism is essential to quantify the importance of each different part of the input to feature learning. It helps the model better extract the key and essential information by assigning different influence factors to each part of the input to focus on important regional features while ignoring unimportant information, which will not increase the model's computational cost and storage cost. Recently, the researchers of deep have paid more focus to the attention model. It is easy to that the attention models play an essential role in all kinds of tasks. In [25], the authors introduce an attention module into a skeleton-based action analysis model to focus on the importance of different video frames. A novel tri-attention module is proposed in [26] to guide the network to detect significant changes in local motion. The innovation of [27] introduces an attention mechanism into action recognition, which uses the LSTM model to learn the critical moving parts of the videos data. In [28], Girdhar et al. proposed a simple but powerful attentional pooling model, which focuses on action recognition and human-object interaction tasks. Meanwhile, the bottom-up and top-down new inference are introduced as the low-rank approximation of the bilinear pooling method, which is used to replace the common mean pooling or max-pooling in the final pooling layer of the CNN network. Experiments are carried out on datasets and good results are obtained. In addition, the author also tried to add information on critical points of a pose, which improved the performance again. In [29], Wang et al. proposed non-local operations, which calculated the feedback of a location as a weighted sum of all location features to capture long-range dependencies. In [30], Woo et al. proposed two attention modules. One is based on channel, and the other is based on spatial, applied attention mechanism simultaneously. Convolutional Block Attention Module (CBAM) inferred attention map in turn along the channel and spatial dimensions. On channel attention, the input is a feature F of H × W × C. We first perform global average pooling and maximum pooling of the two spaces respectively to get two-channel descriptions of 1 × 1 × C. Then, they are respectively sent into a two-layer neural network. Then, the weight coefficient is obtained by adding the two features after a sigmoid activation function. Finally, the new scaled feature can be obtained by multiplying the weight coefficient with the original feature F multiply to get a scaled new feature. Spatial attention is similar to channel attention, except that the former gets two H × W × 1 channel descriptions and stitches these two descriptions together according to the channels. Based on this research, we propose our attention block, which can be embedded into our extract spatiotemporal (EST) bottleneck, which forms our S3D RANs together. 3 MODEL DESIGN This section reviews the typical 3D CNNs architectures and introduces the novel human action recognition model based on deep ResNets to extract collaborative spatiotemporal features. Our S3D RANs encode collaborative spatiotemporal features from three video views by imposing a weight-sharing constraint on the learnable parameters. Furthermore, to enhance the networks to focus more on the salient features for action recognition, we propose a novel view and channel-wise attention mechanism that can improve the quality of spatiotemporal encodings throughout its feature hierarchy. Then, we introduce more details about each component of our architectures. 3D convolutions Let x denote a video clip whose input format is N × C × T × H × W, where N, C, T, H and W represents the clip length, number of channels, temporal information, height and width of each frame, respectively. 3D CNNs are the most natural way to encode spatiotemporal information. For example, we denote the 3D convolutional filters with the size of T × H × W, where T is the time dimension of a kernel that can be understood as that how many frames are processed and H × W is the spatial kernel size. Therefore, if there is a 2D convolutional filter with a size of H × W, it can be naturally transformed into a 3D convolutional filter with the size of T × H × W by joining an increased time dimension. In classical and efficient CNN models like ResNets, 1 × 1 and 3 × 3 are the two most common filters. Given a residual unit consisted of 1 × 1 and 3 × 3 convolutions as shown in Figure 4a, we may easily transform the intermediate 3 × 3 filters into 3 × 3 × 3 (C3D3 × 3 × 3) as shown in Figure 4b, or inflate the first 1 × 1 filter into 1 × 1 × 3 (C3D3 × 1 × 1) as shown in Figure 4c. Compared to 2D CNNs, 3D CNNs achieve comparable performance to capture temporal features owing to performing convolution and pooling operations along the time dimension. In 3D CNNs, these operations are directly performed temporal and spatial together, while in 2D CNNs, they are done only spatially. Nevertheless, this will bring substantial computational cost and cause high model complexity. FIGURE 4Open in figure viewerPowerPoint Comparison of various residual units In our experiments, we inflate the middle 1 × 3 × 3 filters for S3D blocks, as illustrated in Figure 4d. For simplicity, we have omitted some activation layers and down-sampling layers. Different from 3D CNNs learned T × H × W jointly, and we actually utilize 2D convolution along with the view of H × W, T × H, T × W, respectively. As noted before, C3D3×3×3 used 3D convolution kernels of 3 × 3 × 3 to extract spatial and temporal features together. Spatial features can be learned along the H and W dimension, and temporal features can be learned along the T dimension. C3D3×1×1 utilized 1D convolution kernels of 3 × 1 × 1 along the T dimension to aggregate temporal feature and then a 2D convolution kernels of 1 × 3 × 3 along the H and W dimension is used to aggregate spatial features. While in our proposed method, we executed 2D convolution kernels of 3 × 3 along with three views: H--W, T–H, and T–W to learn spatial and temporal features complimentary, and then weighted summation of the three generated feature maps. An example instantiation of the network is illustrated in Table 1. For all convolutional layers, we apply appropriate padding and stride in the spatial and temporal dimension, respectively, so that the size of the convolutional layers from input to output does not change. We use the pooling layer to reduce the dimension. TABLE 1. An example instantiation of the S3D RANs. The backbone is ResNet-50 layer name Output size Kernels Strides input 16 × 224 ×224 none none conv1 16 × 112 × 112 1 × 7 × 7, 64 1, 22 pool1 16 × 56 × 56 3 × 3 × 3, max pooling 1, 22 block1 16 × 56 × 56 1 × 1 × 1 , 64 1 × 3 × 3 , 3 × 1 × 3 , 3 × 3 × 1 , 64 1 × 1 × 1 , 256 × 3 $\left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {1 \times 1 \times 1,64}\\ { \def\eqcellsep{&}\begin{array}{*{20}{c}} {1 \times 3 \times 3,}&\quad {3 \times 1 \times 3,}&\quad {3 \times 3 \times 1,64} \end{array} }\\ {1 \times 1 \times 1,256} \end{array} } \right] \times 3$ 1, 12 pool2 8 × 56 × 56 3 × 1 × 1, max pooling 2, 12 block2 8 × 28 × 28 1 × 1 × 1 , 128 1 × 3 × 3 , 3 × 1 × 3 , 3 × 3 × 1 , 128 1 × 1 × 1 , 512 × 4 $\left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {1 \times 1 \times 1,128}\\ { \def\eqcellsep{&}\begin{array}{*{20}{c}} {1 \times 3 \times 3,}&\quad {3 \times 1 \times 3,}&\quad {3 \times 3 \times 1,128} \end{array} }\\ {1 \times 1 \times 1,512} \end{array} } \right] \times 4$ 1, 22 block3 4 × 14 × 14 1 × 1 × 1 , 256 1 × 3 × 3 , 3 × 1 × 3 , 3 × 3 × 1 , 256 1 × 1 × 1 , 1024 × 6 $\left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {1 \times 1 \times 1,256}\\ { \def\eqcellsep{&}\begin{array}{*{20}{c}} {1 \times 3 \times 3,}&\quad {3 \times 1 \times 3,}&\quad {3 \times 3 \times 1,256} \end{array} }\\ {1 \times 1 \times 1,1024} \end{array} } \right] \times 6$ 2, 22 block4 4 × 7 × 7 1 × 1 × 1 , 512 1 × 3 × 3 , 3 × 1 × 3 , 3 × 3 × 1 , 512 1 × 1 × 1 , 2048 × 3 $\left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {1 \times 1 \times 1,512}\\ { \def\eqcellsep{&}\begin{array}{*{20}{c}} {1 \times 3 \times 3,}&\quad {3 \times 1 \times 3,}&\quad {3 \times 3 \times 1,512} \end{array} }\\ {1 \times 1 \times 1,2048} \end{array} } \right] \times 3$ 1, 22 pool3 1 × 1 × 1 4 × 7 × 7, average pooling 1, 12 fc 1 × 1 × 1 2048 × num_classes 1, 12 Spatial striding is performed on the first residual unit of each block. The dimensions of kernels are denoted by {T ×H ×W, C} for temporal-spatial, and channel sizes. Strides are denoted as {temporal stride, spatial stride2}. Convolutional residual blocks are shown in brackets, next to the number of times each block is repeated in the stack. This vector is fed to a fully-connected layer that outputs the class probabilities through a softmax function. Extract spatiotemporal features As mentioned above, we apply 2D 3 × 3 convolutions rather than 3D convolutions along the multi-view of videos data, the front view (H–W), left view (T–H), and vertical view (T–W). Two core ideas of the convolutional neural network: Local connectivity; Convolution kernel parameter sharing. A key role of the two is to reduce the number of parameters, make the operation simple and efficient, and can operate on large-scale datasets. In order to keep the same parameters as single-view 2D, we share the parameters of the three-view convolutions. Then we aggregate the three resulting feature maps with a weighted summation. We calculate the convolution of the three views separately. Denote a volume x ∈ R T × H × W × C $x \in {\mathbb{R}^{T \times H \times W \times C}}$ as the input feature maps, where T is the time span, H refers to the height of the raw frames, W represents the width in the spatial area, and C is the number of channels. In order to extract spatio-temporal features employ a spurious-3D convolution and then get a feature map x ′ ∈ R T ′ × H ′ × W ′ × C ′ $x^{\prime} \in {\mathbb{R}^{T^{\prime} \times H^{\prime} \times W^{\prime} \times C^{\prime}}}$ . In fact, we express it as a 5D tensor of χ ∈ R N × C × T × H × W $\chi \in {\mathbb{R}^{N \times C \times T \times H \times W}}$ , where N is similar to batch-size. We can compute the output feature maps by: x h w = x × ω 1 × h × w , x t w = x × ω t × 1 × w , x t h = x × ω t × h × 1 . \begin{equation} \def\eqcellsep{&}\begin{array}{l} {x_{hw}} = x \times {\omega _{1 \times h \times w}},\\[4.5pt] {x_{tw}} = x \times {\omega _{t \times 1 \times w}},\\[4.5pt] {x_{th}} = x \times {\omega _{t \times h \times 1}}. \end{array} \end{equation} (1)where * stands for 3D convolution, and ω represents the filters of the three views. Therefore, the networks can learn features of the single-view from H–W, T–W, T–H, respectively. 3D CNNs employ a 3D convolution of 3 × 3 × 3 to extract spatiotemporal features jointly. Generally, the parameters of 3D convolution with a kernel size of k × k × k are about k3, while S3D RANs contain 3 k 2 − 3 k + 1 $3{k^2} - 3k + 1$ [15]. Because we can get some similarity of colours and edge features from the single view, we remove the corner parameters, which are the shared region of three views. Sharing parameters of different views can reduce the