# Multi-Scale Receptive Fields Convolutional Network for Action Recognition
## 2 Related Work
There are also some works that combine ConvNets with recurrent neural networks (RNN) [27] for action recognition, such as LRCN [28], scLSTM [29], and CapsGaNet [30]. Although RNNs have a strong ability to process data with temporal relationships, their training processes are unstable, which makes them hard to train. Besides, it is still challenging for 2D ConvNets to learn temporal information directly from raw video frames. Optical flow is an effective motion representation for describing object movement and provides orthogonal information compared with RGB images. This allows 2D ConvNets to learn temporal information. However, calculating optical flow is computationally intensive and requires significant storage. A natural way to understand a video is by using 3D convolution kernels to handle video data, which leads to the usage of 3D ConvNets. The seminal work for using 3D ConvNets for action recognition is [31]. However, the network in [31] is not deep enough to show the potential of 3D ConvNets. Therefore, [PERSON] et al. [7] proposed to use a deeper 3D ConvNet, named C3D, for training and testing. Although the performance of C3D on standard benchmarks is not as good as expected, it shows strong generalization capability and is available as a generic feature extractor for other video tasks. However, using 3D kernels leads to an increase in the number of parameters, making 3D ConvNets hard to optimize and require large-scale action recognition datasets. This situation was alleviated when [PERSON] et al. [8] proposed inflated 3D ConvNets (I3D). The main contribution of I3D is inflating the ImageNet pre-trained 2D model weights to their counterparts in the 3D model. Thus, 3D ConvNets can avoid being trained from scratch. Moreover, I3D achieves high recognition accuracy on UCF101 and HMDB51 after pre-training on a new large-scale dataset, Kinetics-400. [PERSON] et al. [32] proposed a method for human activity recognition using depth videos with both spatial and temporal characteristics. They constructed a hierarchical difference image and fed it to a pre-trained CNN, which then classified human activities by combining the videos. Although this method achieved better accuracy than skeleton-based models, it should be noted that the limited availability of datasets may have contributed to this result.There are also works aiming to reduce the training complexity of 3D ConvNets. For example, P3D [16] and R(2+1)D [17] explore the idea of 3D factorization into two separate operations, a 2D spatial convolution and a 1D temporal convolution. This split operation allows spatial and temporal features to be optimized separately, thus reducing the optimization difficulty of the model. The difference between P3D and R(2+1)D is how they arrange the two factorized operations and how they formulate each residual block. [PERSON] et al. [33] propose another way of simplifying 3D ConvNets. They integrate 2D and 3D ConvNets in a single network, termed MiCTNet. MiCTNet is proposed on the hypothesis of the uneven distribution of spatiotemporal information in video, aiming to generate deeper and more informative feature maps. S3D [19] combines the merits of the approaches mentioned above, proving that it is feasible to replace 3D convolution with low-cost 2D convolution. Table 1 shows the number of parameters of ResNet-50 in 2D, 3D, and P3D structures. To avoid a large number of parameters, our network employs the structure of P3D networks. In this paper, we aim to handle the situation that the size of moving objects in different video samples is not consistent. Some of them look very small in a video frame while some occupy the whole frame. We propose a method of multi-scale receptive fields to process large or small moving objects. The structure of P3D is adopted, with dilated convolution being deployed in some related layers to change the receptive field.