# Multi-Scale Receptive Fields Convolutional Network for Action Recognition
###### Abstract  
Extracting good action representations from video frames is an intricate challenge due to the presence of moving objects of various sizes across current action recognition datasets. Most of the current action recognition methodologies have paid scant attention to this characteristic and have relied on deep learning models to automatically solve it. In this paper, we introduce a multi-scale receptive fields convolutional network (MSRFNet), which is fashioned after the pseudo-3D residual network architecture to mitigate the impact of scale variation in moving objects. The crux of MSRFNet is the integration of a multi-scale receptive fields block, which incorporates multiple dilated convolution layers that share identical convolutional parameters, but feature different receptive fields. MSRFNet leverages three scales of receptive fields to extract features from moving objects of diverse sizes, striving to produce scale-specific feature maps with a uniform representational power. Through visualization of the attention of MSRFNet, we analyze how the model re-allocates its attention to moving objects after implementing the multi-scale receptive fields approach. Experimental results on the benchmark dataset demonstrate that MSRFNet achieves improvement of 3.2% on UCFi01, improvement of 5.8% on HMDB51, and improvement of 7.7% on Kinetics-400 compared with the baseline. Compared with state-of-the-art techniques, MSRFNet gets comparable or superior results, thereby affirming the effectiveness of the proposed approach.  
action recognition; pseudo-3D residual networks; multi-scale receptive fields +
Footnote †: journal:  
two separate operations, a 2D spatial convolution (e.g., \(1\times 3\times 3\)) and a 1D temporal convolution (e.g., \(3\times 1\times 1\)). This operation simplifies 3D ConvNets and reduces the number of parameters. Other examples include R(2+1)D [17], TSM [18], S3D [19], etc.  
Motivated by methods that improve computational efficiency, we propose a network, called multi-scale receptive fields ConvNet (MSRFNet), consisting of pseudo-3D operations. The key to our method is a multi-scale receptive fields block, which uses a variety of receptive fields to extract features from input video frames. This block is designed to handle variously sized moving objects, which vary in size across different video samples. The input frames are processed with receptive fields of different sizes to account for this variation. MSRFNet improves upon the performance of existing methods due to the introduction of the multi-scale receptive fields block.  
The contributions in this work can be summarized as follows:  
* The multi-scale receptive fields block is utilized to handle variously sized moving objects, which can extract better action representations.
* The different branches in MSRFNBock shares common parameters to learn action features collaboratively. Meanwhile, it does not result in an increase in the number of parameters of MSRFNet.
* MSRFNet provides comparable or even better results than other leading methods on three benchmarks, UCF101 [12], HMDB51 [11], and Kinetics-400 [13].  
The rest of the paper is organised as follows. In Section 2, we discuss some related works for action recognition. In Section 3, a detailed explanation of MSRFNet is provided. Section 4 shows the experimental results with our analysis. Section 5 concludes the paper.