# Dilated Multi-Temporal Modeling for Action Recognition
## 1 Introduction  
Action recognition is a crucial task in the field of computer vision, as it allows for the automatic identification of actions and behaviors in video sequences. In this task, temporal modeling is used to capture the motion information in a video, which is essential for accurately identifying the actions being performed. Several methods have been proposed for temporal modeling, including two-stream networks [1], 3D CNNs [2; 3], and 13, 6934. The 13, 6934.  
structure by stacking feature maps. These methods, building multi-branch networks, constructing frame pyramids, and stacking feature maps, have employed costly strategies.  
In this paper, we propose a dilated multi-temporal (DMT) module offering a novel solution to modeling multi-temporal information without the need for multi-branch networks, input-level frame pyramids, or stacking feature maps, which can be integrated into the existing backbone architecture. As demonstrated in [11], dilated convolution [12] can effectively increase the receptive field size without adding additional parameters or computations. Thus, we argue that time-dilated convolutions can also extend the receptive field in the temporal dimension and the use of dilated CNNs can extract the long-range temporal features. Furthermore, this characteristics of dilation motivates us to adopt CNNs with different dilation rates to model different temporal information--small rates for shorter times and bigger rates for longer times. As depicted in the rightmost part of Figure 1, the DMT module utilizes dilated convolutions with different dilation rates in the different channels of the feature maps in order to capture information at multiple scales. Additionally, we employ a strategy with low computational overhead to efficiently merge temporal information at different scales. This enables our model to integrate information from multiple scales in a manner that preserves relevant details while minimizing computational requirements. This is particularly useful for applications where computational efficiency is a priority, such as real-time video action analysis or large-scale data processing.  
We evaluate the performance of the proposed DMT on the task of video-based action classification. The proposed approach leads to enhanced performance on several benchmark datasets, such as FineGym99 [13], Something-Something V1(5 thV1) [14], and Kinetics400 [15]. In addition, we conduct ablation experiments on FineGym99 to further validate the effectiveness of our approach. The dataset was chosen because it contains fine-grained action classes with substantial variance in their durations, making it an ideal candidate for evaluating the ability of models for multi-temporal modeling. The experiments demonstrate the ability of the DMT module to perform accurate action recognition. Overall, this work contributes to the field of action recognition by providing a practical solution for modeling multi-temporal information in action recognition.  
The main innovations of our work can be summarized as follows:  
* The DMT module allows for modeling of multi-temporal information without the need for multi-branch networks or input-level frame pyramids.
* In different channels, dilated convolutions with different rates enable modeling of different temporal scales.
* Partial channel modeling of temporal information enables the integration of information from multiple scales while minimizing extra computational requirements.  
Figure 1: (**a**) The original feature maps without temporal modeling. (**b**) Performing convolution along the temporal dimension using depthwise convolutions with kernels of stationary size. (**c**) Performing convolution along the temporal dimension utilizing different dilation rates across different sections of the feature map channels. This technique allows for an increased receptive field and for the ability to capture multiple temporal information.