TAM: Temporal Adaptive Module for Video Recognition

1 INTRODUCTION . Deep learning has brought great progress for various recognition tasks in image domain , such as image classification ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) , object detection ( Ren et al. , 2017 ) , and instance segmentation ( He et al. , 2017 ) . The key to these successes is to devise flexible and efficient architectures that are capable of learning powerful visual representations from large-scale image datasets ( Deng et al. , 2009 ) . However , deep learning research progress in video understanding is relatively more slowly , partially due to the high complexity of video data . The core technical problem in video understanding is to design an effective temporal module , that is expected to be able to capture complex temporal structure with high flexibility , while yet to be of low computational consumption for processing high dimensional video data efficiently . 3D Convolutional Neural Networks ( 3D CNNs ) ( Ji et al. , 2010 ; Tran et al. , 2015 ) have turned out to be mainstream architectures for video modeling ( Carreira & Zisserman , 2017 ; Feichtenhofer et al. , 2019 ; Tran et al. , 2018 ; Qiu et al. , 2017 ) . The 3D convolution is a natural extension over its 2D counterparts and provides a learnable operator for video recognition . However , this simple extension lacks specific consideration about the temporal properties in video data and might as well lead to high computational cost . Therefore , recent methods aim to improve 3D CNNs from two different aspects by combining a lightweight temporal module with 2D CNNs to improve efficiency ( e.g. , TSN ( Wang et al. , 2016 ) , TSM ( Lin et al. , 2019 ) ) , or designing a dedicated temporal module to better capture temporal relation ( e.g. , Nonlocal Net ( Wang et al. , 2018b ) , ARTNet ( Wang et al. , 2018a ) , STM ( Jiang et al. , 2019 ) ) . However , how to devise a temporal module with high efficiency and strong flexibility still remains to be an unsolved problem in video recognition . Consequently , we aim at advancing the current video architectures along this direction . In this paper , we focus on devising a principled adaptive module to capture temporal information in a more flexible way . In general , we observe that video data is with extremely complex dynamics along the temporal dimension due to factors such as camera motion and various speed . Thus 3D convolutions ( temporal convolutions ) might lack enough representation power to describe motion diversity by simply employing a fixed number of video invariant kernels . To deal with such complex temporal variations in videos , we argue that adaptive temporal kernels for each video are effective and as well necessary to describe motion patterns . To this end , as shown in Figure 1 , we present a two-level adaptive modeling scheme to decompose this video specific temporal kernel into a location sensitive importance map and a location invariant ( also video adaptive ) aggregation kernel . This unique design allows the location sensitive importance map to focus on enhancing discriminative temporal information from a local view , and enables the location invariant aggregation weights to capture temporal dependencies guided by a global view of the input video sequence . Specifically , the design of temporal adaptive module ( TAM ) strictly follows two principles : high efficiency and strong flexibility . To ensure our TAM with a low computational cost , we first squeeze the feature map by employing a global spatial pooling , and then establish our TAM in a channelwise manner to keep the efficiency . Our TAM is composed of two branches : a local branch ( L ) and a global branch ( G ) . As shown in Fig . 2 , TAM is implemented in an efficient way . The local branch employs temporal convolutions to produce the location sensitive importance maps to discriminate the local feature , while the global branch uses fully connected layers to produce the location invariant kernel for temporal aggregation . The importance map generated by a local temporal window focuses on short-term motion modeling and the aggregation kernel using a global view pays more attention to the long-term temporal information . Furthermore , our TAM could be flexibly plugged into the existing 2D CNNs to yield an efficient video recognition architecture , termed as TANet . We validate the proposed TANet on the task of action classification in video recognition . Particularly , we first study the performance of the TANet on the Kinetics-400 dataset . We demonstrate that our TAM is better at capturing temporal information than other several counterparts , such as temporal pooling , temporal convolution , TSM ( Lin et al. , 2019 ) , and Non-local block ( Wang et al. , 2018b ) . Our TANet is able to yield a very competitive accuracy with the FLOPs similar to 2D CNNs . We further test our TANet on the motion dominated dataset of Something-Something , where the state-of-the-art performance is also achieved . 2 RELATED WORKS . Video understanding is a core topic in the field of computer vision . At early stage , a lot of traditional methods ( Le et al. , 2011 ; Kläser et al. , 2008 ; Sadanand & Corso , 2012 ; Willems et al. , 2008 ) have designed various hand-crafted features to encode the video data , but these methods are too inflexible when generalized to other video tasks . Recently , since the rapid development of video understanding has been much benefited from deep learning methods ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ) , especially in video recognition , a series of CNNs-based methods were proposed to learn spatiotemporal representation , and the differences with our method will be clarified later . Furthermore , our work also relates to dynamic convolution and attention in CNNs . CNNs-based Methods for Action Recognition . Since the deep learning method has been wildly used in the image tasks , there are many attempts ( Karpathy et al. , 2014 ; Simonyan & Zisserman , 2014 ; Wang et al. , 2016 ; Zhou et al. , 2018 ; He et al. , 2019 ; Lin et al. , 2019 ) based on 2D CNNs devoted to modeling the video clips . In particular , Wang et al . ( 2016 ) used the frames sparsely sampled from the whole video to learn the long-range information by aggregating scores after the last fully-connected layer . Lin et al . ( 2019 ) shifted the channels along the temporal dimension in an efficient way , which yields a good performance with 2D CNNs . By a simple extension from spatial domain to spatiotemporal domain , 3D convolution ( Ji et al. , 2010 ; Tran et al. , 2015 ) was proposed to capture the motion information encoded in video clips . Due to the release of large-scale Kinetics dataset ( Kay et al. , 2017 ) , 3D CNNs ( Carreira & Zisserman , 2017 ) were wildly used in action recognition . Its variants ( Qiu et al. , 2017 ; Tran et al. , 2018 ; Xie et al. , 2018 ) decomposed the 3D convolution into a spatial 2D convolution and a temporal 1D convolution to learn the spatiotemporal features . And Feichtenhofer et al . ( 2019 ) designed a network with dual paths to learn the spatiotemporal features and achieved a promising accuracy in video understanding . The methods aforementioned all share a common insight that they are video invariant and ignore the inherent temporal diversities in videos . As opposed to these methods , we design a two-level adaptive modeling scheme by decomposing the video specific operation into a location sensitive excitation and a location invariant convolution with adaptive kernel for each video clip . Attention in Action Recognition . The local branch in TAM mostly relates to SENet ( Hu et al. , 2018 ) . But the SENet learned modulation weights for each channel of feature maps . Several methods ( Liu et al. , 2019b ; Diba et al. , 2018 ) also resorted to the attention to learn more discriminative features in videos . Different from these methods , the local branch keeps the temporal information to learn the location sensitive importances . Wang et al . ( 2018b ) designed a non-local block which can be seen as self-attention to capture long-range dependencies . Our TANet captures the long-range dependencies by simply stacking more TAM , and keep the efficiency of networks . Dynamic Convolutions . Jia et al . ( 2016 ) first proposed the dynamic filters on the tasks of video and stereo prediction , and designed a convolutional encoder-decoder as filter-generating network . Several works ( Yang et al. , 2019 ; Chen et al. , 2020 ) in image tasks attempted to generate aggregation weights for a set of convolutional kernels , and then produce a dynamic kernel . Our motivation are different from these methods . We aim to use this temporal adaptive module to deal with temporal variations in videos . Specifically , we design an efficient form to implement this temporal dynamic kernel based on input feature maps , which is critical for understanding the video content . 3 METHOD . 3.1 THE OVERVIEW OF TEMPORAL ADAPTIVE MODULE . As we discussed in Sec.1 , video data typically exhibit the complex temporal dynamics caused by many factors such as camera motion and speed variations . Therefore , we aim to tackle this issue by introducing a temporal adaptive module ( TAM ) with video specific kernels , unlike the sharing convolutional kernel in 3D CNNs . our TAM could be easily integrated into the existing 2D CNNs ( e.g. , ResNet ) to yield a video network architecture , as shown in Figure 2 . We will give an overview of TAM and then describe its technical details . Formally , let X ∈ RC×T×H×W denote the feature maps for a video clip , where C represents the number of channels , and T , H , W are its spatiotemporal dimensions . For efficiency , our TAM only focus on temporal modeling and the spatial pattern is expected to captured by 2D convolutions . Therefore , we first employ a global spatial average pooling to squeeze the feature map as follows : X̂c , t = φ ( X ) c , t = 1 H ×W ∑ i , j Xc , t , j , i , ( 1 ) where c , t , j , i is the index of different dimensions ( in channel , time , height and width ) , and X̂ ∈ RC×T aggregates the spatial information of X . For simplicity , we here use φ to denote the function . that aggregates the spatial information . Our proposed temporal adaptive module is established based on this squeezed 1D temporal signal for a high efficiency . Our TAM is composed of two branches : a local branch L and a global branch G , which aims to learn a location sensitive importance map to enhance discriminative features and then produces the location invariant weights to adaptively aggregate temporal information in a convolutional manner . More specifically , the TAM is formulated as follows : Y = G ( X ) ⊗ ( L ( X ) X ) , ( 2 ) where ⊗ denotes convolutional operator and denotes element-wise multiplication . It is worth noting that these two branches focus on different aspects of temporal information , where the local branch tries to capture the short term information to attend important features by using a temporal convolution , while the global branch aims to incorporate long-range temporal structure to guide adaptive temporal aggregation with fully connected layers . Disentangling kernel learning procedures into local and global branches turns out to be an effective way in experiments . The two branches will be introduced in the following sections .

Summary: This paper presents a new temporal adaptive module (TAM) to generate video-specific temporal kernels based on its own feature maps. TAM proposes a unique two-level adaptive modeling scheme by decoupling dynamic kernel into a location sensitive importance map and a location invariant aggregation weight. The importance map is learned in a local temporal window to capture short term information, while the aggregation weight is generated from a global view with a focus on long-term structure.