TAda! Temporally-Adaptive Convolutions for Video Understanding

1 INTRODUCTION . Convolutions are an indispensable operation in modern deep vision models ( He et al. , 2016 ; Szegedy et al. , 2015 ; Krizhevsky et al. , 2012 ) , whose different variants have driven the state-of-the-art performances of convolutional neural networks ( CNNs ) in many visual tasks ( Xie et al. , 2017 ; Dai et al. , 2017 ; Zhou et al. , 2019 ) and application scenarios ( Howard et al. , 2017 ; Yang et al. , 2019 ) . In the video paradigm , compared to the 3D convolutions ( Tran et al. , 2015 ) , the combination of 2D spatial convolutions and 1D temporal convolutions are more widely preferred owing to its efficiency ( Wang et al. , 2016 ; Lin et al. , 2019a ; Liu et al. , 2021 ; Qiu et al. , 2017 ) . Nevertheless , 1D temporal convolutions still introduce non-negligible computation overhead on top of the spatial convolutions . Therefore , we seek to directly equip the spatial convolutions with temporal modelling abilities . One essential property of the convolutions is the translation invariance ( Ruderman & Bialek , 1994 ; Simoncelli & Olshausen , 2001 ) , resulted from its local connectivity and shared weights . However , recent works in dynamic filtering have shown that strictly shard weights for all pixels may be suboptimal for modelling various spatial contents ( Zhou et al. , 2021 ; Wu et al. , 2018 ) . Given the diverse nature of the temporal dynamics in videos , we hypothesize that the temporal modelling could also benefit from relaxed invariance along the temporal dimension . This means that the convolution weights for different time steps are no longer strictly shared . Existing dynamic filter networks can achive this , but with two drawbacks . First , it is difficult for most of them ( Zhou et al. , 2021 ; Yang et al. , 2019 ) to leverage the existing pre-trained models such as ResNet . This is critical in video applications , since training video models from scratch is highly resource demanding ( Feichtenhofer et al. , 2019 ; Feichtenhofer , 2020 ) and prone to over-fitting on small datasets . Second , for most dynamic filters , the weights are generated with respect to its spatial context ( Zhou et al. , 2021 ; Jia et al. , 2016 ) or the global descriptor ( Chen et al. , 2020a ; Yang et al. , 2019 ) , which is difficult to capture the temporal variations between frames . 1In this work , we use spatial convolutions and 2D convolutions interchangeably . In this work , we present the Temporally-Adaptive Convolution ( TAdaConv ) for video understanding , where the convolution kernel weights are no longer fixed across different frames . Specifically , the convolution kernel for the t-th frame Wt is factorized to the multiplication of the base weight and a calibration weight : Wt = αt ·Wb , where the calibration weight αt is adaptively generated from the input data for all channels in the base weight Wb . For each frame , we generate the calibration weight based on the frame descriptors of its adjacent time steps as well as the global descriptor , which effectively encodes the local and global temporal dynamics in videos . The difference between TAdaConv and the spatial convolutions is visualized in Fig . 1 . The main advantages of this factorization are three-fold : ( i ) TAdaConv can be easily plugged into any existing models to enhance temporal modelling , and their pre-trained weights can still be exploited ; ( ii ) the temporal modelling ability can be highly promoted with the help of the temporallyadaptive weight ; ( iii ) in comparison with temporal convolutions that often operate on the learned 2D feature maps , TAdaConv is more efficient by directly operating on the convolution kernels . TAdaConv is proposed as a drop-in replacement for the spatial convolutions in existing models . It can both serve as a stand-alone temporal modelling module for 2D networks , or be inserted into existing convolutional video models to further enhance the ability to model temporal dynamics . For efficiency , we construct TAda2D by replacing the spatial convolutions in ResNet ( He et al. , 2016 ) , which leads to on par or better performance than the state-of-the-arts . Used as an enhancement of existing video models , TAdaConv leads to notable improvements on multiple video datasets . The strong performance and the consistent improvements demonstrate that the temporally adaptive convolutions can be an important operation for modelling complex temporal dynamics in videos . 2 RELATED WORK . ConvNets for temporal modelling . A fundamental difference between videos and images lies in the temporal dimension , which makes temporal modeling an important research area for understanding videos . Recent deep CNNs for video understanding can be divided into two types . One models spatio-temporal information jointly by 3D convolutions ( Carreira & Zisserman , 2017 ; Tran et al. , 2015 ; Feichtenhofer , 2020 ; Tran et al. , 2019 ) . The other builds upon 2D networks , where most approaches employ 2D convolutions that share weights among all the frames for spatial modelling , and design additional operations for temporal modelling , such as temporal shift ( Lin et al. , 2019a ) , temporal difference ( Wang et al. , 2021 ; Jiang et al. , 2019 ) , temporal convolution ( Jiang et al. , 2019 ; Tran et al. , 2018 ; Liu et al. , 2021 ) and correlation operation ( Wang et al. , 2020 ) , etc . This work directly empowers the spatial convolutions with temporal modelling abilities , which can be further coupled with other temporal modelling operations for stronger video recognition performances . Dynamic networks . Dynamic networks refer to networks with content-adaptive weights or modules , such as dynamic filters/convolutions ( Jia et al. , 2016 ; Yang et al. , 2019 ; Li et al. , 2021 ) , dynamic activations ( Li et al. , 2020d ; Chen et al. , 2020b ) , and dynamic routing ( Wang et al. , 2018b ; Li et al. , 2020c ) , etc . Dynamic networks have demonstrated their exceeding network capacity and thus performance compared to the static ones . In video understanding , dynamic filter ( Liu et al. , 2021 ) or temporal aggregation ( Meng et al. , 2021 ) also demonstrate strong capability of temporal modelling . Recently , some convolutions applies spatially-adaptive weights ( Elsayed et al. , 2020 ; Chen et al. , 2021 ) , showing the benefit of relaxing the spatial invariance in modelling diverse visual contents . Hence , the proposed TAdaConv enhances temporal modelling by relaxing the invariance along the temporal dimension . TAdaConv has two key differences from previous works : ( i ) the convolution weight in TAdaConv is factorized into a base weight and a calibration weight , being able to exploit pre-trained weights ; ( ii ) the calibration weights are generated with respect to the temporal contexts . 3 TADACONV : TEMPORALLY-ADAPTIVE CONVOLUTIONS . In this work , we seek to empower the spatial convolutions with temporal modelling abilities . Inspired by the calibration process of temporal convolutions ( Sec . 3.1 ) , TAdaConv dynamically calibrates the convolution weights for each frame according to its temporal context ( Sec . 3.2 ) . 3.1 REVISITING TEMPORAL CONVOLUTIONS . We first revisit the temporal convolutions to showcase its underlying process and its relation to dynamic filters . We consider depth-wise temporal convolution for simplicity , which is more widely used because of its efficiency ( Liu et al. , 2021 ; Jiang et al. , 2019 ) . Formally , for a 3×1×1 temporal convolution filter parameterized by β = [ β1 , β2 , β3 ] and placed ( ignoring normalizations ) after the 2D convolution parameterized by W , the output feature x̃t of the t-th frame can be obtained by : x̃t = β1 · δ ( W ∗ xt−1 ) + β2 · δ ( W ∗ xt ) + β3 · δ ( W ∗ xt+1 ) , ( 1 ) where the · indicates the element-wise multiplication , ∗ denotes the convolution over the spatial dimension and δ denotes ReLU activation ( Nair & Hinton , 2010 ) . It can be rewritten as follows : x̃t = Wt−1 ∗ xt−1 +Wt ∗ xt +Wt+1 ∗ xt+1 , ( 2 ) where Wi , jt−1 = M i , j t−1 · β1 ·W , W i , j t = M i , j t · β2 ·W and W i , j t+1 = M i , j t+1 · β3 ·W are spatiotemporal location adaptive convolution weights . Mt ∈ RC×H×W is a dynamic tensor , with its value dependent on the result of the spatial convolutions ( see Appendix A for details ) . Hence , the temporal convolutions in the ( 2+1 ) D convolution essentially performs ( i ) weight calibration on the spatial convolutions and ( ii ) feature aggregation between adjacent frames . However , if the temporal modelling is achieved by coupling temporal convolutions to spatial convolutions , a non-negligible computation overhead is still introduced ( see Table 2 ) . 3.2 FORMULATION OF TADACONV . For efficiency , we set out to directly empower the spatial convolutions with temporal modelling abilities . Inspired by the recent finding that the relaxation of spatial invariance strengthens spatial modelling ( Zhou et al. , 2021 ; Elsayed et al. , 2020 ) , we hypothesize that temporally adaptive weights can also help temporal modelling . Therefore , the convolution weights in a TAdaConv layer are varied on a frame-by-frame basis . Since we observe that previous dynamic filters can hardly utilize the pretrained weights , we take inspiration from our observation in the temporal convolutions and factorize the weights for the t-th frame Wt into the multiplication of a base weight Wb shared for all frames , and a calibration weight αt that are different for each time step : x̃t = Wt ∗ xt = ( αt ·Wb ) ∗ xt , ( 3 ) Calibration weight generation . To allow for the TAdaConv to model temporal dynamics , it is crucial that the calibration weight αt for the t-th frame takes into account not only the current frame , but more importantly , its temporal context , i.e. , αt = G ( ... , xt−1 , xt , xt+1 , ... ) . Otherwise TAdaConv would degenerate to a set of unrelated spatial convolutions with different weights applied on different frames . We show an instantiation of the generation function G in Fig . 2 ( b ) . In our design , we aim for efficiency and the ability to capture inter-frame temporal dynamics . For efficiency , we operate on the frame description vectors v ∈ RT×C obtained by the global average pooling over the spatial dimension GAPs for each frame , i.e. , vt = GAPs ( xt ) . For temporal modelling , we apply stacked two-layer 1D convolutions F with a dimension reduction ratio of r on the local temporal context vadjt = { vt−1 , vt , vt+1 } obtained from x adj t = { xt−1 , xt , xt+1 } : F ( xadjt ) = Conv1DC/r→C ( δ ( BN ( Conv1DC→C/r ( v adj t ) ) ) ) . ( 4 ) where δ and BN denote the ReLU ( Nair & Hinton , 2010 ) and batchnorm ( Ioffe & Szegedy , 2015 ) . In order for a larger inter-frame field of view in complement to the local 1D convolution , we further incorporate global temporal information by adding a global descriptor g to the weight generation process F through a linear mapping function FC : F ( xadjt , g ) = Conv1DC/r→C ( δ ( BN ( Conv1DC→C/r ( v adj t ) + FC C→C/r ( g ) ) ) , ( 5 ) where g = GAPst ( x ) with GAPst being global average pooling on spatial and temporal dimensions . Initialization . The TAdaConv is designed to be readily inserted into existing models by simply replacing the 2D convolutions . For an effective use of the pre-trained weights , TAdaConv is initialized to behave exactly the same as the standard convolution . This is achieved by zero-initializing the weight of the last convolution in F and adding a constant vector 1 to the formulation : αt = G ( x ) = 1+ F ( xadjt , g ) . ( 6 ) In this way , at initial state , Wt = 1 ·Wb = Wb , where we load Wb with the pre-trained weights . Calibration dimension . The base weight Wb ∈ RCo×Ci×k 2 can be calibrated in different dimensions . We instantiate the calibration on the Cin dimension ( αt ∈ R1×Cin×1 ) , as the weight generation based on the input features yields a more precise estimation for the relation of the input channels than the output channels or spatial structures ( empirical analysis in Table 7 ) . Comparison with other dynamic filters . Table 1 compares TAdaConv with existing dynamic filters . Mixtue-of-experts based dynamic filters such as CondConv dynamically aggregates multiple kernels to generate the weights that are shared for all locations . The weights in most other dynamic filters are completely generated from the input , such as DynamicFilter ( Jia et al. , 2016 ) and DDF ( Zhou et al. , 2021 ) in images and TAM ( Liu et al. , 2021 ) in videos . Compared to image based ones , TAdaConv achieves temporal modelling by generating weights from the local and global temporal context . Compared to TANet ( Liu et al. , 2021 ) , TAdaConv is better at temopral modellling because of temporally adaptive weights . Further , TAdaConv can effectively generate weights identical to the pre-trained ones , while it is difficult for previous approaches to exploit pre-trained models . More detailed comparisons of dynamic filters are included in Appendix J . Comparison with temporal convolutions . Table 2 compares the TAdaConv with R ( 2+1 ) D in parameters and FLOPs , which shows most of our additional computation overhead on top of the spatial convolution is an order of magnitude less than the temporal convolution . For detailed computation analysis and comparison with other temporal modelling approaches , please refer to Appendix B .

Summary: This paper proposes TAdaConv that calibrates kernel weights of convolutional layers adaptively according to the temporal dynamics of the input tensor. It is designed to incorporate both the local and global temporal context by using stacked two-layer 1D convolutional operations and global average pooling. Since it works exactly the same as the original convolutional layer at the initial stage, it is easy to insert TAdaConv into existing ConvNet architectures. On top of this, the authors construct TAda2D networks by introducing the temporal feature aggregation module that is based on a strided temporal average pooling. Many experiments performed on video/action classification and localization demonstrate the effectiveness of the proposed module.