TAda! Temporally-Adaptive Convolutions for Video Understanding

1 INTRODUCTION . Convolutions are an indispensable operation in modern deep vision models ( He et al. , 2016 ; Szegedy et al. , 2015 ; Krizhevsky et al. , 2012 ) , whose different variants have driven the state-of-the-art performances of convolutional neural networks ( CNNs ) in many visual tasks ( Xie et al. , 2017 ; Dai et al. , 2017 ; Zhou et al. , 2019 ) and application scenarios ( Howard et al. , 2017 ; Yang et al. , 2019 ) . In the video paradigm , compared to the 3D convolutions ( Tran et al. , 2015 ) , the combination of 2D spatial convolutions and 1D temporal convolutions are more widely preferred owing to its efficiency ( Wang et al. , 2016 ; Lin et al. , 2019a ; Liu et al. , 2021 ; Qiu et al. , 2017 ) . Nevertheless , 1D temporal convolutions still introduce non-negligible computation overhead on top of the spatial convolutions . Therefore , we seek to directly equip the spatial convolutions with temporal modelling abilities . One essential property of the convolutions is the translation invariance ( Ruderman & Bialek , 1994 ; Simoncelli & Olshausen , 2001 ) , resulted from its local connectivity and shared weights . However , recent works in dynamic filtering have shown that strictly shard weights for all pixels may be suboptimal for modelling various spatial contents ( Zhou et al. , 2021 ; Wu et al. , 2018 ) . Given the diverse nature of the temporal dynamics in videos , we hypothesize that the temporal modelling could also benefit from relaxed invariance along the temporal dimension . This means that the convolution weights for different time steps are no longer strictly shared . Existing dynamic filter networks can achive this , but with two drawbacks . First , it is difficult for most of them ( Zhou et al. , 2021 ; Yang et al. , 2019 ) to leverage the existing pre-trained models such as ResNet . This is critical in video applications , since training video models from scratch is highly resource demanding ( Feichtenhofer et al. , 2019 ; Feichtenhofer , 2020 ) and prone to over-fitting on small datasets . Second , for most dynamic filters , the weights are generated with respect to its spatial context ( Zhou et al. , 2021 ; Jia et al. , 2016 ) or the global descriptor ( Chen et al. , 2020a ; Yang et al. , 2019 ) , which is difficult to capture the temporal variations between frames . 1In this work , we use spatial convolutions and 2D convolutions interchangeably . In this work , we present the Temporally-Adaptive Convolution ( TAdaConv ) for video understanding , where the convolution kernel weights are no longer fixed across different frames . Specifically , the convolution kernel for the t-th frame Wt is factorized to the multiplication of the base weight and a calibration weight : Wt = αt ·Wb , where the calibration weight αt is adaptively generated from the input data for all channels in the base weight Wb . For each frame , we generate the calibration weight based on the frame descriptors of its adjacent time steps as well as the global descriptor , which effectively encodes the local and global temporal dynamics in videos . The difference between TAdaConv and the spatial convolutions is visualized in Fig . 1 . The main advantages of this factorization are three-fold : ( i ) TAdaConv can be easily plugged into any existing models to enhance temporal modelling , and their pre-trained weights can still be exploited ; ( ii ) the temporal modelling ability can be highly promoted with the help of the temporallyadaptive weight ; ( iii ) in comparison with temporal convolutions that often operate on the learned 2D feature maps , TAdaConv is more efficient by directly operating on the convolution kernels . TAdaConv is proposed as a drop-in replacement for the spatial convolutions in existing models . It can both serve as a stand-alone temporal modelling module for 2D networks , or be inserted into existing convolutional video models to further enhance the ability to model temporal dynamics . For efficiency , we construct TAda2D by replacing the spatial convolutions in ResNet ( He et al. , 2016 ) , which leads to on par or better performance than the state-of-the-arts . Used as an enhancement of existing video models , TAdaConv leads to notable improvements on multiple video datasets . The strong performance and the consistent improvements demonstrate that the temporally adaptive convolutions can be an important operation for modelling complex temporal dynamics in videos . 2 RELATED WORK . ConvNets for temporal modelling . A fundamental difference between videos and images lies in the temporal dimension , which makes temporal modeling an important research area for understanding videos . Recent deep CNNs for video understanding can be divided into two types . One models spatio-temporal information jointly by 3D convolutions ( Carreira & Zisserman , 2017 ; Tran et al. , 2015 ; Feichtenhofer , 2020 ; Tran et al. , 2019 ) . The other builds upon 2D networks , where most approaches employ 2D convolutions that share weights among all the frames for spatial modelling , and design additional operations for temporal modelling , such as temporal shift ( Lin et al. , 2019a ) , temporal difference ( Wang et al. , 2021 ; Jiang et al. , 2019 ) , temporal convolution ( Jiang et al. , 2019 ; Tran et al. , 2018 ; Liu et al. , 2021 ) and correlation operation ( Wang et al. , 2020 ) , etc . This work directly empowers the spatial convolutions with temporal modelling abilities , which can be further coupled with other temporal modelling operations for stronger video recognition performances . Dynamic networks . Dynamic networks refer to networks with content-adaptive weights or modules , such as dynamic filters/convolutions ( Jia et al. , 2016 ; Yang et al. , 2019 ; Li et al. , 2021 ) , dynamic activations ( Li et al. , 2020d ; Chen et al. , 2020b ) , and dynamic routing ( Wang et al. , 2018b ; Li et al. , 2020c ) , etc . Dynamic networks have demonstrated their exceeding network capacity and thus performance compared to the static ones . In video understanding , dynamic filter ( Liu et al. , 2021 ) or temporal aggregation ( Meng et al. , 2021 ) also demonstrate strong capability of temporal modelling . Recently , some convolutions applies spatially-adaptive weights ( Elsayed et al. , 2020 ; Chen et al. , 2021 ) , showing the benefit of relaxing the spatial invariance in modelling diverse visual contents . Hence , the proposed TAdaConv enhances temporal modelling by relaxing the invariance along the temporal dimension . TAdaConv has two key differences from previous works : ( i ) the convolution weight in TAdaConv is factorized into a base weight and a calibration weight , being able to exploit pre-trained weights ; ( ii ) the calibration weights are generated with respect to the temporal contexts . 3 TADACONV : TEMPORALLY-ADAPTIVE CONVOLUTIONS . In this work , we seek to empower the spatial convolutions with temporal modelling abilities . Inspired by the calibration process of temporal convolutions ( Sec . 3.1 ) , TAdaConv dynamically calibrates the convolution weights for each frame according to its temporal context ( Sec . 3.2 ) . 3.1 REVISITING TEMPORAL CONVOLUTIONS . We first revisit the temporal convolutions to showcase its underlying process and its relation to dynamic filters . We consider depth-wise temporal convolution for simplicity , which is more widely used because of its efficiency ( Liu et al. , 2021 ; Jiang et al. , 2019 ) . Formally , for a 3×1×1 temporal convolution filter parameterized by β = [ β1 , β2 , β3 ] and placed ( ignoring normalizations ) after the 2D convolution parameterized by W , the output feature x̃t of the t-th frame can be obtained by : x̃t = β1 · δ ( W ∗ xt−1 ) + β2 · δ ( W ∗ xt ) + β3 · δ ( W ∗ xt+1 ) , ( 1 ) where the · indicates the element-wise multiplication , ∗ denotes the convolution over the spatial dimension and δ denotes ReLU activation ( Nair & Hinton , 2010 ) . It can be rewritten as follows : x̃t = Wt−1 ∗ xt−1 +Wt ∗ xt +Wt+1 ∗ xt+1 , ( 2 ) where Wi , jt−1 = M i , j t−1 · β1 ·W , W i , j t = M i , j t · β2 ·W and W i , j t+1 = M i , j t+1 · β3 ·W are spatiotemporal location adaptive convolution weights . Mt ∈ RC×H×W is a dynamic tensor , with its value dependent on the result of the spatial convolutions ( see Appendix A for details ) . Hence , the temporal convolutions in the ( 2+1 ) D convolution essentially performs ( i ) weight calibration on the spatial convolutions and ( ii ) feature aggregation between adjacent frames . However , if the temporal modelling is achieved by coupling temporal convolutions to spatial convolutions , a non-negligible computation overhead is still introduced ( see Table 2 ) . 3.2 FORMULATION OF TADACONV . For efficiency , we set out to directly empower the spatial convolutions with temporal modelling abilities . Inspired by the recent finding that the relaxation of spatial invariance strengthens spatial modelling ( Zhou et al. , 2021 ; Elsayed et al. , 2020 ) , we hypothesize that temporally adaptive weights can also help temporal modelling . Therefore , the convolution weights in a TAdaConv layer are varied on a frame-by-frame basis . Since we observe that previous dynamic filters can hardly utilize the pretrained weights , we take inspiration from our observation in the temporal convolutions and factorize the weights for the t-th frame Wt into the multiplication of a base weight Wb shared for all frames , and a calibration weight αt that are different for each time step : x̃t = Wt ∗ xt = ( αt ·Wb ) ∗ xt , ( 3 ) Calibration weight generation . To allow for the TAdaConv to model temporal dynamics , it is crucial that the calibration weight αt for the t-th frame takes into account not only the current frame , but more importantly , its temporal context , i.e. , αt = G ( ... , xt−1 , xt , xt+1 , ... ) . Otherwise TAdaConv would degenerate to a set of unrelated spatial convolutions with different weights applied on different frames . We show an instantiation of the generation function G in Fig . 2 ( b ) . In our design , we aim for efficiency and the ability to capture inter-frame temporal dynamics . For efficiency , we operate on the frame description vectors v ∈ RT×C obtained by the global average pooling over the spatial dimension GAPs for each frame , i.e. , vt = GAPs ( xt ) . For temporal modelling , we apply stacked two-layer 1D convolutions F with a dimension reduction ratio of r on the local temporal context vadjt = { vt−1 , vt , vt+1 } obtained from x adj t = { xt−1 , xt , xt+1 } : F ( xadjt ) = Conv1DC/r→C ( δ ( BN ( Conv1DC→C/r ( v adj t ) ) ) ) . ( 4 ) where δ and BN denote the ReLU ( Nair & Hinton , 2010 ) and batchnorm ( Ioffe & Szegedy , 2015 ) . In order for a larger inter-frame field of view in complement to the local 1D convolution , we further incorporate global temporal information by adding a global descriptor g to the weight generation process F through a linear mapping function FC : F ( xadjt , g ) = Conv1DC/r→C ( δ ( BN ( Conv1DC→C/r ( v adj t ) + FC C→C/r ( g ) ) ) , ( 5 ) where g = GAPst ( x ) with GAPst being global average pooling on spatial and temporal dimensions . Initialization . The TAdaConv is designed to be readily inserted into existing models by simply replacing the 2D convolutions . For an effective use of the pre-trained weights , TAdaConv is initialized to behave exactly the same as the standard convolution . This is achieved by zero-initializing the weight of the last convolution in F and adding a constant vector 1 to the formulation : αt = G ( x ) = 1+ F ( xadjt , g ) . ( 6 ) In this way , at initial state , Wt = 1 ·Wb = Wb , where we load Wb with the pre-trained weights . Calibration dimension . The base weight Wb ∈ RCo×Ci×k 2 can be calibrated in different dimensions . We instantiate the calibration on the Cin dimension ( αt ∈ R1×Cin×1 ) , as the weight generation based on the input features yields a more precise estimation for the relation of the input channels than the output channels or spatial structures ( empirical analysis in Table 7 ) . Comparison with other dynamic filters . Table 1 compares TAdaConv with existing dynamic filters . Mixtue-of-experts based dynamic filters such as CondConv dynamically aggregates multiple kernels to generate the weights that are shared for all locations . The weights in most other dynamic filters are completely generated from the input , such as DynamicFilter ( Jia et al. , 2016 ) and DDF ( Zhou et al. , 2021 ) in images and TAM ( Liu et al. , 2021 ) in videos . Compared to image based ones , TAdaConv achieves temporal modelling by generating weights from the local and global temporal context . Compared to TANet ( Liu et al. , 2021 ) , TAdaConv is better at temopral modellling because of temporally adaptive weights . Further , TAdaConv can effectively generate weights identical to the pre-trained ones , while it is difficult for previous approaches to exploit pre-trained models . More detailed comparisons of dynamic filters are included in Appendix J . Comparison with temporal convolutions . Table 2 compares the TAdaConv with R ( 2+1 ) D in parameters and FLOPs , which shows most of our additional computation overhead on top of the spatial convolution is an order of magnitude less than the temporal convolution . For detailed computation analysis and comparison with other temporal modelling approaches , please refer to Appendix B .

Summary: This work seeks to improve performance of video understanding models though the use of spatial convolutions which are dynamically adapted in time when applied to a sequence of frames. The novelty of this work lies in the fact that the weights of the spatial convolutions are adapted, rather than applying a temporal excitation to feature maps to achieve the same purpose. To achieve this, a drop-in replacement for spatial convolutions is designed which uses pooled local and global frame descriptors to produce the temporal calibration weights. Furthermore, this operation is used as part of a temporal feature aggregation block that is used as part of 2D CNN architecture. The benefits of the approach include that it can make effective use of pre-trained weights and that it can improve model capacity for a marginal increase in computation. Experimentally, it is show large performance improvements over a baseline and match state of the art on significant datasets.